2024-04-30T08:09:09.201 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-30T08:09:09.201 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_pdonnell@teuthology --name pdonnell-2024-04-30_05:04:19-fs-wip-pdonnell-testing-20240429.210911-debug-distro-default-smithi --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} --verbose --archive /home/teuthworker/archive/pdonnell-2024-04-30_05:04:19-fs-wip-pdonnell-testing-20240429.210911-debug-distro-default-smithi/7680601 -- /home/teuthworker/archive/pdonnell-2024-04-30_05:04:19-fs-wip-pdonnell-testing-20240429.210911-debug-distro-default-smithi/7680601/orig.config.yaml 2024-04-30T08:09:09.256 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-30T08:09:09.348 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-04-30_05:04:19-fs-wip-pdonnell-testing-20240429.210911-debug-distro-default-smithi/7680601 branch: wip-pdonnell-testing-20240429.210911-debug description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} email: pdonnell@redhat.com first_in_suite: false job_id: '7680601' kernel: client: branch: testing kdb: true sha1: distro ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: pdonnell-2024-04-30_05:04:19-fs-wip-pdonnell-testing-20240429.210911-debug-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-pdonnell-testing-20240429.210911-debug ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 filestore flush min: 0 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd heartbeat grace: 60 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request - SLOW_OPS - slow request sha1: 8feaf7d9864e980041b042fd5d964ac18adfd20f subvols: create: 2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.dir_split install: ceph: flavor: default sha1: 8feaf7d9864e980041b042fd5d964ac18adfd20f kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - nowsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20240429.210911-debug sha1: 8feaf7d9864e980041b042fd5d964ac18adfd20f owner: scheduled_pdonnell@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 4382 sha1: 8feaf7d9864e980041b042fd5d964ac18adfd20f sleep_before_teardown: 0 subset: 105/512 suite: fs suite_branch: wip-pdonnell-testing-20240429.210911-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_8feaf7d9864e980041b042fd5d964ac18adfd20f/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 8feaf7d9864e980041b042fd5d964ac18adfd20f syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi032.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEU3Y/7URMK0Jo+Za/Cngq+21csc7ZJUzb/JDPnzUl01zYOQiFmUXAflXU88IPq7tql29/6zYUAEKA4YzeYGahY= smithi071.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCwOv6xI7VHUgInYJlYquG4or26SYiH/PW8rDSDC7Cm9qaMTgr+P8l1M3oepRFSVIBhWr07qy/paGUqt1CWv3UQ= smithi189.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCnlkcjxTxuYmuUCdpJxn4UaVQC4NPv7vfYg3VsGfGtOr72bwi2gmaUSpofdjWuo0ynIGlYWKVhoNs+6brFD0Os= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/ffsb.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-30_05:04:19 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-30T08:09:09.348 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_8feaf7d9864e980041b042fd5d964ac18adfd20f/qa; will attempt to use it 2024-04-30T08:09:09.349 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_8feaf7d9864e980041b042fd5d964ac18adfd20f/qa/tasks 2024-04-30T08:09:09.349 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-30T08:09:09.351 INFO:teuthology.task.internal:Checking packages... 2024-04-30T08:09:09.377 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '8feaf7d9864e980041b042fd5d964ac18adfd20f' 2024-04-30T08:09:09.377 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-30T08:09:09.377 INFO:teuthology.packaging:ref: None 2024-04-30T08:09:09.377 INFO:teuthology.packaging:tag: None 2024-04-30T08:09:09.377 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20240429.210911-debug 2024-04-30T08:09:09.377 INFO:teuthology.packaging:sha1: 8feaf7d9864e980041b042fd5d964ac18adfd20f 2024-04-30T08:09:09.377 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-pdonnell-testing-20240429.210911-debug 2024-04-30T08:09:09.524 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3348.g8feaf7d9 2024-04-30T08:09:09.526 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-30T08:09:09.532 INFO:teuthology.task.internal:no buildpackages task found 2024-04-30T08:09:09.532 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-30T08:09:09.554 INFO:teuthology.task.internal:Saving configuration 2024-04-30T08:09:09.574 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-30T08:09:09.641 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-30T08:09:09.666 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi032.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-30_05:04:19-fs-wip-pdonnell-testing-20240429.210911-debug-distro-default-smithi/7680601', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-30 08:03:51.713627', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEU3Y/7URMK0Jo+Za/Cngq+21csc7ZJUzb/JDPnzUl01zYOQiFmUXAflXU88IPq7tql29/6zYUAEKA4YzeYGahY='} 2024-04-30T08:09:09.691 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi071.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-30_05:04:19-fs-wip-pdonnell-testing-20240429.210911-debug-distro-default-smithi/7680601', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-30 08:03:51.712458', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCwOv6xI7VHUgInYJlYquG4or26SYiH/PW8rDSDC7Cm9qaMTgr+P8l1M3oepRFSVIBhWr07qy/paGUqt1CWv3UQ='} 2024-04-30T08:09:09.713 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi189.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-30_05:04:19-fs-wip-pdonnell-testing-20240429.210911-debug-distro-default-smithi/7680601', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-30 08:03:51.710685', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCnlkcjxTxuYmuUCdpJxn4UaVQC4NPv7vfYg3VsGfGtOr72bwi2gmaUSpofdjWuo0ynIGlYWKVhoNs+6brFD0Os='} 2024-04-30T08:09:09.713 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-30T08:09:09.768 INFO:teuthology.task.internal:roles: ubuntu@smithi032.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-30T08:09:09.768 INFO:teuthology.task.internal:roles: ubuntu@smithi071.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-30T08:09:09.768 INFO:teuthology.task.internal:roles: ubuntu@smithi189.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-30T08:09:09.768 INFO:teuthology.run_tasks:Running task console_log... 2024-04-30T08:09:09.905 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f74af87f0d0>, signals=[15]) 2024-04-30T08:09:09.905 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-30T08:09:09.911 INFO:teuthology.task.internal:Opening connections... 2024-04-30T08:09:09.911 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi032.front.sepia.ceph.com 2024-04-30T08:09:09.913 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-30T08:09:09.992 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi071.front.sepia.ceph.com 2024-04-30T08:09:09.993 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-30T08:09:10.065 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi189.front.sepia.ceph.com 2024-04-30T08:09:10.066 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-30T08:09:10.141 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-30T08:09:10.147 DEBUG:teuthology.orchestra.run.smithi032:> uname -m 2024-04-30T08:09:10.165 INFO:teuthology.orchestra.run.smithi032.stdout:x86_64 2024-04-30T08:09:10.166 DEBUG:teuthology.orchestra.run.smithi032:> cat /etc/os-release 2024-04-30T08:09:10.227 INFO:teuthology.orchestra.run.smithi032.stdout:NAME="CentOS Stream" 2024-04-30T08:09:10.227 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION="9" 2024-04-30T08:09:10.228 INFO:teuthology.orchestra.run.smithi032.stdout:ID="centos" 2024-04-30T08:09:10.228 INFO:teuthology.orchestra.run.smithi032.stdout:ID_LIKE="rhel fedora" 2024-04-30T08:09:10.228 INFO:teuthology.orchestra.run.smithi032.stdout:VERSION_ID="9" 2024-04-30T08:09:10.228 INFO:teuthology.orchestra.run.smithi032.stdout:PLATFORM_ID="platform:el9" 2024-04-30T08:09:10.228 INFO:teuthology.orchestra.run.smithi032.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-30T08:09:10.228 INFO:teuthology.orchestra.run.smithi032.stdout:ANSI_COLOR="0;31" 2024-04-30T08:09:10.228 INFO:teuthology.orchestra.run.smithi032.stdout:LOGO="fedora-logo-icon" 2024-04-30T08:09:10.229 INFO:teuthology.orchestra.run.smithi032.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-30T08:09:10.229 INFO:teuthology.orchestra.run.smithi032.stdout:HOME_URL="https://centos.org/" 2024-04-30T08:09:10.229 INFO:teuthology.orchestra.run.smithi032.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-30T08:09:10.229 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-30T08:09:10.229 INFO:teuthology.orchestra.run.smithi032.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-30T08:09:10.230 INFO:teuthology.lock.ops:Updating smithi032.front.sepia.ceph.com on lock server 2024-04-30T08:09:10.266 DEBUG:teuthology.orchestra.run.smithi071:> uname -m 2024-04-30T08:09:10.296 INFO:teuthology.orchestra.run.smithi071.stdout:x86_64 2024-04-30T08:09:10.296 DEBUG:teuthology.orchestra.run.smithi071:> cat /etc/os-release 2024-04-30T08:09:10.357 INFO:teuthology.orchestra.run.smithi071.stdout:NAME="CentOS Stream" 2024-04-30T08:09:10.357 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION="9" 2024-04-30T08:09:10.357 INFO:teuthology.orchestra.run.smithi071.stdout:ID="centos" 2024-04-30T08:09:10.357 INFO:teuthology.orchestra.run.smithi071.stdout:ID_LIKE="rhel fedora" 2024-04-30T08:09:10.357 INFO:teuthology.orchestra.run.smithi071.stdout:VERSION_ID="9" 2024-04-30T08:09:10.357 INFO:teuthology.orchestra.run.smithi071.stdout:PLATFORM_ID="platform:el9" 2024-04-30T08:09:10.357 INFO:teuthology.orchestra.run.smithi071.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-30T08:09:10.358 INFO:teuthology.orchestra.run.smithi071.stdout:ANSI_COLOR="0;31" 2024-04-30T08:09:10.358 INFO:teuthology.orchestra.run.smithi071.stdout:LOGO="fedora-logo-icon" 2024-04-30T08:09:10.358 INFO:teuthology.orchestra.run.smithi071.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-30T08:09:10.358 INFO:teuthology.orchestra.run.smithi071.stdout:HOME_URL="https://centos.org/" 2024-04-30T08:09:10.358 INFO:teuthology.orchestra.run.smithi071.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-30T08:09:10.358 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-30T08:09:10.358 INFO:teuthology.orchestra.run.smithi071.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-30T08:09:10.359 INFO:teuthology.lock.ops:Updating smithi071.front.sepia.ceph.com on lock server 2024-04-30T08:09:10.382 DEBUG:teuthology.orchestra.run.smithi189:> uname -m 2024-04-30T08:09:10.401 INFO:teuthology.orchestra.run.smithi189.stdout:x86_64 2024-04-30T08:09:10.401 DEBUG:teuthology.orchestra.run.smithi189:> cat /etc/os-release 2024-04-30T08:09:10.459 INFO:teuthology.orchestra.run.smithi189.stdout:NAME="CentOS Stream" 2024-04-30T08:09:10.459 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION="9" 2024-04-30T08:09:10.459 INFO:teuthology.orchestra.run.smithi189.stdout:ID="centos" 2024-04-30T08:09:10.459 INFO:teuthology.orchestra.run.smithi189.stdout:ID_LIKE="rhel fedora" 2024-04-30T08:09:10.459 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION_ID="9" 2024-04-30T08:09:10.460 INFO:teuthology.orchestra.run.smithi189.stdout:PLATFORM_ID="platform:el9" 2024-04-30T08:09:10.460 INFO:teuthology.orchestra.run.smithi189.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-30T08:09:10.460 INFO:teuthology.orchestra.run.smithi189.stdout:ANSI_COLOR="0;31" 2024-04-30T08:09:10.460 INFO:teuthology.orchestra.run.smithi189.stdout:LOGO="fedora-logo-icon" 2024-04-30T08:09:10.460 INFO:teuthology.orchestra.run.smithi189.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-30T08:09:10.460 INFO:teuthology.orchestra.run.smithi189.stdout:HOME_URL="https://centos.org/" 2024-04-30T08:09:10.460 INFO:teuthology.orchestra.run.smithi189.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-30T08:09:10.460 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-30T08:09:10.460 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-30T08:09:10.460 INFO:teuthology.lock.ops:Updating smithi189.front.sepia.ceph.com on lock server 2024-04-30T08:09:10.482 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-30T08:09:10.501 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-30T08:09:10.557 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-30T08:09:10.557 DEBUG:teuthology.orchestra.run.smithi032:> test '!' -e /home/ubuntu/cephtest 2024-04-30T08:09:10.560 DEBUG:teuthology.orchestra.run.smithi071:> test '!' -e /home/ubuntu/cephtest 2024-04-30T08:09:10.562 DEBUG:teuthology.orchestra.run.smithi189:> test '!' -e /home/ubuntu/cephtest 2024-04-30T08:09:10.578 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-30T08:09:10.599 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-30T08:09:10.600 DEBUG:teuthology.orchestra.run.smithi032:> test -z $(ls -A /var/lib/ceph) 2024-04-30T08:09:10.617 DEBUG:teuthology.orchestra.run.smithi071:> test -z $(ls -A /var/lib/ceph) 2024-04-30T08:09:10.620 DEBUG:teuthology.orchestra.run.smithi189:> test -z $(ls -A /var/lib/ceph) 2024-04-30T08:09:10.736 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-30T08:09:10.816 INFO:teuthology.run_tasks:Running task kernel... 2024-04-30T08:09:10.835 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'kdb': True, 'sha1': 'distro'} 2024-04-30T08:09:10.835 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-04-30T08:09:10.835 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-04-30T08:09:10.836 DEBUG:teuthology.orchestra.run.smithi032:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-30T08:09:10.853 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-30T08:09:10.854 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-04-30T08:09:11.093 DEBUG:teuthology.task.kernel:sha1 for client.0 is 806ec36b21129f113ce356c007dd59694d340b37 2024-04-30T08:09:11.093 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "806ec36b21129f113ce356c007dd59694d340b37"... 2024-04-30T08:09:11.093 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2024-04-30T08:09:11.111 INFO:teuthology.orchestra.run.smithi032.stdout:5.14.0-437.el9.x86_64 2024-04-30T08:09:11.111 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 806ec36b21129f113ce356c007dd59694d340b37 2024-04-30T08:09:11.111 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-30T08:09:11.112 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum upgrade -y linux-firmware 2024-04-30T08:09:21.745 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - BaseOS 915 kB/s | 8.0 MB 00:09 2024-04-30T08:09:24.108 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - AppStream 16 MB/s | 19 MB 00:01 2024-04-30T08:09:36.276 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - CRB 746 kB/s | 6.2 MB 00:08 2024-04-30T08:09:37.698 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - Extras packages 78 kB/s | 16 kB 00:00 2024-04-30T08:09:40.291 INFO:teuthology.orchestra.run.smithi032.stdout:Extra Packages for Enterprise Linux 8.5 MB/s | 21 MB 00:02 2024-04-30T08:09:45.567 INFO:teuthology.orchestra.run.smithi032.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-04-30T08:09:47.616 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-30T08:09:47.617 INFO:teuthology.orchestra.run.smithi032.stdout:Nothing to do. 2024-04-30T08:09:47.618 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-30T08:09:47.733 INFO:teuthology.task.kernel:Downloading kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-30T08:09:47.734 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-04-30T08:09:47.959 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ 2024-04-30T08:09:47.959 DEBUG:teuthology.orchestra.run.smithi032:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-30T08:09:47.962 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi032.front.sepia.ceph.com to complete... 2024-04-30T08:10:00.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30 08:10:00 URL:https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm [357367792/357367792] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-30T08:10:00.090 INFO:teuthology.orchestra.run.smithi032.stderr:FINISHED --2024-04-30 08:10:00-- 2024-04-30T08:10:00.091 INFO:teuthology.orchestra.run.smithi032.stderr:Total wall clock time: 12s 2024-04-30T08:10:00.091 INFO:teuthology.orchestra.run.smithi032.stderr:Downloaded: 1 files, 341M in 12s (28.8 MB/s) 2024-04-30T08:10:00.093 INFO:teuthology.task.kernel:Installing kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-30T08:10:00.094 DEBUG:teuthology.orchestra.run.smithi032:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-30T08:10:02.322 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying... ######################################## 2024-04-30T08:10:02.437 INFO:teuthology.orchestra.run.smithi032.stdout:Preparing... ######################################## 2024-04-30T08:10:02.556 INFO:teuthology.orchestra.run.smithi032.stdout:Updating / installing... 2024-04-30T08:10:10.843 INFO:teuthology.orchestra.run.smithi032.stdout:kernel-6.9.0_rc3_g806ec36b2112-1 ######################################## 2024-04-30T08:10:51.028 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi032.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-30T08:10:51.028 DEBUG:teuthology.orchestra.run.smithi032:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-30T08:10:51.103 INFO:teuthology.orchestra.run.smithi032.stdout:/boot/System.map-6.9.0-rc3-g806ec36b2112 2024-04-30T08:10:51.103 INFO:teuthology.orchestra.run.smithi032.stdout:/boot/config-6.9.0-rc3-g806ec36b2112 2024-04-30T08:10:51.103 INFO:teuthology.orchestra.run.smithi032.stdout:/boot/vmlinuz-6.9.0-rc3-g806ec36b2112 2024-04-30T08:10:51.104 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/System.map 2024-04-30T08:10:51.104 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/config 2024-04-30T08:10:51.104 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel 2024-04-30T08:10:51.104 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch 2024-04-30T08:10:51.104 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86 2024-04-30T08:10:51.104 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto 2024-04-30T08:10:51.104 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-30T08:10:51.104 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-30T08:10:51.104 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-30T08:10:51.104 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-30T08:10:51.104 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-30T08:10:51.104 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-30T08:10:51.104 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-30T08:10:51.105 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-30T08:10:51.105 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-30T08:10:51.105 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-30T08:10:51.105 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-30T08:10:51.105 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-30T08:10:51.105 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-30T08:10:51.105 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-30T08:10:51.105 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-30T08:10:51.105 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-30T08:10:51.105 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-30T08:10:51.105 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-30T08:10:51.105 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-30T08:10:51.106 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-30T08:10:51.106 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel 2024-04-30T08:10:51.106 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu 2024-04-30T08:10:51.106 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce 2024-04-30T08:10:51.106 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-30T08:10:51.106 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm 2024-04-30T08:10:51.106 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-30T08:10:51.106 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-30T08:10:51.106 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm.ko 2024-04-30T08:10:51.106 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto 2024-04-30T08:10:51.106 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ansi_cprng.ko 2024-04-30T08:10:51.106 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/anubis.ko 2024-04-30T08:10:51.107 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/arc4.ko 2024-04-30T08:10:51.107 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx 2024-04-30T08:10:51.107 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_memcpy.ko 2024-04-30T08:10:51.107 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_pq.ko 2024-04-30T08:10:51.107 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-30T08:10:51.107 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_tx.ko 2024-04-30T08:10:51.107 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_xor.ko 2024-04-30T08:10:51.107 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/raid6test.ko 2024-04-30T08:10:51.107 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authenc.ko 2024-04-30T08:10:51.107 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authencesn.ko 2024-04-30T08:10:51.107 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blake2b_generic.ko 2024-04-30T08:10:51.107 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_common.ko 2024-04-30T08:10:51.108 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_generic.ko 2024-04-30T08:10:51.108 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/camellia_generic.ko 2024-04-30T08:10:51.108 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast5_generic.ko 2024-04-30T08:10:51.108 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast6_generic.ko 2024-04-30T08:10:51.108 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast_common.ko 2024-04-30T08:10:51.108 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ccm.ko 2024-04-30T08:10:51.108 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cmac.ko 2024-04-30T08:10:51.108 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crc32_generic.ko 2024-04-30T08:10:51.108 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_engine.ko 2024-04-30T08:10:51.108 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_user.ko 2024-04-30T08:10:51.108 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cts.ko 2024-04-30T08:10:51.108 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/des_generic.ko 2024-04-30T08:10:51.108 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/dh_generic.ko 2024-04-30T08:10:51.108 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/echainiv.ko 2024-04-30T08:10:51.109 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/essiv.ko 2024-04-30T08:10:51.109 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/fcrypt.ko 2024-04-30T08:10:51.109 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/khazad.ko 2024-04-30T08:10:51.109 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4.ko 2024-04-30T08:10:51.109 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4hc.ko 2024-04-30T08:10:51.109 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/md4.ko 2024-04-30T08:10:51.109 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/michael_mic.ko 2024-04-30T08:10:51.109 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcbc.ko 2024-04-30T08:10:51.109 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcrypt.ko 2024-04-30T08:10:51.109 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/rmd160.ko 2024-04-30T08:10:51.109 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/seed.ko 2024-04-30T08:10:51.109 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/serpent_generic.ko 2024-04-30T08:10:51.109 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tcrypt.ko 2024-04-30T08:10:51.109 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tea.ko 2024-04-30T08:10:51.110 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_common.ko 2024-04-30T08:10:51.110 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_generic.ko 2024-04-30T08:10:51.110 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/vmac.ko 2024-04-30T08:10:51.110 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/wp512.ko 2024-04-30T08:10:51.110 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xcbc.ko 2024-04-30T08:10:51.110 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xor.ko 2024-04-30T08:10:51.110 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xxhash_generic.ko 2024-04-30T08:10:51.110 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers 2024-04-30T08:10:51.110 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi 2024-04-30T08:10:51.110 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-30T08:10:51.110 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_pad.ko 2024-04-30T08:10:51.110 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/ec_sys.ko 2024-04-30T08:10:51.110 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/platform_profile.ko 2024-04-30T08:10:51.111 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbs.ko 2024-04-30T08:10:51.111 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbshc.ko 2024-04-30T08:10:51.111 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/video.ko 2024-04-30T08:10:51.111 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata 2024-04-30T08:10:51.111 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/acard-ahci.ko 2024-04-30T08:10:51.111 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ahci_platform.ko 2024-04-30T08:10:51.111 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ata_generic.ko 2024-04-30T08:10:51.111 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/libahci_platform.ko 2024-04-30T08:10:51.111 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_acpi.ko 2024-04-30T08:10:51.111 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ali.ko 2024-04-30T08:10:51.111 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_amd.ko 2024-04-30T08:10:51.111 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_artop.ko 2024-04-30T08:10:51.111 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atiixp.ko 2024-04-30T08:10:51.112 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atp867x.ko 2024-04-30T08:10:51.112 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd640.ko 2024-04-30T08:10:51.112 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd64x.ko 2024-04-30T08:10:51.112 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cypress.ko 2024-04-30T08:10:51.112 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_efar.ko 2024-04-30T08:10:51.112 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt366.ko 2024-04-30T08:10:51.112 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt37x.ko 2024-04-30T08:10:51.112 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-30T08:10:51.112 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-30T08:10:51.112 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it8213.ko 2024-04-30T08:10:51.112 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it821x.ko 2024-04-30T08:10:51.112 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_jmicron.ko 2024-04-30T08:10:51.112 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_marvell.ko 2024-04-30T08:10:51.113 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_mpiix.ko 2024-04-30T08:10:51.113 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_netcell.ko 2024-04-30T08:10:51.113 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ninja32.ko 2024-04-30T08:10:51.113 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87410.ko 2024-04-30T08:10:51.113 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87415.ko 2024-04-30T08:10:51.113 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_oldpiix.ko 2024-04-30T08:10:51.113 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_opti.ko 2024-04-30T08:10:51.113 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_optidma.ko 2024-04-30T08:10:51.113 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pcmcia.ko 2024-04-30T08:10:51.113 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-30T08:10:51.113 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-30T08:10:51.113 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_piccolo.ko 2024-04-30T08:10:51.113 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_rdc.ko 2024-04-30T08:10:51.114 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sch.ko 2024-04-30T08:10:51.114 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_serverworks.ko 2024-04-30T08:10:51.114 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sil680.ko 2024-04-30T08:10:51.114 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sis.ko 2024-04-30T08:10:51.114 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sl82c105.ko 2024-04-30T08:10:51.114 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_triflex.ko 2024-04-30T08:10:51.114 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_via.ko 2024-04-30T08:10:51.114 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pdc_adma.ko 2024-04-30T08:10:51.114 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_inic162x.ko 2024-04-30T08:10:51.114 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_mv.ko 2024-04-30T08:10:51.114 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_nv.ko 2024-04-30T08:10:51.114 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_promise.ko 2024-04-30T08:10:51.115 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_qstor.ko 2024-04-30T08:10:51.115 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil.ko 2024-04-30T08:10:51.115 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil24.ko 2024-04-30T08:10:51.115 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sis.ko 2024-04-30T08:10:51.115 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_svw.ko 2024-04-30T08:10:51.115 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sx4.ko 2024-04-30T08:10:51.115 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_uli.ko 2024-04-30T08:10:51.115 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_via.ko 2024-04-30T08:10:51.115 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_vsc.ko 2024-04-30T08:10:51.115 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay 2024-04-30T08:10:51.115 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay/ks0108.ko 2024-04-30T08:10:51.115 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base 2024-04-30T08:10:51.115 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap 2024-04-30T08:10:51.115 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-30T08:10:51.116 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-30T08:10:51.116 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma 2024-04-30T08:10:51.116 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma/bcma.ko 2024-04-30T08:10:51.116 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block 2024-04-30T08:10:51.116 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe 2024-04-30T08:10:51.116 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe/aoe.ko 2024-04-30T08:10:51.116 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/brd.ko 2024-04-30T08:10:51.116 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd 2024-04-30T08:10:51.116 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd/drbd.ko 2024-04-30T08:10:51.116 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/floppy.ko 2024-04-30T08:10:51.116 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/loop.ko 2024-04-30T08:10:51.116 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx 2024-04-30T08:10:51.116 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-30T08:10:51.116 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/nbd.ko 2024-04-30T08:10:51.117 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk 2024-04-30T08:10:51.117 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk/null_blk.ko 2024-04-30T08:10:51.117 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/pktcdvd.ko 2024-04-30T08:10:51.117 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/rbd.ko 2024-04-30T08:10:51.117 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/virtio_blk.ko 2024-04-30T08:10:51.117 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback 2024-04-30T08:10:51.117 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-30T08:10:51.117 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkfront.ko 2024-04-30T08:10:51.117 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram 2024-04-30T08:10:51.117 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram/zram.ko 2024-04-30T08:10:51.117 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char 2024-04-30T08:10:51.117 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hangcheck-timer.ko 2024-04-30T08:10:51.118 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random 2024-04-30T08:10:51.118 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-30T08:10:51.118 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-30T08:10:51.118 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-30T08:10:51.118 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/via-rng.ko 2024-04-30T08:10:51.118 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-30T08:10:51.118 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi 2024-04-30T08:10:51.118 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-30T08:10:51.118 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-30T08:10:51.118 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-30T08:10:51.118 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-30T08:10:51.118 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-30T08:10:51.118 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-30T08:10:51.118 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/lp.ko 2024-04-30T08:10:51.119 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave 2024-04-30T08:10:51.119 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave/mwave.ko 2024-04-30T08:10:51.119 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ppdev.ko 2024-04-30T08:10:51.119 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tlclk.ko 2024-04-30T08:10:51.119 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm 2024-04-30T08:10:51.119 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm.ko 2024-04-30T08:10:51.119 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-30T08:10:51.119 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-30T08:10:51.119 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-30T08:10:51.119 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-30T08:10:51.119 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-30T08:10:51.119 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/uv_mmtimer.ko 2024-04-30T08:10:51.119 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/virtio_console.ko 2024-04-30T08:10:51.119 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq 2024-04-30T08:10:51.120 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-30T08:10:51.120 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-30T08:10:51.120 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-30T08:10:51.120 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-30T08:10:51.120 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-30T08:10:51.120 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-30T08:10:51.120 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto 2024-04-30T08:10:51.120 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp 2024-04-30T08:10:51.120 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-30T08:10:51.120 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp.ko 2024-04-30T08:10:51.120 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel 2024-04-30T08:10:51.120 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat 2024-04-30T08:10:51.120 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common 2024-04-30T08:10:51.120 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-30T08:10:51.121 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-30T08:10:51.121 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-30T08:10:51.121 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-aes.ko 2024-04-30T08:10:51.121 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-sha.ko 2024-04-30T08:10:51.121 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio 2024-04-30T08:10:51.121 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-30T08:10:51.121 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca 2024-04-30T08:10:51.121 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca/dca.ko 2024-04-30T08:10:51.121 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq 2024-04-30T08:10:51.121 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-30T08:10:51.121 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma 2024-04-30T08:10:51.121 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw 2024-04-30T08:10:51.121 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-30T08:10:51.121 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat 2024-04-30T08:10:51.122 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-30T08:10:51.122 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac 2024-04-30T08:10:51.122 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/amd64_edac.ko 2024-04-30T08:10:51.122 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/e752x_edac.ko 2024-04-30T08:10:51.122 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/edac_mce_amd.ko 2024-04-30T08:10:51.122 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3000_edac.ko 2024-04-30T08:10:51.122 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3200_edac.ko 2024-04-30T08:10:51.122 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5100_edac.ko 2024-04-30T08:10:51.122 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5400_edac.ko 2024-04-30T08:10:51.122 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7300_edac.ko 2024-04-30T08:10:51.122 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7core_edac.ko 2024-04-30T08:10:51.122 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i82975x_edac.ko 2024-04-30T08:10:51.122 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/ie31200_edac.ko 2024-04-30T08:10:51.122 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/sb_edac.ko 2024-04-30T08:10:51.123 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/x38_edac.ko 2024-04-30T08:10:51.123 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware 2024-04-30T08:10:51.123 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/edd.ko 2024-04-30T08:10:51.123 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-30T08:10:51.123 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio 2024-04-30T08:10:51.123 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-ich.ko 2024-04-30T08:10:51.123 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-30T08:10:51.123 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid 2024-04-30T08:10:51.123 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-appleir.ko 2024-04-30T08:10:51.123 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-aureal.ko 2024-04-30T08:10:51.123 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-axff.ko 2024-04-30T08:10:51.123 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-dr.ko 2024-04-30T08:10:51.123 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elecom.ko 2024-04-30T08:10:51.123 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elo.ko 2024-04-30T08:10:51.124 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-emsff.ko 2024-04-30T08:10:51.124 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gaff.ko 2024-04-30T08:10:51.124 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gt683r.ko 2024-04-30T08:10:51.124 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gyration.ko 2024-04-30T08:10:51.124 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-30T08:10:51.124 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-30T08:10:51.124 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtekff.ko 2024-04-30T08:10:51.124 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-hyperv.ko 2024-04-30T08:10:51.124 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-icade.ko 2024-04-30T08:10:51.124 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-keytouch.ko 2024-04-30T08:10:51.124 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-kye.ko 2024-04-30T08:10:51.124 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lcpower.ko 2024-04-30T08:10:51.124 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-led.ko 2024-04-30T08:10:51.124 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lenovo.ko 2024-04-30T08:10:51.125 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-30T08:10:51.125 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-30T08:10:51.125 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-multitouch.ko 2024-04-30T08:10:51.125 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-ortek.ko 2024-04-30T08:10:51.125 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-penmount.ko 2024-04-30T08:10:51.125 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-petalynx.ko 2024-04-30T08:10:51.125 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-picolcd.ko 2024-04-30T08:10:51.125 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-pl.ko 2024-04-30T08:10:51.125 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-plantronics.ko 2024-04-30T08:10:51.125 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-primax.ko 2024-04-30T08:10:51.125 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-rmi.ko 2024-04-30T08:10:51.126 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-30T08:10:51.126 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-common.ko 2024-04-30T08:10:51.126 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-30T08:10:51.126 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-30T08:10:51.126 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-30T08:10:51.126 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-30T08:10:51.126 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-30T08:10:51.126 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-30T08:10:51.126 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-30T08:10:51.126 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-30T08:10:51.126 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-30T08:10:51.126 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat.ko 2024-04-30T08:10:51.126 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-saitek.ko 2024-04-30T08:10:51.127 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-samsung.ko 2024-04-30T08:10:51.127 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-30T08:10:51.127 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sjoy.ko 2024-04-30T08:10:51.127 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sony.ko 2024-04-30T08:10:51.127 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-speedlink.ko 2024-04-30T08:10:51.127 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-steelseries.ko 2024-04-30T08:10:51.127 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sunplus.ko 2024-04-30T08:10:51.127 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-30T08:10:51.127 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tivo.ko 2024-04-30T08:10:51.127 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tmff.ko 2024-04-30T08:10:51.127 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-topseed.ko 2024-04-30T08:10:51.127 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-twinhan.ko 2024-04-30T08:10:51.128 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-uclogic.ko 2024-04-30T08:10:51.128 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-waltop.ko 2024-04-30T08:10:51.128 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-wiimote.ko 2024-04-30T08:10:51.128 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-xinmo.ko 2024-04-30T08:10:51.128 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zpff.ko 2024-04-30T08:10:51.128 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zydacron.ko 2024-04-30T08:10:51.128 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/uhid.ko 2024-04-30T08:10:51.128 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/wacom.ko 2024-04-30T08:10:51.128 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv 2024-04-30T08:10:51.128 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_balloon.ko 2024-04-30T08:10:51.128 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_utils.ko 2024-04-30T08:10:51.128 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_vmbus.ko 2024-04-30T08:10:51.128 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon 2024-04-30T08:10:51.128 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru.ko 2024-04-30T08:10:51.129 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru3.ko 2024-04-30T08:10:51.129 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-30T08:10:51.129 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7414.ko 2024-04-30T08:10:51.129 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7418.ko 2024-04-30T08:10:51.129 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adc128d818.ko 2024-04-30T08:10:51.129 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1025.ko 2024-04-30T08:10:51.129 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1026.ko 2024-04-30T08:10:51.129 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1029.ko 2024-04-30T08:10:51.129 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1031.ko 2024-04-30T08:10:51.129 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm9240.ko 2024-04-30T08:10:51.129 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ads7828.ko 2024-04-30T08:10:51.129 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7410.ko 2024-04-30T08:10:51.129 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7411.ko 2024-04-30T08:10:51.130 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7462.ko 2024-04-30T08:10:51.130 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7470.ko 2024-04-30T08:10:51.130 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7475.ko 2024-04-30T08:10:51.130 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7x10.ko 2024-04-30T08:10:51.130 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/amc6821.ko 2024-04-30T08:10:51.130 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/applesmc.ko 2024-04-30T08:10:51.130 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asb100.ko 2024-04-30T08:10:51.130 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asc7621.ko 2024-04-30T08:10:51.130 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-30T08:10:51.130 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/atxp1.ko 2024-04-30T08:10:51.130 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/coretemp.ko 2024-04-30T08:10:51.130 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-30T08:10:51.131 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dme1737.ko 2024-04-30T08:10:51.131 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds1621.ko 2024-04-30T08:10:51.131 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds620.ko 2024-04-30T08:10:51.131 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc1403.ko 2024-04-30T08:10:51.131 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc6w201.ko 2024-04-30T08:10:51.131 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71805f.ko 2024-04-30T08:10:51.131 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71882fg.ko 2024-04-30T08:10:51.131 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f75375s.ko 2024-04-30T08:10:51.131 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fam15h_power.ko 2024-04-30T08:10:51.131 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fschmd.ko 2024-04-30T08:10:51.131 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g760a.ko 2024-04-30T08:10:51.131 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g762.ko 2024-04-30T08:10:51.131 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl518sm.ko 2024-04-30T08:10:51.131 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl520sm.ko 2024-04-30T08:10:51.132 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-30T08:10:51.132 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5500_temp.ko 2024-04-30T08:10:51.132 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5k_amb.ko 2024-04-30T08:10:51.132 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmaem.ko 2024-04-30T08:10:51.132 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmpex.ko 2024-04-30T08:10:51.132 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina209.ko 2024-04-30T08:10:51.132 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina2xx.ko 2024-04-30T08:10:51.132 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/it87.ko 2024-04-30T08:10:51.132 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k10temp.ko 2024-04-30T08:10:51.132 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k8temp.ko 2024-04-30T08:10:51.132 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lineage-pem.ko 2024-04-30T08:10:51.132 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm63.ko 2024-04-30T08:10:51.132 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm73.ko 2024-04-30T08:10:51.132 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm75.ko 2024-04-30T08:10:51.133 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm77.ko 2024-04-30T08:10:51.133 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm78.ko 2024-04-30T08:10:51.133 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm80.ko 2024-04-30T08:10:51.133 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm83.ko 2024-04-30T08:10:51.133 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm85.ko 2024-04-30T08:10:51.133 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm87.ko 2024-04-30T08:10:51.133 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm90.ko 2024-04-30T08:10:51.133 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm92.ko 2024-04-30T08:10:51.133 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm93.ko 2024-04-30T08:10:51.133 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95234.ko 2024-04-30T08:10:51.133 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95241.ko 2024-04-30T08:10:51.133 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95245.ko 2024-04-30T08:10:51.134 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc2945.ko 2024-04-30T08:10:51.134 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4151.ko 2024-04-30T08:10:51.134 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4215.ko 2024-04-30T08:10:51.134 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4222.ko 2024-04-30T08:10:51.134 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4245.ko 2024-04-30T08:10:51.134 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4260.ko 2024-04-30T08:10:51.134 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4261.ko 2024-04-30T08:10:51.134 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max16065.ko 2024-04-30T08:10:51.134 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1619.ko 2024-04-30T08:10:51.134 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1668.ko 2024-04-30T08:10:51.134 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max197.ko 2024-04-30T08:10:51.134 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6639.ko 2024-04-30T08:10:51.134 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6650.ko 2024-04-30T08:10:51.135 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6697.ko 2024-04-30T08:10:51.135 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/mcp3021.ko 2024-04-30T08:10:51.135 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6683.ko 2024-04-30T08:10:51.135 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775-core.ko 2024-04-30T08:10:51.135 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775.ko 2024-04-30T08:10:51.135 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct7802.ko 2024-04-30T08:10:51.135 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87360.ko 2024-04-30T08:10:51.135 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87427.ko 2024-04-30T08:10:51.135 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pcf8591.ko 2024-04-30T08:10:51.135 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus 2024-04-30T08:10:51.135 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-30T08:10:51.135 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-30T08:10:51.135 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-30T08:10:51.136 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-30T08:10:51.136 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-30T08:10:51.136 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-30T08:10:51.136 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-30T08:10:51.136 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-30T08:10:51.136 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-30T08:10:51.136 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-30T08:10:51.136 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-30T08:10:51.136 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-30T08:10:51.136 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/powr1220.ko 2024-04-30T08:10:51.136 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5627.ko 2024-04-30T08:10:51.136 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5636.ko 2024-04-30T08:10:51.136 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-30T08:10:51.137 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht15.ko 2024-04-30T08:10:51.137 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht21.ko 2024-04-30T08:10:51.137 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/shtc1.ko 2024-04-30T08:10:51.137 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sis5595.ko 2024-04-30T08:10:51.137 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47b397.ko 2024-04-30T08:10:51.137 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m1.ko 2024-04-30T08:10:51.137 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m192.ko 2024-04-30T08:10:51.137 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/thmc50.ko 2024-04-30T08:10:51.137 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp102.ko 2024-04-30T08:10:51.137 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp103.ko 2024-04-30T08:10:51.137 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp401.ko 2024-04-30T08:10:51.137 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp421.ko 2024-04-30T08:10:51.137 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via-cputemp.ko 2024-04-30T08:10:51.138 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via686a.ko 2024-04-30T08:10:51.138 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt1211.ko 2024-04-30T08:10:51.138 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt8231.ko 2024-04-30T08:10:51.138 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627ehf.ko 2024-04-30T08:10:51.138 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627hf.ko 2024-04-30T08:10:51.138 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83781d.ko 2024-04-30T08:10:51.138 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83791d.ko 2024-04-30T08:10:51.138 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83792d.ko 2024-04-30T08:10:51.138 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83793.ko 2024-04-30T08:10:51.138 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83795.ko 2024-04-30T08:10:51.138 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l785ts.ko 2024-04-30T08:10:51.138 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l786ng.ko 2024-04-30T08:10:51.138 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c 2024-04-30T08:10:51.139 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos 2024-04-30T08:10:51.139 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-30T08:10:51.139 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-30T08:10:51.139 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses 2024-04-30T08:10:51.139 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-30T08:10:51.139 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-30T08:10:51.139 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-30T08:10:51.139 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-30T08:10:51.139 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-30T08:10:51.139 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-30T08:10:51.139 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-30T08:10:51.139 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-30T08:10:51.139 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-30T08:10:51.140 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-30T08:10:51.140 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-30T08:10:51.140 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-30T08:10:51.140 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-30T08:10:51.140 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-30T08:10:51.140 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-30T08:10:51.140 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-30T08:10:51.140 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-30T08:10:51.140 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-30T08:10:51.140 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-30T08:10:51.140 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-30T08:10:51.140 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-30T08:10:51.140 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-30T08:10:51.140 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-30T08:10:51.141 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-dev.ko 2024-04-30T08:10:51.141 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-mux.ko 2024-04-30T08:10:51.141 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-30T08:10:51.141 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-smbus.ko 2024-04-30T08:10:51.141 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-stub.ko 2024-04-30T08:10:51.141 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband 2024-04-30T08:10:51.141 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core 2024-04-30T08:10:51.141 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-30T08:10:51.141 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_core.ko 2024-04-30T08:10:51.141 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-30T08:10:51.141 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-30T08:10:51.141 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-30T08:10:51.141 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-30T08:10:51.141 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-30T08:10:51.141 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw 2024-04-30T08:10:51.142 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4 2024-04-30T08:10:51.142 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-30T08:10:51.142 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4 2024-04-30T08:10:51.142 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-30T08:10:51.142 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5 2024-04-30T08:10:51.142 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-30T08:10:51.142 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca 2024-04-30T08:10:51.142 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-30T08:10:51.142 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp 2024-04-30T08:10:51.142 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib 2024-04-30T08:10:51.142 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-30T08:10:51.142 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser 2024-04-30T08:10:51.142 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-30T08:10:51.143 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert 2024-04-30T08:10:51.143 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-30T08:10:51.143 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp 2024-04-30T08:10:51.143 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-30T08:10:51.143 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt 2024-04-30T08:10:51.143 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-30T08:10:51.143 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input 2024-04-30T08:10:51.143 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport 2024-04-30T08:10:51.143 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-30T08:10:51.143 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-30T08:10:51.143 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/gameport.ko 2024-04-30T08:10:51.143 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/joydev.ko 2024-04-30T08:10:51.143 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard 2024-04-30T08:10:51.143 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-30T08:10:51.144 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse 2024-04-30T08:10:51.144 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/appletouch.ko 2024-04-30T08:10:51.144 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/bcm5974.ko 2024-04-30T08:10:51.144 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/cyapatp.ko 2024-04-30T08:10:51.144 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-30T08:10:51.144 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/sermouse.ko 2024-04-30T08:10:51.144 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-30T08:10:51.144 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-30T08:10:51.144 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-30T08:10:51.144 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4 2024-04-30T08:10:51.144 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-30T08:10:51.144 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio 2024-04-30T08:10:51.144 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/altera_ps2.ko 2024-04-30T08:10:51.144 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/arc_ps2.ko 2024-04-30T08:10:51.144 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-30T08:10:51.145 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/serio_raw.ko 2024-04-30T08:10:51.145 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/sparse-keymap.ko 2024-04-30T08:10:51.145 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds 2024-04-30T08:10:51.145 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-blinkm.ko 2024-04-30T08:10:51.145 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lm3530.ko 2024-04-30T08:10:51.145 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lp3944.ko 2024-04-30T08:10:51.145 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-ss4200.ko 2024-04-30T08:10:51.145 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger 2024-04-30T08:10:51.145 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-30T08:10:51.145 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-30T08:10:51.145 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-30T08:10:51.145 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-30T08:10:51.146 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-30T08:10:51.146 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-30T08:10:51.146 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-30T08:10:51.146 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-30T08:10:51.146 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-30T08:10:51.146 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md 2024-04-30T08:10:51.146 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache 2024-04-30T08:10:51.146 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache/bcache.ko 2024-04-30T08:10:51.146 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-bio-prison.ko 2024-04-30T08:10:51.146 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache-smq.ko 2024-04-30T08:10:51.146 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache.ko 2024-04-30T08:10:51.146 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-crypt.ko 2024-04-30T08:10:51.147 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-delay.ko 2024-04-30T08:10:51.147 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-flakey.ko 2024-04-30T08:10:51.147 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-log-userspace.ko 2024-04-30T08:10:51.147 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-multipath.ko 2024-04-30T08:10:51.147 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-queue-length.ko 2024-04-30T08:10:51.147 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-raid.ko 2024-04-30T08:10:51.147 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-round-robin.ko 2024-04-30T08:10:51.147 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-service-time.ko 2024-04-30T08:10:51.147 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-switch.ko 2024-04-30T08:10:51.147 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-thin-pool.ko 2024-04-30T08:10:51.147 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-verity.ko 2024-04-30T08:10:51.147 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data 2024-04-30T08:10:51.147 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-30T08:10:51.147 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid0.ko 2024-04-30T08:10:51.148 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid1.ko 2024-04-30T08:10:51.148 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid10.ko 2024-04-30T08:10:51.148 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid456.ko 2024-04-30T08:10:51.148 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message 2024-04-30T08:10:51.148 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion 2024-04-30T08:10:51.148 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptbase.ko 2024-04-30T08:10:51.148 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptctl.ko 2024-04-30T08:10:51.148 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptfc.ko 2024-04-30T08:10:51.148 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptlan.ko 2024-04-30T08:10:51.148 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptsas.ko 2024-04-30T08:10:51.148 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptscsih.ko 2024-04-30T08:10:51.148 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptspi.ko 2024-04-30T08:10:51.148 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd 2024-04-30T08:10:51.148 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_ich.ko 2024-04-30T08:10:51.149 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_sch.ko 2024-04-30T08:10:51.149 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/mfd-core.ko 2024-04-30T08:10:51.149 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/sm501.ko 2024-04-30T08:10:51.149 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/viperboard.ko 2024-04-30T08:10:51.149 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/vx855.ko 2024-04-30T08:10:51.149 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/wl1273-core.ko 2024-04-30T08:10:51.149 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc 2024-04-30T08:10:51.149 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl 2024-04-30T08:10:51.149 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-30T08:10:51.149 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds9802als.ko 2024-04-30T08:10:51.149 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds990x.ko 2024-04-30T08:10:51.149 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/bh1770glc.ko 2024-04-30T08:10:51.149 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710 2024-04-30T08:10:51.149 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710/cb710.ko 2024-04-30T08:10:51.150 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom 2024-04-30T08:10:51.150 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/at24.ko 2024-04-30T08:10:51.150 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-30T08:10:51.150 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/max6875.ko 2024-04-30T08:10:51.150 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/enclosure.ko 2024-04-30T08:10:51.150 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/hpilo.ko 2024-04-30T08:10:51.150 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29003.ko 2024-04-30T08:10:51.150 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29020.ko 2024-04-30T08:10:51.150 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d 2024-04-30T08:10:51.150 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-30T08:10:51.150 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-30T08:10:51.150 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei 2024-04-30T08:10:51.150 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-me.ko 2024-04-30T08:10:51.150 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-txe.ko 2024-04-30T08:10:51.151 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei.ko 2024-04-30T08:10:51.151 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru 2024-04-30T08:10:51.151 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-30T08:10:51.151 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp 2024-04-30T08:10:51.151 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-30T08:10:51.151 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-30T08:10:51.151 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-30T08:10:51.151 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_7xx1.ko 2024-04-30T08:10:51.151 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_core.ko 2024-04-30T08:10:51.151 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tsl2550.ko 2024-04-30T08:10:51.151 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_balloon.ko 2024-04-30T08:10:51.151 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci 2024-04-30T08:10:51.151 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-30T08:10:51.152 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net 2024-04-30T08:10:51.152 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding 2024-04-30T08:10:51.152 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding/bonding.ko 2024-04-30T08:10:51.152 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa 2024-04-30T08:10:51.152 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53 2024-04-30T08:10:51.152 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-30T08:10:51.152 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-30T08:10:51.152 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-30T08:10:51.152 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx 2024-04-30T08:10:51.152 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-30T08:10:51.152 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dummy.ko 2024-04-30T08:10:51.152 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/eql.ko 2024-04-30T08:10:51.152 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet 2024-04-30T08:10:51.152 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com 2024-04-30T08:10:51.153 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-30T08:10:51.153 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-30T08:10:51.153 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-30T08:10:51.153 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-30T08:10:51.153 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390 2024-04-30T08:10:51.153 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-30T08:10:51.153 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-30T08:10:51.153 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-30T08:10:51.153 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-30T08:10:51.153 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec 2024-04-30T08:10:51.153 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-30T08:10:51.153 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere 2024-04-30T08:10:51.153 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-30T08:10:51.153 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon 2024-04-30T08:10:51.153 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-30T08:10:51.154 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera 2024-04-30T08:10:51.154 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-30T08:10:51.154 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd 2024-04-30T08:10:51.154 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-30T08:10:51.154 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-30T08:10:51.154 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-30T08:10:51.154 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros 2024-04-30T08:10:51.154 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx 2024-04-30T08:10:51.154 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-30T08:10:51.154 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-30T08:10:51.154 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-30T08:10:51.154 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-30T08:10:51.154 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-30T08:10:51.155 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx 2024-04-30T08:10:51.155 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-30T08:10:51.155 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-30T08:10:51.155 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom 2024-04-30T08:10:51.155 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-30T08:10:51.155 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-30T08:10:51.155 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-30T08:10:51.155 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-30T08:10:51.155 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-30T08:10:51.155 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet 2024-04-30T08:10:51.155 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-30T08:10:51.155 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-30T08:10:51.156 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade 2024-04-30T08:10:51.156 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna 2024-04-30T08:10:51.156 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-30T08:10:51.156 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio 2024-04-30T08:10:51.156 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-30T08:10:51.156 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-30T08:10:51.156 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-30T08:10:51.156 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-30T08:10:51.156 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-30T08:10:51.156 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-30T08:10:51.156 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-30T08:10:51.156 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-30T08:10:51.156 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-30T08:10:51.156 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-30T08:10:51.157 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco 2024-04-30T08:10:51.157 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic 2024-04-30T08:10:51.157 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-30T08:10:51.157 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec 2024-04-30T08:10:51.157 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip 2024-04-30T08:10:51.157 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-30T08:10:51.157 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-30T08:10:51.157 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-30T08:10:51.157 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-30T08:10:51.157 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-30T08:10:51.157 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-30T08:10:51.157 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink 2024-04-30T08:10:51.158 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-30T08:10:51.158 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-30T08:10:51.158 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dnet.ko 2024-04-30T08:10:51.158 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex 2024-04-30T08:10:51.158 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet 2024-04-30T08:10:51.158 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-30T08:10:51.158 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ethoc.ko 2024-04-30T08:10:51.158 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/fealnx.ko 2024-04-30T08:10:51.158 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel 2024-04-30T08:10:51.158 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-30T08:10:51.158 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000 2024-04-30T08:10:51.158 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-30T08:10:51.158 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e 2024-04-30T08:10:51.158 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-30T08:10:51.159 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k 2024-04-30T08:10:51.159 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-30T08:10:51.159 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e 2024-04-30T08:10:51.159 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-30T08:10:51.159 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf 2024-04-30T08:10:51.159 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-30T08:10:51.159 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb 2024-04-30T08:10:51.159 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-30T08:10:51.159 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf 2024-04-30T08:10:51.159 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-30T08:10:51.159 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-30T08:10:51.159 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-30T08:10:51.160 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-30T08:10:51.160 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-30T08:10:51.160 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/jme.ko 2024-04-30T08:10:51.160 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell 2024-04-30T08:10:51.160 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-30T08:10:51.160 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-30T08:10:51.160 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-30T08:10:51.160 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox 2024-04-30T08:10:51.160 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-30T08:10:51.160 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-30T08:10:51.160 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-30T08:10:51.160 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-30T08:10:51.160 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-30T08:10:51.161 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-30T08:10:51.161 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel 2024-04-30T08:10:51.161 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-30T08:10:51.161 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom 2024-04-30T08:10:51.161 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-30T08:10:51.161 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-30T08:10:51.161 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi 2024-04-30T08:10:51.161 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-30T08:10:51.161 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-30T08:10:51.161 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion 2024-04-30T08:10:51.161 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-30T08:10:51.161 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia 2024-04-30T08:10:51.161 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-30T08:10:51.162 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines 2024-04-30T08:10:51.162 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-30T08:10:51.162 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-30T08:10:51.162 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic 2024-04-30T08:10:51.162 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-30T08:10:51.162 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-30T08:10:51.162 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-30T08:10:51.162 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-30T08:10:51.162 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-30T08:10:51.162 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc 2024-04-30T08:10:51.162 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-30T08:10:51.162 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek 2024-04-30T08:10:51.163 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-30T08:10:51.163 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-30T08:10:51.163 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-30T08:10:51.163 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-30T08:10:51.163 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker 2024-04-30T08:10:51.163 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-30T08:10:51.163 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc 2024-04-30T08:10:51.163 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-30T08:10:51.163 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan 2024-04-30T08:10:51.163 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-30T08:10:51.163 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis 2024-04-30T08:10:51.164 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-30T08:10:51.164 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-30T08:10:51.164 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc 2024-04-30T08:10:51.164 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-30T08:10:51.164 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-30T08:10:51.164 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-30T08:10:51.164 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-30T08:10:51.164 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro 2024-04-30T08:10:51.164 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-30T08:10:51.164 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-30T08:10:51.165 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-30T08:10:51.165 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun 2024-04-30T08:10:51.165 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-30T08:10:51.165 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-30T08:10:51.165 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-30T08:10:51.165 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-30T08:10:51.165 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti 2024-04-30T08:10:51.165 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-30T08:10:51.165 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti 2024-04-30T08:10:51.165 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-30T08:10:51.165 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via 2024-04-30T08:10:51.165 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-30T08:10:51.165 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-30T08:10:51.166 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet 2024-04-30T08:10:51.166 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-30T08:10:51.166 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-30T08:10:51.166 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom 2024-04-30T08:10:51.166 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-30T08:10:51.166 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/geneve.ko 2024-04-30T08:10:51.166 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv 2024-04-30T08:10:51.166 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-30T08:10:51.166 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154 2024-04-30T08:10:51.166 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-30T08:10:51.166 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ifb.ko 2024-04-30T08:10:51.166 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan 2024-04-30T08:10:51.166 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-30T08:10:51.167 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvlan.ko 2024-04-30T08:10:51.167 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvtap.ko 2024-04-30T08:10:51.167 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio 2024-04-30T08:10:51.167 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio.ko 2024-04-30T08:10:51.167 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-30T08:10:51.167 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-30T08:10:51.167 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-30T08:10:51.167 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mii.ko 2024-04-30T08:10:51.167 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/net_failover.ko 2024-04-30T08:10:51.167 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/netconsole.ko 2024-04-30T08:10:51.167 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/nlmon.ko 2024-04-30T08:10:51.168 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs 2024-04-30T08:10:51.168 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-30T08:10:51.168 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-30T08:10:51.168 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy 2024-04-30T08:10:51.168 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/amd.ko 2024-04-30T08:10:51.168 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-30T08:10:51.168 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-30T08:10:51.168 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-30T08:10:51.168 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm87xx.ko 2024-04-30T08:10:51.168 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/broadcom.ko 2024-04-30T08:10:51.168 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/cicada.ko 2024-04-30T08:10:51.168 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/davicom.ko 2024-04-30T08:10:51.168 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/dp83640.ko 2024-04-30T08:10:51.169 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/et1011c.ko 2024-04-30T08:10:51.169 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/icplus.ko 2024-04-30T08:10:51.169 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/lxt.ko 2024-04-30T08:10:51.169 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/marvell.ko 2024-04-30T08:10:51.169 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/micrel.ko 2024-04-30T08:10:51.169 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/national.ko 2024-04-30T08:10:51.169 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/phylink.ko 2024-04-30T08:10:51.169 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/qsemi.ko 2024-04-30T08:10:51.169 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/realtek.ko 2024-04-30T08:10:51.169 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/smsc.ko 2024-04-30T08:10:51.169 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/ste10Xp.ko 2024-04-30T08:10:51.169 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/vitesse.ko 2024-04-30T08:10:51.169 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp 2024-04-30T08:10:51.169 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-30T08:10:51.170 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_async.ko 2024-04-30T08:10:51.170 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-30T08:10:51.170 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-30T08:10:51.170 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-30T08:10:51.170 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-30T08:10:51.170 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppoe.ko 2024-04-30T08:10:51.170 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppox.ko 2024-04-30T08:10:51.170 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pptp.ko 2024-04-30T08:10:51.170 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip 2024-04-30T08:10:51.170 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slhc.ko 2024-04-30T08:10:51.170 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slip.ko 2024-04-30T08:10:51.170 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/sungem_phy.ko 2024-04-30T08:10:51.171 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tap.ko 2024-04-30T08:10:51.171 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team 2024-04-30T08:10:51.171 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team.ko 2024-04-30T08:10:51.171 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-30T08:10:51.171 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-30T08:10:51.171 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-30T08:10:51.171 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_random.ko 2024-04-30T08:10:51.171 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-30T08:10:51.171 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tun.ko 2024-04-30T08:10:51.171 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/veth.ko 2024-04-30T08:10:51.171 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/virtio_net.ko 2024-04-30T08:10:51.171 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3 2024-04-30T08:10:51.172 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-30T08:10:51.172 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan 2024-04-30T08:10:51.172 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan/vxlan.ko 2024-04-30T08:10:51.172 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback 2024-04-30T08:10:51.172 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-30T08:10:51.172 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netfront.ko 2024-04-30T08:10:51.172 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb 2024-04-30T08:10:51.172 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb/ntb.ko 2024-04-30T08:10:51.172 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme 2024-04-30T08:10:51.172 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host 2024-04-30T08:10:51.172 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme-core.ko 2024-04-30T08:10:51.172 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme.ko 2024-04-30T08:10:51.172 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport 2024-04-30T08:10:51.173 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport.ko 2024-04-30T08:10:51.173 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_cs.ko 2024-04-30T08:10:51.173 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_pc.ko 2024-04-30T08:10:51.173 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_serial.ko 2024-04-30T08:10:51.173 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci 2024-04-30T08:10:51.173 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug 2024-04-30T08:10:51.173 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-30T08:10:51.173 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie 2024-04-30T08:10:51.173 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-30T08:10:51.173 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/xen-pcifront.ko 2024-04-30T08:10:51.173 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia 2024-04-30T08:10:51.173 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/i82092.ko 2024-04-30T08:10:51.174 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/pd6729.ko 2024-04-30T08:10:51.174 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-30T08:10:51.174 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl 2024-04-30T08:10:51.174 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel 2024-04-30T08:10:51.174 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-30T08:10:51.174 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform 2024-04-30T08:10:51.174 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86 2024-04-30T08:10:51.174 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-30T08:10:51.174 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acerhdf.ko 2024-04-30T08:10:51.174 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-30T08:10:51.174 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-30T08:10:51.174 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-30T08:10:51.174 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-30T08:10:51.175 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-30T08:10:51.175 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-30T08:10:51.175 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-30T08:10:51.175 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-30T08:10:51.175 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-30T08:10:51.175 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/hdaps.ko 2024-04-30T08:10:51.175 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel 2024-04-30T08:10:51.175 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-30T08:10:51.175 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel_ips.ko 2024-04-30T08:10:51.175 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-30T08:10:51.175 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-30T08:10:51.175 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-30T08:10:51.175 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-30T08:10:51.175 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-30T08:10:51.176 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-30T08:10:51.176 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-30T08:10:51.176 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-30T08:10:51.176 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-30T08:10:51.176 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi.ko 2024-04-30T08:10:51.176 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap 2024-04-30T08:10:51.176 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-30T08:10:51.176 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-30T08:10:51.176 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps 2024-04-30T08:10:51.176 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients 2024-04-30T08:10:51.176 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-30T08:10:51.176 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-30T08:10:51.176 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps_parport.ko 2024-04-30T08:10:51.176 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/pps_core.ko 2024-04-30T08:10:51.177 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp 2024-04-30T08:10:51.177 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp.ko 2024-04-30T08:10:51.177 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp_kvm.ko 2024-04-30T08:10:51.177 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras 2024-04-30T08:10:51.177 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd 2024-04-30T08:10:51.177 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl 2024-04-30T08:10:51.177 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-30T08:10:51.177 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/fmpm.ko 2024-04-30T08:10:51.177 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc 2024-04-30T08:10:51.177 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-30T08:10:51.177 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-30T08:10:51.177 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-30T08:10:51.177 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-30T08:10:51.177 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-30T08:10:51.177 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-30T08:10:51.178 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-30T08:10:51.178 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-30T08:10:51.178 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-30T08:10:51.178 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-30T08:10:51.178 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-em3027.ko 2024-04-30T08:10:51.178 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-30T08:10:51.178 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-30T08:10:51.178 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-30T08:10:51.178 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-30T08:10:51.178 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-30T08:10:51.178 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-30T08:10:51.178 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-max6900.ko 2024-04-30T08:10:51.178 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-30T08:10:51.179 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-30T08:10:51.179 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-30T08:10:51.179 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-30T08:10:51.179 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-30T08:10:51.179 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-30T08:10:51.179 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-30T08:10:51.179 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-30T08:10:51.179 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-30T08:10:51.179 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-30T08:10:51.179 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-30T08:10:51.179 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-30T08:10:51.179 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-x1205.ko 2024-04-30T08:10:51.179 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi 2024-04-30T08:10:51.179 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-9xxx.ko 2024-04-30T08:10:51.180 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-sas.ko 2024-04-30T08:10:51.180 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-xxxx.ko 2024-04-30T08:10:51.180 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/BusLogic.ko 2024-04-30T08:10:51.180 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/a100u2w.ko 2024-04-30T08:10:51.180 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid 2024-04-30T08:10:51.180 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-30T08:10:51.180 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/advansys.ko 2024-04-30T08:10:51.180 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx 2024-04-30T08:10:51.180 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-30T08:10:51.180 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-30T08:10:51.180 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/am53c974.ko 2024-04-30T08:10:51.180 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr 2024-04-30T08:10:51.180 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-30T08:10:51.180 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/atp870u.ko 2024-04-30T08:10:51.181 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi 2024-04-30T08:10:51.181 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-30T08:10:51.181 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa 2024-04-30T08:10:51.181 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa/bfa.ko 2024-04-30T08:10:51.181 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc 2024-04-30T08:10:51.181 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-30T08:10:51.181 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i 2024-04-30T08:10:51.181 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-30T08:10:51.181 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ch.ko 2024-04-30T08:10:51.181 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor 2024-04-30T08:10:51.181 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-30T08:10:51.181 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi 2024-04-30T08:10:51.181 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-30T08:10:51.181 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-30T08:10:51.182 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-30T08:10:51.182 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-30T08:10:51.182 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-30T08:10:51.182 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dc395x.ko 2024-04-30T08:10:51.182 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler 2024-04-30T08:10:51.182 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-30T08:10:51.182 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-30T08:10:51.182 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-30T08:10:51.182 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-30T08:10:51.183 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dmx3191d.ko 2024-04-30T08:10:51.183 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r 2024-04-30T08:10:51.183 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-30T08:10:51.183 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esp_scsi.ko 2024-04-30T08:10:51.183 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe 2024-04-30T08:10:51.183 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-30T08:10:51.183 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-30T08:10:51.183 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic 2024-04-30T08:10:51.183 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic/fnic.ko 2024-04-30T08:10:51.183 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hpsa.ko 2024-04-30T08:10:51.183 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hptiop.ko 2024-04-30T08:10:51.183 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hv_storvsc.ko 2024-04-30T08:10:51.183 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/initio.ko 2024-04-30T08:10:51.183 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ipr.ko 2024-04-30T08:10:51.184 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ips.ko 2024-04-30T08:10:51.184 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci 2024-04-30T08:10:51.184 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci/isci.ko 2024-04-30T08:10:51.184 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-30T08:10:51.184 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-30T08:10:51.184 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc 2024-04-30T08:10:51.184 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc/libfc.ko 2024-04-30T08:10:51.184 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi.ko 2024-04-30T08:10:51.184 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-30T08:10:51.184 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas 2024-04-30T08:10:51.184 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas/libsas.ko 2024-04-30T08:10:51.184 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc 2024-04-30T08:10:51.184 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-30T08:10:51.184 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid 2024-04-30T08:10:51.185 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid.ko 2024-04-30T08:10:51.185 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-30T08:10:51.185 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-30T08:10:51.185 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-30T08:10:51.185 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas 2024-04-30T08:10:51.185 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-30T08:10:51.185 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas 2024-04-30T08:10:51.185 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-30T08:10:51.185 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvumi.ko 2024-04-30T08:10:51.185 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001 2024-04-30T08:10:51.185 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-30T08:10:51.185 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pmcraid.ko 2024-04-30T08:10:51.185 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla1280.ko 2024-04-30T08:10:51.185 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx 2024-04-30T08:10:51.185 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-30T08:10:51.186 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-30T08:10:51.186 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx 2024-04-30T08:10:51.186 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-30T08:10:51.186 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/raid_class.ko 2024-04-30T08:10:51.186 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_debug.ko 2024-04-30T08:10:51.186 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-30T08:10:51.186 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-30T08:10:51.186 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-30T08:10:51.186 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-30T08:10:51.186 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-30T08:10:51.186 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ses.ko 2024-04-30T08:10:51.186 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/st.ko 2024-04-30T08:10:51.186 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/stex.ko 2024-04-30T08:10:51.186 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2 2024-04-30T08:10:51.187 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-30T08:10:51.187 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/virtio_scsi.ko 2024-04-30T08:10:51.187 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-30T08:10:51.187 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/wd719x.ko 2024-04-30T08:10:51.187 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb 2024-04-30T08:10:51.187 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb/ssb.ko 2024-04-30T08:10:51.187 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target 2024-04-30T08:10:51.187 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi 2024-04-30T08:10:51.187 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-30T08:10:51.187 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback 2024-04-30T08:10:51.187 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-30T08:10:51.187 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_file.ko 2024-04-30T08:10:51.187 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_iblock.ko 2024-04-30T08:10:51.188 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_mod.ko 2024-04-30T08:10:51.188 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_pscsi.ko 2024-04-30T08:10:51.188 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_user.ko 2024-04-30T08:10:51.188 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc 2024-04-30T08:10:51.188 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-30T08:10:51.188 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal 2024-04-30T08:10:51.188 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel 2024-04-30T08:10:51.188 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal 2024-04-30T08:10:51.188 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-30T08:10:51.188 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-30T08:10:51.188 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-30T08:10:51.188 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-30T08:10:51.188 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-30T08:10:51.189 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-30T08:10:51.189 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-30T08:10:51.189 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-30T08:10:51.189 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-30T08:10:51.189 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-30T08:10:51.189 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-30T08:10:51.189 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-30T08:10:51.189 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-30T08:10:51.189 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-30T08:10:51.189 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-30T08:10:51.189 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-30T08:10:51.189 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-30T08:10:51.189 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-30T08:10:51.190 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty 2024-04-30T08:10:51.190 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless 2024-04-30T08:10:51.190 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-30T08:10:51.190 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_gsm.ko 2024-04-30T08:10:51.190 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_hdlc.ko 2024-04-30T08:10:51.190 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/nozomi.ko 2024-04-30T08:10:51.190 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial 2024-04-30T08:10:51.190 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250 2024-04-30T08:10:51.190 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-30T08:10:51.190 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/arc_uart.ko 2024-04-30T08:10:51.190 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm 2024-04-30T08:10:51.190 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-30T08:10:51.191 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs 2024-04-30T08:10:51.191 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core 2024-04-30T08:10:51.191 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-30T08:10:51.191 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host 2024-04-30T08:10:51.191 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-30T08:10:51.191 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio 2024-04-30T08:10:51.191 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio.ko 2024-04-30T08:10:51.191 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_aec.ko 2024-04-30T08:10:51.191 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_cif.ko 2024-04-30T08:10:51.191 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_pci_generic.ko 2024-04-30T08:10:51.191 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_sercos3.ko 2024-04-30T08:10:51.191 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb 2024-04-30T08:10:51.192 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class 2024-04-30T08:10:51.192 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-acm.ko 2024-04-30T08:10:51.192 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-30T08:10:51.192 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usblp.ko 2024-04-30T08:10:51.192 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usbtmc.ko 2024-04-30T08:10:51.192 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host 2024-04-30T08:10:51.192 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-30T08:10:51.192 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image 2024-04-30T08:10:51.192 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/mdc800.ko 2024-04-30T08:10:51.192 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/microtek.ko 2024-04-30T08:10:51.192 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc 2024-04-30T08:10:51.192 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/adutux.ko 2024-04-30T08:10:51.192 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/appledisplay.ko 2024-04-30T08:10:51.192 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi26.ko 2024-04-30T08:10:51.193 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi62.ko 2024-04-30T08:10:51.193 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ezusb.ko 2024-04-30T08:10:51.193 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/idmouse.ko 2024-04-30T08:10:51.193 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/iowarrior.ko 2024-04-30T08:10:51.193 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-30T08:10:51.193 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ldusb.ko 2024-04-30T08:10:51.193 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/legousbtower.ko 2024-04-30T08:10:51.193 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga 2024-04-30T08:10:51.193 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-30T08:10:51.193 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-30T08:10:51.193 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usb3503.ko 2024-04-30T08:10:51.193 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usblcd.ko 2024-04-30T08:10:51.193 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-30T08:10:51.193 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/uss720.ko 2024-04-30T08:10:51.194 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/yurex.ko 2024-04-30T08:10:51.194 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy 2024-04-30T08:10:51.194 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy/phy-generic.ko 2024-04-30T08:10:51.194 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial 2024-04-30T08:10:51.194 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/aircable.ko 2024-04-30T08:10:51.194 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ark3116.ko 2024-04-30T08:10:51.194 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-30T08:10:51.194 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ch341.ko 2024-04-30T08:10:51.194 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cp210x.ko 2024-04-30T08:10:51.194 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cyberjack.ko 2024-04-30T08:10:51.194 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-30T08:10:51.194 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-30T08:10:51.194 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/empeg.ko 2024-04-30T08:10:51.195 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-30T08:10:51.195 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-30T08:10:51.195 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-30T08:10:51.195 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_ti.ko 2024-04-30T08:10:51.195 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipaq.ko 2024-04-30T08:10:51.195 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipw.ko 2024-04-30T08:10:51.195 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ir-usb.ko 2024-04-30T08:10:51.195 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-30T08:10:51.195 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan.ko 2024-04-30T08:10:51.195 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-30T08:10:51.195 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-30T08:10:51.195 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-30T08:10:51.195 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mct_u232.ko 2024-04-30T08:10:51.195 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7720.ko 2024-04-30T08:10:51.196 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7840.ko 2024-04-30T08:10:51.196 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/navman.ko 2024-04-30T08:10:51.196 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/omninet.ko 2024-04-30T08:10:51.196 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/opticon.ko 2024-04-30T08:10:51.196 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/option.ko 2024-04-30T08:10:51.196 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/oti6858.ko 2024-04-30T08:10:51.196 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/pl2303.ko 2024-04-30T08:10:51.196 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcaux.ko 2024-04-30T08:10:51.196 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcserial.ko 2024-04-30T08:10:51.196 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/quatech2.ko 2024-04-30T08:10:51.196 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/safe_serial.ko 2024-04-30T08:10:51.196 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/sierra.ko 2024-04-30T08:10:51.196 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-30T08:10:51.197 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ssu100.ko 2024-04-30T08:10:51.197 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/symbolserial.ko 2024-04-30T08:10:51.197 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-30T08:10:51.197 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-30T08:10:51.197 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_debug.ko 2024-04-30T08:10:51.197 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-30T08:10:51.197 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/visor.ko 2024-04-30T08:10:51.197 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/whiteheat.ko 2024-04-30T08:10:51.197 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-30T08:10:51.197 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage 2024-04-30T08:10:51.197 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/uas.ko 2024-04-30T08:10:51.197 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-30T08:10:51.197 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-30T08:10:51.197 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-30T08:10:51.198 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-30T08:10:51.198 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-30T08:10:51.198 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-30T08:10:51.198 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-30T08:10:51.198 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-karma.ko 2024-04-30T08:10:51.198 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-30T08:10:51.198 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-30T08:10:51.198 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-30T08:10:51.198 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-30T08:10:51.198 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-30T08:10:51.198 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/usb-storage.ko 2024-04-30T08:10:51.198 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip 2024-04-30T08:10:51.199 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-30T08:10:51.199 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-30T08:10:51.199 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-30T08:10:51.199 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio 2024-04-30T08:10:51.199 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci 2024-04-30T08:10:51.199 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-30T08:10:51.199 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-30T08:10:51.199 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio.ko 2024-04-30T08:10:51.199 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-30T08:10:51.199 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost 2024-04-30T08:10:51.199 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost.ko 2024-04-30T08:10:51.199 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-30T08:10:51.200 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_net.ko 2024-04-30T08:10:51.200 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_scsi.ko 2024-04-30T08:10:51.200 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video 2024-04-30T08:10:51.200 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight 2024-04-30T08:10:51.200 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/apple_bl.ko 2024-04-30T08:10:51.200 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/lcd.ko 2024-04-30T08:10:51.200 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-30T08:10:51.200 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio 2024-04-30T08:10:51.200 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio.ko 2024-04-30T08:10:51.200 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_balloon.ko 2024-04-30T08:10:51.200 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_mmio.ko 2024-04-30T08:10:51.200 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci.ko 2024-04-30T08:10:51.200 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-30T08:10:51.201 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-30T08:10:51.201 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_ring.ko 2024-04-30T08:10:51.201 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1 2024-04-30T08:10:51.201 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters 2024-04-30T08:10:51.201 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2482.ko 2024-04-30T08:10:51.201 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2490.ko 2024-04-30T08:10:51.201 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves 2024-04-30T08:10:51.201 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-30T08:10:51.201 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-30T08:10:51.201 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-30T08:10:51.201 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-30T08:10:51.201 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-30T08:10:51.201 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-30T08:10:51.202 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-30T08:10:51.202 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-30T08:10:51.202 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-30T08:10:51.202 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-30T08:10:51.202 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-30T08:10:51.202 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/wire.ko 2024-04-30T08:10:51.202 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog 2024-04-30T08:10:51.202 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-30T08:10:51.202 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-30T08:10:51.202 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-30T08:10:51.202 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/hpwdt.ko 2024-04-30T08:10:51.202 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/i6300esb.ko 2024-04-30T08:10:51.202 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-30T08:10:51.203 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-30T08:10:51.203 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ib700wdt.ko 2024-04-30T08:10:51.203 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ibmasr.ko 2024-04-30T08:10:51.203 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-30T08:10:51.203 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-30T08:10:51.203 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it87_wdt.ko 2024-04-30T08:10:51.203 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/machzwd.ko 2024-04-30T08:10:51.203 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/nv_tco.ko 2024-04-30T08:10:51.203 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-30T08:10:51.203 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-30T08:10:51.203 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-30T08:10:51.203 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-30T08:10:51.204 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/softdog.ko 2024-04-30T08:10:51.204 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-30T08:10:51.204 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/via_wdt.ko 2024-04-30T08:10:51.204 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-30T08:10:51.204 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-30T08:10:51.204 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-30T08:10:51.204 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/wdt_pci.ko 2024-04-30T08:10:51.204 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/xen_wdt.ko 2024-04-30T08:10:51.204 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen 2024-04-30T08:10:51.204 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-30T08:10:51.204 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-evtchn.ko 2024-04-30T08:10:51.204 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntalloc.ko 2024-04-30T08:10:51.204 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntdev.ko 2024-04-30T08:10:51.204 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback 2024-04-30T08:10:51.204 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-30T08:10:51.205 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-privcmd.ko 2024-04-30T08:10:51.205 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs 2024-04-30T08:10:51.205 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-30T08:10:51.205 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs 2024-04-30T08:10:51.205 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p 2024-04-30T08:10:51.205 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p/9p.ko 2024-04-30T08:10:51.205 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs 2024-04-30T08:10:51.205 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs/affs.ko 2024-04-30T08:10:51.205 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs 2024-04-30T08:10:51.205 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs/befs.ko 2024-04-30T08:10:51.205 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/binfmt_misc.ko 2024-04-30T08:10:51.205 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs 2024-04-30T08:10:51.205 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs/btrfs.ko 2024-04-30T08:10:51.206 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles 2024-04-30T08:10:51.206 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles/cachefiles.ko 2024-04-30T08:10:51.206 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph 2024-04-30T08:10:51.206 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph/ceph.ko 2024-04-30T08:10:51.206 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda 2024-04-30T08:10:51.206 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda/coda.ko 2024-04-30T08:10:51.206 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs 2024-04-30T08:10:51.206 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs/cramfs.ko 2024-04-30T08:10:51.206 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm 2024-04-30T08:10:51.206 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm/dlm.ko 2024-04-30T08:10:51.206 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs 2024-04-30T08:10:51.206 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-30T08:10:51.206 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs 2024-04-30T08:10:51.206 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs/f2fs.ko 2024-04-30T08:10:51.207 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat 2024-04-30T08:10:51.207 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/fat.ko 2024-04-30T08:10:51.207 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/msdos.ko 2024-04-30T08:10:51.207 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/vfat.ko 2024-04-30T08:10:51.207 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse 2024-04-30T08:10:51.207 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/cuse.ko 2024-04-30T08:10:51.207 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/fuse.ko 2024-04-30T08:10:51.207 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2 2024-04-30T08:10:51.207 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2/gfs2.ko 2024-04-30T08:10:51.207 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs 2024-04-30T08:10:51.207 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs/hfs.ko 2024-04-30T08:10:51.207 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus 2024-04-30T08:10:51.207 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus/hfsplus.ko 2024-04-30T08:10:51.208 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs 2024-04-30T08:10:51.208 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs/isofs.ko 2024-04-30T08:10:51.208 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs 2024-04-30T08:10:51.208 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs/jfs.ko 2024-04-30T08:10:51.208 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd 2024-04-30T08:10:51.208 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd/lockd.ko 2024-04-30T08:10:51.208 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix 2024-04-30T08:10:51.208 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix/minix.ko 2024-04-30T08:10:51.208 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs 2024-04-30T08:10:51.208 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs/netfs.ko 2024-04-30T08:10:51.208 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs 2024-04-30T08:10:51.208 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout 2024-04-30T08:10:51.208 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-30T08:10:51.208 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout 2024-04-30T08:10:51.209 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-30T08:10:51.209 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout 2024-04-30T08:10:51.209 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-30T08:10:51.209 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfs.ko 2024-04-30T08:10:51.209 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv3.ko 2024-04-30T08:10:51.209 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv4.ko 2024-04-30T08:10:51.209 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common 2024-04-30T08:10:51.209 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/grace.ko 2024-04-30T08:10:51.209 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/nfs_acl.ko 2024-04-30T08:10:51.209 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd 2024-04-30T08:10:51.209 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd/nfsd.ko 2024-04-30T08:10:51.209 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2 2024-04-30T08:10:51.209 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2/nilfs2.ko 2024-04-30T08:10:51.209 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls 2024-04-30T08:10:51.210 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-celtic.ko 2024-04-30T08:10:51.210 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-centeuro.ko 2024-04-30T08:10:51.210 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-croatian.ko 2024-04-30T08:10:51.210 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-cyrillic.ko 2024-04-30T08:10:51.210 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-gaelic.ko 2024-04-30T08:10:51.210 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-greek.ko 2024-04-30T08:10:51.210 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-iceland.ko 2024-04-30T08:10:51.210 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-inuit.ko 2024-04-30T08:10:51.210 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-roman.ko 2024-04-30T08:10:51.210 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-romanian.ko 2024-04-30T08:10:51.210 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-turkish.ko 2024-04-30T08:10:51.210 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1250.ko 2024-04-30T08:10:51.210 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1251.ko 2024-04-30T08:10:51.210 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1255.ko 2024-04-30T08:10:51.211 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp737.ko 2024-04-30T08:10:51.211 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp775.ko 2024-04-30T08:10:51.211 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp850.ko 2024-04-30T08:10:51.211 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp852.ko 2024-04-30T08:10:51.211 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp855.ko 2024-04-30T08:10:51.211 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp857.ko 2024-04-30T08:10:51.211 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp860.ko 2024-04-30T08:10:51.211 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp861.ko 2024-04-30T08:10:51.211 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp862.ko 2024-04-30T08:10:51.211 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp863.ko 2024-04-30T08:10:51.211 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp864.ko 2024-04-30T08:10:51.211 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp865.ko 2024-04-30T08:10:51.211 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp866.ko 2024-04-30T08:10:51.212 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp869.ko 2024-04-30T08:10:51.212 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp874.ko 2024-04-30T08:10:51.212 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp932.ko 2024-04-30T08:10:51.212 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp936.ko 2024-04-30T08:10:51.212 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp949.ko 2024-04-30T08:10:51.212 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp950.ko 2024-04-30T08:10:51.212 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_euc-jp.ko 2024-04-30T08:10:51.212 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-1.ko 2024-04-30T08:10:51.212 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-13.ko 2024-04-30T08:10:51.212 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-14.ko 2024-04-30T08:10:51.212 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-15.ko 2024-04-30T08:10:51.212 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-2.ko 2024-04-30T08:10:51.212 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-3.ko 2024-04-30T08:10:51.212 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-4.ko 2024-04-30T08:10:51.213 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-5.ko 2024-04-30T08:10:51.213 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-6.ko 2024-04-30T08:10:51.213 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-7.ko 2024-04-30T08:10:51.213 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-9.ko 2024-04-30T08:10:51.213 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-r.ko 2024-04-30T08:10:51.213 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-ru.ko 2024-04-30T08:10:51.213 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-u.ko 2024-04-30T08:10:51.213 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-30T08:10:51.213 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_utf8.ko 2024-04-30T08:10:51.213 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2 2024-04-30T08:10:51.213 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster 2024-04-30T08:10:51.213 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-30T08:10:51.213 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm 2024-04-30T08:10:51.213 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-30T08:10:51.214 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs 2024-04-30T08:10:51.214 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-30T08:10:51.214 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2.ko 2024-04-30T08:10:51.214 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-30T08:10:51.214 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-30T08:10:51.214 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-30T08:10:51.214 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs 2024-04-30T08:10:51.214 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs/overlay.ko 2024-04-30T08:10:51.214 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore 2024-04-30T08:10:51.214 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore/ramoops.ko 2024-04-30T08:10:51.214 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs 2024-04-30T08:10:51.214 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs/reiserfs.ko 2024-04-30T08:10:51.214 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs 2024-04-30T08:10:51.214 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs/romfs.ko 2024-04-30T08:10:51.215 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb 2024-04-30T08:10:51.215 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client 2024-04-30T08:10:51.215 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client/cifs.ko 2024-04-30T08:10:51.215 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common 2024-04-30T08:10:51.215 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_arc4.ko 2024-04-30T08:10:51.215 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_md4.ko 2024-04-30T08:10:51.215 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs 2024-04-30T08:10:51.215 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs/squashfs.ko 2024-04-30T08:10:51.215 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv 2024-04-30T08:10:51.215 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv/sysv.ko 2024-04-30T08:10:51.215 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf 2024-04-30T08:10:51.215 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf/udf.ko 2024-04-30T08:10:51.215 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs 2024-04-30T08:10:51.215 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs/ufs.ko 2024-04-30T08:10:51.216 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs 2024-04-30T08:10:51.216 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs/xfs.ko 2024-04-30T08:10:51.216 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel 2024-04-30T08:10:51.216 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu 2024-04-30T08:10:51.216 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu/rcutorture.ko 2024-04-30T08:10:51.216 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/torture.ko 2024-04-30T08:10:51.216 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace 2024-04-30T08:10:51.216 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-30T08:10:51.216 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib 2024-04-30T08:10:51.216 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/asn1_encoder.ko 2024-04-30T08:10:51.216 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc-itu-t.ko 2024-04-30T08:10:51.216 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc8.ko 2024-04-30T08:10:51.216 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto 2024-04-30T08:10:51.216 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libarc4.ko 2024-04-30T08:10:51.217 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libdes.ko 2024-04-30T08:10:51.217 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/libcrc32c.ko 2024-04-30T08:10:51.217 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lru_cache.ko 2024-04-30T08:10:51.217 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4 2024-04-30T08:10:51.217 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4_compress.ko 2024-04-30T08:10:51.217 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4hc_compress.ko 2024-04-30T08:10:51.217 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math 2024-04-30T08:10:51.217 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math/cordic.ko 2024-04-30T08:10:51.217 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6 2024-04-30T08:10:51.217 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6/raid6_pq.ko 2024-04-30T08:10:51.217 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon 2024-04-30T08:10:51.217 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-30T08:10:51.217 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_bm.ko 2024-04-30T08:10:51.217 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_fsm.ko 2024-04-30T08:10:51.218 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_kmp.ko 2024-04-30T08:10:51.218 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd 2024-04-30T08:10:51.218 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd/zstd_compress.ko 2024-04-30T08:10:51.218 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm 2024-04-30T08:10:51.218 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm/hwpoison-inject.ko 2024-04-30T08:10:51.218 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net 2024-04-30T08:10:51.218 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan 2024-04-30T08:10:51.218 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/6lowpan.ko 2024-04-30T08:10:51.218 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_dest.ko 2024-04-30T08:10:51.218 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_fragment.ko 2024-04-30T08:10:51.218 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_hop.ko 2024-04-30T08:10:51.218 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-30T08:10:51.218 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_mobility.ko 2024-04-30T08:10:51.219 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_routing.ko 2024-04-30T08:10:51.219 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_udp.ko 2024-04-30T08:10:51.219 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802 2024-04-30T08:10:51.219 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/garp.ko 2024-04-30T08:10:51.219 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/mrp.ko 2024-04-30T08:10:51.219 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/p8022.ko 2024-04-30T08:10:51.219 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/psnap.ko 2024-04-30T08:10:51.219 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/stp.ko 2024-04-30T08:10:51.219 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q 2024-04-30T08:10:51.219 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q/8021q.ko 2024-04-30T08:10:51.219 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p 2024-04-30T08:10:51.219 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet.ko 2024-04-30T08:10:51.219 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_fd.ko 2024-04-30T08:10:51.219 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_rdma.ko 2024-04-30T08:10:51.220 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_virtio.ko 2024-04-30T08:10:51.220 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv 2024-04-30T08:10:51.220 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv/batman-adv.ko 2024-04-30T08:10:51.220 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge 2024-04-30T08:10:51.220 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/br_netfilter.ko 2024-04-30T08:10:51.220 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/bridge.ko 2024-04-30T08:10:51.220 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter 2024-04-30T08:10:51.220 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-30T08:10:51.220 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-30T08:10:51.220 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-30T08:10:51.220 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-30T08:10:51.220 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-30T08:10:51.220 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-30T08:10:51.220 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-30T08:10:51.221 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-30T08:10:51.221 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-30T08:10:51.221 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-30T08:10:51.221 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-30T08:10:51.221 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-30T08:10:51.221 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-30T08:10:51.221 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-30T08:10:51.221 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-30T08:10:51.221 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-30T08:10:51.221 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-30T08:10:51.221 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-30T08:10:51.221 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-30T08:10:51.222 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-30T08:10:51.222 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtables.ko 2024-04-30T08:10:51.222 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-30T08:10:51.222 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-30T08:10:51.222 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-30T08:10:51.222 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph 2024-04-30T08:10:51.222 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph/libceph.ko 2024-04-30T08:10:51.222 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core 2024-04-30T08:10:51.222 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/failover.ko 2024-04-30T08:10:51.222 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/pktgen.ko 2024-04-30T08:10:51.222 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp 2024-04-30T08:10:51.222 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp.ko 2024-04-30T08:10:51.222 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_diag.ko 2024-04-30T08:10:51.223 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv4.ko 2024-04-30T08:10:51.223 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv6.ko 2024-04-30T08:10:51.223 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver 2024-04-30T08:10:51.223 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver/dns_resolver.ko 2024-04-30T08:10:51.223 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa 2024-04-30T08:10:51.223 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/dsa_core.ko 2024-04-30T08:10:51.223 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_brcm.ko 2024-04-30T08:10:51.223 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_dsa.ko 2024-04-30T08:10:51.223 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_none.ko 2024-04-30T08:10:51.223 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_trailer.ko 2024-04-30T08:10:51.223 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154 2024-04-30T08:10:51.223 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan 2024-04-30T08:10:51.223 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-30T08:10:51.223 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154.ko 2024-04-30T08:10:51.223 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-30T08:10:51.224 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4 2024-04-30T08:10:51.224 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ah4.ko 2024-04-30T08:10:51.224 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/esp4.ko 2024-04-30T08:10:51.224 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/fou.ko 2024-04-30T08:10:51.224 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/gre.ko 2024-04-30T08:10:51.224 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/inet_diag.ko 2024-04-30T08:10:51.224 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_gre.ko 2024-04-30T08:10:51.224 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_tunnel.ko 2024-04-30T08:10:51.224 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_vti.ko 2024-04-30T08:10:51.224 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipcomp.ko 2024-04-30T08:10:51.224 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipip.ko 2024-04-30T08:10:51.224 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter 2024-04-30T08:10:51.224 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-30T08:10:51.224 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-30T08:10:51.225 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-30T08:10:51.225 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-30T08:10:51.225 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-30T08:10:51.225 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-30T08:10:51.225 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-30T08:10:51.225 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-30T08:10:51.225 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-30T08:10:51.225 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-30T08:10:51.225 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-30T08:10:51.225 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-30T08:10:51.225 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-30T08:10:51.225 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-30T08:10:51.225 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-30T08:10:51.226 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-30T08:10:51.226 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-30T08:10:51.226 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-30T08:10:51.226 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-30T08:10:51.226 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-30T08:10:51.226 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-30T08:10:51.226 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_bic.ko 2024-04-30T08:10:51.226 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_dctcp.ko 2024-04-30T08:10:51.226 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_diag.ko 2024-04-30T08:10:51.226 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_highspeed.ko 2024-04-30T08:10:51.226 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_htcp.ko 2024-04-30T08:10:51.226 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_hybla.ko 2024-04-30T08:10:51.226 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_illinois.ko 2024-04-30T08:10:51.227 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_lp.ko 2024-04-30T08:10:51.227 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_scalable.ko 2024-04-30T08:10:51.227 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_vegas.ko 2024-04-30T08:10:51.227 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_veno.ko 2024-04-30T08:10:51.227 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_westwood.ko 2024-04-30T08:10:51.227 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_yeah.ko 2024-04-30T08:10:51.227 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tunnel4.ko 2024-04-30T08:10:51.227 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_diag.ko 2024-04-30T08:10:51.227 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_tunnel.ko 2024-04-30T08:10:51.227 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-30T08:10:51.227 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6 2024-04-30T08:10:51.227 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ah6.ko 2024-04-30T08:10:51.227 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/esp6.ko 2024-04-30T08:10:51.227 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/fou6.ko 2024-04-30T08:10:51.228 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_tunnel.ko 2024-04-30T08:10:51.228 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-30T08:10:51.228 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_vti.ko 2024-04-30T08:10:51.228 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ipcomp6.ko 2024-04-30T08:10:51.228 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter 2024-04-30T08:10:51.228 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-30T08:10:51.228 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-30T08:10:51.228 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-30T08:10:51.228 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-30T08:10:51.228 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-30T08:10:51.228 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-30T08:10:51.228 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-30T08:10:51.228 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-30T08:10:51.229 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-30T08:10:51.229 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-30T08:10:51.229 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-30T08:10:51.229 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-30T08:10:51.229 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-30T08:10:51.229 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-30T08:10:51.229 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-30T08:10:51.229 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-30T08:10:51.229 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-30T08:10:51.229 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-30T08:10:51.229 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-30T08:10:51.229 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-30T08:10:51.229 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-30T08:10:51.229 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-30T08:10:51.229 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-30T08:10:51.230 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-30T08:10:51.230 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/sit.ko 2024-04-30T08:10:51.230 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/tunnel6.ko 2024-04-30T08:10:51.230 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-30T08:10:51.230 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key 2024-04-30T08:10:51.230 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key/af_key.ko 2024-04-30T08:10:51.230 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp 2024-04-30T08:10:51.230 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_core.ko 2024-04-30T08:10:51.230 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-30T08:10:51.230 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_eth.ko 2024-04-30T08:10:51.230 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip.ko 2024-04-30T08:10:51.230 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip6.ko 2024-04-30T08:10:51.230 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_netlink.ko 2024-04-30T08:10:51.231 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ppp.ko 2024-04-30T08:10:51.231 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc 2024-04-30T08:10:51.231 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc/llc.ko 2024-04-30T08:10:51.231 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154 2024-04-30T08:10:51.231 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154/mac802154.ko 2024-04-30T08:10:51.231 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls 2024-04-30T08:10:51.231 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls/mpls_gso.ko 2024-04-30T08:10:51.231 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter 2024-04-30T08:10:51.231 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset 2024-04-30T08:10:51.231 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set.ko 2024-04-30T08:10:51.231 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-30T08:10:51.231 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-30T08:10:51.231 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-30T08:10:51.231 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-30T08:10:51.232 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-30T08:10:51.232 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-30T08:10:51.232 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-30T08:10:51.232 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-30T08:10:51.232 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-30T08:10:51.232 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-30T08:10:51.232 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-30T08:10:51.232 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-30T08:10:51.232 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-30T08:10:51.232 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-30T08:10:51.232 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-30T08:10:51.232 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs 2024-04-30T08:10:51.232 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-30T08:10:51.232 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-30T08:10:51.233 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-30T08:10:51.233 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-30T08:10:51.233 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-30T08:10:51.233 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-30T08:10:51.233 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-30T08:10:51.233 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-30T08:10:51.233 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-30T08:10:51.233 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-30T08:10:51.233 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-30T08:10:51.233 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-30T08:10:51.233 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-30T08:10:51.233 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-30T08:10:51.233 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conncount.ko 2024-04-30T08:10:51.233 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack.ko 2024-04-30T08:10:51.234 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-30T08:10:51.234 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-30T08:10:51.234 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-30T08:10:51.234 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-30T08:10:51.234 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-30T08:10:51.234 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-30T08:10:51.234 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-30T08:10:51.234 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-30T08:10:51.234 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-30T08:10:51.234 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-30T08:10:51.234 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-30T08:10:51.234 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-30T08:10:51.234 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-30T08:10:51.234 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table.ko 2024-04-30T08:10:51.235 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-30T08:10:51.235 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_log_syslog.ko 2024-04-30T08:10:51.235 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat.ko 2024-04-30T08:10:51.235 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-30T08:10:51.235 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-30T08:10:51.235 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_irc.ko 2024-04-30T08:10:51.235 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_sip.ko 2024-04-30T08:10:51.235 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-30T08:10:51.235 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-30T08:10:51.235 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_tables.ko 2024-04-30T08:10:51.235 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink.ko 2024-04-30T08:10:51.235 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-30T08:10:51.235 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_log.ko 2024-04-30T08:10:51.235 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-30T08:10:51.236 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-30T08:10:51.236 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_chain_nat.ko 2024-04-30T08:10:51.236 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_compat.ko 2024-04-30T08:10:51.236 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_ct.ko 2024-04-30T08:10:51.236 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-30T08:10:51.236 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fib.ko 2024-04-30T08:10:51.236 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-30T08:10:51.236 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_hash.ko 2024-04-30T08:10:51.236 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_limit.ko 2024-04-30T08:10:51.236 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_log.ko 2024-04-30T08:10:51.236 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_masq.ko 2024-04-30T08:10:51.236 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_nat.ko 2024-04-30T08:10:51.236 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_numgen.ko 2024-04-30T08:10:51.236 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_queue.ko 2024-04-30T08:10:51.237 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_quota.ko 2024-04-30T08:10:51.237 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_redir.ko 2024-04-30T08:10:51.237 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject.ko 2024-04-30T08:10:51.237 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject_inet.ko 2024-04-30T08:10:51.237 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_socket.ko 2024-04-30T08:10:51.237 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_tunnel.ko 2024-04-30T08:10:51.237 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_AUDIT.ko 2024-04-30T08:10:51.237 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-30T08:10:51.237 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-30T08:10:51.237 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-30T08:10:51.237 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CT.ko 2024-04-30T08:10:51.237 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_DSCP.ko 2024-04-30T08:10:51.237 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HL.ko 2024-04-30T08:10:51.237 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HMARK.ko 2024-04-30T08:10:51.238 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-30T08:10:51.238 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LED.ko 2024-04-30T08:10:51.238 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LOG.ko 2024-04-30T08:10:51.238 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-30T08:10:51.238 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NETMAP.ko 2024-04-30T08:10:51.238 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFLOG.ko 2024-04-30T08:10:51.238 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-30T08:10:51.238 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_RATEEST.ko 2024-04-30T08:10:51.238 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-30T08:10:51.238 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_SECMARK.ko 2024-04-30T08:10:51.238 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-30T08:10:51.238 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-30T08:10:51.238 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TEE.ko 2024-04-30T08:10:51.238 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TPROXY.ko 2024-04-30T08:10:51.239 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TRACE.ko 2024-04-30T08:10:51.239 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_addrtype.ko 2024-04-30T08:10:51.239 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_bpf.ko 2024-04-30T08:10:51.239 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cgroup.ko 2024-04-30T08:10:51.239 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cluster.ko 2024-04-30T08:10:51.239 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_comment.ko 2024-04-30T08:10:51.239 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connbytes.ko 2024-04-30T08:10:51.239 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlabel.ko 2024-04-30T08:10:51.239 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlimit.ko 2024-04-30T08:10:51.239 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connmark.ko 2024-04-30T08:10:51.239 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_conntrack.ko 2024-04-30T08:10:51.239 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cpu.ko 2024-04-30T08:10:51.239 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dccp.ko 2024-04-30T08:10:51.239 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_devgroup.ko 2024-04-30T08:10:51.239 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dscp.ko 2024-04-30T08:10:51.240 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ecn.ko 2024-04-30T08:10:51.240 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_esp.ko 2024-04-30T08:10:51.240 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hashlimit.ko 2024-04-30T08:10:51.240 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_helper.ko 2024-04-30T08:10:51.240 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hl.ko 2024-04-30T08:10:51.240 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipcomp.ko 2024-04-30T08:10:51.240 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_iprange.ko 2024-04-30T08:10:51.240 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipvs.ko 2024-04-30T08:10:51.240 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_l2tp.ko 2024-04-30T08:10:51.240 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_length.ko 2024-04-30T08:10:51.240 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_limit.ko 2024-04-30T08:10:51.240 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mac.ko 2024-04-30T08:10:51.241 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mark.ko 2024-04-30T08:10:51.241 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_multiport.ko 2024-04-30T08:10:51.241 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nat.ko 2024-04-30T08:10:51.241 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nfacct.ko 2024-04-30T08:10:51.241 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_osf.ko 2024-04-30T08:10:51.241 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_owner.ko 2024-04-30T08:10:51.241 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_physdev.ko 2024-04-30T08:10:51.241 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_pkttype.ko 2024-04-30T08:10:51.241 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_policy.ko 2024-04-30T08:10:51.241 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_quota.ko 2024-04-30T08:10:51.241 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_rateest.ko 2024-04-30T08:10:51.241 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_realm.ko 2024-04-30T08:10:51.241 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_recent.ko 2024-04-30T08:10:51.241 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_sctp.ko 2024-04-30T08:10:51.242 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_set.ko 2024-04-30T08:10:51.242 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_socket.ko 2024-04-30T08:10:51.242 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_state.ko 2024-04-30T08:10:51.242 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_statistic.ko 2024-04-30T08:10:51.242 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_string.ko 2024-04-30T08:10:51.242 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_tcpmss.ko 2024-04-30T08:10:51.242 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_time.ko 2024-04-30T08:10:51.242 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_u32.ko 2024-04-30T08:10:51.242 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink 2024-04-30T08:10:51.242 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink/netlink_diag.ko 2024-04-30T08:10:51.242 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh 2024-04-30T08:10:51.242 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh/nsh.ko 2024-04-30T08:10:51.242 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch 2024-04-30T08:10:51.243 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/openvswitch.ko 2024-04-30T08:10:51.243 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-geneve.ko 2024-04-30T08:10:51.243 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-gre.ko 2024-04-30T08:10:51.243 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-vxlan.ko 2024-04-30T08:10:51.243 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet 2024-04-30T08:10:51.243 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet/af_packet_diag.ko 2024-04-30T08:10:51.243 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds 2024-04-30T08:10:51.243 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds.ko 2024-04-30T08:10:51.243 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_rdma.ko 2024-04-30T08:10:51.243 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_tcp.ko 2024-04-30T08:10:51.243 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched 2024-04-30T08:10:51.243 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_csum.ko 2024-04-30T08:10:51.243 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_gact.ko 2024-04-30T08:10:51.243 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_mirred.ko 2024-04-30T08:10:51.244 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_nat.ko 2024-04-30T08:10:51.244 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_pedit.ko 2024-04-30T08:10:51.244 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_police.ko 2024-04-30T08:10:51.244 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_simple.ko 2024-04-30T08:10:51.244 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_skbedit.ko 2024-04-30T08:10:51.244 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_vlan.ko 2024-04-30T08:10:51.244 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_basic.ko 2024-04-30T08:10:51.244 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_bpf.ko 2024-04-30T08:10:51.244 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_flow.ko 2024-04-30T08:10:51.244 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_fw.ko 2024-04-30T08:10:51.244 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_route.ko 2024-04-30T08:10:51.244 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_u32.ko 2024-04-30T08:10:51.244 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_cmp.ko 2024-04-30T08:10:51.244 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_ipset.ko 2024-04-30T08:10:51.245 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_meta.ko 2024-04-30T08:10:51.245 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_nbyte.ko 2024-04-30T08:10:51.245 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_text.ko 2024-04-30T08:10:51.245 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_u32.ko 2024-04-30T08:10:51.245 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_choke.ko 2024-04-30T08:10:51.245 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_codel.ko 2024-04-30T08:10:51.245 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_drr.ko 2024-04-30T08:10:51.245 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_fq.ko 2024-04-30T08:10:51.245 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_gred.ko 2024-04-30T08:10:51.245 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hfsc.ko 2024-04-30T08:10:51.245 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hhf.ko 2024-04-30T08:10:51.245 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_htb.ko 2024-04-30T08:10:51.245 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_ingress.ko 2024-04-30T08:10:51.245 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio.ko 2024-04-30T08:10:51.245 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio_lib.ko 2024-04-30T08:10:51.246 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_multiq.ko 2024-04-30T08:10:51.246 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_netem.ko 2024-04-30T08:10:51.246 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_pie.ko 2024-04-30T08:10:51.246 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_plug.ko 2024-04-30T08:10:51.246 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_prio.ko 2024-04-30T08:10:51.246 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_qfq.ko 2024-04-30T08:10:51.246 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_red.ko 2024-04-30T08:10:51.246 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfb.ko 2024-04-30T08:10:51.246 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfq.ko 2024-04-30T08:10:51.246 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_tbf.ko 2024-04-30T08:10:51.246 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_teql.ko 2024-04-30T08:10:51.246 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp 2024-04-30T08:10:51.246 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp.ko 2024-04-30T08:10:51.246 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp_diag.ko 2024-04-30T08:10:51.247 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc 2024-04-30T08:10:51.247 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss 2024-04-30T08:10:51.247 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-30T08:10:51.247 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-30T08:10:51.247 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/sunrpc.ko 2024-04-30T08:10:51.247 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma 2024-04-30T08:10:51.247 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-30T08:10:51.247 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc 2024-04-30T08:10:51.247 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc.ko 2024-04-30T08:10:51.247 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc_diag.ko 2024-04-30T08:10:51.247 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix 2024-04-30T08:10:51.247 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix/unix_diag.ko 2024-04-30T08:10:51.247 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock 2024-04-30T08:10:51.247 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-30T08:10:51.248 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-30T08:10:51.248 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock.ko 2024-04-30T08:10:51.248 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-30T08:10:51.248 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-30T08:10:51.248 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm 2024-04-30T08:10:51.248 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-30T08:10:51.248 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security 2024-04-30T08:10:51.248 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys 2024-04-30T08:10:51.248 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys 2024-04-30T08:10:51.248 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-30T08:10:51.248 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys 2024-04-30T08:10:51.248 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys/trusted.ko 2024-04-30T08:10:51.248 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias 2024-04-30T08:10:51.248 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias.bin 2024-04-30T08:10:51.249 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin 2024-04-30T08:10:51.249 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.alias.bin 2024-04-30T08:10:51.249 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.bin 2024-04-30T08:10:51.249 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.modinfo 2024-04-30T08:10:51.249 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep 2024-04-30T08:10:51.249 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep.bin 2024-04-30T08:10:51.249 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.devname 2024-04-30T08:10:51.249 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.order 2024-04-30T08:10:51.249 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.softdep 2024-04-30T08:10:51.249 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols 2024-04-30T08:10:51.249 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols.bin 2024-04-30T08:10:51.249 INFO:teuthology.orchestra.run.smithi032.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-30T08:10:51.254 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g806ec36b2112 2024-04-30T08:10:51.254 DEBUG:teuthology.orchestra.run.smithi032:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-30T08:10:51.277 INFO:teuthology.orchestra.run.smithi032.stdout:postinstall scriptlet (using /bin/sh): 2024-04-30T08:10:51.277 INFO:teuthology.orchestra.run.smithi032.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-30T08:10:51.277 INFO:teuthology.orchestra.run.smithi032.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g806ec36b2112 /lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-30T08:10:51.277 INFO:teuthology.orchestra.run.smithi032.stdout:fi 2024-04-30T08:10:51.277 INFO:teuthology.orchestra.run.smithi032.stdout:for file in vmlinuz System.map config; do 2024-04-30T08:10:51.277 INFO:teuthology.orchestra.run.smithi032.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112"; then 2024-04-30T08:10:51.277 INFO:teuthology.orchestra.run.smithi032.stdout: cp "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112" 2024-04-30T08:10:51.277 INFO:teuthology.orchestra.run.smithi032.stdout: fi 2024-04-30T08:10:51.278 INFO:teuthology.orchestra.run.smithi032.stdout:done 2024-04-30T08:10:51.278 INFO:teuthology.orchestra.run.smithi032.stdout:if [ -d "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" ] && \ 2024-04-30T08:10:51.278 INFO:teuthology.orchestra.run.smithi032.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" >/dev/null 2>&1; then 2024-04-30T08:10:51.278 INFO:teuthology.orchestra.run.smithi032.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-30T08:10:51.278 INFO:teuthology.orchestra.run.smithi032.stdout: cp -r "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-30T08:10:51.278 INFO:teuthology.orchestra.run.smithi032.stdout:fi 2024-04-30T08:10:51.278 INFO:teuthology.orchestra.run.smithi032.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep" ]; then 2024-04-30T08:10:51.278 INFO:teuthology.orchestra.run.smithi032.stdout: /usr/sbin/depmod 6.9.0-rc3-g806ec36b2112 2024-04-30T08:10:51.278 INFO:teuthology.orchestra.run.smithi032.stdout:fi 2024-04-30T08:10:51.278 INFO:teuthology.orchestra.run.smithi032.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-30T08:10:51.278 INFO:teuthology.orchestra.run.smithi032.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-30T08:10:51.278 INFO:teuthology.orchestra.run.smithi032.stdout:kernel-install remove 6.9.0-rc3-g806ec36b2112 2024-04-30T08:10:51.278 INFO:teuthology.orchestra.run.smithi032.stdout:fi 2024-04-30T08:10:51.279 DEBUG:teuthology.orchestra.run.smithi032:> sudo rpm -qi grub2-tools 2024-04-30T08:10:51.353 INFO:teuthology.orchestra.run.smithi032.stdout:Name : grub2-tools 2024-04-30T08:10:51.353 INFO:teuthology.orchestra.run.smithi032.stdout:Epoch : 1 2024-04-30T08:10:51.353 INFO:teuthology.orchestra.run.smithi032.stdout:Version : 2.06 2024-04-30T08:10:51.353 INFO:teuthology.orchestra.run.smithi032.stdout:Release : 68.el9 2024-04-30T08:10:51.353 INFO:teuthology.orchestra.run.smithi032.stdout:Architecture: x86_64 2024-04-30T08:10:51.353 INFO:teuthology.orchestra.run.smithi032.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-04-30T08:10:51.353 INFO:teuthology.orchestra.run.smithi032.stdout:Group : Unspecified 2024-04-30T08:10:51.353 INFO:teuthology.orchestra.run.smithi032.stdout:Size : 8264254 2024-04-30T08:10:51.353 INFO:teuthology.orchestra.run.smithi032.stdout:License : GPLv3+ 2024-04-30T08:10:51.353 INFO:teuthology.orchestra.run.smithi032.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-30T08:10:51.354 INFO:teuthology.orchestra.run.smithi032.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-30T08:10:51.354 INFO:teuthology.orchestra.run.smithi032.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-30T08:10:51.354 INFO:teuthology.orchestra.run.smithi032.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-30T08:10:51.354 INFO:teuthology.orchestra.run.smithi032.stdout:Packager : builder@centos.org 2024-04-30T08:10:51.354 INFO:teuthology.orchestra.run.smithi032.stdout:Vendor : CentOS 2024-04-30T08:10:51.354 INFO:teuthology.orchestra.run.smithi032.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-30T08:10:51.354 INFO:teuthology.orchestra.run.smithi032.stdout:Summary : Support tools for GRUB. 2024-04-30T08:10:51.354 INFO:teuthology.orchestra.run.smithi032.stdout:Description : 2024-04-30T08:10:51.354 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:10:51.354 INFO:teuthology.orchestra.run.smithi032.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-30T08:10:51.354 INFO:teuthology.orchestra.run.smithi032.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-30T08:10:51.354 INFO:teuthology.orchestra.run.smithi032.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-30T08:10:51.354 INFO:teuthology.orchestra.run.smithi032.stdout:hardware devices. 2024-04-30T08:10:51.355 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:10:51.355 INFO:teuthology.orchestra.run.smithi032.stdout:This subpackage provides tools for support of all platforms. 2024-04-30T08:10:51.356 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-30T08:10:51.356 INFO:teuthology.task.kernel:Updating grub on smithi032 to boot 6.9.0-rc3-g806ec36b2112 2024-04-30T08:10:51.356 DEBUG:teuthology.orchestra.run.smithi032:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-30T08:10:52.105 INFO:teuthology.orchestra.run.smithi032.stderr:Generating grub configuration file ... 2024-04-30T08:10:53.478 INFO:teuthology.orchestra.run.smithi032.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-30T08:10:53.500 INFO:teuthology.orchestra.run.smithi032.stderr:done 2024-04-30T08:10:53.503 DEBUG:teuthology.orchestra.run.smithi032:> mktemp 2024-04-30T08:10:53.523 INFO:teuthology.orchestra.run.smithi032.stdout:/tmp/tmp.31Tx9vdHWY 2024-04-30T08:10:53.523 DEBUG:teuthology.orchestra.run.smithi032:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.31Tx9vdHWY 2024-04-30T08:10:53.589 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0666 /tmp/tmp.31Tx9vdHWY 2024-04-30T08:10:53.729 DEBUG:teuthology.orchestra.remote:smithi032:/tmp/tmp.31Tx9vdHWY is 6KB 2024-04-30T08:10:53.777 DEBUG:teuthology.orchestra.run.smithi032:> rm -fr /tmp/tmp.31Tx9vdHWY 2024-04-30T08:10:53.792 DEBUG:teuthology.orchestra.run.smithi032:> sudo /bin/ls /boot/loader/entries || true 2024-04-30T08:10:53.854 INFO:teuthology.orchestra.run.smithi032.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-04-30T08:10:53.854 INFO:teuthology.orchestra.run.smithi032.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-04-30T08:10:53.854 INFO:teuthology.orchestra.run.smithi032.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-04-30T08:10:53.854 INFO:teuthology.orchestra.run.smithi032.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112.conf 2024-04-30T08:10:53.855 DEBUG:teuthology.orchestra.run.smithi032:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112 2024-04-30T08:10:54.020 DEBUG:teuthology.orchestra.run.smithi032:> sudo shutdown -r now 2024-04-30T08:11:24.023 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-04-30T08:11:24.023 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-04-30T08:11:24.024 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-30T08:11:42.494 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.32 2024-04-30T08:11:51.495 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-04-30T08:11:51.496 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-30T08:12:09.886 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.32 2024-04-30T08:12:21.888 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-04-30T08:12:21.889 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-30T08:12:24.958 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.32 2024-04-30T08:12:39.959 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-04-30T08:12:39.960 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-30T08:12:40.243 DEBUG:teuthology.orchestra.run.smithi032:> true 2024-04-30T08:12:40.737 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-04-30T08:12:40.737 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.9.0-rc3-g806ec36b2112"... 2024-04-30T08:12:40.738 DEBUG:teuthology.orchestra.run.smithi032:> uname -r 2024-04-30T08:12:40.777 INFO:teuthology.orchestra.run.smithi032.stdout:6.9.0-rc3-g806ec36b2112 2024-04-30T08:12:40.778 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g806ec36b2112 vs 6.9.0-rc3-g806ec36b2112 2024-04-30T08:12:40.778 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-30T08:12:40.778 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-30T08:12:41.779 DEBUG:teuthology.parallel:result is None 2024-04-30T08:12:41.780 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-30T08:12:41.817 INFO:teuthology.task.internal:Creating test directory... 2024-04-30T08:12:41.817 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-30T08:12:41.825 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-30T08:12:41.828 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-30T08:12:41.873 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-30T08:12:41.882 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-30T08:12:41.933 INFO:teuthology.task.internal:Creating archive directory... 2024-04-30T08:12:41.934 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-30T08:12:41.945 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-30T08:12:41.948 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-30T08:12:42.065 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-30T08:12:42.080 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-30T08:12:42.080 DEBUG:teuthology.orchestra.run.smithi071:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-30T08:12:42.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-30T08:12:42.097 DEBUG:teuthology.orchestra.run.smithi189:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-30T08:12:42.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-30T08:12:42.114 DEBUG:teuthology.orchestra.run.smithi032:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-30T08:12:42.121 DEBUG:teuthology.orchestra.run.smithi071:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-30T08:12:42.140 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-30T08:12:42.186 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-30T08:12:42.206 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-30T08:12:42.220 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-30T08:12:42.238 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-30T08:12:42.306 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-30T08:12:42.507 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-30T08:12:42.512 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-30T08:12:42.520 INFO:teuthology.task.internal:Configuring sudo... 2024-04-30T08:12:42.520 DEBUG:teuthology.orchestra.run.smithi032:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-30T08:12:42.527 DEBUG:teuthology.orchestra.run.smithi071:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-30T08:12:42.529 DEBUG:teuthology.orchestra.run.smithi189:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-30T08:12:42.596 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-30T08:12:42.605 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-30T08:12:42.605 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-30T08:12:42.612 DEBUG:teuthology.orchestra.run.smithi071:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-30T08:12:42.614 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-30T08:12:42.657 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-30T08:12:42.734 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-30T08:12:42.826 DEBUG:teuthology.orchestra.run.smithi032:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-30T08:12:42.895 DEBUG:teuthology.orchestra.run.smithi032:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-30T08:12:42.972 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-30T08:12:42.972 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-30T08:12:43.036 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-30T08:12:43.080 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-30T08:12:43.133 DEBUG:teuthology.orchestra.run.smithi071:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-30T08:12:43.216 DEBUG:teuthology.orchestra.run.smithi071:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-30T08:12:43.257 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-30T08:12:43.258 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-30T08:12:43.323 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-30T08:12:43.368 DEBUG:teuthology.orchestra.run.smithi189:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-30T08:12:43.414 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-30T08:12:43.501 DEBUG:teuthology.orchestra.run.smithi189:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-30T08:12:43.540 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-30T08:12:43.540 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-30T08:12:43.606 DEBUG:teuthology.orchestra.run.smithi032:> sudo service rsyslog restart 2024-04-30T08:12:43.617 DEBUG:teuthology.orchestra.run.smithi071:> sudo service rsyslog restart 2024-04-30T08:12:43.620 DEBUG:teuthology.orchestra.run.smithi189:> sudo service rsyslog restart 2024-04-30T08:12:43.658 INFO:teuthology.orchestra.run.smithi071.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-30T08:12:43.687 INFO:teuthology.orchestra.run.smithi189.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-30T08:12:43.774 INFO:teuthology.orchestra.run.smithi032.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-30T08:12:44.302 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-30T08:12:44.310 INFO:teuthology.task.internal:Starting timer... 2024-04-30T08:12:44.310 INFO:teuthology.run_tasks:Running task pcp... 2024-04-30T08:12:44.381 INFO:teuthology.run_tasks:Running task selinux... 2024-04-30T08:12:44.405 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-30T08:12:44.406 DEBUG:teuthology.orchestra.run.smithi032:> sudo service auditd rotate 2024-04-30T08:12:44.574 INFO:teuthology.orchestra.run.smithi032.stdout:Rotating logs: 2024-04-30T08:12:44.582 DEBUG:teuthology.orchestra.run.smithi071:> sudo service auditd rotate 2024-04-30T08:12:44.651 INFO:teuthology.orchestra.run.smithi071.stdout:Rotating logs: 2024-04-30T08:12:44.653 DEBUG:teuthology.orchestra.run.smithi189:> sudo service auditd rotate 2024-04-30T08:12:44.725 INFO:teuthology.orchestra.run.smithi189.stdout:Rotating logs: 2024-04-30T08:12:44.727 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-30T08:12:44.728 DEBUG:teuthology.orchestra.run.smithi032:> /usr/sbin/getenforce 2024-04-30T08:12:44.867 INFO:teuthology.orchestra.run.smithi032.stdout:Permissive 2024-04-30T08:12:44.868 DEBUG:teuthology.orchestra.run.smithi071:> /usr/sbin/getenforce 2024-04-30T08:12:44.892 INFO:teuthology.orchestra.run.smithi071.stdout:Permissive 2024-04-30T08:12:44.893 DEBUG:teuthology.orchestra.run.smithi189:> /usr/sbin/getenforce 2024-04-30T08:12:44.917 INFO:teuthology.orchestra.run.smithi189.stdout:Permissive 2024-04-30T08:12:44.917 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi032.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi071.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi189.front.sepia.ceph.com': 'permissive'} 2024-04-30T08:12:44.917 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-30T08:12:45.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-30T08:12:45.046 DEBUG:teuthology.orchestra.run.smithi071:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-30T08:12:45.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-30T08:12:45.072 DEBUG:teuthology.orchestra.run.smithi189:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-30T08:12:45.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-30T08:12:45.098 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-30T08:12:45.099 DEBUG:teuthology.orchestra.run.smithi032:> sudo /usr/sbin/setenforce permissive 2024-04-30T08:12:45.162 DEBUG:teuthology.orchestra.run.smithi071:> sudo /usr/sbin/setenforce permissive 2024-04-30T08:12:45.195 DEBUG:teuthology.orchestra.run.smithi189:> sudo /usr/sbin/setenforce permissive 2024-04-30T08:12:45.219 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-30T08:12:45.231 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-30T08:12:45.247 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-30T08:12:45.249 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi032.front.sepia.ceph.com,smithi071.front.sepia.ceph.com,smithi189.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-30T08:18:23.841 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi032.front.sepia.ceph.com'), Remote(name='ubuntu@smithi071.front.sepia.ceph.com'), Remote(name='ubuntu@smithi189.front.sepia.ceph.com')] 2024-04-30T08:18:23.842 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-04-30T08:18:23.843 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi032.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-30T08:18:23.971 DEBUG:teuthology.orchestra.run.smithi032:> true 2024-04-30T08:18:24.133 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi032.front.sepia.ceph.com' 2024-04-30T08:18:24.134 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-04-30T08:18:24.135 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi071.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-30T08:18:24.208 DEBUG:teuthology.orchestra.run.smithi071:> true 2024-04-30T08:18:24.286 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi071.front.sepia.ceph.com' 2024-04-30T08:18:24.287 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-04-30T08:18:24.287 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-30T08:18:24.369 DEBUG:teuthology.orchestra.run.smithi189:> true 2024-04-30T08:18:24.451 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-04-30T08:18:24.452 INFO:teuthology.run_tasks:Running task clock... 2024-04-30T08:18:24.465 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-30T08:18:24.465 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-30T08:18:24.465 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-30T08:18:24.473 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-30T08:18:24.473 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-30T08:18:24.476 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-30T08:18:24.476 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-30T08:18:24.505 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-30T08:18:24.520 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-30T08:18:24.523 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-30T08:18:24.538 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-30T08:18:24.544 INFO:teuthology.orchestra.run.smithi071.stderr:sudo: ntpd: command not found 2024-04-30T08:18:24.555 INFO:teuthology.orchestra.run.smithi071.stdout:506 Cannot talk to daemon 2024-04-30T08:18:24.562 INFO:teuthology.orchestra.run.smithi189.stderr:sudo: ntpd: command not found 2024-04-30T08:18:24.566 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-30T08:18:24.568 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-30T08:18:24.573 INFO:teuthology.orchestra.run.smithi189.stdout:506 Cannot talk to daemon 2024-04-30T08:18:24.581 INFO:teuthology.orchestra.run.smithi071.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-30T08:18:24.588 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-30T08:18:24.601 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-30T08:18:24.610 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-30T08:18:24.640 INFO:teuthology.orchestra.run.smithi071.stderr:bash: line 1: ntpq: command not found 2024-04-30T08:18:24.643 INFO:teuthology.orchestra.run.smithi071.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-30T08:18:24.644 INFO:teuthology.orchestra.run.smithi071.stdout:=============================================================================== 2024-04-30T08:18:24.644 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-30T08:18:24.644 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-30T08:18:24.644 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-30T08:18:24.644 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-30T08:18:24.657 INFO:teuthology.orchestra.run.smithi189.stderr:bash: line 1: ntpq: command not found 2024-04-30T08:18:24.661 INFO:teuthology.orchestra.run.smithi189.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-30T08:18:24.661 INFO:teuthology.orchestra.run.smithi189.stdout:=============================================================================== 2024-04-30T08:18:24.661 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-30T08:18:24.661 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-30T08:18:24.661 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-30T08:18:24.661 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-30T08:18:24.688 INFO:teuthology.orchestra.run.smithi032.stderr:sudo: ntpd: command not found 2024-04-30T08:18:24.723 INFO:teuthology.orchestra.run.smithi032.stdout:506 Cannot talk to daemon 2024-04-30T08:18:24.766 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-30T08:18:24.812 INFO:teuthology.orchestra.run.smithi032.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-30T08:18:24.951 INFO:teuthology.orchestra.run.smithi032.stderr:bash: line 1: ntpq: command not found 2024-04-30T08:18:24.961 INFO:teuthology.orchestra.run.smithi032.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-30T08:18:24.961 INFO:teuthology.orchestra.run.smithi032.stdout:=============================================================================== 2024-04-30T08:18:24.961 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-30T08:18:24.961 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-30T08:18:24.961 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-30T08:18:24.961 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-30T08:18:24.962 INFO:teuthology.run_tasks:Running task install... 2024-04-30T08:18:24.971 DEBUG:teuthology.task.install:project ceph 2024-04-30T08:18:24.971 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '8feaf7d9864e980041b042fd5d964ac18adfd20f'}} 2024-04-30T08:18:24.971 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '8feaf7d9864e980041b042fd5d964ac18adfd20f'} 2024-04-30T08:18:24.971 INFO:teuthology.task.install:Using flavor: default 2024-04-30T08:18:24.977 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-30T08:18:24.977 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-30T08:18:24.977 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8feaf7d9864e980041b042fd5d964ac18adfd20f', 'tag': None, 'wait_for_package': False} 2024-04-30T08:18:24.978 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8feaf7d9864e980041b042fd5d964ac18adfd20f 2024-04-30T08:18:24.980 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8feaf7d9864e980041b042fd5d964ac18adfd20f', 'tag': None, 'wait_for_package': False} 2024-04-30T08:18:24.980 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8feaf7d9864e980041b042fd5d964ac18adfd20f 2024-04-30T08:18:24.981 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '8feaf7d9864e980041b042fd5d964ac18adfd20f', 'tag': None, 'wait_for_package': False} 2024-04-30T08:18:24.981 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8feaf7d9864e980041b042fd5d964ac18adfd20f 2024-04-30T08:18:25.132 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240429.210911-debug/8feaf7d9864e980041b042fd5d964ac18adfd20f/centos/9/flavors/default/ 2024-04-30T08:18:25.132 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3348.g8feaf7d9 2024-04-30T08:18:25.238 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240429.210911-debug/8feaf7d9864e980041b042fd5d964ac18adfd20f/centos/9/flavors/default/ 2024-04-30T08:18:25.238 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3348.g8feaf7d9 2024-04-30T08:18:25.243 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240429.210911-debug/8feaf7d9864e980041b042fd5d964ac18adfd20f/centos/9/flavors/default/ 2024-04-30T08:18:25.243 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3348.g8feaf7d9 2024-04-30T08:18:25.308 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240429.210911-debug/8feaf7d9864e980041b042fd5d964ac18adfd20f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240429.210911-debug/8feaf7d9864e980041b042fd5d964ac18adfd20f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240429.210911-debug/8feaf7d9864e980041b042fd5d964ac18adfd20f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-30T08:18:25.308 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-30T08:18:25.308 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-30T08:18:25.405 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-30T08:18:25.406 DEBUG:teuthology.orchestra.run.smithi032:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8feaf7d9864e980041b042fd5d964ac18adfd20f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-30T08:18:25.408 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240429.210911-debug/8feaf7d9864e980041b042fd5d964ac18adfd20f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240429.210911-debug/8feaf7d9864e980041b042fd5d964ac18adfd20f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240429.210911-debug/8feaf7d9864e980041b042fd5d964ac18adfd20f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-30T08:18:25.408 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-30T08:18:25.408 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-30T08:18:25.414 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240429.210911-debug/8feaf7d9864e980041b042fd5d964ac18adfd20f/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240429.210911-debug/8feaf7d9864e980041b042fd5d964ac18adfd20f/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240429.210911-debug/8feaf7d9864e980041b042fd5d964ac18adfd20f/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-30T08:18:25.414 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-30T08:18:25.415 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-30T08:18:25.437 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-30T08:18:25.437 DEBUG:teuthology.orchestra.run.smithi071:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8feaf7d9864e980041b042fd5d964ac18adfd20f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-30T08:18:25.441 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-30T08:18:25.441 DEBUG:teuthology.orchestra.run.smithi189:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/8feaf7d9864e980041b042fd5d964ac18adfd20f/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-30T08:18:25.506 DEBUG:teuthology.orchestra.run.smithi071:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-30T08:18:25.508 DEBUG:teuthology.orchestra.run.smithi189:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-30T08:18:25.541 DEBUG:teuthology.orchestra.run.smithi032:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-30T08:18:25.583 DEBUG:teuthology.orchestra.run.smithi189:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-30T08:18:25.584 DEBUG:teuthology.orchestra.run.smithi071:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-30T08:18:25.647 INFO:teuthology.orchestra.run.smithi189.stdout:check_obsoletes = 1 2024-04-30T08:18:25.649 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum clean all 2024-04-30T08:18:25.650 INFO:teuthology.orchestra.run.smithi071.stdout:check_obsoletes = 1 2024-04-30T08:18:25.651 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum clean all 2024-04-30T08:18:25.661 DEBUG:teuthology.orchestra.run.smithi032:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-30T08:18:25.745 INFO:teuthology.orchestra.run.smithi032.stdout:check_obsoletes = 1 2024-04-30T08:18:25.750 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum clean all 2024-04-30T08:18:25.879 INFO:teuthology.orchestra.run.smithi071.stdout:50 files removed 2024-04-30T08:18:25.881 INFO:teuthology.orchestra.run.smithi189.stdout:50 files removed 2024-04-30T08:18:25.906 DEBUG:teuthology.orchestra.run.smithi071:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-30T08:18:25.909 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-30T08:18:26.076 INFO:teuthology.orchestra.run.smithi032.stdout:50 files removed 2024-04-30T08:18:26.116 DEBUG:teuthology.orchestra.run.smithi032:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-30T08:18:26.569 INFO:teuthology.orchestra.run.smithi189.stdout:ceph packages for x86_64 211 kB/s | 84 kB 00:00 2024-04-30T08:18:26.648 INFO:teuthology.orchestra.run.smithi071.stdout:ceph packages for x86_64 174 kB/s | 84 kB 00:00 2024-04-30T08:18:26.952 INFO:teuthology.orchestra.run.smithi189.stdout:ceph noarch packages 34 kB/s | 12 kB 00:00 2024-04-30T08:18:27.103 INFO:teuthology.orchestra.run.smithi032.stdout:ceph packages for x86_64 153 kB/s | 84 kB 00:00 2024-04-30T08:18:27.307 INFO:teuthology.orchestra.run.smithi071.stdout:ceph noarch packages 19 kB/s | 12 kB 00:00 2024-04-30T08:18:27.370 INFO:teuthology.orchestra.run.smithi189.stdout:ceph source packages 4.9 kB/s | 1.9 kB 00:00 2024-04-30T08:18:27.670 INFO:teuthology.orchestra.run.smithi032.stdout:ceph noarch packages 25 kB/s | 12 kB 00:00 2024-04-30T08:18:27.716 INFO:teuthology.orchestra.run.smithi071.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-04-30T08:18:28.513 INFO:teuthology.orchestra.run.smithi032.stdout:ceph source packages 2.5 kB/s | 1.9 kB 00:00 2024-04-30T08:18:28.733 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - BaseOS 8.2 MB/s | 8.0 MB 00:00 2024-04-30T08:18:32.234 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - BaseOS 2.2 MB/s | 8.0 MB 00:03 2024-04-30T08:18:32.970 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - AppStream 6.0 MB/s | 19 MB 00:03 2024-04-30T08:18:37.035 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.2 MB 00:00 2024-04-30T08:18:38.594 INFO:teuthology.orchestra.run.smithi071.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 16 kB 00:00 2024-04-30T08:18:41.043 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - BaseOS 604 kB/s | 8.0 MB 00:13 2024-04-30T08:18:42.670 INFO:teuthology.orchestra.run.smithi071.stdout:Extra Packages for Enterprise Linux 5.3 MB/s | 21 MB 00:03 2024-04-30T08:18:44.319 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - AppStream 8.6 MB/s | 19 MB 00:02 2024-04-30T08:18:47.686 INFO:teuthology.orchestra.run.smithi071.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-30T08:18:49.070 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - CRB 5.0 MB/s | 6.2 MB 00:01 2024-04-30T08:18:49.821 INFO:teuthology.orchestra.run.smithi071.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-30T08:18:49.822 INFO:teuthology.orchestra.run.smithi071.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-30T08:18:49.833 INFO:teuthology.orchestra.run.smithi071.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-30T08:18:49.834 INFO:teuthology.orchestra.run.smithi071.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-30T08:18:49.834 INFO:teuthology.orchestra.run.smithi071.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-30T08:18:49.835 INFO:teuthology.orchestra.run.smithi071.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-30T08:18:49.837 INFO:teuthology.orchestra.run.smithi071.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-30T08:18:49.839 INFO:teuthology.orchestra.run.smithi071.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-30T08:18:49.840 INFO:teuthology.orchestra.run.smithi071.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-30T08:18:49.901 INFO:teuthology.orchestra.run.smithi071.stdout:Dependencies resolved. 2024-04-30T08:18:49.915 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================= 2024-04-30T08:18:49.915 INFO:teuthology.orchestra.run.smithi071.stdout: Package Arch Version Repository Size 2024-04-30T08:18:49.915 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================= 2024-04-30T08:18:49.915 INFO:teuthology.orchestra.run.smithi071.stdout:Installing: 2024-04-30T08:18:49.915 INFO:teuthology.orchestra.run.smithi071.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-30T08:18:49.916 INFO:teuthology.orchestra.run.smithi071.stdout: ceph x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 6.6 k 2024-04-30T08:18:49.916 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 5.4 M 2024-04-30T08:18:49.916 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 863 k 2024-04-30T08:18:49.916 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 147 k 2024-04-30T08:18:49.916 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 1.5 M 2024-04-30T08:18:49.916 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3348.g8feaf7d9.el9 ceph-noarch 142 k 2024-04-30T08:18:49.916 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3348.g8feaf7d9.el9 ceph-noarch 3.5 M 2024-04-30T08:18:49.916 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3348.g8feaf7d9.el9 ceph-noarch 7.4 M 2024-04-30T08:18:49.916 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-rook noarch 2:19.0.0-3348.g8feaf7d9.el9 ceph-noarch 49 k 2024-04-30T08:18:49.916 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 10 M 2024-04-30T08:18:49.916 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 43 M 2024-04-30T08:18:49.916 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-volume noarch 2:19.0.0-3348.g8feaf7d9.el9 ceph-noarch 273 k 2024-04-30T08:18:49.917 INFO:teuthology.orchestra.run.smithi071.stdout: cephadm noarch 2:19.0.0-3348.g8feaf7d9.el9 ceph-noarch 762 k 2024-04-30T08:18:49.917 INFO:teuthology.orchestra.run.smithi071.stdout: cephfs-mirror x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 224 k 2024-04-30T08:18:49.917 INFO:teuthology.orchestra.run.smithi071.stdout: cephfs-shell x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 27 k 2024-04-30T08:18:49.917 INFO:teuthology.orchestra.run.smithi071.stdout: cephfs-top noarch 2:19.0.0-3348.g8feaf7d9.el9 ceph-noarch 18 k 2024-04-30T08:18:49.917 INFO:teuthology.orchestra.run.smithi071.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-30T08:18:49.917 INFO:teuthology.orchestra.run.smithi071.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-30T08:18:49.917 INFO:teuthology.orchestra.run.smithi071.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-30T08:18:49.917 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-devel x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 32 k 2024-04-30T08:18:49.917 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2 x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 756 k 2024-04-30T08:18:49.917 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 127 k 2024-04-30T08:18:49.917 INFO:teuthology.orchestra.run.smithi071.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-30T08:18:49.917 INFO:teuthology.orchestra.run.smithi071.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-30T08:18:49.917 INFO:teuthology.orchestra.run.smithi071.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-30T08:18:49.918 INFO:teuthology.orchestra.run.smithi071.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-30T08:18:49.918 INFO:teuthology.orchestra.run.smithi071.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-30T08:18:49.918 INFO:teuthology.orchestra.run.smithi071.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-30T08:18:49.918 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 161 k 2024-04-30T08:18:49.918 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 321 k 2024-04-30T08:18:49.918 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 298 k 2024-04-30T08:18:49.918 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 100 k 2024-04-30T08:18:49.918 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 86 k 2024-04-30T08:18:49.918 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 3.0 M 2024-04-30T08:18:49.918 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 169 k 2024-04-30T08:18:49.918 INFO:teuthology.orchestra.run.smithi071.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-30T08:18:49.919 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrading: 2024-04-30T08:18:49.919 INFO:teuthology.orchestra.run.smithi071.stdout: librados2 x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 3.3 M 2024-04-30T08:18:49.919 INFO:teuthology.orchestra.run.smithi071.stdout: librbd1 x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 3.1 M 2024-04-30T08:18:49.919 INFO:teuthology.orchestra.run.smithi071.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-30T08:18:49.919 INFO:teuthology.orchestra.run.smithi071.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-30T08:18:49.919 INFO:teuthology.orchestra.run.smithi071.stdout:Installing dependencies: 2024-04-30T08:18:49.919 INFO:teuthology.orchestra.run.smithi071.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-30T08:18:49.919 INFO:teuthology.orchestra.run.smithi071.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-30T08:18:49.919 INFO:teuthology.orchestra.run.smithi071.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-30T08:18:49.919 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 20 M 2024-04-30T08:18:49.919 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3348.g8feaf7d9.el9 ceph-noarch 34 k 2024-04-30T08:18:49.919 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 2.3 M 2024-04-30T08:18:49.920 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3348.g8feaf7d9.el9 ceph-noarch 249 k 2024-04-30T08:18:49.920 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 4.9 M 2024-04-30T08:18:49.920 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 18 M 2024-04-30T08:18:49.920 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3348.g8feaf7d9.el9 ceph-noarch 16 k 2024-04-30T08:18:49.920 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-selinux x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 25 k 2024-04-30T08:18:49.920 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-30T08:18:49.920 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-30T08:18:49.920 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-30T08:18:49.920 INFO:teuthology.orchestra.run.smithi071.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-30T08:18:49.920 INFO:teuthology.orchestra.run.smithi071.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-30T08:18:49.921 INFO:teuthology.orchestra.run.smithi071.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-30T08:18:49.921 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-30T08:18:49.921 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-30T08:18:49.921 INFO:teuthology.orchestra.run.smithi071.stdout: libcephsqlite x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 158 k 2024-04-30T08:18:49.921 INFO:teuthology.orchestra.run.smithi071.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-30T08:18:49.921 INFO:teuthology.orchestra.run.smithi071.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-30T08:18:49.921 INFO:teuthology.orchestra.run.smithi071.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-30T08:18:49.921 INFO:teuthology.orchestra.run.smithi071.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-30T08:18:49.921 INFO:teuthology.orchestra.run.smithi071.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-30T08:18:49.921 INFO:teuthology.orchestra.run.smithi071.stdout: libradosstriper1 x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 497 k 2024-04-30T08:18:49.921 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-30T08:18:49.921 INFO:teuthology.orchestra.run.smithi071.stdout: librgw2 x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 5.2 M 2024-04-30T08:18:49.922 INFO:teuthology.orchestra.run.smithi071.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-30T08:18:49.922 INFO:teuthology.orchestra.run.smithi071.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-30T08:18:49.922 INFO:teuthology.orchestra.run.smithi071.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-30T08:18:49.922 INFO:teuthology.orchestra.run.smithi071.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-30T08:18:49.922 INFO:teuthology.orchestra.run.smithi071.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-30T08:18:49.922 INFO:teuthology.orchestra.run.smithi071.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-30T08:18:49.922 INFO:teuthology.orchestra.run.smithi071.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-30T08:18:49.922 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-30T08:18:49.922 INFO:teuthology.orchestra.run.smithi071.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-30T08:18:49.922 INFO:teuthology.orchestra.run.smithi071.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-30T08:18:49.922 INFO:teuthology.orchestra.run.smithi071.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-30T08:18:49.922 INFO:teuthology.orchestra.run.smithi071.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-30T08:18:49.922 INFO:teuthology.orchestra.run.smithi071.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-30T08:18:49.923 INFO:teuthology.orchestra.run.smithi071.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-30T08:18:49.923 INFO:teuthology.orchestra.run.smithi071.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-30T08:18:49.923 INFO:teuthology.orchestra.run.smithi071.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-30T08:18:49.923 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-30T08:18:49.923 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 45 k 2024-04-30T08:18:49.923 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 135 k 2024-04-30T08:18:49.923 INFO:teuthology.orchestra.run.smithi071.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-30T08:18:49.923 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-30T08:18:49.923 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-30T08:18:49.923 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-30T08:18:49.923 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-30T08:18:49.923 INFO:teuthology.orchestra.run.smithi071.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-30T08:18:49.924 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-30T08:18:49.924 INFO:teuthology.orchestra.run.smithi071.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-30T08:18:49.924 INFO:teuthology.orchestra.run.smithi071.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-30T08:18:49.924 INFO:teuthology.orchestra.run.smithi071.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-30T08:18:49.924 INFO:teuthology.orchestra.run.smithi071.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-30T08:18:49.924 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-30T08:18:49.924 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-30T08:18:49.924 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-30T08:18:49.924 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-30T08:18:49.924 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-30T08:18:49.924 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-30T08:18:49.924 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-30T08:18:49.924 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-30T08:18:49.925 INFO:teuthology.orchestra.run.smithi071.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-30T08:18:49.925 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-30T08:18:49.925 INFO:teuthology.orchestra.run.smithi071.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-30T08:18:49.925 INFO:teuthology.orchestra.run.smithi071.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-30T08:18:49.925 INFO:teuthology.orchestra.run.smithi071.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-30T08:18:49.925 INFO:teuthology.orchestra.run.smithi071.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-30T08:18:49.925 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-30T08:18:49.925 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-30T08:18:49.925 INFO:teuthology.orchestra.run.smithi071.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-30T08:18:49.925 INFO:teuthology.orchestra.run.smithi071.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-30T08:18:49.925 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-30T08:18:49.925 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-30T08:18:49.926 INFO:teuthology.orchestra.run.smithi071.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-30T08:18:49.926 INFO:teuthology.orchestra.run.smithi071.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-30T08:18:49.926 INFO:teuthology.orchestra.run.smithi071.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-30T08:18:49.926 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-30T08:18:49.926 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-30T08:18:49.926 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-30T08:18:49.926 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-30T08:18:49.926 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-30T08:18:49.926 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-30T08:18:49.926 INFO:teuthology.orchestra.run.smithi071.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-30T08:18:49.926 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-30T08:18:49.926 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-30T08:18:49.926 INFO:teuthology.orchestra.run.smithi071.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-30T08:18:49.927 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-30T08:18:49.927 INFO:teuthology.orchestra.run.smithi071.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-30T08:18:49.927 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-30T08:18:49.927 INFO:teuthology.orchestra.run.smithi071.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-30T08:18:49.927 INFO:teuthology.orchestra.run.smithi071.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-30T08:18:49.927 INFO:teuthology.orchestra.run.smithi071.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-30T08:18:49.927 INFO:teuthology.orchestra.run.smithi071.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-30T08:18:49.927 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-30T08:18:49.927 INFO:teuthology.orchestra.run.smithi071.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-30T08:18:49.927 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-30T08:18:49.927 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-30T08:18:49.927 INFO:teuthology.orchestra.run.smithi071.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-30T08:18:49.927 INFO:teuthology.orchestra.run.smithi071.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-30T08:18:49.928 INFO:teuthology.orchestra.run.smithi071.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-30T08:18:49.928 INFO:teuthology.orchestra.run.smithi071.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-30T08:18:49.928 INFO:teuthology.orchestra.run.smithi071.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-30T08:18:49.928 INFO:teuthology.orchestra.run.smithi071.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-30T08:18:49.928 INFO:teuthology.orchestra.run.smithi071.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-30T08:18:49.928 INFO:teuthology.orchestra.run.smithi071.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-30T08:18:49.928 INFO:teuthology.orchestra.run.smithi071.stdout:Installing weak dependencies: 2024-04-30T08:18:49.928 INFO:teuthology.orchestra.run.smithi071.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-30T08:18:49.928 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-30T08:18:49.928 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction Summary 2024-04-30T08:18:49.928 INFO:teuthology.orchestra.run.smithi071.stdout:======================================================================================= 2024-04-30T08:18:49.928 INFO:teuthology.orchestra.run.smithi071.stdout:Install 148 Packages 2024-04-30T08:18:49.928 INFO:teuthology.orchestra.run.smithi071.stdout:Upgrade 4 Packages 2024-04-30T08:18:49.929 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-30T08:18:49.929 INFO:teuthology.orchestra.run.smithi071.stdout:Total download size: 214 M 2024-04-30T08:18:49.929 INFO:teuthology.orchestra.run.smithi071.stdout:Downloading Packages: 2024-04-30T08:18:51.054 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - Extras packages 19 kB/s | 16 kB 00:00 2024-04-30T08:18:52.079 INFO:teuthology.orchestra.run.smithi071.stdout:(1/152): ceph-19.0.0-3348.g8feaf7d9.el9.x86_64. 41 kB/s | 6.6 kB 00:00 2024-04-30T08:18:52.271 INFO:teuthology.orchestra.run.smithi071.stdout:(2/152): ceph-fuse-19.0.0-3348.g8feaf7d9.el9.x8 4.4 MB/s | 863 kB 00:00 2024-04-30T08:18:52.330 INFO:teuthology.orchestra.run.smithi071.stdout:(3/152): ceph-immutable-object-cache-19.0.0-334 2.5 MB/s | 147 kB 00:00 2024-04-30T08:18:52.597 INFO:teuthology.orchestra.run.smithi071.stdout:(4/152): ceph-mds-19.0.0-3348.g8feaf7d9.el9.x86 8.5 MB/s | 2.3 MB 00:00 2024-04-30T08:18:52.822 INFO:teuthology.orchestra.run.smithi071.stdout:(5/152): ceph-mgr-19.0.0-3348.g8feaf7d9.el9.x86 6.7 MB/s | 1.5 MB 00:00 2024-04-30T08:18:53.456 INFO:teuthology.orchestra.run.smithi071.stdout:(6/152): ceph-mon-19.0.0-3348.g8feaf7d9.el9.x86 7.7 MB/s | 4.9 MB 00:00 2024-04-30T08:18:53.639 INFO:teuthology.orchestra.run.smithi071.stdout:(7/152): ceph-common-19.0.0-3348.g8feaf7d9.el9. 11 MB/s | 20 MB 00:01 2024-04-30T08:18:53.781 INFO:teuthology.orchestra.run.smithi071.stdout:(8/152): ceph-base-19.0.0-3348.g8feaf7d9.el9.x8 2.9 MB/s | 5.4 MB 00:01 2024-04-30T08:18:54.207 INFO:teuthology.orchestra.run.smithi071.stdout:(9/152): ceph-radosgw-19.0.0-3348.g8feaf7d9.el9 18 MB/s | 10 MB 00:00 2024-04-30T08:18:54.232 INFO:teuthology.orchestra.run.smithi071.stdout:(10/152): ceph-selinux-19.0.0-3348.g8feaf7d9.el 56 kB/s | 25 kB 00:00 2024-04-30T08:18:54.432 INFO:teuthology.orchestra.run.smithi071.stdout:(11/152): cephfs-mirror-19.0.0-3348.g8feaf7d9.e 1.1 MB/s | 224 kB 00:00 2024-04-30T08:18:54.549 INFO:teuthology.orchestra.run.smithi071.stdout:(12/152): cephfs-shell-19.0.0-3348.g8feaf7d9.el 232 kB/s | 27 kB 00:00 2024-04-30T08:18:54.674 INFO:teuthology.orchestra.run.smithi071.stdout:(13/152): libcephfs-devel-19.0.0-3348.g8feaf7d9 253 kB/s | 32 kB 00:00 2024-04-30T08:18:55.191 INFO:teuthology.orchestra.run.smithi071.stdout:(14/152): libcephfs2-19.0.0-3348.g8feaf7d9.el9. 1.4 MB/s | 756 kB 00:00 2024-04-30T08:18:55.291 INFO:teuthology.orchestra.run.smithi071.stdout:(15/152): libcephsqlite-19.0.0-3348.g8feaf7d9.e 1.5 MB/s | 158 kB 00:00 2024-04-30T08:18:55.375 INFO:teuthology.orchestra.run.smithi071.stdout:(16/152): librados-devel-19.0.0-3348.g8feaf7d9. 1.5 MB/s | 127 kB 00:00 2024-04-30T08:18:55.542 INFO:teuthology.orchestra.run.smithi071.stdout:(17/152): libradosstriper1-19.0.0-3348.g8feaf7d 2.9 MB/s | 497 kB 00:00 2024-04-30T08:18:55.899 INFO:teuthology.orchestra.run.smithi189.stdout:Extra Packages for Enterprise Linux 4.4 MB/s | 21 MB 00:04 2024-04-30T08:18:56.618 INFO:teuthology.orchestra.run.smithi071.stdout:(18/152): ceph-test-19.0.0-3348.g8feaf7d9.el9.x 18 MB/s | 43 MB 00:02 2024-04-30T08:18:56.701 INFO:teuthology.orchestra.run.smithi071.stdout:(19/152): librgw2-19.0.0-3348.g8feaf7d9.el9.x86 4.5 MB/s | 5.2 MB 00:01 2024-04-30T08:18:56.735 INFO:teuthology.orchestra.run.smithi071.stdout:(20/152): python3-ceph-argparse-19.0.0-3348.g8f 386 kB/s | 45 kB 00:00 2024-04-30T08:18:56.927 INFO:teuthology.orchestra.run.smithi071.stdout:(21/152): ceph-osd-19.0.0-3348.g8feaf7d9.el9.x8 5.1 MB/s | 18 MB 00:03 2024-04-30T08:18:56.952 INFO:teuthology.orchestra.run.smithi071.stdout:(22/152): python3-ceph-common-19.0.0-3348.g8fea 539 kB/s | 135 kB 00:00 2024-04-30T08:18:56.978 INFO:teuthology.orchestra.run.smithi071.stdout:(23/152): python3-cephfs-19.0.0-3348.g8feaf7d9. 666 kB/s | 161 kB 00:00 2024-04-30T08:18:57.003 INFO:teuthology.orchestra.run.smithi071.stdout:(24/152): python3-rados-19.0.0-3348.g8feaf7d9.e 4.2 MB/s | 321 kB 00:00 2024-04-30T08:18:57.028 INFO:teuthology.orchestra.run.smithi071.stdout:(25/152): python3-rgw-19.0.0-3348.g8feaf7d9.el9 1.9 MB/s | 100 kB 00:00 2024-04-30T08:18:57.054 INFO:teuthology.orchestra.run.smithi071.stdout:(26/152): rbd-fuse-19.0.0-3348.g8feaf7d9.el9.x8 1.7 MB/s | 86 kB 00:00 2024-04-30T08:18:57.079 INFO:teuthology.orchestra.run.smithi071.stdout:(27/152): python3-rbd-19.0.0-3348.g8feaf7d9.el9 2.3 MB/s | 298 kB 00:00 2024-04-30T08:18:57.121 INFO:teuthology.orchestra.run.smithi071.stdout:(28/152): rbd-nbd-19.0.0-3348.g8feaf7d9.el9.x86 2.5 MB/s | 169 kB 00:00 2024-04-30T08:18:57.163 INFO:teuthology.orchestra.run.smithi071.stdout:(29/152): ceph-grafana-dashboards-19.0.0-3348.g 409 kB/s | 34 kB 00:00 2024-04-30T08:18:57.230 INFO:teuthology.orchestra.run.smithi071.stdout:(30/152): rbd-mirror-19.0.0-3348.g8feaf7d9.el9. 15 MB/s | 3.0 MB 00:00 2024-04-30T08:18:57.255 INFO:teuthology.orchestra.run.smithi071.stdout:(31/152): ceph-mgr-cephadm-19.0.0-3348.g8feaf7d 1.0 MB/s | 142 kB 00:00 2024-04-30T08:18:57.330 INFO:teuthology.orchestra.run.smithi071.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3348.g8f 3.3 MB/s | 249 kB 00:00 2024-04-30T08:18:57.406 INFO:teuthology.orchestra.run.smithi071.stdout:(33/152): ceph-mgr-rook-19.0.0-3348.g8feaf7d9.e 658 kB/s | 49 kB 00:00 2024-04-30T08:18:57.489 INFO:teuthology.orchestra.run.smithi071.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3348.g8 197 kB/s | 16 kB 00:00 2024-04-30T08:18:57.581 INFO:teuthology.orchestra.run.smithi071.stdout:(35/152): ceph-volume-19.0.0-3348.g8feaf7d9.el9 2.9 MB/s | 273 kB 00:00 2024-04-30T08:18:57.682 INFO:teuthology.orchestra.run.smithi071.stdout:(36/152): ceph-mgr-diskprediction-local-19.0.0- 16 MB/s | 7.4 MB 00:00 2024-04-30T08:18:57.732 INFO:teuthology.orchestra.run.smithi071.stdout:(37/152): cephadm-19.0.0-3348.g8feaf7d9.el9.noa 5.0 MB/s | 762 kB 00:00 2024-04-30T08:18:57.783 INFO:teuthology.orchestra.run.smithi071.stdout:(38/152): ceph-mgr-dashboard-19.0.0-3348.g8feaf 5.7 MB/s | 3.5 MB 00:00 2024-04-30T08:18:57.958 INFO:teuthology.orchestra.run.smithi071.stdout:(39/152): cephfs-top-19.0.0-3348.g8feaf7d9.el9. 66 kB/s | 18 kB 00:00 2024-04-30T08:18:57.983 INFO:teuthology.orchestra.run.smithi071.stdout:(40/152): fuse-2.9.9-15.el9.x86_64.rpm 321 kB/s | 80 kB 00:00 2024-04-30T08:18:58.050 INFO:teuthology.orchestra.run.smithi071.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 195 kB 00:00 2024-04-30T08:18:58.117 INFO:teuthology.orchestra.run.smithi071.stdout:(42/152): libgfortran-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 803 kB 00:00 2024-04-30T08:18:58.175 INFO:teuthology.orchestra.run.smithi071.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-04-30T08:18:58.217 INFO:teuthology.orchestra.run.smithi071.stdout:(44/152): python3-cryptography-36.0.1-4.el9.x86 8.6 MB/s | 1.2 MB 00:00 2024-04-30T08:18:58.259 INFO:teuthology.orchestra.run.smithi071.stdout:(45/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 917 kB/s | 253 kB 00:00 2024-04-30T08:18:58.293 INFO:teuthology.orchestra.run.smithi071.stdout:(46/152): python3-pycparser-2.20-6.el9.noarch.r 1.1 MB/s | 135 kB 00:00 2024-04-30T08:18:58.318 INFO:teuthology.orchestra.run.smithi071.stdout:(47/152): python3-requests-2.25.1-8.el9.noarch. 1.2 MB/s | 125 kB 00:00 2024-04-30T08:18:58.386 INFO:teuthology.orchestra.run.smithi071.stdout:(48/152): python3-urllib3-1.26.5-5.el9.noarch.r 1.7 MB/s | 215 kB 00:00 2024-04-30T08:18:58.419 INFO:teuthology.orchestra.run.smithi071.stdout:(49/152): automake-1.16.2-8.el9.noarch.rpm 5.4 MB/s | 689 kB 00:00 2024-04-30T08:18:58.461 INFO:teuthology.orchestra.run.smithi071.stdout:(50/152): bison-3.7.4-5.el9.x86_64.rpm 6.5 MB/s | 943 kB 00:00 2024-04-30T08:18:58.486 INFO:teuthology.orchestra.run.smithi071.stdout:(51/152): boost-program-options-1.75.0-8.el9.x8 1.0 MB/s | 107 kB 00:00 2024-04-30T08:18:58.512 INFO:teuthology.orchestra.run.smithi071.stdout:(52/152): elfutils-libelf-devel-0.190-2.el9.x86 233 kB/s | 21 kB 00:00 2024-04-30T08:18:58.537 INFO:teuthology.orchestra.run.smithi071.stdout:(53/152): flex-2.6.4-9.el9.x86_64.rpm 4.1 MB/s | 313 kB 00:00 2024-04-30T08:18:58.562 INFO:teuthology.orchestra.run.smithi071.stdout:(54/152): flexiblas-3.0.4-8.el9.x86_64.rpm 433 kB/s | 33 kB 00:00 2024-04-30T08:18:58.654 INFO:teuthology.orchestra.run.smithi071.stdout:(55/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 21 MB/s | 3.0 MB 00:00 2024-04-30T08:18:58.679 INFO:teuthology.orchestra.run.smithi071.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 95 kB/s | 11 kB 00:00 2024-04-30T08:18:58.705 INFO:teuthology.orchestra.run.smithi071.stdout:(57/152): flexiblas-openblas-openmp-3.0.4-8.el9 109 kB/s | 18 kB 00:00 2024-04-30T08:18:58.730 INFO:teuthology.orchestra.run.smithi071.stdout:(58/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 160 kB 00:00 2024-04-30T08:18:58.755 INFO:teuthology.orchestra.run.smithi071.stdout:(59/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 599 kB/s | 45 kB 00:00 2024-04-30T08:18:58.806 INFO:teuthology.orchestra.run.smithi071.stdout:(60/152): librdkafka-1.6.1-102.el9.x86_64.rpm 6.4 MB/s | 662 kB 00:00 2024-04-30T08:18:58.839 INFO:teuthology.orchestra.run.smithi071.stdout:(61/152): libtool-2.4.6-45.el9.x86_64.rpm 5.2 MB/s | 582 kB 00:00 2024-04-30T08:18:58.865 INFO:teuthology.orchestra.run.smithi071.stdout:(62/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 233 kB/s | 25 kB 00:00 2024-04-30T08:18:58.890 INFO:teuthology.orchestra.run.smithi071.stdout:(63/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 565 kB/s | 47 kB 00:00 2024-04-30T08:18:58.915 INFO:teuthology.orchestra.run.smithi071.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.8 MB/s | 292 kB 00:00 2024-04-30T08:18:58.949 INFO:teuthology.orchestra.run.smithi071.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 2.2 MB/s | 188 kB 00:00 2024-04-30T08:18:58.974 INFO:teuthology.orchestra.run.smithi071.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 420 kB/s | 35 kB 00:00 2024-04-30T08:18:59.100 INFO:teuthology.orchestra.run.smithi071.stdout:(67/152): openblas-openmp-0.3.21-2.el9.x86_64.r 26 MB/s | 4.7 MB 00:00 2024-04-30T08:18:59.183 INFO:teuthology.orchestra.run.smithi071.stdout:(68/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 19 MB/s | 4.4 MB 00:00 2024-04-30T08:18:59.242 INFO:teuthology.orchestra.run.smithi071.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 6.3 MB/s | 911 kB 00:00 2024-04-30T08:18:59.259 INFO:teuthology.orchestra.run.smithi071.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.7 MB/s | 133 kB 00:00 2024-04-30T08:18:59.359 INFO:teuthology.orchestra.run.smithi071.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 10 MB/s | 1.0 MB 00:00 2024-04-30T08:18:59.451 INFO:teuthology.orchestra.run.smithi071.stdout:(72/152): postgresql-server-13.14-1.el9.x86_64. 28 MB/s | 5.8 MB 00:00 2024-04-30T08:18:59.493 INFO:teuthology.orchestra.run.smithi071.stdout:(73/152): postgresql-13.14-1.el9.x86_64.rpm 3.1 MB/s | 1.6 MB 00:00 2024-04-30T08:18:59.518 INFO:teuthology.orchestra.run.smithi071.stdout:(74/152): python3-attrs-20.3.0-7.el9.noarch.rpm 549 kB/s | 87 kB 00:00 2024-04-30T08:18:59.577 INFO:teuthology.orchestra.run.smithi071.stdout:(75/152): python3-devel-3.9.18-3.el9.x86_64.rpm 2.9 MB/s | 244 kB 00:00 2024-04-30T08:18:59.669 INFO:teuthology.orchestra.run.smithi071.stdout:(76/152): python3-babel-2.9.1-2.el9.noarch.rpm 27 MB/s | 6.0 MB 00:00 2024-04-30T08:18:59.694 INFO:teuthology.orchestra.run.smithi071.stdout:(77/152): python3-jinja2-2.11.3-5.el9.noarch.rp 1.4 MB/s | 248 kB 00:00 2024-04-30T08:18:59.720 INFO:teuthology.orchestra.run.smithi071.stdout:(78/152): python3-jsonpatch-1.21-16.el9.noarch. 180 kB/s | 26 kB 00:00 2024-04-30T08:18:59.745 INFO:teuthology.orchestra.run.smithi071.stdout:(79/152): python3-jsonpointer-2.0-4.el9.noarch. 253 kB/s | 19 kB 00:00 2024-04-30T08:18:59.770 INFO:teuthology.orchestra.run.smithi071.stdout:(80/152): python3-mako-1.1.4-6.el9.noarch.rpm 2.2 MB/s | 172 kB 00:00 2024-04-30T08:18:59.796 INFO:teuthology.orchestra.run.smithi071.stdout:(81/152): python3-markupsafe-1.1.1-12.el9.x86_6 460 kB/s | 35 kB 00:00 2024-04-30T08:18:59.854 INFO:teuthology.orchestra.run.smithi071.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.9 MB/s | 250 kB 00:00 2024-04-30T08:18:59.930 INFO:teuthology.orchestra.run.smithi071.stdout:(83/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 28 MB/s | 5.1 MB 00:00 2024-04-30T08:18:59.955 INFO:teuthology.orchestra.run.smithi071.stdout:(84/152): python3-oauthlib-3.1.1-5.el9.noarch.r 1.4 MB/s | 222 kB 00:00 2024-04-30T08:18:59.980 INFO:teuthology.orchestra.run.smithi071.stdout:(85/152): python3-packaging-20.9-5.el9.noarch.r 614 kB/s | 77 kB 00:00 2024-04-30T08:18:59.997 INFO:teuthology.orchestra.run.smithi071.stdout:(86/152): python3-prettytable-0.7.2-27.el9.noar 628 kB/s | 42 kB 00:00 2024-04-30T08:19:00.022 INFO:teuthology.orchestra.run.smithi071.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 3.9 MB/s | 269 kB 00:00 2024-04-30T08:19:00.048 INFO:teuthology.orchestra.run.smithi071.stdout:(88/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 159 kB 00:00 2024-04-30T08:19:00.073 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - AppStream 732 kB/s | 19 MB 00:26 2024-04-30T08:19:00.073 INFO:teuthology.orchestra.run.smithi071.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 3.6 MB/s | 279 kB 00:00 2024-04-30T08:19:00.098 INFO:teuthology.orchestra.run.smithi071.stdout:(90/152): python3-pytz-2021.1-5.el9.noarch.rpm 673 kB/s | 51 kB 00:00 2024-04-30T08:19:00.124 INFO:teuthology.orchestra.run.smithi071.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 710 kB/s | 54 kB 00:00 2024-04-30T08:19:00.182 INFO:teuthology.orchestra.run.smithi071.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 498 kB/s | 42 kB 00:00 2024-04-30T08:19:00.216 INFO:teuthology.orchestra.run.smithi071.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 480 kB/s | 44 kB 00:00 2024-04-30T08:19:00.266 INFO:teuthology.orchestra.run.smithi071.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 221 kB 00:00 2024-04-30T08:19:00.300 INFO:teuthology.orchestra.run.smithi071.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 429 kB/s | 36 kB 00:00 2024-04-30T08:19:00.350 INFO:teuthology.orchestra.run.smithi071.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 565 kB/s | 47 kB 00:00 2024-04-30T08:19:00.526 INFO:teuthology.orchestra.run.smithi071.stdout:(97/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 36 MB/s | 16 MB 00:00 2024-04-30T08:19:00.560 INFO:teuthology.orchestra.run.smithi071.stdout:(98/152): socat-1.7.4.1-5.el9.x86_64.rpm 1.1 MB/s | 305 kB 00:00 2024-04-30T08:19:00.585 INFO:teuthology.orchestra.run.smithi071.stdout:(99/152): uuid-1.6.2-55.el9.x86_64.rpm 247 kB/s | 58 kB 00:00 2024-04-30T08:19:00.610 INFO:teuthology.orchestra.run.smithi071.stdout:(100/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.1 MB/s | 94 kB 00:00 2024-04-30T08:19:00.636 INFO:teuthology.orchestra.run.smithi071.stdout:(101/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 842 kB/s | 64 kB 00:00 2024-04-30T08:19:00.661 INFO:teuthology.orchestra.run.smithi071.stdout:(102/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 593 kB/s | 45 kB 00:00 2024-04-30T08:19:00.778 INFO:teuthology.orchestra.run.smithi071.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 133 kB/s | 22 kB 00:00 2024-04-30T08:19:01.005 INFO:teuthology.orchestra.run.smithi189.stdout:lab-extras 43 kB/s | 1.7 kB 00:00 2024-04-30T08:19:01.011 INFO:teuthology.orchestra.run.smithi071.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 2.2 MB/s | 863 kB 00:00 2024-04-30T08:19:01.162 INFO:teuthology.orchestra.run.smithi071.stdout:(105/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-04-30T08:19:01.204 INFO:teuthology.orchestra.run.smithi071.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 725 kB/s | 309 kB 00:00 2024-04-30T08:19:01.229 INFO:teuthology.orchestra.run.smithi071.stdout:(107/152): grpc-data-1.46.7-10.el9.noarch.rpm 89 kB/s | 19 kB 00:00 2024-04-30T08:19:01.321 INFO:teuthology.orchestra.run.smithi071.stdout:(108/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 222 kB/s | 26 kB 00:00 2024-04-30T08:19:01.422 INFO:teuthology.orchestra.run.smithi071.stdout:(109/152): libunwind-1.6.2-1.el9.x86_64.rpm 674 kB/s | 67 kB 00:00 2024-04-30T08:19:01.447 INFO:teuthology.orchestra.run.smithi071.stdout:(110/152): liboath-2.6.7-2.el9.x86_64.rpm 223 kB/s | 49 kB 00:00 2024-04-30T08:19:01.522 INFO:teuthology.orchestra.run.smithi071.stdout:(111/152): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-04-30T08:19:01.547 INFO:teuthology.orchestra.run.smithi071.stdout:(112/152): luarocks-3.9.2-1.el9.noarch.rpm 1.2 MB/s | 151 kB 00:00 2024-04-30T08:19:01.623 INFO:teuthology.orchestra.run.smithi071.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 5.4 MB/s | 548 kB 00:00 2024-04-30T08:19:01.648 INFO:teuthology.orchestra.run.smithi071.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 433 kB/s | 43 kB 00:00 2024-04-30T08:19:01.706 INFO:teuthology.orchestra.run.smithi071.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 385 kB/s | 32 kB 00:00 2024-04-30T08:19:01.732 INFO:teuthology.orchestra.run.smithi071.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 179 kB/s | 15 kB 00:00 2024-04-30T08:19:01.799 INFO:teuthology.orchestra.run.smithi071.stdout:(117/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 2.3 MB/s | 839 kB 00:00 2024-04-30T08:19:01.824 INFO:teuthology.orchestra.run.smithi071.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.4 MB/s | 172 kB 00:00 2024-04-30T08:19:01.858 INFO:teuthology.orchestra.run.smithi071.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 2.8 MB/s | 358 kB 00:00 2024-04-30T08:19:01.893 INFO:teuthology.orchestra.run.smithi071.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-04-30T08:19:01.917 INFO:teuthology.orchestra.run.smithi071.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 382 kB/s | 35 kB 00:00 2024-04-30T08:19:01.959 INFO:teuthology.orchestra.run.smithi071.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 2.0 MB/s | 206 kB 00:00 2024-04-30T08:19:02.009 INFO:teuthology.orchestra.run.smithi071.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 1.5 MB/s | 144 kB 00:00 2024-04-30T08:19:02.035 INFO:teuthology.orchestra.run.smithi071.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 142 kB/s | 11 kB 00:00 2024-04-30T08:19:02.110 INFO:teuthology.orchestra.run.smithi071.stdout:(125/152): python3-grpcio-1.46.7-10.el9.x86_64. 9.3 MB/s | 2.0 MB 00:00 2024-04-30T08:19:02.152 INFO:teuthology.orchestra.run.smithi071.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 124 kB/s | 18 kB 00:00 2024-04-30T08:19:02.177 INFO:teuthology.orchestra.run.smithi071.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 163 kB/s | 23 kB 00:00 2024-04-30T08:19:02.203 INFO:teuthology.orchestra.run.smithi071.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 211 kB/s | 19 kB 00:00 2024-04-30T08:19:02.261 INFO:teuthology.orchestra.run.smithi071.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 236 kB/s | 20 kB 00:00 2024-04-30T08:19:02.386 INFO:teuthology.orchestra.run.smithi071.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 252 kB/s | 46 kB 00:00 2024-04-30T08:19:02.411 INFO:teuthology.orchestra.run.smithi071.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 525 kB/s | 79 kB 00:00 2024-04-30T08:19:02.445 INFO:teuthology.orchestra.run.smithi071.stdout:(132/152): python3-kubernetes-26.1.0-2.el9.noar 3.8 MB/s | 1.0 MB 00:00 2024-04-30T08:19:02.462 INFO:teuthology.orchestra.run.smithi071.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 763 kB/s | 58 kB 00:00 2024-04-30T08:19:02.496 INFO:teuthology.orchestra.run.smithi071.stdout:(134/152): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-04-30T08:19:02.521 INFO:teuthology.orchestra.run.smithi071.stdout:(135/152): python3-portend-3.1.0-2.el9.noarch.r 217 kB/s | 16 kB 00:00 2024-04-30T08:19:02.538 INFO:teuthology.orchestra.run.smithi071.stdout:(136/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-04-30T08:19:02.580 INFO:teuthology.orchestra.run.smithi071.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 336 kB/s | 28 kB 00:00 2024-04-30T08:19:02.634 INFO:teuthology.orchestra.run.smithi071.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 274 kB/s | 31 kB 00:00 2024-04-30T08:19:02.665 INFO:teuthology.orchestra.run.smithi071.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-04-30T08:19:02.689 INFO:teuthology.orchestra.run.smithi071.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 543 kB/s | 59 kB 00:00 2024-04-30T08:19:02.715 INFO:teuthology.orchestra.run.smithi071.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 445 kB/s | 36 kB 00:00 2024-04-30T08:19:02.748 INFO:teuthology.orchestra.run.smithi071.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 608 kB/s | 51 kB 00:00 2024-04-30T08:19:02.774 INFO:teuthology.orchestra.run.smithi071.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-04-30T08:19:02.799 INFO:teuthology.orchestra.run.smithi071.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 1.0 MB/s | 90 kB 00:00 2024-04-30T08:19:02.858 INFO:teuthology.orchestra.run.smithi071.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 239 kB/s | 20 kB 00:00 2024-04-30T08:19:02.883 INFO:teuthology.orchestra.run.smithi071.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-04-30T08:19:02.975 INFO:teuthology.orchestra.run.smithi071.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-04-30T08:19:03.114 INFO:teuthology.orchestra.run.smithi189.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-30T08:19:03.115 INFO:teuthology.orchestra.run.smithi189.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-30T08:19:03.126 INFO:teuthology.orchestra.run.smithi189.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-30T08:19:03.127 INFO:teuthology.orchestra.run.smithi189.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-30T08:19:03.128 INFO:teuthology.orchestra.run.smithi189.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-30T08:19:03.128 INFO:teuthology.orchestra.run.smithi189.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-30T08:19:03.130 INFO:teuthology.orchestra.run.smithi189.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-30T08:19:03.132 INFO:teuthology.orchestra.run.smithi189.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-30T08:19:03.133 INFO:teuthology.orchestra.run.smithi189.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-30T08:19:03.195 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-04-30T08:19:03.208 INFO:teuthology.orchestra.run.smithi189.stdout:======================================================================================= 2024-04-30T08:19:03.208 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-04-30T08:19:03.208 INFO:teuthology.orchestra.run.smithi189.stdout:======================================================================================= 2024-04-30T08:19:03.208 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-04-30T08:19:03.209 INFO:teuthology.orchestra.run.smithi189.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-30T08:19:03.209 INFO:teuthology.orchestra.run.smithi189.stdout: ceph x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 6.6 k 2024-04-30T08:19:03.209 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-base x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 5.4 M 2024-04-30T08:19:03.209 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-fuse x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 863 k 2024-04-30T08:19:03.209 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 147 k 2024-04-30T08:19:03.209 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 1.5 M 2024-04-30T08:19:03.209 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3348.g8feaf7d9.el9 ceph-noarch 142 k 2024-04-30T08:19:03.209 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3348.g8feaf7d9.el9 ceph-noarch 3.5 M 2024-04-30T08:19:03.209 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3348.g8feaf7d9.el9 ceph-noarch 7.4 M 2024-04-30T08:19:03.209 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-rook noarch 2:19.0.0-3348.g8feaf7d9.el9 ceph-noarch 49 k 2024-04-30T08:19:03.209 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-radosgw x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 10 M 2024-04-30T08:19:03.209 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-test x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 43 M 2024-04-30T08:19:03.209 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-volume noarch 2:19.0.0-3348.g8feaf7d9.el9 ceph-noarch 273 k 2024-04-30T08:19:03.210 INFO:teuthology.orchestra.run.smithi189.stdout: cephadm noarch 2:19.0.0-3348.g8feaf7d9.el9 ceph-noarch 762 k 2024-04-30T08:19:03.210 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-mirror x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 224 k 2024-04-30T08:19:03.210 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-shell x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 27 k 2024-04-30T08:19:03.210 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-top noarch 2:19.0.0-3348.g8feaf7d9.el9 ceph-noarch 18 k 2024-04-30T08:19:03.210 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-30T08:19:03.210 INFO:teuthology.orchestra.run.smithi189.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-30T08:19:03.210 INFO:teuthology.orchestra.run.smithi189.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-30T08:19:03.210 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs-devel x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 32 k 2024-04-30T08:19:03.210 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs2 x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 756 k 2024-04-30T08:19:03.210 INFO:teuthology.orchestra.run.smithi189.stdout: librados-devel x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 127 k 2024-04-30T08:19:03.210 INFO:teuthology.orchestra.run.smithi189.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-30T08:19:03.210 INFO:teuthology.orchestra.run.smithi189.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-30T08:19:03.211 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-30T08:19:03.211 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-30T08:19:03.211 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-30T08:19:03.211 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-30T08:19:03.211 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cephfs x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 161 k 2024-04-30T08:19:03.211 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rados x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 321 k 2024-04-30T08:19:03.211 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rbd x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 298 k 2024-04-30T08:19:03.211 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rgw x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 100 k 2024-04-30T08:19:03.211 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-fuse x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 86 k 2024-04-30T08:19:03.211 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-mirror x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 3.0 M 2024-04-30T08:19:03.211 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-nbd x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 169 k 2024-04-30T08:19:03.211 INFO:teuthology.orchestra.run.smithi189.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-30T08:19:03.211 INFO:teuthology.orchestra.run.smithi189.stdout:Upgrading: 2024-04-30T08:19:03.212 INFO:teuthology.orchestra.run.smithi189.stdout: librados2 x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 3.3 M 2024-04-30T08:19:03.212 INFO:teuthology.orchestra.run.smithi189.stdout: librbd1 x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 3.1 M 2024-04-30T08:19:03.212 INFO:teuthology.orchestra.run.smithi189.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-30T08:19:03.212 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-30T08:19:03.212 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-04-30T08:19:03.212 INFO:teuthology.orchestra.run.smithi189.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-30T08:19:03.212 INFO:teuthology.orchestra.run.smithi189.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-30T08:19:03.212 INFO:teuthology.orchestra.run.smithi189.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-30T08:19:03.212 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-common x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 20 M 2024-04-30T08:19:03.212 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3348.g8feaf7d9.el9 ceph-noarch 34 k 2024-04-30T08:19:03.212 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mds x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 2.3 M 2024-04-30T08:19:03.212 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3348.g8feaf7d9.el9 ceph-noarch 249 k 2024-04-30T08:19:03.212 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mon x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 4.9 M 2024-04-30T08:19:03.212 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-osd x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 18 M 2024-04-30T08:19:03.213 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3348.g8feaf7d9.el9 ceph-noarch 16 k 2024-04-30T08:19:03.213 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-selinux x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 25 k 2024-04-30T08:19:03.213 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-30T08:19:03.213 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-30T08:19:03.213 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-30T08:19:03.213 INFO:teuthology.orchestra.run.smithi189.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-30T08:19:03.213 INFO:teuthology.orchestra.run.smithi189.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-30T08:19:03.213 INFO:teuthology.orchestra.run.smithi189.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-30T08:19:03.213 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-30T08:19:03.213 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-30T08:19:03.213 INFO:teuthology.orchestra.run.smithi189.stdout: libcephsqlite x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 158 k 2024-04-30T08:19:03.213 INFO:teuthology.orchestra.run.smithi189.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-30T08:19:03.213 INFO:teuthology.orchestra.run.smithi189.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-30T08:19:03.214 INFO:teuthology.orchestra.run.smithi189.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-30T08:19:03.214 INFO:teuthology.orchestra.run.smithi189.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-30T08:19:03.214 INFO:teuthology.orchestra.run.smithi189.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-30T08:19:03.214 INFO:teuthology.orchestra.run.smithi189.stdout: libradosstriper1 x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 497 k 2024-04-30T08:19:03.214 INFO:teuthology.orchestra.run.smithi189.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-30T08:19:03.214 INFO:teuthology.orchestra.run.smithi189.stdout: librgw2 x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 5.2 M 2024-04-30T08:19:03.214 INFO:teuthology.orchestra.run.smithi189.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-30T08:19:03.214 INFO:teuthology.orchestra.run.smithi189.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-30T08:19:03.214 INFO:teuthology.orchestra.run.smithi189.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-30T08:19:03.214 INFO:teuthology.orchestra.run.smithi189.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-30T08:19:03.214 INFO:teuthology.orchestra.run.smithi189.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-30T08:19:03.214 INFO:teuthology.orchestra.run.smithi189.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-30T08:19:03.215 INFO:teuthology.orchestra.run.smithi189.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-30T08:19:03.215 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-30T08:19:03.215 INFO:teuthology.orchestra.run.smithi189.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-30T08:19:03.215 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-30T08:19:03.215 INFO:teuthology.orchestra.run.smithi189.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-30T08:19:03.215 INFO:teuthology.orchestra.run.smithi189.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-30T08:19:03.215 INFO:teuthology.orchestra.run.smithi189.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-30T08:19:03.215 INFO:teuthology.orchestra.run.smithi189.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-30T08:19:03.215 INFO:teuthology.orchestra.run.smithi189.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-30T08:19:03.215 INFO:teuthology.orchestra.run.smithi189.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-30T08:19:03.215 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-30T08:19:03.215 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-argparse x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 45 k 2024-04-30T08:19:03.215 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-common x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 135 k 2024-04-30T08:19:03.216 INFO:teuthology.orchestra.run.smithi189.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-30T08:19:03.216 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-30T08:19:03.216 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-30T08:19:03.216 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-30T08:19:03.216 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-30T08:19:03.216 INFO:teuthology.orchestra.run.smithi189.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-30T08:19:03.216 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-30T08:19:03.216 INFO:teuthology.orchestra.run.smithi189.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-30T08:19:03.216 INFO:teuthology.orchestra.run.smithi189.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-30T08:19:03.216 INFO:teuthology.orchestra.run.smithi189.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-30T08:19:03.216 INFO:teuthology.orchestra.run.smithi189.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-30T08:19:03.216 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-30T08:19:03.216 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-30T08:19:03.217 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-30T08:19:03.217 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-30T08:19:03.217 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-30T08:19:03.217 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-30T08:19:03.217 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-30T08:19:03.217 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-30T08:19:03.217 INFO:teuthology.orchestra.run.smithi189.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-30T08:19:03.217 INFO:teuthology.orchestra.run.smithi189.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-30T08:19:03.217 INFO:teuthology.orchestra.run.smithi189.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-30T08:19:03.217 INFO:teuthology.orchestra.run.smithi189.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-30T08:19:03.217 INFO:teuthology.orchestra.run.smithi189.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-30T08:19:03.217 INFO:teuthology.orchestra.run.smithi189.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-30T08:19:03.217 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-30T08:19:03.218 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-30T08:19:03.218 INFO:teuthology.orchestra.run.smithi189.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-30T08:19:03.218 INFO:teuthology.orchestra.run.smithi189.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-30T08:19:03.218 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-30T08:19:03.218 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-30T08:19:03.218 INFO:teuthology.orchestra.run.smithi189.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-30T08:19:03.218 INFO:teuthology.orchestra.run.smithi189.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-30T08:19:03.218 INFO:teuthology.orchestra.run.smithi189.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-30T08:19:03.218 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-30T08:19:03.218 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-30T08:19:03.218 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-30T08:19:03.218 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-30T08:19:03.218 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-30T08:19:03.219 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-30T08:19:03.219 INFO:teuthology.orchestra.run.smithi189.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-30T08:19:03.219 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-30T08:19:03.219 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-30T08:19:03.219 INFO:teuthology.orchestra.run.smithi189.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-30T08:19:03.219 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-30T08:19:03.219 INFO:teuthology.orchestra.run.smithi189.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-30T08:19:03.219 INFO:teuthology.orchestra.run.smithi189.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-30T08:19:03.219 INFO:teuthology.orchestra.run.smithi189.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-30T08:19:03.219 INFO:teuthology.orchestra.run.smithi189.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-30T08:19:03.219 INFO:teuthology.orchestra.run.smithi189.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-30T08:19:03.219 INFO:teuthology.orchestra.run.smithi189.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-30T08:19:03.219 INFO:teuthology.orchestra.run.smithi189.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-30T08:19:03.219 INFO:teuthology.orchestra.run.smithi189.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-30T08:19:03.220 INFO:teuthology.orchestra.run.smithi189.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-30T08:19:03.220 INFO:teuthology.orchestra.run.smithi189.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-30T08:19:03.220 INFO:teuthology.orchestra.run.smithi189.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-30T08:19:03.220 INFO:teuthology.orchestra.run.smithi189.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-30T08:19:03.220 INFO:teuthology.orchestra.run.smithi189.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-30T08:19:03.220 INFO:teuthology.orchestra.run.smithi189.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-30T08:19:03.220 INFO:teuthology.orchestra.run.smithi189.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-30T08:19:03.220 INFO:teuthology.orchestra.run.smithi189.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-30T08:19:03.220 INFO:teuthology.orchestra.run.smithi189.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-30T08:19:03.220 INFO:teuthology.orchestra.run.smithi189.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-30T08:19:03.220 INFO:teuthology.orchestra.run.smithi189.stdout:Installing weak dependencies: 2024-04-30T08:19:03.220 INFO:teuthology.orchestra.run.smithi189.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-30T08:19:03.220 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-30T08:19:03.220 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-04-30T08:19:03.221 INFO:teuthology.orchestra.run.smithi189.stdout:======================================================================================= 2024-04-30T08:19:03.221 INFO:teuthology.orchestra.run.smithi189.stdout:Install 148 Packages 2024-04-30T08:19:03.221 INFO:teuthology.orchestra.run.smithi189.stdout:Upgrade 4 Packages 2024-04-30T08:19:03.221 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-30T08:19:03.221 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 214 M 2024-04-30T08:19:03.221 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-04-30T08:19:03.267 INFO:teuthology.orchestra.run.smithi071.stdout:(148/152): librbd1-19.0.0-3348.g8feaf7d9.el9.x8 11 MB/s | 3.1 MB 00:00 2024-04-30T08:19:03.434 INFO:teuthology.orchestra.run.smithi071.stdout:(149/152): librados2-19.0.0-3348.g8feaf7d9.el9. 6.0 MB/s | 3.3 MB 00:00 2024-04-30T08:19:03.492 INFO:teuthology.orchestra.run.smithi071.stdout:(150/152): openssl-3.2.1-1.el9.x86_64.rpm 6.0 MB/s | 1.3 MB 00:00 2024-04-30T08:19:03.668 INFO:teuthology.orchestra.run.smithi071.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 10 MB/s | 2.4 MB 00:00 2024-04-30T08:19:03.835 INFO:teuthology.orchestra.run.smithi071.stdout:(152/152): python3-werkzeug-2.0.3-3.el9.1.noarc 393 kB/s | 427 kB 00:01 2024-04-30T08:19:03.840 INFO:teuthology.orchestra.run.smithi071.stdout:-------------------------------------------------------------------------------- 2024-04-30T08:19:03.840 INFO:teuthology.orchestra.run.smithi071.stdout:Total 15 MB/s | 214 MB 00:13 2024-04-30T08:19:04.173 INFO:teuthology.orchestra.run.smithi189.stdout:(1/152): ceph-19.0.0-3348.g8feaf7d9.el9.x86_64. 38 kB/s | 6.6 kB 00:00 2024-04-30T08:19:04.407 INFO:teuthology.orchestra.run.smithi189.stdout:(2/152): ceph-fuse-19.0.0-3348.g8feaf7d9.el9.x8 3.6 MB/s | 863 kB 00:00 2024-04-30T08:19:04.482 INFO:teuthology.orchestra.run.smithi189.stdout:(3/152): ceph-immutable-object-cache-19.0.0-334 1.9 MB/s | 147 kB 00:00 2024-04-30T08:19:04.511 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.2 MB 00:00 2024-04-30T08:19:04.565 INFO:teuthology.orchestra.run.smithi189.stdout:(4/152): ceph-base-19.0.0-3348.g8feaf7d9.el9.x8 9.5 MB/s | 5.4 MB 00:00 2024-04-30T08:19:04.699 INFO:teuthology.orchestra.run.smithi189.stdout:(5/152): ceph-mds-19.0.0-3348.g8feaf7d9.el9.x86 10 MB/s | 2.3 MB 00:00 2024-04-30T08:19:04.741 INFO:teuthology.orchestra.run.smithi189.stdout:(6/152): ceph-mgr-19.0.0-3348.g8feaf7d9.el9.x86 8.6 MB/s | 1.5 MB 00:00 2024-04-30T08:19:04.955 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction check 2024-04-30T08:19:05.098 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction check succeeded. 2024-04-30T08:19:05.098 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction test 2024-04-30T08:19:05.183 INFO:teuthology.orchestra.run.smithi189.stdout:(7/152): ceph-mon-19.0.0-3348.g8feaf7d9.el9.x86 10 MB/s | 4.9 MB 00:00 2024-04-30T08:19:06.075 INFO:teuthology.orchestra.run.smithi189.stdout:(8/152): ceph-osd-19.0.0-3348.g8feaf7d9.el9.x86 13 MB/s | 18 MB 00:01 2024-04-30T08:19:06.158 INFO:teuthology.orchestra.run.smithi189.stdout:(9/152): ceph-selinux-19.0.0-3348.g8feaf7d9.el9 303 kB/s | 25 kB 00:00 2024-04-30T08:19:06.267 INFO:teuthology.orchestra.run.smithi189.stdout:(10/152): ceph-radosgw-19.0.0-3348.g8feaf7d9.el 9.3 MB/s | 10 MB 00:01 2024-04-30T08:19:06.342 INFO:teuthology.orchestra.run.smithi189.stdout:(11/152): cephfs-mirror-19.0.0-3348.g8feaf7d9.e 2.9 MB/s | 224 kB 00:00 2024-04-30T08:19:06.418 INFO:teuthology.orchestra.run.smithi189.stdout:(12/152): cephfs-shell-19.0.0-3348.g8feaf7d9.el 360 kB/s | 27 kB 00:00 2024-04-30T08:19:06.501 INFO:teuthology.orchestra.run.smithi189.stdout:(13/152): libcephfs-devel-19.0.0-3348.g8feaf7d9 381 kB/s | 32 kB 00:00 2024-04-30T08:19:06.668 INFO:teuthology.orchestra.run.smithi189.stdout:(14/152): libcephfs2-19.0.0-3348.g8feaf7d9.el9. 4.4 MB/s | 756 kB 00:00 2024-04-30T08:19:06.743 INFO:teuthology.orchestra.run.smithi189.stdout:(15/152): libcephsqlite-19.0.0-3348.g8feaf7d9.e 2.1 MB/s | 158 kB 00:00 2024-04-30T08:19:06.810 INFO:teuthology.orchestra.run.smithi189.stdout:(16/152): librados-devel-19.0.0-3348.g8feaf7d9. 1.9 MB/s | 127 kB 00:00 2024-04-30T08:19:06.820 INFO:teuthology.orchestra.run.smithi071.stdout:Transaction test succeeded. 2024-04-30T08:19:06.822 INFO:teuthology.orchestra.run.smithi071.stdout:Running transaction 2024-04-30T08:19:06.894 INFO:teuthology.orchestra.run.smithi189.stdout:(17/152): libradosstriper1-19.0.0-3348.g8feaf7d 5.8 MB/s | 497 kB 00:00 2024-04-30T08:19:07.494 INFO:teuthology.orchestra.run.smithi189.stdout:(18/152): librgw2-19.0.0-3348.g8feaf7d9.el9.x86 8.6 MB/s | 5.2 MB 00:00 2024-04-30T08:19:07.711 INFO:teuthology.orchestra.run.smithi189.stdout:(19/152): ceph-common-19.0.0-3348.g8feaf7d9.el9 5.3 MB/s | 20 MB 00:03 2024-04-30T08:19:07.736 INFO:teuthology.orchestra.run.smithi189.stdout:(20/152): python3-ceph-argparse-19.0.0-3348.g8f 187 kB/s | 45 kB 00:00 2024-04-30T08:19:07.770 INFO:teuthology.orchestra.run.smithi032.stdout:CentOS Stream 9 - Extras packages 8.5 kB/s | 16 kB 00:01 2024-04-30T08:19:07.787 INFO:teuthology.orchestra.run.smithi189.stdout:(21/152): python3-cephfs-19.0.0-3348.g8feaf7d9. 3.2 MB/s | 161 kB 00:00 2024-04-30T08:19:07.812 INFO:teuthology.orchestra.run.smithi189.stdout:(22/152): python3-ceph-common-19.0.0-3348.g8fea 1.3 MB/s | 135 kB 00:00 2024-04-30T08:19:07.870 INFO:teuthology.orchestra.run.smithi189.stdout:(23/152): python3-rados-19.0.0-3348.g8feaf7d9.e 3.8 MB/s | 321 kB 00:00 2024-04-30T08:19:07.921 INFO:teuthology.orchestra.run.smithi189.stdout:(24/152): python3-rbd-19.0.0-3348.g8feaf7d9.el9 2.7 MB/s | 298 kB 00:00 2024-04-30T08:19:07.946 INFO:teuthology.orchestra.run.smithi189.stdout:(25/152): python3-rgw-19.0.0-3348.g8feaf7d9.el9 1.3 MB/s | 100 kB 00:00 2024-04-30T08:19:08.255 INFO:teuthology.orchestra.run.smithi189.stdout:(26/152): rbd-mirror-19.0.0-3348.g8feaf7d9.el9. 9.9 MB/s | 3.0 MB 00:00 2024-04-30T08:19:08.272 INFO:teuthology.orchestra.run.smithi189.stdout:(27/152): rbd-fuse-19.0.0-3348.g8feaf7d9.el9.x8 245 kB/s | 86 kB 00:00 2024-04-30T08:19:08.314 INFO:teuthology.orchestra.run.smithi189.stdout:(28/152): rbd-nbd-19.0.0-3348.g8feaf7d9.el9.x86 2.8 MB/s | 169 kB 00:00 2024-04-30T08:19:08.356 INFO:teuthology.orchestra.run.smithi189.stdout:(29/152): ceph-grafana-dashboards-19.0.0-3348.g 409 kB/s | 34 kB 00:00 2024-04-30T08:19:08.381 INFO:teuthology.orchestra.run.smithi189.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3348.g8feaf7d 2.1 MB/s | 142 kB 00:00 2024-04-30T08:19:09.157 INFO:teuthology.orchestra.run.smithi189.stdout:(31/152): ceph-test-19.0.0-3348.g8feaf7d9.el9.x 14 MB/s | 43 MB 00:02 2024-04-30T08:19:09.224 INFO:teuthology.orchestra.run.smithi189.stdout:(32/152): ceph-mgr-dashboard-19.0.0-3348.g8feaf 4.1 MB/s | 3.5 MB 00:00 2024-04-30T08:19:09.249 INFO:teuthology.orchestra.run.smithi189.stdout:(33/152): ceph-mgr-modules-core-19.0.0-3348.g8f 2.7 MB/s | 249 kB 00:00 2024-04-30T08:19:09.341 INFO:teuthology.orchestra.run.smithi189.stdout:(34/152): ceph-mgr-diskprediction-local-19.0.0- 7.7 MB/s | 7.4 MB 00:00 2024-04-30T08:19:09.366 INFO:teuthology.orchestra.run.smithi189.stdout:(35/152): ceph-prometheus-alerts-19.0.0-3348.g8 140 kB/s | 16 kB 00:00 2024-04-30T08:19:09.391 INFO:teuthology.orchestra.run.smithi189.stdout:(36/152): ceph-mgr-rook-19.0.0-3348.g8feaf7d9.e 295 kB/s | 49 kB 00:00 2024-04-30T08:19:09.418 INFO:teuthology.orchestra.run.smithi189.stdout:(37/152): ceph-volume-19.0.0-3348.g8feaf7d9.el9 3.5 MB/s | 273 kB 00:00 2024-04-30T08:19:09.450 INFO:teuthology.orchestra.run.smithi189.stdout:(38/152): cephadm-19.0.0-3348.g8feaf7d9.el9.noa 8.9 MB/s | 762 kB 00:00 2024-04-30T08:19:09.475 INFO:teuthology.orchestra.run.smithi189.stdout:(39/152): cephfs-top-19.0.0-3348.g8feaf7d9.el9. 218 kB/s | 18 kB 00:00 2024-04-30T08:19:09.509 INFO:teuthology.orchestra.run.smithi189.stdout:(40/152): fuse-2.9.9-15.el9.x86_64.rpm 871 kB/s | 80 kB 00:00 2024-04-30T08:19:09.584 INFO:teuthology.orchestra.run.smithi189.stdout:(41/152): libgfortran-11.4.1-3.el9.x86_64.rpm 5.9 MB/s | 803 kB 00:00 2024-04-30T08:19:09.618 INFO:teuthology.orchestra.run.smithi189.stdout:(42/152): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-04-30T08:19:09.643 INFO:teuthology.orchestra.run.smithi189.stdout:(43/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.8 MB/s | 253 kB 00:00 2024-04-30T08:19:09.702 INFO:teuthology.orchestra.run.smithi189.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 106 kB 00:00 2024-04-30T08:19:09.735 INFO:teuthology.orchestra.run.smithi189.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 1.4 MB/s | 135 kB 00:00 2024-04-30T08:19:09.769 INFO:teuthology.orchestra.run.smithi189.stdout:(46/152): python3-cryptography-36.0.1-4.el9.x86 6.7 MB/s | 1.2 MB 00:00 2024-04-30T08:19:09.811 INFO:teuthology.orchestra.run.smithi189.stdout:(47/152): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-04-30T08:19:09.861 INFO:teuthology.orchestra.run.smithi189.stdout:(48/152): python3-urllib3-1.26.5-5.el9.noarch.r 1.7 MB/s | 215 kB 00:00 2024-04-30T08:19:10.103 INFO:teuthology.orchestra.run.smithi189.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 443 kB/s | 107 kB 00:00 2024-04-30T08:19:10.137 INFO:teuthology.orchestra.run.smithi189.stdout:(50/152): automake-1.16.2-8.el9.noarch.rpm 1.8 MB/s | 689 kB 00:00 2024-04-30T08:19:10.162 INFO:teuthology.orchestra.run.smithi189.stdout:(51/152): bison-3.7.4-5.el9.x86_64.rpm 2.6 MB/s | 943 kB 00:00 2024-04-30T08:19:10.187 INFO:teuthology.orchestra.run.smithi189.stdout:(52/152): elfutils-libelf-devel-0.190-2.el9.x86 256 kB/s | 21 kB 00:00 2024-04-30T08:19:10.221 INFO:teuthology.orchestra.run.smithi189.stdout:(53/152): flex-2.6.4-9.el9.x86_64.rpm 3.7 MB/s | 313 kB 00:00 2024-04-30T08:19:10.246 INFO:teuthology.orchestra.run.smithi189.stdout:(54/152): flexiblas-3.0.4-8.el9.x86_64.rpm 390 kB/s | 33 kB 00:00 2024-04-30T08:19:10.296 INFO:teuthology.orchestra.run.smithi189.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 242 kB/s | 18 kB 00:00 2024-04-30T08:19:10.322 INFO:teuthology.orchestra.run.smithi189.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 148 kB/s | 11 kB 00:00 2024-04-30T08:19:10.389 INFO:teuthology.orchestra.run.smithi189.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.7 MB/s | 160 kB 00:00 2024-04-30T08:19:10.447 INFO:teuthology.orchestra.run.smithi189.stdout:(58/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 12 MB/s | 3.0 MB 00:00 2024-04-30T08:19:10.465 INFO:teuthology.orchestra.run.smithi189.stdout:(59/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 318 kB/s | 45 kB 00:00 2024-04-30T08:19:10.540 INFO:teuthology.orchestra.run.smithi189.stdout:(60/152): librdkafka-1.6.1-102.el9.x86_64.rpm 4.3 MB/s | 662 kB 00:00 2024-04-30T08:19:10.565 INFO:teuthology.orchestra.run.smithi189.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 254 kB/s | 25 kB 00:00 2024-04-30T08:19:10.590 INFO:teuthology.orchestra.run.smithi189.stdout:(62/152): libtool-2.4.6-45.el9.x86_64.rpm 4.0 MB/s | 582 kB 00:00 2024-04-30T08:19:10.649 INFO:teuthology.orchestra.run.smithi189.stdout:(63/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 436 kB/s | 47 kB 00:00 2024-04-30T08:19:10.674 INFO:teuthology.orchestra.run.smithi189.stdout:(64/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.6 MB/s | 292 kB 00:00 2024-04-30T08:19:10.733 INFO:teuthology.orchestra.run.smithi189.stdout:(65/152): lua-5.4.4-4.el9.x86_64.rpm 1.3 MB/s | 188 kB 00:00 2024-04-30T08:19:10.758 INFO:teuthology.orchestra.run.smithi189.stdout:(66/152): openblas-0.3.21-2.el9.x86_64.rpm 324 kB/s | 35 kB 00:00 2024-04-30T08:19:11.075 INFO:teuthology.orchestra.run.smithi189.stdout:(67/152): openblas-openmp-0.3.21-2.el9.x86_64.r 12 MB/s | 4.7 MB 00:00 2024-04-30T08:19:11.238 INFO:teuthology.orchestra.run.smithi189.stdout:(68/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 8.8 MB/s | 4.4 MB 00:00 2024-04-30T08:19:11.275 INFO:teuthology.orchestra.run.smithi189.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 4.4 MB/s | 911 kB 00:00 2024-04-30T08:19:11.342 INFO:teuthology.orchestra.run.smithi189.stdout:(70/152): postgresql-13.14-1.el9.x86_64.rpm 2.7 MB/s | 1.6 MB 00:00 2024-04-30T08:19:11.368 INFO:teuthology.orchestra.run.smithi189.stdout:(71/152): postgresql-private-libs-13.14-1.el9.x 1.0 MB/s | 133 kB 00:00 2024-04-30T08:19:11.493 INFO:teuthology.orchestra.run.smithi189.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 699 kB/s | 87 kB 00:00 2024-04-30T08:19:11.527 INFO:teuthology.orchestra.run.smithi189.stdout:(73/152): protobuf-3.14.0-13.el9.x86_64.rpm 5.5 MB/s | 1.0 MB 00:00 2024-04-30T08:19:11.644 INFO:teuthology.orchestra.run.smithi189.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 2.0 MB/s | 244 kB 00:00 2024-04-30T08:19:11.794 INFO:teuthology.orchestra.run.smithi189.stdout:(75/152): postgresql-server-13.14-1.el9.x86_64. 11 MB/s | 5.8 MB 00:00 2024-04-30T08:19:11.819 INFO:teuthology.orchestra.run.smithi189.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 1.4 MB/s | 248 kB 00:00 2024-04-30T08:19:11.878 INFO:teuthology.orchestra.run.smithi189.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 307 kB/s | 26 kB 00:00 2024-04-30T08:19:11.903 INFO:teuthology.orchestra.run.smithi189.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 229 kB/s | 19 kB 00:00 2024-04-30T08:19:11.970 INFO:teuthology.orchestra.run.smithi189.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 1.8 MB/s | 172 kB 00:00 2024-04-30T08:19:11.995 INFO:teuthology.orchestra.run.smithi189.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 378 kB/s | 35 kB 00:00 2024-04-30T08:19:12.212 INFO:teuthology.orchestra.run.smithi189.stdout:(81/152): python3-babel-2.9.1-2.el9.noarch.rpm 8.3 MB/s | 6.0 MB 00:00 2024-04-30T08:19:12.237 INFO:teuthology.orchestra.run.smithi189.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.0 MB/s | 250 kB 00:00 2024-04-30T08:19:12.338 INFO:teuthology.orchestra.run.smithi189.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 1.7 MB/s | 222 kB 00:00 2024-04-30T08:19:12.363 INFO:teuthology.orchestra.run.smithi189.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 616 kB/s | 77 kB 00:00 2024-04-30T08:19:12.480 INFO:teuthology.orchestra.run.smithi189.stdout:(85/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 10 MB/s | 5.1 MB 00:00 2024-04-30T08:19:12.505 INFO:teuthology.orchestra.run.smithi189.stdout:(86/152): python3-prettytable-0.7.2-27.el9.noar 252 kB/s | 42 kB 00:00 2024-04-30T08:19:12.547 INFO:teuthology.orchestra.run.smithi189.stdout:(87/152): python3-protobuf-3.14.0-13.el9.noarch 1.4 MB/s | 269 kB 00:00 2024-04-30T08:19:12.590 INFO:teuthology.orchestra.run.smithi189.stdout:(88/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-04-30T08:19:12.615 INFO:teuthology.orchestra.run.smithi189.stdout:(89/152): python3-pyasn1-modules-0.4.8-6.el9.no 2.5 MB/s | 279 kB 00:00 2024-04-30T08:19:12.648 INFO:teuthology.orchestra.run.smithi189.stdout:(90/152): python3-pytz-2021.1-5.el9.noarch.rpm 507 kB/s | 51 kB 00:00 2024-04-30T08:19:12.652 INFO:teuthology.orchestra.run.smithi071.stdout: Preparing : 1/1 2024-04-30T08:19:12.682 INFO:teuthology.orchestra.run.smithi189.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 584 kB/s | 54 kB 00:00 2024-04-30T08:19:12.747 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-30T08:19:12.785 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-30T08:19:12.793 INFO:teuthology.orchestra.run.smithi189.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 288 kB/s | 42 kB 00:00 2024-04-30T08:19:12.816 INFO:teuthology.orchestra.run.smithi189.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 331 kB/s | 44 kB 00:00 2024-04-30T08:19:12.890 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-30T08:19:12.923 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-30T08:19:12.924 INFO:teuthology.orchestra.run.smithi189.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 1.6 MB/s | 221 kB 00:00 2024-04-30T08:19:12.941 INFO:teuthology.orchestra.run.smithi189.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 287 kB/s | 36 kB 00:00 2024-04-30T08:19:12.948 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-30T08:19:12.994 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-30T08:19:13.008 INFO:teuthology.orchestra.run.smithi189.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 567 kB/s | 47 kB 00:00 2024-04-30T08:19:13.084 INFO:teuthology.orchestra.run.smithi189.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 772 kB/s | 58 kB 00:00 2024-04-30T08:19:13.117 INFO:teuthology.orchestra.run.smithi189.stdout:(98/152): socat-1.7.4.1-5.el9.x86_64.rpm 1.7 MB/s | 305 kB 00:00 2024-04-30T08:19:13.168 INFO:teuthology.orchestra.run.smithi189.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.1 MB/s | 94 kB 00:00 2024-04-30T08:19:13.193 INFO:teuthology.orchestra.run.smithi189.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 847 kB/s | 64 kB 00:00 2024-04-30T08:19:13.251 INFO:teuthology.orchestra.run.smithi189.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 537 kB/s | 45 kB 00:00 2024-04-30T08:19:13.336 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-30T08:19:13.377 INFO:teuthology.orchestra.run.smithi189.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 121 kB/s | 22 kB 00:00 2024-04-30T08:19:13.450 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : librados2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 8/156 2024-04-30T08:19:13.560 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 8/156 2024-04-30T08:19:13.584 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephfs2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 9/156 2024-04-30T08:19:13.675 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephfs2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 9/156 2024-04-30T08:19:13.714 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rados-2:19.0.0-3348.g8feaf7d9.el9.x86_64 10/156 2024-04-30T08:19:13.744 INFO:teuthology.orchestra.run.smithi189.stdout:(103/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 14 MB/s | 16 MB 00:01 2024-04-30T08:19:13.811 INFO:teuthology.orchestra.run.smithi189.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 1.5 MB/s | 863 kB 00:00 2024-04-30T08:19:13.865 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-30T08:19:13.911 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-30T08:19:13.947 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephsqlite-2:19.0.0-3348.g8feaf7d9.el9.x86_64 13/156 2024-04-30T08:19:14.052 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3348.g8feaf7d9.el9.x86_64 13/156 2024-04-30T08:19:14.082 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libradosstriper1-2:19.0.0-3348.g8feaf7d9.el9.x86 14/156 2024-04-30T08:19:14.094 INFO:teuthology.orchestra.run.smithi189.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 76 kB/s | 19 kB 00:00 2024-04-30T08:19:14.144 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3348.g8feaf7d9.el9.x86 14/156 2024-04-30T08:19:14.166 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-30T08:19:14.185 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-30T08:19:14.186 INFO:teuthology.orchestra.run.smithi189.stdout:(106/152): abseil-cpp-20211102.0-3.el9.x86_64.r 677 kB/s | 548 kB 00:00 2024-04-30T08:19:14.220 INFO:teuthology.orchestra.run.smithi189.stdout:(107/152): gperftools-libs-2.9.1-2.el9.x86_64.r 650 kB/s | 309 kB 00:00 2024-04-30T08:19:14.278 INFO:teuthology.orchestra.run.smithi189.stdout:(108/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 283 kB/s | 26 kB 00:00 2024-04-30T08:19:14.304 INFO:teuthology.orchestra.run.smithi189.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 581 kB/s | 49 kB 00:00 2024-04-30T08:19:14.384 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-30T08:19:14.395 INFO:teuthology.orchestra.run.smithi189.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 576 kB/s | 67 kB 00:00 2024-04-30T08:19:14.416 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-30T08:19:14.421 INFO:teuthology.orchestra.run.smithi189.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 1.3 MB/s | 151 kB 00:00 2024-04-30T08:19:14.524 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-30T08:19:14.563 INFO:teuthology.orchestra.run.smithi189.stdout:(112/152): python3-asyncssh-2.13.2-5.el9.noarch 3.8 MB/s | 548 kB 00:00 2024-04-30T08:19:14.619 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-30T08:19:14.630 INFO:teuthology.orchestra.run.smithi189.stdout:(113/152): libarrow-9.0.0-11.el9.x86_64.rpm 8.3 MB/s | 4.4 MB 00:00 2024-04-30T08:19:14.657 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-30T08:19:14.664 INFO:teuthology.orchestra.run.smithi189.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 432 kB/s | 43 kB 00:00 2024-04-30T08:19:14.692 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-30T08:19:14.705 INFO:teuthology.orchestra.run.smithi189.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 426 kB/s | 32 kB 00:00 2024-04-30T08:19:14.739 INFO:teuthology.orchestra.run.smithi189.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 200 kB/s | 15 kB 00:00 2024-04-30T08:19:14.757 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-30T08:19:14.773 INFO:teuthology.orchestra.run.smithi189.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-04-30T08:19:14.818 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-30T08:19:14.831 INFO:teuthology.orchestra.run.smithi189.stdout:(118/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 1.9 MB/s | 839 kB 00:00 2024-04-30T08:19:14.862 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-30T08:19:14.865 INFO:teuthology.orchestra.run.smithi189.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 2.8 MB/s | 358 kB 00:00 2024-04-30T08:19:14.890 INFO:teuthology.orchestra.run.smithi189.stdout:(120/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.5 MB/s | 303 kB 00:00 2024-04-30T08:19:14.894 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-30T08:19:14.915 INFO:teuthology.orchestra.run.smithi189.stdout:(121/152): python3-colorama-0.4.4-12.el9.noarch 421 kB/s | 35 kB 00:00 2024-04-30T08:19:14.922 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-30T08:19:14.949 INFO:teuthology.orchestra.run.smithi189.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 2.4 MB/s | 206 kB 00:00 2024-04-30T08:19:14.992 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-30T08:19:15.008 INFO:teuthology.orchestra.run.smithi189.stdout:(123/152): python3-grpcio-1.46.7-10.el9.x86_64. 17 MB/s | 2.0 MB 00:00 2024-04-30T08:19:15.033 INFO:teuthology.orchestra.run.smithi189.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 128 kB/s | 11 kB 00:00 2024-04-30T08:19:15.045 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-30T08:19:15.058 INFO:teuthology.orchestra.run.smithi189.stdout:(125/152): python3-grpcio-tools-1.46.7-10.el9.x 1.0 MB/s | 144 kB 00:00 2024-04-30T08:19:15.076 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-30T08:19:15.084 INFO:teuthology.orchestra.run.smithi189.stdout:(126/152): python3-jaraco-classes-3.2.1-5.el9.n 234 kB/s | 18 kB 00:00 2024-04-30T08:19:15.109 INFO:teuthology.orchestra.run.smithi189.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 307 kB/s | 23 kB 00:00 2024-04-30T08:19:15.119 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-30T08:19:15.134 INFO:teuthology.orchestra.run.smithi189.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 257 kB/s | 19 kB 00:00 2024-04-30T08:19:15.159 INFO:teuthology.orchestra.run.smithi189.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 261 kB/s | 20 kB 00:00 2024-04-30T08:19:15.193 INFO:teuthology.orchestra.run.smithi189.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 12 MB/s | 1.0 MB 00:00 2024-04-30T08:19:15.197 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-30T08:19:15.227 INFO:teuthology.orchestra.run.smithi189.stdout:(131/152): python3-logutils-0.3.5-21.el9.noarch 503 kB/s | 46 kB 00:00 2024-04-30T08:19:15.230 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ceph-common-2:19.0.0-3348.g8feaf7d9.el9. 31/156 2024-04-30T08:19:15.252 INFO:teuthology.orchestra.run.smithi189.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 855 kB/s | 79 kB 00:00 2024-04-30T08:19:15.273 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ceph-argparse-2:19.0.0-3348.g8feaf7d9.el 32/156 2024-04-30T08:19:15.277 INFO:teuthology.orchestra.run.smithi189.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 689 kB/s | 58 kB 00:00 2024-04-30T08:19:15.302 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cephfs-2:19.0.0-3348.g8feaf7d9.el9.x86_6 33/156 2024-04-30T08:19:15.319 INFO:teuthology.orchestra.run.smithi189.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 246 kB/s | 16 kB 00:00 2024-04-30T08:19:15.353 INFO:teuthology.orchestra.run.smithi189.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-04-30T08:19:15.373 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-30T08:19:15.386 INFO:teuthology.orchestra.run.smithi189.stdout:(136/152): python3-pecan-1.4.2-3.el9.noarch.rpm 1.7 MB/s | 272 kB 00:00 2024-04-30T08:19:15.412 INFO:teuthology.orchestra.run.smithi189.stdout:(137/152): python3-pyperclip-1.8.2-2.el9.noarch 307 kB/s | 28 kB 00:00 2024-04-30T08:19:15.417 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-30T08:19:15.437 INFO:teuthology.orchestra.run.smithi189.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 367 kB/s | 31 kB 00:00 2024-04-30T08:19:15.487 INFO:teuthology.orchestra.run.smithi189.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 786 kB/s | 59 kB 00:00 2024-04-30T08:19:15.492 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-30T08:19:15.513 INFO:teuthology.orchestra.run.smithi189.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 475 kB/s | 36 kB 00:00 2024-04-30T08:19:15.538 INFO:teuthology.orchestra.run.smithi189.stdout:(141/152): python3-routes-2.5.1-5.el9.noarch.rp 1.2 MB/s | 188 kB 00:00 2024-04-30T08:19:15.563 INFO:teuthology.orchestra.run.smithi189.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 675 kB/s | 51 kB 00:00 2024-04-30T08:19:15.588 INFO:teuthology.orchestra.run.smithi189.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-04-30T08:19:15.627 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-30T08:19:15.639 INFO:teuthology.orchestra.run.smithi189.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 889 kB/s | 90 kB 00:00 2024-04-30T08:19:15.664 INFO:teuthology.orchestra.run.smithi189.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 265 kB/s | 20 kB 00:00 2024-04-30T08:19:15.678 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-30T08:19:15.690 INFO:teuthology.orchestra.run.smithi189.stdout:(146/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-04-30T08:19:15.765 INFO:teuthology.orchestra.run.smithi189.stdout:(147/152): thrift-0.15.0-2.el9.x86_64.rpm 16 MB/s | 1.6 MB 00:00 2024-04-30T08:19:15.790 INFO:teuthology.orchestra.run.smithi189.stdout:(148/152): re2-20211101-3.el9.x86_64.rpm 1.2 MB/s | 192 kB 00:00 2024-04-30T08:19:15.899 INFO:teuthology.orchestra.run.smithi189.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 12 MB/s | 1.3 MB 00:00 2024-04-30T08:19:15.990 INFO:teuthology.orchestra.run.smithi189.stdout:(150/152): librados2-19.0.0-3348.g8feaf7d9.el9. 11 MB/s | 3.3 MB 00:00 2024-04-30T08:19:16.082 INFO:teuthology.orchestra.run.smithi189.stdout:(151/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 13 MB/s | 2.4 MB 00:00 2024-04-30T08:19:16.217 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-30T08:19:16.274 INFO:teuthology.orchestra.run.smithi189.stdout:(152/152): librbd1-19.0.0-3348.g8feaf7d9.el9.x8 6.1 MB/s | 3.1 MB 00:00 2024-04-30T08:19:16.282 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-04-30T08:19:16.282 INFO:teuthology.orchestra.run.smithi189.stdout:Total 16 MB/s | 214 MB 00:13 2024-04-30T08:19:16.299 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-30T08:19:16.328 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-30T08:19:16.371 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-30T08:19:16.400 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librados-devel-2:19.0.0-3348.g8feaf7d9.el9.x86_6 43/156 2024-04-30T08:19:16.442 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-30T08:19:16.590 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-30T08:19:16.653 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-30T08:19:16.708 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-30T08:19:16.781 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-30T08:19:16.812 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-30T08:19:16.849 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-30T08:19:16.896 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-30T08:19:16.932 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-30T08:19:16.975 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-30T08:19:17.026 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-30T08:19:17.163 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-30T08:19:17.214 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-30T08:19:17.245 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-30T08:19:17.286 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-30T08:19:17.367 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-04-30T08:19:17.381 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-30T08:19:17.420 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-30T08:19:17.460 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-30T08:19:17.486 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-30T08:19:17.513 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-30T08:19:17.516 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-04-30T08:19:17.516 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-04-30T08:19:17.586 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-30T08:19:17.712 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-30T08:19:17.751 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-30T08:19:17.779 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-30T08:19:17.814 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-30T08:19:17.853 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-30T08:19:17.885 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-30T08:19:17.915 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-30T08:19:17.949 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-30T08:19:17.983 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-30T08:19:18.025 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-30T08:19:18.076 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-30T08:19:18.163 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-30T08:19:18.409 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-30T08:19:18.447 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-30T08:19:18.496 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-30T08:19:18.605 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-30T08:19:18.657 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-30T08:19:19.254 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-04-30T08:19:19.255 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-04-30T08:19:19.386 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-30T08:19:19.437 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-30T08:19:19.470 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-30T08:19:19.570 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-30T08:19:19.620 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-30T08:19:19.708 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-30T08:19:19.740 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-30T08:19:19.919 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-30T08:19:19.954 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-30T08:19:20.031 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-30T08:19:20.679 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-30T08:19:20.795 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-30T08:19:22.026 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-30T08:19:22.110 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-30T08:19:22.137 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-30T08:19:22.171 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-30T08:19:22.492 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-30T08:19:22.543 INFO:teuthology.orchestra.run.smithi071.stdout: Upgrading : librbd1-2:19.0.0-3348.g8feaf7d9.el9.x86_64 99/156 2024-04-30T08:19:22.633 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librbd1-2:19.0.0-3348.g8feaf7d9.el9.x86_64 99/156 2024-04-30T08:19:22.664 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rbd-2:19.0.0-3348.g8feaf7d9.el9.x86_64 100/156 2024-04-30T08:19:22.700 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-30T08:19:23.190 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-30T08:19:23.221 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : librgw2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 103/156 2024-04-30T08:19:23.292 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librgw2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 103/156 2024-04-30T08:19:23.314 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-rgw-2:19.0.0-3348.g8feaf7d9.el9.x86_64 104/156 2024-04-30T08:19:25.396 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-04-30T08:19:25.504 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-30T08:19:25.514 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-common-2:19.0.0-3348.g8feaf7d9.el9.x86_64 105/156 2024-04-30T08:19:25.542 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-30T08:19:25.553 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-common-2:19.0.0-3348.g8feaf7d9.el9.x86_64 105/156 2024-04-30T08:19:25.612 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-30T08:19:25.646 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-30T08:19:25.672 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-30T08:19:25.714 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-30T08:19:26.049 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-30T08:19:26.114 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-common-2:19.0.0-3348.g8feaf7d9.el9.x86_64 105/156 2024-04-30T08:19:26.156 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : librados2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 8/156 2024-04-30T08:19:26.165 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-base-2:19.0.0-3348.g8feaf7d9.el9.x86_64 106/156 2024-04-30T08:19:26.259 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-base-2:19.0.0-3348.g8feaf7d9.el9.x86_64 106/156 2024-04-30T08:19:26.259 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-30T08:19:26.259 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-30T08:19:26.259 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-30T08:19:26.280 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librados2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 8/156 2024-04-30T08:19:26.291 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-selinux-2:19.0.0-3348.g8feaf7d9.el9.x86_64 107/156 2024-04-30T08:19:26.315 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephfs2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 9/156 2024-04-30T08:19:26.406 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libcephfs2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 9/156 2024-04-30T08:19:26.445 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rados-2:19.0.0-3348.g8feaf7d9.el9.x86_64 10/156 2024-04-30T08:19:26.597 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-30T08:19:26.650 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-30T08:19:26.678 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephsqlite-2:19.0.0-3348.g8feaf7d9.el9.x86_64 13/156 2024-04-30T08:19:26.765 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3348.g8feaf7d9.el9.x86_64 13/156 2024-04-30T08:19:26.787 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libradosstriper1-2:19.0.0-3348.g8feaf7d9.el9.x86 14/156 2024-04-30T08:19:26.850 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3348.g8feaf7d9.el9.x86 14/156 2024-04-30T08:19:26.872 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-30T08:19:26.892 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-30T08:19:27.089 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-30T08:19:27.121 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-30T08:19:27.229 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-30T08:19:27.334 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-30T08:19:27.371 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-30T08:19:27.405 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-30T08:19:27.471 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-30T08:19:27.505 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-30T08:19:27.548 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-30T08:19:27.581 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-30T08:19:27.609 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-30T08:19:27.678 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-30T08:19:27.731 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-30T08:19:27.761 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-30T08:19:27.796 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-30T08:19:27.873 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-30T08:19:27.908 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ceph-common-2:19.0.0-3348.g8feaf7d9.el9. 31/156 2024-04-30T08:19:27.950 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ceph-argparse-2:19.0.0-3348.g8feaf7d9.el 32/156 2024-04-30T08:19:27.979 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cephfs-2:19.0.0-3348.g8feaf7d9.el9.x86_6 33/156 2024-04-30T08:19:28.049 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-30T08:19:28.094 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-30T08:19:28.169 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-30T08:19:28.304 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-30T08:19:28.355 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-30T08:19:28.881 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-30T08:19:28.892 INFO:teuthology.orchestra.run.smithi032.stdout:Extra Packages for Enterprise Linux 1.0 MB/s | 21 MB 00:20 2024-04-30T08:19:28.977 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-30T08:19:29.004 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-30T08:19:29.047 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-30T08:19:29.085 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librados-devel-2:19.0.0-3348.g8feaf7d9.el9.x86_6 43/156 2024-04-30T08:19:29.127 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-30T08:19:29.280 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-30T08:19:29.338 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-30T08:19:29.393 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-30T08:19:29.470 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-30T08:19:29.505 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-30T08:19:29.542 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-30T08:19:29.589 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-30T08:19:29.624 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-30T08:19:29.667 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-30T08:19:29.709 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-30T08:19:29.849 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-30T08:19:29.890 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-30T08:19:29.920 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-30T08:19:29.962 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-30T08:19:30.058 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-30T08:19:30.095 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-30T08:19:30.135 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-30T08:19:30.177 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-30T08:19:30.202 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-30T08:19:30.277 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-30T08:19:30.405 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-30T08:19:30.443 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-30T08:19:30.462 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-30T08:19:30.496 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-30T08:19:30.535 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-30T08:19:30.567 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-30T08:19:30.597 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-30T08:19:30.632 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-30T08:19:30.665 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-30T08:19:30.849 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-30T08:19:30.883 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-30T08:19:30.972 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-30T08:19:31.029 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-30T08:19:31.060 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-30T08:19:31.108 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-30T08:19:31.209 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-30T08:19:31.264 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-30T08:19:32.011 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-30T08:19:32.059 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-30T08:19:32.093 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-30T08:19:32.194 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-30T08:19:32.243 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-30T08:19:32.333 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-30T08:19:32.370 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-30T08:19:32.538 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-30T08:19:32.568 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-30T08:19:32.645 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-30T08:19:33.304 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-30T08:19:33.417 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-30T08:19:34.662 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-30T08:19:34.746 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-30T08:19:34.795 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-30T08:19:34.824 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-30T08:19:35.029 INFO:teuthology.orchestra.run.smithi032.stdout:lab-extras 28 kB/s | 1.7 kB 00:00 2024-04-30T08:19:35.144 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-30T08:19:35.195 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : librbd1-2:19.0.0-3348.g8feaf7d9.el9.x86_64 99/156 2024-04-30T08:19:35.286 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librbd1-2:19.0.0-3348.g8feaf7d9.el9.x86_64 99/156 2024-04-30T08:19:35.317 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rbd-2:19.0.0-3348.g8feaf7d9.el9.x86_64 100/156 2024-04-30T08:19:35.353 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-30T08:19:35.844 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-30T08:19:35.890 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librgw2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 103/156 2024-04-30T08:19:35.961 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librgw2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 103/156 2024-04-30T08:19:35.984 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rgw-2:19.0.0-3348.g8feaf7d9.el9.x86_64 104/156 2024-04-30T08:19:37.540 INFO:teuthology.orchestra.run.smithi032.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-30T08:19:37.542 INFO:teuthology.orchestra.run.smithi032.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-30T08:19:37.563 INFO:teuthology.orchestra.run.smithi032.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-30T08:19:37.564 INFO:teuthology.orchestra.run.smithi032.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-30T08:19:37.565 INFO:teuthology.orchestra.run.smithi032.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-30T08:19:37.567 INFO:teuthology.orchestra.run.smithi032.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-30T08:19:37.569 INFO:teuthology.orchestra.run.smithi032.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-30T08:19:37.573 INFO:teuthology.orchestra.run.smithi032.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-30T08:19:37.574 INFO:teuthology.orchestra.run.smithi032.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-30T08:19:37.651 INFO:teuthology.orchestra.run.smithi032.stdout:Dependencies resolved. 2024-04-30T08:19:37.672 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================= 2024-04-30T08:19:37.672 INFO:teuthology.orchestra.run.smithi032.stdout: Package Arch Version Repository Size 2024-04-30T08:19:37.672 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================= 2024-04-30T08:19:37.672 INFO:teuthology.orchestra.run.smithi032.stdout:Installing: 2024-04-30T08:19:37.672 INFO:teuthology.orchestra.run.smithi032.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-30T08:19:37.672 INFO:teuthology.orchestra.run.smithi032.stdout: ceph x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 6.6 k 2024-04-30T08:19:37.672 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 5.4 M 2024-04-30T08:19:37.672 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-fuse x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 863 k 2024-04-30T08:19:37.672 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 147 k 2024-04-30T08:19:37.672 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 1.5 M 2024-04-30T08:19:37.672 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3348.g8feaf7d9.el9 ceph-noarch 142 k 2024-04-30T08:19:37.673 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3348.g8feaf7d9.el9 ceph-noarch 3.5 M 2024-04-30T08:19:37.673 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3348.g8feaf7d9.el9 ceph-noarch 7.4 M 2024-04-30T08:19:37.673 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-rook noarch 2:19.0.0-3348.g8feaf7d9.el9 ceph-noarch 49 k 2024-04-30T08:19:37.673 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-radosgw x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 10 M 2024-04-30T08:19:37.673 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-test x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 43 M 2024-04-30T08:19:37.673 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-volume noarch 2:19.0.0-3348.g8feaf7d9.el9 ceph-noarch 273 k 2024-04-30T08:19:37.673 INFO:teuthology.orchestra.run.smithi032.stdout: cephadm noarch 2:19.0.0-3348.g8feaf7d9.el9 ceph-noarch 762 k 2024-04-30T08:19:37.673 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-mirror x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 224 k 2024-04-30T08:19:37.673 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-shell x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 27 k 2024-04-30T08:19:37.673 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-top noarch 2:19.0.0-3348.g8feaf7d9.el9 ceph-noarch 18 k 2024-04-30T08:19:37.673 INFO:teuthology.orchestra.run.smithi032.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-30T08:19:37.673 INFO:teuthology.orchestra.run.smithi032.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-30T08:19:37.673 INFO:teuthology.orchestra.run.smithi032.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-30T08:19:37.674 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs-devel x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 32 k 2024-04-30T08:19:37.674 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs2 x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 756 k 2024-04-30T08:19:37.674 INFO:teuthology.orchestra.run.smithi032.stdout: librados-devel x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 127 k 2024-04-30T08:19:37.674 INFO:teuthology.orchestra.run.smithi032.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-30T08:19:37.674 INFO:teuthology.orchestra.run.smithi032.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-30T08:19:37.674 INFO:teuthology.orchestra.run.smithi032.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-30T08:19:37.674 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-30T08:19:37.674 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-30T08:19:37.674 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-30T08:19:37.674 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cephfs x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 161 k 2024-04-30T08:19:37.674 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rados x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 321 k 2024-04-30T08:19:37.675 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rbd x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 298 k 2024-04-30T08:19:37.675 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rgw x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 100 k 2024-04-30T08:19:37.675 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-fuse x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 86 k 2024-04-30T08:19:37.675 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-mirror x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 3.0 M 2024-04-30T08:19:37.675 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-nbd x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 169 k 2024-04-30T08:19:37.675 INFO:teuthology.orchestra.run.smithi032.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-30T08:19:37.675 INFO:teuthology.orchestra.run.smithi032.stdout:Upgrading: 2024-04-30T08:19:37.675 INFO:teuthology.orchestra.run.smithi032.stdout: librados2 x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 3.3 M 2024-04-30T08:19:37.675 INFO:teuthology.orchestra.run.smithi032.stdout: librbd1 x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 3.1 M 2024-04-30T08:19:37.675 INFO:teuthology.orchestra.run.smithi032.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-30T08:19:37.675 INFO:teuthology.orchestra.run.smithi032.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-30T08:19:37.676 INFO:teuthology.orchestra.run.smithi032.stdout:Installing dependencies: 2024-04-30T08:19:37.676 INFO:teuthology.orchestra.run.smithi032.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-30T08:19:37.676 INFO:teuthology.orchestra.run.smithi032.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-30T08:19:37.676 INFO:teuthology.orchestra.run.smithi032.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-30T08:19:37.676 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-common x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 20 M 2024-04-30T08:19:37.676 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3348.g8feaf7d9.el9 ceph-noarch 34 k 2024-04-30T08:19:37.676 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mds x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 2.3 M 2024-04-30T08:19:37.676 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3348.g8feaf7d9.el9 ceph-noarch 249 k 2024-04-30T08:19:37.676 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mon x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 4.9 M 2024-04-30T08:19:37.676 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-osd x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 18 M 2024-04-30T08:19:37.676 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3348.g8feaf7d9.el9 ceph-noarch 16 k 2024-04-30T08:19:37.676 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-selinux x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 25 k 2024-04-30T08:19:37.676 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-30T08:19:37.677 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-30T08:19:37.677 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-30T08:19:37.677 INFO:teuthology.orchestra.run.smithi032.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-30T08:19:37.677 INFO:teuthology.orchestra.run.smithi032.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-30T08:19:37.677 INFO:teuthology.orchestra.run.smithi032.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-30T08:19:37.677 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-04-30T08:19:37.677 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-04-30T08:19:37.677 INFO:teuthology.orchestra.run.smithi032.stdout: libcephsqlite x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 158 k 2024-04-30T08:19:37.677 INFO:teuthology.orchestra.run.smithi032.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-30T08:19:37.677 INFO:teuthology.orchestra.run.smithi032.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-30T08:19:37.677 INFO:teuthology.orchestra.run.smithi032.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-30T08:19:37.677 INFO:teuthology.orchestra.run.smithi032.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-30T08:19:37.678 INFO:teuthology.orchestra.run.smithi032.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-30T08:19:37.678 INFO:teuthology.orchestra.run.smithi032.stdout: libradosstriper1 x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 497 k 2024-04-30T08:19:37.678 INFO:teuthology.orchestra.run.smithi032.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-30T08:19:37.678 INFO:teuthology.orchestra.run.smithi032.stdout: librgw2 x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 5.2 M 2024-04-30T08:19:37.678 INFO:teuthology.orchestra.run.smithi032.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-30T08:19:37.678 INFO:teuthology.orchestra.run.smithi032.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-30T08:19:37.678 INFO:teuthology.orchestra.run.smithi032.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-30T08:19:37.678 INFO:teuthology.orchestra.run.smithi032.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-30T08:19:37.678 INFO:teuthology.orchestra.run.smithi032.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-30T08:19:37.678 INFO:teuthology.orchestra.run.smithi032.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-30T08:19:37.679 INFO:teuthology.orchestra.run.smithi032.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-30T08:19:37.679 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-30T08:19:37.679 INFO:teuthology.orchestra.run.smithi032.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-04-30T08:19:37.679 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-30T08:19:37.679 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-30T08:19:37.679 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-30T08:19:37.679 INFO:teuthology.orchestra.run.smithi032.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-30T08:19:37.679 INFO:teuthology.orchestra.run.smithi032.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-30T08:19:37.679 INFO:teuthology.orchestra.run.smithi032.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-30T08:19:37.679 INFO:teuthology.orchestra.run.smithi032.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-30T08:19:37.679 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-30T08:19:37.679 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-argparse x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 45 k 2024-04-30T08:19:37.680 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-common x86_64 2:19.0.0-3348.g8feaf7d9.el9 ceph 135 k 2024-04-30T08:19:37.680 INFO:teuthology.orchestra.run.smithi032.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-30T08:19:37.680 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-30T08:19:37.680 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-30T08:19:37.680 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-30T08:19:37.680 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-30T08:19:37.680 INFO:teuthology.orchestra.run.smithi032.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-30T08:19:37.680 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-30T08:19:37.680 INFO:teuthology.orchestra.run.smithi032.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-30T08:19:37.680 INFO:teuthology.orchestra.run.smithi032.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-30T08:19:37.680 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-30T08:19:37.681 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-30T08:19:37.681 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-30T08:19:37.681 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-30T08:19:37.681 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-30T08:19:37.681 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-30T08:19:37.681 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-30T08:19:37.681 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-30T08:19:37.681 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-30T08:19:37.681 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-30T08:19:37.681 INFO:teuthology.orchestra.run.smithi032.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-30T08:19:37.681 INFO:teuthology.orchestra.run.smithi032.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-30T08:19:37.681 INFO:teuthology.orchestra.run.smithi032.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-30T08:19:37.681 INFO:teuthology.orchestra.run.smithi032.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-30T08:19:37.682 INFO:teuthology.orchestra.run.smithi032.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-30T08:19:37.682 INFO:teuthology.orchestra.run.smithi032.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-30T08:19:37.682 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-30T08:19:37.682 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-30T08:19:37.682 INFO:teuthology.orchestra.run.smithi032.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-30T08:19:37.682 INFO:teuthology.orchestra.run.smithi032.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-30T08:19:37.682 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-30T08:19:37.682 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-30T08:19:37.682 INFO:teuthology.orchestra.run.smithi032.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-30T08:19:37.682 INFO:teuthology.orchestra.run.smithi032.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-30T08:19:37.682 INFO:teuthology.orchestra.run.smithi032.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-30T08:19:37.682 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-30T08:19:37.683 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-30T08:19:37.683 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-30T08:19:37.683 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-30T08:19:37.683 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-30T08:19:37.683 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-30T08:19:37.683 INFO:teuthology.orchestra.run.smithi032.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-30T08:19:37.683 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-30T08:19:37.683 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-30T08:19:37.683 INFO:teuthology.orchestra.run.smithi032.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-30T08:19:37.683 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-30T08:19:37.683 INFO:teuthology.orchestra.run.smithi032.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-30T08:19:37.683 INFO:teuthology.orchestra.run.smithi032.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-30T08:19:37.683 INFO:teuthology.orchestra.run.smithi032.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-30T08:19:37.684 INFO:teuthology.orchestra.run.smithi032.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-30T08:19:37.684 INFO:teuthology.orchestra.run.smithi032.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-30T08:19:37.684 INFO:teuthology.orchestra.run.smithi032.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-30T08:19:37.684 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-30T08:19:37.684 INFO:teuthology.orchestra.run.smithi032.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-30T08:19:37.684 INFO:teuthology.orchestra.run.smithi032.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-30T08:19:37.684 INFO:teuthology.orchestra.run.smithi032.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-30T08:19:37.684 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-30T08:19:37.684 INFO:teuthology.orchestra.run.smithi032.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-30T08:19:37.684 INFO:teuthology.orchestra.run.smithi032.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-30T08:19:37.684 INFO:teuthology.orchestra.run.smithi032.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-30T08:19:37.684 INFO:teuthology.orchestra.run.smithi032.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-30T08:19:37.684 INFO:teuthology.orchestra.run.smithi032.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-30T08:19:37.685 INFO:teuthology.orchestra.run.smithi032.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-30T08:19:37.685 INFO:teuthology.orchestra.run.smithi032.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-30T08:19:37.685 INFO:teuthology.orchestra.run.smithi032.stdout:Installing weak dependencies: 2024-04-30T08:19:37.685 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-30T08:19:37.685 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:19:37.685 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction Summary 2024-04-30T08:19:37.685 INFO:teuthology.orchestra.run.smithi032.stdout:======================================================================================= 2024-04-30T08:19:37.685 INFO:teuthology.orchestra.run.smithi032.stdout:Install 148 Packages 2024-04-30T08:19:37.685 INFO:teuthology.orchestra.run.smithi032.stdout:Upgrade 4 Packages 2024-04-30T08:19:37.685 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:19:37.686 INFO:teuthology.orchestra.run.smithi032.stdout:Total download size: 214 M 2024-04-30T08:19:37.686 INFO:teuthology.orchestra.run.smithi032.stdout:Downloading Packages: 2024-04-30T08:19:37.985 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3348.g8feaf7d9.el9.x86_64 107/156 2024-04-30T08:19:37.985 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /sys 2024-04-30T08:19:37.985 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /proc 2024-04-30T08:19:37.985 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /mnt 2024-04-30T08:19:37.985 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /var/tmp 2024-04-30T08:19:37.985 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /home 2024-04-30T08:19:37.986 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /root 2024-04-30T08:19:37.986 INFO:teuthology.orchestra.run.smithi071.stdout:skipping the directory /tmp 2024-04-30T08:19:37.986 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-30T08:19:38.160 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-volume-2:19.0.0-3348.g8feaf7d9.el9.noarch 108/156 2024-04-30T08:19:38.175 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-volume-2:19.0.0-3348.g8feaf7d9.el9.noarch 108/156 2024-04-30T08:19:38.176 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-30T08:19:38.176 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-30T08:19:38.176 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-30T08:19:38.194 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-common-2:19.0.0-3348.g8feaf7d9.el9.x86_64 105/156 2024-04-30T08:19:38.230 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-common-2:19.0.0-3348.g8feaf7d9.el9.x86_64 105/156 2024-04-30T08:19:38.794 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-common-2:19.0.0-3348.g8feaf7d9.el9.x86_64 105/156 2024-04-30T08:19:38.806 INFO:teuthology.orchestra.run.smithi032.stdout:(1/152): ceph-19.0.0-3348.g8feaf7d9.el9.x86_64. 23 kB/s | 6.6 kB 00:00 2024-04-30T08:19:38.842 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-base-2:19.0.0-3348.g8feaf7d9.el9.x86_64 106/156 2024-04-30T08:19:38.930 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-base-2:19.0.0-3348.g8feaf7d9.el9.x86_64 106/156 2024-04-30T08:19:38.930 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-30T08:19:38.930 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-30T08:19:38.931 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-30T08:19:38.960 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-selinux-2:19.0.0-3348.g8feaf7d9.el9.x86_64 107/156 2024-04-30T08:19:39.123 INFO:teuthology.orchestra.run.smithi032.stdout:(2/152): ceph-fuse-19.0.0-3348.g8feaf7d9.el9.x8 2.7 MB/s | 863 kB 00:00 2024-04-30T08:19:39.240 INFO:teuthology.orchestra.run.smithi032.stdout:(3/152): ceph-base-19.0.0-3348.g8feaf7d9.el9.x8 7.5 MB/s | 5.4 MB 00:00 2024-04-30T08:19:39.274 INFO:teuthology.orchestra.run.smithi032.stdout:(4/152): ceph-immutable-object-cache-19.0.0-334 986 kB/s | 147 kB 00:00 2024-04-30T08:19:39.416 INFO:teuthology.orchestra.run.smithi032.stdout:(5/152): ceph-mds-19.0.0-3348.g8feaf7d9.el9.x86 13 MB/s | 2.3 MB 00:00 2024-04-30T08:19:39.608 INFO:teuthology.orchestra.run.smithi032.stdout:(6/152): ceph-mgr-19.0.0-3348.g8feaf7d9.el9.x86 4.5 MB/s | 1.5 MB 00:00 2024-04-30T08:19:39.733 INFO:teuthology.orchestra.run.smithi032.stdout:(7/152): ceph-mon-19.0.0-3348.g8feaf7d9.el9.x86 15 MB/s | 4.9 MB 00:00 2024-04-30T08:19:39.754 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-osd-2:19.0.0-3348.g8feaf7d9.el9.x86_64 109/156 2024-04-30T08:19:39.781 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-osd-2:19.0.0-3348.g8feaf7d9.el9.x86_64 109/156 2024-04-30T08:19:39.781 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-30T08:19:39.781 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-30T08:19:39.781 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-30T08:19:39.781 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-30T08:19:39.781 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-30T08:19:40.013 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mds-2:19.0.0-3348.g8feaf7d9.el9.x86_64 110/156 2024-04-30T08:19:40.036 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mds-2:19.0.0-3348.g8feaf7d9.el9.x86_64 110/156 2024-04-30T08:19:40.036 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-30T08:19:40.036 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-30T08:19:40.037 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-30T08:19:40.037 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-30T08:19:40.037 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-30T08:19:40.433 INFO:teuthology.orchestra.run.smithi032.stdout:(8/152): ceph-radosgw-19.0.0-3348.g8feaf7d9.el9 14 MB/s | 10 MB 00:00 2024-04-30T08:19:40.506 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mon-2:19.0.0-3348.g8feaf7d9.el9.x86_64 111/156 2024-04-30T08:19:40.517 INFO:teuthology.orchestra.run.smithi032.stdout:(9/152): ceph-selinux-19.0.0-3348.g8feaf7d9.el9 305 kB/s | 25 kB 00:00 2024-04-30T08:19:40.528 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mon-2:19.0.0-3348.g8feaf7d9.el9.x86_64 111/156 2024-04-30T08:19:40.528 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-30T08:19:40.528 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-30T08:19:40.529 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-30T08:19:40.529 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-30T08:19:40.529 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-30T08:19:40.638 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-30T08:19:40.704 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-30T08:19:40.759 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-30T08:19:40.946 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-30T08:19:41.000 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-30T08:19:41.078 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-30T08:19:41.167 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-30T08:19:41.329 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-30T08:19:41.367 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-30T08:19:41.399 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-30T08:19:41.427 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-30T08:19:41.759 INFO:teuthology.orchestra.run.smithi032.stdout:(10/152): ceph-osd-19.0.0-3348.g8feaf7d9.el9.x8 8.3 MB/s | 18 MB 00:02 2024-04-30T08:19:41.842 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: cephadm-2:19.0.0-3348.g8feaf7d9.el9.noarch 123/156 2024-04-30T08:19:41.866 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : cephadm-2:19.0.0-3348.g8feaf7d9.el9.noarch 123/156 2024-04-30T08:19:41.908 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3348.g8feaf7d9.e 124/156 2024-04-30T08:19:41.968 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3348.g8feaf7d9. 125/156 2024-04-30T08:19:41.998 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3348.g8feaf7d9.el9.noa 126/156 2024-04-30T08:19:42.134 INFO:teuthology.orchestra.run.smithi032.stdout:(11/152): cephfs-mirror-19.0.0-3348.g8feaf7d9.e 601 kB/s | 224 kB 00:00 2024-04-30T08:19:42.243 INFO:teuthology.orchestra.run.smithi032.stdout:(12/152): cephfs-shell-19.0.0-3348.g8feaf7d9.el 252 kB/s | 27 kB 00:00 2024-04-30T08:19:42.326 INFO:teuthology.orchestra.run.smithi032.stdout:(13/152): libcephfs-devel-19.0.0-3348.g8feaf7d9 386 kB/s | 32 kB 00:00 2024-04-30T08:19:42.769 INFO:teuthology.orchestra.run.smithi032.stdout:(14/152): ceph-common-19.0.0-3348.g8feaf7d9.el9 4.6 MB/s | 20 MB 00:04 2024-04-30T08:19:42.818 INFO:teuthology.orchestra.run.smithi032.stdout:(15/152): libcephfs2-19.0.0-3348.g8feaf7d9.el9. 1.5 MB/s | 756 kB 00:00 2024-04-30T08:19:42.852 INFO:teuthology.orchestra.run.smithi032.stdout:(16/152): libcephsqlite-19.0.0-3348.g8feaf7d9.e 1.9 MB/s | 158 kB 00:00 2024-04-30T08:19:42.886 INFO:teuthology.orchestra.run.smithi032.stdout:(17/152): librados-devel-19.0.0-3348.g8feaf7d9. 1.9 MB/s | 127 kB 00:00 2024-04-30T08:19:42.961 INFO:teuthology.orchestra.run.smithi032.stdout:(18/152): libradosstriper1-19.0.0-3348.g8feaf7d 4.5 MB/s | 497 kB 00:00 2024-04-30T08:19:43.032 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3348.g8feaf7d9.el9.noa 126/156 2024-04-30T08:19:43.053 INFO:teuthology.orchestra.run.smithi032.stdout:(19/152): python3-ceph-argparse-19.0.0-3348.g8f 497 kB/s | 45 kB 00:00 2024-04-30T08:19:43.066 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3348.g8feaf7d9.el9.n 127/156 2024-04-30T08:19:43.154 INFO:teuthology.orchestra.run.smithi032.stdout:(20/152): python3-ceph-common-19.0.0-3348.g8fea 1.3 MB/s | 135 kB 00:00 2024-04-30T08:19:43.262 INFO:teuthology.orchestra.run.smithi032.stdout:(21/152): python3-cephfs-19.0.0-3348.g8feaf7d9. 1.5 MB/s | 161 kB 00:00 2024-04-30T08:19:43.404 INFO:teuthology.orchestra.run.smithi032.stdout:(22/152): python3-rados-19.0.0-3348.g8feaf7d9.e 2.2 MB/s | 321 kB 00:00 2024-04-30T08:19:43.888 INFO:teuthology.orchestra.run.smithi032.stdout:(23/152): ceph-test-19.0.0-3348.g8feaf7d9.el9.x 13 MB/s | 43 MB 00:03 2024-04-30T08:19:43.953 INFO:teuthology.orchestra.run.smithi032.stdout:(24/152): python3-rbd-19.0.0-3348.g8feaf7d9.el9 544 kB/s | 298 kB 00:00 2024-04-30T08:19:44.027 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3348.g8feaf7d9.el9.n 127/156 2024-04-30T08:19:44.058 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3348.g8fe 128/156 2024-04-30T08:19:44.072 INFO:teuthology.orchestra.run.smithi032.stdout:(25/152): librgw2-19.0.0-3348.g8feaf7d9.el9.x86 4.4 MB/s | 5.2 MB 00:01 2024-04-30T08:19:44.097 INFO:teuthology.orchestra.run.smithi032.stdout:(26/152): python3-rgw-19.0.0-3348.g8feaf7d9.el9 480 kB/s | 100 kB 00:00 2024-04-30T08:19:44.123 INFO:teuthology.orchestra.run.smithi032.stdout:(27/152): rbd-fuse-19.0.0-3348.g8feaf7d9.el9.x8 511 kB/s | 86 kB 00:00 2024-04-30T08:19:44.156 INFO:teuthology.orchestra.run.smithi032.stdout:(28/152): rbd-nbd-19.0.0-3348.g8feaf7d9.el9.x86 2.9 MB/s | 169 kB 00:00 2024-04-30T08:19:44.161 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3348.g8fe 128/156 2024-04-30T08:19:44.190 INFO:teuthology.orchestra.run.smithi032.stdout:(29/152): ceph-grafana-dashboards-19.0.0-3348.g 519 kB/s | 34 kB 00:00 2024-04-30T08:19:44.224 INFO:teuthology.orchestra.run.smithi032.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3348.g8feaf7d 2.1 MB/s | 142 kB 00:00 2024-04-30T08:19:44.335 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3348.g8feaf7d9.el 129/156 2024-04-30T08:19:44.361 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-2:19.0.0-3348.g8feaf7d9.el9.x86_64 130/156 2024-04-30T08:19:44.384 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3348.g8feaf7d9.el9.x86_64 130/156 2024-04-30T08:19:44.384 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-30T08:19:44.384 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-30T08:19:44.384 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-30T08:19:44.384 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-30T08:19:44.384 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-30T08:19:44.424 INFO:teuthology.orchestra.run.smithi032.stdout:(31/152): rbd-mirror-19.0.0-3348.g8feaf7d9.el9. 8.7 MB/s | 3.0 MB 00:00 2024-04-30T08:19:44.428 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-mgr-rook-2:19.0.0-3348.g8feaf7d9.el9.noarch 131/156 2024-04-30T08:19:44.439 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3348.g8feaf7d9.el9.noarch 131/156 2024-04-30T08:19:44.532 INFO:teuthology.orchestra.run.smithi032.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3348.g8f 2.3 MB/s | 249 kB 00:00 2024-04-30T08:19:44.538 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-2:19.0.0-3348.g8feaf7d9.el9.x86_64 132/156 2024-04-30T08:19:44.585 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-fuse-2:19.0.0-3348.g8feaf7d9.el9.x86_64 133/156 2024-04-30T08:19:44.600 INFO:teuthology.orchestra.run.smithi032.stdout:(33/152): ceph-mgr-rook-19.0.0-3348.g8feaf7d9.e 743 kB/s | 49 kB 00:00 2024-04-30T08:19:44.691 INFO:teuthology.orchestra.run.smithi032.stdout:(34/152): ceph-mgr-dashboard-19.0.0-3348.g8feaf 7.1 MB/s | 3.5 MB 00:00 2024-04-30T08:19:44.708 INFO:teuthology.orchestra.run.smithi032.stdout:(35/152): ceph-prometheus-alerts-19.0.0-3348.g8 153 kB/s | 16 kB 00:00 2024-04-30T08:19:44.859 INFO:teuthology.orchestra.run.smithi032.stdout:(36/152): ceph-mgr-diskprediction-local-19.0.0- 12 MB/s | 7.4 MB 00:00 2024-04-30T08:19:44.892 INFO:teuthology.orchestra.run.smithi032.stdout:(37/152): ceph-volume-19.0.0-3348.g8feaf7d9.el9 1.3 MB/s | 273 kB 00:00 2024-04-30T08:19:44.926 INFO:teuthology.orchestra.run.smithi032.stdout:(38/152): cephadm-19.0.0-3348.g8feaf7d9.el9.noa 3.4 MB/s | 762 kB 00:00 2024-04-30T08:19:44.951 INFO:teuthology.orchestra.run.smithi032.stdout:(39/152): cephfs-top-19.0.0-3348.g8feaf7d9.el9. 201 kB/s | 18 kB 00:00 2024-04-30T08:19:45.110 INFO:teuthology.orchestra.run.smithi032.stdout:(40/152): fuse-2.9.9-15.el9.x86_64.rpm 371 kB/s | 80 kB 00:00 2024-04-30T08:19:45.210 INFO:teuthology.orchestra.run.smithi032.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 757 kB/s | 195 kB 00:00 2024-04-30T08:19:45.310 INFO:teuthology.orchestra.run.smithi032.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.2 MB/s | 253 kB 00:00 2024-04-30T08:19:45.394 INFO:teuthology.orchestra.run.smithi032.stdout:(43/152): libgfortran-11.4.1-3.el9.x86_64.rpm 1.7 MB/s | 803 kB 00:00 2024-04-30T08:19:45.461 INFO:teuthology.orchestra.run.smithi032.stdout:(44/152): python3-ply-3.11-14.el9.noarch.rpm 711 kB/s | 106 kB 00:00 2024-04-30T08:19:45.486 INFO:teuthology.orchestra.run.smithi032.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 1.4 MB/s | 135 kB 00:00 2024-04-30T08:19:45.522 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-30T08:19:45.555 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-radosgw-2:19.0.0-3348.g8feaf7d9.el9.x86_64 135/156 2024-04-30T08:19:45.578 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3348.g8feaf7d9.el9.x86_64 135/156 2024-04-30T08:19:45.578 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-30T08:19:45.578 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-30T08:19:45.578 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-30T08:19:45.579 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-30T08:19:45.579 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-30T08:19:45.586 INFO:teuthology.orchestra.run.smithi032.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 1.0 MB/s | 125 kB 00:00 2024-04-30T08:19:45.612 INFO:teuthology.orchestra.run.smithi032.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 1.7 MB/s | 215 kB 00:00 2024-04-30T08:19:45.622 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3348.g8feaf 136/156 2024-04-30T08:19:45.645 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3348.g8feaf 136/156 2024-04-30T08:19:45.645 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-30T08:19:45.645 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-30T08:19:45.645 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-30T08:19:45.690 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : cephfs-mirror-2:19.0.0-3348.g8feaf7d9.el9.x86_64 137/156 2024-04-30T08:19:45.712 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3348.g8feaf7d9.el9.x86_64 137/156 2024-04-30T08:19:45.713 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-30T08:19:45.713 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-30T08:19:45.713 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-30T08:19:45.713 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-30T08:19:45.713 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-30T08:19:45.729 INFO:teuthology.orchestra.run.smithi032.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 4.8 MB/s | 689 kB 00:00 2024-04-30T08:19:45.771 INFO:teuthology.orchestra.run.smithi032.stdout:(49/152): bison-3.7.4-5.el9.x86_64.rpm 5.8 MB/s | 943 kB 00:00 2024-04-30T08:19:45.821 INFO:teuthology.orchestra.run.smithi032.stdout:(50/152): elfutils-libelf-devel-0.190-2.el9.x86 438 kB/s | 21 kB 00:00 2024-04-30T08:19:45.846 INFO:teuthology.orchestra.run.smithi032.stdout:(51/152): boost-program-options-1.75.0-8.el9.x8 919 kB/s | 107 kB 00:00 2024-04-30T08:19:45.897 INFO:teuthology.orchestra.run.smithi032.stdout:(52/152): flexiblas-3.0.4-8.el9.x86_64.rpm 666 kB/s | 33 kB 00:00 2024-04-30T08:19:45.947 INFO:teuthology.orchestra.run.smithi032.stdout:(53/152): python3-cryptography-36.0.1-4.el9.x86 1.7 MB/s | 1.2 MB 00:00 2024-04-30T08:19:45.981 INFO:teuthology.orchestra.run.smithi032.stdout:(54/152): flex-2.6.4-9.el9.x86_64.rpm 1.9 MB/s | 313 kB 00:00 2024-04-30T08:19:46.023 INFO:teuthology.orchestra.run.smithi032.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 244 kB/s | 18 kB 00:00 2024-04-30T08:19:46.024 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-mirror-2:19.0.0-3348.g8feaf7d9.el9.x86_64 138/156 2024-04-30T08:19:46.046 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3348.g8feaf7d9.el9.x86_64 138/156 2024-04-30T08:19:46.046 INFO:teuthology.orchestra.run.smithi071.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-30T08:19:46.046 INFO:teuthology.orchestra.run.smithi071.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-30T08:19:46.046 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-30T08:19:46.046 INFO:teuthology.orchestra.run.smithi071.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-30T08:19:46.047 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-30T08:19:46.048 INFO:teuthology.orchestra.run.smithi032.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 169 kB/s | 11 kB 00:00 2024-04-30T08:19:46.098 INFO:teuthology.orchestra.run.smithi032.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.1 MB/s | 160 kB 00:00 2024-04-30T08:19:46.124 INFO:teuthology.orchestra.run.smithi032.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 610 kB/s | 45 kB 00:00 2024-04-30T08:19:46.216 INFO:teuthology.orchestra.run.smithi032.stdout:(59/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 9.4 MB/s | 3.0 MB 00:00 2024-04-30T08:19:46.283 INFO:teuthology.orchestra.run.smithi032.stdout:(60/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 387 kB/s | 25 kB 00:00 2024-04-30T08:19:46.325 INFO:teuthology.orchestra.run.smithi032.stdout:(61/152): libtool-2.4.6-45.el9.x86_64.rpm 2.9 MB/s | 582 kB 00:00 2024-04-30T08:19:46.358 INFO:teuthology.orchestra.run.smithi032.stdout:(62/152): librdkafka-1.6.1-102.el9.x86_64.rpm 2.8 MB/s | 662 kB 00:00 2024-04-30T08:19:46.383 INFO:teuthology.orchestra.run.smithi032.stdout:(63/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 476 kB/s | 47 kB 00:00 2024-04-30T08:19:46.434 INFO:teuthology.orchestra.run.smithi032.stdout:(64/152): openblas-0.3.21-2.el9.x86_64.rpm 718 kB/s | 35 kB 00:00 2024-04-30T08:19:46.468 INFO:teuthology.orchestra.run.smithi032.stdout:(65/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.0 MB/s | 292 kB 00:00 2024-04-30T08:19:46.484 INFO:teuthology.orchestra.run.smithi032.stdout:(66/152): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 188 kB 00:00 2024-04-30T08:19:46.668 INFO:teuthology.orchestra.run.smithi032.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 8.7 MB/s | 1.6 MB 00:00 2024-04-30T08:19:46.837 INFO:teuthology.orchestra.run.smithi032.stdout:(68/152): openblas-openmp-0.3.21-2.el9.x86_64.r 12 MB/s | 4.7 MB 00:00 2024-04-30T08:19:46.952 INFO:teuthology.orchestra.run.smithi032.stdout:(69/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 9.2 MB/s | 4.4 MB 00:00 2024-04-30T08:19:46.994 INFO:teuthology.orchestra.run.smithi032.stdout:(70/152): postgresql-contrib-13.14-1.el9.x86_64 2.7 MB/s | 911 kB 00:00 2024-04-30T08:19:47.019 INFO:teuthology.orchestra.run.smithi032.stdout:(71/152): postgresql-private-libs-13.14-1.el9.x 737 kB/s | 133 kB 00:00 2024-04-30T08:19:47.078 INFO:teuthology.orchestra.run.smithi032.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.5 MB/s | 87 kB 00:00 2024-04-30T08:19:47.161 INFO:teuthology.orchestra.run.smithi032.stdout:(73/152): protobuf-3.14.0-13.el9.x86_64.rpm 6.1 MB/s | 1.0 MB 00:00 2024-04-30T08:19:47.245 INFO:teuthology.orchestra.run.smithi032.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 2.9 MB/s | 244 kB 00:00 2024-04-30T08:19:47.329 INFO:teuthology.orchestra.run.smithi032.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 2.9 MB/s | 248 kB 00:00 2024-04-30T08:19:47.462 INFO:teuthology.orchestra.run.smithi032.stdout:(76/152): postgresql-server-13.14-1.el9.x86_64. 11 MB/s | 5.8 MB 00:00 2024-04-30T08:19:47.488 INFO:teuthology.orchestra.run.smithi032.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 162 kB/s | 26 kB 00:00 2024-04-30T08:19:47.530 INFO:teuthology.orchestra.run.smithi032.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 290 kB/s | 19 kB 00:00 2024-04-30T08:19:47.655 INFO:teuthology.orchestra.run.smithi032.stdout:(79/152): python3-babel-2.9.1-2.el9.noarch.rpm 10 MB/s | 6.0 MB 00:00 2024-04-30T08:19:47.680 INFO:teuthology.orchestra.run.smithi032.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 232 kB/s | 35 kB 00:00 2024-04-30T08:19:47.706 INFO:teuthology.orchestra.run.smithi032.stdout:(81/152): python3-mako-1.1.4-6.el9.noarch.rpm 795 kB/s | 172 kB 00:00 2024-04-30T08:19:47.789 INFO:teuthology.orchestra.run.smithi032.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.3 MB/s | 250 kB 00:00 2024-04-30T08:19:47.823 INFO:teuthology.orchestra.run.smithi032.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 1.9 MB/s | 222 kB 00:00 2024-04-30T08:19:47.882 INFO:teuthology.orchestra.run.smithi032.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 847 kB/s | 77 kB 00:00 2024-04-30T08:19:47.907 INFO:teuthology.orchestra.run.smithi032.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 511 kB/s | 42 kB 00:00 2024-04-30T08:19:48.016 INFO:teuthology.orchestra.run.smithi032.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 2.0 MB/s | 269 kB 00:00 2024-04-30T08:19:48.041 INFO:teuthology.orchestra.run.smithi032.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.2 MB/s | 159 kB 00:00 2024-04-30T08:19:48.125 INFO:teuthology.orchestra.run.smithi032.stdout:(88/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 5.1 MB 00:00 2024-04-30T08:19:48.150 INFO:teuthology.orchestra.run.smithi032.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 474 kB/s | 51 kB 00:00 2024-04-30T08:19:48.183 INFO:teuthology.orchestra.run.smithi032.stdout:(90/152): python3-pyasn1-modules-0.4.8-6.el9.no 1.6 MB/s | 279 kB 00:00 2024-04-30T08:19:48.209 INFO:teuthology.orchestra.run.smithi032.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 644 kB/s | 54 kB 00:00 2024-04-30T08:19:48.259 INFO:teuthology.orchestra.run.smithi032.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 556 kB/s | 42 kB 00:00 2024-04-30T08:19:48.284 INFO:teuthology.orchestra.run.smithi032.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 594 kB/s | 44 kB 00:00 2024-04-30T08:19:48.344 INFO:teuthology.orchestra.run.smithi032.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 620 kB/s | 36 kB 00:00 2024-04-30T08:19:48.376 INFO:teuthology.orchestra.run.smithi032.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 1.9 MB/s | 221 kB 00:00 2024-04-30T08:19:48.435 INFO:teuthology.orchestra.run.smithi032.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 532 kB/s | 47 kB 00:00 2024-04-30T08:19:48.494 INFO:teuthology.orchestra.run.smithi032.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 2.6 MB/s | 305 kB 00:00 2024-04-30T08:19:48.519 INFO:teuthology.orchestra.run.smithi032.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 701 kB/s | 58 kB 00:00 2024-04-30T08:19:48.620 INFO:teuthology.orchestra.run.smithi032.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 755 kB/s | 94 kB 00:00 2024-04-30T08:19:48.645 INFO:teuthology.orchestra.run.smithi032.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 509 kB/s | 64 kB 00:00 2024-04-30T08:19:48.712 INFO:teuthology.orchestra.run.smithi032.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 491 kB/s | 45 kB 00:00 2024-04-30T08:19:48.973 INFO:teuthology.orchestra.run.smithi032.stdout:(102/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 20 MB/s | 16 MB 00:00 2024-04-30T08:19:49.454 INFO:teuthology.orchestra.run.smithi032.stdout:(103/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-04-30T08:19:49.538 INFO:teuthology.orchestra.run.smithi032.stdout:(104/152): lua-devel-5.4.4-4.el9.x86_64.rpm 25 kB/s | 22 kB 00:00 2024-04-30T08:19:49.571 INFO:teuthology.orchestra.run.smithi032.stdout:(105/152): gperftools-libs-2.9.1-2.el9.x86_64.r 2.6 MB/s | 309 kB 00:00 2024-04-30T08:19:49.730 INFO:teuthology.orchestra.run.smithi032.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 102 kB/s | 19 kB 00:00 2024-04-30T08:19:49.822 INFO:teuthology.orchestra.run.smithi032.stdout:(107/152): libarrow-doc-9.0.0-11.el9.noarch.rpm 287 kB/s | 26 kB 00:00 2024-04-30T08:19:49.905 INFO:teuthology.orchestra.run.smithi032.stdout:(108/152): libarrow-9.0.0-11.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-04-30T08:19:49.931 INFO:teuthology.orchestra.run.smithi032.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 449 kB/s | 49 kB 00:00 2024-04-30T08:19:49.981 INFO:teuthology.orchestra.run.smithi032.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 899 kB/s | 67 kB 00:00 2024-04-30T08:19:50.090 INFO:teuthology.orchestra.run.smithi032.stdout:(111/152): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.6 MB/s | 839 kB 00:00 2024-04-30T08:19:50.123 INFO:teuthology.orchestra.run.smithi032.stdout:(112/152): luarocks-3.9.2-1.el9.noarch.rpm 787 kB/s | 151 kB 00:00 2024-04-30T08:19:50.199 INFO:teuthology.orchestra.run.smithi032.stdout:(113/152): python3-asyncssh-2.13.2-5.el9.noarch 5.0 MB/s | 548 kB 00:00 2024-04-30T08:19:50.224 INFO:teuthology.orchestra.run.smithi032.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 434 kB/s | 43 kB 00:00 2024-04-30T08:19:50.302 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : ceph-test-2:19.0.0-3348.g8feaf7d9.el9.x86_64 139/156 2024-04-30T08:19:50.308 INFO:teuthology.orchestra.run.smithi032.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 297 kB/s | 32 kB 00:00 2024-04-30T08:19:50.333 INFO:teuthology.orchestra.run.smithi032.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 139 kB/s | 15 kB 00:00 2024-04-30T08:19:50.345 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-fuse-2:19.0.0-3348.g8feaf7d9.el9.x86_64 140/156 2024-04-30T08:19:50.384 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : rbd-nbd-2:19.0.0-3348.g8feaf7d9.el9.x86_64 141/156 2024-04-30T08:19:50.392 INFO:teuthology.orchestra.run.smithi032.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.0 MB/s | 172 kB 00:00 2024-04-30T08:19:50.416 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-30T08:19:50.443 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-30T08:19:50.492 INFO:teuthology.orchestra.run.smithi032.stdout:(118/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.0 MB/s | 303 kB 00:00 2024-04-30T08:19:50.526 INFO:teuthology.orchestra.run.smithi032.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 1.8 MB/s | 358 kB 00:00 2024-04-30T08:19:50.584 INFO:teuthology.orchestra.run.smithi032.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 388 kB/s | 35 kB 00:00 2024-04-30T08:19:50.630 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : cephfs-shell-2:19.0.0-3348.g8feaf7d9.el9.x86_64 144/156 2024-04-30T08:19:50.643 INFO:teuthology.orchestra.run.smithi032.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 1.7 MB/s | 206 kB 00:00 2024-04-30T08:19:50.673 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-30T08:19:50.703 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libcephfs-devel-2:19.0.0-3348.g8feaf7d9.el9.x86_ 146/156 2024-04-30T08:19:50.752 INFO:teuthology.orchestra.run.smithi032.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 1.3 MB/s | 144 kB 00:00 2024-04-30T08:19:50.781 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3348.g8feaf7d9.el9.x86_64 107/156 2024-04-30T08:19:50.781 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /sys 2024-04-30T08:19:50.781 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /proc 2024-04-30T08:19:50.781 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /mnt 2024-04-30T08:19:50.781 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /var/tmp 2024-04-30T08:19:50.781 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /home 2024-04-30T08:19:50.782 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /root 2024-04-30T08:19:50.782 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /tmp 2024-04-30T08:19:50.782 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-30T08:19:50.869 INFO:teuthology.orchestra.run.smithi032.stdout:(123/152): python3-grpcio-1.46.7-10.el9.x86_64. 7.2 MB/s | 2.0 MB 00:00 2024-04-30T08:19:50.902 INFO:teuthology.orchestra.run.smithi032.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 71 kB/s | 11 kB 00:00 2024-04-30T08:19:50.928 INFO:teuthology.orchestra.run.smithi032.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 308 kB/s | 18 kB 00:00 2024-04-30T08:19:50.956 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-volume-2:19.0.0-3348.g8feaf7d9.el9.noarch 108/156 2024-04-30T08:19:50.969 INFO:teuthology.orchestra.run.smithi032.stdout:(126/152): protobuf-compiler-3.14.0-13.el9.x86_ 383 kB/s | 863 kB 00:02 2024-04-30T08:19:50.972 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-volume-2:19.0.0-3348.g8feaf7d9.el9.noarch 108/156 2024-04-30T08:19:50.972 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-30T08:19:50.972 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-30T08:19:50.972 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-30T08:19:50.995 INFO:teuthology.orchestra.run.smithi032.stdout:(127/152): python3-jaraco-collections-3.0.0-8.e 253 kB/s | 23 kB 00:00 2024-04-30T08:19:51.020 INFO:teuthology.orchestra.run.smithi032.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 212 kB/s | 19 kB 00:00 2024-04-30T08:19:51.112 INFO:teuthology.orchestra.run.smithi032.stdout:(129/152): python3-logutils-0.3.5-21.el9.noarch 512 kB/s | 46 kB 00:00 2024-04-30T08:19:51.162 INFO:teuthology.orchestra.run.smithi032.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 7.3 MB/s | 1.0 MB 00:00 2024-04-30T08:19:51.188 INFO:teuthology.orchestra.run.smithi032.stdout:(131/152): python3-jaraco-text-3.2.0-6.el9.noar 91 kB/s | 20 kB 00:00 2024-04-30T08:19:51.221 INFO:teuthology.orchestra.run.smithi032.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 729 kB/s | 79 kB 00:00 2024-04-30T08:19:51.255 INFO:teuthology.orchestra.run.smithi032.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 630 kB/s | 58 kB 00:00 2024-04-30T08:19:51.297 INFO:teuthology.orchestra.run.smithi032.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 220 kB/s | 16 kB 00:00 2024-04-30T08:19:51.326 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-30T08:19:51.339 INFO:teuthology.orchestra.run.smithi032.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.1 MB/s | 90 kB 00:00 2024-04-30T08:19:51.367 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-30T08:19:51.388 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-30T08:19:51.406 INFO:teuthology.orchestra.run.smithi032.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 260 kB/s | 28 kB 00:00 2024-04-30T08:19:51.456 INFO:teuthology.orchestra.run.smithi032.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 264 kB/s | 31 kB 00:00 2024-04-30T08:19:51.507 INFO:teuthology.orchestra.run.smithi032.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-04-30T08:19:51.540 INFO:teuthology.orchestra.run.smithi032.stdout:(139/152): python3-pecan-1.4.2-3.el9.noarch.rpm 774 kB/s | 272 kB 00:00 2024-04-30T08:19:51.566 INFO:teuthology.orchestra.run.smithi032.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 546 kB/s | 59 kB 00:00 2024-04-30T08:19:51.591 INFO:teuthology.orchestra.run.smithi032.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 431 kB/s | 36 kB 00:00 2024-04-30T08:19:51.616 INFO:teuthology.orchestra.run.smithi032.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 684 kB/s | 51 kB 00:00 2024-04-30T08:19:51.650 INFO:teuthology.orchestra.run.smithi032.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.7 MB/s | 230 kB 00:00 2024-04-30T08:19:51.675 INFO:teuthology.orchestra.run.smithi032.stdout:(144/152): python3-websocket-client-1.2.3-2.el9 1.1 MB/s | 90 kB 00:00 2024-04-30T08:19:51.709 INFO:teuthology.orchestra.run.smithi032.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 346 kB/s | 20 kB 00:00 2024-04-30T08:19:51.759 INFO:teuthology.orchestra.run.smithi032.stdout:(146/152): python3-werkzeug-2.0.3-3.el9.1.noarc 2.9 MB/s | 427 kB 00:00 2024-04-30T08:19:51.793 INFO:teuthology.orchestra.run.smithi032.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 1.6 MB/s | 192 kB 00:00 2024-04-30T08:19:51.868 INFO:teuthology.orchestra.run.smithi032.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-04-30T08:19:52.135 INFO:teuthology.orchestra.run.smithi032.stdout:(149/152): librbd1-19.0.0-3348.g8feaf7d9.el9.x8 9.1 MB/s | 3.1 MB 00:00 2024-04-30T08:19:52.210 INFO:teuthology.orchestra.run.smithi032.stdout:(150/152): librados2-19.0.0-3348.g8feaf7d9.el9. 7.4 MB/s | 3.3 MB 00:00 2024-04-30T08:19:52.227 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : cephfs-top-2:19.0.0-3348.g8feaf7d9.el9.noarch 148/156 2024-04-30T08:19:52.313 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-30T08:19:52.346 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-30T08:19:52.438 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-30T08:19:52.444 INFO:teuthology.orchestra.run.smithi032.stdout:(151/152): openssl-3.2.1-1.el9.x86_64.rpm 2.3 MB/s | 1.3 MB 00:00 2024-04-30T08:19:52.476 INFO:teuthology.orchestra.run.smithi071.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-30T08:19:52.476 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-30T08:19:52.544 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-osd-2:19.0.0-3348.g8feaf7d9.el9.x86_64 109/156 2024-04-30T08:19:52.568 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-30T08:19:52.570 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-osd-2:19.0.0-3348.g8feaf7d9.el9.x86_64 109/156 2024-04-30T08:19:52.570 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-30T08:19:52.570 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-30T08:19:52.570 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-30T08:19:52.570 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-30T08:19:52.571 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-30T08:19:52.601 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-30T08:19:52.601 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-30T08:19:52.691 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-30T08:19:52.727 INFO:teuthology.orchestra.run.smithi032.stdout:(152/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 4.0 MB/s | 2.4 MB 00:00 2024-04-30T08:19:52.734 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-30T08:19:52.735 INFO:teuthology.orchestra.run.smithi032.stdout:Total 14 MB/s | 214 MB 00:15 2024-04-30T08:19:52.783 INFO:teuthology.orchestra.run.smithi071.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-30T08:19:52.795 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mds-2:19.0.0-3348.g8feaf7d9.el9.x86_64 110/156 2024-04-30T08:19:52.818 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mds-2:19.0.0-3348.g8feaf7d9.el9.x86_64 110/156 2024-04-30T08:19:52.818 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-30T08:19:52.818 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-30T08:19:52.818 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-30T08:19:52.818 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-30T08:19:52.818 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-30T08:19:53.279 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mon-2:19.0.0-3348.g8feaf7d9.el9.x86_64 111/156 2024-04-30T08:19:53.301 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mon-2:19.0.0-3348.g8feaf7d9.el9.x86_64 111/156 2024-04-30T08:19:53.302 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-30T08:19:53.302 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-30T08:19:53.302 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-30T08:19:53.302 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-30T08:19:53.302 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-30T08:19:53.401 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-30T08:19:53.451 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-30T08:19:53.506 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-30T08:19:53.694 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-30T08:19:53.738 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-30T08:19:53.819 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-30T08:19:53.907 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-30T08:19:54.047 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-30T08:19:54.096 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-30T08:19:54.129 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-30T08:19:54.156 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-30T08:19:54.607 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: cephadm-2:19.0.0-3348.g8feaf7d9.el9.noarch 123/156 2024-04-30T08:19:54.638 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephadm-2:19.0.0-3348.g8feaf7d9.el9.noarch 123/156 2024-04-30T08:19:54.670 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3348.g8feaf7d9.e 124/156 2024-04-30T08:19:54.732 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3348.g8feaf7d9. 125/156 2024-04-30T08:19:54.769 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3348.g8feaf7d9.el9.noa 126/156 2024-04-30T08:19:55.758 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction check 2024-04-30T08:19:55.816 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3348.g8feaf7d9.el9.noa 126/156 2024-04-30T08:19:55.865 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3348.g8feaf7d9.el9.n 127/156 2024-04-30T08:19:56.007 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction check succeeded. 2024-04-30T08:19:56.007 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction test 2024-04-30T08:19:56.833 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3348.g8feaf7d9.el9.n 127/156 2024-04-30T08:19:56.865 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3348.g8fe 128/156 2024-04-30T08:19:56.970 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3348.g8fe 128/156 2024-04-30T08:19:57.141 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3348.g8feaf7d9.el 129/156 2024-04-30T08:19:57.176 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-2:19.0.0-3348.g8feaf7d9.el9.x86_64 130/156 2024-04-30T08:19:57.199 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3348.g8feaf7d9.el9.x86_64 130/156 2024-04-30T08:19:57.199 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-30T08:19:57.199 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-30T08:19:57.199 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-30T08:19:57.199 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-30T08:19:57.199 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-30T08:19:57.268 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-rook-2:19.0.0-3348.g8feaf7d9.el9.noarch 131/156 2024-04-30T08:19:57.279 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3348.g8feaf7d9.el9.noarch 131/156 2024-04-30T08:19:57.379 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-2:19.0.0-3348.g8feaf7d9.el9.x86_64 132/156 2024-04-30T08:19:57.433 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-fuse-2:19.0.0-3348.g8feaf7d9.el9.x86_64 133/156 2024-04-30T08:19:58.371 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-30T08:19:58.395 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-radosgw-2:19.0.0-3348.g8feaf7d9.el9.x86_64 135/156 2024-04-30T08:19:58.418 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3348.g8feaf7d9.el9.x86_64 135/156 2024-04-30T08:19:58.418 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-30T08:19:58.418 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-30T08:19:58.419 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-30T08:19:58.419 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-30T08:19:58.419 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-30T08:19:58.463 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3348.g8feaf 136/156 2024-04-30T08:19:58.485 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3348.g8feaf 136/156 2024-04-30T08:19:58.485 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-30T08:19:58.485 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-30T08:19:58.485 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-30T08:19:58.530 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephfs-mirror-2:19.0.0-3348.g8feaf7d9.el9.x86_64 137/156 2024-04-30T08:19:58.553 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3348.g8feaf7d9.el9.x86_64 137/156 2024-04-30T08:19:58.553 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-30T08:19:58.553 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-30T08:19:58.553 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-30T08:19:58.553 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-30T08:19:58.553 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-30T08:19:58.598 INFO:teuthology.orchestra.run.smithi032.stdout:Transaction test succeeded. 2024-04-30T08:19:58.598 INFO:teuthology.orchestra.run.smithi032.stdout:Running transaction 2024-04-30T08:19:58.840 INFO:teuthology.orchestra.run.smithi071.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-30T08:19:58.841 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-2:19.0.0-3348.g8feaf7d9.el9.x86_64 1/156 2024-04-30T08:19:58.841 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-base-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2/156 2024-04-30T08:19:58.841 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-common-2:19.0.0-3348.g8feaf7d9.el9.x86_64 3/156 2024-04-30T08:19:58.841 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-fuse-2:19.0.0-3348.g8feaf7d9.el9.x86_64 4/156 2024-04-30T08:19:58.841 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3348.g8feaf 5/156 2024-04-30T08:19:58.841 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mds-2:19.0.0-3348.g8feaf7d9.el9.x86_64 6/156 2024-04-30T08:19:58.841 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-2:19.0.0-3348.g8feaf7d9.el9.x86_64 7/156 2024-04-30T08:19:58.841 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mon-2:19.0.0-3348.g8feaf7d9.el9.x86_64 8/156 2024-04-30T08:19:58.841 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-osd-2:19.0.0-3348.g8feaf7d9.el9.x86_64 9/156 2024-04-30T08:19:58.841 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-radosgw-2:19.0.0-3348.g8feaf7d9.el9.x86_64 10/156 2024-04-30T08:19:58.841 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-selinux-2:19.0.0-3348.g8feaf7d9.el9.x86_64 11/156 2024-04-30T08:19:58.841 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-test-2:19.0.0-3348.g8feaf7d9.el9.x86_64 12/156 2024-04-30T08:19:58.842 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : cephfs-mirror-2:19.0.0-3348.g8feaf7d9.el9.x86_64 13/156 2024-04-30T08:19:58.842 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : cephfs-shell-2:19.0.0-3348.g8feaf7d9.el9.x86_64 14/156 2024-04-30T08:19:58.842 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephfs-devel-2:19.0.0-3348.g8feaf7d9.el9.x86_ 15/156 2024-04-30T08:19:58.842 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephfs2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 16/156 2024-04-30T08:19:58.842 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libcephsqlite-2:19.0.0-3348.g8feaf7d9.el9.x86_64 17/156 2024-04-30T08:19:58.842 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados-devel-2:19.0.0-3348.g8feaf7d9.el9.x86_6 18/156 2024-04-30T08:19:58.842 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libradosstriper1-2:19.0.0-3348.g8feaf7d9.el9.x86 19/156 2024-04-30T08:19:58.842 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librgw2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 20/156 2024-04-30T08:19:58.842 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3348.g8feaf7d9.el 21/156 2024-04-30T08:19:58.842 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ceph-common-2:19.0.0-3348.g8feaf7d9.el9. 22/156 2024-04-30T08:19:58.842 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cephfs-2:19.0.0-3348.g8feaf7d9.el9.x86_6 23/156 2024-04-30T08:19:58.842 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rados-2:19.0.0-3348.g8feaf7d9.el9.x86_64 24/156 2024-04-30T08:19:58.842 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rbd-2:19.0.0-3348.g8feaf7d9.el9.x86_64 25/156 2024-04-30T08:19:58.843 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rgw-2:19.0.0-3348.g8feaf7d9.el9.x86_64 26/156 2024-04-30T08:19:58.843 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-fuse-2:19.0.0-3348.g8feaf7d9.el9.x86_64 27/156 2024-04-30T08:19:58.843 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-mirror-2:19.0.0-3348.g8feaf7d9.el9.x86_64 28/156 2024-04-30T08:19:58.843 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : rbd-nbd-2:19.0.0-3348.g8feaf7d9.el9.x86_64 29/156 2024-04-30T08:19:58.843 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3348.g8feaf7d9. 30/156 2024-04-30T08:19:58.843 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3348.g8feaf7d9.el9.noa 31/156 2024-04-30T08:19:58.843 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3348.g8feaf7d9.el9.n 32/156 2024-04-30T08:19:58.843 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3348.g8fe 33/156 2024-04-30T08:19:58.843 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3348.g8feaf7d9.el 34/156 2024-04-30T08:19:58.843 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3348.g8feaf7d9.el9.noarch 35/156 2024-04-30T08:19:58.843 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3348.g8feaf7d9.e 36/156 2024-04-30T08:19:58.843 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : ceph-volume-2:19.0.0-3348.g8feaf7d9.el9.noarch 37/156 2024-04-30T08:19:58.843 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : cephadm-2:19.0.0-3348.g8feaf7d9.el9.noarch 38/156 2024-04-30T08:19:58.844 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : cephfs-top-2:19.0.0-3348.g8feaf7d9.el9.noarch 39/156 2024-04-30T08:19:58.844 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-30T08:19:58.844 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-30T08:19:58.844 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-30T08:19:58.844 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-30T08:19:58.844 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-30T08:19:58.844 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-30T08:19:58.844 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-30T08:19:58.844 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-30T08:19:58.844 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-30T08:19:58.844 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-30T08:19:58.844 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-30T08:19:58.844 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-30T08:19:58.845 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-30T08:19:58.845 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-30T08:19:58.845 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-30T08:19:58.845 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-30T08:19:58.845 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-30T08:19:58.845 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-30T08:19:58.845 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-30T08:19:58.845 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-30T08:19:58.845 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-30T08:19:58.845 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-30T08:19:58.845 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-30T08:19:58.845 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-30T08:19:58.845 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-30T08:19:58.845 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-30T08:19:58.846 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-30T08:19:58.846 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-30T08:19:58.846 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-30T08:19:58.846 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-30T08:19:58.846 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-30T08:19:58.846 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-30T08:19:58.846 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-30T08:19:58.846 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-30T08:19:58.846 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-30T08:19:58.846 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-30T08:19:58.846 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-30T08:19:58.846 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-30T08:19:58.846 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-30T08:19:58.847 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-30T08:19:58.847 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-30T08:19:58.847 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-30T08:19:58.847 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-30T08:19:58.847 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-30T08:19:58.847 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-30T08:19:58.847 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-30T08:19:58.847 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-30T08:19:58.847 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-30T08:19:58.847 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-30T08:19:58.847 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-30T08:19:58.847 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-30T08:19:58.848 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-30T08:19:58.848 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-30T08:19:58.848 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-30T08:19:58.848 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-30T08:19:58.848 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-30T08:19:58.848 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-30T08:19:58.848 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-30T08:19:58.848 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-30T08:19:58.848 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-30T08:19:58.848 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-30T08:19:58.848 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-30T08:19:58.848 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-30T08:19:58.848 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-30T08:19:58.849 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-30T08:19:58.849 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-30T08:19:58.849 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-30T08:19:58.849 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-30T08:19:58.849 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-30T08:19:58.849 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-30T08:19:58.849 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-30T08:19:58.849 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-30T08:19:58.849 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-30T08:19:58.849 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-30T08:19:58.849 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-30T08:19:58.849 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-30T08:19:58.849 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-30T08:19:58.849 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-30T08:19:58.850 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-30T08:19:58.850 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-30T08:19:58.850 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-30T08:19:58.850 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-30T08:19:58.850 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-30T08:19:58.850 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-30T08:19:58.850 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-30T08:19:58.850 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-30T08:19:58.850 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-30T08:19:58.850 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-30T08:19:58.850 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-30T08:19:58.850 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-30T08:19:58.850 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-30T08:19:58.850 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-30T08:19:58.851 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-30T08:19:58.851 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-30T08:19:58.851 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-30T08:19:58.851 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-30T08:19:58.851 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-30T08:19:58.851 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-30T08:19:58.851 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-30T08:19:58.851 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-30T08:19:58.851 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-30T08:19:58.851 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-30T08:19:58.851 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-30T08:19:58.851 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-30T08:19:58.851 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-30T08:19:58.851 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-30T08:19:58.851 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-30T08:19:58.852 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-30T08:19:58.852 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-30T08:19:58.852 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 149/156 2024-04-30T08:19:58.852 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-30T08:19:58.852 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librbd1-2:19.0.0-3348.g8feaf7d9.el9.x86_64 151/156 2024-04-30T08:19:58.852 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-30T08:19:58.852 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-30T08:19:58.852 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-30T08:19:58.852 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-30T08:19:58.872 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-mirror-2:19.0.0-3348.g8feaf7d9.el9.x86_64 138/156 2024-04-30T08:19:58.895 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3348.g8feaf7d9.el9.x86_64 138/156 2024-04-30T08:19:58.895 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-30T08:19:58.895 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-30T08:19:58.895 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-30T08:19:58.895 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-30T08:19:58.895 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-30T08:20:00.948 INFO:teuthology.orchestra.run.smithi071.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-30T08:20:00.948 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-30T08:20:00.948 INFO:teuthology.orchestra.run.smithi071.stdout:Upgraded: 2024-04-30T08:20:00.949 INFO:teuthology.orchestra.run.smithi071.stdout: librados2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:00.949 INFO:teuthology.orchestra.run.smithi071.stdout: librbd1-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:00.949 INFO:teuthology.orchestra.run.smithi071.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-30T08:20:00.949 INFO:teuthology.orchestra.run.smithi071.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-30T08:20:00.949 INFO:teuthology.orchestra.run.smithi071.stdout:Installed: 2024-04-30T08:20:00.949 INFO:teuthology.orchestra.run.smithi071.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-30T08:20:00.949 INFO:teuthology.orchestra.run.smithi071.stdout: automake-1.16.2-8.el9.noarch 2024-04-30T08:20:00.949 INFO:teuthology.orchestra.run.smithi071.stdout: bison-3.7.4-5.el9.x86_64 2024-04-30T08:20:00.949 INFO:teuthology.orchestra.run.smithi071.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-30T08:20:00.949 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:00.949 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-base-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:00.949 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-common-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:00.950 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-fuse-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:00.950 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-grafana-dashboards-2:19.0.0-3348.g8feaf7d9.el9.noarch 2024-04-30T08:20:00.950 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-immutable-object-cache-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:00.950 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mds-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:00.950 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:00.950 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-cephadm-2:19.0.0-3348.g8feaf7d9.el9.noarch 2024-04-30T08:20:00.950 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-dashboard-2:19.0.0-3348.g8feaf7d9.el9.noarch 2024-04-30T08:20:00.950 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3348.g8feaf7d9.el9.noarch 2024-04-30T08:20:00.950 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-modules-core-2:19.0.0-3348.g8feaf7d9.el9.noarch 2024-04-30T08:20:00.950 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mgr-rook-2:19.0.0-3348.g8feaf7d9.el9.noarch 2024-04-30T08:20:00.950 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-mon-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:00.950 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-osd-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:00.951 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-prometheus-alerts-2:19.0.0-3348.g8feaf7d9.el9.noarch 2024-04-30T08:20:00.951 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-radosgw-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:00.951 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-selinux-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:00.951 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-test-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:00.951 INFO:teuthology.orchestra.run.smithi071.stdout: ceph-volume-2:19.0.0-3348.g8feaf7d9.el9.noarch 2024-04-30T08:20:00.951 INFO:teuthology.orchestra.run.smithi071.stdout: cephadm-2:19.0.0-3348.g8feaf7d9.el9.noarch 2024-04-30T08:20:00.951 INFO:teuthology.orchestra.run.smithi071.stdout: cephfs-mirror-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:00.951 INFO:teuthology.orchestra.run.smithi071.stdout: cephfs-shell-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:00.951 INFO:teuthology.orchestra.run.smithi071.stdout: cephfs-top-2:19.0.0-3348.g8feaf7d9.el9.noarch 2024-04-30T08:20:00.951 INFO:teuthology.orchestra.run.smithi071.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-30T08:20:00.951 INFO:teuthology.orchestra.run.smithi071.stdout: flex-2.6.4-9.el9.x86_64 2024-04-30T08:20:00.951 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-30T08:20:00.952 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-30T08:20:00.952 INFO:teuthology.orchestra.run.smithi071.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-30T08:20:00.952 INFO:teuthology.orchestra.run.smithi071.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-30T08:20:00.952 INFO:teuthology.orchestra.run.smithi071.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-30T08:20:00.952 INFO:teuthology.orchestra.run.smithi071.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-30T08:20:00.952 INFO:teuthology.orchestra.run.smithi071.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-30T08:20:00.952 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-30T08:20:00.952 INFO:teuthology.orchestra.run.smithi071.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-30T08:20:00.952 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs-devel-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:00.952 INFO:teuthology.orchestra.run.smithi071.stdout: libcephfs2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:00.952 INFO:teuthology.orchestra.run.smithi071.stdout: libcephsqlite-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:00.952 INFO:teuthology.orchestra.run.smithi071.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-30T08:20:00.952 INFO:teuthology.orchestra.run.smithi071.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-30T08:20:00.953 INFO:teuthology.orchestra.run.smithi071.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-30T08:20:00.953 INFO:teuthology.orchestra.run.smithi071.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-30T08:20:00.953 INFO:teuthology.orchestra.run.smithi071.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-30T08:20:00.953 INFO:teuthology.orchestra.run.smithi071.stdout: librados-devel-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:00.953 INFO:teuthology.orchestra.run.smithi071.stdout: libradosstriper1-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:00.953 INFO:teuthology.orchestra.run.smithi071.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-30T08:20:00.953 INFO:teuthology.orchestra.run.smithi071.stdout: librgw2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:00.953 INFO:teuthology.orchestra.run.smithi071.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-30T08:20:00.953 INFO:teuthology.orchestra.run.smithi071.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-30T08:20:00.953 INFO:teuthology.orchestra.run.smithi071.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-30T08:20:00.953 INFO:teuthology.orchestra.run.smithi071.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-30T08:20:00.953 INFO:teuthology.orchestra.run.smithi071.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-30T08:20:00.953 INFO:teuthology.orchestra.run.smithi071.stdout: lua-5.4.4-4.el9.x86_64 2024-04-30T08:20:00.953 INFO:teuthology.orchestra.run.smithi071.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-30T08:20:00.954 INFO:teuthology.orchestra.run.smithi071.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-30T08:20:00.954 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-30T08:20:00.954 INFO:teuthology.orchestra.run.smithi071.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-30T08:20:00.954 INFO:teuthology.orchestra.run.smithi071.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-30T08:20:00.954 INFO:teuthology.orchestra.run.smithi071.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-30T08:20:00.954 INFO:teuthology.orchestra.run.smithi071.stdout: postgresql-13.14-1.el9.x86_64 2024-04-30T08:20:00.954 INFO:teuthology.orchestra.run.smithi071.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-30T08:20:00.954 INFO:teuthology.orchestra.run.smithi071.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-30T08:20:00.954 INFO:teuthology.orchestra.run.smithi071.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-30T08:20:00.955 INFO:teuthology.orchestra.run.smithi071.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-30T08:20:00.955 INFO:teuthology.orchestra.run.smithi071.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-30T08:20:00.955 INFO:teuthology.orchestra.run.smithi071.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-30T08:20:00.955 INFO:teuthology.orchestra.run.smithi071.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-30T08:20:00.955 INFO:teuthology.orchestra.run.smithi071.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-30T08:20:00.955 INFO:teuthology.orchestra.run.smithi071.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-30T08:20:00.955 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-30T08:20:00.955 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-argparse-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:00.955 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ceph-common-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:00.955 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cephfs-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:00.955 INFO:teuthology.orchestra.run.smithi071.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-30T08:20:00.955 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-30T08:20:00.955 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-30T08:20:00.955 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-30T08:20:00.956 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-30T08:20:00.956 INFO:teuthology.orchestra.run.smithi071.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-30T08:20:00.956 INFO:teuthology.orchestra.run.smithi071.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-30T08:20:00.956 INFO:teuthology.orchestra.run.smithi071.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-30T08:20:00.956 INFO:teuthology.orchestra.run.smithi071.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-30T08:20:00.956 INFO:teuthology.orchestra.run.smithi071.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-30T08:20:00.956 INFO:teuthology.orchestra.run.smithi071.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-30T08:20:00.956 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-30T08:20:00.956 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-30T08:20:00.956 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-30T08:20:00.956 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-30T08:20:00.956 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-30T08:20:00.957 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-30T08:20:00.957 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-30T08:20:00.957 INFO:teuthology.orchestra.run.smithi071.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-30T08:20:00.957 INFO:teuthology.orchestra.run.smithi071.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-30T08:20:00.957 INFO:teuthology.orchestra.run.smithi071.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-30T08:20:00.957 INFO:teuthology.orchestra.run.smithi071.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-30T08:20:00.957 INFO:teuthology.orchestra.run.smithi071.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-30T08:20:00.957 INFO:teuthology.orchestra.run.smithi071.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-30T08:20:00.957 INFO:teuthology.orchestra.run.smithi071.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-30T08:20:00.957 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-30T08:20:00.957 INFO:teuthology.orchestra.run.smithi071.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-30T08:20:00.958 INFO:teuthology.orchestra.run.smithi071.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-30T08:20:00.958 INFO:teuthology.orchestra.run.smithi071.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-30T08:20:00.958 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-30T08:20:00.958 INFO:teuthology.orchestra.run.smithi071.stdout: python3-ply-3.11-14.el9.noarch 2024-04-30T08:20:00.958 INFO:teuthology.orchestra.run.smithi071.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-30T08:20:00.958 INFO:teuthology.orchestra.run.smithi071.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-30T08:20:00.958 INFO:teuthology.orchestra.run.smithi071.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-30T08:20:00.958 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-30T08:20:00.958 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-30T08:20:00.958 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-30T08:20:00.958 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-30T08:20:00.958 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-30T08:20:00.958 INFO:teuthology.orchestra.run.smithi071.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-30T08:20:00.959 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rados-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:00.959 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rbd-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:00.959 INFO:teuthology.orchestra.run.smithi071.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-30T08:20:00.959 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-30T08:20:00.959 INFO:teuthology.orchestra.run.smithi071.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-30T08:20:00.959 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rgw-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:00.959 INFO:teuthology.orchestra.run.smithi071.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-30T08:20:00.959 INFO:teuthology.orchestra.run.smithi071.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-30T08:20:00.959 INFO:teuthology.orchestra.run.smithi071.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-30T08:20:00.959 INFO:teuthology.orchestra.run.smithi071.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-30T08:20:00.959 INFO:teuthology.orchestra.run.smithi071.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-30T08:20:00.959 INFO:teuthology.orchestra.run.smithi071.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-30T08:20:00.960 INFO:teuthology.orchestra.run.smithi071.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-30T08:20:00.960 INFO:teuthology.orchestra.run.smithi071.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-30T08:20:00.960 INFO:teuthology.orchestra.run.smithi071.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-30T08:20:00.960 INFO:teuthology.orchestra.run.smithi071.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-30T08:20:00.960 INFO:teuthology.orchestra.run.smithi071.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-30T08:20:00.960 INFO:teuthology.orchestra.run.smithi071.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-30T08:20:00.960 INFO:teuthology.orchestra.run.smithi071.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-30T08:20:00.960 INFO:teuthology.orchestra.run.smithi071.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-30T08:20:00.960 INFO:teuthology.orchestra.run.smithi071.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-30T08:20:00.960 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-fuse-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:00.960 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-mirror-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:00.960 INFO:teuthology.orchestra.run.smithi071.stdout: rbd-nbd-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:00.961 INFO:teuthology.orchestra.run.smithi071.stdout: re2-1:20211101-3.el9.x86_64 2024-04-30T08:20:00.961 INFO:teuthology.orchestra.run.smithi071.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-30T08:20:00.961 INFO:teuthology.orchestra.run.smithi071.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-30T08:20:00.961 INFO:teuthology.orchestra.run.smithi071.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-30T08:20:00.961 INFO:teuthology.orchestra.run.smithi071.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-30T08:20:00.961 INFO:teuthology.orchestra.run.smithi071.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-30T08:20:00.961 INFO:teuthology.orchestra.run.smithi071.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-30T08:20:00.962 INFO:teuthology.orchestra.run.smithi071.stdout: 2024-04-30T08:20:00.962 INFO:teuthology.orchestra.run.smithi071.stdout:Complete! 2024-04-30T08:20:01.386 DEBUG:teuthology.parallel:result is None 2024-04-30T08:20:03.202 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-test-2:19.0.0-3348.g8feaf7d9.el9.x86_64 139/156 2024-04-30T08:20:03.243 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-fuse-2:19.0.0-3348.g8feaf7d9.el9.x86_64 140/156 2024-04-30T08:20:03.284 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-nbd-2:19.0.0-3348.g8feaf7d9.el9.x86_64 141/156 2024-04-30T08:20:03.316 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-30T08:20:03.343 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-30T08:20:03.534 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephfs-shell-2:19.0.0-3348.g8feaf7d9.el9.x86_64 144/156 2024-04-30T08:20:03.581 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-30T08:20:03.610 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephfs-devel-2:19.0.0-3348.g8feaf7d9.el9.x86_ 146/156 2024-04-30T08:20:04.173 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-30T08:20:04.217 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-30T08:20:04.236 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-30T08:20:05.110 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephfs-top-2:19.0.0-3348.g8feaf7d9.el9.noarch 148/156 2024-04-30T08:20:05.196 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-30T08:20:05.229 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-30T08:20:05.274 INFO:teuthology.orchestra.run.smithi032.stdout: Preparing : 1/1 2024-04-30T08:20:05.320 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-30T08:20:05.359 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-30T08:20:05.359 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-30T08:20:05.361 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-30T08:20:05.434 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-30T08:20:05.435 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-30T08:20:05.468 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-30T08:20:05.468 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-30T08:20:05.541 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-30T08:20:05.558 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-30T08:20:05.596 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-30T08:20:05.624 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-30T08:20:05.628 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-30T08:20:05.764 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-30T08:20:06.232 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-30T08:20:06.343 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : librados2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 8/156 2024-04-30T08:20:06.571 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 8/156 2024-04-30T08:20:06.611 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephfs2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 9/156 2024-04-30T08:20:06.755 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephfs2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 9/156 2024-04-30T08:20:06.819 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rados-2:19.0.0-3348.g8feaf7d9.el9.x86_64 10/156 2024-04-30T08:20:07.159 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-30T08:20:07.228 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-30T08:20:07.257 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephsqlite-2:19.0.0-3348.g8feaf7d9.el9.x86_64 13/156 2024-04-30T08:20:07.425 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3348.g8feaf7d9.el9.x86_64 13/156 2024-04-30T08:20:07.458 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libradosstriper1-2:19.0.0-3348.g8feaf7d9.el9.x86 14/156 2024-04-30T08:20:07.550 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3348.g8feaf7d9.el9.x86 14/156 2024-04-30T08:20:07.601 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-30T08:20:07.688 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-30T08:20:07.981 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-30T08:20:08.044 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-30T08:20:08.246 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-30T08:20:08.375 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 18/156 2024-04-30T08:20:08.420 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-30T08:20:08.469 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-30T08:20:08.573 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-30T08:20:08.624 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-30T08:20:08.688 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-30T08:20:08.761 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-30T08:20:08.803 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-30T08:20:08.942 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-30T08:20:09.013 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-30T08:20:09.045 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-30T08:20:09.102 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-30T08:20:09.210 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-30T08:20:09.248 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ceph-common-2:19.0.0-3348.g8feaf7d9.el9. 31/156 2024-04-30T08:20:09.333 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ceph-argparse-2:19.0.0-3348.g8feaf7d9.el 32/156 2024-04-30T08:20:09.373 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cephfs-2:19.0.0-3348.g8feaf7d9.el9.x86_6 33/156 2024-04-30T08:20:09.496 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-30T08:20:09.552 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-30T08:20:09.664 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-30T08:20:09.852 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-30T08:20:09.939 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-30T08:20:10.666 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-30T08:20:10.772 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-30T08:20:10.813 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-30T08:20:10.863 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-30T08:20:10.909 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librados-devel-2:19.0.0-3348.g8feaf7d9.el9.x86_6 43/156 2024-04-30T08:20:10.994 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-30T08:20:11.319 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-30T08:20:11.382 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-30T08:20:11.460 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-30T08:20:11.555 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-30T08:20:11.588 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-30T08:20:11.588 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-2:19.0.0-3348.g8feaf7d9.el9.x86_64 1/156 2024-04-30T08:20:11.588 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-base-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2/156 2024-04-30T08:20:11.589 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-common-2:19.0.0-3348.g8feaf7d9.el9.x86_64 3/156 2024-04-30T08:20:11.589 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-fuse-2:19.0.0-3348.g8feaf7d9.el9.x86_64 4/156 2024-04-30T08:20:11.589 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3348.g8feaf 5/156 2024-04-30T08:20:11.589 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mds-2:19.0.0-3348.g8feaf7d9.el9.x86_64 6/156 2024-04-30T08:20:11.589 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-2:19.0.0-3348.g8feaf7d9.el9.x86_64 7/156 2024-04-30T08:20:11.589 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mon-2:19.0.0-3348.g8feaf7d9.el9.x86_64 8/156 2024-04-30T08:20:11.589 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-osd-2:19.0.0-3348.g8feaf7d9.el9.x86_64 9/156 2024-04-30T08:20:11.589 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-radosgw-2:19.0.0-3348.g8feaf7d9.el9.x86_64 10/156 2024-04-30T08:20:11.589 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-selinux-2:19.0.0-3348.g8feaf7d9.el9.x86_64 11/156 2024-04-30T08:20:11.589 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-test-2:19.0.0-3348.g8feaf7d9.el9.x86_64 12/156 2024-04-30T08:20:11.589 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephfs-mirror-2:19.0.0-3348.g8feaf7d9.el9.x86_64 13/156 2024-04-30T08:20:11.590 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephfs-shell-2:19.0.0-3348.g8feaf7d9.el9.x86_64 14/156 2024-04-30T08:20:11.590 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephfs-devel-2:19.0.0-3348.g8feaf7d9.el9.x86_ 15/156 2024-04-30T08:20:11.590 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephfs2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 16/156 2024-04-30T08:20:11.590 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephsqlite-2:19.0.0-3348.g8feaf7d9.el9.x86_64 17/156 2024-04-30T08:20:11.590 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados-devel-2:19.0.0-3348.g8feaf7d9.el9.x86_6 18/156 2024-04-30T08:20:11.590 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libradosstriper1-2:19.0.0-3348.g8feaf7d9.el9.x86 19/156 2024-04-30T08:20:11.590 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librgw2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 20/156 2024-04-30T08:20:11.590 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3348.g8feaf7d9.el 21/156 2024-04-30T08:20:11.590 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ceph-common-2:19.0.0-3348.g8feaf7d9.el9. 22/156 2024-04-30T08:20:11.590 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cephfs-2:19.0.0-3348.g8feaf7d9.el9.x86_6 23/156 2024-04-30T08:20:11.590 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rados-2:19.0.0-3348.g8feaf7d9.el9.x86_64 24/156 2024-04-30T08:20:11.590 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rbd-2:19.0.0-3348.g8feaf7d9.el9.x86_64 25/156 2024-04-30T08:20:11.590 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rgw-2:19.0.0-3348.g8feaf7d9.el9.x86_64 26/156 2024-04-30T08:20:11.591 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-fuse-2:19.0.0-3348.g8feaf7d9.el9.x86_64 27/156 2024-04-30T08:20:11.591 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-mirror-2:19.0.0-3348.g8feaf7d9.el9.x86_64 28/156 2024-04-30T08:20:11.591 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-nbd-2:19.0.0-3348.g8feaf7d9.el9.x86_64 29/156 2024-04-30T08:20:11.591 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3348.g8feaf7d9. 30/156 2024-04-30T08:20:11.591 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3348.g8feaf7d9.el9.noa 31/156 2024-04-30T08:20:11.591 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3348.g8feaf7d9.el9.n 32/156 2024-04-30T08:20:11.591 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3348.g8fe 33/156 2024-04-30T08:20:11.591 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3348.g8feaf7d9.el 34/156 2024-04-30T08:20:11.591 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3348.g8feaf7d9.el9.noarch 35/156 2024-04-30T08:20:11.591 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3348.g8feaf7d9.e 36/156 2024-04-30T08:20:11.591 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-volume-2:19.0.0-3348.g8feaf7d9.el9.noarch 37/156 2024-04-30T08:20:11.591 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephadm-2:19.0.0-3348.g8feaf7d9.el9.noarch 38/156 2024-04-30T08:20:11.591 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephfs-top-2:19.0.0-3348.g8feaf7d9.el9.noarch 39/156 2024-04-30T08:20:11.592 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-30T08:20:11.592 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-30T08:20:11.592 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-30T08:20:11.592 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-30T08:20:11.592 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-30T08:20:11.592 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-30T08:20:11.592 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-30T08:20:11.592 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-30T08:20:11.592 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-30T08:20:11.592 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-30T08:20:11.592 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-30T08:20:11.593 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-30T08:20:11.593 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-30T08:20:11.593 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-30T08:20:11.593 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-30T08:20:11.593 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-30T08:20:11.593 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-30T08:20:11.593 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-30T08:20:11.593 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-30T08:20:11.593 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-30T08:20:11.593 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-30T08:20:11.593 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-30T08:20:11.593 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-30T08:20:11.593 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-30T08:20:11.594 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-30T08:20:11.594 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-30T08:20:11.594 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-30T08:20:11.594 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-30T08:20:11.594 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-30T08:20:11.594 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-30T08:20:11.594 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-30T08:20:11.594 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-30T08:20:11.594 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-30T08:20:11.594 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-30T08:20:11.594 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-30T08:20:11.594 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-30T08:20:11.594 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-30T08:20:11.595 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-30T08:20:11.595 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-30T08:20:11.595 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-30T08:20:11.595 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-30T08:20:11.595 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-30T08:20:11.595 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-30T08:20:11.595 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-30T08:20:11.595 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-30T08:20:11.595 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-30T08:20:11.595 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-30T08:20:11.596 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-30T08:20:11.596 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-30T08:20:11.596 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-30T08:20:11.596 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-30T08:20:11.596 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-30T08:20:11.596 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-30T08:20:11.596 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-30T08:20:11.596 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-30T08:20:11.596 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-30T08:20:11.596 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-30T08:20:11.596 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-30T08:20:11.596 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-30T08:20:11.596 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-30T08:20:11.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-30T08:20:11.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-30T08:20:11.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-30T08:20:11.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-30T08:20:11.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-30T08:20:11.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-30T08:20:11.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-30T08:20:11.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-30T08:20:11.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-30T08:20:11.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-30T08:20:11.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-30T08:20:11.597 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-30T08:20:11.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-30T08:20:11.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-30T08:20:11.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-30T08:20:11.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-30T08:20:11.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-30T08:20:11.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-30T08:20:11.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-30T08:20:11.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-30T08:20:11.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-30T08:20:11.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-30T08:20:11.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-30T08:20:11.598 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-30T08:20:11.599 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-30T08:20:11.599 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-30T08:20:11.599 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-30T08:20:11.599 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-30T08:20:11.599 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-30T08:20:11.599 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-30T08:20:11.599 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-30T08:20:11.599 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-30T08:20:11.599 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-30T08:20:11.599 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-30T08:20:11.599 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-30T08:20:11.599 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-30T08:20:11.599 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-30T08:20:11.599 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-30T08:20:11.600 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-30T08:20:11.600 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-30T08:20:11.600 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-30T08:20:11.600 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-30T08:20:11.600 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-30T08:20:11.600 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-30T08:20:11.600 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-30T08:20:11.600 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-30T08:20:11.600 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-30T08:20:11.600 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-30T08:20:11.600 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-30T08:20:11.600 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 149/156 2024-04-30T08:20:11.600 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-30T08:20:11.600 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librbd1-2:19.0.0-3348.g8feaf7d9.el9.x86_64 151/156 2024-04-30T08:20:11.601 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-30T08:20:11.601 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-30T08:20:11.601 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-30T08:20:11.601 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-30T08:20:11.689 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-30T08:20:11.733 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-30T08:20:11.815 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-30T08:20:11.866 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-30T08:20:11.931 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-30T08:20:11.995 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-30T08:20:12.207 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-30T08:20:12.286 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-30T08:20:12.325 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-30T08:20:12.387 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-30T08:20:12.527 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-30T08:20:12.565 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-30T08:20:12.606 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-30T08:20:12.638 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-30T08:20:12.671 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 63/156 2024-04-30T08:20:12.863 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-30T08:20:13.057 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-30T08:20:13.103 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-30T08:20:13.153 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-30T08:20:13.199 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-30T08:20:13.265 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-30T08:20:13.317 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-30T08:20:13.366 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-30T08:20:13.437 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-30T08:20:13.487 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-30T08:20:13.566 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-30T08:20:13.618 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-30T08:20:13.618 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-30T08:20:13.618 INFO:teuthology.orchestra.run.smithi189.stdout:Upgraded: 2024-04-30T08:20:13.618 INFO:teuthology.orchestra.run.smithi189.stdout: librados2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:13.618 INFO:teuthology.orchestra.run.smithi189.stdout: librbd1-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:13.618 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-30T08:20:13.618 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-30T08:20:13.619 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-04-30T08:20:13.619 INFO:teuthology.orchestra.run.smithi189.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-30T08:20:13.619 INFO:teuthology.orchestra.run.smithi189.stdout: automake-1.16.2-8.el9.noarch 2024-04-30T08:20:13.619 INFO:teuthology.orchestra.run.smithi189.stdout: bison-3.7.4-5.el9.x86_64 2024-04-30T08:20:13.619 INFO:teuthology.orchestra.run.smithi189.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-30T08:20:13.619 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:13.619 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-base-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:13.619 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-common-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:13.619 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-fuse-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:13.619 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-grafana-dashboards-2:19.0.0-3348.g8feaf7d9.el9.noarch 2024-04-30T08:20:13.619 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-immutable-object-cache-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:13.619 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mds-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:13.619 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:13.620 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-cephadm-2:19.0.0-3348.g8feaf7d9.el9.noarch 2024-04-30T08:20:13.620 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-dashboard-2:19.0.0-3348.g8feaf7d9.el9.noarch 2024-04-30T08:20:13.620 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3348.g8feaf7d9.el9.noarch 2024-04-30T08:20:13.620 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-modules-core-2:19.0.0-3348.g8feaf7d9.el9.noarch 2024-04-30T08:20:13.620 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-rook-2:19.0.0-3348.g8feaf7d9.el9.noarch 2024-04-30T08:20:13.620 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mon-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:13.620 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-osd-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:13.620 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-prometheus-alerts-2:19.0.0-3348.g8feaf7d9.el9.noarch 2024-04-30T08:20:13.620 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-radosgw-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:13.620 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-selinux-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:13.620 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-test-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:13.620 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-volume-2:19.0.0-3348.g8feaf7d9.el9.noarch 2024-04-30T08:20:13.620 INFO:teuthology.orchestra.run.smithi189.stdout: cephadm-2:19.0.0-3348.g8feaf7d9.el9.noarch 2024-04-30T08:20:13.621 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-mirror-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:13.621 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-shell-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:13.621 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-top-2:19.0.0-3348.g8feaf7d9.el9.noarch 2024-04-30T08:20:13.621 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-30T08:20:13.621 INFO:teuthology.orchestra.run.smithi189.stdout: flex-2.6.4-9.el9.x86_64 2024-04-30T08:20:13.621 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-30T08:20:13.621 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-30T08:20:13.621 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-30T08:20:13.621 INFO:teuthology.orchestra.run.smithi189.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-30T08:20:13.621 INFO:teuthology.orchestra.run.smithi189.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-30T08:20:13.621 INFO:teuthology.orchestra.run.smithi189.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-30T08:20:13.621 INFO:teuthology.orchestra.run.smithi189.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-30T08:20:13.621 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-30T08:20:13.622 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-30T08:20:13.622 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs-devel-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:13.622 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:13.622 INFO:teuthology.orchestra.run.smithi189.stdout: libcephsqlite-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:13.622 INFO:teuthology.orchestra.run.smithi189.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-30T08:20:13.622 INFO:teuthology.orchestra.run.smithi189.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-30T08:20:13.622 INFO:teuthology.orchestra.run.smithi189.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-30T08:20:13.622 INFO:teuthology.orchestra.run.smithi189.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-30T08:20:13.622 INFO:teuthology.orchestra.run.smithi189.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-30T08:20:13.622 INFO:teuthology.orchestra.run.smithi189.stdout: librados-devel-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:13.622 INFO:teuthology.orchestra.run.smithi189.stdout: libradosstriper1-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:13.622 INFO:teuthology.orchestra.run.smithi189.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-30T08:20:13.623 INFO:teuthology.orchestra.run.smithi189.stdout: librgw2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:13.623 INFO:teuthology.orchestra.run.smithi189.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-30T08:20:13.623 INFO:teuthology.orchestra.run.smithi189.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-30T08:20:13.623 INFO:teuthology.orchestra.run.smithi189.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-30T08:20:13.623 INFO:teuthology.orchestra.run.smithi189.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-30T08:20:13.623 INFO:teuthology.orchestra.run.smithi189.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-30T08:20:13.623 INFO:teuthology.orchestra.run.smithi189.stdout: lua-5.4.4-4.el9.x86_64 2024-04-30T08:20:13.623 INFO:teuthology.orchestra.run.smithi189.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-30T08:20:13.623 INFO:teuthology.orchestra.run.smithi189.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-30T08:20:13.623 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-30T08:20:13.623 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-30T08:20:13.623 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-30T08:20:13.623 INFO:teuthology.orchestra.run.smithi189.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-30T08:20:13.624 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-13.14-1.el9.x86_64 2024-04-30T08:20:13.624 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-30T08:20:13.624 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-30T08:20:13.624 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-30T08:20:13.624 INFO:teuthology.orchestra.run.smithi189.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-30T08:20:13.624 INFO:teuthology.orchestra.run.smithi189.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-30T08:20:13.624 INFO:teuthology.orchestra.run.smithi189.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-30T08:20:13.624 INFO:teuthology.orchestra.run.smithi189.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-30T08:20:13.624 INFO:teuthology.orchestra.run.smithi189.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-30T08:20:13.624 INFO:teuthology.orchestra.run.smithi189.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-30T08:20:13.624 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-30T08:20:13.624 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-argparse-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:13.624 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-common-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:13.625 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cephfs-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:13.625 INFO:teuthology.orchestra.run.smithi189.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-30T08:20:13.625 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-30T08:20:13.625 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-30T08:20:13.625 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-30T08:20:13.625 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-30T08:20:13.625 INFO:teuthology.orchestra.run.smithi189.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-30T08:20:13.625 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-30T08:20:13.625 INFO:teuthology.orchestra.run.smithi189.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-30T08:20:13.625 INFO:teuthology.orchestra.run.smithi189.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-30T08:20:13.625 INFO:teuthology.orchestra.run.smithi189.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-30T08:20:13.625 INFO:teuthology.orchestra.run.smithi189.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-30T08:20:13.625 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-30T08:20:13.626 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-30T08:20:13.626 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-30T08:20:13.626 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-30T08:20:13.626 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-30T08:20:13.626 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-30T08:20:13.626 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-30T08:20:13.626 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-30T08:20:13.626 INFO:teuthology.orchestra.run.smithi189.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-30T08:20:13.626 INFO:teuthology.orchestra.run.smithi189.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-30T08:20:13.626 INFO:teuthology.orchestra.run.smithi189.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-30T08:20:13.626 INFO:teuthology.orchestra.run.smithi189.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-30T08:20:13.626 INFO:teuthology.orchestra.run.smithi189.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-30T08:20:13.626 INFO:teuthology.orchestra.run.smithi189.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-30T08:20:13.627 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-30T08:20:13.627 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-30T08:20:13.627 INFO:teuthology.orchestra.run.smithi189.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-30T08:20:13.627 INFO:teuthology.orchestra.run.smithi189.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-30T08:20:13.627 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-30T08:20:13.627 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ply-3.11-14.el9.noarch 2024-04-30T08:20:13.627 INFO:teuthology.orchestra.run.smithi189.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-30T08:20:13.627 INFO:teuthology.orchestra.run.smithi189.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-30T08:20:13.627 INFO:teuthology.orchestra.run.smithi189.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-30T08:20:13.627 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-30T08:20:13.627 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-30T08:20:13.627 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-30T08:20:13.627 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-30T08:20:13.628 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-30T08:20:13.628 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-30T08:20:13.628 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rados-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:13.628 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rbd-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:13.628 INFO:teuthology.orchestra.run.smithi189.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-30T08:20:13.628 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-30T08:20:13.628 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-30T08:20:13.628 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rgw-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:13.628 INFO:teuthology.orchestra.run.smithi189.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-30T08:20:13.628 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-30T08:20:13.628 INFO:teuthology.orchestra.run.smithi189.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-30T08:20:13.628 INFO:teuthology.orchestra.run.smithi189.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-30T08:20:13.628 INFO:teuthology.orchestra.run.smithi189.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-30T08:20:13.629 INFO:teuthology.orchestra.run.smithi189.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-30T08:20:13.629 INFO:teuthology.orchestra.run.smithi189.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-30T08:20:13.629 INFO:teuthology.orchestra.run.smithi189.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-30T08:20:13.629 INFO:teuthology.orchestra.run.smithi189.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-30T08:20:13.629 INFO:teuthology.orchestra.run.smithi189.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-30T08:20:13.629 INFO:teuthology.orchestra.run.smithi189.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-30T08:20:13.629 INFO:teuthology.orchestra.run.smithi189.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-30T08:20:13.629 INFO:teuthology.orchestra.run.smithi189.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-30T08:20:13.629 INFO:teuthology.orchestra.run.smithi189.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-30T08:20:13.629 INFO:teuthology.orchestra.run.smithi189.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-30T08:20:13.629 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-fuse-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:13.629 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-mirror-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:13.629 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-nbd-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:20:13.630 INFO:teuthology.orchestra.run.smithi189.stdout: re2-1:20211101-3.el9.x86_64 2024-04-30T08:20:13.630 INFO:teuthology.orchestra.run.smithi189.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-30T08:20:13.630 INFO:teuthology.orchestra.run.smithi189.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-30T08:20:13.630 INFO:teuthology.orchestra.run.smithi189.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-30T08:20:13.630 INFO:teuthology.orchestra.run.smithi189.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-30T08:20:13.630 INFO:teuthology.orchestra.run.smithi189.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-30T08:20:13.631 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-30T08:20:13.631 INFO:teuthology.orchestra.run.smithi189.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-30T08:20:13.631 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-30T08:20:13.631 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-04-30T08:20:13.767 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-30T08:20:13.823 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-30T08:20:13.866 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-30T08:20:13.918 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-30T08:20:13.976 DEBUG:teuthology.parallel:result is None 2024-04-30T08:20:14.067 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-30T08:20:14.130 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-30T08:20:15.085 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-30T08:20:15.155 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-30T08:20:15.188 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-30T08:20:15.312 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-30T08:20:15.393 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-30T08:20:15.505 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-30T08:20:15.548 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-30T08:20:15.792 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-30T08:20:15.837 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-30T08:20:15.951 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-30T08:20:16.828 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-30T08:20:16.989 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-30T08:20:18.702 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-30T08:20:18.805 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-30T08:20:18.849 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-30T08:20:18.896 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-30T08:20:19.329 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-30T08:20:19.382 INFO:teuthology.orchestra.run.smithi032.stdout: Upgrading : librbd1-2:19.0.0-3348.g8feaf7d9.el9.x86_64 99/156 2024-04-30T08:20:19.507 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librbd1-2:19.0.0-3348.g8feaf7d9.el9.x86_64 99/156 2024-04-30T08:20:19.540 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rbd-2:19.0.0-3348.g8feaf7d9.el9.x86_64 100/156 2024-04-30T08:20:19.623 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-30T08:20:20.214 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 102/156 2024-04-30T08:20:20.243 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : librgw2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 103/156 2024-04-30T08:20:20.368 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librgw2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 103/156 2024-04-30T08:20:20.387 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-rgw-2:19.0.0-3348.g8feaf7d9.el9.x86_64 104/156 2024-04-30T08:20:23.064 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-common-2:19.0.0-3348.g8feaf7d9.el9.x86_64 105/156 2024-04-30T08:20:23.108 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-common-2:19.0.0-3348.g8feaf7d9.el9.x86_64 105/156 2024-04-30T08:20:23.817 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-common-2:19.0.0-3348.g8feaf7d9.el9.x86_64 105/156 2024-04-30T08:20:23.862 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-base-2:19.0.0-3348.g8feaf7d9.el9.x86_64 106/156 2024-04-30T08:20:24.029 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-base-2:19.0.0-3348.g8feaf7d9.el9.x86_64 106/156 2024-04-30T08:20:24.029 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-30T08:20:24.029 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-30T08:20:24.030 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:20:24.071 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-selinux-2:19.0.0-3348.g8feaf7d9.el9.x86_64 107/156 2024-04-30T08:20:37.726 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3348.g8feaf7d9.el9.x86_64 107/156 2024-04-30T08:20:37.727 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /sys 2024-04-30T08:20:37.727 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /proc 2024-04-30T08:20:37.727 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /mnt 2024-04-30T08:20:37.727 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /var/tmp 2024-04-30T08:20:37.727 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /home 2024-04-30T08:20:37.727 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /root 2024-04-30T08:20:37.727 INFO:teuthology.orchestra.run.smithi032.stdout:skipping the directory /tmp 2024-04-30T08:20:37.727 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:20:37.988 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-volume-2:19.0.0-3348.g8feaf7d9.el9.noarch 108/156 2024-04-30T08:20:38.035 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-volume-2:19.0.0-3348.g8feaf7d9.el9.noarch 108/156 2024-04-30T08:20:38.035 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-30T08:20:38.036 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-30T08:20:38.036 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:20:39.867 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-osd-2:19.0.0-3348.g8feaf7d9.el9.x86_64 109/156 2024-04-30T08:20:39.968 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-osd-2:19.0.0-3348.g8feaf7d9.el9.x86_64 109/156 2024-04-30T08:20:39.968 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-30T08:20:39.968 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-30T08:20:39.968 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-30T08:20:39.969 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-30T08:20:39.969 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:20:40.251 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mds-2:19.0.0-3348.g8feaf7d9.el9.x86_64 110/156 2024-04-30T08:20:40.337 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mds-2:19.0.0-3348.g8feaf7d9.el9.x86_64 110/156 2024-04-30T08:20:40.337 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-30T08:20:40.337 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-30T08:20:40.337 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-30T08:20:40.337 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-30T08:20:40.338 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:20:40.901 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mon-2:19.0.0-3348.g8feaf7d9.el9.x86_64 111/156 2024-04-30T08:20:40.999 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mon-2:19.0.0-3348.g8feaf7d9.el9.x86_64 111/156 2024-04-30T08:20:41.000 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-30T08:20:41.000 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-30T08:20:41.000 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-30T08:20:41.000 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-30T08:20:41.000 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:20:41.136 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-30T08:20:41.233 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-30T08:20:41.313 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-30T08:20:41.565 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-30T08:20:41.633 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-30T08:20:41.722 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-30T08:20:41.856 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-30T08:20:42.029 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-30T08:20:42.081 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-30T08:20:42.122 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-30T08:20:42.155 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-30T08:20:42.841 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: cephadm-2:19.0.0-3348.g8feaf7d9.el9.noarch 123/156 2024-04-30T08:20:42.883 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : cephadm-2:19.0.0-3348.g8feaf7d9.el9.noarch 123/156 2024-04-30T08:20:42.964 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3348.g8feaf7d9.e 124/156 2024-04-30T08:20:43.083 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3348.g8feaf7d9. 125/156 2024-04-30T08:20:43.118 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3348.g8feaf7d9.el9.noa 126/156 2024-04-30T08:20:44.485 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3348.g8feaf7d9.el9.noa 126/156 2024-04-30T08:20:44.529 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3348.g8feaf7d9.el9.n 127/156 2024-04-30T08:20:45.696 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3348.g8feaf7d9.el9.n 127/156 2024-04-30T08:20:45.720 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3348.g8fe 128/156 2024-04-30T08:20:45.875 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3348.g8fe 128/156 2024-04-30T08:20:46.077 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3348.g8feaf7d9.el 129/156 2024-04-30T08:20:46.108 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-2:19.0.0-3348.g8feaf7d9.el9.x86_64 130/156 2024-04-30T08:20:46.205 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3348.g8feaf7d9.el9.x86_64 130/156 2024-04-30T08:20:46.205 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-30T08:20:46.205 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-30T08:20:46.205 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-30T08:20:46.206 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-30T08:20:46.206 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:20:46.267 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-mgr-rook-2:19.0.0-3348.g8feaf7d9.el9.noarch 131/156 2024-04-30T08:20:46.308 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3348.g8feaf7d9.el9.noarch 131/156 2024-04-30T08:20:46.440 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-2:19.0.0-3348.g8feaf7d9.el9.x86_64 132/156 2024-04-30T08:20:46.522 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-fuse-2:19.0.0-3348.g8feaf7d9.el9.x86_64 133/156 2024-04-30T08:20:47.624 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-30T08:20:47.661 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-radosgw-2:19.0.0-3348.g8feaf7d9.el9.x86_64 135/156 2024-04-30T08:20:47.747 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3348.g8feaf7d9.el9.x86_64 135/156 2024-04-30T08:20:47.747 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-30T08:20:47.747 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-30T08:20:47.747 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-30T08:20:47.747 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-30T08:20:47.747 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:20:47.804 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3348.g8feaf 136/156 2024-04-30T08:20:47.902 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3348.g8feaf 136/156 2024-04-30T08:20:47.902 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-30T08:20:47.902 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-30T08:20:47.902 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:20:47.979 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : cephfs-mirror-2:19.0.0-3348.g8feaf7d9.el9.x86_64 137/156 2024-04-30T08:20:48.085 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3348.g8feaf7d9.el9.x86_64 137/156 2024-04-30T08:20:48.085 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-30T08:20:48.085 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-30T08:20:48.085 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-30T08:20:48.086 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-30T08:20:48.086 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:20:48.488 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-mirror-2:19.0.0-3348.g8feaf7d9.el9.x86_64 138/156 2024-04-30T08:20:48.570 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3348.g8feaf7d9.el9.x86_64 138/156 2024-04-30T08:20:48.570 INFO:teuthology.orchestra.run.smithi032.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-30T08:20:48.571 INFO:teuthology.orchestra.run.smithi032.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-30T08:20:48.571 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-30T08:20:48.571 INFO:teuthology.orchestra.run.smithi032.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-30T08:20:48.571 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:20:53.539 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : ceph-test-2:19.0.0-3348.g8feaf7d9.el9.x86_64 139/156 2024-04-30T08:20:53.590 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-fuse-2:19.0.0-3348.g8feaf7d9.el9.x86_64 140/156 2024-04-30T08:20:53.680 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : rbd-nbd-2:19.0.0-3348.g8feaf7d9.el9.x86_64 141/156 2024-04-30T08:20:53.725 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-30T08:20:53.773 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-30T08:20:54.132 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : cephfs-shell-2:19.0.0-3348.g8feaf7d9.el9.x86_64 144/156 2024-04-30T08:20:54.182 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-30T08:20:54.210 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libcephfs-devel-2:19.0.0-3348.g8feaf7d9.el9.x86_ 146/156 2024-04-30T08:20:55.115 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-30T08:20:55.157 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-30T08:20:55.223 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-30T08:20:56.674 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : cephfs-top-2:19.0.0-3348.g8feaf7d9.el9.noarch 148/156 2024-04-30T08:20:56.781 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-30T08:20:56.867 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-30T08:20:57.054 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-30T08:20:57.096 INFO:teuthology.orchestra.run.smithi032.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-30T08:20:57.096 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-30T08:20:57.214 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-30T08:20:57.261 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-30T08:20:57.261 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-30T08:20:57.393 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-30T08:20:57.528 INFO:teuthology.orchestra.run.smithi032.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-30T08:20:57.812 INFO:teuthology.orchestra.run.smithi032.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-30T08:20:57.813 INFO:teuthology.orchestra.run.smithi032.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-30T08:20:57.813 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:21:03.578 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-2:19.0.0-3348.g8feaf7d9.el9.x86_64 1/156 2024-04-30T08:21:03.578 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-base-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2/156 2024-04-30T08:21:03.578 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-common-2:19.0.0-3348.g8feaf7d9.el9.x86_64 3/156 2024-04-30T08:21:03.578 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-fuse-2:19.0.0-3348.g8feaf7d9.el9.x86_64 4/156 2024-04-30T08:21:03.578 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3348.g8feaf 5/156 2024-04-30T08:21:03.578 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mds-2:19.0.0-3348.g8feaf7d9.el9.x86_64 6/156 2024-04-30T08:21:03.578 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-2:19.0.0-3348.g8feaf7d9.el9.x86_64 7/156 2024-04-30T08:21:03.579 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mon-2:19.0.0-3348.g8feaf7d9.el9.x86_64 8/156 2024-04-30T08:21:03.579 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-osd-2:19.0.0-3348.g8feaf7d9.el9.x86_64 9/156 2024-04-30T08:21:03.579 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-radosgw-2:19.0.0-3348.g8feaf7d9.el9.x86_64 10/156 2024-04-30T08:21:03.579 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-selinux-2:19.0.0-3348.g8feaf7d9.el9.x86_64 11/156 2024-04-30T08:21:03.579 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-test-2:19.0.0-3348.g8feaf7d9.el9.x86_64 12/156 2024-04-30T08:21:03.579 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : cephfs-mirror-2:19.0.0-3348.g8feaf7d9.el9.x86_64 13/156 2024-04-30T08:21:03.579 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : cephfs-shell-2:19.0.0-3348.g8feaf7d9.el9.x86_64 14/156 2024-04-30T08:21:03.579 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephfs-devel-2:19.0.0-3348.g8feaf7d9.el9.x86_ 15/156 2024-04-30T08:21:03.579 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephfs2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 16/156 2024-04-30T08:21:03.579 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libcephsqlite-2:19.0.0-3348.g8feaf7d9.el9.x86_64 17/156 2024-04-30T08:21:03.579 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados-devel-2:19.0.0-3348.g8feaf7d9.el9.x86_6 18/156 2024-04-30T08:21:03.579 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libradosstriper1-2:19.0.0-3348.g8feaf7d9.el9.x86 19/156 2024-04-30T08:21:03.580 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librgw2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 20/156 2024-04-30T08:21:03.580 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3348.g8feaf7d9.el 21/156 2024-04-30T08:21:03.580 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ceph-common-2:19.0.0-3348.g8feaf7d9.el9. 22/156 2024-04-30T08:21:03.580 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cephfs-2:19.0.0-3348.g8feaf7d9.el9.x86_6 23/156 2024-04-30T08:21:03.580 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rados-2:19.0.0-3348.g8feaf7d9.el9.x86_64 24/156 2024-04-30T08:21:03.580 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rbd-2:19.0.0-3348.g8feaf7d9.el9.x86_64 25/156 2024-04-30T08:21:03.580 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rgw-2:19.0.0-3348.g8feaf7d9.el9.x86_64 26/156 2024-04-30T08:21:03.580 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-fuse-2:19.0.0-3348.g8feaf7d9.el9.x86_64 27/156 2024-04-30T08:21:03.580 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-mirror-2:19.0.0-3348.g8feaf7d9.el9.x86_64 28/156 2024-04-30T08:21:03.580 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : rbd-nbd-2:19.0.0-3348.g8feaf7d9.el9.x86_64 29/156 2024-04-30T08:21:03.592 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3348.g8feaf7d9. 30/156 2024-04-30T08:21:03.592 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3348.g8feaf7d9.el9.noa 31/156 2024-04-30T08:21:03.592 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3348.g8feaf7d9.el9.n 32/156 2024-04-30T08:21:03.592 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3348.g8fe 33/156 2024-04-30T08:21:03.592 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3348.g8feaf7d9.el 34/156 2024-04-30T08:21:03.592 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3348.g8feaf7d9.el9.noarch 35/156 2024-04-30T08:21:03.592 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3348.g8feaf7d9.e 36/156 2024-04-30T08:21:03.592 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : ceph-volume-2:19.0.0-3348.g8feaf7d9.el9.noarch 37/156 2024-04-30T08:21:03.592 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : cephadm-2:19.0.0-3348.g8feaf7d9.el9.noarch 38/156 2024-04-30T08:21:03.593 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : cephfs-top-2:19.0.0-3348.g8feaf7d9.el9.noarch 39/156 2024-04-30T08:21:03.593 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-30T08:21:03.593 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-30T08:21:03.593 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-30T08:21:03.593 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-30T08:21:03.593 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-30T08:21:03.593 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-30T08:21:03.593 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-30T08:21:03.593 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-30T08:21:03.593 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-30T08:21:03.594 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-30T08:21:03.594 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-30T08:21:03.594 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-30T08:21:03.594 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-30T08:21:03.594 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-30T08:21:03.594 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-30T08:21:03.594 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-30T08:21:03.594 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-30T08:21:03.594 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-30T08:21:03.594 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-30T08:21:03.594 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-30T08:21:03.594 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-30T08:21:03.595 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-30T08:21:03.595 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-30T08:21:03.595 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-30T08:21:03.595 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-30T08:21:03.595 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-30T08:21:03.595 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-30T08:21:03.595 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-30T08:21:03.595 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-30T08:21:03.595 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-30T08:21:03.595 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-30T08:21:03.595 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-30T08:21:03.595 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-30T08:21:03.596 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-30T08:21:03.596 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-30T08:21:03.596 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-30T08:21:03.596 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-30T08:21:03.596 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-30T08:21:03.596 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-30T08:21:03.596 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-30T08:21:03.596 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-30T08:21:03.596 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-30T08:21:03.596 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-30T08:21:03.596 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-30T08:21:03.596 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-30T08:21:03.596 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-30T08:21:03.597 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-30T08:21:03.597 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-30T08:21:03.597 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-30T08:21:03.597 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-30T08:21:03.597 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-30T08:21:03.597 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-30T08:21:03.597 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-30T08:21:03.597 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-30T08:21:03.597 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-30T08:21:03.597 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-30T08:21:03.597 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-30T08:21:03.597 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-30T08:21:03.597 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-30T08:21:03.597 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-30T08:21:03.598 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-30T08:21:03.598 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-30T08:21:03.598 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-30T08:21:03.598 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-30T08:21:03.598 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-30T08:21:03.598 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-30T08:21:03.598 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-30T08:21:03.598 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-30T08:21:03.598 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/156 2024-04-30T08:21:03.598 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/156 2024-04-30T08:21:03.598 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-30T08:21:03.598 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-30T08:21:03.598 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-30T08:21:03.599 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/156 2024-04-30T08:21:03.599 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-30T08:21:03.599 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-30T08:21:03.599 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-30T08:21:03.599 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-30T08:21:03.599 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-30T08:21:03.599 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-30T08:21:03.599 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-30T08:21:03.599 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-30T08:21:03.599 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-30T08:21:03.599 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-30T08:21:03.599 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-30T08:21:03.599 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-30T08:21:03.600 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-30T08:21:03.600 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-30T08:21:03.600 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-30T08:21:03.600 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-30T08:21:03.600 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-30T08:21:03.600 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-30T08:21:03.600 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-30T08:21:03.600 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-30T08:21:03.600 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-30T08:21:03.600 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-30T08:21:03.600 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-30T08:21:03.600 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-30T08:21:03.600 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-30T08:21:03.600 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-30T08:21:03.601 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-30T08:21:03.601 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-30T08:21:03.601 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-30T08:21:03.601 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-30T08:21:03.601 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-30T08:21:03.601 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-30T08:21:03.601 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-30T08:21:03.601 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-30T08:21:03.601 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-30T08:21:03.601 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 149/156 2024-04-30T08:21:03.601 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-30T08:21:03.601 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librbd1-2:19.0.0-3348.g8feaf7d9.el9.x86_64 151/156 2024-04-30T08:21:03.601 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-30T08:21:03.602 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-30T08:21:03.602 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-30T08:21:03.602 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-30T08:21:06.051 INFO:teuthology.orchestra.run.smithi032.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-30T08:21:06.052 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:21:06.052 INFO:teuthology.orchestra.run.smithi032.stdout:Upgraded: 2024-04-30T08:21:06.052 INFO:teuthology.orchestra.run.smithi032.stdout: librados2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:21:06.052 INFO:teuthology.orchestra.run.smithi032.stdout: librbd1-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:21:06.052 INFO:teuthology.orchestra.run.smithi032.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-30T08:21:06.052 INFO:teuthology.orchestra.run.smithi032.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-30T08:21:06.052 INFO:teuthology.orchestra.run.smithi032.stdout:Installed: 2024-04-30T08:21:06.052 INFO:teuthology.orchestra.run.smithi032.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-30T08:21:06.052 INFO:teuthology.orchestra.run.smithi032.stdout: automake-1.16.2-8.el9.noarch 2024-04-30T08:21:06.052 INFO:teuthology.orchestra.run.smithi032.stdout: bison-3.7.4-5.el9.x86_64 2024-04-30T08:21:06.053 INFO:teuthology.orchestra.run.smithi032.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-30T08:21:06.053 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:21:06.053 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-base-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:21:06.053 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-common-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:21:06.053 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-fuse-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:21:06.053 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-grafana-dashboards-2:19.0.0-3348.g8feaf7d9.el9.noarch 2024-04-30T08:21:06.053 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-immutable-object-cache-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:21:06.053 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mds-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:21:06.053 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:21:06.053 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-cephadm-2:19.0.0-3348.g8feaf7d9.el9.noarch 2024-04-30T08:21:06.053 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-dashboard-2:19.0.0-3348.g8feaf7d9.el9.noarch 2024-04-30T08:21:06.053 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3348.g8feaf7d9.el9.noarch 2024-04-30T08:21:06.054 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-modules-core-2:19.0.0-3348.g8feaf7d9.el9.noarch 2024-04-30T08:21:06.054 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mgr-rook-2:19.0.0-3348.g8feaf7d9.el9.noarch 2024-04-30T08:21:06.054 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-mon-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:21:06.054 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-osd-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:21:06.054 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-prometheus-alerts-2:19.0.0-3348.g8feaf7d9.el9.noarch 2024-04-30T08:21:06.054 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-radosgw-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:21:06.054 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-selinux-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:21:06.054 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-test-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:21:06.054 INFO:teuthology.orchestra.run.smithi032.stdout: ceph-volume-2:19.0.0-3348.g8feaf7d9.el9.noarch 2024-04-30T08:21:06.054 INFO:teuthology.orchestra.run.smithi032.stdout: cephadm-2:19.0.0-3348.g8feaf7d9.el9.noarch 2024-04-30T08:21:06.054 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-mirror-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:21:06.054 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-shell-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:21:06.055 INFO:teuthology.orchestra.run.smithi032.stdout: cephfs-top-2:19.0.0-3348.g8feaf7d9.el9.noarch 2024-04-30T08:21:06.055 INFO:teuthology.orchestra.run.smithi032.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-30T08:21:06.055 INFO:teuthology.orchestra.run.smithi032.stdout: flex-2.6.4-9.el9.x86_64 2024-04-30T08:21:06.055 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-30T08:21:06.055 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-30T08:21:06.055 INFO:teuthology.orchestra.run.smithi032.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-30T08:21:06.055 INFO:teuthology.orchestra.run.smithi032.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-30T08:21:06.055 INFO:teuthology.orchestra.run.smithi032.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-30T08:21:06.055 INFO:teuthology.orchestra.run.smithi032.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-30T08:21:06.055 INFO:teuthology.orchestra.run.smithi032.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-30T08:21:06.055 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow-9.0.0-11.el9.x86_64 2024-04-30T08:21:06.055 INFO:teuthology.orchestra.run.smithi032.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-04-30T08:21:06.055 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs-devel-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:21:06.056 INFO:teuthology.orchestra.run.smithi032.stdout: libcephfs2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:21:06.056 INFO:teuthology.orchestra.run.smithi032.stdout: libcephsqlite-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:21:06.056 INFO:teuthology.orchestra.run.smithi032.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-30T08:21:06.056 INFO:teuthology.orchestra.run.smithi032.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-30T08:21:06.056 INFO:teuthology.orchestra.run.smithi032.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-30T08:21:06.056 INFO:teuthology.orchestra.run.smithi032.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-30T08:21:06.056 INFO:teuthology.orchestra.run.smithi032.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-30T08:21:06.056 INFO:teuthology.orchestra.run.smithi032.stdout: librados-devel-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:21:06.056 INFO:teuthology.orchestra.run.smithi032.stdout: libradosstriper1-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:21:06.056 INFO:teuthology.orchestra.run.smithi032.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-30T08:21:06.056 INFO:teuthology.orchestra.run.smithi032.stdout: librgw2-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:21:06.056 INFO:teuthology.orchestra.run.smithi032.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-30T08:21:06.056 INFO:teuthology.orchestra.run.smithi032.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-30T08:21:06.057 INFO:teuthology.orchestra.run.smithi032.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-30T08:21:06.057 INFO:teuthology.orchestra.run.smithi032.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-30T08:21:06.057 INFO:teuthology.orchestra.run.smithi032.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-30T08:21:06.057 INFO:teuthology.orchestra.run.smithi032.stdout: lua-5.4.4-4.el9.x86_64 2024-04-30T08:21:06.057 INFO:teuthology.orchestra.run.smithi032.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-30T08:21:06.057 INFO:teuthology.orchestra.run.smithi032.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-30T08:21:06.057 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-30T08:21:06.057 INFO:teuthology.orchestra.run.smithi032.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-30T08:21:06.057 INFO:teuthology.orchestra.run.smithi032.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-30T08:21:06.057 INFO:teuthology.orchestra.run.smithi032.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-04-30T08:21:06.057 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-13.14-1.el9.x86_64 2024-04-30T08:21:06.057 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-30T08:21:06.057 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-30T08:21:06.058 INFO:teuthology.orchestra.run.smithi032.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-30T08:21:06.058 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-30T08:21:06.058 INFO:teuthology.orchestra.run.smithi032.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-30T08:21:06.058 INFO:teuthology.orchestra.run.smithi032.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-30T08:21:06.058 INFO:teuthology.orchestra.run.smithi032.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-30T08:21:06.058 INFO:teuthology.orchestra.run.smithi032.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-30T08:21:06.058 INFO:teuthology.orchestra.run.smithi032.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-30T08:21:06.058 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-30T08:21:06.058 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-argparse-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:21:06.058 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ceph-common-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:21:06.058 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cephfs-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:21:06.058 INFO:teuthology.orchestra.run.smithi032.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-30T08:21:06.058 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-30T08:21:06.059 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-30T08:21:06.059 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-30T08:21:06.059 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-30T08:21:06.059 INFO:teuthology.orchestra.run.smithi032.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-30T08:21:06.059 INFO:teuthology.orchestra.run.smithi032.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-30T08:21:06.059 INFO:teuthology.orchestra.run.smithi032.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-30T08:21:06.059 INFO:teuthology.orchestra.run.smithi032.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-30T08:21:06.059 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-30T08:21:06.059 INFO:teuthology.orchestra.run.smithi032.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-30T08:21:06.059 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-30T08:21:06.059 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-30T08:21:06.059 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-30T08:21:06.059 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-30T08:21:06.060 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-30T08:21:06.060 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-30T08:21:06.060 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-30T08:21:06.060 INFO:teuthology.orchestra.run.smithi032.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-30T08:21:06.060 INFO:teuthology.orchestra.run.smithi032.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-30T08:21:06.060 INFO:teuthology.orchestra.run.smithi032.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-30T08:21:06.060 INFO:teuthology.orchestra.run.smithi032.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-30T08:21:06.060 INFO:teuthology.orchestra.run.smithi032.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-30T08:21:06.060 INFO:teuthology.orchestra.run.smithi032.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-30T08:21:06.060 INFO:teuthology.orchestra.run.smithi032.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-30T08:21:06.060 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-30T08:21:06.060 INFO:teuthology.orchestra.run.smithi032.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-30T08:21:06.061 INFO:teuthology.orchestra.run.smithi032.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-30T08:21:06.061 INFO:teuthology.orchestra.run.smithi032.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-30T08:21:06.061 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-30T08:21:06.061 INFO:teuthology.orchestra.run.smithi032.stdout: python3-ply-3.11-14.el9.noarch 2024-04-30T08:21:06.061 INFO:teuthology.orchestra.run.smithi032.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-30T08:21:06.061 INFO:teuthology.orchestra.run.smithi032.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-30T08:21:06.061 INFO:teuthology.orchestra.run.smithi032.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-30T08:21:06.061 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-30T08:21:06.061 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-30T08:21:06.061 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-30T08:21:06.061 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-30T08:21:06.061 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-30T08:21:06.061 INFO:teuthology.orchestra.run.smithi032.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-30T08:21:06.062 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rados-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:21:06.062 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rbd-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:21:06.062 INFO:teuthology.orchestra.run.smithi032.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-30T08:21:06.062 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-30T08:21:06.062 INFO:teuthology.orchestra.run.smithi032.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-30T08:21:06.062 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rgw-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:21:06.062 INFO:teuthology.orchestra.run.smithi032.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-30T08:21:06.062 INFO:teuthology.orchestra.run.smithi032.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-30T08:21:06.062 INFO:teuthology.orchestra.run.smithi032.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-30T08:21:06.062 INFO:teuthology.orchestra.run.smithi032.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-30T08:21:06.062 INFO:teuthology.orchestra.run.smithi032.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-30T08:21:06.062 INFO:teuthology.orchestra.run.smithi032.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-30T08:21:06.062 INFO:teuthology.orchestra.run.smithi032.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-30T08:21:06.063 INFO:teuthology.orchestra.run.smithi032.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-30T08:21:06.063 INFO:teuthology.orchestra.run.smithi032.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-30T08:21:06.063 INFO:teuthology.orchestra.run.smithi032.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-30T08:21:06.063 INFO:teuthology.orchestra.run.smithi032.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-30T08:21:06.063 INFO:teuthology.orchestra.run.smithi032.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-30T08:21:06.063 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-30T08:21:06.063 INFO:teuthology.orchestra.run.smithi032.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-30T08:21:06.063 INFO:teuthology.orchestra.run.smithi032.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-30T08:21:06.063 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-fuse-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:21:06.063 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-mirror-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:21:06.063 INFO:teuthology.orchestra.run.smithi032.stdout: rbd-nbd-2:19.0.0-3348.g8feaf7d9.el9.x86_64 2024-04-30T08:21:06.063 INFO:teuthology.orchestra.run.smithi032.stdout: re2-1:20211101-3.el9.x86_64 2024-04-30T08:21:06.064 INFO:teuthology.orchestra.run.smithi032.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-30T08:21:06.064 INFO:teuthology.orchestra.run.smithi032.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-30T08:21:06.064 INFO:teuthology.orchestra.run.smithi032.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-30T08:21:06.064 INFO:teuthology.orchestra.run.smithi032.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-30T08:21:06.064 INFO:teuthology.orchestra.run.smithi032.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-30T08:21:06.064 INFO:teuthology.orchestra.run.smithi032.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-30T08:21:06.064 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:21:06.064 INFO:teuthology.orchestra.run.smithi032.stdout:Complete! 2024-04-30T08:21:06.647 DEBUG:teuthology.parallel:result is None 2024-04-30T08:21:06.647 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8feaf7d9864e980041b042fd5d964ac18adfd20f 2024-04-30T08:21:06.873 DEBUG:teuthology.orchestra.run.smithi032:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-30T08:21:06.942 INFO:teuthology.orchestra.run.smithi032.stdout:19.0.0-3348.g8feaf7d9.el9 2024-04-30T08:21:06.942 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3348.g8feaf7d9.el9 2024-04-30T08:21:06.942 INFO:teuthology.task.install:The correct ceph version 19.0.0-3348.g8feaf7d9 is installed. 2024-04-30T08:21:06.944 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8feaf7d9864e980041b042fd5d964ac18adfd20f 2024-04-30T08:21:07.168 DEBUG:teuthology.orchestra.run.smithi071:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-30T08:21:07.193 INFO:teuthology.orchestra.run.smithi071.stdout:19.0.0-3348.g8feaf7d9.el9 2024-04-30T08:21:07.193 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3348.g8feaf7d9.el9 2024-04-30T08:21:07.193 INFO:teuthology.task.install:The correct ceph version 19.0.0-3348.g8feaf7d9 is installed. 2024-04-30T08:21:07.195 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8feaf7d9864e980041b042fd5d964ac18adfd20f 2024-04-30T08:21:07.336 DEBUG:teuthology.orchestra.run.smithi189:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-30T08:21:07.362 INFO:teuthology.orchestra.run.smithi189.stdout:19.0.0-3348.g8feaf7d9.el9 2024-04-30T08:21:07.363 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3348.g8feaf7d9.el9 2024-04-30T08:21:07.363 INFO:teuthology.task.install:The correct ceph version 19.0.0-3348.g8feaf7d9 is installed. 2024-04-30T08:21:07.364 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-30T08:21:07.365 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-30T08:21:07.365 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-30T08:21:07.442 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-30T08:21:07.442 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-30T08:21:07.473 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-30T08:21:07.473 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-30T08:21:07.506 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-30T08:21:07.507 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-30T08:21:07.507 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/daemon-helper 2024-04-30T08:21:07.589 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-30T08:21:07.730 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-30T08:21:07.730 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/daemon-helper 2024-04-30T08:21:07.759 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-30T08:21:07.825 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-30T08:21:07.825 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/daemon-helper 2024-04-30T08:21:07.854 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-30T08:21:07.919 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-30T08:21:07.919 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-30T08:21:07.919 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-30T08:21:07.989 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-30T08:21:08.096 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-30T08:21:08.096 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-30T08:21:08.120 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-30T08:21:08.203 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-30T08:21:08.203 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-30T08:21:08.227 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-30T08:21:08.292 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-30T08:21:08.292 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-30T08:21:08.292 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/usr/bin/stdin-killer 2024-04-30T08:21:08.380 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-30T08:21:08.485 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-30T08:21:08.485 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/usr/bin/stdin-killer 2024-04-30T08:21:08.510 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-30T08:21:08.573 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-30T08:21:08.573 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/stdin-killer 2024-04-30T08:21:08.597 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-30T08:21:08.661 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-30T08:21:08.769 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd heartbeat grace': 60, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request', 'SLOW_OPS', 'slow request'], 'sha1': '8feaf7d9864e980041b042fd5d964ac18adfd20f', 'subvols': {'create': 2}} 2024-04-30T08:21:08.769 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f 2024-04-30T08:21:08.769 INFO:tasks.cephadm:Cluster fsid is 98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:21:08.769 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-30T08:21:08.770 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.32', 'mon.b': '172.21.15.71', 'mon.c': '172.21.15.189'} 2024-04-30T08:21:08.770 INFO:tasks.cephadm:First mon is mon.a on smithi032 2024-04-30T08:21:08.770 INFO:tasks.cephadm:First mgr is x 2024-04-30T08:21:08.770 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-30T08:21:08.770 DEBUG:teuthology.orchestra.run.smithi032:> sudo hostname $(hostname -s) 2024-04-30T08:21:08.836 DEBUG:teuthology.orchestra.run.smithi071:> sudo hostname $(hostname -s) 2024-04-30T08:21:08.859 DEBUG:teuthology.orchestra.run.smithi189:> sudo hostname $(hostname -s) 2024-04-30T08:21:08.883 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-30T08:21:08.883 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=8feaf7d9864e980041b042fd5d964ac18adfd20f 2024-04-30T08:21:09.106 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '8feaf7d9864e980041b042fd5d964ac18adfd20f', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78949/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3348-g8feaf7d9', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3348.g8feaf7d9'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240429.210911-debug/8feaf7d9864e980041b042fd5d964ac18adfd20f/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-30 03:40:01.399403', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pdonnell-testing-20240429.210911-debug', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-pdonnell-testing-20240429.210911-debug/8feaf7d9864e980041b042fd5d964ac18adfd20f/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}, {'status': 'ready', 'sha1': '8feaf7d9864e980041b042fd5d964ac18adfd20f', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78947/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3348-g8feaf7d9', 'node_name': '172.21.5.40+adami10', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3348.g8feaf7d9'}, 'url': 'https://1.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240429.210911-debug/8feaf7d9864e980041b042fd5d964ac18adfd20f/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-29 23:09:46.570698', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-pdonnell-testing-20240429.210911-debug', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/wip-pdonnell-testing-20240429.210911-debug/8feaf7d9864e980041b042fd5d964ac18adfd20f/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-30T08:21:09.333 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-pdonnell-testing-20240429.210911-debug, sha1 8feaf7d9864e980041b042fd5d964ac18adfd20f from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=8feaf7d9864e980041b042fd5d964ac18adfd20f 2024-04-30T08:21:09.335 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240429.210911-debug/8feaf7d9864e980041b042fd5d964ac18adfd20f/centos/9/x86_64/flavors/default/cephadm 2024-04-30T08:21:09.335 DEBUG:teuthology.orchestra.run.smithi032:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240429.210911-debug/8feaf7d9864e980041b042fd5d964ac18adfd20f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-30T08:21:11.337 INFO:teuthology.orchestra.run.smithi032.stdout:-rw-r--r--. 1 ubuntu ubuntu 781599 Apr 30 08:21 /home/ubuntu/cephtest/cephadm 2024-04-30T08:21:11.338 DEBUG:teuthology.orchestra.run.smithi071:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240429.210911-debug/8feaf7d9864e980041b042fd5d964ac18adfd20f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-30T08:21:12.109 INFO:teuthology.orchestra.run.smithi071.stdout:-rw-r--r--. 1 ubuntu ubuntu 781599 Apr 30 08:21 /home/ubuntu/cephtest/cephadm 2024-04-30T08:21:12.162 DEBUG:teuthology.orchestra.run.smithi189:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-pdonnell-testing-20240429.210911-debug/8feaf7d9864e980041b042fd5d964ac18adfd20f/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-30T08:21:13.167 INFO:teuthology.orchestra.run.smithi189.stdout:-rw-r--r--. 1 ubuntu ubuntu 781599 Apr 30 08:21 /home/ubuntu/cephtest/cephadm 2024-04-30T08:21:13.167 DEBUG:teuthology.orchestra.run.smithi032:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-30T08:21:13.216 DEBUG:teuthology.orchestra.run.smithi071:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-30T08:21:13.235 DEBUG:teuthology.orchestra.run.smithi189:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-30T08:21:13.299 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f on all hosts... 2024-04-30T08:21:13.300 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f pull 2024-04-30T08:21:13.305 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f pull 2024-04-30T08:21:13.307 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f pull 2024-04-30T08:21:14.818 INFO:teuthology.orchestra.run.smithi189.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f... 2024-04-30T08:21:14.872 INFO:teuthology.orchestra.run.smithi071.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f... 2024-04-30T08:21:15.411 INFO:teuthology.orchestra.run.smithi032.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f... 2024-04-30T08:21:44.704 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-30T08:21:44.704 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_version": "ceph version 19.0.0-3348-g8feaf7d9 (8feaf7d9864e980041b042fd5d964ac18adfd20f) squid (dev)", 2024-04-30T08:21:44.704 INFO:teuthology.orchestra.run.smithi189.stdout: "image_id": "16056ace0edc7f73516cf28297f1db04dda872801fb5b17b9157b0d1ccf38f9b", 2024-04-30T08:21:44.704 INFO:teuthology.orchestra.run.smithi189.stdout: "repo_digests": [ 2024-04-30T08:21:44.704 INFO:teuthology.orchestra.run.smithi189.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de" 2024-04-30T08:21:44.704 INFO:teuthology.orchestra.run.smithi189.stdout: ] 2024-04-30T08:21:44.704 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-30T08:21:54.143 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-30T08:21:54.143 INFO:teuthology.orchestra.run.smithi071.stdout: "ceph_version": "ceph version 19.0.0-3348-g8feaf7d9 (8feaf7d9864e980041b042fd5d964ac18adfd20f) squid (dev)", 2024-04-30T08:21:54.143 INFO:teuthology.orchestra.run.smithi071.stdout: "image_id": "16056ace0edc7f73516cf28297f1db04dda872801fb5b17b9157b0d1ccf38f9b", 2024-04-30T08:21:54.143 INFO:teuthology.orchestra.run.smithi071.stdout: "repo_digests": [ 2024-04-30T08:21:54.143 INFO:teuthology.orchestra.run.smithi071.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de" 2024-04-30T08:21:54.143 INFO:teuthology.orchestra.run.smithi071.stdout: ] 2024-04-30T08:21:54.144 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-30T08:21:58.184 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-30T08:21:58.184 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_version": "ceph version 19.0.0-3348-g8feaf7d9 (8feaf7d9864e980041b042fd5d964ac18adfd20f) squid (dev)", 2024-04-30T08:21:58.184 INFO:teuthology.orchestra.run.smithi032.stdout: "image_id": "16056ace0edc7f73516cf28297f1db04dda872801fb5b17b9157b0d1ccf38f9b", 2024-04-30T08:21:58.184 INFO:teuthology.orchestra.run.smithi032.stdout: "repo_digests": [ 2024-04-30T08:21:58.184 INFO:teuthology.orchestra.run.smithi032.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de" 2024-04-30T08:21:58.184 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-30T08:21:58.185 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-30T08:21:58.238 DEBUG:teuthology.orchestra.run.smithi032:> sudo mkdir -p /etc/ceph 2024-04-30T08:21:58.335 DEBUG:teuthology.orchestra.run.smithi071:> sudo mkdir -p /etc/ceph 2024-04-30T08:21:58.367 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /etc/ceph 2024-04-30T08:21:58.400 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 777 /etc/ceph 2024-04-30T08:21:58.493 DEBUG:teuthology.orchestra.run.smithi071:> sudo chmod 777 /etc/ceph 2024-04-30T08:21:58.520 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 777 /etc/ceph 2024-04-30T08:21:58.548 INFO:tasks.cephadm:Writing seed config... 2024-04-30T08:21:58.549 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-30T08:21:58.550 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-30T08:21:58.550 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-30T08:21:58.550 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-30T08:21:58.550 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-30T08:21:58.550 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-30T08:21:58.550 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-30T08:21:58.550 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-30T08:21:58.550 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-30T08:21:58.550 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-30T08:21:58.550 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-30T08:21:58.550 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-30T08:21:58.550 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-30T08:21:58.550 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-30T08:21:58.550 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-30T08:21:58.551 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-30T08:21:58.551 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-30T08:21:58.551 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-04-30T08:21:58.551 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-30T08:21:58.551 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-30T08:21:58.551 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-30T08:21:58.551 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-30T08:21:58.551 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-30T08:21:58.551 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-30T08:21:58.551 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-30T08:21:58.551 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-30T08:21:58.551 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-30T08:21:58.551 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-30T08:21:58.551 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-30T08:21:58.552 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-30T08:21:58.552 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-30T08:21:58.552 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-04-30T08:21:58.552 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-30T08:21:58.552 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-30T08:21:58.552 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-30T08:21:58.552 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-30T08:21:58.552 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-30T08:21:58.552 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-30T08:21:58.552 INFO:tasks.cephadm: override: [osd] filestore flush min = 0 2024-04-30T08:21:58.552 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-30T08:21:58.552 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-30T08:21:58.553 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-30T08:21:58.553 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-30T08:21:58.553 INFO:tasks.cephadm: override: [osd] osd heartbeat grace = 60 2024-04-30T08:21:58.553 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-30T08:21:58.553 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-30T08:21:58.553 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-30T08:21:58.553 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-30T08:21:58.554 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-30T08:21:58.554 DEBUG:teuthology.orchestra.run.smithi032:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-30T08:21:58.614 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 98c0a670-06ca-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 filestore flush min = 0 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd heartbeat grace = 60 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-30T08:21:58.615 DEBUG:teuthology.orchestra.run.smithi032:mon.a> sudo journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mon.a.service 2024-04-30T08:21:58.663 DEBUG:teuthology.orchestra.run.smithi032:mgr.x> sudo journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mgr.x.service 2024-04-30T08:21:58.708 INFO:tasks.cephadm:Bootstrapping... 2024-04-30T08:21:58.709 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f -v bootstrap --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.32 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-30T08:21:58.988 INFO:teuthology.orchestra.run.smithi032.stdout:-------------------------------------------------------------------------------- 2024-04-30T08:21:58.988 INFO:teuthology.orchestra.run.smithi032.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f', '-v', 'bootstrap', '--fsid', '98c0a670-06ca-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.32', '--skip-admin-label'] 2024-04-30T08:21:58.989 INFO:teuthology.orchestra.run.smithi032.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-30T08:21:58.989 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying podman|docker is present... 2024-04-30T08:21:59.058 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-30T08:21:59.059 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying lvm2 is present... 2024-04-30T08:21:59.059 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying time synchronization is in place... 2024-04-30T08:21:59.075 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-30T08:21:59.075 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-30T08:21:59.090 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-30T08:21:59.091 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-04-30T08:21:59.111 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout enabled 2024-04-30T08:21:59.130 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout active 2024-04-30T08:21:59.131 INFO:teuthology.orchestra.run.smithi032.stdout:Unit chronyd.service is enabled and running 2024-04-30T08:21:59.131 INFO:teuthology.orchestra.run.smithi032.stdout:Repeating the final host check... 2024-04-30T08:21:59.189 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-30T08:21:59.190 INFO:teuthology.orchestra.run.smithi032.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-30T08:21:59.190 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl is present 2024-04-30T08:21:59.190 INFO:teuthology.orchestra.run.smithi032.stdout:lvcreate is present 2024-04-30T08:21:59.206 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-30T08:21:59.206 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-30T08:21:59.227 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-30T08:21:59.227 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-04-30T08:21:59.251 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout enabled 2024-04-30T08:21:59.275 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout active 2024-04-30T08:21:59.275 INFO:teuthology.orchestra.run.smithi032.stdout:Unit chronyd.service is enabled and running 2024-04-30T08:21:59.275 INFO:teuthology.orchestra.run.smithi032.stdout:Host looks OK 2024-04-30T08:21:59.275 INFO:teuthology.orchestra.run.smithi032.stdout:Cluster fsid: 98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:21:59.275 INFO:teuthology.orchestra.run.smithi032.stdout:Acquiring lock 139642458946720 on /run/cephadm/98c0a670-06ca-11ef-bc93-c7b262605968.lock 2024-04-30T08:21:59.276 INFO:teuthology.orchestra.run.smithi032.stdout:Lock 139642458946720 acquired on /run/cephadm/98c0a670-06ca-11ef-bc93-c7b262605968.lock 2024-04-30T08:21:59.277 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying IP 172.21.15.32 port 3300 ... 2024-04-30T08:21:59.278 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying IP 172.21.15.32 port 6789 ... 2024-04-30T08:21:59.279 INFO:teuthology.orchestra.run.smithi032.stdout:Base mon IP(s) is [172.21.15.32:3300, 172.21.15.32:6789], mon addrv is [v2:172.21.15.32:3300,v1:172.21.15.32:6789] 2024-04-30T08:21:59.294 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.32 metric 100 2024-04-30T08:21:59.295 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.32 metric 100 2024-04-30T08:21:59.307 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-30T08:21:59.308 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-30T08:21:59.308 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-30T08:21:59.319 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-30T08:21:59.320 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-30T08:21:59.320 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-30T08:21:59.320 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-30T08:21:59.320 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-04-30T08:21:59.321 INFO:teuthology.orchestra.run.smithi032.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-30T08:21:59.323 INFO:teuthology.orchestra.run.smithi032.stdout:Mon IP `172.21.15.32` is in CIDR network `172.21.0.0/20` 2024-04-30T08:21:59.324 INFO:teuthology.orchestra.run.smithi032.stdout:Mon IP `172.21.15.32` is in CIDR network `172.21.0.0/20` 2024-04-30T08:21:59.324 INFO:teuthology.orchestra.run.smithi032.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-30T08:21:59.327 INFO:teuthology.orchestra.run.smithi032.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-30T08:21:59.330 INFO:teuthology.orchestra.run.smithi032.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f... 2024-04-30T08:22:00.006 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stdout 16056ace0edc7f73516cf28297f1db04dda872801fb5b17b9157b0d1ccf38f9b 2024-04-30T08:22:00.006 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f... 2024-04-30T08:22:00.006 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Getting image source signatures 2024-04-30T08:22:00.006 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Copying blob sha256:e2f2fd2adca92139da4b4b907fa90a6775d3eb4eddaa977c42ef37c66eea70bd 2024-04-30T08:22:00.006 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Copying blob sha256:2ecc180639a47cc64dfe0f4d1e10509407f10934b4f1a2e15a87d6988e587a87 2024-04-30T08:22:00.006 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Copying config sha256:16056ace0edc7f73516cf28297f1db04dda872801fb5b17b9157b0d1ccf38f9b 2024-04-30T08:22:00.006 INFO:teuthology.orchestra.run.smithi032.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-30T08:22:01.327 INFO:teuthology.orchestra.run.smithi032.stdout:ceph: stdout ceph version 19.0.0-3348-g8feaf7d9 (8feaf7d9864e980041b042fd5d964ac18adfd20f) squid (dev) 2024-04-30T08:22:01.327 INFO:teuthology.orchestra.run.smithi032.stdout:Ceph version: ceph version 19.0.0-3348-g8feaf7d9 (8feaf7d9864e980041b042fd5d964ac18adfd20f) squid (dev) 2024-04-30T08:22:01.327 INFO:teuthology.orchestra.run.smithi032.stdout:Extracting ceph user uid/gid from container image... 2024-04-30T08:22:02.436 INFO:teuthology.orchestra.run.smithi032.stdout:stat: stdout 167 167 2024-04-30T08:22:02.436 INFO:teuthology.orchestra.run.smithi032.stdout:Creating initial keys... 2024-04-30T08:22:03.679 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-authtool: stdout AQArqjBmW+IYARAA/d71/2whmHUnT+xbaq+fzQ== 2024-04-30T08:22:04.861 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-authtool: stdout AQAsqjBmAJ2nDRAAZ6xP0wuqbf5N5UlFZw4TaQ== 2024-04-30T08:22:06.162 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph-authtool: stdout AQAtqjBmwrWyHxAAtGBxn+BQuPrqoPT7UXowTQ== 2024-04-30T08:22:06.164 INFO:teuthology.orchestra.run.smithi032.stdout:Creating initial monmap... 2024-04-30T08:22:07.470 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-30T08:22:07.471 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-30T08:22:07.471 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:22:07.471 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-30T08:22:07.471 INFO:teuthology.orchestra.run.smithi032.stdout:monmaptool for a [v2:172.21.15.32:3300,v1:172.21.15.32:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-30T08:22:07.471 INFO:teuthology.orchestra.run.smithi032.stdout:setting min_mon_release = quincy 2024-04-30T08:22:07.471 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: set fsid to 98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:22:07.472 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-30T08:22:07.472 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:22:07.473 INFO:teuthology.orchestra.run.smithi032.stdout:Creating mon... 2024-04-30T08:22:08.650 INFO:teuthology.orchestra.run.smithi032.stdout:create mon.a on 2024-04-30T08:22:09.008 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-30T08:22:09.243 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-30T08:22:09.547 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-98c0a670-06ca-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-98c0a670-06ca-11ef-bc93-c7b262605968.target. 2024-04-30T08:22:09.547 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-98c0a670-06ca-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-98c0a670-06ca-11ef-bc93-c7b262605968.target. 2024-04-30T08:22:09.953 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mon.a 2024-04-30T08:22:09.953 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Failed to reset failed state of unit ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-30T08:22:10.257 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-98c0a670-06ca-11ef-bc93-c7b262605968.target.wants/ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-98c0a670-06ca-11ef-bc93-c7b262605968@.service. 2024-04-30T08:22:10.949 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:10 smithi032 podman[27943]: 2024-04-30 08:22:10.717942962 +0000 UTC m=+0.155251866 container create 92fb155cb52d428a4eec4f412b2cd18a352451c043c9c0a8b65d9ad7bcdaf39e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mon-a, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-30T08:22:10.949 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:10 smithi032 podman[27943]: 2024-04-30 08:22:10.946797949 +0000 UTC m=+0.384106860 container init 92fb155cb52d428a4eec4f412b2cd18a352451c043c9c0a8b65d9ad7bcdaf39e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mon-a, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-30T08:22:11.068 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-30T08:22:11.068 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout disabled 2024-04-30T08:22:11.092 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-30T08:22:11.092 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-04-30T08:22:11.092 INFO:teuthology.orchestra.run.smithi032.stdout:firewalld.service is not enabled 2024-04-30T08:22:11.092 INFO:teuthology.orchestra.run.smithi032.stdout:Not possible to enable service . firewalld.service is not available 2024-04-30T08:22:11.093 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mon to start... 2024-04-30T08:22:11.093 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mon... 2024-04-30T08:22:11.246 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:10 smithi032 podman[27943]: 2024-04-30 08:22:10.958261479 +0000 UTC m=+0.395570390 container start 92fb155cb52d428a4eec4f412b2cd18a352451c043c9c0a8b65d9ad7bcdaf39e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True) 2024-04-30T08:22:11.246 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:11 smithi032 bash[27943]: 92fb155cb52d428a4eec4f412b2cd18a352451c043c9c0a8b65d9ad7bcdaf39e 2024-04-30T08:22:11.246 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:11 smithi032 systemd[1]: Started Ceph mon.a for 98c0a670-06ca-11ef-bc93-c7b262605968. 2024-04-30T08:22:11.246 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:11 smithi032 ceph-mon[27961]: mkfs 98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:22:11.246 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:11 smithi032 ceph-mon[27961]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-30T08:22:12.408 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:12 smithi032 ceph-mon[27961]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-30T08:22:12.408 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:12 smithi032 ceph-mon[27961]: monmap epoch 1 2024-04-30T08:22:12.408 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:12 smithi032 ceph-mon[27961]: fsid 98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:22:12.408 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:12 smithi032 ceph-mon[27961]: last_changed 2024-04-30T08:22:06.865404+0000 2024-04-30T08:22:12.408 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:12 smithi032 ceph-mon[27961]: created 2024-04-30T08:22:06.865404+0000 2024-04-30T08:22:12.408 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:12 smithi032 ceph-mon[27961]: min_mon_release 19 (squid) 2024-04-30T08:22:12.408 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:12 smithi032 ceph-mon[27961]: election_strategy: 1 2024-04-30T08:22:12.408 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:12 smithi032 ceph-mon[27961]: 0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-04-30T08:22:12.408 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:12 smithi032 ceph-mon[27961]: fsmap 2024-04-30T08:22:12.408 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:12 smithi032 ceph-mon[27961]: osdmap e1: 0 total, 0 up, 0 in 2024-04-30T08:22:12.409 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:12 smithi032 ceph-mon[27961]: mgrmap e1: no daemons active 2024-04-30T08:22:12.409 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:12 smithi032 ceph-mon[27961]: from='client.? 172.21.15.32:0/1874860907' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-30T08:22:12.663 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout cluster: 2024-04-30T08:22:12.664 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout id: 98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:22:12.664 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-30T08:22:12.664 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-30T08:22:12.664 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout services: 2024-04-30T08:22:12.664 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.781147s) 2024-04-30T08:22:12.664 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-30T08:22:12.664 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-30T08:22:12.664 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-30T08:22:12.664 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout data: 2024-04-30T08:22:12.664 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-30T08:22:12.664 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-30T08:22:12.664 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-30T08:22:12.665 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout pgs: 2024-04-30T08:22:12.670 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-30T08:22:12.670 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.805+0000 7f0e13200700 1 Processor -- start 2024-04-30T08:22:12.670 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.805+0000 7f0e13200700 1 -- start start 2024-04-30T08:22:12.670 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.806+0000 7f0e13200700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e0c07cc40 0x7f0e0c10c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:12.670 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.806+0000 7f0e13200700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0c10cd70 con 0x7f0e0c07cc40 2024-04-30T08:22:12.670 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.807+0000 7f0e0be00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e0c07cc40 0x7f0e0c10c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:22:12.670 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.807+0000 7f0e0be00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e0c07cc40 0x7f0e0c10c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46706/0 (socket says 172.21.15.32:46706) 2024-04-30T08:22:12.670 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.807+0000 7f0e0be00700 1 -- 172.21.15.32:0/2330389891 learned_addr learned my addr 172.21.15.32:0/2330389891 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:22:12.670 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.808+0000 7f0e0be00700 1 -- 172.21.15.32:0/2330389891 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e0c10cef0 con 0x7f0e0c07cc40 2024-04-30T08:22:12.670 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.808+0000 7f0e0be00700 1 --2- 172.21.15.32:0/2330389891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e0c07cc40 0x7f0e0c10c7a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc0098a0 tx=0x7f0dfc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cdb29a4fda9d7f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:22:12.670 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.808+0000 7f0e0b400700 1 -- 172.21.15.32:0/2330389891 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0dfc004160 con 0x7f0e0c07cc40 2024-04-30T08:22:12.670 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.809+0000 7f0e0b400700 1 -- 172.21.15.32:0/2330389891 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dfc004300 con 0x7f0e0c07cc40 2024-04-30T08:22:12.671 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.809+0000 7f0e0b400700 1 -- 172.21.15.32:0/2330389891 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0dfc0036a0 con 0x7f0e0c07cc40 2024-04-30T08:22:12.671 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.809+0000 7f0e13200700 1 -- 172.21.15.32:0/2330389891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e0c07cc40 msgr2=0x7f0e0c10c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:22:12.671 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.809+0000 7f0e13200700 1 --2- 172.21.15.32:0/2330389891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e0c07cc40 0x7f0e0c10c7a0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc0098a0 tx=0x7f0dfc009bb0 comp rx=0 tx=0).stop 2024-04-30T08:22:12.671 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.809+0000 7f0e13200700 1 -- 172.21.15.32:0/2330389891 shutdown_connections 2024-04-30T08:22:12.671 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.809+0000 7f0e13200700 1 --2- 172.21.15.32:0/2330389891 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e0c07cc40 0x7f0e0c10c7a0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:12.671 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.809+0000 7f0e13200700 1 -- 172.21.15.32:0/2330389891 >> 172.21.15.32:0/2330389891 conn(0x7f0e0c104230 msgr2=0x7f0e0c106680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:22:12.671 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.810+0000 7f0e13200700 1 -- 172.21.15.32:0/2330389891 shutdown_connections 2024-04-30T08:22:12.671 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.810+0000 7f0e13200700 1 -- 172.21.15.32:0/2330389891 wait complete. 2024-04-30T08:22:12.671 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.811+0000 7f0e13200700 1 Processor -- start 2024-04-30T08:22:12.671 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.812+0000 7f0e13200700 1 -- start start 2024-04-30T08:22:12.671 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.812+0000 7f0e13200700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e0c19f920 0x7f0e0c19fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:12.671 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.812+0000 7f0e13200700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0e0c10d420 con 0x7f0e0c19f920 2024-04-30T08:22:12.672 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.812+0000 7f0e0be00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e0c19f920 0x7f0e0c19fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:22:12.672 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.812+0000 7f0e0be00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e0c19f920 0x7f0e0c19fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46718/0 (socket says 172.21.15.32:46718) 2024-04-30T08:22:12.672 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.812+0000 7f0e0be00700 1 -- 172.21.15.32:0/1874860907 learned_addr learned my addr 172.21.15.32:0/1874860907 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:22:12.672 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.813+0000 7f0e0be00700 1 -- 172.21.15.32:0/1874860907 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e0c1a02d0 con 0x7f0e0c19f920 2024-04-30T08:22:12.672 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.813+0000 7f0e0be00700 1 --2- 172.21.15.32:0/1874860907 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e0c19f920 0x7f0e0c19fd50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc000c00 tx=0x7f0dfc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:22:12.672 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.813+0000 7f0e09600700 1 -- 172.21.15.32:0/1874860907 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0dfc003ca0 con 0x7f0e0c19f920 2024-04-30T08:22:12.672 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.814+0000 7f0e09600700 1 -- 172.21.15.32:0/1874860907 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0dfc003e40 con 0x7f0e0c19f920 2024-04-30T08:22:12.672 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.814+0000 7f0e09600700 1 -- 172.21.15.32:0/1874860907 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0dfc011780 con 0x7f0e0c19f920 2024-04-30T08:22:12.672 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.814+0000 7f0e13200700 1 -- 172.21.15.32:0/1874860907 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e0c1a0560 con 0x7f0e0c19f920 2024-04-30T08:22:12.672 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.814+0000 7f0e13200700 1 -- 172.21.15.32:0/1874860907 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e0c1a3090 con 0x7f0e0c19f920 2024-04-30T08:22:12.672 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.814+0000 7f0e09600700 1 -- 172.21.15.32:0/1874860907 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f0dfc011920 con 0x7f0e0c19f920 2024-04-30T08:22:12.672 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.814+0000 7f0e09600700 1 -- 172.21.15.32:0/1874860907 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0dfc025d80 con 0x7f0e0c19f920 2024-04-30T08:22:12.672 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.815+0000 7f0e13200700 1 -- 172.21.15.32:0/1874860907 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0e0c064d70 con 0x7f0e0c19f920 2024-04-30T08:22:12.672 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.819+0000 7f0e09600700 1 -- 172.21.15.32:0/1874860907 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f0dfc004260 con 0x7f0e0c19f920 2024-04-30T08:22:12.673 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.900+0000 7f0e13200700 1 -- 172.21.15.32:0/1874860907 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f0e0c064320 con 0x7f0e0c19f920 2024-04-30T08:22:12.673 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.901+0000 7f0e09600700 1 -- 172.21.15.32:0/1874860907 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f0dfc011d50 con 0x7f0e0c19f920 2024-04-30T08:22:12.673 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.903+0000 7f0e13200700 1 -- 172.21.15.32:0/1874860907 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e0c19f920 msgr2=0x7f0e0c19fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:22:12.673 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.903+0000 7f0e13200700 1 --2- 172.21.15.32:0/1874860907 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e0c19f920 0x7f0e0c19fd50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc000c00 tx=0x7f0dfc004750 comp rx=0 tx=0).stop 2024-04-30T08:22:12.673 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.903+0000 7f0e13200700 1 -- 172.21.15.32:0/1874860907 shutdown_connections 2024-04-30T08:22:12.673 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.903+0000 7f0e13200700 1 --2- 172.21.15.32:0/1874860907 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0e0c19f920 0x7f0e0c19fd50 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:12.673 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.903+0000 7f0e13200700 1 -- 172.21.15.32:0/1874860907 >> 172.21.15.32:0/1874860907 conn(0x7f0e0c104230 msgr2=0x7f0e0c106040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:22:12.673 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.903+0000 7f0e13200700 1 -- 172.21.15.32:0/1874860907 shutdown_connections 2024-04-30T08:22:12.673 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:11.903+0000 7f0e13200700 1 -- 172.21.15.32:0/1874860907 wait complete. 2024-04-30T08:22:12.673 INFO:teuthology.orchestra.run.smithi032.stdout:mon is available 2024-04-30T08:22:12.673 INFO:teuthology.orchestra.run.smithi032.stdout:Assimilating anything we can from ceph.conf... 2024-04-30T08:22:13.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:13 smithi032 ceph-mon[27961]: from='client.? 172.21.15.32:0/3382538159' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-30T08:22:13.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:13 smithi032 ceph-mon[27961]: from='client.? 172.21.15.32:0/3382538159' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-30T08:22:14.117 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-30T08:22:14.117 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [global] 2024-04-30T08:22:14.117 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout fsid = 98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:22:14.117 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-30T08:22:14.117 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.32:3300,v1:172.21.15.32:6789] 2024-04-30T08:22:14.117 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-30T08:22:14.117 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-30T08:22:14.117 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-30T08:22:14.117 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-30T08:22:14.117 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-30T08:22:14.117 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-30T08:22:14.123 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-30T08:22:14.123 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-30T08:22:14.124 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [osd] 2024-04-30T08:22:14.124 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2024-04-30T08:22:14.124 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-30T08:22:14.124 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-30T08:22:14.124 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-30T08:22:14.124 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-30T08:22:14.124 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-30T08:22:14.124 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.357+0000 7f94e1c00700 1 Processor -- start 2024-04-30T08:22:14.124 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.357+0000 7f94e1c00700 1 -- start start 2024-04-30T08:22:14.124 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.358+0000 7f94e1c00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94dc07ed10 0x7f94dc07d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:14.124 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.358+0000 7f94e1c00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f94dc07d770 con 0x7f94dc07ed10 2024-04-30T08:22:14.125 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.359+0000 7f94daa00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94dc07ed10 0x7f94dc07d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:22:14.125 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.359+0000 7f94daa00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94dc07ed10 0x7f94dc07d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46732/0 (socket says 172.21.15.32:46732) 2024-04-30T08:22:14.125 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.359+0000 7f94daa00700 1 -- 172.21.15.32:0/3113510915 learned_addr learned my addr 172.21.15.32:0/3113510915 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:22:14.125 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.359+0000 7f94daa00700 1 -- 172.21.15.32:0/3113510915 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94dc07d8f0 con 0x7f94dc07ed10 2024-04-30T08:22:14.125 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.360+0000 7f94daa00700 1 --2- 172.21.15.32:0/3113510915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94dc07ed10 0x7f94dc07d230 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f94cc0098a0 tx=0x7f94cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1833015f189eba2 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:22:14.125 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.360+0000 7f94d9600700 1 -- 172.21.15.32:0/3113510915 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94cc004160 con 0x7f94dc07ed10 2024-04-30T08:22:14.125 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.360+0000 7f94d9600700 1 -- 172.21.15.32:0/3113510915 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94cc004300 con 0x7f94dc07ed10 2024-04-30T08:22:14.125 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.360+0000 7f94d9600700 1 -- 172.21.15.32:0/3113510915 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94cc0036a0 con 0x7f94dc07ed10 2024-04-30T08:22:14.125 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.361+0000 7f94e1c00700 1 -- 172.21.15.32:0/3113510915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94dc07ed10 msgr2=0x7f94dc07d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:22:14.125 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.361+0000 7f94e1c00700 1 --2- 172.21.15.32:0/3113510915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94dc07ed10 0x7f94dc07d230 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f94cc0098a0 tx=0x7f94cc009bb0 comp rx=0 tx=0).stop 2024-04-30T08:22:14.125 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.361+0000 7f94e1c00700 1 -- 172.21.15.32:0/3113510915 shutdown_connections 2024-04-30T08:22:14.125 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.361+0000 7f94e1c00700 1 --2- 172.21.15.32:0/3113510915 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94dc07ed10 0x7f94dc07d230 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:14.126 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.361+0000 7f94e1c00700 1 -- 172.21.15.32:0/3113510915 >> 172.21.15.32:0/3113510915 conn(0x7f94dc105300 msgr2=0x7f94dc1077d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:22:14.126 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.361+0000 7f94e1c00700 1 -- 172.21.15.32:0/3113510915 shutdown_connections 2024-04-30T08:22:14.126 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.361+0000 7f94e1c00700 1 -- 172.21.15.32:0/3113510915 wait complete. 2024-04-30T08:22:14.126 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.363+0000 7f94e1c00700 1 Processor -- start 2024-04-30T08:22:14.126 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.363+0000 7f94e1c00700 1 -- start start 2024-04-30T08:22:14.126 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.363+0000 7f94e1c00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94dc07ed10 0x7f94dc19f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:14.126 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.363+0000 7f94e1c00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f94dc07de20 con 0x7f94dc07ed10 2024-04-30T08:22:14.126 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.364+0000 7f94daa00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94dc07ed10 0x7f94dc19f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:22:14.126 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.364+0000 7f94daa00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94dc07ed10 0x7f94dc19f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46744/0 (socket says 172.21.15.32:46744) 2024-04-30T08:22:14.126 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.364+0000 7f94daa00700 1 -- 172.21.15.32:0/3382538159 learned_addr learned my addr 172.21.15.32:0/3382538159 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:22:14.127 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.364+0000 7f94daa00700 1 -- 172.21.15.32:0/3382538159 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94dc19fec0 con 0x7f94dc07ed10 2024-04-30T08:22:14.127 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.365+0000 7f94daa00700 1 --2- 172.21.15.32:0/3382538159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94dc07ed10 0x7f94dc19f980 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f94cc000c00 tx=0x7f94cc004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:22:14.127 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.365+0000 7f94d3400700 1 -- 172.21.15.32:0/3382538159 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94cc003db0 con 0x7f94dc07ed10 2024-04-30T08:22:14.127 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.365+0000 7f94d3400700 1 -- 172.21.15.32:0/3382538159 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f94cc011630 con 0x7f94dc07ed10 2024-04-30T08:22:14.127 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.365+0000 7f94d3400700 1 -- 172.21.15.32:0/3382538159 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f94cc011950 con 0x7f94dc07ed10 2024-04-30T08:22:14.127 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.365+0000 7f94e1c00700 1 -- 172.21.15.32:0/3382538159 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94dc1a0150 con 0x7f94dc07ed10 2024-04-30T08:22:14.127 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.365+0000 7f94e1c00700 1 -- 172.21.15.32:0/3382538159 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94dc1a2d60 con 0x7f94dc07ed10 2024-04-30T08:22:14.127 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.365+0000 7f94d3400700 1 -- 172.21.15.32:0/3382538159 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f94cc011af0 con 0x7f94dc07ed10 2024-04-30T08:22:14.127 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.365+0000 7f94d3400700 1 -- 172.21.15.32:0/3382538159 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f94cc01c450 con 0x7f94dc07ed10 2024-04-30T08:22:14.127 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.366+0000 7f94e1c00700 1 -- 172.21.15.32:0/3382538159 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94dc064d70 con 0x7f94dc07ed10 2024-04-30T08:22:14.127 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.370+0000 7f94d3400700 1 -- 172.21.15.32:0/3382538159 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f94cc032020 con 0x7f94dc07ed10 2024-04-30T08:22:14.127 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.447+0000 7f94e1c00700 1 -- 172.21.15.32:0/3382538159 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f94dc02e890 con 0x7f94dc07ed10 2024-04-30T08:22:14.127 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.455+0000 7f94d3400700 1 -- 172.21.15.32:0/3382538159 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+490 (secure 0 0 0) 0x7f94cc004260 con 0x7f94dc07ed10 2024-04-30T08:22:14.128 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.457+0000 7f94e1c00700 1 -- 172.21.15.32:0/3382538159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94dc07ed10 msgr2=0x7f94dc19f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:22:14.128 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.457+0000 7f94e1c00700 1 --2- 172.21.15.32:0/3382538159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94dc07ed10 0x7f94dc19f980 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f94cc000c00 tx=0x7f94cc004750 comp rx=0 tx=0).stop 2024-04-30T08:22:14.128 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.457+0000 7f94e1c00700 1 -- 172.21.15.32:0/3382538159 shutdown_connections 2024-04-30T08:22:14.128 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.457+0000 7f94e1c00700 1 --2- 172.21.15.32:0/3382538159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94dc07ed10 0x7f94dc19f980 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:14.128 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.457+0000 7f94e1c00700 1 -- 172.21.15.32:0/3382538159 >> 172.21.15.32:0/3382538159 conn(0x7f94dc105300 msgr2=0x7f94dc106100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:22:14.128 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.457+0000 7f94e1c00700 1 -- 172.21.15.32:0/3382538159 shutdown_connections 2024-04-30T08:22:14.128 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:13.457+0000 7f94e1c00700 1 -- 172.21.15.32:0/3382538159 wait complete. 2024-04-30T08:22:14.128 INFO:teuthology.orchestra.run.smithi032.stdout:Generating new minimal ceph.conf... 2024-04-30T08:22:15.176 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:14 smithi032 ceph-mon[27961]: from='client.? 172.21.15.32:0/2935342602' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:22:15.544 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.780+0000 7f824a800700 1 Processor -- start 2024-04-30T08:22:15.544 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.781+0000 7f824a800700 1 -- start start 2024-04-30T08:22:15.544 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.781+0000 7f824a800700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8244109cb0 0x7f824410a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:15.544 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.781+0000 7f824a800700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f824410a6b0 con 0x7f8244109cb0 2024-04-30T08:22:15.544 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.782+0000 7f8243400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8244109cb0 0x7f824410a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:22:15.544 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.782+0000 7f8243400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8244109cb0 0x7f824410a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46754/0 (socket says 172.21.15.32:46754) 2024-04-30T08:22:15.544 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.782+0000 7f8243400700 1 -- 172.21.15.32:0/2043395420 learned_addr learned my addr 172.21.15.32:0/2043395420 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:22:15.545 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.782+0000 7f8243400700 1 -- 172.21.15.32:0/2043395420 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f824410ade0 con 0x7f8244109cb0 2024-04-30T08:22:15.545 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.783+0000 7f8243400700 1 --2- 172.21.15.32:0/2043395420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8244109cb0 0x7f824410a0c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f8234009e80 tx=0x7f823400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f67e1917474e1645 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:22:15.545 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.783+0000 7f8242000700 1 -- 172.21.15.32:0/2043395420 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8234017070 con 0x7f8244109cb0 2024-04-30T08:22:15.545 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.783+0000 7f8242000700 1 -- 172.21.15.32:0/2043395420 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f823400b8c0 con 0x7f8244109cb0 2024-04-30T08:22:15.545 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.783+0000 7f8242000700 1 -- 172.21.15.32:0/2043395420 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8234004490 con 0x7f8244109cb0 2024-04-30T08:22:15.545 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.784+0000 7f824a800700 1 -- 172.21.15.32:0/2043395420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8244109cb0 msgr2=0x7f824410a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:22:15.545 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.784+0000 7f824a800700 1 --2- 172.21.15.32:0/2043395420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8244109cb0 0x7f824410a0c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f8234009e80 tx=0x7f823400b330 comp rx=0 tx=0).stop 2024-04-30T08:22:15.545 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.784+0000 7f824a800700 1 -- 172.21.15.32:0/2043395420 shutdown_connections 2024-04-30T08:22:15.545 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.784+0000 7f824a800700 1 --2- 172.21.15.32:0/2043395420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8244109cb0 0x7f824410a0c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:15.545 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.784+0000 7f824a800700 1 -- 172.21.15.32:0/2043395420 >> 172.21.15.32:0/2043395420 conn(0x7f8244105350 msgr2=0x7f82441077a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:22:15.545 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.784+0000 7f824a800700 1 -- 172.21.15.32:0/2043395420 shutdown_connections 2024-04-30T08:22:15.545 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.784+0000 7f824a800700 1 -- 172.21.15.32:0/2043395420 wait complete. 2024-04-30T08:22:15.546 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.785+0000 7f824a800700 1 Processor -- start 2024-04-30T08:22:15.546 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.785+0000 7f824a800700 1 -- start start 2024-04-30T08:22:15.546 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.786+0000 7f824a800700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8244109cb0 0x7f824419dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:15.546 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.786+0000 7f824a800700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f824410e010 con 0x7f8244109cb0 2024-04-30T08:22:15.546 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.786+0000 7f8243400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8244109cb0 0x7f824419dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:22:15.546 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.786+0000 7f8243400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8244109cb0 0x7f824419dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46768/0 (socket says 172.21.15.32:46768) 2024-04-30T08:22:15.546 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.786+0000 7f8243400700 1 -- 172.21.15.32:0/2935342602 learned_addr learned my addr 172.21.15.32:0/2935342602 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:22:15.546 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.786+0000 7f8243400700 1 -- 172.21.15.32:0/2935342602 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f824419e520 con 0x7f8244109cb0 2024-04-30T08:22:15.546 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.787+0000 7f8243400700 1 --2- 172.21.15.32:0/2935342602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8244109cb0 0x7f824419dfe0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8234009fd0 tx=0x7f823400bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:22:15.546 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.787+0000 7f823be00700 1 -- 172.21.15.32:0/2935342602 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8234020020 con 0x7f8244109cb0 2024-04-30T08:22:15.546 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.787+0000 7f823be00700 1 -- 172.21.15.32:0/2935342602 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f8234012430 con 0x7f8244109cb0 2024-04-30T08:22:15.546 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.787+0000 7f823be00700 1 -- 172.21.15.32:0/2935342602 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8234017040 con 0x7f8244109cb0 2024-04-30T08:22:15.546 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.787+0000 7f824a800700 1 -- 172.21.15.32:0/2935342602 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f824419e7b0 con 0x7f8244109cb0 2024-04-30T08:22:15.547 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.787+0000 7f824a800700 1 -- 172.21.15.32:0/2935342602 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82441a13c0 con 0x7f8244109cb0 2024-04-30T08:22:15.547 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.788+0000 7f823be00700 1 -- 172.21.15.32:0/2935342602 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8234023020 con 0x7f8244109cb0 2024-04-30T08:22:15.547 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.788+0000 7f823be00700 1 -- 172.21.15.32:0/2935342602 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f823400f050 con 0x7f8244109cb0 2024-04-30T08:22:15.547 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.789+0000 7f824a800700 1 -- 172.21.15.32:0/2935342602 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8244064d70 con 0x7f8244109cb0 2024-04-30T08:22:15.547 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.792+0000 7f823be00700 1 -- 172.21.15.32:0/2935342602 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f823402a020 con 0x7f8244109cb0 2024-04-30T08:22:15.547 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.870+0000 7f824a800700 1 -- 172.21.15.32:0/2935342602 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f82441a1590 con 0x7f8244109cb0 2024-04-30T08:22:15.547 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.871+0000 7f823be00700 1 -- 172.21.15.32:0/2935342602 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7f823400b860 con 0x7f8244109cb0 2024-04-30T08:22:15.547 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.872+0000 7f824a800700 1 -- 172.21.15.32:0/2935342602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8244109cb0 msgr2=0x7f824419dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:22:15.547 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.872+0000 7f824a800700 1 --2- 172.21.15.32:0/2935342602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8244109cb0 0x7f824419dfe0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f8234009fd0 tx=0x7f823400bd50 comp rx=0 tx=0).stop 2024-04-30T08:22:15.547 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.872+0000 7f824a800700 1 -- 172.21.15.32:0/2935342602 shutdown_connections 2024-04-30T08:22:15.547 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.872+0000 7f824a800700 1 --2- 172.21.15.32:0/2935342602 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8244109cb0 0x7f824419dfe0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:15.547 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.872+0000 7f824a800700 1 -- 172.21.15.32:0/2935342602 >> 172.21.15.32:0/2935342602 conn(0x7f8244105350 msgr2=0x7f8244105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:22:15.547 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.872+0000 7f824a800700 1 -- 172.21.15.32:0/2935342602 shutdown_connections 2024-04-30T08:22:15.547 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:14.872+0000 7f824a800700 1 -- 172.21.15.32:0/2935342602 wait complete. 2024-04-30T08:22:15.548 INFO:teuthology.orchestra.run.smithi032.stdout:Restarting the monitor... 2024-04-30T08:22:15.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:15 smithi032 systemd[1]: Stopping Ceph mon.a for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:22:16.334 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:16 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mon-a[27957]: 2024-04-30T08:22:16.064+0000 7f5eeca00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-30T08:22:16.402 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:16 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mon-a[27957]: 2024-04-30T08:22:16.064+0000 7f5eeca00700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-30T08:22:16.612 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:16 smithi032 podman[28207]: 2024-04-30 08:22:16.334120203 +0000 UTC m=+0.441421410 container died 92fb155cb52d428a4eec4f412b2cd18a352451c043c9c0a8b65d9ad7bcdaf39e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-30T08:22:16.910 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:16 smithi032 podman[28207]: 2024-04-30 08:22:16.724464597 +0000 UTC m=+0.831765826 container cleanup 92fb155cb52d428a4eec4f412b2cd18a352451c043c9c0a8b65d9ad7bcdaf39e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mon-a, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d) 2024-04-30T08:22:16.910 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:16 smithi032 bash[28207]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mon-a 2024-04-30T08:22:17.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:16 smithi032 podman[28221]: 2024-04-30 08:22:16.9082348 +0000 UTC m=+0.559532088 container remove 92fb155cb52d428a4eec4f412b2cd18a352451c043c9c0a8b65d9ad7bcdaf39e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mon-a, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-30T08:22:18.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:17 smithi032 systemd[1]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-30T08:22:18.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:17 smithi032 systemd[1]: Stopped Ceph mon.a for 98c0a670-06ca-11ef-bc93-c7b262605968. 2024-04-30T08:22:18.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:17 smithi032 systemd[1]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mon.a.service: Consumed 2.075s CPU time. 2024-04-30T08:22:18.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:17 smithi032 systemd[1]: Starting Ceph mon.a for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:22:18.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:18 smithi032 podman[28313]: 2024-04-30 08:22:18.211297899 +0000 UTC m=+0.034176911 image pull 16056ace0edc7f73516cf28297f1db04dda872801fb5b17b9157b0d1ccf38f9b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f 2024-04-30T08:22:22.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:22 smithi032 podman[28313]: 2024-04-30 08:22:22.596725447 +0000 UTC m=+4.419604466 container create 9a3de84cb1e909976d461d7e88cd0ad533211e8248979910e513c4fd2f0e933f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, io.buildah.version=1.33.5, org.label-schema.build-date=20240430) 2024-04-30T08:22:26.201 INFO:teuthology.orchestra.run.smithi032.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-30T08:22:26.358 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 podman[28313]: 2024-04-30 08:22:26.094364673 +0000 UTC m=+7.917243716 container init 9a3de84cb1e909976d461d7e88cd0ad533211e8248979910e513c4fd2f0e933f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mon-a, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-30T08:22:26.358 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 podman[28313]: 2024-04-30 08:22:26.105624263 +0000 UTC m=+7.928503306 container start 9a3de84cb1e909976d461d7e88cd0ad533211e8248979910e513c4fd2f0e933f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d) 2024-04-30T08:22:26.358 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 bash[28313]: 9a3de84cb1e909976d461d7e88cd0ad533211e8248979910e513c4fd2f0e933f 2024-04-30T08:22:26.358 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 systemd[1]: Started Ceph mon.a for 98c0a670-06ca-11ef-bc93-c7b262605968. 2024-04-30T08:22:26.358 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: set uid:gid to 167:167 (ceph:ceph) 2024-04-30T08:22:26.358 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: ceph version 19.0.0-3348-g8feaf7d9 (8feaf7d9864e980041b042fd5d964ac18adfd20f) squid (dev), process ceph-mon, pid 2 2024-04-30T08:22:26.358 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: pidfile_write: ignore empty --pid-file 2024-04-30T08:22:26.358 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: load: jerasure load: lrc 2024-04-30T08:22:26.358 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: RocksDB version: 7.9.2 2024-04-30T08:22:26.358 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Git sha 0 2024-04-30T08:22:26.359 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Compile date 2024-04-29 22:33:18 2024-04-30T08:22:26.359 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: DB SUMMARY 2024-04-30T08:22:26.359 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: DB Session ID: 4DVLRH1H9W6XROUBRTJM 2024-04-30T08:22:26.359 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: CURRENT file: CURRENT 2024-04-30T08:22:26.359 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: IDENTITY file: IDENTITY 2024-04-30T08:22:26.359 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-30T08:22:26.359 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-30T08:22:26.359 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 105130 ; 2024-04-30T08:22:26.359 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.error_if_exists: 0 2024-04-30T08:22:26.359 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.create_if_missing: 0 2024-04-30T08:22:26.359 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.paranoid_checks: 1 2024-04-30T08:22:26.359 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-30T08:22:26.359 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-30T08:22:26.360 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-30T08:22:26.360 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.env: 0x56368aae4500 2024-04-30T08:22:26.360 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.fs: PosixFileSystem 2024-04-30T08:22:26.360 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.info_log: 0x5636b9bb0040 2024-04-30T08:22:26.360 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.max_file_opening_threads: 16 2024-04-30T08:22:26.360 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.statistics: (nil) 2024-04-30T08:22:26.360 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.use_fsync: 0 2024-04-30T08:22:26.361 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.max_log_file_size: 0 2024-04-30T08:22:26.361 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-30T08:22:26.361 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-30T08:22:26.361 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.keep_log_file_num: 1000 2024-04-30T08:22:26.361 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.recycle_log_file_num: 0 2024-04-30T08:22:26.361 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.allow_fallocate: 1 2024-04-30T08:22:26.361 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.allow_mmap_reads: 0 2024-04-30T08:22:26.361 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.allow_mmap_writes: 0 2024-04-30T08:22:26.361 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.use_direct_reads: 0 2024-04-30T08:22:26.361 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-30T08:22:26.361 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.create_missing_column_families: 0 2024-04-30T08:22:26.361 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.db_log_dir: 2024-04-30T08:22:26.361 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.wal_dir: 2024-04-30T08:22:26.362 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-30T08:22:26.362 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-30T08:22:26.362 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-30T08:22:26.362 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-30T08:22:26.362 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-30T08:22:26.362 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-30T08:22:26.362 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.advise_random_on_open: 1 2024-04-30T08:22:26.362 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.db_write_buffer_size: 0 2024-04-30T08:22:26.362 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.write_buffer_manager: 0x5636b9a7a8c0 2024-04-30T08:22:26.362 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-30T08:22:26.362 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-30T08:22:26.362 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-30T08:22:26.363 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.rate_limiter: (nil) 2024-04-30T08:22:26.363 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-30T08:22:26.363 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.wal_recovery_mode: 2 2024-04-30T08:22:26.363 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.enable_thread_tracking: 0 2024-04-30T08:22:26.363 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.enable_pipelined_write: 0 2024-04-30T08:22:26.363 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.unordered_write: 0 2024-04-30T08:22:26.363 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-30T08:22:26.363 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-30T08:22:26.363 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-30T08:22:26.363 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-30T08:22:26.364 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.row_cache: None 2024-04-30T08:22:26.364 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.wal_filter: None 2024-04-30T08:22:26.364 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-30T08:22:26.364 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.allow_ingest_behind: 0 2024-04-30T08:22:26.364 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.two_write_queues: 0 2024-04-30T08:22:26.364 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.manual_wal_flush: 0 2024-04-30T08:22:26.364 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.wal_compression: 0 2024-04-30T08:22:26.364 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.atomic_flush: 0 2024-04-30T08:22:26.364 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-30T08:22:26.364 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-30T08:22:26.365 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-30T08:22:26.365 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.log_readahead_size: 0 2024-04-30T08:22:26.365 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-30T08:22:26.365 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.best_efforts_recovery: 0 2024-04-30T08:22:26.365 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-30T08:22:26.365 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-30T08:22:26.365 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.allow_data_in_errors: 0 2024-04-30T08:22:26.365 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.db_host_id: __hostname__ 2024-04-30T08:22:26.365 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-30T08:22:26.365 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.max_background_jobs: 2 2024-04-30T08:22:26.365 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.max_background_compactions: -1 2024-04-30T08:22:26.365 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.max_subcompactions: 1 2024-04-30T08:22:26.366 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-30T08:22:26.366 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-30T08:22:26.366 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-30T08:22:26.366 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.max_total_wal_size: 0 2024-04-30T08:22:26.366 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-30T08:22:26.366 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-30T08:22:26.366 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-30T08:22:26.366 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-30T08:22:26.366 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.max_open_files: -1 2024-04-30T08:22:26.366 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.bytes_per_sync: 0 2024-04-30T08:22:26.366 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-30T08:22:26.366 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-30T08:22:26.367 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.compaction_readahead_size: 0 2024-04-30T08:22:26.367 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.max_background_flushes: -1 2024-04-30T08:22:26.367 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Compression algorithms supported: 2024-04-30T08:22:26.367 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-30T08:22:26.367 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: kZSTD supported: 0 2024-04-30T08:22:26.367 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: kXpressCompression supported: 0 2024-04-30T08:22:26.367 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: kLZ4HCCompression supported: 1 2024-04-30T08:22:26.367 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: kZlibCompression supported: 1 2024-04-30T08:22:26.367 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: kSnappyCompression supported: 1 2024-04-30T08:22:26.367 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: kLZ4Compression supported: 1 2024-04-30T08:22:26.367 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: kBZip2Compression supported: 0 2024-04-30T08:22:26.367 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-30T08:22:26.367 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-30T08:22:26.368 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-30T08:22:26.368 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-30T08:22:26.368 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-30T08:22:26.368 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.merge_operator: 2024-04-30T08:22:26.368 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.compaction_filter: None 2024-04-30T08:22:26.368 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.compaction_filter_factory: None 2024-04-30T08:22:26.368 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.sst_partitioner_factory: None 2024-04-30T08:22:26.368 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-30T08:22:26.368 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-30T08:22:26.368 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5636b9bb0200) 2024-04-30T08:22:26.368 INFO:journalctl@ceph.mon.a.smithi032.stdout: cache_index_and_filter_blocks: 1 2024-04-30T08:22:26.368 INFO:journalctl@ceph.mon.a.smithi032.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-30T08:22:26.368 INFO:journalctl@ceph.mon.a.smithi032.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-30T08:22:26.369 INFO:journalctl@ceph.mon.a.smithi032.stdout: pin_top_level_index_and_filter: 1 2024-04-30T08:22:26.369 INFO:journalctl@ceph.mon.a.smithi032.stdout: index_type: 0 2024-04-30T08:22:26.369 INFO:journalctl@ceph.mon.a.smithi032.stdout: data_block_index_type: 0 2024-04-30T08:22:26.369 INFO:journalctl@ceph.mon.a.smithi032.stdout: index_shortening: 1 2024-04-30T08:22:26.369 INFO:journalctl@ceph.mon.a.smithi032.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-30T08:22:26.369 INFO:journalctl@ceph.mon.a.smithi032.stdout: checksum: 4 2024-04-30T08:22:26.369 INFO:journalctl@ceph.mon.a.smithi032.stdout: no_block_cache: 0 2024-04-30T08:22:26.369 INFO:journalctl@ceph.mon.a.smithi032.stdout: block_cache: 0x5636b9aad1f0 2024-04-30T08:22:26.369 INFO:journalctl@ceph.mon.a.smithi032.stdout: block_cache_name: BinnedLRUCache 2024-04-30T08:22:26.369 INFO:journalctl@ceph.mon.a.smithi032.stdout: block_cache_options: 2024-04-30T08:22:26.369 INFO:journalctl@ceph.mon.a.smithi032.stdout: capacity : 536870912 2024-04-30T08:22:26.369 INFO:journalctl@ceph.mon.a.smithi032.stdout: num_shard_bits : 4 2024-04-30T08:22:26.369 INFO:journalctl@ceph.mon.a.smithi032.stdout: strict_capacity_limit : 0 2024-04-30T08:22:26.369 INFO:journalctl@ceph.mon.a.smithi032.stdout: high_pri_pool_ratio: 0.000 2024-04-30T08:22:26.369 INFO:journalctl@ceph.mon.a.smithi032.stdout: block_cache_compressed: (nil) 2024-04-30T08:22:26.370 INFO:journalctl@ceph.mon.a.smithi032.stdout: persistent_cache: (nil) 2024-04-30T08:22:26.370 INFO:journalctl@ceph.mon.a.smithi032.stdout: block_size: 4096 2024-04-30T08:22:26.370 INFO:journalctl@ceph.mon.a.smithi032.stdout: block_size_deviation: 10 2024-04-30T08:22:26.370 INFO:journalctl@ceph.mon.a.smithi032.stdout: block_restart_interval: 16 2024-04-30T08:22:26.370 INFO:journalctl@ceph.mon.a.smithi032.stdout: index_block_restart_interval: 1 2024-04-30T08:22:26.370 INFO:journalctl@ceph.mon.a.smithi032.stdout: metadata_block_size: 4096 2024-04-30T08:22:26.370 INFO:journalctl@ceph.mon.a.smithi032.stdout: partition_filters: 0 2024-04-30T08:22:26.370 INFO:journalctl@ceph.mon.a.smithi032.stdout: use_delta_encoding: 1 2024-04-30T08:22:26.370 INFO:journalctl@ceph.mon.a.smithi032.stdout: filter_policy: bloomfilter 2024-04-30T08:22:26.370 INFO:journalctl@ceph.mon.a.smithi032.stdout: whole_key_filtering: 1 2024-04-30T08:22:26.370 INFO:journalctl@ceph.mon.a.smithi032.stdout: verify_compression: 0 2024-04-30T08:22:26.370 INFO:journalctl@ceph.mon.a.smithi032.stdout: read_amp_bytes_per_bit: 0 2024-04-30T08:22:26.370 INFO:journalctl@ceph.mon.a.smithi032.stdout: format_version: 5 2024-04-30T08:22:26.371 INFO:journalctl@ceph.mon.a.smithi032.stdout: enable_index_compression: 1 2024-04-30T08:22:26.371 INFO:journalctl@ceph.mon.a.smithi032.stdout: block_align: 0 2024-04-30T08:22:26.371 INFO:journalctl@ceph.mon.a.smithi032.stdout: max_auto_readahead_size: 262144 2024-04-30T08:22:26.372 INFO:journalctl@ceph.mon.a.smithi032.stdout: prepopulate_block_cache: 0 2024-04-30T08:22:26.372 INFO:journalctl@ceph.mon.a.smithi032.stdout: initial_auto_readahead_size: 8192 2024-04-30T08:22:26.372 INFO:journalctl@ceph.mon.a.smithi032.stdout: num_file_reads_for_auto_readahead: 2 2024-04-30T08:22:26.372 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.write_buffer_size: 33554432 2024-04-30T08:22:26.372 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.max_write_buffer_number: 2 2024-04-30T08:22:26.372 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.compression: NoCompression 2024-04-30T08:22:26.372 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.bottommost_compression: Disabled 2024-04-30T08:22:26.373 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.prefix_extractor: nullptr 2024-04-30T08:22:26.373 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-30T08:22:26.373 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.num_levels: 7 2024-04-30T08:22:26.373 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-30T08:22:26.373 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-30T08:22:26.373 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-30T08:22:26.373 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-30T08:22:26.373 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-30T08:22:26.373 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-30T08:22:26.373 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-30T08:22:26.373 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-30T08:22:26.373 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-30T08:22:26.374 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-30T08:22:26.374 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-30T08:22:26.374 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-30T08:22:26.374 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-30T08:22:26.374 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.compression_opts.level: 32767 2024-04-30T08:22:26.374 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.compression_opts.strategy: 0 2024-04-30T08:22:26.374 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-30T08:22:26.374 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-30T08:22:26.374 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-30T08:22:26.374 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-30T08:22:26.375 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.compression_opts.enabled: false 2024-04-30T08:22:26.375 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-30T08:22:26.375 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-30T08:22:26.375 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-30T08:22:26.375 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-30T08:22:26.375 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.target_file_size_base: 67108864 2024-04-30T08:22:26.375 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-30T08:22:26.375 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-30T08:22:26.375 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-30T08:22:26.375 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-30T08:22:26.375 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-30T08:22:26.376 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-30T08:22:26.376 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-30T08:22:26.376 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-30T08:22:26.376 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-30T08:22:26.376 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-30T08:22:26.376 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-30T08:22:26.376 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-30T08:22:26.376 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-30T08:22:26.376 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-30T08:22:26.376 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.arena_block_size: 1048576 2024-04-30T08:22:26.377 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-30T08:22:26.377 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-30T08:22:26.377 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.disable_auto_compactions: 0 2024-04-30T08:22:26.377 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-30T08:22:26.377 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-30T08:22:26.377 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-30T08:22:26.377 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-30T08:22:26.377 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-30T08:22:26.377 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-30T08:22:26.377 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-30T08:22:26.377 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-30T08:22:26.377 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-30T08:22:26.378 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-30T08:22:26.378 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-30T08:22:26.378 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.inplace_update_support: 0 2024-04-30T08:22:26.378 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-30T08:22:26.378 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-30T08:22:26.378 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-30T08:22:26.378 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-30T08:22:26.378 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.bloom_locality: 0 2024-04-30T08:22:26.378 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.max_successive_merges: 0 2024-04-30T08:22:26.378 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-30T08:22:26.378 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.paranoid_file_checks: 0 2024-04-30T08:22:26.379 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.force_consistency_checks: 1 2024-04-30T08:22:26.379 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.report_bg_io_stats: 0 2024-04-30T08:22:26.379 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.ttl: 2592000 2024-04-30T08:22:26.379 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-30T08:22:26.379 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-30T08:22:26.379 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-30T08:22:26.379 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.enable_blob_files: false 2024-04-30T08:22:26.379 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.min_blob_size: 0 2024-04-30T08:22:26.379 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.blob_file_size: 268435456 2024-04-30T08:22:26.380 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-30T08:22:26.380 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-30T08:22:26.380 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-30T08:22:26.380 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-30T08:22:26.380 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-30T08:22:26.380 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.blob_file_starting_level: 0 2024-04-30T08:22:26.380 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-30T08:22:26.380 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-30T08:22:26.381 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-30T08:22:26.381 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 82aaa059-e88a-44f3-84ab-22e6183e9c90 2024-04-30T08:22:26.381 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714465346245161, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-30T08:22:26.381 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-30T08:22:26.381 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714465346247426, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 100309, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 326, "table_properties": {"data_size": 98198, "index_size": 303, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14175, "raw_average_key_size": 49, "raw_value_size": 90460, "raw_average_value_size": 317, "num_data_blocks": 12, "num_entries": 285, "num_filter_entries": 285, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714465346, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "82aaa059-e88a-44f3-84ab-22e6183e9c90", "db_session_id": "4DVLRH1H9W6XROUBRTJM", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-30T08:22:26.381 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714465346247589, "job": 1, "event": "recovery_finished"} 2024-04-30T08:22:26.381 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-30T08:22:26.381 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-30T08:22:26.381 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5636b9a91c00 2024-04-30T08:22:26.381 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: DB pointer 0x5636be884000 2024-04-30T08:22:26.382 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-30T08:22:26.382 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-30T08:22:26.382 INFO:journalctl@ceph.mon.a.smithi032.stdout: ** DB Stats ** 2024-04-30T08:22:26.382 INFO:journalctl@ceph.mon.a.smithi032.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-30T08:22:26.382 INFO:journalctl@ceph.mon.a.smithi032.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-30T08:22:26.382 INFO:journalctl@ceph.mon.a.smithi032.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-30T08:22:26.382 INFO:journalctl@ceph.mon.a.smithi032.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-30T08:22:26.382 INFO:journalctl@ceph.mon.a.smithi032.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-30T08:22:26.382 INFO:journalctl@ceph.mon.a.smithi032.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-30T08:22:26.382 INFO:journalctl@ceph.mon.a.smithi032.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-30T08:22:26.383 INFO:journalctl@ceph.mon.a.smithi032.stdout: 2024-04-30T08:22:26.383 INFO:journalctl@ceph.mon.a.smithi032.stdout: ** Compaction Stats [default] ** 2024-04-30T08:22:26.383 INFO:journalctl@ceph.mon.a.smithi032.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-30T08:22:26.383 INFO:journalctl@ceph.mon.a.smithi032.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-30T08:22:26.383 INFO:journalctl@ceph.mon.a.smithi032.stdout: L0 2/0 99.82 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 77.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-30T08:22:26.383 INFO:journalctl@ceph.mon.a.smithi032.stdout: Sum 2/0 99.82 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 77.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-30T08:22:26.383 INFO:journalctl@ceph.mon.a.smithi032.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 77.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-30T08:22:26.383 INFO:journalctl@ceph.mon.a.smithi032.stdout: 2024-04-30T08:22:26.384 INFO:journalctl@ceph.mon.a.smithi032.stdout: ** Compaction Stats [default] ** 2024-04-30T08:22:26.384 INFO:journalctl@ceph.mon.a.smithi032.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-30T08:22:26.384 INFO:journalctl@ceph.mon.a.smithi032.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-30T08:22:26.384 INFO:journalctl@ceph.mon.a.smithi032.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 77.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-30T08:22:26.384 INFO:journalctl@ceph.mon.a.smithi032.stdout: 2024-04-30T08:22:26.384 INFO:journalctl@ceph.mon.a.smithi032.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-30T08:22:26.384 INFO:journalctl@ceph.mon.a.smithi032.stdout: 2024-04-30T08:22:26.384 INFO:journalctl@ceph.mon.a.smithi032.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-30T08:22:26.384 INFO:journalctl@ceph.mon.a.smithi032.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-30T08:22:26.384 INFO:journalctl@ceph.mon.a.smithi032.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-30T08:22:26.384 INFO:journalctl@ceph.mon.a.smithi032.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-30T08:22:26.385 INFO:journalctl@ceph.mon.a.smithi032.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-30T08:22:26.385 INFO:journalctl@ceph.mon.a.smithi032.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-30T08:22:26.385 INFO:journalctl@ceph.mon.a.smithi032.stdout: Cumulative compaction: 0.00 GB write, 12.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-30T08:22:26.385 INFO:journalctl@ceph.mon.a.smithi032.stdout: Interval compaction: 0.00 GB write, 12.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-30T08:22:26.385 INFO:journalctl@ceph.mon.a.smithi032.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-30T08:22:26.385 INFO:journalctl@ceph.mon.a.smithi032.stdout: Block cache BinnedLRUCache@0x5636b9aad1f0#2 capacity: 512.00 MB usage: 1.53 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.5e-05 secs_since: 0 2024-04-30T08:22:26.385 INFO:journalctl@ceph.mon.a.smithi032.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.03 KB,0.000196695%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-30T08:22:26.385 INFO:journalctl@ceph.mon.a.smithi032.stdout: 2024-04-30T08:22:26.385 INFO:journalctl@ceph.mon.a.smithi032.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-30T08:22:26.385 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: starting mon.a rank 0 at public addrs [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] at bind addrs [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:22:26.385 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: mon.a@-1(???) e1 preinit fsid 98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:22:26.385 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: mon.a@-1(???).mds e1 new map 2024-04-30T08:22:26.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: mon.a@-1(???).mds e1 print_map 2024-04-30T08:22:26.386 INFO:journalctl@ceph.mon.a.smithi032.stdout: e1 2024-04-30T08:22:26.386 INFO:journalctl@ceph.mon.a.smithi032.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-30T08:22:26.386 INFO:journalctl@ceph.mon.a.smithi032.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-30T08:22:26.386 INFO:journalctl@ceph.mon.a.smithi032.stdout: legacy client fscid: -1 2024-04-30T08:22:26.386 INFO:journalctl@ceph.mon.a.smithi032.stdout: 2024-04-30T08:22:26.386 INFO:journalctl@ceph.mon.a.smithi032.stdout: No filesystems configured 2024-04-30T08:22:26.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-30T08:22:26.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-30T08:22:26.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-30T08:22:26.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-30T08:22:26.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-30T08:22:26.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-30T08:22:26.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: monmap epoch 1 2024-04-30T08:22:26.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: fsid 98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:22:26.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: last_changed 2024-04-30T08:22:06.865404+0000 2024-04-30T08:22:26.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: created 2024-04-30T08:22:06.865404+0000 2024-04-30T08:22:26.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: min_mon_release 19 (squid) 2024-04-30T08:22:26.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: election_strategy: 1 2024-04-30T08:22:26.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: 0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-04-30T08:22:26.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: fsmap 2024-04-30T08:22:26.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: osdmap e1: 0 total, 0 up, 0 in 2024-04-30T08:22:26.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:26 smithi032 ceph-mon[28331]: mgrmap e1: no daemons active 2024-04-30T08:22:27.751 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.958+0000 7f29c7400700 1 Processor -- start 2024-04-30T08:22:27.756 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.958+0000 7f29c7400700 1 -- start start 2024-04-30T08:22:27.757 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.959+0000 7f29c7400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29c0109cb0 0x7f29c010a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:27.757 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.959+0000 7f29c7400700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f29c010a6b0 con 0x7f29c0109cb0 2024-04-30T08:22:27.757 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.960+0000 7f29c4a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29c0109cb0 0x7f29c010a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:22:27.757 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.960+0000 7f29c4a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29c0109cb0 0x7f29c010a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51280/0 (socket says 172.21.15.32:51280) 2024-04-30T08:22:27.757 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.960+0000 7f29c4a00700 1 -- 172.21.15.32:0/1291688741 learned_addr learned my addr 172.21.15.32:0/1291688741 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:22:27.757 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.960+0000 7f29c4a00700 1 -- 172.21.15.32:0/1291688741 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29c010ade0 con 0x7f29c0109cb0 2024-04-30T08:22:27.757 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.961+0000 7f29c4a00700 1 --2- 172.21.15.32:0/1291688741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29c0109cb0 0x7f29c010a0c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f29b40098a0 tx=0x7f29b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99828df92e661fa1 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:22:27.757 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.961+0000 7f29bf400700 1 -- 172.21.15.32:0/1291688741 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f29b4004160 con 0x7f29c0109cb0 2024-04-30T08:22:27.757 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.961+0000 7f29bf400700 1 -- 172.21.15.32:0/1291688741 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f29b4004300 con 0x7f29c0109cb0 2024-04-30T08:22:27.757 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.961+0000 7f29bf400700 1 -- 172.21.15.32:0/1291688741 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f29b4011af0 con 0x7f29c0109cb0 2024-04-30T08:22:27.757 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.962+0000 7f29c7400700 1 -- 172.21.15.32:0/1291688741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29c0109cb0 msgr2=0x7f29c010a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:22:27.757 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.962+0000 7f29c7400700 1 --2- 172.21.15.32:0/1291688741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29c0109cb0 0x7f29c010a0c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f29b40098a0 tx=0x7f29b4009bb0 comp rx=0 tx=0).stop 2024-04-30T08:22:27.758 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.962+0000 7f29c7400700 1 -- 172.21.15.32:0/1291688741 shutdown_connections 2024-04-30T08:22:27.758 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.962+0000 7f29c7400700 1 --2- 172.21.15.32:0/1291688741 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29c0109cb0 0x7f29c010a0c0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:27.758 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.962+0000 7f29c7400700 1 -- 172.21.15.32:0/1291688741 >> 172.21.15.32:0/1291688741 conn(0x7f29c0105350 msgr2=0x7f29c01077a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:22:27.758 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.962+0000 7f29c7400700 1 -- 172.21.15.32:0/1291688741 shutdown_connections 2024-04-30T08:22:27.758 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.963+0000 7f29c7400700 1 -- 172.21.15.32:0/1291688741 wait complete. 2024-04-30T08:22:27.758 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.964+0000 7f29c7400700 1 Processor -- start 2024-04-30T08:22:27.758 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.964+0000 7f29c7400700 1 -- start start 2024-04-30T08:22:27.758 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.964+0000 7f29c7400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29c0109cb0 0x7f29c01a0270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:27.758 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.964+0000 7f29c7400700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f29c010e010 con 0x7f29c0109cb0 2024-04-30T08:22:27.758 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.965+0000 7f29c4a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29c0109cb0 0x7f29c01a0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:22:27.758 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.965+0000 7f29c4a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29c0109cb0 0x7f29c01a0270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51290/0 (socket says 172.21.15.32:51290) 2024-04-30T08:22:27.758 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.965+0000 7f29c4a00700 1 -- 172.21.15.32:0/3882202010 learned_addr learned my addr 172.21.15.32:0/3882202010 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:22:27.758 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.965+0000 7f29c4a00700 1 -- 172.21.15.32:0/3882202010 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29c01a07b0 con 0x7f29c0109cb0 2024-04-30T08:22:27.759 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.965+0000 7f29c4a00700 1 --2- 172.21.15.32:0/3882202010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29c0109cb0 0x7f29c01a0270 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f29b40090e0 tx=0x7f29b4011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:22:27.759 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.966+0000 7f29bd600700 1 -- 172.21.15.32:0/3882202010 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f29b40039e0 con 0x7f29c0109cb0 2024-04-30T08:22:27.759 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.966+0000 7f29c7400700 1 -- 172.21.15.32:0/3882202010 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29c01a0a40 con 0x7f29c0109cb0 2024-04-30T08:22:27.759 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.966+0000 7f29bd600700 1 -- 172.21.15.32:0/3882202010 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f29b4003b80 con 0x7f29c0109cb0 2024-04-30T08:22:27.759 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.966+0000 7f29bd600700 1 -- 172.21.15.32:0/3882202010 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f29b4024690 con 0x7f29c0109cb0 2024-04-30T08:22:27.759 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.966+0000 7f29c7400700 1 -- 172.21.15.32:0/3882202010 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29c01a3650 con 0x7f29c0109cb0 2024-04-30T08:22:27.759 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.966+0000 7f29bd600700 1 -- 172.21.15.32:0/3882202010 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f29b4022020 con 0x7f29c0109cb0 2024-04-30T08:22:27.759 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.966+0000 7f29bd600700 1 -- 172.21.15.32:0/3882202010 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f29b402dc40 con 0x7f29c0109cb0 2024-04-30T08:22:27.759 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.967+0000 7f29c7400700 1 -- 172.21.15.32:0/3882202010 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29c0064d70 con 0x7f29c0109cb0 2024-04-30T08:22:27.759 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:26.971+0000 7f29bd600700 1 -- 172.21.15.32:0/3882202010 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f29b4043e50 con 0x7f29c0109cb0 2024-04-30T08:22:27.759 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:27.050+0000 7f29c7400700 1 -- 172.21.15.32:0/3882202010 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f29c002e8b0 con 0x7f29c0109cb0 2024-04-30T08:22:27.759 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:27.056+0000 7f29bd600700 1 -- 172.21.15.32:0/3882202010 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f29b4032020 con 0x7f29c0109cb0 2024-04-30T08:22:27.759 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:27.057+0000 7f29c7400700 1 -- 172.21.15.32:0/3882202010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29c0109cb0 msgr2=0x7f29c01a0270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:22:27.760 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:27.057+0000 7f29c7400700 1 --2- 172.21.15.32:0/3882202010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29c0109cb0 0x7f29c01a0270 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f29b40090e0 tx=0x7f29b4011940 comp rx=0 tx=0).stop 2024-04-30T08:22:27.760 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:27.057+0000 7f29c7400700 1 -- 172.21.15.32:0/3882202010 shutdown_connections 2024-04-30T08:22:27.760 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:27.057+0000 7f29c7400700 1 --2- 172.21.15.32:0/3882202010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29c0109cb0 0x7f29c01a0270 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:27.760 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:27.057+0000 7f29c7400700 1 -- 172.21.15.32:0/3882202010 >> 172.21.15.32:0/3882202010 conn(0x7f29c0105350 msgr2=0x7f29c007d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:22:27.760 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:27.057+0000 7f29c7400700 1 -- 172.21.15.32:0/3882202010 shutdown_connections 2024-04-30T08:22:27.760 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:27.058+0000 7f29c7400700 1 -- 172.21.15.32:0/3882202010 wait complete. 2024-04-30T08:22:27.760 INFO:teuthology.orchestra.run.smithi032.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-30T08:22:27.774 INFO:teuthology.orchestra.run.smithi032.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-30T08:22:27.774 INFO:teuthology.orchestra.run.smithi032.stdout:Creating mgr... 2024-04-30T08:22:27.774 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-30T08:22:27.775 INFO:teuthology.orchestra.run.smithi032.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-30T08:22:28.198 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mgr.x 2024-04-30T08:22:28.199 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Failed to reset failed state of unit ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-30T08:22:28.316 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:28 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3882202010' entity='client.admin' 2024-04-30T08:22:28.464 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-98c0a670-06ca-11ef-bc93-c7b262605968.target.wants/ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-98c0a670-06ca-11ef-bc93-c7b262605968@.service. 2024-04-30T08:22:28.691 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:28 smithi032 systemd[1]: Starting Ceph mgr.x for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:22:29.192 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:28 smithi032 podman[28556]: 2024-04-30 08:22:28.750502395 +0000 UTC m=+0.035547030 image pull 16056ace0edc7f73516cf28297f1db04dda872801fb5b17b9157b0d1ccf38f9b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f 2024-04-30T08:22:29.571 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:29 smithi032 podman[28556]: 2024-04-30 08:22:29.251450949 +0000 UTC m=+0.536495622 container create 55eff9d8db7ceca692cabe245b81c28a16acf7f28fc49dde0d53b36da014ee1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-30T08:22:29.571 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:29 smithi032 podman[28556]: 2024-04-30 08:22:29.487640704 +0000 UTC m=+0.772685356 container init 55eff9d8db7ceca692cabe245b81c28a16acf7f28fc49dde0d53b36da014ee1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_CLEAN=True) 2024-04-30T08:22:29.571 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:29 smithi032 podman[28556]: 2024-04-30 08:22:29.497819892 +0000 UTC m=+0.782864572 container start 55eff9d8db7ceca692cabe245b81c28a16acf7f28fc49dde0d53b36da014ee1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux ) 2024-04-30T08:22:29.621 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-30T08:22:29.622 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout disabled 2024-04-30T08:22:29.640 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-30T08:22:29.641 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-04-30T08:22:29.641 INFO:teuthology.orchestra.run.smithi032.stdout:firewalld.service is not enabled 2024-04-30T08:22:29.641 INFO:teuthology.orchestra.run.smithi032.stdout:Not possible to enable service . firewalld.service is not available 2024-04-30T08:22:29.659 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-30T08:22:29.659 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout disabled 2024-04-30T08:22:29.676 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-30T08:22:29.676 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-04-30T08:22:29.676 INFO:teuthology.orchestra.run.smithi032.stdout:firewalld.service is not enabled 2024-04-30T08:22:29.676 INFO:teuthology.orchestra.run.smithi032.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-30T08:22:29.676 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mgr to start... 2024-04-30T08:22:29.676 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mgr... 2024-04-30T08:22:29.827 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:29 smithi032 bash[28556]: 55eff9d8db7ceca692cabe245b81c28a16acf7f28fc49dde0d53b36da014ee1a 2024-04-30T08:22:29.828 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:29 smithi032 systemd[1]: Started Ceph mgr.x for 98c0a670-06ca-11ef-bc93-c7b262605968. 2024-04-30T08:22:30.099 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:29 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:29.825+0000 7f95bdadd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-30T08:22:30.370 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:30 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:30.096+0000 7f95bdadd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-30T08:22:30.371 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:30 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:30.240+0000 7f95bdadd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-30T08:22:30.670 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:30 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3466682425' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-30T08:22:30.941 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:30 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:30.668+0000 7f95bdadd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-30T08:22:30.941 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:30 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:30.790+0000 7f95bdadd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-30T08:22:31.215 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-30T08:22:31.215 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-30T08:22:31.215 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsid": "98c0a670-06ca-11ef-bc93-c7b262605968", 2024-04-30T08:22:31.215 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "health": { 2024-04-30T08:22:31.216 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-30T08:22:31.216 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-30T08:22:31.216 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-30T08:22:31.216 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-30T08:22:31.216 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-30T08:22:31.216 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-30T08:22:31.216 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 0 2024-04-30T08:22:31.216 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-30T08:22:31.216 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-30T08:22:31.216 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "a" 2024-04-30T08:22:31.216 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-30T08:22:31.216 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_age": 4, 2024-04-30T08:22:31.216 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-30T08:22:31.217 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-30T08:22:31.217 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-30T08:22:31.217 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-30T08:22:31.217 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-30T08:22:31.217 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-30T08:22:31.217 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-30T08:22:31.217 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-30T08:22:31.217 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-30T08:22:31.217 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-30T08:22:31.225 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-30T08:22:31.226 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-30T08:22:31.226 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-30T08:22:31.226 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-30T08:22:31.226 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-30T08:22:31.226 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-30T08:22:31.226 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-30T08:22:31.226 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-30T08:22:31.226 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-30T08:22:31.226 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-30T08:22:31.226 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-30T08:22:31.226 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-30T08:22:31.226 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-30T08:22:31.226 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-30T08:22:31.227 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-30T08:22:31.227 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-30T08:22:31.227 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-30T08:22:31.227 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-30T08:22:31.227 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-30T08:22:31.227 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-30T08:22:31.227 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-30T08:22:31.227 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-30T08:22:31.227 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-30T08:22:31.227 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-30T08:22:31.227 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-30T08:22:31.227 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "restful" 2024-04-30T08:22:31.227 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-30T08:22:31.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-30T08:22:31.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-30T08:22:31.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-30T08:22:31.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-30T08:22:31.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modified": "2024-04-30T08:22:11.123688+0000", 2024-04-30T08:22:31.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-30T08:22:31.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-30T08:22:31.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-30T08:22:31.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-30T08:22:31.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.368+0000 7f1b69c00700 1 Processor -- start 2024-04-30T08:22:31.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.368+0000 7f1b69c00700 1 -- start start 2024-04-30T08:22:31.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.368+0000 7f1b69c00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b6407ed50 0x7f1b6407d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:31.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.369+0000 7f1b69c00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1b6407d7b0 con 0x7f1b6407ed50 2024-04-30T08:22:31.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.369+0000 7f1b63e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b6407ed50 0x7f1b6407d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:22:31.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.369+0000 7f1b63e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b6407ed50 0x7f1b6407d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51316/0 (socket says 172.21.15.32:51316) 2024-04-30T08:22:31.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.369+0000 7f1b63e00700 1 -- 172.21.15.32:0/3463973617 learned_addr learned my addr 172.21.15.32:0/3463973617 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:22:31.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.370+0000 7f1b63e00700 1 -- 172.21.15.32:0/3463973617 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b6407d930 con 0x7f1b6407ed50 2024-04-30T08:22:31.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.370+0000 7f1b63e00700 1 --2- 172.21.15.32:0/3463973617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b6407ed50 0x7f1b6407d270 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1b540098a0 tx=0x7f1b54009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a404575fa20d948 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:22:31.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.370+0000 7f1b62a00700 1 -- 172.21.15.32:0/3463973617 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b54004160 con 0x7f1b6407ed50 2024-04-30T08:22:31.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.371+0000 7f1b62a00700 1 -- 172.21.15.32:0/3463973617 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1b54004300 con 0x7f1b6407ed50 2024-04-30T08:22:31.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.371+0000 7f1b62a00700 1 -- 172.21.15.32:0/3463973617 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b54011af0 con 0x7f1b6407ed50 2024-04-30T08:22:31.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.371+0000 7f1b69c00700 1 -- 172.21.15.32:0/3463973617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b6407ed50 msgr2=0x7f1b6407d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:22:31.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.371+0000 7f1b69c00700 1 --2- 172.21.15.32:0/3463973617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b6407ed50 0x7f1b6407d270 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1b540098a0 tx=0x7f1b54009bb0 comp rx=0 tx=0).stop 2024-04-30T08:22:31.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.372+0000 7f1b69c00700 1 -- 172.21.15.32:0/3463973617 shutdown_connections 2024-04-30T08:22:31.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.372+0000 7f1b69c00700 1 --2- 172.21.15.32:0/3463973617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b6407ed50 0x7f1b6407d270 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:31.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.372+0000 7f1b69c00700 1 -- 172.21.15.32:0/3463973617 >> 172.21.15.32:0/3463973617 conn(0x7f1b641050d0 msgr2=0x7f1b641075a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:22:31.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.372+0000 7f1b69c00700 1 -- 172.21.15.32:0/3463973617 shutdown_connections 2024-04-30T08:22:31.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.372+0000 7f1b69c00700 1 -- 172.21.15.32:0/3463973617 wait complete. 2024-04-30T08:22:31.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.373+0000 7f1b69c00700 1 Processor -- start 2024-04-30T08:22:31.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.373+0000 7f1b69c00700 1 -- start start 2024-04-30T08:22:31.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.373+0000 7f1b69c00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b6407ed50 0x7f1b641a0020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:31.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.373+0000 7f1b69c00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1b6410fff0 con 0x7f1b6407ed50 2024-04-30T08:22:31.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.374+0000 7f1b63e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b6407ed50 0x7f1b641a0020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:22:31.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.374+0000 7f1b63e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b6407ed50 0x7f1b641a0020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51318/0 (socket says 172.21.15.32:51318) 2024-04-30T08:22:31.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.374+0000 7f1b63e00700 1 -- 172.21.15.32:0/3466682425 learned_addr learned my addr 172.21.15.32:0/3466682425 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:22:31.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.374+0000 7f1b63e00700 1 -- 172.21.15.32:0/3466682425 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b641a0560 con 0x7f1b6407ed50 2024-04-30T08:22:31.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.375+0000 7f1b63e00700 1 --2- 172.21.15.32:0/3466682425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b6407ed50 0x7f1b641a0020 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1b540117b0 tx=0x7f1b54011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:22:31.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.375+0000 7f1b60c00700 1 -- 172.21.15.32:0/3466682425 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b540039c0 con 0x7f1b6407ed50 2024-04-30T08:22:31.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.375+0000 7f1b60c00700 1 -- 172.21.15.32:0/3466682425 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1b54003b60 con 0x7f1b6407ed50 2024-04-30T08:22:31.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.375+0000 7f1b60c00700 1 -- 172.21.15.32:0/3466682425 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b540196a0 con 0x7f1b6407ed50 2024-04-30T08:22:31.230 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.375+0000 7f1b69c00700 1 -- 172.21.15.32:0/3466682425 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b641a07f0 con 0x7f1b6407ed50 2024-04-30T08:22:31.231 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.375+0000 7f1b69c00700 1 -- 172.21.15.32:0/3466682425 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b641a3400 con 0x7f1b6407ed50 2024-04-30T08:22:31.231 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.375+0000 7f1b60c00700 1 -- 172.21.15.32:0/3466682425 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1b54019ca0 con 0x7f1b6407ed50 2024-04-30T08:22:31.231 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.376+0000 7f1b60c00700 1 -- 172.21.15.32:0/3466682425 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f1b54024730 con 0x7f1b6407ed50 2024-04-30T08:22:31.231 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.376+0000 7f1b69c00700 1 -- 172.21.15.32:0/3466682425 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b64064d70 con 0x7f1b6407ed50 2024-04-30T08:22:31.231 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.379+0000 7f1b60c00700 1 -- 172.21.15.32:0/3466682425 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f1b540042a0 con 0x7f1b6407ed50 2024-04-30T08:22:31.231 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.460+0000 7f1b69c00700 1 -- 172.21.15.32:0/3466682425 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f1b641a35d0 con 0x7f1b6407ed50 2024-04-30T08:22:31.231 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.461+0000 7f1b60c00700 1 -- 172.21.15.32:0/3466682425 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f1b5401ab00 con 0x7f1b6407ed50 2024-04-30T08:22:31.234 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.463+0000 7f1b69c00700 1 -- 172.21.15.32:0/3466682425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b6407ed50 msgr2=0x7f1b641a0020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:22:31.234 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.463+0000 7f1b69c00700 1 --2- 172.21.15.32:0/3466682425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b6407ed50 0x7f1b641a0020 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1b540117b0 tx=0x7f1b54011b70 comp rx=0 tx=0).stop 2024-04-30T08:22:31.234 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.463+0000 7f1b69c00700 1 -- 172.21.15.32:0/3466682425 shutdown_connections 2024-04-30T08:22:31.234 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.463+0000 7f1b69c00700 1 --2- 172.21.15.32:0/3466682425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1b6407ed50 0x7f1b641a0020 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:31.234 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.463+0000 7f1b69c00700 1 -- 172.21.15.32:0/3466682425 >> 172.21.15.32:0/3466682425 conn(0x7f1b641050d0 msgr2=0x7f1b64105ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:22:31.234 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.463+0000 7f1b69c00700 1 -- 172.21.15.32:0/3466682425 shutdown_connections 2024-04-30T08:22:31.234 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:30.463+0000 7f1b69c00700 1 -- 172.21.15.32:0/3466682425 wait complete. 2024-04-30T08:22:31.234 INFO:teuthology.orchestra.run.smithi032.stdout:mgr not available, waiting (1/15)... 2024-04-30T08:22:32.691 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:32 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:32.438+0000 7f95bdadd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-30T08:22:32.691 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:32 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:32.675+0000 7f95bdadd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-30T08:22:33.191 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:32 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:32.897+0000 7f95bdadd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-30T08:22:33.586 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:33 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:33.298+0000 7f95bdadd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-30T08:22:33.586 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:33 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:33.426+0000 7f95bdadd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-30T08:22:33.837 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:33 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:33.685+0000 7f95bdadd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-30T08:22:34.190 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:34 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3658632735' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-30T08:22:34.191 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:33 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:33.954+0000 7f95bdadd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-30T08:22:34.635 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-30T08:22:34.635 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-30T08:22:34.635 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsid": "98c0a670-06ca-11ef-bc93-c7b262605968", 2024-04-30T08:22:34.635 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "health": { 2024-04-30T08:22:34.635 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-30T08:22:34.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-30T08:22:34.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-30T08:22:34.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-30T08:22:34.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-30T08:22:34.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-30T08:22:34.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 0 2024-04-30T08:22:34.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-30T08:22:34.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-30T08:22:34.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "a" 2024-04-30T08:22:34.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-30T08:22:34.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-04-30T08:22:34.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-30T08:22:34.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-30T08:22:34.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-30T08:22:34.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-30T08:22:34.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-30T08:22:34.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-30T08:22:34.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-30T08:22:34.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-30T08:22:34.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-30T08:22:34.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-30T08:22:34.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-30T08:22:34.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-30T08:22:34.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-30T08:22:34.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-30T08:22:34.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-30T08:22:34.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-30T08:22:34.648 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-30T08:22:34.648 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-30T08:22:34.648 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-30T08:22:34.648 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-30T08:22:34.648 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-30T08:22:34.648 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-30T08:22:34.648 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-30T08:22:34.648 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-30T08:22:34.648 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-30T08:22:34.648 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-30T08:22:34.648 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-30T08:22:34.648 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-30T08:22:34.649 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-30T08:22:34.649 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-30T08:22:34.649 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-30T08:22:34.649 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-30T08:22:34.649 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-30T08:22:34.649 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-30T08:22:34.649 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-30T08:22:34.649 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "restful" 2024-04-30T08:22:34.649 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-30T08:22:34.649 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-30T08:22:34.649 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-30T08:22:34.650 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-30T08:22:34.650 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-30T08:22:34.650 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modified": "2024-04-30T08:22:11.123688+0000", 2024-04-30T08:22:34.650 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-30T08:22:34.650 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-30T08:22:34.650 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-30T08:22:34.650 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-30T08:22:34.650 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.927+0000 7fc925400700 1 Processor -- start 2024-04-30T08:22:34.650 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.928+0000 7fc925400700 1 -- start start 2024-04-30T08:22:34.650 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.928+0000 7fc925400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc920109cb0 0x7fc92010a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:34.650 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.928+0000 7fc925400700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc92010a6b0 con 0x7fc920109cb0 2024-04-30T08:22:34.650 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.928+0000 7fc91e800700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc920109cb0 0x7fc92010a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:22:34.650 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.929+0000 7fc91e800700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc920109cb0 0x7fc92010a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51334/0 (socket says 172.21.15.32:51334) 2024-04-30T08:22:34.651 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.929+0000 7fc91e800700 1 -- 172.21.15.32:0/3901946620 learned_addr learned my addr 172.21.15.32:0/3901946620 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:22:34.651 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.929+0000 7fc91e800700 1 -- 172.21.15.32:0/3901946620 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc92010ade0 con 0x7fc920109cb0 2024-04-30T08:22:34.651 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.930+0000 7fc91e800700 1 --2- 172.21.15.32:0/3901946620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc920109cb0 0x7fc92010a0c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc9100098a0 tx=0x7fc910009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c15b1e0384db80b server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:22:34.651 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.930+0000 7fc91d400700 1 -- 172.21.15.32:0/3901946620 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc910004160 con 0x7fc920109cb0 2024-04-30T08:22:34.651 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.930+0000 7fc91d400700 1 -- 172.21.15.32:0/3901946620 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc910004300 con 0x7fc920109cb0 2024-04-30T08:22:34.651 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.930+0000 7fc91d400700 1 -- 172.21.15.32:0/3901946620 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc910011af0 con 0x7fc920109cb0 2024-04-30T08:22:34.651 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.931+0000 7fc925400700 1 -- 172.21.15.32:0/3901946620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc920109cb0 msgr2=0x7fc92010a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:22:34.651 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.931+0000 7fc925400700 1 --2- 172.21.15.32:0/3901946620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc920109cb0 0x7fc92010a0c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc9100098a0 tx=0x7fc910009bb0 comp rx=0 tx=0).stop 2024-04-30T08:22:34.651 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.931+0000 7fc925400700 1 -- 172.21.15.32:0/3901946620 shutdown_connections 2024-04-30T08:22:34.651 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.931+0000 7fc925400700 1 --2- 172.21.15.32:0/3901946620 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc920109cb0 0x7fc92010a0c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:34.651 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.931+0000 7fc925400700 1 -- 172.21.15.32:0/3901946620 >> 172.21.15.32:0/3901946620 conn(0x7fc920105350 msgr2=0x7fc9201077a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:22:34.651 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.931+0000 7fc925400700 1 -- 172.21.15.32:0/3901946620 shutdown_connections 2024-04-30T08:22:34.651 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.931+0000 7fc925400700 1 -- 172.21.15.32:0/3901946620 wait complete. 2024-04-30T08:22:34.652 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.932+0000 7fc925400700 1 Processor -- start 2024-04-30T08:22:34.652 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.933+0000 7fc925400700 1 -- start start 2024-04-30T08:22:34.653 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.933+0000 7fc925400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc920109cb0 0x7fc92007ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:34.653 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.933+0000 7fc925400700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc92010e010 con 0x7fc920109cb0 2024-04-30T08:22:34.653 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.933+0000 7fc91e800700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc920109cb0 0x7fc92007ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:22:34.653 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.933+0000 7fc91e800700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc920109cb0 0x7fc92007ea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51350/0 (socket says 172.21.15.32:51350) 2024-04-30T08:22:34.653 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.933+0000 7fc91e800700 1 -- 172.21.15.32:0/3658632735 learned_addr learned my addr 172.21.15.32:0/3658632735 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:22:34.653 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.934+0000 7fc91e800700 1 -- 172.21.15.32:0/3658632735 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc92007efa0 con 0x7fc920109cb0 2024-04-30T08:22:34.653 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.934+0000 7fc91e800700 1 --2- 172.21.15.32:0/3658632735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc920109cb0 0x7fc92007ea60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc9100090e0 tx=0x7fc910011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:22:34.653 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.934+0000 7fc917400700 1 -- 172.21.15.32:0/3658632735 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc9100039e0 con 0x7fc920109cb0 2024-04-30T08:22:34.653 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.934+0000 7fc917400700 1 -- 172.21.15.32:0/3658632735 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc910003b80 con 0x7fc920109cb0 2024-04-30T08:22:34.653 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.934+0000 7fc925400700 1 -- 172.21.15.32:0/3658632735 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc92007d1f0 con 0x7fc920109cb0 2024-04-30T08:22:34.654 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.934+0000 7fc917400700 1 -- 172.21.15.32:0/3658632735 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc910024690 con 0x7fc920109cb0 2024-04-30T08:22:34.654 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.935+0000 7fc925400700 1 -- 172.21.15.32:0/3658632735 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc92007d6d0 con 0x7fc920109cb0 2024-04-30T08:22:34.654 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.935+0000 7fc917400700 1 -- 172.21.15.32:0/3658632735 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fc910022020 con 0x7fc920109cb0 2024-04-30T08:22:34.654 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.935+0000 7fc917400700 1 -- 172.21.15.32:0/3658632735 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc91002dbe0 con 0x7fc920109cb0 2024-04-30T08:22:34.654 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.936+0000 7fc925400700 1 -- 172.21.15.32:0/3658632735 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc920064d70 con 0x7fc920109cb0 2024-04-30T08:22:34.654 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:33.938+0000 7fc917400700 1 -- 172.21.15.32:0/3658632735 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fc910043e50 con 0x7fc920109cb0 2024-04-30T08:22:34.654 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:34.017+0000 7fc925400700 1 -- 172.21.15.32:0/3658632735 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fc92007ddc0 con 0x7fc920109cb0 2024-04-30T08:22:34.654 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:34.018+0000 7fc917400700 1 -- 172.21.15.32:0/3658632735 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fc910032020 con 0x7fc920109cb0 2024-04-30T08:22:34.654 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:34.020+0000 7fc925400700 1 -- 172.21.15.32:0/3658632735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc920109cb0 msgr2=0x7fc92007ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:22:34.654 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:34.020+0000 7fc925400700 1 --2- 172.21.15.32:0/3658632735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc920109cb0 0x7fc92007ea60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc9100090e0 tx=0x7fc910011940 comp rx=0 tx=0).stop 2024-04-30T08:22:34.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:34.020+0000 7fc925400700 1 -- 172.21.15.32:0/3658632735 shutdown_connections 2024-04-30T08:22:34.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:34.020+0000 7fc925400700 1 --2- 172.21.15.32:0/3658632735 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc920109cb0 0x7fc92007ea60 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:34.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:34.020+0000 7fc925400700 1 -- 172.21.15.32:0/3658632735 >> 172.21.15.32:0/3658632735 conn(0x7fc920105350 msgr2=0x7fc920105cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:22:34.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:34.020+0000 7fc925400700 1 -- 172.21.15.32:0/3658632735 shutdown_connections 2024-04-30T08:22:34.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:34.020+0000 7fc925400700 1 -- 172.21.15.32:0/3658632735 wait complete. 2024-04-30T08:22:34.655 INFO:teuthology.orchestra.run.smithi032.stdout:mgr not available, waiting (2/15)... 2024-04-30T08:22:35.191 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:34 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:34.836+0000 7f95bdadd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-30T08:22:35.691 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:35 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:35.412+0000 7f95bdadd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-30T08:22:35.691 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:35 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:35.544+0000 7f95bdadd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-30T08:22:36.191 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:35 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:35.820+0000 7f95bdadd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-30T08:22:36.805 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:36 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:36.485+0000 7f95bdadd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-30T08:22:36.805 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:36 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:36.613+0000 7f95bdadd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-30T08:22:37.085 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:36 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:36.961+0000 7f95bdadd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-30T08:22:37.407 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:37 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:37.149+0000 7f95bdadd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-30T08:22:37.407 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:37 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:37.283+0000 7f95bdadd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-30T08:22:37.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:37 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3482952534' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-30T08:22:37.691 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:37 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:37.445+0000 7f95bdadd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-30T08:22:38.021 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:37 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:37.823+0000 7f95bdadd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-30T08:22:38.034 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-30T08:22:38.035 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-30T08:22:38.035 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsid": "98c0a670-06ca-11ef-bc93-c7b262605968", 2024-04-30T08:22:38.035 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "health": { 2024-04-30T08:22:38.035 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-30T08:22:38.035 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-30T08:22:38.035 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-30T08:22:38.035 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-30T08:22:38.035 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-30T08:22:38.035 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-30T08:22:38.035 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 0 2024-04-30T08:22:38.035 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-30T08:22:38.035 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-30T08:22:38.036 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "a" 2024-04-30T08:22:38.036 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-30T08:22:38.036 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_age": 11, 2024-04-30T08:22:38.036 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-30T08:22:38.036 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-30T08:22:38.036 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-30T08:22:38.036 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-30T08:22:38.036 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-30T08:22:38.036 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-30T08:22:38.036 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-30T08:22:38.036 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-30T08:22:38.036 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-30T08:22:38.044 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-30T08:22:38.044 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-30T08:22:38.044 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-30T08:22:38.044 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-30T08:22:38.045 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-30T08:22:38.045 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-30T08:22:38.045 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-30T08:22:38.045 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-30T08:22:38.045 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-30T08:22:38.045 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-30T08:22:38.045 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-30T08:22:38.045 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-30T08:22:38.045 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-30T08:22:38.045 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-30T08:22:38.045 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-30T08:22:38.045 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-30T08:22:38.045 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-30T08:22:38.045 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-30T08:22:38.046 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-30T08:22:38.046 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-30T08:22:38.046 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-30T08:22:38.046 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-30T08:22:38.046 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-30T08:22:38.046 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-30T08:22:38.046 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-30T08:22:38.046 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-30T08:22:38.046 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "restful" 2024-04-30T08:22:38.046 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-30T08:22:38.046 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-30T08:22:38.046 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-30T08:22:38.046 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-30T08:22:38.047 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-30T08:22:38.047 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modified": "2024-04-30T08:22:11.123688+0000", 2024-04-30T08:22:38.047 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-30T08:22:38.047 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-30T08:22:38.047 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-30T08:22:38.047 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-30T08:22:38.047 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.315+0000 7fe96d200700 1 Processor -- start 2024-04-30T08:22:38.047 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.315+0000 7fe96d200700 1 -- start start 2024-04-30T08:22:38.047 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.315+0000 7fe96d200700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe96810a500 0x7fe96810c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:38.047 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.315+0000 7fe96d200700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe96807cdd0 con 0x7fe96810a500 2024-04-30T08:22:38.047 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.316+0000 7fe967400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe96810a500 0x7fe96810c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:22:38.047 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.316+0000 7fe967400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe96810a500 0x7fe96810c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33526/0 (socket says 172.21.15.32:33526) 2024-04-30T08:22:38.047 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.316+0000 7fe967400700 1 -- 172.21.15.32:0/3772616489 learned_addr learned my addr 172.21.15.32:0/3772616489 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:22:38.048 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.316+0000 7fe967400700 1 -- 172.21.15.32:0/3772616489 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe96810ce60 con 0x7fe96810a500 2024-04-30T08:22:38.048 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.317+0000 7fe967400700 1 --2- 172.21.15.32:0/3772616489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe96810a500 0x7fe96810c920 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe9580098a0 tx=0x7fe958009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe49668990b418dd server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:22:38.048 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.317+0000 7fe966000700 1 -- 172.21.15.32:0/3772616489 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe958004160 con 0x7fe96810a500 2024-04-30T08:22:38.048 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.317+0000 7fe966000700 1 -- 172.21.15.32:0/3772616489 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe958004300 con 0x7fe96810a500 2024-04-30T08:22:38.048 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.317+0000 7fe966000700 1 -- 172.21.15.32:0/3772616489 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe958011af0 con 0x7fe96810a500 2024-04-30T08:22:38.048 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.318+0000 7fe96d200700 1 -- 172.21.15.32:0/3772616489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe96810a500 msgr2=0x7fe96810c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:22:38.048 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.318+0000 7fe96d200700 1 --2- 172.21.15.32:0/3772616489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe96810a500 0x7fe96810c920 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe9580098a0 tx=0x7fe958009bb0 comp rx=0 tx=0).stop 2024-04-30T08:22:38.048 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.318+0000 7fe96d200700 1 -- 172.21.15.32:0/3772616489 shutdown_connections 2024-04-30T08:22:38.048 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.318+0000 7fe96d200700 1 --2- 172.21.15.32:0/3772616489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe96810a500 0x7fe96810c920 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:38.048 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.318+0000 7fe96d200700 1 -- 172.21.15.32:0/3772616489 >> 172.21.15.32:0/3772616489 conn(0x7fe968103fa0 msgr2=0x7fe968106410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:22:38.048 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.319+0000 7fe96d200700 1 -- 172.21.15.32:0/3772616489 shutdown_connections 2024-04-30T08:22:38.048 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.319+0000 7fe96d200700 1 -- 172.21.15.32:0/3772616489 wait complete. 2024-04-30T08:22:38.048 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.320+0000 7fe96d200700 1 Processor -- start 2024-04-30T08:22:38.048 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.320+0000 7fe96d200700 1 -- start start 2024-04-30T08:22:38.049 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.320+0000 7fe96d200700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe96810a500 0x7fe96819ff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:38.049 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.320+0000 7fe96d200700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe96810dd60 con 0x7fe96810a500 2024-04-30T08:22:38.049 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.320+0000 7fe967400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe96810a500 0x7fe96819ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:22:38.049 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.320+0000 7fe967400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe96810a500 0x7fe96819ff70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33532/0 (socket says 172.21.15.32:33532) 2024-04-30T08:22:38.049 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.321+0000 7fe967400700 1 -- 172.21.15.32:0/3482952534 learned_addr learned my addr 172.21.15.32:0/3482952534 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:22:38.049 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.321+0000 7fe967400700 1 -- 172.21.15.32:0/3482952534 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9681a04b0 con 0x7fe96810a500 2024-04-30T08:22:38.049 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.321+0000 7fe967400700 1 --2- 172.21.15.32:0/3482952534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe96810a500 0x7fe96819ff70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe958009fb0 tx=0x7fe958011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:22:38.049 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.322+0000 7fe95fe00700 1 -- 172.21.15.32:0/3482952534 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe9580036a0 con 0x7fe96810a500 2024-04-30T08:22:38.049 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.322+0000 7fe95fe00700 1 -- 172.21.15.32:0/3482952534 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fe95801a430 con 0x7fe96810a500 2024-04-30T08:22:38.049 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.322+0000 7fe96d200700 1 -- 172.21.15.32:0/3482952534 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9681a0740 con 0x7fe96810a500 2024-04-30T08:22:38.049 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.322+0000 7fe95fe00700 1 -- 172.21.15.32:0/3482952534 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe958019550 con 0x7fe96810a500 2024-04-30T08:22:38.049 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.322+0000 7fe96d200700 1 -- 172.21.15.32:0/3482952534 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9681a3350 con 0x7fe96810a500 2024-04-30T08:22:38.049 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.322+0000 7fe95fe00700 1 -- 172.21.15.32:0/3482952534 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe958019b90 con 0x7fe96810a500 2024-04-30T08:22:38.050 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.322+0000 7fe95fe00700 1 -- 172.21.15.32:0/3482952534 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe958024730 con 0x7fe96810a500 2024-04-30T08:22:38.050 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.323+0000 7fe96d200700 1 -- 172.21.15.32:0/3482952534 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe968064d70 con 0x7fe96810a500 2024-04-30T08:22:38.050 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.326+0000 7fe95fe00700 1 -- 172.21.15.32:0/3482952534 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fe958043e50 con 0x7fe96810a500 2024-04-30T08:22:38.050 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.405+0000 7fe96d200700 1 -- 172.21.15.32:0/3482952534 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fe9681a3520 con 0x7fe96810a500 2024-04-30T08:22:38.050 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.406+0000 7fe95fe00700 1 -- 172.21.15.32:0/3482952534 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7fe958032020 con 0x7fe96810a500 2024-04-30T08:22:38.050 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.408+0000 7fe96d200700 1 -- 172.21.15.32:0/3482952534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe96810a500 msgr2=0x7fe96819ff70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:22:38.050 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.408+0000 7fe96d200700 1 --2- 172.21.15.32:0/3482952534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe96810a500 0x7fe96819ff70 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe958009fb0 tx=0x7fe958011940 comp rx=0 tx=0).stop 2024-04-30T08:22:38.050 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.408+0000 7fe96d200700 1 -- 172.21.15.32:0/3482952534 shutdown_connections 2024-04-30T08:22:38.050 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.408+0000 7fe96d200700 1 --2- 172.21.15.32:0/3482952534 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe96810a500 0x7fe96819ff70 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:38.050 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.408+0000 7fe96d200700 1 -- 172.21.15.32:0/3482952534 >> 172.21.15.32:0/3482952534 conn(0x7fe968103fa0 msgr2=0x7fe968108440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:22:38.050 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.408+0000 7fe96d200700 1 -- 172.21.15.32:0/3482952534 shutdown_connections 2024-04-30T08:22:38.050 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:37.408+0000 7fe96d200700 1 -- 172.21.15.32:0/3482952534 wait complete. 2024-04-30T08:22:38.050 INFO:teuthology.orchestra.run.smithi032.stdout:mgr not available, waiting (3/15)... 2024-04-30T08:22:38.930 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:38 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:38.632+0000 7f95bdadd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-30T08:22:39.191 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:38 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:38.928+0000 7f95bdadd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-30T08:22:39.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:38 smithi032 ceph-mon[28331]: Activating manager daemon x 2024-04-30T08:22:39.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:38 smithi032 ceph-mon[28331]: mgrmap e2: x(active, starting, since 0.0132388s) 2024-04-30T08:22:39.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:38 smithi032 ceph-mon[28331]: from='mgr.14100 172.21.15.32:0/3827278482' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-30T08:22:39.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:38 smithi032 ceph-mon[28331]: from='mgr.14100 172.21.15.32:0/3827278482' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-30T08:22:39.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:38 smithi032 ceph-mon[28331]: from='mgr.14100 172.21.15.32:0/3827278482' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-30T08:22:39.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:38 smithi032 ceph-mon[28331]: from='mgr.14100 172.21.15.32:0/3827278482' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-30T08:22:39.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:38 smithi032 ceph-mon[28331]: from='mgr.14100 172.21.15.32:0/3827278482' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-30T08:22:39.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:38 smithi032 ceph-mon[28331]: Manager daemon x is now available 2024-04-30T08:22:39.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:38 smithi032 ceph-mon[28331]: from='mgr.14100 172.21.15.32:0/3827278482' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-30T08:22:39.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:38 smithi032 ceph-mon[28331]: from='mgr.14100 172.21.15.32:0/3827278482' entity='mgr.x' 2024-04-30T08:22:39.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:38 smithi032 ceph-mon[28331]: from='mgr.14100 172.21.15.32:0/3827278482' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-30T08:22:39.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:38 smithi032 ceph-mon[28331]: from='mgr.14100 172.21.15.32:0/3827278482' entity='mgr.x' 2024-04-30T08:22:40.316 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:39 smithi032 ceph-mon[28331]: from='mgr.14100 172.21.15.32:0/3827278482' entity='mgr.x' 2024-04-30T08:22:40.316 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:39 smithi032 ceph-mon[28331]: mgrmap e3: x(active, since 1.02675s) 2024-04-30T08:22:41.635 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-30T08:22:41.635 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-30T08:22:41.635 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsid": "98c0a670-06ca-11ef-bc93-c7b262605968", 2024-04-30T08:22:41.635 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "health": { 2024-04-30T08:22:41.635 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-30T08:22:41.635 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-30T08:22:41.635 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-30T08:22:41.635 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-30T08:22:41.635 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-30T08:22:41.635 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-30T08:22:41.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 0 2024-04-30T08:22:41.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-30T08:22:41.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-30T08:22:41.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "a" 2024-04-30T08:22:41.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-30T08:22:41.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-04-30T08:22:41.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-30T08:22:41.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-30T08:22:41.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-30T08:22:41.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-30T08:22:41.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-30T08:22:41.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-30T08:22:41.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-30T08:22:41.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-30T08:22:41.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-30T08:22:41.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-30T08:22:41.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-30T08:22:41.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-30T08:22:41.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-30T08:22:41.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-30T08:22:41.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-30T08:22:41.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-30T08:22:41.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-30T08:22:41.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-30T08:22:41.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-30T08:22:41.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-30T08:22:41.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-30T08:22:41.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-30T08:22:41.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-30T08:22:41.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-30T08:22:41.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-30T08:22:41.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-30T08:22:41.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-30T08:22:41.640 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-30T08:22:41.640 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-30T08:22:41.640 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-30T08:22:41.640 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-30T08:22:41.640 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-30T08:22:41.640 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-30T08:22:41.641 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-30T08:22:41.641 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-30T08:22:41.641 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "restful" 2024-04-30T08:22:41.642 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ], 2024-04-30T08:22:41.642 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-30T08:22:41.642 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-30T08:22:41.642 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-30T08:22:41.642 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-30T08:22:41.642 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "modified": "2024-04-30T08:22:11.123688+0000", 2024-04-30T08:22:41.643 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-30T08:22:41.643 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout }, 2024-04-30T08:22:41.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-30T08:22:41.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-30T08:22:41.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.690+0000 7f1310c00700 1 Processor -- start 2024-04-30T08:22:41.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.690+0000 7f1310c00700 1 -- start start 2024-04-30T08:22:41.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.691+0000 7f1310c00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f130c109a30 0x7f130c109e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:41.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.691+0000 7f1310c00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f130c10a430 con 0x7f130c109a30 2024-04-30T08:22:41.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.691+0000 7f130b400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f130c109a30 0x7f130c109e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:22:41.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.691+0000 7f130b400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f130c109a30 0x7f130c109e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33594/0 (socket says 172.21.15.32:33594) 2024-04-30T08:22:41.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.691+0000 7f130b400700 1 -- 172.21.15.32:0/4155347655 learned_addr learned my addr 172.21.15.32:0/4155347655 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:22:41.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.692+0000 7f130b400700 1 -- 172.21.15.32:0/4155347655 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f130c10ab60 con 0x7f130c109a30 2024-04-30T08:22:41.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.692+0000 7f130b400700 1 --2- 172.21.15.32:0/4155347655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f130c109a30 0x7f130c109e40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f12fc0098a0 tx=0x7f12fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65b1e14eebb83e65 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:22:41.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.693+0000 7f130a000700 1 -- 172.21.15.32:0/4155347655 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12fc004160 con 0x7f130c109a30 2024-04-30T08:22:41.646 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.693+0000 7f130a000700 1 -- 172.21.15.32:0/4155347655 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f12fc004300 con 0x7f130c109a30 2024-04-30T08:22:41.650 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.693+0000 7f130a000700 1 -- 172.21.15.32:0/4155347655 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12fc011af0 con 0x7f130c109a30 2024-04-30T08:22:41.650 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.693+0000 7f1310c00700 1 -- 172.21.15.32:0/4155347655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f130c109a30 msgr2=0x7f130c109e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:22:41.650 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.693+0000 7f1310c00700 1 --2- 172.21.15.32:0/4155347655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f130c109a30 0x7f130c109e40 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f12fc0098a0 tx=0x7f12fc009bb0 comp rx=0 tx=0).stop 2024-04-30T08:22:41.650 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.694+0000 7f1310c00700 1 -- 172.21.15.32:0/4155347655 shutdown_connections 2024-04-30T08:22:41.650 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.694+0000 7f1310c00700 1 --2- 172.21.15.32:0/4155347655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f130c109a30 0x7f130c109e40 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:41.650 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.694+0000 7f1310c00700 1 -- 172.21.15.32:0/4155347655 >> 172.21.15.32:0/4155347655 conn(0x7f130c105090 msgr2=0x7f130c107520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:22:41.650 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.694+0000 7f1310c00700 1 -- 172.21.15.32:0/4155347655 shutdown_connections 2024-04-30T08:22:41.650 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.694+0000 7f1310c00700 1 -- 172.21.15.32:0/4155347655 wait complete. 2024-04-30T08:22:41.650 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.695+0000 7f1310c00700 1 Processor -- start 2024-04-30T08:22:41.651 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.695+0000 7f1310c00700 1 -- start start 2024-04-30T08:22:41.651 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.695+0000 7f1310c00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f130c109a30 0x7f130c1a0060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:41.651 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.695+0000 7f1310c00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f130c10df10 con 0x7f130c109a30 2024-04-30T08:22:41.651 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.696+0000 7f130b400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f130c109a30 0x7f130c1a0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:22:41.651 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.696+0000 7f130b400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f130c109a30 0x7f130c1a0060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33602/0 (socket says 172.21.15.32:33602) 2024-04-30T08:22:41.651 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.696+0000 7f130b400700 1 -- 172.21.15.32:0/3005217656 learned_addr learned my addr 172.21.15.32:0/3005217656 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:22:41.651 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.696+0000 7f130b400700 1 -- 172.21.15.32:0/3005217656 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f130c1a05a0 con 0x7f130c109a30 2024-04-30T08:22:41.651 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.696+0000 7f130b400700 1 --2- 172.21.15.32:0/3005217656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f130c109a30 0x7f130c1a0060 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f12fc011890 tx=0x7f12fc011970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:22:41.651 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.697+0000 7f1303e00700 1 -- 172.21.15.32:0/3005217656 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12fc0039e0 con 0x7f130c109a30 2024-04-30T08:22:41.651 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.697+0000 7f1303e00700 1 -- 172.21.15.32:0/3005217656 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f12fc003b80 con 0x7f130c109a30 2024-04-30T08:22:41.651 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.697+0000 7f1310c00700 1 -- 172.21.15.32:0/3005217656 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f130c1a0830 con 0x7f130c109a30 2024-04-30T08:22:41.652 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.697+0000 7f1303e00700 1 -- 172.21.15.32:0/3005217656 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f12fc0246e0 con 0x7f130c109a30 2024-04-30T08:22:41.652 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.697+0000 7f1310c00700 1 -- 172.21.15.32:0/3005217656 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f130c1a3440 con 0x7f130c109a30 2024-04-30T08:22:41.652 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.698+0000 7f1310c00700 1 -- 172.21.15.32:0/3005217656 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f130c064d70 con 0x7f130c109a30 2024-04-30T08:22:41.652 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.698+0000 7f1303e00700 1 -- 172.21.15.32:0/3005217656 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 3) ==== 49399+0+0 (secure 0 0 0) 0x7f12fc022020 con 0x7f130c109a30 2024-04-30T08:22:41.652 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.699+0000 7f1303e00700 1 --2- 172.21.15.32:0/3005217656 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f12f003d450 0x7f12f003f960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:41.652 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.699+0000 7f1303e00700 1 -- 172.21.15.32:0/3005217656 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f12fc051f40 con 0x7f130c109a30 2024-04-30T08:22:41.652 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.699+0000 7f130aa00700 1 --2- 172.21.15.32:0/3005217656 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f12f003d450 0x7f12f003f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:22:41.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.700+0000 7f130aa00700 1 --2- 172.21.15.32:0/3005217656 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f12f003d450 0x7f12f003f960 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f12f8006fd0 tx=0x7f12f8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:22:41.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.706+0000 7f1303e00700 1 -- 172.21.15.32:0/3005217656 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f12fc0212c0 con 0x7f130c109a30 2024-04-30T08:22:41.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:40.995+0000 7f1303e00700 1 -- 172.21.15.32:0/3005217656 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mgrmap(e 4) ==== 49505+0+0 (secure 0 0 0) 0x7f12fc019c70 con 0x7f130c109a30 2024-04-30T08:22:41.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:41.024+0000 7f1310c00700 1 -- 172.21.15.32:0/3005217656 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f130c1a3ac0 con 0x7f130c109a30 2024-04-30T08:22:41.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:41.026+0000 7f1303e00700 1 -- 172.21.15.32:0/3005217656 <== mon.0 v2:172.21.15.32:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f12fc017e90 con 0x7f130c109a30 2024-04-30T08:22:41.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:41.029+0000 7f1310c00700 1 -- 172.21.15.32:0/3005217656 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f12f003d450 msgr2=0x7f12f003f960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:22:41.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:41.029+0000 7f1310c00700 1 --2- 172.21.15.32:0/3005217656 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f12f003d450 0x7f12f003f960 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f12f8006fd0 tx=0x7f12f8006e40 comp rx=0 tx=0).stop 2024-04-30T08:22:41.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:41.029+0000 7f1310c00700 1 -- 172.21.15.32:0/3005217656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f130c109a30 msgr2=0x7f130c1a0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:22:41.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:41.029+0000 7f1310c00700 1 --2- 172.21.15.32:0/3005217656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f130c109a30 0x7f130c1a0060 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f12fc011890 tx=0x7f12fc011970 comp rx=0 tx=0).stop 2024-04-30T08:22:41.656 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:41.029+0000 7f1310c00700 1 -- 172.21.15.32:0/3005217656 shutdown_connections 2024-04-30T08:22:41.656 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:41.029+0000 7f1310c00700 1 --2- 172.21.15.32:0/3005217656 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f12f003d450 0x7f12f003f960 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:41.656 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:41.029+0000 7f1310c00700 1 --2- 172.21.15.32:0/3005217656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f130c109a30 0x7f130c1a0060 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:41.656 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:41.029+0000 7f1310c00700 1 -- 172.21.15.32:0/3005217656 >> 172.21.15.32:0/3005217656 conn(0x7f130c105090 msgr2=0x7f130c1059f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:22:41.656 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:41.029+0000 7f1310c00700 1 -- 172.21.15.32:0/3005217656 shutdown_connections 2024-04-30T08:22:41.656 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:41.029+0000 7f1310c00700 1 -- 172.21.15.32:0/3005217656 wait complete. 2024-04-30T08:22:41.656 INFO:teuthology.orchestra.run.smithi032.stdout:mgr is available 2024-04-30T08:22:42.346 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:41 smithi032 ceph-mon[28331]: mgrmap e4: x(active, since 2s) 2024-04-30T08:22:42.346 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:41 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3005217656' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-30T08:22:43.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:42 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2290086677' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-30T08:22:43.262 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-30T08:22:43.263 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [global] 2024-04-30T08:22:43.263 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout fsid = 98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:22:43.263 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_cluster_log_file_level = debug 2024-04-30T08:22:43.263 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-30T08:22:43.263 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-30T08:22:43.263 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-30T08:22:43.263 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-30T08:22:43.263 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-30T08:22:43.263 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-30T08:22:43.263 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-30T08:22:43.263 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 2024-04-30T08:22:43.267 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout [osd] 2024-04-30T08:22:43.267 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2024-04-30T08:22:43.267 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-30T08:22:43.267 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-30T08:22:43.267 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-30T08:22:43.267 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-30T08:22:43.267 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-30T08:22:43.267 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.345+0000 7fc9b6600700 1 Processor -- start 2024-04-30T08:22:43.267 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.345+0000 7fc9b6600700 1 -- start start 2024-04-30T08:22:43.268 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.346+0000 7fc9b6600700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9b0109ce0 0x7fc9b010a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:43.268 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.346+0000 7fc9b6600700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b010a6e0 con 0x7fc9b0109ce0 2024-04-30T08:22:43.268 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.346+0000 7fc9af400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9b0109ce0 0x7fc9b010a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:22:43.268 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.346+0000 7fc9af400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9b0109ce0 0x7fc9b010a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33612/0 (socket says 172.21.15.32:33612) 2024-04-30T08:22:43.268 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.346+0000 7fc9af400700 1 -- 172.21.15.32:0/3710783900 learned_addr learned my addr 172.21.15.32:0/3710783900 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:22:43.268 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.347+0000 7fc9af400700 1 -- 172.21.15.32:0/3710783900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9b010adc0 con 0x7fc9b0109ce0 2024-04-30T08:22:43.268 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.347+0000 7fc9af400700 1 --2- 172.21.15.32:0/3710783900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9b0109ce0 0x7fc9b010a0f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc9a00098a0 tx=0x7fc9a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b05afa05083f9688 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:22:43.268 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.347+0000 7fc9aea00700 1 -- 172.21.15.32:0/3710783900 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc9a0004160 con 0x7fc9b0109ce0 2024-04-30T08:22:43.268 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.347+0000 7fc9aea00700 1 -- 172.21.15.32:0/3710783900 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc9a0004300 con 0x7fc9b0109ce0 2024-04-30T08:22:43.268 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.347+0000 7fc9aea00700 1 -- 172.21.15.32:0/3710783900 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc9a0011af0 con 0x7fc9b0109ce0 2024-04-30T08:22:43.268 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.348+0000 7fc9b6600700 1 -- 172.21.15.32:0/3710783900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9b0109ce0 msgr2=0x7fc9b010a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:22:43.268 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.348+0000 7fc9b6600700 1 --2- 172.21.15.32:0/3710783900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9b0109ce0 0x7fc9b010a0f0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc9a00098a0 tx=0x7fc9a0009bb0 comp rx=0 tx=0).stop 2024-04-30T08:22:43.268 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.348+0000 7fc9b6600700 1 -- 172.21.15.32:0/3710783900 shutdown_connections 2024-04-30T08:22:43.269 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.348+0000 7fc9b6600700 1 --2- 172.21.15.32:0/3710783900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9b0109ce0 0x7fc9b010a0f0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:43.269 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.348+0000 7fc9b6600700 1 -- 172.21.15.32:0/3710783900 >> 172.21.15.32:0/3710783900 conn(0x7fc9b0105300 msgr2=0x7fc9b01077d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:22:43.269 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.349+0000 7fc9b6600700 1 -- 172.21.15.32:0/3710783900 shutdown_connections 2024-04-30T08:22:43.269 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.349+0000 7fc9b6600700 1 -- 172.21.15.32:0/3710783900 wait complete. 2024-04-30T08:22:43.269 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.350+0000 7fc9b6600700 1 Processor -- start 2024-04-30T08:22:43.269 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.350+0000 7fc9b6600700 1 -- start start 2024-04-30T08:22:43.269 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.350+0000 7fc9b6600700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9b0109ce0 0x7fc9b019e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:43.269 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.350+0000 7fc9b6600700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b010e170 con 0x7fc9b0109ce0 2024-04-30T08:22:43.269 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.351+0000 7fc9af400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9b0109ce0 0x7fc9b019e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:22:43.269 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.351+0000 7fc9af400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9b0109ce0 0x7fc9b019e090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33620/0 (socket says 172.21.15.32:33620) 2024-04-30T08:22:43.269 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.351+0000 7fc9af400700 1 -- 172.21.15.32:0/2290086677 learned_addr learned my addr 172.21.15.32:0/2290086677 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:22:43.269 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.351+0000 7fc9af400700 1 -- 172.21.15.32:0/2290086677 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9b019e5d0 con 0x7fc9b0109ce0 2024-04-30T08:22:43.269 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.351+0000 7fc9af400700 1 --2- 172.21.15.32:0/2290086677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9b0109ce0 0x7fc9b019e090 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0011890 tx=0x7fc9a0011970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:22:43.272 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.352+0000 7fc9acc00700 1 -- 172.21.15.32:0/2290086677 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc9a00039e0 con 0x7fc9b0109ce0 2024-04-30T08:22:43.272 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.352+0000 7fc9acc00700 1 -- 172.21.15.32:0/2290086677 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fc9a0003b80 con 0x7fc9b0109ce0 2024-04-30T08:22:43.272 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.352+0000 7fc9acc00700 1 -- 172.21.15.32:0/2290086677 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc9a00246e0 con 0x7fc9b0109ce0 2024-04-30T08:22:43.272 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.352+0000 7fc9b6600700 1 -- 172.21.15.32:0/2290086677 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9b019e860 con 0x7fc9b0109ce0 2024-04-30T08:22:43.272 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.352+0000 7fc9b6600700 1 -- 172.21.15.32:0/2290086677 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9b01a1470 con 0x7fc9b0109ce0 2024-04-30T08:22:43.272 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.353+0000 7fc9b6600700 1 -- 172.21.15.32:0/2290086677 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9b0064d70 con 0x7fc9b0109ce0 2024-04-30T08:22:43.272 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.353+0000 7fc9acc00700 1 -- 172.21.15.32:0/2290086677 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 4) ==== 49505+0+0 (secure 0 0 0) 0x7fc9a0022020 con 0x7fc9b0109ce0 2024-04-30T08:22:43.272 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.354+0000 7fc9acc00700 1 --2- 172.21.15.32:0/2290086677 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7fc99403d570 0x7fc99403fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:43.273 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.354+0000 7fc9acc00700 1 -- 172.21.15.32:0/2290086677 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fc9a0052170 con 0x7fc9b0109ce0 2024-04-30T08:22:43.273 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.354+0000 7fc9a6a00700 1 --2- 172.21.15.32:0/2290086677 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7fc99403d570 0x7fc99403fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:22:43.273 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.355+0000 7fc9a6a00700 1 --2- 172.21.15.32:0/2290086677 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7fc99403d570 0x7fc99403fa80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc99800ad30 tx=0x7fc9980093f0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:22:43.273 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.360+0000 7fc9acc00700 1 -- 172.21.15.32:0/2290086677 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc9a00212c0 con 0x7fc9b0109ce0 2024-04-30T08:22:43.273 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.592+0000 7fc9b6600700 1 -- 172.21.15.32:0/2290086677 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7fc9b01a2c70 con 0x7fc9b0109ce0 2024-04-30T08:22:43.273 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.595+0000 7fc9acc00700 1 -- 172.21.15.32:0/2290086677 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+434 (secure 0 0 0) 0x7fc9a00094f0 con 0x7fc9b0109ce0 2024-04-30T08:22:43.273 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.599+0000 7fc9b6600700 1 -- 172.21.15.32:0/2290086677 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7fc99403d570 msgr2=0x7fc99403fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:22:43.273 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.599+0000 7fc9b6600700 1 --2- 172.21.15.32:0/2290086677 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7fc99403d570 0x7fc99403fa80 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fc99800ad30 tx=0x7fc9980093f0 comp rx=0 tx=0).stop 2024-04-30T08:22:43.273 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.599+0000 7fc9b6600700 1 -- 172.21.15.32:0/2290086677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9b0109ce0 msgr2=0x7fc9b019e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:22:43.273 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.599+0000 7fc9b6600700 1 --2- 172.21.15.32:0/2290086677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9b0109ce0 0x7fc9b019e090 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc9a0011890 tx=0x7fc9a0011970 comp rx=0 tx=0).stop 2024-04-30T08:22:43.273 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.600+0000 7fc9b6600700 1 -- 172.21.15.32:0/2290086677 shutdown_connections 2024-04-30T08:22:43.273 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.600+0000 7fc9b6600700 1 --2- 172.21.15.32:0/2290086677 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7fc99403d570 0x7fc99403fa80 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:43.273 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.600+0000 7fc9b6600700 1 --2- 172.21.15.32:0/2290086677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc9b0109ce0 0x7fc9b019e090 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:43.273 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.600+0000 7fc9b6600700 1 -- 172.21.15.32:0/2290086677 >> 172.21.15.32:0/2290086677 conn(0x7fc9b0105300 msgr2=0x7fc9b0105cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:22:43.274 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.600+0000 7fc9b6600700 1 -- 172.21.15.32:0/2290086677 shutdown_connections 2024-04-30T08:22:43.274 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:42.600+0000 7fc9b6600700 1 -- 172.21.15.32:0/2290086677 wait complete. 2024-04-30T08:22:43.274 INFO:teuthology.orchestra.run.smithi032.stdout:Enabling cephadm module... 2024-04-30T08:22:44.685 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:44 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2377468883' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-30T08:22:44.686 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:44 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: ignoring --setuser ceph since I am not root 2024-04-30T08:22:44.686 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:44 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: ignoring --setgroup ceph since I am not root 2024-04-30T08:22:44.686 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:44 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:44.684+0000 7fe058216200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-30T08:22:45.015 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.937+0000 7f518e000700 1 Processor -- start 2024-04-30T08:22:45.016 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.937+0000 7f518e000700 1 -- start start 2024-04-30T08:22:45.016 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.938+0000 7f518e000700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5188109c80 0x7f518810a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:45.016 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.938+0000 7f518e000700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f518810a680 con 0x7f5188109c80 2024-04-30T08:22:45.016 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.938+0000 7f5187400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5188109c80 0x7f518810a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:22:45.016 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.938+0000 7f5187400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5188109c80 0x7f518810a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33630/0 (socket says 172.21.15.32:33630) 2024-04-30T08:22:45.016 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.938+0000 7f5187400700 1 -- 172.21.15.32:0/3703172588 learned_addr learned my addr 172.21.15.32:0/3703172588 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:22:45.016 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.939+0000 7f5187400700 1 -- 172.21.15.32:0/3703172588 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f518810adb0 con 0x7f5188109c80 2024-04-30T08:22:45.016 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.939+0000 7f5187400700 1 --2- 172.21.15.32:0/3703172588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5188109c80 0x7f518810a090 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f51780098a0 tx=0x7f5178009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=828d90dd2e0f4857 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:22:45.016 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.939+0000 7f5186000700 1 -- 172.21.15.32:0/3703172588 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5178004160 con 0x7f5188109c80 2024-04-30T08:22:45.016 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.940+0000 7f5186000700 1 -- 172.21.15.32:0/3703172588 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5178004300 con 0x7f5188109c80 2024-04-30T08:22:45.016 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.940+0000 7f5186000700 1 -- 172.21.15.32:0/3703172588 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5178011af0 con 0x7f5188109c80 2024-04-30T08:22:45.017 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.940+0000 7f518e000700 1 -- 172.21.15.32:0/3703172588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5188109c80 msgr2=0x7f518810a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:22:45.017 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.940+0000 7f518e000700 1 --2- 172.21.15.32:0/3703172588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5188109c80 0x7f518810a090 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f51780098a0 tx=0x7f5178009bb0 comp rx=0 tx=0).stop 2024-04-30T08:22:45.017 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.941+0000 7f518e000700 1 -- 172.21.15.32:0/3703172588 shutdown_connections 2024-04-30T08:22:45.017 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.941+0000 7f518e000700 1 --2- 172.21.15.32:0/3703172588 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5188109c80 0x7f518810a090 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:45.017 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.941+0000 7f518e000700 1 -- 172.21.15.32:0/3703172588 >> 172.21.15.32:0/3703172588 conn(0x7f5188105300 msgr2=0x7f5188107770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:22:45.017 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.941+0000 7f518e000700 1 -- 172.21.15.32:0/3703172588 shutdown_connections 2024-04-30T08:22:45.017 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.941+0000 7f518e000700 1 -- 172.21.15.32:0/3703172588 wait complete. 2024-04-30T08:22:45.017 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.942+0000 7f518e000700 1 Processor -- start 2024-04-30T08:22:45.017 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.942+0000 7f518e000700 1 -- start start 2024-04-30T08:22:45.017 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.942+0000 7f518e000700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5188109c80 0x7f51881a01e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:45.017 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.942+0000 7f518e000700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f518810dfe0 con 0x7f5188109c80 2024-04-30T08:22:45.017 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.943+0000 7f5187400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5188109c80 0x7f51881a01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:22:45.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.943+0000 7f5187400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5188109c80 0x7f51881a01e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33642/0 (socket says 172.21.15.32:33642) 2024-04-30T08:22:45.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.943+0000 7f5187400700 1 -- 172.21.15.32:0/2377468883 learned_addr learned my addr 172.21.15.32:0/2377468883 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:22:45.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.943+0000 7f5187400700 1 -- 172.21.15.32:0/2377468883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51881a0720 con 0x7f5188109c80 2024-04-30T08:22:45.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.944+0000 7f5187400700 1 --2- 172.21.15.32:0/2377468883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5188109c80 0x7f51881a01e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5178009fb0 tx=0x7f5178011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:22:45.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.944+0000 7f517fe00700 1 -- 172.21.15.32:0/2377468883 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f51780036a0 con 0x7f5188109c80 2024-04-30T08:22:45.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.944+0000 7f517fe00700 1 -- 172.21.15.32:0/2377468883 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5178003cd0 con 0x7f5188109c80 2024-04-30T08:22:45.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.944+0000 7f518e000700 1 -- 172.21.15.32:0/2377468883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51881a09b0 con 0x7f5188109c80 2024-04-30T08:22:45.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.944+0000 7f517fe00700 1 -- 172.21.15.32:0/2377468883 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f517801a430 con 0x7f5188109c80 2024-04-30T08:22:45.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.944+0000 7f518e000700 1 -- 172.21.15.32:0/2377468883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51881a35c0 con 0x7f5188109c80 2024-04-30T08:22:45.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.945+0000 7f518e000700 1 -- 172.21.15.32:0/2377468883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5188064d70 con 0x7f5188109c80 2024-04-30T08:22:45.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.945+0000 7f517fe00700 1 -- 172.21.15.32:0/2377468883 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 4) ==== 49505+0+0 (secure 0 0 0) 0x7f5178003840 con 0x7f5188109c80 2024-04-30T08:22:45.019 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.946+0000 7f517fe00700 1 --2- 172.21.15.32:0/2377468883 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f516c03d520 0x7f516c03fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:45.019 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.946+0000 7f517fe00700 1 -- 172.21.15.32:0/2377468883 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5178052340 con 0x7f5188109c80 2024-04-30T08:22:45.019 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.946+0000 7f5186a00700 1 --2- 172.21.15.32:0/2377468883 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f516c03d520 0x7f516c03fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:22:45.019 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.947+0000 7f5186a00700 1 --2- 172.21.15.32:0/2377468883 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f516c03d520 0x7f516c03fa30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5174006fd0 tx=0x7f5174006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:22:45.019 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:43.951+0000 7f517fe00700 1 -- 172.21.15.32:0/2377468883 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5178024440 con 0x7f5188109c80 2024-04-30T08:22:45.019 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:44.245+0000 7f518e000700 1 -- 172.21.15.32:0/2377468883 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f51881a3c40 con 0x7f5188109c80 2024-04-30T08:22:45.019 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:44.299+0000 7f517fe00700 1 -- 172.21.15.32:0/2377468883 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f5178051d60 con 0x7f5188109c80 2024-04-30T08:22:45.019 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:44.302+0000 7f517fe00700 1 -- 172.21.15.32:0/2377468883 <== mon.0 v2:172.21.15.32:3300/0 8 ==== mgrmap(e 5) ==== 49516+0+0 (secure 0 0 0) 0x7f5178018a40 con 0x7f5188109c80 2024-04-30T08:22:45.019 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:44.303+0000 7f518e000700 1 -- 172.21.15.32:0/2377468883 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f516c03d520 msgr2=0x7f516c03fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:22:45.019 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:44.303+0000 7f518e000700 1 --2- 172.21.15.32:0/2377468883 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f516c03d520 0x7f516c03fa30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5174006fd0 tx=0x7f5174006e40 comp rx=0 tx=0).stop 2024-04-30T08:22:45.019 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:44.303+0000 7f518e000700 1 -- 172.21.15.32:0/2377468883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5188109c80 msgr2=0x7f51881a01e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:22:45.020 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:44.303+0000 7f518e000700 1 --2- 172.21.15.32:0/2377468883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5188109c80 0x7f51881a01e0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5178009fb0 tx=0x7f5178011890 comp rx=0 tx=0).stop 2024-04-30T08:22:45.020 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:44.304+0000 7f518e000700 1 -- 172.21.15.32:0/2377468883 shutdown_connections 2024-04-30T08:22:45.020 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:44.304+0000 7f518e000700 1 --2- 172.21.15.32:0/2377468883 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f516c03d520 0x7f516c03fa30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:45.020 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:44.304+0000 7f518e000700 1 --2- 172.21.15.32:0/2377468883 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5188109c80 0x7f51881a01e0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:45.020 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:44.304+0000 7f518e000700 1 -- 172.21.15.32:0/2377468883 >> 172.21.15.32:0/2377468883 conn(0x7f5188105300 msgr2=0x7f518807d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:22:45.020 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:44.304+0000 7f518e000700 1 -- 172.21.15.32:0/2377468883 shutdown_connections 2024-04-30T08:22:45.020 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:44.305+0000 7f518e000700 1 -- 172.21.15.32:0/2377468883 wait complete. 2024-04-30T08:22:45.308 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:44 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:44.950+0000 7fe058216200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-30T08:22:45.308 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:45 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:45.093+0000 7fe058216200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-30T08:22:45.635 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:45 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:45.508+0000 7fe058216200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-30T08:22:45.635 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:45 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2377468883' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-30T08:22:45.635 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:45 smithi032 ceph-mon[28331]: mgrmap e5: x(active, since 5s) 2024-04-30T08:22:45.894 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:45 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:45.634+0000 7fe058216200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-30T08:22:46.587 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:46 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1003516609' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-30T08:22:46.602 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-30T08:22:46.602 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-30T08:22:46.602 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-30T08:22:46.602 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-30T08:22:46.602 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-30T08:22:46.603 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-30T08:22:46.603 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.684+0000 7f6a5e800700 1 Processor -- start 2024-04-30T08:22:46.603 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.684+0000 7f6a5e800700 1 -- start start 2024-04-30T08:22:46.603 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.684+0000 7f6a5e800700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a5807ed50 0x7f6a5807d270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:46.603 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.684+0000 7f6a5e800700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6a5807d7b0 con 0x7f6a5807ed50 2024-04-30T08:22:46.603 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.685+0000 7f6a5d400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a5807ed50 0x7f6a5807d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:22:46.603 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.685+0000 7f6a5d400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a5807ed50 0x7f6a5807d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33684/0 (socket says 172.21.15.32:33684) 2024-04-30T08:22:46.604 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.685+0000 7f6a5d400700 1 -- 172.21.15.32:0/393695250 learned_addr learned my addr 172.21.15.32:0/393695250 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:22:46.604 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.685+0000 7f6a5d400700 1 -- 172.21.15.32:0/393695250 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a5807d930 con 0x7f6a5807ed50 2024-04-30T08:22:46.604 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.686+0000 7f6a5d400700 1 --2- 172.21.15.32:0/393695250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a5807ed50 0x7f6a5807d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c009e80 tx=0x7f6a4c00b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5205a00196825faa server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:22:46.604 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.686+0000 7f6a57e00700 1 -- 172.21.15.32:0/393695250 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6a4c017070 con 0x7f6a5807ed50 2024-04-30T08:22:46.605 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.686+0000 7f6a57e00700 1 -- 172.21.15.32:0/393695250 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6a4c00b8c0 con 0x7f6a5807ed50 2024-04-30T08:22:46.605 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.686+0000 7f6a57e00700 1 -- 172.21.15.32:0/393695250 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6a4c004490 con 0x7f6a5807ed50 2024-04-30T08:22:46.605 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.687+0000 7f6a5e800700 1 -- 172.21.15.32:0/393695250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a5807ed50 msgr2=0x7f6a5807d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:22:46.605 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.687+0000 7f6a5e800700 1 --2- 172.21.15.32:0/393695250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a5807ed50 0x7f6a5807d270 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c009e80 tx=0x7f6a4c00b330 comp rx=0 tx=0).stop 2024-04-30T08:22:46.605 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.687+0000 7f6a5e800700 1 -- 172.21.15.32:0/393695250 shutdown_connections 2024-04-30T08:22:46.605 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.687+0000 7f6a5e800700 1 --2- 172.21.15.32:0/393695250 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a5807ed50 0x7f6a5807d270 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:46.605 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.687+0000 7f6a5e800700 1 -- 172.21.15.32:0/393695250 >> 172.21.15.32:0/393695250 conn(0x7f6a581050d0 msgr2=0x7f6a581075a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:22:46.605 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.687+0000 7f6a5e800700 1 -- 172.21.15.32:0/393695250 shutdown_connections 2024-04-30T08:22:46.605 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.687+0000 7f6a5e800700 1 -- 172.21.15.32:0/393695250 wait complete. 2024-04-30T08:22:46.605 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.688+0000 7f6a5e800700 1 Processor -- start 2024-04-30T08:22:46.605 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.688+0000 7f6a5e800700 1 -- start start 2024-04-30T08:22:46.606 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.689+0000 7f6a5e800700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a5807ed50 0x7f6a5819ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:46.606 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.689+0000 7f6a5e800700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6a5810fff0 con 0x7f6a5807ed50 2024-04-30T08:22:46.606 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.689+0000 7f6a5d400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a5807ed50 0x7f6a5819ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:22:46.606 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.689+0000 7f6a5d400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a5807ed50 0x7f6a5819ffc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:33696/0 (socket says 172.21.15.32:33696) 2024-04-30T08:22:46.606 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.689+0000 7f6a5d400700 1 -- 172.21.15.32:0/1003516609 learned_addr learned my addr 172.21.15.32:0/1003516609 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:22:46.606 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.689+0000 7f6a5d400700 1 -- 172.21.15.32:0/1003516609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a581a0500 con 0x7f6a5807ed50 2024-04-30T08:22:46.606 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.690+0000 7f6a5d400700 1 --2- 172.21.15.32:0/1003516609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a5807ed50 0x7f6a5819ffc0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c00b7b0 tx=0x7f6a4c00bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:22:46.606 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.690+0000 7f6a56000700 1 -- 172.21.15.32:0/1003516609 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6a4c020020 con 0x7f6a5807ed50 2024-04-30T08:22:46.606 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.690+0000 7f6a56000700 1 -- 172.21.15.32:0/1003516609 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6a4c012430 con 0x7f6a5807ed50 2024-04-30T08:22:46.607 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.690+0000 7f6a56000700 1 -- 172.21.15.32:0/1003516609 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6a4c017040 con 0x7f6a5807ed50 2024-04-30T08:22:46.607 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.690+0000 7f6a5e800700 1 -- 172.21.15.32:0/1003516609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a581a0790 con 0x7f6a5807ed50 2024-04-30T08:22:46.607 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.690+0000 7f6a5e800700 1 -- 172.21.15.32:0/1003516609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a581a33a0 con 0x7f6a5807ed50 2024-04-30T08:22:46.607 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.691+0000 7f6a5e800700 1 -- 172.21.15.32:0/1003516609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a58064d70 con 0x7f6a5807ed50 2024-04-30T08:22:46.607 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.692+0000 7f6a56000700 1 -- 172.21.15.32:0/1003516609 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 49516+0+0 (secure 0 0 0) 0x7f6a4c023020 con 0x7f6a5807ed50 2024-04-30T08:22:46.607 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.692+0000 7f6a56000700 1 --2- 172.21.15.32:0/1003516609 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f6a4003d570 0x7f6a4003fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:46.607 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.692+0000 7f6a56000700 1 -- 172.21.15.32:0/1003516609 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f6a4c00f050 con 0x7f6a5807ed50 2024-04-30T08:22:46.607 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.692+0000 7f6a5ca00700 1 -- 172.21.15.32:0/1003516609 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f6a4003d570 msgr2=0x7f6a4003fa80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/464872591 2024-04-30T08:22:46.607 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.692+0000 7f6a5ca00700 1 --2- 172.21.15.32:0/1003516609 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f6a4003d570 0x7f6a4003fa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-30T08:22:46.607 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.698+0000 7f6a56000700 1 -- 172.21.15.32:0/1003516609 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6a4c056070 con 0x7f6a5807ed50 2024-04-30T08:22:46.608 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.893+0000 7f6a5ca00700 1 -- 172.21.15.32:0/1003516609 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f6a4003d570 msgr2=0x7f6a4003fa80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/464872591 2024-04-30T08:22:46.608 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.893+0000 7f6a5ca00700 1 --2- 172.21.15.32:0/1003516609 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f6a4003d570 0x7f6a4003fa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-30T08:22:46.608 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.982+0000 7f6a5e800700 1 -- 172.21.15.32:0/1003516609 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f6a581a3a70 con 0x7f6a5807ed50 2024-04-30T08:22:46.608 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.983+0000 7f6a56000700 1 -- 172.21.15.32:0/1003516609 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f6a4c02a020 con 0x7f6a5807ed50 2024-04-30T08:22:46.608 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.986+0000 7f6a5e800700 1 -- 172.21.15.32:0/1003516609 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f6a4003d570 msgr2=0x7f6a4003fa80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-30T08:22:46.608 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.986+0000 7f6a5e800700 1 --2- 172.21.15.32:0/1003516609 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f6a4003d570 0x7f6a4003fa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:46.608 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.986+0000 7f6a5e800700 1 -- 172.21.15.32:0/1003516609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a5807ed50 msgr2=0x7f6a5819ffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:22:46.608 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.986+0000 7f6a5e800700 1 --2- 172.21.15.32:0/1003516609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a5807ed50 0x7f6a5819ffc0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f6a4c00b7b0 tx=0x7f6a4c00bd50 comp rx=0 tx=0).stop 2024-04-30T08:22:46.609 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.987+0000 7f6a5e800700 1 -- 172.21.15.32:0/1003516609 shutdown_connections 2024-04-30T08:22:46.609 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.987+0000 7f6a5e800700 1 --2- 172.21.15.32:0/1003516609 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f6a4003d570 0x7f6a4003fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:46.609 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.987+0000 7f6a5e800700 1 --2- 172.21.15.32:0/1003516609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6a5807ed50 0x7f6a5819ffc0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:46.609 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.987+0000 7f6a5e800700 1 -- 172.21.15.32:0/1003516609 >> 172.21.15.32:0/1003516609 conn(0x7f6a581050d0 msgr2=0x7f6a58105a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:22:46.609 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.987+0000 7f6a5e800700 1 -- 172.21.15.32:0/1003516609 shutdown_connections 2024-04-30T08:22:46.609 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:45.987+0000 7f6a5e800700 1 -- 172.21.15.32:0/1003516609 wait complete. 2024-04-30T08:22:46.609 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for the mgr to restart... 2024-04-30T08:22:46.609 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mgr epoch 5... 2024-04-30T08:22:47.691 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:47 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:47.253+0000 7fe058216200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-30T08:22:47.691 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:47 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:47.493+0000 7fe058216200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-30T08:22:48.115 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:47 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:47.716+0000 7fe058216200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-30T08:22:48.441 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:48 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:48.113+0000 7fe058216200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-30T08:22:48.441 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:48 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:48.240+0000 7fe058216200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-30T08:22:48.763 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:48 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:48.499+0000 7fe058216200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-30T08:22:48.764 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:48 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:48.762+0000 7fe058216200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-30T08:22:49.941 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:49 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:49.579+0000 7fe058216200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-30T08:22:50.441 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:50 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:50.160+0000 7fe058216200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-30T08:22:50.441 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:50 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:50.291+0000 7fe058216200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-30T08:22:50.941 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:50 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:50.568+0000 7fe058216200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-30T08:22:51.691 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:51 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:51.237+0000 7fe058216200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-30T08:22:51.691 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:51 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:51.367+0000 7fe058216200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-30T08:22:52.020 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:51 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:51.702+0000 7fe058216200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-30T08:22:52.020 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:51 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:51.888+0000 7fe058216200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-30T08:22:52.020 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:52 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:52.019+0000 7fe058216200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-30T08:22:52.441 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:52 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:52.177+0000 7fe058216200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-30T08:22:52.941 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:52 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:52.554+0000 7fe058216200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-30T08:22:53.637 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:53 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:53.340+0000 7fe058216200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-30T08:22:53.941 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:22:53 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:22:53.635+0000 7fe058216200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-30T08:22:53.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:53 smithi032 ceph-mon[28331]: Active manager daemon x restarted 2024-04-30T08:22:53.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:53 smithi032 ceph-mon[28331]: Activating manager daemon x 2024-04-30T08:22:53.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:53 smithi032 ceph-mon[28331]: osdmap e2: 0 total, 0 up, 0 in 2024-04-30T08:22:53.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:53 smithi032 ceph-mon[28331]: mgrmap e6: x(active, starting, since 0.0114009s) 2024-04-30T08:22:53.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:53 smithi032 ceph-mon[28331]: from='mgr.14120 172.21.15.32:0/1220014720' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-30T08:22:53.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:53 smithi032 ceph-mon[28331]: from='mgr.14120 172.21.15.32:0/1220014720' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-30T08:22:53.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:53 smithi032 ceph-mon[28331]: from='mgr.14120 172.21.15.32:0/1220014720' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-30T08:22:53.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:53 smithi032 ceph-mon[28331]: from='mgr.14120 172.21.15.32:0/1220014720' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-30T08:22:53.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:53 smithi032 ceph-mon[28331]: from='mgr.14120 172.21.15.32:0/1220014720' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-30T08:22:53.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:53 smithi032 ceph-mon[28331]: Manager daemon x is now available 2024-04-30T08:22:54.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:54 smithi032 ceph-mon[28331]: Found migration_current of "None". Setting to last migration. 2024-04-30T08:22:54.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:54 smithi032 ceph-mon[28331]: from='mgr.14120 172.21.15.32:0/1220014720' entity='mgr.x' 2024-04-30T08:22:54.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:54 smithi032 ceph-mon[28331]: from='mgr.14120 172.21.15.32:0/1220014720' entity='mgr.x' 2024-04-30T08:22:54.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:54 smithi032 ceph-mon[28331]: from='mgr.14120 172.21.15.32:0/1220014720' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:22:54.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:54 smithi032 ceph-mon[28331]: from='mgr.14120 172.21.15.32:0/1220014720' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:22:54.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:54 smithi032 ceph-mon[28331]: from='mgr.14120 172.21.15.32:0/1220014720' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-30T08:22:54.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:54 smithi032 ceph-mon[28331]: from='mgr.14120 172.21.15.32:0/1220014720' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-30T08:22:54.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:54 smithi032 ceph-mon[28331]: from='mgr.14120 172.21.15.32:0/1220014720' entity='mgr.x' 2024-04-30T08:22:54.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:54 smithi032 ceph-mon[28331]: from='mgr.14120 172.21.15.32:0/1220014720' entity='mgr.x' 2024-04-30T08:22:54.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:54 smithi032 ceph-mon[28331]: mgrmap e7: x(active, since 1.02033s) 2024-04-30T08:22:55.283 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-30T08:22:55.283 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-30T08:22:55.283 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-30T08:22:55.283 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-30T08:22:55.284 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.255+0000 7f304ce00700 1 Processor -- start 2024-04-30T08:22:55.284 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.255+0000 7f304ce00700 1 -- start start 2024-04-30T08:22:55.284 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.255+0000 7f304ce00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f304807ed10 0x7f304807d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:55.284 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.255+0000 7f304ce00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f304807d770 con 0x7f304807ed10 2024-04-30T08:22:55.284 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.256+0000 7f3047400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f304807ed10 0x7f304807d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:22:55.284 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.256+0000 7f3047400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f304807ed10 0x7f304807d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35986/0 (socket says 172.21.15.32:35986) 2024-04-30T08:22:55.284 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.256+0000 7f3047400700 1 -- 172.21.15.32:0/1974184087 learned_addr learned my addr 172.21.15.32:0/1974184087 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:22:55.284 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.256+0000 7f3047400700 1 -- 172.21.15.32:0/1974184087 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f304807d8f0 con 0x7f304807ed10 2024-04-30T08:22:55.290 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.257+0000 7f3047400700 1 --2- 172.21.15.32:0/1974184087 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f304807ed10 0x7f304807d230 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f30380098a0 tx=0x7f3038009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56663c2fca5b5519 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:22:55.290 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.257+0000 7f3046000700 1 -- 172.21.15.32:0/1974184087 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3038004160 con 0x7f304807ed10 2024-04-30T08:22:55.290 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.257+0000 7f3046000700 1 -- 172.21.15.32:0/1974184087 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3038004300 con 0x7f304807ed10 2024-04-30T08:22:55.290 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.257+0000 7f3046000700 1 -- 172.21.15.32:0/1974184087 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3038011af0 con 0x7f304807ed10 2024-04-30T08:22:55.291 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.258+0000 7f304ce00700 1 -- 172.21.15.32:0/1974184087 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f304807ed10 msgr2=0x7f304807d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:22:55.291 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.258+0000 7f304ce00700 1 --2- 172.21.15.32:0/1974184087 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f304807ed10 0x7f304807d230 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f30380098a0 tx=0x7f3038009bb0 comp rx=0 tx=0).stop 2024-04-30T08:22:55.291 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.258+0000 7f304ce00700 1 -- 172.21.15.32:0/1974184087 shutdown_connections 2024-04-30T08:22:55.291 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.258+0000 7f304ce00700 1 --2- 172.21.15.32:0/1974184087 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f304807ed10 0x7f304807d230 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:55.291 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.258+0000 7f304ce00700 1 -- 172.21.15.32:0/1974184087 >> 172.21.15.32:0/1974184087 conn(0x7f30481050d0 msgr2=0x7f30481075a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:22:55.291 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.259+0000 7f304ce00700 1 -- 172.21.15.32:0/1974184087 shutdown_connections 2024-04-30T08:22:55.291 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.259+0000 7f304ce00700 1 -- 172.21.15.32:0/1974184087 wait complete. 2024-04-30T08:22:55.291 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.260+0000 7f304ce00700 1 Processor -- start 2024-04-30T08:22:55.291 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.260+0000 7f304ce00700 1 -- start start 2024-04-30T08:22:55.291 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.260+0000 7f304ce00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f304807ed10 0x7f30481a0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:55.291 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.260+0000 7f304ce00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f304810ffd0 con 0x7f304807ed10 2024-04-30T08:22:55.291 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.260+0000 7f3047400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f304807ed10 0x7f30481a0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:22:55.291 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.260+0000 7f3047400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f304807ed10 0x7f30481a0010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36000/0 (socket says 172.21.15.32:36000) 2024-04-30T08:22:55.292 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.260+0000 7f3047400700 1 -- 172.21.15.32:0/3952884431 learned_addr learned my addr 172.21.15.32:0/3952884431 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:22:55.292 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.261+0000 7f3047400700 1 -- 172.21.15.32:0/3952884431 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30481a0550 con 0x7f304807ed10 2024-04-30T08:22:55.292 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.261+0000 7f3047400700 1 --2- 172.21.15.32:0/3952884431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f304807ed10 0x7f30481a0010 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f30380117b0 tx=0x7f3038011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:22:55.292 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.261+0000 7f303fe00700 1 -- 172.21.15.32:0/3952884431 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f30380039c0 con 0x7f304807ed10 2024-04-30T08:22:55.292 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.262+0000 7f303fe00700 1 -- 172.21.15.32:0/3952884431 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f3038003b60 con 0x7f304807ed10 2024-04-30T08:22:55.292 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.262+0000 7f304ce00700 1 -- 172.21.15.32:0/3952884431 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30481a07e0 con 0x7f304807ed10 2024-04-30T08:22:55.292 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.262+0000 7f303fe00700 1 -- 172.21.15.32:0/3952884431 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f30380196a0 con 0x7f304807ed10 2024-04-30T08:22:55.292 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.262+0000 7f304ce00700 1 -- 172.21.15.32:0/3952884431 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30481a33f0 con 0x7f304807ed10 2024-04-30T08:22:55.292 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.263+0000 7f303fe00700 1 -- 172.21.15.32:0/3952884431 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 5) ==== 49516+0+0 (secure 0 0 0) 0x7f3038019910 con 0x7f304807ed10 2024-04-30T08:22:55.292 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.263+0000 7f303fe00700 1 --2- 172.21.15.32:0/3952884431 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f302c03d5c0 0x7f302c03fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:55.292 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.263+0000 7f303fe00700 1 -- 172.21.15.32:0/3952884431 --> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f302c0401e0 con 0x7f302c03d5c0 2024-04-30T08:22:55.292 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.264+0000 7f303fe00700 1 -- 172.21.15.32:0/3952884431 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f3038052900 con 0x7f304807ed10 2024-04-30T08:22:55.292 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.264+0000 7f3046a00700 1 -- 172.21.15.32:0/3952884431 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f302c03d5c0 msgr2=0x7f302c03fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/464872591 2024-04-30T08:22:55.293 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.264+0000 7f3046a00700 1 --2- 172.21.15.32:0/3952884431 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f302c03d5c0 0x7f302c03fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-30T08:22:55.293 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.465+0000 7f3046a00700 1 -- 172.21.15.32:0/3952884431 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f302c03d5c0 msgr2=0x7f302c03fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/464872591 2024-04-30T08:22:55.293 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.465+0000 7f3046a00700 1 --2- 172.21.15.32:0/3952884431 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f302c03d5c0 0x7f302c03fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-30T08:22:55.293 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.866+0000 7f3046a00700 1 -- 172.21.15.32:0/3952884431 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f302c03d5c0 msgr2=0x7f302c03fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/464872591 2024-04-30T08:22:55.293 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:47.866+0000 7f3046a00700 1 --2- 172.21.15.32:0/3952884431 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f302c03d5c0 0x7f302c03fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-30T08:22:55.293 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:48.667+0000 7f3046a00700 1 -- 172.21.15.32:0/3952884431 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f302c03d5c0 msgr2=0x7f302c03fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/464872591 2024-04-30T08:22:55.293 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:48.667+0000 7f3046a00700 1 --2- 172.21.15.32:0/3952884431 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f302c03d5c0 0x7f302c03fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-30T08:22:55.293 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:50.269+0000 7f3046a00700 1 -- 172.21.15.32:0/3952884431 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f302c03d5c0 msgr2=0x7f302c03fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/464872591 2024-04-30T08:22:55.293 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:50.269+0000 7f3046a00700 1 --2- 172.21.15.32:0/3952884431 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f302c03d5c0 0x7f302c03fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-30T08:22:55.293 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:53.473+0000 7f3046a00700 1 -- 172.21.15.32:0/3952884431 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f302c03d5c0 msgr2=0x7f302c03fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/464872591 2024-04-30T08:22:55.293 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:53.473+0000 7f3046a00700 1 --2- 172.21.15.32:0/3952884431 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f302c03d5c0 0x7f302c03fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-30T08:22:55.293 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:53.646+0000 7f303fe00700 1 -- 172.21.15.32:0/3952884431 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mgrmap(e 6) ==== 49283+0+0 (secure 0 0 0) 0x7f3038019c50 con 0x7f304807ed10 2024-04-30T08:22:55.293 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:53.646+0000 7f303fe00700 1 -- 172.21.15.32:0/3952884431 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f302c03d5c0 msgr2=0x7f302c03fad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-30T08:22:55.294 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:53.646+0000 7f303fe00700 1 --2- 172.21.15.32:0/3952884431 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f302c03d5c0 0x7f302c03fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:55.294 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:54.652+0000 7f303fe00700 1 -- 172.21.15.32:0/3952884431 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mgrmap(e 7) ==== 49410+0+0 (secure 0 0 0) 0x7f3038018350 con 0x7f304807ed10 2024-04-30T08:22:55.294 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:54.652+0000 7f303fe00700 1 --2- 172.21.15.32:0/3952884431 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f302c040f00 0x7f302c043320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:55.294 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:54.652+0000 7f303fe00700 1 -- 172.21.15.32:0/3952884431 --> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f303804b270 con 0x7f302c040f00 2024-04-30T08:22:55.294 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:54.655+0000 7f3046a00700 1 --2- 172.21.15.32:0/3952884431 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f302c040f00 0x7f302c043320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:22:55.294 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:54.658+0000 7f3046a00700 1 --2- 172.21.15.32:0/3952884431 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f302c040f00 0x7f302c043320 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f3034003a10 tx=0x7f30340092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:22:55.294 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:54.660+0000 7f303fe00700 1 -- 172.21.15.32:0/3952884431 <== mgr.14120 v2:172.21.15.32:6800/4019351306 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f303804b270 con 0x7f302c040f00 2024-04-30T08:22:55.294 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:54.670+0000 7f304ce00700 1 -- 172.21.15.32:0/3952884431 --> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f3048064d70 con 0x7f302c040f00 2024-04-30T08:22:55.294 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:54.670+0000 7f303fe00700 1 -- 172.21.15.32:0/3952884431 <== mgr.14120 v2:172.21.15.32:6800/4019351306 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f3048064d70 con 0x7f302c040f00 2024-04-30T08:22:55.295 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:54.671+0000 7f304ce00700 1 -- 172.21.15.32:0/3952884431 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f302c040f00 msgr2=0x7f302c043320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:22:55.295 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:54.671+0000 7f304ce00700 1 --2- 172.21.15.32:0/3952884431 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f302c040f00 0x7f302c043320 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f3034003a10 tx=0x7f30340092b0 comp rx=0 tx=0).stop 2024-04-30T08:22:55.295 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:54.671+0000 7f304ce00700 1 -- 172.21.15.32:0/3952884431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f304807ed10 msgr2=0x7f30481a0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:22:55.295 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:54.671+0000 7f304ce00700 1 --2- 172.21.15.32:0/3952884431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f304807ed10 0x7f30481a0010 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f30380117b0 tx=0x7f3038011b70 comp rx=0 tx=0).stop 2024-04-30T08:22:55.295 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:54.671+0000 7f304ce00700 1 -- 172.21.15.32:0/3952884431 shutdown_connections 2024-04-30T08:22:55.295 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:54.671+0000 7f304ce00700 1 --2- 172.21.15.32:0/3952884431 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f302c040f00 0x7f302c043320 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:55.295 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:54.671+0000 7f304ce00700 1 --2- 172.21.15.32:0/3952884431 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f304807ed10 0x7f30481a0010 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:55.295 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:54.671+0000 7f304ce00700 1 --2- 172.21.15.32:0/3952884431 >> [v2:172.21.15.32:6800/464872591,v1:172.21.15.32:6801/464872591] conn(0x7f302c03d5c0 0x7f302c03fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:55.296 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:54.671+0000 7f304ce00700 1 -- 172.21.15.32:0/3952884431 >> 172.21.15.32:0/3952884431 conn(0x7f30481050d0 msgr2=0x7f3048105a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:22:55.296 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:54.671+0000 7f304ce00700 1 -- 172.21.15.32:0/3952884431 shutdown_connections 2024-04-30T08:22:55.296 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:54.671+0000 7f304ce00700 1 -- 172.21.15.32:0/3952884431 wait complete. 2024-04-30T08:22:55.296 INFO:teuthology.orchestra.run.smithi032.stdout:mgr epoch 5 is available 2024-04-30T08:22:55.296 INFO:teuthology.orchestra.run.smithi032.stdout:Setting orchestrator backend to cephadm... 2024-04-30T08:22:55.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:55 smithi032 ceph-mon[28331]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-30T08:22:55.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:55 smithi032 ceph-mon[28331]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-30T08:22:55.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:55 smithi032 ceph-mon[28331]: [30/Apr/2024:08:22:54] ENGINE Bus STARTING 2024-04-30T08:22:55.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:55 smithi032 ceph-mon[28331]: [30/Apr/2024:08:22:54] ENGINE Serving on http://172.21.15.32:8765 2024-04-30T08:22:55.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:55 smithi032 ceph-mon[28331]: [30/Apr/2024:08:22:55] ENGINE Serving on https://172.21.15.32:7150 2024-04-30T08:22:55.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:55 smithi032 ceph-mon[28331]: [30/Apr/2024:08:22:55] ENGINE Bus STARTED 2024-04-30T08:22:55.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:55 smithi032 ceph-mon[28331]: from='mgr.14120 172.21.15.32:0/1220014720' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:22:56.933 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:56 smithi032 ceph-mon[28331]: mgrmap e8: x(active, since 2s) 2024-04-30T08:22:56.933 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:56 smithi032 ceph-mon[28331]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:22:56.933 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:56 smithi032 ceph-mon[28331]: from='mgr.14120 172.21.15.32:0/1220014720' entity='mgr.x' 2024-04-30T08:22:56.933 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:56 smithi032 ceph-mon[28331]: from='mgr.14120 172.21.15.32:0/1220014720' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:22:56.945 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.021+0000 7f6f34e00700 1 Processor -- start 2024-04-30T08:22:56.946 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.022+0000 7f6f34e00700 1 -- start start 2024-04-30T08:22:56.946 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.022+0000 7f6f34e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f30109cb0 0x7f6f3010a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:56.946 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.022+0000 7f6f34e00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6f3010a6b0 con 0x7f6f30109cb0 2024-04-30T08:22:56.947 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.023+0000 7f6f2de00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f30109cb0 0x7f6f3010a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:22:56.947 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.023+0000 7f6f2de00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f30109cb0 0x7f6f3010a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36068/0 (socket says 172.21.15.32:36068) 2024-04-30T08:22:56.947 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.023+0000 7f6f2de00700 1 -- 172.21.15.32:0/82523288 learned_addr learned my addr 172.21.15.32:0/82523288 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:22:56.947 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.023+0000 7f6f2de00700 1 -- 172.21.15.32:0/82523288 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f3010ade0 con 0x7f6f30109cb0 2024-04-30T08:22:56.947 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.024+0000 7f6f2de00700 1 --2- 172.21.15.32:0/82523288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f30109cb0 0x7f6f3010a0c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6f200098a0 tx=0x7f6f20009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81dd2315b2e27577 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:22:56.947 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.024+0000 7f6f2d400700 1 -- 172.21.15.32:0/82523288 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f20004160 con 0x7f6f30109cb0 2024-04-30T08:22:56.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.024+0000 7f6f2d400700 1 -- 172.21.15.32:0/82523288 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6f20004300 con 0x7f6f30109cb0 2024-04-30T08:22:56.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.024+0000 7f6f2d400700 1 -- 172.21.15.32:0/82523288 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f20011af0 con 0x7f6f30109cb0 2024-04-30T08:22:56.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.025+0000 7f6f34e00700 1 -- 172.21.15.32:0/82523288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f30109cb0 msgr2=0x7f6f3010a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:22:56.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.025+0000 7f6f34e00700 1 --2- 172.21.15.32:0/82523288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f30109cb0 0x7f6f3010a0c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f6f200098a0 tx=0x7f6f20009bb0 comp rx=0 tx=0).stop 2024-04-30T08:22:56.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.025+0000 7f6f34e00700 1 -- 172.21.15.32:0/82523288 shutdown_connections 2024-04-30T08:22:56.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.025+0000 7f6f34e00700 1 --2- 172.21.15.32:0/82523288 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f30109cb0 0x7f6f3010a0c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:56.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.025+0000 7f6f34e00700 1 -- 172.21.15.32:0/82523288 >> 172.21.15.32:0/82523288 conn(0x7f6f30105350 msgr2=0x7f6f301077a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:22:56.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.025+0000 7f6f34e00700 1 -- 172.21.15.32:0/82523288 shutdown_connections 2024-04-30T08:22:56.949 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.025+0000 7f6f34e00700 1 -- 172.21.15.32:0/82523288 wait complete. 2024-04-30T08:22:56.950 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.026+0000 7f6f34e00700 1 Processor -- start 2024-04-30T08:22:56.950 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.026+0000 7f6f34e00700 1 -- start start 2024-04-30T08:22:56.952 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.026+0000 7f6f34e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f30109cb0 0x7f6f3019e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:56.952 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.027+0000 7f6f34e00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6f3010e010 con 0x7f6f30109cb0 2024-04-30T08:22:56.952 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.027+0000 7f6f2de00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f30109cb0 0x7f6f3019e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:22:56.952 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.027+0000 7f6f2de00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f30109cb0 0x7f6f3019e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:36080/0 (socket says 172.21.15.32:36080) 2024-04-30T08:22:56.952 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.027+0000 7f6f2de00700 1 -- 172.21.15.32:0/2371668788 learned_addr learned my addr 172.21.15.32:0/2371668788 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:22:56.952 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.027+0000 7f6f2de00700 1 -- 172.21.15.32:0/2371668788 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f3019e580 con 0x7f6f30109cb0 2024-04-30T08:22:56.953 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.027+0000 7f6f2de00700 1 --2- 172.21.15.32:0/2371668788 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f30109cb0 0x7f6f3019e040 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6f200090e0 tx=0x7f6f20011940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:22:56.953 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.028+0000 7f6f26a00700 1 -- 172.21.15.32:0/2371668788 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f200039e0 con 0x7f6f30109cb0 2024-04-30T08:22:56.953 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.028+0000 7f6f26a00700 1 -- 172.21.15.32:0/2371668788 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f6f20003b80 con 0x7f6f30109cb0 2024-04-30T08:22:56.953 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.028+0000 7f6f26a00700 1 -- 172.21.15.32:0/2371668788 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6f20024690 con 0x7f6f30109cb0 2024-04-30T08:22:56.953 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.028+0000 7f6f34e00700 1 -- 172.21.15.32:0/2371668788 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f3019e810 con 0x7f6f30109cb0 2024-04-30T08:22:56.953 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.028+0000 7f6f34e00700 1 -- 172.21.15.32:0/2371668788 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f301a1420 con 0x7f6f30109cb0 2024-04-30T08:22:56.953 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.029+0000 7f6f34e00700 1 -- 172.21.15.32:0/2371668788 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f30064d70 con 0x7f6f30109cb0 2024-04-30T08:22:56.953 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.029+0000 7f6f26a00700 1 -- 172.21.15.32:0/2371668788 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f6f20022020 con 0x7f6f30109cb0 2024-04-30T08:22:56.953 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.030+0000 7f6f26a00700 1 --2- 172.21.15.32:0/2371668788 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f6f1403d5c0 0x7f6f1403fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:56.953 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.030+0000 7f6f26a00700 1 -- 172.21.15.32:0/2371668788 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f6f200525c0 con 0x7f6f30109cb0 2024-04-30T08:22:56.953 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.030+0000 7f6f27e00700 1 --2- 172.21.15.32:0/2371668788 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f6f1403d5c0 0x7f6f1403fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:22:56.955 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.031+0000 7f6f27e00700 1 --2- 172.21.15.32:0/2371668788 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f6f1403d5c0 0x7f6f1403fad0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6f18006fd0 tx=0x7f6f18006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:22:56.955 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.036+0000 7f6f26a00700 1 -- 172.21.15.32:0/2371668788 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6f20051f10 con 0x7f6f30109cb0 2024-04-30T08:22:56.955 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.287+0000 7f6f34e00700 1 -- 172.21.15.32:0/2371668788 --> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f6f30106420 con 0x7f6f1403d5c0 2024-04-30T08:22:56.956 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.298+0000 7f6f26a00700 1 -- 172.21.15.32:0/2371668788 <== mgr.14120 v2:172.21.15.32:6800/4019351306 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f6f30106420 con 0x7f6f1403d5c0 2024-04-30T08:22:56.956 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.301+0000 7f6f34e00700 1 -- 172.21.15.32:0/2371668788 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f6f1403d5c0 msgr2=0x7f6f1403fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:22:56.956 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.301+0000 7f6f34e00700 1 --2- 172.21.15.32:0/2371668788 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f6f1403d5c0 0x7f6f1403fad0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6f18006fd0 tx=0x7f6f18006e40 comp rx=0 tx=0).stop 2024-04-30T08:22:56.956 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.301+0000 7f6f34e00700 1 -- 172.21.15.32:0/2371668788 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f30109cb0 msgr2=0x7f6f3019e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:22:56.956 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.301+0000 7f6f34e00700 1 --2- 172.21.15.32:0/2371668788 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f30109cb0 0x7f6f3019e040 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f6f200090e0 tx=0x7f6f20011940 comp rx=0 tx=0).stop 2024-04-30T08:22:56.956 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.302+0000 7f6f34e00700 1 -- 172.21.15.32:0/2371668788 shutdown_connections 2024-04-30T08:22:56.956 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.302+0000 7f6f34e00700 1 --2- 172.21.15.32:0/2371668788 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f6f1403d5c0 0x7f6f1403fad0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:56.956 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.302+0000 7f6f34e00700 1 --2- 172.21.15.32:0/2371668788 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6f30109cb0 0x7f6f3019e040 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:56.956 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.302+0000 7f6f34e00700 1 -- 172.21.15.32:0/2371668788 >> 172.21.15.32:0/2371668788 conn(0x7f6f30105350 msgr2=0x7f6f30105cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:22:56.956 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.302+0000 7f6f34e00700 1 -- 172.21.15.32:0/2371668788 shutdown_connections 2024-04-30T08:22:56.956 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:56.302+0000 7f6f34e00700 1 -- 172.21.15.32:0/2371668788 wait complete. 2024-04-30T08:22:58.634 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-30T08:22:58.634 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.656+0000 7f5d49000700 1 Processor -- start 2024-04-30T08:22:58.635 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.657+0000 7f5d49000700 1 -- start start 2024-04-30T08:22:58.635 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.657+0000 7f5d49000700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d4407ed30 0x7f5d4407d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:58.635 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.657+0000 7f5d49000700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4407d790 con 0x7f5d4407ed30 2024-04-30T08:22:58.635 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.658+0000 7f5d43400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d4407ed30 0x7f5d4407d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:22:58.635 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.658+0000 7f5d43400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d4407ed30 0x7f5d4407d250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37774/0 (socket says 172.21.15.32:37774) 2024-04-30T08:22:58.635 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.658+0000 7f5d43400700 1 -- 172.21.15.32:0/1229516706 learned_addr learned my addr 172.21.15.32:0/1229516706 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:22:58.635 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.658+0000 7f5d43400700 1 -- 172.21.15.32:0/1229516706 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d4407d910 con 0x7f5d4407ed30 2024-04-30T08:22:58.635 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.658+0000 7f5d43400700 1 --2- 172.21.15.32:0/1229516706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d4407ed30 0x7f5d4407d250 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5d340098a0 tx=0x7f5d34009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d748de5c04dcc7e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:22:58.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.659+0000 7f5d42000700 1 -- 172.21.15.32:0/1229516706 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d34004160 con 0x7f5d4407ed30 2024-04-30T08:22:58.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.659+0000 7f5d42000700 1 -- 172.21.15.32:0/1229516706 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5d34004300 con 0x7f5d4407ed30 2024-04-30T08:22:58.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.659+0000 7f5d42000700 1 -- 172.21.15.32:0/1229516706 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d34011af0 con 0x7f5d4407ed30 2024-04-30T08:22:58.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.660+0000 7f5d49000700 1 -- 172.21.15.32:0/1229516706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d4407ed30 msgr2=0x7f5d4407d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:22:58.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.660+0000 7f5d49000700 1 --2- 172.21.15.32:0/1229516706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d4407ed30 0x7f5d4407d250 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f5d340098a0 tx=0x7f5d34009bb0 comp rx=0 tx=0).stop 2024-04-30T08:22:58.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.660+0000 7f5d49000700 1 -- 172.21.15.32:0/1229516706 shutdown_connections 2024-04-30T08:22:58.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.660+0000 7f5d49000700 1 --2- 172.21.15.32:0/1229516706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d4407ed30 0x7f5d4407d250 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:58.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.660+0000 7f5d49000700 1 -- 172.21.15.32:0/1229516706 >> 172.21.15.32:0/1229516706 conn(0x7f5d441050d0 msgr2=0x7f5d441075a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:22:58.636 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.660+0000 7f5d49000700 1 -- 172.21.15.32:0/1229516706 shutdown_connections 2024-04-30T08:22:58.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.660+0000 7f5d49000700 1 -- 172.21.15.32:0/1229516706 wait complete. 2024-04-30T08:22:58.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.661+0000 7f5d49000700 1 Processor -- start 2024-04-30T08:22:58.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.661+0000 7f5d49000700 1 -- start start 2024-04-30T08:22:58.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.662+0000 7f5d49000700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d4407ed30 0x7f5d441a0000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:58.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.662+0000 7f5d49000700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5d4410ffb0 con 0x7f5d4407ed30 2024-04-30T08:22:58.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.662+0000 7f5d43400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d4407ed30 0x7f5d441a0000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:22:58.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.662+0000 7f5d43400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d4407ed30 0x7f5d441a0000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37776/0 (socket says 172.21.15.32:37776) 2024-04-30T08:22:58.637 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.662+0000 7f5d43400700 1 -- 172.21.15.32:0/531875893 learned_addr learned my addr 172.21.15.32:0/531875893 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:22:58.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.662+0000 7f5d43400700 1 -- 172.21.15.32:0/531875893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d441a0540 con 0x7f5d4407ed30 2024-04-30T08:22:58.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.663+0000 7f5d43400700 1 --2- 172.21.15.32:0/531875893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d4407ed30 0x7f5d441a0000 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5d340117b0 tx=0x7f5d34011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:22:58.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.663+0000 7f5d3be00700 1 -- 172.21.15.32:0/531875893 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d340039c0 con 0x7f5d4407ed30 2024-04-30T08:22:58.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.663+0000 7f5d3be00700 1 -- 172.21.15.32:0/531875893 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5d34003b60 con 0x7f5d4407ed30 2024-04-30T08:22:58.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.663+0000 7f5d3be00700 1 -- 172.21.15.32:0/531875893 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d340196a0 con 0x7f5d4407ed30 2024-04-30T08:22:58.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.663+0000 7f5d49000700 1 -- 172.21.15.32:0/531875893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d441a07d0 con 0x7f5d4407ed30 2024-04-30T08:22:58.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.663+0000 7f5d49000700 1 -- 172.21.15.32:0/531875893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d441a33e0 con 0x7f5d4407ed30 2024-04-30T08:22:58.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.665+0000 7f5d3be00700 1 -- 172.21.15.32:0/531875893 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f5d3401a430 con 0x7f5d4407ed30 2024-04-30T08:22:58.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.665+0000 7f5d49000700 1 -- 172.21.15.32:0/531875893 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d44064d70 con 0x7f5d4407ed30 2024-04-30T08:22:58.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.665+0000 7f5d3be00700 1 --2- 172.21.15.32:0/531875893 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f5d2803d620 0x7f5d2803fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:22:58.638 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.665+0000 7f5d3be00700 1 -- 172.21.15.32:0/531875893 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f5d34052a90 con 0x7f5d4407ed30 2024-04-30T08:22:58.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.665+0000 7f5d42a00700 1 --2- 172.21.15.32:0/531875893 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f5d2803d620 0x7f5d2803fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:22:58.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.666+0000 7f5d42a00700 1 --2- 172.21.15.32:0/531875893 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f5d2803d620 0x7f5d2803fb30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5d30006fd0 tx=0x7f5d30006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:22:58.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.672+0000 7f5d3be00700 1 -- 172.21.15.32:0/531875893 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5d3400b3e0 con 0x7f5d4407ed30 2024-04-30T08:22:58.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.905+0000 7f5d49000700 1 -- 172.21.15.32:0/531875893 --> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f5d44106160 con 0x7f5d2803d620 2024-04-30T08:22:58.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.907+0000 7f5d3be00700 1 -- 172.21.15.32:0/531875893 <== mgr.14120 v2:172.21.15.32:6800/4019351306 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f5d44106160 con 0x7f5d2803d620 2024-04-30T08:22:58.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.909+0000 7f5d49000700 1 -- 172.21.15.32:0/531875893 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f5d2803d620 msgr2=0x7f5d2803fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:22:58.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.909+0000 7f5d49000700 1 --2- 172.21.15.32:0/531875893 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f5d2803d620 0x7f5d2803fb30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f5d30006fd0 tx=0x7f5d30006e40 comp rx=0 tx=0).stop 2024-04-30T08:22:58.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.909+0000 7f5d49000700 1 -- 172.21.15.32:0/531875893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d4407ed30 msgr2=0x7f5d441a0000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:22:58.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.909+0000 7f5d49000700 1 --2- 172.21.15.32:0/531875893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d4407ed30 0x7f5d441a0000 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f5d340117b0 tx=0x7f5d34011b70 comp rx=0 tx=0).stop 2024-04-30T08:22:58.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.909+0000 7f5d49000700 1 -- 172.21.15.32:0/531875893 shutdown_connections 2024-04-30T08:22:58.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.909+0000 7f5d49000700 1 --2- 172.21.15.32:0/531875893 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f5d2803d620 0x7f5d2803fb30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:58.639 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.909+0000 7f5d49000700 1 --2- 172.21.15.32:0/531875893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d4407ed30 0x7f5d441a0000 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:22:58.640 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.909+0000 7f5d49000700 1 -- 172.21.15.32:0/531875893 >> 172.21.15.32:0/531875893 conn(0x7f5d441050d0 msgr2=0x7f5d441059f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:22:58.640 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.910+0000 7f5d49000700 1 -- 172.21.15.32:0/531875893 shutdown_connections 2024-04-30T08:22:58.640 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:57.910+0000 7f5d49000700 1 -- 172.21.15.32:0/531875893 wait complete. 2024-04-30T08:22:58.640 INFO:teuthology.orchestra.run.smithi032.stdout:Generating ssh key... 2024-04-30T08:22:58.814 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:22:58 smithi032 ceph-mon[28331]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:23:00.909 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:00 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: Generating public/private rsa key pair. 2024-04-30T08:23:00.909 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:00 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: Your identification has been saved in /tmp/tmpbim8jzm1/key. 2024-04-30T08:23:00.909 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:00 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: Your public key has been saved in /tmp/tmpbim8jzm1/key.pub. 2024-04-30T08:23:00.909 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:00 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: The key fingerprint is: 2024-04-30T08:23:00.909 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:00 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: SHA256:HjOx+ImtkFsiAMiJsYca6FHkPlBgogykOyDzeILzxLA ceph-98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:23:00.909 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:00 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: The key's randomart image is: 2024-04-30T08:23:00.910 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:00 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: +---[RSA 3072]----+ 2024-04-30T08:23:00.910 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:00 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: |=+o+ | 2024-04-30T08:23:00.910 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:00 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: |%+= | 2024-04-30T08:23:00.910 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:00 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: |/*.. . | 2024-04-30T08:23:00.910 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:00 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: |B@+ . o | 2024-04-30T08:23:00.910 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:00 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: |E.=o . S | 2024-04-30T08:23:00.910 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:00 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: | O .. = = | 2024-04-30T08:23:00.910 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:00 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: | o + o = | 2024-04-30T08:23:00.910 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:00 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: | . = . | 2024-04-30T08:23:00.910 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:00 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: | . . | 2024-04-30T08:23:00.910 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:00 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: +----[SHA256]-----+ 2024-04-30T08:23:01.220 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.350+0000 7fa576800700 1 Processor -- start 2024-04-30T08:23:01.221 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.350+0000 7fa576800700 1 -- start start 2024-04-30T08:23:01.221 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.351+0000 7fa576800700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa57007cc60 0x7fa57010e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:01.221 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.351+0000 7fa576800700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa57010eef0 con 0x7fa57007cc60 2024-04-30T08:23:01.221 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.351+0000 7fa575400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa57007cc60 0x7fa57010e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:01.221 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.351+0000 7fa575400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa57007cc60 0x7fa57010e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37782/0 (socket says 172.21.15.32:37782) 2024-04-30T08:23:01.221 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.351+0000 7fa575400700 1 -- 172.21.15.32:0/3733005091 learned_addr learned my addr 172.21.15.32:0/3733005091 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:23:01.221 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.352+0000 7fa575400700 1 -- 172.21.15.32:0/3733005091 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa57010f070 con 0x7fa57007cc60 2024-04-30T08:23:01.223 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.352+0000 7fa575400700 1 --2- 172.21.15.32:0/3733005091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa57007cc60 0x7fa57010e9b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa5640098a0 tx=0x7fa564009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b95da3007f7d2af9 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:01.223 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.353+0000 7fa56fe00700 1 -- 172.21.15.32:0/3733005091 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa564004160 con 0x7fa57007cc60 2024-04-30T08:23:01.223 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.353+0000 7fa56fe00700 1 -- 172.21.15.32:0/3733005091 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa564004300 con 0x7fa57007cc60 2024-04-30T08:23:01.223 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.353+0000 7fa56fe00700 1 -- 172.21.15.32:0/3733005091 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa564011af0 con 0x7fa57007cc60 2024-04-30T08:23:01.223 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.354+0000 7fa576800700 1 -- 172.21.15.32:0/3733005091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa57007cc60 msgr2=0x7fa57010e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:01.223 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.354+0000 7fa576800700 1 --2- 172.21.15.32:0/3733005091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa57007cc60 0x7fa57010e9b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fa5640098a0 tx=0x7fa564009bb0 comp rx=0 tx=0).stop 2024-04-30T08:23:01.223 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.354+0000 7fa576800700 1 -- 172.21.15.32:0/3733005091 shutdown_connections 2024-04-30T08:23:01.223 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.354+0000 7fa576800700 1 --2- 172.21.15.32:0/3733005091 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa57007cc60 0x7fa57010e9b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:01.223 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.354+0000 7fa576800700 1 -- 172.21.15.32:0/3733005091 >> 172.21.15.32:0/3733005091 conn(0x7fa570103fa0 msgr2=0x7fa570106410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:01.223 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.354+0000 7fa576800700 1 -- 172.21.15.32:0/3733005091 shutdown_connections 2024-04-30T08:23:01.223 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.355+0000 7fa576800700 1 -- 172.21.15.32:0/3733005091 wait complete. 2024-04-30T08:23:01.223 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.356+0000 7fa576800700 1 Processor -- start 2024-04-30T08:23:01.225 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.356+0000 7fa576800700 1 -- start start 2024-04-30T08:23:01.225 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.356+0000 7fa576800700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa57007cc60 0x7fa57019ff60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:01.225 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.356+0000 7fa576800700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa57010ff80 con 0x7fa57007cc60 2024-04-30T08:23:01.225 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.356+0000 7fa575400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa57007cc60 0x7fa57019ff60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:01.225 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.356+0000 7fa575400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa57007cc60 0x7fa57019ff60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37794/0 (socket says 172.21.15.32:37794) 2024-04-30T08:23:01.226 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.356+0000 7fa575400700 1 -- 172.21.15.32:0/2639463111 learned_addr learned my addr 172.21.15.32:0/2639463111 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:23:01.226 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.357+0000 7fa575400700 1 -- 172.21.15.32:0/2639463111 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5701a04a0 con 0x7fa57007cc60 2024-04-30T08:23:01.226 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.357+0000 7fa575400700 1 --2- 172.21.15.32:0/2639463111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa57007cc60 0x7fa57019ff60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa5640117b0 tx=0x7fa564011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:01.226 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.357+0000 7fa56e000700 1 -- 172.21.15.32:0/2639463111 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa564003af0 con 0x7fa57007cc60 2024-04-30T08:23:01.226 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.358+0000 7fa56e000700 1 -- 172.21.15.32:0/2639463111 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7fa564020070 con 0x7fa57007cc60 2024-04-30T08:23:01.226 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.358+0000 7fa576800700 1 -- 172.21.15.32:0/2639463111 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5701a0730 con 0x7fa57007cc60 2024-04-30T08:23:01.226 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.358+0000 7fa56e000700 1 -- 172.21.15.32:0/2639463111 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa564019630 con 0x7fa57007cc60 2024-04-30T08:23:01.226 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.358+0000 7fa576800700 1 -- 172.21.15.32:0/2639463111 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5701a3340 con 0x7fa57007cc60 2024-04-30T08:23:01.226 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.359+0000 7fa576800700 1 -- 172.21.15.32:0/2639463111 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa570064d70 con 0x7fa57007cc60 2024-04-30T08:23:01.226 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.359+0000 7fa56e000700 1 -- 172.21.15.32:0/2639463111 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7fa56401a430 con 0x7fa57007cc60 2024-04-30T08:23:01.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.360+0000 7fa56e000700 1 --2- 172.21.15.32:0/2639463111 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7fa55803d5c0 0x7fa55803fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:01.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.360+0000 7fa56e000700 1 -- 172.21.15.32:0/2639463111 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa5640528a0 con 0x7fa57007cc60 2024-04-30T08:23:01.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.360+0000 7fa574a00700 1 --2- 172.21.15.32:0/2639463111 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7fa55803d5c0 0x7fa55803fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:01.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.361+0000 7fa574a00700 1 --2- 172.21.15.32:0/2639463111 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7fa55803d5c0 0x7fa55803fad0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa55c006fd0 tx=0x7fa55c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:01.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.365+0000 7fa56e000700 1 -- 172.21.15.32:0/2639463111 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa564021de0 con 0x7fa57007cc60 2024-04-30T08:23:01.228 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:22:59.593+0000 7fa576800700 1 -- 172.21.15.32:0/2639463111 --> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fa570105010 con 0x7fa55803d5c0 2024-04-30T08:23:01.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:00.514+0000 7fa56e000700 1 -- 172.21.15.32:0/2639463111 <== mgr.14120 v2:172.21.15.32:6800/4019351306 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fa570105010 con 0x7fa55803d5c0 2024-04-30T08:23:01.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:00.516+0000 7fa576800700 1 -- 172.21.15.32:0/2639463111 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7fa55803d5c0 msgr2=0x7fa55803fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:01.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:00.516+0000 7fa576800700 1 --2- 172.21.15.32:0/2639463111 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7fa55803d5c0 0x7fa55803fad0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fa55c006fd0 tx=0x7fa55c006e40 comp rx=0 tx=0).stop 2024-04-30T08:23:01.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:00.516+0000 7fa576800700 1 -- 172.21.15.32:0/2639463111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa57007cc60 msgr2=0x7fa57019ff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:01.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:00.516+0000 7fa576800700 1 --2- 172.21.15.32:0/2639463111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa57007cc60 0x7fa57019ff60 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fa5640117b0 tx=0x7fa564011b70 comp rx=0 tx=0).stop 2024-04-30T08:23:01.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:00.516+0000 7fa576800700 1 -- 172.21.15.32:0/2639463111 shutdown_connections 2024-04-30T08:23:01.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:00.516+0000 7fa576800700 1 --2- 172.21.15.32:0/2639463111 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7fa55803d5c0 0x7fa55803fad0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:01.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:00.516+0000 7fa576800700 1 --2- 172.21.15.32:0/2639463111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa57007cc60 0x7fa57019ff60 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:01.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:00.516+0000 7fa576800700 1 -- 172.21.15.32:0/2639463111 >> 172.21.15.32:0/2639463111 conn(0x7fa570103fa0 msgr2=0x7fa5701048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:01.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:00.517+0000 7fa576800700 1 -- 172.21.15.32:0/2639463111 shutdown_connections 2024-04-30T08:23:01.229 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:00.517+0000 7fa576800700 1 -- 172.21.15.32:0/2639463111 wait complete. 2024-04-30T08:23:01.755 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:01 smithi032 ceph-mon[28331]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:23:01.755 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:01 smithi032 ceph-mon[28331]: Generating ssh key... 2024-04-30T08:23:01.755 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:01 smithi032 ceph-mon[28331]: from='mgr.14120 172.21.15.32:0/1220014720' entity='mgr.x' 2024-04-30T08:23:01.756 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:01 smithi032 ceph-mon[28331]: from='mgr.14120 172.21.15.32:0/1220014720' entity='mgr.x' 2024-04-30T08:23:02.833 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:02 smithi032 ceph-mon[28331]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:23:02.843 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAebVzc5PNbSUQhRM83qqlc+1tN2IJZ3mtGdHIQzTNbXelQTm4K61cFqb2kCuL2ZS8TTReRVjyNDsYaJ8HLj4JQ2SclnBwIKshH2vXnqUKzACxDLFYe5ajSupZZCPJqF4+NUMF5ZATa5UQcPRJ+fxt//dHsJqYS2oN1aqT35RL+xL3KVQI4kjSKwndp4twbZNMXc8ABdplGqgkjKgfpWhz0mfEIwmUQOVxazLnH0UPSdQjKgAI/YbLRD28Lv0wnkPWwGRjcmrIVzQh43oqWSqSSuB4FI+HTfU/auTHSXdMDLmuf3nIB+FiC/f9lGW81Ko5nj3qM6uMy1eNHIj1XiuQj2dUHnDqYQJEt52JHMC7IFSyYDoLF2oUNNj6FVJTTV2JgTXzGk1XL6alaTG5qDm5DT4hDM/aSeg8fjOhxsbMe9pewleloEDy4ocg/N8lc5/eu9M3ad2cbE60FDjW0ibIAoCW7KIrq75d2FtK2ca3jUvlIYgi0IZ8Z3CFjntsltc= ceph-98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:23:02.845 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.914+0000 7f0070800700 1 Processor -- start 2024-04-30T08:23:02.845 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.914+0000 7f0070800700 1 -- start start 2024-04-30T08:23:02.845 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.915+0000 7f0070800700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f006807ed40 0x7f006807d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:02.846 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.915+0000 7f0070800700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f006807d750 con 0x7f006807ed40 2024-04-30T08:23:02.846 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.915+0000 7f006de00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f006807ed40 0x7f006807d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:02.846 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.915+0000 7f006de00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f006807ed40 0x7f006807d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37804/0 (socket says 172.21.15.32:37804) 2024-04-30T08:23:02.846 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.915+0000 7f006de00700 1 -- 172.21.15.32:0/4035716574 learned_addr learned my addr 172.21.15.32:0/4035716574 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:23:02.846 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.916+0000 7f006de00700 1 -- 172.21.15.32:0/4035716574 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f006807d8d0 con 0x7f006807ed40 2024-04-30T08:23:02.846 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.916+0000 7f006de00700 1 --2- 172.21.15.32:0/4035716574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f006807ed40 0x7f006807d210 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f005c0098a0 tx=0x7f005c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fabd6a34681d6402 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:02.846 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.916+0000 7f006ca00700 1 -- 172.21.15.32:0/4035716574 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f005c004160 con 0x7f006807ed40 2024-04-30T08:23:02.846 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.917+0000 7f006ca00700 1 -- 172.21.15.32:0/4035716574 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f005c004300 con 0x7f006807ed40 2024-04-30T08:23:02.846 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.917+0000 7f006ca00700 1 -- 172.21.15.32:0/4035716574 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f005c011af0 con 0x7f006807ed40 2024-04-30T08:23:02.846 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.917+0000 7f0070800700 1 -- 172.21.15.32:0/4035716574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f006807ed40 msgr2=0x7f006807d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:02.846 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.917+0000 7f0070800700 1 --2- 172.21.15.32:0/4035716574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f006807ed40 0x7f006807d210 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f005c0098a0 tx=0x7f005c009bb0 comp rx=0 tx=0).stop 2024-04-30T08:23:02.846 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.918+0000 7f0070800700 1 -- 172.21.15.32:0/4035716574 shutdown_connections 2024-04-30T08:23:02.847 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.918+0000 7f0070800700 1 --2- 172.21.15.32:0/4035716574 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f006807ed40 0x7f006807d210 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:02.850 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.918+0000 7f0070800700 1 -- 172.21.15.32:0/4035716574 >> 172.21.15.32:0/4035716574 conn(0x7f00681052f0 msgr2=0x7f0068107740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:02.850 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.918+0000 7f0070800700 1 -- 172.21.15.32:0/4035716574 shutdown_connections 2024-04-30T08:23:02.850 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.918+0000 7f0070800700 1 -- 172.21.15.32:0/4035716574 wait complete. 2024-04-30T08:23:02.850 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.919+0000 7f0070800700 1 Processor -- start 2024-04-30T08:23:02.851 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.919+0000 7f0070800700 1 -- start start 2024-04-30T08:23:02.851 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.919+0000 7f0070800700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f006807ed40 0x7f00681a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:02.851 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.919+0000 7f0070800700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0068110130 con 0x7f006807ed40 2024-04-30T08:23:02.851 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.920+0000 7f006de00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f006807ed40 0x7f00681a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:02.851 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.920+0000 7f006de00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f006807ed40 0x7f00681a0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37814/0 (socket says 172.21.15.32:37814) 2024-04-30T08:23:02.851 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.920+0000 7f006de00700 1 -- 172.21.15.32:0/2448696324 learned_addr learned my addr 172.21.15.32:0/2448696324 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:23:02.851 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.920+0000 7f006de00700 1 -- 172.21.15.32:0/2448696324 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00681a06c0 con 0x7f006807ed40 2024-04-30T08:23:02.851 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.921+0000 7f006de00700 1 --2- 172.21.15.32:0/2448696324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f006807ed40 0x7f00681a0180 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f005c0117b0 tx=0x7f005c011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:02.851 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.921+0000 7f0062a00700 1 -- 172.21.15.32:0/2448696324 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f005c003af0 con 0x7f006807ed40 2024-04-30T08:23:02.851 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.921+0000 7f0062a00700 1 -- 172.21.15.32:0/2448696324 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f005c020070 con 0x7f006807ed40 2024-04-30T08:23:02.851 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.921+0000 7f0070800700 1 -- 172.21.15.32:0/2448696324 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00681a0950 con 0x7f006807ed40 2024-04-30T08:23:02.851 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.921+0000 7f0062a00700 1 -- 172.21.15.32:0/2448696324 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f005c019630 con 0x7f006807ed40 2024-04-30T08:23:02.851 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.921+0000 7f0070800700 1 -- 172.21.15.32:0/2448696324 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00681a3560 con 0x7f006807ed40 2024-04-30T08:23:02.852 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.922+0000 7f0070800700 1 -- 172.21.15.32:0/2448696324 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0068064d70 con 0x7f006807ed40 2024-04-30T08:23:02.852 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.923+0000 7f0062a00700 1 -- 172.21.15.32:0/2448696324 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f005c01a430 con 0x7f006807ed40 2024-04-30T08:23:02.852 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.923+0000 7f0062a00700 1 --2- 172.21.15.32:0/2448696324 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f005003d5c0 0x7f005003fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:02.852 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.923+0000 7f0062a00700 1 -- 172.21.15.32:0/2448696324 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f005c0528d0 con 0x7f006807ed40 2024-04-30T08:23:02.852 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.924+0000 7f006d400700 1 --2- 172.21.15.32:0/2448696324 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f005003d5c0 0x7f005003fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:02.852 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.925+0000 7f006d400700 1 --2- 172.21.15.32:0/2448696324 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f005003d5c0 0x7f005003fad0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0058006fd0 tx=0x7f0058006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:02.852 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:01.931+0000 7f0062a00700 1 -- 172.21.15.32:0/2448696324 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f005c0042a0 con 0x7f006807ed40 2024-04-30T08:23:02.852 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:02.161+0000 7f0070800700 1 -- 172.21.15.32:0/2448696324 --> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7f0068107520 con 0x7f005003d5c0 2024-04-30T08:23:02.852 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:02.164+0000 7f0062a00700 1 -- 172.21.15.32:0/2448696324 <== mgr.14120 v2:172.21.15.32:6800/4019351306 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7f0068107520 con 0x7f005003d5c0 2024-04-30T08:23:02.852 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:02.167+0000 7f0070800700 1 -- 172.21.15.32:0/2448696324 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f005003d5c0 msgr2=0x7f005003fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:02.852 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:02.167+0000 7f0070800700 1 --2- 172.21.15.32:0/2448696324 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f005003d5c0 0x7f005003fad0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f0058006fd0 tx=0x7f0058006e40 comp rx=0 tx=0).stop 2024-04-30T08:23:02.852 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:02.167+0000 7f0070800700 1 -- 172.21.15.32:0/2448696324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f006807ed40 msgr2=0x7f00681a0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:02.852 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:02.167+0000 7f0070800700 1 --2- 172.21.15.32:0/2448696324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f006807ed40 0x7f00681a0180 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f005c0117b0 tx=0x7f005c011b70 comp rx=0 tx=0).stop 2024-04-30T08:23:02.853 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:02.168+0000 7f0070800700 1 -- 172.21.15.32:0/2448696324 shutdown_connections 2024-04-30T08:23:02.853 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:02.168+0000 7f0070800700 1 --2- 172.21.15.32:0/2448696324 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f005003d5c0 0x7f005003fad0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:02.853 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:02.168+0000 7f0070800700 1 --2- 172.21.15.32:0/2448696324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f006807ed40 0x7f00681a0180 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:02.853 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:02.168+0000 7f0070800700 1 -- 172.21.15.32:0/2448696324 >> 172.21.15.32:0/2448696324 conn(0x7f00681052f0 msgr2=0x7f0068106d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:02.853 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:02.168+0000 7f0070800700 1 -- 172.21.15.32:0/2448696324 shutdown_connections 2024-04-30T08:23:02.853 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:02.168+0000 7f0070800700 1 -- 172.21.15.32:0/2448696324 wait complete. 2024-04-30T08:23:02.854 INFO:teuthology.orchestra.run.smithi032.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-30T08:23:02.854 INFO:teuthology.orchestra.run.smithi032.stdout:Adding key to root@localhost authorized_keys... 2024-04-30T08:23:02.854 INFO:teuthology.orchestra.run.smithi032.stdout:Adding host smithi032... 2024-04-30T08:23:04.862 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:04 smithi032 ceph-mon[28331]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi032", "addr": "172.21.15.32", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:23:05.765 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:05 smithi032 ceph-mon[28331]: Deploying cephadm binary to smithi032 2024-04-30T08:23:10.737 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Added host 'smithi032' with addr '172.21.15.32' 2024-04-30T08:23:10.737 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.503+0000 7f1396000700 1 Processor -- start 2024-04-30T08:23:10.737 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.503+0000 7f1396000700 1 -- start start 2024-04-30T08:23:10.738 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.504+0000 7f1396000700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f139007cca0 0x7f139010e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:10.738 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.504+0000 7f1396000700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f139010eeb0 con 0x7f139007cca0 2024-04-30T08:23:10.738 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.504+0000 7f1394c00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f139007cca0 0x7f139010e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:10.738 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.504+0000 7f1394c00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f139007cca0 0x7f139010e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37820/0 (socket says 172.21.15.32:37820) 2024-04-30T08:23:10.738 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.504+0000 7f1394c00700 1 -- 172.21.15.32:0/1982171590 learned_addr learned my addr 172.21.15.32:0/1982171590 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:23:10.738 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.505+0000 7f1394c00700 1 -- 172.21.15.32:0/1982171590 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f139010f030 con 0x7f139007cca0 2024-04-30T08:23:10.738 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.505+0000 7f1394c00700 1 --2- 172.21.15.32:0/1982171590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f139007cca0 0x7f139010e970 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f13840098a0 tx=0x7f1384009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4356d60a66a2488f server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:10.738 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.505+0000 7f138f400700 1 -- 172.21.15.32:0/1982171590 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1384004160 con 0x7f139007cca0 2024-04-30T08:23:10.738 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.506+0000 7f138f400700 1 -- 172.21.15.32:0/1982171590 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1384004300 con 0x7f139007cca0 2024-04-30T08:23:10.738 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.506+0000 7f138f400700 1 -- 172.21.15.32:0/1982171590 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1384011af0 con 0x7f139007cca0 2024-04-30T08:23:10.738 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.506+0000 7f1396000700 1 -- 172.21.15.32:0/1982171590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f139007cca0 msgr2=0x7f139010e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:10.738 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.506+0000 7f1396000700 1 --2- 172.21.15.32:0/1982171590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f139007cca0 0x7f139010e970 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f13840098a0 tx=0x7f1384009bb0 comp rx=0 tx=0).stop 2024-04-30T08:23:10.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.507+0000 7f1396000700 1 -- 172.21.15.32:0/1982171590 shutdown_connections 2024-04-30T08:23:10.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.507+0000 7f1396000700 1 --2- 172.21.15.32:0/1982171590 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f139007cca0 0x7f139010e970 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:10.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.507+0000 7f1396000700 1 -- 172.21.15.32:0/1982171590 >> 172.21.15.32:0/1982171590 conn(0x7f1390103fa0 msgr2=0x7f1390106410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:10.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.507+0000 7f1396000700 1 -- 172.21.15.32:0/1982171590 shutdown_connections 2024-04-30T08:23:10.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.507+0000 7f1396000700 1 -- 172.21.15.32:0/1982171590 wait complete. 2024-04-30T08:23:10.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.508+0000 7f1396000700 1 Processor -- start 2024-04-30T08:23:10.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.508+0000 7f1396000700 1 -- start start 2024-04-30T08:23:10.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.508+0000 7f1396000700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f139007cca0 0x7f139019ff40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:10.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.508+0000 7f1396000700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f139010ff90 con 0x7f139007cca0 2024-04-30T08:23:10.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.509+0000 7f1394c00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f139007cca0 0x7f139019ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:10.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.509+0000 7f1394c00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f139007cca0 0x7f139019ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37824/0 (socket says 172.21.15.32:37824) 2024-04-30T08:23:10.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.509+0000 7f1394c00700 1 -- 172.21.15.32:0/3523967701 learned_addr learned my addr 172.21.15.32:0/3523967701 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:23:10.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.509+0000 7f1394c00700 1 -- 172.21.15.32:0/3523967701 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13901a0480 con 0x7f139007cca0 2024-04-30T08:23:10.739 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.510+0000 7f1394c00700 1 --2- 172.21.15.32:0/3523967701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f139007cca0 0x7f139019ff40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f13840117b0 tx=0x7f1384011b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:10.740 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.510+0000 7f138d600700 1 -- 172.21.15.32:0/3523967701 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1384003af0 con 0x7f139007cca0 2024-04-30T08:23:10.740 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.510+0000 7f138d600700 1 -- 172.21.15.32:0/3523967701 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f1384020070 con 0x7f139007cca0 2024-04-30T08:23:10.740 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.510+0000 7f138d600700 1 -- 172.21.15.32:0/3523967701 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1384019630 con 0x7f139007cca0 2024-04-30T08:23:10.740 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.510+0000 7f1396000700 1 -- 172.21.15.32:0/3523967701 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13901a0710 con 0x7f139007cca0 2024-04-30T08:23:10.740 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.510+0000 7f1396000700 1 -- 172.21.15.32:0/3523967701 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13901a3320 con 0x7f139007cca0 2024-04-30T08:23:10.740 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.511+0000 7f1396000700 1 -- 172.21.15.32:0/3523967701 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1390064d70 con 0x7f139007cca0 2024-04-30T08:23:10.740 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.511+0000 7f138d600700 1 -- 172.21.15.32:0/3523967701 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f138401a9c0 con 0x7f139007cca0 2024-04-30T08:23:10.741 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.512+0000 7f138d600700 1 --2- 172.21.15.32:0/3523967701 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f137803d5c0 0x7f137803fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:10.741 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.512+0000 7f138d600700 1 -- 172.21.15.32:0/3523967701 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f13840529a0 con 0x7f139007cca0 2024-04-30T08:23:10.741 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.512+0000 7f138fe00700 1 --2- 172.21.15.32:0/3523967701 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f137803d5c0 0x7f137803fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:10.741 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.513+0000 7f138fe00700 1 --2- 172.21.15.32:0/3523967701 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f137803d5c0 0x7f137803fad0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f137c006fd0 tx=0x7f137c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:10.741 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.518+0000 7f138d600700 1 -- 172.21.15.32:0/3523967701 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f13840172b0 con 0x7f139007cca0 2024-04-30T08:23:10.741 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:03.754+0000 7f1396000700 1 -- 172.21.15.32:0/3523967701 --> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi032", "addr": "172.21.15.32", "target": ["mon-mgr", ""]}) -- 0x7f1390104f20 con 0x7f137803d5c0 2024-04-30T08:23:10.742 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:10.004+0000 7f138d600700 1 -- 172.21.15.32:0/3523967701 <== mgr.14120 v2:172.21.15.32:6800/4019351306 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f1390104f20 con 0x7f137803d5c0 2024-04-30T08:23:10.742 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:10.008+0000 7f1396000700 1 -- 172.21.15.32:0/3523967701 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f137803d5c0 msgr2=0x7f137803fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:10.742 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:10.008+0000 7f1396000700 1 --2- 172.21.15.32:0/3523967701 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f137803d5c0 0x7f137803fad0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f137c006fd0 tx=0x7f137c006e40 comp rx=0 tx=0).stop 2024-04-30T08:23:10.742 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:10.008+0000 7f1396000700 1 -- 172.21.15.32:0/3523967701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f139007cca0 msgr2=0x7f139019ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:10.742 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:10.008+0000 7f1396000700 1 --2- 172.21.15.32:0/3523967701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f139007cca0 0x7f139019ff40 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f13840117b0 tx=0x7f1384011b70 comp rx=0 tx=0).stop 2024-04-30T08:23:10.742 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:10.008+0000 7f1396000700 1 -- 172.21.15.32:0/3523967701 shutdown_connections 2024-04-30T08:23:10.743 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:10.008+0000 7f1396000700 1 --2- 172.21.15.32:0/3523967701 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f137803d5c0 0x7f137803fad0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:10.743 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:10.008+0000 7f1396000700 1 --2- 172.21.15.32:0/3523967701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f139007cca0 0x7f139019ff40 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:10.743 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:10.008+0000 7f1396000700 1 -- 172.21.15.32:0/3523967701 >> 172.21.15.32:0/3523967701 conn(0x7f1390103fa0 msgr2=0x7f1390104810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:10.743 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:10.008+0000 7f1396000700 1 -- 172.21.15.32:0/3523967701 shutdown_connections 2024-04-30T08:23:10.743 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:10.008+0000 7f1396000700 1 -- 172.21.15.32:0/3523967701 wait complete. 2024-04-30T08:23:10.743 INFO:teuthology.orchestra.run.smithi032.stdout:Deploying unmanaged mon service... 2024-04-30T08:23:11.380 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:11 smithi032 ceph-mon[28331]: from='mgr.14120 172.21.15.32:0/1220014720' entity='mgr.x' 2024-04-30T08:23:11.380 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:11 smithi032 ceph-mon[28331]: Added host smithi032 2024-04-30T08:23:11.381 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:11 smithi032 ceph-mon[28331]: from='mgr.14120 172.21.15.32:0/1220014720' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:23:12.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:12 smithi032 ceph-mon[28331]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:23:12.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:12 smithi032 ceph-mon[28331]: Saving service mon spec with placement count:5 2024-04-30T08:23:12.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:12 smithi032 ceph-mon[28331]: from='mgr.14120 172.21.15.32:0/1220014720' entity='mgr.x' 2024-04-30T08:23:13.288 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-30T08:23:13.300 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.487+0000 7f5d96000700 1 Processor -- start 2024-04-30T08:23:13.300 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.488+0000 7f5d96000700 1 -- start start 2024-04-30T08:23:13.300 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.488+0000 7f5d96000700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d90110160 0x7f5d90110590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:13.300 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.488+0000 7f5d96000700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5d90110ba0 con 0x7f5d90110160 2024-04-30T08:23:13.300 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.489+0000 7f5d8f400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d90110160 0x7f5d90110590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:13.301 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.489+0000 7f5d8f400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d90110160 0x7f5d90110590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52110/0 (socket says 172.21.15.32:52110) 2024-04-30T08:23:13.301 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.489+0000 7f5d8f400700 1 -- 172.21.15.32:0/4229013238 learned_addr learned my addr 172.21.15.32:0/4229013238 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:23:13.302 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.489+0000 7f5d8f400700 1 -- 172.21.15.32:0/4229013238 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d90111300 con 0x7f5d90110160 2024-04-30T08:23:13.303 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.490+0000 7f5d8f400700 1 --2- 172.21.15.32:0/4229013238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d90110160 0x7f5d90110590 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5d800098a0 tx=0x7f5d80009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4131b13b460fb586 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:13.303 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.490+0000 7f5d8ea00700 1 -- 172.21.15.32:0/4229013238 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d80004160 con 0x7f5d90110160 2024-04-30T08:23:13.303 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.490+0000 7f5d8ea00700 1 -- 172.21.15.32:0/4229013238 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5d80004300 con 0x7f5d90110160 2024-04-30T08:23:13.303 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.490+0000 7f5d8ea00700 1 -- 172.21.15.32:0/4229013238 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d80011af0 con 0x7f5d90110160 2024-04-30T08:23:13.303 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.491+0000 7f5d96000700 1 -- 172.21.15.32:0/4229013238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d90110160 msgr2=0x7f5d90110590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:13.303 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.491+0000 7f5d96000700 1 --2- 172.21.15.32:0/4229013238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d90110160 0x7f5d90110590 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f5d800098a0 tx=0x7f5d80009bb0 comp rx=0 tx=0).stop 2024-04-30T08:23:13.303 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.491+0000 7f5d96000700 1 -- 172.21.15.32:0/4229013238 shutdown_connections 2024-04-30T08:23:13.303 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.491+0000 7f5d96000700 1 --2- 172.21.15.32:0/4229013238 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d90110160 0x7f5d90110590 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:13.304 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.491+0000 7f5d96000700 1 -- 172.21.15.32:0/4229013238 >> 172.21.15.32:0/4229013238 conn(0x7f5d9007f330 msgr2=0x7f5d900817e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:13.304 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.491+0000 7f5d96000700 1 -- 172.21.15.32:0/4229013238 shutdown_connections 2024-04-30T08:23:13.304 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.491+0000 7f5d96000700 1 -- 172.21.15.32:0/4229013238 wait complete. 2024-04-30T08:23:13.304 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.492+0000 7f5d96000700 1 Processor -- start 2024-04-30T08:23:13.304 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.492+0000 7f5d96000700 1 -- start start 2024-04-30T08:23:13.304 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.493+0000 7f5d96000700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d90110160 0x7f5d901a44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:13.304 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.493+0000 7f5d96000700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5d901145c0 con 0x7f5d90110160 2024-04-30T08:23:13.305 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.493+0000 7f5d8f400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d90110160 0x7f5d901a44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:13.305 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.493+0000 7f5d8f400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d90110160 0x7f5d901a44e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52116/0 (socket says 172.21.15.32:52116) 2024-04-30T08:23:13.305 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.493+0000 7f5d8f400700 1 -- 172.21.15.32:0/2097690978 learned_addr learned my addr 172.21.15.32:0/2097690978 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:23:13.305 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.493+0000 7f5d8f400700 1 -- 172.21.15.32:0/2097690978 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d901a4a20 con 0x7f5d90110160 2024-04-30T08:23:13.305 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.494+0000 7f5d8f400700 1 --2- 172.21.15.32:0/2097690978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d90110160 0x7f5d901a44e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5d80009fb0 tx=0x7f5d80011890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:13.305 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.494+0000 7f5d8cc00700 1 -- 172.21.15.32:0/2097690978 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d800036a0 con 0x7f5d90110160 2024-04-30T08:23:13.305 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.494+0000 7f5d8cc00700 1 -- 172.21.15.32:0/2097690978 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1070+0+0 (secure 0 0 0) 0x7f5d80024440 con 0x7f5d90110160 2024-04-30T08:23:13.305 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.494+0000 7f5d96000700 1 -- 172.21.15.32:0/2097690978 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d901a4cb0 con 0x7f5d90110160 2024-04-30T08:23:13.305 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.494+0000 7f5d8cc00700 1 -- 172.21.15.32:0/2097690978 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5d8001a5c0 con 0x7f5d90110160 2024-04-30T08:23:13.306 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.494+0000 7f5d96000700 1 -- 172.21.15.32:0/2097690978 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d901a78c0 con 0x7f5d90110160 2024-04-30T08:23:13.306 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.496+0000 7f5d96000700 1 -- 172.21.15.32:0/2097690978 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d90064d70 con 0x7f5d90110160 2024-04-30T08:23:13.306 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.496+0000 7f5d8cc00700 1 -- 172.21.15.32:0/2097690978 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f5d8001a880 con 0x7f5d90110160 2024-04-30T08:23:13.306 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.496+0000 7f5d8cc00700 1 --2- 172.21.15.32:0/2097690978 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f5d7403d520 0x7f5d7403fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:13.306 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.496+0000 7f5d8cc00700 1 -- 172.21.15.32:0/2097690978 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f5d800526f0 con 0x7f5d90110160 2024-04-30T08:23:13.306 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.496+0000 7f5d86a00700 1 --2- 172.21.15.32:0/2097690978 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f5d7403d520 0x7f5d7403fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:13.306 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.497+0000 7f5d86a00700 1 --2- 172.21.15.32:0/2097690978 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f5d7403d520 0x7f5d7403fa30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5d78006fd0 tx=0x7f5d78006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:13.306 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.502+0000 7f5d8cc00700 1 -- 172.21.15.32:0/2097690978 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5d8002c350 con 0x7f5d90110160 2024-04-30T08:23:13.307 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.758+0000 7f5d96000700 1 -- 172.21.15.32:0/2097690978 --> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f5d900803d0 con 0x7f5d7403d520 2024-04-30T08:23:13.307 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.770+0000 7f5d8cc00700 1 -- 172.21.15.32:0/2097690978 <== mgr.14120 v2:172.21.15.32:6800/4019351306 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f5d900803d0 con 0x7f5d7403d520 2024-04-30T08:23:13.307 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.773+0000 7f5d96000700 1 -- 172.21.15.32:0/2097690978 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f5d7403d520 msgr2=0x7f5d7403fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:13.307 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.773+0000 7f5d96000700 1 --2- 172.21.15.32:0/2097690978 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f5d7403d520 0x7f5d7403fa30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5d78006fd0 tx=0x7f5d78006e40 comp rx=0 tx=0).stop 2024-04-30T08:23:13.307 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.773+0000 7f5d96000700 1 -- 172.21.15.32:0/2097690978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d90110160 msgr2=0x7f5d901a44e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:13.307 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.773+0000 7f5d96000700 1 --2- 172.21.15.32:0/2097690978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d90110160 0x7f5d901a44e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f5d80009fb0 tx=0x7f5d80011890 comp rx=0 tx=0).stop 2024-04-30T08:23:13.307 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.774+0000 7f5d96000700 1 -- 172.21.15.32:0/2097690978 shutdown_connections 2024-04-30T08:23:13.307 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.774+0000 7f5d96000700 1 --2- 172.21.15.32:0/2097690978 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f5d7403d520 0x7f5d7403fa30 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:13.307 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.774+0000 7f5d96000700 1 --2- 172.21.15.32:0/2097690978 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d90110160 0x7f5d901a44e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:13.308 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.774+0000 7f5d96000700 1 -- 172.21.15.32:0/2097690978 >> 172.21.15.32:0/2097690978 conn(0x7f5d9007f330 msgr2=0x7f5d9007fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:13.308 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.774+0000 7f5d96000700 1 -- 172.21.15.32:0/2097690978 shutdown_connections 2024-04-30T08:23:13.308 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:11.774+0000 7f5d96000700 1 -- 172.21.15.32:0/2097690978 wait complete. 2024-04-30T08:23:13.308 INFO:teuthology.orchestra.run.smithi032.stdout:Deploying unmanaged mgr service... 2024-04-30T08:23:14.321 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:14 smithi032 ceph-mon[28331]: from='mgr.14120 172.21.15.32:0/1220014720' entity='mgr.x' 2024-04-30T08:23:14.321 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:14 smithi032 ceph-mon[28331]: from='mgr.14120 172.21.15.32:0/1220014720' entity='mgr.x' 2024-04-30T08:23:15.017 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-30T08:23:15.017 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.050+0000 7f4caf200700 1 Processor -- start 2024-04-30T08:23:15.017 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.050+0000 7f4caf200700 1 -- start start 2024-04-30T08:23:15.017 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.050+0000 7f4caf200700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ca807ed40 0x7f4ca807d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:15.017 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.050+0000 7f4caf200700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca807d750 con 0x7f4ca807ed40 2024-04-30T08:23:15.017 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.051+0000 7f4ca7e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ca807ed40 0x7f4ca807d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:15.017 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.051+0000 7f4ca7e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ca807ed40 0x7f4ca807d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52124/0 (socket says 172.21.15.32:52124) 2024-04-30T08:23:15.017 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.051+0000 7f4ca7e00700 1 -- 172.21.15.32:0/912298199 learned_addr learned my addr 172.21.15.32:0/912298199 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:23:15.017 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.051+0000 7f4ca7e00700 1 -- 172.21.15.32:0/912298199 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ca807d8d0 con 0x7f4ca807ed40 2024-04-30T08:23:15.017 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.052+0000 7f4ca7e00700 1 --2- 172.21.15.32:0/912298199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ca807ed40 0x7f4ca807d210 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4c980098a0 tx=0x7f4c98009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73f6e4745c47665f server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:15.017 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.052+0000 7f4ca6a00700 1 -- 172.21.15.32:0/912298199 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c98004160 con 0x7f4ca807ed40 2024-04-30T08:23:15.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.052+0000 7f4ca6a00700 1 -- 172.21.15.32:0/912298199 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c98004300 con 0x7f4ca807ed40 2024-04-30T08:23:15.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.052+0000 7f4ca6a00700 1 -- 172.21.15.32:0/912298199 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c98011a20 con 0x7f4ca807ed40 2024-04-30T08:23:15.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.053+0000 7f4caf200700 1 -- 172.21.15.32:0/912298199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ca807ed40 msgr2=0x7f4ca807d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:15.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.053+0000 7f4caf200700 1 --2- 172.21.15.32:0/912298199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ca807ed40 0x7f4ca807d210 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4c980098a0 tx=0x7f4c98009bb0 comp rx=0 tx=0).stop 2024-04-30T08:23:15.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.053+0000 7f4caf200700 1 -- 172.21.15.32:0/912298199 shutdown_connections 2024-04-30T08:23:15.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.053+0000 7f4caf200700 1 --2- 172.21.15.32:0/912298199 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ca807ed40 0x7f4ca807d210 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:15.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.053+0000 7f4caf200700 1 -- 172.21.15.32:0/912298199 >> 172.21.15.32:0/912298199 conn(0x7f4ca8105350 msgr2=0x7f4ca81077a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:15.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.054+0000 7f4caf200700 1 -- 172.21.15.32:0/912298199 shutdown_connections 2024-04-30T08:23:15.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.054+0000 7f4caf200700 1 -- 172.21.15.32:0/912298199 wait complete. 2024-04-30T08:23:15.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.055+0000 7f4caf200700 1 Processor -- start 2024-04-30T08:23:15.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.055+0000 7f4caf200700 1 -- start start 2024-04-30T08:23:15.018 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.056+0000 7f4caf200700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ca807ed40 0x7f4ca81a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:15.019 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.056+0000 7f4caf200700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4ca8110230 con 0x7f4ca807ed40 2024-04-30T08:23:15.019 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.056+0000 7f4ca7e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ca807ed40 0x7f4ca81a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:15.019 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.056+0000 7f4ca7e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ca807ed40 0x7f4ca81a0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52130/0 (socket says 172.21.15.32:52130) 2024-04-30T08:23:15.019 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.056+0000 7f4ca7e00700 1 -- 172.21.15.32:0/128229713 learned_addr learned my addr 172.21.15.32:0/128229713 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:23:15.019 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.057+0000 7f4ca7e00700 1 -- 172.21.15.32:0/128229713 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ca81a06c0 con 0x7f4ca807ed40 2024-04-30T08:23:15.019 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.057+0000 7f4ca7e00700 1 --2- 172.21.15.32:0/128229713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ca807ed40 0x7f4ca81a0180 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4c98004660 tx=0x7f4c98011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:15.019 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.058+0000 7f4ca4c00700 1 -- 172.21.15.32:0/128229713 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c98003a60 con 0x7f4ca807ed40 2024-04-30T08:23:15.019 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.058+0000 7f4ca4c00700 1 -- 172.21.15.32:0/128229713 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c9801b8f0 con 0x7f4ca807ed40 2024-04-30T08:23:15.019 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.058+0000 7f4caf200700 1 -- 172.21.15.32:0/128229713 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ca81a0950 con 0x7f4ca807ed40 2024-04-30T08:23:15.019 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.058+0000 7f4ca4c00700 1 -- 172.21.15.32:0/128229713 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4c98023890 con 0x7f4ca807ed40 2024-04-30T08:23:15.019 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.058+0000 7f4caf200700 1 -- 172.21.15.32:0/128229713 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ca81a3560 con 0x7f4ca807ed40 2024-04-30T08:23:15.019 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.060+0000 7f4caf200700 1 -- 172.21.15.32:0/128229713 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ca8064d70 con 0x7f4ca807ed40 2024-04-30T08:23:15.019 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.060+0000 7f4ca4c00700 1 -- 172.21.15.32:0/128229713 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f4c9802c3f0 con 0x7f4ca807ed40 2024-04-30T08:23:15.019 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.061+0000 7f4ca4c00700 1 --2- 172.21.15.32:0/128229713 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f4c8c03d570 0x7f4c8c03fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:15.020 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.061+0000 7f4ca4c00700 1 -- 172.21.15.32:0/128229713 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4c98052d40 con 0x7f4ca807ed40 2024-04-30T08:23:15.020 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.061+0000 7f4ca7400700 1 --2- 172.21.15.32:0/128229713 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f4c8c03d570 0x7f4c8c03fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:15.020 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.062+0000 7f4ca7400700 1 --2- 172.21.15.32:0/128229713 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f4c8c03d570 0x7f4c8c03fa80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4c94006fd0 tx=0x7f4c94006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:15.020 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.067+0000 7f4ca4c00700 1 -- 172.21.15.32:0/128229713 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4c9801f030 con 0x7f4ca807ed40 2024-04-30T08:23:15.020 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.319+0000 7f4caf200700 1 -- 172.21.15.32:0/128229713 --> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f4ca81a37f0 con 0x7f4c8c03d570 2024-04-30T08:23:15.020 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.329+0000 7f4ca4c00700 1 -- 172.21.15.32:0/128229713 <== mgr.14120 v2:172.21.15.32:6800/4019351306 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f4ca81a37f0 con 0x7f4c8c03d570 2024-04-30T08:23:15.020 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.332+0000 7f4caf200700 1 -- 172.21.15.32:0/128229713 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f4c8c03d570 msgr2=0x7f4c8c03fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:15.020 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.332+0000 7f4caf200700 1 --2- 172.21.15.32:0/128229713 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f4c8c03d570 0x7f4c8c03fa80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4c94006fd0 tx=0x7f4c94006e40 comp rx=0 tx=0).stop 2024-04-30T08:23:15.020 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.332+0000 7f4caf200700 1 -- 172.21.15.32:0/128229713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ca807ed40 msgr2=0x7f4ca81a0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:15.020 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.332+0000 7f4caf200700 1 --2- 172.21.15.32:0/128229713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ca807ed40 0x7f4ca81a0180 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f4c98004660 tx=0x7f4c98011aa0 comp rx=0 tx=0).stop 2024-04-30T08:23:15.020 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.332+0000 7f4caf200700 1 -- 172.21.15.32:0/128229713 shutdown_connections 2024-04-30T08:23:15.020 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.332+0000 7f4caf200700 1 --2- 172.21.15.32:0/128229713 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f4c8c03d570 0x7f4c8c03fa80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:15.020 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.332+0000 7f4caf200700 1 --2- 172.21.15.32:0/128229713 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ca807ed40 0x7f4ca81a0180 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:15.021 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.332+0000 7f4caf200700 1 -- 172.21.15.32:0/128229713 >> 172.21.15.32:0/128229713 conn(0x7f4ca8105350 msgr2=0x7f4ca8105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:15.021 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.333+0000 7f4caf200700 1 -- 172.21.15.32:0/128229713 shutdown_connections 2024-04-30T08:23:15.021 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:14.333+0000 7f4caf200700 1 -- 172.21.15.32:0/128229713 wait complete. 2024-04-30T08:23:15.690 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:15 smithi032 ceph-mon[28331]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:23:15.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:15 smithi032 ceph-mon[28331]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:23:15.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:15 smithi032 ceph-mon[28331]: Saving service mgr spec with placement count:2 2024-04-30T08:23:15.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:15 smithi032 ceph-mon[28331]: from='mgr.14120 172.21.15.32:0/1220014720' entity='mgr.x' 2024-04-30T08:23:16.921 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.729+0000 7f3198e00700 1 Processor -- start 2024-04-30T08:23:16.922 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.729+0000 7f3198e00700 1 -- start start 2024-04-30T08:23:16.922 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.730+0000 7f3198e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3194109bf0 0x7f319410a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:16.922 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.730+0000 7f3198e00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f319410a5f0 con 0x7f3194109bf0 2024-04-30T08:23:16.922 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.730+0000 7f3191e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3194109bf0 0x7f319410a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:16.922 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.730+0000 7f3191e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3194109bf0 0x7f319410a000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52138/0 (socket says 172.21.15.32:52138) 2024-04-30T08:23:16.922 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.730+0000 7f3191e00700 1 -- 172.21.15.32:0/3117073045 learned_addr learned my addr 172.21.15.32:0/3117073045 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:23:16.922 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.731+0000 7f3191e00700 1 -- 172.21.15.32:0/3117073045 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f319410ad20 con 0x7f3194109bf0 2024-04-30T08:23:16.922 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.731+0000 7f3191e00700 1 --2- 172.21.15.32:0/3117073045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3194109bf0 0x7f319410a000 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f31840098a0 tx=0x7f3184009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=483dddea0a723457 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:16.922 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.731+0000 7f3190a00700 1 -- 172.21.15.32:0/3117073045 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3184004160 con 0x7f3194109bf0 2024-04-30T08:23:16.922 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.731+0000 7f3190a00700 1 -- 172.21.15.32:0/3117073045 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3184004300 con 0x7f3194109bf0 2024-04-30T08:23:16.922 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.731+0000 7f3190a00700 1 -- 172.21.15.32:0/3117073045 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3184011a20 con 0x7f3194109bf0 2024-04-30T08:23:16.923 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.732+0000 7f3198e00700 1 -- 172.21.15.32:0/3117073045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3194109bf0 msgr2=0x7f319410a000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:16.923 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.732+0000 7f3198e00700 1 --2- 172.21.15.32:0/3117073045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3194109bf0 0x7f319410a000 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f31840098a0 tx=0x7f3184009bb0 comp rx=0 tx=0).stop 2024-04-30T08:23:16.923 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.732+0000 7f3198e00700 1 -- 172.21.15.32:0/3117073045 shutdown_connections 2024-04-30T08:23:16.923 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.732+0000 7f3198e00700 1 --2- 172.21.15.32:0/3117073045 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3194109bf0 0x7f319410a000 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:16.923 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.732+0000 7f3198e00700 1 -- 172.21.15.32:0/3117073045 >> 172.21.15.32:0/3117073045 conn(0x7f31941052f0 msgr2=0x7f3194107740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:16.923 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.732+0000 7f3198e00700 1 -- 172.21.15.32:0/3117073045 shutdown_connections 2024-04-30T08:23:16.923 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.732+0000 7f3198e00700 1 -- 172.21.15.32:0/3117073045 wait complete. 2024-04-30T08:23:16.923 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.733+0000 7f3198e00700 1 Processor -- start 2024-04-30T08:23:16.923 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.734+0000 7f3198e00700 1 -- start start 2024-04-30T08:23:16.923 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.734+0000 7f3198e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3194109bf0 0x7f319419df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:16.923 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.734+0000 7f3198e00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f319410e0d0 con 0x7f3194109bf0 2024-04-30T08:23:16.923 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.734+0000 7f3191e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3194109bf0 0x7f319419df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:16.923 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.734+0000 7f3191e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3194109bf0 0x7f319419df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52142/0 (socket says 172.21.15.32:52142) 2024-04-30T08:23:16.924 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.734+0000 7f3191e00700 1 -- 172.21.15.32:0/2445121606 learned_addr learned my addr 172.21.15.32:0/2445121606 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:23:16.924 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.735+0000 7f3191e00700 1 -- 172.21.15.32:0/2445121606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f319419e4d0 con 0x7f3194109bf0 2024-04-30T08:23:16.924 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.735+0000 7f3191e00700 1 --2- 172.21.15.32:0/2445121606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3194109bf0 0x7f319419df90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3184003e70 tx=0x7f3184011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:16.924 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.735+0000 7f318aa00700 1 -- 172.21.15.32:0/2445121606 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3184003af0 con 0x7f3194109bf0 2024-04-30T08:23:16.924 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.735+0000 7f318aa00700 1 -- 172.21.15.32:0/2445121606 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f318401a8e0 con 0x7f3194109bf0 2024-04-30T08:23:16.924 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.735+0000 7f318aa00700 1 -- 172.21.15.32:0/2445121606 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3184024870 con 0x7f3194109bf0 2024-04-30T08:23:16.924 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.736+0000 7f3198e00700 1 -- 172.21.15.32:0/2445121606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f319419e760 con 0x7f3194109bf0 2024-04-30T08:23:16.924 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.736+0000 7f3198e00700 1 -- 172.21.15.32:0/2445121606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31941a1370 con 0x7f3194109bf0 2024-04-30T08:23:16.924 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.737+0000 7f3198e00700 1 -- 172.21.15.32:0/2445121606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3194064d70 con 0x7f3194109bf0 2024-04-30T08:23:16.924 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.737+0000 7f318aa00700 1 -- 172.21.15.32:0/2445121606 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7f3184022020 con 0x7f3194109bf0 2024-04-30T08:23:16.924 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.738+0000 7f318aa00700 1 --2- 172.21.15.32:0/2445121606 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f317803d570 0x7f317803fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:16.924 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.738+0000 7f318aa00700 1 -- 172.21.15.32:0/2445121606 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f31840526d0 con 0x7f3194109bf0 2024-04-30T08:23:16.924 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.738+0000 7f3191400700 1 --2- 172.21.15.32:0/2445121606 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f317803d570 0x7f317803fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:16.925 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.739+0000 7f3191400700 1 --2- 172.21.15.32:0/2445121606 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f317803d570 0x7f317803fa80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3180006fd0 tx=0x7f3180006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:16.925 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.743+0000 7f318aa00700 1 -- 172.21.15.32:0/2445121606 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f318401ad90 con 0x7f3194109bf0 2024-04-30T08:23:16.925 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.974+0000 7f3198e00700 1 -- 172.21.15.32:0/2445121606 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f31941a1660 con 0x7f3194109bf0 2024-04-30T08:23:16.925 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.978+0000 7f318aa00700 1 -- 172.21.15.32:0/2445121606 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f3184018a40 con 0x7f3194109bf0 2024-04-30T08:23:16.925 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.981+0000 7f3198e00700 1 -- 172.21.15.32:0/2445121606 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f317803d570 msgr2=0x7f317803fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:16.925 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.981+0000 7f3198e00700 1 --2- 172.21.15.32:0/2445121606 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f317803d570 0x7f317803fa80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f3180006fd0 tx=0x7f3180006e40 comp rx=0 tx=0).stop 2024-04-30T08:23:16.925 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.981+0000 7f3198e00700 1 -- 172.21.15.32:0/2445121606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3194109bf0 msgr2=0x7f319419df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:16.925 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.981+0000 7f3198e00700 1 --2- 172.21.15.32:0/2445121606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3194109bf0 0x7f319419df90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f3184003e70 tx=0x7f3184011aa0 comp rx=0 tx=0).stop 2024-04-30T08:23:16.925 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.981+0000 7f3198e00700 1 -- 172.21.15.32:0/2445121606 shutdown_connections 2024-04-30T08:23:16.925 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.981+0000 7f3198e00700 1 --2- 172.21.15.32:0/2445121606 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f317803d570 0x7f317803fa80 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:16.925 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.981+0000 7f3198e00700 1 --2- 172.21.15.32:0/2445121606 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3194109bf0 0x7f319419df90 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:16.925 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.981+0000 7f3198e00700 1 -- 172.21.15.32:0/2445121606 >> 172.21.15.32:0/2445121606 conn(0x7f31941052f0 msgr2=0x7f3194106c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:16.925 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.982+0000 7f3198e00700 1 -- 172.21.15.32:0/2445121606 shutdown_connections 2024-04-30T08:23:16.925 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:15.982+0000 7f3198e00700 1 -- 172.21.15.32:0/2445121606 wait complete. 2024-04-30T08:23:17.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:16 smithi032 ceph-mon[28331]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:23:17.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:16 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2445121606' entity='client.admin' 2024-04-30T08:23:17.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:16 smithi032 ceph-mon[28331]: from='mgr.14120 172.21.15.32:0/1220014720' entity='mgr.x' 2024-04-30T08:23:18.499 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.564+0000 7fbe47e00700 1 Processor -- start 2024-04-30T08:23:18.499 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.565+0000 7fbe47e00700 1 -- start start 2024-04-30T08:23:18.499 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.565+0000 7fbe47e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe4810bc90 0x7fbe4810c0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:18.499 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.565+0000 7fbe47e00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbe4810c690 con 0x7fbe4810bc90 2024-04-30T08:23:18.499 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.566+0000 7fbe46a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe4810bc90 0x7fbe4810c0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:18.499 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.566+0000 7fbe46a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe4810bc90 0x7fbe4810c0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56962/0 (socket says 172.21.15.32:56962) 2024-04-30T08:23:18.499 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.566+0000 7fbe46a00700 1 -- 172.21.15.32:0/311891937 learned_addr learned my addr 172.21.15.32:0/311891937 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:23:18.499 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.566+0000 7fbe46a00700 1 -- 172.21.15.32:0/311891937 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe4810cd20 con 0x7fbe4810bc90 2024-04-30T08:23:18.499 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.567+0000 7fbe46a00700 1 --2- 172.21.15.32:0/311891937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe4810bc90 0x7fbe4810c0a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbe380098a0 tx=0x7fbe38009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6c0001e139331e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:18.499 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.567+0000 7fbe45600700 1 -- 172.21.15.32:0/311891937 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe38004160 con 0x7fbe4810bc90 2024-04-30T08:23:18.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.567+0000 7fbe45600700 1 -- 172.21.15.32:0/311891937 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe38004300 con 0x7fbe4810bc90 2024-04-30T08:23:18.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.567+0000 7fbe45600700 1 -- 172.21.15.32:0/311891937 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe38011a20 con 0x7fbe4810bc90 2024-04-30T08:23:18.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.568+0000 7fbe47e00700 1 -- 172.21.15.32:0/311891937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe4810bc90 msgr2=0x7fbe4810c0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:18.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.568+0000 7fbe47e00700 1 --2- 172.21.15.32:0/311891937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe4810bc90 0x7fbe4810c0a0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fbe380098a0 tx=0x7fbe38009bb0 comp rx=0 tx=0).stop 2024-04-30T08:23:18.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.568+0000 7fbe47e00700 1 -- 172.21.15.32:0/311891937 shutdown_connections 2024-04-30T08:23:18.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.568+0000 7fbe47e00700 1 --2- 172.21.15.32:0/311891937 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe4810bc90 0x7fbe4810c0a0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:18.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.568+0000 7fbe47e00700 1 -- 172.21.15.32:0/311891937 >> 172.21.15.32:0/311891937 conn(0x7fbe481074b0 msgr2=0x7fbe481098d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:18.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.568+0000 7fbe47e00700 1 -- 172.21.15.32:0/311891937 shutdown_connections 2024-04-30T08:23:18.500 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.568+0000 7fbe47e00700 1 -- 172.21.15.32:0/311891937 wait complete. 2024-04-30T08:23:18.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.569+0000 7fbe47e00700 1 Processor -- start 2024-04-30T08:23:18.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.569+0000 7fbe47e00700 1 -- start start 2024-04-30T08:23:18.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.570+0000 7fbe47e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe4810bc90 0x7fbe4819ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:18.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.570+0000 7fbe47e00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbe481100d0 con 0x7fbe4810bc90 2024-04-30T08:23:18.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.570+0000 7fbe46a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe4810bc90 0x7fbe4819ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:18.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.570+0000 7fbe46a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe4810bc90 0x7fbe4819ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56978/0 (socket says 172.21.15.32:56978) 2024-04-30T08:23:18.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.570+0000 7fbe46a00700 1 -- 172.21.15.32:0/1603032359 learned_addr learned my addr 172.21.15.32:0/1603032359 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:23:18.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.571+0000 7fbe46a00700 1 -- 172.21.15.32:0/1603032359 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe481a04f0 con 0x7fbe4810bc90 2024-04-30T08:23:18.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.571+0000 7fbe46a00700 1 --2- 172.21.15.32:0/1603032359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe4810bc90 0x7fbe4819ffb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbe38003e70 tx=0x7fbe38011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:18.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.571+0000 7fbe3f400700 1 -- 172.21.15.32:0/1603032359 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe38003aa0 con 0x7fbe4810bc90 2024-04-30T08:23:18.501 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.571+0000 7fbe3f400700 1 -- 172.21.15.32:0/1603032359 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe3801a8e0 con 0x7fbe4810bc90 2024-04-30T08:23:18.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.571+0000 7fbe47e00700 1 -- 172.21.15.32:0/1603032359 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe481a0780 con 0x7fbe4810bc90 2024-04-30T08:23:18.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.571+0000 7fbe3f400700 1 -- 172.21.15.32:0/1603032359 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbe38024870 con 0x7fbe4810bc90 2024-04-30T08:23:18.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.571+0000 7fbe47e00700 1 -- 172.21.15.32:0/1603032359 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe481a3390 con 0x7fbe4810bc90 2024-04-30T08:23:18.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.572+0000 7fbe47e00700 1 -- 172.21.15.32:0/1603032359 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe48064d70 con 0x7fbe4810bc90 2024-04-30T08:23:18.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.573+0000 7fbe3f400700 1 -- 172.21.15.32:0/1603032359 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7fbe38022020 con 0x7fbe4810bc90 2024-04-30T08:23:18.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.574+0000 7fbe3f400700 1 --2- 172.21.15.32:0/1603032359 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7fbe2c03d5c0 0x7fbe2c03fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:18.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.574+0000 7fbe3f400700 1 -- 172.21.15.32:0/1603032359 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fbe38052780 con 0x7fbe4810bc90 2024-04-30T08:23:18.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.574+0000 7fbe46000700 1 --2- 172.21.15.32:0/1603032359 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7fbe2c03d5c0 0x7fbe2c03fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:18.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.575+0000 7fbe46000700 1 --2- 172.21.15.32:0/1603032359 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7fbe2c03d5c0 0x7fbe2c03fad0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fbe34006fd0 tx=0x7fbe34006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:18.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.580+0000 7fbe3f400700 1 -- 172.21.15.32:0/1603032359 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbe38017e90 con 0x7fbe4810bc90 2024-04-30T08:23:18.502 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.815+0000 7fbe47e00700 1 -- 172.21.15.32:0/1603032359 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7fbe481a3680 con 0x7fbe4810bc90 2024-04-30T08:23:18.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.819+0000 7fbe3f400700 1 -- 172.21.15.32:0/1603032359 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7fbe380212c0 con 0x7fbe4810bc90 2024-04-30T08:23:18.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.823+0000 7fbe47e00700 1 -- 172.21.15.32:0/1603032359 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7fbe2c03d5c0 msgr2=0x7fbe2c03fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:18.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.823+0000 7fbe47e00700 1 --2- 172.21.15.32:0/1603032359 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7fbe2c03d5c0 0x7fbe2c03fad0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fbe34006fd0 tx=0x7fbe34006e40 comp rx=0 tx=0).stop 2024-04-30T08:23:18.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.823+0000 7fbe47e00700 1 -- 172.21.15.32:0/1603032359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe4810bc90 msgr2=0x7fbe4819ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:18.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.823+0000 7fbe47e00700 1 --2- 172.21.15.32:0/1603032359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe4810bc90 0x7fbe4819ffb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fbe38003e70 tx=0x7fbe38011aa0 comp rx=0 tx=0).stop 2024-04-30T08:23:18.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.823+0000 7fbe47e00700 1 -- 172.21.15.32:0/1603032359 shutdown_connections 2024-04-30T08:23:18.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.823+0000 7fbe47e00700 1 --2- 172.21.15.32:0/1603032359 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7fbe2c03d5c0 0x7fbe2c03fad0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:18.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.823+0000 7fbe47e00700 1 --2- 172.21.15.32:0/1603032359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe4810bc90 0x7fbe4819ffb0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:18.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.823+0000 7fbe47e00700 1 -- 172.21.15.32:0/1603032359 >> 172.21.15.32:0/1603032359 conn(0x7fbe481074b0 msgr2=0x7fbe48108d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:18.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.824+0000 7fbe47e00700 1 -- 172.21.15.32:0/1603032359 shutdown_connections 2024-04-30T08:23:18.503 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:17.824+0000 7fbe47e00700 1 -- 172.21.15.32:0/1603032359 wait complete. 2024-04-30T08:23:18.503 INFO:teuthology.orchestra.run.smithi032.stdout:Enabling the dashboard module... 2024-04-30T08:23:19.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:18 smithi032 ceph-mon[28331]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:23:19.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:18 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1603032359' entity='client.admin' 2024-04-30T08:23:20.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:19 smithi032 ceph-mon[28331]: from='mgr.14120 172.21.15.32:0/1220014720' entity='mgr.x' 2024-04-30T08:23:20.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:19 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1026669802' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-30T08:23:20.469 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:20 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: ignoring --setuser ceph since I am not root 2024-04-30T08:23:20.469 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:20 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: ignoring --setgroup ceph since I am not root 2024-04-30T08:23:20.469 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:20 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:23:20.404+0000 7fbbc5252200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-30T08:23:20.761 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:20 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:23:20.674+0000 7fbbc5252200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-30T08:23:21.064 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:20 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1026669802' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-30T08:23:21.064 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:20 smithi032 ceph-mon[28331]: mgrmap e9: x(active, since 26s) 2024-04-30T08:23:21.064 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:20 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:23:20.812+0000 7fbbc5252200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-30T08:23:21.352 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:21 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:23:21.231+0000 7fbbc5252200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-30T08:23:21.352 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:21 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:23:21.350+0000 7fbbc5252200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-30T08:23:21.623 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.205+0000 7feb76a00700 1 Processor -- start 2024-04-30T08:23:21.623 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.205+0000 7feb76a00700 1 -- start start 2024-04-30T08:23:21.624 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.206+0000 7feb76a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb7007ed10 0x7feb7007d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:21.624 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.206+0000 7feb76a00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7feb7007d770 con 0x7feb7007ed10 2024-04-30T08:23:21.624 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.206+0000 7feb75600700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb7007ed10 0x7feb7007d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:21.624 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.206+0000 7feb75600700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb7007ed10 0x7feb7007d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56986/0 (socket says 172.21.15.32:56986) 2024-04-30T08:23:21.624 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.206+0000 7feb75600700 1 -- 172.21.15.32:0/4231463142 learned_addr learned my addr 172.21.15.32:0/4231463142 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:23:21.624 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.207+0000 7feb75600700 1 -- 172.21.15.32:0/4231463142 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb7007d8f0 con 0x7feb7007ed10 2024-04-30T08:23:21.624 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.207+0000 7feb75600700 1 --2- 172.21.15.32:0/4231463142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb7007ed10 0x7feb7007d230 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7feb640098a0 tx=0x7feb64009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc057cba286c4b77 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:21.624 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.208+0000 7feb6fe00700 1 -- 172.21.15.32:0/4231463142 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feb64004160 con 0x7feb7007ed10 2024-04-30T08:23:21.624 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.208+0000 7feb6fe00700 1 -- 172.21.15.32:0/4231463142 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb64004300 con 0x7feb7007ed10 2024-04-30T08:23:21.624 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.208+0000 7feb6fe00700 1 -- 172.21.15.32:0/4231463142 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feb64011a20 con 0x7feb7007ed10 2024-04-30T08:23:21.624 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.209+0000 7feb76a00700 1 -- 172.21.15.32:0/4231463142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb7007ed10 msgr2=0x7feb7007d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:21.625 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.209+0000 7feb76a00700 1 --2- 172.21.15.32:0/4231463142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb7007ed10 0x7feb7007d230 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7feb640098a0 tx=0x7feb64009bb0 comp rx=0 tx=0).stop 2024-04-30T08:23:21.625 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.209+0000 7feb76a00700 1 -- 172.21.15.32:0/4231463142 shutdown_connections 2024-04-30T08:23:21.625 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.209+0000 7feb76a00700 1 --2- 172.21.15.32:0/4231463142 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb7007ed10 0x7feb7007d230 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:21.625 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.209+0000 7feb76a00700 1 -- 172.21.15.32:0/4231463142 >> 172.21.15.32:0/4231463142 conn(0x7feb701050d0 msgr2=0x7feb701075a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:21.625 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.209+0000 7feb76a00700 1 -- 172.21.15.32:0/4231463142 shutdown_connections 2024-04-30T08:23:21.625 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.209+0000 7feb76a00700 1 -- 172.21.15.32:0/4231463142 wait complete. 2024-04-30T08:23:21.625 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.210+0000 7feb76a00700 1 Processor -- start 2024-04-30T08:23:21.625 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.210+0000 7feb76a00700 1 -- start start 2024-04-30T08:23:21.625 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.211+0000 7feb76a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb7007ed10 0x7feb7019ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:21.625 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.211+0000 7feb76a00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7feb70110010 con 0x7feb7007ed10 2024-04-30T08:23:21.625 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.211+0000 7feb75600700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb7007ed10 0x7feb7019ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:21.625 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.211+0000 7feb75600700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb7007ed10 0x7feb7019ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56998/0 (socket says 172.21.15.32:56998) 2024-04-30T08:23:21.625 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.211+0000 7feb75600700 1 -- 172.21.15.32:0/1026669802 learned_addr learned my addr 172.21.15.32:0/1026669802 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:23:21.625 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.212+0000 7feb75600700 1 -- 172.21.15.32:0/1026669802 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb701a04e0 con 0x7feb7007ed10 2024-04-30T08:23:21.625 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.212+0000 7feb75600700 1 --2- 172.21.15.32:0/1026669802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb7007ed10 0x7feb7019ffa0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7feb64004660 tx=0x7feb64011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:21.626 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.212+0000 7feb6e000700 1 -- 172.21.15.32:0/1026669802 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feb64003a60 con 0x7feb7007ed10 2024-04-30T08:23:21.626 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.212+0000 7feb6e000700 1 -- 172.21.15.32:0/1026669802 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feb6401b8f0 con 0x7feb7007ed10 2024-04-30T08:23:21.626 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.212+0000 7feb6e000700 1 -- 172.21.15.32:0/1026669802 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feb64023890 con 0x7feb7007ed10 2024-04-30T08:23:21.626 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.212+0000 7feb76a00700 1 -- 172.21.15.32:0/1026669802 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb701a0770 con 0x7feb7007ed10 2024-04-30T08:23:21.626 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.212+0000 7feb76a00700 1 -- 172.21.15.32:0/1026669802 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb701a3380 con 0x7feb7007ed10 2024-04-30T08:23:21.626 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.213+0000 7feb76a00700 1 -- 172.21.15.32:0/1026669802 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb70064d70 con 0x7feb7007ed10 2024-04-30T08:23:21.626 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.214+0000 7feb6e000700 1 -- 172.21.15.32:0/1026669802 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 8) ==== 49516+0+0 (secure 0 0 0) 0x7feb6401b440 con 0x7feb7007ed10 2024-04-30T08:23:21.626 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.214+0000 7feb6e000700 1 --2- 172.21.15.32:0/1026669802 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7feb5803d570 0x7feb5803fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:21.626 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.214+0000 7feb6e000700 1 -- 172.21.15.32:0/1026669802 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7feb64052d70 con 0x7feb7007ed10 2024-04-30T08:23:21.627 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.215+0000 7feb74c00700 1 --2- 172.21.15.32:0/1026669802 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7feb5803d570 0x7feb5803fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:21.627 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.216+0000 7feb74c00700 1 --2- 172.21.15.32:0/1026669802 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7feb5803d570 0x7feb5803fa80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7feb5c006fd0 tx=0x7feb5c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:21.627 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.220+0000 7feb6e000700 1 -- 172.21.15.32:0/1026669802 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feb6401f030 con 0x7feb7007ed10 2024-04-30T08:23:21.627 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.514+0000 7feb76a00700 1 -- 172.21.15.32:0/1026669802 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7feb701a3550 con 0x7feb7007ed10 2024-04-30T08:23:21.627 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.988+0000 7feb6e000700 1 -- 172.21.15.32:0/1026669802 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7feb6401bda0 con 0x7feb7007ed10 2024-04-30T08:23:21.627 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.991+0000 7feb6e000700 1 -- 172.21.15.32:0/1026669802 <== mon.0 v2:172.21.15.32:3300/0 8 ==== mgrmap(e 9) ==== 49529+0+0 (secure 0 0 0) 0x7feb6400bad0 con 0x7feb7007ed10 2024-04-30T08:23:21.627 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.992+0000 7feb76a00700 1 -- 172.21.15.32:0/1026669802 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7feb5803d570 msgr2=0x7feb5803fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:21.627 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.992+0000 7feb76a00700 1 --2- 172.21.15.32:0/1026669802 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7feb5803d570 0x7feb5803fa80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7feb5c006fd0 tx=0x7feb5c006e40 comp rx=0 tx=0).stop 2024-04-30T08:23:21.627 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.992+0000 7feb76a00700 1 -- 172.21.15.32:0/1026669802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb7007ed10 msgr2=0x7feb7019ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:21.628 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.992+0000 7feb76a00700 1 --2- 172.21.15.32:0/1026669802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb7007ed10 0x7feb7019ffa0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7feb64004660 tx=0x7feb64011aa0 comp rx=0 tx=0).stop 2024-04-30T08:23:21.628 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.993+0000 7feb76a00700 1 -- 172.21.15.32:0/1026669802 shutdown_connections 2024-04-30T08:23:21.628 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.993+0000 7feb76a00700 1 --2- 172.21.15.32:0/1026669802 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7feb5803d570 0x7feb5803fa80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:21.628 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.993+0000 7feb76a00700 1 --2- 172.21.15.32:0/1026669802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feb7007ed10 0x7feb7019ffa0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:21.628 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.993+0000 7feb76a00700 1 -- 172.21.15.32:0/1026669802 >> 172.21.15.32:0/1026669802 conn(0x7feb701050d0 msgr2=0x7feb70105a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:21.628 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.994+0000 7feb76a00700 1 -- 172.21.15.32:0/1026669802 shutdown_connections 2024-04-30T08:23:21.628 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:19.994+0000 7feb76a00700 1 -- 172.21.15.32:0/1026669802 wait complete. 2024-04-30T08:23:22.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:22 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/4279727053' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-30T08:23:23.240 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:22 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:23:22.995+0000 7fbbc5252200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-30T08:23:23.240 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:23 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:23:23.238+0000 7fbbc5252200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-30T08:23:23.320 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-30T08:23:23.321 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-30T08:23:23.321 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-30T08:23:23.322 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-30T08:23:23.322 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-30T08:23:23.322 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-30T08:23:23.322 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.284+0000 7f0829800700 1 Processor -- start 2024-04-30T08:23:23.322 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.284+0000 7f0829800700 1 -- start start 2024-04-30T08:23:23.322 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.284+0000 7f0829800700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0824109cb0 0x7f082410a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:23.324 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.284+0000 7f0829800700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f082410a6b0 con 0x7f0824109cb0 2024-04-30T08:23:23.324 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.285+0000 7f0822a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0824109cb0 0x7f082410a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:23.324 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.285+0000 7f0822a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0824109cb0 0x7f082410a0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57044/0 (socket says 172.21.15.32:57044) 2024-04-30T08:23:23.324 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.285+0000 7f0822a00700 1 -- 172.21.15.32:0/1262276962 learned_addr learned my addr 172.21.15.32:0/1262276962 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:23:23.324 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.286+0000 7f0822a00700 1 -- 172.21.15.32:0/1262276962 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f082410ade0 con 0x7f0824109cb0 2024-04-30T08:23:23.325 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.286+0000 7f0822a00700 1 --2- 172.21.15.32:0/1262276962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0824109cb0 0x7f082410a0c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f08140098a0 tx=0x7f0814009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad1183cffd8648d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:23.325 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.286+0000 7f0821600700 1 -- 172.21.15.32:0/1262276962 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0814004160 con 0x7f0824109cb0 2024-04-30T08:23:23.325 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.286+0000 7f0821600700 1 -- 172.21.15.32:0/1262276962 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0814004300 con 0x7f0824109cb0 2024-04-30T08:23:23.325 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.286+0000 7f0821600700 1 -- 172.21.15.32:0/1262276962 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0814011a20 con 0x7f0824109cb0 2024-04-30T08:23:23.325 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.287+0000 7f0829800700 1 -- 172.21.15.32:0/1262276962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0824109cb0 msgr2=0x7f082410a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:23.325 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.287+0000 7f0829800700 1 --2- 172.21.15.32:0/1262276962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0824109cb0 0x7f082410a0c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f08140098a0 tx=0x7f0814009bb0 comp rx=0 tx=0).stop 2024-04-30T08:23:23.327 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.287+0000 7f0829800700 1 -- 172.21.15.32:0/1262276962 shutdown_connections 2024-04-30T08:23:23.327 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.288+0000 7f0829800700 1 --2- 172.21.15.32:0/1262276962 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0824109cb0 0x7f082410a0c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:23.328 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.288+0000 7f0829800700 1 -- 172.21.15.32:0/1262276962 >> 172.21.15.32:0/1262276962 conn(0x7f0824105350 msgr2=0x7f08241077a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:23.328 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.288+0000 7f0829800700 1 -- 172.21.15.32:0/1262276962 shutdown_connections 2024-04-30T08:23:23.328 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.288+0000 7f0829800700 1 -- 172.21.15.32:0/1262276962 wait complete. 2024-04-30T08:23:23.328 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.289+0000 7f0829800700 1 Processor -- start 2024-04-30T08:23:23.328 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.289+0000 7f0829800700 1 -- start start 2024-04-30T08:23:23.328 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.289+0000 7f0829800700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0824109cb0 0x7f082419dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:23.328 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.289+0000 7f0829800700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f082410e190 con 0x7f0824109cb0 2024-04-30T08:23:23.328 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.289+0000 7f0822a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0824109cb0 0x7f082419dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:23.328 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.289+0000 7f0822a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0824109cb0 0x7f082419dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57048/0 (socket says 172.21.15.32:57048) 2024-04-30T08:23:23.328 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.289+0000 7f0822a00700 1 -- 172.21.15.32:0/4279727053 learned_addr learned my addr 172.21.15.32:0/4279727053 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:23:23.328 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.290+0000 7f0822a00700 1 -- 172.21.15.32:0/4279727053 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f082419e520 con 0x7f0824109cb0 2024-04-30T08:23:23.328 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.290+0000 7f0822a00700 1 --2- 172.21.15.32:0/4279727053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0824109cb0 0x7f082419dfe0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0814011a20 tx=0x7f0814011b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:23.328 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.291+0000 7f081b400700 1 -- 172.21.15.32:0/4279727053 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08140039c0 con 0x7f0824109cb0 2024-04-30T08:23:23.329 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.291+0000 7f081b400700 1 -- 172.21.15.32:0/4279727053 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0814003b60 con 0x7f0824109cb0 2024-04-30T08:23:23.329 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.291+0000 7f0829800700 1 -- 172.21.15.32:0/4279727053 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f082419e7b0 con 0x7f0824109cb0 2024-04-30T08:23:23.329 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.291+0000 7f081b400700 1 -- 172.21.15.32:0/4279727053 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0814019700 con 0x7f0824109cb0 2024-04-30T08:23:23.329 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.291+0000 7f0829800700 1 -- 172.21.15.32:0/4279727053 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08241a13c0 con 0x7f0824109cb0 2024-04-30T08:23:23.329 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.292+0000 7f0829800700 1 -- 172.21.15.32:0/4279727053 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0824064d70 con 0x7f0824109cb0 2024-04-30T08:23:23.329 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.292+0000 7f081b400700 1 -- 172.21.15.32:0/4279727053 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 9) ==== 49529+0+0 (secure 0 0 0) 0x7f081401ab00 con 0x7f0824109cb0 2024-04-30T08:23:23.329 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.293+0000 7f081b400700 1 --2- 172.21.15.32:0/4279727053 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f080803d570 0x7f080803fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:23.329 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.293+0000 7f081b400700 1 -- 172.21.15.32:0/4279727053 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f0814052b30 con 0x7f0824109cb0 2024-04-30T08:23:23.332 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.293+0000 7f0822000700 1 -- 172.21.15.32:0/4279727053 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f080803d570 msgr2=0x7f080803fa80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/4019351306 2024-04-30T08:23:23.332 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.293+0000 7f0822000700 1 --2- 172.21.15.32:0/4279727053 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f080803d570 0x7f080803fa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-30T08:23:23.332 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.298+0000 7f081b400700 1 -- 172.21.15.32:0/4279727053 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0814018d60 con 0x7f0824109cb0 2024-04-30T08:23:23.332 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.494+0000 7f0822000700 1 -- 172.21.15.32:0/4279727053 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f080803d570 msgr2=0x7f080803fa80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/4019351306 2024-04-30T08:23:23.332 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.494+0000 7f0822000700 1 --2- 172.21.15.32:0/4279727053 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f080803d570 0x7f080803fa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-30T08:23:23.332 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.582+0000 7f0829800700 1 -- 172.21.15.32:0/4279727053 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f08241a16b0 con 0x7f0824109cb0 2024-04-30T08:23:23.332 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.583+0000 7f081b400700 1 -- 172.21.15.32:0/4279727053 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f081401a860 con 0x7f0824109cb0 2024-04-30T08:23:23.332 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.586+0000 7f0829800700 1 -- 172.21.15.32:0/4279727053 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f080803d570 msgr2=0x7f080803fa80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-30T08:23:23.332 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.586+0000 7f0829800700 1 --2- 172.21.15.32:0/4279727053 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f080803d570 0x7f080803fa80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:23.332 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.586+0000 7f0829800700 1 -- 172.21.15.32:0/4279727053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0824109cb0 msgr2=0x7f082419dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:23.333 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.586+0000 7f0829800700 1 --2- 172.21.15.32:0/4279727053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0824109cb0 0x7f082419dfe0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f0814011a20 tx=0x7f0814011b00 comp rx=0 tx=0).stop 2024-04-30T08:23:23.333 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.586+0000 7f0829800700 1 -- 172.21.15.32:0/4279727053 shutdown_connections 2024-04-30T08:23:23.333 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.586+0000 7f0829800700 1 --2- 172.21.15.32:0/4279727053 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f080803d570 0x7f080803fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:23.333 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.586+0000 7f0829800700 1 --2- 172.21.15.32:0/4279727053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0824109cb0 0x7f082419dfe0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:23.333 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.586+0000 7f0829800700 1 -- 172.21.15.32:0/4279727053 >> 172.21.15.32:0/4279727053 conn(0x7f0824105350 msgr2=0x7f0824105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:23.333 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.586+0000 7f0829800700 1 -- 172.21.15.32:0/4279727053 shutdown_connections 2024-04-30T08:23:23.333 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:22.586+0000 7f0829800700 1 -- 172.21.15.32:0/4279727053 wait complete. 2024-04-30T08:23:23.333 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for the mgr to restart... 2024-04-30T08:23:23.333 INFO:teuthology.orchestra.run.smithi032.stdout:Waiting for mgr epoch 9... 2024-04-30T08:23:23.608 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:23 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:23:23.464+0000 7fbbc5252200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-30T08:23:24.191 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:23 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:23:23.882+0000 7fbbc5252200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-30T08:23:24.191 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:24 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:23:24.016+0000 7fbbc5252200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-30T08:23:24.547 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:24 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:23:24.278+0000 7fbbc5252200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-30T08:23:24.548 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:24 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:23:24.546+0000 7fbbc5252200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-30T08:23:25.691 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:25 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:23:25.370+0000 7fbbc5252200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-30T08:23:26.373 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:25 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:23:25.957+0000 7fbbc5252200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-30T08:23:26.373 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:26 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:23:26.091+0000 7fbbc5252200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-30T08:23:26.691 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:26 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:23:26.371+0000 7fbbc5252200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-30T08:23:27.398 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:27 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:23:27.055+0000 7fbbc5252200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-30T08:23:27.398 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:27 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:23:27.187+0000 7fbbc5252200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-30T08:23:27.691 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:27 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:23:27.531+0000 7fbbc5252200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-30T08:23:28.030 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:27 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:23:27.725+0000 7fbbc5252200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-30T08:23:28.030 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:27 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:23:27.862+0000 7fbbc5252200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-30T08:23:28.416 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:28 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:23:28.027+0000 7fbbc5252200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-30T08:23:28.416 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:28 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:23:28.414+0000 7fbbc5252200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-30T08:23:29.441 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:29 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:23:29.216+0000 7fbbc5252200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-30T08:23:29.941 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:23:29 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:23:29.521+0000 7fbbc5252200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-30T08:23:29.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:29 smithi032 ceph-mon[28331]: Active manager daemon x restarted 2024-04-30T08:23:29.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:29 smithi032 ceph-mon[28331]: Activating manager daemon x 2024-04-30T08:23:29.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:29 smithi032 ceph-mon[28331]: osdmap e3: 0 total, 0 up, 0 in 2024-04-30T08:23:29.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:29 smithi032 ceph-mon[28331]: mgrmap e10: x(active, starting, since 0.0114595s) 2024-04-30T08:23:29.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:29 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-30T08:23:29.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:29 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-30T08:23:29.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:29 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-30T08:23:29.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:29 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-30T08:23:29.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:29 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-30T08:23:29.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:29 smithi032 ceph-mon[28331]: Manager daemon x is now available 2024-04-30T08:23:30.906 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:30 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:23:30.906 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:30 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-30T08:23:30.907 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:30 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-30T08:23:30.907 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:30 smithi032 ceph-mon[28331]: mgrmap e11: x(active, since 1.01121s) 2024-04-30T08:23:31.641 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout { 2024-04-30T08:23:31.641 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-30T08:23:31.641 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-30T08:23:31.641 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout } 2024-04-30T08:23:31.641 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:23.997+0000 7f739ce00700 1 Processor -- start 2024-04-30T08:23:31.641 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:23.998+0000 7f739ce00700 1 -- start start 2024-04-30T08:23:31.641 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:23.998+0000 7f739ce00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f739810be40 0x7f739810c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:31.641 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:23.998+0000 7f739ce00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f739810c840 con 0x7f739810be40 2024-04-30T08:23:31.641 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:23.998+0000 7f7395e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f739810be40 0x7f739810c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:31.641 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:23.998+0000 7f7395e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f739810be40 0x7f739810c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57056/0 (socket says 172.21.15.32:57056) 2024-04-30T08:23:31.642 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:23.999+0000 7f7395e00700 1 -- 172.21.15.32:0/2676709519 learned_addr learned my addr 172.21.15.32:0/2676709519 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:23:31.642 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:23.999+0000 7f7395e00700 1 -- 172.21.15.32:0/2676709519 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f739810cf20 con 0x7f739810be40 2024-04-30T08:23:31.642 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:23.999+0000 7f7395e00700 1 --2- 172.21.15.32:0/2676709519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f739810be40 0x7f739810c250 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f73880098a0 tx=0x7f7388009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22456cdc6ec6b20b server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:31.642 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:24.000+0000 7f7394a00700 1 -- 172.21.15.32:0/2676709519 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7388004160 con 0x7f739810be40 2024-04-30T08:23:31.642 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:24.000+0000 7f7394a00700 1 -- 172.21.15.32:0/2676709519 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7388004300 con 0x7f739810be40 2024-04-30T08:23:31.642 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:24.000+0000 7f7394a00700 1 -- 172.21.15.32:0/2676709519 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7388011a20 con 0x7f739810be40 2024-04-30T08:23:31.642 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:24.000+0000 7f739ce00700 1 -- 172.21.15.32:0/2676709519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f739810be40 msgr2=0x7f739810c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:31.642 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:24.000+0000 7f739ce00700 1 --2- 172.21.15.32:0/2676709519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f739810be40 0x7f739810c250 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f73880098a0 tx=0x7f7388009bb0 comp rx=0 tx=0).stop 2024-04-30T08:23:31.642 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:24.001+0000 7f739ce00700 1 -- 172.21.15.32:0/2676709519 shutdown_connections 2024-04-30T08:23:31.642 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:24.001+0000 7f739ce00700 1 --2- 172.21.15.32:0/2676709519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f739810be40 0x7f739810c250 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:31.642 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:24.001+0000 7f739ce00700 1 -- 172.21.15.32:0/2676709519 >> 172.21.15.32:0/2676709519 conn(0x7f7398107680 msgr2=0x7f7398109aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:31.642 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:24.001+0000 7f739ce00700 1 -- 172.21.15.32:0/2676709519 shutdown_connections 2024-04-30T08:23:31.643 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:24.001+0000 7f739ce00700 1 -- 172.21.15.32:0/2676709519 wait complete. 2024-04-30T08:23:31.643 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:24.002+0000 7f739ce00700 1 Processor -- start 2024-04-30T08:23:31.643 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:24.002+0000 7f739ce00700 1 -- start start 2024-04-30T08:23:31.643 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:24.002+0000 7f739ce00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f739810be40 0x7f73981a0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:31.643 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:24.002+0000 7f739ce00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f73981102d0 con 0x7f739810be40 2024-04-30T08:23:31.643 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:24.003+0000 7f7395e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f739810be40 0x7f73981a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:31.643 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:24.003+0000 7f7395e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f739810be40 0x7f73981a0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57058/0 (socket says 172.21.15.32:57058) 2024-04-30T08:23:31.643 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:24.003+0000 7f7395e00700 1 -- 172.21.15.32:0/4279667520 learned_addr learned my addr 172.21.15.32:0/4279667520 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:23:31.643 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:24.003+0000 7f7395e00700 1 -- 172.21.15.32:0/4279667520 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f73981a06a0 con 0x7f739810be40 2024-04-30T08:23:31.643 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:24.003+0000 7f7395e00700 1 --2- 172.21.15.32:0/4279667520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f739810be40 0x7f73981a0160 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7388003e70 tx=0x7f7388011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:31.643 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:24.004+0000 7f738ea00700 1 -- 172.21.15.32:0/4279667520 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7388003aa0 con 0x7f739810be40 2024-04-30T08:23:31.643 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:24.004+0000 7f738ea00700 1 -- 172.21.15.32:0/4279667520 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f738801a8e0 con 0x7f739810be40 2024-04-30T08:23:31.643 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:24.004+0000 7f739ce00700 1 -- 172.21.15.32:0/4279667520 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f73981a0930 con 0x7f739810be40 2024-04-30T08:23:31.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:24.004+0000 7f738ea00700 1 -- 172.21.15.32:0/4279667520 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7388024870 con 0x7f739810be40 2024-04-30T08:23:31.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:24.004+0000 7f739ce00700 1 -- 172.21.15.32:0/4279667520 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f73981a3540 con 0x7f739810be40 2024-04-30T08:23:31.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:24.005+0000 7f738ea00700 1 -- 172.21.15.32:0/4279667520 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 9) ==== 49529+0+0 (secure 0 0 0) 0x7f7388022020 con 0x7f739810be40 2024-04-30T08:23:31.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:24.006+0000 7f738ea00700 1 --2- 172.21.15.32:0/4279667520 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f737c03d5c0 0x7f737c03fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:31.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:24.006+0000 7f738ea00700 1 -- 172.21.15.32:0/4279667520 --> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f737c0401e0 con 0x7f737c03d5c0 2024-04-30T08:23:31.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:24.006+0000 7f738ea00700 1 -- 172.21.15.32:0/4279667520 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7388052900 con 0x7f739810be40 2024-04-30T08:23:31.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:24.006+0000 7f7395400700 1 -- 172.21.15.32:0/4279667520 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f737c03d5c0 msgr2=0x7f737c03fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/4019351306 2024-04-30T08:23:31.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:24.006+0000 7f7395400700 1 --2- 172.21.15.32:0/4279667520 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f737c03d5c0 0x7f737c03fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-30T08:23:31.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:24.207+0000 7f7395400700 1 -- 172.21.15.32:0/4279667520 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f737c03d5c0 msgr2=0x7f737c03fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/4019351306 2024-04-30T08:23:31.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:24.207+0000 7f7395400700 1 --2- 172.21.15.32:0/4279667520 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f737c03d5c0 0x7f737c03fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-30T08:23:31.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:24.608+0000 7f7395400700 1 -- 172.21.15.32:0/4279667520 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f737c03d5c0 msgr2=0x7f737c03fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/4019351306 2024-04-30T08:23:31.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:24.608+0000 7f7395400700 1 --2- 172.21.15.32:0/4279667520 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f737c03d5c0 0x7f737c03fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-30T08:23:31.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:25.409+0000 7f7395400700 1 -- 172.21.15.32:0/4279667520 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f737c03d5c0 msgr2=0x7f737c03fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/4019351306 2024-04-30T08:23:31.644 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:25.409+0000 7f7395400700 1 --2- 172.21.15.32:0/4279667520 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f737c03d5c0 0x7f737c03fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-30T08:23:31.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:27.011+0000 7f7395400700 1 -- 172.21.15.32:0/4279667520 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f737c03d5c0 msgr2=0x7f737c03fad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.32:6800/4019351306 2024-04-30T08:23:31.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:27.011+0000 7f7395400700 1 --2- 172.21.15.32:0/4279667520 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f737c03d5c0 0x7f737c03fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-30T08:23:31.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:29.532+0000 7f738ea00700 1 -- 172.21.15.32:0/4279667520 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mgrmap(e 10) ==== 49296+0+0 (secure 0 0 0) 0x7f738802d5d0 con 0x7f739810be40 2024-04-30T08:23:31.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:29.532+0000 7f738ea00700 1 -- 172.21.15.32:0/4279667520 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f737c03d5c0 msgr2=0x7f737c03fad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-30T08:23:31.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:29.532+0000 7f738ea00700 1 --2- 172.21.15.32:0/4279667520 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f737c03d5c0 0x7f737c03fad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:31.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:30.534+0000 7f738ea00700 1 -- 172.21.15.32:0/4279667520 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mgrmap(e 11) ==== 49423+0+0 (secure 0 0 0) 0x7f7388054510 con 0x7f739810be40 2024-04-30T08:23:31.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:30.534+0000 7f738ea00700 1 --2- 172.21.15.32:0/4279667520 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f7388052490 0x7f737c043000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:31.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:30.534+0000 7f738ea00700 1 -- 172.21.15.32:0/4279667520 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f7388052000 con 0x7f7388052490 2024-04-30T08:23:31.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:30.535+0000 7f7395400700 1 --2- 172.21.15.32:0/4279667520 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f7388052490 0x7f737c043000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:31.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:30.538+0000 7f7395400700 1 --2- 172.21.15.32:0/4279667520 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f7388052490 0x7f737c043000 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7384003a10 tx=0x7f73840092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:31.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:30.539+0000 7f738ea00700 1 -- 172.21.15.32:0/4279667520 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== command_reply(tid 0: 0 ) ==== 8+0+9564 (secure 0 0 0) 0x7f7388052000 con 0x7f7388052490 2024-04-30T08:23:31.645 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:30.552+0000 7f739ce00700 1 -- 172.21.15.32:0/4279667520 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f7398064d70 con 0x7f7388052490 2024-04-30T08:23:31.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:30.552+0000 7f738ea00700 1 -- 172.21.15.32:0/4279667520 <== mgr.14152 v2:172.21.15.32:6800/2930853529 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f7398064d70 con 0x7f7388052490 2024-04-30T08:23:31.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:30.553+0000 7f739ce00700 1 -- 172.21.15.32:0/4279667520 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f7388052490 msgr2=0x7f737c043000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:31.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:30.553+0000 7f739ce00700 1 --2- 172.21.15.32:0/4279667520 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f7388052490 0x7f737c043000 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f7384003a10 tx=0x7f73840092b0 comp rx=0 tx=0).stop 2024-04-30T08:23:31.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:30.553+0000 7f739ce00700 1 -- 172.21.15.32:0/4279667520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f739810be40 msgr2=0x7f73981a0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:31.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:30.553+0000 7f739ce00700 1 --2- 172.21.15.32:0/4279667520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f739810be40 0x7f73981a0160 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f7388003e70 tx=0x7f7388011aa0 comp rx=0 tx=0).stop 2024-04-30T08:23:31.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:30.553+0000 7f739ce00700 1 -- 172.21.15.32:0/4279667520 shutdown_connections 2024-04-30T08:23:31.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:30.553+0000 7f739ce00700 1 --2- 172.21.15.32:0/4279667520 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f7388052490 0x7f737c043000 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:31.647 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:30.553+0000 7f739ce00700 1 --2- 172.21.15.32:0/4279667520 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f739810be40 0x7f73981a0160 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:31.648 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:30.553+0000 7f739ce00700 1 --2- 172.21.15.32:0/4279667520 >> [v2:172.21.15.32:6800/4019351306,v1:172.21.15.32:6801/4019351306] conn(0x7f737c03d5c0 0x7f737c03fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:31.648 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:30.553+0000 7f739ce00700 1 -- 172.21.15.32:0/4279667520 >> 172.21.15.32:0/4279667520 conn(0x7f7398107680 msgr2=0x7f7398108f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:31.648 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:30.553+0000 7f739ce00700 1 -- 172.21.15.32:0/4279667520 shutdown_connections 2024-04-30T08:23:31.648 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:30.553+0000 7f739ce00700 1 -- 172.21.15.32:0/4279667520 wait complete. 2024-04-30T08:23:31.648 INFO:teuthology.orchestra.run.smithi032.stdout:mgr epoch 9 is available 2024-04-30T08:23:31.648 INFO:teuthology.orchestra.run.smithi032.stdout:Generating a dashboard self-signed certificate... 2024-04-30T08:23:31.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:31 smithi032 ceph-mon[28331]: [30/Apr/2024:08:23:30] ENGINE Bus STARTING 2024-04-30T08:23:31.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:31 smithi032 ceph-mon[28331]: [30/Apr/2024:08:23:30] ENGINE Serving on https://172.21.15.32:7150 2024-04-30T08:23:31.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:31 smithi032 ceph-mon[28331]: [30/Apr/2024:08:23:31] ENGINE Serving on http://172.21.15.32:8765 2024-04-30T08:23:31.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:31 smithi032 ceph-mon[28331]: [30/Apr/2024:08:23:31] ENGINE Bus STARTED 2024-04-30T08:23:32.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:32 smithi032 ceph-mon[28331]: mgrmap e12: x(active, since 2s) 2024-04-30T08:23:32.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:32 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:23:33.856 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-30T08:23:33.856 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.783+0000 7f9262400700 1 Processor -- start 2024-04-30T08:23:33.856 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.783+0000 7f9262400700 1 -- start start 2024-04-30T08:23:33.856 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.784+0000 7f9262400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f925c07ed40 0x7f925c07d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:33.856 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.784+0000 7f9262400700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f925c07d750 con 0x7f925c07ed40 2024-04-30T08:23:33.856 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.784+0000 7f925b400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f925c07ed40 0x7f925c07d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:33.856 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.784+0000 7f925b400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f925c07ed40 0x7f925c07d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60534/0 (socket says 172.21.15.32:60534) 2024-04-30T08:23:33.856 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.784+0000 7f925b400700 1 -- 172.21.15.32:0/2438125696 learned_addr learned my addr 172.21.15.32:0/2438125696 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:23:33.856 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.785+0000 7f925b400700 1 -- 172.21.15.32:0/2438125696 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f925c07d8d0 con 0x7f925c07ed40 2024-04-30T08:23:33.857 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.785+0000 7f925b400700 1 --2- 172.21.15.32:0/2438125696 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f925c07ed40 0x7f925c07d210 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f924c0098a0 tx=0x7f924c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=635117ceb4a13b4f server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:33.857 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.786+0000 7f925a000700 1 -- 172.21.15.32:0/2438125696 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f924c004160 con 0x7f925c07ed40 2024-04-30T08:23:33.857 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.786+0000 7f925a000700 1 -- 172.21.15.32:0/2438125696 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f924c004300 con 0x7f925c07ed40 2024-04-30T08:23:33.857 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.786+0000 7f925a000700 1 -- 172.21.15.32:0/2438125696 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f924c011a20 con 0x7f925c07ed40 2024-04-30T08:23:33.857 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.787+0000 7f9262400700 1 -- 172.21.15.32:0/2438125696 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f925c07ed40 msgr2=0x7f925c07d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:33.857 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.787+0000 7f9262400700 1 --2- 172.21.15.32:0/2438125696 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f925c07ed40 0x7f925c07d210 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f924c0098a0 tx=0x7f924c009bb0 comp rx=0 tx=0).stop 2024-04-30T08:23:33.857 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.787+0000 7f9262400700 1 -- 172.21.15.32:0/2438125696 shutdown_connections 2024-04-30T08:23:33.857 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.787+0000 7f9262400700 1 --2- 172.21.15.32:0/2438125696 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f925c07ed40 0x7f925c07d210 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:33.857 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.787+0000 7f9262400700 1 -- 172.21.15.32:0/2438125696 >> 172.21.15.32:0/2438125696 conn(0x7f925c105370 msgr2=0x7f925c1077e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:33.857 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.787+0000 7f9262400700 1 -- 172.21.15.32:0/2438125696 shutdown_connections 2024-04-30T08:23:33.857 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.787+0000 7f9262400700 1 -- 172.21.15.32:0/2438125696 wait complete. 2024-04-30T08:23:33.857 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.788+0000 7f9262400700 1 Processor -- start 2024-04-30T08:23:33.857 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.788+0000 7f9262400700 1 -- start start 2024-04-30T08:23:33.857 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.789+0000 7f9262400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f925c07ed40 0x7f925c1a01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:33.858 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.789+0000 7f9262400700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f925c110270 con 0x7f925c07ed40 2024-04-30T08:23:33.858 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.789+0000 7f925b400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f925c07ed40 0x7f925c1a01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:33.858 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.789+0000 7f925b400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f925c07ed40 0x7f925c1a01a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60548/0 (socket says 172.21.15.32:60548) 2024-04-30T08:23:33.858 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.789+0000 7f925b400700 1 -- 172.21.15.32:0/3793338938 learned_addr learned my addr 172.21.15.32:0/3793338938 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:23:33.858 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.790+0000 7f925b400700 1 -- 172.21.15.32:0/3793338938 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f925c1a06e0 con 0x7f925c07ed40 2024-04-30T08:23:33.858 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.790+0000 7f925b400700 1 --2- 172.21.15.32:0/3793338938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f925c07ed40 0x7f925c1a01a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f924c000c00 tx=0x7f924c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:33.858 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.790+0000 7f9253e00700 1 -- 172.21.15.32:0/3793338938 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f924c003aa0 con 0x7f925c07ed40 2024-04-30T08:23:33.858 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.790+0000 7f9253e00700 1 -- 172.21.15.32:0/3793338938 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f924c01b8f0 con 0x7f925c07ed40 2024-04-30T08:23:33.858 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.790+0000 7f9262400700 1 -- 172.21.15.32:0/3793338938 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f925c1a0970 con 0x7f925c07ed40 2024-04-30T08:23:33.858 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.790+0000 7f9253e00700 1 -- 172.21.15.32:0/3793338938 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f924c023890 con 0x7f925c07ed40 2024-04-30T08:23:33.858 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.791+0000 7f9262400700 1 -- 172.21.15.32:0/3793338938 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f925c1a3580 con 0x7f925c07ed40 2024-04-30T08:23:33.858 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.792+0000 7f9262400700 1 -- 172.21.15.32:0/3793338938 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f925c064d70 con 0x7f925c07ed40 2024-04-30T08:23:33.858 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.792+0000 7f9253e00700 1 -- 172.21.15.32:0/3793338938 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 12) ==== 49529+0+0 (secure 0 0 0) 0x7f924c01b440 con 0x7f925c07ed40 2024-04-30T08:23:33.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.793+0000 7f9253e00700 1 --2- 172.21.15.32:0/3793338938 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f924003d5c0 0x7f924003fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:33.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.793+0000 7f9253e00700 1 -- 172.21.15.32:0/3793338938 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f924c052d50 con 0x7f925c07ed40 2024-04-30T08:23:33.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.793+0000 7f925aa00700 1 --2- 172.21.15.32:0/3793338938 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f924003d5c0 0x7f924003fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:33.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.794+0000 7f925aa00700 1 --2- 172.21.15.32:0/3793338938 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f924003d5c0 0x7f924003fad0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9248006fd0 tx=0x7f9248006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:33.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:32.799+0000 7f9253e00700 1 -- 172.21.15.32:0/3793338938 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f924c056030 con 0x7f925c07ed40 2024-04-30T08:23:33.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:33.039+0000 7f9262400700 1 -- 172.21.15.32:0/3793338938 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f925c106320 con 0x7f924003d5c0 2024-04-30T08:23:33.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:33.189+0000 7f9253e00700 1 -- 172.21.15.32:0/3793338938 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f925c106320 con 0x7f924003d5c0 2024-04-30T08:23:33.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:33.192+0000 7f9262400700 1 -- 172.21.15.32:0/3793338938 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f924003d5c0 msgr2=0x7f924003fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:33.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:33.192+0000 7f9262400700 1 --2- 172.21.15.32:0/3793338938 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f924003d5c0 0x7f924003fad0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9248006fd0 tx=0x7f9248006e40 comp rx=0 tx=0).stop 2024-04-30T08:23:33.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:33.192+0000 7f9262400700 1 -- 172.21.15.32:0/3793338938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f925c07ed40 msgr2=0x7f925c1a01a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:33.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:33.192+0000 7f9262400700 1 --2- 172.21.15.32:0/3793338938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f925c07ed40 0x7f925c1a01a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f924c000c00 tx=0x7f924c011aa0 comp rx=0 tx=0).stop 2024-04-30T08:23:33.859 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:33.192+0000 7f9262400700 1 -- 172.21.15.32:0/3793338938 shutdown_connections 2024-04-30T08:23:33.860 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:33.192+0000 7f9262400700 1 --2- 172.21.15.32:0/3793338938 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f924003d5c0 0x7f924003fad0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:33.860 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:33.192+0000 7f9262400700 1 --2- 172.21.15.32:0/3793338938 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f925c07ed40 0x7f925c1a01a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:33.860 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:33.192+0000 7f9262400700 1 -- 172.21.15.32:0/3793338938 >> 172.21.15.32:0/3793338938 conn(0x7f925c105370 msgr2=0x7f925c105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:33.860 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:33.193+0000 7f9262400700 1 -- 172.21.15.32:0/3793338938 shutdown_connections 2024-04-30T08:23:33.860 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:33.193+0000 7f9262400700 1 -- 172.21.15.32:0/3793338938 wait complete. 2024-04-30T08:23:33.860 INFO:teuthology.orchestra.run.smithi032.stdout:Creating initial admin user... 2024-04-30T08:23:34.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:34 smithi032 ceph-mon[28331]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:23:34.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:34 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:23:34.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:34 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:23:35.764 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:35 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:23:35.764 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:35 smithi032 ceph-mon[28331]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:23:35.764 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:35 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:23:35.770 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$U.yRqQSnyj1x73joR9HlXuyp9lF3PeX0F5g3ZkV8gxLP/mrM1rpUK", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1714465415, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-30T08:23:35.771 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.541+0000 7f2548e00700 1 Processor -- start 2024-04-30T08:23:35.772 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.541+0000 7f2548e00700 1 -- start start 2024-04-30T08:23:35.772 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.541+0000 7f2548e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f254407ed60 0x7f254407d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:35.772 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.541+0000 7f2548e00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f254407d770 con 0x7f254407ed60 2024-04-30T08:23:35.772 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.542+0000 7f2541e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f254407ed60 0x7f254407d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:35.772 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.542+0000 7f2541e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f254407ed60 0x7f254407d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60550/0 (socket says 172.21.15.32:60550) 2024-04-30T08:23:35.772 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.542+0000 7f2541e00700 1 -- 172.21.15.32:0/749519718 learned_addr learned my addr 172.21.15.32:0/749519718 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:23:35.773 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.542+0000 7f2541e00700 1 -- 172.21.15.32:0/749519718 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f254407d8f0 con 0x7f254407ed60 2024-04-30T08:23:35.773 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.543+0000 7f2541e00700 1 --2- 172.21.15.32:0/749519718 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f254407ed60 0x7f254407d230 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f25340098a0 tx=0x7f2534009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7106ce48a78876a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:35.773 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.543+0000 7f2540a00700 1 -- 172.21.15.32:0/749519718 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2534004160 con 0x7f254407ed60 2024-04-30T08:23:35.773 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.543+0000 7f2540a00700 1 -- 172.21.15.32:0/749519718 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2534004300 con 0x7f254407ed60 2024-04-30T08:23:35.773 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.543+0000 7f2540a00700 1 -- 172.21.15.32:0/749519718 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2534011a20 con 0x7f254407ed60 2024-04-30T08:23:35.775 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.544+0000 7f2548e00700 1 -- 172.21.15.32:0/749519718 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f254407ed60 msgr2=0x7f254407d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:35.775 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.544+0000 7f2548e00700 1 --2- 172.21.15.32:0/749519718 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f254407ed60 0x7f254407d230 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f25340098a0 tx=0x7f2534009bb0 comp rx=0 tx=0).stop 2024-04-30T08:23:35.775 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.544+0000 7f2548e00700 1 -- 172.21.15.32:0/749519718 shutdown_connections 2024-04-30T08:23:35.775 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.544+0000 7f2548e00700 1 --2- 172.21.15.32:0/749519718 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f254407ed60 0x7f254407d230 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:35.775 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.544+0000 7f2548e00700 1 -- 172.21.15.32:0/749519718 >> 172.21.15.32:0/749519718 conn(0x7f2544105350 msgr2=0x7f25441077a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:35.775 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.544+0000 7f2548e00700 1 -- 172.21.15.32:0/749519718 shutdown_connections 2024-04-30T08:23:35.775 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.545+0000 7f2548e00700 1 -- 172.21.15.32:0/749519718 wait complete. 2024-04-30T08:23:35.775 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.546+0000 7f2548e00700 1 Processor -- start 2024-04-30T08:23:35.775 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.546+0000 7f2548e00700 1 -- start start 2024-04-30T08:23:35.776 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.546+0000 7f2548e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f254407ed60 0x7f25441a0160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:35.776 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.546+0000 7f2548e00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2544110270 con 0x7f254407ed60 2024-04-30T08:23:35.776 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.547+0000 7f2541e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f254407ed60 0x7f25441a0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:35.776 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.547+0000 7f2541e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f254407ed60 0x7f25441a0160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60552/0 (socket says 172.21.15.32:60552) 2024-04-30T08:23:35.776 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.547+0000 7f2541e00700 1 -- 172.21.15.32:0/976819926 learned_addr learned my addr 172.21.15.32:0/976819926 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:23:35.776 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.547+0000 7f2541e00700 1 -- 172.21.15.32:0/976819926 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25441a06a0 con 0x7f254407ed60 2024-04-30T08:23:35.776 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.547+0000 7f2541e00700 1 --2- 172.21.15.32:0/976819926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f254407ed60 0x7f25441a0160 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2534009520 tx=0x7f2534011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:35.776 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.548+0000 7f253aa00700 1 -- 172.21.15.32:0/976819926 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f25340039e0 con 0x7f254407ed60 2024-04-30T08:23:35.776 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.548+0000 7f253aa00700 1 -- 172.21.15.32:0/976819926 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f253401b8f0 con 0x7f254407ed60 2024-04-30T08:23:35.779 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.548+0000 7f253aa00700 1 -- 172.21.15.32:0/976819926 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f25340197d0 con 0x7f254407ed60 2024-04-30T08:23:35.779 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.548+0000 7f2548e00700 1 -- 172.21.15.32:0/976819926 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25441a0930 con 0x7f254407ed60 2024-04-30T08:23:35.779 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.548+0000 7f2548e00700 1 -- 172.21.15.32:0/976819926 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25441a3540 con 0x7f254407ed60 2024-04-30T08:23:35.779 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.549+0000 7f2548e00700 1 -- 172.21.15.32:0/976819926 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2544064d70 con 0x7f254407ed60 2024-04-30T08:23:35.779 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.549+0000 7f253aa00700 1 -- 172.21.15.32:0/976819926 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 12) ==== 49529+0+0 (secure 0 0 0) 0x7f253401b440 con 0x7f254407ed60 2024-04-30T08:23:35.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.550+0000 7f253aa00700 1 --2- 172.21.15.32:0/976819926 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f252803d620 0x7f252803fb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:35.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.550+0000 7f253aa00700 1 -- 172.21.15.32:0/976819926 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f2534052ef0 con 0x7f254407ed60 2024-04-30T08:23:35.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.550+0000 7f2541400700 1 --2- 172.21.15.32:0/976819926 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f252803d620 0x7f252803fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:35.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.551+0000 7f2541400700 1 --2- 172.21.15.32:0/976819926 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f252803d620 0x7f252803fb30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2530006fd0 tx=0x7f2530006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:35.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.556+0000 7f253aa00700 1 -- 172.21.15.32:0/976819926 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f253401f030 con 0x7f254407ed60 2024-04-30T08:23:35.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:34.828+0000 7f2548e00700 1 -- 172.21.15.32:0/976819926 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f25441a37d0 con 0x7f252803d620 2024-04-30T08:23:35.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:35.087+0000 7f253aa00700 1 -- 172.21.15.32:0/976819926 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f25441a37d0 con 0x7f252803d620 2024-04-30T08:23:35.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:35.090+0000 7f2548e00700 1 -- 172.21.15.32:0/976819926 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f252803d620 msgr2=0x7f252803fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:35.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:35.090+0000 7f2548e00700 1 --2- 172.21.15.32:0/976819926 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f252803d620 0x7f252803fb30 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2530006fd0 tx=0x7f2530006e40 comp rx=0 tx=0).stop 2024-04-30T08:23:35.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:35.090+0000 7f2548e00700 1 -- 172.21.15.32:0/976819926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f254407ed60 msgr2=0x7f25441a0160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:35.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:35.090+0000 7f2548e00700 1 --2- 172.21.15.32:0/976819926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f254407ed60 0x7f25441a0160 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f2534009520 tx=0x7f2534011aa0 comp rx=0 tx=0).stop 2024-04-30T08:23:35.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:35.090+0000 7f2548e00700 1 -- 172.21.15.32:0/976819926 shutdown_connections 2024-04-30T08:23:35.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:35.090+0000 7f2548e00700 1 --2- 172.21.15.32:0/976819926 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f252803d620 0x7f252803fb30 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:35.780 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:35.090+0000 7f2548e00700 1 --2- 172.21.15.32:0/976819926 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f254407ed60 0x7f25441a0160 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:35.781 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:35.090+0000 7f2548e00700 1 -- 172.21.15.32:0/976819926 >> 172.21.15.32:0/976819926 conn(0x7f2544105350 msgr2=0x7f2544105bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:35.781 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:35.091+0000 7f2548e00700 1 -- 172.21.15.32:0/976819926 shutdown_connections 2024-04-30T08:23:35.781 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:35.091+0000 7f2548e00700 1 -- 172.21.15.32:0/976819926 wait complete. 2024-04-30T08:23:35.781 INFO:teuthology.orchestra.run.smithi032.stdout:Fetching dashboard port number... 2024-04-30T08:23:37.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:37 smithi032 ceph-mon[28331]: mgrmap e13: x(active, since 6s) 2024-04-30T08:23:37.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:37 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1022052102' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-30T08:23:37.959 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stdout 8443 2024-04-30T08:23:37.968 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.651+0000 7f67f7200700 1 Processor -- start 2024-04-30T08:23:37.968 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.651+0000 7f67f7200700 1 -- start start 2024-04-30T08:23:37.968 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.652+0000 7f67f7200700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f007ed60 0x7f67f007d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:37.975 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.652+0000 7f67f7200700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f67f007d770 con 0x7f67f007ed60 2024-04-30T08:23:37.975 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.652+0000 7f67efe00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f007ed60 0x7f67f007d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:37.975 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.652+0000 7f67efe00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f007ed60 0x7f67f007d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60558/0 (socket says 172.21.15.32:60558) 2024-04-30T08:23:37.975 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.652+0000 7f67efe00700 1 -- 172.21.15.32:0/2027592285 learned_addr learned my addr 172.21.15.32:0/2027592285 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:23:37.975 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.653+0000 7f67efe00700 1 -- 172.21.15.32:0/2027592285 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67f007d8f0 con 0x7f67f007ed60 2024-04-30T08:23:37.976 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.653+0000 7f67efe00700 1 --2- 172.21.15.32:0/2027592285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f007ed60 0x7f67f007d230 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f67e00098a0 tx=0x7f67e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e6985b169ec334f server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:37.976 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.654+0000 7f67eea00700 1 -- 172.21.15.32:0/2027592285 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f67e0004160 con 0x7f67f007ed60 2024-04-30T08:23:37.976 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.654+0000 7f67eea00700 1 -- 172.21.15.32:0/2027592285 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f67e0004300 con 0x7f67f007ed60 2024-04-30T08:23:37.976 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.654+0000 7f67eea00700 1 -- 172.21.15.32:0/2027592285 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f67e0011a20 con 0x7f67f007ed60 2024-04-30T08:23:37.976 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.655+0000 7f67f7200700 1 -- 172.21.15.32:0/2027592285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f007ed60 msgr2=0x7f67f007d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:37.976 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.655+0000 7f67f7200700 1 --2- 172.21.15.32:0/2027592285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f007ed60 0x7f67f007d230 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f67e00098a0 tx=0x7f67e0009bb0 comp rx=0 tx=0).stop 2024-04-30T08:23:37.976 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.655+0000 7f67f7200700 1 -- 172.21.15.32:0/2027592285 shutdown_connections 2024-04-30T08:23:37.976 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.655+0000 7f67f7200700 1 --2- 172.21.15.32:0/2027592285 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f007ed60 0x7f67f007d230 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:37.976 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.655+0000 7f67f7200700 1 -- 172.21.15.32:0/2027592285 >> 172.21.15.32:0/2027592285 conn(0x7f67f0105350 msgr2=0x7f67f01077a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:37.976 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.655+0000 7f67f7200700 1 -- 172.21.15.32:0/2027592285 shutdown_connections 2024-04-30T08:23:37.977 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.655+0000 7f67f7200700 1 -- 172.21.15.32:0/2027592285 wait complete. 2024-04-30T08:23:37.977 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.656+0000 7f67f7200700 1 Processor -- start 2024-04-30T08:23:37.977 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.657+0000 7f67f7200700 1 -- start start 2024-04-30T08:23:37.977 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.657+0000 7f67f7200700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f007ed60 0x7f67f01a0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:37.977 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.657+0000 7f67f7200700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f67f0110270 con 0x7f67f007ed60 2024-04-30T08:23:37.977 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.657+0000 7f67efe00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f007ed60 0x7f67f01a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:37.977 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.657+0000 7f67efe00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f007ed60 0x7f67f01a0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60574/0 (socket says 172.21.15.32:60574) 2024-04-30T08:23:37.978 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.657+0000 7f67efe00700 1 -- 172.21.15.32:0/1022052102 learned_addr learned my addr 172.21.15.32:0/1022052102 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:23:37.978 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.658+0000 7f67efe00700 1 -- 172.21.15.32:0/1022052102 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67f01a06c0 con 0x7f67f007ed60 2024-04-30T08:23:37.978 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.658+0000 7f67efe00700 1 --2- 172.21.15.32:0/1022052102 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f007ed60 0x7f67f01a0180 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f67e0009520 tx=0x7f67e0011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:37.978 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.658+0000 7f67ecc00700 1 -- 172.21.15.32:0/1022052102 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f67e00039e0 con 0x7f67f007ed60 2024-04-30T08:23:37.978 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.658+0000 7f67ecc00700 1 -- 172.21.15.32:0/1022052102 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f67e001b8f0 con 0x7f67f007ed60 2024-04-30T08:23:37.978 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.658+0000 7f67ecc00700 1 -- 172.21.15.32:0/1022052102 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f67e00197d0 con 0x7f67f007ed60 2024-04-30T08:23:37.978 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.659+0000 7f67f7200700 1 -- 172.21.15.32:0/1022052102 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67f01a0950 con 0x7f67f007ed60 2024-04-30T08:23:37.978 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.659+0000 7f67f7200700 1 -- 172.21.15.32:0/1022052102 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67f01a3560 con 0x7f67f007ed60 2024-04-30T08:23:37.978 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.660+0000 7f67f7200700 1 -- 172.21.15.32:0/1022052102 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67f0064d70 con 0x7f67f007ed60 2024-04-30T08:23:37.978 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.660+0000 7f67ecc00700 1 -- 172.21.15.32:0/1022052102 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f67e001b440 con 0x7f67f007ed60 2024-04-30T08:23:37.978 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.661+0000 7f67ecc00700 1 --2- 172.21.15.32:0/1022052102 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f67d403d6c0 0x7f67d403fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:37.978 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.661+0000 7f67ecc00700 1 -- 172.21.15.32:0/1022052102 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f67e0052ff0 con 0x7f67f007ed60 2024-04-30T08:23:37.979 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.662+0000 7f67ef400700 1 --2- 172.21.15.32:0/1022052102 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f67d403d6c0 0x7f67d403fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:37.979 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.662+0000 7f67ef400700 1 --2- 172.21.15.32:0/1022052102 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f67d403d6c0 0x7f67d403fbd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f67dc006fd0 tx=0x7f67dc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:37.979 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.668+0000 7f67ecc00700 1 -- 172.21.15.32:0/1022052102 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f67e001f030 con 0x7f67f007ed60 2024-04-30T08:23:37.979 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.903+0000 7f67f7200700 1 -- 172.21.15.32:0/1022052102 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f67f01a3730 con 0x7f67f007ed60 2024-04-30T08:23:37.979 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.904+0000 7f67ecc00700 1 -- 172.21.15.32:0/1022052102 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f67e000e260 con 0x7f67f007ed60 2024-04-30T08:23:37.979 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.907+0000 7f67f7200700 1 -- 172.21.15.32:0/1022052102 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f67d403d6c0 msgr2=0x7f67d403fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:37.979 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.907+0000 7f67f7200700 1 --2- 172.21.15.32:0/1022052102 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f67d403d6c0 0x7f67d403fbd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f67dc006fd0 tx=0x7f67dc006e40 comp rx=0 tx=0).stop 2024-04-30T08:23:37.980 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.907+0000 7f67f7200700 1 -- 172.21.15.32:0/1022052102 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f007ed60 msgr2=0x7f67f01a0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:37.980 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.907+0000 7f67f7200700 1 --2- 172.21.15.32:0/1022052102 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f007ed60 0x7f67f01a0180 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f67e0009520 tx=0x7f67e0011aa0 comp rx=0 tx=0).stop 2024-04-30T08:23:37.980 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.907+0000 7f67f7200700 1 -- 172.21.15.32:0/1022052102 shutdown_connections 2024-04-30T08:23:37.980 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.907+0000 7f67f7200700 1 --2- 172.21.15.32:0/1022052102 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f67d403d6c0 0x7f67d403fbd0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:37.980 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.907+0000 7f67f7200700 1 --2- 172.21.15.32:0/1022052102 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67f007ed60 0x7f67f01a0180 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:37.980 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.907+0000 7f67f7200700 1 -- 172.21.15.32:0/1022052102 >> 172.21.15.32:0/1022052102 conn(0x7f67f0105350 msgr2=0x7f67f0105c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:37.980 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.907+0000 7f67f7200700 1 -- 172.21.15.32:0/1022052102 shutdown_connections 2024-04-30T08:23:37.980 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:36.907+0000 7f67f7200700 1 -- 172.21.15.32:0/1022052102 wait complete. 2024-04-30T08:23:38.007 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-30T08:23:38.008 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout disabled 2024-04-30T08:23:38.035 INFO:teuthology.orchestra.run.smithi032.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-30T08:23:38.035 INFO:teuthology.orchestra.run.smithi032.stdout:systemctl: stdout inactive 2024-04-30T08:23:38.035 INFO:teuthology.orchestra.run.smithi032.stdout:firewalld.service is not enabled 2024-04-30T08:23:38.036 INFO:teuthology.orchestra.run.smithi032.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-30T08:23:38.043 INFO:teuthology.orchestra.run.smithi032.stdout:Ceph Dashboard is now available at: 2024-04-30T08:23:38.043 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:23:38.043 INFO:teuthology.orchestra.run.smithi032.stdout: URL: https://smithi032.front.sepia.ceph.com:8443/ 2024-04-30T08:23:38.043 INFO:teuthology.orchestra.run.smithi032.stdout: User: admin 2024-04-30T08:23:38.044 INFO:teuthology.orchestra.run.smithi032.stdout: Password: 67po839m9o 2024-04-30T08:23:38.044 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:23:38.045 INFO:teuthology.orchestra.run.smithi032.stdout:Saving cluster configuration to /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/config directory 2024-04-30T08:23:38.046 INFO:teuthology.orchestra.run.smithi032.stdout:Enabling autotune for osd_memory_target 2024-04-30T08:23:39.652 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.737+0000 7f4aa1800700 1 Processor -- start 2024-04-30T08:23:39.653 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.737+0000 7f4aa1800700 1 -- start start 2024-04-30T08:23:39.653 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.738+0000 7f4aa1800700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a9c110160 0x7f4a9c110590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:39.653 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.738+0000 7f4aa1800700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9c110ba0 con 0x7f4a9c110160 2024-04-30T08:23:39.653 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.738+0000 7f4a9aa00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a9c110160 0x7f4a9c110590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:39.653 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.738+0000 7f4a9aa00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a9c110160 0x7f4a9c110590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54990/0 (socket says 172.21.15.32:54990) 2024-04-30T08:23:39.653 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.738+0000 7f4a9aa00700 1 -- 172.21.15.32:0/1281828334 learned_addr learned my addr 172.21.15.32:0/1281828334 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:23:39.653 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.739+0000 7f4a9aa00700 1 -- 172.21.15.32:0/1281828334 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a9c111300 con 0x7f4a9c110160 2024-04-30T08:23:39.653 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.739+0000 7f4a9aa00700 1 --2- 172.21.15.32:0/1281828334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a9c110160 0x7f4a9c110590 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c0098a0 tx=0x7f4a8c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5ffd559e6232fd5 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:39.654 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.739+0000 7f4a9a000700 1 -- 172.21.15.32:0/1281828334 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a8c004160 con 0x7f4a9c110160 2024-04-30T08:23:39.654 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.739+0000 7f4a9a000700 1 -- 172.21.15.32:0/1281828334 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a8c004300 con 0x7f4a9c110160 2024-04-30T08:23:39.654 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.739+0000 7f4a9a000700 1 -- 172.21.15.32:0/1281828334 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a8c011a20 con 0x7f4a9c110160 2024-04-30T08:23:39.654 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.740+0000 7f4aa1800700 1 -- 172.21.15.32:0/1281828334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a9c110160 msgr2=0x7f4a9c110590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:39.654 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.740+0000 7f4aa1800700 1 --2- 172.21.15.32:0/1281828334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a9c110160 0x7f4a9c110590 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c0098a0 tx=0x7f4a8c009bb0 comp rx=0 tx=0).stop 2024-04-30T08:23:39.654 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.740+0000 7f4aa1800700 1 -- 172.21.15.32:0/1281828334 shutdown_connections 2024-04-30T08:23:39.654 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.740+0000 7f4aa1800700 1 --2- 172.21.15.32:0/1281828334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a9c110160 0x7f4a9c110590 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:39.654 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.740+0000 7f4aa1800700 1 -- 172.21.15.32:0/1281828334 >> 172.21.15.32:0/1281828334 conn(0x7f4a9c07f330 msgr2=0x7f4a9c0817e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:39.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.741+0000 7f4aa1800700 1 -- 172.21.15.32:0/1281828334 shutdown_connections 2024-04-30T08:23:39.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.741+0000 7f4aa1800700 1 -- 172.21.15.32:0/1281828334 wait complete. 2024-04-30T08:23:39.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.742+0000 7f4aa1800700 1 Processor -- start 2024-04-30T08:23:39.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.742+0000 7f4aa1800700 1 -- start start 2024-04-30T08:23:39.655 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.742+0000 7f4aa1800700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a9c110160 0x7f4a9c1a44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:39.656 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.742+0000 7f4aa1800700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9c1145c0 con 0x7f4a9c110160 2024-04-30T08:23:39.656 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.742+0000 7f4a9aa00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a9c110160 0x7f4a9c1a44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:39.656 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.742+0000 7f4a9aa00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a9c110160 0x7f4a9c1a44f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54998/0 (socket says 172.21.15.32:54998) 2024-04-30T08:23:39.656 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.742+0000 7f4a9aa00700 1 -- 172.21.15.32:0/311857411 learned_addr learned my addr 172.21.15.32:0/311857411 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:23:39.656 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.743+0000 7f4a9aa00700 1 -- 172.21.15.32:0/311857411 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a9c1a4a30 con 0x7f4a9c110160 2024-04-30T08:23:39.656 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.743+0000 7f4a9aa00700 1 --2- 172.21.15.32:0/311857411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a9c110160 0x7f4a9c1a44f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c003f00 tx=0x7f4a8c011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:39.657 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.743+0000 7f4a93400700 1 -- 172.21.15.32:0/311857411 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a8c003aa0 con 0x7f4a9c110160 2024-04-30T08:23:39.657 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.743+0000 7f4a93400700 1 -- 172.21.15.32:0/311857411 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a8c01b8f0 con 0x7f4a9c110160 2024-04-30T08:23:39.657 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.743+0000 7f4a93400700 1 -- 172.21.15.32:0/311857411 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a8c023890 con 0x7f4a9c110160 2024-04-30T08:23:39.657 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.743+0000 7f4aa1800700 1 -- 172.21.15.32:0/311857411 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a9c1a4cc0 con 0x7f4a9c110160 2024-04-30T08:23:39.657 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.744+0000 7f4aa1800700 1 -- 172.21.15.32:0/311857411 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a9c1a78d0 con 0x7f4a9c110160 2024-04-30T08:23:39.657 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.745+0000 7f4aa1800700 1 -- 172.21.15.32:0/311857411 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a9c064d70 con 0x7f4a9c110160 2024-04-30T08:23:39.658 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.745+0000 7f4a93400700 1 -- 172.21.15.32:0/311857411 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f4a8c01b440 con 0x7f4a9c110160 2024-04-30T08:23:39.658 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.746+0000 7f4a93400700 1 --2- 172.21.15.32:0/311857411 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4a8003d6c0 0x7f4a8003fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:39.658 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.746+0000 7f4a93400700 1 -- 172.21.15.32:0/311857411 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4a8c052e30 con 0x7f4a9c110160 2024-04-30T08:23:39.658 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.746+0000 7f4a93e00700 1 --2- 172.21.15.32:0/311857411 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4a8003d6c0 0x7f4a8003fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:39.659 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.747+0000 7f4a93e00700 1 --2- 172.21.15.32:0/311857411 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4a8003d6c0 0x7f4a8003fbd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4a84006fd0 tx=0x7f4a84006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:39.659 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.751+0000 7f4a93400700 1 -- 172.21.15.32:0/311857411 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4a8c01f030 con 0x7f4a9c110160 2024-04-30T08:23:39.659 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.982+0000 7f4aa1800700 1 -- 172.21.15.32:0/311857411 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f4a9c1a7aa0 con 0x7f4a9c110160 2024-04-30T08:23:39.659 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.984+0000 7f4a93400700 1 -- 172.21.15.32:0/311857411 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f4a8c01bba0 con 0x7f4a9c110160 2024-04-30T08:23:39.660 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.987+0000 7f4aa1800700 1 -- 172.21.15.32:0/311857411 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4a8003d6c0 msgr2=0x7f4a8003fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:39.660 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.987+0000 7f4aa1800700 1 --2- 172.21.15.32:0/311857411 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4a8003d6c0 0x7f4a8003fbd0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f4a84006fd0 tx=0x7f4a84006e40 comp rx=0 tx=0).stop 2024-04-30T08:23:39.660 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.987+0000 7f4aa1800700 1 -- 172.21.15.32:0/311857411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a9c110160 msgr2=0x7f4a9c1a44f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:39.660 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.987+0000 7f4aa1800700 1 --2- 172.21.15.32:0/311857411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a9c110160 0x7f4a9c1a44f0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c003f00 tx=0x7f4a8c011aa0 comp rx=0 tx=0).stop 2024-04-30T08:23:39.661 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.987+0000 7f4aa1800700 1 -- 172.21.15.32:0/311857411 shutdown_connections 2024-04-30T08:23:39.661 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.987+0000 7f4aa1800700 1 --2- 172.21.15.32:0/311857411 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4a8003d6c0 0x7f4a8003fbd0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:39.661 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.987+0000 7f4aa1800700 1 --2- 172.21.15.32:0/311857411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a9c110160 0x7f4a9c1a44f0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:39.661 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.987+0000 7f4aa1800700 1 -- 172.21.15.32:0/311857411 >> 172.21.15.32:0/311857411 conn(0x7f4a9c07f330 msgr2=0x7f4a9c07fc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:39.661 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.987+0000 7f4aa1800700 1 -- 172.21.15.32:0/311857411 shutdown_connections 2024-04-30T08:23:39.661 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:38.987+0000 7f4aa1800700 1 -- 172.21.15.32:0/311857411 wait complete. 2024-04-30T08:23:41.325 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.310+0000 7fe267e00700 1 Processor -- start 2024-04-30T08:23:41.325 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.310+0000 7fe267e00700 1 -- start start 2024-04-30T08:23:41.328 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.311+0000 7fe267e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe26807ed60 0x7fe26807d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:41.329 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.311+0000 7fe267e00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe26807d770 con 0x7fe26807ed60 2024-04-30T08:23:41.329 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.311+0000 7fe266a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe26807ed60 0x7fe26807d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:41.329 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.311+0000 7fe266a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe26807ed60 0x7fe26807d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55002/0 (socket says 172.21.15.32:55002) 2024-04-30T08:23:41.329 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.311+0000 7fe266a00700 1 -- 172.21.15.32:0/2038651675 learned_addr learned my addr 172.21.15.32:0/2038651675 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:23:41.329 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.312+0000 7fe266a00700 1 -- 172.21.15.32:0/2038651675 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe26807d8f0 con 0x7fe26807ed60 2024-04-30T08:23:41.329 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.312+0000 7fe266a00700 1 --2- 172.21.15.32:0/2038651675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe26807ed60 0x7fe26807d230 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe2580098a0 tx=0x7fe258009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2057035be01dc00 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:41.329 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.312+0000 7fe265600700 1 -- 172.21.15.32:0/2038651675 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe258004160 con 0x7fe26807ed60 2024-04-30T08:23:41.329 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.312+0000 7fe265600700 1 -- 172.21.15.32:0/2038651675 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe258004300 con 0x7fe26807ed60 2024-04-30T08:23:41.329 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.312+0000 7fe265600700 1 -- 172.21.15.32:0/2038651675 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe258011a20 con 0x7fe26807ed60 2024-04-30T08:23:41.329 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.313+0000 7fe267e00700 1 -- 172.21.15.32:0/2038651675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe26807ed60 msgr2=0x7fe26807d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:41.329 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.313+0000 7fe267e00700 1 --2- 172.21.15.32:0/2038651675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe26807ed60 0x7fe26807d230 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe2580098a0 tx=0x7fe258009bb0 comp rx=0 tx=0).stop 2024-04-30T08:23:41.329 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.314+0000 7fe267e00700 1 -- 172.21.15.32:0/2038651675 shutdown_connections 2024-04-30T08:23:41.330 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.314+0000 7fe267e00700 1 --2- 172.21.15.32:0/2038651675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe26807ed60 0x7fe26807d230 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:41.330 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.314+0000 7fe267e00700 1 -- 172.21.15.32:0/2038651675 >> 172.21.15.32:0/2038651675 conn(0x7fe268105090 msgr2=0x7fe268107520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:41.330 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.314+0000 7fe267e00700 1 -- 172.21.15.32:0/2038651675 shutdown_connections 2024-04-30T08:23:41.330 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.314+0000 7fe267e00700 1 -- 172.21.15.32:0/2038651675 wait complete. 2024-04-30T08:23:41.330 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.315+0000 7fe267e00700 1 Processor -- start 2024-04-30T08:23:41.330 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.316+0000 7fe267e00700 1 -- start start 2024-04-30T08:23:41.330 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.316+0000 7fe267e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe26807ed60 0x7fe26819ff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:41.330 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.316+0000 7fe267e00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe26810fff0 con 0x7fe26807ed60 2024-04-30T08:23:41.330 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.316+0000 7fe266a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe26807ed60 0x7fe26819ff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:41.330 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.316+0000 7fe266a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe26807ed60 0x7fe26819ff50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55014/0 (socket says 172.21.15.32:55014) 2024-04-30T08:23:41.330 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.317+0000 7fe266a00700 1 -- 172.21.15.32:0/950437424 learned_addr learned my addr 172.21.15.32:0/950437424 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:23:41.330 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.317+0000 7fe266a00700 1 -- 172.21.15.32:0/950437424 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2681a0490 con 0x7fe26807ed60 2024-04-30T08:23:41.330 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.317+0000 7fe266a00700 1 --2- 172.21.15.32:0/950437424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe26807ed60 0x7fe26819ff50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe258000c00 tx=0x7fe258011aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:41.331 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.318+0000 7fe25f400700 1 -- 172.21.15.32:0/950437424 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe258003aa0 con 0x7fe26807ed60 2024-04-30T08:23:41.331 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.318+0000 7fe25f400700 1 -- 172.21.15.32:0/950437424 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe25801b8f0 con 0x7fe26807ed60 2024-04-30T08:23:41.334 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.318+0000 7fe25f400700 1 -- 172.21.15.32:0/950437424 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe258023890 con 0x7fe26807ed60 2024-04-30T08:23:41.334 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.318+0000 7fe267e00700 1 -- 172.21.15.32:0/950437424 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2681a0720 con 0x7fe26807ed60 2024-04-30T08:23:41.334 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.318+0000 7fe267e00700 1 -- 172.21.15.32:0/950437424 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2681a3330 con 0x7fe26807ed60 2024-04-30T08:23:41.334 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.319+0000 7fe267e00700 1 -- 172.21.15.32:0/950437424 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe268064d70 con 0x7fe26807ed60 2024-04-30T08:23:41.334 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.319+0000 7fe25f400700 1 -- 172.21.15.32:0/950437424 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7fe25801b440 con 0x7fe26807ed60 2024-04-30T08:23:41.334 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.320+0000 7fe25f400700 1 --2- 172.21.15.32:0/950437424 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe24c03d6c0 0x7fe24c03fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:41.335 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.320+0000 7fe25f400700 1 -- 172.21.15.32:0/950437424 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fe258052eb0 con 0x7fe26807ed60 2024-04-30T08:23:41.335 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.320+0000 7fe266000700 1 --2- 172.21.15.32:0/950437424 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe24c03d6c0 0x7fe24c03fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:41.335 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.321+0000 7fe266000700 1 --2- 172.21.15.32:0/950437424 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe24c03d6c0 0x7fe24c03fbd0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe254006fd0 tx=0x7fe254006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:41.335 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.325+0000 7fe25f400700 1 -- 172.21.15.32:0/950437424 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe25801f030 con 0x7fe26807ed60 2024-04-30T08:23:41.335 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.674+0000 7fe267e00700 1 -- 172.21.15.32:0/950437424 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7fe2681a3500 con 0x7fe26807ed60 2024-04-30T08:23:41.335 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.677+0000 7fe25f400700 1 -- 172.21.15.32:0/950437424 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7fe25800f2e0 con 0x7fe26807ed60 2024-04-30T08:23:41.335 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.680+0000 7fe267e00700 1 -- 172.21.15.32:0/950437424 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe24c03d6c0 msgr2=0x7fe24c03fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:41.335 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.680+0000 7fe267e00700 1 --2- 172.21.15.32:0/950437424 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe24c03d6c0 0x7fe24c03fbd0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe254006fd0 tx=0x7fe254006e40 comp rx=0 tx=0).stop 2024-04-30T08:23:41.335 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.680+0000 7fe267e00700 1 -- 172.21.15.32:0/950437424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe26807ed60 msgr2=0x7fe26819ff50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:41.335 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.680+0000 7fe267e00700 1 --2- 172.21.15.32:0/950437424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe26807ed60 0x7fe26819ff50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe258000c00 tx=0x7fe258011aa0 comp rx=0 tx=0).stop 2024-04-30T08:23:41.335 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.680+0000 7fe267e00700 1 -- 172.21.15.32:0/950437424 shutdown_connections 2024-04-30T08:23:41.335 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.680+0000 7fe267e00700 1 --2- 172.21.15.32:0/950437424 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe24c03d6c0 0x7fe24c03fbd0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:41.335 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.680+0000 7fe267e00700 1 --2- 172.21.15.32:0/950437424 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe26807ed60 0x7fe26819ff50 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:41.336 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.680+0000 7fe267e00700 1 -- 172.21.15.32:0/950437424 >> 172.21.15.32:0/950437424 conn(0x7fe268105090 msgr2=0x7fe268105930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:41.336 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.681+0000 7fe267e00700 1 -- 172.21.15.32:0/950437424 shutdown_connections 2024-04-30T08:23:41.336 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr 2024-04-30T08:23:40.681+0000 7fe267e00700 1 -- 172.21.15.32:0/950437424 wait complete. 2024-04-30T08:23:41.336 INFO:teuthology.orchestra.run.smithi032.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-30T08:23:41.336 INFO:teuthology.orchestra.run.smithi032.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-30T08:23:41.336 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:23:41.336 INFO:teuthology.orchestra.run.smithi032.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-30T08:23:41.336 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:23:41.336 INFO:teuthology.orchestra.run.smithi032.stdout:Or, if you are only running a single cluster on this host: 2024-04-30T08:23:41.336 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:23:41.336 INFO:teuthology.orchestra.run.smithi032.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-30T08:23:41.336 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:23:41.336 INFO:teuthology.orchestra.run.smithi032.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-30T08:23:41.336 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:23:41.337 INFO:teuthology.orchestra.run.smithi032.stdout: ceph telemetry on 2024-04-30T08:23:41.337 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:23:41.337 INFO:teuthology.orchestra.run.smithi032.stdout:For more information see: 2024-04-30T08:23:41.337 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:23:41.337 INFO:teuthology.orchestra.run.smithi032.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-30T08:23:41.337 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:23:41.337 INFO:teuthology.orchestra.run.smithi032.stdout:Bootstrap complete. 2024-04-30T08:23:41.436 INFO:tasks.cephadm:Fetching config... 2024-04-30T08:23:41.436 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-30T08:23:41.436 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-30T08:23:41.495 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-30T08:23:41.495 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-30T08:23:41.496 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-30T08:23:41.551 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-30T08:23:41.551 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-30T08:23:41.551 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-30T08:23:41.678 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-30T08:23:41.678 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-30T08:23:41.678 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-30T08:23:41.788 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-30T08:23:41.789 DEBUG:teuthology.orchestra.run.smithi032:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAebVzc5PNbSUQhRM83qqlc+1tN2IJZ3mtGdHIQzTNbXelQTm4K61cFqb2kCuL2ZS8TTReRVjyNDsYaJ8HLj4JQ2SclnBwIKshH2vXnqUKzACxDLFYe5ajSupZZCPJqF4+NUMF5ZATa5UQcPRJ+fxt//dHsJqYS2oN1aqT35RL+xL3KVQI4kjSKwndp4twbZNMXc8ABdplGqgkjKgfpWhz0mfEIwmUQOVxazLnH0UPSdQjKgAI/YbLRD28Lv0wnkPWwGRjcmrIVzQh43oqWSqSSuB4FI+HTfU/auTHSXdMDLmuf3nIB+FiC/f9lGW81Ko5nj3qM6uMy1eNHIj1XiuQj2dUHnDqYQJEt52JHMC7IFSyYDoLF2oUNNj6FVJTTV2JgTXzGk1XL6alaTG5qDm5DT4hDM/aSeg8fjOhxsbMe9pewleloEDy4ocg/N8lc5/eu9M3ad2cbE60FDjW0ibIAoCW7KIrq75d2FtK2ca3jUvlIYgi0IZ8Z3CFjntsltc= ceph-98c0a670-06ca-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-30T08:23:41.914 INFO:teuthology.orchestra.run.smithi032.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAebVzc5PNbSUQhRM83qqlc+1tN2IJZ3mtGdHIQzTNbXelQTm4K61cFqb2kCuL2ZS8TTReRVjyNDsYaJ8HLj4JQ2SclnBwIKshH2vXnqUKzACxDLFYe5ajSupZZCPJqF4+NUMF5ZATa5UQcPRJ+fxt//dHsJqYS2oN1aqT35RL+xL3KVQI4kjSKwndp4twbZNMXc8ABdplGqgkjKgfpWhz0mfEIwmUQOVxazLnH0UPSdQjKgAI/YbLRD28Lv0wnkPWwGRjcmrIVzQh43oqWSqSSuB4FI+HTfU/auTHSXdMDLmuf3nIB+FiC/f9lGW81Ko5nj3qM6uMy1eNHIj1XiuQj2dUHnDqYQJEt52JHMC7IFSyYDoLF2oUNNj6FVJTTV2JgTXzGk1XL6alaTG5qDm5DT4hDM/aSeg8fjOhxsbMe9pewleloEDy4ocg/N8lc5/eu9M3ad2cbE60FDjW0ibIAoCW7KIrq75d2FtK2ca3jUvlIYgi0IZ8Z3CFjntsltc= ceph-98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:23:41.915 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:41 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/950437424' entity='client.admin' 2024-04-30T08:23:41.943 DEBUG:teuthology.orchestra.run.smithi071:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAebVzc5PNbSUQhRM83qqlc+1tN2IJZ3mtGdHIQzTNbXelQTm4K61cFqb2kCuL2ZS8TTReRVjyNDsYaJ8HLj4JQ2SclnBwIKshH2vXnqUKzACxDLFYe5ajSupZZCPJqF4+NUMF5ZATa5UQcPRJ+fxt//dHsJqYS2oN1aqT35RL+xL3KVQI4kjSKwndp4twbZNMXc8ABdplGqgkjKgfpWhz0mfEIwmUQOVxazLnH0UPSdQjKgAI/YbLRD28Lv0wnkPWwGRjcmrIVzQh43oqWSqSSuB4FI+HTfU/auTHSXdMDLmuf3nIB+FiC/f9lGW81Ko5nj3qM6uMy1eNHIj1XiuQj2dUHnDqYQJEt52JHMC7IFSyYDoLF2oUNNj6FVJTTV2JgTXzGk1XL6alaTG5qDm5DT4hDM/aSeg8fjOhxsbMe9pewleloEDy4ocg/N8lc5/eu9M3ad2cbE60FDjW0ibIAoCW7KIrq75d2FtK2ca3jUvlIYgi0IZ8Z3CFjntsltc= ceph-98c0a670-06ca-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-30T08:23:41.988 INFO:teuthology.orchestra.run.smithi071.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAebVzc5PNbSUQhRM83qqlc+1tN2IJZ3mtGdHIQzTNbXelQTm4K61cFqb2kCuL2ZS8TTReRVjyNDsYaJ8HLj4JQ2SclnBwIKshH2vXnqUKzACxDLFYe5ajSupZZCPJqF4+NUMF5ZATa5UQcPRJ+fxt//dHsJqYS2oN1aqT35RL+xL3KVQI4kjSKwndp4twbZNMXc8ABdplGqgkjKgfpWhz0mfEIwmUQOVxazLnH0UPSdQjKgAI/YbLRD28Lv0wnkPWwGRjcmrIVzQh43oqWSqSSuB4FI+HTfU/auTHSXdMDLmuf3nIB+FiC/f9lGW81Ko5nj3qM6uMy1eNHIj1XiuQj2dUHnDqYQJEt52JHMC7IFSyYDoLF2oUNNj6FVJTTV2JgTXzGk1XL6alaTG5qDm5DT4hDM/aSeg8fjOhxsbMe9pewleloEDy4ocg/N8lc5/eu9M3ad2cbE60FDjW0ibIAoCW7KIrq75d2FtK2ca3jUvlIYgi0IZ8Z3CFjntsltc= ceph-98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:23:42.001 DEBUG:teuthology.orchestra.run.smithi189:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAebVzc5PNbSUQhRM83qqlc+1tN2IJZ3mtGdHIQzTNbXelQTm4K61cFqb2kCuL2ZS8TTReRVjyNDsYaJ8HLj4JQ2SclnBwIKshH2vXnqUKzACxDLFYe5ajSupZZCPJqF4+NUMF5ZATa5UQcPRJ+fxt//dHsJqYS2oN1aqT35RL+xL3KVQI4kjSKwndp4twbZNMXc8ABdplGqgkjKgfpWhz0mfEIwmUQOVxazLnH0UPSdQjKgAI/YbLRD28Lv0wnkPWwGRjcmrIVzQh43oqWSqSSuB4FI+HTfU/auTHSXdMDLmuf3nIB+FiC/f9lGW81Ko5nj3qM6uMy1eNHIj1XiuQj2dUHnDqYQJEt52JHMC7IFSyYDoLF2oUNNj6FVJTTV2JgTXzGk1XL6alaTG5qDm5DT4hDM/aSeg8fjOhxsbMe9pewleloEDy4ocg/N8lc5/eu9M3ad2cbE60FDjW0ibIAoCW7KIrq75d2FtK2ca3jUvlIYgi0IZ8Z3CFjntsltc= ceph-98c0a670-06ca-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-30T08:23:42.048 INFO:teuthology.orchestra.run.smithi189.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAebVzc5PNbSUQhRM83qqlc+1tN2IJZ3mtGdHIQzTNbXelQTm4K61cFqb2kCuL2ZS8TTReRVjyNDsYaJ8HLj4JQ2SclnBwIKshH2vXnqUKzACxDLFYe5ajSupZZCPJqF4+NUMF5ZATa5UQcPRJ+fxt//dHsJqYS2oN1aqT35RL+xL3KVQI4kjSKwndp4twbZNMXc8ABdplGqgkjKgfpWhz0mfEIwmUQOVxazLnH0UPSdQjKgAI/YbLRD28Lv0wnkPWwGRjcmrIVzQh43oqWSqSSuB4FI+HTfU/auTHSXdMDLmuf3nIB+FiC/f9lGW81Ko5nj3qM6uMy1eNHIj1XiuQj2dUHnDqYQJEt52JHMC7IFSyYDoLF2oUNNj6FVJTTV2JgTXzGk1XL6alaTG5qDm5DT4hDM/aSeg8fjOhxsbMe9pewleloEDy4ocg/N8lc5/eu9M3ad2cbE60FDjW0ibIAoCW7KIrq75d2FtK2ca3jUvlIYgi0IZ8Z3CFjntsltc= ceph-98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:23:42.059 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-30T08:23:42.528 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:23:45.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.365+0000 7f4a62200700 1 -- 172.21.15.32:0/3494943180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a5c0776d0 msgr2=0x7f4a5c075b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:45.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.365+0000 7f4a62200700 1 --2- 172.21.15.32:0/3494943180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a5c0776d0 0x7f4a5c075b00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c009910 tx=0x7f4a4c009c20 comp rx=0 tx=0).stop 2024-04-30T08:23:45.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.366+0000 7f4a62200700 1 -- 172.21.15.32:0/3494943180 shutdown_connections 2024-04-30T08:23:45.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.366+0000 7f4a62200700 1 --2- 172.21.15.32:0/3494943180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a5c0776d0 0x7f4a5c075b00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:45.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.366+0000 7f4a62200700 1 -- 172.21.15.32:0/3494943180 >> 172.21.15.32:0/3494943180 conn(0x7f4a5c0ff1b0 msgr2=0x7f4a5c101660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:45.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.366+0000 7f4a62200700 1 -- 172.21.15.32:0/3494943180 shutdown_connections 2024-04-30T08:23:45.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.366+0000 7f4a62200700 1 -- 172.21.15.32:0/3494943180 wait complete. 2024-04-30T08:23:45.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.367+0000 7f4a62200700 1 Processor -- start 2024-04-30T08:23:45.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.368+0000 7f4a62200700 1 -- start start 2024-04-30T08:23:45.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.368+0000 7f4a62200700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a5c0776d0 0x7f4a5c19b870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:45.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.368+0000 7f4a62200700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4a5c10bd20 con 0x7f4a5c0776d0 2024-04-30T08:23:45.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.369+0000 7f4a5b400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a5c0776d0 0x7f4a5c19b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:45.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.369+0000 7f4a5b400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a5c0776d0 0x7f4a5c19b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55042/0 (socket says 172.21.15.32:55042) 2024-04-30T08:23:45.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.369+0000 7f4a5b400700 1 -- 172.21.15.32:0/1047003197 learned_addr learned my addr 172.21.15.32:0/1047003197 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:23:45.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.369+0000 7f4a5b400700 1 -- 172.21.15.32:0/1047003197 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a5c19bdb0 con 0x7f4a5c0776d0 2024-04-30T08:23:45.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.369+0000 7f4a5b400700 1 --2- 172.21.15.32:0/1047003197 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a5c0776d0 0x7f4a5c19b870 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c006010 tx=0x7f4a4c005ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:45.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.370+0000 7f4a53e00700 1 -- 172.21.15.32:0/1047003197 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a4c026070 con 0x7f4a5c0776d0 2024-04-30T08:23:45.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.370+0000 7f4a53e00700 1 -- 172.21.15.32:0/1047003197 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4a4c0044a0 con 0x7f4a5c0776d0 2024-04-30T08:23:45.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.370+0000 7f4a62200700 1 -- 172.21.15.32:0/1047003197 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a5c19c040 con 0x7f4a5c0776d0 2024-04-30T08:23:45.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.370+0000 7f4a53e00700 1 -- 172.21.15.32:0/1047003197 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a4c0039d0 con 0x7f4a5c0776d0 2024-04-30T08:23:45.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.370+0000 7f4a62200700 1 -- 172.21.15.32:0/1047003197 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a5c19ec50 con 0x7f4a5c0776d0 2024-04-30T08:23:45.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.371+0000 7f4a53e00700 1 -- 172.21.15.32:0/1047003197 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f4a4c029070 con 0x7f4a5c0776d0 2024-04-30T08:23:45.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.372+0000 7f4a62200700 1 -- 172.21.15.32:0/1047003197 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a5c10c4a0 con 0x7f4a5c0776d0 2024-04-30T08:23:45.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.372+0000 7f4a53e00700 1 --2- 172.21.15.32:0/1047003197 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4a4003d6c0 0x7f4a4003fbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:45.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.372+0000 7f4a53e00700 1 -- 172.21.15.32:0/1047003197 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4a4c01e070 con 0x7f4a5c0776d0 2024-04-30T08:23:45.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.372+0000 7f4a5aa00700 1 --2- 172.21.15.32:0/1047003197 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4a4003d6c0 0x7f4a4003fbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:45.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.373+0000 7f4a5aa00700 1 --2- 172.21.15.32:0/1047003197 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4a4003d6c0 0x7f4a4003fbd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4a48006fd0 tx=0x7f4a48006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:45.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.378+0000 7f4a53e00700 1 -- 172.21.15.32:0/1047003197 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4a4c013070 con 0x7f4a5c0776d0 2024-04-30T08:23:45.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.629+0000 7f4a62200700 1 -- 172.21.15.32:0/1047003197 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7f4a5c02e6a0 con 0x7f4a5c0776d0 2024-04-30T08:23:45.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.633+0000 7f4a53e00700 1 -- 172.21.15.32:0/1047003197 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7f4a4c0282c0 con 0x7f4a5c0776d0 2024-04-30T08:23:45.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.636+0000 7f4a62200700 1 -- 172.21.15.32:0/1047003197 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4a4003d6c0 msgr2=0x7f4a4003fbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:45.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.636+0000 7f4a62200700 1 --2- 172.21.15.32:0/1047003197 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4a4003d6c0 0x7f4a4003fbd0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4a48006fd0 tx=0x7f4a48006e40 comp rx=0 tx=0).stop 2024-04-30T08:23:45.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.636+0000 7f4a62200700 1 -- 172.21.15.32:0/1047003197 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a5c0776d0 msgr2=0x7f4a5c19b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:45.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.636+0000 7f4a62200700 1 --2- 172.21.15.32:0/1047003197 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a5c0776d0 0x7f4a5c19b870 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c006010 tx=0x7f4a4c005ad0 comp rx=0 tx=0).stop 2024-04-30T08:23:45.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.636+0000 7f4a62200700 1 -- 172.21.15.32:0/1047003197 shutdown_connections 2024-04-30T08:23:45.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.636+0000 7f4a62200700 1 --2- 172.21.15.32:0/1047003197 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4a4003d6c0 0x7f4a4003fbd0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:45.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.636+0000 7f4a62200700 1 --2- 172.21.15.32:0/1047003197 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4a5c0776d0 0x7f4a5c19b870 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:45.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.636+0000 7f4a62200700 1 -- 172.21.15.32:0/1047003197 >> 172.21.15.32:0/1047003197 conn(0x7f4a5c0ff1b0 msgr2=0x7f4a5c0ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:45.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.637+0000 7f4a62200700 1 -- 172.21.15.32:0/1047003197 shutdown_connections 2024-04-30T08:23:45.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:45.637+0000 7f4a62200700 1 -- 172.21.15.32:0/1047003197 wait complete. 2024-04-30T08:23:45.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:45 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:23:45.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:45 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:23:45.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:45 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi032", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:23:45.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:45 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:23:45.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:45 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:23:45.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:45 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:23:46.454 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-30T08:23:46.454 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-30T08:23:46.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:46 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1047003197' entity='client.admin' 2024-04-30T08:23:46.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:46 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:23:46.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:46 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:23:47.530 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:23:49.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.601+0000 7f08a7a00700 1 -- 172.21.15.32:0/1775115080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08a00776d0 msgr2=0x7f08a0075b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:49.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.601+0000 7f08a7a00700 1 --2- 172.21.15.32:0/1775115080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08a00776d0 0x7f08a0075b00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0894009910 tx=0x7f0894009c20 comp rx=0 tx=0).stop 2024-04-30T08:23:49.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.601+0000 7f08a7a00700 1 -- 172.21.15.32:0/1775115080 shutdown_connections 2024-04-30T08:23:49.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.601+0000 7f08a7a00700 1 --2- 172.21.15.32:0/1775115080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08a00776d0 0x7f08a0075b00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:49.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.601+0000 7f08a7a00700 1 -- 172.21.15.32:0/1775115080 >> 172.21.15.32:0/1775115080 conn(0x7f08a00ff1b0 msgr2=0x7f08a0101660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:49.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.601+0000 7f08a7a00700 1 -- 172.21.15.32:0/1775115080 shutdown_connections 2024-04-30T08:23:49.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.601+0000 7f08a7a00700 1 -- 172.21.15.32:0/1775115080 wait complete. 2024-04-30T08:23:49.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.602+0000 7f08a7a00700 1 Processor -- start 2024-04-30T08:23:49.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.602+0000 7f08a7a00700 1 -- start start 2024-04-30T08:23:49.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.603+0000 7f08a7a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08a00776d0 0x7f08a019fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:49.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.603+0000 7f08a7a00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f08a01101c0 con 0x7f08a00776d0 2024-04-30T08:23:49.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.603+0000 7f08a5000700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08a00776d0 0x7f08a019fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:49.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.603+0000 7f08a5000700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08a00776d0 0x7f08a019fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34296/0 (socket says 172.21.15.32:34296) 2024-04-30T08:23:49.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.603+0000 7f08a5000700 1 -- 172.21.15.32:0/956599901 learned_addr learned my addr 172.21.15.32:0/956599901 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:23:49.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.604+0000 7f08a5000700 1 -- 172.21.15.32:0/956599901 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08a01a01e0 con 0x7f08a00776d0 2024-04-30T08:23:49.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.604+0000 7f08a5000700 1 --2- 172.21.15.32:0/956599901 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08a00776d0 0x7f08a019fca0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0894006010 tx=0x7f0894005ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:49.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.604+0000 7f089d600700 1 -- 172.21.15.32:0/956599901 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0894026070 con 0x7f08a00776d0 2024-04-30T08:23:49.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.604+0000 7f089d600700 1 -- 172.21.15.32:0/956599901 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f08940044a0 con 0x7f08a00776d0 2024-04-30T08:23:49.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.604+0000 7f08a7a00700 1 -- 172.21.15.32:0/956599901 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08a01a0470 con 0x7f08a00776d0 2024-04-30T08:23:49.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.604+0000 7f089d600700 1 -- 172.21.15.32:0/956599901 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f08940039d0 con 0x7f08a00776d0 2024-04-30T08:23:49.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.604+0000 7f08a7a00700 1 -- 172.21.15.32:0/956599901 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08a01a3080 con 0x7f08a00776d0 2024-04-30T08:23:49.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.606+0000 7f08a7a00700 1 -- 172.21.15.32:0/956599901 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08a01108a0 con 0x7f08a00776d0 2024-04-30T08:23:49.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.606+0000 7f089d600700 1 -- 172.21.15.32:0/956599901 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f0894029070 con 0x7f08a00776d0 2024-04-30T08:23:49.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.607+0000 7f089d600700 1 --2- 172.21.15.32:0/956599901 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f088803d670 0x7f088803fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:49.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.607+0000 7f089d600700 1 -- 172.21.15.32:0/956599901 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f089401e070 con 0x7f08a00776d0 2024-04-30T08:23:49.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.607+0000 7f089fe00700 1 --2- 172.21.15.32:0/956599901 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f088803d670 0x7f088803fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:49.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.608+0000 7f089fe00700 1 --2- 172.21.15.32:0/956599901 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f088803d670 0x7f088803fb80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f088c006fd0 tx=0x7f088c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:49.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.613+0000 7f089d600700 1 -- 172.21.15.32:0/956599901 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0894013070 con 0x7f08a00776d0 2024-04-30T08:23:49.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.849+0000 7f08a7a00700 1 -- 172.21.15.32:0/956599901 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f08a002e6a0 con 0x7f088803d670 2024-04-30T08:23:49.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.859+0000 7f089d600700 1 -- 172.21.15.32:0/956599901 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f08a002e6a0 con 0x7f088803d670 2024-04-30T08:23:49.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.862+0000 7f08a7a00700 1 -- 172.21.15.32:0/956599901 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f088803d670 msgr2=0x7f088803fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:49.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.862+0000 7f08a7a00700 1 --2- 172.21.15.32:0/956599901 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f088803d670 0x7f088803fb80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f088c006fd0 tx=0x7f088c006e40 comp rx=0 tx=0).stop 2024-04-30T08:23:49.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.862+0000 7f08a7a00700 1 -- 172.21.15.32:0/956599901 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08a00776d0 msgr2=0x7f08a019fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:49.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.862+0000 7f08a7a00700 1 --2- 172.21.15.32:0/956599901 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08a00776d0 0x7f08a019fca0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0894006010 tx=0x7f0894005ad0 comp rx=0 tx=0).stop 2024-04-30T08:23:49.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.862+0000 7f08a7a00700 1 -- 172.21.15.32:0/956599901 shutdown_connections 2024-04-30T08:23:49.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.862+0000 7f08a7a00700 1 --2- 172.21.15.32:0/956599901 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f088803d670 0x7f088803fb80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:49.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.862+0000 7f08a7a00700 1 --2- 172.21.15.32:0/956599901 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f08a00776d0 0x7f08a019fca0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:49.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.862+0000 7f08a7a00700 1 -- 172.21.15.32:0/956599901 >> 172.21.15.32:0/956599901 conn(0x7f08a00ff1b0 msgr2=0x7f08a00ffeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:49.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.863+0000 7f08a7a00700 1 -- 172.21.15.32:0/956599901 shutdown_connections 2024-04-30T08:23:49.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:49.863+0000 7f08a7a00700 1 -- 172.21.15.32:0/956599901 wait complete. 2024-04-30T08:23:50.690 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi071 2024-04-30T08:23:50.690 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-30T08:23:50.690 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/etc/ceph/ceph.conf 2024-04-30T08:23:50.709 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-30T08:23:50.709 DEBUG:teuthology.orchestra.run.smithi071:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-30T08:23:50.766 INFO:tasks.cephadm:Adding host smithi071 to orchestrator... 2024-04-30T08:23:50.766 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph orch host add smithi071 2024-04-30T08:23:50.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:50 smithi032 ceph-mon[28331]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:23:50.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:50 smithi032 ceph-mon[28331]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:23:50.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:50 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:23:50.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:50 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:23:50.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:50 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:23:50.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:50 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:23:50.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:50 smithi032 ceph-mon[28331]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-30T08:23:51.188 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:23:51.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:51 smithi032 ceph-mon[28331]: Updating smithi032:/var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-30T08:23:51.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:51 smithi032 ceph-mon[28331]: Updating smithi032:/etc/ceph/ceph.client.admin.keyring 2024-04-30T08:23:53.118 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:52 smithi032 ceph-mon[28331]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:23:53.118 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:52 smithi032 ceph-mon[28331]: Updating smithi032:/var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-30T08:23:53.118 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:23:53.118 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:23:53.119 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:23:53.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:53.370+0000 7fa725400700 1 -- 172.21.15.32:0/2523107349 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa72010a350 msgr2=0x7fa72010a760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:53.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:53.370+0000 7fa725400700 1 --2- 172.21.15.32:0/2523107349 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa72010a350 0x7fa72010a760 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa710009910 tx=0x7fa710009c20 comp rx=0 tx=0).stop 2024-04-30T08:23:53.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:53.370+0000 7fa725400700 1 -- 172.21.15.32:0/2523107349 shutdown_connections 2024-04-30T08:23:53.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:53.370+0000 7fa725400700 1 --2- 172.21.15.32:0/2523107349 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa72010a350 0x7fa72010a760 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:53.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:53.370+0000 7fa725400700 1 -- 172.21.15.32:0/2523107349 >> 172.21.15.32:0/2523107349 conn(0x7fa720079830 msgr2=0x7fa720108530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:53.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:53.370+0000 7fa725400700 1 -- 172.21.15.32:0/2523107349 shutdown_connections 2024-04-30T08:23:53.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:53.370+0000 7fa725400700 1 -- 172.21.15.32:0/2523107349 wait complete. 2024-04-30T08:23:53.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:53.371+0000 7fa725400700 1 Processor -- start 2024-04-30T08:23:53.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:53.371+0000 7fa725400700 1 -- start start 2024-04-30T08:23:53.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:53.372+0000 7fa725400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa72010a350 0x7fa72019fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:53.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:53.372+0000 7fa725400700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa720114640 con 0x7fa72010a350 2024-04-30T08:23:53.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:53.372+0000 7fa71e800700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa72010a350 0x7fa72019fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:53.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:53.372+0000 7fa71e800700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa72010a350 0x7fa72019fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34306/0 (socket says 172.21.15.32:34306) 2024-04-30T08:23:53.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:53.372+0000 7fa71e800700 1 -- 172.21.15.32:0/1707950952 learned_addr learned my addr 172.21.15.32:0/1707950952 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:23:53.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:53.373+0000 7fa71e800700 1 -- 172.21.15.32:0/1707950952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7201a0420 con 0x7fa72010a350 2024-04-30T08:23:53.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:53.373+0000 7fa71e800700 1 --2- 172.21.15.32:0/1707950952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa72010a350 0x7fa72019fee0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa710006010 tx=0x7fa710005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:53.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:53.373+0000 7fa717400700 1 -- 172.21.15.32:0/1707950952 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa710026070 con 0x7fa72010a350 2024-04-30T08:23:53.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:53.373+0000 7fa717400700 1 -- 172.21.15.32:0/1707950952 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa71000bba0 con 0x7fa72010a350 2024-04-30T08:23:53.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:53.373+0000 7fa725400700 1 -- 172.21.15.32:0/1707950952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7201a0710 con 0x7fa72010a350 2024-04-30T08:23:53.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:53.373+0000 7fa717400700 1 -- 172.21.15.32:0/1707950952 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa710003910 con 0x7fa72010a350 2024-04-30T08:23:53.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:53.374+0000 7fa725400700 1 -- 172.21.15.32:0/1707950952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7201a4710 con 0x7fa72010a350 2024-04-30T08:23:53.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:53.375+0000 7fa725400700 1 -- 172.21.15.32:0/1707950952 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa72010b800 con 0x7fa72010a350 2024-04-30T08:23:53.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:53.375+0000 7fa717400700 1 -- 172.21.15.32:0/1707950952 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7fa710029070 con 0x7fa72010a350 2024-04-30T08:23:53.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:53.375+0000 7fa717400700 1 --2- 172.21.15.32:0/1707950952 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa70403d610 0x7fa70403fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:23:53.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:53.376+0000 7fa717400700 1 -- 172.21.15.32:0/1707950952 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fa71001e070 con 0x7fa72010a350 2024-04-30T08:23:53.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:53.376+0000 7fa717e00700 1 --2- 172.21.15.32:0/1707950952 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa70403d610 0x7fa70403fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:23:53.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:53.377+0000 7fa717e00700 1 --2- 172.21.15.32:0/1707950952 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa70403d610 0x7fa70403fb20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa708006fd0 tx=0x7fa708006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:23:53.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:53.381+0000 7fa717400700 1 -- 172.21.15.32:0/1707950952 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa710013070 con 0x7fa72010a350 2024-04-30T08:23:53.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:53.617+0000 7fa725400700 1 -- 172.21.15.32:0/1707950952 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi071", "target": ["mon-mgr", ""]}) -- 0x7fa7201a1240 con 0x7fa70403d610 2024-04-30T08:23:55.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:54 smithi032 ceph-mon[28331]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:23:55.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:54 smithi032 ceph-mon[28331]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi071", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:23:56.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:55 smithi032 ceph-mon[28331]: Deploying cephadm binary to smithi071 2024-04-30T08:23:57.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:56 smithi032 ceph-mon[28331]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:23:58.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:58.724+0000 7fa717400700 1 -- 172.21.15.32:0/1707950952 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fa7201a1240 con 0x7fa70403d610 2024-04-30T08:23:58.727 INFO:teuthology.orchestra.run.smithi032.stdout:Added host 'smithi071' with addr '172.21.15.71' 2024-04-30T08:23:58.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:58.728+0000 7fa725400700 1 -- 172.21.15.32:0/1707950952 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa70403d610 msgr2=0x7fa70403fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:58.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:58.728+0000 7fa725400700 1 --2- 172.21.15.32:0/1707950952 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa70403d610 0x7fa70403fb20 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa708006fd0 tx=0x7fa708006e40 comp rx=0 tx=0).stop 2024-04-30T08:23:58.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:58.728+0000 7fa725400700 1 -- 172.21.15.32:0/1707950952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa72010a350 msgr2=0x7fa72019fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:23:58.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:58.728+0000 7fa725400700 1 --2- 172.21.15.32:0/1707950952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa72010a350 0x7fa72019fee0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fa710006010 tx=0x7fa710005780 comp rx=0 tx=0).stop 2024-04-30T08:23:58.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:58.728+0000 7fa725400700 1 -- 172.21.15.32:0/1707950952 shutdown_connections 2024-04-30T08:23:58.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:58.728+0000 7fa725400700 1 --2- 172.21.15.32:0/1707950952 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa70403d610 0x7fa70403fb20 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:58.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:58.728+0000 7fa725400700 1 --2- 172.21.15.32:0/1707950952 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa72010a350 0x7fa72019fee0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:23:58.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:58.728+0000 7fa725400700 1 -- 172.21.15.32:0/1707950952 >> 172.21.15.32:0/1707950952 conn(0x7fa720079830 msgr2=0x7fa72010bcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:23:58.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:58.728+0000 7fa725400700 1 -- 172.21.15.32:0/1707950952 shutdown_connections 2024-04-30T08:23:58.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:23:58.728+0000 7fa725400700 1 -- 172.21.15.32:0/1707950952 wait complete. 2024-04-30T08:23:59.065 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:58 smithi032 ceph-mon[28331]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:23:59.065 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:58 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:23:59.065 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:58 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:23:59.408 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-30T08:23:59.867 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:24:00.178 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:59 smithi032 ceph-mon[28331]: Added host smithi071 2024-04-30T08:24:00.178 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:59 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:00.178 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:23:59 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:01.189 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:00 smithi032 ceph-mon[28331]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:02.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.079+0000 7f3486e00700 1 -- 172.21.15.32:0/3794561195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3480075c20 msgr2=0x7f3480076030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:02.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.079+0000 7f3486e00700 1 --2- 172.21.15.32:0/3794561195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3480075c20 0x7f3480076030 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3474009910 tx=0x7f3474009c20 comp rx=0 tx=0).stop 2024-04-30T08:24:02.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.079+0000 7f3486e00700 1 -- 172.21.15.32:0/3794561195 shutdown_connections 2024-04-30T08:24:02.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.079+0000 7f3486e00700 1 --2- 172.21.15.32:0/3794561195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3480075c20 0x7f3480076030 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:02.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.079+0000 7f3486e00700 1 -- 172.21.15.32:0/3794561195 >> 172.21.15.32:0/3794561195 conn(0x7f34800ff400 msgr2=0x7f3480101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:24:02.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.079+0000 7f3486e00700 1 -- 172.21.15.32:0/3794561195 shutdown_connections 2024-04-30T08:24:02.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.079+0000 7f3486e00700 1 -- 172.21.15.32:0/3794561195 wait complete. 2024-04-30T08:24:02.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.081+0000 7f3486e00700 1 Processor -- start 2024-04-30T08:24:02.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.081+0000 7f3486e00700 1 -- start start 2024-04-30T08:24:02.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.081+0000 7f3486e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3480075c20 0x7f34801a1560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:24:02.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.081+0000 7f3486e00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f34801100c0 con 0x7f3480075c20 2024-04-30T08:24:02.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.082+0000 7f3485a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3480075c20 0x7f34801a1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:24:02.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.082+0000 7f3485a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3480075c20 0x7f34801a1560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:54992/0 (socket says 172.21.15.32:54992) 2024-04-30T08:24:02.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.082+0000 7f3485a00700 1 -- 172.21.15.32:0/3824007141 learned_addr learned my addr 172.21.15.32:0/3824007141 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:24:02.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.082+0000 7f3485a00700 1 -- 172.21.15.32:0/3824007141 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34801a1aa0 con 0x7f3480075c20 2024-04-30T08:24:02.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.083+0000 7f3485a00700 1 --2- 172.21.15.32:0/3824007141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3480075c20 0x7f34801a1560 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3474006010 tx=0x7f3474005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:24:02.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.083+0000 7f347e000700 1 -- 172.21.15.32:0/3824007141 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3474026070 con 0x7f3480075c20 2024-04-30T08:24:02.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.083+0000 7f347e000700 1 -- 172.21.15.32:0/3824007141 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f347400bc00 con 0x7f3480075c20 2024-04-30T08:24:02.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.083+0000 7f3486e00700 1 -- 172.21.15.32:0/3824007141 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34801a1d30 con 0x7f3480075c20 2024-04-30T08:24:02.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.083+0000 7f347e000700 1 -- 172.21.15.32:0/3824007141 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3474003910 con 0x7f3480075c20 2024-04-30T08:24:02.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.083+0000 7f3486e00700 1 -- 172.21.15.32:0/3824007141 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f348019b7a0 con 0x7f3480075c20 2024-04-30T08:24:02.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.084+0000 7f3486e00700 1 -- 172.21.15.32:0/3824007141 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3480077190 con 0x7f3480075c20 2024-04-30T08:24:02.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.084+0000 7f347e000700 1 -- 172.21.15.32:0/3824007141 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f3474029070 con 0x7f3480075c20 2024-04-30T08:24:02.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.085+0000 7f347e000700 1 --2- 172.21.15.32:0/3824007141 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f346803d610 0x7f346803fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:24:02.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.085+0000 7f347e000700 1 -- 172.21.15.32:0/3824007141 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f347401e070 con 0x7f3480075c20 2024-04-30T08:24:02.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.085+0000 7f3485000700 1 --2- 172.21.15.32:0/3824007141 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f346803d610 0x7f346803fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:24:02.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.086+0000 7f3485000700 1 --2- 172.21.15.32:0/3824007141 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f346803d610 0x7f346803fb20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f346c006fd0 tx=0x7f346c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:24:02.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.091+0000 7f347e000700 1 -- 172.21.15.32:0/3824007141 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3474013070 con 0x7f3480075c20 2024-04-30T08:24:02.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.324+0000 7f3486e00700 1 -- 172.21.15.32:0/3824007141 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f348019ccb0 con 0x7f346803d610 2024-04-30T08:24:02.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.327+0000 7f347e000700 1 -- 172.21.15.32:0/3824007141 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+159 (secure 0 0 0) 0x7f348019ccb0 con 0x7f346803d610 2024-04-30T08:24:02.329 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:24:02.329 INFO:teuthology.orchestra.run.smithi032.stdout:[{"addr": "172.21.15.32", "hostname": "smithi032", "labels": [], "status": ""}, {"addr": "172.21.15.71", "hostname": "smithi071", "labels": [], "status": ""}] 2024-04-30T08:24:02.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.330+0000 7f3486e00700 1 -- 172.21.15.32:0/3824007141 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f346803d610 msgr2=0x7f346803fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:02.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.330+0000 7f3486e00700 1 --2- 172.21.15.32:0/3824007141 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f346803d610 0x7f346803fb20 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f346c006fd0 tx=0x7f346c006e40 comp rx=0 tx=0).stop 2024-04-30T08:24:02.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.330+0000 7f3486e00700 1 -- 172.21.15.32:0/3824007141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3480075c20 msgr2=0x7f34801a1560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:02.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.330+0000 7f3486e00700 1 --2- 172.21.15.32:0/3824007141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3480075c20 0x7f34801a1560 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3474006010 tx=0x7f3474005780 comp rx=0 tx=0).stop 2024-04-30T08:24:02.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.330+0000 7f3486e00700 1 -- 172.21.15.32:0/3824007141 shutdown_connections 2024-04-30T08:24:02.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.330+0000 7f3486e00700 1 --2- 172.21.15.32:0/3824007141 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f346803d610 0x7f346803fb20 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:02.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.330+0000 7f3486e00700 1 --2- 172.21.15.32:0/3824007141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3480075c20 0x7f34801a1560 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:02.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.330+0000 7f3486e00700 1 -- 172.21.15.32:0/3824007141 >> 172.21.15.32:0/3824007141 conn(0x7f34800ff400 msgr2=0x7f348010bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:24:02.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.330+0000 7f3486e00700 1 -- 172.21.15.32:0/3824007141 shutdown_connections 2024-04-30T08:24:02.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:02.331+0000 7f3486e00700 1 -- 172.21.15.32:0/3824007141 wait complete. 2024-04-30T08:24:02.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:02 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:02.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:02 smithi032 ceph-mon[28331]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:03.215 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi189 2024-04-30T08:24:03.215 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-30T08:24:03.215 DEBUG:teuthology.orchestra.run.smithi189:> dd of=/etc/ceph/ceph.conf 2024-04-30T08:24:03.234 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-30T08:24:03.234 DEBUG:teuthology.orchestra.run.smithi189:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-30T08:24:03.293 INFO:tasks.cephadm:Adding host smithi189 to orchestrator... 2024-04-30T08:24:03.293 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph orch host add smithi189 2024-04-30T08:24:03.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:03 smithi032 ceph-mon[28331]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:24:03.763 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:24:04.929 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:04 smithi032 ceph-mon[28331]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:05.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:05.847+0000 7f439b000700 1 -- 172.21.15.32:0/1372957001 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4394102860 msgr2=0x7f4394102c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:05.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:05.847+0000 7f439b000700 1 --2- 172.21.15.32:0/1372957001 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4394102860 0x7f4394102c70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f4384009910 tx=0x7f4384009c20 comp rx=0 tx=0).stop 2024-04-30T08:24:05.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:05.847+0000 7f439b000700 1 -- 172.21.15.32:0/1372957001 shutdown_connections 2024-04-30T08:24:05.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:05.847+0000 7f439b000700 1 --2- 172.21.15.32:0/1372957001 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4394102860 0x7f4394102c70 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:05.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:05.847+0000 7f439b000700 1 -- 172.21.15.32:0/1372957001 >> 172.21.15.32:0/1372957001 conn(0x7f4394078500 msgr2=0x7f4394078960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:24:05.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:05.848+0000 7f439b000700 1 -- 172.21.15.32:0/1372957001 shutdown_connections 2024-04-30T08:24:05.850 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:05 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:05.850 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:05 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:05.850 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:05 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:05.850 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:05 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:05.850 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:05 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi071", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:24:05.850 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:05 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:05.851 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:05 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:24:05.851 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:05 smithi032 ceph-mon[28331]: Updating smithi071:/etc/ceph/ceph.conf 2024-04-30T08:24:05.851 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:05 smithi032 ceph-mon[28331]: Updating smithi071:/var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-30T08:24:05.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:05.848+0000 7f439b000700 1 -- 172.21.15.32:0/1372957001 wait complete. 2024-04-30T08:24:05.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:05.849+0000 7f439b000700 1 Processor -- start 2024-04-30T08:24:05.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:05.849+0000 7f439b000700 1 -- start start 2024-04-30T08:24:05.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:05.850+0000 7f439b000700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4394102860 0x7f43941a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:24:05.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:05.850+0000 7f439b000700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f43941102a0 con 0x7f4394102860 2024-04-30T08:24:05.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:05.850+0000 7f4393e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4394102860 0x7f43941a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:24:05.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:05.850+0000 7f4393e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4394102860 0x7f43941a16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55018/0 (socket says 172.21.15.32:55018) 2024-04-30T08:24:05.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:05.850+0000 7f4393e00700 1 -- 172.21.15.32:0/1790390610 learned_addr learned my addr 172.21.15.32:0/1790390610 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:24:05.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:05.851+0000 7f4393e00700 1 -- 172.21.15.32:0/1790390610 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43941a1c30 con 0x7f4394102860 2024-04-30T08:24:05.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:05.851+0000 7f4393e00700 1 --2- 172.21.15.32:0/1790390610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4394102860 0x7f43941a16f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f43840056d0 tx=0x7f43840057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:24:05.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:05.851+0000 7f4390c00700 1 -- 172.21.15.32:0/1790390610 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4384027070 con 0x7f4394102860 2024-04-30T08:24:05.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:05.852+0000 7f4390c00700 1 -- 172.21.15.32:0/1790390610 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f438400be10 con 0x7f4394102860 2024-04-30T08:24:05.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:05.852+0000 7f439b000700 1 -- 172.21.15.32:0/1790390610 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43941a1ec0 con 0x7f4394102860 2024-04-30T08:24:05.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:05.852+0000 7f4390c00700 1 -- 172.21.15.32:0/1790390610 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4384022d20 con 0x7f4394102860 2024-04-30T08:24:05.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:05.852+0000 7f439b000700 1 -- 172.21.15.32:0/1790390610 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f439419b910 con 0x7f4394102860 2024-04-30T08:24:05.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:05.853+0000 7f439b000700 1 -- 172.21.15.32:0/1790390610 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f439406b5c0 con 0x7f4394102860 2024-04-30T08:24:05.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:05.853+0000 7f4390c00700 1 -- 172.21.15.32:0/1790390610 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f438402a070 con 0x7f4394102860 2024-04-30T08:24:05.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:05.854+0000 7f4390c00700 1 --2- 172.21.15.32:0/1790390610 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f437803d610 0x7f437803fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:24:05.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:05.854+0000 7f4390c00700 1 -- 172.21.15.32:0/1790390610 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f438401e030 con 0x7f4394102860 2024-04-30T08:24:05.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:05.854+0000 7f4393400700 1 --2- 172.21.15.32:0/1790390610 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f437803d610 0x7f437803fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:24:05.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:05.855+0000 7f4393400700 1 --2- 172.21.15.32:0/1790390610 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f437803d610 0x7f437803fb20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4380006fd0 tx=0x7f4380006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:24:05.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:05.859+0000 7f4390c00700 1 -- 172.21.15.32:0/1790390610 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4384013070 con 0x7f4394102860 2024-04-30T08:24:06.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:06.103+0000 7f439b000700 1 -- 172.21.15.32:0/1790390610 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi189", "target": ["mon-mgr", ""]}) -- 0x7f439419ce40 con 0x7f437803d610 2024-04-30T08:24:07.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:06 smithi032 ceph-mon[28331]: Updating smithi071:/etc/ceph/ceph.client.admin.keyring 2024-04-30T08:24:07.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:06 smithi032 ceph-mon[28331]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:07.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:06 smithi032 ceph-mon[28331]: Updating smithi071:/var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-30T08:24:07.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:06 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:07.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:06 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:07.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:06 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:07.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:06 smithi032 ceph-mon[28331]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi189", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:24:08.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:07 smithi032 ceph-mon[28331]: Deploying cephadm binary to smithi189 2024-04-30T08:24:09.445 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:08 smithi032 ceph-mon[28331]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:11.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:11.018+0000 7f4390c00700 1 -- 172.21.15.32:0/1790390610 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f439419ce40 con 0x7f437803d610 2024-04-30T08:24:11.022 INFO:teuthology.orchestra.run.smithi032.stdout:Added host 'smithi189' with addr '172.21.15.189' 2024-04-30T08:24:11.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:11.022+0000 7f439b000700 1 -- 172.21.15.32:0/1790390610 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f437803d610 msgr2=0x7f437803fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:11.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:11.022+0000 7f439b000700 1 --2- 172.21.15.32:0/1790390610 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f437803d610 0x7f437803fb20 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f4380006fd0 tx=0x7f4380006e40 comp rx=0 tx=0).stop 2024-04-30T08:24:11.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:11.022+0000 7f439b000700 1 -- 172.21.15.32:0/1790390610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4394102860 msgr2=0x7f43941a16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:11.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:11.022+0000 7f439b000700 1 --2- 172.21.15.32:0/1790390610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4394102860 0x7f43941a16f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f43840056d0 tx=0x7f43840057b0 comp rx=0 tx=0).stop 2024-04-30T08:24:11.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:11.022+0000 7f439b000700 1 -- 172.21.15.32:0/1790390610 shutdown_connections 2024-04-30T08:24:11.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:11.022+0000 7f439b000700 1 --2- 172.21.15.32:0/1790390610 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f437803d610 0x7f437803fb20 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:11.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:11.022+0000 7f439b000700 1 --2- 172.21.15.32:0/1790390610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4394102860 0x7f43941a16f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:11.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:11.022+0000 7f439b000700 1 -- 172.21.15.32:0/1790390610 >> 172.21.15.32:0/1790390610 conn(0x7f4394078500 msgr2=0x7f4394100830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:24:11.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:11.023+0000 7f439b000700 1 -- 172.21.15.32:0/1790390610 shutdown_connections 2024-04-30T08:24:11.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:11.023+0000 7f439b000700 1 -- 172.21.15.32:0/1790390610 wait complete. 2024-04-30T08:24:11.415 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:10 smithi032 ceph-mon[28331]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:11.823 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-30T08:24:12.281 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:24:12.440 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:12 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:12.440 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:12 smithi032 ceph-mon[28331]: Added host smithi189 2024-04-30T08:24:12.440 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:12 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:24:12.440 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:12 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:13.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:13 smithi032 ceph-mon[28331]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:13.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:13 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:14.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.333+0000 7fbf9b200700 1 -- 172.21.15.32:0/3271211492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf9c075c80 msgr2=0x7fbf9c076090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:14.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.333+0000 7fbf9b200700 1 --2- 172.21.15.32:0/3271211492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf9c075c80 0x7fbf9c076090 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c009910 tx=0x7fbf8c009c20 comp rx=0 tx=0).stop 2024-04-30T08:24:14.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.333+0000 7fbf9b200700 1 -- 172.21.15.32:0/3271211492 shutdown_connections 2024-04-30T08:24:14.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.333+0000 7fbf9b200700 1 --2- 172.21.15.32:0/3271211492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf9c075c80 0x7fbf9c076090 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:14.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.333+0000 7fbf9b200700 1 -- 172.21.15.32:0/3271211492 >> 172.21.15.32:0/3271211492 conn(0x7fbf9c0ff400 msgr2=0x7fbf9c101850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:24:14.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.333+0000 7fbf9b200700 1 -- 172.21.15.32:0/3271211492 shutdown_connections 2024-04-30T08:24:14.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.333+0000 7fbf9b200700 1 -- 172.21.15.32:0/3271211492 wait complete. 2024-04-30T08:24:14.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.335+0000 7fbf9b200700 1 Processor -- start 2024-04-30T08:24:14.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.335+0000 7fbf9b200700 1 -- start start 2024-04-30T08:24:14.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.336+0000 7fbf9b200700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf9c075c80 0x7fbf9c1a15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:24:14.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.336+0000 7fbf9b200700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbf9c1100c0 con 0x7fbf9c075c80 2024-04-30T08:24:14.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.336+0000 7fbf99e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf9c075c80 0x7fbf9c1a15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:24:14.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.336+0000 7fbf99e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf9c075c80 0x7fbf9c1a15f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51376/0 (socket says 172.21.15.32:51376) 2024-04-30T08:24:14.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.336+0000 7fbf99e00700 1 -- 172.21.15.32:0/3966667355 learned_addr learned my addr 172.21.15.32:0/3966667355 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:24:14.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.336+0000 7fbf99e00700 1 -- 172.21.15.32:0/3966667355 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf9c1a1b30 con 0x7fbf9c075c80 2024-04-30T08:24:14.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.337+0000 7fbf99e00700 1 --2- 172.21.15.32:0/3966667355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf9c075c80 0x7fbf9c1a15f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c006010 tx=0x7fbf8c005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:24:14.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.337+0000 7fbf92a00700 1 -- 172.21.15.32:0/3966667355 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbf8c026070 con 0x7fbf9c075c80 2024-04-30T08:24:14.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.337+0000 7fbf92a00700 1 -- 172.21.15.32:0/3966667355 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf8c00bc00 con 0x7fbf9c075c80 2024-04-30T08:24:14.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.337+0000 7fbf92a00700 1 -- 172.21.15.32:0/3966667355 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbf8c003910 con 0x7fbf9c075c80 2024-04-30T08:24:14.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.337+0000 7fbf9b200700 1 -- 172.21.15.32:0/3966667355 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf9c1a1dc0 con 0x7fbf9c075c80 2024-04-30T08:24:14.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.337+0000 7fbf9b200700 1 -- 172.21.15.32:0/3966667355 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf9c19b7d0 con 0x7fbf9c075c80 2024-04-30T08:24:14.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.339+0000 7fbf9b200700 1 -- 172.21.15.32:0/3966667355 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf9c0771f0 con 0x7fbf9c075c80 2024-04-30T08:24:14.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.339+0000 7fbf92a00700 1 -- 172.21.15.32:0/3966667355 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7fbf8c029070 con 0x7fbf9c075c80 2024-04-30T08:24:14.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.339+0000 7fbf92a00700 1 --2- 172.21.15.32:0/3966667355 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbf8003d670 0x7fbf8003fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:24:14.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.339+0000 7fbf92a00700 1 -- 172.21.15.32:0/3966667355 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fbf8c01e070 con 0x7fbf9c075c80 2024-04-30T08:24:14.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.340+0000 7fbf99400700 1 --2- 172.21.15.32:0/3966667355 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbf8003d670 0x7fbf8003fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:24:14.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.340+0000 7fbf99400700 1 --2- 172.21.15.32:0/3966667355 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbf8003d670 0x7fbf8003fb80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fbf84006fd0 tx=0x7fbf84006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:24:14.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.345+0000 7fbf92a00700 1 -- 172.21.15.32:0/3966667355 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbf8c013070 con 0x7fbf9c075c80 2024-04-30T08:24:14.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.583+0000 7fbf9b200700 1 -- 172.21.15.32:0/3966667355 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fbf9c19cc90 con 0x7fbf8003d670 2024-04-30T08:24:14.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.585+0000 7fbf92a00700 1 -- 172.21.15.32:0/3966667355 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+239 (secure 0 0 0) 0x7fbf9c19cc90 con 0x7fbf8003d670 2024-04-30T08:24:14.587 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:24:14.587 INFO:teuthology.orchestra.run.smithi032.stdout:[{"addr": "172.21.15.32", "hostname": "smithi032", "labels": [], "status": ""}, {"addr": "172.21.15.71", "hostname": "smithi071", "labels": [], "status": ""}, {"addr": "172.21.15.189", "hostname": "smithi189", "labels": [], "status": ""}] 2024-04-30T08:24:14.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.589+0000 7fbf9b200700 1 -- 172.21.15.32:0/3966667355 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbf8003d670 msgr2=0x7fbf8003fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:14.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.589+0000 7fbf9b200700 1 --2- 172.21.15.32:0/3966667355 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbf8003d670 0x7fbf8003fb80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fbf84006fd0 tx=0x7fbf84006e40 comp rx=0 tx=0).stop 2024-04-30T08:24:14.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.589+0000 7fbf9b200700 1 -- 172.21.15.32:0/3966667355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf9c075c80 msgr2=0x7fbf9c1a15f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:14.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.589+0000 7fbf9b200700 1 --2- 172.21.15.32:0/3966667355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf9c075c80 0x7fbf9c1a15f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fbf8c006010 tx=0x7fbf8c005780 comp rx=0 tx=0).stop 2024-04-30T08:24:14.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.589+0000 7fbf9b200700 1 -- 172.21.15.32:0/3966667355 shutdown_connections 2024-04-30T08:24:14.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.589+0000 7fbf9b200700 1 --2- 172.21.15.32:0/3966667355 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbf8003d670 0x7fbf8003fb80 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:14.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.589+0000 7fbf9b200700 1 --2- 172.21.15.32:0/3966667355 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf9c075c80 0x7fbf9c1a15f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:14.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.589+0000 7fbf9b200700 1 -- 172.21.15.32:0/3966667355 >> 172.21.15.32:0/3966667355 conn(0x7fbf9c0ff400 msgr2=0x7fbf9c10bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:24:14.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.589+0000 7fbf9b200700 1 -- 172.21.15.32:0/3966667355 shutdown_connections 2024-04-30T08:24:14.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:14.590+0000 7fbf9b200700 1 -- 172.21.15.32:0/3966667355 wait complete. 2024-04-30T08:24:14.927 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:14 smithi032 ceph-mon[28331]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:14.928 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:14 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:15.339 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-30T08:24:15.340 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-30T08:24:15.800 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:24:16.086 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:15 smithi032 ceph-mon[28331]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:24:17.047 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:16 smithi032 ceph-mon[28331]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:18.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:18.032+0000 7f6bc3200700 1 -- 172.21.15.32:0/3129435528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bc410bb90 msgr2=0x7f6bc410bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:18.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:18.032+0000 7f6bc3200700 1 --2- 172.21.15.32:0/3129435528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bc410bb90 0x7f6bc410bfa0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f6bb4009910 tx=0x7f6bb4009c20 comp rx=0 tx=0).stop 2024-04-30T08:24:18.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:18.032+0000 7f6bc3200700 1 -- 172.21.15.32:0/3129435528 shutdown_connections 2024-04-30T08:24:18.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:18.032+0000 7f6bc3200700 1 --2- 172.21.15.32:0/3129435528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bc410bb90 0x7f6bc410bfa0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:18.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:18.032+0000 7f6bc3200700 1 -- 172.21.15.32:0/3129435528 >> 172.21.15.32:0/3129435528 conn(0x7f6bc4101630 msgr2=0x7f6bc4103a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:24:18.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:18.032+0000 7f6bc3200700 1 -- 172.21.15.32:0/3129435528 shutdown_connections 2024-04-30T08:24:18.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:18.032+0000 7f6bc3200700 1 -- 172.21.15.32:0/3129435528 wait complete. 2024-04-30T08:24:18.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:18.034+0000 7f6bc3200700 1 Processor -- start 2024-04-30T08:24:18.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:18.034+0000 7f6bc3200700 1 -- start start 2024-04-30T08:24:18.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:18.035+0000 7f6bc3200700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bc410bb90 0x7f6bc41a1570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:24:18.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:18.035+0000 7f6bc3200700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6bc4110050 con 0x7f6bc410bb90 2024-04-30T08:24:18.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:18.035+0000 7f6bc1e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bc410bb90 0x7f6bc41a1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:24:18.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:18.035+0000 7f6bc1e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bc410bb90 0x7f6bc41a1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35592/0 (socket says 172.21.15.32:35592) 2024-04-30T08:24:18.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:18.035+0000 7f6bc1e00700 1 -- 172.21.15.32:0/3402431792 learned_addr learned my addr 172.21.15.32:0/3402431792 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:24:18.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:18.036+0000 7f6bc1e00700 1 -- 172.21.15.32:0/3402431792 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bc41a1ab0 con 0x7f6bc410bb90 2024-04-30T08:24:18.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:18.036+0000 7f6bc1e00700 1 --2- 172.21.15.32:0/3402431792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bc410bb90 0x7f6bc41a1570 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6bb40092e0 tx=0x7f6bb4005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:24:18.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:18.036+0000 7f6bbaa00700 1 -- 172.21.15.32:0/3402431792 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6bb4026070 con 0x7f6bc410bb90 2024-04-30T08:24:18.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:18.036+0000 7f6bbaa00700 1 -- 172.21.15.32:0/3402431792 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6bb400b6f0 con 0x7f6bc410bb90 2024-04-30T08:24:18.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:18.036+0000 7f6bbaa00700 1 -- 172.21.15.32:0/3402431792 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6bb40037f0 con 0x7f6bc410bb90 2024-04-30T08:24:18.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:18.036+0000 7f6bc3200700 1 -- 172.21.15.32:0/3402431792 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bc41a1da0 con 0x7f6bc410bb90 2024-04-30T08:24:18.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:18.036+0000 7f6bc3200700 1 -- 172.21.15.32:0/3402431792 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bc419b8d0 con 0x7f6bc410bb90 2024-04-30T08:24:18.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:18.038+0000 7f6bc3200700 1 -- 172.21.15.32:0/3402431792 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bc41071d0 con 0x7f6bc410bb90 2024-04-30T08:24:18.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:18.038+0000 7f6bbaa00700 1 -- 172.21.15.32:0/3402431792 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f6bb4029070 con 0x7f6bc410bb90 2024-04-30T08:24:18.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:18.039+0000 7f6bbaa00700 1 --2- 172.21.15.32:0/3402431792 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f6ba803d670 0x7f6ba803fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:24:18.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:18.039+0000 7f6bbaa00700 1 -- 172.21.15.32:0/3402431792 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f6bb401e070 con 0x7f6bc410bb90 2024-04-30T08:24:18.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:18.039+0000 7f6bbbe00700 1 --2- 172.21.15.32:0/3402431792 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f6ba803d670 0x7f6ba803fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:24:18.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:18.040+0000 7f6bbbe00700 1 --2- 172.21.15.32:0/3402431792 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f6ba803d670 0x7f6ba803fb80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6bac006fd0 tx=0x7f6bac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:24:18.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:18.045+0000 7f6bbaa00700 1 -- 172.21.15.32:0/3402431792 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6bb4013070 con 0x7f6bc410bb90 2024-04-30T08:24:18.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:17 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:18.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:17 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:18.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:17 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:18.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:17 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:18.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:17 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:24:18.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:17 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:18.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:17 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:24:18.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:17 smithi032 ceph-mon[28331]: Updating smithi189:/etc/ceph/ceph.conf 2024-04-30T08:24:18.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:18.281+0000 7f6bc3200700 1 -- 172.21.15.32:0/3402431792 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f6bc41a1f70 con 0x7f6bc410bb90 2024-04-30T08:24:19.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:19.264+0000 7f6bbaa00700 1 -- 172.21.15.32:0/3402431792 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f6bb4021410 con 0x7f6bc410bb90 2024-04-30T08:24:19.266 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:18 smithi032 ceph-mon[28331]: Updating smithi189:/var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-30T08:24:19.266 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:18 smithi032 ceph-mon[28331]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:19.266 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:18 smithi032 ceph-mon[28331]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-04-30T08:24:19.266 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:18 smithi032 ceph-mon[28331]: Updating smithi189:/var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-30T08:24:19.266 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:18 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:19.267 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:18 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:19.267 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:18 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:19.267 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:18 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3402431792' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-30T08:24:19.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:19.268+0000 7f6bc3200700 1 -- 172.21.15.32:0/3402431792 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f6ba803d670 msgr2=0x7f6ba803fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:19.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:19.268+0000 7f6bc3200700 1 --2- 172.21.15.32:0/3402431792 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f6ba803d670 0x7f6ba803fb80 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f6bac006fd0 tx=0x7f6bac006e40 comp rx=0 tx=0).stop 2024-04-30T08:24:19.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:19.268+0000 7f6bc3200700 1 -- 172.21.15.32:0/3402431792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bc410bb90 msgr2=0x7f6bc41a1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:19.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:19.268+0000 7f6bc3200700 1 --2- 172.21.15.32:0/3402431792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bc410bb90 0x7f6bc41a1570 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f6bb40092e0 tx=0x7f6bb4005780 comp rx=0 tx=0).stop 2024-04-30T08:24:19.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:19.268+0000 7f6bc3200700 1 -- 172.21.15.32:0/3402431792 shutdown_connections 2024-04-30T08:24:19.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:19.268+0000 7f6bc3200700 1 --2- 172.21.15.32:0/3402431792 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f6ba803d670 0x7f6ba803fb80 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:19.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:19.268+0000 7f6bc3200700 1 --2- 172.21.15.32:0/3402431792 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6bc410bb90 0x7f6bc41a1570 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:19.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:19.268+0000 7f6bc3200700 1 -- 172.21.15.32:0/3402431792 >> 172.21.15.32:0/3402431792 conn(0x7f6bc4101630 msgr2=0x7f6bc4102640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:24:19.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:19.269+0000 7f6bc3200700 1 -- 172.21.15.32:0/3402431792 shutdown_connections 2024-04-30T08:24:19.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:19.269+0000 7f6bc3200700 1 -- 172.21.15.32:0/3402431792 wait complete. 2024-04-30T08:24:19.271 INFO:teuthology.orchestra.run.smithi032.stderr:adjusted tunables profile to default 2024-04-30T08:24:20.070 INFO:tasks.cephadm:Adding mon.a on smithi032 2024-04-30T08:24:20.070 INFO:tasks.cephadm:Adding mon.b on smithi071 2024-04-30T08:24:20.070 INFO:tasks.cephadm:Adding mon.c on smithi189 2024-04-30T08:24:20.070 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi032:172.21.15.32=a;smithi071:172.21.15.71=b;smithi189:172.21.15.189=c' 2024-04-30T08:24:20.300 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-30T08:24:20.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:20 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3402431792' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-30T08:24:20.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:20 smithi032 ceph-mon[28331]: osdmap e4: 0 total, 0 up, 0 in 2024-04-30T08:24:20.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:20 smithi032 ceph-mon[28331]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:21.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.690+0000 7f36916ca700 1 -- 172.21.15.189:0/83318406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f368c1089e0 msgr2=0x7f368c108df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:21.691 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.690+0000 7f36916ca700 1 --2- 172.21.15.189:0/83318406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f368c1089e0 0x7f368c108df0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f367c009910 tx=0x7f367c009c20 comp rx=0 tx=0).stop 2024-04-30T08:24:21.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.690+0000 7f36916ca700 1 -- 172.21.15.189:0/83318406 shutdown_connections 2024-04-30T08:24:21.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.690+0000 7f36916ca700 1 --2- 172.21.15.189:0/83318406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f368c1089e0 0x7f368c108df0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:21.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.690+0000 7f36916ca700 1 -- 172.21.15.189:0/83318406 >> 172.21.15.189:0/83318406 conn(0x7f368c078500 msgr2=0x7f368c078960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:24:21.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.690+0000 7f36916ca700 1 -- 172.21.15.189:0/83318406 shutdown_connections 2024-04-30T08:24:21.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.690+0000 7f36916ca700 1 -- 172.21.15.189:0/83318406 wait complete. 2024-04-30T08:24:21.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.690+0000 7f36916ca700 1 Processor -- start 2024-04-30T08:24:21.692 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.691+0000 7f36916ca700 1 -- start start 2024-04-30T08:24:21.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.691+0000 7f36916ca700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f368c1089e0 0x7f368c19a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:24:21.693 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.691+0000 7f36916ca700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f368c10cf30 con 0x7f368c1089e0 2024-04-30T08:24:21.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.691+0000 7f368affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f368c1089e0 0x7f368c19a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:24:21.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.691+0000 7f368affd700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f368c1089e0 0x7f368c19a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.189:57094/0 (socket says 172.21.15.189:57094) 2024-04-30T08:24:21.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.691+0000 7f368affd700 1 -- 172.21.15.189:0/1023865787 learned_addr learned my addr 172.21.15.189:0/1023865787 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-30T08:24:21.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.692+0000 7f368affd700 1 -- 172.21.15.189:0/1023865787 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f368c06bc50 con 0x7f368c1089e0 2024-04-30T08:24:21.694 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.693+0000 7f368affd700 1 --2- 172.21.15.189:0/1023865787 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f368c1089e0 0x7f368c19a6f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f367c0092e0 tx=0x7f367c005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:24:21.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.693+0000 7f3673fff700 1 -- 172.21.15.189:0/1023865787 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f367c026070 con 0x7f368c1089e0 2024-04-30T08:24:21.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.693+0000 7f36916ca700 1 -- 172.21.15.189:0/1023865787 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f368c19acf0 con 0x7f368c1089e0 2024-04-30T08:24:21.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.693+0000 7f36916ca700 1 -- 172.21.15.189:0/1023865787 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f368c19b1b0 con 0x7f368c1089e0 2024-04-30T08:24:21.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.693+0000 7f3673fff700 1 -- 172.21.15.189:0/1023865787 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f367c00b6f0 con 0x7f368c1089e0 2024-04-30T08:24:21.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.693+0000 7f3673fff700 1 -- 172.21.15.189:0/1023865787 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f367c0037f0 con 0x7f368c1089e0 2024-04-30T08:24:21.695 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.694+0000 7f36916ca700 1 -- 172.21.15.189:0/1023865787 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f368c1040f0 con 0x7f368c1089e0 2024-04-30T08:24:21.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.695+0000 7f3673fff700 1 -- 172.21.15.189:0/1023865787 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f367c029070 con 0x7f368c1089e0 2024-04-30T08:24:21.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.695+0000 7f3673fff700 1 --2- 172.21.15.189:0/1023865787 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f367403d610 0x7f367403fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:24:21.696 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.695+0000 7f3673fff700 1 -- 172.21.15.189:0/1023865787 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f367c01e070 con 0x7f368c1089e0 2024-04-30T08:24:21.697 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.696+0000 7f368a7fc700 1 --2- 172.21.15.189:0/1023865787 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f367403d610 0x7f367403fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:24:21.698 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.697+0000 7f368a7fc700 1 --2- 172.21.15.189:0/1023865787 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f367403d610 0x7f367403fb20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3680006fd0 tx=0x7f3680006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:24:21.712 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.712+0000 7f3673fff700 1 -- 172.21.15.189:0/1023865787 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f367c013070 con 0x7f368c1089e0 2024-04-30T08:24:21.922 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.921+0000 7f36916ca700 1 -- 172.21.15.189:0/1023865787 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi032:172.21.15.32=a;smithi071:172.21.15.71=b;smithi189:172.21.15.189=c", "target": ["mon-mgr", ""]}) -- 0x7f368c06b3a0 con 0x7f367403d610 2024-04-30T08:24:21.938 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.937+0000 7f3673fff700 1 -- 172.21.15.189:0/1023865787 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f368c06b3a0 con 0x7f367403d610 2024-04-30T08:24:21.938 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled mon update... 2024-04-30T08:24:21.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.939+0000 7f36916ca700 1 -- 172.21.15.189:0/1023865787 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f367403d610 msgr2=0x7f367403fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:21.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.939+0000 7f36916ca700 1 --2- 172.21.15.189:0/1023865787 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f367403d610 0x7f367403fb20 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3680006fd0 tx=0x7f3680006e40 comp rx=0 tx=0).stop 2024-04-30T08:24:21.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.939+0000 7f36916ca700 1 -- 172.21.15.189:0/1023865787 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f368c1089e0 msgr2=0x7f368c19a6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:21.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.939+0000 7f36916ca700 1 --2- 172.21.15.189:0/1023865787 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f368c1089e0 0x7f368c19a6f0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f367c0092e0 tx=0x7f367c005780 comp rx=0 tx=0).stop 2024-04-30T08:24:21.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.939+0000 7f36916ca700 1 -- 172.21.15.189:0/1023865787 shutdown_connections 2024-04-30T08:24:21.940 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.939+0000 7f36916ca700 1 --2- 172.21.15.189:0/1023865787 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f367403d610 0x7f367403fb20 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:21.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.939+0000 7f36916ca700 1 --2- 172.21.15.189:0/1023865787 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f368c1089e0 0x7f368c19a6f0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:21.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.939+0000 7f36916ca700 1 -- 172.21.15.189:0/1023865787 >> 172.21.15.189:0/1023865787 conn(0x7f368c078500 msgr2=0x7f368c0ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:24:21.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.939+0000 7f36916ca700 1 -- 172.21.15.189:0/1023865787 shutdown_connections 2024-04-30T08:24:21.941 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:21.939+0000 7f36916ca700 1 -- 172.21.15.189:0/1023865787 wait complete. 2024-04-30T08:24:22.586 DEBUG:teuthology.orchestra.run.smithi071:mon.b> sudo journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mon.b.service 2024-04-30T08:24:22.589 DEBUG:teuthology.orchestra.run.smithi189:mon.c> sudo journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mon.c.service 2024-04-30T08:24:22.591 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-30T08:24:22.591 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-30T08:24:23.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:22 smithi032 ceph-mon[28331]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:23.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:22 smithi032 ceph-mon[28331]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi032:172.21.15.32=a;smithi071:172.21.15.71=b;smithi189:172.21.15.189=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:24:23.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:22 smithi032 ceph-mon[28331]: Saving service mon spec with placement smithi032:172.21.15.32=a;smithi071:172.21.15.71=b;smithi189:172.21.15.189=c;count:3 2024-04-30T08:24:23.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:22 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:23.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:22 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:24:23.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:22 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:23.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:22 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:24:23.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:22 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:23.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:22 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-30T08:24:23.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:22 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:23.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:22 smithi032 ceph-mon[28331]: Deploying daemon mon.c on smithi189 2024-04-30T08:24:23.296 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-30T08:24:25.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:24 smithi032 ceph-mon[28331]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:25.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:25.975+0000 7ff49a616700 1 -- 172.21.15.189:0/1694198515 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff494108b40 msgr2=0x7ff494108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:25.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:25.975+0000 7ff49a616700 1 --2- 172.21.15.189:0/1694198515 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff494108b40 0x7ff494108f50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff484009910 tx=0x7ff484009c20 comp rx=0 tx=0).stop 2024-04-30T08:24:25.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:25.975+0000 7ff49a616700 1 -- 172.21.15.189:0/1694198515 shutdown_connections 2024-04-30T08:24:25.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:25.975+0000 7ff49a616700 1 --2- 172.21.15.189:0/1694198515 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff494108b40 0x7ff494108f50 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:25.977 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:25.975+0000 7ff49a616700 1 -- 172.21.15.189:0/1694198515 >> 172.21.15.189:0/1694198515 conn(0x7ff4940fdd80 msgr2=0x7ff4941001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:24:25.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:25.975+0000 7ff49a616700 1 -- 172.21.15.189:0/1694198515 shutdown_connections 2024-04-30T08:24:25.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:25.975+0000 7ff49a616700 1 -- 172.21.15.189:0/1694198515 wait complete. 2024-04-30T08:24:25.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:25.976+0000 7ff49a616700 1 Processor -- start 2024-04-30T08:24:25.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:25.976+0000 7ff49a616700 1 -- start start 2024-04-30T08:24:25.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:25.976+0000 7ff49a616700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff494108b40 0x7ff49419e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:24:25.978 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:25.976+0000 7ff49a616700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff49410d060 con 0x7ff494108b40 2024-04-30T08:24:25.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:25.977+0000 7ff493fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff494108b40 0x7ff49419e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:24:25.979 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:25.977+0000 7ff493fff700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff494108b40 0x7ff49419e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.189:39932/0 (socket says 172.21.15.189:39932) 2024-04-30T08:24:25.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:25.977+0000 7ff493fff700 1 -- 172.21.15.189:0/548230897 learned_addr learned my addr 172.21.15.189:0/548230897 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-30T08:24:25.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:25.978+0000 7ff493fff700 1 -- 172.21.15.189:0/548230897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff49419ea40 con 0x7ff494108b40 2024-04-30T08:24:25.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:25.978+0000 7ff493fff700 1 --2- 172.21.15.189:0/548230897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff494108b40 0x7ff49419e500 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff4840056d0 tx=0x7ff4840057b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:24:25.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:25.978+0000 7ff4917fa700 1 -- 172.21.15.189:0/548230897 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff484027070 con 0x7ff494108b40 2024-04-30T08:24:25.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:25.979+0000 7ff49a616700 1 -- 172.21.15.189:0/548230897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff49419ecd0 con 0x7ff494108b40 2024-04-30T08:24:25.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:25.979+0000 7ff49a616700 1 -- 172.21.15.189:0/548230897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff494198690 con 0x7ff494108b40 2024-04-30T08:24:25.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:25.979+0000 7ff4917fa700 1 -- 172.21.15.189:0/548230897 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff48400be10 con 0x7ff494108b40 2024-04-30T08:24:25.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:25.979+0000 7ff4917fa700 1 -- 172.21.15.189:0/548230897 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff484022cd0 con 0x7ff494108b40 2024-04-30T08:24:25.980 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:25.979+0000 7ff49a616700 1 -- 172.21.15.189:0/548230897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff480005360 con 0x7ff494108b40 2024-04-30T08:24:25.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:25.981+0000 7ff4917fa700 1 -- 172.21.15.189:0/548230897 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7ff48402a070 con 0x7ff494108b40 2024-04-30T08:24:25.981 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:25.981+0000 7ff4917fa700 1 --2- 172.21.15.189:0/548230897 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff47c03d610 0x7ff47c03fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:24:25.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:25.981+0000 7ff4917fa700 1 -- 172.21.15.189:0/548230897 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff48401e030 con 0x7ff494108b40 2024-04-30T08:24:25.982 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:25.981+0000 7ff4937fe700 1 --2- 172.21.15.189:0/548230897 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff47c03d610 0x7ff47c03fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:24:25.983 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:25.982+0000 7ff4937fe700 1 --2- 172.21.15.189:0/548230897 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff47c03d610 0x7ff47c03fb20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff488006fd0 tx=0x7ff488006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:24:25.997 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:25.996+0000 7ff4917fa700 1 -- 172.21.15.189:0/548230897 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff484013070 con 0x7ff494108b40 2024-04-30T08:24:26.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:26 smithi189 systemd[1]: Starting Ceph mon.c for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:24:26.300 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:26.299+0000 7ff49a616700 1 -- 172.21.15.189:0/548230897 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7ff480005920 con 0x7ff494108b40 2024-04-30T08:24:26.301 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:26.300+0000 7ff4917fa700 1 -- 172.21.15.189:0/548230897 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7ff484035090 con 0x7ff494108b40 2024-04-30T08:24:26.301 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-30T08:24:26.302 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":1,"fsid":"98c0a670-06ca-11ef-bc93-c7b262605968","modified":"2024-04-30T08:22:06.865404Z","created":"2024-04-30T08:22:06.865404Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-30T08:24:26.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:26.303+0000 7ff49a616700 1 -- 172.21.15.189:0/548230897 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff47c03d610 msgr2=0x7ff47c03fb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:26.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:26.303+0000 7ff49a616700 1 --2- 172.21.15.189:0/548230897 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff47c03d610 0x7ff47c03fb20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7ff488006fd0 tx=0x7ff488006e40 comp rx=0 tx=0).stop 2024-04-30T08:24:26.304 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:26.303+0000 7ff49a616700 1 -- 172.21.15.189:0/548230897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff494108b40 msgr2=0x7ff49419e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:26.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:26.303+0000 7ff49a616700 1 --2- 172.21.15.189:0/548230897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff494108b40 0x7ff49419e500 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff4840056d0 tx=0x7ff4840057b0 comp rx=0 tx=0).stop 2024-04-30T08:24:26.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:26.303+0000 7ff49a616700 1 -- 172.21.15.189:0/548230897 shutdown_connections 2024-04-30T08:24:26.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:26.303+0000 7ff49a616700 1 --2- 172.21.15.189:0/548230897 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff47c03d610 0x7ff47c03fb20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:26.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:26.303+0000 7ff49a616700 1 --2- 172.21.15.189:0/548230897 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff494108b40 0x7ff49419e500 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:26.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:26.303+0000 7ff49a616700 1 -- 172.21.15.189:0/548230897 >> 172.21.15.189:0/548230897 conn(0x7ff4940fdd80 msgr2=0x7ff4940fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:24:26.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:26.303+0000 7ff49a616700 1 -- 172.21.15.189:0/548230897 shutdown_connections 2024-04-30T08:24:26.305 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:26.303+0000 7ff49a616700 1 -- 172.21.15.189:0/548230897 wait complete. 2024-04-30T08:24:26.305 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 1 2024-04-30T08:24:26.636 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:26 smithi189 podman[33001]: 2024-04-30 08:24:26.3133836 +0000 UTC m=+0.019195982 image pull 16056ace0edc7f73516cf28297f1db04dda872801fb5b17b9157b0d1ccf38f9b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de 2024-04-30T08:24:27.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:26 smithi032 ceph-mon[28331]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:27.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:26 smithi032 ceph-mon[28331]: from='client.? 172.21.15.189:0/548230897' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-30T08:24:27.233 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 podman[33001]: 2024-04-30 08:24:27.022950718 +0000 UTC m=+0.728763070 container create bf88bed5fe55d41eb96b923f63c7f14003672d82eafa509460d7a8f06986f14d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mon-c, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0) 2024-04-30T08:24:27.233 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 podman[33001]: 2024-04-30 08:24:27.233350173 +0000 UTC m=+0.939162540 container init bf88bed5fe55d41eb96b923f63c7f14003672d82eafa509460d7a8f06986f14d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mon-c, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-30T08:24:27.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 podman[33001]: 2024-04-30 08:24:27.236428518 +0000 UTC m=+0.942240872 container start bf88bed5fe55d41eb96b923f63c7f14003672d82eafa509460d7a8f06986f14d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-30T08:24:27.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: set uid:gid to 167:167 (ceph:ceph) 2024-04-30T08:24:27.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: ceph version 19.0.0-3348-g8feaf7d9 (8feaf7d9864e980041b042fd5d964ac18adfd20f) squid (dev), process ceph-mon, pid 2 2024-04-30T08:24:27.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: pidfile_write: ignore empty --pid-file 2024-04-30T08:24:27.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: load: jerasure load: lrc 2024-04-30T08:24:27.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: RocksDB version: 7.9.2 2024-04-30T08:24:27.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Git sha 0 2024-04-30T08:24:27.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Compile date 2024-04-29 22:33:18 2024-04-30T08:24:27.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: DB SUMMARY 2024-04-30T08:24:27.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: DB Session ID: QKWLU3JG4PB2KTDXXEO2 2024-04-30T08:24:27.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: CURRENT file: CURRENT 2024-04-30T08:24:27.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: IDENTITY file: IDENTITY 2024-04-30T08:24:27.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-30T08:24:27.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-04-30T08:24:27.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-04-30T08:24:27.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.error_if_exists: 0 2024-04-30T08:24:27.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.create_if_missing: 0 2024-04-30T08:24:27.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.paranoid_checks: 1 2024-04-30T08:24:27.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-30T08:24:27.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-30T08:24:27.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-30T08:24:27.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.env: 0x556341d66500 2024-04-30T08:24:27.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.fs: PosixFileSystem 2024-04-30T08:24:27.553 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.info_log: 0x5563436affe0 2024-04-30T08:24:27.553 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.max_file_opening_threads: 16 2024-04-30T08:24:27.553 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.statistics: (nil) 2024-04-30T08:24:27.553 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.use_fsync: 0 2024-04-30T08:24:27.553 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.max_log_file_size: 0 2024-04-30T08:24:27.553 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-30T08:24:27.553 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-30T08:24:27.553 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.keep_log_file_num: 1000 2024-04-30T08:24:27.553 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.recycle_log_file_num: 0 2024-04-30T08:24:27.553 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.allow_fallocate: 1 2024-04-30T08:24:27.553 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.allow_mmap_reads: 0 2024-04-30T08:24:27.553 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.allow_mmap_writes: 0 2024-04-30T08:24:27.554 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.use_direct_reads: 0 2024-04-30T08:24:27.554 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-30T08:24:27.554 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.create_missing_column_families: 0 2024-04-30T08:24:27.554 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.db_log_dir: 2024-04-30T08:24:27.554 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.wal_dir: 2024-04-30T08:24:27.554 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-30T08:24:27.554 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-30T08:24:27.554 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-30T08:24:27.554 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-30T08:24:27.555 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-30T08:24:27.555 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-30T08:24:27.555 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.advise_random_on_open: 1 2024-04-30T08:24:27.555 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.db_write_buffer_size: 0 2024-04-30T08:24:27.555 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.write_buffer_manager: 0x5563436808c0 2024-04-30T08:24:27.555 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-30T08:24:27.555 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-30T08:24:27.555 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-30T08:24:27.555 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.rate_limiter: (nil) 2024-04-30T08:24:27.555 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-30T08:24:27.555 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.wal_recovery_mode: 2 2024-04-30T08:24:27.555 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.enable_thread_tracking: 0 2024-04-30T08:24:27.555 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.enable_pipelined_write: 0 2024-04-30T08:24:27.555 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.unordered_write: 0 2024-04-30T08:24:27.556 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-30T08:24:27.556 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-30T08:24:27.556 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-30T08:24:27.556 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-30T08:24:27.556 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.row_cache: None 2024-04-30T08:24:27.556 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.wal_filter: None 2024-04-30T08:24:27.556 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-30T08:24:27.556 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.allow_ingest_behind: 0 2024-04-30T08:24:27.556 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.two_write_queues: 0 2024-04-30T08:24:27.556 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.manual_wal_flush: 0 2024-04-30T08:24:27.556 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.wal_compression: 0 2024-04-30T08:24:27.556 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.atomic_flush: 0 2024-04-30T08:24:27.556 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-30T08:24:27.556 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-30T08:24:27.558 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-30T08:24:27.558 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.log_readahead_size: 0 2024-04-30T08:24:27.558 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-30T08:24:27.558 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.best_efforts_recovery: 0 2024-04-30T08:24:27.558 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-30T08:24:27.558 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-30T08:24:27.558 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.allow_data_in_errors: 0 2024-04-30T08:24:27.559 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.db_host_id: __hostname__ 2024-04-30T08:24:27.559 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-30T08:24:27.559 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.max_background_jobs: 2 2024-04-30T08:24:27.559 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.max_background_compactions: -1 2024-04-30T08:24:27.559 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.max_subcompactions: 1 2024-04-30T08:24:27.559 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-30T08:24:27.559 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-30T08:24:27.559 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-30T08:24:27.559 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.max_total_wal_size: 0 2024-04-30T08:24:27.559 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-30T08:24:27.559 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-30T08:24:27.559 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-30T08:24:27.559 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-30T08:24:27.560 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.max_open_files: -1 2024-04-30T08:24:27.560 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.bytes_per_sync: 0 2024-04-30T08:24:27.560 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-30T08:24:27.560 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-30T08:24:27.560 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.compaction_readahead_size: 0 2024-04-30T08:24:27.560 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.max_background_flushes: -1 2024-04-30T08:24:27.560 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Compression algorithms supported: 2024-04-30T08:24:27.560 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-30T08:24:27.560 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: kZSTD supported: 0 2024-04-30T08:24:27.560 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: kXpressCompression supported: 0 2024-04-30T08:24:27.560 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: kLZ4HCCompression supported: 1 2024-04-30T08:24:27.560 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: kZlibCompression supported: 1 2024-04-30T08:24:27.560 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: kSnappyCompression supported: 1 2024-04-30T08:24:27.560 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: kLZ4Compression supported: 1 2024-04-30T08:24:27.561 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: kBZip2Compression supported: 0 2024-04-30T08:24:27.561 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-30T08:24:27.561 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-30T08:24:27.561 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-04-30T08:24:27.561 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-30T08:24:27.561 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-30T08:24:27.561 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.merge_operator: 2024-04-30T08:24:27.561 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.compaction_filter: None 2024-04-30T08:24:27.561 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.compaction_filter_factory: None 2024-04-30T08:24:27.561 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.sst_partitioner_factory: None 2024-04-30T08:24:27.561 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-30T08:24:27.561 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-30T08:24:27.561 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5563437b61c0) 2024-04-30T08:24:27.562 INFO:journalctl@ceph.mon.c.smithi189.stdout: cache_index_and_filter_blocks: 1 2024-04-30T08:24:27.562 INFO:journalctl@ceph.mon.c.smithi189.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-30T08:24:27.562 INFO:journalctl@ceph.mon.c.smithi189.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-30T08:24:27.562 INFO:journalctl@ceph.mon.c.smithi189.stdout: pin_top_level_index_and_filter: 1 2024-04-30T08:24:27.562 INFO:journalctl@ceph.mon.c.smithi189.stdout: index_type: 0 2024-04-30T08:24:27.562 INFO:journalctl@ceph.mon.c.smithi189.stdout: data_block_index_type: 0 2024-04-30T08:24:27.562 INFO:journalctl@ceph.mon.c.smithi189.stdout: index_shortening: 1 2024-04-30T08:24:27.562 INFO:journalctl@ceph.mon.c.smithi189.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-30T08:24:27.562 INFO:journalctl@ceph.mon.c.smithi189.stdout: checksum: 4 2024-04-30T08:24:27.562 INFO:journalctl@ceph.mon.c.smithi189.stdout: no_block_cache: 0 2024-04-30T08:24:27.562 INFO:journalctl@ceph.mon.c.smithi189.stdout: block_cache: 0x5563436b31f0 2024-04-30T08:24:27.562 INFO:journalctl@ceph.mon.c.smithi189.stdout: block_cache_name: BinnedLRUCache 2024-04-30T08:24:27.563 INFO:journalctl@ceph.mon.c.smithi189.stdout: block_cache_options: 2024-04-30T08:24:27.563 INFO:journalctl@ceph.mon.c.smithi189.stdout: capacity : 536870912 2024-04-30T08:24:27.563 INFO:journalctl@ceph.mon.c.smithi189.stdout: num_shard_bits : 4 2024-04-30T08:24:27.563 INFO:journalctl@ceph.mon.c.smithi189.stdout: strict_capacity_limit : 0 2024-04-30T08:24:27.563 INFO:journalctl@ceph.mon.c.smithi189.stdout: high_pri_pool_ratio: 0.000 2024-04-30T08:24:27.563 INFO:journalctl@ceph.mon.c.smithi189.stdout: block_cache_compressed: (nil) 2024-04-30T08:24:27.563 INFO:journalctl@ceph.mon.c.smithi189.stdout: persistent_cache: (nil) 2024-04-30T08:24:27.563 INFO:journalctl@ceph.mon.c.smithi189.stdout: block_size: 4096 2024-04-30T08:24:27.563 INFO:journalctl@ceph.mon.c.smithi189.stdout: block_size_deviation: 10 2024-04-30T08:24:27.563 INFO:journalctl@ceph.mon.c.smithi189.stdout: block_restart_interval: 16 2024-04-30T08:24:27.563 INFO:journalctl@ceph.mon.c.smithi189.stdout: index_block_restart_interval: 1 2024-04-30T08:24:27.563 INFO:journalctl@ceph.mon.c.smithi189.stdout: metadata_block_size: 4096 2024-04-30T08:24:27.564 INFO:journalctl@ceph.mon.c.smithi189.stdout: partition_filters: 0 2024-04-30T08:24:27.564 INFO:journalctl@ceph.mon.c.smithi189.stdout: use_delta_encoding: 1 2024-04-30T08:24:27.564 INFO:journalctl@ceph.mon.c.smithi189.stdout: filter_policy: bloomfilter 2024-04-30T08:24:27.564 INFO:journalctl@ceph.mon.c.smithi189.stdout: whole_key_filtering: 1 2024-04-30T08:24:27.564 INFO:journalctl@ceph.mon.c.smithi189.stdout: verify_compression: 0 2024-04-30T08:24:27.564 INFO:journalctl@ceph.mon.c.smithi189.stdout: read_amp_bytes_per_bit: 0 2024-04-30T08:24:27.564 INFO:journalctl@ceph.mon.c.smithi189.stdout: format_version: 5 2024-04-30T08:24:27.564 INFO:journalctl@ceph.mon.c.smithi189.stdout: enable_index_compression: 1 2024-04-30T08:24:27.564 INFO:journalctl@ceph.mon.c.smithi189.stdout: block_align: 0 2024-04-30T08:24:27.564 INFO:journalctl@ceph.mon.c.smithi189.stdout: max_auto_readahead_size: 262144 2024-04-30T08:24:27.564 INFO:journalctl@ceph.mon.c.smithi189.stdout: prepopulate_block_cache: 0 2024-04-30T08:24:27.564 INFO:journalctl@ceph.mon.c.smithi189.stdout: initial_auto_readahead_size: 8192 2024-04-30T08:24:27.564 INFO:journalctl@ceph.mon.c.smithi189.stdout: num_file_reads_for_auto_readahead: 2 2024-04-30T08:24:27.564 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.write_buffer_size: 33554432 2024-04-30T08:24:27.564 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.max_write_buffer_number: 2 2024-04-30T08:24:27.565 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.compression: NoCompression 2024-04-30T08:24:27.565 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.bottommost_compression: Disabled 2024-04-30T08:24:27.565 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.prefix_extractor: nullptr 2024-04-30T08:24:27.565 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-30T08:24:27.565 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.num_levels: 7 2024-04-30T08:24:27.565 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-30T08:24:27.565 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-30T08:24:27.565 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-30T08:24:27.565 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-30T08:24:27.565 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-30T08:24:27.565 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-30T08:24:27.565 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-30T08:24:27.566 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-30T08:24:27.566 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-30T08:24:27.566 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-30T08:24:27.566 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-30T08:24:27.566 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-30T08:24:27.566 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-30T08:24:27.566 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.compression_opts.level: 32767 2024-04-30T08:24:27.566 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.compression_opts.strategy: 0 2024-04-30T08:24:27.566 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-30T08:24:27.566 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-30T08:24:27.566 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-30T08:24:27.566 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-30T08:24:27.566 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.compression_opts.enabled: false 2024-04-30T08:24:27.566 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-30T08:24:27.567 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-30T08:24:27.567 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-30T08:24:27.567 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-30T08:24:27.567 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.target_file_size_base: 67108864 2024-04-30T08:24:27.567 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-30T08:24:27.567 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-30T08:24:27.567 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-30T08:24:27.567 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-30T08:24:27.567 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-30T08:24:27.567 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-30T08:24:27.567 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-30T08:24:27.567 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-30T08:24:27.567 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-30T08:24:27.567 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-30T08:24:27.568 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-30T08:24:27.568 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-30T08:24:27.568 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-30T08:24:27.568 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-30T08:24:27.568 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.arena_block_size: 1048576 2024-04-30T08:24:27.568 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-30T08:24:27.568 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-30T08:24:27.568 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.disable_auto_compactions: 0 2024-04-30T08:24:27.568 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-30T08:24:27.568 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-30T08:24:27.568 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-30T08:24:27.568 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-30T08:24:27.568 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-30T08:24:27.568 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-30T08:24:27.569 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-30T08:24:27.569 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-30T08:24:27.569 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-30T08:24:27.569 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-30T08:24:27.569 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-30T08:24:27.569 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.inplace_update_support: 0 2024-04-30T08:24:27.569 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-30T08:24:27.569 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-30T08:24:27.569 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-30T08:24:27.569 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-30T08:24:27.569 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.bloom_locality: 0 2024-04-30T08:24:27.569 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.max_successive_merges: 0 2024-04-30T08:24:27.569 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-30T08:24:27.569 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.paranoid_file_checks: 0 2024-04-30T08:24:27.570 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.force_consistency_checks: 1 2024-04-30T08:24:27.570 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.report_bg_io_stats: 0 2024-04-30T08:24:27.570 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.ttl: 2592000 2024-04-30T08:24:27.570 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-30T08:24:27.570 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-30T08:24:27.570 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-30T08:24:27.570 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.enable_blob_files: false 2024-04-30T08:24:27.570 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.min_blob_size: 0 2024-04-30T08:24:27.570 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.blob_file_size: 268435456 2024-04-30T08:24:27.570 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-30T08:24:27.570 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-30T08:24:27.570 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-30T08:24:27.570 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-30T08:24:27.571 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-30T08:24:27.571 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.blob_file_starting_level: 0 2024-04-30T08:24:27.571 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-30T08:24:27.571 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-30T08:24:27.571 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-30T08:24:27.571 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 73248893-3375-470f-bf45-6e75d1e4087b 2024-04-30T08:24:27.571 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714465467279476, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-30T08:24:27.571 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-30T08:24:27.571 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714465467279903, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1648, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714465467, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "73248893-3375-470f-bf45-6e75d1e4087b", "db_session_id": "QKWLU3JG4PB2KTDXXEO2", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-30T08:24:27.571 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714465467280005, "job": 1, "event": "recovery_finished"} 2024-04-30T08:24:27.571 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-30T08:24:27.571 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-30T08:24:27.571 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x556343697c00 2024-04-30T08:24:27.572 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: DB pointer 0x55634848a000 2024-04-30T08:24:27.572 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-30T08:24:27.572 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-30T08:24:27.572 INFO:journalctl@ceph.mon.c.smithi189.stdout: ** DB Stats ** 2024-04-30T08:24:27.572 INFO:journalctl@ceph.mon.c.smithi189.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-30T08:24:27.572 INFO:journalctl@ceph.mon.c.smithi189.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-30T08:24:27.572 INFO:journalctl@ceph.mon.c.smithi189.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-30T08:24:27.572 INFO:journalctl@ceph.mon.c.smithi189.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-30T08:24:27.572 INFO:journalctl@ceph.mon.c.smithi189.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-30T08:24:27.572 INFO:journalctl@ceph.mon.c.smithi189.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-30T08:24:27.572 INFO:journalctl@ceph.mon.c.smithi189.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-30T08:24:27.572 INFO:journalctl@ceph.mon.c.smithi189.stdout: 2024-04-30T08:24:27.572 INFO:journalctl@ceph.mon.c.smithi189.stdout: ** Compaction Stats [default] ** 2024-04-30T08:24:27.572 INFO:journalctl@ceph.mon.c.smithi189.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-30T08:24:27.573 INFO:journalctl@ceph.mon.c.smithi189.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-30T08:24:27.573 INFO:journalctl@ceph.mon.c.smithi189.stdout: L0 1/0 1.61 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-30T08:24:27.573 INFO:journalctl@ceph.mon.c.smithi189.stdout: Sum 1/0 1.61 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-30T08:24:27.573 INFO:journalctl@ceph.mon.c.smithi189.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-30T08:24:27.573 INFO:journalctl@ceph.mon.c.smithi189.stdout: 2024-04-30T08:24:27.573 INFO:journalctl@ceph.mon.c.smithi189.stdout: ** Compaction Stats [default] ** 2024-04-30T08:24:27.573 INFO:journalctl@ceph.mon.c.smithi189.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-30T08:24:27.573 INFO:journalctl@ceph.mon.c.smithi189.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-30T08:24:27.573 INFO:journalctl@ceph.mon.c.smithi189.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-30T08:24:27.573 INFO:journalctl@ceph.mon.c.smithi189.stdout: 2024-04-30T08:24:27.573 INFO:journalctl@ceph.mon.c.smithi189.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-30T08:24:27.573 INFO:journalctl@ceph.mon.c.smithi189.stdout: 2024-04-30T08:24:27.573 INFO:journalctl@ceph.mon.c.smithi189.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-30T08:24:27.573 INFO:journalctl@ceph.mon.c.smithi189.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-30T08:24:27.574 INFO:journalctl@ceph.mon.c.smithi189.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-30T08:24:27.574 INFO:journalctl@ceph.mon.c.smithi189.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-30T08:24:27.574 INFO:journalctl@ceph.mon.c.smithi189.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-30T08:24:27.574 INFO:journalctl@ceph.mon.c.smithi189.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-30T08:24:27.574 INFO:journalctl@ceph.mon.c.smithi189.stdout: Cumulative compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-30T08:24:27.574 INFO:journalctl@ceph.mon.c.smithi189.stdout: Interval compaction: 0.00 GB write, 0.62 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-30T08:24:27.574 INFO:journalctl@ceph.mon.c.smithi189.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-30T08:24:27.574 INFO:journalctl@ceph.mon.c.smithi189.stdout: Block cache BinnedLRUCache@0x5563436b31f0#2 capacity: 512.00 MB usage: 0.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-04-30T08:24:27.574 INFO:journalctl@ceph.mon.c.smithi189.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.12 KB,2.23517e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-30T08:24:27.574 INFO:journalctl@ceph.mon.c.smithi189.stdout: 2024-04-30T08:24:27.574 INFO:journalctl@ceph.mon.c.smithi189.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-30T08:24:27.574 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-04-30T08:24:27.575 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: using public_addr v2:172.21.15.189:0/0 -> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-30T08:24:27.575 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: starting mon.c rank -1 at public addrs [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] at bind addrs [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:24:27.575 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: mon.c@-1(???) e0 preinit fsid 98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:24:27.575 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: mon.c@-1(synchronizing).mds e1 new map 2024-04-30T08:24:27.575 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: mon.c@-1(synchronizing).mds e1 print_map 2024-04-30T08:24:27.575 INFO:journalctl@ceph.mon.c.smithi189.stdout: e1 2024-04-30T08:24:27.575 INFO:journalctl@ceph.mon.c.smithi189.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-30T08:24:27.575 INFO:journalctl@ceph.mon.c.smithi189.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-30T08:24:27.575 INFO:journalctl@ceph.mon.c.smithi189.stdout: legacy client fscid: -1 2024-04-30T08:24:27.575 INFO:journalctl@ceph.mon.c.smithi189.stdout: 2024-04-30T08:24:27.575 INFO:journalctl@ceph.mon.c.smithi189.stdout: No filesystems configured 2024-04-30T08:24:27.575 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-30T08:24:27.575 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-30T08:24:27.576 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-30T08:24:27.576 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-30T08:24:27.576 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-30T08:24:27.576 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-30T08:24:27.576 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-30T08:24:27.576 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-30T08:24:27.576 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-30T08:24:27.576 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-30T08:24:27.576 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:27.576 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:27.576 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:24:27.576 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: Added host smithi071 2024-04-30T08:24:27.576 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:27.576 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:27.577 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:27.577 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:27.577 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:27.577 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:24:27.577 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:27.577 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:27.577 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:27.577 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:27.577 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:27.577 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi071", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:24:27.577 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:27.577 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:24:27.577 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: Updating smithi071:/etc/ceph/ceph.conf 2024-04-30T08:24:27.577 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: Updating smithi071:/var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-30T08:24:27.578 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: Updating smithi071:/etc/ceph/ceph.client.admin.keyring 2024-04-30T08:24:27.578 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:27.578 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: Updating smithi071:/var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-30T08:24:27.578 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:27.578 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:27.578 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:27.578 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi189", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:24:27.578 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: Deploying cephadm binary to smithi189 2024-04-30T08:24:27.578 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:27.578 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:27.578 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:27.578 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: Added host smithi189 2024-04-30T08:24:27.578 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:24:27.578 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:27.579 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:27.579 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:27.579 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:27.579 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:27.579 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:24:27.579 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:27.579 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:27.579 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:27.579 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:27.579 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:27.579 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:24:27.579 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:27.579 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:24:27.580 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: Updating smithi189:/etc/ceph/ceph.conf 2024-04-30T08:24:27.580 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: Updating smithi189:/var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-30T08:24:27.580 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:27.580 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-04-30T08:24:27.580 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: Updating smithi189:/var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-30T08:24:27.580 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:27.580 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:27.580 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:27.580 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3402431792' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-30T08:24:27.580 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3402431792' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-30T08:24:27.580 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: osdmap e4: 0 total, 0 up, 0 in 2024-04-30T08:24:27.580 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:27.580 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:27.580 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi032:172.21.15.32=a;smithi071:172.21.15.71=b;smithi189:172.21.15.189=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:24:27.581 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: Saving service mon spec with placement smithi032:172.21.15.32=a;smithi071:172.21.15.71=b;smithi189:172.21.15.189=c;count:3 2024-04-30T08:24:27.581 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:27.581 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:24:27.581 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:27.581 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:24:27.581 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:27.581 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-30T08:24:27.581 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:27.581 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: Deploying daemon mon.c on smithi189 2024-04-30T08:24:27.581 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:27.581 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:27.581 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: from='client.? 172.21.15.189:0/548230897' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-30T08:24:27.581 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 ceph-mon[33038]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-30T08:24:27.581 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 bash[33001]: bf88bed5fe55d41eb96b923f63c7f14003672d82eafa509460d7a8f06986f14d 2024-04-30T08:24:27.581 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:27 smithi189 systemd[1]: Started Ceph mon.c for 98c0a670-06ca-11ef-bc93-c7b262605968. 2024-04-30T08:24:27.951 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-30T08:24:27.952 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-30T08:24:28.185 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.c/config 2024-04-30T08:24:32.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:32 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-30T08:24:32.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:32 smithi032 ceph-mon[28331]: mon.a calling monitor election 2024-04-30T08:24:32.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:32 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:24:32.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:32 smithi032 ceph-mon[28331]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:32.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:32 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:24:32.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:32 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:24:32.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:32 smithi032 ceph-mon[28331]: mon.c calling monitor election 2024-04-30T08:24:32.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:32 smithi032 ceph-mon[28331]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:32.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:32 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:24:32.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:32 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:24:32.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:32 smithi032 ceph-mon[28331]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:32.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:32 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:24:32.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:32 smithi032 ceph-mon[28331]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-30T08:24:32.694 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:32 smithi032 ceph-mon[28331]: monmap epoch 2 2024-04-30T08:24:32.694 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:32 smithi032 ceph-mon[28331]: fsid 98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:24:32.694 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:32 smithi032 ceph-mon[28331]: last_changed 2024-04-30T08:24:27.318454+0000 2024-04-30T08:24:32.694 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:32 smithi032 ceph-mon[28331]: created 2024-04-30T08:22:06.865404+0000 2024-04-30T08:24:32.694 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:32 smithi032 ceph-mon[28331]: min_mon_release 19 (squid) 2024-04-30T08:24:32.694 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:32 smithi032 ceph-mon[28331]: election_strategy: 1 2024-04-30T08:24:32.694 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:32 smithi032 ceph-mon[28331]: 0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-04-30T08:24:32.694 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:32 smithi032 ceph-mon[28331]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-04-30T08:24:32.694 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:32 smithi032 ceph-mon[28331]: fsmap 2024-04-30T08:24:32.694 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:32 smithi032 ceph-mon[28331]: osdmap e4: 0 total, 0 up, 0 in 2024-04-30T08:24:32.694 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:32 smithi032 ceph-mon[28331]: mgrmap e13: x(active, since 62s) 2024-04-30T08:24:32.694 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:32 smithi032 ceph-mon[28331]: overall HEALTH_OK 2024-04-30T08:24:32.695 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:32 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:32.695 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:32 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:32.756 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.755+0000 7f56188de700 1 -- 172.21.15.189:0/3269063757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55fc0035c0 msgr2=0x7f55fc005aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:32.757 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.755+0000 7f56188de700 1 --2- 172.21.15.189:0/3269063757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55fc0035c0 0x7f55fc005aa0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f561406b240 tx=0x7f560400feb0 comp rx=0 tx=0).stop 2024-04-30T08:24:32.757 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:32 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-30T08:24:32.757 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:32 smithi189 ceph-mon[33038]: mon.a calling monitor election 2024-04-30T08:24:32.757 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:32 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:24:32.757 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:32 smithi189 ceph-mon[33038]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:32.757 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:32 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:24:32.757 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:32 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:24:32.757 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:32 smithi189 ceph-mon[33038]: mon.c calling monitor election 2024-04-30T08:24:32.758 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:32 smithi189 ceph-mon[33038]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:32.758 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:32 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:24:32.758 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:32 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:24:32.758 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:32 smithi189 ceph-mon[33038]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:32.758 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:32 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:24:32.758 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:32 smithi189 ceph-mon[33038]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-30T08:24:32.758 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:32 smithi189 ceph-mon[33038]: monmap epoch 2 2024-04-30T08:24:32.758 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:32 smithi189 ceph-mon[33038]: fsid 98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:24:32.758 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:32 smithi189 ceph-mon[33038]: last_changed 2024-04-30T08:24:27.318454+0000 2024-04-30T08:24:32.758 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:32 smithi189 ceph-mon[33038]: created 2024-04-30T08:22:06.865404+0000 2024-04-30T08:24:32.758 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:32 smithi189 ceph-mon[33038]: min_mon_release 19 (squid) 2024-04-30T08:24:32.758 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:32 smithi189 ceph-mon[33038]: election_strategy: 1 2024-04-30T08:24:32.758 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:32 smithi189 ceph-mon[33038]: 0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-04-30T08:24:32.759 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:32 smithi189 ceph-mon[33038]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-04-30T08:24:32.759 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:32 smithi189 ceph-mon[33038]: fsmap 2024-04-30T08:24:32.759 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:32 smithi189 ceph-mon[33038]: osdmap e4: 0 total, 0 up, 0 in 2024-04-30T08:24:32.759 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:32 smithi189 ceph-mon[33038]: mgrmap e13: x(active, since 62s) 2024-04-30T08:24:32.759 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:32 smithi189 ceph-mon[33038]: overall HEALTH_OK 2024-04-30T08:24:32.759 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:32 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:32.759 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:32 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:32.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.755+0000 7f56188de700 1 -- 172.21.15.189:0/3269063757 shutdown_connections 2024-04-30T08:24:32.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.755+0000 7f56188de700 1 --2- 172.21.15.189:0/3269063757 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55fc0035c0 0x7f55fc005aa0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:32.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.755+0000 7f56188de700 1 -- 172.21.15.189:0/3269063757 >> 172.21.15.189:0/3269063757 conn(0x7f56140fe5c0 msgr2=0x7f5614100a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:24:32.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.755+0000 7f56188de700 1 -- 172.21.15.189:0/3269063757 shutdown_connections 2024-04-30T08:24:32.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.755+0000 7f56188de700 1 -- 172.21.15.189:0/3269063757 wait complete. 2024-04-30T08:24:32.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.756+0000 7f56188de700 1 Processor -- start 2024-04-30T08:24:32.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.756+0000 7f56188de700 1 -- start start 2024-04-30T08:24:32.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.757+0000 7f56188de700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55fc0035c0 0x7f561419f5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:24:32.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.757+0000 7f56188de700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f561419faf0 0x7f56141997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:24:32.761 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.757+0000 7f56188de700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f561410bb70 con 0x7f55fc0035c0 2024-04-30T08:24:32.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.757+0000 7f56188de700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f561410b9f0 con 0x7f561419faf0 2024-04-30T08:24:32.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.757+0000 7f5611d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f561419faf0 0x7f56141997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:24:32.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.757+0000 7f5611d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f561419faf0 0x7f56141997f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:45352/0 (socket says 172.21.15.189:45352) 2024-04-30T08:24:32.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.757+0000 7f5611d9b700 1 -- 172.21.15.189:0/3973830186 learned_addr learned my addr 172.21.15.189:0/3973830186 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-30T08:24:32.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.757+0000 7f5611d9b700 1 -- 172.21.15.189:0/3973830186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f561419faf0 msgr2=0x7f56141997f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-30T08:24:32.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.757+0000 7f5611d9b700 1 -- 172.21.15.189:0/3973830186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f561419faf0 msgr2=0x7f56141997f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-30T08:24:32.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.757+0000 7f5611d9b700 1 --2- 172.21.15.189:0/3973830186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f561419faf0 0x7f56141997f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-30T08:24:32.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.757+0000 7f5611d9b700 1 --2- 172.21.15.189:0/3973830186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f561419faf0 0x7f56141997f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-30T08:24:32.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.757+0000 7f561259c700 1 --2- 172.21.15.189:0/3973830186 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55fc0035c0 0x7f561419f5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:24:32.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.758+0000 7f561259c700 1 -- 172.21.15.189:0/3973830186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f561419faf0 msgr2=0x7f56141997f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-30T08:24:32.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.758+0000 7f561259c700 1 --2- 172.21.15.189:0/3973830186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f561419faf0 0x7f56141997f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:32.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.758+0000 7f561259c700 1 -- 172.21.15.189:0/3973830186 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56141a00e0 con 0x7f55fc0035c0 2024-04-30T08:24:32.762 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.759+0000 7f561259c700 1 --2- 172.21.15.189:0/3973830186 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55fc0035c0 0x7f561419f5b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5604010010 tx=0x7f5604005cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:24:32.763 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.759+0000 7f56037fe700 1 -- 172.21.15.189:0/3973830186 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f5604013960 con 0x7f55fc0035c0 2024-04-30T08:24:32.763 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.759+0000 7f56188de700 1 -- 172.21.15.189:0/3973830186 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5614199d90 con 0x7f55fc0035c0 2024-04-30T08:24:32.763 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.759+0000 7f56037fe700 1 -- 172.21.15.189:0/3973830186 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56040044e0 con 0x7f55fc0035c0 2024-04-30T08:24:32.763 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.759+0000 7f56188de700 1 -- 172.21.15.189:0/3973830186 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f561419a320 con 0x7f55fc0035c0 2024-04-30T08:24:32.763 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.759+0000 7f56037fe700 1 -- 172.21.15.189:0/3973830186 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f5604003ce0 con 0x7f55fc0035c0 2024-04-30T08:24:32.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.761+0000 7f56188de700 1 -- 172.21.15.189:0/3973830186 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f561406ba40 con 0x7f55fc0035c0 2024-04-30T08:24:32.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.761+0000 7f56037fe700 1 -- 172.21.15.189:0/3973830186 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f5604023070 con 0x7f55fc0035c0 2024-04-30T08:24:32.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.762+0000 7f56037fe700 1 --2- 172.21.15.189:0/3973830186 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f55f403d550 0x7f55f403fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:24:32.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.762+0000 7f56037fe700 1 -- 172.21.15.189:0/3973830186 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f5604020050 con 0x7f55fc0035c0 2024-04-30T08:24:32.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.762+0000 7f5611d9b700 1 --2- 172.21.15.189:0/3973830186 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f55f403d550 0x7f55f403fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:24:32.764 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.763+0000 7f5611d9b700 1 --2- 172.21.15.189:0/3973830186 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f55f403d550 0x7f55f403fa60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5614109940 tx=0x7f5608008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:24:32.779 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:32.778+0000 7f56037fe700 1 -- 172.21.15.189:0/3973830186 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5604028020 con 0x7f55fc0035c0 2024-04-30T08:24:33.068 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:33.067+0000 7f56188de700 1 -- 172.21.15.189:0/3973830186 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f561419a5d0 con 0x7f55fc0035c0 2024-04-30T08:24:33.069 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:33.068+0000 7f56037fe700 1 -- 172.21.15.189:0/3973830186 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f5604024400 con 0x7f55fc0035c0 2024-04-30T08:24:33.069 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-30T08:24:33.069 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":2,"fsid":"98c0a670-06ca-11ef-bc93-c7b262605968","modified":"2024-04-30T08:24:27.318454Z","created":"2024-04-30T08:22:06.865404Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-30T08:24:33.071 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:33.070+0000 7f56188de700 1 -- 172.21.15.189:0/3973830186 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f55f403d550 msgr2=0x7f55f403fa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:33.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:33.070+0000 7f56188de700 1 --2- 172.21.15.189:0/3973830186 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f55f403d550 0x7f55f403fa60 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f5614109940 tx=0x7f5608008040 comp rx=0 tx=0).stop 2024-04-30T08:24:33.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:33.070+0000 7f56188de700 1 -- 172.21.15.189:0/3973830186 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55fc0035c0 msgr2=0x7f561419f5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:33.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:33.070+0000 7f56188de700 1 --2- 172.21.15.189:0/3973830186 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55fc0035c0 0x7f561419f5b0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f5604010010 tx=0x7f5604005cc0 comp rx=0 tx=0).stop 2024-04-30T08:24:33.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:33.070+0000 7f56188de700 1 -- 172.21.15.189:0/3973830186 shutdown_connections 2024-04-30T08:24:33.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:33.070+0000 7f56188de700 1 --2- 172.21.15.189:0/3973830186 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f55f403d550 0x7f55f403fa60 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:33.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:33.070+0000 7f56188de700 1 --2- 172.21.15.189:0/3973830186 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f55fc0035c0 0x7f561419f5b0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:33.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:33.070+0000 7f56188de700 1 --2- 172.21.15.189:0/3973830186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f561419faf0 0x7f56141997f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:33.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:33.070+0000 7f56188de700 1 -- 172.21.15.189:0/3973830186 >> 172.21.15.189:0/3973830186 conn(0x7f56140fe5c0 msgr2=0x7f56141015a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:24:33.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:33.071+0000 7f56188de700 1 -- 172.21.15.189:0/3973830186 shutdown_connections 2024-04-30T08:24:33.072 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:33.071+0000 7f56188de700 1 -- 172.21.15.189:0/3973830186 wait complete. 2024-04-30T08:24:33.073 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 2 2024-04-30T08:24:33.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:33 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:33.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:33 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-30T08:24:33.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:33 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:33.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:33 smithi032 ceph-mon[28331]: Deploying daemon mon.b on smithi071 2024-04-30T08:24:33.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:33 smithi032 ceph-mon[28331]: from='client.? 172.21.15.189:0/3973830186' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-30T08:24:33.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:33 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:24:33.712 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:33 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:33.712 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:33 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-30T08:24:33.712 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:33 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:33.712 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:33 smithi189 ceph-mon[33038]: Deploying daemon mon.b on smithi071 2024-04-30T08:24:33.712 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:33 smithi189 ceph-mon[33038]: from='client.? 172.21.15.189:0/3973830186' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-30T08:24:33.712 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:33 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:24:34.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:34 smithi032 ceph-mon[28331]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:34.691 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:24:34 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:24:34.320+0000 7fbb8a000700 -1 mgr.server handle_report got status from non-daemon mon.c 2024-04-30T08:24:34.737 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-30T08:24:34.738 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-30T08:24:34.763 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:34 smithi189 ceph-mon[33038]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:34.974 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.c/config 2024-04-30T08:24:36.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 podman[33366]: 2024-04-30 08:24:36.1012619 +0000 UTC m=+0.018144890 image pull 16056ace0edc7f73516cf28297f1db04dda872801fb5b17b9157b0d1ccf38f9b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de 2024-04-30T08:24:36.474 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.473+0000 7f136959a700 1 -- 172.21.15.189:0/3919158984 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f135c005540 con 0x7f136c108ab0 2024-04-30T08:24:36.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.474+0000 7f1370971700 1 -- 172.21.15.189:0/3919158984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f136c108ab0 msgr2=0x7f136c108ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:36.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.474+0000 7f1370971700 1 --2- 172.21.15.189:0/3919158984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f136c108ab0 0x7f136c108ec0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f135c009910 tx=0x7f135c009c20 comp rx=0 tx=0).stop 2024-04-30T08:24:36.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.474+0000 7f1370971700 1 -- 172.21.15.189:0/3919158984 shutdown_connections 2024-04-30T08:24:36.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.474+0000 7f1370971700 1 --2- 172.21.15.189:0/3919158984 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f136c108ab0 0x7f136c108ec0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:36.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.474+0000 7f1370971700 1 -- 172.21.15.189:0/3919158984 >> 172.21.15.189:0/3919158984 conn(0x7f136c0fe5c0 msgr2=0x7f136c100a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:24:36.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.474+0000 7f1370971700 1 -- 172.21.15.189:0/3919158984 shutdown_connections 2024-04-30T08:24:36.475 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.474+0000 7f1370971700 1 -- 172.21.15.189:0/3919158984 wait complete. 2024-04-30T08:24:36.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.475+0000 7f1370971700 1 Processor -- start 2024-04-30T08:24:36.476 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.475+0000 7f1370971700 1 -- start start 2024-04-30T08:24:36.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.475+0000 7f1370971700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f136c108ab0 0x7f136c1a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:24:36.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.475+0000 7f1370971700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f136c1a1f00 0x7f136c19bb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:24:36.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.475+0000 7f1370971700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f136c06c0c0 con 0x7f136c1a1f00 2024-04-30T08:24:36.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.475+0000 7f1370971700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f136c06bf40 con 0x7f136c108ab0 2024-04-30T08:24:36.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.475+0000 7f136a59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f136c108ab0 0x7f136c1a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:24:36.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.475+0000 7f136a59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f136c108ab0 0x7f136c1a19c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39672/0 (socket says 172.21.15.189:39672) 2024-04-30T08:24:36.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.475+0000 7f136a59c700 1 -- 172.21.15.189:0/607363900 learned_addr learned my addr 172.21.15.189:0/607363900 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-30T08:24:36.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.475+0000 7f136a59c700 1 -- 172.21.15.189:0/607363900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f136c108ab0 msgr2=0x7f136c1a19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-30T08:24:36.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.475+0000 7f136a59c700 1 -- 172.21.15.189:0/607363900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f136c108ab0 msgr2=0x7f136c1a19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-30T08:24:36.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.475+0000 7f136a59c700 1 --2- 172.21.15.189:0/607363900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f136c108ab0 0x7f136c1a19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-30T08:24:36.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.475+0000 7f136a59c700 1 --2- 172.21.15.189:0/607363900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f136c108ab0 0x7f136c1a19c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-30T08:24:36.477 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.476+0000 7f1369d9b700 1 --2- 172.21.15.189:0/607363900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f136c1a1f00 0x7f136c19bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:24:36.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.476+0000 7f1369d9b700 1 -- 172.21.15.189:0/607363900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f136c108ab0 msgr2=0x7f136c1a19c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-30T08:24:36.478 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.476+0000 7f1369d9b700 1 --2- 172.21.15.189:0/607363900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f136c108ab0 0x7f136c1a19c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:36.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.476+0000 7f1369d9b700 1 -- 172.21.15.189:0/607363900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f136c1a24a0 con 0x7f136c1a1f00 2024-04-30T08:24:36.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.477+0000 7f1369d9b700 1 --2- 172.21.15.189:0/607363900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f136c1a1f00 0x7f136c19bb50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f136000b4d0 tx=0x7f136000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:24:36.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.477+0000 7f135b7fe700 1 -- 172.21.15.189:0/607363900 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f1360011630 con 0x7f136c1a1f00 2024-04-30T08:24:36.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.477+0000 7f1370971700 1 -- 172.21.15.189:0/607363900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f136c19c150 con 0x7f136c1a1f00 2024-04-30T08:24:36.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.477+0000 7f135b7fe700 1 -- 172.21.15.189:0/607363900 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1360011c80 con 0x7f136c1a1f00 2024-04-30T08:24:36.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.477+0000 7f135b7fe700 1 -- 172.21.15.189:0/607363900 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f1360010da0 con 0x7f136c1a1f00 2024-04-30T08:24:36.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.477+0000 7f1370971700 1 -- 172.21.15.189:0/607363900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f136c19c6e0 con 0x7f136c1a1f00 2024-04-30T08:24:36.479 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.478+0000 7f1370971700 1 -- 172.21.15.189:0/607363900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f136c19c9f0 con 0x7f136c1a1f00 2024-04-30T08:24:36.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.479+0000 7f135b7fe700 1 -- 172.21.15.189:0/607363900 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f1360018020 con 0x7f136c1a1f00 2024-04-30T08:24:36.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.479+0000 7f135b7fe700 1 --2- 172.21.15.189:0/607363900 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f135403d3b0 0x7f135403f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:24:36.480 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.479+0000 7f135b7fe700 1 -- 172.21.15.189:0/607363900 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f13600514a0 con 0x7f136c1a1f00 2024-04-30T08:24:36.481 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.480+0000 7f136a59c700 1 --2- 172.21.15.189:0/607363900 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f135403d3b0 0x7f135403f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:24:36.482 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.481+0000 7f136a59c700 1 --2- 172.21.15.189:0/607363900 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f135403d3b0 0x7f135403f8c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f135c009750 tx=0x7f135c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:24:36.498 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.497+0000 7f135b7fe700 1 -- 172.21.15.189:0/607363900 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f136001f020 con 0x7f136c1a1f00 2024-04-30T08:24:36.500 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.500+0000 7f135b7fe700 1 -- 172.21.15.189:0/607363900 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1360022580 con 0x7f136c1a1f00 2024-04-30T08:24:36.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 podman[33366]: 2024-04-30 08:24:36.408080551 +0000 UTC m=+0.324963531 container init 7615c446928d9168f2c655b7af67defffaa49ddbc2afe0a94052bb819dde3244 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mon-b, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-30T08:24:36.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 podman[33366]: 2024-04-30 08:24:36.410931144 +0000 UTC m=+0.327814124 container start 7615c446928d9168f2c655b7af67defffaa49ddbc2afe0a94052bb819dde3244 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mon-b, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-30T08:24:36.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: set uid:gid to 167:167 (ceph:ceph) 2024-04-30T08:24:36.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: ceph version 19.0.0-3348-g8feaf7d9 (8feaf7d9864e980041b042fd5d964ac18adfd20f) squid (dev), process ceph-mon, pid 2 2024-04-30T08:24:36.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: pidfile_write: ignore empty --pid-file 2024-04-30T08:24:36.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: load: jerasure load: lrc 2024-04-30T08:24:36.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: RocksDB version: 7.9.2 2024-04-30T08:24:36.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Git sha 0 2024-04-30T08:24:36.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Compile date 2024-04-29 22:33:18 2024-04-30T08:24:36.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: DB SUMMARY 2024-04-30T08:24:36.756 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: DB Session ID: 67AQ66G83J1M7QYCLE24 2024-04-30T08:24:36.756 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: CURRENT file: CURRENT 2024-04-30T08:24:36.756 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: IDENTITY file: IDENTITY 2024-04-30T08:24:36.756 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-04-30T08:24:36.756 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-04-30T08:24:36.756 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-04-30T08:24:36.756 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.error_if_exists: 0 2024-04-30T08:24:36.756 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.create_if_missing: 0 2024-04-30T08:24:36.756 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.paranoid_checks: 1 2024-04-30T08:24:36.756 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-30T08:24:36.756 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-30T08:24:36.756 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-30T08:24:36.756 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.env: 0x562488fa4500 2024-04-30T08:24:36.757 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.fs: PosixFileSystem 2024-04-30T08:24:36.757 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.info_log: 0x56248b097fe0 2024-04-30T08:24:36.757 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.max_file_opening_threads: 16 2024-04-30T08:24:36.757 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.statistics: (nil) 2024-04-30T08:24:36.757 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.use_fsync: 0 2024-04-30T08:24:36.757 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.max_log_file_size: 0 2024-04-30T08:24:36.757 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-30T08:24:36.757 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-30T08:24:36.757 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.keep_log_file_num: 1000 2024-04-30T08:24:36.757 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.recycle_log_file_num: 0 2024-04-30T08:24:36.757 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.allow_fallocate: 1 2024-04-30T08:24:36.757 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.allow_mmap_reads: 0 2024-04-30T08:24:36.757 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.allow_mmap_writes: 0 2024-04-30T08:24:36.757 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.use_direct_reads: 0 2024-04-30T08:24:36.758 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-30T08:24:36.758 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.create_missing_column_families: 0 2024-04-30T08:24:36.758 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.db_log_dir: 2024-04-30T08:24:36.758 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.wal_dir: 2024-04-30T08:24:36.758 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-30T08:24:36.758 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-30T08:24:36.758 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-30T08:24:36.758 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-30T08:24:36.758 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-30T08:24:36.758 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-30T08:24:36.758 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.advise_random_on_open: 1 2024-04-30T08:24:36.758 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.db_write_buffer_size: 0 2024-04-30T08:24:36.758 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.write_buffer_manager: 0x56248b0688c0 2024-04-30T08:24:36.758 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-30T08:24:36.759 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-30T08:24:36.759 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-30T08:24:36.759 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.rate_limiter: (nil) 2024-04-30T08:24:36.759 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-30T08:24:36.759 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.wal_recovery_mode: 2 2024-04-30T08:24:36.759 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.enable_thread_tracking: 0 2024-04-30T08:24:36.759 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.enable_pipelined_write: 0 2024-04-30T08:24:36.759 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.unordered_write: 0 2024-04-30T08:24:36.759 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-30T08:24:36.759 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-30T08:24:36.759 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-30T08:24:36.759 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-30T08:24:36.759 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.row_cache: None 2024-04-30T08:24:36.759 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.wal_filter: None 2024-04-30T08:24:36.760 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-30T08:24:36.760 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.allow_ingest_behind: 0 2024-04-30T08:24:36.760 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.two_write_queues: 0 2024-04-30T08:24:36.760 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.manual_wal_flush: 0 2024-04-30T08:24:36.760 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.wal_compression: 0 2024-04-30T08:24:36.760 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.atomic_flush: 0 2024-04-30T08:24:36.760 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-30T08:24:36.760 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-30T08:24:36.760 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-30T08:24:36.760 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.log_readahead_size: 0 2024-04-30T08:24:36.760 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-30T08:24:36.760 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.best_efforts_recovery: 0 2024-04-30T08:24:36.760 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-30T08:24:36.760 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-30T08:24:36.761 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.allow_data_in_errors: 0 2024-04-30T08:24:36.761 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.db_host_id: __hostname__ 2024-04-30T08:24:36.761 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-30T08:24:36.761 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.max_background_jobs: 2 2024-04-30T08:24:36.761 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.max_background_compactions: -1 2024-04-30T08:24:36.761 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.max_subcompactions: 1 2024-04-30T08:24:36.761 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-30T08:24:36.761 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-30T08:24:36.761 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-30T08:24:36.761 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.max_total_wal_size: 0 2024-04-30T08:24:36.761 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-30T08:24:36.761 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-30T08:24:36.761 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-30T08:24:36.761 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-30T08:24:36.762 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.max_open_files: -1 2024-04-30T08:24:36.762 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.bytes_per_sync: 0 2024-04-30T08:24:36.762 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-30T08:24:36.762 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-30T08:24:36.762 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.compaction_readahead_size: 0 2024-04-30T08:24:36.762 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.max_background_flushes: -1 2024-04-30T08:24:36.762 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Compression algorithms supported: 2024-04-30T08:24:36.762 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-30T08:24:36.762 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: kZSTD supported: 0 2024-04-30T08:24:36.762 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: kXpressCompression supported: 0 2024-04-30T08:24:36.762 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: kLZ4HCCompression supported: 1 2024-04-30T08:24:36.762 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: kZlibCompression supported: 1 2024-04-30T08:24:36.762 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: kSnappyCompression supported: 1 2024-04-30T08:24:36.762 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: kLZ4Compression supported: 1 2024-04-30T08:24:36.763 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: kBZip2Compression supported: 0 2024-04-30T08:24:36.763 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-30T08:24:36.763 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-30T08:24:36.763 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-04-30T08:24:36.763 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-30T08:24:36.763 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-30T08:24:36.763 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.merge_operator: 2024-04-30T08:24:36.763 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.compaction_filter: None 2024-04-30T08:24:36.763 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.compaction_filter_factory: None 2024-04-30T08:24:36.763 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.sst_partitioner_factory: None 2024-04-30T08:24:36.763 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-30T08:24:36.763 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-30T08:24:36.763 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56248b19e1c0) 2024-04-30T08:24:36.763 INFO:journalctl@ceph.mon.b.smithi071.stdout: cache_index_and_filter_blocks: 1 2024-04-30T08:24:36.763 INFO:journalctl@ceph.mon.b.smithi071.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-30T08:24:36.764 INFO:journalctl@ceph.mon.b.smithi071.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-30T08:24:36.764 INFO:journalctl@ceph.mon.b.smithi071.stdout: pin_top_level_index_and_filter: 1 2024-04-30T08:24:36.764 INFO:journalctl@ceph.mon.b.smithi071.stdout: index_type: 0 2024-04-30T08:24:36.764 INFO:journalctl@ceph.mon.b.smithi071.stdout: data_block_index_type: 0 2024-04-30T08:24:36.764 INFO:journalctl@ceph.mon.b.smithi071.stdout: index_shortening: 1 2024-04-30T08:24:36.764 INFO:journalctl@ceph.mon.b.smithi071.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-30T08:24:36.764 INFO:journalctl@ceph.mon.b.smithi071.stdout: checksum: 4 2024-04-30T08:24:36.764 INFO:journalctl@ceph.mon.b.smithi071.stdout: no_block_cache: 0 2024-04-30T08:24:36.764 INFO:journalctl@ceph.mon.b.smithi071.stdout: block_cache: 0x56248b09b1f0 2024-04-30T08:24:36.764 INFO:journalctl@ceph.mon.b.smithi071.stdout: block_cache_name: BinnedLRUCache 2024-04-30T08:24:36.764 INFO:journalctl@ceph.mon.b.smithi071.stdout: block_cache_options: 2024-04-30T08:24:36.764 INFO:journalctl@ceph.mon.b.smithi071.stdout: capacity : 536870912 2024-04-30T08:24:36.764 INFO:journalctl@ceph.mon.b.smithi071.stdout: num_shard_bits : 4 2024-04-30T08:24:36.764 INFO:journalctl@ceph.mon.b.smithi071.stdout: strict_capacity_limit : 0 2024-04-30T08:24:36.765 INFO:journalctl@ceph.mon.b.smithi071.stdout: high_pri_pool_ratio: 0.000 2024-04-30T08:24:36.765 INFO:journalctl@ceph.mon.b.smithi071.stdout: block_cache_compressed: (nil) 2024-04-30T08:24:36.765 INFO:journalctl@ceph.mon.b.smithi071.stdout: persistent_cache: (nil) 2024-04-30T08:24:36.765 INFO:journalctl@ceph.mon.b.smithi071.stdout: block_size: 4096 2024-04-30T08:24:36.765 INFO:journalctl@ceph.mon.b.smithi071.stdout: block_size_deviation: 10 2024-04-30T08:24:36.765 INFO:journalctl@ceph.mon.b.smithi071.stdout: block_restart_interval: 16 2024-04-30T08:24:36.765 INFO:journalctl@ceph.mon.b.smithi071.stdout: index_block_restart_interval: 1 2024-04-30T08:24:36.765 INFO:journalctl@ceph.mon.b.smithi071.stdout: metadata_block_size: 4096 2024-04-30T08:24:36.765 INFO:journalctl@ceph.mon.b.smithi071.stdout: partition_filters: 0 2024-04-30T08:24:36.765 INFO:journalctl@ceph.mon.b.smithi071.stdout: use_delta_encoding: 1 2024-04-30T08:24:36.765 INFO:journalctl@ceph.mon.b.smithi071.stdout: filter_policy: bloomfilter 2024-04-30T08:24:36.765 INFO:journalctl@ceph.mon.b.smithi071.stdout: whole_key_filtering: 1 2024-04-30T08:24:36.765 INFO:journalctl@ceph.mon.b.smithi071.stdout: verify_compression: 0 2024-04-30T08:24:36.765 INFO:journalctl@ceph.mon.b.smithi071.stdout: read_amp_bytes_per_bit: 0 2024-04-30T08:24:36.765 INFO:journalctl@ceph.mon.b.smithi071.stdout: format_version: 5 2024-04-30T08:24:36.766 INFO:journalctl@ceph.mon.b.smithi071.stdout: enable_index_compression: 1 2024-04-30T08:24:36.766 INFO:journalctl@ceph.mon.b.smithi071.stdout: block_align: 0 2024-04-30T08:24:36.766 INFO:journalctl@ceph.mon.b.smithi071.stdout: max_auto_readahead_size: 262144 2024-04-30T08:24:36.766 INFO:journalctl@ceph.mon.b.smithi071.stdout: prepopulate_block_cache: 0 2024-04-30T08:24:36.766 INFO:journalctl@ceph.mon.b.smithi071.stdout: initial_auto_readahead_size: 8192 2024-04-30T08:24:36.768 INFO:journalctl@ceph.mon.b.smithi071.stdout: num_file_reads_for_auto_readahead: 2 2024-04-30T08:24:36.768 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.write_buffer_size: 33554432 2024-04-30T08:24:36.768 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.max_write_buffer_number: 2 2024-04-30T08:24:36.768 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.compression: NoCompression 2024-04-30T08:24:36.768 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.bottommost_compression: Disabled 2024-04-30T08:24:36.768 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.prefix_extractor: nullptr 2024-04-30T08:24:36.768 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-30T08:24:36.768 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.num_levels: 7 2024-04-30T08:24:36.768 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-30T08:24:36.768 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-30T08:24:36.768 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-30T08:24:36.768 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-30T08:24:36.768 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-30T08:24:36.769 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-30T08:24:36.769 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-30T08:24:36.769 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-30T08:24:36.769 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-30T08:24:36.769 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-30T08:24:36.769 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-30T08:24:36.769 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-30T08:24:36.769 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-30T08:24:36.769 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.compression_opts.level: 32767 2024-04-30T08:24:36.769 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.compression_opts.strategy: 0 2024-04-30T08:24:36.769 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-30T08:24:36.769 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-30T08:24:36.769 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-30T08:24:36.769 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-30T08:24:36.769 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.compression_opts.enabled: false 2024-04-30T08:24:36.770 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-30T08:24:36.770 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-30T08:24:36.770 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-30T08:24:36.770 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-30T08:24:36.770 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.target_file_size_base: 67108864 2024-04-30T08:24:36.770 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-30T08:24:36.770 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-30T08:24:36.770 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-30T08:24:36.770 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-30T08:24:36.770 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-30T08:24:36.770 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-30T08:24:36.770 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-30T08:24:36.770 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-30T08:24:36.770 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-30T08:24:36.771 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-30T08:24:36.771 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-30T08:24:36.771 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-30T08:24:36.771 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-30T08:24:36.771 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-30T08:24:36.771 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.arena_block_size: 1048576 2024-04-30T08:24:36.771 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-30T08:24:36.771 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-30T08:24:36.771 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.disable_auto_compactions: 0 2024-04-30T08:24:36.771 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-30T08:24:36.771 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-30T08:24:36.771 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-30T08:24:36.771 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-30T08:24:36.771 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-30T08:24:36.772 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-30T08:24:36.772 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-30T08:24:36.772 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-30T08:24:36.772 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-30T08:24:36.772 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-30T08:24:36.772 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-30T08:24:36.772 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.inplace_update_support: 0 2024-04-30T08:24:36.772 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-30T08:24:36.772 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-30T08:24:36.772 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-30T08:24:36.772 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-30T08:24:36.772 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.bloom_locality: 0 2024-04-30T08:24:36.772 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.max_successive_merges: 0 2024-04-30T08:24:36.772 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-30T08:24:36.773 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.paranoid_file_checks: 0 2024-04-30T08:24:36.773 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.force_consistency_checks: 1 2024-04-30T08:24:36.773 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.report_bg_io_stats: 0 2024-04-30T08:24:36.773 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.ttl: 2592000 2024-04-30T08:24:36.773 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-30T08:24:36.773 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-30T08:24:36.773 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-30T08:24:36.773 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.enable_blob_files: false 2024-04-30T08:24:36.773 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.min_blob_size: 0 2024-04-30T08:24:36.773 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.blob_file_size: 268435456 2024-04-30T08:24:36.773 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-30T08:24:36.773 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-30T08:24:36.773 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-30T08:24:36.773 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-30T08:24:36.774 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-30T08:24:36.774 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.blob_file_starting_level: 0 2024-04-30T08:24:36.774 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-30T08:24:36.774 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-04-30T08:24:36.774 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-04-30T08:24:36.774 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 4e5f5440-f428-43ee-9337-4e03ef59b153 2024-04-30T08:24:36.774 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714465476462008, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-04-30T08:24:36.774 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-04-30T08:24:36.774 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714465476462480, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1773, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1714465476, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "4e5f5440-f428-43ee-9337-4e03ef59b153", "db_session_id": "67AQ66G83J1M7QYCLE24", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-04-30T08:24:36.774 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: EVENT_LOG_v1 {"time_micros": 1714465476462575, "job": 1, "event": "recovery_finished"} 2024-04-30T08:24:36.774 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-04-30T08:24:36.774 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-30T08:24:36.774 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x56248b07fc00 2024-04-30T08:24:36.774 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: DB pointer 0x56248fe72000 2024-04-30T08:24:36.775 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-30T08:24:36.775 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-30T08:24:36.775 INFO:journalctl@ceph.mon.b.smithi071.stdout: ** DB Stats ** 2024-04-30T08:24:36.775 INFO:journalctl@ceph.mon.b.smithi071.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-30T08:24:36.775 INFO:journalctl@ceph.mon.b.smithi071.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-30T08:24:36.775 INFO:journalctl@ceph.mon.b.smithi071.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-30T08:24:36.775 INFO:journalctl@ceph.mon.b.smithi071.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-30T08:24:36.775 INFO:journalctl@ceph.mon.b.smithi071.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-30T08:24:36.775 INFO:journalctl@ceph.mon.b.smithi071.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-30T08:24:36.775 INFO:journalctl@ceph.mon.b.smithi071.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-30T08:24:36.775 INFO:journalctl@ceph.mon.b.smithi071.stdout: 2024-04-30T08:24:36.775 INFO:journalctl@ceph.mon.b.smithi071.stdout: ** Compaction Stats [default] ** 2024-04-30T08:24:36.775 INFO:journalctl@ceph.mon.b.smithi071.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-30T08:24:36.775 INFO:journalctl@ceph.mon.b.smithi071.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-30T08:24:36.776 INFO:journalctl@ceph.mon.b.smithi071.stdout: L0 1/0 1.73 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-30T08:24:36.776 INFO:journalctl@ceph.mon.b.smithi071.stdout: Sum 1/0 1.73 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-30T08:24:36.776 INFO:journalctl@ceph.mon.b.smithi071.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-30T08:24:36.776 INFO:journalctl@ceph.mon.b.smithi071.stdout: 2024-04-30T08:24:36.776 INFO:journalctl@ceph.mon.b.smithi071.stdout: ** Compaction Stats [default] ** 2024-04-30T08:24:36.776 INFO:journalctl@ceph.mon.b.smithi071.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-30T08:24:36.776 INFO:journalctl@ceph.mon.b.smithi071.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-30T08:24:36.776 INFO:journalctl@ceph.mon.b.smithi071.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-04-30T08:24:36.776 INFO:journalctl@ceph.mon.b.smithi071.stdout: 2024-04-30T08:24:36.776 INFO:journalctl@ceph.mon.b.smithi071.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-30T08:24:36.776 INFO:journalctl@ceph.mon.b.smithi071.stdout: 2024-04-30T08:24:36.776 INFO:journalctl@ceph.mon.b.smithi071.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-30T08:24:36.776 INFO:journalctl@ceph.mon.b.smithi071.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-30T08:24:36.776 INFO:journalctl@ceph.mon.b.smithi071.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-30T08:24:36.776 INFO:journalctl@ceph.mon.b.smithi071.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-30T08:24:36.777 INFO:journalctl@ceph.mon.b.smithi071.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-30T08:24:36.777 INFO:journalctl@ceph.mon.b.smithi071.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-30T08:24:36.777 INFO:journalctl@ceph.mon.b.smithi071.stdout: Cumulative compaction: 0.00 GB write, 0.13 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-30T08:24:36.777 INFO:journalctl@ceph.mon.b.smithi071.stdout: Interval compaction: 0.00 GB write, 0.13 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-30T08:24:36.777 INFO:journalctl@ceph.mon.b.smithi071.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-30T08:24:36.777 INFO:journalctl@ceph.mon.b.smithi071.stdout: Block cache BinnedLRUCache@0x56248b09b1f0#2 capacity: 512.00 MB usage: 0.23 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-04-30T08:24:36.777 INFO:journalctl@ceph.mon.b.smithi071.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.12 KB,2.23517e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-04-30T08:24:36.777 INFO:journalctl@ceph.mon.b.smithi071.stdout: 2024-04-30T08:24:36.777 INFO:journalctl@ceph.mon.b.smithi071.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-30T08:24:36.777 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-04-30T08:24:36.777 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: using public_addr v2:172.21.15.71:0/0 -> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] 2024-04-30T08:24:36.777 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: starting mon.b rank -1 at public addrs [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] at bind addrs [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:24:36.777 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: mon.b@-1(???) e0 preinit fsid 98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:24:36.777 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 bash[33366]: 7615c446928d9168f2c655b7af67defffaa49ddbc2afe0a94052bb819dde3244 2024-04-30T08:24:36.778 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 systemd[1]: Started Ceph mon.b for 98c0a670-06ca-11ef-bc93-c7b262605968. 2024-04-30T08:24:36.778 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: mon.b@-1(synchronizing).mds e1 new map 2024-04-30T08:24:36.778 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: mon.b@-1(synchronizing).mds e1 print_map 2024-04-30T08:24:36.778 INFO:journalctl@ceph.mon.b.smithi071.stdout: e1 2024-04-30T08:24:36.778 INFO:journalctl@ceph.mon.b.smithi071.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-30T08:24:36.778 INFO:journalctl@ceph.mon.b.smithi071.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-30T08:24:36.778 INFO:journalctl@ceph.mon.b.smithi071.stdout: legacy client fscid: -1 2024-04-30T08:24:36.778 INFO:journalctl@ceph.mon.b.smithi071.stdout: 2024-04-30T08:24:36.778 INFO:journalctl@ceph.mon.b.smithi071.stdout: No filesystems configured 2024-04-30T08:24:36.778 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-04-30T08:24:36.778 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-04-30T08:24:36.778 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-04-30T08:24:36.778 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-04-30T08:24:36.779 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-04-30T08:24:36.779 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-04-30T08:24:36.779 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-30T08:24:36.779 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-30T08:24:36.779 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-30T08:24:36.779 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-04-30T08:24:36.779 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:36.779 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:36.779 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:24:36.779 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: Added host smithi071 2024-04-30T08:24:36.779 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:36.779 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:36.779 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:36.779 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:36.779 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:36.780 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:24:36.780 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:36.780 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:36.780 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:36.780 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:36.780 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:36.780 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi071", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:24:36.780 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:36.780 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:24:36.780 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: Updating smithi071:/etc/ceph/ceph.conf 2024-04-30T08:24:36.780 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: Updating smithi071:/var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-30T08:24:36.780 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: Updating smithi071:/etc/ceph/ceph.client.admin.keyring 2024-04-30T08:24:36.780 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:36.780 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: Updating smithi071:/var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-30T08:24:36.781 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:36.781 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:36.781 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:36.781 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi189", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:24:36.781 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: Deploying cephadm binary to smithi189 2024-04-30T08:24:36.781 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:36.781 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:36.781 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:36.781 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: Added host smithi189 2024-04-30T08:24:36.781 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:24:36.781 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:36.781 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:36.781 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:36.781 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:36.781 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:36.782 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:24:36.782 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:36.782 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:36.782 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:36.782 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:36.782 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:36.782 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:24:36.782 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:36.782 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:24:36.782 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: Updating smithi189:/etc/ceph/ceph.conf 2024-04-30T08:24:36.782 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: Updating smithi189:/var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-30T08:24:36.782 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:36.782 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-04-30T08:24:36.783 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: Updating smithi189:/var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-30T08:24:36.783 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:36.783 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:36.783 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:36.783 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3402431792' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-30T08:24:36.783 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3402431792' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-30T08:24:36.783 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: osdmap e4: 0 total, 0 up, 0 in 2024-04-30T08:24:36.783 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:36.783 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:36.783 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi032:172.21.15.32=a;smithi071:172.21.15.71=b;smithi189:172.21.15.189=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:24:36.783 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: Saving service mon spec with placement smithi032:172.21.15.32=a;smithi071:172.21.15.71=b;smithi189:172.21.15.189=c;count:3 2024-04-30T08:24:36.783 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:36.783 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:24:36.783 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:36.783 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:24:36.784 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:36.784 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-30T08:24:36.784 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:36.784 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: Deploying daemon mon.c on smithi189 2024-04-30T08:24:36.784 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:36.784 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:36.784 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='client.? 172.21.15.189:0/548230897' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-30T08:24:36.784 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-30T08:24:36.784 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: mon.a calling monitor election 2024-04-30T08:24:36.784 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:24:36.784 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:36.784 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:24:36.784 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:24:36.784 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: mon.c calling monitor election 2024-04-30T08:24:36.785 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:36.785 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:24:36.785 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:24:36.785 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:36.785 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:24:36.785 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-30T08:24:36.785 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: monmap epoch 2 2024-04-30T08:24:36.785 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: fsid 98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:24:36.785 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: last_changed 2024-04-30T08:24:27.318454+0000 2024-04-30T08:24:36.785 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: created 2024-04-30T08:22:06.865404+0000 2024-04-30T08:24:36.785 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: min_mon_release 19 (squid) 2024-04-30T08:24:36.785 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: election_strategy: 1 2024-04-30T08:24:36.785 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: 0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-04-30T08:24:36.785 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-04-30T08:24:36.785 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: fsmap 2024-04-30T08:24:36.786 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: osdmap e4: 0 total, 0 up, 0 in 2024-04-30T08:24:36.786 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: mgrmap e13: x(active, since 62s) 2024-04-30T08:24:36.786 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: overall HEALTH_OK 2024-04-30T08:24:36.786 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:36.786 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:36.786 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:36.786 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-30T08:24:36.786 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:36.786 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: Deploying daemon mon.b on smithi071 2024-04-30T08:24:36.786 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='client.? 172.21.15.189:0/3973830186' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-30T08:24:36.786 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:24:36.786 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:36.786 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:36 smithi071 ceph-mon[33385]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-30T08:24:36.787 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:36.775+0000 7f1370971700 1 -- 172.21.15.189:0/607363900 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f136c19cff0 con 0x7f136c1a1f00 2024-04-30T08:24:41.524 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:41.522+0000 7f135b7fe700 1 -- 172.21.15.189:0/607363900 <== mon.0 v2:172.21.15.32:3300/0 8 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1279 (secure 0 0 0) 0x7f1360016220 con 0x7f136c1a1f00 2024-04-30T08:24:41.524 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-04-30T08:24:41.524 INFO:teuthology.orchestra.run.smithi189.stdout:{"epoch":3,"fsid":"98c0a670-06ca-11ef-bc93-c7b262605968","modified":"2024-04-30T08:24:36.497982Z","created":"2024-04-30T08:22:06.865404Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:3300","nonce":0},{"type":"v1","addr":"172.21.15.32:6789","nonce":0}]},"addr":"172.21.15.32:6789/0","public_addr":"172.21.15.32:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:3300","nonce":0},{"type":"v1","addr":"172.21.15.71:6789","nonce":0}]},"addr":"172.21.15.71:6789/0","public_addr":"172.21.15.71:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-30T08:24:41.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:41.525+0000 7f1370971700 1 -- 172.21.15.189:0/607363900 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f135403d3b0 msgr2=0x7f135403f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:41.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:41.525+0000 7f1370971700 1 --2- 172.21.15.189:0/607363900 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f135403d3b0 0x7f135403f8c0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f135c009750 tx=0x7f135c01a040 comp rx=0 tx=0).stop 2024-04-30T08:24:41.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:41.525+0000 7f1370971700 1 -- 172.21.15.189:0/607363900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f136c1a1f00 msgr2=0x7f136c19bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:41.526 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:41.525+0000 7f1370971700 1 --2- 172.21.15.189:0/607363900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f136c1a1f00 0x7f136c19bb50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f136000b4d0 tx=0x7f136000b890 comp rx=0 tx=0).stop 2024-04-30T08:24:41.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:41.525+0000 7f1370971700 1 -- 172.21.15.189:0/607363900 shutdown_connections 2024-04-30T08:24:41.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:41.525+0000 7f1370971700 1 --2- 172.21.15.189:0/607363900 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f135403d3b0 0x7f135403f8c0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:41.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:41.525+0000 7f1370971700 1 --2- 172.21.15.189:0/607363900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f136c108ab0 0x7f136c1a19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:41.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:41.525+0000 7f1370971700 1 --2- 172.21.15.189:0/607363900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f136c1a1f00 0x7f136c19bb50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:41.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:41.525+0000 7f1370971700 1 -- 172.21.15.189:0/607363900 >> 172.21.15.189:0/607363900 conn(0x7f136c0fe5c0 msgr2=0x7f136c100f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:24:41.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:41.525+0000 7f1370971700 1 -- 172.21.15.189:0/607363900 shutdown_connections 2024-04-30T08:24:41.527 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:41.525+0000 7f1370971700 1 -- 172.21.15.189:0/607363900 wait complete. 2024-04-30T08:24:41.527 INFO:teuthology.orchestra.run.smithi189.stderr:dumped monmap epoch 3 2024-04-30T08:24:41.782 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:41 smithi189 ceph-mon[33038]: mon.c calling monitor election 2024-04-30T08:24:41.782 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:41 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-30T08:24:41.782 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:41 smithi189 ceph-mon[33038]: mon.a calling monitor election 2024-04-30T08:24:41.782 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:41 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:24:41.782 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:41 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:24:41.782 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:41 smithi189 ceph-mon[33038]: from='client.? 172.21.15.189:0/607363900' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-30T08:24:41.783 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:41 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:24:41.783 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:41 smithi189 ceph-mon[33038]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:41.783 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:41 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:24:41.783 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:41 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:24:41.783 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:41 smithi189 ceph-mon[33038]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:41.783 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:41 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:24:41.783 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:41 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:24:41.783 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:41 smithi189 ceph-mon[33038]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-30T08:24:41.783 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:41 smithi189 ceph-mon[33038]: monmap epoch 3 2024-04-30T08:24:41.783 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:41 smithi189 ceph-mon[33038]: fsid 98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:24:41.783 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:41 smithi189 ceph-mon[33038]: last_changed 2024-04-30T08:24:36.497982+0000 2024-04-30T08:24:41.783 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:41 smithi189 ceph-mon[33038]: created 2024-04-30T08:22:06.865404+0000 2024-04-30T08:24:41.783 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:41 smithi189 ceph-mon[33038]: min_mon_release 19 (squid) 2024-04-30T08:24:41.783 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:41 smithi189 ceph-mon[33038]: election_strategy: 1 2024-04-30T08:24:41.784 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:41 smithi189 ceph-mon[33038]: 0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-04-30T08:24:41.784 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:41 smithi189 ceph-mon[33038]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-04-30T08:24:41.784 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:41 smithi189 ceph-mon[33038]: 2: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-30T08:24:41.784 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:41 smithi189 ceph-mon[33038]: fsmap 2024-04-30T08:24:41.784 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:41 smithi189 ceph-mon[33038]: osdmap e4: 0 total, 0 up, 0 in 2024-04-30T08:24:41.784 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:41 smithi189 ceph-mon[33038]: mgrmap e13: x(active, since 71s) 2024-04-30T08:24:41.784 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:41 smithi189 ceph-mon[33038]: overall HEALTH_OK 2024-04-30T08:24:41.784 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:41 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:41.784 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:41 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:41.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:41 smithi032 ceph-mon[28331]: mon.c calling monitor election 2024-04-30T08:24:41.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:41 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-30T08:24:41.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:41 smithi032 ceph-mon[28331]: mon.a calling monitor election 2024-04-30T08:24:41.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:41 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:24:41.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:41 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:24:41.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:41 smithi032 ceph-mon[28331]: from='client.? 172.21.15.189:0/607363900' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-30T08:24:41.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:41 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:24:41.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:41 smithi032 ceph-mon[28331]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:41.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:41 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:24:41.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:41 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:24:41.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:41 smithi032 ceph-mon[28331]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:41.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:41 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:24:41.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:41 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:24:41.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:41 smithi032 ceph-mon[28331]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-30T08:24:41.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:41 smithi032 ceph-mon[28331]: monmap epoch 3 2024-04-30T08:24:41.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:41 smithi032 ceph-mon[28331]: fsid 98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:24:41.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:41 smithi032 ceph-mon[28331]: last_changed 2024-04-30T08:24:36.497982+0000 2024-04-30T08:24:41.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:41 smithi032 ceph-mon[28331]: created 2024-04-30T08:22:06.865404+0000 2024-04-30T08:24:41.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:41 smithi032 ceph-mon[28331]: min_mon_release 19 (squid) 2024-04-30T08:24:41.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:41 smithi032 ceph-mon[28331]: election_strategy: 1 2024-04-30T08:24:41.945 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:41 smithi032 ceph-mon[28331]: 0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-04-30T08:24:41.945 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:41 smithi032 ceph-mon[28331]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-04-30T08:24:41.945 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:41 smithi032 ceph-mon[28331]: 2: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-30T08:24:41.945 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:41 smithi032 ceph-mon[28331]: fsmap 2024-04-30T08:24:41.945 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:41 smithi032 ceph-mon[28331]: osdmap e4: 0 total, 0 up, 0 in 2024-04-30T08:24:41.945 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:41 smithi032 ceph-mon[28331]: mgrmap e13: x(active, since 71s) 2024-04-30T08:24:41.945 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:41 smithi032 ceph-mon[28331]: overall HEALTH_OK 2024-04-30T08:24:41.945 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:41 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:41.945 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:41 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:42.137 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-30T08:24:42.137 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-30T08:24:42.571 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:24:42.887 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:42 smithi189 ceph-mon[33038]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:42.887 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:42 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:42.887 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:42 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:42.887 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:42 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:24:42.887 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:42 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:24:42.898 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:42 smithi032 ceph-mon[28331]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:42.898 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:42 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:42.898 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:42 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:42.899 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:42 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:24:42.899 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:42 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:24:43.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:43 smithi071 ceph-mon[33385]: mon.c calling monitor election 2024-04-30T08:24:43.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:43 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-30T08:24:43.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:43 smithi071 ceph-mon[33385]: mon.a calling monitor election 2024-04-30T08:24:43.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:43 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:24:43.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:43 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:24:43.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:43 smithi071 ceph-mon[33385]: from='client.? 172.21.15.189:0/607363900' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-30T08:24:43.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:43 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:24:43.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:43 smithi071 ceph-mon[33385]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:43.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:43 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:24:43.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:43 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:24:43.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:43 smithi071 ceph-mon[33385]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:43.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:43 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:24:43.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:43 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:24:43.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:43 smithi071 ceph-mon[33385]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-30T08:24:43.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:43 smithi071 ceph-mon[33385]: monmap epoch 3 2024-04-30T08:24:43.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:43 smithi071 ceph-mon[33385]: fsid 98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:24:43.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:43 smithi071 ceph-mon[33385]: last_changed 2024-04-30T08:24:36.497982+0000 2024-04-30T08:24:43.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:43 smithi071 ceph-mon[33385]: created 2024-04-30T08:22:06.865404+0000 2024-04-30T08:24:43.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:43 smithi071 ceph-mon[33385]: min_mon_release 19 (squid) 2024-04-30T08:24:43.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:43 smithi071 ceph-mon[33385]: election_strategy: 1 2024-04-30T08:24:43.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:43 smithi071 ceph-mon[33385]: 0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-04-30T08:24:43.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:43 smithi071 ceph-mon[33385]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-04-30T08:24:43.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:43 smithi071 ceph-mon[33385]: 2: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-30T08:24:43.756 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:43 smithi071 ceph-mon[33385]: fsmap 2024-04-30T08:24:43.756 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:43 smithi071 ceph-mon[33385]: osdmap e4: 0 total, 0 up, 0 in 2024-04-30T08:24:43.756 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:43 smithi071 ceph-mon[33385]: mgrmap e13: x(active, since 71s) 2024-04-30T08:24:43.756 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:43 smithi071 ceph-mon[33385]: overall HEALTH_OK 2024-04-30T08:24:43.756 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:43 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:43.756 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:43 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:43.756 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:43 smithi071 ceph-mon[33385]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:43.756 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:43 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:43.756 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:43 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:43.756 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:43 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:24:43.756 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:43 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:24:44.380 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:44 smithi032 ceph-mon[28331]: mon.b calling monitor election 2024-04-30T08:24:44.381 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:44 smithi032 ceph-mon[28331]: mon.b calling monitor election 2024-04-30T08:24:44.381 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:44 smithi032 ceph-mon[28331]: mon.c calling monitor election 2024-04-30T08:24:44.381 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:44 smithi032 ceph-mon[28331]: mon.a calling monitor election 2024-04-30T08:24:44.381 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:44 smithi032 ceph-mon[28331]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-30T08:24:44.381 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:44 smithi032 ceph-mon[28331]: monmap epoch 3 2024-04-30T08:24:44.381 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:44 smithi032 ceph-mon[28331]: fsid 98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:24:44.381 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:44 smithi032 ceph-mon[28331]: last_changed 2024-04-30T08:24:36.497982+0000 2024-04-30T08:24:44.381 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:44 smithi032 ceph-mon[28331]: created 2024-04-30T08:22:06.865404+0000 2024-04-30T08:24:44.381 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:44 smithi032 ceph-mon[28331]: min_mon_release 19 (squid) 2024-04-30T08:24:44.381 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:44 smithi032 ceph-mon[28331]: election_strategy: 1 2024-04-30T08:24:44.381 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:44 smithi032 ceph-mon[28331]: 0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-04-30T08:24:44.381 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:44 smithi032 ceph-mon[28331]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-04-30T08:24:44.381 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:44 smithi032 ceph-mon[28331]: 2: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-30T08:24:44.381 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:44 smithi032 ceph-mon[28331]: fsmap 2024-04-30T08:24:44.382 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:44 smithi032 ceph-mon[28331]: osdmap e4: 0 total, 0 up, 0 in 2024-04-30T08:24:44.382 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:44 smithi032 ceph-mon[28331]: mgrmap e13: x(active, since 73s) 2024-04-30T08:24:44.382 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:44 smithi032 ceph-mon[28331]: overall HEALTH_OK 2024-04-30T08:24:44.382 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:44 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:44.382 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:44 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:44.382 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:44 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:44.382 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:44 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:44.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:44 smithi071 ceph-mon[33385]: mon.b calling monitor election 2024-04-30T08:24:44.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:44 smithi071 ceph-mon[33385]: mon.b calling monitor election 2024-04-30T08:24:44.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:44 smithi071 ceph-mon[33385]: mon.c calling monitor election 2024-04-30T08:24:44.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:44 smithi071 ceph-mon[33385]: mon.a calling monitor election 2024-04-30T08:24:44.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:44 smithi071 ceph-mon[33385]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-30T08:24:44.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:44 smithi071 ceph-mon[33385]: monmap epoch 3 2024-04-30T08:24:44.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:44 smithi071 ceph-mon[33385]: fsid 98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:24:44.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:44 smithi071 ceph-mon[33385]: last_changed 2024-04-30T08:24:36.497982+0000 2024-04-30T08:24:44.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:44 smithi071 ceph-mon[33385]: created 2024-04-30T08:22:06.865404+0000 2024-04-30T08:24:44.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:44 smithi071 ceph-mon[33385]: min_mon_release 19 (squid) 2024-04-30T08:24:44.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:44 smithi071 ceph-mon[33385]: election_strategy: 1 2024-04-30T08:24:44.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:44 smithi071 ceph-mon[33385]: 0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-04-30T08:24:44.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:44 smithi071 ceph-mon[33385]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-04-30T08:24:44.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:44 smithi071 ceph-mon[33385]: 2: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-30T08:24:44.506 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:44 smithi071 ceph-mon[33385]: fsmap 2024-04-30T08:24:44.506 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:44 smithi071 ceph-mon[33385]: osdmap e4: 0 total, 0 up, 0 in 2024-04-30T08:24:44.506 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:44 smithi071 ceph-mon[33385]: mgrmap e13: x(active, since 73s) 2024-04-30T08:24:44.506 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:44 smithi071 ceph-mon[33385]: overall HEALTH_OK 2024-04-30T08:24:44.506 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:44 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:44.506 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:44 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:44.506 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:44 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:44.506 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:44 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:44.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:44 smithi189 ceph-mon[33038]: mon.b calling monitor election 2024-04-30T08:24:44.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:44 smithi189 ceph-mon[33038]: mon.b calling monitor election 2024-04-30T08:24:44.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:44 smithi189 ceph-mon[33038]: mon.c calling monitor election 2024-04-30T08:24:44.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:44 smithi189 ceph-mon[33038]: mon.a calling monitor election 2024-04-30T08:24:44.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:44 smithi189 ceph-mon[33038]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-30T08:24:44.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:44 smithi189 ceph-mon[33038]: monmap epoch 3 2024-04-30T08:24:44.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:44 smithi189 ceph-mon[33038]: fsid 98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:24:44.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:44 smithi189 ceph-mon[33038]: last_changed 2024-04-30T08:24:36.497982+0000 2024-04-30T08:24:44.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:44 smithi189 ceph-mon[33038]: created 2024-04-30T08:22:06.865404+0000 2024-04-30T08:24:44.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:44 smithi189 ceph-mon[33038]: min_mon_release 19 (squid) 2024-04-30T08:24:44.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:44 smithi189 ceph-mon[33038]: election_strategy: 1 2024-04-30T08:24:44.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:44 smithi189 ceph-mon[33038]: 0: [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] mon.a 2024-04-30T08:24:44.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:44 smithi189 ceph-mon[33038]: 1: [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon.c 2024-04-30T08:24:44.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:44 smithi189 ceph-mon[33038]: 2: [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] mon.b 2024-04-30T08:24:44.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:44 smithi189 ceph-mon[33038]: fsmap 2024-04-30T08:24:44.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:44 smithi189 ceph-mon[33038]: osdmap e4: 0 total, 0 up, 0 in 2024-04-30T08:24:44.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:44 smithi189 ceph-mon[33038]: mgrmap e13: x(active, since 73s) 2024-04-30T08:24:44.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:44 smithi189 ceph-mon[33038]: overall HEALTH_OK 2024-04-30T08:24:44.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:44 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:44.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:44 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:44.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:44 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:44.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:44 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:44.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.631+0000 7f2d76400700 1 -- 172.21.15.32:0/70643137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d701089e0 msgr2=0x7f2d70108df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:44.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.631+0000 7f2d76400700 1 --2- 172.21.15.32:0/70643137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d701089e0 0x7f2d70108df0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2d60009910 tx=0x7f2d60009c20 comp rx=0 tx=0).stop 2024-04-30T08:24:44.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.631+0000 7f2d76400700 1 -- 172.21.15.32:0/70643137 shutdown_connections 2024-04-30T08:24:44.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.631+0000 7f2d76400700 1 --2- 172.21.15.32:0/70643137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d701089e0 0x7f2d70108df0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:44.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.631+0000 7f2d76400700 1 -- 172.21.15.32:0/70643137 >> 172.21.15.32:0/70643137 conn(0x7f2d70078500 msgr2=0x7f2d70078960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:24:44.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.631+0000 7f2d76400700 1 -- 172.21.15.32:0/70643137 shutdown_connections 2024-04-30T08:24:44.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.631+0000 7f2d76400700 1 -- 172.21.15.32:0/70643137 wait complete. 2024-04-30T08:24:44.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.632+0000 7f2d76400700 1 Processor -- start 2024-04-30T08:24:44.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.633+0000 7f2d76400700 1 -- start start 2024-04-30T08:24:44.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.633+0000 7f2d76400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d701089e0 0x7f2d7019aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:24:44.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.633+0000 7f2d76400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d7019af50 0x7f2d7019b360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:24:44.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.633+0000 7f2d76400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d701a3a80 0x7f2d701a5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:24:44.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.633+0000 7f2d76400700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2d7010cf30 con 0x7f2d701a3a80 2024-04-30T08:24:44.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.633+0000 7f2d76400700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f2d7010cdb0 con 0x7f2d7019af50 2024-04-30T08:24:44.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.633+0000 7f2d76400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2d7010d0b0 con 0x7f2d701089e0 2024-04-30T08:24:44.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.634+0000 7f2d6ea00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d7019af50 0x7f2d7019b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:24:44.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.634+0000 7f2d6f400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d701089e0 0x7f2d7019aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:24:44.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.634+0000 7f2d6fe00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d701a3a80 0x7f2d701a5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:24:44.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.634+0000 7f2d6ea00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d7019af50 0x7f2d7019b360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:35448/0 (socket says 172.21.15.32:35448) 2024-04-30T08:24:44.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.634+0000 7f2d6ea00700 1 -- 172.21.15.32:0/3683532236 learned_addr learned my addr 172.21.15.32:0/3683532236 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:24:44.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.634+0000 7f2d6ea00700 1 -- 172.21.15.32:0/3683532236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d701089e0 msgr2=0x7f2d7019aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:44.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.634+0000 7f2d6ea00700 1 --2- 172.21.15.32:0/3683532236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d701089e0 0x7f2d7019aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:44.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.634+0000 7f2d6ea00700 1 -- 172.21.15.32:0/3683532236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d701a3a80 msgr2=0x7f2d701a5ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:44.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.634+0000 7f2d6ea00700 1 --2- 172.21.15.32:0/3683532236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d701a3a80 0x7f2d701a5ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:44.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.634+0000 7f2d6ea00700 1 -- 172.21.15.32:0/3683532236 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d701a6630 con 0x7f2d7019af50 2024-04-30T08:24:44.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.635+0000 7f2d6ea00700 1 --2- 172.21.15.32:0/3683532236 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d7019af50 0x7f2d7019b360 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c00b780 tx=0x7f2d5c00ba90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:24:44.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.635+0000 7f2d67e00700 1 -- 172.21.15.32:0/3683532236 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d5c016070 con 0x7f2d7019af50 2024-04-30T08:24:44.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.635+0000 7f2d67e00700 1 -- 172.21.15.32:0/3683532236 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2d5c0036a0 con 0x7f2d7019af50 2024-04-30T08:24:44.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.635+0000 7f2d67e00700 1 -- 172.21.15.32:0/3683532236 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d5c0119c0 con 0x7f2d7019af50 2024-04-30T08:24:44.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.635+0000 7f2d76400700 1 -- 172.21.15.32:0/3683532236 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d701a6920 con 0x7f2d7019af50 2024-04-30T08:24:44.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.635+0000 7f2d76400700 1 -- 172.21.15.32:0/3683532236 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d701a6e10 con 0x7f2d7019af50 2024-04-30T08:24:44.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.636+0000 7f2d76400700 1 -- 172.21.15.32:0/3683532236 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d701040f0 con 0x7f2d7019af50 2024-04-30T08:24:44.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.637+0000 7f2d67e00700 1 -- 172.21.15.32:0/3683532236 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7f2d5c019070 con 0x7f2d7019af50 2024-04-30T08:24:44.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.637+0000 7f2d67e00700 1 --2- 172.21.15.32:0/3683532236 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f2d5403daa0 0x7f2d5403ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:24:44.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.637+0000 7f2d67e00700 1 -- 172.21.15.32:0/3683532236 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f2d5c050180 con 0x7f2d7019af50 2024-04-30T08:24:44.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.637+0000 7f2d6f400700 1 --2- 172.21.15.32:0/3683532236 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f2d5403daa0 0x7f2d5403ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:24:44.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.638+0000 7f2d6f400700 1 --2- 172.21.15.32:0/3683532236 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f2d5403daa0 0x7f2d5403ffb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2d60005e00 tx=0x7f2d6001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:24:44.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.643+0000 7f2d67e00700 1 -- 172.21.15.32:0/3683532236 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2d5c003850 con 0x7f2d7019af50 2024-04-30T08:24:44.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.879+0000 7f2d76400700 1 -- 172.21.15.32:0/3683532236 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f2d701a7210 con 0x7f2d7019af50 2024-04-30T08:24:44.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.880+0000 7f2d67e00700 1 -- 172.21.15.32:0/3683532236 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+273 (secure 0 0 0) 0x7f2d5c01a3b0 con 0x7f2d7019af50 2024-04-30T08:24:44.881 INFO:teuthology.orchestra.run.smithi032.stdout:# minimal ceph.conf for 98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:24:44.881 INFO:teuthology.orchestra.run.smithi032.stdout:[global] 2024-04-30T08:24:44.882 INFO:teuthology.orchestra.run.smithi032.stdout: fsid = 98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:24:44.882 INFO:teuthology.orchestra.run.smithi032.stdout: mon_host = [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-04-30T08:24:44.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.883+0000 7f2d76400700 1 -- 172.21.15.32:0/3683532236 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f2d5403daa0 msgr2=0x7f2d5403ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:44.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.883+0000 7f2d76400700 1 --2- 172.21.15.32:0/3683532236 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f2d5403daa0 0x7f2d5403ffb0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f2d60005e00 tx=0x7f2d6001a040 comp rx=0 tx=0).stop 2024-04-30T08:24:44.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.883+0000 7f2d76400700 1 -- 172.21.15.32:0/3683532236 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d7019af50 msgr2=0x7f2d7019b360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:44.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.883+0000 7f2d76400700 1 --2- 172.21.15.32:0/3683532236 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d7019af50 0x7f2d7019b360 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f2d5c00b780 tx=0x7f2d5c00ba90 comp rx=0 tx=0).stop 2024-04-30T08:24:44.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.883+0000 7f2d76400700 1 -- 172.21.15.32:0/3683532236 shutdown_connections 2024-04-30T08:24:44.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.883+0000 7f2d76400700 1 --2- 172.21.15.32:0/3683532236 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f2d5403daa0 0x7f2d5403ffb0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:44.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.883+0000 7f2d76400700 1 --2- 172.21.15.32:0/3683532236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2d701a3a80 0x7f2d701a5ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:44.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.883+0000 7f2d76400700 1 --2- 172.21.15.32:0/3683532236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d701089e0 0x7f2d7019aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:44.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.883+0000 7f2d76400700 1 --2- 172.21.15.32:0/3683532236 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2d7019af50 0x7f2d7019b360 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:44.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.883+0000 7f2d76400700 1 -- 172.21.15.32:0/3683532236 >> 172.21.15.32:0/3683532236 conn(0x7f2d70078500 msgr2=0x7f2d700ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:24:44.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.883+0000 7f2d76400700 1 -- 172.21.15.32:0/3683532236 shutdown_connections 2024-04-30T08:24:44.885 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:44.883+0000 7f2d76400700 1 -- 172.21.15.32:0/3683532236 wait complete. 2024-04-30T08:24:45.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:45 smithi032 ceph-mon[28331]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-30T08:24:45.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:45 smithi032 ceph-mon[28331]: Updating smithi071:/etc/ceph/ceph.conf 2024-04-30T08:24:45.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:45 smithi032 ceph-mon[28331]: Updating smithi189:/etc/ceph/ceph.conf 2024-04-30T08:24:45.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:45 smithi032 ceph-mon[28331]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:45.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:45 smithi032 ceph-mon[28331]: Updating smithi071:/var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-30T08:24:45.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:45 smithi032 ceph-mon[28331]: Updating smithi189:/var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-30T08:24:45.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:45 smithi032 ceph-mon[28331]: Updating smithi032:/var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-30T08:24:45.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:45 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:24:45.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:45 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:45.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:45 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:45.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:45 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:45.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:45 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:45.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:45 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3683532236' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:45.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:45 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:45.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:45 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:45.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:45 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:45.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:45 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:45.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:45 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-30T08:24:45.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:45 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-30T08:24:45.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:45 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:45.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:45 smithi071 ceph-mon[33385]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-30T08:24:45.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:45 smithi071 ceph-mon[33385]: Updating smithi071:/etc/ceph/ceph.conf 2024-04-30T08:24:45.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:45 smithi071 ceph-mon[33385]: Updating smithi189:/etc/ceph/ceph.conf 2024-04-30T08:24:45.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:45 smithi071 ceph-mon[33385]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:45.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:45 smithi071 ceph-mon[33385]: Updating smithi071:/var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-30T08:24:45.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:45 smithi071 ceph-mon[33385]: Updating smithi189:/var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-30T08:24:45.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:45 smithi071 ceph-mon[33385]: Updating smithi032:/var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-30T08:24:45.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:45 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:24:45.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:45 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:45.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:45 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:45.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:45 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:45.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:45 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:45.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:45 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3683532236' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:45.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:45 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:45.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:45 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:45.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:45 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:45.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:45 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:45.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:45 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-30T08:24:45.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:45 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-30T08:24:45.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:45 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:45.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:45 smithi189 ceph-mon[33038]: Updating smithi032:/etc/ceph/ceph.conf 2024-04-30T08:24:45.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:45 smithi189 ceph-mon[33038]: Updating smithi071:/etc/ceph/ceph.conf 2024-04-30T08:24:45.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:45 smithi189 ceph-mon[33038]: Updating smithi189:/etc/ceph/ceph.conf 2024-04-30T08:24:45.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:45 smithi189 ceph-mon[33038]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:45.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:45 smithi189 ceph-mon[33038]: Updating smithi071:/var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-30T08:24:45.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:45 smithi189 ceph-mon[33038]: Updating smithi189:/var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-30T08:24:45.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:45 smithi189 ceph-mon[33038]: Updating smithi032:/var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-30T08:24:45.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:45 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:24:45.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:45 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:45.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:45 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:45.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:45 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:45.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:45 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:45.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:45 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3683532236' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:45.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:45 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:45.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:45 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:45.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:45 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:45.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:45 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:45.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:45 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-30T08:24:45.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:45 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-30T08:24:45.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:45 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:45.593 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-30T08:24:45.594 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-30T08:24:45.594 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/ceph.conf 2024-04-30T08:24:45.662 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-30T08:24:45.663 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-30T08:24:45.775 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-30T08:24:45.775 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/ceph/ceph.conf 2024-04-30T08:24:45.807 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-30T08:24:45.807 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-30T08:24:45.875 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-30T08:24:45.875 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.conf 2024-04-30T08:24:45.907 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-30T08:24:45.907 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-30T08:24:45.976 INFO:tasks.cephadm:Adding mgr.x on smithi032 2024-04-30T08:24:45.976 INFO:tasks.cephadm:Adding mgr.y on smithi071 2024-04-30T08:24:45.976 INFO:tasks.cephadm:Adding mgr.z on smithi189 2024-04-30T08:24:45.977 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi032=x;smithi071=y;smithi189=z' 2024-04-30T08:24:46.256 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.c/config 2024-04-30T08:24:46.379 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:46 smithi032 ceph-mon[28331]: Reconfiguring mon.a (unknown last config time)... 2024-04-30T08:24:46.379 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:46 smithi032 ceph-mon[28331]: Reconfiguring daemon mon.a on smithi032 2024-04-30T08:24:46.468 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:46 smithi189 ceph-mon[33038]: Reconfiguring mon.a (unknown last config time)... 2024-04-30T08:24:46.468 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:46 smithi189 ceph-mon[33038]: Reconfiguring daemon mon.a on smithi032 2024-04-30T08:24:46.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:46 smithi071 ceph-mon[33385]: Reconfiguring mon.a (unknown last config time)... 2024-04-30T08:24:46.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:46 smithi071 ceph-mon[33385]: Reconfiguring daemon mon.a on smithi032 2024-04-30T08:24:47.413 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:47 smithi189 ceph-mon[33038]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:47.413 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:47 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:47.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:47 smithi032 ceph-mon[28331]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:47.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:47 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:47.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:47 smithi071 ceph-mon[33385]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:47.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:47 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:47.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.639+0000 7ff33706e700 1 -- 172.21.15.189:0/3998216906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff330103c60 msgr2=0x7ff330104070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:47.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.639+0000 7ff33706e700 1 --2- 172.21.15.189:0/3998216906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff330103c60 0x7ff330104070 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7ff324009910 tx=0x7ff324009c20 comp rx=0 tx=0).stop 2024-04-30T08:24:47.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.639+0000 7ff33706e700 1 -- 172.21.15.189:0/3998216906 shutdown_connections 2024-04-30T08:24:47.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.639+0000 7ff33706e700 1 --2- 172.21.15.189:0/3998216906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff330103c60 0x7ff330104070 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:47.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.639+0000 7ff33706e700 1 -- 172.21.15.189:0/3998216906 >> 172.21.15.189:0/3998216906 conn(0x7ff3300ff2a0 msgr2=0x7ff330101750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:24:47.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.640+0000 7ff33706e700 1 -- 172.21.15.189:0/3998216906 shutdown_connections 2024-04-30T08:24:47.641 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.640+0000 7ff33706e700 1 -- 172.21.15.189:0/3998216906 wait complete. 2024-04-30T08:24:47.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.640+0000 7ff33706e700 1 Processor -- start 2024-04-30T08:24:47.645 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.640+0000 7ff33706e700 1 -- start start 2024-04-30T08:24:47.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.641+0000 7ff33706e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff330103c60 0x7ff330199900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:24:47.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.641+0000 7ff33706e700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff330199e40 0x7ff33019edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:24:47.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.641+0000 7ff33706e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff33019f310 0x7ff3301a1730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:24:47.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.641+0000 7ff33706e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff330109c20 con 0x7ff330103c60 2024-04-30T08:24:47.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.641+0000 7ff33706e700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff330109aa0 con 0x7ff330199e40 2024-04-30T08:24:47.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.641+0000 7ff33706e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff330109da0 con 0x7ff33019f310 2024-04-30T08:24:47.646 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.641+0000 7ff33560b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff33019f310 0x7ff3301a1730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:24:47.647 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.641+0000 7ff33560b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff33019f310 0x7ff3301a1730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57020/0 (socket says 172.21.15.189:57020) 2024-04-30T08:24:47.649 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.641+0000 7ff33560b700 1 -- 172.21.15.189:0/2425967282 learned_addr learned my addr 172.21.15.189:0/2425967282 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-30T08:24:47.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.641+0000 7ff32ffff700 1 --2- 172.21.15.189:0/2425967282 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff330199e40 0x7ff33019edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:24:47.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.641+0000 7ff33560b700 1 -- 172.21.15.189:0/2425967282 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff330199e40 msgr2=0x7ff33019edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:47.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.641+0000 7ff33560b700 1 --2- 172.21.15.189:0/2425967282 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff330199e40 0x7ff33019edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:47.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.641+0000 7ff33560b700 1 -- 172.21.15.189:0/2425967282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff330103c60 msgr2=0x7ff330199900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:47.650 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.641+0000 7ff33560b700 1 --2- 172.21.15.189:0/2425967282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff330103c60 0x7ff330199900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:47.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.641+0000 7ff33560b700 1 -- 172.21.15.189:0/2425967282 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3301a1ca0 con 0x7ff33019f310 2024-04-30T08:24:47.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.641+0000 7ff33560b700 1 --2- 172.21.15.189:0/2425967282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff33019f310 0x7ff3301a1730 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff32800ba50 tx=0x7ff32800bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:24:47.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.641+0000 7ff32dffb700 1 -- 172.21.15.189:0/2425967282 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff328019070 con 0x7ff33019f310 2024-04-30T08:24:47.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.642+0000 7ff32dffb700 1 -- 172.21.15.189:0/2425967282 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff328014440 con 0x7ff33019f310 2024-04-30T08:24:47.651 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.642+0000 7ff33706e700 1 -- 172.21.15.189:0/2425967282 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3301a1f90 con 0x7ff33019f310 2024-04-30T08:24:47.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.642+0000 7ff32dffb700 1 -- 172.21.15.189:0/2425967282 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff32800ce60 con 0x7ff33019f310 2024-04-30T08:24:47.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.642+0000 7ff33706e700 1 -- 172.21.15.189:0/2425967282 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3301a2400 con 0x7ff33019f310 2024-04-30T08:24:47.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.643+0000 7ff32dffb700 1 -- 172.21.15.189:0/2425967282 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7ff32801c070 con 0x7ff33019f310 2024-04-30T08:24:47.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.643+0000 7ff32dffb700 1 --2- 172.21.15.189:0/2425967282 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff32003daa0 0x7ff32003ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:24:47.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.643+0000 7ff32dffb700 1 -- 172.21.15.189:0/2425967282 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7ff328051b50 con 0x7ff33019f310 2024-04-30T08:24:47.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.643+0000 7ff33706e700 1 -- 172.21.15.189:0/2425967282 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff330064d70 con 0x7ff33019f310 2024-04-30T08:24:47.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.643+0000 7ff334e0a700 1 --2- 172.21.15.189:0/2425967282 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff32003daa0 0x7ff32003ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:24:47.652 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.644+0000 7ff334e0a700 1 --2- 172.21.15.189:0/2425967282 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff32003daa0 0x7ff32003ffb0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff324006010 tx=0x7ff324005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:24:47.653 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.649+0000 7ff32dffb700 1 -- 172.21.15.189:0/2425967282 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff328012c60 con 0x7ff33019f310 2024-04-30T08:24:47.861 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.860+0000 7ff33706e700 1 -- 172.21.15.189:0/2425967282 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi032=x;smithi071=y;smithi189=z", "target": ["mon-mgr", ""]}) -- 0x7ff3301010c0 con 0x7ff32003daa0 2024-04-30T08:24:47.877 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.876+0000 7ff32dffb700 1 -- 172.21.15.189:0/2425967282 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7ff3301010c0 con 0x7ff32003daa0 2024-04-30T08:24:47.878 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled mgr update... 2024-04-30T08:24:47.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.878+0000 7ff33706e700 1 -- 172.21.15.189:0/2425967282 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff32003daa0 msgr2=0x7ff32003ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:47.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.878+0000 7ff33706e700 1 --2- 172.21.15.189:0/2425967282 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff32003daa0 0x7ff32003ffb0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7ff324006010 tx=0x7ff324005dc0 comp rx=0 tx=0).stop 2024-04-30T08:24:47.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.878+0000 7ff33706e700 1 -- 172.21.15.189:0/2425967282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff33019f310 msgr2=0x7ff3301a1730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:47.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.878+0000 7ff33706e700 1 --2- 172.21.15.189:0/2425967282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff33019f310 0x7ff3301a1730 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff32800ba50 tx=0x7ff32800bd60 comp rx=0 tx=0).stop 2024-04-30T08:24:47.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.878+0000 7ff33706e700 1 -- 172.21.15.189:0/2425967282 shutdown_connections 2024-04-30T08:24:47.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.878+0000 7ff33706e700 1 --2- 172.21.15.189:0/2425967282 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff32003daa0 0x7ff32003ffb0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:47.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.878+0000 7ff33706e700 1 --2- 172.21.15.189:0/2425967282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff33019f310 0x7ff3301a1730 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:47.880 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.878+0000 7ff33706e700 1 --2- 172.21.15.189:0/2425967282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff330103c60 0x7ff330199900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:47.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.878+0000 7ff33706e700 1 --2- 172.21.15.189:0/2425967282 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff330199e40 0x7ff33019edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:47.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.878+0000 7ff33706e700 1 -- 172.21.15.189:0/2425967282 >> 172.21.15.189:0/2425967282 conn(0x7ff3300ff2a0 msgr2=0x7ff3301009b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:24:47.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.879+0000 7ff33706e700 1 -- 172.21.15.189:0/2425967282 shutdown_connections 2024-04-30T08:24:47.881 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:24:47.879+0000 7ff33706e700 1 -- 172.21.15.189:0/2425967282 wait complete. 2024-04-30T08:24:48.431 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:48 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:48.431 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:48 smithi189 ceph-mon[33038]: Reconfiguring mon.b (monmap changed)... 2024-04-30T08:24:48.432 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:48 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-30T08:24:48.432 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:48 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-30T08:24:48.432 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:48 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:48.432 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:48 smithi189 ceph-mon[33038]: Reconfiguring daemon mon.b on smithi071 2024-04-30T08:24:48.432 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:48 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:48.433 DEBUG:teuthology.orchestra.run.smithi071:mgr.y> sudo journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mgr.y.service 2024-04-30T08:24:48.436 DEBUG:teuthology.orchestra.run.smithi189:mgr.z> sudo journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mgr.z.service 2024-04-30T08:24:48.439 INFO:tasks.cephadm:Deploying OSDs... 2024-04-30T08:24:48.440 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-30T08:24:48.440 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/scratch_devs of=/dev/stdout 2024-04-30T08:24:48.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:48 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:48.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:48 smithi032 ceph-mon[28331]: Reconfiguring mon.b (monmap changed)... 2024-04-30T08:24:48.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:48 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-30T08:24:48.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:48 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-30T08:24:48.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:48 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:48.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:48 smithi032 ceph-mon[28331]: Reconfiguring daemon mon.b on smithi071 2024-04-30T08:24:48.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:48 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:48.455 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:48 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:48.456 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:48 smithi071 ceph-mon[33385]: Reconfiguring mon.b (monmap changed)... 2024-04-30T08:24:48.456 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:48 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-30T08:24:48.456 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:48 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-30T08:24:48.456 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:48 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:48.456 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:48 smithi071 ceph-mon[33385]: Reconfiguring daemon mon.b on smithi071 2024-04-30T08:24:48.456 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:48 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:48.503 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-30T08:24:48.504 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_1 2024-04-30T08:24:48.565 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-30T08:24:48.565 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-30T08:24:48.565 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 6h/6d Inode: 633 Links: 1 2024-04-30T08:24:48.565 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-30T08:24:48.565 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-04-30T08:24:48.566 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-04-30 08:23:46.248093660 +0000 2024-04-30T08:24:48.566 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-04-30 08:17:55.507374780 +0000 2024-04-30T08:24:48.566 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-04-30 08:17:55.507374780 +0000 2024-04-30T08:24:48.566 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-04-30 08:17:55.507374780 +0000 2024-04-30T08:24:48.566 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-30T08:24:48.651 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-04-30T08:24:48.651 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-04-30T08:24:48.651 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000284363 s, 1.8 MB/s 2024-04-30T08:24:48.654 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-30T08:24:48.713 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_2 2024-04-30T08:24:48.768 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-30T08:24:48.768 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-30T08:24:48.768 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 6h/6d Inode: 651 Links: 1 2024-04-30T08:24:48.768 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-30T08:24:48.768 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-04-30T08:24:48.768 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-04-30 08:23:46.352091606 +0000 2024-04-30T08:24:48.769 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-04-30 08:17:56.126362604 +0000 2024-04-30T08:24:48.769 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-04-30 08:17:56.126362604 +0000 2024-04-30T08:24:48.769 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-04-30 08:17:56.126362604 +0000 2024-04-30T08:24:48.769 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-30T08:24:48.857 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-04-30T08:24:48.858 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-04-30T08:24:48.858 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000302821 s, 1.7 MB/s 2024-04-30T08:24:48.865 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-30T08:24:48.923 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_3 2024-04-30T08:24:48.978 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-30T08:24:48.978 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-30T08:24:48.978 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 6h/6d Inode: 671 Links: 1 2024-04-30T08:24:48.978 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-30T08:24:48.978 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-04-30T08:24:48.978 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-04-30 08:23:46.454089590 +0000 2024-04-30T08:24:48.979 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-04-30 08:17:56.742350487 +0000 2024-04-30T08:24:48.979 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-04-30 08:17:56.743350467 +0000 2024-04-30T08:24:48.979 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-04-30 08:17:56.742350487 +0000 2024-04-30T08:24:48.979 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-30T08:24:49.066 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-04-30T08:24:49.067 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-04-30T08:24:49.067 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000286481 s, 1.8 MB/s 2024-04-30T08:24:49.071 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-30T08:24:49.131 DEBUG:teuthology.orchestra.run.smithi032:> stat /dev/vg_nvme/lv_4 2024-04-30T08:24:49.186 INFO:teuthology.orchestra.run.smithi032.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-30T08:24:49.186 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-30T08:24:49.186 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 6h/6d Inode: 690 Links: 1 2024-04-30T08:24:49.186 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-30T08:24:49.186 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:device_t:s0 2024-04-30T08:24:49.186 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-04-30 08:23:46.528088128 +0000 2024-04-30T08:24:49.186 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-04-30 08:17:57.362338291 +0000 2024-04-30T08:24:49.186 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-04-30 08:17:57.362338291 +0000 2024-04-30T08:24:49.187 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-04-30 08:17:57.362338291 +0000 2024-04-30T08:24:49.187 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-30T08:24:49.266 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records in 2024-04-30T08:24:49.266 INFO:teuthology.orchestra.run.smithi032.stderr:1+0 records out 2024-04-30T08:24:49.266 INFO:teuthology.orchestra.run.smithi032.stderr:512 bytes copied, 0.000254856 s, 2.0 MB/s 2024-04-30T08:24:49.270 DEBUG:teuthology.orchestra.run.smithi032:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-30T08:24:49.325 DEBUG:teuthology.orchestra.run.smithi071:> set -ex 2024-04-30T08:24:49.325 DEBUG:teuthology.orchestra.run.smithi071:> dd if=/scratch_devs of=/dev/stdout 2024-04-30T08:24:49.344 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-30T08:24:49.344 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_1 2024-04-30T08:24:49.402 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-30T08:24:49.402 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-30T08:24:49.402 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 667 Links: 1 2024-04-30T08:24:49.402 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-30T08:24:49.402 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-04-30T08:24:49.402 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-04-30 08:24:35.882394992 +0000 2024-04-30T08:24:49.402 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-04-30 08:17:20.537041182 +0000 2024-04-30T08:24:49.402 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-04-30 08:17:20.537041182 +0000 2024-04-30T08:24:49.403 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-04-30 08:17:20.537041182 +0000 2024-04-30T08:24:49.403 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-30T08:24:49.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:49 smithi032 ceph-mon[28331]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:49.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:49 smithi032 ceph-mon[28331]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi032=x;smithi071=y;smithi189=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:24:49.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:49 smithi032 ceph-mon[28331]: Saving service mgr spec with placement smithi032=x;smithi071=y;smithi189=z;count:3 2024-04-30T08:24:49.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:49 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:49.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:49 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:49.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:49 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-30T08:24:49.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:49 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-30T08:24:49.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:49 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:49.465 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:49 smithi071 ceph-mon[33385]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:49.465 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:49 smithi071 ceph-mon[33385]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi032=x;smithi071=y;smithi189=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:24:49.465 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:49 smithi071 ceph-mon[33385]: Saving service mgr spec with placement smithi032=x;smithi071=y;smithi189=z;count:3 2024-04-30T08:24:49.465 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:49 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:49.465 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:49 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:49.465 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:49 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-30T08:24:49.465 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:49 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-30T08:24:49.465 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:49 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:49.467 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-04-30T08:24:49.467 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-04-30T08:24:49.468 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000256071 s, 2.0 MB/s 2024-04-30T08:24:49.469 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-30T08:24:49.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:49 smithi189 ceph-mon[33038]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:49.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:49 smithi189 ceph-mon[33038]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi032=x;smithi071=y;smithi189=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:24:49.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:49 smithi189 ceph-mon[33038]: Saving service mgr spec with placement smithi032=x;smithi071=y;smithi189=z;count:3 2024-04-30T08:24:49.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:49 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:49.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:49 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:49.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:49 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-30T08:24:49.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:49 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-30T08:24:49.470 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:49 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:49.526 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_2 2024-04-30T08:24:49.583 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-30T08:24:49.584 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-30T08:24:49.584 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-30T08:24:49.584 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-30T08:24:49.584 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-04-30T08:24:49.584 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-04-30 08:24:35.882394992 +0000 2024-04-30T08:24:49.584 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-04-30 08:17:20.878035103 +0000 2024-04-30T08:24:49.584 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-04-30 08:17:20.878035103 +0000 2024-04-30T08:24:49.584 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-04-30 08:17:20.878035103 +0000 2024-04-30T08:24:49.584 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-30T08:24:49.649 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-04-30T08:24:49.649 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-04-30T08:24:49.650 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000247813 s, 2.1 MB/s 2024-04-30T08:24:49.651 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-30T08:24:49.709 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_3 2024-04-30T08:24:49.765 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-30T08:24:49.766 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-30T08:24:49.766 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-30T08:24:49.766 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-30T08:24:49.766 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-04-30T08:24:49.766 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-04-30 08:24:35.882394992 +0000 2024-04-30T08:24:49.766 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-04-30 08:17:21.212029149 +0000 2024-04-30T08:24:49.766 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-04-30 08:17:21.212029149 +0000 2024-04-30T08:24:49.766 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-04-30 08:17:21.212029149 +0000 2024-04-30T08:24:49.766 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-30T08:24:49.830 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-04-30T08:24:49.830 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-04-30T08:24:49.830 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000243964 s, 2.1 MB/s 2024-04-30T08:24:49.831 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-30T08:24:49.888 DEBUG:teuthology.orchestra.run.smithi071:> stat /dev/vg_nvme/lv_4 2024-04-30T08:24:49.944 INFO:teuthology.orchestra.run.smithi071.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-30T08:24:49.944 INFO:teuthology.orchestra.run.smithi071.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-30T08:24:49.945 INFO:teuthology.orchestra.run.smithi071.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-04-30T08:24:49.945 INFO:teuthology.orchestra.run.smithi071.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-30T08:24:49.945 INFO:teuthology.orchestra.run.smithi071.stdout:Context: system_u:object_r:device_t:s0 2024-04-30T08:24:49.945 INFO:teuthology.orchestra.run.smithi071.stdout:Access: 2024-04-30 08:24:35.883394974 +0000 2024-04-30T08:24:49.945 INFO:teuthology.orchestra.run.smithi071.stdout:Modify: 2024-04-30 08:17:21.551023106 +0000 2024-04-30T08:24:49.945 INFO:teuthology.orchestra.run.smithi071.stdout:Change: 2024-04-30 08:17:21.551023106 +0000 2024-04-30T08:24:49.945 INFO:teuthology.orchestra.run.smithi071.stdout: Birth: 2024-04-30 08:17:21.551023106 +0000 2024-04-30T08:24:49.946 DEBUG:teuthology.orchestra.run.smithi071:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-30T08:24:50.007 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records in 2024-04-30T08:24:50.007 INFO:teuthology.orchestra.run.smithi071.stderr:1+0 records out 2024-04-30T08:24:50.008 INFO:teuthology.orchestra.run.smithi071.stderr:512 bytes copied, 0.000201508 s, 2.5 MB/s 2024-04-30T08:24:50.009 DEBUG:teuthology.orchestra.run.smithi071:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-30T08:24:50.064 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-04-30T08:24:50.064 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/scratch_devs of=/dev/stdout 2024-04-30T08:24:50.082 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-30T08:24:50.082 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_1 2024-04-30T08:24:50.138 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-30T08:24:50.138 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-30T08:24:50.138 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-04-30T08:24:50.139 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-30T08:24:50.139 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-04-30T08:24:50.139 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-04-30 08:24:26.095677710 +0000 2024-04-30T08:24:50.139 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-04-30 08:16:45.015974327 +0000 2024-04-30T08:24:50.139 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-04-30 08:16:45.015974327 +0000 2024-04-30T08:24:50.139 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-04-30 08:16:45.015974327 +0000 2024-04-30T08:24:50.139 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-30T08:24:50.204 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-04-30T08:24:50.204 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-04-30T08:24:50.204 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000195593 s, 2.6 MB/s 2024-04-30T08:24:50.206 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-30T08:24:50.263 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_2 2024-04-30T08:24:50.319 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-30T08:24:50.319 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-30T08:24:50.319 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-30T08:24:50.319 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-30T08:24:50.319 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-04-30T08:24:50.319 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-04-30 08:24:26.095677710 +0000 2024-04-30T08:24:50.319 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-04-30 08:16:45.354967247 +0000 2024-04-30T08:24:50.319 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-04-30 08:16:45.354967247 +0000 2024-04-30T08:24:50.319 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-04-30 08:16:45.354967247 +0000 2024-04-30T08:24:50.319 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-30T08:24:50.381 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-04-30T08:24:50.382 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-04-30T08:24:50.382 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000199784 s, 2.6 MB/s 2024-04-30T08:24:50.384 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-30T08:24:50.439 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_3 2024-04-30T08:24:50.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:50 smithi032 ceph-mon[28331]: Reconfiguring mon.c (monmap changed)... 2024-04-30T08:24:50.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:50 smithi032 ceph-mon[28331]: Reconfiguring daemon mon.c on smithi189 2024-04-30T08:24:50.495 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-30T08:24:50.495 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-30T08:24:50.495 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-04-30T08:24:50.495 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-30T08:24:50.495 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-04-30T08:24:50.495 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-04-30 08:24:26.096677690 +0000 2024-04-30T08:24:50.495 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-04-30 08:16:45.712959771 +0000 2024-04-30T08:24:50.495 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-04-30 08:16:45.712959771 +0000 2024-04-30T08:24:50.495 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-04-30 08:16:45.712959771 +0000 2024-04-30T08:24:50.496 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-30T08:24:50.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:50 smithi071 ceph-mon[33385]: Reconfiguring mon.c (monmap changed)... 2024-04-30T08:24:50.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:50 smithi071 ceph-mon[33385]: Reconfiguring daemon mon.c on smithi189 2024-04-30T08:24:50.549 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:50 smithi189 ceph-mon[33038]: Reconfiguring mon.c (monmap changed)... 2024-04-30T08:24:50.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:50 smithi189 ceph-mon[33038]: Reconfiguring daemon mon.c on smithi189 2024-04-30T08:24:50.558 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-04-30T08:24:50.558 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-04-30T08:24:50.558 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000214635 s, 2.4 MB/s 2024-04-30T08:24:50.559 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-30T08:24:50.615 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_4 2024-04-30T08:24:50.670 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-30T08:24:50.671 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-30T08:24:50.671 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-04-30T08:24:50.671 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-30T08:24:50.671 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-04-30T08:24:50.671 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-04-30 08:24:26.096677690 +0000 2024-04-30T08:24:50.671 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-04-30 08:16:46.045952817 +0000 2024-04-30T08:24:50.671 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-04-30 08:16:46.045952817 +0000 2024-04-30T08:24:50.671 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-04-30 08:16:46.045952817 +0000 2024-04-30T08:24:50.671 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-30T08:24:50.733 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-04-30T08:24:50.733 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-04-30T08:24:50.733 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000246684 s, 2.1 MB/s 2024-04-30T08:24:50.734 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-30T08:24:50.790 INFO:tasks.cephadm:Deploying osd.0 on smithi032 with /dev/vg_nvme/lv_4... 2024-04-30T08:24:50.790 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-30T08:24:51.249 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:24:51.272 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:51 smithi189 ceph-mon[33038]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:51.272 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:51 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:51.272 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:51 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:51.272 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:51 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:24:51.272 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:51 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:51.272 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:51 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:24:51.272 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:51 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:51.272 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:51 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-30T08:24:51.272 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:51 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-30T08:24:51.272 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:51 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-30T08:24:51.272 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:51 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:51.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:51 smithi032 ceph-mon[28331]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:51.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:51 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:51.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:51 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:51.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:51 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:24:51.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:51 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:51.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:51 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:24:51.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:51 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:51.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:51 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-30T08:24:51.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:51 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-30T08:24:51.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:51 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-30T08:24:51.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:51 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:51.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:51 smithi071 ceph-mon[33385]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:51.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:51 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:51.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:51 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:51.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:51 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:24:51.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:51 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:51.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:51 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:24:51.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:51 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:51.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:51 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-30T08:24:51.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:51 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-30T08:24:51.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:51 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-30T08:24:51.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:51 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:52.469 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:52 smithi032 ceph-mon[28331]: Deploying daemon mgr.z on smithi189 2024-04-30T08:24:52.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:52 smithi071 ceph-mon[33385]: Deploying daemon mgr.z on smithi189 2024-04-30T08:24:52.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:52 smithi189 ceph-mon[33038]: Deploying daemon mgr.z on smithi189 2024-04-30T08:24:52.551 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:24:52 smithi189 systemd[1]: Starting Ceph mgr.z for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:24:52.851 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:24:52 smithi189 podman[34892]: 2024-04-30 08:24:52.48485364 +0000 UTC m=+0.017183861 image pull 16056ace0edc7f73516cf28297f1db04dda872801fb5b17b9157b0d1ccf38f9b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de 2024-04-30T08:24:52.851 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:24:52 smithi189 podman[34892]: 2024-04-30 08:24:52.601725228 +0000 UTC m=+0.134055458 container create b90455e4a74856c308ad9119d2b65297cff0a79e8a07fe42a68add492ee481c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-z, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-30T08:24:52.851 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:24:52 smithi189 podman[34892]: 2024-04-30 08:24:52.812502303 +0000 UTC m=+0.344832511 container init b90455e4a74856c308ad9119d2b65297cff0a79e8a07fe42a68add492ee481c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-30T08:24:52.851 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:24:52 smithi189 podman[34892]: 2024-04-30 08:24:52.815386125 +0000 UTC m=+0.347716323 container start b90455e4a74856c308ad9119d2b65297cff0a79e8a07fe42a68add492ee481c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-z, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240430, io.buildah.version=1.33.5) 2024-04-30T08:24:52.852 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:24:52 smithi189 ceph-mgr[34911]: -- 172.21.15.189:0/3080094471 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5598de8a25a0 con 0x5598e362c480 2024-04-30T08:24:53.170 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:24:52 smithi189 bash[34892]: b90455e4a74856c308ad9119d2b65297cff0a79e8a07fe42a68add492ee481c7 2024-04-30T08:24:53.170 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:24:52 smithi189 systemd[1]: Started Ceph mgr.z for 98c0a670-06ca-11ef-bc93-c7b262605968. 2024-04-30T08:24:53.170 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:24:52 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-z[34907]: 2024-04-30T08:24:52.981+0000 7fa4be2f8200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-30T08:24:53.170 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:24:53 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-z[34907]: 2024-04-30T08:24:53.169+0000 7fa4be2f8200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-30T08:24:53.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:53 smithi071 ceph-mon[33385]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:53.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:53 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:53.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:53 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:53.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:53 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:53.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:53 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-30T08:24:53.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:53 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-30T08:24:53.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:53 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-30T08:24:53.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:53 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:53.550 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:24:53 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-z[34907]: 2024-04-30T08:24:53.272+0000 7fa4be2f8200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-30T08:24:53.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:53 smithi189 ceph-mon[33038]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:53.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:53 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:53.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:53 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:53.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:53 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:53.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:53 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-30T08:24:53.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:53 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-30T08:24:53.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:53 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-30T08:24:53.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:53 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:53.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:53 smithi032 ceph-mon[28331]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:53.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:53 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:53.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:53 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:53.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:53 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:53.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:53 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-30T08:24:53.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:53 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-30T08:24:53.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:53 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-30T08:24:53.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:53 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:54.050 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:24:53 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-z[34907]: 2024-04-30T08:24:53.581+0000 7fa4be2f8200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-30T08:24:54.050 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:24:53 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-z[34907]: 2024-04-30T08:24:53.667+0000 7fa4be2f8200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-30T08:24:54.228 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-30T08:24:54.229 INFO:teuthology.orchestra.run.smithi032.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-30T08:24:54.229 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 10+0 records in 2024-04-30T08:24:54.229 INFO:teuthology.orchestra.run.smithi032.stderr:10+0 records out 2024-04-30T08:24:54.229 INFO:teuthology.orchestra.run.smithi032.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0387466 s, 271 MB/s 2024-04-30T08:24:54.229 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping successful for: 2024-04-30T08:24:54.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:54 smithi071 ceph-mon[33385]: Deploying daemon mgr.y on smithi071 2024-04-30T08:24:54.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:54 smithi189 ceph-mon[33038]: Deploying daemon mgr.y on smithi071 2024-04-30T08:24:54.651 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:54 smithi032 ceph-mon[28331]: Deploying daemon mgr.y on smithi071 2024-04-30T08:24:55.031 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi032:vg_nvme/lv_4 2024-04-30T08:24:55.133 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:24:54 smithi071 systemd[1]: Starting Ceph mgr.y for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:24:55.211 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:24:54 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-z[34907]: 2024-04-30T08:24:54.876+0000 7fa4be2f8200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-30T08:24:55.211 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:24:55 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-z[34907]: 2024-04-30T08:24:55.050+0000 7fa4be2f8200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-30T08:24:55.451 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:55 smithi071 ceph-mon[33385]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:55.451 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:24:55 smithi071 podman[34758]: 2024-04-30 08:24:55.132368177 +0000 UTC m=+0.093249220 container create b69f6e6706d923cc833b8d2f70598cbae2555ca76f34772da9070ec2a372e30b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-y, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-30T08:24:55.451 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:24:55 smithi071 podman[34758]: 2024-04-30 08:24:55.056902077 +0000 UTC m=+0.017783132 image pull 16056ace0edc7f73516cf28297f1db04dda872801fb5b17b9157b0d1ccf38f9b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de 2024-04-30T08:24:55.451 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:24:55 smithi071 podman[34758]: 2024-04-30 08:24:55.284630567 +0000 UTC m=+0.245511611 container init b69f6e6706d923cc833b8d2f70598cbae2555ca76f34772da9070ec2a372e30b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-y, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.build-date=20240430, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-30T08:24:55.452 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:24:55 smithi071 podman[34758]: 2024-04-30 08:24:55.287650067 +0000 UTC m=+0.248531123 container start b69f6e6706d923cc833b8d2f70598cbae2555ca76f34772da9070ec2a372e30b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-y, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-30T08:24:55.452 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:24:55 smithi071 ceph-mgr[34777]: -- 172.21.15.71:0/584482624 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5614637e25a0 con 0x56146856c480 2024-04-30T08:24:55.452 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:24:55 smithi071 bash[34758]: b69f6e6706d923cc833b8d2f70598cbae2555ca76f34772da9070ec2a372e30b 2024-04-30T08:24:55.452 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:24:55 smithi071 systemd[1]: Started Ceph mgr.y for 98c0a670-06ca-11ef-bc93-c7b262605968. 2024-04-30T08:24:55.452 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:24:55 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-y[34773]: 2024-04-30T08:24:55.450+0000 7faad382c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-30T08:24:55.479 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:24:55.537 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:55 smithi189 ceph-mon[33038]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:55.537 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:24:55 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-z[34907]: 2024-04-30T08:24:55.229+0000 7fa4be2f8200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-30T08:24:55.537 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:24:55 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-z[34907]: 2024-04-30T08:24:55.535+0000 7fa4be2f8200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-30T08:24:55.626 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:55 smithi032 ceph-mon[28331]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:55.740 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:24:55 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-y[34773]: 2024-04-30T08:24:55.636+0000 7faad382c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-30T08:24:55.741 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:24:55 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-y[34773]: 2024-04-30T08:24:55.739+0000 7faad382c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-30T08:24:55.800 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:24:55 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-z[34907]: 2024-04-30T08:24:55.631+0000 7fa4be2f8200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-30T08:24:56.220 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:24:55 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-z[34907]: 2024-04-30T08:24:55.845+0000 7fa4be2f8200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-30T08:24:56.220 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:24:56 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-z[34907]: 2024-04-30T08:24:56.048+0000 7fa4be2f8200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-30T08:24:56.421 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:24:56 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-y[34773]: 2024-04-30T08:24:56.066+0000 7faad382c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-30T08:24:56.422 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:24:56 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-y[34773]: 2024-04-30T08:24:56.153+0000 7faad382c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-30T08:24:56.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:56 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:56.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:56 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:56.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:56 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:56.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:56 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:56.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:56 smithi032 ceph-mon[28331]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:56.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:56 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:24:56.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:56 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:56.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:56 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:56.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:56 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:56.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:56 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:56.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:56 smithi189 ceph-mon[33038]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:56.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:56 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:24:56.739 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:56 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:56.739 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:56 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:56.739 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:56 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:56.739 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:56 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:56.739 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:56 smithi071 ceph-mon[33385]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:56.739 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:56 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:24:57.050 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:24:56 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-z[34907]: 2024-04-30T08:24:56.701+0000 7fa4be2f8200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-30T08:24:57.433 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:24:57 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-z[34907]: 2024-04-30T08:24:57.136+0000 7fa4be2f8200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-30T08:24:57.433 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:24:57 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-z[34907]: 2024-04-30T08:24:57.231+0000 7fa4be2f8200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-30T08:24:57.433 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:24:57 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-z[34907]: 2024-04-30T08:24:57.432+0000 7fa4be2f8200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-30T08:24:57.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.563+0000 7fc5bc800700 1 -- 172.21.15.32:0/1658518421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5b4104290 msgr2=0x7fc5b41046a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:57.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.563+0000 7fc5bc800700 1 --2- 172.21.15.32:0/1658518421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5b4104290 0x7fc5b41046a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc5a8009990 tx=0x7fc5a8009ca0 comp rx=0 tx=0).stop 2024-04-30T08:24:57.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.563+0000 7fc5bc800700 1 -- 172.21.15.32:0/1658518421 shutdown_connections 2024-04-30T08:24:57.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.563+0000 7fc5bc800700 1 --2- 172.21.15.32:0/1658518421 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5b4105d60 0x7fc5b410c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:57.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.563+0000 7fc5bc800700 1 --2- 172.21.15.32:0/1658518421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5b4104290 0x7fc5b41046a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:57.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.563+0000 7fc5bc800700 1 --2- 172.21.15.32:0/1658518421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5b41053c0 0x7fc5b41057d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:57.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.563+0000 7fc5bc800700 1 -- 172.21.15.32:0/1658518421 >> 172.21.15.32:0/1658518421 conn(0x7fc5b40ff910 msgr2=0x7fc5b4101d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:24:57.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.564+0000 7fc5bc800700 1 -- 172.21.15.32:0/1658518421 shutdown_connections 2024-04-30T08:24:57.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.564+0000 7fc5bc800700 1 -- 172.21.15.32:0/1658518421 wait complete. 2024-04-30T08:24:57.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.565+0000 7fc5bc800700 1 Processor -- start 2024-04-30T08:24:57.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.566+0000 7fc5bc800700 1 -- start start 2024-04-30T08:24:57.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.566+0000 7fc5bc800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5b4104290 0x7fc5b419cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:24:57.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.566+0000 7fc5bc800700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5b41053c0 0x7fc5b419cff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:24:57.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.566+0000 7fc5bc800700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5b4105d60 0x7fc5b41a3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:24:57.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.566+0000 7fc5bc800700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b410fd30 con 0x7fc5b41053c0 2024-04-30T08:24:57.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.566+0000 7fc5bc800700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b410fbb0 con 0x7fc5b4105d60 2024-04-30T08:24:57.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.566+0000 7fc5bc800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc5b410feb0 con 0x7fc5b4104290 2024-04-30T08:24:57.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.567+0000 7fc5b9e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5b4104290 0x7fc5b419cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:24:57.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.567+0000 7fc5ba800700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5b4105d60 0x7fc5b41a3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:24:57.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.567+0000 7fc5b9e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5b4104290 0x7fc5b419cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:53176/0 (socket says 172.21.15.32:53176) 2024-04-30T08:24:57.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.567+0000 7fc5ba800700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5b4105d60 0x7fc5b41a3f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:43666/0 (socket says 172.21.15.32:43666) 2024-04-30T08:24:57.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.567+0000 7fc5b9e00700 1 -- 172.21.15.32:0/1286958595 learned_addr learned my addr 172.21.15.32:0/1286958595 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:24:57.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.567+0000 7fc5b9400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5b41053c0 0x7fc5b419cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:24:57.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.567+0000 7fc5b9e00700 1 -- 172.21.15.32:0/1286958595 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5b4105d60 msgr2=0x7fc5b41a3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:57.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.567+0000 7fc5b9e00700 1 --2- 172.21.15.32:0/1286958595 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5b4105d60 0x7fc5b41a3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:57.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.567+0000 7fc5b9e00700 1 -- 172.21.15.32:0/1286958595 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5b41053c0 msgr2=0x7fc5b419cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:24:57.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.567+0000 7fc5b9e00700 1 --2- 172.21.15.32:0/1286958595 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5b41053c0 0x7fc5b419cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:24:57.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.567+0000 7fc5b9e00700 1 -- 172.21.15.32:0/1286958595 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5b41a4610 con 0x7fc5b4104290 2024-04-30T08:24:57.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.568+0000 7fc5b9e00700 1 --2- 172.21.15.32:0/1286958595 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5b4104290 0x7fc5b419cab0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc5a800b6c0 tx=0x7fc5a8009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:24:57.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.568+0000 7fc5aea00700 1 -- 172.21.15.32:0/1286958595 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a8026070 con 0x7fc5b4104290 2024-04-30T08:24:57.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.568+0000 7fc5aea00700 1 -- 172.21.15.32:0/1286958595 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5a8021410 con 0x7fc5b4104290 2024-04-30T08:24:57.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.568+0000 7fc5bc800700 1 -- 172.21.15.32:0/1286958595 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5b41a48a0 con 0x7fc5b4104290 2024-04-30T08:24:57.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.568+0000 7fc5aea00700 1 -- 172.21.15.32:0/1286958595 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5a8017650 con 0x7fc5b4104290 2024-04-30T08:24:57.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.568+0000 7fc5bc800700 1 -- 172.21.15.32:0/1286958595 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5b41a4d50 con 0x7fc5b4104290 2024-04-30T08:24:57.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.570+0000 7fc5bc800700 1 -- 172.21.15.32:0/1286958595 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5b4051210 con 0x7fc5b4104290 2024-04-30T08:24:57.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.570+0000 7fc5aea00700 1 -- 172.21.15.32:0/1286958595 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 13) ==== 49572+0+0 (secure 0 0 0) 0x7fc5a8029070 con 0x7fc5b4104290 2024-04-30T08:24:57.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.571+0000 7fc5aea00700 1 --2- 172.21.15.32:0/1286958595 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc59c03d850 0x7fc59c03fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:24:57.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.571+0000 7fc5aea00700 1 -- 172.21.15.32:0/1286958595 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fc5a80526b0 con 0x7fc5b4104290 2024-04-30T08:24:57.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.571+0000 7fc5b9400700 1 --2- 172.21.15.32:0/1286958595 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc59c03d850 0x7fc59c03fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:24:57.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.572+0000 7fc5b9400700 1 --2- 172.21.15.32:0/1286958595 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc59c03d850 0x7fc59c03fd60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc5b419db60 tx=0x7fc5a4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:24:57.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.577+0000 7fc5aea00700 1 -- 172.21.15.32:0/1286958595 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc5a8013070 con 0x7fc5b4104290 2024-04-30T08:24:57.753 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:24:57 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-y[34773]: 2024-04-30T08:24:57.424+0000 7faad382c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-30T08:24:57.753 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:24:57 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-y[34773]: 2024-04-30T08:24:57.595+0000 7faad382c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-30T08:24:57.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:24:57.824+0000 7fc5bc800700 1 -- 172.21.15.32:0/1286958595 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fc5b4077060 con 0x7fc59c03d850 2024-04-30T08:24:58.050 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:24:57 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-y[34773]: 2024-04-30T08:24:57.764+0000 7faad382c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-30T08:24:58.051 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:24:58 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-y[34773]: 2024-04-30T08:24:58.049+0000 7faad382c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-30T08:24:58.287 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:24:57 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-z[34907]: 2024-04-30T08:24:57.953+0000 7fa4be2f8200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-30T08:24:58.287 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:24:58 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-z[34907]: 2024-04-30T08:24:58.044+0000 7fa4be2f8200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-30T08:24:58.287 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:58 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:58.288 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:58 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:58.288 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:58 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:58.288 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:58 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:58.288 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:58 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:58.288 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:58 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:24:58.288 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:58 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:58.288 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:58 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-30T08:24:58.288 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:58 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-30T08:24:58.288 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:58 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:58.288 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:58 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:24:58.288 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:58 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:24:58.288 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:58 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:58.345 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:24:58 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-y[34773]: 2024-04-30T08:24:58.142+0000 7faad382c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-30T08:24:58.345 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:24:58 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-y[34773]: 2024-04-30T08:24:58.344+0000 7faad382c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-30T08:24:58.345 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:58 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:58.345 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:58 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:58.345 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:58 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:58.345 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:58 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:58.345 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:58 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:58.345 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:58 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:24:58.345 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:58 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:58.346 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:58 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-30T08:24:58.346 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:58 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-30T08:24:58.346 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:58 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:58.346 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:58 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:24:58.346 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:58 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:24:58.346 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:58 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:58.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:58 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:58.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:58 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:58.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:58 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:58.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:58 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:58.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:58 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:58.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:58 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:24:58.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:58 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:24:58.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:58 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-30T08:24:58.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:58 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-30T08:24:58.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:58 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:58.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:58 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:24:58.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:58 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:24:58.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:58 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:24:58.550 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:24:58 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-z[34907]: 2024-04-30T08:24:58.286+0000 7fa4be2f8200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-30T08:24:58.550 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:24:58 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-z[34907]: 2024-04-30T08:24:58.418+0000 7fa4be2f8200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-30T08:24:58.550 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:24:58 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-z[34907]: 2024-04-30T08:24:58.511+0000 7fa4be2f8200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-30T08:24:58.753 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:24:58 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-y[34773]: 2024-04-30T08:24:58.533+0000 7faad382c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-30T08:24:58.929 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:24:58 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-z[34907]: 2024-04-30T08:24:58.626+0000 7fa4be2f8200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-30T08:24:58.929 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:24:58 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-z[34907]: 2024-04-30T08:24:58.928+0000 7fa4be2f8200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-30T08:24:59.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:59 smithi189 ceph-mon[33038]: Reconfiguring mgr.x (unknown last config time)... 2024-04-30T08:24:59.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:59 smithi189 ceph-mon[33038]: Reconfiguring daemon mgr.x on smithi032 2024-04-30T08:24:59.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:59 smithi189 ceph-mon[33038]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:59.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:24:59 smithi189 ceph-mon[33038]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:24:59.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:59 smithi032 ceph-mon[28331]: Reconfiguring mgr.x (unknown last config time)... 2024-04-30T08:24:59.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:59 smithi032 ceph-mon[28331]: Reconfiguring daemon mgr.x on smithi032 2024-04-30T08:24:59.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:59 smithi032 ceph-mon[28331]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:59.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:24:59 smithi032 ceph-mon[28331]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:24:59.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:59 smithi071 ceph-mon[33385]: Reconfiguring mgr.x (unknown last config time)... 2024-04-30T08:24:59.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:59 smithi071 ceph-mon[33385]: Reconfiguring daemon mgr.x on smithi032 2024-04-30T08:24:59.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:59 smithi071 ceph-mon[33385]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:24:59.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:24:59 smithi071 ceph-mon[33385]: from='client.24121 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:24:59.504 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:24:59 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-y[34773]: 2024-04-30T08:24:59.129+0000 7faad382c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-30T08:24:59.800 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:24:59 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-z[34907]: 2024-04-30T08:24:59.536+0000 7fa4be2f8200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-30T08:24:59.800 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:24:59 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-z[34907]: 2024-04-30T08:24:59.754+0000 7fa4be2f8200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-30T08:24:59.849 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:24:59 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-y[34773]: 2024-04-30T08:24:59.550+0000 7faad382c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-30T08:24:59.849 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:24:59 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-y[34773]: 2024-04-30T08:24:59.645+0000 7faad382c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-30T08:24:59.849 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:24:59 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-y[34773]: 2024-04-30T08:24:59.847+0000 7faad382c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-30T08:25:00.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:25:00.629+0000 7fc5aea00700 1 -- 172.21.15.32:0/1286958595 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mgrmap(e 14) ==== 98518+0+0 (secure 0 0 0) 0x7fc5a8017d10 con 0x7fc5b4104290 2024-04-30T08:25:00.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:00 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:00.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:00 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:00.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:00 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:25:00.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:00 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:25:00.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:00 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:25:00.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:00 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:00.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:00 smithi032 ceph-mon[28331]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:00.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:00 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:00.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:00 smithi032 ceph-mon[28331]: from='mgr.? 172.21.15.189:0/467361785' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-30T08:25:00.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:00 smithi032 ceph-mon[28331]: from='mgr.? 172.21.15.189:0/467361785' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-30T08:25:00.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:00 smithi032 ceph-mon[28331]: Standby manager daemon z started 2024-04-30T08:25:00.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:00 smithi032 ceph-mon[28331]: from='mgr.? 172.21.15.189:0/467361785' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-30T08:25:00.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:00 smithi032 ceph-mon[28331]: from='mgr.? 172.21.15.189:0/467361785' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-30T08:25:00.722 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:25:00 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-y[34773]: 2024-04-30T08:25:00.386+0000 7faad382c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-30T08:25:00.723 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:25:00 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-y[34773]: 2024-04-30T08:25:00.477+0000 7faad382c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-30T08:25:00.723 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:00 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:00.723 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:00 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:00.723 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:00 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:25:00.723 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:00 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:25:00.723 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:00 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:25:00.723 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:00 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:00.723 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:00 smithi071 ceph-mon[33385]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:00.723 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:00 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:00.723 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:00 smithi071 ceph-mon[33385]: from='mgr.? 172.21.15.189:0/467361785' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-30T08:25:00.724 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:00 smithi071 ceph-mon[33385]: from='mgr.? 172.21.15.189:0/467361785' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-30T08:25:00.724 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:00 smithi071 ceph-mon[33385]: Standby manager daemon z started 2024-04-30T08:25:00.724 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:00 smithi071 ceph-mon[33385]: from='mgr.? 172.21.15.189:0/467361785' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-30T08:25:00.724 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:00 smithi071 ceph-mon[33385]: from='mgr.? 172.21.15.189:0/467361785' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-30T08:25:00.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:00 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:00.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:00 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:00.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:00 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:25:00.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:00 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:25:00.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:00 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:25:00.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:00 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:00.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:00 smithi189 ceph-mon[33038]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:00.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:00 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:00.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:00 smithi189 ceph-mon[33038]: from='mgr.? 172.21.15.189:0/467361785' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-30T08:25:00.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:00 smithi189 ceph-mon[33038]: from='mgr.? 172.21.15.189:0/467361785' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-30T08:25:00.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:00 smithi189 ceph-mon[33038]: Standby manager daemon z started 2024-04-30T08:25:00.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:00 smithi189 ceph-mon[33038]: from='mgr.? 172.21.15.189:0/467361785' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-30T08:25:00.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:00 smithi189 ceph-mon[33038]: from='mgr.? 172.21.15.189:0/467361785' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-30T08:25:01.003 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:25:00 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-y[34773]: 2024-04-30T08:25:00.720+0000 7faad382c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-30T08:25:01.003 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:25:00 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-y[34773]: 2024-04-30T08:25:00.854+0000 7faad382c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-30T08:25:01.003 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:25:00 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-y[34773]: 2024-04-30T08:25:00.949+0000 7faad382c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-30T08:25:01.371 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:25:01 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-y[34773]: 2024-04-30T08:25:01.064+0000 7faad382c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-30T08:25:01.371 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:25:01 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-y[34773]: 2024-04-30T08:25:01.369+0000 7faad382c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-30T08:25:01.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:01 smithi032 ceph-mon[28331]: mgrmap e14: x(active, since 91s), standbys: z 2024-04-30T08:25:01.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:01 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-30T08:25:01.966 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:01 smithi071 ceph-mon[33385]: mgrmap e14: x(active, since 91s), standbys: z 2024-04-30T08:25:01.967 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:01 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-30T08:25:02.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:01 smithi189 ceph-mon[33038]: mgrmap e14: x(active, since 91s), standbys: z 2024-04-30T08:25:02.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:01 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-30T08:25:02.253 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:25:01 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-y[34773]: 2024-04-30T08:25:01.964+0000 7faad382c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-30T08:25:02.253 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:25:02 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-y[34773]: 2024-04-30T08:25:02.179+0000 7faad382c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-30T08:25:02.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:25:02.649+0000 7fc5aea00700 1 -- 172.21.15.32:0/1286958595 <== mon.1 v2:172.21.15.189:3300/0 8 ==== mgrmap(e 15) ==== 147464+0+0 (secure 0 0 0) 0x7fc5a8009d10 con 0x7fc5b4104290 2024-04-30T08:25:02.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:02 smithi032 ceph-mon[28331]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:02.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:02 smithi032 ceph-mon[28331]: from='mgr.? 172.21.15.71:0/3536673330' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-30T08:25:02.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:02 smithi032 ceph-mon[28331]: Standby manager daemon y started 2024-04-30T08:25:02.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:02 smithi032 ceph-mon[28331]: from='mgr.? 172.21.15.71:0/3536673330' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-30T08:25:02.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:02 smithi032 ceph-mon[28331]: from='mgr.? 172.21.15.71:0/3536673330' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-30T08:25:02.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:02 smithi032 ceph-mon[28331]: from='mgr.? 172.21.15.71:0/3536673330' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-30T08:25:03.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:02 smithi071 ceph-mon[33385]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:03.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:02 smithi071 ceph-mon[33385]: from='mgr.? 172.21.15.71:0/3536673330' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-30T08:25:03.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:02 smithi071 ceph-mon[33385]: Standby manager daemon y started 2024-04-30T08:25:03.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:02 smithi071 ceph-mon[33385]: from='mgr.? 172.21.15.71:0/3536673330' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-30T08:25:03.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:02 smithi071 ceph-mon[33385]: from='mgr.? 172.21.15.71:0/3536673330' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-30T08:25:03.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:02 smithi071 ceph-mon[33385]: from='mgr.? 172.21.15.71:0/3536673330' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-30T08:25:03.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:02 smithi189 ceph-mon[33038]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:03.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:02 smithi189 ceph-mon[33038]: from='mgr.? 172.21.15.71:0/3536673330' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-30T08:25:03.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:02 smithi189 ceph-mon[33038]: Standby manager daemon y started 2024-04-30T08:25:03.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:02 smithi189 ceph-mon[33038]: from='mgr.? 172.21.15.71:0/3536673330' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-30T08:25:03.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:02 smithi189 ceph-mon[33038]: from='mgr.? 172.21.15.71:0/3536673330' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-30T08:25:03.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:02 smithi189 ceph-mon[33038]: from='mgr.? 172.21.15.71:0/3536673330' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-30T08:25:03.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:03 smithi032 ceph-mon[28331]: mgrmap e15: x(active, since 93s), standbys: z, y 2024-04-30T08:25:03.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:03 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-30T08:25:03.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:03 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1749213771' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7e060c8-1cbb-418f-9a03-06693f6b3f52"}]: dispatch 2024-04-30T08:25:03.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:03 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1749213771' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a7e060c8-1cbb-418f-9a03-06693f6b3f52"}]': finished 2024-04-30T08:25:03.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:03 smithi032 ceph-mon[28331]: osdmap e5: 1 total, 0 up, 1 in 2024-04-30T08:25:03.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:03 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-30T08:25:04.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:03 smithi071 ceph-mon[33385]: mgrmap e15: x(active, since 93s), standbys: z, y 2024-04-30T08:25:04.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:03 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-30T08:25:04.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:03 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1749213771' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7e060c8-1cbb-418f-9a03-06693f6b3f52"}]: dispatch 2024-04-30T08:25:04.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:03 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1749213771' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a7e060c8-1cbb-418f-9a03-06693f6b3f52"}]': finished 2024-04-30T08:25:04.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:03 smithi071 ceph-mon[33385]: osdmap e5: 1 total, 0 up, 1 in 2024-04-30T08:25:04.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:03 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-30T08:25:04.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:03 smithi189 ceph-mon[33038]: mgrmap e15: x(active, since 93s), standbys: z, y 2024-04-30T08:25:04.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:03 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-30T08:25:04.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:03 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1749213771' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a7e060c8-1cbb-418f-9a03-06693f6b3f52"}]: dispatch 2024-04-30T08:25:04.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:03 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1749213771' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a7e060c8-1cbb-418f-9a03-06693f6b3f52"}]': finished 2024-04-30T08:25:04.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:03 smithi189 ceph-mon[33038]: osdmap e5: 1 total, 0 up, 1 in 2024-04-30T08:25:04.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:03 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-30T08:25:04.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:04 smithi032 ceph-mon[28331]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:04.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:04 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3016268561' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:25:05.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:04 smithi071 ceph-mon[33385]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:05.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:04 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3016268561' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:25:05.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:04 smithi189 ceph-mon[33038]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:05.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:04 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3016268561' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:25:06.818 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:06 smithi189 ceph-mon[33038]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:06.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:06 smithi032 ceph-mon[28331]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:07.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:06 smithi071 ceph-mon[33385]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:08.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:08 smithi032 ceph-mon[28331]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:09.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:08 smithi071 ceph-mon[33385]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:09.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:08 smithi189 ceph-mon[33038]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:10.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:10 smithi032 ceph-mon[28331]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:11.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:10 smithi071 ceph-mon[33385]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:11.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:10 smithi189 ceph-mon[33038]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:12.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:12 smithi032 ceph-mon[28331]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:13.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:12 smithi071 ceph-mon[33385]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:13.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:12 smithi189 ceph-mon[33038]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:13.931 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:13 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-30T08:25:13.931 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:13 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:25:13.931 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:13 smithi032 ceph-mon[28331]: Deploying daemon osd.0 on smithi032 2024-04-30T08:25:14.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:13 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-30T08:25:14.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:13 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:25:14.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:13 smithi071 ceph-mon[33385]: Deploying daemon osd.0 on smithi032 2024-04-30T08:25:14.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:13 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-30T08:25:14.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:13 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:25:14.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:13 smithi189 ceph-mon[33038]: Deploying daemon osd.0 on smithi032 2024-04-30T08:25:14.907 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:14 smithi032 ceph-mon[28331]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:15.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:14 smithi071 ceph-mon[33385]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:15.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:14 smithi189 ceph-mon[33038]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:16.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:16 smithi032 ceph-mon[28331]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:17.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:16 smithi071 ceph-mon[33385]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:17.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:16 smithi189 ceph-mon[33038]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:18.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:18 smithi032 ceph-mon[28331]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:19.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:18 smithi071 ceph-mon[33385]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:19.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:18 smithi189 ceph-mon[33038]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:20.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:20 smithi032 ceph-mon[28331]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:21.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:20 smithi071 ceph-mon[33385]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:21.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:20 smithi189 ceph-mon[33038]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:23.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:23 smithi032 ceph-mon[28331]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:23.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:23 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:25:23.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:23 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:23.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:23 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:23.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:23 smithi071 ceph-mon[33385]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:23.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:23 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:25:23.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:23 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:23.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:23 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:23.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:23 smithi189 ceph-mon[33038]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:23.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:23 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:25:23.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:23 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:23.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:23 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:25.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:25 smithi032 ceph-mon[28331]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:25.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:25 smithi071 ceph-mon[33385]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:25.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:25 smithi189 ceph-mon[33038]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:26.954 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:26 smithi032 ceph-mon[28331]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:26.954 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:26 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:26.954 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:26 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:26.954 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:26 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:25:26.954 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:26 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:25:26.954 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:26 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:26.954 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:26 smithi032 ceph-mon[28331]: from='osd.0 [v2:172.21.15.32:6802/2624700402,v1:172.21.15.32:6803/2624700402]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-30T08:25:26.954 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:26 smithi032 ceph-mon[28331]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-30T08:25:27.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:26 smithi071 ceph-mon[33385]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:27.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:26 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:27.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:26 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:27.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:26 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:25:27.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:26 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:25:27.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:26 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:27.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:26 smithi071 ceph-mon[33385]: from='osd.0 [v2:172.21.15.32:6802/2624700402,v1:172.21.15.32:6803/2624700402]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-30T08:25:27.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:26 smithi071 ceph-mon[33385]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-30T08:25:27.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:26 smithi189 ceph-mon[33038]: pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:27.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:26 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:27.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:26 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:27.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:26 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:25:27.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:26 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:25:27.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:26 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:27.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:26 smithi189 ceph-mon[33038]: from='osd.0 [v2:172.21.15.32:6802/2624700402,v1:172.21.15.32:6803/2624700402]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-30T08:25:27.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:26 smithi189 ceph-mon[33038]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-30T08:25:27.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:25:27.833+0000 7fc5aea00700 1 -- 172.21.15.32:0/1286958595 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fc5b4077060 con 0x7fc59c03d850 2024-04-30T08:25:27.835 INFO:teuthology.orchestra.run.smithi032.stdout:Created osd(s) 0 on host 'smithi032' 2024-04-30T08:25:27.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:25:27.837+0000 7fc5bc800700 1 -- 172.21.15.32:0/1286958595 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc59c03d850 msgr2=0x7fc59c03fd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:25:27.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:25:27.837+0000 7fc5bc800700 1 --2- 172.21.15.32:0/1286958595 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc59c03d850 0x7fc59c03fd60 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fc5b419db60 tx=0x7fc5a4009260 comp rx=0 tx=0).stop 2024-04-30T08:25:27.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:25:27.837+0000 7fc5bc800700 1 -- 172.21.15.32:0/1286958595 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5b4104290 msgr2=0x7fc5b419cab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:25:27.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:25:27.837+0000 7fc5bc800700 1 --2- 172.21.15.32:0/1286958595 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5b4104290 0x7fc5b419cab0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fc5a800b6c0 tx=0x7fc5a8009580 comp rx=0 tx=0).stop 2024-04-30T08:25:27.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:25:27.837+0000 7fc5bc800700 1 -- 172.21.15.32:0/1286958595 shutdown_connections 2024-04-30T08:25:27.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:25:27.837+0000 7fc5bc800700 1 --2- 172.21.15.32:0/1286958595 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc59c03d850 0x7fc59c03fd60 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:25:27.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:25:27.837+0000 7fc5bc800700 1 --2- 172.21.15.32:0/1286958595 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5b4105d60 0x7fc5b41a3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:25:27.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:25:27.837+0000 7fc5bc800700 1 --2- 172.21.15.32:0/1286958595 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5b4104290 0x7fc5b419cab0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:25:27.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:25:27.837+0000 7fc5bc800700 1 --2- 172.21.15.32:0/1286958595 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5b41053c0 0x7fc5b419cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:25:27.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:25:27.837+0000 7fc5bc800700 1 -- 172.21.15.32:0/1286958595 >> 172.21.15.32:0/1286958595 conn(0x7fc5b40ff910 msgr2=0x7fc5b4075940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:25:27.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:25:27.837+0000 7fc5bc800700 1 -- 172.21.15.32:0/1286958595 shutdown_connections 2024-04-30T08:25:27.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:25:27.837+0000 7fc5bc800700 1 -- 172.21.15.32:0/1286958595 wait complete. 2024-04-30T08:25:28.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:28 smithi071 ceph-mon[33385]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-30T08:25:28.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:28 smithi071 ceph-mon[33385]: from='osd.0 [v2:172.21.15.32:6802/2624700402,v1:172.21.15.32:6803/2624700402]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-30T08:25:28.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:28 smithi071 ceph-mon[33385]: osdmap e6: 1 total, 0 up, 1 in 2024-04-30T08:25:28.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:28 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-30T08:25:28.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:28 smithi071 ceph-mon[33385]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-30T08:25:28.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:28 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:28.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:28 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:28.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:28 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:25:28.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:28 smithi189 ceph-mon[33038]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-30T08:25:28.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:28 smithi189 ceph-mon[33038]: from='osd.0 [v2:172.21.15.32:6802/2624700402,v1:172.21.15.32:6803/2624700402]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-30T08:25:28.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:28 smithi189 ceph-mon[33038]: osdmap e6: 1 total, 0 up, 1 in 2024-04-30T08:25:28.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:28 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-30T08:25:28.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:28 smithi189 ceph-mon[33038]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-30T08:25:28.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:28 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:28.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:28 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:28.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:28 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:25:28.608 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:28 smithi032 ceph-mon[28331]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-30T08:25:28.608 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:28 smithi032 ceph-mon[28331]: from='osd.0 [v2:172.21.15.32:6802/2624700402,v1:172.21.15.32:6803/2624700402]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-30T08:25:28.608 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:28 smithi032 ceph-mon[28331]: osdmap e6: 1 total, 0 up, 1 in 2024-04-30T08:25:28.609 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:28 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-30T08:25:28.609 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:28 smithi032 ceph-mon[28331]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-30T08:25:28.609 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:28 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:28.609 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:28 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:28.609 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:28 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:25:28.663 DEBUG:teuthology.orchestra.run.smithi032:osd.0> sudo journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.0.service 2024-04-30T08:25:28.668 INFO:tasks.cephadm:Deploying osd.1 on smithi071 with /dev/vg_nvme/lv_4... 2024-04-30T08:25:28.668 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-30T08:25:28.911 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.b/config 2024-04-30T08:25:29.132 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 30 08:25:28 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-0[43279]: 2024-04-30T08:25:28.876+0000 7f7f33200700 -1 osd.0 0 waiting for initial osdmap 2024-04-30T08:25:29.132 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 30 08:25:28 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-0[43279]: 2024-04-30T08:25:28.896+0000 7f7f29000700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-30T08:25:29.294 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:29 smithi071 ceph-mon[33385]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:29.294 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:29 smithi071 ceph-mon[33385]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-30T08:25:29.294 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:29 smithi071 ceph-mon[33385]: osdmap e7: 1 total, 0 up, 1 in 2024-04-30T08:25:29.294 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:29 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-30T08:25:29.294 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:29 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-30T08:25:29.294 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:29 smithi071 ceph-mon[33385]: from='osd.0 ' entity='osd.0' 2024-04-30T08:25:29.412 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:29 smithi032 ceph-mon[28331]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:29.413 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:29 smithi032 ceph-mon[28331]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-30T08:25:29.413 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:29 smithi032 ceph-mon[28331]: osdmap e7: 1 total, 0 up, 1 in 2024-04-30T08:25:29.413 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:29 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-30T08:25:29.413 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:29 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-30T08:25:29.413 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:29 smithi032 ceph-mon[28331]: from='osd.0 ' entity='osd.0' 2024-04-30T08:25:29.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:29 smithi189 ceph-mon[33038]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:29.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:29 smithi189 ceph-mon[33038]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-30T08:25:29.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:29 smithi189 ceph-mon[33038]: osdmap e7: 1 total, 0 up, 1 in 2024-04-30T08:25:29.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:29 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-30T08:25:29.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:29 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-30T08:25:29.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:29 smithi189 ceph-mon[33038]: from='osd.0 ' entity='osd.0' 2024-04-30T08:25:30.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:30 smithi032 ceph-mon[28331]: purged_snaps scrub starts 2024-04-30T08:25:30.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:30 smithi032 ceph-mon[28331]: purged_snaps scrub ok 2024-04-30T08:25:30.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:30 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-30T08:25:30.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:30 smithi032 ceph-mon[28331]: osd.0 [v2:172.21.15.32:6802/2624700402,v1:172.21.15.32:6803/2624700402] boot 2024-04-30T08:25:30.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:30 smithi032 ceph-mon[28331]: osdmap e8: 1 total, 1 up, 1 in 2024-04-30T08:25:30.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:30 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-30T08:25:30.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:30 smithi071 ceph-mon[33385]: purged_snaps scrub starts 2024-04-30T08:25:30.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:30 smithi071 ceph-mon[33385]: purged_snaps scrub ok 2024-04-30T08:25:30.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:30 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-30T08:25:30.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:30 smithi071 ceph-mon[33385]: osd.0 [v2:172.21.15.32:6802/2624700402,v1:172.21.15.32:6803/2624700402] boot 2024-04-30T08:25:30.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:30 smithi071 ceph-mon[33385]: osdmap e8: 1 total, 1 up, 1 in 2024-04-30T08:25:30.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:30 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-30T08:25:30.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:30 smithi189 ceph-mon[33038]: purged_snaps scrub starts 2024-04-30T08:25:30.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:30 smithi189 ceph-mon[33038]: purged_snaps scrub ok 2024-04-30T08:25:30.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:30 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-30T08:25:30.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:30 smithi189 ceph-mon[33038]: osd.0 [v2:172.21.15.32:6802/2624700402,v1:172.21.15.32:6803/2624700402] boot 2024-04-30T08:25:30.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:30 smithi189 ceph-mon[33038]: osdmap e8: 1 total, 1 up, 1 in 2024-04-30T08:25:30.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:30 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-30T08:25:30.735 INFO:teuthology.orchestra.run.smithi071.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-30T08:25:30.735 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-30T08:25:30.735 INFO:teuthology.orchestra.run.smithi071.stderr: stderr: 10+0 records in 2024-04-30T08:25:30.735 INFO:teuthology.orchestra.run.smithi071.stderr:10+0 records out 2024-04-30T08:25:30.736 INFO:teuthology.orchestra.run.smithi071.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113942 s, 920 MB/s 2024-04-30T08:25:30.736 INFO:teuthology.orchestra.run.smithi071.stderr:--> Zapping successful for: 2024-04-30T08:25:31.378 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:31 smithi071 ceph-mon[33385]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:31.379 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:31 smithi071 ceph-mon[33385]: osdmap e9: 1 total, 1 up, 1 in 2024-04-30T08:25:31.408 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi071:vg_nvme/lv_4 2024-04-30T08:25:31.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:31 smithi189 ceph-mon[33038]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:31.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:31 smithi189 ceph-mon[33038]: osdmap e9: 1 total, 1 up, 1 in 2024-04-30T08:25:31.650 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.b/config 2024-04-30T08:25:31.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:31 smithi032 ceph-mon[28331]: pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-30T08:25:31.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:31 smithi032 ceph-mon[28331]: osdmap e9: 1 total, 1 up, 1 in 2024-04-30T08:25:33.092 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.090+0000 7f6c289a2700 1 -- 172.21.15.71:0/2106092138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c241022a0 msgr2=0x7f6c241026b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:25:33.092 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.090+0000 7f6c289a2700 1 --2- 172.21.15.71:0/2106092138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c241022a0 0x7f6c241026b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6c18009990 tx=0x7f6c18009ca0 comp rx=0 tx=0).stop 2024-04-30T08:25:33.092 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.090+0000 7f6c289a2700 1 -- 172.21.15.71:0/2106092138 shutdown_connections 2024-04-30T08:25:33.092 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.090+0000 7f6c289a2700 1 --2- 172.21.15.71:0/2106092138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c24102bf0 0x7f6c2410c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:25:33.092 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.090+0000 7f6c289a2700 1 --2- 172.21.15.71:0/2106092138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c24105850 0x7f6c24105c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:25:33.092 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.090+0000 7f6c289a2700 1 --2- 172.21.15.71:0/2106092138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c241022a0 0x7f6c241026b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:25:33.092 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.090+0000 7f6c289a2700 1 -- 172.21.15.71:0/2106092138 >> 172.21.15.71:0/2106092138 conn(0x7f6c240fdf70 msgr2=0x7f6c24100420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:25:33.092 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.090+0000 7f6c289a2700 1 -- 172.21.15.71:0/2106092138 shutdown_connections 2024-04-30T08:25:33.092 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.090+0000 7f6c289a2700 1 -- 172.21.15.71:0/2106092138 wait complete. 2024-04-30T08:25:33.092 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.091+0000 7f6c289a2700 1 Processor -- start 2024-04-30T08:25:33.092 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.091+0000 7f6c289a2700 1 -- start start 2024-04-30T08:25:33.094 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.091+0000 7f6c289a2700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c241022a0 0x7f6c2419a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:25:33.094 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.091+0000 7f6c289a2700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c24102bf0 0x7f6c2419ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:25:33.094 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.091+0000 7f6c289a2700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c24105850 0x7f6c241a1e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:25:33.095 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.091+0000 7f6c289a2700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6c2410fde0 con 0x7f6c24102bf0 2024-04-30T08:25:33.095 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.091+0000 7f6c289a2700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6c2410fc60 con 0x7f6c24105850 2024-04-30T08:25:33.095 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.091+0000 7f6c289a2700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6c2410ff60 con 0x7f6c241022a0 2024-04-30T08:25:33.095 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.091+0000 7f6c22d9d700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c24105850 0x7f6c241a1e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:25:33.095 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.091+0000 7f6c22d9d700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c24105850 0x7f6c241a1e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:57094/0 (socket says 172.21.15.71:57094) 2024-04-30T08:25:33.095 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.091+0000 7f6c22d9d700 1 -- 172.21.15.71:0/2216663946 learned_addr learned my addr 172.21.15.71:0/2216663946 (peer_addr_for_me v2:172.21.15.71:0/0) 2024-04-30T08:25:33.095 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.091+0000 7f6c2259c700 1 --2- 172.21.15.71:0/2216663946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c241022a0 0x7f6c2419a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:25:33.095 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.092+0000 7f6c22d9d700 1 -- 172.21.15.71:0/2216663946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c241022a0 msgr2=0x7f6c2419a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:25:33.095 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.092+0000 7f6c22d9d700 1 --2- 172.21.15.71:0/2216663946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c241022a0 0x7f6c2419a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:25:33.095 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.092+0000 7f6c22d9d700 1 -- 172.21.15.71:0/2216663946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c24102bf0 msgr2=0x7f6c2419ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:25:33.095 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.092+0000 7f6c22d9d700 1 --2- 172.21.15.71:0/2216663946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c24102bf0 0x7f6c2419ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:25:33.095 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.092+0000 7f6c22d9d700 1 -- 172.21.15.71:0/2216663946 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c241a2480 con 0x7f6c24105850 2024-04-30T08:25:33.096 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.092+0000 7f6c2259c700 1 --2- 172.21.15.71:0/2216663946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c241022a0 0x7f6c2419a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-30T08:25:33.096 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.092+0000 7f6c22d9d700 1 --2- 172.21.15.71:0/2216663946 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c24105850 0x7f6c241a1e00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c00b970 tx=0x7f6c1c00bd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:25:33.096 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.092+0000 7f6c137fe700 1 -- 172.21.15.71:0/2216663946 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c1c019070 con 0x7f6c24105850 2024-04-30T08:25:33.096 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.092+0000 7f6c137fe700 1 -- 172.21.15.71:0/2216663946 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c1c014440 con 0x7f6c24105850 2024-04-30T08:25:33.096 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.092+0000 7f6c137fe700 1 -- 172.21.15.71:0/2216663946 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c1c00ce60 con 0x7f6c24105850 2024-04-30T08:25:33.096 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.092+0000 7f6c289a2700 1 -- 172.21.15.71:0/2216663946 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c241a2770 con 0x7f6c24105850 2024-04-30T08:25:33.096 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.092+0000 7f6c289a2700 1 -- 172.21.15.71:0/2216663946 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c241a2d80 con 0x7f6c24105850 2024-04-30T08:25:33.096 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.093+0000 7f6c289a2700 1 -- 172.21.15.71:0/2216663946 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c24051210 con 0x7f6c24105850 2024-04-30T08:25:33.097 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.094+0000 7f6c137fe700 1 -- 172.21.15.71:0/2216663946 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 15) ==== 147464+0+0 (secure 0 0 0) 0x7f6c1c01c070 con 0x7f6c24105850 2024-04-30T08:25:33.097 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.095+0000 7f6c137fe700 1 --2- 172.21.15.71:0/2216663946 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f6c0c0afb70 0x7f6c0c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:25:33.097 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.095+0000 7f6c137fe700 1 -- 172.21.15.71:0/2216663946 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f6c1c0dc9e0 con 0x7f6c24105850 2024-04-30T08:25:33.097 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.095+0000 7f6c2259c700 1 --2- 172.21.15.71:0/2216663946 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f6c0c0afb70 0x7f6c0c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:25:33.097 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.096+0000 7f6c2259c700 1 --2- 172.21.15.71:0/2216663946 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f6c0c0afb70 0x7f6c0c0b2080 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6c14005fd0 tx=0x7f6c14005dc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:25:33.099 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.099+0000 7f6c137fe700 1 -- 172.21.15.71:0/2216663946 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6c1c0a7030 con 0x7f6c24105850 2024-04-30T08:25:33.307 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:33.306+0000 7f6c289a2700 1 -- 172.21.15.71:0/2216663946 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f6c24063b50 con 0x7f6c0c0afb70 2024-04-30T08:25:33.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:33 smithi071 ceph-mon[33385]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:33.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:33 smithi189 ceph-mon[33038]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:33.599 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:33 smithi032 ceph-mon[28331]: pgmap v60: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:34.484 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:34 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:25:34.485 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:34 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:25:34.485 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:34 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:25:34.485 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:34 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:34.485 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:34 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:34.485 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:34 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:25:34.485 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:34 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:34.485 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:34 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:25:34.485 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:34 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:25:34.485 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:34 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:34.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:34 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:25:35.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:34 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:25:35.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:34 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:25:35.195 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:34 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:35.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:34 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:35.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:34 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:25:35.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:34 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:35.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:34 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:25:35.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:34 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:25:35.196 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:34 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:35.197 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:34 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:25:35.197 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:34 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:25:35.198 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:34 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:25:35.198 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:34 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:35.198 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:34 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:35.198 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:34 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:25:35.198 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:34 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:35.198 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:34 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:25:35.198 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:34 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:25:35.198 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:34 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:35.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:35 smithi071 ceph-mon[33385]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:25:35.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:35 smithi071 ceph-mon[33385]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:35.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:35 smithi071 ceph-mon[33385]: Detected new or changed devices on smithi032 2024-04-30T08:25:35.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:35 smithi071 ceph-mon[33385]: Adjusting osd_memory_target on smithi032 to 17094M 2024-04-30T08:25:35.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:35 smithi189 ceph-mon[33038]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:25:35.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:35 smithi189 ceph-mon[33038]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:35.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:35 smithi189 ceph-mon[33038]: Detected new or changed devices on smithi032 2024-04-30T08:25:35.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:35 smithi189 ceph-mon[33038]: Adjusting osd_memory_target on smithi032 to 17094M 2024-04-30T08:25:35.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:35 smithi032 ceph-mon[28331]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:25:35.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:35 smithi032 ceph-mon[28331]: pgmap v61: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:35.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:35 smithi032 ceph-mon[28331]: Detected new or changed devices on smithi032 2024-04-30T08:25:35.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:35 smithi032 ceph-mon[28331]: Adjusting osd_memory_target on smithi032 to 17094M 2024-04-30T08:25:36.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:36 smithi071 ceph-mon[33385]: from='client.? 172.21.15.71:0/2981904670' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72ef2fb1-e5ca-48ef-aa3a-4796a64b922f"}]: dispatch 2024-04-30T08:25:36.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:36 smithi071 ceph-mon[33385]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72ef2fb1-e5ca-48ef-aa3a-4796a64b922f"}]: dispatch 2024-04-30T08:25:36.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:36 smithi071 ceph-mon[33385]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "72ef2fb1-e5ca-48ef-aa3a-4796a64b922f"}]': finished 2024-04-30T08:25:36.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:36 smithi071 ceph-mon[33385]: osdmap e10: 2 total, 1 up, 2 in 2024-04-30T08:25:36.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:36 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-30T08:25:36.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:36 smithi189 ceph-mon[33038]: from='client.? 172.21.15.71:0/2981904670' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72ef2fb1-e5ca-48ef-aa3a-4796a64b922f"}]: dispatch 2024-04-30T08:25:36.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:36 smithi189 ceph-mon[33038]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72ef2fb1-e5ca-48ef-aa3a-4796a64b922f"}]: dispatch 2024-04-30T08:25:36.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:36 smithi189 ceph-mon[33038]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "72ef2fb1-e5ca-48ef-aa3a-4796a64b922f"}]': finished 2024-04-30T08:25:36.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:36 smithi189 ceph-mon[33038]: osdmap e10: 2 total, 1 up, 2 in 2024-04-30T08:25:36.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:36 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-30T08:25:36.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:36 smithi032 ceph-mon[28331]: from='client.? 172.21.15.71:0/2981904670' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72ef2fb1-e5ca-48ef-aa3a-4796a64b922f"}]: dispatch 2024-04-30T08:25:36.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:36 smithi032 ceph-mon[28331]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "72ef2fb1-e5ca-48ef-aa3a-4796a64b922f"}]: dispatch 2024-04-30T08:25:36.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:36 smithi032 ceph-mon[28331]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "72ef2fb1-e5ca-48ef-aa3a-4796a64b922f"}]': finished 2024-04-30T08:25:36.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:36 smithi032 ceph-mon[28331]: osdmap e10: 2 total, 1 up, 2 in 2024-04-30T08:25:36.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:36 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-30T08:25:37.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:37 smithi071 ceph-mon[33385]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:37.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:37 smithi071 ceph-mon[33385]: from='client.? 172.21.15.71:0/4013840642' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:25:37.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:37 smithi189 ceph-mon[33038]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:37.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:37 smithi189 ceph-mon[33038]: from='client.? 172.21.15.71:0/4013840642' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:25:37.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:37 smithi032 ceph-mon[28331]: pgmap v62: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:37.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:37 smithi032 ceph-mon[28331]: from='client.? 172.21.15.71:0/4013840642' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:25:39.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:39 smithi189 ceph-mon[33038]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:39.558 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:39 smithi071 ceph-mon[33385]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:39.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:39 smithi032 ceph-mon[28331]: pgmap v64: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:41.447 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:41 smithi071 ceph-mon[33385]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:41.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:41 smithi189 ceph-mon[33038]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:41.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:41 smithi032 ceph-mon[28331]: pgmap v65: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:42.472 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:42 smithi071 ceph-mon[33385]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:42.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:42 smithi032 ceph-mon[28331]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:42.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:42 smithi189 ceph-mon[33038]: pgmap v66: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:43.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:43 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-30T08:25:43.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:43 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:25:43.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:43 smithi032 ceph-mon[28331]: Deploying daemon osd.1 on smithi071 2024-04-30T08:25:43.697 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:43 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-30T08:25:43.697 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:43 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:25:43.697 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:43 smithi071 ceph-mon[33385]: Deploying daemon osd.1 on smithi071 2024-04-30T08:25:43.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:43 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-30T08:25:43.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:43 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:25:43.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:43 smithi189 ceph-mon[33038]: Deploying daemon osd.1 on smithi071 2024-04-30T08:25:44.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:44 smithi032 ceph-mon[28331]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:44.696 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:44 smithi071 ceph-mon[33385]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:44.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:44 smithi189 ceph-mon[33038]: pgmap v67: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:46.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:46 smithi032 ceph-mon[28331]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:47.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:46 smithi071 ceph-mon[33385]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:47.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:46 smithi189 ceph-mon[33038]: pgmap v68: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:48.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:48 smithi032 ceph-mon[28331]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:48.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:48 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:48.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:48 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:48.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:48 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:25:49.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:48 smithi071 ceph-mon[33385]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:49.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:48 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:49.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:48 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:49.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:48 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:25:49.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:48 smithi189 ceph-mon[33038]: pgmap v69: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:49.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:48 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:49.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:48 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:49.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:48 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:25:50.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:50 smithi032 ceph-mon[28331]: pgmap v70: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:51.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:50 smithi071 ceph-mon[33385]: pgmap v70: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:51.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:50 smithi189 ceph-mon[33038]: pgmap v70: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:52.199 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:52.198+0000 7f6c137fe700 1 -- 172.21.15.71:0/2216663946 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6c24063b50 con 0x7f6c0c0afb70 2024-04-30T08:25:52.199 INFO:teuthology.orchestra.run.smithi071.stdout:Created osd(s) 1 on host 'smithi071' 2024-04-30T08:25:52.201 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:52.200+0000 7f6c289a2700 1 -- 172.21.15.71:0/2216663946 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f6c0c0afb70 msgr2=0x7f6c0c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:25:52.201 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:52.200+0000 7f6c289a2700 1 --2- 172.21.15.71:0/2216663946 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f6c0c0afb70 0x7f6c0c0b2080 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f6c14005fd0 tx=0x7f6c14005dc0 comp rx=0 tx=0).stop 2024-04-30T08:25:52.201 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:52.200+0000 7f6c289a2700 1 -- 172.21.15.71:0/2216663946 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c24105850 msgr2=0x7f6c241a1e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:25:52.201 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:52.200+0000 7f6c289a2700 1 --2- 172.21.15.71:0/2216663946 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c24105850 0x7f6c241a1e00 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c00b970 tx=0x7f6c1c00bd30 comp rx=0 tx=0).stop 2024-04-30T08:25:52.201 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:52.200+0000 7f6c289a2700 1 -- 172.21.15.71:0/2216663946 shutdown_connections 2024-04-30T08:25:52.201 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:52.200+0000 7f6c289a2700 1 --2- 172.21.15.71:0/2216663946 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f6c0c0afb70 0x7f6c0c0b2080 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:25:52.201 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:52.200+0000 7f6c289a2700 1 --2- 172.21.15.71:0/2216663946 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c24105850 0x7f6c241a1e00 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:25:52.201 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:52.200+0000 7f6c289a2700 1 --2- 172.21.15.71:0/2216663946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c241022a0 0x7f6c2419a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:25:52.201 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:52.200+0000 7f6c289a2700 1 --2- 172.21.15.71:0/2216663946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c24102bf0 0x7f6c2419ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:25:52.201 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:52.200+0000 7f6c289a2700 1 -- 172.21.15.71:0/2216663946 >> 172.21.15.71:0/2216663946 conn(0x7f6c240fdf70 msgr2=0x7f6c241087f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:25:52.201 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:52.200+0000 7f6c289a2700 1 -- 172.21.15.71:0/2216663946 shutdown_connections 2024-04-30T08:25:52.201 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:25:52.200+0000 7f6c289a2700 1 -- 172.21.15.71:0/2216663946 wait complete. 2024-04-30T08:25:52.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:52 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:52.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:52 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:52.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:52 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:25:52.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:52 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:25:52.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:52 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:52.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:52 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:25:52.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:52 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:52.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:52 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:52.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:52 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:52.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:52 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:52.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:52 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:25:52.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:52 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:25:52.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:52 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:52.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:52 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:25:52.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:52 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:52.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:52 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:52.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:52.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:52.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:25:52.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:25:52.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:52.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:25:52.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:52.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:52.818 DEBUG:teuthology.orchestra.run.smithi071:osd.1> sudo journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.1.service 2024-04-30T08:25:52.820 INFO:tasks.cephadm:Deploying osd.2 on smithi189 with /dev/vg_nvme/lv_4... 2024-04-30T08:25:52.820 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-30T08:25:53.063 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.c/config 2024-04-30T08:25:53.447 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:53 smithi189 ceph-mon[33038]: pgmap v71: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:53.447 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:53 smithi189 ceph-mon[33038]: from='osd.1 [v2:172.21.15.71:6800/444649711,v1:172.21.15.71:6801/444649711]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-30T08:25:53.447 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:53 smithi189 ceph-mon[33038]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-30T08:25:53.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:53 smithi071 ceph-mon[33385]: pgmap v71: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:53.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:53 smithi071 ceph-mon[33385]: from='osd.1 [v2:172.21.15.71:6800/444649711,v1:172.21.15.71:6801/444649711]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-30T08:25:53.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:53 smithi071 ceph-mon[33385]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-30T08:25:53.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:53 smithi032 ceph-mon[28331]: pgmap v71: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:53.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:53 smithi032 ceph-mon[28331]: from='osd.1 [v2:172.21.15.71:6800/444649711,v1:172.21.15.71:6801/444649711]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-30T08:25:53.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:53 smithi032 ceph-mon[28331]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-30T08:25:54.292 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:54 smithi189 ceph-mon[33038]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-30T08:25:54.292 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:54 smithi189 ceph-mon[33038]: from='osd.1 [v2:172.21.15.71:6800/444649711,v1:172.21.15.71:6801/444649711]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-04-30T08:25:54.292 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:54 smithi189 ceph-mon[33038]: osdmap e11: 2 total, 1 up, 2 in 2024-04-30T08:25:54.292 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-30T08:25:54.292 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:54 smithi189 ceph-mon[33038]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-04-30T08:25:54.292 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:54 smithi189 ceph-mon[33038]: pgmap v73: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:54.498 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:54 smithi071 ceph-mon[33385]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-30T08:25:54.498 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:54 smithi071 ceph-mon[33385]: from='osd.1 [v2:172.21.15.71:6800/444649711,v1:172.21.15.71:6801/444649711]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-04-30T08:25:54.498 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:54 smithi071 ceph-mon[33385]: osdmap e11: 2 total, 1 up, 2 in 2024-04-30T08:25:54.499 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-30T08:25:54.499 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:54 smithi071 ceph-mon[33385]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-04-30T08:25:54.499 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:54 smithi071 ceph-mon[33385]: pgmap v73: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:54.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:54 smithi032 ceph-mon[28331]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-30T08:25:54.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:54 smithi032 ceph-mon[28331]: from='osd.1 [v2:172.21.15.71:6800/444649711,v1:172.21.15.71:6801/444649711]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-04-30T08:25:54.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:54 smithi032 ceph-mon[28331]: osdmap e11: 2 total, 1 up, 2 in 2024-04-30T08:25:54.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-30T08:25:54.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:54 smithi032 ceph-mon[28331]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-04-30T08:25:54.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:54 smithi032 ceph-mon[28331]: pgmap v73: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:54.848 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-30T08:25:54.848 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-30T08:25:54.848 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-04-30T08:25:54.848 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-04-30T08:25:54.848 INFO:teuthology.orchestra.run.smithi189.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0141329 s, 742 MB/s 2024-04-30T08:25:54.848 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-04-30T08:25:55.090 INFO:journalctl@ceph.osd.1.smithi071.stdout:Apr 30 08:25:54 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-1[37970]: 2024-04-30T08:25:54.825+0000 7f8effeaa700 -1 osd.1 0 waiting for initial osdmap 2024-04-30T08:25:55.090 INFO:journalctl@ceph.osd.1.smithi071.stdout:Apr 30 08:25:54 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-1[37970]: 2024-04-30T08:25:54.828+0000 7f8ef7c57700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-30T08:25:55.407 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi189:vg_nvme/lv_4 2024-04-30T08:25:55.417 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:55 smithi071 ceph-mon[33385]: purged_snaps scrub starts 2024-04-30T08:25:55.417 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:55 smithi071 ceph-mon[33385]: purged_snaps scrub ok 2024-04-30T08:25:55.417 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:55 smithi071 ceph-mon[33385]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-04-30T08:25:55.417 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:55 smithi071 ceph-mon[33385]: osdmap e12: 2 total, 1 up, 2 in 2024-04-30T08:25:55.418 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:55 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-30T08:25:55.418 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:55 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-30T08:25:55.418 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:55 smithi071 ceph-mon[33385]: from='osd.1 ' entity='osd.1' 2024-04-30T08:25:55.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:55 smithi189 ceph-mon[33038]: purged_snaps scrub starts 2024-04-30T08:25:55.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:55 smithi189 ceph-mon[33038]: purged_snaps scrub ok 2024-04-30T08:25:55.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:55 smithi189 ceph-mon[33038]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-04-30T08:25:55.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:55 smithi189 ceph-mon[33038]: osdmap e12: 2 total, 1 up, 2 in 2024-04-30T08:25:55.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:55 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-30T08:25:55.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:55 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-30T08:25:55.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:55 smithi189 ceph-mon[33038]: from='osd.1 ' entity='osd.1' 2024-04-30T08:25:55.652 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.c/config 2024-04-30T08:25:55.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:55 smithi032 ceph-mon[28331]: purged_snaps scrub starts 2024-04-30T08:25:55.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:55 smithi032 ceph-mon[28331]: purged_snaps scrub ok 2024-04-30T08:25:55.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:55 smithi032 ceph-mon[28331]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-04-30T08:25:55.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:55 smithi032 ceph-mon[28331]: osdmap e12: 2 total, 1 up, 2 in 2024-04-30T08:25:55.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:55 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-30T08:25:55.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:55 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-30T08:25:55.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:55 smithi032 ceph-mon[28331]: from='osd.1 ' entity='osd.1' 2024-04-30T08:25:56.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:56 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-30T08:25:56.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:56 smithi189 ceph-mon[33038]: pgmap v75: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:56.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:56 smithi189 ceph-mon[33038]: Detected new or changed devices on smithi071 2024-04-30T08:25:56.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:56 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:56.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:56 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:56.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:56 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:25:56.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:56 smithi189 ceph-mon[33038]: Adjusting osd_memory_target on smithi071 to 17088M 2024-04-30T08:25:56.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:56 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:56.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:56 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:25:56.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:56 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:25:56.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:56 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:56.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:56 smithi189 ceph-mon[33038]: osd.1 [v2:172.21.15.71:6800/444649711,v1:172.21.15.71:6801/444649711] boot 2024-04-30T08:25:56.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:56 smithi189 ceph-mon[33038]: osdmap e13: 2 total, 2 up, 2 in 2024-04-30T08:25:56.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:56 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-30T08:25:56.594 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:56 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-30T08:25:56.594 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:56 smithi032 ceph-mon[28331]: pgmap v75: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:56.594 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:56 smithi032 ceph-mon[28331]: Detected new or changed devices on smithi071 2024-04-30T08:25:56.594 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:56 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:56.594 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:56 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:56.594 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:56 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:25:56.594 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:56 smithi032 ceph-mon[28331]: Adjusting osd_memory_target on smithi071 to 17088M 2024-04-30T08:25:56.594 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:56 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:56.594 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:56 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:25:56.595 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:56 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:25:56.595 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:56 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:56.595 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:56 smithi032 ceph-mon[28331]: osd.1 [v2:172.21.15.71:6800/444649711,v1:172.21.15.71:6801/444649711] boot 2024-04-30T08:25:56.595 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:56 smithi032 ceph-mon[28331]: osdmap e13: 2 total, 2 up, 2 in 2024-04-30T08:25:56.595 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:56 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-30T08:25:56.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:56 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-30T08:25:56.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:56 smithi071 ceph-mon[33385]: pgmap v75: 0 pgs: ; 0 B data, 31 MiB used, 89 GiB / 89 GiB avail 2024-04-30T08:25:56.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:56 smithi071 ceph-mon[33385]: Detected new or changed devices on smithi071 2024-04-30T08:25:56.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:56 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:56.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:56 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:56.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:56 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:25:56.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:56 smithi071 ceph-mon[33385]: Adjusting osd_memory_target on smithi071 to 17088M 2024-04-30T08:25:56.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:56 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:56.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:56 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:25:56.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:56 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:25:56.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:56 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:25:56.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:56 smithi071 ceph-mon[33385]: osd.1 [v2:172.21.15.71:6800/444649711,v1:172.21.15.71:6801/444649711] boot 2024-04-30T08:25:56.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:56 smithi071 ceph-mon[33385]: osdmap e13: 2 total, 2 up, 2 in 2024-04-30T08:25:56.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:56 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-30T08:25:57.171 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.168+0000 7fe404f3a700 1 -- 172.21.15.189:0/2614479882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4001042d0 msgr2=0x7fe4001046e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:25:57.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.168+0000 7fe404f3a700 1 --2- 172.21.15.189:0/2614479882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4001042d0 0x7fe4001046e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8009990 tx=0x7fe3e8009ca0 comp rx=0 tx=0).stop 2024-04-30T08:25:57.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.168+0000 7fe404f3a700 1 -- 172.21.15.189:0/2614479882 shutdown_connections 2024-04-30T08:25:57.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.168+0000 7fe404f3a700 1 --2- 172.21.15.189:0/2614479882 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe400104c70 0x7fe40010b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:25:57.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.168+0000 7fe404f3a700 1 --2- 172.21.15.189:0/2614479882 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4001031a0 0x7fe4001035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:25:57.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.168+0000 7fe404f3a700 1 --2- 172.21.15.189:0/2614479882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4001042d0 0x7fe4001046e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:25:57.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.168+0000 7fe404f3a700 1 -- 172.21.15.189:0/2614479882 >> 172.21.15.189:0/2614479882 conn(0x7fe4000fe7e0 msgr2=0x7fe400100c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:25:57.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.168+0000 7fe404f3a700 1 -- 172.21.15.189:0/2614479882 shutdown_connections 2024-04-30T08:25:57.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.168+0000 7fe404f3a700 1 -- 172.21.15.189:0/2614479882 wait complete. 2024-04-30T08:25:57.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.169+0000 7fe404f3a700 1 Processor -- start 2024-04-30T08:25:57.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.169+0000 7fe404f3a700 1 -- start start 2024-04-30T08:25:57.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.169+0000 7fe404f3a700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4001031a0 0x7fe4000763c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:25:57.172 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.169+0000 7fe404f3a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4001042d0 0x7fe40019f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:25:57.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.169+0000 7fe404f3a700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe400104c70 0x7fe4001a1be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:25:57.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.169+0000 7fe404f3a700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe40010ec90 con 0x7fe4001031a0 2024-04-30T08:25:57.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.169+0000 7fe404f3a700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe40010eb10 con 0x7fe400104c70 2024-04-30T08:25:57.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.169+0000 7fe404f3a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe40010ee10 con 0x7fe4001042d0 2024-04-30T08:25:57.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.169+0000 7fe3f5d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4001042d0 0x7fe40019f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:25:57.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.169+0000 7fe3f5d9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4001042d0 0x7fe40019f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:40218/0 (socket says 172.21.15.189:40218) 2024-04-30T08:25:57.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.169+0000 7fe3f5d9b700 1 -- 172.21.15.189:0/166237834 learned_addr learned my addr 172.21.15.189:0/166237834 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-30T08:25:57.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.170+0000 7fe3fed9d700 1 --2- 172.21.15.189:0/166237834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe400104c70 0x7fe4001a1be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:25:57.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.170+0000 7fe3f5d9b700 1 -- 172.21.15.189:0/166237834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe400104c70 msgr2=0x7fe4001a1be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:25:57.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.170+0000 7fe3f5d9b700 1 --2- 172.21.15.189:0/166237834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe400104c70 0x7fe4001a1be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:25:57.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.170+0000 7fe3f5d9b700 1 -- 172.21.15.189:0/166237834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4001031a0 msgr2=0x7fe4000763c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:25:57.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.170+0000 7fe3f5d9b700 1 --2- 172.21.15.189:0/166237834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4001031a0 0x7fe4000763c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:25:57.173 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.170+0000 7fe3f5d9b700 1 -- 172.21.15.189:0/166237834 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4001a2410 con 0x7fe4001042d0 2024-04-30T08:25:57.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.170+0000 7fe3f5d9b700 1 --2- 172.21.15.189:0/166237834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4001042d0 0x7fe40019f650 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8003800 tx=0x7fe3e80047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:25:57.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.170+0000 7fe3f7fff700 1 -- 172.21.15.189:0/166237834 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e8004830 con 0x7fe4001042d0 2024-04-30T08:25:57.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.170+0000 7fe3f7fff700 1 -- 172.21.15.189:0/166237834 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe3e8005020 con 0x7fe4001042d0 2024-04-30T08:25:57.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.170+0000 7fe404f3a700 1 -- 172.21.15.189:0/166237834 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4001a26a0 con 0x7fe4001042d0 2024-04-30T08:25:57.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.170+0000 7fe3f7fff700 1 -- 172.21.15.189:0/166237834 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe3e8003e40 con 0x7fe4001042d0 2024-04-30T08:25:57.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.170+0000 7fe404f3a700 1 -- 172.21.15.189:0/166237834 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4001a2b70 con 0x7fe4001042d0 2024-04-30T08:25:57.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.171+0000 7fe404f3a700 1 -- 172.21.15.189:0/166237834 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe400051210 con 0x7fe4001042d0 2024-04-30T08:25:57.174 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.173+0000 7fe3f7fff700 1 -- 172.21.15.189:0/166237834 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 15) ==== 147464+0+0 (secure 0 0 0) 0x7fe3e8029060 con 0x7fe4001042d0 2024-04-30T08:25:57.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.174+0000 7fe3f7fff700 1 --2- 172.21.15.189:0/166237834 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe3ec0afb70 0x7fe3ec0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:25:57.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.174+0000 7fe3f7fff700 1 -- 172.21.15.189:0/166237834 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(13..13 src has 1..13) ==== 2309+0+0 (secure 0 0 0) 0x7fe3e80de180 con 0x7fe4001042d0 2024-04-30T08:25:57.175 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.175+0000 7fe3fe59c700 1 --2- 172.21.15.189:0/166237834 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe3ec0afb70 0x7fe3ec0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:25:57.176 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.176+0000 7fe3fe59c700 1 --2- 172.21.15.189:0/166237834 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe3ec0afb70 0x7fe3ec0b2080 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe4001041d0 tx=0x7fe3f0006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:25:57.179 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.178+0000 7fe3f7fff700 1 -- 172.21.15.189:0/166237834 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe3e8013070 con 0x7fe4001042d0 2024-04-30T08:25:57.388 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:25:57.387+0000 7fe404f3a700 1 -- 172.21.15.189:0/166237834 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fe400063b50 con 0x7fe3ec0afb70 2024-04-30T08:25:58.670 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:58 smithi189 ceph-mon[33038]: osdmap e14: 2 total, 2 up, 2 in 2024-04-30T08:25:58.671 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:58 smithi189 ceph-mon[33038]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:25:58.671 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:58 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:25:58.671 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:58 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:25:58.671 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:58 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:25:58.671 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:25:58 smithi189 ceph-mon[33038]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:25:58.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:58 smithi032 ceph-mon[28331]: osdmap e14: 2 total, 2 up, 2 in 2024-04-30T08:25:58.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:58 smithi032 ceph-mon[28331]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:25:58.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:58 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:25:58.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:58 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:25:58.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:58 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:25:58.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:25:58 smithi032 ceph-mon[28331]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:25:58.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:58 smithi071 ceph-mon[33385]: osdmap e14: 2 total, 2 up, 2 in 2024-04-30T08:25:58.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:58 smithi071 ceph-mon[33385]: from='client.24163 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:25:58.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:58 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:25:58.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:58 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:25:58.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:58 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:25:58.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:25:58 smithi071 ceph-mon[33385]: pgmap v78: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:26:00.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:00 smithi032 ceph-mon[28331]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:26:00.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:00 smithi032 ceph-mon[28331]: from='client.? 172.21.15.189:0/536335748' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "704e5e8a-935e-47c7-ae28-27119876ac09"}]: dispatch 2024-04-30T08:26:00.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:00 smithi032 ceph-mon[28331]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "704e5e8a-935e-47c7-ae28-27119876ac09"}]: dispatch 2024-04-30T08:26:00.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:00 smithi032 ceph-mon[28331]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "704e5e8a-935e-47c7-ae28-27119876ac09"}]': finished 2024-04-30T08:26:00.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:00 smithi032 ceph-mon[28331]: osdmap e15: 3 total, 2 up, 3 in 2024-04-30T08:26:00.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:00 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-30T08:26:01.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:00 smithi071 ceph-mon[33385]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:26:01.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:00 smithi071 ceph-mon[33385]: from='client.? 172.21.15.189:0/536335748' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "704e5e8a-935e-47c7-ae28-27119876ac09"}]: dispatch 2024-04-30T08:26:01.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:00 smithi071 ceph-mon[33385]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "704e5e8a-935e-47c7-ae28-27119876ac09"}]: dispatch 2024-04-30T08:26:01.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:00 smithi071 ceph-mon[33385]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "704e5e8a-935e-47c7-ae28-27119876ac09"}]': finished 2024-04-30T08:26:01.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:00 smithi071 ceph-mon[33385]: osdmap e15: 3 total, 2 up, 3 in 2024-04-30T08:26:01.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:00 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-30T08:26:01.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:00 smithi189 ceph-mon[33038]: pgmap v79: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:26:01.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:00 smithi189 ceph-mon[33038]: from='client.? 172.21.15.189:0/536335748' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "704e5e8a-935e-47c7-ae28-27119876ac09"}]: dispatch 2024-04-30T08:26:01.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:00 smithi189 ceph-mon[33038]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "704e5e8a-935e-47c7-ae28-27119876ac09"}]: dispatch 2024-04-30T08:26:01.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:00 smithi189 ceph-mon[33038]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "704e5e8a-935e-47c7-ae28-27119876ac09"}]': finished 2024-04-30T08:26:01.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:00 smithi189 ceph-mon[33038]: osdmap e15: 3 total, 2 up, 3 in 2024-04-30T08:26:01.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:00 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-30T08:26:01.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:01 smithi032 ceph-mon[28331]: from='client.? 172.21.15.189:0/1153643324' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:26:02.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:01 smithi071 ceph-mon[33385]: from='client.? 172.21.15.189:0/1153643324' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:26:02.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:01 smithi189 ceph-mon[33038]: from='client.? 172.21.15.189:0/1153643324' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:26:02.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:02 smithi032 ceph-mon[28331]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:26:03.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:02 smithi071 ceph-mon[33385]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:26:03.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:02 smithi189 ceph-mon[33038]: pgmap v81: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:26:04.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:04 smithi032 ceph-mon[28331]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:26:04.978 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:04 smithi071 ceph-mon[33385]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:26:05.002 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:04 smithi189 ceph-mon[33038]: pgmap v82: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:26:06.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:06 smithi189 ceph-mon[33038]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:26:06.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:06 smithi032 ceph-mon[28331]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:26:07.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:06 smithi071 ceph-mon[33385]: pgmap v83: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:26:07.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:07 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-30T08:26:07.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:07 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:26:07.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:07 smithi032 ceph-mon[28331]: Deploying daemon osd.2 on smithi189 2024-04-30T08:26:07.960 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:07 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-30T08:26:07.960 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:07 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:26:07.960 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:07 smithi189 ceph-mon[33038]: Deploying daemon osd.2 on smithi189 2024-04-30T08:26:08.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:07 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-30T08:26:08.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:07 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:26:08.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:07 smithi071 ceph-mon[33385]: Deploying daemon osd.2 on smithi189 2024-04-30T08:26:08.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:08 smithi032 ceph-mon[28331]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:26:08.948 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:08 smithi189 ceph-mon[33038]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:26:09.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:08 smithi071 ceph-mon[33385]: pgmap v84: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:26:10.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:10 smithi189 ceph-mon[33038]: pgmap v85: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:26:10.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:10 smithi032 ceph-mon[28331]: pgmap v85: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:26:11.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:10 smithi071 ceph-mon[33385]: pgmap v85: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:26:12.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:12 smithi032 ceph-mon[28331]: pgmap v86: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:26:14.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:12 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:14.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:12 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:26:14.392 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:12 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:14.393 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:12 smithi071 ceph-mon[33385]: pgmap v86: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:26:14.393 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:12 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:14.393 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:12 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:26:14.393 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:12 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:14.393 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:12 smithi189 ceph-mon[33038]: pgmap v86: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:26:14.394 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:12 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:14.394 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:12 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:26:14.394 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:12 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:14.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:14 smithi032 ceph-mon[28331]: pgmap v87: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:26:15.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:14 smithi071 ceph-mon[33385]: pgmap v87: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:26:15.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:14 smithi189 ceph-mon[33038]: pgmap v87: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:26:15.674 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 2 on host 'smithi189' 2024-04-30T08:26:15.675 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:26:15.672+0000 7fe3f7fff700 1 -- 172.21.15.189:0/166237834 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe400063b50 con 0x7fe3ec0afb70 2024-04-30T08:26:15.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:26:15.674+0000 7fe404f3a700 1 -- 172.21.15.189:0/166237834 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe3ec0afb70 msgr2=0x7fe3ec0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:26:15.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:26:15.674+0000 7fe404f3a700 1 --2- 172.21.15.189:0/166237834 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe3ec0afb70 0x7fe3ec0b2080 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe4001041d0 tx=0x7fe3f0006d20 comp rx=0 tx=0).stop 2024-04-30T08:26:15.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:26:15.674+0000 7fe404f3a700 1 -- 172.21.15.189:0/166237834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4001042d0 msgr2=0x7fe40019f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:26:15.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:26:15.674+0000 7fe404f3a700 1 --2- 172.21.15.189:0/166237834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4001042d0 0x7fe40019f650 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe3e8003800 tx=0x7fe3e80047b0 comp rx=0 tx=0).stop 2024-04-30T08:26:15.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:26:15.674+0000 7fe404f3a700 1 -- 172.21.15.189:0/166237834 shutdown_connections 2024-04-30T08:26:15.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:26:15.674+0000 7fe404f3a700 1 --2- 172.21.15.189:0/166237834 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe3ec0afb70 0x7fe3ec0b2080 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:26:15.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:26:15.674+0000 7fe404f3a700 1 --2- 172.21.15.189:0/166237834 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe400104c70 0x7fe4001a1be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:26:15.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:26:15.674+0000 7fe404f3a700 1 --2- 172.21.15.189:0/166237834 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe4001031a0 0x7fe4000763c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:26:15.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:26:15.674+0000 7fe404f3a700 1 --2- 172.21.15.189:0/166237834 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4001042d0 0x7fe40019f650 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:26:15.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:26:15.674+0000 7fe404f3a700 1 -- 172.21.15.189:0/166237834 >> 172.21.15.189:0/166237834 conn(0x7fe4000fe7e0 msgr2=0x7fe400107500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:26:15.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:26:15.675+0000 7fe404f3a700 1 -- 172.21.15.189:0/166237834 shutdown_connections 2024-04-30T08:26:15.676 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:26:15.675+0000 7fe404f3a700 1 -- 172.21.15.189:0/166237834 wait complete. 2024-04-30T08:26:16.307 DEBUG:teuthology.orchestra.run.smithi189:osd.2> sudo journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.2.service 2024-04-30T08:26:16.309 INFO:tasks.cephadm:Deploying osd.3 on smithi032 with /dev/vg_nvme/lv_3... 2024-04-30T08:26:16.309 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-30T08:26:16.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:16 smithi032 ceph-mon[28331]: pgmap v88: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:26:16.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:16 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:16.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:16 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:16.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:16 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:16.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:16 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:16.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:16 smithi032 ceph-mon[28331]: from='osd.2 [v2:172.21.15.189:6800/3255528671,v1:172.21.15.189:6801/3255528671]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-30T08:26:16.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:16 smithi032 ceph-mon[28331]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-30T08:26:16.769 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:26:17.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:16 smithi071 ceph-mon[33385]: pgmap v88: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:26:17.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:16 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:17.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:16 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:17.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:16 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:17.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:16 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:17.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:16 smithi071 ceph-mon[33385]: from='osd.2 [v2:172.21.15.189:6800/3255528671,v1:172.21.15.189:6801/3255528671]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-30T08:26:17.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:16 smithi071 ceph-mon[33385]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-30T08:26:17.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:16 smithi189 ceph-mon[33038]: pgmap v88: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:26:17.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:16 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:17.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:16 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:17.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:16 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:17.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:16 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:17.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:16 smithi189 ceph-mon[33038]: from='osd.2 [v2:172.21.15.189:6800/3255528671,v1:172.21.15.189:6801/3255528671]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-30T08:26:17.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:16 smithi189 ceph-mon[33038]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-30T08:26:18.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:17 smithi071 ceph-mon[33385]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-30T08:26:18.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:17 smithi071 ceph-mon[33385]: from='osd.2 [v2:172.21.15.189:6800/3255528671,v1:172.21.15.189:6801/3255528671]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-30T08:26:18.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:17 smithi071 ceph-mon[33385]: osdmap e16: 3 total, 2 up, 3 in 2024-04-30T08:26:18.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:17 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-30T08:26:18.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:17 smithi071 ceph-mon[33385]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-30T08:26:18.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:17 smithi189 ceph-mon[33038]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-30T08:26:18.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:17 smithi189 ceph-mon[33038]: from='osd.2 [v2:172.21.15.189:6800/3255528671,v1:172.21.15.189:6801/3255528671]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-30T08:26:18.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:17 smithi189 ceph-mon[33038]: osdmap e16: 3 total, 2 up, 3 in 2024-04-30T08:26:18.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:17 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-30T08:26:18.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:17 smithi189 ceph-mon[33038]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-30T08:26:18.106 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:17 smithi032 ceph-mon[28331]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-30T08:26:18.106 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:17 smithi032 ceph-mon[28331]: from='osd.2 [v2:172.21.15.189:6800/3255528671,v1:172.21.15.189:6801/3255528671]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-30T08:26:18.106 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:17 smithi032 ceph-mon[28331]: osdmap e16: 3 total, 2 up, 3 in 2024-04-30T08:26:18.106 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:17 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-30T08:26:18.106 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:17 smithi032 ceph-mon[28331]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-30T08:26:18.550 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 30 08:26:18 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-2[38240]: 2024-04-30T08:26:18.161+0000 7f5b32e05700 -1 osd.2 0 waiting for initial osdmap 2024-04-30T08:26:18.550 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 30 08:26:18 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-2[38240]: 2024-04-30T08:26:18.165+0000 7f5b2abb2700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-30T08:26:18.996 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:18 smithi189 ceph-mon[33038]: pgmap v90: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:26:18.997 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:18 smithi189 ceph-mon[33038]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-30T08:26:18.997 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:18 smithi189 ceph-mon[33038]: osdmap e17: 3 total, 2 up, 3 in 2024-04-30T08:26:18.997 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:18 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-30T08:26:18.997 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:18 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-30T08:26:18.997 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:18 smithi189 ceph-mon[33038]: from='osd.2 ' entity='osd.2' 2024-04-30T08:26:18.997 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:18 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-30T08:26:19.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:18 smithi071 ceph-mon[33385]: pgmap v90: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:26:19.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:18 smithi071 ceph-mon[33385]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-30T08:26:19.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:18 smithi071 ceph-mon[33385]: osdmap e17: 3 total, 2 up, 3 in 2024-04-30T08:26:19.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:18 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-30T08:26:19.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:18 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-30T08:26:19.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:18 smithi071 ceph-mon[33385]: from='osd.2 ' entity='osd.2' 2024-04-30T08:26:19.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:18 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-30T08:26:19.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:18 smithi032 ceph-mon[28331]: pgmap v90: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:26:19.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:18 smithi032 ceph-mon[28331]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-30T08:26:19.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:18 smithi032 ceph-mon[28331]: osdmap e17: 3 total, 2 up, 3 in 2024-04-30T08:26:19.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:18 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-30T08:26:19.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:18 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-30T08:26:19.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:18 smithi032 ceph-mon[28331]: from='osd.2 ' entity='osd.2' 2024-04-30T08:26:19.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:18 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-30T08:26:19.707 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-30T08:26:19.707 INFO:teuthology.orchestra.run.smithi032.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-30T08:26:19.707 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 10+0 records in 2024-04-30T08:26:19.708 INFO:teuthology.orchestra.run.smithi032.stderr:10+0 records out 2024-04-30T08:26:19.708 INFO:teuthology.orchestra.run.smithi032.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0425622 s, 246 MB/s 2024-04-30T08:26:19.708 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping successful for: 2024-04-30T08:26:20.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:20 smithi032 ceph-mon[28331]: purged_snaps scrub starts 2024-04-30T08:26:20.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:20 smithi032 ceph-mon[28331]: purged_snaps scrub ok 2024-04-30T08:26:20.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:20 smithi032 ceph-mon[28331]: osd.2 [v2:172.21.15.189:6800/3255528671,v1:172.21.15.189:6801/3255528671] boot 2024-04-30T08:26:20.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:20 smithi032 ceph-mon[28331]: osdmap e18: 3 total, 3 up, 3 in 2024-04-30T08:26:20.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:20 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-30T08:26:20.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:20 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:20.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:20 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:20.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:20 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:26:20.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:20 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:20.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:20 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:26:20.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:20 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:26:20.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:20 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:20.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:20 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:26:20.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:20 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:26:20.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:20 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:26:20.444 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:20 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:20.444 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:20 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-30T08:26:20.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:20 smithi071 ceph-mon[33385]: purged_snaps scrub starts 2024-04-30T08:26:20.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:20 smithi071 ceph-mon[33385]: purged_snaps scrub ok 2024-04-30T08:26:20.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:20 smithi071 ceph-mon[33385]: osd.2 [v2:172.21.15.189:6800/3255528671,v1:172.21.15.189:6801/3255528671] boot 2024-04-30T08:26:20.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:20 smithi071 ceph-mon[33385]: osdmap e18: 3 total, 3 up, 3 in 2024-04-30T08:26:20.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:20 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-30T08:26:20.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:20 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:20.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:20 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:20.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:20 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:26:20.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:20 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:20.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:20 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:26:20.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:20 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:26:20.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:20 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:20.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:20 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:26:20.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:20 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:26:20.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:20 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:26:20.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:20 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:20.506 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:20 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-30T08:26:20.532 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi032:vg_nvme/lv_3 2024-04-30T08:26:20.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:20 smithi189 ceph-mon[33038]: purged_snaps scrub starts 2024-04-30T08:26:20.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:20 smithi189 ceph-mon[33038]: purged_snaps scrub ok 2024-04-30T08:26:20.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:20 smithi189 ceph-mon[33038]: osd.2 [v2:172.21.15.189:6800/3255528671,v1:172.21.15.189:6801/3255528671] boot 2024-04-30T08:26:20.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:20 smithi189 ceph-mon[33038]: osdmap e18: 3 total, 3 up, 3 in 2024-04-30T08:26:20.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:20 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-30T08:26:20.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:20 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:20.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:20 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:20.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:20 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:26:20.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:20 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:20.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:20 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:26:20.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:20 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:26:20.553 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:20 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:20.553 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:20 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:26:20.553 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:20 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:26:20.553 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:20 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:26:20.554 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:20 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:20.554 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:20 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-30T08:26:20.996 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:26:21.409 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 30 08:26:21 smithi032 sudo[47854]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-30T08:26:21.409 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 30 08:26:21 smithi032 sudo[47854]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-30T08:26:21.409 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 30 08:26:21 smithi032 sudo[47854]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-30T08:26:21.410 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 30 08:26:21 smithi032 sudo[47854]: pam_unix(sudo:session): session closed for user root 2024-04-30T08:26:21.410 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:21 smithi032 ceph-mon[28331]: Detected new or changed devices on smithi189 2024-04-30T08:26:21.410 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:21 smithi032 ceph-mon[28331]: Adjusting osd_memory_target on smithi189 to 17088M 2024-04-30T08:26:21.410 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:21 smithi032 ceph-mon[28331]: pgmap v93: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:26:21.411 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:21 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-30T08:26:21.411 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:21 smithi032 ceph-mon[28331]: osdmap e19: 3 total, 3 up, 3 in 2024-04-30T08:26:21.411 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:21 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-30T08:26:21.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:21 smithi071 ceph-mon[33385]: Detected new or changed devices on smithi189 2024-04-30T08:26:21.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:21 smithi071 ceph-mon[33385]: Adjusting osd_memory_target on smithi189 to 17088M 2024-04-30T08:26:21.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:21 smithi071 ceph-mon[33385]: pgmap v93: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:26:21.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:21 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-30T08:26:21.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:21 smithi071 ceph-mon[33385]: osdmap e19: 3 total, 3 up, 3 in 2024-04-30T08:26:21.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:21 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-30T08:26:21.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:21 smithi189 ceph-mon[33038]: Detected new or changed devices on smithi189 2024-04-30T08:26:21.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:21 smithi189 ceph-mon[33038]: Adjusting osd_memory_target on smithi189 to 17088M 2024-04-30T08:26:21.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:21 smithi189 ceph-mon[33038]: pgmap v93: 0 pgs: ; 0 B data, 61 MiB used, 179 GiB / 179 GiB avail 2024-04-30T08:26:21.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:21 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-30T08:26:21.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:21 smithi189 ceph-mon[33038]: osdmap e19: 3 total, 3 up, 3 in 2024-04-30T08:26:21.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:21 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-30T08:26:21.691 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 30 08:26:21 smithi032 sudo[47857]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-30T08:26:21.691 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 30 08:26:21 smithi032 sudo[47857]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-30T08:26:21.691 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 30 08:26:21 smithi032 sudo[47857]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-30T08:26:21.691 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 30 08:26:21 smithi032 sudo[47857]: pam_unix(sudo:session): session closed for user root 2024-04-30T08:26:21.843 INFO:journalctl@ceph.osd.1.smithi071.stdout:Apr 30 08:26:21 smithi071 sudo[42103]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-30T08:26:21.844 INFO:journalctl@ceph.osd.1.smithi071.stdout:Apr 30 08:26:21 smithi071 sudo[42103]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-30T08:26:21.844 INFO:journalctl@ceph.osd.1.smithi071.stdout:Apr 30 08:26:21 smithi071 sudo[42103]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-30T08:26:21.844 INFO:journalctl@ceph.osd.1.smithi071.stdout:Apr 30 08:26:21 smithi071 sudo[42103]: pam_unix(sudo:session): session closed for user root 2024-04-30T08:26:21.844 INFO:journalctl@ceph.osd.1.smithi071.stdout:Apr 30 08:26:21 smithi071 sudo[42106]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-30T08:26:21.844 INFO:journalctl@ceph.osd.1.smithi071.stdout:Apr 30 08:26:21 smithi071 sudo[42106]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-30T08:26:21.844 INFO:journalctl@ceph.osd.1.smithi071.stdout:Apr 30 08:26:21 smithi071 sudo[42106]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-30T08:26:21.844 INFO:journalctl@ceph.osd.1.smithi071.stdout:Apr 30 08:26:21 smithi071 sudo[42106]: pam_unix(sudo:session): session closed for user root 2024-04-30T08:26:22.029 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:21 smithi032 sudo[47873]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-30T08:26:22.029 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:21 smithi032 sudo[47873]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-30T08:26:22.029 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:21 smithi032 sudo[47873]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-30T08:26:22.029 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:21 smithi032 sudo[47873]: pam_unix(sudo:session): session closed for user root 2024-04-30T08:26:22.029 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:21 smithi032 sudo[47876]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-30T08:26:22.029 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:21 smithi032 sudo[47876]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-30T08:26:22.029 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:21 smithi032 sudo[47876]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-30T08:26:22.030 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:21 smithi032 sudo[47876]: pam_unix(sudo:session): session closed for user root 2024-04-30T08:26:22.050 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 30 08:26:21 smithi189 sudo[42136]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-30T08:26:22.050 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 30 08:26:21 smithi189 sudo[42136]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-30T08:26:22.050 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 30 08:26:21 smithi189 sudo[42136]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-30T08:26:22.050 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 30 08:26:21 smithi189 sudo[42136]: pam_unix(sudo:session): session closed for user root 2024-04-30T08:26:22.050 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 30 08:26:21 smithi189 sudo[42139]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-30T08:26:22.050 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 30 08:26:21 smithi189 sudo[42139]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-30T08:26:22.050 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 30 08:26:21 smithi189 sudo[42139]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-30T08:26:22.050 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 30 08:26:21 smithi189 sudo[42139]: pam_unix(sudo:session): session closed for user root 2024-04-30T08:26:22.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:21 smithi189 sudo[42142]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-30T08:26:22.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:21 smithi189 sudo[42142]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-30T08:26:22.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:21 smithi189 sudo[42142]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-30T08:26:22.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:21 smithi189 sudo[42142]: pam_unix(sudo:session): session closed for user root 2024-04-30T08:26:22.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:21 smithi189 sudo[42145]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-30T08:26:22.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:21 smithi189 sudo[42145]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-30T08:26:22.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:21 smithi189 sudo[42145]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-30T08:26:22.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:21 smithi189 sudo[42145]: pam_unix(sudo:session): session closed for user root 2024-04-30T08:26:22.209 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:21 smithi071 sudo[42109]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-30T08:26:22.209 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:21 smithi071 sudo[42109]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-30T08:26:22.209 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:21 smithi071 sudo[42109]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-30T08:26:22.209 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:21 smithi071 sudo[42109]: pam_unix(sudo:session): session closed for user root 2024-04-30T08:26:22.209 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:21 smithi071 sudo[42112]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-30T08:26:22.209 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:21 smithi071 sudo[42112]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-30T08:26:22.209 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:21 smithi071 sudo[42112]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-30T08:26:22.209 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:21 smithi071 sudo[42112]: pam_unix(sudo:session): session closed for user root 2024-04-30T08:26:22.210 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:22 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-30T08:26:22.210 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:22 smithi071 ceph-mon[33385]: osdmap e20: 3 total, 3 up, 3 in 2024-04-30T08:26:22.210 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:22 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:22.210 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:22 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:22.210 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:22 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:22.210 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:22 smithi071 ceph-mon[33385]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-30T08:26:22.210 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:22 smithi071 ceph-mon[33385]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-30T08:26:22.210 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:22 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-30T08:26:22.210 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:22 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:26:22.210 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:22 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:26:22.210 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:22 smithi071 ceph-mon[33385]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-30T08:26:22.210 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:22 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-30T08:26:22.210 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:22 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:26:22.211 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:22 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:26:22.211 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:22 smithi071 ceph-mon[33385]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-30T08:26:22.339 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:22 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-30T08:26:22.340 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:22 smithi032 ceph-mon[28331]: osdmap e20: 3 total, 3 up, 3 in 2024-04-30T08:26:22.340 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:22 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:22.340 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:22 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:22.340 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:22 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:22.340 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:22 smithi032 ceph-mon[28331]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-30T08:26:22.340 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:22 smithi032 ceph-mon[28331]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-30T08:26:22.340 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:22 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-30T08:26:22.340 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:22 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:26:22.340 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:22 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:26:22.340 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:22 smithi032 ceph-mon[28331]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-30T08:26:22.340 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:22 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-30T08:26:22.340 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:22 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:26:22.341 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:22 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:26:22.341 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:22 smithi032 ceph-mon[28331]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-30T08:26:22.341 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:22 smithi032 ceph-mon[28331]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-30T08:26:22.341 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:22 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-30T08:26:22.341 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:22 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:26:22.341 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:22 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:26:22.341 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:22 smithi032 ceph-mon[28331]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-30T08:26:22.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:22 smithi071 ceph-mon[33385]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-30T08:26:22.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:22 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-30T08:26:22.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:22 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:26:22.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:22 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:26:22.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:22 smithi071 ceph-mon[33385]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-30T08:26:22.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:22 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-30T08:26:22.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:22 smithi189 ceph-mon[33038]: osdmap e20: 3 total, 3 up, 3 in 2024-04-30T08:26:22.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:22 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:22.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:22 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:22.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:22 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:22.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:22 smithi189 ceph-mon[33038]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-30T08:26:22.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:22 smithi189 ceph-mon[33038]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-30T08:26:22.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:22 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-30T08:26:22.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:22 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:26:22.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:22 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:26:22.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:22 smithi189 ceph-mon[33038]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-30T08:26:22.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:22 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-30T08:26:22.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:22 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:26:22.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:22 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:26:22.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:22 smithi189 ceph-mon[33038]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-30T08:26:22.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:22 smithi189 ceph-mon[33038]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-30T08:26:22.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:22 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-30T08:26:22.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:22 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:26:22.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:22 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:26:22.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:22 smithi189 ceph-mon[33038]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-30T08:26:23.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.239+0000 7f508ca00700 1 -- 172.21.15.32:0/1762830791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f508810c180 msgr2=0x7f508810e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:26:23.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.239+0000 7f508ca00700 1 --2- 172.21.15.32:0/1762830791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f508810c180 0x7f508810e690 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f508000b0a0 tx=0x7f508000b3b0 comp rx=0 tx=0).stop 2024-04-30T08:26:23.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.239+0000 7f508ca00700 1 -- 172.21.15.32:0/1762830791 shutdown_connections 2024-04-30T08:26:23.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.239+0000 7f508ca00700 1 --2- 172.21.15.32:0/1762830791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f508810c180 0x7f508810e690 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:26:23.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.239+0000 7f508ca00700 1 --2- 172.21.15.32:0/1762830791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50880786e0 0x7f5088076bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:26:23.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.239+0000 7f508ca00700 1 --2- 172.21.15.32:0/1762830791 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f50880770f0 0x7f5088077520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:26:23.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.239+0000 7f508ca00700 1 -- 172.21.15.32:0/1762830791 >> 172.21.15.32:0/1762830791 conn(0x7f50880ff920 msgr2=0x7f5088101db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:26:23.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.240+0000 7f508ca00700 1 -- 172.21.15.32:0/1762830791 shutdown_connections 2024-04-30T08:26:23.242 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:23 smithi032 ceph-mon[28331]: pgmap v96: 1 pgs: 1 active+clean; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:23.242 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:23 smithi032 ceph-mon[28331]: osdmap e21: 3 total, 3 up, 3 in 2024-04-30T08:26:23.242 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:23 smithi032 ceph-mon[28331]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-30T08:26:23.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.240+0000 7f508ca00700 1 -- 172.21.15.32:0/1762830791 wait complete. 2024-04-30T08:26:23.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.241+0000 7f508ca00700 1 Processor -- start 2024-04-30T08:26:23.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.241+0000 7f508ca00700 1 -- start start 2024-04-30T08:26:23.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.241+0000 7f508ca00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50880770f0 0x7f508819ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:26:23.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.241+0000 7f508ca00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f50880786e0 0x7f508819cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:26:23.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.241+0000 7f508ca00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f508810c180 0x7f50881a3f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:26:23.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.241+0000 7f508ca00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5088111f60 con 0x7f508810c180 2024-04-30T08:26:23.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.241+0000 7f508ca00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5088111de0 con 0x7f50880786e0 2024-04-30T08:26:23.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.241+0000 7f508ca00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f50881120e0 con 0x7f50880770f0 2024-04-30T08:26:23.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.242+0000 7f5086800700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f508810c180 0x7f50881a3f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:26:23.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.242+0000 7f5085e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50880770f0 0x7f508819ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:26:23.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.242+0000 7f5086800700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f508810c180 0x7f50881a3f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42950/0 (socket says 172.21.15.32:42950) 2024-04-30T08:26:23.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.242+0000 7f5085e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50880770f0 0x7f508819ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:47626/0 (socket says 172.21.15.32:47626) 2024-04-30T08:26:23.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.242+0000 7f5086800700 1 -- 172.21.15.32:0/591091338 learned_addr learned my addr 172.21.15.32:0/591091338 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:26:23.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.242+0000 7f5085400700 1 --2- 172.21.15.32:0/591091338 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f50880786e0 0x7f508819cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:26:23.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.242+0000 7f5085e00700 1 -- 172.21.15.32:0/591091338 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f50880786e0 msgr2=0x7f508819cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:26:23.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.242+0000 7f5085e00700 1 --2- 172.21.15.32:0/591091338 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f50880786e0 0x7f508819cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:26:23.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.242+0000 7f5085e00700 1 -- 172.21.15.32:0/591091338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f508810c180 msgr2=0x7f50881a3f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:26:23.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.242+0000 7f5085e00700 1 --2- 172.21.15.32:0/591091338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f508810c180 0x7f50881a3f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:26:23.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.242+0000 7f5085e00700 1 -- 172.21.15.32:0/591091338 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50881a4580 con 0x7f50880770f0 2024-04-30T08:26:23.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.242+0000 7f5086800700 1 --2- 172.21.15.32:0/591091338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f508810c180 0x7f50881a3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:26:23.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.243+0000 7f5085e00700 1 --2- 172.21.15.32:0/591091338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50880770f0 0x7f508819ca20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f507800e9c0 tx=0x7f507800ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:26:23.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.243+0000 7f507ea00700 1 -- 172.21.15.32:0/591091338 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f507800c9c0 con 0x7f50880770f0 2024-04-30T08:26:23.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.243+0000 7f508ca00700 1 -- 172.21.15.32:0/591091338 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50881a4870 con 0x7f50880770f0 2024-04-30T08:26:23.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.243+0000 7f507ea00700 1 -- 172.21.15.32:0/591091338 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f507800cb60 con 0x7f50880770f0 2024-04-30T08:26:23.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.243+0000 7f508ca00700 1 -- 172.21.15.32:0/591091338 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50881a4e00 con 0x7f50880770f0 2024-04-30T08:26:23.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.243+0000 7f507ea00700 1 -- 172.21.15.32:0/591091338 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5078018420 con 0x7f50880770f0 2024-04-30T08:26:23.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.245+0000 7f508ca00700 1 -- 172.21.15.32:0/591091338 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f508802e590 con 0x7f50880770f0 2024-04-30T08:26:23.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.246+0000 7f507ea00700 1 -- 172.21.15.32:0/591091338 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f5078010430 con 0x7f50880770f0 2024-04-30T08:26:23.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.248+0000 7f507ea00700 1 --2- 172.21.15.32:0/591091338 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f506c0afc80 0x7f506c0b2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:26:23.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.248+0000 7f507ea00700 1 -- 172.21.15.32:0/591091338 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3237+0+0 (secure 0 0 0) 0x7f50780dd930 con 0x7f50880770f0 2024-04-30T08:26:23.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.248+0000 7f5085400700 1 --2- 172.21.15.32:0/591091338 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f506c0afc80 0x7f506c0b2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:26:23.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.249+0000 7f5085400700 1 --2- 172.21.15.32:0/591091338 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f506c0afc80 0x7f506c0b2190 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f508819dad0 tx=0x7f50740073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:26:23.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.253+0000 7f507ea00700 1 -- 172.21.15.32:0/591091338 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f50780a7840 con 0x7f50880770f0 2024-04-30T08:26:23.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:23.493+0000 7f508ca00700 1 -- 172.21.15.32:0/591091338 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f5088063b50 con 0x7f506c0afc80 2024-04-30T08:26:23.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:23 smithi071 ceph-mon[33385]: pgmap v96: 1 pgs: 1 active+clean; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:23.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:23 smithi071 ceph-mon[33385]: osdmap e21: 3 total, 3 up, 3 in 2024-04-30T08:26:23.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:23 smithi071 ceph-mon[33385]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-30T08:26:23.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:23 smithi189 ceph-mon[33038]: pgmap v96: 1 pgs: 1 active+clean; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:23.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:23 smithi189 ceph-mon[33038]: osdmap e21: 3 total, 3 up, 3 in 2024-04-30T08:26:23.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:23 smithi189 ceph-mon[33038]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-30T08:26:24.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:24 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:26:24.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:24 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:26:24.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:24 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:26:24.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:24 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:26:24.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:24 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:26:24.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:24 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:26:24.594 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:24 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:26:24.594 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:24 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:26:24.594 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:24 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:26:25.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:25 smithi071 ceph-mon[33385]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:26:25.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:25 smithi071 ceph-mon[33385]: pgmap v98: 1 pgs: 1 active+clean; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:25.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:25 smithi189 ceph-mon[33038]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:26:25.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:25 smithi189 ceph-mon[33038]: pgmap v98: 1 pgs: 1 active+clean; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:25.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:25 smithi032 ceph-mon[28331]: from='client.24199 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:26:25.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:25 smithi032 ceph-mon[28331]: pgmap v98: 1 pgs: 1 active+clean; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:27.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:27 smithi071 ceph-mon[33385]: pgmap v99: 1 pgs: 1 active+clean; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:27.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:27 smithi189 ceph-mon[33038]: pgmap v99: 1 pgs: 1 active+clean; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:27.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:27 smithi032 ceph-mon[28331]: pgmap v99: 1 pgs: 1 active+clean; 0 B data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:28.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:28 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2137196698' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "575004f7-6559-4a62-863e-e33078b0eebf"}]: dispatch 2024-04-30T08:26:28.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:28 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2137196698' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "575004f7-6559-4a62-863e-e33078b0eebf"}]': finished 2024-04-30T08:26:28.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:28 smithi071 ceph-mon[33385]: osdmap e22: 4 total, 3 up, 4 in 2024-04-30T08:26:28.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:28 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-30T08:26:28.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:28 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2137196698' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "575004f7-6559-4a62-863e-e33078b0eebf"}]: dispatch 2024-04-30T08:26:28.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:28 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2137196698' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "575004f7-6559-4a62-863e-e33078b0eebf"}]': finished 2024-04-30T08:26:28.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:28 smithi189 ceph-mon[33038]: osdmap e22: 4 total, 3 up, 4 in 2024-04-30T08:26:28.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:28 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-30T08:26:28.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:28 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2137196698' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "575004f7-6559-4a62-863e-e33078b0eebf"}]: dispatch 2024-04-30T08:26:28.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:28 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2137196698' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "575004f7-6559-4a62-863e-e33078b0eebf"}]': finished 2024-04-30T08:26:28.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:28 smithi032 ceph-mon[28331]: osdmap e22: 4 total, 3 up, 4 in 2024-04-30T08:26:28.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:28 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-30T08:26:29.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:29 smithi189 ceph-mon[33038]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 9.2 KiB/s rd, 358 KiB/s wr, 29 op/s 2024-04-30T08:26:29.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:29 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2443998486' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:26:29.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:29 smithi032 ceph-mon[28331]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 9.2 KiB/s rd, 358 KiB/s wr, 29 op/s 2024-04-30T08:26:29.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:29 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2443998486' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:26:29.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:29 smithi071 ceph-mon[33385]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 9.2 KiB/s rd, 358 KiB/s wr, 29 op/s 2024-04-30T08:26:29.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:29 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2443998486' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:26:31.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:31 smithi189 ceph-mon[33038]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 6.9 KiB/s rd, 269 KiB/s wr, 21 op/s 2024-04-30T08:26:31.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:31 smithi032 ceph-mon[28331]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 6.9 KiB/s rd, 269 KiB/s wr, 21 op/s 2024-04-30T08:26:31.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:31 smithi071 ceph-mon[33385]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 6.9 KiB/s rd, 269 KiB/s wr, 21 op/s 2024-04-30T08:26:33.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:33 smithi032 ceph-mon[28331]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 5.9 KiB/s rd, 230 KiB/s wr, 18 op/s 2024-04-30T08:26:33.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:33 smithi189 ceph-mon[33038]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 5.9 KiB/s rd, 230 KiB/s wr, 18 op/s 2024-04-30T08:26:33.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:33 smithi071 ceph-mon[33385]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 5.9 KiB/s rd, 230 KiB/s wr, 18 op/s 2024-04-30T08:26:35.549 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:35 smithi032 ceph-mon[28331]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 5.5 KiB/s rd, 215 KiB/s wr, 17 op/s 2024-04-30T08:26:35.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:35 smithi189 ceph-mon[33038]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 5.5 KiB/s rd, 215 KiB/s wr, 17 op/s 2024-04-30T08:26:35.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:35 smithi071 ceph-mon[33385]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 5.5 KiB/s rd, 215 KiB/s wr, 17 op/s 2024-04-30T08:26:36.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:36 smithi032 ceph-mon[28331]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 5.5 KiB/s rd, 215 KiB/s wr, 17 op/s 2024-04-30T08:26:36.723 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:36 smithi189 ceph-mon[33038]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 5.5 KiB/s rd, 215 KiB/s wr, 17 op/s 2024-04-30T08:26:36.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:36 smithi071 ceph-mon[33385]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail; 5.5 KiB/s rd, 215 KiB/s wr, 17 op/s 2024-04-30T08:26:37.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:37 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-30T08:26:37.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:37 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:26:38.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:37 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-30T08:26:38.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:37 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:26:38.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:37 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-30T08:26:38.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:37 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:26:38.926 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:38 smithi032 ceph-mon[28331]: Deploying daemon osd.3 on smithi032 2024-04-30T08:26:38.926 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:38 smithi032 ceph-mon[28331]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:39.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:38 smithi071 ceph-mon[33385]: Deploying daemon osd.3 on smithi032 2024-04-30T08:26:39.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:38 smithi071 ceph-mon[33385]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:39.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:38 smithi189 ceph-mon[33038]: Deploying daemon osd.3 on smithi032 2024-04-30T08:26:39.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:38 smithi189 ceph-mon[33038]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:40.859 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:40 smithi032 ceph-mon[28331]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:41.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:40 smithi071 ceph-mon[33385]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:41.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:40 smithi189 ceph-mon[33038]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:42.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:42 smithi032 ceph-mon[28331]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:43.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:42 smithi071 ceph-mon[33385]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:43.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:42 smithi189 ceph-mon[33038]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:44.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:44 smithi032 ceph-mon[28331]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:45.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:44 smithi071 ceph-mon[33385]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:45.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:44 smithi189 ceph-mon[33038]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:46.918 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:46 smithi032 ceph-mon[28331]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:47.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:46 smithi071 ceph-mon[33385]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:47.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:46 smithi189 ceph-mon[33038]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:48.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:48 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:48.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:48 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:26:48.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:48 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:48.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:48 smithi032 ceph-mon[28331]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:48.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:48 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:48.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:48 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:26:48.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:48 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:48.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:48 smithi071 ceph-mon[33385]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:48.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:48 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:48.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:48 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:26:48.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:48 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:48.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:48 smithi189 ceph-mon[33038]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:50.875 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:50 smithi032 ceph-mon[28331]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:51.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:50 smithi071 ceph-mon[33385]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:51.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:50 smithi189 ceph-mon[33038]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:52.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:51 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:52.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:51 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:52.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:51 smithi189 ceph-mon[33038]: from='osd.3 [v2:172.21.15.32:6810/1384702112,v1:172.21.15.32:6811/1384702112]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-30T08:26:52.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:51 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:52.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:51 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:52.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:51 smithi032 ceph-mon[28331]: from='osd.3 [v2:172.21.15.32:6810/1384702112,v1:172.21.15.32:6811/1384702112]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-30T08:26:52.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:51 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:52.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:51 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:52.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:51 smithi071 ceph-mon[33385]: from='osd.3 [v2:172.21.15.32:6810/1384702112,v1:172.21.15.32:6811/1384702112]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-30T08:26:53.155 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:52 smithi071 ceph-mon[33385]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:53.155 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:52 smithi071 ceph-mon[33385]: from='osd.3 [v2:172.21.15.32:6810/1384702112,v1:172.21.15.32:6811/1384702112]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-30T08:26:53.155 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:52 smithi071 ceph-mon[33385]: osdmap e23: 4 total, 3 up, 4 in 2024-04-30T08:26:53.155 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:52 smithi071 ceph-mon[33385]: from='osd.3 [v2:172.21.15.32:6810/1384702112,v1:172.21.15.32:6811/1384702112]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-30T08:26:53.155 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:52 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-30T08:26:53.155 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:52 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:26:53.155 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:52 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:26:53.155 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:52 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:53.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:52 smithi032 ceph-mon[28331]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:53.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:52 smithi032 ceph-mon[28331]: from='osd.3 [v2:172.21.15.32:6810/1384702112,v1:172.21.15.32:6811/1384702112]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-30T08:26:53.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:52 smithi032 ceph-mon[28331]: osdmap e23: 4 total, 3 up, 4 in 2024-04-30T08:26:53.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:52 smithi032 ceph-mon[28331]: from='osd.3 [v2:172.21.15.32:6810/1384702112,v1:172.21.15.32:6811/1384702112]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-30T08:26:53.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-30T08:26:53.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:26:53.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:26:53.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:53.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:52 smithi189 ceph-mon[33038]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:53.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:52 smithi189 ceph-mon[33038]: from='osd.3 [v2:172.21.15.32:6810/1384702112,v1:172.21.15.32:6811/1384702112]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-30T08:26:53.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:52 smithi189 ceph-mon[33038]: osdmap e23: 4 total, 3 up, 4 in 2024-04-30T08:26:53.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:52 smithi189 ceph-mon[33038]: from='osd.3 [v2:172.21.15.32:6810/1384702112,v1:172.21.15.32:6811/1384702112]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-30T08:26:53.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:52 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-30T08:26:53.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:52 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:26:53.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:52 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:26:53.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:52 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:53.367 INFO:teuthology.orchestra.run.smithi032.stdout:Created osd(s) 3 on host 'smithi032' 2024-04-30T08:26:53.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:53.365+0000 7f507ea00700 1 -- 172.21.15.32:0/591091338 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f5088063b50 con 0x7f506c0afc80 2024-04-30T08:26:53.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:53.369+0000 7f508ca00700 1 -- 172.21.15.32:0/591091338 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f506c0afc80 msgr2=0x7f506c0b2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:26:53.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:53.369+0000 7f508ca00700 1 --2- 172.21.15.32:0/591091338 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f506c0afc80 0x7f506c0b2190 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f508819dad0 tx=0x7f50740073d0 comp rx=0 tx=0).stop 2024-04-30T08:26:53.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:53.369+0000 7f508ca00700 1 -- 172.21.15.32:0/591091338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50880770f0 msgr2=0x7f508819ca20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:26:53.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:53.369+0000 7f508ca00700 1 --2- 172.21.15.32:0/591091338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50880770f0 0x7f508819ca20 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f507800e9c0 tx=0x7f507800ecd0 comp rx=0 tx=0).stop 2024-04-30T08:26:53.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:53.369+0000 7f508ca00700 1 -- 172.21.15.32:0/591091338 shutdown_connections 2024-04-30T08:26:53.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:53.369+0000 7f508ca00700 1 --2- 172.21.15.32:0/591091338 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f506c0afc80 0x7f506c0b2190 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:26:53.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:53.369+0000 7f508ca00700 1 --2- 172.21.15.32:0/591091338 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f508810c180 0x7f50881a3f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:26:53.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:53.369+0000 7f508ca00700 1 --2- 172.21.15.32:0/591091338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50880770f0 0x7f508819ca20 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:26:53.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:53.369+0000 7f508ca00700 1 --2- 172.21.15.32:0/591091338 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f50880786e0 0x7f508819cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:26:53.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:53.369+0000 7f508ca00700 1 -- 172.21.15.32:0/591091338 >> 172.21.15.32:0/591091338 conn(0x7f50880ff920 msgr2=0x7f5088101db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:26:53.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:53.370+0000 7f508ca00700 1 -- 172.21.15.32:0/591091338 shutdown_connections 2024-04-30T08:26:53.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:26:53.370+0000 7f508ca00700 1 -- 172.21.15.32:0/591091338 wait complete. 2024-04-30T08:26:54.151 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:53 smithi032 ceph-mon[28331]: from='osd.3 [v2:172.21.15.32:6810/1384702112,v1:172.21.15.32:6811/1384702112]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-30T08:26:54.151 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:53 smithi032 ceph-mon[28331]: osdmap e24: 4 total, 3 up, 4 in 2024-04-30T08:26:54.151 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:53 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-30T08:26:54.151 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:53 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-30T08:26:54.152 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:53 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:54.152 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:53 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:54.152 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:53 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:26:54.217 DEBUG:teuthology.orchestra.run.smithi032:osd.3> sudo journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.3.service 2024-04-30T08:26:54.222 INFO:tasks.cephadm:Deploying osd.4 on smithi071 with /dev/vg_nvme/lv_3... 2024-04-30T08:26:54.222 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-30T08:26:54.244 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:53 smithi071 ceph-mon[33385]: from='osd.3 [v2:172.21.15.32:6810/1384702112,v1:172.21.15.32:6811/1384702112]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-30T08:26:54.244 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:53 smithi071 ceph-mon[33385]: osdmap e24: 4 total, 3 up, 4 in 2024-04-30T08:26:54.244 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:53 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-30T08:26:54.244 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:53 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-30T08:26:54.245 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:53 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:54.245 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:53 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:54.245 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:53 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:26:54.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:53 smithi189 ceph-mon[33038]: from='osd.3 [v2:172.21.15.32:6810/1384702112,v1:172.21.15.32:6811/1384702112]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-30T08:26:54.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:53 smithi189 ceph-mon[33038]: osdmap e24: 4 total, 3 up, 4 in 2024-04-30T08:26:54.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:53 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-30T08:26:54.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:53 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-30T08:26:54.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:53 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:54.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:53 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:26:54.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:53 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:26:54.457 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.b/config 2024-04-30T08:26:54.891 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:54 smithi071 ceph-mon[33385]: purged_snaps scrub starts 2024-04-30T08:26:54.891 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:54 smithi071 ceph-mon[33385]: purged_snaps scrub ok 2024-04-30T08:26:54.891 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:54 smithi071 ceph-mon[33385]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:54.892 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-30T08:26:54.892 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:54 smithi071 ceph-mon[33385]: osd.3 [v2:172.21.15.32:6810/1384702112,v1:172.21.15.32:6811/1384702112] boot 2024-04-30T08:26:54.892 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:54 smithi071 ceph-mon[33385]: osdmap e25: 4 total, 4 up, 4 in 2024-04-30T08:26:54.892 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-30T08:26:55.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:54 smithi032 ceph-mon[28331]: purged_snaps scrub starts 2024-04-30T08:26:55.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:54 smithi032 ceph-mon[28331]: purged_snaps scrub ok 2024-04-30T08:26:55.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:54 smithi032 ceph-mon[28331]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:55.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-30T08:26:55.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:54 smithi032 ceph-mon[28331]: osd.3 [v2:172.21.15.32:6810/1384702112,v1:172.21.15.32:6811/1384702112] boot 2024-04-30T08:26:55.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:54 smithi032 ceph-mon[28331]: osdmap e25: 4 total, 4 up, 4 in 2024-04-30T08:26:55.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-30T08:26:55.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:54 smithi189 ceph-mon[33038]: purged_snaps scrub starts 2024-04-30T08:26:55.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:54 smithi189 ceph-mon[33038]: purged_snaps scrub ok 2024-04-30T08:26:55.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:54 smithi189 ceph-mon[33038]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 92 MiB used, 268 GiB / 268 GiB avail 2024-04-30T08:26:55.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-30T08:26:55.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:54 smithi189 ceph-mon[33038]: osd.3 [v2:172.21.15.32:6810/1384702112,v1:172.21.15.32:6811/1384702112] boot 2024-04-30T08:26:55.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:54 smithi189 ceph-mon[33038]: osdmap e25: 4 total, 4 up, 4 in 2024-04-30T08:26:55.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-30T08:26:56.015 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:55 smithi032 ceph-mon[28331]: osdmap e26: 4 total, 4 up, 4 in 2024-04-30T08:26:56.103 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:55 smithi071 ceph-mon[33385]: osdmap e26: 4 total, 4 up, 4 in 2024-04-30T08:26:56.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:55 smithi189 ceph-mon[33038]: osdmap e26: 4 total, 4 up, 4 in 2024-04-30T08:26:56.849 INFO:teuthology.orchestra.run.smithi071.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-30T08:26:56.850 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-30T08:26:56.850 INFO:teuthology.orchestra.run.smithi071.stderr: stderr: 10+0 records in 2024-04-30T08:26:56.850 INFO:teuthology.orchestra.run.smithi071.stderr:10+0 records out 2024-04-30T08:26:56.850 INFO:teuthology.orchestra.run.smithi071.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130467 s, 804 MB/s 2024-04-30T08:26:56.850 INFO:teuthology.orchestra.run.smithi071.stderr:--> Zapping successful for: 2024-04-30T08:26:57.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:56 smithi032 ceph-mon[28331]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-30T08:26:57.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:56 smithi032 ceph-mon[28331]: osdmap e27: 4 total, 4 up, 4 in 2024-04-30T08:26:57.224 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:56 smithi071 ceph-mon[33385]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-30T08:26:57.224 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:56 smithi071 ceph-mon[33385]: osdmap e27: 4 total, 4 up, 4 in 2024-04-30T08:26:57.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:56 smithi189 ceph-mon[33038]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-30T08:26:57.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:56 smithi189 ceph-mon[33038]: osdmap e27: 4 total, 4 up, 4 in 2024-04-30T08:26:57.509 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi071:vg_nvme/lv_3 2024-04-30T08:26:57.791 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.b/config 2024-04-30T08:26:58.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:26:58 smithi032 ceph-mon[28331]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-04-30T08:26:58.947 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:26:58 smithi071 ceph-mon[33385]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-04-30T08:26:59.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:26:58 smithi189 ceph-mon[33038]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 121 KiB/s, 0 objects/s recovering 2024-04-30T08:26:59.539 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.500+0000 7fcc7c47f700 1 -- 172.21.15.71:0/276716492 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc74107420 msgr2=0x7fcc741098c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:26:59.539 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.500+0000 7fcc7c47f700 1 --2- 172.21.15.71:0/276716492 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc74107420 0x7fcc741098c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fcc64009990 tx=0x7fcc64009ca0 comp rx=0 tx=0).stop 2024-04-30T08:26:59.540 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.500+0000 7fcc7c47f700 1 -- 172.21.15.71:0/276716492 shutdown_connections 2024-04-30T08:26:59.540 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.500+0000 7fcc7c47f700 1 --2- 172.21.15.71:0/276716492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc74109f50 0x7fcc7410c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:26:59.540 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.500+0000 7fcc7c47f700 1 --2- 172.21.15.71:0/276716492 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc740765c0 0x7fcc74106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:26:59.540 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.500+0000 7fcc7c47f700 1 --2- 172.21.15.71:0/276716492 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc74107420 0x7fcc741098c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:26:59.540 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.500+0000 7fcc7c47f700 1 -- 172.21.15.71:0/276716492 >> 172.21.15.71:0/276716492 conn(0x7fcc740fe7e0 msgr2=0x7fcc74100c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:26:59.540 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.500+0000 7fcc7c47f700 1 -- 172.21.15.71:0/276716492 shutdown_connections 2024-04-30T08:26:59.540 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.500+0000 7fcc7c47f700 1 -- 172.21.15.71:0/276716492 wait complete. 2024-04-30T08:26:59.540 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.500+0000 7fcc7c47f700 1 Processor -- start 2024-04-30T08:26:59.540 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.501+0000 7fcc7c47f700 1 -- start start 2024-04-30T08:26:59.540 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.501+0000 7fcc7c47f700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc740765c0 0x7fcc741046a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:26:59.540 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.501+0000 7fcc7c47f700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc74107420 0x7fcc74102d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:26:59.540 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.501+0000 7fcc7c47f700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc74109f50 0x7fcc74103290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:26:59.541 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.501+0000 7fcc7c47f700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7410fd10 con 0x7fcc740765c0 2024-04-30T08:26:59.541 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.501+0000 7fcc7c47f700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7410fb90 con 0x7fcc74107420 2024-04-30T08:26:59.541 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.501+0000 7fcc7c47f700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcc7410fe90 con 0x7fcc74109f50 2024-04-30T08:26:59.541 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.501+0000 7fcc79a1a700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc74107420 0x7fcc74102d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:26:59.541 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.501+0000 7fcc79a1a700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc74107420 0x7fcc74102d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:49856/0 (socket says 172.21.15.71:49856) 2024-04-30T08:26:59.541 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.501+0000 7fcc79a1a700 1 -- 172.21.15.71:0/727425758 learned_addr learned my addr 172.21.15.71:0/727425758 (peer_addr_for_me v2:172.21.15.71:0/0) 2024-04-30T08:26:59.541 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.501+0000 7fcc7aa1c700 1 --2- 172.21.15.71:0/727425758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc74109f50 0x7fcc74103290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:26:59.541 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.501+0000 7fcc79a1a700 1 -- 172.21.15.71:0/727425758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc74109f50 msgr2=0x7fcc74103290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:26:59.541 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.501+0000 7fcc79a1a700 1 --2- 172.21.15.71:0/727425758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc74109f50 0x7fcc74103290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:26:59.541 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.501+0000 7fcc79a1a700 1 -- 172.21.15.71:0/727425758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc740765c0 msgr2=0x7fcc741046a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:26:59.541 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.501+0000 7fcc79a1a700 1 --2- 172.21.15.71:0/727425758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc740765c0 0x7fcc741046a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:26:59.541 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.501+0000 7fcc79a1a700 1 -- 172.21.15.71:0/727425758 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc741039f0 con 0x7fcc74107420 2024-04-30T08:26:59.542 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.501+0000 7fcc79a1a700 1 --2- 172.21.15.71:0/727425758 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc74107420 0x7fcc74102d50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcc64003800 tx=0x7fcc64009f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:26:59.542 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.501+0000 7fcc6b7fe700 1 -- 172.21.15.71:0/727425758 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc64027070 con 0x7fcc74107420 2024-04-30T08:26:59.542 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.502+0000 7fcc6b7fe700 1 -- 172.21.15.71:0/727425758 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc64022470 con 0x7fcc74107420 2024-04-30T08:26:59.542 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.502+0000 7fcc7c47f700 1 -- 172.21.15.71:0/727425758 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc741a48e0 con 0x7fcc74107420 2024-04-30T08:26:59.542 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.502+0000 7fcc6b7fe700 1 -- 172.21.15.71:0/727425758 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc6400f6e0 con 0x7fcc74107420 2024-04-30T08:26:59.542 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.502+0000 7fcc7c47f700 1 -- 172.21.15.71:0/727425758 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc741a4c80 con 0x7fcc74107420 2024-04-30T08:26:59.542 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.503+0000 7fcc7c47f700 1 -- 172.21.15.71:0/727425758 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc74051210 con 0x7fcc74107420 2024-04-30T08:26:59.542 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.504+0000 7fcc6b7fe700 1 -- 172.21.15.71:0/727425758 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fcc6402a070 con 0x7fcc74107420 2024-04-30T08:26:59.542 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.505+0000 7fcc6b7fe700 1 --2- 172.21.15.71:0/727425758 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fcc600afb80 0x7fcc600b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:26:59.542 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.505+0000 7fcc6b7fe700 1 -- 172.21.15.71:0/727425758 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(27..27 src has 1..27) ==== 3669+0+0 (secure 0 0 0) 0x7fcc640de910 con 0x7fcc74107420 2024-04-30T08:26:59.542 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.505+0000 7fcc7a21b700 1 --2- 172.21.15.71:0/727425758 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fcc600afb80 0x7fcc600b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:26:59.542 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.506+0000 7fcc7a21b700 1 --2- 172.21.15.71:0/727425758 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fcc600afb80 0x7fcc600b2090 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcc740762c0 tx=0x7fcc70005c80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:26:59.542 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.510+0000 7fcc6b7fe700 1 -- 172.21.15.71:0/727425758 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcc64013070 con 0x7fcc74107420 2024-04-30T08:26:59.721 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:26:59.720+0000 7fcc7c47f700 1 -- 172.21.15.71:0/727425758 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fcc74063b50 con 0x7fcc600afb80 2024-04-30T08:27:00.604 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:00 smithi071 ceph-mon[33385]: Detected new or changed devices on smithi032 2024-04-30T08:27:00.604 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:00 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:00.604 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:00 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:00.604 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:00 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:27:00.604 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:00 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:27:00.604 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:00 smithi071 ceph-mon[33385]: Adjusting osd_memory_target on smithi032 to 8547M 2024-04-30T08:27:00.604 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:00 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:00.604 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:00 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:27:00.604 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:00 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:27:00.605 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:00 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:00.605 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:00 smithi071 ceph-mon[33385]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-30T08:27:00.605 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:00 smithi071 ceph-mon[33385]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:27:00.605 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:00 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:27:00.605 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:00 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:27:00.605 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:00 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:27:00.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:00 smithi032 ceph-mon[28331]: Detected new or changed devices on smithi032 2024-04-30T08:27:00.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:00 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:00.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:00 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:00.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:00 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:27:00.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:00 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:27:00.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:00 smithi032 ceph-mon[28331]: Adjusting osd_memory_target on smithi032 to 8547M 2024-04-30T08:27:00.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:00 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:00.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:00 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:27:00.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:00 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:27:00.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:00 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:00.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:00 smithi032 ceph-mon[28331]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-30T08:27:00.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:00 smithi032 ceph-mon[28331]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:27:00.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:00 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:27:00.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:00 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:27:00.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:00 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:27:00.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:00 smithi189 ceph-mon[33038]: Detected new or changed devices on smithi032 2024-04-30T08:27:00.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:00 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:00.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:00 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:00.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:00 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:27:00.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:00 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:27:00.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:00 smithi189 ceph-mon[33038]: Adjusting osd_memory_target on smithi032 to 8547M 2024-04-30T08:27:00.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:00 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:00.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:00 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:27:00.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:00 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:27:00.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:00 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:00.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:00 smithi189 ceph-mon[33038]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-30T08:27:00.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:00 smithi189 ceph-mon[33038]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:27:00.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:00 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:27:00.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:00 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:27:00.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:00 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:27:02.941 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:02 smithi071 ceph-mon[33385]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-30T08:27:02.941 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:02 smithi071 ceph-mon[33385]: from='client.? 172.21.15.71:0/1466235476' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1c1e894-074d-4c78-89ef-9e5a114268bd"}]: dispatch 2024-04-30T08:27:02.941 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:02 smithi071 ceph-mon[33385]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1c1e894-074d-4c78-89ef-9e5a114268bd"}]: dispatch 2024-04-30T08:27:02.941 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:02 smithi071 ceph-mon[33385]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e1c1e894-074d-4c78-89ef-9e5a114268bd"}]': finished 2024-04-30T08:27:02.941 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:02 smithi071 ceph-mon[33385]: osdmap e28: 5 total, 4 up, 5 in 2024-04-30T08:27:02.941 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:02 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-30T08:27:02.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:02 smithi032 ceph-mon[28331]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-30T08:27:02.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:02 smithi032 ceph-mon[28331]: from='client.? 172.21.15.71:0/1466235476' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1c1e894-074d-4c78-89ef-9e5a114268bd"}]: dispatch 2024-04-30T08:27:02.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:02 smithi032 ceph-mon[28331]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1c1e894-074d-4c78-89ef-9e5a114268bd"}]: dispatch 2024-04-30T08:27:02.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:02 smithi032 ceph-mon[28331]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e1c1e894-074d-4c78-89ef-9e5a114268bd"}]': finished 2024-04-30T08:27:02.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:02 smithi032 ceph-mon[28331]: osdmap e28: 5 total, 4 up, 5 in 2024-04-30T08:27:02.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:02 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-30T08:27:03.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:02 smithi189 ceph-mon[33038]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-30T08:27:03.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:02 smithi189 ceph-mon[33038]: from='client.? 172.21.15.71:0/1466235476' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1c1e894-074d-4c78-89ef-9e5a114268bd"}]: dispatch 2024-04-30T08:27:03.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:02 smithi189 ceph-mon[33038]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e1c1e894-074d-4c78-89ef-9e5a114268bd"}]: dispatch 2024-04-30T08:27:03.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:02 smithi189 ceph-mon[33038]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e1c1e894-074d-4c78-89ef-9e5a114268bd"}]': finished 2024-04-30T08:27:03.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:02 smithi189 ceph-mon[33038]: osdmap e28: 5 total, 4 up, 5 in 2024-04-30T08:27:03.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:02 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-30T08:27:03.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:03 smithi032 ceph-mon[28331]: from='client.? 172.21.15.71:0/128522655' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:27:04.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:03 smithi071 ceph-mon[33385]: from='client.? 172.21.15.71:0/128522655' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:27:04.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:03 smithi189 ceph-mon[33038]: from='client.? 172.21.15.71:0/128522655' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:27:04.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:04 smithi032 ceph-mon[28331]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-30T08:27:05.006 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:04 smithi071 ceph-mon[33385]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-30T08:27:05.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:04 smithi189 ceph-mon[33038]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-30T08:27:06.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:06 smithi032 ceph-mon[28331]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-30T08:27:06.976 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:06 smithi071 ceph-mon[33385]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-30T08:27:07.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:06 smithi189 ceph-mon[33038]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-30T08:27:08.927 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:08 smithi071 ceph-mon[33385]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-30T08:27:08.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:08 smithi032 ceph-mon[28331]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-30T08:27:09.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:08 smithi189 ceph-mon[33038]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-30T08:27:10.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:10 smithi032 ceph-mon[28331]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-30T08:27:10.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:10 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-30T08:27:10.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:10 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:27:10.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:10 smithi032 ceph-mon[28331]: Deploying daemon osd.4 on smithi071 2024-04-30T08:27:11.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:10 smithi071 ceph-mon[33385]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-30T08:27:11.005 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:10 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-30T08:27:11.005 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:10 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:27:11.005 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:10 smithi071 ceph-mon[33385]: Deploying daemon osd.4 on smithi071 2024-04-30T08:27:11.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:10 smithi189 ceph-mon[33038]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-30T08:27:11.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:10 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-30T08:27:11.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:10 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:27:11.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:10 smithi189 ceph-mon[33038]: Deploying daemon osd.4 on smithi071 2024-04-30T08:27:12.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:12 smithi032 ceph-mon[28331]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-30T08:27:13.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:12 smithi071 ceph-mon[33385]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-30T08:27:13.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:12 smithi189 ceph-mon[33038]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-30T08:27:14.844 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:14 smithi071 ceph-mon[33385]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-30T08:27:15.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:14 smithi189 ceph-mon[33038]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-30T08:27:15.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:14 smithi032 ceph-mon[28331]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-30T08:27:17.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:16 smithi071 ceph-mon[33385]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-30T08:27:17.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:16 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:17.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:16 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:17.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:16 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:27:17.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:16 smithi189 ceph-mon[33038]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-30T08:27:17.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:16 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:17.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:16 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:17.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:16 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:27:17.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:16 smithi032 ceph-mon[28331]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-30T08:27:17.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:16 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:17.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:16 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:17.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:16 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:27:18.748 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:18 smithi071 ceph-mon[33385]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-30T08:27:19.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:18 smithi189 ceph-mon[33038]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-30T08:27:19.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:18 smithi032 ceph-mon[28331]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-30T08:27:19.561 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:27:19.560+0000 7fcc6b7fe700 1 -- 172.21.15.71:0/727425758 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fcc74063b50 con 0x7fcc600afb80 2024-04-30T08:27:19.562 INFO:teuthology.orchestra.run.smithi071.stdout:Created osd(s) 4 on host 'smithi071' 2024-04-30T08:27:19.563 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:27:19.562+0000 7fcc7c47f700 1 -- 172.21.15.71:0/727425758 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fcc600afb80 msgr2=0x7fcc600b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:27:19.563 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:27:19.562+0000 7fcc7c47f700 1 --2- 172.21.15.71:0/727425758 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fcc600afb80 0x7fcc600b2090 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fcc740762c0 tx=0x7fcc70005c80 comp rx=0 tx=0).stop 2024-04-30T08:27:19.563 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:27:19.562+0000 7fcc7c47f700 1 -- 172.21.15.71:0/727425758 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc74107420 msgr2=0x7fcc74102d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:27:19.563 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:27:19.562+0000 7fcc7c47f700 1 --2- 172.21.15.71:0/727425758 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc74107420 0x7fcc74102d50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcc64003800 tx=0x7fcc64009f90 comp rx=0 tx=0).stop 2024-04-30T08:27:19.563 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:27:19.562+0000 7fcc7c47f700 1 -- 172.21.15.71:0/727425758 shutdown_connections 2024-04-30T08:27:19.563 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:27:19.562+0000 7fcc7c47f700 1 --2- 172.21.15.71:0/727425758 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fcc600afb80 0x7fcc600b2090 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:27:19.564 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:27:19.562+0000 7fcc7c47f700 1 --2- 172.21.15.71:0/727425758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc74109f50 0x7fcc74103290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:27:19.564 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:27:19.562+0000 7fcc7c47f700 1 --2- 172.21.15.71:0/727425758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcc740765c0 0x7fcc741046a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:27:19.564 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:27:19.562+0000 7fcc7c47f700 1 --2- 172.21.15.71:0/727425758 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcc74107420 0x7fcc74102d50 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:27:19.564 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:27:19.562+0000 7fcc7c47f700 1 -- 172.21.15.71:0/727425758 >> 172.21.15.71:0/727425758 conn(0x7fcc740fe7e0 msgr2=0x7fcc741054c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:27:19.564 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:27:19.562+0000 7fcc7c47f700 1 -- 172.21.15.71:0/727425758 shutdown_connections 2024-04-30T08:27:19.564 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:27:19.562+0000 7fcc7c47f700 1 -- 172.21.15.71:0/727425758 wait complete. 2024-04-30T08:27:20.233 DEBUG:teuthology.orchestra.run.smithi071:osd.4> sudo journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.4.service 2024-04-30T08:27:20.235 INFO:tasks.cephadm:Deploying osd.5 on smithi189 with /dev/vg_nvme/lv_3... 2024-04-30T08:27:20.235 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-30T08:27:20.495 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.c/config 2024-04-30T08:27:20.776 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:20 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:20.776 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:20 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:20.776 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:20 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:20.776 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:20 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:20.776 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:20 smithi189 ceph-mon[33038]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-30T08:27:20.776 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:20 smithi189 ceph-mon[33038]: from='osd.4 [v2:172.21.15.71:6808/4250534574,v1:172.21.15.71:6809/4250534574]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-30T08:27:20.776 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:20 smithi189 ceph-mon[33038]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-30T08:27:20.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:20 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:20.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:20 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:20.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:20 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:20.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:20 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:20.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:20 smithi032 ceph-mon[28331]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-30T08:27:20.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:20 smithi032 ceph-mon[28331]: from='osd.4 [v2:172.21.15.71:6808/4250534574,v1:172.21.15.71:6809/4250534574]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-30T08:27:20.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:20 smithi032 ceph-mon[28331]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-30T08:27:21.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:20 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:21.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:20 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:21.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:20 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:21.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:20 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:21.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:20 smithi071 ceph-mon[33385]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-30T08:27:21.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:20 smithi071 ceph-mon[33385]: from='osd.4 [v2:172.21.15.71:6808/4250534574,v1:172.21.15.71:6809/4250534574]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-30T08:27:21.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:20 smithi071 ceph-mon[33385]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-30T08:27:21.918 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:21 smithi189 ceph-mon[33038]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-30T08:27:21.918 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:21 smithi189 ceph-mon[33038]: from='osd.4 [v2:172.21.15.71:6808/4250534574,v1:172.21.15.71:6809/4250534574]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-04-30T08:27:21.918 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:21 smithi189 ceph-mon[33038]: osdmap e29: 5 total, 4 up, 5 in 2024-04-30T08:27:21.918 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:21 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-30T08:27:21.918 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:21 smithi189 ceph-mon[33038]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-04-30T08:27:21.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:21 smithi032 ceph-mon[28331]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-30T08:27:21.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:21 smithi032 ceph-mon[28331]: from='osd.4 [v2:172.21.15.71:6808/4250534574,v1:172.21.15.71:6809/4250534574]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-04-30T08:27:21.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:21 smithi032 ceph-mon[28331]: osdmap e29: 5 total, 4 up, 5 in 2024-04-30T08:27:21.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:21 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-30T08:27:21.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:21 smithi032 ceph-mon[28331]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-04-30T08:27:22.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:21 smithi071 ceph-mon[33385]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-30T08:27:22.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:21 smithi071 ceph-mon[33385]: from='osd.4 [v2:172.21.15.71:6808/4250534574,v1:172.21.15.71:6809/4250534574]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-04-30T08:27:22.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:21 smithi071 ceph-mon[33385]: osdmap e29: 5 total, 4 up, 5 in 2024-04-30T08:27:22.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:21 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-30T08:27:22.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:21 smithi071 ceph-mon[33385]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-04-30T08:27:22.004 INFO:journalctl@ceph.osd.4.smithi071.stdout:Apr 30 08:27:21 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-4[44913]: 2024-04-30T08:27:21.605+0000 7f07155d2700 -1 osd.4 0 waiting for initial osdmap 2024-04-30T08:27:22.004 INFO:journalctl@ceph.osd.4.smithi071.stdout:Apr 30 08:27:21 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-4[44913]: 2024-04-30T08:27:21.612+0000 7f0711387700 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-30T08:27:22.482 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-30T08:27:22.482 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-30T08:27:22.482 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-04-30T08:27:22.482 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-04-30T08:27:22.482 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0125872 s, 833 MB/s 2024-04-30T08:27:22.483 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-04-30T08:27:22.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:22 smithi071 ceph-mon[33385]: purged_snaps scrub starts 2024-04-30T08:27:22.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:22 smithi071 ceph-mon[33385]: purged_snaps scrub ok 2024-04-30T08:27:22.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:22 smithi071 ceph-mon[33385]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-04-30T08:27:22.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:22 smithi071 ceph-mon[33385]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-30T08:27:22.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:22 smithi071 ceph-mon[33385]: osdmap e30: 5 total, 4 up, 5 in 2024-04-30T08:27:22.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:22 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-30T08:27:22.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:22 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-30T08:27:22.849 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:22 smithi189 ceph-mon[33038]: purged_snaps scrub starts 2024-04-30T08:27:22.849 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:22 smithi189 ceph-mon[33038]: purged_snaps scrub ok 2024-04-30T08:27:22.849 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:22 smithi189 ceph-mon[33038]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-04-30T08:27:22.849 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:22 smithi189 ceph-mon[33038]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-30T08:27:22.850 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:22 smithi189 ceph-mon[33038]: osdmap e30: 5 total, 4 up, 5 in 2024-04-30T08:27:22.850 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:22 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-30T08:27:22.850 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:22 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-30T08:27:22.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:22 smithi032 ceph-mon[28331]: purged_snaps scrub starts 2024-04-30T08:27:22.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:22 smithi032 ceph-mon[28331]: purged_snaps scrub ok 2024-04-30T08:27:22.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:22 smithi032 ceph-mon[28331]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-04-30T08:27:22.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:22 smithi032 ceph-mon[28331]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-30T08:27:22.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:22 smithi032 ceph-mon[28331]: osdmap e30: 5 total, 4 up, 5 in 2024-04-30T08:27:22.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:22 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-30T08:27:22.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:22 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-30T08:27:23.144 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi189:vg_nvme/lv_3 2024-04-30T08:27:23.386 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.c/config 2024-04-30T08:27:23.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:23 smithi189 ceph-mon[33038]: osd.4 [v2:172.21.15.71:6808/4250534574,v1:172.21.15.71:6809/4250534574] boot 2024-04-30T08:27:23.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:23 smithi189 ceph-mon[33038]: osdmap e31: 5 total, 5 up, 5 in 2024-04-30T08:27:23.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:23 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-30T08:27:23.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:23 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:23.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:23 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:23.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:23 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:27:23.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:23 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:27:23.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:23 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:23.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:23 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:27:23.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:23 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:27:23.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:23 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:23.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:23 smithi032 ceph-mon[28331]: osd.4 [v2:172.21.15.71:6808/4250534574,v1:172.21.15.71:6809/4250534574] boot 2024-04-30T08:27:23.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:23 smithi032 ceph-mon[28331]: osdmap e31: 5 total, 5 up, 5 in 2024-04-30T08:27:23.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:23 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-30T08:27:23.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:23 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:23.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:23 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:23.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:23 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:27:23.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:23 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:27:23.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:23 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:23.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:23 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:27:23.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:23 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:27:23.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:23 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:24.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:23 smithi071 ceph-mon[33385]: osd.4 [v2:172.21.15.71:6808/4250534574,v1:172.21.15.71:6809/4250534574] boot 2024-04-30T08:27:24.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:23 smithi071 ceph-mon[33385]: osdmap e31: 5 total, 5 up, 5 in 2024-04-30T08:27:24.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:23 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-30T08:27:24.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:23 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:24.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:23 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:24.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:23 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:27:24.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:23 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:27:24.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:23 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:24.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:23 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:27:24.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:23 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:27:24.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:23 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:24.785 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:24 smithi189 ceph-mon[33038]: Detected new or changed devices on smithi071 2024-04-30T08:27:24.785 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:24 smithi189 ceph-mon[33038]: Adjusting osd_memory_target on smithi071 to 8544M 2024-04-30T08:27:24.785 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:24 smithi189 ceph-mon[33038]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-30T08:27:24.785 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:24 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:27:24.785 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:24 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:27:24.785 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:24 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:27:24.786 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:24 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:24.786 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:24 smithi189 ceph-mon[33038]: osdmap e32: 5 total, 5 up, 5 in 2024-04-30T08:27:24.931 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:24 smithi071 ceph-mon[33385]: Detected new or changed devices on smithi071 2024-04-30T08:27:24.931 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:24 smithi071 ceph-mon[33385]: Adjusting osd_memory_target on smithi071 to 8544M 2024-04-30T08:27:24.931 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:24 smithi071 ceph-mon[33385]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-30T08:27:24.931 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:24 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:27:24.931 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:24 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:27:24.931 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:24 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:27:24.932 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:24 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:24.932 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:24 smithi071 ceph-mon[33385]: osdmap e32: 5 total, 5 up, 5 in 2024-04-30T08:27:24.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:24 smithi032 ceph-mon[28331]: Detected new or changed devices on smithi071 2024-04-30T08:27:24.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:24 smithi032 ceph-mon[28331]: Adjusting osd_memory_target on smithi071 to 8544M 2024-04-30T08:27:24.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:24 smithi032 ceph-mon[28331]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 119 MiB used, 357 GiB / 358 GiB avail 2024-04-30T08:27:24.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:24 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:27:24.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:24 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:27:24.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:24 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:27:24.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:24 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:24.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:24 smithi032 ceph-mon[28331]: osdmap e32: 5 total, 5 up, 5 in 2024-04-30T08:27:25.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.014+0000 7f38af900700 1 -- 172.21.15.189:0/2019349685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38a8102930 msgr2=0x7f38a8102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:27:25.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.014+0000 7f38af900700 1 --2- 172.21.15.189:0/2019349685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38a8102930 0x7f38a8102d40 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f3898009990 tx=0x7f3898009ca0 comp rx=0 tx=0).stop 2024-04-30T08:27:25.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.014+0000 7f38af900700 1 -- 172.21.15.189:0/2019349685 shutdown_connections 2024-04-30T08:27:25.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.014+0000 7f38af900700 1 --2- 172.21.15.189:0/2019349685 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38a8104400 0x7f38a810acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:27:25.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.014+0000 7f38af900700 1 --2- 172.21.15.189:0/2019349685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38a8102930 0x7f38a8102d40 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:27:25.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.014+0000 7f38af900700 1 --2- 172.21.15.189:0/2019349685 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f38a8103a60 0x7f38a8103e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:27:25.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.014+0000 7f38af900700 1 -- 172.21.15.189:0/2019349685 >> 172.21.15.189:0/2019349685 conn(0x7f38a80fdf70 msgr2=0x7f38a8100420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:27:25.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.014+0000 7f38af900700 1 -- 172.21.15.189:0/2019349685 shutdown_connections 2024-04-30T08:27:25.017 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.014+0000 7f38af900700 1 -- 172.21.15.189:0/2019349685 wait complete. 2024-04-30T08:27:25.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.014+0000 7f38af900700 1 Processor -- start 2024-04-30T08:27:25.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.015+0000 7f38af900700 1 -- start start 2024-04-30T08:27:25.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.015+0000 7f38af900700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f38a8102930 0x7f38a80ffea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:27:25.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.015+0000 7f38af900700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38a8103a60 0x7f38a819f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:27:25.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.015+0000 7f38af900700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38a8104400 0x7f38a81a1c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:27:25.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.015+0000 7f38af900700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f38a810e3f0 con 0x7f38a8104400 2024-04-30T08:27:25.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.015+0000 7f38af900700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f38a810e270 con 0x7f38a8102930 2024-04-30T08:27:25.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.015+0000 7f38af900700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f38a810e570 con 0x7f38a8103a60 2024-04-30T08:27:25.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.015+0000 7f38ace9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38a8103a60 0x7f38a819f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:27:25.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.015+0000 7f38ace9b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38a8103a60 0x7f38a819f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:58646/0 (socket says 172.21.15.189:58646) 2024-04-30T08:27:25.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.015+0000 7f38ace9b700 1 -- 172.21.15.189:0/3298523433 learned_addr learned my addr 172.21.15.189:0/3298523433 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-30T08:27:25.018 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.015+0000 7f38ad69c700 1 --2- 172.21.15.189:0/3298523433 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f38a8102930 0x7f38a80ffea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:27:25.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.015+0000 7f38ace9b700 1 -- 172.21.15.189:0/3298523433 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f38a8102930 msgr2=0x7f38a80ffea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:27:25.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.015+0000 7f38ace9b700 1 --2- 172.21.15.189:0/3298523433 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f38a8102930 0x7f38a80ffea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:27:25.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.015+0000 7f38ace9b700 1 -- 172.21.15.189:0/3298523433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38a8104400 msgr2=0x7f38a81a1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:27:25.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.015+0000 7f38ace9b700 1 --2- 172.21.15.189:0/3298523433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38a8104400 0x7f38a81a1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:27:25.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.015+0000 7f38ace9b700 1 -- 172.21.15.189:0/3298523433 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38a81a2460 con 0x7f38a8103a60 2024-04-30T08:27:25.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.015+0000 7f38ace9b700 1 --2- 172.21.15.189:0/3298523433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38a8103a60 0x7f38a819f6a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f38a400d950 tx=0x7f38a400dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:27:25.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.016+0000 7f389e7fc700 1 -- 172.21.15.189:0/3298523433 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a4016070 con 0x7f38a8103a60 2024-04-30T08:27:25.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.016+0000 7f38af900700 1 -- 172.21.15.189:0/3298523433 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38a81a2750 con 0x7f38a8103a60 2024-04-30T08:27:25.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.016+0000 7f389e7fc700 1 -- 172.21.15.189:0/3298523433 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38a4009df0 con 0x7f38a8103a60 2024-04-30T08:27:25.019 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.016+0000 7f389e7fc700 1 -- 172.21.15.189:0/3298523433 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38a400f3c0 con 0x7f38a8103a60 2024-04-30T08:27:25.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.016+0000 7f38af900700 1 -- 172.21.15.189:0/3298523433 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38a81a2d10 con 0x7f38a8103a60 2024-04-30T08:27:25.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.017+0000 7f38af900700 1 -- 172.21.15.189:0/3298523433 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38a8051210 con 0x7f38a8103a60 2024-04-30T08:27:25.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.018+0000 7f389e7fc700 1 -- 172.21.15.189:0/3298523433 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f38a4019070 con 0x7f38a8103a60 2024-04-30T08:27:25.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.019+0000 7f389e7fc700 1 --2- 172.21.15.189:0/3298523433 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f38940afb80 0x7f38940b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:27:25.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.019+0000 7f389e7fc700 1 -- 172.21.15.189:0/3298523433 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(32..32 src has 1..32) ==== 4101+0+0 (secure 0 0 0) 0x7f38a40dd200 con 0x7f38a8103a60 2024-04-30T08:27:25.020 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.019+0000 7f38ad69c700 1 --2- 172.21.15.189:0/3298523433 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f38940afb80 0x7f38940b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:27:25.021 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.021+0000 7f38ad69c700 1 --2- 172.21.15.189:0/3298523433 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f38940afb80 0x7f38940b2090 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f38980038a0 tx=0x7f389801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:27:25.024 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.024+0000 7f389e7fc700 1 -- 172.21.15.189:0/3298523433 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f38a40a6e50 con 0x7f38a8103a60 2024-04-30T08:27:25.229 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:25.228+0000 7f38af900700 1 -- 172.21.15.189:0/3298523433 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f38a8063b50 con 0x7f38940afb80 2024-04-30T08:27:25.895 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:25 smithi189 ceph-mon[33038]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:27:25.895 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:25 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:27:25.895 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:25 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:27:25.895 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:25 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:27:25.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:25 smithi032 ceph-mon[28331]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:27:25.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:25 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:27:25.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:25 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:27:25.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:25 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:27:26.009 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:25 smithi071 ceph-mon[33385]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:27:26.009 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:25 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:27:26.009 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:25 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:27:26.009 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:25 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:27:26.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:26 smithi189 ceph-mon[33038]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:26.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:26 smithi032 ceph-mon[28331]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:27.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:26 smithi071 ceph-mon[33385]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:28.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:28 smithi032 ceph-mon[28331]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:28.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:28 smithi032 ceph-mon[28331]: from='client.? 172.21.15.189:0/2883079208' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12c022d4-fac6-40c4-ae7a-691571bee5d0"}]: dispatch 2024-04-30T08:27:28.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:28 smithi032 ceph-mon[28331]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12c022d4-fac6-40c4-ae7a-691571bee5d0"}]: dispatch 2024-04-30T08:27:28.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:28 smithi032 ceph-mon[28331]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12c022d4-fac6-40c4-ae7a-691571bee5d0"}]': finished 2024-04-30T08:27:28.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:28 smithi032 ceph-mon[28331]: osdmap e33: 6 total, 5 up, 6 in 2024-04-30T08:27:28.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:28 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-30T08:27:29.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:28 smithi071 ceph-mon[33385]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:29.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:28 smithi071 ceph-mon[33385]: from='client.? 172.21.15.189:0/2883079208' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12c022d4-fac6-40c4-ae7a-691571bee5d0"}]: dispatch 2024-04-30T08:27:29.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:28 smithi071 ceph-mon[33385]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12c022d4-fac6-40c4-ae7a-691571bee5d0"}]: dispatch 2024-04-30T08:27:29.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:28 smithi071 ceph-mon[33385]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12c022d4-fac6-40c4-ae7a-691571bee5d0"}]': finished 2024-04-30T08:27:29.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:28 smithi071 ceph-mon[33385]: osdmap e33: 6 total, 5 up, 6 in 2024-04-30T08:27:29.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:28 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-30T08:27:29.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:28 smithi189 ceph-mon[33038]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:29.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:28 smithi189 ceph-mon[33038]: from='client.? 172.21.15.189:0/2883079208' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12c022d4-fac6-40c4-ae7a-691571bee5d0"}]: dispatch 2024-04-30T08:27:29.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:28 smithi189 ceph-mon[33038]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "12c022d4-fac6-40c4-ae7a-691571bee5d0"}]: dispatch 2024-04-30T08:27:29.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:28 smithi189 ceph-mon[33038]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "12c022d4-fac6-40c4-ae7a-691571bee5d0"}]': finished 2024-04-30T08:27:29.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:28 smithi189 ceph-mon[33038]: osdmap e33: 6 total, 5 up, 6 in 2024-04-30T08:27:29.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:28 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-30T08:27:29.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:29 smithi032 ceph-mon[28331]: from='client.? 172.21.15.189:0/1192957308' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:27:30.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:29 smithi071 ceph-mon[33385]: from='client.? 172.21.15.189:0/1192957308' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:27:30.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:29 smithi189 ceph-mon[33038]: from='client.? 172.21.15.189:0/1192957308' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:27:30.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:30 smithi032 ceph-mon[28331]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:31.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:30 smithi071 ceph-mon[33385]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:31.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:30 smithi189 ceph-mon[33038]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:32.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:32 smithi032 ceph-mon[28331]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:32.961 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:32 smithi189 ceph-mon[33038]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:33.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:32 smithi071 ceph-mon[33385]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:34.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:34 smithi189 ceph-mon[33038]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:34.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:34 smithi032 ceph-mon[28331]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:35.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:34 smithi071 ceph-mon[33385]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:35.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:35 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-30T08:27:35.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:35 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:27:36.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:35 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-30T08:27:36.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:35 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:27:36.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:35 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-30T08:27:36.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:35 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:27:36.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:36 smithi032 ceph-mon[28331]: Deploying daemon osd.5 on smithi189 2024-04-30T08:27:36.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:36 smithi032 ceph-mon[28331]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:36.955 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:36 smithi189 ceph-mon[33038]: Deploying daemon osd.5 on smithi189 2024-04-30T08:27:36.955 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:36 smithi189 ceph-mon[33038]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:37.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:36 smithi071 ceph-mon[33385]: Deploying daemon osd.5 on smithi189 2024-04-30T08:27:37.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:36 smithi071 ceph-mon[33385]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:38.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:38 smithi032 ceph-mon[28331]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:39.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:38 smithi071 ceph-mon[33385]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:39.036 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:38 smithi189 ceph-mon[33038]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:41.002 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:40 smithi189 ceph-mon[33038]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:41.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:40 smithi071 ceph-mon[33385]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:41.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:40 smithi032 ceph-mon[28331]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:42.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:42 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:42.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:42 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:42.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:42 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:27:42.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:42 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:42.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:42 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:42.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:42 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:27:42.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:42 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:42.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:42 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:42.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:42 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:27:43.401 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:43 smithi189 ceph-mon[33038]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:43.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:43 smithi032 ceph-mon[28331]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:43.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:43 smithi071 ceph-mon[33385]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:45.075 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:45.074+0000 7f389e7fc700 1 -- 172.21.15.189:0/3298523433 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f38a8063b50 con 0x7f38940afb80 2024-04-30T08:27:45.076 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 5 on host 'smithi189' 2024-04-30T08:27:45.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:45.077+0000 7f38af900700 1 -- 172.21.15.189:0/3298523433 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f38940afb80 msgr2=0x7f38940b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:27:45.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:45.077+0000 7f38af900700 1 --2- 172.21.15.189:0/3298523433 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f38940afb80 0x7f38940b2090 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f38980038a0 tx=0x7f389801a040 comp rx=0 tx=0).stop 2024-04-30T08:27:45.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:45.077+0000 7f38af900700 1 -- 172.21.15.189:0/3298523433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38a8103a60 msgr2=0x7f38a819f6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:27:45.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:45.077+0000 7f38af900700 1 --2- 172.21.15.189:0/3298523433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38a8103a60 0x7f38a819f6a0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f38a400d950 tx=0x7f38a400dc60 comp rx=0 tx=0).stop 2024-04-30T08:27:45.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:45.077+0000 7f38af900700 1 -- 172.21.15.189:0/3298523433 shutdown_connections 2024-04-30T08:27:45.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:45.077+0000 7f38af900700 1 --2- 172.21.15.189:0/3298523433 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f38940afb80 0x7f38940b2090 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:27:45.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:45.077+0000 7f38af900700 1 --2- 172.21.15.189:0/3298523433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38a8104400 0x7f38a81a1c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:27:45.078 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:45.077+0000 7f38af900700 1 --2- 172.21.15.189:0/3298523433 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f38a8102930 0x7f38a80ffea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:27:45.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:45.077+0000 7f38af900700 1 --2- 172.21.15.189:0/3298523433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38a8103a60 0x7f38a819f6a0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:27:45.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:45.077+0000 7f38af900700 1 -- 172.21.15.189:0/3298523433 >> 172.21.15.189:0/3298523433 conn(0x7f38a80fdf70 msgr2=0x7f38a81076c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:27:45.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:45.077+0000 7f38af900700 1 -- 172.21.15.189:0/3298523433 shutdown_connections 2024-04-30T08:27:45.079 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:27:45.077+0000 7f38af900700 1 -- 172.21.15.189:0/3298523433 wait complete. 2024-04-30T08:27:45.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:45 smithi189 ceph-mon[33038]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:45.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:45 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:45.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:45 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:45.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:45 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:27:45.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:45 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:27:45.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:45 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:45.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:45 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:45.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:45 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:45.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:45 smithi032 ceph-mon[28331]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:45.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:45 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:45.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:45 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:45.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:45 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:27:45.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:45 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:27:45.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:45 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:45.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:45 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:45.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:45 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:45.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:45 smithi071 ceph-mon[33385]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:45.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:45 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:45.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:45 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:45.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:45 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:27:45.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:45 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:27:45.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:45 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:45.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:45 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:45.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:45 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:45.765 DEBUG:teuthology.orchestra.run.smithi189:osd.5> sudo journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.5.service 2024-04-30T08:27:45.767 INFO:tasks.cephadm:Deploying osd.6 on smithi032 with /dev/vg_nvme/lv_2... 2024-04-30T08:27:45.767 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-30T08:27:46.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:46 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:27:46.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:46 smithi032 ceph-mon[28331]: from='osd.5 [v2:172.21.15.189:6808/4245346198,v1:172.21.15.189:6809/4245346198]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-30T08:27:46.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:46 smithi032 ceph-mon[28331]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-30T08:27:46.224 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:27:46.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:46 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:27:46.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:46 smithi071 ceph-mon[33385]: from='osd.5 [v2:172.21.15.189:6808/4245346198,v1:172.21.15.189:6809/4245346198]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-30T08:27:46.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:46 smithi071 ceph-mon[33385]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-30T08:27:46.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:46 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:27:46.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:46 smithi189 ceph-mon[33038]: from='osd.5 [v2:172.21.15.189:6808/4245346198,v1:172.21.15.189:6809/4245346198]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-30T08:27:46.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:46 smithi189 ceph-mon[33038]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-30T08:27:47.223 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:47 smithi189 ceph-mon[33038]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:47.223 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:47 smithi189 ceph-mon[33038]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-30T08:27:47.223 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:47 smithi189 ceph-mon[33038]: osdmap e34: 6 total, 5 up, 6 in 2024-04-30T08:27:47.223 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:47 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-30T08:27:47.223 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:47 smithi189 ceph-mon[33038]: from='osd.5 [v2:172.21.15.189:6808/4245346198,v1:172.21.15.189:6809/4245346198]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-30T08:27:47.224 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:47 smithi189 ceph-mon[33038]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-30T08:27:47.224 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 30 08:27:47 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-5[45058]: 2024-04-30T08:27:47.079+0000 7f45f762c700 -1 osd.5 0 waiting for initial osdmap 2024-04-30T08:27:47.224 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 30 08:27:47 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-5[45058]: 2024-04-30T08:27:47.087+0000 7f45f33e1700 -1 osd.5 35 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-30T08:27:47.351 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:47 smithi032 ceph-mon[28331]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:47.351 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:47 smithi032 ceph-mon[28331]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-30T08:27:47.351 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:47 smithi032 ceph-mon[28331]: osdmap e34: 6 total, 5 up, 6 in 2024-04-30T08:27:47.351 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:47 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-30T08:27:47.351 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:47 smithi032 ceph-mon[28331]: from='osd.5 [v2:172.21.15.189:6808/4245346198,v1:172.21.15.189:6809/4245346198]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-30T08:27:47.351 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:47 smithi032 ceph-mon[28331]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-30T08:27:47.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:47 smithi071 ceph-mon[33385]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:47.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:47 smithi071 ceph-mon[33385]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-30T08:27:47.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:47 smithi071 ceph-mon[33385]: osdmap e34: 6 total, 5 up, 6 in 2024-04-30T08:27:47.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:47 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-30T08:27:47.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:47 smithi071 ceph-mon[33385]: from='osd.5 [v2:172.21.15.189:6808/4245346198,v1:172.21.15.189:6809/4245346198]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-30T08:27:47.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:47 smithi071 ceph-mon[33385]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-30T08:27:48.287 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:48 smithi032 ceph-mon[28331]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-30T08:27:48.287 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:48 smithi032 ceph-mon[28331]: osdmap e35: 6 total, 5 up, 6 in 2024-04-30T08:27:48.287 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:48 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-30T08:27:48.287 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:48 smithi032 ceph-mon[28331]: osd.5 [v2:172.21.15.189:6808/4245346198,v1:172.21.15.189:6809/4245346198] boot 2024-04-30T08:27:48.287 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:48 smithi032 ceph-mon[28331]: osdmap e36: 6 total, 6 up, 6 in 2024-04-30T08:27:48.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:48 smithi189 ceph-mon[33038]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-30T08:27:48.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:48 smithi189 ceph-mon[33038]: osdmap e35: 6 total, 5 up, 6 in 2024-04-30T08:27:48.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:48 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-30T08:27:48.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:48 smithi189 ceph-mon[33038]: osd.5 [v2:172.21.15.189:6808/4245346198,v1:172.21.15.189:6809/4245346198] boot 2024-04-30T08:27:48.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:48 smithi189 ceph-mon[33038]: osdmap e36: 6 total, 6 up, 6 in 2024-04-30T08:27:48.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:48 smithi071 ceph-mon[33385]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-30T08:27:48.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:48 smithi071 ceph-mon[33385]: osdmap e35: 6 total, 5 up, 6 in 2024-04-30T08:27:48.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:48 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-30T08:27:48.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:48 smithi071 ceph-mon[33385]: osd.5 [v2:172.21.15.189:6808/4245346198,v1:172.21.15.189:6809/4245346198] boot 2024-04-30T08:27:48.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:48 smithi071 ceph-mon[33385]: osdmap e36: 6 total, 6 up, 6 in 2024-04-30T08:27:49.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:49 smithi032 ceph-mon[28331]: purged_snaps scrub starts 2024-04-30T08:27:49.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:49 smithi032 ceph-mon[28331]: purged_snaps scrub ok 2024-04-30T08:27:49.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:49 smithi032 ceph-mon[28331]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:49.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:49 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-30T08:27:49.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:49 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:49.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:49 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:49.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:49 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:27:49.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:49 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:27:49.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:49 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:49.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:49 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:27:49.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:49 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:27:49.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:49 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:49.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:49 smithi032 ceph-mon[28331]: osdmap e37: 6 total, 6 up, 6 in 2024-04-30T08:27:49.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:49 smithi071 ceph-mon[33385]: purged_snaps scrub starts 2024-04-30T08:27:49.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:49 smithi071 ceph-mon[33385]: purged_snaps scrub ok 2024-04-30T08:27:49.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:49 smithi071 ceph-mon[33385]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:49.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:49 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-30T08:27:49.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:49 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:49.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:49 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:49.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:49 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:27:49.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:49 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:27:49.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:49 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:49.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:49 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:27:49.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:49 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:27:49.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:49 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:49.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:49 smithi071 ceph-mon[33385]: osdmap e37: 6 total, 6 up, 6 in 2024-04-30T08:27:49.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:49 smithi189 ceph-mon[33038]: purged_snaps scrub starts 2024-04-30T08:27:49.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:49 smithi189 ceph-mon[33038]: purged_snaps scrub ok 2024-04-30T08:27:49.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:49 smithi189 ceph-mon[33038]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 146 MiB used, 447 GiB / 447 GiB avail 2024-04-30T08:27:49.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:49 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-30T08:27:49.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:49 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:49.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:49 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:49.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:49 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:27:49.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:49 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:27:49.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:49 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:49.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:49 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:27:49.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:49 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:27:49.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:49 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:27:49.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:49 smithi189 ceph-mon[33038]: osdmap e37: 6 total, 6 up, 6 in 2024-04-30T08:27:49.574 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-30T08:27:49.575 INFO:teuthology.orchestra.run.smithi032.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-30T08:27:49.575 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 10+0 records in 2024-04-30T08:27:49.575 INFO:teuthology.orchestra.run.smithi032.stderr:10+0 records out 2024-04-30T08:27:49.575 INFO:teuthology.orchestra.run.smithi032.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0311179 s, 337 MB/s 2024-04-30T08:27:49.575 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping successful for: 2024-04-30T08:27:50.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:50 smithi189 ceph-mon[33038]: Detected new or changed devices on smithi189 2024-04-30T08:27:50.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:50 smithi189 ceph-mon[33038]: Adjusting osd_memory_target on smithi189 to 8544M 2024-04-30T08:27:50.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:50 smithi189 ceph-mon[33038]: osdmap e38: 6 total, 6 up, 6 in 2024-04-30T08:27:50.404 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:50 smithi032 ceph-mon[28331]: Detected new or changed devices on smithi189 2024-04-30T08:27:50.404 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:50 smithi032 ceph-mon[28331]: Adjusting osd_memory_target on smithi189 to 8544M 2024-04-30T08:27:50.404 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:50 smithi032 ceph-mon[28331]: osdmap e38: 6 total, 6 up, 6 in 2024-04-30T08:27:50.479 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi032:vg_nvme/lv_2 2024-04-30T08:27:50.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:50 smithi071 ceph-mon[33385]: Detected new or changed devices on smithi189 2024-04-30T08:27:50.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:50 smithi071 ceph-mon[33385]: Adjusting osd_memory_target on smithi189 to 8544M 2024-04-30T08:27:50.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:50 smithi071 ceph-mon[33385]: osdmap e38: 6 total, 6 up, 6 in 2024-04-30T08:27:50.942 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:27:51.424 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:51 smithi032 ceph-mon[28331]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:27:51.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:51 smithi071 ceph-mon[33385]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:27:51.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:51 smithi189 ceph-mon[33038]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 172 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:27:53.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.152+0000 7f56e9800700 1 -- 172.21.15.32:0/3549938552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56e406c220 msgr2=0x7f56e4106e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:27:53.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.152+0000 7f56e9800700 1 --2- 172.21.15.32:0/3549938552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56e406c220 0x7f56e4106e00 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f56d4009990 tx=0x7f56d4009ca0 comp rx=0 tx=0).stop 2024-04-30T08:27:53.154 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:53 smithi032 ceph-mon[28331]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 127 KiB/s, 0 objects/s recovering 2024-04-30T08:27:53.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.152+0000 7f56e9800700 1 -- 172.21.15.32:0/3549938552 shutdown_connections 2024-04-30T08:27:53.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.152+0000 7f56e9800700 1 --2- 172.21.15.32:0/3549938552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56e4109f40 0x7f56e410c450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:27:53.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.152+0000 7f56e9800700 1 --2- 172.21.15.32:0/3549938552 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56e406c220 0x7f56e4106e00 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:27:53.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.152+0000 7f56e9800700 1 --2- 172.21.15.32:0/3549938552 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56e41073d0 0x7f56e41098b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:27:53.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.152+0000 7f56e9800700 1 -- 172.21.15.32:0/3549938552 >> 172.21.15.32:0/3549938552 conn(0x7f56e40fe7d0 msgr2=0x7f56e4100c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:27:53.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.152+0000 7f56e9800700 1 -- 172.21.15.32:0/3549938552 shutdown_connections 2024-04-30T08:27:53.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.152+0000 7f56e9800700 1 -- 172.21.15.32:0/3549938552 wait complete. 2024-04-30T08:27:53.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.153+0000 7f56e9800700 1 Processor -- start 2024-04-30T08:27:53.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.153+0000 7f56e9800700 1 -- start start 2024-04-30T08:27:53.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.154+0000 7f56e9800700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56e406c220 0x7f56e4104650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:27:53.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.154+0000 7f56e9800700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56e41073d0 0x7f56e4102d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:27:53.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.154+0000 7f56e9800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56e4109f40 0x7f56e4103240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:27:53.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.154+0000 7f56e9800700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f56e410fd20 con 0x7f56e406c220 2024-04-30T08:27:53.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.154+0000 7f56e9800700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f56e410fba0 con 0x7f56e41073d0 2024-04-30T08:27:53.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.154+0000 7f56e9800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f56e410fea0 con 0x7f56e4109f40 2024-04-30T08:27:53.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.154+0000 7f56e2a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56e406c220 0x7f56e4104650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:27:53.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.154+0000 7f56e2a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56e406c220 0x7f56e4104650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44904/0 (socket says 172.21.15.32:44904) 2024-04-30T08:27:53.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.154+0000 7f56e2a00700 1 -- 172.21.15.32:0/2720527981 learned_addr learned my addr 172.21.15.32:0/2720527981 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:27:53.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.154+0000 7f56e2000700 1 --2- 172.21.15.32:0/2720527981 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56e41073d0 0x7f56e4102d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:27:53.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.154+0000 7f56e3400700 1 --2- 172.21.15.32:0/2720527981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56e4109f40 0x7f56e4103240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:27:53.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.155+0000 7f56e2000700 1 -- 172.21.15.32:0/2720527981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56e4109f40 msgr2=0x7f56e4103240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:27:53.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.155+0000 7f56e2000700 1 --2- 172.21.15.32:0/2720527981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56e4109f40 0x7f56e4103240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:27:53.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.155+0000 7f56e2000700 1 -- 172.21.15.32:0/2720527981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56e406c220 msgr2=0x7f56e4104650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:27:53.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.155+0000 7f56e2000700 1 --2- 172.21.15.32:0/2720527981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56e406c220 0x7f56e4104650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:27:53.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.155+0000 7f56e2000700 1 -- 172.21.15.32:0/2720527981 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56e41039a0 con 0x7f56e41073d0 2024-04-30T08:27:53.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.155+0000 7f56e2000700 1 --2- 172.21.15.32:0/2720527981 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56e41073d0 0x7f56e4102d00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f56d000e9f0 tx=0x7f56d000ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:27:53.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.155+0000 7f56db400700 1 -- 172.21.15.32:0/2720527981 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d000cb70 con 0x7f56e41073d0 2024-04-30T08:27:53.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.155+0000 7f56db400700 1 -- 172.21.15.32:0/2720527981 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f56d000ee90 con 0x7f56e41073d0 2024-04-30T08:27:53.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.155+0000 7f56e9800700 1 -- 172.21.15.32:0/2720527981 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56e41a4940 con 0x7f56e41073d0 2024-04-30T08:27:53.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.155+0000 7f56db400700 1 -- 172.21.15.32:0/2720527981 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d0003d40 con 0x7f56e41073d0 2024-04-30T08:27:53.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.156+0000 7f56e9800700 1 -- 172.21.15.32:0/2720527981 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56e41a4e70 con 0x7f56e41073d0 2024-04-30T08:27:53.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.157+0000 7f56e9800700 1 -- 172.21.15.32:0/2720527981 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56e4051210 con 0x7f56e41073d0 2024-04-30T08:27:53.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.159+0000 7f56db400700 1 -- 172.21.15.32:0/2720527981 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f56d0003ee0 con 0x7f56e41073d0 2024-04-30T08:27:53.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.160+0000 7f56db400700 1 --2- 172.21.15.32:0/2720527981 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f56c80afc00 0x7f56c80b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:27:53.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.160+0000 7f56db400700 1 -- 172.21.15.32:0/2720527981 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(38..38 src has 1..38) ==== 4533+0+0 (secure 0 0 0) 0x7f56d00dde70 con 0x7f56e41073d0 2024-04-30T08:27:53.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.161+0000 7f56e2a00700 1 --2- 172.21.15.32:0/2720527981 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f56c80afc00 0x7f56c80b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:27:53.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.161+0000 7f56e2a00700 1 --2- 172.21.15.32:0/2720527981 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f56c80afc00 0x7f56c80b2110 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f56d400b2e0 tx=0x7f56d4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:27:53.165 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.165+0000 7f56db400700 1 -- 172.21.15.32:0/2720527981 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f56d00a7870 con 0x7f56e41073d0 2024-04-30T08:27:53.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:27:53.408+0000 7f56e9800700 1 -- 172.21.15.32:0/2720527981 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f56e4063b50 con 0x7f56c80afc00 2024-04-30T08:27:53.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:53 smithi071 ceph-mon[33385]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 127 KiB/s, 0 objects/s recovering 2024-04-30T08:27:53.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:53 smithi189 ceph-mon[33038]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 127 KiB/s, 0 objects/s recovering 2024-04-30T08:27:54.410 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:27:54.410 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:27:54.411 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:27:54.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:27:54.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:27:54.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:27:54.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:27:54.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:27:54.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:27:55.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:55 smithi032 ceph-mon[28331]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:27:55.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:55 smithi032 ceph-mon[28331]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-30T08:27:55.502 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:55 smithi071 ceph-mon[33385]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:27:55.502 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:55 smithi071 ceph-mon[33385]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-30T08:27:55.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:55 smithi189 ceph-mon[33038]: from='client.24269 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:27:55.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:55 smithi189 ceph-mon[33038]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-30T08:27:57.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:57 smithi032 ceph-mon[28331]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-30T08:27:57.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:57 smithi071 ceph-mon[33385]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-30T08:27:57.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:57 smithi189 ceph-mon[33038]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-04-30T08:27:58.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:58 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/203059202' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b413f8ee-3d4a-415a-a2ba-d68437b7330d"}]: dispatch 2024-04-30T08:27:58.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:58 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/203059202' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b413f8ee-3d4a-415a-a2ba-d68437b7330d"}]': finished 2024-04-30T08:27:58.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:58 smithi032 ceph-mon[28331]: osdmap e39: 7 total, 6 up, 7 in 2024-04-30T08:27:58.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:58 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-30T08:27:58.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:58 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/203059202' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b413f8ee-3d4a-415a-a2ba-d68437b7330d"}]: dispatch 2024-04-30T08:27:58.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:58 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/203059202' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b413f8ee-3d4a-415a-a2ba-d68437b7330d"}]': finished 2024-04-30T08:27:58.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:58 smithi071 ceph-mon[33385]: osdmap e39: 7 total, 6 up, 7 in 2024-04-30T08:27:58.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:58 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-30T08:27:58.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:58 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/203059202' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b413f8ee-3d4a-415a-a2ba-d68437b7330d"}]: dispatch 2024-04-30T08:27:58.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:58 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/203059202' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b413f8ee-3d4a-415a-a2ba-d68437b7330d"}]': finished 2024-04-30T08:27:58.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:58 smithi189 ceph-mon[33038]: osdmap e39: 7 total, 6 up, 7 in 2024-04-30T08:27:58.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:58 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-30T08:27:59.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:59 smithi032 ceph-mon[28331]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-30T08:27:59.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:27:59 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2436251019' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:27:59.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:59 smithi071 ceph-mon[33385]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-30T08:27:59.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:27:59 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2436251019' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:27:59.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:59 smithi189 ceph-mon[33038]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 68 KiB/s, 0 objects/s recovering 2024-04-30T08:27:59.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:27:59 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2436251019' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:28:01.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:01 smithi032 ceph-mon[28331]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-30T08:28:01.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:01 smithi071 ceph-mon[33385]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-30T08:28:01.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:01 smithi189 ceph-mon[33038]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail; 60 KiB/s, 0 objects/s recovering 2024-04-30T08:28:03.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:03 smithi032 ceph-mon[28331]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:03.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:03 smithi071 ceph-mon[33385]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:03.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:03 smithi189 ceph-mon[33038]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:05.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:05 smithi032 ceph-mon[28331]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:05.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:05 smithi071 ceph-mon[33385]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:05.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:05 smithi189 ceph-mon[33038]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:07.420 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:07 smithi032 ceph-mon[28331]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:07.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:07 smithi071 ceph-mon[33385]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:07.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:07 smithi189 ceph-mon[33038]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:08.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:08 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-30T08:28:08.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:08 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:28:08.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:08 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-30T08:28:08.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:08 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:28:08.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:08 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-30T08:28:08.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:08 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:28:09.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:09 smithi032 ceph-mon[28331]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:09.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:09 smithi032 ceph-mon[28331]: Deploying daemon osd.6 on smithi032 2024-04-30T08:28:09.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:09 smithi071 ceph-mon[33385]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:09.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:09 smithi071 ceph-mon[33385]: Deploying daemon osd.6 on smithi032 2024-04-30T08:28:09.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:09 smithi189 ceph-mon[33038]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:09.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:09 smithi189 ceph-mon[33038]: Deploying daemon osd.6 on smithi032 2024-04-30T08:28:11.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:11 smithi032 ceph-mon[28331]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:11.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:11 smithi071 ceph-mon[33385]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:11.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:11 smithi189 ceph-mon[33038]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:13.423 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:13 smithi032 ceph-mon[28331]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:13.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:13 smithi071 ceph-mon[33385]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:13.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:13 smithi189 ceph-mon[33038]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:15.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:15 smithi071 ceph-mon[33385]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:15.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:15 smithi189 ceph-mon[33038]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:15.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:15 smithi032 ceph-mon[28331]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:17.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:17 smithi032 ceph-mon[28331]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:17.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:17 smithi071 ceph-mon[33385]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:17.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:17 smithi189 ceph-mon[33038]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:18.878 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:18 smithi032 ceph-mon[28331]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:18.878 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:18 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:18.878 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:18 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:18.878 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:18 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:28:19.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:18 smithi071 ceph-mon[33385]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:19.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:18 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:19.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:18 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:19.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:18 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:28:19.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:18 smithi189 ceph-mon[33038]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:19.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:18 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:19.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:18 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:19.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:18 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:28:20.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:20 smithi032 ceph-mon[28331]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:21.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:20 smithi071 ceph-mon[33385]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:21.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:20 smithi189 ceph-mon[33038]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:22.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:22 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:22.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:22 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:22.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:22 smithi032 ceph-mon[28331]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:22.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:22 smithi032 ceph-mon[28331]: from='osd.6 [v2:172.21.15.32:6818/1910389190,v1:172.21.15.32:6819/1910389190]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-30T08:28:22.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:22 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:22.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:22 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:22.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:22 smithi071 ceph-mon[33385]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:22.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:22 smithi071 ceph-mon[33385]: from='osd.6 [v2:172.21.15.32:6818/1910389190,v1:172.21.15.32:6819/1910389190]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-30T08:28:22.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:22 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:22.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:22 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:22.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:22 smithi189 ceph-mon[33038]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:22.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:22 smithi189 ceph-mon[33038]: from='osd.6 [v2:172.21.15.32:6818/1910389190,v1:172.21.15.32:6819/1910389190]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-30T08:28:23.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:28:23.430+0000 7f56db400700 1 -- 172.21.15.32:0/2720527981 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f56e4063b50 con 0x7f56c80afc00 2024-04-30T08:28:23.434 INFO:teuthology.orchestra.run.smithi032.stdout:Created osd(s) 6 on host 'smithi032' 2024-04-30T08:28:23.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:28:23.434+0000 7f56e9800700 1 -- 172.21.15.32:0/2720527981 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f56c80afc00 msgr2=0x7f56c80b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:28:23.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:28:23.434+0000 7f56e9800700 1 --2- 172.21.15.32:0/2720527981 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f56c80afc00 0x7f56c80b2110 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f56d400b2e0 tx=0x7f56d4005fb0 comp rx=0 tx=0).stop 2024-04-30T08:28:23.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:28:23.434+0000 7f56e9800700 1 -- 172.21.15.32:0/2720527981 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56e41073d0 msgr2=0x7f56e4102d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:28:23.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:28:23.434+0000 7f56e9800700 1 --2- 172.21.15.32:0/2720527981 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56e41073d0 0x7f56e4102d00 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f56d000e9f0 tx=0x7f56d000ed00 comp rx=0 tx=0).stop 2024-04-30T08:28:23.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:28:23.434+0000 7f56e9800700 1 -- 172.21.15.32:0/2720527981 shutdown_connections 2024-04-30T08:28:23.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:28:23.434+0000 7f56e9800700 1 --2- 172.21.15.32:0/2720527981 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f56c80afc00 0x7f56c80b2110 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:28:23.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:28:23.435+0000 7f56e9800700 1 --2- 172.21.15.32:0/2720527981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56e4109f40 0x7f56e4103240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:28:23.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:28:23.435+0000 7f56e9800700 1 --2- 172.21.15.32:0/2720527981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f56e406c220 0x7f56e4104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:28:23.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:28:23.435+0000 7f56e9800700 1 --2- 172.21.15.32:0/2720527981 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f56e41073d0 0x7f56e4102d00 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:28:23.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:28:23.435+0000 7f56e9800700 1 -- 172.21.15.32:0/2720527981 >> 172.21.15.32:0/2720527981 conn(0x7f56e40fe7d0 msgr2=0x7f56e41055d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:28:23.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:28:23.435+0000 7f56e9800700 1 -- 172.21.15.32:0/2720527981 shutdown_connections 2024-04-30T08:28:23.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:28:23.435+0000 7f56e9800700 1 -- 172.21.15.32:0/2720527981 wait complete. 2024-04-30T08:28:23.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:23 smithi032 ceph-mon[28331]: from='osd.6 [v2:172.21.15.32:6818/1910389190,v1:172.21.15.32:6819/1910389190]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-30T08:28:23.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:23 smithi032 ceph-mon[28331]: osdmap e40: 7 total, 6 up, 7 in 2024-04-30T08:28:23.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:23 smithi032 ceph-mon[28331]: from='osd.6 [v2:172.21.15.32:6818/1910389190,v1:172.21.15.32:6819/1910389190]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-30T08:28:23.694 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:23 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-30T08:28:23.694 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:23 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:28:23.694 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:23 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:28:23.694 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:23 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:23.694 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:23 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:23.694 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:23 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:23.694 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:23 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:28:23.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:23 smithi189 ceph-mon[33038]: from='osd.6 [v2:172.21.15.32:6818/1910389190,v1:172.21.15.32:6819/1910389190]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-30T08:28:23.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:23 smithi189 ceph-mon[33038]: osdmap e40: 7 total, 6 up, 7 in 2024-04-30T08:28:23.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:23 smithi189 ceph-mon[33038]: from='osd.6 [v2:172.21.15.32:6818/1910389190,v1:172.21.15.32:6819/1910389190]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-30T08:28:23.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:23 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-30T08:28:23.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:23 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:28:23.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:23 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:28:23.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:23 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:23.802 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:23 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:23.802 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:23 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:23.802 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:23 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:28:24.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:23 smithi071 ceph-mon[33385]: from='osd.6 [v2:172.21.15.32:6818/1910389190,v1:172.21.15.32:6819/1910389190]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-30T08:28:24.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:23 smithi071 ceph-mon[33385]: osdmap e40: 7 total, 6 up, 7 in 2024-04-30T08:28:24.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:23 smithi071 ceph-mon[33385]: from='osd.6 [v2:172.21.15.32:6818/1910389190,v1:172.21.15.32:6819/1910389190]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-30T08:28:24.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:23 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-30T08:28:24.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:23 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:28:24.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:23 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:28:24.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:23 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:24.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:23 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:24.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:23 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:24.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:23 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:28:24.458 DEBUG:teuthology.orchestra.run.smithi032:osd.6> sudo journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.6.service 2024-04-30T08:28:24.463 INFO:tasks.cephadm:Deploying osd.7 on smithi071 with /dev/vg_nvme/lv_2... 2024-04-30T08:28:24.464 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-30T08:28:24.710 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.b/config 2024-04-30T08:28:24.750 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:24 smithi032 ceph-mon[28331]: from='osd.6 [v2:172.21.15.32:6818/1910389190,v1:172.21.15.32:6819/1910389190]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-30T08:28:24.750 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:24 smithi032 ceph-mon[28331]: osdmap e41: 7 total, 6 up, 7 in 2024-04-30T08:28:24.750 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:24 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-30T08:28:24.750 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:24 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-30T08:28:24.751 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:24 smithi032 ceph-mon[28331]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:24.751 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:24 smithi032 ceph-mon[28331]: from='osd.6 [v2:172.21.15.32:6818/1910389190,v1:172.21.15.32:6819/1910389190]' entity='osd.6' 2024-04-30T08:28:24.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:24 smithi071 ceph-mon[33385]: from='osd.6 [v2:172.21.15.32:6818/1910389190,v1:172.21.15.32:6819/1910389190]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-30T08:28:24.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:24 smithi071 ceph-mon[33385]: osdmap e41: 7 total, 6 up, 7 in 2024-04-30T08:28:24.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:24 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-30T08:28:24.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:24 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-30T08:28:24.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:24 smithi071 ceph-mon[33385]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:24.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:24 smithi071 ceph-mon[33385]: from='osd.6 [v2:172.21.15.32:6818/1910389190,v1:172.21.15.32:6819/1910389190]' entity='osd.6' 2024-04-30T08:28:24.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:24 smithi189 ceph-mon[33038]: from='osd.6 [v2:172.21.15.32:6818/1910389190,v1:172.21.15.32:6819/1910389190]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-30T08:28:24.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:24 smithi189 ceph-mon[33038]: osdmap e41: 7 total, 6 up, 7 in 2024-04-30T08:28:24.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:24 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-30T08:28:24.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:24 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-30T08:28:24.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:24 smithi189 ceph-mon[33038]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 173 MiB used, 536 GiB / 536 GiB avail 2024-04-30T08:28:24.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:24 smithi189 ceph-mon[33038]: from='osd.6 [v2:172.21.15.32:6818/1910389190,v1:172.21.15.32:6819/1910389190]' entity='osd.6' 2024-04-30T08:28:25.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:25 smithi189 ceph-mon[33038]: purged_snaps scrub starts 2024-04-30T08:28:25.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:25 smithi189 ceph-mon[33038]: purged_snaps scrub ok 2024-04-30T08:28:25.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:25 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-30T08:28:25.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:25 smithi189 ceph-mon[33038]: osd.6 [v2:172.21.15.32:6818/1910389190,v1:172.21.15.32:6819/1910389190] boot 2024-04-30T08:28:25.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:25 smithi189 ceph-mon[33038]: osdmap e42: 7 total, 7 up, 7 in 2024-04-30T08:28:25.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:25 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-30T08:28:25.832 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:25 smithi032 ceph-mon[28331]: purged_snaps scrub starts 2024-04-30T08:28:25.832 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:25 smithi032 ceph-mon[28331]: purged_snaps scrub ok 2024-04-30T08:28:25.832 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:25 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-30T08:28:25.832 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:25 smithi032 ceph-mon[28331]: osd.6 [v2:172.21.15.32:6818/1910389190,v1:172.21.15.32:6819/1910389190] boot 2024-04-30T08:28:25.832 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:25 smithi032 ceph-mon[28331]: osdmap e42: 7 total, 7 up, 7 in 2024-04-30T08:28:25.832 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:25 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-30T08:28:25.868 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:25 smithi071 ceph-mon[33385]: purged_snaps scrub starts 2024-04-30T08:28:25.869 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:25 smithi071 ceph-mon[33385]: purged_snaps scrub ok 2024-04-30T08:28:25.869 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:25 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-30T08:28:25.869 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:25 smithi071 ceph-mon[33385]: osd.6 [v2:172.21.15.32:6818/1910389190,v1:172.21.15.32:6819/1910389190] boot 2024-04-30T08:28:25.869 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:25 smithi071 ceph-mon[33385]: osdmap e42: 7 total, 7 up, 7 in 2024-04-30T08:28:25.869 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:25 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-30T08:28:26.750 INFO:teuthology.orchestra.run.smithi071.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-30T08:28:26.750 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-30T08:28:26.750 INFO:teuthology.orchestra.run.smithi071.stderr: stderr: 10+0 records in 2024-04-30T08:28:26.750 INFO:teuthology.orchestra.run.smithi071.stderr:10+0 records out 2024-04-30T08:28:26.750 INFO:teuthology.orchestra.run.smithi071.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0140081 s, 749 MB/s 2024-04-30T08:28:26.750 INFO:teuthology.orchestra.run.smithi071.stderr:--> Zapping successful for: 2024-04-30T08:28:26.774 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:26 smithi071 ceph-mon[33385]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:26.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:26 smithi189 ceph-mon[33038]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:26.863 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:26 smithi032 ceph-mon[28331]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:27.415 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi071:vg_nvme/lv_2 2024-04-30T08:28:27.670 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.b/config 2024-04-30T08:28:27.704 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:27 smithi071 ceph-mon[33385]: osdmap e43: 7 total, 7 up, 7 in 2024-04-30T08:28:27.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:27 smithi189 ceph-mon[33038]: osdmap e43: 7 total, 7 up, 7 in 2024-04-30T08:28:27.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:27 smithi032 ceph-mon[28331]: osdmap e43: 7 total, 7 up, 7 in 2024-04-30T08:28:28.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:28 smithi071 ceph-mon[33385]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:28.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:28 smithi189 ceph-mon[33038]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:28.889 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:28 smithi032 ceph-mon[28331]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:29.340 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.339+0000 7f0a1a6f5700 1 -- 172.21.15.71:0/3073663360 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a14104c40 msgr2=0x7f0a1410b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:28:29.340 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.339+0000 7f0a1a6f5700 1 --2- 172.21.15.71:0/3073663360 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a14104c40 0x7f0a1410b6d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c00b0a0 tx=0x7f0a0c00b3b0 comp rx=0 tx=0).stop 2024-04-30T08:28:29.340 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.339+0000 7f0a1a6f5700 1 -- 172.21.15.71:0/3073663360 shutdown_connections 2024-04-30T08:28:29.340 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.339+0000 7f0a1a6f5700 1 --2- 172.21.15.71:0/3073663360 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a14104c40 0x7f0a1410b6d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:28:29.341 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.339+0000 7f0a1a6f5700 1 --2- 172.21.15.71:0/3073663360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a14103700 0x7f0a14103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:28:29.341 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.339+0000 7f0a1a6f5700 1 --2- 172.21.15.71:0/3073663360 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a141040e0 0x7f0a141045b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:28:29.341 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.339+0000 7f0a1a6f5700 1 -- 172.21.15.71:0/3073663360 >> 172.21.15.71:0/3073663360 conn(0x7f0a140fdf70 msgr2=0x7f0a14100420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:28:29.341 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.339+0000 7f0a1a6f5700 1 -- 172.21.15.71:0/3073663360 shutdown_connections 2024-04-30T08:28:29.341 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.339+0000 7f0a1a6f5700 1 -- 172.21.15.71:0/3073663360 wait complete. 2024-04-30T08:28:29.343 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.340+0000 7f0a1a6f5700 1 Processor -- start 2024-04-30T08:28:29.343 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.340+0000 7f0a1a6f5700 1 -- start start 2024-04-30T08:28:29.343 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.340+0000 7f0a1a6f5700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a14103700 0x7f0a14102dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:28:29.343 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.340+0000 7f0a1a6f5700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a141040e0 0x7f0a1419f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:28:29.343 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.340+0000 7f0a1a6f5700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a14104c40 0x7f0a141a1b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:28:29.343 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.340+0000 7f0a1a6f5700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0a1410ec40 con 0x7f0a14104c40 2024-04-30T08:28:29.343 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.340+0000 7f0a1a6f5700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f0a1410eac0 con 0x7f0a141040e0 2024-04-30T08:28:29.343 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.340+0000 7f0a1a6f5700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0a1410edc0 con 0x7f0a14103700 2024-04-30T08:28:29.343 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.340+0000 7f0a137fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a141040e0 0x7f0a1419f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:28:29.343 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.340+0000 7f0a137fe700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a141040e0 0x7f0a1419f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:36420/0 (socket says 172.21.15.71:36420) 2024-04-30T08:28:29.344 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.340+0000 7f0a137fe700 1 -- 172.21.15.71:0/874081312 learned_addr learned my addr 172.21.15.71:0/874081312 (peer_addr_for_me v2:172.21.15.71:0/0) 2024-04-30T08:28:29.344 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.341+0000 7f0a13fff700 1 --2- 172.21.15.71:0/874081312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a14103700 0x7f0a14102dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:28:29.344 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.341+0000 7f0a137fe700 1 -- 172.21.15.71:0/874081312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a14103700 msgr2=0x7f0a14102dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:28:29.344 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.341+0000 7f0a137fe700 1 --2- 172.21.15.71:0/874081312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a14103700 0x7f0a14102dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:28:29.344 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.341+0000 7f0a137fe700 1 -- 172.21.15.71:0/874081312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a14104c40 msgr2=0x7f0a141a1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:28:29.344 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.341+0000 7f0a137fe700 1 --2- 172.21.15.71:0/874081312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a14104c40 0x7f0a141a1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:28:29.344 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.341+0000 7f0a137fe700 1 -- 172.21.15.71:0/874081312 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a141a2360 con 0x7f0a141040e0 2024-04-30T08:28:29.344 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.341+0000 7f0a13fff700 1 --2- 172.21.15.71:0/874081312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a14103700 0x7f0a14102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-30T08:28:29.344 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.341+0000 7f0a137fe700 1 --2- 172.21.15.71:0/874081312 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a141040e0 0x7f0a1419f5c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0a0800c920 tx=0x7f0a0800cce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:28:29.344 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.341+0000 7f0a117fa700 1 -- 172.21.15.71:0/874081312 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a08007b00 con 0x7f0a141040e0 2024-04-30T08:28:29.344 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.341+0000 7f0a117fa700 1 -- 172.21.15.71:0/874081312 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a08004ce0 con 0x7f0a141040e0 2024-04-30T08:28:29.344 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.341+0000 7f0a1a6f5700 1 -- 172.21.15.71:0/874081312 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a141a2650 con 0x7f0a141040e0 2024-04-30T08:28:29.344 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.341+0000 7f0a117fa700 1 -- 172.21.15.71:0/874081312 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a08005650 con 0x7f0a141040e0 2024-04-30T08:28:29.345 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.341+0000 7f0a1a6f5700 1 -- 172.21.15.71:0/874081312 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a141a2b90 con 0x7f0a141040e0 2024-04-30T08:28:29.345 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.342+0000 7f0a1a6f5700 1 -- 172.21.15.71:0/874081312 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a14051210 con 0x7f0a141040e0 2024-04-30T08:28:29.345 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.343+0000 7f0a117fa700 1 -- 172.21.15.71:0/874081312 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f0a08004750 con 0x7f0a141040e0 2024-04-30T08:28:29.345 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.344+0000 7f0a117fa700 1 --2- 172.21.15.71:0/874081312 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f09fc0afb80 0x7f09fc0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:28:29.345 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.344+0000 7f0a117fa700 1 -- 172.21.15.71:0/874081312 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(43..43 src has 1..43) ==== 4965+0+0 (secure 0 0 0) 0x7f0a080ddf30 con 0x7f0a141040e0 2024-04-30T08:28:29.346 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.344+0000 7f0a13fff700 1 --2- 172.21.15.71:0/874081312 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f09fc0afb80 0x7f09fc0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:28:29.346 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.345+0000 7f0a13fff700 1 --2- 172.21.15.71:0/874081312 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f09fc0afb80 0x7f09fc0b2090 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0a1406c1e0 tx=0x7f0a04009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:28:29.348 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.348+0000 7f0a117fa700 1 -- 172.21.15.71:0/874081312 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0a08014030 con 0x7f0a141040e0 2024-04-30T08:28:29.558 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:29.558+0000 7f0a1a6f5700 1 -- 172.21.15.71:0/874081312 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f0a14063b50 con 0x7f09fc0afb80 2024-04-30T08:28:30.484 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:30 smithi071 ceph-mon[33385]: Detected new or changed devices on smithi032 2024-04-30T08:28:30.484 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:30 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:30.484 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:30 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:30.485 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:30 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:28:30.485 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:30 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:28:30.485 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:30 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:28:30.485 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:30 smithi071 ceph-mon[33385]: Adjusting osd_memory_target on smithi032 to 5698M 2024-04-30T08:28:30.485 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:30 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:30.485 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:30 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:28:30.485 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:30 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:28:30.485 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:30 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:30.485 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:30 smithi071 ceph-mon[33385]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:28:30.485 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:30 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:28:30.485 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:30 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:28:30.485 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:30 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:28:30.485 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:30 smithi071 ceph-mon[33385]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:30.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:30 smithi032 ceph-mon[28331]: Detected new or changed devices on smithi032 2024-04-30T08:28:30.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:30 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:30.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:30 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:30.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:30 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:28:30.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:30 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:28:30.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:30 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:28:30.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:30 smithi032 ceph-mon[28331]: Adjusting osd_memory_target on smithi032 to 5698M 2024-04-30T08:28:30.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:30 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:30.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:30 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:28:30.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:30 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:28:30.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:30 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:30.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:30 smithi032 ceph-mon[28331]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:28:30.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:30 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:28:30.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:30 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:28:30.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:30 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:28:30.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:30 smithi032 ceph-mon[28331]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:30.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:30 smithi189 ceph-mon[33038]: Detected new or changed devices on smithi032 2024-04-30T08:28:30.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:30 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:30.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:30 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:30.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:30 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:28:30.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:30 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:28:30.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:30 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:28:30.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:30 smithi189 ceph-mon[33038]: Adjusting osd_memory_target on smithi032 to 5698M 2024-04-30T08:28:30.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:30 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:30.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:30 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:28:30.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:30 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:28:30.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:30 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:30.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:30 smithi189 ceph-mon[33038]: from='client.24296 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:28:30.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:30 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:28:30.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:30 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:28:30.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:30 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:28:30.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:30 smithi189 ceph-mon[33038]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:32.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:32 smithi032 ceph-mon[28331]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:32.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:32 smithi032 ceph-mon[28331]: from='client.? 172.21.15.71:0/2960684149' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aac64fb3-fc13-4126-b842-7b06ad5ce238"}]: dispatch 2024-04-30T08:28:32.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:32 smithi032 ceph-mon[28331]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aac64fb3-fc13-4126-b842-7b06ad5ce238"}]: dispatch 2024-04-30T08:28:32.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:32 smithi032 ceph-mon[28331]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aac64fb3-fc13-4126-b842-7b06ad5ce238"}]': finished 2024-04-30T08:28:32.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:32 smithi032 ceph-mon[28331]: osdmap e44: 8 total, 7 up, 8 in 2024-04-30T08:28:32.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:32 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-30T08:28:33.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:32 smithi071 ceph-mon[33385]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:33.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:32 smithi071 ceph-mon[33385]: from='client.? 172.21.15.71:0/2960684149' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aac64fb3-fc13-4126-b842-7b06ad5ce238"}]: dispatch 2024-04-30T08:28:33.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:32 smithi071 ceph-mon[33385]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aac64fb3-fc13-4126-b842-7b06ad5ce238"}]: dispatch 2024-04-30T08:28:33.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:32 smithi071 ceph-mon[33385]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aac64fb3-fc13-4126-b842-7b06ad5ce238"}]': finished 2024-04-30T08:28:33.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:32 smithi071 ceph-mon[33385]: osdmap e44: 8 total, 7 up, 8 in 2024-04-30T08:28:33.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:32 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-30T08:28:33.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:32 smithi189 ceph-mon[33038]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:33.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:32 smithi189 ceph-mon[33038]: from='client.? 172.21.15.71:0/2960684149' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aac64fb3-fc13-4126-b842-7b06ad5ce238"}]: dispatch 2024-04-30T08:28:33.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:32 smithi189 ceph-mon[33038]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "aac64fb3-fc13-4126-b842-7b06ad5ce238"}]: dispatch 2024-04-30T08:28:33.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:32 smithi189 ceph-mon[33038]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "aac64fb3-fc13-4126-b842-7b06ad5ce238"}]': finished 2024-04-30T08:28:33.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:32 smithi189 ceph-mon[33038]: osdmap e44: 8 total, 7 up, 8 in 2024-04-30T08:28:33.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:32 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-30T08:28:33.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:33 smithi032 ceph-mon[28331]: from='client.? 172.21.15.71:0/2068219687' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:28:34.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:33 smithi071 ceph-mon[33385]: from='client.? 172.21.15.71:0/2068219687' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:28:34.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:33 smithi189 ceph-mon[33038]: from='client.? 172.21.15.71:0/2068219687' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:28:35.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:34 smithi071 ceph-mon[33385]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:35.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:34 smithi189 ceph-mon[33038]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:35.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:34 smithi032 ceph-mon[28331]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:36.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:36 smithi071 ceph-mon[33385]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:37.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:36 smithi189 ceph-mon[33038]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:37.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:36 smithi032 ceph-mon[28331]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:39.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:38 smithi071 ceph-mon[33385]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:39.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:38 smithi189 ceph-mon[33038]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:39.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:38 smithi032 ceph-mon[28331]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:40.736 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:40 smithi071 ceph-mon[33385]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:40.736 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:40 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-30T08:28:40.736 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:40 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:28:40.736 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:40 smithi071 ceph-mon[33385]: Deploying daemon osd.7 on smithi071 2024-04-30T08:28:41.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:40 smithi189 ceph-mon[33038]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:41.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:40 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-30T08:28:41.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:40 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:28:41.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:40 smithi189 ceph-mon[33038]: Deploying daemon osd.7 on smithi071 2024-04-30T08:28:41.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:40 smithi032 ceph-mon[28331]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:41.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:40 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-30T08:28:41.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:40 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:28:41.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:40 smithi032 ceph-mon[28331]: Deploying daemon osd.7 on smithi071 2024-04-30T08:28:42.975 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:42 smithi071 ceph-mon[33385]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:43.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:42 smithi189 ceph-mon[33038]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:43.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:42 smithi032 ceph-mon[28331]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:44.891 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:44 smithi071 ceph-mon[33385]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:45.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:44 smithi189 ceph-mon[33038]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:45.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:44 smithi032 ceph-mon[28331]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:47.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:46 smithi071 ceph-mon[33385]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:47.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:46 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:47.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:46 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:47.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:46 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:28:47.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:46 smithi189 ceph-mon[33038]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:47.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:46 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:47.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:46 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:47.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:46 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:28:47.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:46 smithi032 ceph-mon[28331]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:47.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:46 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:47.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:46 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:47.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:46 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:28:49.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:48 smithi071 ceph-mon[33385]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:49.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:48 smithi189 ceph-mon[33038]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:49.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:48 smithi032 ceph-mon[28331]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:49.996 INFO:teuthology.orchestra.run.smithi071.stdout:Created osd(s) 7 on host 'smithi071' 2024-04-30T08:28:49.996 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:49.994+0000 7f0a117fa700 1 -- 172.21.15.71:0/874081312 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0a14063b50 con 0x7f09fc0afb80 2024-04-30T08:28:49.999 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:49.996+0000 7f0a1a6f5700 1 -- 172.21.15.71:0/874081312 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f09fc0afb80 msgr2=0x7f09fc0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:28:49.999 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:49.996+0000 7f0a1a6f5700 1 --2- 172.21.15.71:0/874081312 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f09fc0afb80 0x7f09fc0b2090 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0a1406c1e0 tx=0x7f0a04009260 comp rx=0 tx=0).stop 2024-04-30T08:28:49.999 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:49.996+0000 7f0a1a6f5700 1 -- 172.21.15.71:0/874081312 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a141040e0 msgr2=0x7f0a1419f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:28:49.999 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:49.996+0000 7f0a1a6f5700 1 --2- 172.21.15.71:0/874081312 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a141040e0 0x7f0a1419f5c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f0a0800c920 tx=0x7f0a0800cce0 comp rx=0 tx=0).stop 2024-04-30T08:28:49.999 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:49.996+0000 7f0a1a6f5700 1 -- 172.21.15.71:0/874081312 shutdown_connections 2024-04-30T08:28:49.999 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:49.996+0000 7f0a1a6f5700 1 --2- 172.21.15.71:0/874081312 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f09fc0afb80 0x7f09fc0b2090 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:28:49.999 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:49.996+0000 7f0a1a6f5700 1 --2- 172.21.15.71:0/874081312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a14104c40 0x7f0a141a1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:28:49.999 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:49.996+0000 7f0a1a6f5700 1 --2- 172.21.15.71:0/874081312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a14103700 0x7f0a14102dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:28:49.999 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:49.996+0000 7f0a1a6f5700 1 --2- 172.21.15.71:0/874081312 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a141040e0 0x7f0a1419f5c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:28:49.999 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:49.996+0000 7f0a1a6f5700 1 -- 172.21.15.71:0/874081312 >> 172.21.15.71:0/874081312 conn(0x7f0a140fdf70 msgr2=0x7f0a141076b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:28:49.999 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:49.997+0000 7f0a1a6f5700 1 -- 172.21.15.71:0/874081312 shutdown_connections 2024-04-30T08:28:49.999 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:28:49.997+0000 7f0a1a6f5700 1 -- 172.21.15.71:0/874081312 wait complete. 2024-04-30T08:28:50.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:50 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:50.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:50 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:50.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:50 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:28:50.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:50 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:28:50.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:50 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:50.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:50 smithi071 ceph-mon[33385]: from='osd.7 [v2:172.21.15.71:6816/692607380,v1:172.21.15.71:6817/692607380]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-30T08:28:50.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:50 smithi071 ceph-mon[33385]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-30T08:28:50.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:50 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:28:50.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:50 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:50.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:50 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:50.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:50 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:50.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:50 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:50.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:50 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:28:50.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:50 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:28:50.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:50 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:50.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:50 smithi189 ceph-mon[33038]: from='osd.7 [v2:172.21.15.71:6816/692607380,v1:172.21.15.71:6817/692607380]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-30T08:28:50.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:50 smithi189 ceph-mon[33038]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-30T08:28:50.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:50 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:28:50.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:50 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:50.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:50 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:50.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:50 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:50.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:50 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:50.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:50 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:28:50.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:50 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:28:50.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:50 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:50.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:50 smithi032 ceph-mon[28331]: from='osd.7 [v2:172.21.15.71:6816/692607380,v1:172.21.15.71:6817/692607380]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-30T08:28:50.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:50 smithi032 ceph-mon[28331]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-30T08:28:50.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:50 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:28:50.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:50 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:50.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:50 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:50.687 DEBUG:teuthology.orchestra.run.smithi071:osd.7> sudo journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.7.service 2024-04-30T08:28:50.689 INFO:tasks.cephadm:Deploying osd.8 on smithi189 with /dev/vg_nvme/lv_2... 2024-04-30T08:28:50.690 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-30T08:28:50.932 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.c/config 2024-04-30T08:28:51.206 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:51 smithi071 ceph-mon[33385]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:51.206 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:51 smithi071 ceph-mon[33385]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-30T08:28:51.206 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:51 smithi071 ceph-mon[33385]: from='osd.7 [v2:172.21.15.71:6816/692607380,v1:172.21.15.71:6817/692607380]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-04-30T08:28:51.206 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:51 smithi071 ceph-mon[33385]: osdmap e45: 8 total, 7 up, 8 in 2024-04-30T08:28:51.206 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:51 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-30T08:28:51.206 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:51 smithi071 ceph-mon[33385]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-04-30T08:28:51.361 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:51 smithi189 ceph-mon[33038]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:51.361 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:51 smithi189 ceph-mon[33038]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-30T08:28:51.361 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:51 smithi189 ceph-mon[33038]: from='osd.7 [v2:172.21.15.71:6816/692607380,v1:172.21.15.71:6817/692607380]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-04-30T08:28:51.361 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:51 smithi189 ceph-mon[33038]: osdmap e45: 8 total, 7 up, 8 in 2024-04-30T08:28:51.361 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:51 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-30T08:28:51.361 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:51 smithi189 ceph-mon[33038]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-04-30T08:28:51.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:51 smithi032 ceph-mon[28331]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:51.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:51 smithi032 ceph-mon[28331]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-30T08:28:51.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:51 smithi032 ceph-mon[28331]: from='osd.7 [v2:172.21.15.71:6816/692607380,v1:172.21.15.71:6817/692607380]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-04-30T08:28:51.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:51 smithi032 ceph-mon[28331]: osdmap e45: 8 total, 7 up, 8 in 2024-04-30T08:28:51.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:51 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-30T08:28:51.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:51 smithi032 ceph-mon[28331]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-04-30T08:28:51.956 INFO:journalctl@ceph.osd.7.smithi071.stdout:Apr 30 08:28:51 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-7[51771]: 2024-04-30T08:28:51.547+0000 7f7772cb0700 -1 osd.7 0 waiting for initial osdmap 2024-04-30T08:28:51.956 INFO:journalctl@ceph.osd.7.smithi071.stdout:Apr 30 08:28:51 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-7[51771]: 2024-04-30T08:28:51.558+0000 7f776aa5d700 -1 osd.7 47 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-30T08:28:52.211 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:52 smithi071 ceph-mon[33385]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-04-30T08:28:52.211 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:52 smithi071 ceph-mon[33385]: osdmap e46: 8 total, 7 up, 8 in 2024-04-30T08:28:52.211 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:52 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-30T08:28:52.211 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:52 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-30T08:28:52.211 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:52 smithi071 ceph-mon[33385]: osdmap e47: 8 total, 7 up, 8 in 2024-04-30T08:28:52.211 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:52 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-30T08:28:52.211 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:52 smithi071 ceph-mon[33385]: from='osd.7 ' entity='osd.7' 2024-04-30T08:28:52.211 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:52 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-30T08:28:52.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:52 smithi189 ceph-mon[33038]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-04-30T08:28:52.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:52 smithi189 ceph-mon[33038]: osdmap e46: 8 total, 7 up, 8 in 2024-04-30T08:28:52.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:52 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-30T08:28:52.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:52 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-30T08:28:52.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:52 smithi189 ceph-mon[33038]: osdmap e47: 8 total, 7 up, 8 in 2024-04-30T08:28:52.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:52 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-30T08:28:52.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:52 smithi189 ceph-mon[33038]: from='osd.7 ' entity='osd.7' 2024-04-30T08:28:52.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:52 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-30T08:28:52.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:52 smithi032 ceph-mon[28331]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-04-30T08:28:52.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:52 smithi032 ceph-mon[28331]: osdmap e46: 8 total, 7 up, 8 in 2024-04-30T08:28:52.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-30T08:28:52.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-30T08:28:52.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:52 smithi032 ceph-mon[28331]: osdmap e47: 8 total, 7 up, 8 in 2024-04-30T08:28:52.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-30T08:28:52.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:52 smithi032 ceph-mon[28331]: from='osd.7 ' entity='osd.7' 2024-04-30T08:28:52.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-30T08:28:52.930 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-30T08:28:52.930 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-30T08:28:52.931 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-04-30T08:28:52.931 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-04-30T08:28:52.931 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124818 s, 840 MB/s 2024-04-30T08:28:52.931 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-04-30T08:28:53.275 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:53 smithi071 ceph-mon[33385]: purged_snaps scrub starts 2024-04-30T08:28:53.276 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:53 smithi071 ceph-mon[33385]: purged_snaps scrub ok 2024-04-30T08:28:53.276 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:53 smithi071 ceph-mon[33385]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:53.276 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:53 smithi071 ceph-mon[33385]: osd.7 [v2:172.21.15.71:6816/692607380,v1:172.21.15.71:6817/692607380] boot 2024-04-30T08:28:53.276 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:53 smithi071 ceph-mon[33385]: osdmap e48: 8 total, 8 up, 8 in 2024-04-30T08:28:53.276 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:53 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-30T08:28:53.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:53 smithi032 ceph-mon[28331]: purged_snaps scrub starts 2024-04-30T08:28:53.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:53 smithi032 ceph-mon[28331]: purged_snaps scrub ok 2024-04-30T08:28:53.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:53 smithi032 ceph-mon[28331]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:53.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:53 smithi032 ceph-mon[28331]: osd.7 [v2:172.21.15.71:6816/692607380,v1:172.21.15.71:6817/692607380] boot 2024-04-30T08:28:53.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:53 smithi032 ceph-mon[28331]: osdmap e48: 8 total, 8 up, 8 in 2024-04-30T08:28:53.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:53 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-30T08:28:53.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:53 smithi189 ceph-mon[33038]: purged_snaps scrub starts 2024-04-30T08:28:53.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:53 smithi189 ceph-mon[33038]: purged_snaps scrub ok 2024-04-30T08:28:53.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:53 smithi189 ceph-mon[33038]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 204 MiB used, 626 GiB / 626 GiB avail 2024-04-30T08:28:53.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:53 smithi189 ceph-mon[33038]: osd.7 [v2:172.21.15.71:6816/692607380,v1:172.21.15.71:6817/692607380] boot 2024-04-30T08:28:53.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:53 smithi189 ceph-mon[33038]: osdmap e48: 8 total, 8 up, 8 in 2024-04-30T08:28:53.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:53 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-30T08:28:53.674 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi189:vg_nvme/lv_2 2024-04-30T08:28:53.916 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.c/config 2024-04-30T08:28:54.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:54 smithi071 ceph-mon[33385]: osdmap e49: 8 total, 8 up, 8 in 2024-04-30T08:28:54.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:54 smithi071 ceph-mon[33385]: pgmap v201: 1 pgs: 1 remapped; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-30T08:28:54.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:54 smithi071 ceph-mon[33385]: Detected new or changed devices on smithi071 2024-04-30T08:28:54.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:54.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:54.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:28:54.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:28:54.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:28:54.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:54 smithi071 ceph-mon[33385]: Adjusting osd_memory_target on smithi071 to 5696M 2024-04-30T08:28:54.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:54.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:28:54.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:28:54.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:54.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:54 smithi189 ceph-mon[33038]: osdmap e49: 8 total, 8 up, 8 in 2024-04-30T08:28:54.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:54 smithi189 ceph-mon[33038]: pgmap v201: 1 pgs: 1 remapped; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-30T08:28:54.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:54 smithi189 ceph-mon[33038]: Detected new or changed devices on smithi071 2024-04-30T08:28:54.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:54.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:54.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:28:54.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:28:54.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:28:54.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:54 smithi189 ceph-mon[33038]: Adjusting osd_memory_target on smithi071 to 5696M 2024-04-30T08:28:54.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:54.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:28:54.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:28:54.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:54.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:54 smithi032 ceph-mon[28331]: osdmap e49: 8 total, 8 up, 8 in 2024-04-30T08:28:54.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:54 smithi032 ceph-mon[28331]: pgmap v201: 1 pgs: 1 remapped; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-30T08:28:54.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:54 smithi032 ceph-mon[28331]: Detected new or changed devices on smithi071 2024-04-30T08:28:54.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:54.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:54.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:28:54.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:28:54.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:28:54.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:54 smithi032 ceph-mon[28331]: Adjusting osd_memory_target on smithi071 to 5696M 2024-04-30T08:28:54.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:54.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:28:54.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:28:54.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:28:55.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.580+0000 7f68d55ab700 1 -- 172.21.15.189:0/2628030852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68d0104c60 msgr2=0x7f68d010b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:28:55.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.580+0000 7f68d55ab700 1 --2- 172.21.15.189:0/2628030852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68d0104c60 0x7f68d010b510 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f68c8009990 tx=0x7f68c8009ca0 comp rx=0 tx=0).stop 2024-04-30T08:28:55.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.580+0000 7f68d55ab700 1 -- 172.21.15.189:0/2628030852 shutdown_connections 2024-04-30T08:28:55.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.580+0000 7f68d55ab700 1 --2- 172.21.15.189:0/2628030852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68d0104c60 0x7f68d010b510 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:28:55.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.580+0000 7f68d55ab700 1 --2- 172.21.15.189:0/2628030852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68d0103190 0x7f68d01035a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:28:55.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.580+0000 7f68d55ab700 1 --2- 172.21.15.189:0/2628030852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68d01042c0 0x7f68d01046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:28:55.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.580+0000 7f68d55ab700 1 -- 172.21.15.189:0/2628030852 >> 172.21.15.189:0/2628030852 conn(0x7f68d00fe7d0 msgr2=0x7f68d0100c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:28:55.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.580+0000 7f68d55ab700 1 -- 172.21.15.189:0/2628030852 shutdown_connections 2024-04-30T08:28:55.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.580+0000 7f68d55ab700 1 -- 172.21.15.189:0/2628030852 wait complete. 2024-04-30T08:28:55.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.581+0000 7f68d55ab700 1 Processor -- start 2024-04-30T08:28:55.584 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.581+0000 7f68d55ab700 1 -- start start 2024-04-30T08:28:55.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.581+0000 7f68d55ab700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68d0103190 0x7f68d0199760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:28:55.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.581+0000 7f68d55ab700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68d01042c0 0x7f68d0199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:28:55.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.581+0000 7f68d55ab700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68d0104c60 0x7f68d01a0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:28:55.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.581+0000 7f68d55ab700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f68d010ec80 con 0x7f68d0104c60 2024-04-30T08:28:55.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.581+0000 7f68d55ab700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f68d010eb00 con 0x7f68d01042c0 2024-04-30T08:28:55.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.581+0000 7f68d55ab700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f68d010ee00 con 0x7f68d0103190 2024-04-30T08:28:55.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.582+0000 7f68ceffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68d0103190 0x7f68d0199760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:28:55.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.582+0000 7f68ceffd700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68d0103190 0x7f68d0199760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:47414/0 (socket says 172.21.15.189:47414) 2024-04-30T08:28:55.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.582+0000 7f68ceffd700 1 -- 172.21.15.189:0/804684528 learned_addr learned my addr 172.21.15.189:0/804684528 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-30T08:28:55.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.582+0000 7f68c67fc700 1 --2- 172.21.15.189:0/804684528 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68d01042c0 0x7f68d0199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:28:55.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.582+0000 7f68ceffd700 1 -- 172.21.15.189:0/804684528 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68d01042c0 msgr2=0x7f68d0199ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:28:55.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.582+0000 7f68ceffd700 1 --2- 172.21.15.189:0/804684528 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68d01042c0 0x7f68d0199ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:28:55.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.582+0000 7f68ceffd700 1 -- 172.21.15.189:0/804684528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68d0104c60 msgr2=0x7f68d01a0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:28:55.585 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.582+0000 7f68ceffd700 1 --2- 172.21.15.189:0/804684528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68d0104c60 0x7f68d01a0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:28:55.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.582+0000 7f68ceffd700 1 -- 172.21.15.189:0/804684528 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68d006b560 con 0x7f68d0103190 2024-04-30T08:28:55.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.582+0000 7f68ceffd700 1 --2- 172.21.15.189:0/804684528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68d0103190 0x7f68d0199760 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f68c000ada0 tx=0x7f68c000c330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:28:55.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.582+0000 7f68ccff9700 1 -- 172.21.15.189:0/804684528 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c0017020 con 0x7f68d0103190 2024-04-30T08:28:55.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.582+0000 7f68ccff9700 1 -- 172.21.15.189:0/804684528 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f68c000a0d0 con 0x7f68d0103190 2024-04-30T08:28:55.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.582+0000 7f68d55ab700 1 -- 172.21.15.189:0/804684528 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68d006b850 con 0x7f68d0103190 2024-04-30T08:28:55.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.582+0000 7f68ccff9700 1 -- 172.21.15.189:0/804684528 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68c0004280 con 0x7f68d0103190 2024-04-30T08:28:55.586 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.582+0000 7f68d55ab700 1 -- 172.21.15.189:0/804684528 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68d006be10 con 0x7f68d0103190 2024-04-30T08:28:55.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.583+0000 7f68d55ab700 1 -- 172.21.15.189:0/804684528 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f68d0051210 con 0x7f68d0103190 2024-04-30T08:28:55.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.585+0000 7f68ccff9700 1 -- 172.21.15.189:0/804684528 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f68c0004ad0 con 0x7f68d0103190 2024-04-30T08:28:55.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.585+0000 7f68ccff9700 1 --2- 172.21.15.189:0/804684528 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f68bc0afb80 0x7f68bc0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:28:55.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.586+0000 7f68ccff9700 1 -- 172.21.15.189:0/804684528 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(50..50 src has 1..50) ==== 5397+0+0 (secure 0 0 0) 0x7f68c00dda80 con 0x7f68d0103190 2024-04-30T08:28:55.587 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.586+0000 7f68c67fc700 1 --2- 172.21.15.189:0/804684528 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f68bc0afb80 0x7f68bc0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:28:55.588 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.587+0000 7f68c67fc700 1 --2- 172.21.15.189:0/804684528 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f68bc0afb80 0x7f68bc0b2090 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f68d019a810 tx=0x7f68b8009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:28:55.590 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.589+0000 7f68ccff9700 1 -- 172.21.15.189:0/804684528 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f68c0021020 con 0x7f68d0103190 2024-04-30T08:28:55.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:55 smithi032 ceph-mon[28331]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-04-30T08:28:55.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:55 smithi032 ceph-mon[28331]: osdmap e50: 8 total, 8 up, 8 in 2024-04-30T08:28:55.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:55 smithi071 ceph-mon[33385]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-04-30T08:28:55.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:55 smithi071 ceph-mon[33385]: osdmap e50: 8 total, 8 up, 8 in 2024-04-30T08:28:55.795 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:28:55.794+0000 7f68d55ab700 1 -- 172.21.15.189:0/804684528 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f68d0063b50 con 0x7f68bc0afb80 2024-04-30T08:28:55.795 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:55 smithi189 ceph-mon[33038]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-04-30T08:28:55.795 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:55 smithi189 ceph-mon[33038]: osdmap e50: 8 total, 8 up, 8 in 2024-04-30T08:28:56.677 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:56 smithi189 ceph-mon[33038]: pgmap v203: 1 pgs: 1 remapped; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-30T08:28:56.678 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:56 smithi189 ceph-mon[33038]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:28:56.678 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:56 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:28:56.678 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:56 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:28:56.678 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:56 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:28:56.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:56 smithi032 ceph-mon[28331]: pgmap v203: 1 pgs: 1 remapped; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-30T08:28:56.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:56 smithi032 ceph-mon[28331]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:28:56.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:56 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:28:56.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:56 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:28:56.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:56 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:28:56.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:56 smithi071 ceph-mon[33385]: pgmap v203: 1 pgs: 1 remapped; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-30T08:28:56.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:56 smithi071 ceph-mon[33385]: from='client.24304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:28:56.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:56 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:28:56.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:56 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:28:56.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:56 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:28:58.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:58 smithi032 ceph-mon[28331]: pgmap v204: 1 pgs: 1 remapped; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-30T08:28:58.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:58 smithi032 ceph-mon[28331]: from='client.? 172.21.15.189:0/2005216188' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5"}]: dispatch 2024-04-30T08:28:58.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:58 smithi032 ceph-mon[28331]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5"}]: dispatch 2024-04-30T08:28:58.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:58 smithi032 ceph-mon[28331]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5"}]': finished 2024-04-30T08:28:58.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:58 smithi032 ceph-mon[28331]: osdmap e51: 9 total, 8 up, 9 in 2024-04-30T08:28:58.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:58 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-30T08:28:59.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:58 smithi071 ceph-mon[33385]: pgmap v204: 1 pgs: 1 remapped; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-30T08:28:59.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:58 smithi071 ceph-mon[33385]: from='client.? 172.21.15.189:0/2005216188' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5"}]: dispatch 2024-04-30T08:28:59.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:58 smithi071 ceph-mon[33385]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5"}]: dispatch 2024-04-30T08:28:59.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:58 smithi071 ceph-mon[33385]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5"}]': finished 2024-04-30T08:28:59.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:58 smithi071 ceph-mon[33385]: osdmap e51: 9 total, 8 up, 9 in 2024-04-30T08:28:59.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:58 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-30T08:28:59.032 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:58 smithi189 ceph-mon[33038]: pgmap v204: 1 pgs: 1 remapped; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-30T08:28:59.032 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:58 smithi189 ceph-mon[33038]: from='client.? 172.21.15.189:0/2005216188' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5"}]: dispatch 2024-04-30T08:28:59.032 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:58 smithi189 ceph-mon[33038]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5"}]: dispatch 2024-04-30T08:28:59.032 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:58 smithi189 ceph-mon[33038]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5"}]': finished 2024-04-30T08:28:59.032 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:58 smithi189 ceph-mon[33038]: osdmap e51: 9 total, 8 up, 9 in 2024-04-30T08:28:59.032 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:58 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-30T08:29:00.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:28:59 smithi071 ceph-mon[33385]: from='client.? 172.21.15.189:0/675895264' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:29:00.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:28:59 smithi189 ceph-mon[33038]: from='client.? 172.21.15.189:0/675895264' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:29:00.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:28:59 smithi032 ceph-mon[28331]: from='client.? 172.21.15.189:0/675895264' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:29:01.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:00 smithi071 ceph-mon[33385]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 185 KiB/s, 0 objects/s recovering 2024-04-30T08:29:01.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:00 smithi071 ceph-mon[33385]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-04-30T08:29:01.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:00 smithi071 ceph-mon[33385]: Cluster is now healthy 2024-04-30T08:29:01.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:00 smithi189 ceph-mon[33038]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 185 KiB/s, 0 objects/s recovering 2024-04-30T08:29:01.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:00 smithi189 ceph-mon[33038]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-04-30T08:29:01.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:00 smithi189 ceph-mon[33038]: Cluster is now healthy 2024-04-30T08:29:01.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:00 smithi032 ceph-mon[28331]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 185 KiB/s, 0 objects/s recovering 2024-04-30T08:29:01.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:00 smithi032 ceph-mon[28331]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-04-30T08:29:01.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:00 smithi032 ceph-mon[28331]: Cluster is now healthy 2024-04-30T08:29:03.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:02 smithi071 ceph-mon[33385]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-30T08:29:03.008 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:02 smithi189 ceph-mon[33038]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-30T08:29:03.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:02 smithi032 ceph-mon[28331]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-30T08:29:04.983 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:04 smithi189 ceph-mon[33038]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 125 KiB/s, 0 objects/s recovering 2024-04-30T08:29:05.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:04 smithi071 ceph-mon[33385]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 125 KiB/s, 0 objects/s recovering 2024-04-30T08:29:05.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:04 smithi032 ceph-mon[28331]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 125 KiB/s, 0 objects/s recovering 2024-04-30T08:29:06.951 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:06 smithi189 ceph-mon[33038]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-04-30T08:29:06.951 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:06 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-30T08:29:06.951 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:06 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:29:06.952 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:06 smithi189 ceph-mon[33038]: Deploying daemon osd.8 on smithi189 2024-04-30T08:29:07.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:06 smithi071 ceph-mon[33385]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-04-30T08:29:07.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:06 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-30T08:29:07.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:06 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:29:07.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:06 smithi071 ceph-mon[33385]: Deploying daemon osd.8 on smithi189 2024-04-30T08:29:07.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:06 smithi032 ceph-mon[28331]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-04-30T08:29:07.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:06 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-30T08:29:07.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:06 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:29:07.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:06 smithi032 ceph-mon[28331]: Deploying daemon osd.8 on smithi189 2024-04-30T08:29:08.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:08 smithi189 ceph-mon[33038]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-04-30T08:29:09.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:08 smithi071 ceph-mon[33385]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-04-30T08:29:09.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:08 smithi032 ceph-mon[28331]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail; 115 KiB/s, 0 objects/s recovering 2024-04-30T08:29:10.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:10 smithi189 ceph-mon[33038]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-30T08:29:11.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:10 smithi071 ceph-mon[33385]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-30T08:29:11.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:10 smithi032 ceph-mon[28331]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-30T08:29:13.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:12 smithi071 ceph-mon[33385]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-30T08:29:13.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:12 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:13.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:12 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:13.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:12 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:29:13.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:12 smithi189 ceph-mon[33038]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-30T08:29:13.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:12 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:13.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:12 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:13.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:12 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:29:13.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:12 smithi032 ceph-mon[28331]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-30T08:29:13.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:12 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:13.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:12 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:13.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:12 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:29:15.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:14 smithi071 ceph-mon[33385]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-30T08:29:15.006 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:14 smithi189 ceph-mon[33038]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-30T08:29:15.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:14 smithi032 ceph-mon[28331]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-30T08:29:15.840 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:29:15.839+0000 7f68ccff9700 1 -- 172.21.15.189:0/804684528 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f68d0063b50 con 0x7f68bc0afb80 2024-04-30T08:29:15.841 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 8 on host 'smithi189' 2024-04-30T08:29:15.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:29:15.842+0000 7f68d55ab700 1 -- 172.21.15.189:0/804684528 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f68bc0afb80 msgr2=0x7f68bc0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:29:15.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:29:15.842+0000 7f68d55ab700 1 --2- 172.21.15.189:0/804684528 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f68bc0afb80 0x7f68bc0b2090 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f68d019a810 tx=0x7f68b8009190 comp rx=0 tx=0).stop 2024-04-30T08:29:15.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:29:15.842+0000 7f68d55ab700 1 -- 172.21.15.189:0/804684528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68d0103190 msgr2=0x7f68d0199760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:29:15.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:29:15.842+0000 7f68d55ab700 1 --2- 172.21.15.189:0/804684528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68d0103190 0x7f68d0199760 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f68c000ada0 tx=0x7f68c000c330 comp rx=0 tx=0).stop 2024-04-30T08:29:15.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:29:15.842+0000 7f68d55ab700 1 -- 172.21.15.189:0/804684528 shutdown_connections 2024-04-30T08:29:15.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:29:15.842+0000 7f68d55ab700 1 --2- 172.21.15.189:0/804684528 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f68bc0afb80 0x7f68bc0b2090 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:29:15.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:29:15.842+0000 7f68d55ab700 1 --2- 172.21.15.189:0/804684528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f68d0104c60 0x7f68d01a0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:29:15.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:29:15.842+0000 7f68d55ab700 1 --2- 172.21.15.189:0/804684528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68d0103190 0x7f68d0199760 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:29:15.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:29:15.842+0000 7f68d55ab700 1 --2- 172.21.15.189:0/804684528 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f68d01042c0 0x7f68d0199ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:29:15.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:29:15.842+0000 7f68d55ab700 1 -- 172.21.15.189:0/804684528 >> 172.21.15.189:0/804684528 conn(0x7f68d00fe7d0 msgr2=0x7f68d01074f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:29:15.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:29:15.842+0000 7f68d55ab700 1 -- 172.21.15.189:0/804684528 shutdown_connections 2024-04-30T08:29:15.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:29:15.842+0000 7f68d55ab700 1 -- 172.21.15.189:0/804684528 wait complete. 2024-04-30T08:29:16.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:15 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:16.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:15 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:16.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:15 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:29:16.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:15 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:29:16.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:15 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:16.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:15 smithi071 ceph-mon[33385]: from='osd.8 [v2:172.21.15.189:6816/3973697069,v1:172.21.15.189:6817/3973697069]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-30T08:29:16.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:15 smithi071 ceph-mon[33385]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-30T08:29:16.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:15 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:16.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:15 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:16.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:15 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:29:16.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:15 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:29:16.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:15 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:16.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:15 smithi189 ceph-mon[33038]: from='osd.8 [v2:172.21.15.189:6816/3973697069,v1:172.21.15.189:6817/3973697069]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-30T08:29:16.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:15 smithi189 ceph-mon[33038]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-30T08:29:16.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:15 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:16.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:15 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:16.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:15 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:29:16.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:15 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:29:16.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:15 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:16.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:15 smithi032 ceph-mon[28331]: from='osd.8 [v2:172.21.15.189:6816/3973697069,v1:172.21.15.189:6817/3973697069]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-30T08:29:16.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:15 smithi032 ceph-mon[28331]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-30T08:29:16.567 DEBUG:teuthology.orchestra.run.smithi189:osd.8> sudo journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.8.service 2024-04-30T08:29:16.570 INFO:tasks.cephadm:Deploying osd.9 on smithi032 with /dev/vg_nvme/lv_1... 2024-04-30T08:29:16.570 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-30T08:29:16.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:16 smithi032 ceph-mon[28331]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-30T08:29:16.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:16 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:16.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:16 smithi032 ceph-mon[28331]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-30T08:29:16.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:16 smithi032 ceph-mon[28331]: from='osd.8 [v2:172.21.15.189:6816/3973697069,v1:172.21.15.189:6817/3973697069]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-30T08:29:16.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:16 smithi032 ceph-mon[28331]: osdmap e52: 9 total, 8 up, 9 in 2024-04-30T08:29:16.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:16 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-30T08:29:16.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:16 smithi032 ceph-mon[28331]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-30T08:29:16.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:16 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:16.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:16 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:29:17.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:16 smithi189 ceph-mon[33038]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-30T08:29:17.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:16 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:17.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:16 smithi189 ceph-mon[33038]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-30T08:29:17.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:16 smithi189 ceph-mon[33038]: from='osd.8 [v2:172.21.15.189:6816/3973697069,v1:172.21.15.189:6817/3973697069]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-30T08:29:17.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:16 smithi189 ceph-mon[33038]: osdmap e52: 9 total, 8 up, 9 in 2024-04-30T08:29:17.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:16 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-30T08:29:17.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:16 smithi189 ceph-mon[33038]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-30T08:29:17.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:16 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:17.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:16 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:29:17.054 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:29:17.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:16 smithi071 ceph-mon[33385]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-30T08:29:17.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:16 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:17.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:16 smithi071 ceph-mon[33385]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-30T08:29:17.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:16 smithi071 ceph-mon[33385]: from='osd.8 [v2:172.21.15.189:6816/3973697069,v1:172.21.15.189:6817/3973697069]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-30T08:29:17.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:16 smithi071 ceph-mon[33385]: osdmap e52: 9 total, 8 up, 9 in 2024-04-30T08:29:17.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:16 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-30T08:29:17.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:16 smithi071 ceph-mon[33385]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-30T08:29:17.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:16 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:17.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:16 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:29:17.523 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 30 08:29:17 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-8[51785]: 2024-04-30T08:29:17.247+0000 7fd623b72700 -1 osd.8 0 waiting for initial osdmap 2024-04-30T08:29:17.523 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 30 08:29:17 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-8[51785]: 2024-04-30T08:29:17.260+0000 7fd61b91f700 -1 osd.8 53 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-30T08:29:18.093 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:17 smithi189 ceph-mon[33038]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-30T08:29:18.093 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:17 smithi189 ceph-mon[33038]: osdmap e53: 9 total, 8 up, 9 in 2024-04-30T08:29:18.093 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:17 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-30T08:29:18.093 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:17 smithi189 ceph-mon[33038]: from='osd.8 ' entity='osd.8' 2024-04-30T08:29:18.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:17 smithi032 ceph-mon[28331]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-30T08:29:18.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:17 smithi032 ceph-mon[28331]: osdmap e53: 9 total, 8 up, 9 in 2024-04-30T08:29:18.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:17 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-30T08:29:18.139 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:17 smithi032 ceph-mon[28331]: from='osd.8 ' entity='osd.8' 2024-04-30T08:29:18.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:17 smithi071 ceph-mon[33385]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-30T08:29:18.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:17 smithi071 ceph-mon[33385]: osdmap e53: 9 total, 8 up, 9 in 2024-04-30T08:29:18.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:17 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-30T08:29:18.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:17 smithi071 ceph-mon[33385]: from='osd.8 ' entity='osd.8' 2024-04-30T08:29:18.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:18 smithi032 ceph-mon[28331]: purged_snaps scrub starts 2024-04-30T08:29:18.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:18 smithi032 ceph-mon[28331]: purged_snaps scrub ok 2024-04-30T08:29:18.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:18 smithi032 ceph-mon[28331]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-30T08:29:18.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:18 smithi032 ceph-mon[28331]: osd.8 [v2:172.21.15.189:6816/3973697069,v1:172.21.15.189:6817/3973697069] boot 2024-04-30T08:29:18.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:18 smithi032 ceph-mon[28331]: osdmap e54: 9 total, 9 up, 9 in 2024-04-30T08:29:18.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:18 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-30T08:29:18.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:18 smithi032 ceph-mon[28331]: osdmap e55: 9 total, 9 up, 9 in 2024-04-30T08:29:19.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:18 smithi071 ceph-mon[33385]: purged_snaps scrub starts 2024-04-30T08:29:19.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:18 smithi071 ceph-mon[33385]: purged_snaps scrub ok 2024-04-30T08:29:19.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:18 smithi071 ceph-mon[33385]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-30T08:29:19.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:18 smithi071 ceph-mon[33385]: osd.8 [v2:172.21.15.189:6816/3973697069,v1:172.21.15.189:6817/3973697069] boot 2024-04-30T08:29:19.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:18 smithi071 ceph-mon[33385]: osdmap e54: 9 total, 9 up, 9 in 2024-04-30T08:29:19.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:18 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-30T08:29:19.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:18 smithi071 ceph-mon[33385]: osdmap e55: 9 total, 9 up, 9 in 2024-04-30T08:29:19.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:18 smithi189 ceph-mon[33038]: purged_snaps scrub starts 2024-04-30T08:29:19.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:18 smithi189 ceph-mon[33038]: purged_snaps scrub ok 2024-04-30T08:29:19.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:18 smithi189 ceph-mon[33038]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 235 MiB used, 715 GiB / 715 GiB avail 2024-04-30T08:29:19.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:18 smithi189 ceph-mon[33038]: osd.8 [v2:172.21.15.189:6816/3973697069,v1:172.21.15.189:6817/3973697069] boot 2024-04-30T08:29:19.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:18 smithi189 ceph-mon[33038]: osdmap e54: 9 total, 9 up, 9 in 2024-04-30T08:29:19.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:18 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-30T08:29:19.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:18 smithi189 ceph-mon[33038]: osdmap e55: 9 total, 9 up, 9 in 2024-04-30T08:29:20.208 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-30T08:29:20.209 INFO:teuthology.orchestra.run.smithi032.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-30T08:29:20.209 INFO:teuthology.orchestra.run.smithi032.stderr: stderr: 10+0 records in 2024-04-30T08:29:20.209 INFO:teuthology.orchestra.run.smithi032.stderr:10+0 records out 2024-04-30T08:29:20.209 INFO:teuthology.orchestra.run.smithi032.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0319569 s, 328 MB/s 2024-04-30T08:29:20.209 INFO:teuthology.orchestra.run.smithi032.stderr:--> Zapping successful for: 2024-04-30T08:29:20.856 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:20 smithi032 ceph-mon[28331]: pgmap v220: 1 pgs: 1 remapped+peering; 577 KiB data, 266 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:20.856 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:20 smithi032 ceph-mon[28331]: Detected new or changed devices on smithi189 2024-04-30T08:29:20.856 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:20 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:20.856 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:20 smithi032 ceph-mon[28331]: osdmap e56: 9 total, 9 up, 9 in 2024-04-30T08:29:20.856 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:20 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:20.856 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:20 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:29:20.856 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:20 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:29:20.856 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:20 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:29:20.856 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:20 smithi032 ceph-mon[28331]: Adjusting osd_memory_target on smithi189 to 5696M 2024-04-30T08:29:20.856 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:20 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:20.857 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:20 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:29:20.857 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:20 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:29:20.857 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:20 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:21.138 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi032:vg_nvme/lv_1 2024-04-30T08:29:21.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:20 smithi071 ceph-mon[33385]: pgmap v220: 1 pgs: 1 remapped+peering; 577 KiB data, 266 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:21.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:20 smithi071 ceph-mon[33385]: Detected new or changed devices on smithi189 2024-04-30T08:29:21.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:20 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:21.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:20 smithi071 ceph-mon[33385]: osdmap e56: 9 total, 9 up, 9 in 2024-04-30T08:29:21.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:20 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:21.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:20 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:29:21.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:20 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:29:21.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:20 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:29:21.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:20 smithi071 ceph-mon[33385]: Adjusting osd_memory_target on smithi189 to 5696M 2024-04-30T08:29:21.255 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:20 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:21.255 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:20 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:29:21.255 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:20 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:29:21.255 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:20 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:21.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:20 smithi189 ceph-mon[33038]: pgmap v220: 1 pgs: 1 remapped+peering; 577 KiB data, 266 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:21.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:20 smithi189 ceph-mon[33038]: Detected new or changed devices on smithi189 2024-04-30T08:29:21.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:20 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:21.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:20 smithi189 ceph-mon[33038]: osdmap e56: 9 total, 9 up, 9 in 2024-04-30T08:29:21.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:20 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:21.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:20 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:29:21.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:20 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:29:21.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:20 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:29:21.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:20 smithi189 ceph-mon[33038]: Adjusting osd_memory_target on smithi189 to 5696M 2024-04-30T08:29:21.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:20 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:21.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:20 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:29:21.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:20 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:29:21.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:20 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:21.609 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:29:23.075 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:22 smithi071 ceph-mon[33385]: pgmap v222: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:23.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:22 smithi032 ceph-mon[28331]: pgmap v222: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:23.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:22 smithi189 ceph-mon[33038]: pgmap v222: 1 pgs: 1 remapped+peering; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:24.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.170+0000 7f3ee5e00700 1 -- 172.21.15.32:0/3729645235 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ee01075d0 msgr2=0x7f3ee01079e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:29:24.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.170+0000 7f3ee5e00700 1 --2- 172.21.15.32:0/3729645235 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ee01075d0 0x7f3ee01079e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc009990 tx=0x7f3ecc009ca0 comp rx=0 tx=0).stop 2024-04-30T08:29:24.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.170+0000 7f3ee5e00700 1 -- 172.21.15.32:0/3729645235 shutdown_connections 2024-04-30T08:29:24.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.170+0000 7f3ee5e00700 1 --2- 172.21.15.32:0/3729645235 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ee0107f70 0x7f3ee010e820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:29:24.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.170+0000 7f3ee5e00700 1 --2- 172.21.15.32:0/3729645235 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ee01064a0 0x7f3ee01068b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:29:24.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.170+0000 7f3ee5e00700 1 --2- 172.21.15.32:0/3729645235 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ee01075d0 0x7f3ee01079e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:29:24.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.170+0000 7f3ee5e00700 1 -- 172.21.15.32:0/3729645235 >> 172.21.15.32:0/3729645235 conn(0x7f3ee0101ae0 msgr2=0x7f3ee0103f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:29:24.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.170+0000 7f3ee5e00700 1 -- 172.21.15.32:0/3729645235 shutdown_connections 2024-04-30T08:29:24.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.170+0000 7f3ee5e00700 1 -- 172.21.15.32:0/3729645235 wait complete. 2024-04-30T08:29:24.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.171+0000 7f3ee5e00700 1 Processor -- start 2024-04-30T08:29:24.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.172+0000 7f3ee5e00700 1 -- start start 2024-04-30T08:29:24.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.172+0000 7f3ee5e00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ee01064a0 0x7f3ee019ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:29:24.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.172+0000 7f3ee5e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ee01075d0 0x7f3ee019cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:29:24.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.172+0000 7f3ee5e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ee0107f70 0x7f3ee01a3f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:29:24.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.172+0000 7f3ee5e00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3ee0111f60 con 0x7f3ee0107f70 2024-04-30T08:29:24.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.172+0000 7f3ee5e00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f3ee0111de0 con 0x7f3ee01064a0 2024-04-30T08:29:24.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.172+0000 7f3ee5e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3ee01120e0 con 0x7f3ee01075d0 2024-04-30T08:29:24.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.173+0000 7f3edea00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ee01064a0 0x7f3ee019ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:29:24.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.173+0000 7f3edea00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ee01064a0 0x7f3ee019ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:37890/0 (socket says 172.21.15.32:37890) 2024-04-30T08:29:24.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.173+0000 7f3edea00700 1 -- 172.21.15.32:0/951172335 learned_addr learned my addr 172.21.15.32:0/951172335 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:29:24.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.173+0000 7f3ede000700 1 --2- 172.21.15.32:0/951172335 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ee01075d0 0x7f3ee019cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:29:24.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.173+0000 7f3edf400700 1 --2- 172.21.15.32:0/951172335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ee0107f70 0x7f3ee01a3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:29:24.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.173+0000 7f3edea00700 1 -- 172.21.15.32:0/951172335 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ee01075d0 msgr2=0x7f3ee019cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:29:24.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.173+0000 7f3edea00700 1 --2- 172.21.15.32:0/951172335 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ee01075d0 0x7f3ee019cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:29:24.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.173+0000 7f3edea00700 1 -- 172.21.15.32:0/951172335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ee0107f70 msgr2=0x7f3ee01a3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:29:24.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.173+0000 7f3edea00700 1 --2- 172.21.15.32:0/951172335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ee0107f70 0x7f3ee01a3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:29:24.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.173+0000 7f3edea00700 1 -- 172.21.15.32:0/951172335 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ee01a45d0 con 0x7f3ee01064a0 2024-04-30T08:29:24.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.173+0000 7f3edf400700 1 --2- 172.21.15.32:0/951172335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ee0107f70 0x7f3ee01a3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:29:24.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.173+0000 7f3edea00700 1 --2- 172.21.15.32:0/951172335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ee01064a0 0x7f3ee019ca70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3ed000e940 tx=0x7f3ed000ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:29:24.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.174+0000 7f3ee4a00700 1 -- 172.21.15.32:0/951172335 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed000cbd0 con 0x7f3ee01064a0 2024-04-30T08:29:24.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.174+0000 7f3ee4a00700 1 -- 172.21.15.32:0/951172335 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ed000ee90 con 0x7f3ee01064a0 2024-04-30T08:29:24.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.174+0000 7f3ee4a00700 1 -- 172.21.15.32:0/951172335 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed0003dc0 con 0x7f3ee01064a0 2024-04-30T08:29:24.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.174+0000 7f3ee5e00700 1 -- 172.21.15.32:0/951172335 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ee01a48c0 con 0x7f3ee01064a0 2024-04-30T08:29:24.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.174+0000 7f3ee5e00700 1 -- 172.21.15.32:0/951172335 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ee01a4db0 con 0x7f3ee01064a0 2024-04-30T08:29:24.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.175+0000 7f3ee5e00700 1 -- 172.21.15.32:0/951172335 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ee0051210 con 0x7f3ee01064a0 2024-04-30T08:29:24.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.177+0000 7f3ee4a00700 1 -- 172.21.15.32:0/951172335 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f3ed0003f60 con 0x7f3ee01064a0 2024-04-30T08:29:24.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.180+0000 7f3ee4a00700 1 --2- 172.21.15.32:0/951172335 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3ec40afc00 0x7f3ec40b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:29:24.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.180+0000 7f3ee4a00700 1 -- 172.21.15.32:0/951172335 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(56..56 src has 1..56) ==== 5829+0+0 (secure 0 0 0) 0x7f3ed00de510 con 0x7f3ee01064a0 2024-04-30T08:29:24.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.180+0000 7f3ede000700 1 --2- 172.21.15.32:0/951172335 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3ec40afc00 0x7f3ec40b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:29:24.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.181+0000 7f3ede000700 1 --2- 172.21.15.32:0/951172335 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3ec40afc00 0x7f3ec40b2110 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc0098e0 tx=0x7f3ecc00b7c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:29:24.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.183+0000 7f3ee4a00700 1 -- 172.21.15.32:0/951172335 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3ed00a7a00 con 0x7f3ee01064a0 2024-04-30T08:29:24.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:24.425+0000 7f3ee5e00700 1 -- 172.21.15.32:0/951172335 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f3ee0063b50 con 0x7f3ec40afc00 2024-04-30T08:29:25.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:24 smithi032 ceph-mon[28331]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-30T08:29:25.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:24 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:29:25.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:24 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:29:25.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:24 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:29:25.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:24 smithi071 ceph-mon[33385]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-30T08:29:25.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:24 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:29:25.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:24 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:29:25.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:24 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:29:25.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:24 smithi189 ceph-mon[33038]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-30T08:29:25.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:24 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:29:25.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:24 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:29:25.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:24 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:29:26.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:25 smithi032 ceph-mon[28331]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:29:26.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:25 smithi071 ceph-mon[33385]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:29:26.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:25 smithi189 ceph-mon[33038]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi032:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:29:27.082 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:26 smithi032 ceph-mon[28331]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-30T08:29:27.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:26 smithi071 ceph-mon[33385]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-30T08:29:27.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:26 smithi189 ceph-mon[33038]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail; 74 KiB/s, 0 objects/s recovering 2024-04-30T08:29:29.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:28 smithi032 ceph-mon[28331]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-30T08:29:29.255 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:28 smithi071 ceph-mon[33385]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-30T08:29:29.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:28 smithi189 ceph-mon[33038]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail; 65 KiB/s, 0 objects/s recovering 2024-04-30T08:29:30.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:29 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2736601235' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef4a1645-ae6e-45c3-82ad-161510e62aa9"}]: dispatch 2024-04-30T08:29:30.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:29 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2736601235' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef4a1645-ae6e-45c3-82ad-161510e62aa9"}]': finished 2024-04-30T08:29:30.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:29 smithi032 ceph-mon[28331]: osdmap e57: 10 total, 9 up, 10 in 2024-04-30T08:29:30.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:29 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-30T08:29:30.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:29 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2736601235' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef4a1645-ae6e-45c3-82ad-161510e62aa9"}]: dispatch 2024-04-30T08:29:30.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:29 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2736601235' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef4a1645-ae6e-45c3-82ad-161510e62aa9"}]': finished 2024-04-30T08:29:30.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:29 smithi071 ceph-mon[33385]: osdmap e57: 10 total, 9 up, 10 in 2024-04-30T08:29:30.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:29 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-30T08:29:30.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:29 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2736601235' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef4a1645-ae6e-45c3-82ad-161510e62aa9"}]: dispatch 2024-04-30T08:29:30.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:29 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2736601235' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef4a1645-ae6e-45c3-82ad-161510e62aa9"}]': finished 2024-04-30T08:29:30.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:29 smithi189 ceph-mon[33038]: osdmap e57: 10 total, 9 up, 10 in 2024-04-30T08:29:30.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:29 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-30T08:29:31.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:30 smithi032 ceph-mon[28331]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-30T08:29:31.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:30 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1178376' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:29:31.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:30 smithi071 ceph-mon[33385]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-30T08:29:31.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:30 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1178376' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:29:31.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:30 smithi189 ceph-mon[33038]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-04-30T08:29:31.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:30 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1178376' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:29:33.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:32 smithi032 ceph-mon[28331]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-30T08:29:33.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:32 smithi071 ceph-mon[33385]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-30T08:29:33.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:32 smithi189 ceph-mon[33038]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-30T08:29:35.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:34 smithi032 ceph-mon[28331]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:35.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:34 smithi071 ceph-mon[33385]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:35.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:34 smithi189 ceph-mon[33038]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:37.041 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:36 smithi032 ceph-mon[28331]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:37.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:36 smithi071 ceph-mon[33385]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:37.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:36 smithi189 ceph-mon[33038]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:39.120 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:38 smithi189 ceph-mon[33038]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:39.120 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:38 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-30T08:29:39.120 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:38 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:29:39.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:38 smithi032 ceph-mon[28331]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:39.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:38 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-30T08:29:39.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:38 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:29:39.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:38 smithi071 ceph-mon[33385]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:39.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:38 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-30T08:29:39.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:38 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:29:40.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:39 smithi032 ceph-mon[28331]: Deploying daemon osd.9 on smithi032 2024-04-30T08:29:40.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:39 smithi071 ceph-mon[33385]: Deploying daemon osd.9 on smithi032 2024-04-30T08:29:40.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:39 smithi189 ceph-mon[33038]: Deploying daemon osd.9 on smithi032 2024-04-30T08:29:41.099 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:40 smithi032 ceph-mon[28331]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:41.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:40 smithi071 ceph-mon[33385]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:41.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:40 smithi189 ceph-mon[33038]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:43.067 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:42 smithi032 ceph-mon[28331]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:43.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:42 smithi071 ceph-mon[33385]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:43.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:42 smithi189 ceph-mon[33038]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:45.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:44 smithi032 ceph-mon[28331]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:45.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:44 smithi071 ceph-mon[33385]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:45.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:44 smithi189 ceph-mon[33038]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:47.121 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:46 smithi032 ceph-mon[28331]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:47.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:46 smithi071 ceph-mon[33385]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:47.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:46 smithi189 ceph-mon[33038]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:49.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:48 smithi032 ceph-mon[28331]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:49.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:48 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:49.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:48 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:29:49.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:48 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:49.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:48 smithi071 ceph-mon[33385]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:49.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:48 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:49.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:48 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:29:49.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:48 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:49.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:48 smithi189 ceph-mon[33038]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:49.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:48 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:49.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:48 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:29:49.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:48 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:51.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:50 smithi032 ceph-mon[28331]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:51.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:50 smithi071 ceph-mon[33385]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:51.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:50 smithi189 ceph-mon[33038]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:53.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:52 smithi032 ceph-mon[28331]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:53.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:53.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:53.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:52 smithi032 ceph-mon[28331]: from='osd.9 [v2:172.21.15.32:6826/46066685,v1:172.21.15.32:6827/46066685]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-30T08:29:53.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:52 smithi071 ceph-mon[33385]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:53.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:52 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:53.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:52 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:53.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:52 smithi071 ceph-mon[33385]: from='osd.9 [v2:172.21.15.32:6826/46066685,v1:172.21.15.32:6827/46066685]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-30T08:29:53.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:52 smithi189 ceph-mon[33038]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:53.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:52 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:53.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:52 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:53.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:52 smithi189 ceph-mon[33038]: from='osd.9 [v2:172.21.15.32:6826/46066685,v1:172.21.15.32:6827/46066685]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-30T08:29:54.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:54.384+0000 7f3ee4a00700 1 -- 172.21.15.32:0/951172335 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3ee0063b50 con 0x7f3ec40afc00 2024-04-30T08:29:54.387 INFO:teuthology.orchestra.run.smithi032.stdout:Created osd(s) 9 on host 'smithi032' 2024-04-30T08:29:54.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:54.387+0000 7f3ee5e00700 1 -- 172.21.15.32:0/951172335 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3ec40afc00 msgr2=0x7f3ec40b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:29:54.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:54.387+0000 7f3ee5e00700 1 --2- 172.21.15.32:0/951172335 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3ec40afc00 0x7f3ec40b2110 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc0098e0 tx=0x7f3ecc00b7c0 comp rx=0 tx=0).stop 2024-04-30T08:29:54.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:54.388+0000 7f3ee5e00700 1 -- 172.21.15.32:0/951172335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ee01064a0 msgr2=0x7f3ee019ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:29:54.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:54.388+0000 7f3ee5e00700 1 --2- 172.21.15.32:0/951172335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ee01064a0 0x7f3ee019ca70 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f3ed000e940 tx=0x7f3ed000ec50 comp rx=0 tx=0).stop 2024-04-30T08:29:54.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:54.388+0000 7f3ee5e00700 1 -- 172.21.15.32:0/951172335 shutdown_connections 2024-04-30T08:29:54.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:54.388+0000 7f3ee5e00700 1 --2- 172.21.15.32:0/951172335 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3ec40afc00 0x7f3ec40b2110 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:29:54.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:54.388+0000 7f3ee5e00700 1 --2- 172.21.15.32:0/951172335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ee0107f70 0x7f3ee01a3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:29:54.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:54.388+0000 7f3ee5e00700 1 --2- 172.21.15.32:0/951172335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ee01064a0 0x7f3ee019ca70 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:29:54.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:54.388+0000 7f3ee5e00700 1 --2- 172.21.15.32:0/951172335 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ee01075d0 0x7f3ee019cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:29:54.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:54.388+0000 7f3ee5e00700 1 -- 172.21.15.32:0/951172335 >> 172.21.15.32:0/951172335 conn(0x7f3ee0101ae0 msgr2=0x7f3ee010a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:29:54.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:54.388+0000 7f3ee5e00700 1 -- 172.21.15.32:0/951172335 shutdown_connections 2024-04-30T08:29:54.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:29:54.388+0000 7f3ee5e00700 1 -- 172.21.15.32:0/951172335 wait complete. 2024-04-30T08:29:54.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:54 smithi032 ceph-mon[28331]: from='osd.9 [v2:172.21.15.32:6826/46066685,v1:172.21.15.32:6827/46066685]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-30T08:29:54.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:54 smithi032 ceph-mon[28331]: osdmap e58: 10 total, 9 up, 10 in 2024-04-30T08:29:54.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:54 smithi032 ceph-mon[28331]: from='osd.9 [v2:172.21.15.32:6826/46066685,v1:172.21.15.32:6827/46066685]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-30T08:29:54.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-30T08:29:54.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:29:54.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:29:54.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:54.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:54 smithi032 ceph-mon[28331]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:54.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:54.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:29:54.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:54.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:54 smithi071 ceph-mon[33385]: from='osd.9 [v2:172.21.15.32:6826/46066685,v1:172.21.15.32:6827/46066685]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-30T08:29:54.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:54 smithi071 ceph-mon[33385]: osdmap e58: 10 total, 9 up, 10 in 2024-04-30T08:29:54.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:54 smithi071 ceph-mon[33385]: from='osd.9 [v2:172.21.15.32:6826/46066685,v1:172.21.15.32:6827/46066685]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-30T08:29:54.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-30T08:29:54.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:29:54.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:29:54.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:54.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:54 smithi071 ceph-mon[33385]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:54.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:54.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:29:54.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:54.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:54 smithi189 ceph-mon[33038]: from='osd.9 [v2:172.21.15.32:6826/46066685,v1:172.21.15.32:6827/46066685]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-30T08:29:54.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:54 smithi189 ceph-mon[33038]: osdmap e58: 10 total, 9 up, 10 in 2024-04-30T08:29:54.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:54 smithi189 ceph-mon[33038]: from='osd.9 [v2:172.21.15.32:6826/46066685,v1:172.21.15.32:6827/46066685]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]: dispatch 2024-04-30T08:29:54.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-30T08:29:54.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:29:54.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:29:54.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:54.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:54 smithi189 ceph-mon[33038]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:54.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:54.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:29:54.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:29:55.348 DEBUG:teuthology.orchestra.run.smithi032:osd.9> sudo journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.9.service 2024-04-30T08:29:55.360 INFO:tasks.cephadm:Deploying osd.10 on smithi071 with /dev/vg_nvme/lv_1... 2024-04-30T08:29:55.360 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-30T08:29:55.563 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:55 smithi032 ceph-mon[28331]: from='osd.9 [v2:172.21.15.32:6826/46066685,v1:172.21.15.32:6827/46066685]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-30T08:29:55.564 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:55 smithi032 ceph-mon[28331]: osdmap e59: 10 total, 9 up, 10 in 2024-04-30T08:29:55.564 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:55 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-30T08:29:55.564 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:55 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-30T08:29:55.610 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.b/config 2024-04-30T08:29:55.733 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:55 smithi071 ceph-mon[33385]: from='osd.9 [v2:172.21.15.32:6826/46066685,v1:172.21.15.32:6827/46066685]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-30T08:29:55.733 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:55 smithi071 ceph-mon[33385]: osdmap e59: 10 total, 9 up, 10 in 2024-04-30T08:29:55.733 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:55 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-30T08:29:55.733 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:55 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-30T08:29:55.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:55 smithi189 ceph-mon[33038]: from='osd.9 [v2:172.21.15.32:6826/46066685,v1:172.21.15.32:6827/46066685]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi032", "root=default"]}]': finished 2024-04-30T08:29:55.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:55 smithi189 ceph-mon[33038]: osdmap e59: 10 total, 9 up, 10 in 2024-04-30T08:29:55.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:55 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-30T08:29:55.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:55 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-30T08:29:56.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:56 smithi071 ceph-mon[33385]: purged_snaps scrub starts 2024-04-30T08:29:56.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:56 smithi071 ceph-mon[33385]: purged_snaps scrub ok 2024-04-30T08:29:56.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:56 smithi071 ceph-mon[33385]: osd.9 [v2:172.21.15.32:6826/46066685,v1:172.21.15.32:6827/46066685] boot 2024-04-30T08:29:56.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:56 smithi071 ceph-mon[33385]: osdmap e60: 10 total, 10 up, 10 in 2024-04-30T08:29:56.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:56 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-30T08:29:56.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:56 smithi071 ceph-mon[33385]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:56.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:56 smithi189 ceph-mon[33038]: purged_snaps scrub starts 2024-04-30T08:29:56.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:56 smithi189 ceph-mon[33038]: purged_snaps scrub ok 2024-04-30T08:29:56.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:56 smithi189 ceph-mon[33038]: osd.9 [v2:172.21.15.32:6826/46066685,v1:172.21.15.32:6827/46066685] boot 2024-04-30T08:29:56.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:56 smithi189 ceph-mon[33038]: osdmap e60: 10 total, 10 up, 10 in 2024-04-30T08:29:56.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:56 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-30T08:29:56.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:56 smithi189 ceph-mon[33038]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:56.847 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:56 smithi032 ceph-mon[28331]: purged_snaps scrub starts 2024-04-30T08:29:56.848 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:56 smithi032 ceph-mon[28331]: purged_snaps scrub ok 2024-04-30T08:29:56.848 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:56 smithi032 ceph-mon[28331]: osd.9 [v2:172.21.15.32:6826/46066685,v1:172.21.15.32:6827/46066685] boot 2024-04-30T08:29:56.848 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:56 smithi032 ceph-mon[28331]: osdmap e60: 10 total, 10 up, 10 in 2024-04-30T08:29:56.848 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:56 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-30T08:29:56.848 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:56 smithi032 ceph-mon[28331]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 267 MiB used, 804 GiB / 805 GiB avail 2024-04-30T08:29:57.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:57 smithi071 ceph-mon[33385]: osdmap e61: 10 total, 10 up, 10 in 2024-04-30T08:29:57.790 INFO:teuthology.orchestra.run.smithi071.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-30T08:29:57.791 INFO:teuthology.orchestra.run.smithi071.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-30T08:29:57.791 INFO:teuthology.orchestra.run.smithi071.stderr: stderr: 10+0 records in 2024-04-30T08:29:57.791 INFO:teuthology.orchestra.run.smithi071.stderr:10+0 records out 2024-04-30T08:29:57.791 INFO:teuthology.orchestra.run.smithi071.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0112449 s, 932 MB/s 2024-04-30T08:29:57.791 INFO:teuthology.orchestra.run.smithi071.stderr:--> Zapping successful for: 2024-04-30T08:29:57.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:57 smithi189 ceph-mon[33038]: osdmap e61: 10 total, 10 up, 10 in 2024-04-30T08:29:57.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:57 smithi032 ceph-mon[28331]: osdmap e61: 10 total, 10 up, 10 in 2024-04-30T08:29:58.538 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi071:vg_nvme/lv_1 2024-04-30T08:29:58.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:29:58 smithi071 ceph-mon[33385]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:29:58.792 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.b/config 2024-04-30T08:29:58.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:29:58 smithi189 ceph-mon[33038]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:29:58.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:29:58 smithi032 ceph-mon[28331]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:00.523 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.520+0000 7faf428c7700 1 -- 172.21.15.71:0/4100851225 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf3c1042d0 msgr2=0x7faf3c1046e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:30:00.523 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.520+0000 7faf428c7700 1 --2- 172.21.15.71:0/4100851225 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf3c1042d0 0x7faf3c1046e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7faf30009990 tx=0x7faf30009ca0 comp rx=0 tx=0).stop 2024-04-30T08:30:00.523 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.520+0000 7faf428c7700 1 -- 172.21.15.71:0/4100851225 shutdown_connections 2024-04-30T08:30:00.524 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.520+0000 7faf428c7700 1 --2- 172.21.15.71:0/4100851225 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf3c104c70 0x7faf3c10b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:00.524 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.520+0000 7faf428c7700 1 --2- 172.21.15.71:0/4100851225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf3c1031a0 0x7faf3c1035b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:00.524 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.520+0000 7faf428c7700 1 --2- 172.21.15.71:0/4100851225 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf3c1042d0 0x7faf3c1046e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:00.524 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.520+0000 7faf428c7700 1 -- 172.21.15.71:0/4100851225 >> 172.21.15.71:0/4100851225 conn(0x7faf3c0fe7e0 msgr2=0x7faf3c100c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:30:00.524 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.520+0000 7faf428c7700 1 -- 172.21.15.71:0/4100851225 shutdown_connections 2024-04-30T08:30:00.524 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.520+0000 7faf428c7700 1 -- 172.21.15.71:0/4100851225 wait complete. 2024-04-30T08:30:00.524 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.521+0000 7faf428c7700 1 Processor -- start 2024-04-30T08:30:00.524 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.521+0000 7faf428c7700 1 -- start start 2024-04-30T08:30:00.524 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.521+0000 7faf428c7700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf3c1031a0 0x7faf3c199820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:30:00.524 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.521+0000 7faf428c7700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf3c1042d0 0x7faf3c199d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:30:00.524 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.521+0000 7faf428c7700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf3c104c70 0x7faf3c1a0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:30:00.524 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.521+0000 7faf428c7700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7faf3c10ece0 con 0x7faf3c1031a0 2024-04-30T08:30:00.524 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.521+0000 7faf428c7700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7faf3c10eb60 con 0x7faf3c104c70 2024-04-30T08:30:00.525 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.521+0000 7faf428c7700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7faf3c10ee60 con 0x7faf3c1042d0 2024-04-30T08:30:00.525 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.521+0000 7faf40e64700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf3c104c70 0x7faf3c1a0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:30:00.525 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.521+0000 7faf40e64700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf3c104c70 0x7faf3c1a0d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.71:52896/0 (socket says 172.21.15.71:52896) 2024-04-30T08:30:00.525 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.521+0000 7faf40e64700 1 -- 172.21.15.71:0/2303729912 learned_addr learned my addr 172.21.15.71:0/2303729912 (peer_addr_for_me v2:172.21.15.71:0/0) 2024-04-30T08:30:00.525 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.521+0000 7faf3b7fe700 1 --2- 172.21.15.71:0/2303729912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf3c1042d0 0x7faf3c199d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:30:00.525 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.522+0000 7faf40e64700 1 -- 172.21.15.71:0/2303729912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf3c1042d0 msgr2=0x7faf3c199d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:30:00.525 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.522+0000 7faf40e64700 1 --2- 172.21.15.71:0/2303729912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf3c1042d0 0x7faf3c199d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:00.525 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.522+0000 7faf40e64700 1 -- 172.21.15.71:0/2303729912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf3c1031a0 msgr2=0x7faf3c199820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:30:00.525 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.522+0000 7faf40e64700 1 --2- 172.21.15.71:0/2303729912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf3c1031a0 0x7faf3c199820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:00.525 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.522+0000 7faf40e64700 1 -- 172.21.15.71:0/2303729912 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf3c0759f0 con 0x7faf3c104c70 2024-04-30T08:30:00.525 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.522+0000 7faf40e64700 1 --2- 172.21.15.71:0/2303729912 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf3c104c70 0x7faf3c1a0d00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7faf3400b970 tx=0x7faf3400bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:30:00.525 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.522+0000 7faf397fa700 1 -- 172.21.15.71:0/2303729912 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf34019070 con 0x7faf3c104c70 2024-04-30T08:30:00.525 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.522+0000 7faf397fa700 1 -- 172.21.15.71:0/2303729912 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faf3400cde0 con 0x7faf3c104c70 2024-04-30T08:30:00.526 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.522+0000 7faf428c7700 1 -- 172.21.15.71:0/2303729912 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf3c075ce0 con 0x7faf3c104c70 2024-04-30T08:30:00.526 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.522+0000 7faf397fa700 1 -- 172.21.15.71:0/2303729912 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf34007370 con 0x7faf3c104c70 2024-04-30T08:30:00.526 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.522+0000 7faf428c7700 1 -- 172.21.15.71:0/2303729912 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf3c0762a0 con 0x7faf3c104c70 2024-04-30T08:30:00.526 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.523+0000 7faf428c7700 1 -- 172.21.15.71:0/2303729912 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf3c051210 con 0x7faf3c104c70 2024-04-30T08:30:00.527 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.524+0000 7faf397fa700 1 -- 172.21.15.71:0/2303729912 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7faf3401c070 con 0x7faf3c104c70 2024-04-30T08:30:00.527 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.525+0000 7faf397fa700 1 --2- 172.21.15.71:0/2303729912 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7faf240afc00 0x7faf240b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:30:00.527 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.525+0000 7faf397fa700 1 -- 172.21.15.71:0/2303729912 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 6261+0+0 (secure 0 0 0) 0x7faf340ddff0 con 0x7faf3c104c70 2024-04-30T08:30:00.527 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.525+0000 7faf3bfff700 1 --2- 172.21.15.71:0/2303729912 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7faf240afc00 0x7faf240b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:30:00.527 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.526+0000 7faf3bfff700 1 --2- 172.21.15.71:0/2303729912 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7faf240afc00 0x7faf240b2110 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7faf3c1041d0 tx=0x7faf2c005d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:30:00.529 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.528+0000 7faf397fa700 1 -- 172.21.15.71:0/2303729912 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faf34023070 con 0x7faf3c104c70 2024-04-30T08:30:00.739 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:00.738+0000 7faf428c7700 1 -- 172.21.15.71:0/2303729912 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7faf3c02e600 con 0x7faf240afc00 2024-04-30T08:30:01.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:00 smithi071 ceph-mon[33385]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:01.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:00 smithi071 ceph-mon[33385]: overall HEALTH_OK 2024-04-30T08:30:01.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:00 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:01.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:00 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:01.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:00 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:01.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:00 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:01.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:00 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:01.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:00 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:01.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:00 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:01.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:00 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:30:01.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:00 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:30:01.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:00 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:01.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:00 smithi189 ceph-mon[33038]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:01.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:00 smithi189 ceph-mon[33038]: overall HEALTH_OK 2024-04-30T08:30:01.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:00 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:01.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:00 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:01.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:00 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:01.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:00 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:01.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:00 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:01.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:00 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:01.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:00 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:01.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:00 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:30:01.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:00 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:30:01.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:00 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:01.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:00 smithi032 ceph-mon[28331]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:01.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:00 smithi032 ceph-mon[28331]: overall HEALTH_OK 2024-04-30T08:30:01.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:00 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:01.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:00 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:01.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:00 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:01.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:00 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:01.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:00 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:01.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:00 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:01.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:00 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:01.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:00 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:30:01.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:00 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:30:01.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:00 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:02.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:01 smithi071 ceph-mon[33385]: Detected new or changed devices on smithi032 2024-04-30T08:30:02.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:01 smithi071 ceph-mon[33385]: Adjusting osd_memory_target on smithi032 to 4273M 2024-04-30T08:30:02.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:01 smithi071 ceph-mon[33385]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:30:02.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:01 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:30:02.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:01 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:30:02.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:01 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:30:02.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:01 smithi189 ceph-mon[33038]: Detected new or changed devices on smithi032 2024-04-30T08:30:02.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:01 smithi189 ceph-mon[33038]: Adjusting osd_memory_target on smithi032 to 4273M 2024-04-30T08:30:02.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:01 smithi189 ceph-mon[33038]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:30:02.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:01 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:30:02.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:01 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:30:02.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:01 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:30:02.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:01 smithi032 ceph-mon[28331]: Detected new or changed devices on smithi032 2024-04-30T08:30:02.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:01 smithi032 ceph-mon[28331]: Adjusting osd_memory_target on smithi032 to 4273M 2024-04-30T08:30:02.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:01 smithi032 ceph-mon[28331]: from='client.24365 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi071:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:30:02.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:01 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:30:02.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:01 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:30:02.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:01 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:30:03.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:02 smithi071 ceph-mon[33385]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:03.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:02 smithi189 ceph-mon[33038]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:03.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:02 smithi032 ceph-mon[28331]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:04.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:03 smithi071 ceph-mon[33385]: from='client.? 172.21.15.71:0/1469592437' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "25b26a38-a644-409f-b941-59efdf04a3ae"}]: dispatch 2024-04-30T08:30:04.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:03 smithi071 ceph-mon[33385]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "25b26a38-a644-409f-b941-59efdf04a3ae"}]: dispatch 2024-04-30T08:30:04.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:03 smithi071 ceph-mon[33385]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "25b26a38-a644-409f-b941-59efdf04a3ae"}]': finished 2024-04-30T08:30:04.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:03 smithi071 ceph-mon[33385]: osdmap e62: 11 total, 10 up, 11 in 2024-04-30T08:30:04.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:03 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-30T08:30:04.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:03 smithi189 ceph-mon[33038]: from='client.? 172.21.15.71:0/1469592437' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "25b26a38-a644-409f-b941-59efdf04a3ae"}]: dispatch 2024-04-30T08:30:04.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:03 smithi189 ceph-mon[33038]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "25b26a38-a644-409f-b941-59efdf04a3ae"}]: dispatch 2024-04-30T08:30:04.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:03 smithi189 ceph-mon[33038]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "25b26a38-a644-409f-b941-59efdf04a3ae"}]': finished 2024-04-30T08:30:04.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:03 smithi189 ceph-mon[33038]: osdmap e62: 11 total, 10 up, 11 in 2024-04-30T08:30:04.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:03 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-30T08:30:04.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:03 smithi032 ceph-mon[28331]: from='client.? 172.21.15.71:0/1469592437' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "25b26a38-a644-409f-b941-59efdf04a3ae"}]: dispatch 2024-04-30T08:30:04.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:03 smithi032 ceph-mon[28331]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "25b26a38-a644-409f-b941-59efdf04a3ae"}]: dispatch 2024-04-30T08:30:04.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:03 smithi032 ceph-mon[28331]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "25b26a38-a644-409f-b941-59efdf04a3ae"}]': finished 2024-04-30T08:30:04.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:03 smithi032 ceph-mon[28331]: osdmap e62: 11 total, 10 up, 11 in 2024-04-30T08:30:04.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:03 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-30T08:30:05.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:04 smithi071 ceph-mon[33385]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:05.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:04 smithi071 ceph-mon[33385]: from='client.? 172.21.15.71:0/806159827' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:30:05.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:04 smithi189 ceph-mon[33038]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:05.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:04 smithi189 ceph-mon[33038]: from='client.? 172.21.15.71:0/806159827' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:30:05.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:04 smithi032 ceph-mon[28331]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:05.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:04 smithi032 ceph-mon[28331]: from='client.? 172.21.15.71:0/806159827' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:30:07.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:06 smithi071 ceph-mon[33385]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:07.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:06 smithi189 ceph-mon[33038]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:07.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:06 smithi032 ceph-mon[28331]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:09.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:08 smithi071 ceph-mon[33385]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:09.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:08 smithi189 ceph-mon[33038]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:09.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:08 smithi032 ceph-mon[28331]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:11.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:10 smithi071 ceph-mon[33385]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:11.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:10 smithi189 ceph-mon[33038]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:11.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:10 smithi032 ceph-mon[28331]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:12.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:12 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-30T08:30:12.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:12 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:30:12.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:12 smithi032 ceph-mon[28331]: Deploying daemon osd.10 on smithi071 2024-04-30T08:30:12.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:12 smithi032 ceph-mon[28331]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:12.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:12 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-30T08:30:12.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:12 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:30:12.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:12 smithi071 ceph-mon[33385]: Deploying daemon osd.10 on smithi071 2024-04-30T08:30:12.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:12 smithi071 ceph-mon[33385]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:12.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:12 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-30T08:30:12.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:12 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:30:12.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:12 smithi189 ceph-mon[33038]: Deploying daemon osd.10 on smithi071 2024-04-30T08:30:12.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:12 smithi189 ceph-mon[33038]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:14.944 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:14 smithi071 ceph-mon[33385]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:15.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:14 smithi189 ceph-mon[33038]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:15.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:14 smithi032 ceph-mon[28331]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:16.928 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:16 smithi071 ceph-mon[33385]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:17.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:16 smithi189 ceph-mon[33038]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:17.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:16 smithi032 ceph-mon[28331]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:18.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:18 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:18.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:18 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:18.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:18 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:30:18.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:18 smithi071 ceph-mon[33385]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:18.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:18 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:18.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:18 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:18.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:18 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:30:18.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:18 smithi189 ceph-mon[33038]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:18.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:18 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:18.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:18 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:18.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:18 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:30:18.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:18 smithi032 ceph-mon[28331]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:20.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:20 smithi071 ceph-mon[33385]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:21.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:20 smithi189 ceph-mon[33038]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:21.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:20 smithi032 ceph-mon[28331]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:21.281 INFO:teuthology.orchestra.run.smithi071.stdout:Created osd(s) 10 on host 'smithi071' 2024-04-30T08:30:21.282 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:21.280+0000 7faf397fa700 1 -- 172.21.15.71:0/2303729912 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7faf3c02e600 con 0x7faf240afc00 2024-04-30T08:30:21.283 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:21.282+0000 7faf428c7700 1 -- 172.21.15.71:0/2303729912 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7faf240afc00 msgr2=0x7faf240b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:30:21.283 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:21.282+0000 7faf428c7700 1 --2- 172.21.15.71:0/2303729912 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7faf240afc00 0x7faf240b2110 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7faf3c1041d0 tx=0x7faf2c005d20 comp rx=0 tx=0).stop 2024-04-30T08:30:21.284 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:21.282+0000 7faf428c7700 1 -- 172.21.15.71:0/2303729912 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf3c104c70 msgr2=0x7faf3c1a0d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:30:21.284 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:21.282+0000 7faf428c7700 1 --2- 172.21.15.71:0/2303729912 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf3c104c70 0x7faf3c1a0d00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7faf3400b970 tx=0x7faf3400bc80 comp rx=0 tx=0).stop 2024-04-30T08:30:21.284 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:21.282+0000 7faf428c7700 1 -- 172.21.15.71:0/2303729912 shutdown_connections 2024-04-30T08:30:21.284 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:21.282+0000 7faf428c7700 1 --2- 172.21.15.71:0/2303729912 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7faf240afc00 0x7faf240b2110 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:21.284 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:21.282+0000 7faf428c7700 1 --2- 172.21.15.71:0/2303729912 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faf3c104c70 0x7faf3c1a0d00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:21.284 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:21.282+0000 7faf428c7700 1 --2- 172.21.15.71:0/2303729912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faf3c1031a0 0x7faf3c199820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:21.284 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:21.282+0000 7faf428c7700 1 --2- 172.21.15.71:0/2303729912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faf3c1042d0 0x7faf3c199d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:21.284 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:21.282+0000 7faf428c7700 1 -- 172.21.15.71:0/2303729912 >> 172.21.15.71:0/2303729912 conn(0x7faf3c0fe7e0 msgr2=0x7faf3c107500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:30:21.285 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:21.283+0000 7faf428c7700 1 -- 172.21.15.71:0/2303729912 shutdown_connections 2024-04-30T08:30:21.285 INFO:teuthology.orchestra.run.smithi071.stderr:2024-04-30T08:30:21.283+0000 7faf428c7700 1 -- 172.21.15.71:0/2303729912 wait complete. 2024-04-30T08:30:21.996 DEBUG:teuthology.orchestra.run.smithi071:osd.10> sudo journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.10.service 2024-04-30T08:30:21.999 INFO:tasks.cephadm:Deploying osd.11 on smithi189 with /dev/vg_nvme/lv_1... 2024-04-30T08:30:21.999 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-30T08:30:22.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:22 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:22.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:22 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:22.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:22 smithi071 ceph-mon[33385]: from='osd.10 [v2:172.21.15.71:6824/1886290293,v1:172.21.15.71:6825/1886290293]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-30T08:30:22.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:22 smithi071 ceph-mon[33385]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-30T08:30:22.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:22 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:22.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:22 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:22.258 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.c/config 2024-04-30T08:30:22.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:22 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:22.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:22 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:22.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:22 smithi189 ceph-mon[33038]: from='osd.10 [v2:172.21.15.71:6824/1886290293,v1:172.21.15.71:6825/1886290293]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-30T08:30:22.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:22 smithi189 ceph-mon[33038]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-30T08:30:22.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:22 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:22.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:22 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:22.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:22 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:22.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:22 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:22.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:22 smithi032 ceph-mon[28331]: from='osd.10 [v2:172.21.15.71:6824/1886290293,v1:172.21.15.71:6825/1886290293]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-30T08:30:22.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:22 smithi032 ceph-mon[28331]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-30T08:30:22.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:22 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:22.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:22 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:23.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:23 smithi071 ceph-mon[33385]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:23.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:23 smithi071 ceph-mon[33385]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-30T08:30:23.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:23 smithi071 ceph-mon[33385]: from='osd.10 [v2:172.21.15.71:6824/1886290293,v1:172.21.15.71:6825/1886290293]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-04-30T08:30:23.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:23 smithi071 ceph-mon[33385]: osdmap e63: 11 total, 10 up, 11 in 2024-04-30T08:30:23.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:23 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-30T08:30:23.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:23 smithi071 ceph-mon[33385]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-04-30T08:30:23.504 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:30:23 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:30:23.247+0000 7f89f2da7700 -1 osd.10 0 waiting for initial osdmap 2024-04-30T08:30:23.504 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:30:23 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:30:23.260+0000 7f89eeb5c700 -1 osd.10 64 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-30T08:30:23.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:23 smithi189 ceph-mon[33038]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:23.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:23 smithi189 ceph-mon[33038]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-30T08:30:23.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:23 smithi189 ceph-mon[33038]: from='osd.10 [v2:172.21.15.71:6824/1886290293,v1:172.21.15.71:6825/1886290293]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-04-30T08:30:23.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:23 smithi189 ceph-mon[33038]: osdmap e63: 11 total, 10 up, 11 in 2024-04-30T08:30:23.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:23 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-30T08:30:23.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:23 smithi189 ceph-mon[33038]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-04-30T08:30:23.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:23 smithi032 ceph-mon[28331]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:23.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:23 smithi032 ceph-mon[28331]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-30T08:30:23.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:23 smithi032 ceph-mon[28331]: from='osd.10 [v2:172.21.15.71:6824/1886290293,v1:172.21.15.71:6825/1886290293]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-04-30T08:30:23.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:23 smithi032 ceph-mon[28331]: osdmap e63: 11 total, 10 up, 11 in 2024-04-30T08:30:23.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:23 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-30T08:30:23.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:23 smithi032 ceph-mon[28331]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]: dispatch 2024-04-30T08:30:24.364 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-30T08:30:24.364 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-30T08:30:24.364 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-04-30T08:30:24.364 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-04-30T08:30:24.364 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124461 s, 842 MB/s 2024-04-30T08:30:24.365 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-04-30T08:30:24.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:24 smithi071 ceph-mon[33385]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-04-30T08:30:24.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:24 smithi071 ceph-mon[33385]: osdmap e64: 11 total, 10 up, 11 in 2024-04-30T08:30:24.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:24 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-30T08:30:24.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:24 smithi071 ceph-mon[33385]: osd.10 [v2:172.21.15.71:6824/1886290293,v1:172.21.15.71:6825/1886290293] boot 2024-04-30T08:30:24.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:24 smithi071 ceph-mon[33385]: osdmap e65: 11 total, 11 up, 11 in 2024-04-30T08:30:24.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:24 smithi189 ceph-mon[33038]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-04-30T08:30:24.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:24 smithi189 ceph-mon[33038]: osdmap e64: 11 total, 10 up, 11 in 2024-04-30T08:30:24.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:24 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-30T08:30:24.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:24 smithi189 ceph-mon[33038]: osd.10 [v2:172.21.15.71:6824/1886290293,v1:172.21.15.71:6825/1886290293] boot 2024-04-30T08:30:24.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:24 smithi189 ceph-mon[33038]: osdmap e65: 11 total, 11 up, 11 in 2024-04-30T08:30:24.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:24 smithi032 ceph-mon[28331]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi071", "root=default"]}]': finished 2024-04-30T08:30:24.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:24 smithi032 ceph-mon[28331]: osdmap e64: 11 total, 10 up, 11 in 2024-04-30T08:30:24.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:24 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-30T08:30:24.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:24 smithi032 ceph-mon[28331]: osd.10 [v2:172.21.15.71:6824/1886290293,v1:172.21.15.71:6825/1886290293] boot 2024-04-30T08:30:24.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:24 smithi032 ceph-mon[28331]: osdmap e65: 11 total, 11 up, 11 in 2024-04-30T08:30:25.073 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi189:vg_nvme/lv_1 2024-04-30T08:30:25.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:25 smithi189 ceph-mon[33038]: purged_snaps scrub starts 2024-04-30T08:30:25.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:25 smithi189 ceph-mon[33038]: purged_snaps scrub ok 2024-04-30T08:30:25.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:25 smithi189 ceph-mon[33038]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:25.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:25 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-30T08:30:25.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:25 smithi189 ceph-mon[33038]: osdmap e66: 11 total, 11 up, 11 in 2024-04-30T08:30:25.330 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.c/config 2024-04-30T08:30:25.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:25 smithi071 ceph-mon[33385]: purged_snaps scrub starts 2024-04-30T08:30:25.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:25 smithi071 ceph-mon[33385]: purged_snaps scrub ok 2024-04-30T08:30:25.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:25 smithi071 ceph-mon[33385]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:25.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:25 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-30T08:30:25.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:25 smithi071 ceph-mon[33385]: osdmap e66: 11 total, 11 up, 11 in 2024-04-30T08:30:25.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:25 smithi032 ceph-mon[28331]: purged_snaps scrub starts 2024-04-30T08:30:25.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:25 smithi032 ceph-mon[28331]: purged_snaps scrub ok 2024-04-30T08:30:25.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:25 smithi032 ceph-mon[28331]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 294 MiB used, 894 GiB / 894 GiB avail 2024-04-30T08:30:25.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:25 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-30T08:30:25.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:25 smithi032 ceph-mon[28331]: osdmap e66: 11 total, 11 up, 11 in 2024-04-30T08:30:26.886 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:26 smithi189 ceph-mon[33038]: Detected new or changed devices on smithi071 2024-04-30T08:30:26.886 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:26 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:26.886 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:26 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:26.887 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:26 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:26.887 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:26 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:26.887 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:26 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:26.887 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:26 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:26.887 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:26 smithi189 ceph-mon[33038]: Adjusting osd_memory_target on smithi071 to 4272M 2024-04-30T08:30:26.887 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:26 smithi189 ceph-mon[33038]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-04-30T08:30:26.887 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:26 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:26.887 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:26 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:30:26.887 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:26 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:30:26.887 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:26 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:26.887 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:26 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:30:26.887 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:26 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:30:26.887 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:26 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:30:26.888 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:26 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:26.888 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:26 smithi189 ceph-mon[33038]: osdmap e67: 11 total, 11 up, 11 in 2024-04-30T08:30:26.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:26 smithi032 ceph-mon[28331]: Detected new or changed devices on smithi071 2024-04-30T08:30:26.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:26 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:26.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:26 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:26.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:26 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:26.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:26 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:26.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:26 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:26.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:26 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:26.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:26 smithi032 ceph-mon[28331]: Adjusting osd_memory_target on smithi071 to 4272M 2024-04-30T08:30:26.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:26 smithi032 ceph-mon[28331]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-04-30T08:30:26.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:26 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:26.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:26 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:30:26.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:26 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:30:26.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:26 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:26.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:26 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:30:26.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:26 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:30:26.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:26 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:30:26.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:26 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:26.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:26 smithi032 ceph-mon[28331]: osdmap e67: 11 total, 11 up, 11 in 2024-04-30T08:30:27.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:26 smithi071 ceph-mon[33385]: Detected new or changed devices on smithi071 2024-04-30T08:30:27.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:26 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:27.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:26 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:27.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:26 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:27.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:26 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:27.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:26 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:27.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:26 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:27.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:26 smithi071 ceph-mon[33385]: Adjusting osd_memory_target on smithi071 to 4272M 2024-04-30T08:30:27.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:26 smithi071 ceph-mon[33385]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-04-30T08:30:27.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:26 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:27.005 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:26 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:30:27.005 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:26 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:30:27.005 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:26 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:27.005 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:26 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:30:27.005 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:26 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:30:27.005 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:26 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:30:27.005 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:26 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:27.005 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:26 smithi071 ceph-mon[33385]: osdmap e67: 11 total, 11 up, 11 in 2024-04-30T08:30:27.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.129+0000 7ffb32b1e700 1 -- 172.21.15.189:0/2062427842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb2c103180 msgr2=0x7ffb2c103590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:30:27.132 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.129+0000 7ffb32b1e700 1 --2- 172.21.15.189:0/2062427842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb2c103180 0x7ffb2c103590 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ffb20009990 tx=0x7ffb20009ca0 comp rx=0 tx=0).stop 2024-04-30T08:30:27.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.130+0000 7ffb32b1e700 1 -- 172.21.15.189:0/2062427842 shutdown_connections 2024-04-30T08:30:27.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.130+0000 7ffb32b1e700 1 --2- 172.21.15.189:0/2062427842 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb2c104c50 0x7ffb2c10b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:27.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.130+0000 7ffb32b1e700 1 --2- 172.21.15.189:0/2062427842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb2c103180 0x7ffb2c103590 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:27.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.130+0000 7ffb32b1e700 1 --2- 172.21.15.189:0/2062427842 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffb2c1042b0 0x7ffb2c1046c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:27.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.130+0000 7ffb32b1e700 1 -- 172.21.15.189:0/2062427842 >> 172.21.15.189:0/2062427842 conn(0x7ffb2c0fe7c0 msgr2=0x7ffb2c100c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:30:27.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.130+0000 7ffb32b1e700 1 -- 172.21.15.189:0/2062427842 shutdown_connections 2024-04-30T08:30:27.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.130+0000 7ffb32b1e700 1 -- 172.21.15.189:0/2062427842 wait complete. 2024-04-30T08:30:27.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.130+0000 7ffb32b1e700 1 Processor -- start 2024-04-30T08:30:27.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.130+0000 7ffb32b1e700 1 -- start start 2024-04-30T08:30:27.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.130+0000 7ffb32b1e700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb2c103180 0x7ffb2c199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:30:27.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.130+0000 7ffb32b1e700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb2c1042b0 0x7ffb2c199dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:30:27.133 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.130+0000 7ffb32b1e700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffb2c104c50 0x7ffb2c1a0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:30:27.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.130+0000 7ffb32b1e700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ffb2c10ecc0 con 0x7ffb2c1042b0 2024-04-30T08:30:27.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.130+0000 7ffb32b1e700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ffb2c10eb40 con 0x7ffb2c104c50 2024-04-30T08:30:27.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.130+0000 7ffb32b1e700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ffb2c10ee40 con 0x7ffb2c103180 2024-04-30T08:30:27.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.131+0000 7ffb308ba700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb2c103180 0x7ffb2c199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:30:27.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.131+0000 7ffb308ba700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb2c103180 0x7ffb2c199890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:57860/0 (socket says 172.21.15.189:57860) 2024-04-30T08:30:27.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.131+0000 7ffb308ba700 1 -- 172.21.15.189:0/1651801110 learned_addr learned my addr 172.21.15.189:0/1651801110 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-30T08:30:27.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.131+0000 7ffb310bb700 1 --2- 172.21.15.189:0/1651801110 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffb2c104c50 0x7ffb2c1a0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:30:27.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.131+0000 7ffb308ba700 1 -- 172.21.15.189:0/1651801110 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffb2c104c50 msgr2=0x7ffb2c1a0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:30:27.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.131+0000 7ffb308ba700 1 --2- 172.21.15.189:0/1651801110 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffb2c104c50 0x7ffb2c1a0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:27.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.131+0000 7ffb308ba700 1 -- 172.21.15.189:0/1651801110 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb2c1042b0 msgr2=0x7ffb2c199dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:30:27.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.131+0000 7ffb308ba700 1 --2- 172.21.15.189:0/1651801110 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb2c1042b0 0x7ffb2c199dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:27.134 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.131+0000 7ffb308ba700 1 -- 172.21.15.189:0/1651801110 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb2c1a13f0 con 0x7ffb2c103180 2024-04-30T08:30:27.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.131+0000 7ffb308ba700 1 --2- 172.21.15.189:0/1651801110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb2c103180 0x7ffb2c199890 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ffb200057c0 tx=0x7ffb200057f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:30:27.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.131+0000 7ffb29ffb700 1 -- 172.21.15.189:0/1651801110 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb20029070 con 0x7ffb2c103180 2024-04-30T08:30:27.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.131+0000 7ffb29ffb700 1 -- 172.21.15.189:0/1651801110 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffb20019d80 con 0x7ffb2c103180 2024-04-30T08:30:27.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.131+0000 7ffb29ffb700 1 -- 172.21.15.189:0/1651801110 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb20024470 con 0x7ffb2c103180 2024-04-30T08:30:27.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.131+0000 7ffb32b1e700 1 -- 172.21.15.189:0/1651801110 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb2c06bf90 con 0x7ffb2c103180 2024-04-30T08:30:27.135 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.132+0000 7ffb32b1e700 1 -- 172.21.15.189:0/1651801110 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb2c06c540 con 0x7ffb2c103180 2024-04-30T08:30:27.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.133+0000 7ffb32b1e700 1 -- 172.21.15.189:0/1651801110 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb2c051210 con 0x7ffb2c103180 2024-04-30T08:30:27.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.134+0000 7ffb29ffb700 1 -- 172.21.15.189:0/1651801110 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7ffb2002c070 con 0x7ffb2c103180 2024-04-30T08:30:27.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.135+0000 7ffb29ffb700 1 --2- 172.21.15.189:0/1651801110 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ffb1c0afc00 0x7ffb1c0b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:30:27.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.135+0000 7ffb29ffb700 1 -- 172.21.15.189:0/1651801110 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(67..67 src has 1..67) ==== 6694+0+0 (secure 0 0 0) 0x7ffb200e1bc0 con 0x7ffb2c103180 2024-04-30T08:30:27.136 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.135+0000 7ffb2bfff700 1 --2- 172.21.15.189:0/1651801110 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ffb1c0afc00 0x7ffb1c0b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:30:27.137 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.136+0000 7ffb2bfff700 1 --2- 172.21.15.189:0/1651801110 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ffb1c0afc00 0x7ffb1c0b2110 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c19a940 tx=0x7ffb18006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:30:27.139 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.138+0000 7ffb29ffb700 1 -- 172.21.15.189:0/1651801110 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffb20060ff0 con 0x7ffb2c103180 2024-04-30T08:30:27.345 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:27.344+0000 7ffb32b1e700 1 -- 172.21.15.189:0/1651801110 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7ffb2c06c860 con 0x7ffb1c0afc00 2024-04-30T08:30:27.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:27 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:30:27.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:27 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:30:27.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:27 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:30:27.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:30:27.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:30:27.989 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:30:28.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:27 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-30T08:30:28.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:27 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-30T08:30:28.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:27 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:30:28.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:28 smithi032 ceph-mon[28331]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:30:28.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:28 smithi032 ceph-mon[28331]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-04-30T08:30:28.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:28 smithi189 ceph-mon[33038]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:30:28.999 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:28 smithi189 ceph-mon[33038]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-04-30T08:30:29.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:28 smithi071 ceph-mon[33385]: from='client.24373 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:30:29.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:28 smithi071 ceph-mon[33385]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-04-30T08:30:31.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:30 smithi071 ceph-mon[33385]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-30T08:30:31.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:30 smithi071 ceph-mon[33385]: from='client.? 172.21.15.189:0/854434333' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab96b3ff-3688-455b-af64-60226c5bfbff"}]: dispatch 2024-04-30T08:30:31.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:30 smithi071 ceph-mon[33385]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab96b3ff-3688-455b-af64-60226c5bfbff"}]: dispatch 2024-04-30T08:30:31.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:30 smithi071 ceph-mon[33385]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ab96b3ff-3688-455b-af64-60226c5bfbff"}]': finished 2024-04-30T08:30:31.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:30 smithi071 ceph-mon[33385]: osdmap e68: 12 total, 11 up, 12 in 2024-04-30T08:30:31.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:30 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-30T08:30:31.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:30 smithi189 ceph-mon[33038]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-30T08:30:31.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:30 smithi189 ceph-mon[33038]: from='client.? 172.21.15.189:0/854434333' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab96b3ff-3688-455b-af64-60226c5bfbff"}]: dispatch 2024-04-30T08:30:31.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:30 smithi189 ceph-mon[33038]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab96b3ff-3688-455b-af64-60226c5bfbff"}]: dispatch 2024-04-30T08:30:31.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:30 smithi189 ceph-mon[33038]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ab96b3ff-3688-455b-af64-60226c5bfbff"}]': finished 2024-04-30T08:30:31.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:30 smithi189 ceph-mon[33038]: osdmap e68: 12 total, 11 up, 12 in 2024-04-30T08:30:31.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:30 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-30T08:30:31.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:30 smithi032 ceph-mon[28331]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-30T08:30:31.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:30 smithi032 ceph-mon[28331]: from='client.? 172.21.15.189:0/854434333' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab96b3ff-3688-455b-af64-60226c5bfbff"}]: dispatch 2024-04-30T08:30:31.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:30 smithi032 ceph-mon[28331]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ab96b3ff-3688-455b-af64-60226c5bfbff"}]: dispatch 2024-04-30T08:30:31.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:30 smithi032 ceph-mon[28331]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ab96b3ff-3688-455b-af64-60226c5bfbff"}]': finished 2024-04-30T08:30:31.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:30 smithi032 ceph-mon[28331]: osdmap e68: 12 total, 11 up, 12 in 2024-04-30T08:30:31.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:30 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-30T08:30:32.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:31 smithi071 ceph-mon[33385]: from='client.? 172.21.15.189:0/2513659105' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:30:32.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:31 smithi189 ceph-mon[33038]: from='client.? 172.21.15.189:0/2513659105' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:30:32.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:31 smithi032 ceph-mon[28331]: from='client.? 172.21.15.189:0/2513659105' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-30T08:30:33.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:32 smithi071 ceph-mon[33385]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-30T08:30:33.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:32 smithi189 ceph-mon[33038]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-30T08:30:33.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:32 smithi032 ceph-mon[28331]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 90 KiB/s, 0 objects/s recovering 2024-04-30T08:30:35.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:34 smithi071 ceph-mon[33385]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-30T08:30:35.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:34 smithi189 ceph-mon[33038]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-30T08:30:35.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:34 smithi032 ceph-mon[28331]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-30T08:30:36.992 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:36 smithi189 ceph-mon[33038]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-30T08:30:37.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:36 smithi071 ceph-mon[33385]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-30T08:30:37.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:36 smithi032 ceph-mon[28331]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-30T08:30:38.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:37 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-30T08:30:38.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:37 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:30:38.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:37 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-30T08:30:38.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:37 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:30:38.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:37 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-30T08:30:38.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:37 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:30:39.043 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:38 smithi032 ceph-mon[28331]: Deploying daemon osd.11 on smithi189 2024-04-30T08:30:39.043 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:38 smithi032 ceph-mon[28331]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-30T08:30:39.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:38 smithi189 ceph-mon[33038]: Deploying daemon osd.11 on smithi189 2024-04-30T08:30:39.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:38 smithi189 ceph-mon[33038]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-30T08:30:39.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:38 smithi071 ceph-mon[33385]: Deploying daemon osd.11 on smithi189 2024-04-30T08:30:39.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:38 smithi071 ceph-mon[33385]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-30T08:30:41.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:40 smithi189 ceph-mon[33038]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-04-30T08:30:41.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:40 smithi032 ceph-mon[28331]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-04-30T08:30:41.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:40 smithi071 ceph-mon[33385]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-04-30T08:30:43.043 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:42 smithi189 ceph-mon[33038]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-04-30T08:30:43.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:42 smithi032 ceph-mon[28331]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-04-30T08:30:43.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:42 smithi071 ceph-mon[33385]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-04-30T08:30:45.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:44 smithi071 ceph-mon[33385]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-04-30T08:30:45.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:44 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:45.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:44 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:45.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:44 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:30:45.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:44 smithi189 ceph-mon[33038]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-04-30T08:30:45.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:44 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:45.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:44 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:45.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:44 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:30:45.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:44 smithi032 ceph-mon[28331]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-04-30T08:30:45.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:44 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:45.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:44 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:45.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:44 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:30:47.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:46 smithi071 ceph-mon[33385]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-04-30T08:30:47.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:46 smithi189 ceph-mon[33038]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-04-30T08:30:47.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:46 smithi032 ceph-mon[28331]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-04-30T08:30:47.835 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:47.833+0000 7ffb29ffb700 1 -- 172.21.15.189:0/1651801110 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7ffb2c06c860 con 0x7ffb1c0afc00 2024-04-30T08:30:47.835 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 11 on host 'smithi189' 2024-04-30T08:30:47.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:47.836+0000 7ffb32b1e700 1 -- 172.21.15.189:0/1651801110 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ffb1c0afc00 msgr2=0x7ffb1c0b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:30:47.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:47.836+0000 7ffb32b1e700 1 --2- 172.21.15.189:0/1651801110 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ffb1c0afc00 0x7ffb1c0b2110 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c19a940 tx=0x7ffb18006d20 comp rx=0 tx=0).stop 2024-04-30T08:30:47.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:47.836+0000 7ffb32b1e700 1 -- 172.21.15.189:0/1651801110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb2c103180 msgr2=0x7ffb2c199890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:30:47.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:47.836+0000 7ffb32b1e700 1 --2- 172.21.15.189:0/1651801110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb2c103180 0x7ffb2c199890 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7ffb200057c0 tx=0x7ffb200057f0 comp rx=0 tx=0).stop 2024-04-30T08:30:47.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:47.836+0000 7ffb32b1e700 1 -- 172.21.15.189:0/1651801110 shutdown_connections 2024-04-30T08:30:47.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:47.836+0000 7ffb32b1e700 1 --2- 172.21.15.189:0/1651801110 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ffb1c0afc00 0x7ffb1c0b2110 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:47.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:47.836+0000 7ffb32b1e700 1 --2- 172.21.15.189:0/1651801110 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffb2c104c50 0x7ffb2c1a0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:47.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:47.836+0000 7ffb32b1e700 1 --2- 172.21.15.189:0/1651801110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb2c103180 0x7ffb2c199890 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:47.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:47.836+0000 7ffb32b1e700 1 --2- 172.21.15.189:0/1651801110 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffb2c1042b0 0x7ffb2c199dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:47.837 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:47.836+0000 7ffb32b1e700 1 -- 172.21.15.189:0/1651801110 >> 172.21.15.189:0/1651801110 conn(0x7ffb2c0fe7c0 msgr2=0x7ffb2c1074e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:30:47.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:47.836+0000 7ffb32b1e700 1 -- 172.21.15.189:0/1651801110 shutdown_connections 2024-04-30T08:30:47.838 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:30:47.836+0000 7ffb32b1e700 1 -- 172.21.15.189:0/1651801110 wait complete. 2024-04-30T08:30:48.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:47 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:48.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:47 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:48.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:47 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:30:48.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:47 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:30:48.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:47 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:48.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:47 smithi189 ceph-mon[33038]: from='osd.11 [v2:172.21.15.189:6824/850185001,v1:172.21.15.189:6825/850185001]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-30T08:30:48.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:47 smithi189 ceph-mon[33038]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-30T08:30:48.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:47 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:48.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:47 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:48.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:47 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:30:48.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:47 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:48.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:47 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:48.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:47 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:30:48.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:47 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:30:48.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:47 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:48.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:47 smithi032 ceph-mon[28331]: from='osd.11 [v2:172.21.15.189:6824/850185001,v1:172.21.15.189:6825/850185001]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-30T08:30:48.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:47 smithi032 ceph-mon[28331]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-30T08:30:48.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:47 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:48.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:47 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:48.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:47 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:30:48.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:47 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:48.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:47 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:48.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:47 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:30:48.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:47 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:30:48.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:47 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:48.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:47 smithi071 ceph-mon[33385]: from='osd.11 [v2:172.21.15.189:6824/850185001,v1:172.21.15.189:6825/850185001]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-30T08:30:48.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:47 smithi071 ceph-mon[33385]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-30T08:30:48.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:47 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:48.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:47 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:48.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:47 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:30:48.467 DEBUG:teuthology.orchestra.run.smithi189:osd.11> sudo journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.11.service 2024-04-30T08:30:48.469 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-30T08:30:48.470 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-30T08:30:48.925 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:30:49.002 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:48 smithi189 ceph-mon[33038]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-04-30T08:30:49.002 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:48 smithi189 ceph-mon[33038]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-30T08:30:49.002 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:48 smithi189 ceph-mon[33038]: from='osd.11 [v2:172.21.15.189:6824/850185001,v1:172.21.15.189:6825/850185001]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-30T08:30:49.002 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:48 smithi189 ceph-mon[33038]: osdmap e69: 12 total, 11 up, 12 in 2024-04-30T08:30:49.002 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:48 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-30T08:30:49.002 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:48 smithi189 ceph-mon[33038]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-30T08:30:49.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:48 smithi032 ceph-mon[28331]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-04-30T08:30:49.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:48 smithi032 ceph-mon[28331]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-30T08:30:49.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:48 smithi032 ceph-mon[28331]: from='osd.11 [v2:172.21.15.189:6824/850185001,v1:172.21.15.189:6825/850185001]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-30T08:30:49.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:48 smithi032 ceph-mon[28331]: osdmap e69: 12 total, 11 up, 12 in 2024-04-30T08:30:49.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:48 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-30T08:30:49.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:48 smithi032 ceph-mon[28331]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-30T08:30:49.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:48 smithi071 ceph-mon[33385]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-04-30T08:30:49.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:48 smithi071 ceph-mon[33385]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-30T08:30:49.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:48 smithi071 ceph-mon[33385]: from='osd.11 [v2:172.21.15.189:6824/850185001,v1:172.21.15.189:6825/850185001]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-30T08:30:49.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:48 smithi071 ceph-mon[33385]: osdmap e69: 12 total, 11 up, 12 in 2024-04-30T08:30:49.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:48 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-30T08:30:49.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:48 smithi071 ceph-mon[33385]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-04-30T08:30:49.674 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:30:49 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:30:49.345+0000 7fe3a73ee700 -1 osd.11 0 waiting for initial osdmap 2024-04-30T08:30:49.674 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:30:49 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:30:49.360+0000 7fe39f19b700 -1 osd.11 70 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-30T08:30:50.194 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:49 smithi032 ceph-mon[28331]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-30T08:30:50.194 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:49 smithi032 ceph-mon[28331]: osdmap e70: 12 total, 11 up, 12 in 2024-04-30T08:30:50.194 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:49 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-30T08:30:50.194 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:49 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-30T08:30:50.195 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:49 smithi032 ceph-mon[28331]: from='osd.11 ' entity='osd.11' 2024-04-30T08:30:50.211 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:49 smithi189 ceph-mon[33038]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-30T08:30:50.211 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:49 smithi189 ceph-mon[33038]: osdmap e70: 12 total, 11 up, 12 in 2024-04-30T08:30:50.211 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:49 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-30T08:30:50.211 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:49 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-30T08:30:50.211 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:49 smithi189 ceph-mon[33038]: from='osd.11 ' entity='osd.11' 2024-04-30T08:30:50.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:49 smithi071 ceph-mon[33385]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-04-30T08:30:50.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:49 smithi071 ceph-mon[33385]: osdmap e70: 12 total, 11 up, 12 in 2024-04-30T08:30:50.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:49 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-30T08:30:50.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:49 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-30T08:30:50.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:49 smithi071 ceph-mon[33385]: from='osd.11 ' entity='osd.11' 2024-04-30T08:30:51.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:50 smithi189 ceph-mon[33038]: purged_snaps scrub starts 2024-04-30T08:30:51.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:50 smithi189 ceph-mon[33038]: purged_snaps scrub ok 2024-04-30T08:30:51.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:50 smithi189 ceph-mon[33038]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-04-30T08:30:51.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:50 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-30T08:30:51.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:50 smithi189 ceph-mon[33038]: osd.11 [v2:172.21.15.189:6824/850185001,v1:172.21.15.189:6825/850185001] boot 2024-04-30T08:30:51.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:50 smithi189 ceph-mon[33038]: osdmap e71: 12 total, 12 up, 12 in 2024-04-30T08:30:51.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:50 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-30T08:30:51.093 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:50 smithi032 ceph-mon[28331]: purged_snaps scrub starts 2024-04-30T08:30:51.093 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:50 smithi032 ceph-mon[28331]: purged_snaps scrub ok 2024-04-30T08:30:51.094 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:50 smithi032 ceph-mon[28331]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-04-30T08:30:51.094 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:50 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-30T08:30:51.094 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:50 smithi032 ceph-mon[28331]: osd.11 [v2:172.21.15.189:6824/850185001,v1:172.21.15.189:6825/850185001] boot 2024-04-30T08:30:51.094 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:50 smithi032 ceph-mon[28331]: osdmap e71: 12 total, 12 up, 12 in 2024-04-30T08:30:51.094 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:50 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-30T08:30:51.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:50 smithi071 ceph-mon[33385]: purged_snaps scrub starts 2024-04-30T08:30:51.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:50 smithi071 ceph-mon[33385]: purged_snaps scrub ok 2024-04-30T08:30:51.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:50 smithi071 ceph-mon[33385]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 322 MiB used, 983 GiB / 983 GiB avail 2024-04-30T08:30:51.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:50 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-30T08:30:51.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:50 smithi071 ceph-mon[33385]: osd.11 [v2:172.21.15.189:6824/850185001,v1:172.21.15.189:6825/850185001] boot 2024-04-30T08:30:51.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:50 smithi071 ceph-mon[33385]: osdmap e71: 12 total, 12 up, 12 in 2024-04-30T08:30:51.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:50 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-30T08:30:51.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.345+0000 7f1ee2c00700 1 -- 172.21.15.32:0/3497520764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1edc1053e0 msgr2=0x7f1edc1057f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:30:51.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.345+0000 7f1ee2c00700 1 --2- 172.21.15.32:0/3497520764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1edc1053e0 0x7f1edc1057f0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1ec8009990 tx=0x7f1ec8009ca0 comp rx=0 tx=0).stop 2024-04-30T08:30:51.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.345+0000 7f1ee2c00700 1 -- 172.21.15.32:0/3497520764 shutdown_connections 2024-04-30T08:30:51.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.345+0000 7f1ee2c00700 1 --2- 172.21.15.32:0/3497520764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1edc105d80 0x7f1edc10c630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:51.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.345+0000 7f1ee2c00700 1 --2- 172.21.15.32:0/3497520764 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1edc1042b0 0x7f1edc1046c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:51.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.345+0000 7f1ee2c00700 1 --2- 172.21.15.32:0/3497520764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1edc1053e0 0x7f1edc1057f0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:51.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.345+0000 7f1ee2c00700 1 -- 172.21.15.32:0/3497520764 >> 172.21.15.32:0/3497520764 conn(0x7f1edc0ff910 msgr2=0x7f1edc101da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:30:51.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.345+0000 7f1ee2c00700 1 -- 172.21.15.32:0/3497520764 shutdown_connections 2024-04-30T08:30:51.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.345+0000 7f1ee2c00700 1 -- 172.21.15.32:0/3497520764 wait complete. 2024-04-30T08:30:51.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.346+0000 7f1ee2c00700 1 Processor -- start 2024-04-30T08:30:51.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.347+0000 7f1ee2c00700 1 -- start start 2024-04-30T08:30:51.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.347+0000 7f1ee2c00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1edc1042b0 0x7f1edc19a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:30:51.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.347+0000 7f1ee2c00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1edc1053e0 0x7f1edc19ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:30:51.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.347+0000 7f1ee2c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1edc105d80 0x7f1edc1a1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:30:51.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.347+0000 7f1ee2c00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1edc10fd70 con 0x7f1edc1053e0 2024-04-30T08:30:51.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.347+0000 7f1ee2c00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1edc10fbf0 con 0x7f1edc1042b0 2024-04-30T08:30:51.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.347+0000 7f1ee2c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1edc10fef0 con 0x7f1edc105d80 2024-04-30T08:30:51.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.347+0000 7f1edb400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1edc1053e0 0x7f1edc19ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:30:51.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.347+0000 7f1ee0c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1edc105d80 0x7f1edc1a1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:30:51.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.347+0000 7f1edbe00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1edc1042b0 0x7f1edc19a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:30:51.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.347+0000 7f1edb400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1edc1053e0 0x7f1edc19ae50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52572/0 (socket says 172.21.15.32:52572) 2024-04-30T08:30:51.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.347+0000 7f1ee0c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1edc105d80 0x7f1edc1a1df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:34544/0 (socket says 172.21.15.32:34544) 2024-04-30T08:30:51.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.347+0000 7f1edb400700 1 -- 172.21.15.32:0/1792660566 learned_addr learned my addr 172.21.15.32:0/1792660566 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:30:51.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.348+0000 7f1ee0c00700 1 -- 172.21.15.32:0/1792660566 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1edc1042b0 msgr2=0x7f1edc19a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:30:51.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.348+0000 7f1ee0c00700 1 --2- 172.21.15.32:0/1792660566 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1edc1042b0 0x7f1edc19a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:51.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.348+0000 7f1ee0c00700 1 -- 172.21.15.32:0/1792660566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1edc1053e0 msgr2=0x7f1edc19ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:30:51.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.348+0000 7f1ee0c00700 1 --2- 172.21.15.32:0/1792660566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1edc1053e0 0x7f1edc19ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:51.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.348+0000 7f1ee0c00700 1 -- 172.21.15.32:0/1792660566 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1edc1a2470 con 0x7f1edc105d80 2024-04-30T08:30:51.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.348+0000 7f1ee0c00700 1 --2- 172.21.15.32:0/1792660566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1edc105d80 0x7f1edc1a1df0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1ed400efc0 tx=0x7f1ed400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:30:51.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.348+0000 7f1ed8c00700 1 -- 172.21.15.32:0/1792660566 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed40092e0 con 0x7f1edc105d80 2024-04-30T08:30:51.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.348+0000 7f1ed8c00700 1 -- 172.21.15.32:0/1792660566 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1ed4004bb0 con 0x7f1edc105d80 2024-04-30T08:30:51.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.349+0000 7f1ed8c00700 1 -- 172.21.15.32:0/1792660566 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ed4004020 con 0x7f1edc105d80 2024-04-30T08:30:51.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.349+0000 7f1ee2c00700 1 -- 172.21.15.32:0/1792660566 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1edc1a2700 con 0x7f1edc105d80 2024-04-30T08:30:51.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.349+0000 7f1ee2c00700 1 -- 172.21.15.32:0/1792660566 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1edc1a2bf0 con 0x7f1edc105d80 2024-04-30T08:30:51.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.350+0000 7f1ee2c00700 1 -- 172.21.15.32:0/1792660566 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1edc051210 con 0x7f1edc105d80 2024-04-30T08:30:51.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.352+0000 7f1ed8c00700 1 -- 172.21.15.32:0/1792660566 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f1ed401b030 con 0x7f1edc105d80 2024-04-30T08:30:51.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.354+0000 7f1ed8c00700 1 --2- 172.21.15.32:0/1792660566 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f1ec00afb80 0x7f1ec00b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:30:51.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.354+0000 7f1ed8c00700 1 -- 172.21.15.32:0/1792660566 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(71..71 src has 1..71) ==== 7127+0+0 (secure 0 0 0) 0x7f1ed40decf0 con 0x7f1edc105d80 2024-04-30T08:30:51.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.354+0000 7f1edbe00700 1 --2- 172.21.15.32:0/1792660566 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f1ec00afb80 0x7f1ec00b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:30:51.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.355+0000 7f1edbe00700 1 --2- 172.21.15.32:0/1792660566 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f1ec00afb80 0x7f1ec00b2090 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc009680 tx=0x7f1ecc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:30:51.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.356+0000 7f1ed8c00700 1 -- 172.21.15.32:0/1792660566 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1ed4022020 con 0x7f1edc105d80 2024-04-30T08:30:51.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.590+0000 7f1ee2c00700 1 -- 172.21.15.32:0/1792660566 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f1edc02e590 con 0x7f1edc105d80 2024-04-30T08:30:51.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.590+0000 7f1ed8c00700 1 -- 172.21.15.32:0/1792660566 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v72) ==== 74+0+133 (secure 0 0 0) 0x7f1ed4010040 con 0x7f1edc105d80 2024-04-30T08:30:51.592 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:30:51.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.594+0000 7f1ee2c00700 1 -- 172.21.15.32:0/1792660566 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f1ec00afb80 msgr2=0x7f1ec00b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:30:51.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.594+0000 7f1ee2c00700 1 --2- 172.21.15.32:0/1792660566 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f1ec00afb80 0x7f1ec00b2090 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f1ecc009680 tx=0x7f1ecc009260 comp rx=0 tx=0).stop 2024-04-30T08:30:51.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.594+0000 7f1ee2c00700 1 -- 172.21.15.32:0/1792660566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1edc105d80 msgr2=0x7f1edc1a1df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:30:51.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.594+0000 7f1ee2c00700 1 --2- 172.21.15.32:0/1792660566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1edc105d80 0x7f1edc1a1df0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f1ed400efc0 tx=0x7f1ed400c3a0 comp rx=0 tx=0).stop 2024-04-30T08:30:51.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.594+0000 7f1ee2c00700 1 -- 172.21.15.32:0/1792660566 shutdown_connections 2024-04-30T08:30:51.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.594+0000 7f1ee2c00700 1 --2- 172.21.15.32:0/1792660566 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f1ec00afb80 0x7f1ec00b2090 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:51.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.594+0000 7f1ee2c00700 1 --2- 172.21.15.32:0/1792660566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1edc105d80 0x7f1edc1a1df0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:51.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.594+0000 7f1ee2c00700 1 --2- 172.21.15.32:0/1792660566 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1edc1042b0 0x7f1edc19a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:51.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.594+0000 7f1ee2c00700 1 --2- 172.21.15.32:0/1792660566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1edc1053e0 0x7f1edc19ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:51.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.594+0000 7f1ee2c00700 1 -- 172.21.15.32:0/1792660566 >> 172.21.15.32:0/1792660566 conn(0x7f1edc0ff910 msgr2=0x7f1edc108610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:30:51.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.594+0000 7f1ee2c00700 1 -- 172.21.15.32:0/1792660566 shutdown_connections 2024-04-30T08:30:51.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:51.594+0000 7f1ee2c00700 1 -- 172.21.15.32:0/1792660566 wait complete. 2024-04-30T08:30:52.558 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":72,"num_osds":12,"num_up_osds":12,"osd_up_since":1714465850,"num_in_osds":12,"osd_in_since":1714465830,"num_remapped_pgs":0} 2024-04-30T08:30:52.558 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-30T08:30:52.630 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:52 smithi032 ceph-mon[28331]: osdmap e72: 12 total, 12 up, 12 in 2024-04-30T08:30:52.630 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:52 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1792660566' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-30T08:30:52.630 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:52 smithi032 ceph-mon[28331]: Detected new or changed devices on smithi189 2024-04-30T08:30:52.630 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:52.630 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:52.630 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:52.630 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:52.630 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:52.631 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:52.631 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:52 smithi032 ceph-mon[28331]: Adjusting osd_memory_target on smithi189 to 4272M 2024-04-30T08:30:52.631 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:52.631 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:30:52.631 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:30:52.631 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:52.631 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:52 smithi032 ceph-mon[28331]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:30:52.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:52 smithi071 ceph-mon[33385]: osdmap e72: 12 total, 12 up, 12 in 2024-04-30T08:30:52.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:52 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1792660566' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-30T08:30:52.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:52 smithi071 ceph-mon[33385]: Detected new or changed devices on smithi189 2024-04-30T08:30:52.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:52 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:52.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:52 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:52.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:52 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:52.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:52 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:52.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:52 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:52.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:52 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:52.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:52 smithi071 ceph-mon[33385]: Adjusting osd_memory_target on smithi189 to 4272M 2024-04-30T08:30:52.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:52 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:52.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:52 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:30:52.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:52 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:30:52.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:52 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:52.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:52 smithi071 ceph-mon[33385]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:30:52.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:52 smithi189 ceph-mon[33038]: osdmap e72: 12 total, 12 up, 12 in 2024-04-30T08:30:52.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:52 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1792660566' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-30T08:30:52.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:52 smithi189 ceph-mon[33038]: Detected new or changed devices on smithi189 2024-04-30T08:30:52.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:52 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:52.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:52 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:52.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:52 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:52.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:52 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:52.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:52 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:52.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:52 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:30:52.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:52 smithi189 ceph-mon[33038]: Adjusting osd_memory_target on smithi189 to 4272M 2024-04-30T08:30:52.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:52 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:52.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:52 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:30:52.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:52 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:30:52.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:52 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:30:52.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:52 smithi189 ceph-mon[33038]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:30:53.006 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:30:54.836 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:54 smithi032 ceph-mon[28331]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:30:55.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:54 smithi071 ceph-mon[33385]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:30:55.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:54 smithi189 ceph-mon[33038]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:30:55.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.612+0000 7f7704c00700 1 -- 172.21.15.32:0/958518246 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77001051e0 msgr2=0x7f77001055f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:30:55.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.612+0000 7f7704c00700 1 --2- 172.21.15.32:0/958518246 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77001051e0 0x7f77001055f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f76f000b0a0 tx=0x7f76f000b3b0 comp rx=0 tx=0).stop 2024-04-30T08:30:55.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.612+0000 7f7704c00700 1 -- 172.21.15.32:0/958518246 shutdown_connections 2024-04-30T08:30:55.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.612+0000 7f7704c00700 1 --2- 172.21.15.32:0/958518246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f770010e650 0x7f7700110b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:55.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.612+0000 7f7704c00700 1 --2- 172.21.15.32:0/958518246 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f77001051e0 0x7f77001055f0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:55.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.612+0000 7f7704c00700 1 --2- 172.21.15.32:0/958518246 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7700105b30 0x7f770010e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:55.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.612+0000 7f7704c00700 1 -- 172.21.15.32:0/958518246 >> 172.21.15.32:0/958518246 conn(0x7f7700078a10 msgr2=0x7f7700078e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:30:55.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.612+0000 7f7704c00700 1 -- 172.21.15.32:0/958518246 shutdown_connections 2024-04-30T08:30:55.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.612+0000 7f7704c00700 1 -- 172.21.15.32:0/958518246 wait complete. 2024-04-30T08:30:55.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.613+0000 7f7704c00700 1 Processor -- start 2024-04-30T08:30:55.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.614+0000 7f7704c00700 1 -- start start 2024-04-30T08:30:55.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.614+0000 7f7704c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77001051e0 0x7f77001a0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:30:55.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.614+0000 7f7704c00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7700105b30 0x7f77001a0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:30:55.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.614+0000 7f7704c00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f770010e650 0x7f770019a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:30:55.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.614+0000 7f7704c00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f77001141f0 con 0x7f7700105b30 2024-04-30T08:30:55.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.614+0000 7f7704c00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f7700114070 con 0x7f770010e650 2024-04-30T08:30:55.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.614+0000 7f7704c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7700114370 con 0x7f77001051e0 2024-04-30T08:30:55.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.614+0000 7f76ffe00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f770010e650 0x7f770019a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:30:55.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.614+0000 7f76f6a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7700105b30 0x7f77001a0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:30:55.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.614+0000 7f76ffe00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f770010e650 0x7f770019a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:54266/0 (socket says 172.21.15.32:54266) 2024-04-30T08:30:55.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.614+0000 7f76ffe00700 1 -- 172.21.15.32:0/2787863747 learned_addr learned my addr 172.21.15.32:0/2787863747 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:30:55.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.614+0000 7f76ff400700 1 --2- 172.21.15.32:0/2787863747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77001051e0 0x7f77001a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:30:55.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.615+0000 7f76ffe00700 1 -- 172.21.15.32:0/2787863747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77001051e0 msgr2=0x7f77001a0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:30:55.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.615+0000 7f76ffe00700 1 --2- 172.21.15.32:0/2787863747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77001051e0 0x7f77001a0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:55.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.615+0000 7f76ffe00700 1 -- 172.21.15.32:0/2787863747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7700105b30 msgr2=0x7f77001a0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:30:55.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.615+0000 7f76ffe00700 1 --2- 172.21.15.32:0/2787863747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7700105b30 0x7f77001a0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:55.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.615+0000 7f76ffe00700 1 -- 172.21.15.32:0/2787863747 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f770019b060 con 0x7f770010e650 2024-04-30T08:30:55.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.615+0000 7f76ffe00700 1 --2- 172.21.15.32:0/2787863747 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f770010e650 0x7f770019a9b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f76f800e940 tx=0x7f76f800ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:30:55.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.615+0000 7f76fcc00700 1 -- 172.21.15.32:0/2787863747 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f800cbd0 con 0x7f770010e650 2024-04-30T08:30:55.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.615+0000 7f76fcc00700 1 -- 172.21.15.32:0/2787863747 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76f800ee90 con 0x7f770010e650 2024-04-30T08:30:55.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.615+0000 7f76fcc00700 1 -- 172.21.15.32:0/2787863747 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76f8005590 con 0x7f770010e650 2024-04-30T08:30:55.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.615+0000 7f7704c00700 1 -- 172.21.15.32:0/2787863747 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f770019b350 con 0x7f770010e650 2024-04-30T08:30:55.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.616+0000 7f7704c00700 1 -- 172.21.15.32:0/2787863747 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7700102e00 con 0x7f770010e650 2024-04-30T08:30:55.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.617+0000 7f7704c00700 1 -- 172.21.15.32:0/2787863747 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f770006be30 con 0x7f770010e650 2024-04-30T08:30:55.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.619+0000 7f76fcc00700 1 -- 172.21.15.32:0/2787863747 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f76f8004750 con 0x7f770010e650 2024-04-30T08:30:55.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.620+0000 7f76fcc00700 1 --2- 172.21.15.32:0/2787863747 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f76e40afb80 0x7f76e40b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:30:55.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.621+0000 7f76fcc00700 1 -- 172.21.15.32:0/2787863747 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f76f80dea20 con 0x7f770010e650 2024-04-30T08:30:55.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.621+0000 7f76ff400700 1 --2- 172.21.15.32:0/2787863747 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f76e40afb80 0x7f76e40b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:30:55.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.621+0000 7f76ff400700 1 --2- 172.21.15.32:0/2787863747 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f76e40afb80 0x7f76e40b2090 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f76f0009fd0 tx=0x7f76f0006210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:30:55.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.625+0000 7f76fcc00700 1 -- 172.21.15.32:0/2787863747 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f76f80a7a00 con 0x7f770010e650 2024-04-30T08:30:55.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.860+0000 7f7704c00700 1 -- 172.21.15.32:0/2787863747 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f770019bee0 con 0x7f770010e650 2024-04-30T08:30:55.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.861+0000 7f76fcc00700 1 -- 172.21.15.32:0/2787863747 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) ==== 74+0+18381 (secure 0 0 0) 0x7f76f80a7300 con 0x7f770010e650 2024-04-30T08:30:55.863 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:30:55.864 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":72,"fsid":"98c0a670-06ca-11ef-bc93-c7b262605968","created":"2024-04-30T08:22:11.122875+0000","modified":"2024-04-30T08:30:51.365145+0000","last_up_change":"2024-04-30T08:30:50.350439+0000","last_in_change":"2024-04-30T08:30:30.210620+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-30T08:26:19.728331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a7e060c8-1cbb-418f-9a03-06693f6b3f52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6803","nonce":2624700402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6805","nonce":2624700402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6809","nonce":2624700402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6807","nonce":2624700402}]},"public_addr":"172.21.15.32:6803/2624700402","cluster_addr":"172.21.15.32:6805/2624700402","heartbeat_back_addr":"172.21.15.32:6809/2624700402","heartbeat_front_addr":"172.21.15.32:6807/2624700402","state":["exists","up"]},{"osd":1,"uuid":"72ef2fb1-e5ca-48ef-aa3a-4796a64b922f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6801","nonce":444649711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6803","nonce":444649711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6807","nonce":444649711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6805","nonce":444649711}]},"public_addr":"172.21.15.71:6801/444649711","cluster_addr":"172.21.15.71:6803/444649711","heartbeat_back_addr":"172.21.15.71:6807/444649711","heartbeat_front_addr":"172.21.15.71:6805/444649711","state":["exists","up"]},{"osd":2,"uuid":"704e5e8a-935e-47c7-ae28-27119876ac09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6801","nonce":3255528671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6803","nonce":3255528671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6807","nonce":3255528671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6805","nonce":3255528671}]},"public_addr":"172.21.15.189:6801/3255528671","cluster_addr":"172.21.15.189:6803/3255528671","heartbeat_back_addr":"172.21.15.189:6807/3255528671","heartbeat_front_addr":"172.21.15.189:6805/3255528671","state":["exists","up"]},{"osd":3,"uuid":"575004f7-6559-4a62-863e-e33078b0eebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6811","nonce":1384702112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6813","nonce":1384702112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6817","nonce":1384702112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6815","nonce":1384702112}]},"public_addr":"172.21.15.32:6811/1384702112","cluster_addr":"172.21.15.32:6813/1384702112","heartbeat_back_addr":"172.21.15.32:6817/1384702112","heartbeat_front_addr":"172.21.15.32:6815/1384702112","state":["exists","up"]},{"osd":4,"uuid":"e1c1e894-074d-4c78-89ef-9e5a114268bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6809","nonce":4250534574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6811","nonce":4250534574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6815","nonce":4250534574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6813","nonce":4250534574}]},"public_addr":"172.21.15.71:6809/4250534574","cluster_addr":"172.21.15.71:6811/4250534574","heartbeat_back_addr":"172.21.15.71:6815/4250534574","heartbeat_front_addr":"172.21.15.71:6813/4250534574","state":["exists","up"]},{"osd":5,"uuid":"12c022d4-fac6-40c4-ae7a-691571bee5d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6809","nonce":4245346198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6811","nonce":4245346198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6815","nonce":4245346198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6813","nonce":4245346198}]},"public_addr":"172.21.15.189:6809/4245346198","cluster_addr":"172.21.15.189:6811/4245346198","heartbeat_back_addr":"172.21.15.189:6815/4245346198","heartbeat_front_addr":"172.21.15.189:6813/4245346198","state":["exists","up"]},{"osd":6,"uuid":"b413f8ee-3d4a-415a-a2ba-d68437b7330d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6819","nonce":1910389190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6821","nonce":1910389190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6825","nonce":1910389190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6823","nonce":1910389190}]},"public_addr":"172.21.15.32:6819/1910389190","cluster_addr":"172.21.15.32:6821/1910389190","heartbeat_back_addr":"172.21.15.32:6825/1910389190","heartbeat_front_addr":"172.21.15.32:6823/1910389190","state":["exists","up"]},{"osd":7,"uuid":"aac64fb3-fc13-4126-b842-7b06ad5ce238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6817","nonce":692607380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6819","nonce":692607380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6823","nonce":692607380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6821","nonce":692607380}]},"public_addr":"172.21.15.71:6817/692607380","cluster_addr":"172.21.15.71:6819/692607380","heartbeat_back_addr":"172.21.15.71:6823/692607380","heartbeat_front_addr":"172.21.15.71:6821/692607380","state":["exists","up"]},{"osd":8,"uuid":"cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6817","nonce":3973697069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6819","nonce":3973697069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6823","nonce":3973697069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6821","nonce":3973697069}]},"public_addr":"172.21.15.189:6817/3973697069","cluster_addr":"172.21.15.189:6819/3973697069","heartbeat_back_addr":"172.21.15.189:6823/3973697069","heartbeat_front_addr":"172.21.15.189:6821/3973697069","state":["exists","up"]},{"osd":9,"uuid":"ef4a1645-ae6e-45c3-82ad-161510e62aa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6827","nonce":46066685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6829","nonce":46066685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6833","nonce":46066685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6831","nonce":46066685}]},"public_addr":"172.21.15.32:6827/46066685","cluster_addr":"172.21.15.32:6829/46066685","heartbeat_back_addr":"172.21.15.32:6833/46066685","heartbeat_front_addr":"172.21.15.32:6831/46066685","state":["exists","up"]},{"osd":10,"uuid":"25b26a38-a644-409f-b941-59efdf04a3ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6825","nonce":1886290293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6827","nonce":1886290293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6831","nonce":1886290293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6829","nonce":1886290293}]},"public_addr":"172.21.15.71:6825/1886290293","cluster_addr":"172.21.15.71:6827/1886290293","heartbeat_back_addr":"172.21.15.71:6831/1886290293","heartbeat_front_addr":"172.21.15.71:6829/1886290293","state":["exists","up"]},{"osd":11,"uuid":"ab96b3ff-3688-455b-af64-60226c5bfbff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6825","nonce":850185001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6827","nonce":850185001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6831","nonce":850185001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6829","nonce":850185001}]},"public_addr":"172.21.15.189:6825/850185001","cluster_addr":"172.21.15.189:6827/850185001","heartbeat_back_addr":"172.21.15.189:6831/850185001","heartbeat_front_addr":"172.21.15.189:6829/850185001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:27.220308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:53.230783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:17.190642+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:52.603799+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:20.568671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:46.158363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:22.706933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:50.665173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:16.602410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:53.814698+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:22.435189+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:48.651949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1266897222":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/2915689533":"2024-05-01T08:23:29.526664+0000","172.21.15.32:6801/4019351306":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/425836193":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/1134075722":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6801/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/21059253":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/3245550894":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/4019351306":"2024-05-01T08:23:29.526664+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-30T08:30:55.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.864+0000 7f7704c00700 1 -- 172.21.15.32:0/2787863747 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f76e40afb80 msgr2=0x7f76e40b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:30:55.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.864+0000 7f7704c00700 1 --2- 172.21.15.32:0/2787863747 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f76e40afb80 0x7f76e40b2090 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f76f0009fd0 tx=0x7f76f0006210 comp rx=0 tx=0).stop 2024-04-30T08:30:55.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.864+0000 7f7704c00700 1 -- 172.21.15.32:0/2787863747 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f770010e650 msgr2=0x7f770019a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:30:55.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.864+0000 7f7704c00700 1 --2- 172.21.15.32:0/2787863747 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f770010e650 0x7f770019a9b0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f76f800e940 tx=0x7f76f800ec50 comp rx=0 tx=0).stop 2024-04-30T08:30:55.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.864+0000 7f7704c00700 1 -- 172.21.15.32:0/2787863747 shutdown_connections 2024-04-30T08:30:55.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.864+0000 7f7704c00700 1 --2- 172.21.15.32:0/2787863747 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f76e40afb80 0x7f76e40b2090 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:55.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.864+0000 7f7704c00700 1 --2- 172.21.15.32:0/2787863747 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f770010e650 0x7f770019a9b0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:55.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.864+0000 7f7704c00700 1 --2- 172.21.15.32:0/2787863747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f77001051e0 0x7f77001a0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:55.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.864+0000 7f7704c00700 1 --2- 172.21.15.32:0/2787863747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7700105b30 0x7f77001a0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:55.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.864+0000 7f7704c00700 1 -- 172.21.15.32:0/2787863747 >> 172.21.15.32:0/2787863747 conn(0x7f7700078a10 msgr2=0x7f77000ffe10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:30:55.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.865+0000 7f7704c00700 1 -- 172.21.15.32:0/2787863747 shutdown_connections 2024-04-30T08:30:55.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:55.865+0000 7f7704c00700 1 -- 172.21.15.32:0/2787863747 wait complete. 2024-04-30T08:30:56.816 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-30T08:26:19.728331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-30T08:30:56.816 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-30T08:30:57.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:56 smithi071 ceph-mon[33385]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:30:57.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:56 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2787863747' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:30:57.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:56 smithi189 ceph-mon[33038]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:30:57.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:56 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2787863747' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:30:57.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:56 smithi032 ceph-mon[28331]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:30:57.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:56 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2787863747' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:30:57.268 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:30:58.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:30:58 smithi032 ceph-mon[28331]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:30:59.006 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:30:58 smithi071 ceph-mon[33385]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:30:59.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:30:58 smithi189 ceph-mon[33038]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:30:59.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.512+0000 7fa3dc800700 1 -- 172.21.15.32:0/107030280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3d4076400 msgr2=0x7fa3d4114d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:30:59.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.512+0000 7fa3dc800700 1 --2- 172.21.15.32:0/107030280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3d4076400 0x7fa3d4114d80 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fa3c4009990 tx=0x7fa3c4009ca0 comp rx=0 tx=0).stop 2024-04-30T08:30:59.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.512+0000 7fa3dc800700 1 -- 172.21.15.32:0/107030280 shutdown_connections 2024-04-30T08:30:59.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.512+0000 7fa3dc800700 1 --2- 172.21.15.32:0/107030280 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3d41152c0 0x7fa3d41176e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:59.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.512+0000 7fa3dc800700 1 --2- 172.21.15.32:0/107030280 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3d4075ab0 0x7fa3d4075ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:59.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.512+0000 7fa3dc800700 1 --2- 172.21.15.32:0/107030280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3d4076400 0x7fa3d4114d80 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:59.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.512+0000 7fa3dc800700 1 -- 172.21.15.32:0/107030280 >> 172.21.15.32:0/107030280 conn(0x7fa3d40fff90 msgr2=0x7fa3d41023b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:30:59.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.512+0000 7fa3dc800700 1 -- 172.21.15.32:0/107030280 shutdown_connections 2024-04-30T08:30:59.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.512+0000 7fa3dc800700 1 -- 172.21.15.32:0/107030280 wait complete. 2024-04-30T08:30:59.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.514+0000 7fa3dc800700 1 Processor -- start 2024-04-30T08:30:59.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.514+0000 7fa3dc800700 1 -- start start 2024-04-30T08:30:59.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.514+0000 7fa3dc800700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3d4075ab0 0x7fa3d41a13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:30:59.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.514+0000 7fa3dc800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3d4076400 0x7fa3d41a1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:30:59.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.514+0000 7fa3dc800700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3d41152c0 0x7fa3d41a5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:30:59.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.514+0000 7fa3dc800700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d411aa10 con 0x7fa3d41152c0 2024-04-30T08:30:59.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.514+0000 7fa3dc800700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d411a890 con 0x7fa3d4075ab0 2024-04-30T08:30:59.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.514+0000 7fa3dc800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa3d411ab90 con 0x7fa3d4076400 2024-04-30T08:30:59.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.515+0000 7fa3d9e00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3d4075ab0 0x7fa3d41a13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:30:59.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.515+0000 7fa3da800700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3d41152c0 0x7fa3d41a5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:30:59.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.515+0000 7fa3d9e00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3d4075ab0 0x7fa3d41a13c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:43568/0 (socket says 172.21.15.32:43568) 2024-04-30T08:30:59.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.515+0000 7fa3d9e00700 1 -- 172.21.15.32:0/3635290347 learned_addr learned my addr 172.21.15.32:0/3635290347 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:30:59.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.515+0000 7fa3d9400700 1 --2- 172.21.15.32:0/3635290347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3d4076400 0x7fa3d41a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:30:59.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.515+0000 7fa3d9e00700 1 -- 172.21.15.32:0/3635290347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3d4076400 msgr2=0x7fa3d41a1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:30:59.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.515+0000 7fa3d9e00700 1 --2- 172.21.15.32:0/3635290347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3d4076400 0x7fa3d41a1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:59.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.515+0000 7fa3d9e00700 1 -- 172.21.15.32:0/3635290347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3d41152c0 msgr2=0x7fa3d41a5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:30:59.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.515+0000 7fa3d9e00700 1 --2- 172.21.15.32:0/3635290347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3d41152c0 0x7fa3d41a5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:59.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.515+0000 7fa3d9e00700 1 -- 172.21.15.32:0/3635290347 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa3d41a6290 con 0x7fa3d4075ab0 2024-04-30T08:30:59.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.515+0000 7fa3d9e00700 1 --2- 172.21.15.32:0/3635290347 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3d4075ab0 0x7fa3d41a13c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa3c800e9a0 tx=0x7fa3c800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:30:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.516+0000 7fa3cea00700 1 -- 172.21.15.32:0/3635290347 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c800ca10 con 0x7fa3d4075ab0 2024-04-30T08:30:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.516+0000 7fa3cea00700 1 -- 172.21.15.32:0/3635290347 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3c800cbb0 con 0x7fa3d4075ab0 2024-04-30T08:30:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.516+0000 7fa3cea00700 1 -- 172.21.15.32:0/3635290347 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8018680 con 0x7fa3d4075ab0 2024-04-30T08:30:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.516+0000 7fa3dc800700 1 -- 172.21.15.32:0/3635290347 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa3d41a6580 con 0x7fa3d4075ab0 2024-04-30T08:30:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.516+0000 7fa3dc800700 1 -- 172.21.15.32:0/3635290347 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3d41adea0 con 0x7fa3d4075ab0 2024-04-30T08:30:59.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.517+0000 7fa3dc800700 1 -- 172.21.15.32:0/3635290347 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa3d40771f0 con 0x7fa3d4075ab0 2024-04-30T08:30:59.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.519+0000 7fa3cea00700 1 -- 172.21.15.32:0/3635290347 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fa3c8010430 con 0x7fa3d4075ab0 2024-04-30T08:30:59.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.521+0000 7fa3cea00700 1 --2- 172.21.15.32:0/3635290347 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa3bc0afb80 0x7fa3bc0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:30:59.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.521+0000 7fa3cea00700 1 -- 172.21.15.32:0/3635290347 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fa3c80de980 con 0x7fa3d4075ab0 2024-04-30T08:30:59.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.521+0000 7fa3d9400700 1 --2- 172.21.15.32:0/3635290347 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa3bc0afb80 0x7fa3bc0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:30:59.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.522+0000 7fa3d9400700 1 --2- 172.21.15.32:0/3635290347 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa3bc0afb80 0x7fa3bc0b2090 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa3d4103440 tx=0x7fa3c400b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:30:59.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.526+0000 7fa3cea00700 1 -- 172.21.15.32:0/3635290347 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa3c80a7960 con 0x7fa3d4075ab0 2024-04-30T08:30:59.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.763+0000 7fa3dc800700 1 -- 172.21.15.32:0/3635290347 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa3d402e590 con 0x7fa3d4075ab0 2024-04-30T08:30:59.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.764+0000 7fa3cea00700 1 -- 172.21.15.32:0/3635290347 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) ==== 93+0+10 (secure 0 0 0) 0x7fa3c80a7260 con 0x7fa3d4075ab0 2024-04-30T08:30:59.766 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-30T08:30:59.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.768+0000 7fa3dc800700 1 -- 172.21.15.32:0/3635290347 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa3bc0afb80 msgr2=0x7fa3bc0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:30:59.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.768+0000 7fa3dc800700 1 --2- 172.21.15.32:0/3635290347 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa3bc0afb80 0x7fa3bc0b2090 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fa3d4103440 tx=0x7fa3c400b260 comp rx=0 tx=0).stop 2024-04-30T08:30:59.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.768+0000 7fa3dc800700 1 -- 172.21.15.32:0/3635290347 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3d4075ab0 msgr2=0x7fa3d41a13c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:30:59.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.768+0000 7fa3dc800700 1 --2- 172.21.15.32:0/3635290347 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3d4075ab0 0x7fa3d41a13c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa3c800e9a0 tx=0x7fa3c800ecb0 comp rx=0 tx=0).stop 2024-04-30T08:30:59.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.769+0000 7fa3dc800700 1 -- 172.21.15.32:0/3635290347 shutdown_connections 2024-04-30T08:30:59.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.769+0000 7fa3dc800700 1 --2- 172.21.15.32:0/3635290347 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa3bc0afb80 0x7fa3bc0b2090 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:59.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.769+0000 7fa3dc800700 1 --2- 172.21.15.32:0/3635290347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa3d41152c0 0x7fa3d41a5bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:59.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.769+0000 7fa3dc800700 1 --2- 172.21.15.32:0/3635290347 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa3d4075ab0 0x7fa3d41a13c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:59.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.769+0000 7fa3dc800700 1 --2- 172.21.15.32:0/3635290347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3d4076400 0x7fa3d41a1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:30:59.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.769+0000 7fa3dc800700 1 -- 172.21.15.32:0/3635290347 >> 172.21.15.32:0/3635290347 conn(0x7fa3d40fff90 msgr2=0x7fa3d41005e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:30:59.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.769+0000 7fa3dc800700 1 -- 172.21.15.32:0/3635290347 shutdown_connections 2024-04-30T08:30:59.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:30:59.769+0000 7fa3dc800700 1 -- 172.21.15.32:0/3635290347 wait complete. 2024-04-30T08:31:00.718 INFO:tasks.cephadm:Adding mds.a on smithi032 2024-04-30T08:31:00.718 INFO:tasks.cephadm:Adding mds.d on smithi032 2024-04-30T08:31:00.718 INFO:tasks.cephadm:Adding mds.g on smithi032 2024-04-30T08:31:00.718 INFO:tasks.cephadm:Adding mds.j on smithi032 2024-04-30T08:31:00.718 INFO:tasks.cephadm:Adding mds.b on smithi071 2024-04-30T08:31:00.718 INFO:tasks.cephadm:Adding mds.e on smithi071 2024-04-30T08:31:00.718 INFO:tasks.cephadm:Adding mds.h on smithi071 2024-04-30T08:31:00.718 INFO:tasks.cephadm:Adding mds.k on smithi071 2024-04-30T08:31:00.718 INFO:tasks.cephadm:Adding mds.c on smithi189 2024-04-30T08:31:00.719 INFO:tasks.cephadm:Adding mds.f on smithi189 2024-04-30T08:31:00.719 INFO:tasks.cephadm:Adding mds.i on smithi189 2024-04-30T08:31:00.719 INFO:tasks.cephadm:Adding mds.l on smithi189 2024-04-30T08:31:00.719 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi032=a;smithi032=d;smithi032=g;smithi032=j;smithi071=b;smithi071=e;smithi071=h;smithi071=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l' 2024-04-30T08:31:00.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:00 smithi032 ceph-mon[28331]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:00.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:00 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3635290347' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:31:00.966 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.c/config 2024-04-30T08:31:01.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:00 smithi071 ceph-mon[33385]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:01.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:00 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3635290347' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:31:01.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:00 smithi189 ceph-mon[33038]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:01.052 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:00 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3635290347' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:31:02.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.796+0000 7fb6f0df1700 1 -- 172.21.15.189:0/3257600693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6ec103a30 msgr2=0x7fb6ec103e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:02.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.796+0000 7fb6f0df1700 1 --2- 172.21.15.189:0/3257600693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6ec103a30 0x7fb6ec103e40 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb6d4009990 tx=0x7fb6d4009ca0 comp rx=0 tx=0).stop 2024-04-30T08:31:02.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.796+0000 7fb6f0df1700 1 -- 172.21.15.189:0/3257600693 shutdown_connections 2024-04-30T08:31:02.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.796+0000 7fb6f0df1700 1 --2- 172.21.15.189:0/3257600693 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6ec1043d0 0x7fb6ec10ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:02.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.796+0000 7fb6f0df1700 1 --2- 172.21.15.189:0/3257600693 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6ec102900 0x7fb6ec102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:02.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.796+0000 7fb6f0df1700 1 --2- 172.21.15.189:0/3257600693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6ec103a30 0x7fb6ec103e40 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:02.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.796+0000 7fb6f0df1700 1 -- 172.21.15.189:0/3257600693 >> 172.21.15.189:0/3257600693 conn(0x7fb6ec0fdf60 msgr2=0x7fb6ec1003f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:02.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.796+0000 7fb6f0df1700 1 -- 172.21.15.189:0/3257600693 shutdown_connections 2024-04-30T08:31:02.800 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.796+0000 7fb6f0df1700 1 -- 172.21.15.189:0/3257600693 wait complete. 2024-04-30T08:31:02.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.797+0000 7fb6f0df1700 1 Processor -- start 2024-04-30T08:31:02.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.797+0000 7fb6f0df1700 1 -- start start 2024-04-30T08:31:02.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.797+0000 7fb6f0df1700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6ec102900 0x7fb6ec19a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:02.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.797+0000 7fb6f0df1700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6ec103a30 0x7fb6ec19ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:02.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.797+0000 7fb6f0df1700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6ec1043d0 0x7fb6ec1a1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:02.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.797+0000 7fb6f0df1700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ec10e3e0 con 0x7fb6ec103a30 2024-04-30T08:31:02.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.797+0000 7fb6f0df1700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ec10e260 con 0x7fb6ec1043d0 2024-04-30T08:31:02.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.797+0000 7fb6f0df1700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb6ec10e560 con 0x7fb6ec102900 2024-04-30T08:31:02.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.797+0000 7fb6ea59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6ec102900 0x7fb6ec19a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:02.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.798+0000 7fb6ea59c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6ec102900 0x7fb6ec19a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:51864/0 (socket says 172.21.15.189:51864) 2024-04-30T08:31:02.801 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.798+0000 7fb6ea59c700 1 -- 172.21.15.189:0/4188049583 learned_addr learned my addr 172.21.15.189:0/4188049583 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-04-30T08:31:02.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.798+0000 7fb6ead9d700 1 --2- 172.21.15.189:0/4188049583 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6ec1043d0 0x7fb6ec1a1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:02.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.798+0000 7fb6ea59c700 1 -- 172.21.15.189:0/4188049583 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6ec1043d0 msgr2=0x7fb6ec1a1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:02.802 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.798+0000 7fb6ea59c700 1 --2- 172.21.15.189:0/4188049583 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6ec1043d0 0x7fb6ec1a1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:02.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.798+0000 7fb6ea59c700 1 -- 172.21.15.189:0/4188049583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6ec103a30 msgr2=0x7fb6ec19ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:02.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.798+0000 7fb6ea59c700 1 --2- 172.21.15.189:0/4188049583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6ec103a30 0x7fb6ec19ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:02.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.798+0000 7fb6ea59c700 1 -- 172.21.15.189:0/4188049583 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6ec1a2430 con 0x7fb6ec102900 2024-04-30T08:31:02.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.798+0000 7fb6ea59c700 1 --2- 172.21.15.189:0/4188049583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6ec102900 0x7fb6ec19a8d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc00c970 tx=0x7fb6dc00cd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:02.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.798+0000 7fb6e3fff700 1 -- 172.21.15.189:0/4188049583 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc004220 con 0x7fb6ec102900 2024-04-30T08:31:02.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.798+0000 7fb6e3fff700 1 -- 172.21.15.189:0/4188049583 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6dc007db0 con 0x7fb6ec102900 2024-04-30T08:31:02.803 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.798+0000 7fb6f0df1700 1 -- 172.21.15.189:0/4188049583 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6ec1a2720 con 0x7fb6ec102900 2024-04-30T08:31:02.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.798+0000 7fb6e3fff700 1 -- 172.21.15.189:0/4188049583 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6dc003d00 con 0x7fb6ec102900 2024-04-30T08:31:02.804 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.798+0000 7fb6f0df1700 1 -- 172.21.15.189:0/4188049583 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6ec1a2ce0 con 0x7fb6ec102900 2024-04-30T08:31:02.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.799+0000 7fb6f0df1700 1 -- 172.21.15.189:0/4188049583 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6ec051210 con 0x7fb6ec102900 2024-04-30T08:31:02.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.801+0000 7fb6e3fff700 1 -- 172.21.15.189:0/4188049583 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fb6dc007900 con 0x7fb6ec102900 2024-04-30T08:31:02.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.801+0000 7fb6e3fff700 1 --2- 172.21.15.189:0/4188049583 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb6d80afb80 0x7fb6d80b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:02.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.801+0000 7fb6e3fff700 1 -- 172.21.15.189:0/4188049583 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fb6dc0de7b0 con 0x7fb6ec102900 2024-04-30T08:31:02.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.802+0000 7fb6e1d9b700 1 --2- 172.21.15.189:0/4188049583 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb6d80afb80 0x7fb6d80b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:02.805 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.803+0000 7fb6e1d9b700 1 --2- 172.21.15.189:0/4188049583 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb6d80afb80 0x7fb6d80b2090 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb6d400b5c0 tx=0x7fb6d4003860 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:02.806 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:02.805+0000 7fb6e3fff700 1 -- 172.21.15.189:0/4188049583 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb6dc014030 con 0x7fb6ec102900 2024-04-30T08:31:03.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:02 smithi071 ceph-mon[33385]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:03.015 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:03.014+0000 7fb6f0df1700 1 -- 172.21.15.189:0/4188049583 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi032=a;smithi032=d;smithi032=g;smithi032=j;smithi071=b;smithi071=e;smithi071=h;smithi071=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l", "target": ["mon-mgr", ""]}) -- 0x7fb6ec109e80 con 0x7fb6d80afb80 2024-04-30T08:31:03.015 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:02 smithi189 ceph-mon[33038]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:03.033 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:03.032+0000 7fb6e3fff700 1 -- 172.21.15.189:0/4188049583 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fb6ec109e80 con 0x7fb6d80afb80 2024-04-30T08:31:03.034 INFO:teuthology.orchestra.run.smithi189.stdout:Scheduled mds.all update... 2024-04-30T08:31:03.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:03.035+0000 7fb6f0df1700 1 -- 172.21.15.189:0/4188049583 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb6d80afb80 msgr2=0x7fb6d80b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:03.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:03.035+0000 7fb6f0df1700 1 --2- 172.21.15.189:0/4188049583 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb6d80afb80 0x7fb6d80b2090 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fb6d400b5c0 tx=0x7fb6d4003860 comp rx=0 tx=0).stop 2024-04-30T08:31:03.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:03.035+0000 7fb6f0df1700 1 -- 172.21.15.189:0/4188049583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6ec102900 msgr2=0x7fb6ec19a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:03.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:03.035+0000 7fb6f0df1700 1 --2- 172.21.15.189:0/4188049583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6ec102900 0x7fb6ec19a8d0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb6dc00c970 tx=0x7fb6dc00cd30 comp rx=0 tx=0).stop 2024-04-30T08:31:03.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:03.035+0000 7fb6f0df1700 1 -- 172.21.15.189:0/4188049583 shutdown_connections 2024-04-30T08:31:03.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:03.035+0000 7fb6f0df1700 1 --2- 172.21.15.189:0/4188049583 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb6d80afb80 0x7fb6d80b2090 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:03.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:03.035+0000 7fb6f0df1700 1 --2- 172.21.15.189:0/4188049583 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb6ec1043d0 0x7fb6ec1a1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:03.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:03.035+0000 7fb6f0df1700 1 --2- 172.21.15.189:0/4188049583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6ec102900 0x7fb6ec19a8d0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:03.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:03.035+0000 7fb6f0df1700 1 --2- 172.21.15.189:0/4188049583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6ec103a30 0x7fb6ec19ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:03.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:03.035+0000 7fb6f0df1700 1 -- 172.21.15.189:0/4188049583 >> 172.21.15.189:0/4188049583 conn(0x7fb6ec0fdf60 msgr2=0x7fb6ec101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:03.036 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:03.035+0000 7fb6f0df1700 1 -- 172.21.15.189:0/4188049583 shutdown_connections 2024-04-30T08:31:03.037 INFO:teuthology.orchestra.run.smithi189.stderr:2024-04-30T08:31:03.035+0000 7fb6f0df1700 1 -- 172.21.15.189:0/4188049583 wait complete. 2024-04-30T08:31:03.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:02 smithi032 ceph-mon[28331]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:03.769 DEBUG:teuthology.orchestra.run.smithi032:mds.a> sudo journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.a.service 2024-04-30T08:31:03.776 DEBUG:teuthology.orchestra.run.smithi032:mds.d> sudo journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.d.service 2024-04-30T08:31:03.820 DEBUG:teuthology.orchestra.run.smithi032:mds.g> sudo journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.g.service 2024-04-30T08:31:03.867 DEBUG:teuthology.orchestra.run.smithi032:mds.j> sudo journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.j.service 2024-04-30T08:31:03.914 DEBUG:teuthology.orchestra.run.smithi071:mds.b> sudo journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.b.service 2024-04-30T08:31:03.917 DEBUG:teuthology.orchestra.run.smithi071:mds.e> sudo journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.e.service 2024-04-30T08:31:03.960 DEBUG:teuthology.orchestra.run.smithi071:mds.h> sudo journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.h.service 2024-04-30T08:31:04.003 DEBUG:teuthology.orchestra.run.smithi071:mds.k> sudo journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.k.service 2024-04-30T08:31:04.046 DEBUG:teuthology.orchestra.run.smithi189:mds.c> sudo journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.c.service 2024-04-30T08:31:04.048 DEBUG:teuthology.orchestra.run.smithi189:mds.f> sudo journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.f.service 2024-04-30T08:31:04.091 DEBUG:teuthology.orchestra.run.smithi189:mds.i> sudo journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.i.service 2024-04-30T08:31:04.137 DEBUG:teuthology.orchestra.run.smithi189:mds.l> sudo journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.l.service 2024-04-30T08:31:04.180 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-30T08:31:04.181 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-30T08:31:04.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:04 smithi032 ceph-mon[28331]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi032=a;smithi032=d;smithi032=g;smithi032=j;smithi071=b;smithi071=e;smithi071=h;smithi071=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:31:04.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:04 smithi032 ceph-mon[28331]: Saving service mds.all spec with placement smithi032=a;smithi032=d;smithi032=g;smithi032=j;smithi071=b;smithi071=e;smithi071=h;smithi071=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l;count:12 2024-04-30T08:31:04.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:04 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:04.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:04 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:31:04.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:04 smithi071 ceph-mon[33385]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi032=a;smithi032=d;smithi032=g;smithi032=j;smithi071=b;smithi071=e;smithi071=h;smithi071=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:31:04.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:04 smithi071 ceph-mon[33385]: Saving service mds.all spec with placement smithi032=a;smithi032=d;smithi032=g;smithi032=j;smithi071=b;smithi071=e;smithi071=h;smithi071=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l;count:12 2024-04-30T08:31:04.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:04 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:04.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:04 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:31:04.298 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:04 smithi189 ceph-mon[33038]: from='client.24418 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi032=a;smithi032=d;smithi032=g;smithi032=j;smithi071=b;smithi071=e;smithi071=h;smithi071=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:31:04.298 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:04 smithi189 ceph-mon[33038]: Saving service mds.all spec with placement smithi032=a;smithi032=d;smithi032=g;smithi032=j;smithi071=b;smithi071=e;smithi071=h;smithi071=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l;count:12 2024-04-30T08:31:04.298 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:04 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:04.298 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:04 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:31:04.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.475+0000 7f807a000640 1 -- 172.21.15.32:0/2726023626 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8074110120 msgr2=0x7f8074110530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:04.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.475+0000 7f807a000640 1 --2- 172.21.15.32:0/2726023626 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8074110120 0x7f8074110530 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f806c009be0 tx=0x7f806c02a6c0 comp rx=0 tx=0).stop 2024-04-30T08:31:04.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.475+0000 7f807a000640 1 -- 172.21.15.32:0/2726023626 shutdown_connections 2024-04-30T08:31:04.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.475+0000 7f807a000640 1 --2- 172.21.15.32:0/2726023626 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8074110120 0x7f8074110530 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:04.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.475+0000 7f807a000640 1 --2- 172.21.15.32:0/2726023626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f807406c290 0x7f807410b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:04.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.475+0000 7f807a000640 1 --2- 172.21.15.32:0/2726023626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f807406bdf0 0x7f807410adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:04.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.475+0000 7f807a000640 1 -- 172.21.15.32:0/2726023626 >> 172.21.15.32:0/2726023626 conn(0x7f8074071670 msgr2=0x7f8074073b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:04.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.475+0000 7f807a000640 1 -- 172.21.15.32:0/2726023626 shutdown_connections 2024-04-30T08:31:04.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.475+0000 7f807a000640 1 -- 172.21.15.32:0/2726023626 wait complete. 2024-04-30T08:31:04.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.476+0000 7f807a000640 1 Processor -- start 2024-04-30T08:31:04.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.476+0000 7f807a000640 1 -- start start 2024-04-30T08:31:04.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.477+0000 7f807a000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f807406bdf0 0x7f80741a8c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:04.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.477+0000 7f807a000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8074110120 0x7f80741a9150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:04.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.477+0000 7f807a000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80741ae0d0 0x7f80741b04f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:04.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.477+0000 7f807a000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f807411e4b0 con 0x7f8074110120 2024-04-30T08:31:04.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.477+0000 7f807a000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f807411e330 con 0x7f807406bdf0 2024-04-30T08:31:04.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.477+0000 7f807a000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f807411e630 con 0x7f80741ae0d0 2024-04-30T08:31:04.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.477+0000 7f8073400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f807406bdf0 0x7f80741a8c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:04.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.477+0000 7f8073400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f807406bdf0 0x7f80741a8c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:43582/0 (socket says 172.21.15.32:43582) 2024-04-30T08:31:04.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.477+0000 7f8073400640 1 -- 172.21.15.32:0/1409883678 learned_addr learned my addr 172.21.15.32:0/1409883678 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:31:04.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.477+0000 7f8073e00640 1 --2- 172.21.15.32:0/1409883678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80741ae0d0 0x7f80741b04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:04.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.477+0000 7f8072a00640 1 --2- 172.21.15.32:0/1409883678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8074110120 0x7f80741a9150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:04.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.478+0000 7f8073400640 1 -- 172.21.15.32:0/1409883678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80741ae0d0 msgr2=0x7f80741b04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:04.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.478+0000 7f8073400640 1 --2- 172.21.15.32:0/1409883678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80741ae0d0 0x7f80741b04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:04.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.478+0000 7f8073400640 1 -- 172.21.15.32:0/1409883678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8074110120 msgr2=0x7f80741a9150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:04.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.478+0000 7f8073400640 1 --2- 172.21.15.32:0/1409883678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8074110120 0x7f80741a9150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:04.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.478+0000 7f8073400640 1 -- 172.21.15.32:0/1409883678 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80741b0b60 con 0x7f807406bdf0 2024-04-30T08:31:04.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.478+0000 7f8073e00640 1 --2- 172.21.15.32:0/1409883678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80741ae0d0 0x7f80741b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:31:04.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.478+0000 7f8073400640 1 --2- 172.21.15.32:0/1409883678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f807406bdf0 0x7f80741a8c10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f806400e970 tx=0x7f806400ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:04.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.478+0000 7f806be00640 1 -- 172.21.15.32:0/1409883678 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f806400cd80 con 0x7f807406bdf0 2024-04-30T08:31:04.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.478+0000 7f806be00640 1 -- 172.21.15.32:0/1409883678 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f80640035f0 con 0x7f807406bdf0 2024-04-30T08:31:04.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.479+0000 7f807a000640 1 -- 172.21.15.32:0/1409883678 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80741b0e50 con 0x7f807406bdf0 2024-04-30T08:31:04.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.479+0000 7f806be00640 1 -- 172.21.15.32:0/1409883678 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8064010690 con 0x7f807406bdf0 2024-04-30T08:31:04.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.479+0000 7f807a000640 1 -- 172.21.15.32:0/1409883678 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80741b12b0 con 0x7f807406bdf0 2024-04-30T08:31:04.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.480+0000 7f807a000640 1 -- 172.21.15.32:0/1409883678 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8074069dc0 con 0x7f807406bdf0 2024-04-30T08:31:04.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.482+0000 7f806be00640 1 -- 172.21.15.32:0/1409883678 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f8064003aa0 con 0x7f807406bdf0 2024-04-30T08:31:04.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.483+0000 7f806be00640 1 --2- 172.21.15.32:0/1409883678 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f80580afaf0 0x7f80580b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:04.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.483+0000 7f806be00640 1 -- 172.21.15.32:0/1409883678 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f80640dea80 con 0x7f807406bdf0 2024-04-30T08:31:04.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.484+0000 7f8072a00640 1 --2- 172.21.15.32:0/1409883678 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f80580afaf0 0x7f80580b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:04.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.484+0000 7f8072a00640 1 --2- 172.21.15.32:0/1409883678 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f80580afaf0 0x7f80580b2000 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f80600096c0 tx=0x7f8060009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:04.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.488+0000 7f806be00640 1 -- 172.21.15.32:0/1409883678 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f80640e0e20 con 0x7f807406bdf0 2024-04-30T08:31:04.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.699+0000 7f807a000640 1 -- 172.21.15.32:0/1409883678 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8074063820 con 0x7f807406bdf0 2024-04-30T08:31:04.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.701+0000 7f806be00640 1 -- 172.21.15.32:0/1409883678 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) ==== 74+0+18381 (secure 0 0 0) 0x7f80640a7a60 con 0x7f807406bdf0 2024-04-30T08:31:04.703 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:31:04.703 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":72,"fsid":"98c0a670-06ca-11ef-bc93-c7b262605968","created":"2024-04-30T08:22:11.122875+0000","modified":"2024-04-30T08:30:51.365145+0000","last_up_change":"2024-04-30T08:30:50.350439+0000","last_in_change":"2024-04-30T08:30:30.210620+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-30T08:26:19.728331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a7e060c8-1cbb-418f-9a03-06693f6b3f52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6803","nonce":2624700402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6805","nonce":2624700402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6809","nonce":2624700402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6807","nonce":2624700402}]},"public_addr":"172.21.15.32:6803/2624700402","cluster_addr":"172.21.15.32:6805/2624700402","heartbeat_back_addr":"172.21.15.32:6809/2624700402","heartbeat_front_addr":"172.21.15.32:6807/2624700402","state":["exists","up"]},{"osd":1,"uuid":"72ef2fb1-e5ca-48ef-aa3a-4796a64b922f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6801","nonce":444649711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6803","nonce":444649711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6807","nonce":444649711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6805","nonce":444649711}]},"public_addr":"172.21.15.71:6801/444649711","cluster_addr":"172.21.15.71:6803/444649711","heartbeat_back_addr":"172.21.15.71:6807/444649711","heartbeat_front_addr":"172.21.15.71:6805/444649711","state":["exists","up"]},{"osd":2,"uuid":"704e5e8a-935e-47c7-ae28-27119876ac09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6801","nonce":3255528671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6803","nonce":3255528671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6807","nonce":3255528671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6805","nonce":3255528671}]},"public_addr":"172.21.15.189:6801/3255528671","cluster_addr":"172.21.15.189:6803/3255528671","heartbeat_back_addr":"172.21.15.189:6807/3255528671","heartbeat_front_addr":"172.21.15.189:6805/3255528671","state":["exists","up"]},{"osd":3,"uuid":"575004f7-6559-4a62-863e-e33078b0eebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6811","nonce":1384702112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6813","nonce":1384702112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6817","nonce":1384702112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6815","nonce":1384702112}]},"public_addr":"172.21.15.32:6811/1384702112","cluster_addr":"172.21.15.32:6813/1384702112","heartbeat_back_addr":"172.21.15.32:6817/1384702112","heartbeat_front_addr":"172.21.15.32:6815/1384702112","state":["exists","up"]},{"osd":4,"uuid":"e1c1e894-074d-4c78-89ef-9e5a114268bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6809","nonce":4250534574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6811","nonce":4250534574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6815","nonce":4250534574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6813","nonce":4250534574}]},"public_addr":"172.21.15.71:6809/4250534574","cluster_addr":"172.21.15.71:6811/4250534574","heartbeat_back_addr":"172.21.15.71:6815/4250534574","heartbeat_front_addr":"172.21.15.71:6813/4250534574","state":["exists","up"]},{"osd":5,"uuid":"12c022d4-fac6-40c4-ae7a-691571bee5d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6809","nonce":4245346198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6811","nonce":4245346198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6815","nonce":4245346198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6813","nonce":4245346198}]},"public_addr":"172.21.15.189:6809/4245346198","cluster_addr":"172.21.15.189:6811/4245346198","heartbeat_back_addr":"172.21.15.189:6815/4245346198","heartbeat_front_addr":"172.21.15.189:6813/4245346198","state":["exists","up"]},{"osd":6,"uuid":"b413f8ee-3d4a-415a-a2ba-d68437b7330d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6819","nonce":1910389190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6821","nonce":1910389190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6825","nonce":1910389190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6823","nonce":1910389190}]},"public_addr":"172.21.15.32:6819/1910389190","cluster_addr":"172.21.15.32:6821/1910389190","heartbeat_back_addr":"172.21.15.32:6825/1910389190","heartbeat_front_addr":"172.21.15.32:6823/1910389190","state":["exists","up"]},{"osd":7,"uuid":"aac64fb3-fc13-4126-b842-7b06ad5ce238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6817","nonce":692607380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6819","nonce":692607380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6823","nonce":692607380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6821","nonce":692607380}]},"public_addr":"172.21.15.71:6817/692607380","cluster_addr":"172.21.15.71:6819/692607380","heartbeat_back_addr":"172.21.15.71:6823/692607380","heartbeat_front_addr":"172.21.15.71:6821/692607380","state":["exists","up"]},{"osd":8,"uuid":"cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6817","nonce":3973697069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6819","nonce":3973697069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6823","nonce":3973697069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6821","nonce":3973697069}]},"public_addr":"172.21.15.189:6817/3973697069","cluster_addr":"172.21.15.189:6819/3973697069","heartbeat_back_addr":"172.21.15.189:6823/3973697069","heartbeat_front_addr":"172.21.15.189:6821/3973697069","state":["exists","up"]},{"osd":9,"uuid":"ef4a1645-ae6e-45c3-82ad-161510e62aa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6827","nonce":46066685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6829","nonce":46066685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6833","nonce":46066685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6831","nonce":46066685}]},"public_addr":"172.21.15.32:6827/46066685","cluster_addr":"172.21.15.32:6829/46066685","heartbeat_back_addr":"172.21.15.32:6833/46066685","heartbeat_front_addr":"172.21.15.32:6831/46066685","state":["exists","up"]},{"osd":10,"uuid":"25b26a38-a644-409f-b941-59efdf04a3ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6825","nonce":1886290293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6827","nonce":1886290293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6831","nonce":1886290293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6829","nonce":1886290293}]},"public_addr":"172.21.15.71:6825/1886290293","cluster_addr":"172.21.15.71:6827/1886290293","heartbeat_back_addr":"172.21.15.71:6831/1886290293","heartbeat_front_addr":"172.21.15.71:6829/1886290293","state":["exists","up"]},{"osd":11,"uuid":"ab96b3ff-3688-455b-af64-60226c5bfbff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6825","nonce":850185001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6827","nonce":850185001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6831","nonce":850185001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6829","nonce":850185001}]},"public_addr":"172.21.15.189:6825/850185001","cluster_addr":"172.21.15.189:6827/850185001","heartbeat_back_addr":"172.21.15.189:6831/850185001","heartbeat_front_addr":"172.21.15.189:6829/850185001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:27.220308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:53.230783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:17.190642+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:52.603799+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:20.568671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:46.158363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:22.706933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:50.665173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:16.602410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:53.814698+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:22.435189+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:48.651949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1266897222":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/2915689533":"2024-05-01T08:23:29.526664+0000","172.21.15.32:6801/4019351306":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/425836193":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/1134075722":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6801/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/21059253":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/3245550894":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/4019351306":"2024-05-01T08:23:29.526664+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-30T08:31:04.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.704+0000 7f807a000640 1 -- 172.21.15.32:0/1409883678 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f80580afaf0 msgr2=0x7f80580b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:04.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.704+0000 7f807a000640 1 --2- 172.21.15.32:0/1409883678 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f80580afaf0 0x7f80580b2000 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f80600096c0 tx=0x7f8060009260 comp rx=0 tx=0).stop 2024-04-30T08:31:04.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.704+0000 7f807a000640 1 -- 172.21.15.32:0/1409883678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f807406bdf0 msgr2=0x7f80741a8c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:04.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.704+0000 7f807a000640 1 --2- 172.21.15.32:0/1409883678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f807406bdf0 0x7f80741a8c10 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f806400e970 tx=0x7f806400ee30 comp rx=0 tx=0).stop 2024-04-30T08:31:04.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.704+0000 7f807a000640 1 -- 172.21.15.32:0/1409883678 shutdown_connections 2024-04-30T08:31:04.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.704+0000 7f807a000640 1 --2- 172.21.15.32:0/1409883678 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f80580afaf0 0x7f80580b2000 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:04.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.704+0000 7f807a000640 1 --2- 172.21.15.32:0/1409883678 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80741ae0d0 0x7f80741b04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:04.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.704+0000 7f807a000640 1 --2- 172.21.15.32:0/1409883678 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8074110120 0x7f80741a9150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:04.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.704+0000 7f807a000640 1 --2- 172.21.15.32:0/1409883678 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f807406bdf0 0x7f80741a8c10 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:04.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.704+0000 7f807a000640 1 -- 172.21.15.32:0/1409883678 >> 172.21.15.32:0/1409883678 conn(0x7f8074071670 msgr2=0x7f8074073a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:04.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.704+0000 7f807a000640 1 -- 172.21.15.32:0/1409883678 shutdown_connections 2024-04-30T08:31:04.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:04.704+0000 7f807a000640 1 -- 172.21.15.32:0/1409883678 wait complete. 2024-04-30T08:31:04.734 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-30T08:26:19.728331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-30T08:31:04.734 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-30T08:31:05.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.072+0000 7fce5e200640 1 -- 172.21.15.32:0/3985948467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce5810e550 msgr2=0x7fce5810f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:05.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.072+0000 7fce5e200640 1 --2- 172.21.15.32:0/3985948467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce5810e550 0x7fce5810f250 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fce48009900 tx=0x7fce4802a410 comp rx=0 tx=0).stop 2024-04-30T08:31:05.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.072+0000 7fce5e200640 1 -- 172.21.15.32:0/3985948467 shutdown_connections 2024-04-30T08:31:05.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.072+0000 7fce5e200640 1 --2- 172.21.15.32:0/3985948467 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fce58114490 0x7fce581168b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:05.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.072+0000 7fce5e200640 1 --2- 172.21.15.32:0/3985948467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce5810f790 0x7fce5810fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:05.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.072+0000 7fce5e200640 1 --2- 172.21.15.32:0/3985948467 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce5810e550 0x7fce5810f250 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:05.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.072+0000 7fce5e200640 1 -- 172.21.15.32:0/3985948467 >> 172.21.15.32:0/3985948467 conn(0x7fce58074990 msgr2=0x7fce5810bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:05.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.072+0000 7fce5e200640 1 -- 172.21.15.32:0/3985948467 shutdown_connections 2024-04-30T08:31:05.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.072+0000 7fce5e200640 1 -- 172.21.15.32:0/3985948467 wait complete. 2024-04-30T08:31:05.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.073+0000 7fce5e200640 1 Processor -- start 2024-04-30T08:31:05.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.073+0000 7fce5e200640 1 -- start start 2024-04-30T08:31:05.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.074+0000 7fce5e200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce5810e550 0x7fce581a4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:05.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.074+0000 7fce5e200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fce5810f790 0x7fce581a5230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:05.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.074+0000 7fce5e200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce58114490 0x7fce581ac1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:05.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.074+0000 7fce5e200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fce5811a070 con 0x7fce58114490 2024-04-30T08:31:05.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.074+0000 7fce5e200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fce58119ef0 con 0x7fce5810f790 2024-04-30T08:31:05.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.074+0000 7fce5e200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fce5811a1f0 con 0x7fce5810e550 2024-04-30T08:31:05.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.074+0000 7fce57400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce5810e550 0x7fce581a4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:05.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.074+0000 7fce57400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce5810e550 0x7fce581a4cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:46950/0 (socket says 172.21.15.32:46950) 2024-04-30T08:31:05.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.074+0000 7fce57400640 1 -- 172.21.15.32:0/3517390278 learned_addr learned my addr 172.21.15.32:0/3517390278 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:31:05.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.074+0000 7fce57e00640 1 --2- 172.21.15.32:0/3517390278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce58114490 0x7fce581ac1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:05.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.074+0000 7fce56a00640 1 --2- 172.21.15.32:0/3517390278 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fce5810f790 0x7fce581a5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:05.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.074+0000 7fce57400640 1 -- 172.21.15.32:0/3517390278 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fce5810f790 msgr2=0x7fce581a5230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:05.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.074+0000 7fce57400640 1 --2- 172.21.15.32:0/3517390278 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fce5810f790 0x7fce581a5230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:05.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.075+0000 7fce57400640 1 -- 172.21.15.32:0/3517390278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce58114490 msgr2=0x7fce581ac1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:05.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.075+0000 7fce57400640 1 --2- 172.21.15.32:0/3517390278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce58114490 0x7fce581ac1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:05.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.075+0000 7fce57400640 1 -- 172.21.15.32:0/3517390278 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce581ac820 con 0x7fce5810e550 2024-04-30T08:31:05.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.075+0000 7fce57400640 1 --2- 172.21.15.32:0/3517390278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce5810e550 0x7fce581a4cf0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fce48006010 tx=0x7fce48004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:05.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.075+0000 7fce4fe00640 1 -- 172.21.15.32:0/3517390278 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce4803f030 con 0x7fce5810e550 2024-04-30T08:31:05.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.075+0000 7fce5e200640 1 -- 172.21.15.32:0/3517390278 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce581acab0 con 0x7fce5810e550 2024-04-30T08:31:05.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.075+0000 7fce4fe00640 1 -- 172.21.15.32:0/3517390278 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fce48003aa0 con 0x7fce5810e550 2024-04-30T08:31:05.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.075+0000 7fce4fe00640 1 -- 172.21.15.32:0/3517390278 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce48035070 con 0x7fce5810e550 2024-04-30T08:31:05.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.075+0000 7fce5e200640 1 -- 172.21.15.32:0/3517390278 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce581acff0 con 0x7fce5810e550 2024-04-30T08:31:05.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.076+0000 7fce5e200640 1 -- 172.21.15.32:0/3517390278 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce581132a0 con 0x7fce5810e550 2024-04-30T08:31:05.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.079+0000 7fce4fe00640 1 -- 172.21.15.32:0/3517390278 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fce48042030 con 0x7fce5810e550 2024-04-30T08:31:05.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.080+0000 7fce4fe00640 1 --2- 172.21.15.32:0/3517390278 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fce3c0afb20 0x7fce3c0b2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:05.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.080+0000 7fce4fe00640 1 -- 172.21.15.32:0/3517390278 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fce480feda0 con 0x7fce5810e550 2024-04-30T08:31:05.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.080+0000 7fce56a00640 1 --2- 172.21.15.32:0/3517390278 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fce3c0afb20 0x7fce3c0b2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:05.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.081+0000 7fce56a00640 1 --2- 172.21.15.32:0/3517390278 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fce3c0afb20 0x7fce3c0b2030 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fce40009bd0 tx=0x7fce40005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:05.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.085+0000 7fce4fe00640 1 -- 172.21.15.32:0/3517390278 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fce480c7730 con 0x7fce5810e550 2024-04-30T08:31:05.268 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:05 smithi189 ceph-mon[33038]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:05.268 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:05 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:05.268 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:05 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:31:05.268 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:05 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:05.268 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:05 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:05.268 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:05 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:05.269 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:05 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:05.269 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:05 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1409883678' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:31:05.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.291+0000 7fce5e200640 1 -- 172.21.15.32:0/3517390278 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fce58063820 con 0x7fce5810e550 2024-04-30T08:31:05.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.292+0000 7fce4fe00640 1 -- 172.21.15.32:0/3517390278 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) ==== 93+0+10 (secure 0 0 0) 0x7fce480cba30 con 0x7fce5810e550 2024-04-30T08:31:05.294 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-30T08:31:05.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.295+0000 7fce5e200640 1 -- 172.21.15.32:0/3517390278 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fce3c0afb20 msgr2=0x7fce3c0b2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:05.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.295+0000 7fce5e200640 1 --2- 172.21.15.32:0/3517390278 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fce3c0afb20 0x7fce3c0b2030 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fce40009bd0 tx=0x7fce40005f90 comp rx=0 tx=0).stop 2024-04-30T08:31:05.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.295+0000 7fce5e200640 1 -- 172.21.15.32:0/3517390278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce5810e550 msgr2=0x7fce581a4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:05.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.295+0000 7fce5e200640 1 --2- 172.21.15.32:0/3517390278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce5810e550 0x7fce581a4cf0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fce48006010 tx=0x7fce48004270 comp rx=0 tx=0).stop 2024-04-30T08:31:05.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.295+0000 7fce5e200640 1 -- 172.21.15.32:0/3517390278 shutdown_connections 2024-04-30T08:31:05.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.295+0000 7fce5e200640 1 --2- 172.21.15.32:0/3517390278 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fce3c0afb20 0x7fce3c0b2030 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:05.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.295+0000 7fce5e200640 1 --2- 172.21.15.32:0/3517390278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fce58114490 0x7fce581ac1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:05.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.295+0000 7fce5e200640 1 --2- 172.21.15.32:0/3517390278 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fce5810f790 0x7fce581a5230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:05.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.295+0000 7fce5e200640 1 --2- 172.21.15.32:0/3517390278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fce5810e550 0x7fce581a4cf0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:05.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.295+0000 7fce5e200640 1 -- 172.21.15.32:0/3517390278 >> 172.21.15.32:0/3517390278 conn(0x7fce58074990 msgr2=0x7fce58110470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:05.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.295+0000 7fce5e200640 1 -- 172.21.15.32:0/3517390278 shutdown_connections 2024-04-30T08:31:05.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.295+0000 7fce5e200640 1 -- 172.21.15.32:0/3517390278 wait complete. 2024-04-30T08:31:05.322 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:05 smithi032 ceph-mon[28331]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:05.322 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:05 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:05.322 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:05 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:31:05.322 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:05 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:05.322 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:05 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:05.323 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:05 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:05.323 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:05 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:05.323 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:05 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1409883678' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:31:05.325 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-30T08:31:05.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:05 smithi071 ceph-mon[33385]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:05.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:05 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:05.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:05 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:31:05.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:05 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:05.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:05 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:05.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:05 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:05.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:05 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:05.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:05 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1409883678' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:31:05.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.609+0000 7fc97d200640 1 -- 172.21.15.32:0/2507975997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc978104dd0 msgr2=0x7fc9781051e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:05.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.609+0000 7fc97d200640 1 --2- 172.21.15.32:0/2507975997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc978104dd0 0x7fc9781051e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fc964009900 tx=0x7fc96402a3f0 comp rx=0 tx=0).stop 2024-04-30T08:31:05.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.609+0000 7fc97d200640 1 -- 172.21.15.32:0/2507975997 shutdown_connections 2024-04-30T08:31:05.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.609+0000 7fc97d200640 1 --2- 172.21.15.32:0/2507975997 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc978109ad0 0x7fc97810bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:05.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.609+0000 7fc97d200640 1 --2- 172.21.15.32:0/2507975997 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc978104dd0 0x7fc9781051e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:05.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.609+0000 7fc97d200640 1 --2- 172.21.15.32:0/2507975997 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc978103b90 0x7fc978104890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:05.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.609+0000 7fc97d200640 1 -- 172.21.15.32:0/2507975997 >> 172.21.15.32:0/2507975997 conn(0x7fc9780fedc0 msgr2=0x7fc978101230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:05.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.609+0000 7fc97d200640 1 -- 172.21.15.32:0/2507975997 shutdown_connections 2024-04-30T08:31:05.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.609+0000 7fc97d200640 1 -- 172.21.15.32:0/2507975997 wait complete. 2024-04-30T08:31:05.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.610+0000 7fc97d200640 1 Processor -- start 2024-04-30T08:31:05.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.611+0000 7fc97d200640 1 -- start start 2024-04-30T08:31:05.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.611+0000 7fc97d200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc978103b90 0x7fc97819c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:05.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.611+0000 7fc97d200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc978104dd0 0x7fc97819ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:05.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.611+0000 7fc97d200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc978109ad0 0x7fc9781a39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:05.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.611+0000 7fc97d200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc97810f690 con 0x7fc978104dd0 2024-04-30T08:31:05.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.611+0000 7fc97d200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc97810f510 con 0x7fc978103b90 2024-04-30T08:31:05.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.611+0000 7fc97d200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc97810f810 con 0x7fc978109ad0 2024-04-30T08:31:05.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.611+0000 7fc975400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc978104dd0 0x7fc97819ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:05.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.611+0000 7fc975e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc978103b90 0x7fc97819c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:05.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.611+0000 7fc975e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc978103b90 0x7fc97819c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:43616/0 (socket says 172.21.15.32:43616) 2024-04-30T08:31:05.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.611+0000 7fc975e00640 1 -- 172.21.15.32:0/3325032614 learned_addr learned my addr 172.21.15.32:0/3325032614 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:31:05.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.611+0000 7fc976800640 1 --2- 172.21.15.32:0/3325032614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc978109ad0 0x7fc9781a39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:05.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.612+0000 7fc975400640 1 -- 172.21.15.32:0/3325032614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc978109ad0 msgr2=0x7fc9781a39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:05.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.612+0000 7fc975400640 1 --2- 172.21.15.32:0/3325032614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc978109ad0 0x7fc9781a39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:05.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.612+0000 7fc975400640 1 -- 172.21.15.32:0/3325032614 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc978103b90 msgr2=0x7fc97819c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:05.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.612+0000 7fc975400640 1 --2- 172.21.15.32:0/3325032614 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc978103b90 0x7fc97819c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:05.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.612+0000 7fc975400640 1 -- 172.21.15.32:0/3325032614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9781a4040 con 0x7fc978104dd0 2024-04-30T08:31:05.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.612+0000 7fc976800640 1 --2- 172.21.15.32:0/3325032614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc978109ad0 0x7fc9781a39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:31:05.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.612+0000 7fc975400640 1 --2- 172.21.15.32:0/3325032614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc978104dd0 0x7fc97819ca40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc964006010 tx=0x7fc964004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:05.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.612+0000 7fc96ea00640 1 -- 172.21.15.32:0/3325032614 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc96403f070 con 0x7fc978104dd0 2024-04-30T08:31:05.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.613+0000 7fc97d200640 1 -- 172.21.15.32:0/3325032614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9781a42d0 con 0x7fc978104dd0 2024-04-30T08:31:05.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.613+0000 7fc96ea00640 1 -- 172.21.15.32:0/3325032614 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc964031910 con 0x7fc978104dd0 2024-04-30T08:31:05.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.613+0000 7fc96ea00640 1 -- 172.21.15.32:0/3325032614 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc964035070 con 0x7fc978104dd0 2024-04-30T08:31:05.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.613+0000 7fc97d200640 1 -- 172.21.15.32:0/3325032614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9781a4810 con 0x7fc978104dd0 2024-04-30T08:31:05.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.614+0000 7fc97d200640 1 -- 172.21.15.32:0/3325032614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc9781a4a70 con 0x7fc978104dd0 2024-04-30T08:31:05.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.616+0000 7fc96ea00640 1 -- 172.21.15.32:0/3325032614 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fc9640037f0 con 0x7fc978104dd0 2024-04-30T08:31:05.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.618+0000 7fc96ea00640 1 --2- 172.21.15.32:0/3325032614 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc95c0afb70 0x7fc95c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:05.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.618+0000 7fc96ea00640 1 -- 172.21.15.32:0/3325032614 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fc9640ff320 con 0x7fc978104dd0 2024-04-30T08:31:05.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.618+0000 7fc975e00640 1 --2- 172.21.15.32:0/3325032614 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc95c0afb70 0x7fc95c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:05.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.619+0000 7fc975e00640 1 --2- 172.21.15.32:0/3325032614 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc95c0afb70 0x7fc95c0b2080 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc968005f40 tx=0x7fc968009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:05.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.621+0000 7fc96ea00640 1 -- 172.21.15.32:0/3325032614 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc9640c7cb0 con 0x7fc978104dd0 2024-04-30T08:31:05.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.871+0000 7fc97d200640 1 -- 172.21.15.32:0/3325032614 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc9781a4c40 con 0x7fc978104dd0 2024-04-30T08:31:05.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.871+0000 7fc96ea00640 1 -- 172.21.15.32:0/3325032614 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7fc9640cbfb0 con 0x7fc978104dd0 2024-04-30T08:31:05.873 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:31:05.873 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-30T08:31:05.874 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 1 2024-04-30T08:31:05.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.874+0000 7fc97d200640 1 -- 172.21.15.32:0/3325032614 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc95c0afb70 msgr2=0x7fc95c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:05.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.874+0000 7fc97d200640 1 --2- 172.21.15.32:0/3325032614 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc95c0afb70 0x7fc95c0b2080 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc968005f40 tx=0x7fc968009500 comp rx=0 tx=0).stop 2024-04-30T08:31:05.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.874+0000 7fc97d200640 1 -- 172.21.15.32:0/3325032614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc978104dd0 msgr2=0x7fc97819ca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:05.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.874+0000 7fc97d200640 1 --2- 172.21.15.32:0/3325032614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc978104dd0 0x7fc97819ca40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc964006010 tx=0x7fc964004270 comp rx=0 tx=0).stop 2024-04-30T08:31:05.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.875+0000 7fc97d200640 1 -- 172.21.15.32:0/3325032614 shutdown_connections 2024-04-30T08:31:05.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.875+0000 7fc97d200640 1 --2- 172.21.15.32:0/3325032614 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc95c0afb70 0x7fc95c0b2080 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:05.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.875+0000 7fc97d200640 1 --2- 172.21.15.32:0/3325032614 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc978109ad0 0x7fc9781a39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:05.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.875+0000 7fc97d200640 1 --2- 172.21.15.32:0/3325032614 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc978104dd0 0x7fc97819ca40 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:05.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.875+0000 7fc97d200640 1 --2- 172.21.15.32:0/3325032614 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc978103b90 0x7fc97819c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:05.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.875+0000 7fc97d200640 1 -- 172.21.15.32:0/3325032614 >> 172.21.15.32:0/3325032614 conn(0x7fc9780fedc0 msgr2=0x7fc9780ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:05.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.875+0000 7fc97d200640 1 -- 172.21.15.32:0/3325032614 shutdown_connections 2024-04-30T08:31:05.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:05.875+0000 7fc97d200640 1 -- 172.21.15.32:0/3325032614 wait complete. 2024-04-30T08:31:06.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:06 smithi032 ceph-mon[28331]: Deploying daemon mds.l on smithi189 2024-04-30T08:31:06.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:06 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3517390278' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:31:06.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:06 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3325032614' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:06.459 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:06 smithi189 ceph-mon[33038]: Deploying daemon mds.l on smithi189 2024-04-30T08:31:06.460 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:06 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3517390278' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:31:06.460 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:06 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3325032614' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:06.509 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:06 smithi071 ceph-mon[33385]: Deploying daemon mds.l on smithi189 2024-04-30T08:31:06.509 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:06 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3517390278' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:31:06.509 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:06 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3325032614' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:06.800 INFO:journalctl@ceph.mds.l.smithi189.stdout:Apr 30 08:31:06 smithi189 systemd[1]: Starting Ceph mds.l for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:31:07.113 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:07 smithi189 ceph-mon[33038]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:07.114 INFO:journalctl@ceph.mds.l.smithi189.stdout:Apr 30 08:31:06 smithi189 podman[62907]: 2024-04-30 08:31:06.751480212 +0000 UTC m=+0.017341859 image pull 16056ace0edc7f73516cf28297f1db04dda872801fb5b17b9157b0d1ccf38f9b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de 2024-04-30T08:31:07.114 INFO:journalctl@ceph.mds.l.smithi189.stdout:Apr 30 08:31:06 smithi189 podman[62907]: 2024-04-30 08:31:06.868036973 +0000 UTC m=+0.133898623 container create 40bea0c1edd36b00c9309a5545600bb0f8cb3db9dbd178b1a479f45274fe2d22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-l, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.schema-version=1.0) 2024-04-30T08:31:07.114 INFO:journalctl@ceph.mds.l.smithi189.stdout:Apr 30 08:31:07 smithi189 podman[62907]: 2024-04-30 08:31:07.113530259 +0000 UTC m=+0.379391907 container init 40bea0c1edd36b00c9309a5545600bb0f8cb3db9dbd178b1a479f45274fe2d22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-l, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-30T08:31:07.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:07 smithi032 ceph-mon[28331]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:07.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:07 smithi071 ceph-mon[33385]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:07.550 INFO:journalctl@ceph.mds.l.smithi189.stdout:Apr 30 08:31:07 smithi189 podman[62907]: 2024-04-30 08:31:07.116387639 +0000 UTC m=+0.382249290 container start 40bea0c1edd36b00c9309a5545600bb0f8cb3db9dbd178b1a479f45274fe2d22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-l, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-30T08:31:07.550 INFO:journalctl@ceph.mds.l.smithi189.stdout:Apr 30 08:31:07 smithi189 ceph-mds[62926]: -- 172.21.15.189:0/3457480473 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556f81593a40 con 0x556f815db600 2024-04-30T08:31:07.550 INFO:journalctl@ceph.mds.l.smithi189.stdout:Apr 30 08:31:07 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-l[62922]: starting mds.l at 2024-04-30T08:31:07.550 INFO:journalctl@ceph.mds.l.smithi189.stdout:Apr 30 08:31:07 smithi189 bash[62907]: 40bea0c1edd36b00c9309a5545600bb0f8cb3db9dbd178b1a479f45274fe2d22 2024-04-30T08:31:07.550 INFO:journalctl@ceph.mds.l.smithi189.stdout:Apr 30 08:31:07 smithi189 systemd[1]: Started Ceph mds.l for 98c0a670-06ca-11ef-bc93-c7b262605968. 2024-04-30T08:31:07.904 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-30T08:31:08.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.187+0000 7fb70bc00640 1 -- 172.21.15.32:0/3329758925 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb704109ad0 msgr2=0x7fb70410bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:08.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.187+0000 7fb70bc00640 1 --2- 172.21.15.32:0/3329758925 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb704109ad0 0x7fb70410bef0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc009900 tx=0x7fb6fc02a3f0 comp rx=0 tx=0).stop 2024-04-30T08:31:08.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.187+0000 7fb70bc00640 1 -- 172.21.15.32:0/3329758925 shutdown_connections 2024-04-30T08:31:08.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.187+0000 7fb70bc00640 1 --2- 172.21.15.32:0/3329758925 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb704109ad0 0x7fb70410bef0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:08.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.187+0000 7fb70bc00640 1 --2- 172.21.15.32:0/3329758925 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb704104dd0 0x7fb7041051e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:08.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.187+0000 7fb70bc00640 1 --2- 172.21.15.32:0/3329758925 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb704103b90 0x7fb704104890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:08.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.187+0000 7fb70bc00640 1 -- 172.21.15.32:0/3329758925 >> 172.21.15.32:0/3329758925 conn(0x7fb7040fedc0 msgr2=0x7fb704101230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:08.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.187+0000 7fb70bc00640 1 -- 172.21.15.32:0/3329758925 shutdown_connections 2024-04-30T08:31:08.189 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.187+0000 7fb70bc00640 1 -- 172.21.15.32:0/3329758925 wait complete. 2024-04-30T08:31:08.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.188+0000 7fb70bc00640 1 Processor -- start 2024-04-30T08:31:08.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.189+0000 7fb70bc00640 1 -- start start 2024-04-30T08:31:08.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.189+0000 7fb70bc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb704103b90 0x7fb7040779e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:08.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.189+0000 7fb70bc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb704104dd0 0x7fb704076080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:08.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.189+0000 7fb70bc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb704109ad0 0x7fb7040765c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:08.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.189+0000 7fb70bc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb70410f710 con 0x7fb704109ad0 2024-04-30T08:31:08.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.189+0000 7fb70bc00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb70410f590 con 0x7fb704104dd0 2024-04-30T08:31:08.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.189+0000 7fb70bc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb70410f890 con 0x7fb704103b90 2024-04-30T08:31:08.191 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.189+0000 7fb703e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb704104dd0 0x7fb704076080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:08.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.189+0000 7fb703e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb704104dd0 0x7fb704076080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:49752/0 (socket says 172.21.15.32:49752) 2024-04-30T08:31:08.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.189+0000 7fb703e00640 1 -- 172.21.15.32:0/3866242065 learned_addr learned my addr 172.21.15.32:0/3866242065 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:31:08.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.189+0000 7fb709200640 1 --2- 172.21.15.32:0/3866242065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb704103b90 0x7fb7040779e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:08.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.189+0000 7fb709c00640 1 --2- 172.21.15.32:0/3866242065 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb704109ad0 0x7fb7040765c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:08.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.190+0000 7fb703e00640 1 -- 172.21.15.32:0/3866242065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb704103b90 msgr2=0x7fb7040779e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:08.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.190+0000 7fb703e00640 1 --2- 172.21.15.32:0/3866242065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb704103b90 0x7fb7040779e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:08.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.190+0000 7fb703e00640 1 -- 172.21.15.32:0/3866242065 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb704109ad0 msgr2=0x7fb7040765c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:08.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.190+0000 7fb703e00640 1 --2- 172.21.15.32:0/3866242065 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb704109ad0 0x7fb7040765c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:08.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.190+0000 7fb703e00640 1 -- 172.21.15.32:0/3866242065 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb704076cf0 con 0x7fb704104dd0 2024-04-30T08:31:08.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.190+0000 7fb709c00640 1 --2- 172.21.15.32:0/3866242065 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb704109ad0 0x7fb7040765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:31:08.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.190+0000 7fb703e00640 1 --2- 172.21.15.32:0/3866242065 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb704104dd0 0x7fb704076080 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb6f0009d50 tx=0x7fb6f000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:08.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.190+0000 7fb701600640 1 -- 172.21.15.32:0/3866242065 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f0003b70 con 0x7fb704104dd0 2024-04-30T08:31:08.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.190+0000 7fb70bc00640 1 -- 172.21.15.32:0/3866242065 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7041a43a0 con 0x7fb704104dd0 2024-04-30T08:31:08.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.191+0000 7fb70bc00640 1 -- 172.21.15.32:0/3866242065 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7041a4890 con 0x7fb704104dd0 2024-04-30T08:31:08.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.191+0000 7fb701600640 1 -- 172.21.15.32:0/3866242065 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6f0003d10 con 0x7fb704104dd0 2024-04-30T08:31:08.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.191+0000 7fb701600640 1 -- 172.21.15.32:0/3866242065 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6f00188d0 con 0x7fb704104dd0 2024-04-30T08:31:08.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.192+0000 7fb70bc00640 1 -- 172.21.15.32:0/3866242065 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb7041081d0 con 0x7fb704104dd0 2024-04-30T08:31:08.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.195+0000 7fb701600640 1 -- 172.21.15.32:0/3866242065 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fb6f0004020 con 0x7fb704104dd0 2024-04-30T08:31:08.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.196+0000 7fb701600640 1 --2- 172.21.15.32:0/3866242065 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb6e80afaf0 0x7fb6e80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:08.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.196+0000 7fb701600640 1 -- 172.21.15.32:0/3866242065 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fb6f00deeb0 con 0x7fb704104dd0 2024-04-30T08:31:08.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.196+0000 7fb709200640 1 --2- 172.21.15.32:0/3866242065 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb6e80afaf0 0x7fb6e80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:08.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.197+0000 7fb709200640 1 --2- 172.21.15.32:0/3866242065 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb6e80afaf0 0x7fb6e80b2000 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb704104790 tx=0x7fb6f4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:08.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.201+0000 7fb701600640 1 -- 172.21.15.32:0/3866242065 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb6f00a7e90 con 0x7fb704104dd0 2024-04-30T08:31:08.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:08 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:boot 2024-04-30T08:31:08.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:08 smithi071 ceph-mon[33385]: fsmap 1 up:standby 2024-04-30T08:31:08.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:08 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-30T08:31:08.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:08 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:08.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:08 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:08.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:08 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:08.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:08 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:08.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:08 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:08.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:08 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:08.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:08 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:boot 2024-04-30T08:31:08.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:08 smithi032 ceph-mon[28331]: fsmap 1 up:standby 2024-04-30T08:31:08.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:08 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-30T08:31:08.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:08 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:08.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:08 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:08.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:08 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:08.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:08 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:08.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:08 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:08.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:08 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:08.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.452+0000 7fb70bc00640 1 -- 172.21.15.32:0/3866242065 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb704063820 con 0x7fb704104dd0 2024-04-30T08:31:08.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.453+0000 7fb701600640 1 -- 172.21.15.32:0/3866242065 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1393 (secure 0 0 0) 0x7fb6f00a7790 con 0x7fb704104dd0 2024-04-30T08:31:08.454 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:31:08.454 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/463203038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":463203038},{"type":"v1","addr":"172.21.15.189:6833","nonce":463203038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-30T08:31:08.454 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 2 2024-04-30T08:31:08.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.456+0000 7fb70bc00640 1 -- 172.21.15.32:0/3866242065 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb6e80afaf0 msgr2=0x7fb6e80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:08.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.456+0000 7fb70bc00640 1 --2- 172.21.15.32:0/3866242065 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb6e80afaf0 0x7fb6e80b2000 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb704104790 tx=0x7fb6f4005f90 comp rx=0 tx=0).stop 2024-04-30T08:31:08.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.456+0000 7fb70bc00640 1 -- 172.21.15.32:0/3866242065 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb704104dd0 msgr2=0x7fb704076080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:08.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.456+0000 7fb70bc00640 1 --2- 172.21.15.32:0/3866242065 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb704104dd0 0x7fb704076080 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fb6f0009d50 tx=0x7fb6f000c6a0 comp rx=0 tx=0).stop 2024-04-30T08:31:08.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.456+0000 7fb70bc00640 1 -- 172.21.15.32:0/3866242065 shutdown_connections 2024-04-30T08:31:08.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.456+0000 7fb70bc00640 1 --2- 172.21.15.32:0/3866242065 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb6e80afaf0 0x7fb6e80b2000 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:08.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.456+0000 7fb70bc00640 1 --2- 172.21.15.32:0/3866242065 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb704109ad0 0x7fb7040765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:08.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.456+0000 7fb70bc00640 1 --2- 172.21.15.32:0/3866242065 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb704104dd0 0x7fb704076080 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:08.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.456+0000 7fb70bc00640 1 --2- 172.21.15.32:0/3866242065 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb704103b90 0x7fb7040779e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:08.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.456+0000 7fb70bc00640 1 -- 172.21.15.32:0/3866242065 >> 172.21.15.32:0/3866242065 conn(0x7fb7040fedc0 msgr2=0x7fb7041021f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:08.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.456+0000 7fb70bc00640 1 -- 172.21.15.32:0/3866242065 shutdown_connections 2024-04-30T08:31:08.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:08.456+0000 7fb70bc00640 1 -- 172.21.15.32:0/3866242065 wait complete. 2024-04-30T08:31:08.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:08 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:boot 2024-04-30T08:31:08.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:08 smithi189 ceph-mon[33038]: fsmap 1 up:standby 2024-04-30T08:31:08.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:08 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-30T08:31:08.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:08 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:08.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:08 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:08.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:08 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:08.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:08 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:08.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:08 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:08.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:08 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:09.672 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:09 smithi071 ceph-mon[33385]: Deploying daemon mds.k on smithi071 2024-04-30T08:31:09.672 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:09 smithi071 ceph-mon[33385]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:09.672 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:09 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3866242065' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:09.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:09 smithi032 ceph-mon[28331]: Deploying daemon mds.k on smithi071 2024-04-30T08:31:09.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:09 smithi032 ceph-mon[28331]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:09.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:09 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3866242065' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:09.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:09 smithi189 ceph-mon[33038]: Deploying daemon mds.k on smithi071 2024-04-30T08:31:09.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:09 smithi189 ceph-mon[33038]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:09.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:09 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3866242065' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:09.942 INFO:journalctl@ceph.mds.k.smithi071.stdout:Apr 30 08:31:09 smithi071 podman[63056]: 2024-04-30 08:31:09.790843165 +0000 UTC m=+0.018112556 image pull 16056ace0edc7f73516cf28297f1db04dda872801fb5b17b9157b0d1ccf38f9b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de 2024-04-30T08:31:09.942 INFO:journalctl@ceph.mds.k.smithi071.stdout:Apr 30 08:31:09 smithi071 podman[63056]: 2024-04-30 08:31:09.94189245 +0000 UTC m=+0.169161832 container create 188e029109ea2f6d81f5b7d365df309d4206f8a762dcc38b7ef529d0aa0869ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-k, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430) 2024-04-30T08:31:10.197 INFO:journalctl@ceph.mds.k.smithi071.stdout:Apr 30 08:31:10 smithi071 podman[63056]: 2024-04-30 08:31:10.161527961 +0000 UTC m=+0.388797343 container init 188e029109ea2f6d81f5b7d365df309d4206f8a762dcc38b7ef529d0aa0869ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-k, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-30T08:31:10.197 INFO:journalctl@ceph.mds.k.smithi071.stdout:Apr 30 08:31:10 smithi071 podman[63056]: 2024-04-30 08:31:10.164047841 +0000 UTC m=+0.391317225 container start 188e029109ea2f6d81f5b7d365df309d4206f8a762dcc38b7ef529d0aa0869ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-k, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-30T08:31:10.484 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-30T08:31:10.503 INFO:journalctl@ceph.mds.k.smithi071.stdout:Apr 30 08:31:10 smithi071 ceph-mds[63075]: -- 172.21.15.71:0/2572829805 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5575c85f1a40 con 0x5575c8639600 2024-04-30T08:31:10.503 INFO:journalctl@ceph.mds.k.smithi071.stdout:Apr 30 08:31:10 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-k[63071]: starting mds.k at 2024-04-30T08:31:10.503 INFO:journalctl@ceph.mds.k.smithi071.stdout:Apr 30 08:31:10 smithi071 bash[63056]: 188e029109ea2f6d81f5b7d365df309d4206f8a762dcc38b7ef529d0aa0869ce 2024-04-30T08:31:10.503 INFO:journalctl@ceph.mds.k.smithi071.stdout:Apr 30 08:31:10 smithi071 systemd[1]: Started Ceph mds.k for 98c0a670-06ca-11ef-bc93-c7b262605968. 2024-04-30T08:31:10.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.758+0000 7ffa9ea00640 1 -- 172.21.15.32:0/2975018371 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa980760a0 msgr2=0x7ffa980764b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:10.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.758+0000 7ffa9ea00640 1 --2- 172.21.15.32:0/2975018371 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa980760a0 0x7ffa980764b0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ffa88009ca0 tx=0x7ffa8802a8c0 comp rx=0 tx=0).stop 2024-04-30T08:31:10.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.758+0000 7ffa9ea00640 1 -- 172.21.15.32:0/2975018371 shutdown_connections 2024-04-30T08:31:10.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.758+0000 7ffa9ea00640 1 --2- 172.21.15.32:0/2975018371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa9810bb80 0x7ffa9810dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:10.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.758+0000 7ffa9ea00640 1 --2- 172.21.15.32:0/2975018371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa980769f0 0x7ffa9810b640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:10.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.758+0000 7ffa9ea00640 1 --2- 172.21.15.32:0/2975018371 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa980760a0 0x7ffa980764b0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:10.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.758+0000 7ffa9ea00640 1 -- 172.21.15.32:0/2975018371 >> 172.21.15.32:0/2975018371 conn(0x7ffa980fedc0 msgr2=0x7ffa98101230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:10.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.758+0000 7ffa9ea00640 1 -- 172.21.15.32:0/2975018371 shutdown_connections 2024-04-30T08:31:10.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.758+0000 7ffa9ea00640 1 -- 172.21.15.32:0/2975018371 wait complete. 2024-04-30T08:31:10.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.759+0000 7ffa9ea00640 1 Processor -- start 2024-04-30T08:31:10.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.759+0000 7ffa9ea00640 1 -- start start 2024-04-30T08:31:10.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.760+0000 7ffa9ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa980760a0 0x7ffa9819c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:10.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.760+0000 7ffa9ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa980769f0 0x7ffa9819c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:10.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.760+0000 7ffa9ea00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa9810bb80 0x7ffa981a3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:10.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.760+0000 7ffa9ea00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ffa981118b0 con 0x7ffa980769f0 2024-04-30T08:31:10.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.760+0000 7ffa9ea00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ffa98111730 con 0x7ffa9810bb80 2024-04-30T08:31:10.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.760+0000 7ffa9ea00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ffa98111a30 con 0x7ffa980760a0 2024-04-30T08:31:10.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.760+0000 7ffa9ca00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa9810bb80 0x7ffa981a3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:10.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.760+0000 7ffa8f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa980769f0 0x7ffa9819c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:10.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.760+0000 7ffa9ca00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa9810bb80 0x7ffa981a3930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:49768/0 (socket says 172.21.15.32:49768) 2024-04-30T08:31:10.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.760+0000 7ffa9ca00640 1 -- 172.21.15.32:0/3100836152 learned_addr learned my addr 172.21.15.32:0/3100836152 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:31:10.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.760+0000 7ffa97e00640 1 --2- 172.21.15.32:0/3100836152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa980760a0 0x7ffa9819c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:10.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.761+0000 7ffa9ca00640 1 -- 172.21.15.32:0/3100836152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa980760a0 msgr2=0x7ffa9819c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:10.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.761+0000 7ffa9ca00640 1 --2- 172.21.15.32:0/3100836152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa980760a0 0x7ffa9819c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:10.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.761+0000 7ffa9ca00640 1 -- 172.21.15.32:0/3100836152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa980769f0 msgr2=0x7ffa9819c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:10.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.761+0000 7ffa9ca00640 1 --2- 172.21.15.32:0/3100836152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa980769f0 0x7ffa9819c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:10.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.761+0000 7ffa9ca00640 1 -- 172.21.15.32:0/3100836152 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa981a3fa0 con 0x7ffa9810bb80 2024-04-30T08:31:10.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.761+0000 7ffa97e00640 1 --2- 172.21.15.32:0/3100836152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa980760a0 0x7ffa9819c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:31:10.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.761+0000 7ffa9ca00640 1 --2- 172.21.15.32:0/3100836152 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa9810bb80 0x7ffa981a3930 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffa9000e970 tx=0x7ffa9000ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:10.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.761+0000 7ffa95600640 1 -- 172.21.15.32:0/3100836152 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9000ccc0 con 0x7ffa9810bb80 2024-04-30T08:31:10.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.761+0000 7ffa95600640 1 -- 172.21.15.32:0/3100836152 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa9000ce60 con 0x7ffa9810bb80 2024-04-30T08:31:10.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.761+0000 7ffa9ea00640 1 -- 172.21.15.32:0/3100836152 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa981a4290 con 0x7ffa9810bb80 2024-04-30T08:31:10.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.761+0000 7ffa95600640 1 -- 172.21.15.32:0/3100836152 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa90010430 con 0x7ffa9810bb80 2024-04-30T08:31:10.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.761+0000 7ffa9ea00640 1 -- 172.21.15.32:0/3100836152 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa981a47e0 con 0x7ffa9810bb80 2024-04-30T08:31:10.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.763+0000 7ffa9ea00640 1 -- 172.21.15.32:0/3100836152 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa9810ad40 con 0x7ffa9810bb80 2024-04-30T08:31:10.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.765+0000 7ffa95600640 1 -- 172.21.15.32:0/3100836152 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7ffa90004ff0 con 0x7ffa9810bb80 2024-04-30T08:31:10.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.766+0000 7ffa95600640 1 --2- 172.21.15.32:0/3100836152 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ffa7c0afb70 0x7ffa7c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:10.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.766+0000 7ffa95600640 1 -- 172.21.15.32:0/3100836152 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7ffa900debd0 con 0x7ffa9810bb80 2024-04-30T08:31:10.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.767+0000 7ffa97e00640 1 --2- 172.21.15.32:0/3100836152 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ffa7c0afb70 0x7ffa7c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:10.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.767+0000 7ffa97e00640 1 --2- 172.21.15.32:0/3100836152 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ffa7c0afb70 0x7ffa7c0b2080 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ffa88000f80 tx=0x7ffa88031b10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:10.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:10.771+0000 7ffa95600640 1 -- 172.21.15.32:0/3100836152 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffa900a7bb0 con 0x7ffa9810bb80 2024-04-30T08:31:11.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:11.023+0000 7ffa9ea00640 1 -- 172.21.15.32:0/3100836152 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa9802dbd0 con 0x7ffa9810bb80 2024-04-30T08:31:11.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:11.024+0000 7ffa95600640 1 -- 172.21.15.32:0/3100836152 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2218 (secure 0 0 0) 0x7ffa900a74b0 con 0x7ffa9810bb80 2024-04-30T08:31:11.025 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:31:11.025 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/463203038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":463203038},{"type":"v1","addr":"172.21.15.189:6833","nonce":463203038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3635143591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3635143591},{"type":"v1","addr":"172.21.15.71:6833","nonce":3635143591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-30T08:31:11.026 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 3 2024-04-30T08:31:11.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:11.027+0000 7ffa9ea00640 1 -- 172.21.15.32:0/3100836152 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ffa7c0afb70 msgr2=0x7ffa7c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:11.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:11.027+0000 7ffa9ea00640 1 --2- 172.21.15.32:0/3100836152 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ffa7c0afb70 0x7ffa7c0b2080 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7ffa88000f80 tx=0x7ffa88031b10 comp rx=0 tx=0).stop 2024-04-30T08:31:11.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:11.027+0000 7ffa9ea00640 1 -- 172.21.15.32:0/3100836152 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa9810bb80 msgr2=0x7ffa981a3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:11.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:11.027+0000 7ffa9ea00640 1 --2- 172.21.15.32:0/3100836152 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa9810bb80 0x7ffa981a3930 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffa9000e970 tx=0x7ffa9000ed70 comp rx=0 tx=0).stop 2024-04-30T08:31:11.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:11.027+0000 7ffa9ea00640 1 -- 172.21.15.32:0/3100836152 shutdown_connections 2024-04-30T08:31:11.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:11.027+0000 7ffa9ea00640 1 --2- 172.21.15.32:0/3100836152 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ffa7c0afb70 0x7ffa7c0b2080 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:11.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:11.027+0000 7ffa9ea00640 1 --2- 172.21.15.32:0/3100836152 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa9810bb80 0x7ffa981a3930 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:11.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:11.027+0000 7ffa9ea00640 1 --2- 172.21.15.32:0/3100836152 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa980769f0 0x7ffa9819c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:11.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:11.027+0000 7ffa9ea00640 1 --2- 172.21.15.32:0/3100836152 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa980760a0 0x7ffa9819c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:11.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:11.027+0000 7ffa9ea00640 1 -- 172.21.15.32:0/3100836152 >> 172.21.15.32:0/3100836152 conn(0x7ffa980fedc0 msgr2=0x7ffa980ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:11.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:11.027+0000 7ffa9ea00640 1 -- 172.21.15.32:0/3100836152 shutdown_connections 2024-04-30T08:31:11.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:11.027+0000 7ffa9ea00640 1 -- 172.21.15.32:0/3100836152 wait complete. 2024-04-30T08:31:11.599 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:11 smithi071 ceph-mon[33385]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:11.599 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:11 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:11.599 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:11 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6832/3635143591,v1:172.21.15.71:6833/3635143591] up:boot 2024-04-30T08:31:11.599 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:11 smithi071 ceph-mon[33385]: fsmap 2 up:standby 2024-04-30T08:31:11.599 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:11 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-30T08:31:11.599 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:11 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:11.600 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:11 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:11.600 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:11 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:11.600 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:11 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:11.600 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:11 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:11.600 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:11 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3100836152' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:11.602 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:11 smithi189 ceph-mon[33038]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:11.602 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:11 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:11.602 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:11 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6832/3635143591,v1:172.21.15.71:6833/3635143591] up:boot 2024-04-30T08:31:11.602 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:11 smithi189 ceph-mon[33038]: fsmap 2 up:standby 2024-04-30T08:31:11.602 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:11 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-30T08:31:11.602 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:11 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:11.602 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:11 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:11.603 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:11 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:11.603 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:11 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:11.603 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:11 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:11.603 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:11 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3100836152' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:11.629 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:11 smithi032 ceph-mon[28331]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:11.630 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:11 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:11.630 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:11 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6832/3635143591,v1:172.21.15.71:6833/3635143591] up:boot 2024-04-30T08:31:11.630 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:11 smithi032 ceph-mon[28331]: fsmap 2 up:standby 2024-04-30T08:31:11.630 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:11 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-30T08:31:11.630 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:11 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:11.631 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:11 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:11.631 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:11 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:11.631 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:11 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:11.631 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:11 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:11.631 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:11 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3100836152' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:12.614 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:12 smithi189 ceph-mon[33038]: Deploying daemon mds.j on smithi032 2024-04-30T08:31:12.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:12 smithi032 ceph-mon[28331]: Deploying daemon mds.j on smithi032 2024-04-30T08:31:12.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:12 smithi071 ceph-mon[33385]: Deploying daemon mds.j on smithi032 2024-04-30T08:31:13.064 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-30T08:31:13.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.431+0000 7f42c6800640 1 -- 172.21.15.32:0/2473267187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42c0075010 msgr2=0x7f42c0075420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:13.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.431+0000 7f42c6800640 1 --2- 172.21.15.32:0/2473267187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42c0075010 0x7f42c0075420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f42b4009900 tx=0x7f42b402a400 comp rx=0 tx=0).stop 2024-04-30T08:31:13.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.431+0000 7f42c6800640 1 -- 172.21.15.32:0/2473267187 shutdown_connections 2024-04-30T08:31:13.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.431+0000 7f42c6800640 1 --2- 172.21.15.32:0/2473267187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42c01100f0 0x7f42c0112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:13.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.431+0000 7f42c6800640 1 --2- 172.21.15.32:0/2473267187 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f42c0075960 0x7f42c010fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:13.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.431+0000 7f42c6800640 1 --2- 172.21.15.32:0/2473267187 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42c0075010 0x7f42c0075420 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:13.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.431+0000 7f42c6800640 1 -- 172.21.15.32:0/2473267187 >> 172.21.15.32:0/2473267187 conn(0x7f42c00feba0 msgr2=0x7f42c0101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:13.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.431+0000 7f42c6800640 1 -- 172.21.15.32:0/2473267187 shutdown_connections 2024-04-30T08:31:13.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.431+0000 7f42c6800640 1 -- 172.21.15.32:0/2473267187 wait complete. 2024-04-30T08:31:13.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.432+0000 7f42c6800640 1 Processor -- start 2024-04-30T08:31:13.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.432+0000 7f42c6800640 1 -- start start 2024-04-30T08:31:13.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.433+0000 7f42c6800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42c0075010 0x7f42c01a06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:13.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.433+0000 7f42c6800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f42c0075960 0x7f42c01a0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:13.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.433+0000 7f42c6800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42c01100f0 0x7f42c01a7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:13.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.433+0000 7f42c6800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f42c0115ad0 con 0x7f42c0075010 2024-04-30T08:31:13.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.433+0000 7f42c6800640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f42c0115950 con 0x7f42c0075960 2024-04-30T08:31:13.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.433+0000 7f42c6800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f42c0115c50 con 0x7f42c01100f0 2024-04-30T08:31:13.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.433+0000 7f42c5400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42c0075010 0x7f42c01a06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:13.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.433+0000 7f42c5400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42c0075010 0x7f42c01a06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40826/0 (socket says 172.21.15.32:40826) 2024-04-30T08:31:13.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.433+0000 7f42c5400640 1 -- 172.21.15.32:0/3093928084 learned_addr learned my addr 172.21.15.32:0/3093928084 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:31:13.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.433+0000 7f42c4a00640 1 --2- 172.21.15.32:0/3093928084 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f42c0075960 0x7f42c01a0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:13.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.433+0000 7f42c5e00640 1 --2- 172.21.15.32:0/3093928084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42c01100f0 0x7f42c01a7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:13.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.434+0000 7f42c4a00640 1 -- 172.21.15.32:0/3093928084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42c01100f0 msgr2=0x7f42c01a7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:13.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.434+0000 7f42c4a00640 1 --2- 172.21.15.32:0/3093928084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42c01100f0 0x7f42c01a7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:13.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.434+0000 7f42c4a00640 1 -- 172.21.15.32:0/3093928084 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42c0075010 msgr2=0x7f42c01a06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:13.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.434+0000 7f42c4a00640 1 --2- 172.21.15.32:0/3093928084 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42c0075010 0x7f42c01a06f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:13.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.434+0000 7f42c4a00640 1 -- 172.21.15.32:0/3093928084 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42c01a8230 con 0x7f42c0075960 2024-04-30T08:31:13.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.434+0000 7f42c5400640 1 --2- 172.21.15.32:0/3093928084 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42c0075010 0x7f42c01a06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:31:13.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.434+0000 7f42c4a00640 1 --2- 172.21.15.32:0/3093928084 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f42c0075960 0x7f42c01a0c30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f42ac00e9f0 tx=0x7f42ac00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:13.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.434+0000 7f42be000640 1 -- 172.21.15.32:0/3093928084 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ac00ccc0 con 0x7f42c0075960 2024-04-30T08:31:13.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.434+0000 7f42c6800640 1 -- 172.21.15.32:0/3093928084 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42c01a8520 con 0x7f42c0075960 2024-04-30T08:31:13.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.434+0000 7f42be000640 1 -- 172.21.15.32:0/3093928084 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f42ac0035f0 con 0x7f42c0075960 2024-04-30T08:31:13.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.434+0000 7f42be000640 1 -- 172.21.15.32:0/3093928084 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42ac00ce60 con 0x7f42c0075960 2024-04-30T08:31:13.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.434+0000 7f42c6800640 1 -- 172.21.15.32:0/3093928084 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42c01a8a70 con 0x7f42c0075960 2024-04-30T08:31:13.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.436+0000 7f42c6800640 1 -- 172.21.15.32:0/3093928084 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f42c0069c60 con 0x7f42c0075960 2024-04-30T08:31:13.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.438+0000 7f42be000640 1 -- 172.21.15.32:0/3093928084 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f42ac0037a0 con 0x7f42c0075960 2024-04-30T08:31:13.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.440+0000 7f42be000640 1 --2- 172.21.15.32:0/3093928084 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f42a80afae0 0x7f42a80b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:13.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:13 smithi032 ceph-mon[28331]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:13.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.440+0000 7f42be000640 1 -- 172.21.15.32:0/3093928084 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f42ac0dea40 con 0x7f42c0075960 2024-04-30T08:31:13.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.441+0000 7f42c5400640 1 --2- 172.21.15.32:0/3093928084 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f42a80afae0 0x7f42a80b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:13.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.442+0000 7f42c5400640 1 --2- 172.21.15.32:0/3093928084 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f42a80afae0 0x7f42a80b1ff0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f42b4004730 tx=0x7f42b40046c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:13.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.444+0000 7f42be000640 1 -- 172.21.15.32:0/3093928084 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f42ac0a7a20 con 0x7f42c0075960 2024-04-30T08:31:13.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.689+0000 7f42c6800640 1 -- 172.21.15.32:0/3093928084 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f42c0063820 con 0x7f42c0075960 2024-04-30T08:31:13.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.691+0000 7f42be000640 1 -- 172.21.15.32:0/3093928084 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2218 (secure 0 0 0) 0x7f42ac0a7320 con 0x7f42c0075960 2024-04-30T08:31:13.692 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:31:13.692 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/463203038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":463203038},{"type":"v1","addr":"172.21.15.189:6833","nonce":463203038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3635143591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3635143591},{"type":"v1","addr":"172.21.15.71:6833","nonce":3635143591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-30T08:31:13.692 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 3 2024-04-30T08:31:13.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.693+0000 7f42c6800640 1 -- 172.21.15.32:0/3093928084 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f42a80afae0 msgr2=0x7f42a80b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:13.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.693+0000 7f42c6800640 1 --2- 172.21.15.32:0/3093928084 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f42a80afae0 0x7f42a80b1ff0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f42b4004730 tx=0x7f42b40046c0 comp rx=0 tx=0).stop 2024-04-30T08:31:13.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.693+0000 7f42c6800640 1 -- 172.21.15.32:0/3093928084 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f42c0075960 msgr2=0x7f42c01a0c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:13.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.693+0000 7f42c6800640 1 --2- 172.21.15.32:0/3093928084 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f42c0075960 0x7f42c01a0c30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f42ac00e9f0 tx=0x7f42ac00edf0 comp rx=0 tx=0).stop 2024-04-30T08:31:13.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.693+0000 7f42c6800640 1 -- 172.21.15.32:0/3093928084 shutdown_connections 2024-04-30T08:31:13.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.693+0000 7f42c6800640 1 --2- 172.21.15.32:0/3093928084 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f42a80afae0 0x7f42a80b1ff0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:13.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.693+0000 7f42c6800640 1 --2- 172.21.15.32:0/3093928084 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f42c01100f0 0x7f42c01a7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:13.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.693+0000 7f42c6800640 1 --2- 172.21.15.32:0/3093928084 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f42c0075960 0x7f42c01a0c30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:13.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.693+0000 7f42c6800640 1 --2- 172.21.15.32:0/3093928084 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f42c0075010 0x7f42c01a06f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:13.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.693+0000 7f42c6800640 1 -- 172.21.15.32:0/3093928084 >> 172.21.15.32:0/3093928084 conn(0x7f42c00feba0 msgr2=0x7f42c0102f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:13.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.693+0000 7f42c6800640 1 -- 172.21.15.32:0/3093928084 shutdown_connections 2024-04-30T08:31:13.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:13.694+0000 7f42c6800640 1 -- 172.21.15.32:0/3093928084 wait complete. 2024-04-30T08:31:13.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:13 smithi071 ceph-mon[33385]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:13.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:13 smithi189 ceph-mon[33038]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:14.230 INFO:journalctl@ceph.mds.j.smithi032.stdout:Apr 30 08:31:14 smithi032 podman[69106]: 2024-04-30 08:31:14.217738196 +0000 UTC m=+0.374670081 container init e7e3c5f5b7232d911f86850fce0d024f4761ecadefa013cf26d5c1df4f7f1ffd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-j, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-04-30T08:31:14.231 INFO:journalctl@ceph.mds.j.smithi032.stdout:Apr 30 08:31:14 smithi032 podman[69106]: 2024-04-30 08:31:14.229485153 +0000 UTC m=+0.386417065 container start e7e3c5f5b7232d911f86850fce0d024f4761ecadefa013cf26d5c1df4f7f1ffd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-j, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-30T08:31:14.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:14 smithi032 ceph-mon[28331]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:14.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:14 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3093928084' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:14.691 INFO:journalctl@ceph.mds.j.smithi032.stdout:Apr 30 08:31:14 smithi032 bash[69106]: e7e3c5f5b7232d911f86850fce0d024f4761ecadefa013cf26d5c1df4f7f1ffd 2024-04-30T08:31:14.692 INFO:journalctl@ceph.mds.j.smithi032.stdout:Apr 30 08:31:14 smithi032 ceph-mds[69125]: -- 172.21.15.32:0/2455899474 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bc32323a40 con 0x55bc3236b180 2024-04-30T08:31:14.692 INFO:journalctl@ceph.mds.j.smithi032.stdout:Apr 30 08:31:14 smithi032 systemd[1]: Started Ceph mds.j for 98c0a670-06ca-11ef-bc93-c7b262605968. 2024-04-30T08:31:14.692 INFO:journalctl@ceph.mds.j.smithi032.stdout:Apr 30 08:31:14 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-j[69120]: starting mds.j at 2024-04-30T08:31:14.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:14 smithi071 ceph-mon[33385]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:14.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:14 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3093928084' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:14.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:14 smithi189 ceph-mon[33038]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:14.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:14 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3093928084' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:15.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:15 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:15.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:15 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:15.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:15 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:15.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:15 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:15.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:15 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:15.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:15 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:15.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:15 smithi032 ceph-mon[28331]: Deploying daemon mds.d on smithi032 2024-04-30T08:31:15.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:15 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.32:6834/2235842676,v1:172.21.15.32:6835/2235842676] up:boot 2024-04-30T08:31:15.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:15 smithi032 ceph-mon[28331]: fsmap 3 up:standby 2024-04-30T08:31:15.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:15 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-30T08:31:15.728 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-30T08:31:15.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:15 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:15.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:15 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:15.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:15 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:15.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:15 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:15.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:15 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:15.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:15 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:15.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:15 smithi071 ceph-mon[33385]: Deploying daemon mds.d on smithi032 2024-04-30T08:31:15.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:15 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.32:6834/2235842676,v1:172.21.15.32:6835/2235842676] up:boot 2024-04-30T08:31:15.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:15 smithi071 ceph-mon[33385]: fsmap 3 up:standby 2024-04-30T08:31:15.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:15 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-30T08:31:15.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:15 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:15.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:15 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:15.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:15 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:15.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:15 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:15.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:15 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:15.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:15 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:15.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:15 smithi189 ceph-mon[33038]: Deploying daemon mds.d on smithi032 2024-04-30T08:31:15.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:15 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.32:6834/2235842676,v1:172.21.15.32:6835/2235842676] up:boot 2024-04-30T08:31:15.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:15 smithi189 ceph-mon[33038]: fsmap 3 up:standby 2024-04-30T08:31:15.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:15 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-30T08:31:16.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.006+0000 7f191e400640 1 -- 172.21.15.32:0/3198858039 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1918105da0 msgr2=0x7f1918106aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:16.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.006+0000 7f191e400640 1 --2- 172.21.15.32:0/3198858039 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1918105da0 0x7f1918106aa0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f1908009900 tx=0x7f190802a410 comp rx=0 tx=0).stop 2024-04-30T08:31:16.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.006+0000 7f191e400640 1 -- 172.21.15.32:0/3198858039 shutdown_connections 2024-04-30T08:31:16.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.006+0000 7f191e400640 1 --2- 172.21.15.32:0/3198858039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f191810bce0 0x7f191810e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:16.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.006+0000 7f191e400640 1 --2- 172.21.15.32:0/3198858039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1918106fe0 0x7f19181073f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:16.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.006+0000 7f191e400640 1 --2- 172.21.15.32:0/3198858039 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1918105da0 0x7f1918106aa0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:16.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.006+0000 7f191e400640 1 -- 172.21.15.32:0/3198858039 >> 172.21.15.32:0/3198858039 conn(0x7f1918100f90 msgr2=0x7f1918103440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:16.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.006+0000 7f191e400640 1 -- 172.21.15.32:0/3198858039 shutdown_connections 2024-04-30T08:31:16.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.006+0000 7f191e400640 1 -- 172.21.15.32:0/3198858039 wait complete. 2024-04-30T08:31:16.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.007+0000 7f191e400640 1 Processor -- start 2024-04-30T08:31:16.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.007+0000 7f191e400640 1 -- start start 2024-04-30T08:31:16.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.008+0000 7f191e400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1918105da0 0x7f191819c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:16.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.008+0000 7f191e400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1918106fe0 0x7f191819ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:16.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.008+0000 7f191e400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f191810bce0 0x7f19181a39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:16.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.008+0000 7f191e400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f19181118f0 con 0x7f1918105da0 2024-04-30T08:31:16.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.008+0000 7f191e400640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1918111770 con 0x7f1918106fe0 2024-04-30T08:31:16.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.008+0000 7f191e400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1918111a70 con 0x7f191810bce0 2024-04-30T08:31:16.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.008+0000 7f1917400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1918105da0 0x7f191819c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:16.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.008+0000 7f1917400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1918105da0 0x7f191819c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:40870/0 (socket says 172.21.15.32:40870) 2024-04-30T08:31:16.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.008+0000 7f1917400640 1 -- 172.21.15.32:0/4183193354 learned_addr learned my addr 172.21.15.32:0/4183193354 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:31:16.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.008+0000 7f190ea00640 1 --2- 172.21.15.32:0/4183193354 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1918106fe0 0x7f191819ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:16.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.008+0000 7f1917e00640 1 --2- 172.21.15.32:0/4183193354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f191810bce0 0x7f19181a39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:16.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.009+0000 7f1917400640 1 -- 172.21.15.32:0/4183193354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f191810bce0 msgr2=0x7f19181a39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:16.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.009+0000 7f1917400640 1 --2- 172.21.15.32:0/4183193354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f191810bce0 0x7f19181a39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:16.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.009+0000 7f1917400640 1 -- 172.21.15.32:0/4183193354 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1918106fe0 msgr2=0x7f191819ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:16.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.009+0000 7f1917400640 1 --2- 172.21.15.32:0/4183193354 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1918106fe0 0x7f191819ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:16.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.009+0000 7f1917400640 1 -- 172.21.15.32:0/4183193354 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19181a4060 con 0x7f1918105da0 2024-04-30T08:31:16.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.009+0000 7f1917400640 1 --2- 172.21.15.32:0/4183193354 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1918105da0 0x7f191819c520 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1908031440 tx=0x7f1908003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:16.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.009+0000 7f1914c00640 1 -- 172.21.15.32:0/4183193354 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f190803f070 con 0x7f1918105da0 2024-04-30T08:31:16.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.009+0000 7f191e400640 1 -- 172.21.15.32:0/4183193354 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19181a42f0 con 0x7f1918105da0 2024-04-30T08:31:16.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.009+0000 7f1914c00640 1 -- 172.21.15.32:0/4183193354 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1908031c60 con 0x7f1918105da0 2024-04-30T08:31:16.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.009+0000 7f1914c00640 1 -- 172.21.15.32:0/4183193354 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1908035030 con 0x7f1918105da0 2024-04-30T08:31:16.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.009+0000 7f191e400640 1 -- 172.21.15.32:0/4183193354 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19181a4830 con 0x7f1918105da0 2024-04-30T08:31:16.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.010+0000 7f191e400640 1 -- 172.21.15.32:0/4183193354 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f191810aaf0 con 0x7f1918105da0 2024-04-30T08:31:16.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.013+0000 7f1914c00640 1 -- 172.21.15.32:0/4183193354 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f1908042070 con 0x7f1918105da0 2024-04-30T08:31:16.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.015+0000 7f1914c00640 1 --2- 172.21.15.32:0/4183193354 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f18fc0afaf0 0x7f18fc0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:16.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.015+0000 7f1914c00640 1 -- 172.21.15.32:0/4183193354 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f19080ff250 con 0x7f1918105da0 2024-04-30T08:31:16.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.015+0000 7f190ea00640 1 --2- 172.21.15.32:0/4183193354 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f18fc0afaf0 0x7f18fc0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:16.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.016+0000 7f190ea00640 1 --2- 172.21.15.32:0/4183193354 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f18fc0afaf0 0x7f18fc0b2000 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f191819d5c0 tx=0x7f1900005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:16.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.018+0000 7f1914c00640 1 -- 172.21.15.32:0/4183193354 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f19080c7be0 con 0x7f1918105da0 2024-04-30T08:31:16.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.266+0000 7f191e400640 1 -- 172.21.15.32:0/4183193354 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f191802dbd0 con 0x7f1918105da0 2024-04-30T08:31:16.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.267+0000 7f1914c00640 1 -- 172.21.15.32:0/4183193354 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3043 (secure 0 0 0) 0x7f19080cbee0 con 0x7f1918105da0 2024-04-30T08:31:16.268 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:31:16.268 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2235842676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2235842676},{"type":"v1","addr":"172.21.15.32:6835","nonce":2235842676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/463203038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":463203038},{"type":"v1","addr":"172.21.15.189:6833","nonce":463203038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3635143591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3635143591},{"type":"v1","addr":"172.21.15.71:6833","nonce":3635143591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-04-30T08:31:16.268 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 4 2024-04-30T08:31:16.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.271+0000 7f191e400640 1 -- 172.21.15.32:0/4183193354 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f18fc0afaf0 msgr2=0x7f18fc0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:16.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.271+0000 7f191e400640 1 --2- 172.21.15.32:0/4183193354 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f18fc0afaf0 0x7f18fc0b2000 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f191819d5c0 tx=0x7f1900005f90 comp rx=0 tx=0).stop 2024-04-30T08:31:16.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.271+0000 7f191e400640 1 -- 172.21.15.32:0/4183193354 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1918105da0 msgr2=0x7f191819c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:16.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.271+0000 7f191e400640 1 --2- 172.21.15.32:0/4183193354 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1918105da0 0x7f191819c520 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1908031440 tx=0x7f1908003c10 comp rx=0 tx=0).stop 2024-04-30T08:31:16.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.271+0000 7f191e400640 1 -- 172.21.15.32:0/4183193354 shutdown_connections 2024-04-30T08:31:16.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.271+0000 7f191e400640 1 --2- 172.21.15.32:0/4183193354 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f18fc0afaf0 0x7f18fc0b2000 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:16.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.271+0000 7f191e400640 1 --2- 172.21.15.32:0/4183193354 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f191810bce0 0x7f19181a39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:16.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.271+0000 7f191e400640 1 --2- 172.21.15.32:0/4183193354 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1918106fe0 0x7f191819ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:16.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.271+0000 7f191e400640 1 --2- 172.21.15.32:0/4183193354 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1918105da0 0x7f191819c520 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:16.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.271+0000 7f191e400640 1 -- 172.21.15.32:0/4183193354 >> 172.21.15.32:0/4183193354 conn(0x7f1918100f90 msgr2=0x7f1918107cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:16.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.271+0000 7f191e400640 1 -- 172.21.15.32:0/4183193354 shutdown_connections 2024-04-30T08:31:16.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:16.271+0000 7f191e400640 1 -- 172.21.15.32:0/4183193354 wait complete. 2024-04-30T08:31:16.572 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:16 smithi032 ceph-mon[28331]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:16.572 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:16 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/4183193354' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:16.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:16 smithi071 ceph-mon[33385]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:16.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:16 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/4183193354' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:16.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:16 smithi189 ceph-mon[33038]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:16.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:16 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/4183193354' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:18.192 INFO:journalctl@ceph.mds.d.smithi032.stdout:Apr 30 08:31:17 smithi032 systemd[1]: Starting Ceph mds.d for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:31:18.299 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-30T08:31:18.568 INFO:journalctl@ceph.mds.d.smithi032.stdout:Apr 30 08:31:18 smithi032 podman[69450]: 2024-04-30 08:31:18.171828937 +0000 UTC m=+0.038885651 image pull 16056ace0edc7f73516cf28297f1db04dda872801fb5b17b9157b0d1ccf38f9b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de 2024-04-30T08:31:18.568 INFO:journalctl@ceph.mds.d.smithi032.stdout:Apr 30 08:31:18 smithi032 podman[69450]: 2024-04-30 08:31:18.30299207 +0000 UTC m=+0.170048849 container create c92cfdfac4b557a78dc31f4ad7d4dc967a4260d11c34d178ea2b33f367a61fdd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-30T08:31:18.568 INFO:journalctl@ceph.mds.d.smithi032.stdout:Apr 30 08:31:18 smithi032 podman[69450]: 2024-04-30 08:31:18.566974726 +0000 UTC m=+0.434031448 container init c92cfdfac4b557a78dc31f4ad7d4dc967a4260d11c34d178ea2b33f367a61fdd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-d, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d) 2024-04-30T08:31:18.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.614+0000 7f49b3a00640 1 -- 172.21.15.32:0/989918418 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f49ac103b90 msgr2=0x7f49ac104890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:18.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.614+0000 7f49b3a00640 1 --2- 172.21.15.32:0/989918418 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f49ac103b90 0x7f49ac104890 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f499c009900 tx=0x7f499c02a410 comp rx=0 tx=0).stop 2024-04-30T08:31:18.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.614+0000 7f49b3a00640 1 -- 172.21.15.32:0/989918418 shutdown_connections 2024-04-30T08:31:18.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.614+0000 7f49b3a00640 1 --2- 172.21.15.32:0/989918418 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49ac109ad0 0x7f49ac10bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:18.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.614+0000 7f49b3a00640 1 --2- 172.21.15.32:0/989918418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49ac104dd0 0x7f49ac1051e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:18.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.614+0000 7f49b3a00640 1 --2- 172.21.15.32:0/989918418 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f49ac103b90 0x7f49ac104890 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:18.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.614+0000 7f49b3a00640 1 -- 172.21.15.32:0/989918418 >> 172.21.15.32:0/989918418 conn(0x7f49ac0fedc0 msgr2=0x7f49ac101230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:18.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.614+0000 7f49b3a00640 1 -- 172.21.15.32:0/989918418 shutdown_connections 2024-04-30T08:31:18.615 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.614+0000 7f49b3a00640 1 -- 172.21.15.32:0/989918418 wait complete. 2024-04-30T08:31:18.616 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.615+0000 7f49b3a00640 1 Processor -- start 2024-04-30T08:31:18.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.615+0000 7f49b3a00640 1 -- start start 2024-04-30T08:31:18.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.616+0000 7f49b3a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49ac103b90 0x7f49ac19a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:18.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.616+0000 7f49b3a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f49ac104dd0 0x7f49ac19a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:18.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.616+0000 7f49b3a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49ac109ad0 0x7f49ac1a17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:18.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.616+0000 7f49b3a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f49ac10f690 con 0x7f49ac103b90 2024-04-30T08:31:18.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.616+0000 7f49b3a00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f49ac10f510 con 0x7f49ac104dd0 2024-04-30T08:31:18.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.616+0000 7f49b3a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f49ac10f810 con 0x7f49ac109ad0 2024-04-30T08:31:18.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.616+0000 7f49b1000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49ac103b90 0x7f49ac19a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:18.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.616+0000 7f49b1000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49ac103b90 0x7f49ac19a2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46768/0 (socket says 172.21.15.32:46768) 2024-04-30T08:31:18.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.616+0000 7f49b1000640 1 -- 172.21.15.32:0/1815928823 learned_addr learned my addr 172.21.15.32:0/1815928823 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:31:18.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.616+0000 7f49a3e00640 1 --2- 172.21.15.32:0/1815928823 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f49ac104dd0 0x7f49ac19a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:18.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.616+0000 7f49b1a00640 1 --2- 172.21.15.32:0/1815928823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49ac109ad0 0x7f49ac1a17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:18.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.617+0000 7f49a3e00640 1 -- 172.21.15.32:0/1815928823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49ac109ad0 msgr2=0x7f49ac1a17a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:18.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.617+0000 7f49a3e00640 1 --2- 172.21.15.32:0/1815928823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49ac109ad0 0x7f49ac1a17a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:18.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.617+0000 7f49a3e00640 1 -- 172.21.15.32:0/1815928823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49ac103b90 msgr2=0x7f49ac19a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:18.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.617+0000 7f49a3e00640 1 --2- 172.21.15.32:0/1815928823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49ac103b90 0x7f49ac19a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:18.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.617+0000 7f49a3e00640 1 -- 172.21.15.32:0/1815928823 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49ac1a1e10 con 0x7f49ac104dd0 2024-04-30T08:31:18.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.617+0000 7f49a3e00640 1 --2- 172.21.15.32:0/1815928823 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f49ac104dd0 0x7f49ac19a810 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f499400e9f0 tx=0x7f499400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:18.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.617+0000 7f49aa000640 1 -- 172.21.15.32:0/1815928823 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f499400ccc0 con 0x7f49ac104dd0 2024-04-30T08:31:18.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.617+0000 7f49aa000640 1 -- 172.21.15.32:0/1815928823 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4994005020 con 0x7f49ac104dd0 2024-04-30T08:31:18.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.617+0000 7f49b3a00640 1 -- 172.21.15.32:0/1815928823 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49ac1a2100 con 0x7f49ac104dd0 2024-04-30T08:31:18.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.617+0000 7f49aa000640 1 -- 172.21.15.32:0/1815928823 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4994004a00 con 0x7f49ac104dd0 2024-04-30T08:31:18.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.617+0000 7f49b3a00640 1 -- 172.21.15.32:0/1815928823 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49ac1a2600 con 0x7f49ac104dd0 2024-04-30T08:31:18.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.619+0000 7f49b3a00640 1 -- 172.21.15.32:0/1815928823 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49ac1088e0 con 0x7f49ac104dd0 2024-04-30T08:31:18.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.621+0000 7f49aa000640 1 -- 172.21.15.32:0/1815928823 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f4994004ba0 con 0x7f49ac104dd0 2024-04-30T08:31:18.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.623+0000 7f49aa000640 1 --2- 172.21.15.32:0/1815928823 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f49900afaf0 0x7f49900b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:18.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.623+0000 7f49aa000640 1 -- 172.21.15.32:0/1815928823 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f49940de820 con 0x7f49ac104dd0 2024-04-30T08:31:18.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.623+0000 7f49b1000640 1 --2- 172.21.15.32:0/1815928823 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f49900afaf0 0x7f49900b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:18.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.624+0000 7f49b1000640 1 --2- 172.21.15.32:0/1815928823 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f49900afaf0 0x7f49900b2000 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f499c031500 tx=0x7f499c005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:18.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.624+0000 7f49aa000640 1 -- 172.21.15.32:0/1815928823 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f49940a7890 con 0x7f49ac104dd0 2024-04-30T08:31:18.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.894+0000 7f49b3a00640 1 -- 172.21.15.32:0/1815928823 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f49ac02dbd0 con 0x7f49ac104dd0 2024-04-30T08:31:18.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.895+0000 7f49aa000640 1 -- 172.21.15.32:0/1815928823 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3868 (secure 0 0 0) 0x7f49940a7190 con 0x7f49ac104dd0 2024-04-30T08:31:18.896 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:31:18.897 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2235842676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2235842676},{"type":"v1","addr":"172.21.15.32:6835","nonce":2235842676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/463203038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":463203038},{"type":"v1","addr":"172.21.15.189:6833","nonce":463203038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3635143591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3635143591},{"type":"v1","addr":"172.21.15.71:6833","nonce":3635143591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24484,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2569449994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2569449994},{"type":"v1","addr":"172.21.15.32:6837","nonce":2569449994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[]} 2024-04-30T08:31:18.897 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 5 2024-04-30T08:31:18.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.898+0000 7f49b3a00640 1 -- 172.21.15.32:0/1815928823 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f49900afaf0 msgr2=0x7f49900b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:18.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.898+0000 7f49b3a00640 1 --2- 172.21.15.32:0/1815928823 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f49900afaf0 0x7f49900b2000 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f499c031500 tx=0x7f499c005d50 comp rx=0 tx=0).stop 2024-04-30T08:31:18.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.898+0000 7f49b3a00640 1 -- 172.21.15.32:0/1815928823 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f49ac104dd0 msgr2=0x7f49ac19a810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:18.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.898+0000 7f49b3a00640 1 --2- 172.21.15.32:0/1815928823 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f49ac104dd0 0x7f49ac19a810 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f499400e9f0 tx=0x7f499400eeb0 comp rx=0 tx=0).stop 2024-04-30T08:31:18.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.899+0000 7f49b3a00640 1 -- 172.21.15.32:0/1815928823 shutdown_connections 2024-04-30T08:31:18.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.899+0000 7f49b3a00640 1 --2- 172.21.15.32:0/1815928823 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f49900afaf0 0x7f49900b2000 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:18.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.899+0000 7f49b3a00640 1 --2- 172.21.15.32:0/1815928823 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49ac109ad0 0x7f49ac1a17a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:18.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.899+0000 7f49b3a00640 1 --2- 172.21.15.32:0/1815928823 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f49ac104dd0 0x7f49ac19a810 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:18.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.899+0000 7f49b3a00640 1 --2- 172.21.15.32:0/1815928823 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f49ac103b90 0x7f49ac19a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:18.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.899+0000 7f49b3a00640 1 -- 172.21.15.32:0/1815928823 >> 172.21.15.32:0/1815928823 conn(0x7f49ac0fedc0 msgr2=0x7f49ac105ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:18.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.899+0000 7f49b3a00640 1 -- 172.21.15.32:0/1815928823 shutdown_connections 2024-04-30T08:31:18.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:18.899+0000 7f49b3a00640 1 -- 172.21.15.32:0/1815928823 wait complete. 2024-04-30T08:31:18.924 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:18 smithi032 ceph-mon[28331]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:18.925 INFO:journalctl@ceph.mds.d.smithi032.stdout:Apr 30 08:31:18 smithi032 podman[69450]: 2024-04-30 08:31:18.57715646 +0000 UTC m=+0.444213180 container start c92cfdfac4b557a78dc31f4ad7d4dc967a4260d11c34d178ea2b33f367a61fdd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-d, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.build-date=20240430, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-30T08:31:18.925 INFO:journalctl@ceph.mds.d.smithi032.stdout:Apr 30 08:31:18 smithi032 bash[69450]: c92cfdfac4b557a78dc31f4ad7d4dc967a4260d11c34d178ea2b33f367a61fdd 2024-04-30T08:31:18.925 INFO:journalctl@ceph.mds.d.smithi032.stdout:Apr 30 08:31:18 smithi032 systemd[1]: Started Ceph mds.d for 98c0a670-06ca-11ef-bc93-c7b262605968. 2024-04-30T08:31:18.925 INFO:journalctl@ceph.mds.d.smithi032.stdout:Apr 30 08:31:18 smithi032 ceph-mds[69489]: -- 172.21.15.32:0/1724966272 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bfab5ada40 con 0x55bfab5f5a80 2024-04-30T08:31:18.925 INFO:journalctl@ceph.mds.d.smithi032.stdout:Apr 30 08:31:18 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-d[69485]: starting mds.d at 2024-04-30T08:31:19.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:18 smithi071 ceph-mon[33385]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:19.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:18 smithi189 ceph-mon[33038]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:20.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:19 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.32:6836/2569449994,v1:172.21.15.32:6837/2569449994] up:boot 2024-04-30T08:31:20.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:19 smithi071 ceph-mon[33385]: fsmap 4 up:standby 2024-04-30T08:31:20.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:19 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-30T08:31:20.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:19 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:20.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:19 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:20.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:19 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:20.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:19 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:20.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:19 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:20.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:19 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:20.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:19 smithi071 ceph-mon[33385]: Deploying daemon mds.e on smithi071 2024-04-30T08:31:20.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:19 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1815928823' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:20.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:19 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.32:6836/2569449994,v1:172.21.15.32:6837/2569449994] up:boot 2024-04-30T08:31:20.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:19 smithi189 ceph-mon[33038]: fsmap 4 up:standby 2024-04-30T08:31:20.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:19 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-30T08:31:20.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:19 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:20.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:19 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:20.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:19 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:20.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:19 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:20.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:19 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:20.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:19 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:20.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:19 smithi189 ceph-mon[33038]: Deploying daemon mds.e on smithi071 2024-04-30T08:31:20.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:19 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1815928823' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:20.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:19 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.32:6836/2569449994,v1:172.21.15.32:6837/2569449994] up:boot 2024-04-30T08:31:20.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:19 smithi032 ceph-mon[28331]: fsmap 4 up:standby 2024-04-30T08:31:20.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:19 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-30T08:31:20.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:19 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:20.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:19 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:20.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:19 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:20.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:19 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:20.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:19 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:20.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:19 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:20.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:19 smithi032 ceph-mon[28331]: Deploying daemon mds.e on smithi071 2024-04-30T08:31:20.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:19 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1815928823' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:20.927 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-30T08:31:21.076 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:20 smithi071 ceph-mon[33385]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:21.077 INFO:journalctl@ceph.mds.e.smithi071.stdout:Apr 30 08:31:21 smithi071 systemd[1]: Starting Ceph mds.e for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:31:21.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:20 smithi032 ceph-mon[28331]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:21.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.220+0000 7f4f63e00640 1 -- 172.21.15.32:0/427164204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f6410baf0 msgr2=0x7f4f6410df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:21.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.220+0000 7f4f63e00640 1 --2- 172.21.15.32:0/427164204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f6410baf0 0x7f4f6410df10 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4f5c009860 tx=0x7f4f5c02a340 comp rx=0 tx=0).stop 2024-04-30T08:31:21.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.220+0000 7f4f63e00640 1 -- 172.21.15.32:0/427164204 shutdown_connections 2024-04-30T08:31:21.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.220+0000 7f4f63e00640 1 --2- 172.21.15.32:0/427164204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f6410baf0 0x7f4f6410df10 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:21.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.220+0000 7f4f63e00640 1 --2- 172.21.15.32:0/427164204 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f64106df0 0x7f4f64107200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:21.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.220+0000 7f4f63e00640 1 --2- 172.21.15.32:0/427164204 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f64105bb0 0x7f4f641068b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:21.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.220+0000 7f4f63e00640 1 -- 172.21.15.32:0/427164204 >> 172.21.15.32:0/427164204 conn(0x7f4f64100da0 msgr2=0x7f4f64103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:21.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.220+0000 7f4f63e00640 1 -- 172.21.15.32:0/427164204 shutdown_connections 2024-04-30T08:31:21.222 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.220+0000 7f4f63e00640 1 -- 172.21.15.32:0/427164204 wait complete. 2024-04-30T08:31:21.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.221+0000 7f4f63e00640 1 Processor -- start 2024-04-30T08:31:21.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.222+0000 7f4f63e00640 1 -- start start 2024-04-30T08:31:21.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.222+0000 7f4f63e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f64105bb0 0x7f4f6419c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:21.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.222+0000 7f4f63e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f64106df0 0x7f4f6419c770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:21.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.222+0000 7f4f63e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f6410baf0 0x7f4f641a3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:21.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.222+0000 7f4f63e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4f641116c0 con 0x7f4f6410baf0 2024-04-30T08:31:21.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.222+0000 7f4f63e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4f64111540 con 0x7f4f64105bb0 2024-04-30T08:31:21.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.222+0000 7f4f63e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4f64111840 con 0x7f4f64106df0 2024-04-30T08:31:21.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.223+0000 7f4f62a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f64105bb0 0x7f4f6419c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:21.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.223+0000 7f4f62a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f64105bb0 0x7f4f6419c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:49832/0 (socket says 172.21.15.32:49832) 2024-04-30T08:31:21.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.223+0000 7f4f5a000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f64106df0 0x7f4f6419c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:21.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.223+0000 7f4f62a00640 1 -- 172.21.15.32:0/3135268076 learned_addr learned my addr 172.21.15.32:0/3135268076 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:31:21.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.223+0000 7f4f63400640 1 --2- 172.21.15.32:0/3135268076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f6410baf0 0x7f4f641a3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:21.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.223+0000 7f4f62a00640 1 -- 172.21.15.32:0/3135268076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f64106df0 msgr2=0x7f4f6419c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:21.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.223+0000 7f4f62a00640 1 --2- 172.21.15.32:0/3135268076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f64106df0 0x7f4f6419c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:21.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.223+0000 7f4f62a00640 1 -- 172.21.15.32:0/3135268076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f6410baf0 msgr2=0x7f4f641a3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:21.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.223+0000 7f4f62a00640 1 --2- 172.21.15.32:0/3135268076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f6410baf0 0x7f4f641a3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:21.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.223+0000 7f4f62a00640 1 -- 172.21.15.32:0/3135268076 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f641a3d70 con 0x7f4f64105bb0 2024-04-30T08:31:21.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.223+0000 7f4f5a000640 1 --2- 172.21.15.32:0/3135268076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f64106df0 0x7f4f6419c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:31:21.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.223+0000 7f4f62a00640 1 --2- 172.21.15.32:0/3135268076 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f64105bb0 0x7f4f6419c230 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4f5400efc0 tx=0x7f4f5400c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:21.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.224+0000 7f4f5be00640 1 -- 172.21.15.32:0/3135268076 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f54003970 con 0x7f4f64105bb0 2024-04-30T08:31:21.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.224+0000 7f4f5be00640 1 -- 172.21.15.32:0/3135268076 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f54004020 con 0x7f4f64105bb0 2024-04-30T08:31:21.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.224+0000 7f4f5be00640 1 -- 172.21.15.32:0/3135268076 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f5400b960 con 0x7f4f64105bb0 2024-04-30T08:31:21.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.224+0000 7f4f63e00640 1 -- 172.21.15.32:0/3135268076 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f641a4060 con 0x7f4f64105bb0 2024-04-30T08:31:21.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.224+0000 7f4f63e00640 1 -- 172.21.15.32:0/3135268076 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f641a4560 con 0x7f4f64105bb0 2024-04-30T08:31:21.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.225+0000 7f4f63e00640 1 -- 172.21.15.32:0/3135268076 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f6410a900 con 0x7f4f64105bb0 2024-04-30T08:31:21.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.228+0000 7f4f5be00640 1 -- 172.21.15.32:0/3135268076 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f4f5401d020 con 0x7f4f64105bb0 2024-04-30T08:31:21.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.229+0000 7f4f5be00640 1 --2- 172.21.15.32:0/3135268076 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4f480afae0 0x7f4f480b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:21.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.230+0000 7f4f5be00640 1 -- 172.21.15.32:0/3135268076 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f4f540de8e0 con 0x7f4f64105bb0 2024-04-30T08:31:21.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.230+0000 7f4f5a000640 1 --2- 172.21.15.32:0/3135268076 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4f480afae0 0x7f4f480b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:21.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.230+0000 7f4f5a000640 1 --2- 172.21.15.32:0/3135268076 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4f480afae0 0x7f4f480b1ff0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4f6419d2d0 tx=0x7f4f4c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:21.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.233+0000 7f4f5be00640 1 -- 172.21.15.32:0/3135268076 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4f54051ff0 con 0x7f4f64105bb0 2024-04-30T08:31:21.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:20 smithi189 ceph-mon[33038]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:21.480 INFO:journalctl@ceph.mds.e.smithi071.stdout:Apr 30 08:31:21 smithi071 podman[63376]: 2024-04-30 08:31:21.19379627 +0000 UTC m=+0.017785351 image pull 16056ace0edc7f73516cf28297f1db04dda872801fb5b17b9157b0d1ccf38f9b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de 2024-04-30T08:31:21.480 INFO:journalctl@ceph.mds.e.smithi071.stdout:Apr 30 08:31:21 smithi071 podman[63376]: 2024-04-30 08:31:21.315655287 +0000 UTC m=+0.139644369 container create fe42b55cc699347dd25c04c28bca38b032bc11133afc52fd740a5df408232123 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-e, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430) 2024-04-30T08:31:21.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.484+0000 7f4f63e00640 1 -- 172.21.15.32:0/3135268076 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4f6402dbd0 con 0x7f4f64105bb0 2024-04-30T08:31:21.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.485+0000 7f4f5be00640 1 -- 172.21.15.32:0/3135268076 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3868 (secure 0 0 0) 0x7f4f540a78c0 con 0x7f4f64105bb0 2024-04-30T08:31:21.486 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:31:21.486 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2235842676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2235842676},{"type":"v1","addr":"172.21.15.32:6835","nonce":2235842676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/463203038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":463203038},{"type":"v1","addr":"172.21.15.189:6833","nonce":463203038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3635143591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3635143591},{"type":"v1","addr":"172.21.15.71:6833","nonce":3635143591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24484,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2569449994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2569449994},{"type":"v1","addr":"172.21.15.32:6837","nonce":2569449994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[]} 2024-04-30T08:31:21.486 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 5 2024-04-30T08:31:21.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.488+0000 7f4f63e00640 1 -- 172.21.15.32:0/3135268076 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4f480afae0 msgr2=0x7f4f480b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:21.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.488+0000 7f4f63e00640 1 --2- 172.21.15.32:0/3135268076 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4f480afae0 0x7f4f480b1ff0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f4f6419d2d0 tx=0x7f4f4c005f90 comp rx=0 tx=0).stop 2024-04-30T08:31:21.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.488+0000 7f4f63e00640 1 -- 172.21.15.32:0/3135268076 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f64105bb0 msgr2=0x7f4f6419c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:21.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.488+0000 7f4f63e00640 1 --2- 172.21.15.32:0/3135268076 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f64105bb0 0x7f4f6419c230 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4f5400efc0 tx=0x7f4f5400c800 comp rx=0 tx=0).stop 2024-04-30T08:31:21.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.488+0000 7f4f63e00640 1 -- 172.21.15.32:0/3135268076 shutdown_connections 2024-04-30T08:31:21.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.488+0000 7f4f63e00640 1 --2- 172.21.15.32:0/3135268076 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4f480afae0 0x7f4f480b1ff0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:21.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.488+0000 7f4f63e00640 1 --2- 172.21.15.32:0/3135268076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f6410baf0 0x7f4f641a3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:21.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.488+0000 7f4f63e00640 1 --2- 172.21.15.32:0/3135268076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f64106df0 0x7f4f6419c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:21.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.488+0000 7f4f63e00640 1 --2- 172.21.15.32:0/3135268076 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f64105bb0 0x7f4f6419c230 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:21.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.488+0000 7f4f63e00640 1 -- 172.21.15.32:0/3135268076 >> 172.21.15.32:0/3135268076 conn(0x7f4f64100da0 msgr2=0x7f4f64107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:21.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.488+0000 7f4f63e00640 1 -- 172.21.15.32:0/3135268076 shutdown_connections 2024-04-30T08:31:21.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:21.488+0000 7f4f63e00640 1 -- 172.21.15.32:0/3135268076 wait complete. 2024-04-30T08:31:22.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:21 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3135268076' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:22.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:21 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:22.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:21 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:22.005 INFO:journalctl@ceph.mds.e.smithi071.stdout:Apr 30 08:31:21 smithi071 podman[63376]: 2024-04-30 08:31:21.612054515 +0000 UTC m=+0.436043599 container init fe42b55cc699347dd25c04c28bca38b032bc11133afc52fd740a5df408232123 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-e, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430) 2024-04-30T08:31:22.005 INFO:journalctl@ceph.mds.e.smithi071.stdout:Apr 30 08:31:21 smithi071 podman[63376]: 2024-04-30 08:31:21.614753151 +0000 UTC m=+0.438742233 container start fe42b55cc699347dd25c04c28bca38b032bc11133afc52fd740a5df408232123 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-e, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-04-30T08:31:22.005 INFO:journalctl@ceph.mds.e.smithi071.stdout:Apr 30 08:31:21 smithi071 ceph-mds[63395]: -- 172.21.15.71:0/2850894725 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x557a901a7a40 con 0x557a901ef600 2024-04-30T08:31:22.005 INFO:journalctl@ceph.mds.e.smithi071.stdout:Apr 30 08:31:21 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-e[63391]: starting mds.e at 2024-04-30T08:31:22.005 INFO:journalctl@ceph.mds.e.smithi071.stdout:Apr 30 08:31:21 smithi071 bash[63376]: fe42b55cc699347dd25c04c28bca38b032bc11133afc52fd740a5df408232123 2024-04-30T08:31:22.005 INFO:journalctl@ceph.mds.e.smithi071.stdout:Apr 30 08:31:21 smithi071 systemd[1]: Started Ceph mds.e for 98c0a670-06ca-11ef-bc93-c7b262605968. 2024-04-30T08:31:22.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:21 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3135268076' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:22.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:21 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:22.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:21 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:22.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:21 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3135268076' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:22.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:21 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:22.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:21 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:23.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:22 smithi032 ceph-mon[28331]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:23.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:22 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:boot 2024-04-30T08:31:23.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:22 smithi032 ceph-mon[28331]: fsmap 5 up:standby 2024-04-30T08:31:23.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:22 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-30T08:31:23.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:22 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:23.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:22 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:23.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:22 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:23.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:22 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:23.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:22 smithi032 ceph-mon[28331]: Deploying daemon mds.f on smithi189 2024-04-30T08:31:23.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:22 smithi071 ceph-mon[33385]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:23.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:22 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:boot 2024-04-30T08:31:23.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:22 smithi071 ceph-mon[33385]: fsmap 5 up:standby 2024-04-30T08:31:23.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:22 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-30T08:31:23.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:22 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:23.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:22 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:23.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:22 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:23.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:22 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:23.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:22 smithi071 ceph-mon[33385]: Deploying daemon mds.f on smithi189 2024-04-30T08:31:23.264 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:22 smithi189 ceph-mon[33038]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:23.264 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:22 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:boot 2024-04-30T08:31:23.264 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:22 smithi189 ceph-mon[33038]: fsmap 5 up:standby 2024-04-30T08:31:23.264 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:22 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-30T08:31:23.264 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:22 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:23.264 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:22 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:23.264 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:22 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:23.264 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:22 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:23.264 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:22 smithi189 ceph-mon[33038]: Deploying daemon mds.f on smithi189 2024-04-30T08:31:23.516 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-30T08:31:23.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.810+0000 7fc03a600640 1 -- 172.21.15.32:0/698492589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc034109fc0 msgr2=0x7fc03410acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:23.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.810+0000 7fc03a600640 1 --2- 172.21.15.32:0/698492589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc034109fc0 0x7fc03410acc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc028009900 tx=0x7fc02802a3f0 comp rx=0 tx=0).stop 2024-04-30T08:31:23.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.810+0000 7fc03a600640 1 -- 172.21.15.32:0/698492589 shutdown_connections 2024-04-30T08:31:23.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.810+0000 7fc03a600640 1 --2- 172.21.15.32:0/698492589 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc03410ff00 0x7fc034112320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:23.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.810+0000 7fc03a600640 1 --2- 172.21.15.32:0/698492589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc03410b200 0x7fc03410b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:23.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.810+0000 7fc03a600640 1 --2- 172.21.15.32:0/698492589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc034109fc0 0x7fc03410acc0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:23.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.810+0000 7fc03a600640 1 -- 172.21.15.32:0/698492589 >> 172.21.15.32:0/698492589 conn(0x7fc034078bb0 msgr2=0x7fc034107a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:23.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.810+0000 7fc03a600640 1 -- 172.21.15.32:0/698492589 shutdown_connections 2024-04-30T08:31:23.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.810+0000 7fc03a600640 1 -- 172.21.15.32:0/698492589 wait complete. 2024-04-30T08:31:23.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.811+0000 7fc03a600640 1 Processor -- start 2024-04-30T08:31:23.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.812+0000 7fc03a600640 1 -- start start 2024-04-30T08:31:23.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.812+0000 7fc03a600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc034109fc0 0x7fc0341a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:23.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.812+0000 7fc03a600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc03410b200 0x7fc0341a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:23.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.812+0000 7fc03a600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc03410ff00 0x7fc0341a7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:23.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.812+0000 7fc03a600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc034115af0 con 0x7fc03410ff00 2024-04-30T08:31:23.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.812+0000 7fc03a600640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc034115970 con 0x7fc03410b200 2024-04-30T08:31:23.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.812+0000 7fc03a600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc034115c70 con 0x7fc034109fc0 2024-04-30T08:31:23.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.812+0000 7fc039c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc03410ff00 0x7fc0341a7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:23.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.812+0000 7fc033e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc03410b200 0x7fc0341a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:23.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.812+0000 7fc039200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc034109fc0 0x7fc0341a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:23.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.812+0000 7fc033e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc03410b200 0x7fc0341a0c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:49850/0 (socket says 172.21.15.32:49850) 2024-04-30T08:31:23.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.812+0000 7fc039c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc03410ff00 0x7fc0341a7b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:46842/0 (socket says 172.21.15.32:46842) 2024-04-30T08:31:23.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.812+0000 7fc039200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc034109fc0 0x7fc0341a06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:60832/0 (socket says 172.21.15.32:60832) 2024-04-30T08:31:23.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.812+0000 7fc033e00640 1 -- 172.21.15.32:0/2686453253 learned_addr learned my addr 172.21.15.32:0/2686453253 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:31:23.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.813+0000 7fc033e00640 1 -- 172.21.15.32:0/2686453253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc034109fc0 msgr2=0x7fc0341a06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:23.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.813+0000 7fc033e00640 1 --2- 172.21.15.32:0/2686453253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc034109fc0 0x7fc0341a06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:23.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.813+0000 7fc033e00640 1 -- 172.21.15.32:0/2686453253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc03410ff00 msgr2=0x7fc0341a7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:23.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.813+0000 7fc033e00640 1 --2- 172.21.15.32:0/2686453253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc03410ff00 0x7fc0341a7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:23.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.813+0000 7fc033e00640 1 -- 172.21.15.32:0/2686453253 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0341a8200 con 0x7fc03410b200 2024-04-30T08:31:23.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.813+0000 7fc039c00640 1 --2- 172.21.15.32:0/2686453253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc03410ff00 0x7fc0341a7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:31:23.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.813+0000 7fc033e00640 1 --2- 172.21.15.32:0/2686453253 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc03410b200 0x7fc0341a0c00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc02000e9f0 tx=0x7fc02000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:23.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.813+0000 7fc031600640 1 -- 172.21.15.32:0/2686453253 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc02000cd80 con 0x7fc03410b200 2024-04-30T08:31:23.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.813+0000 7fc03a600640 1 -- 172.21.15.32:0/2686453253 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0341a84f0 con 0x7fc03410b200 2024-04-30T08:31:23.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.813+0000 7fc031600640 1 -- 172.21.15.32:0/2686453253 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0200108e0 con 0x7fc03410b200 2024-04-30T08:31:23.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.813+0000 7fc031600640 1 -- 172.21.15.32:0/2686453253 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc020018980 con 0x7fc03410b200 2024-04-30T08:31:23.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.814+0000 7fc03a600640 1 -- 172.21.15.32:0/2686453253 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0341a8a90 con 0x7fc03410b200 2024-04-30T08:31:23.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.815+0000 7fc03a600640 1 -- 172.21.15.32:0/2686453253 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc03410ed10 con 0x7fc03410b200 2024-04-30T08:31:23.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.818+0000 7fc031600640 1 -- 172.21.15.32:0/2686453253 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fc020010430 con 0x7fc03410b200 2024-04-30T08:31:23.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.819+0000 7fc031600640 1 --2- 172.21.15.32:0/2686453253 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc01c0afae0 0x7fc01c0b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:23.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.820+0000 7fc031600640 1 -- 172.21.15.32:0/2686453253 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fc0200defd0 con 0x7fc03410b200 2024-04-30T08:31:23.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.820+0000 7fc039200640 1 --2- 172.21.15.32:0/2686453253 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc01c0afae0 0x7fc01c0b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:23.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.821+0000 7fc031600640 1 -- 172.21.15.32:0/2686453253 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc0200a7960 con 0x7fc03410b200 2024-04-30T08:31:23.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:23.821+0000 7fc039200640 1 --2- 172.21.15.32:0/2686453253 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc01c0afae0 0x7fc01c0b1ff0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc02802a9e0 tx=0x7fc0280314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:24.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:24.068+0000 7fc03a600640 1 -- 172.21.15.32:0/2686453253 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc03402dbd0 con 0x7fc03410b200 2024-04-30T08:31:24.070 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:24.069+0000 7fc031600640 1 -- 172.21.15.32:0/2686453253 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4693 (secure 0 0 0) 0x7fc0200abc60 con 0x7fc03410b200 2024-04-30T08:31:24.070 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:31:24.070 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2235842676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2235842676},{"type":"v1","addr":"172.21.15.32:6835","nonce":2235842676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/463203038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":463203038},{"type":"v1","addr":"172.21.15.189:6833","nonce":463203038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3635143591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3635143591},{"type":"v1","addr":"172.21.15.71:6833","nonce":3635143591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24484,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2569449994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2569449994},{"type":"v1","addr":"172.21.15.32:6837","nonce":2569449994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6835/1558303056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1558303056},{"type":"v1","addr":"172.21.15.71:6835","nonce":1558303056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-30T08:31:24.071 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 6 2024-04-30T08:31:24.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:24.072+0000 7fc03a600640 1 -- 172.21.15.32:0/2686453253 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc01c0afae0 msgr2=0x7fc01c0b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:24.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:24.072+0000 7fc03a600640 1 --2- 172.21.15.32:0/2686453253 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc01c0afae0 0x7fc01c0b1ff0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fc02802a9e0 tx=0x7fc0280314a0 comp rx=0 tx=0).stop 2024-04-30T08:31:24.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:24.072+0000 7fc03a600640 1 -- 172.21.15.32:0/2686453253 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc03410b200 msgr2=0x7fc0341a0c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:24.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:24.072+0000 7fc03a600640 1 --2- 172.21.15.32:0/2686453253 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc03410b200 0x7fc0341a0c00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc02000e9f0 tx=0x7fc02000eeb0 comp rx=0 tx=0).stop 2024-04-30T08:31:24.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:24.072+0000 7fc03a600640 1 -- 172.21.15.32:0/2686453253 shutdown_connections 2024-04-30T08:31:24.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:24.072+0000 7fc03a600640 1 --2- 172.21.15.32:0/2686453253 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc01c0afae0 0x7fc01c0b1ff0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:24.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:24.072+0000 7fc03a600640 1 --2- 172.21.15.32:0/2686453253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc03410ff00 0x7fc0341a7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:24.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:24.072+0000 7fc03a600640 1 --2- 172.21.15.32:0/2686453253 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc03410b200 0x7fc0341a0c00 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:24.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:24.072+0000 7fc03a600640 1 --2- 172.21.15.32:0/2686453253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc034109fc0 0x7fc0341a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:24.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:24.072+0000 7fc03a600640 1 -- 172.21.15.32:0/2686453253 >> 172.21.15.32:0/2686453253 conn(0x7fc034078bb0 msgr2=0x7fc03410bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:24.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:24.073+0000 7fc03a600640 1 -- 172.21.15.32:0/2686453253 shutdown_connections 2024-04-30T08:31:24.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:24.073+0000 7fc03a600640 1 -- 172.21.15.32:0/2686453253 wait complete. 2024-04-30T08:31:24.300 INFO:journalctl@ceph.mds.f.smithi189.stdout:Apr 30 08:31:24 smithi189 systemd[1]: Starting Ceph mds.f for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:31:24.638 INFO:journalctl@ceph.mds.f.smithi189.stdout:Apr 30 08:31:24 smithi189 podman[63217]: 2024-04-30 08:31:24.243888979 +0000 UTC m=+0.019052242 image pull 16056ace0edc7f73516cf28297f1db04dda872801fb5b17b9157b0d1ccf38f9b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de 2024-04-30T08:31:24.639 INFO:journalctl@ceph.mds.f.smithi189.stdout:Apr 30 08:31:24 smithi189 podman[63217]: 2024-04-30 08:31:24.351645166 +0000 UTC m=+0.126808397 container create c9190cc367e4b7a614cbd637841661c0dac2e8e3e0a677d1b1608ea031099663 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-f, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-30T08:31:24.639 INFO:journalctl@ceph.mds.f.smithi189.stdout:Apr 30 08:31:24 smithi189 podman[63217]: 2024-04-30 08:31:24.638275833 +0000 UTC m=+0.413439068 container init c9190cc367e4b7a614cbd637841661c0dac2e8e3e0a677d1b1608ea031099663 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-f, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, ceph=True, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d) 2024-04-30T08:31:25.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:24 smithi189 ceph-mon[33038]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:25.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:24 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2686453253' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:25.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:24 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:25.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:24 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:25.051 INFO:journalctl@ceph.mds.f.smithi189.stdout:Apr 30 08:31:24 smithi189 podman[63217]: 2024-04-30 08:31:24.640926652 +0000 UTC m=+0.416089884 container start c9190cc367e4b7a614cbd637841661c0dac2e8e3e0a677d1b1608ea031099663 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-f, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-04-30T08:31:25.051 INFO:journalctl@ceph.mds.f.smithi189.stdout:Apr 30 08:31:24 smithi189 ceph-mds[63236]: -- 172.21.15.189:0/2287178687 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5561a55dda40 con 0x5561a5625180 2024-04-30T08:31:25.051 INFO:journalctl@ceph.mds.f.smithi189.stdout:Apr 30 08:31:24 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-f[63232]: starting mds.f at 2024-04-30T08:31:25.051 INFO:journalctl@ceph.mds.f.smithi189.stdout:Apr 30 08:31:24 smithi189 bash[63217]: c9190cc367e4b7a614cbd637841661c0dac2e8e3e0a677d1b1608ea031099663 2024-04-30T08:31:25.051 INFO:journalctl@ceph.mds.f.smithi189.stdout:Apr 30 08:31:24 smithi189 systemd[1]: Started Ceph mds.f for 98c0a670-06ca-11ef-bc93-c7b262605968. 2024-04-30T08:31:25.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:24 smithi032 ceph-mon[28331]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:25.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:24 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2686453253' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:25.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:24 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:25.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:24 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:25.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:24 smithi071 ceph-mon[33385]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:25.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:24 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2686453253' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:25.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:24 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:25.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:24 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:26.102 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-30T08:31:26.135 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:25 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6834/278686565,v1:172.21.15.189:6835/278686565] up:boot 2024-04-30T08:31:26.135 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:25 smithi032 ceph-mon[28331]: fsmap 6 up:standby 2024-04-30T08:31:26.135 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:25 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-30T08:31:26.135 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:25 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:26.135 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:25 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:26.135 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:25 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:26.135 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:25 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:26.136 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:25 smithi032 ceph-mon[28331]: Deploying daemon mds.c on smithi189 2024-04-30T08:31:26.233 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:25 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6834/278686565,v1:172.21.15.189:6835/278686565] up:boot 2024-04-30T08:31:26.233 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:25 smithi189 ceph-mon[33038]: fsmap 6 up:standby 2024-04-30T08:31:26.233 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:25 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-30T08:31:26.233 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:25 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:26.233 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:25 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:26.233 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:25 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:26.233 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:25 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:26.233 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:25 smithi189 ceph-mon[33038]: Deploying daemon mds.c on smithi189 2024-04-30T08:31:26.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:25 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6834/278686565,v1:172.21.15.189:6835/278686565] up:boot 2024-04-30T08:31:26.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:25 smithi071 ceph-mon[33385]: fsmap 6 up:standby 2024-04-30T08:31:26.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:25 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-30T08:31:26.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:25 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:26.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:25 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:26.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:25 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:26.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:25 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:26.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:25 smithi071 ceph-mon[33385]: Deploying daemon mds.c on smithi189 2024-04-30T08:31:26.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.391+0000 7f4ddf000640 1 -- 172.21.15.32:0/2826653260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4dd8103ac0 msgr2=0x7f4dd8103ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:26.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.391+0000 7f4ddf000640 1 --2- 172.21.15.32:0/2826653260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4dd8103ac0 0x7f4dd8103ed0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f4dc4009860 tx=0x7f4dc402a340 comp rx=0 tx=0).stop 2024-04-30T08:31:26.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.392+0000 7f4ddf000640 1 -- 172.21.15.32:0/2826653260 shutdown_connections 2024-04-30T08:31:26.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.392+0000 7f4ddf000640 1 --2- 172.21.15.32:0/2826653260 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4dd81087c0 0x7f4dd810abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:26.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.392+0000 7f4ddf000640 1 --2- 172.21.15.32:0/2826653260 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4dd8103ac0 0x7f4dd8103ed0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:26.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.392+0000 7f4ddf000640 1 --2- 172.21.15.32:0/2826653260 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4dd8102880 0x7f4dd8103580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:26.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.392+0000 7f4ddf000640 1 -- 172.21.15.32:0/2826653260 >> 172.21.15.32:0/2826653260 conn(0x7f4dd80fda70 msgr2=0x7f4dd80fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:26.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.392+0000 7f4ddf000640 1 -- 172.21.15.32:0/2826653260 shutdown_connections 2024-04-30T08:31:26.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.392+0000 7f4ddf000640 1 -- 172.21.15.32:0/2826653260 wait complete. 2024-04-30T08:31:26.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.393+0000 7f4ddf000640 1 Processor -- start 2024-04-30T08:31:26.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.393+0000 7f4ddf000640 1 -- start start 2024-04-30T08:31:26.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.393+0000 7f4ddf000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4dd8102880 0x7f4dd819a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:26.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.393+0000 7f4ddf000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4dd8103ac0 0x7f4dd819a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:26.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.393+0000 7f4ddf000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4dd81087c0 0x7f4dd81a1550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:26.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.393+0000 7f4ddf000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd810e380 con 0x7f4dd81087c0 2024-04-30T08:31:26.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.393+0000 7f4ddf000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd810e200 con 0x7f4dd8102880 2024-04-30T08:31:26.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.393+0000 7f4ddf000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4dd810e500 con 0x7f4dd8103ac0 2024-04-30T08:31:26.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.394+0000 7f4ddd200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4dd8103ac0 0x7f4dd819a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:26.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.394+0000 7f4ddd200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4dd8103ac0 0x7f4dd819a5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:60852/0 (socket says 172.21.15.32:60852) 2024-04-30T08:31:26.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.394+0000 7f4ddd200640 1 -- 172.21.15.32:0/2027831198 learned_addr learned my addr 172.21.15.32:0/2027831198 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:31:26.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.394+0000 7f4dde600640 1 --2- 172.21.15.32:0/2027831198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4dd81087c0 0x7f4dd81a1550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:26.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.394+0000 7f4dddc00640 1 --2- 172.21.15.32:0/2027831198 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4dd8102880 0x7f4dd819a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:26.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.394+0000 7f4dddc00640 1 -- 172.21.15.32:0/2027831198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4dd8103ac0 msgr2=0x7f4dd819a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:26.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.394+0000 7f4dddc00640 1 --2- 172.21.15.32:0/2027831198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4dd8103ac0 0x7f4dd819a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:26.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.394+0000 7f4dddc00640 1 -- 172.21.15.32:0/2027831198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4dd81087c0 msgr2=0x7f4dd81a1550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:26.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.394+0000 7f4dddc00640 1 --2- 172.21.15.32:0/2027831198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4dd81087c0 0x7f4dd81a1550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:26.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.394+0000 7f4dddc00640 1 -- 172.21.15.32:0/2027831198 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4dd81a1bc0 con 0x7f4dd8102880 2024-04-30T08:31:26.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.394+0000 7f4dde600640 1 --2- 172.21.15.32:0/2027831198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4dd81087c0 0x7f4dd81a1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:31:26.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.395+0000 7f4dddc00640 1 --2- 172.21.15.32:0/2027831198 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4dd8102880 0x7f4dd819a080 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4dd8103480 tx=0x7f4dcc00ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:26.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.395+0000 7f4dd6000640 1 -- 172.21.15.32:0/2027831198 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc004440 con 0x7f4dd8102880 2024-04-30T08:31:26.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.395+0000 7f4dd6000640 1 -- 172.21.15.32:0/2027831198 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4dcc00cd70 con 0x7f4dd8102880 2024-04-30T08:31:26.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.395+0000 7f4dd6000640 1 -- 172.21.15.32:0/2027831198 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dcc019480 con 0x7f4dd8102880 2024-04-30T08:31:26.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.395+0000 7f4ddf000640 1 -- 172.21.15.32:0/2027831198 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4dd81a1eb0 con 0x7f4dd8102880 2024-04-30T08:31:26.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.395+0000 7f4ddf000640 1 -- 172.21.15.32:0/2027831198 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4dd81a2310 con 0x7f4dd8102880 2024-04-30T08:31:26.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.396+0000 7f4ddf000640 1 -- 172.21.15.32:0/2027831198 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4dd81a25d0 con 0x7f4dd8102880 2024-04-30T08:31:26.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.398+0000 7f4dd6000640 1 -- 172.21.15.32:0/2027831198 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f4dcc003680 con 0x7f4dd8102880 2024-04-30T08:31:26.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.400+0000 7f4dd6000640 1 --2- 172.21.15.32:0/2027831198 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4dc00afae0 0x7f4dc00b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:26.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.400+0000 7f4dd6000640 1 -- 172.21.15.32:0/2027831198 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f4dcc0dedb0 con 0x7f4dd8102880 2024-04-30T08:31:26.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.400+0000 7f4ddd200640 1 --2- 172.21.15.32:0/2027831198 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4dc00afae0 0x7f4dc00b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:26.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.401+0000 7f4ddd200640 1 --2- 172.21.15.32:0/2027831198 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4dc00afae0 0x7f4dc00b1ff0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4dc40303f0 tx=0x7f4dc4030470 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:26.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.402+0000 7f4dd6000640 1 -- 172.21.15.32:0/2027831198 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4dcc077ff0 con 0x7f4dd8102880 2024-04-30T08:31:26.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.650+0000 7f4ddf000640 1 -- 172.21.15.32:0/2027831198 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4dd81a27a0 con 0x7f4dd8102880 2024-04-30T08:31:26.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.652+0000 7f4dd6000640 1 -- 172.21.15.32:0/2027831198 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5518 (secure 0 0 0) 0x7f4dcc0a1e20 con 0x7f4dd8102880 2024-04-30T08:31:26.653 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:31:26.653 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2235842676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2235842676},{"type":"v1","addr":"172.21.15.32:6835","nonce":2235842676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/463203038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":463203038},{"type":"v1","addr":"172.21.15.189:6833","nonce":463203038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3635143591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3635143591},{"type":"v1","addr":"172.21.15.71:6833","nonce":3635143591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24484,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2569449994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2569449994},{"type":"v1","addr":"172.21.15.32:6837","nonce":2569449994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/278686565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":278686565},{"type":"v1","addr":"172.21.15.189:6835","nonce":278686565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6835/1558303056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1558303056},{"type":"v1","addr":"172.21.15.71:6835","nonce":1558303056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-30T08:31:26.654 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 7 2024-04-30T08:31:26.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.654+0000 7f4ddf000640 1 -- 172.21.15.32:0/2027831198 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4dc00afae0 msgr2=0x7f4dc00b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:26.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.654+0000 7f4ddf000640 1 --2- 172.21.15.32:0/2027831198 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4dc00afae0 0x7f4dc00b1ff0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f4dc40303f0 tx=0x7f4dc4030470 comp rx=0 tx=0).stop 2024-04-30T08:31:26.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.654+0000 7f4ddf000640 1 -- 172.21.15.32:0/2027831198 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4dd8102880 msgr2=0x7f4dd819a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:26.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.654+0000 7f4ddf000640 1 --2- 172.21.15.32:0/2027831198 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4dd8102880 0x7f4dd819a080 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f4dd8103480 tx=0x7f4dcc00ee50 comp rx=0 tx=0).stop 2024-04-30T08:31:26.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.654+0000 7f4ddf000640 1 -- 172.21.15.32:0/2027831198 shutdown_connections 2024-04-30T08:31:26.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.654+0000 7f4ddf000640 1 --2- 172.21.15.32:0/2027831198 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4dc00afae0 0x7f4dc00b1ff0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:26.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.654+0000 7f4ddf000640 1 --2- 172.21.15.32:0/2027831198 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4dd81087c0 0x7f4dd81a1550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:26.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.654+0000 7f4ddf000640 1 --2- 172.21.15.32:0/2027831198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4dd8103ac0 0x7f4dd819a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:26.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.654+0000 7f4ddf000640 1 --2- 172.21.15.32:0/2027831198 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4dd8102880 0x7f4dd819a080 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:26.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.654+0000 7f4ddf000640 1 -- 172.21.15.32:0/2027831198 >> 172.21.15.32:0/2027831198 conn(0x7f4dd80fda70 msgr2=0x7f4dd8069f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:26.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.654+0000 7f4ddf000640 1 -- 172.21.15.32:0/2027831198 shutdown_connections 2024-04-30T08:31:26.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:26.654+0000 7f4ddf000640 1 -- 172.21.15.32:0/2027831198 wait complete. 2024-04-30T08:31:27.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:26 smithi032 ceph-mon[28331]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:27.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:26 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2027831198' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:27.210 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:26 smithi189 ceph-mon[33038]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:27.210 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:26 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2027831198' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:27.211 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 30 08:31:26 smithi189 systemd[1]: Starting Ceph mds.c for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:31:27.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:26 smithi071 ceph-mon[33385]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:27.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:26 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2027831198' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:27.534 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 30 08:31:27 smithi189 podman[63521]: 2024-04-30 08:31:27.112604613 +0000 UTC m=+0.017953334 image pull 16056ace0edc7f73516cf28297f1db04dda872801fb5b17b9157b0d1ccf38f9b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de 2024-04-30T08:31:27.534 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 30 08:31:27 smithi189 podman[63521]: 2024-04-30 08:31:27.236095641 +0000 UTC m=+0.141444364 container create 2d6e1472fdad879e1c1cde2e4ce8871cb533c0b2987b0d65b0a716345ded38f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-30T08:31:27.534 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 30 08:31:27 smithi189 podman[63521]: 2024-04-30 08:31:27.533532089 +0000 UTC m=+0.438880810 container init 2d6e1472fdad879e1c1cde2e4ce8871cb533c0b2987b0d65b0a716345ded38f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, io.buildah.version=1.33.5) 2024-04-30T08:31:27.800 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 30 08:31:27 smithi189 podman[63521]: 2024-04-30 08:31:27.536211253 +0000 UTC m=+0.441559978 container start 2d6e1472fdad879e1c1cde2e4ce8871cb533c0b2987b0d65b0a716345ded38f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.build-date=20240430) 2024-04-30T08:31:27.800 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 30 08:31:27 smithi189 ceph-mds[63540]: -- 172.21.15.189:0/2168107195 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5634a11c1a40 con 0x5634a1209a80 2024-04-30T08:31:27.800 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 30 08:31:27 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-c[63536]: starting mds.c at 2024-04-30T08:31:27.801 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 30 08:31:27 smithi189 bash[63521]: 2d6e1472fdad879e1c1cde2e4ce8871cb533c0b2987b0d65b0a716345ded38f3 2024-04-30T08:31:27.801 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 30 08:31:27 smithi189 systemd[1]: Started Ceph mds.c for 98c0a670-06ca-11ef-bc93-c7b262605968. 2024-04-30T08:31:28.685 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-30T08:31:28.824 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:28 smithi071 ceph-mon[33385]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:28.824 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:28 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:28.824 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:28 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:28.824 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:28 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:28.824 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:28 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:28.825 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:28 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:28.825 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:28 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:28.825 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:28 smithi071 ceph-mon[33385]: Deploying daemon mds.h on smithi071 2024-04-30T08:31:28.825 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:28 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] up:boot 2024-04-30T08:31:28.825 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:28 smithi071 ceph-mon[33385]: fsmap 7 up:standby 2024-04-30T08:31:28.825 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:28 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-30T08:31:28.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.982+0000 7fd540800640 1 -- 172.21.15.32:0/1309851320 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd538106fb0 msgr2=0x7fd5381073c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:28.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.982+0000 7fd540800640 1 --2- 172.21.15.32:0/1309851320 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd538106fb0 0x7fd5381073c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fd528009900 tx=0x7fd52802a410 comp rx=0 tx=0).stop 2024-04-30T08:31:28.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.983+0000 7fd540800640 1 -- 172.21.15.32:0/1309851320 shutdown_connections 2024-04-30T08:31:28.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.983+0000 7fd540800640 1 --2- 172.21.15.32:0/1309851320 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd53810bcb0 0x7fd53810e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:28.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.983+0000 7fd540800640 1 --2- 172.21.15.32:0/1309851320 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd538106fb0 0x7fd5381073c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:28.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.983+0000 7fd540800640 1 --2- 172.21.15.32:0/1309851320 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd538105d70 0x7fd538106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:28.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.983+0000 7fd540800640 1 -- 172.21.15.32:0/1309851320 >> 172.21.15.32:0/1309851320 conn(0x7fd538100f40 msgr2=0x7fd538103410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:28.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.983+0000 7fd540800640 1 -- 172.21.15.32:0/1309851320 shutdown_connections 2024-04-30T08:31:28.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.983+0000 7fd540800640 1 -- 172.21.15.32:0/1309851320 wait complete. 2024-04-30T08:31:28.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.984+0000 7fd540800640 1 Processor -- start 2024-04-30T08:31:28.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.984+0000 7fd540800640 1 -- start start 2024-04-30T08:31:28.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.985+0000 7fd540800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd538105d70 0x7fd53819c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:28.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.985+0000 7fd540800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd538106fb0 0x7fd53819c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:28.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.985+0000 7fd540800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd53810bcb0 0x7fd5381a3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:28.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.985+0000 7fd540800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd5381118e0 con 0x7fd538105d70 2024-04-30T08:31:28.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.985+0000 7fd540800640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd538111760 con 0x7fd53810bcb0 2024-04-30T08:31:28.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.985+0000 7fd540800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd538111a60 con 0x7fd538106fb0 2024-04-30T08:31:28.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.985+0000 7fd53e800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd53810bcb0 0x7fd5381a3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:28.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.985+0000 7fd53de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd538105d70 0x7fd53819c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:28.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.985+0000 7fd53e800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd53810bcb0 0x7fd5381a3950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:58330/0 (socket says 172.21.15.32:58330) 2024-04-30T08:31:28.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.985+0000 7fd53d400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd538106fb0 0x7fd53819c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:28.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.985+0000 7fd53e800640 1 -- 172.21.15.32:0/2882158717 learned_addr learned my addr 172.21.15.32:0/2882158717 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:31:28.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.986+0000 7fd53e800640 1 -- 172.21.15.32:0/2882158717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd538106fb0 msgr2=0x7fd53819c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:28.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.986+0000 7fd53e800640 1 --2- 172.21.15.32:0/2882158717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd538106fb0 0x7fd53819c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:28.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.986+0000 7fd53e800640 1 -- 172.21.15.32:0/2882158717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd538105d70 msgr2=0x7fd53819c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:28.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.986+0000 7fd53e800640 1 --2- 172.21.15.32:0/2882158717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd538105d70 0x7fd53819c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:28.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.986+0000 7fd53e800640 1 -- 172.21.15.32:0/2882158717 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5381a3fc0 con 0x7fd53810bcb0 2024-04-30T08:31:28.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.986+0000 7fd53e800640 1 --2- 172.21.15.32:0/2882158717 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd53810bcb0 0x7fd5381a3950 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd534009fd0 tx=0x7fd53400b260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:28.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.986+0000 7fd532a00640 1 -- 172.21.15.32:0/2882158717 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd534017070 con 0x7fd53810bcb0 2024-04-30T08:31:28.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.987+0000 7fd532a00640 1 -- 172.21.15.32:0/2882158717 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd534003e20 con 0x7fd53810bcb0 2024-04-30T08:31:28.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.987+0000 7fd532a00640 1 -- 172.21.15.32:0/2882158717 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd534010460 con 0x7fd53810bcb0 2024-04-30T08:31:28.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.987+0000 7fd540800640 1 -- 172.21.15.32:0/2882158717 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5381a42b0 con 0x7fd53810bcb0 2024-04-30T08:31:28.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.987+0000 7fd540800640 1 -- 172.21.15.32:0/2882158717 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5381a47d0 con 0x7fd53810bcb0 2024-04-30T08:31:28.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.988+0000 7fd540800640 1 -- 172.21.15.32:0/2882158717 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd53810aac0 con 0x7fd53810bcb0 2024-04-30T08:31:28.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.990+0000 7fd532a00640 1 -- 172.21.15.32:0/2882158717 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fd534003970 con 0x7fd53810bcb0 2024-04-30T08:31:28.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.992+0000 7fd532a00640 1 --2- 172.21.15.32:0/2882158717 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd5200afb70 0x7fd5200b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:28.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.992+0000 7fd532a00640 1 -- 172.21.15.32:0/2882158717 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fd5340de490 con 0x7fd53810bcb0 2024-04-30T08:31:28.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.992+0000 7fd53de00640 1 --2- 172.21.15.32:0/2882158717 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd5200afb70 0x7fd5200b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:28.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.993+0000 7fd53de00640 1 --2- 172.21.15.32:0/2882158717 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd5200afb70 0x7fd5200b2080 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd538106970 tx=0x7fd52c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:28.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:28.996+0000 7fd532a00640 1 -- 172.21.15.32:0/2882158717 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd5340a6e20 con 0x7fd53810bcb0 2024-04-30T08:31:29.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:28 smithi189 ceph-mon[33038]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:29.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:28 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:29.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:28 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:29.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:28 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:29.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:28 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:29.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:28 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:29.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:28 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:29.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:28 smithi189 ceph-mon[33038]: Deploying daemon mds.h on smithi071 2024-04-30T08:31:29.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:28 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] up:boot 2024-04-30T08:31:29.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:28 smithi189 ceph-mon[33038]: fsmap 7 up:standby 2024-04-30T08:31:29.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:28 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-30T08:31:29.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:28 smithi032 ceph-mon[28331]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:29.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:28 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:29.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:28 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:29.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:28 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:29.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:28 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:29.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:28 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:29.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:28 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:29.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:28 smithi032 ceph-mon[28331]: Deploying daemon mds.h on smithi071 2024-04-30T08:31:29.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:28 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] up:boot 2024-04-30T08:31:29.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:28 smithi032 ceph-mon[28331]: fsmap 7 up:standby 2024-04-30T08:31:29.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:28 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-30T08:31:29.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:29.243+0000 7fd540800640 1 -- 172.21.15.32:0/2882158717 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd53802dbd0 con 0x7fd53810bcb0 2024-04-30T08:31:29.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:29.244+0000 7fd532a00640 1 -- 172.21.15.32:0/2882158717 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6343 (secure 0 0 0) 0x7fd5340ab120 con 0x7fd53810bcb0 2024-04-30T08:31:29.246 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:31:29.246 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2235842676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2235842676},{"type":"v1","addr":"172.21.15.32:6835","nonce":2235842676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/463203038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":463203038},{"type":"v1","addr":"172.21.15.189:6833","nonce":463203038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3635143591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3635143591},{"type":"v1","addr":"172.21.15.71:6833","nonce":3635143591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24484,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2569449994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2569449994},{"type":"v1","addr":"172.21.15.32:6837","nonce":2569449994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/278686565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":278686565},{"type":"v1","addr":"172.21.15.189:6835","nonce":278686565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6835/1558303056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1558303056},{"type":"v1","addr":"172.21.15.71:6835","nonce":1558303056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/686520873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":686520873},{"type":"v1","addr":"172.21.15.189:6837","nonce":686520873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-30T08:31:29.247 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 8 2024-04-30T08:31:29.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:29.247+0000 7fd540800640 1 -- 172.21.15.32:0/2882158717 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd5200afb70 msgr2=0x7fd5200b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:29.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:29.247+0000 7fd540800640 1 --2- 172.21.15.32:0/2882158717 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd5200afb70 0x7fd5200b2080 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd538106970 tx=0x7fd52c005f90 comp rx=0 tx=0).stop 2024-04-30T08:31:29.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:29.247+0000 7fd540800640 1 -- 172.21.15.32:0/2882158717 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd53810bcb0 msgr2=0x7fd5381a3950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:29.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:29.247+0000 7fd540800640 1 --2- 172.21.15.32:0/2882158717 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd53810bcb0 0x7fd5381a3950 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd534009fd0 tx=0x7fd53400b260 comp rx=0 tx=0).stop 2024-04-30T08:31:29.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:29.247+0000 7fd540800640 1 -- 172.21.15.32:0/2882158717 shutdown_connections 2024-04-30T08:31:29.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:29.247+0000 7fd540800640 1 --2- 172.21.15.32:0/2882158717 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd5200afb70 0x7fd5200b2080 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:29.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:29.247+0000 7fd540800640 1 --2- 172.21.15.32:0/2882158717 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd53810bcb0 0x7fd5381a3950 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:29.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:29.247+0000 7fd540800640 1 --2- 172.21.15.32:0/2882158717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd538106fb0 0x7fd53819c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:29.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:29.247+0000 7fd540800640 1 --2- 172.21.15.32:0/2882158717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd538105d70 0x7fd53819c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:29.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:29.247+0000 7fd540800640 1 -- 172.21.15.32:0/2882158717 >> 172.21.15.32:0/2882158717 conn(0x7fd538100f40 msgr2=0x7fd538107c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:29.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:29.247+0000 7fd540800640 1 -- 172.21.15.32:0/2882158717 shutdown_connections 2024-04-30T08:31:29.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:29.247+0000 7fd540800640 1 -- 172.21.15.32:0/2882158717 wait complete. 2024-04-30T08:31:30.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:29 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2882158717' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:30.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:29 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2882158717' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:30.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:29 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2882158717' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:30.416 INFO:journalctl@ceph.mds.h.smithi071.stdout:Apr 30 08:31:30 smithi071 systemd[1]: Starting Ceph mds.h for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:31:30.689 INFO:journalctl@ceph.mds.h.smithi071.stdout:Apr 30 08:31:30 smithi071 podman[63700]: 2024-04-30 08:31:30.318301483 +0000 UTC m=+0.017988241 image pull 16056ace0edc7f73516cf28297f1db04dda872801fb5b17b9157b0d1ccf38f9b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de 2024-04-30T08:31:30.689 INFO:journalctl@ceph.mds.h.smithi071.stdout:Apr 30 08:31:30 smithi071 podman[63700]: 2024-04-30 08:31:30.426096965 +0000 UTC m=+0.125783763 container create 478762dd0b2a628b4afdca4f410480b202edbbdb814a91530c2e5b349ffcc9e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-h, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-30T08:31:30.689 INFO:journalctl@ceph.mds.h.smithi071.stdout:Apr 30 08:31:30 smithi071 podman[63700]: 2024-04-30 08:31:30.688707892 +0000 UTC m=+0.388394647 container init 478762dd0b2a628b4afdca4f410480b202edbbdb814a91530c2e5b349ffcc9e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True) 2024-04-30T08:31:31.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:30 smithi071 ceph-mon[33385]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:31.003 INFO:journalctl@ceph.mds.h.smithi071.stdout:Apr 30 08:31:30 smithi071 podman[63700]: 2024-04-30 08:31:30.691412027 +0000 UTC m=+0.391098783 container start 478762dd0b2a628b4afdca4f410480b202edbbdb814a91530c2e5b349ffcc9e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-h, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-30T08:31:31.004 INFO:journalctl@ceph.mds.h.smithi071.stdout:Apr 30 08:31:30 smithi071 ceph-mds[63720]: -- 172.21.15.71:0/955527069 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5556ee4a3a40 con 0x5556ee4eb180 2024-04-30T08:31:31.004 INFO:journalctl@ceph.mds.h.smithi071.stdout:Apr 30 08:31:30 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-h[63716]: starting mds.h at 2024-04-30T08:31:31.004 INFO:journalctl@ceph.mds.h.smithi071.stdout:Apr 30 08:31:30 smithi071 bash[63700]: 478762dd0b2a628b4afdca4f410480b202edbbdb814a91530c2e5b349ffcc9e2 2024-04-30T08:31:31.004 INFO:journalctl@ceph.mds.h.smithi071.stdout:Apr 30 08:31:30 smithi071 systemd[1]: Started Ceph mds.h for 98c0a670-06ca-11ef-bc93-c7b262605968. 2024-04-30T08:31:31.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:30 smithi189 ceph-mon[33038]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:31.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:30 smithi032 ceph-mon[28331]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:31.275 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-30T08:31:31.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.572+0000 7f92f7200640 1 -- 172.21.15.32:0/3657218852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92f8105bb0 msgr2=0x7f92f81068b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:31.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.572+0000 7f92f7200640 1 --2- 172.21.15.32:0/3657218852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92f8105bb0 0x7f92f81068b0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f92e8009900 tx=0x7f92e802a410 comp rx=0 tx=0).stop 2024-04-30T08:31:31.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.572+0000 7f92f7200640 1 -- 172.21.15.32:0/3657218852 shutdown_connections 2024-04-30T08:31:31.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.572+0000 7f92f7200640 1 --2- 172.21.15.32:0/3657218852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92f810baf0 0x7f92f810df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:31.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.572+0000 7f92f7200640 1 --2- 172.21.15.32:0/3657218852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92f8106df0 0x7f92f8107200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:31.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.572+0000 7f92f7200640 1 --2- 172.21.15.32:0/3657218852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92f8105bb0 0x7f92f81068b0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:31.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.572+0000 7f92f7200640 1 -- 172.21.15.32:0/3657218852 >> 172.21.15.32:0/3657218852 conn(0x7f92f8100da0 msgr2=0x7f92f8103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:31.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.572+0000 7f92f7200640 1 -- 172.21.15.32:0/3657218852 shutdown_connections 2024-04-30T08:31:31.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.572+0000 7f92f7200640 1 -- 172.21.15.32:0/3657218852 wait complete. 2024-04-30T08:31:31.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.573+0000 7f92f7200640 1 Processor -- start 2024-04-30T08:31:31.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.573+0000 7f92f7200640 1 -- start start 2024-04-30T08:31:31.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.573+0000 7f92f7200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92f8105bb0 0x7f92f819c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:31.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.573+0000 7f92f7200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92f8106df0 0x7f92f819c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:31.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.574+0000 7f92f7200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92f810baf0 0x7f92f81a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:31.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.574+0000 7f92f7200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f92f8111690 con 0x7f92f8106df0 2024-04-30T08:31:31.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.574+0000 7f92f7200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f92f8111510 con 0x7f92f810baf0 2024-04-30T08:31:31.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.574+0000 7f92f7200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f92f8111810 con 0x7f92f8105bb0 2024-04-30T08:31:31.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.574+0000 7f92f5e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92f8105bb0 0x7f92f819c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:31.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.574+0000 7f92f5400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92f8106df0 0x7f92f819c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:31.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.574+0000 7f92f5e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92f8105bb0 0x7f92f819c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:35470/0 (socket says 172.21.15.32:35470) 2024-04-30T08:31:31.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.574+0000 7f92f5e00640 1 -- 172.21.15.32:0/4109262331 learned_addr learned my addr 172.21.15.32:0/4109262331 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:31:31.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.574+0000 7f92f6800640 1 --2- 172.21.15.32:0/4109262331 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92f810baf0 0x7f92f81a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:31.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.574+0000 7f92f5e00640 1 -- 172.21.15.32:0/4109262331 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92f810baf0 msgr2=0x7f92f81a3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:31.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.574+0000 7f92f5e00640 1 --2- 172.21.15.32:0/4109262331 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92f810baf0 0x7f92f81a3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:31.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.574+0000 7f92f5e00640 1 -- 172.21.15.32:0/4109262331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92f8106df0 msgr2=0x7f92f819c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:31.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.574+0000 7f92f5e00640 1 --2- 172.21.15.32:0/4109262331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92f8106df0 0x7f92f819c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:31.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.574+0000 7f92f5e00640 1 -- 172.21.15.32:0/4109262331 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92f81a3de0 con 0x7f92f8105bb0 2024-04-30T08:31:31.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.574+0000 7f92f5400640 1 --2- 172.21.15.32:0/4109262331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92f8106df0 0x7f92f819c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:31:31.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.574+0000 7f92f6800640 1 --2- 172.21.15.32:0/4109262331 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92f810baf0 0x7f92f81a3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:31:31.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.575+0000 7f92f5e00640 1 --2- 172.21.15.32:0/4109262331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92f8105bb0 0x7f92f819c2a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f92e8006010 tx=0x7f92e8004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:31.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.575+0000 7f92eea00640 1 -- 172.21.15.32:0/4109262331 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e803f030 con 0x7f92f8105bb0 2024-04-30T08:31:31.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.575+0000 7f92eea00640 1 -- 172.21.15.32:0/4109262331 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f92e8031910 con 0x7f92f8105bb0 2024-04-30T08:31:31.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.575+0000 7f92f7200640 1 -- 172.21.15.32:0/4109262331 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92f81a4070 con 0x7f92f8105bb0 2024-04-30T08:31:31.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.575+0000 7f92eea00640 1 -- 172.21.15.32:0/4109262331 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92e8035070 con 0x7f92f8105bb0 2024-04-30T08:31:31.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.575+0000 7f92f7200640 1 -- 172.21.15.32:0/4109262331 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92f81a45b0 con 0x7f92f8105bb0 2024-04-30T08:31:31.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.576+0000 7f92f7200640 1 -- 172.21.15.32:0/4109262331 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92f810a900 con 0x7f92f8105bb0 2024-04-30T08:31:31.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.578+0000 7f92eea00640 1 -- 172.21.15.32:0/4109262331 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f92e8042030 con 0x7f92f8105bb0 2024-04-30T08:31:31.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.580+0000 7f92eea00640 1 --2- 172.21.15.32:0/4109262331 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f92dc0afb60 0x7f92dc0b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:31.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.580+0000 7f92eea00640 1 -- 172.21.15.32:0/4109262331 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f92e80feba0 con 0x7f92f8105bb0 2024-04-30T08:31:31.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.580+0000 7f92f5400640 1 --2- 172.21.15.32:0/4109262331 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f92dc0afb60 0x7f92dc0b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:31.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.581+0000 7f92f5400640 1 --2- 172.21.15.32:0/4109262331 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f92dc0afb60 0x7f92dc0b2070 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f92e4009fd0 tx=0x7f92e4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:31.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.582+0000 7f92eea00640 1 -- 172.21.15.32:0/4109262331 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f92e80c7b80 con 0x7f92f8105bb0 2024-04-30T08:31:31.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.829+0000 7f92f7200640 1 -- 172.21.15.32:0/4109262331 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f92f802dbd0 con 0x7f92f8105bb0 2024-04-30T08:31:31.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.830+0000 7f92eea00640 1 -- 172.21.15.32:0/4109262331 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7168 (secure 0 0 0) 0x7f92e80c7480 con 0x7f92f8105bb0 2024-04-30T08:31:31.832 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:31:31.832 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2235842676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2235842676},{"type":"v1","addr":"172.21.15.32:6835","nonce":2235842676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/463203038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":463203038},{"type":"v1","addr":"172.21.15.189:6833","nonce":463203038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3635143591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3635143591},{"type":"v1","addr":"172.21.15.71:6833","nonce":3635143591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24484,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2569449994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2569449994},{"type":"v1","addr":"172.21.15.32:6837","nonce":2569449994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/278686565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":278686565},{"type":"v1","addr":"172.21.15.189:6835","nonce":278686565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6835/1558303056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1558303056},{"type":"v1","addr":"172.21.15.71:6835","nonce":1558303056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/686520873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":686520873},{"type":"v1","addr":"172.21.15.189:6837","nonce":686520873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/2324039932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2324039932},{"type":"v1","addr":"172.21.15.71:6837","nonce":2324039932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-04-30T08:31:31.833 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 9 2024-04-30T08:31:31.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.833+0000 7f92f7200640 1 -- 172.21.15.32:0/4109262331 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f92dc0afb60 msgr2=0x7f92dc0b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:31.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.833+0000 7f92f7200640 1 --2- 172.21.15.32:0/4109262331 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f92dc0afb60 0x7f92dc0b2070 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f92e4009fd0 tx=0x7f92e4009500 comp rx=0 tx=0).stop 2024-04-30T08:31:31.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.833+0000 7f92f7200640 1 -- 172.21.15.32:0/4109262331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92f8105bb0 msgr2=0x7f92f819c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:31.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.833+0000 7f92f7200640 1 --2- 172.21.15.32:0/4109262331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92f8105bb0 0x7f92f819c2a0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f92e8006010 tx=0x7f92e8004270 comp rx=0 tx=0).stop 2024-04-30T08:31:31.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.834+0000 7f92f7200640 1 -- 172.21.15.32:0/4109262331 shutdown_connections 2024-04-30T08:31:31.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.834+0000 7f92f7200640 1 --2- 172.21.15.32:0/4109262331 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f92dc0afb60 0x7f92dc0b2070 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:31.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.834+0000 7f92f7200640 1 --2- 172.21.15.32:0/4109262331 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f92f810baf0 0x7f92f81a3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:31.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.834+0000 7f92f7200640 1 --2- 172.21.15.32:0/4109262331 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f92f8106df0 0x7f92f819c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:31.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.834+0000 7f92f7200640 1 --2- 172.21.15.32:0/4109262331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92f8105bb0 0x7f92f819c2a0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:31.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.834+0000 7f92f7200640 1 -- 172.21.15.32:0/4109262331 >> 172.21.15.32:0/4109262331 conn(0x7f92f8100da0 msgr2=0x7f92f8107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:31.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.834+0000 7f92f7200640 1 -- 172.21.15.32:0/4109262331 shutdown_connections 2024-04-30T08:31:31.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:31.834+0000 7f92f7200640 1 -- 172.21.15.32:0/4109262331 wait complete. 2024-04-30T08:31:31.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:31 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] up:boot 2024-04-30T08:31:31.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:31 smithi189 ceph-mon[33038]: fsmap 8 up:standby 2024-04-30T08:31:31.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:31 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-30T08:31:31.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:31 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-30T08:31:31.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:31 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:31.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:31 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:31.838 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:31 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:31.839 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:31 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:31.839 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:31 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:31.839 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:31 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:31.839 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:31 smithi189 ceph-mon[33038]: Deploying daemon mds.i on smithi189 2024-04-30T08:31:32.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:31 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] up:boot 2024-04-30T08:31:32.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:31 smithi032 ceph-mon[28331]: fsmap 8 up:standby 2024-04-30T08:31:32.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:31 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-30T08:31:32.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:31 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-30T08:31:32.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:31 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:32.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:31 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:32.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:31 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:32.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:31 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:32.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:31 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:32.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:31 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:32.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:31 smithi032 ceph-mon[28331]: Deploying daemon mds.i on smithi189 2024-04-30T08:31:32.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:31 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] up:boot 2024-04-30T08:31:32.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:31 smithi071 ceph-mon[33385]: fsmap 8 up:standby 2024-04-30T08:31:32.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:31 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-30T08:31:32.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:31 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-30T08:31:32.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:31 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:32.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:31 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:32.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:31 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:32.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:31 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:32.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:31 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:32.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:31 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:32.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:31 smithi071 ceph-mon[33385]: Deploying daemon mds.i on smithi189 2024-04-30T08:31:33.009 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:32 smithi189 ceph-mon[33038]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:33.009 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:32 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/4109262331' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:33.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:32 smithi032 ceph-mon[28331]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:33.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:32 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/4109262331' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:33.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:32 smithi071 ceph-mon[33385]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:33.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:32 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/4109262331' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:33.300 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 30 08:31:33 smithi189 systemd[1]: Starting Ceph mds.i for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:31:33.712 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 30 08:31:33 smithi189 podman[63830]: 2024-04-30 08:31:33.315637392 +0000 UTC m=+0.018114078 image pull 16056ace0edc7f73516cf28297f1db04dda872801fb5b17b9157b0d1ccf38f9b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de 2024-04-30T08:31:33.712 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 30 08:31:33 smithi189 podman[63830]: 2024-04-30 08:31:33.434141192 +0000 UTC m=+0.136617911 container create 7800c09b89351ddddbc70e5fe35e0f433ca1aa3030af4abe43d07492b371ecf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-i, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-30T08:31:33.712 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 30 08:31:33 smithi189 podman[63830]: 2024-04-30 08:31:33.711856793 +0000 UTC m=+0.414333470 container init 7800c09b89351ddddbc70e5fe35e0f433ca1aa3030af4abe43d07492b371ecf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-i, org.label-schema.build-date=20240430, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-30T08:31:33.864 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-30T08:31:34.050 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 30 08:31:33 smithi189 podman[63830]: 2024-04-30 08:31:33.714604635 +0000 UTC m=+0.417081321 container start 7800c09b89351ddddbc70e5fe35e0f433ca1aa3030af4abe43d07492b371ecf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-i, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430) 2024-04-30T08:31:34.140 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 30 08:31:33 smithi189 ceph-mds[63849]: -- 172.21.15.189:0/192832613 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555deafefa40 con 0x555deb037600 2024-04-30T08:31:34.140 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 30 08:31:33 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-i[63845]: starting mds.i at 2024-04-30T08:31:34.140 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 30 08:31:33 smithi189 bash[63830]: 7800c09b89351ddddbc70e5fe35e0f433ca1aa3030af4abe43d07492b371ecf7 2024-04-30T08:31:34.140 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 30 08:31:33 smithi189 systemd[1]: Started Ceph mds.i for 98c0a670-06ca-11ef-bc93-c7b262605968. 2024-04-30T08:31:34.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.144+0000 7f02bfe00640 1 -- 172.21.15.32:0/3736527724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02c0075010 msgr2=0x7f02c0075420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:34.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.144+0000 7f02bfe00640 1 --2- 172.21.15.32:0/3736527724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02c0075010 0x7f02c0075420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f02b0009900 tx=0x7f02b002a3f0 comp rx=0 tx=0).stop 2024-04-30T08:31:34.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.145+0000 7f02bfe00640 1 -- 172.21.15.32:0/3736527724 shutdown_connections 2024-04-30T08:31:34.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.145+0000 7f02bfe00640 1 --2- 172.21.15.32:0/3736527724 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f02c010b900 0x7f02c010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:34.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.145+0000 7f02bfe00640 1 --2- 172.21.15.32:0/3736527724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02c0075960 0x7f02c010b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:34.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.145+0000 7f02bfe00640 1 --2- 172.21.15.32:0/3736527724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02c0075010 0x7f02c0075420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:34.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.145+0000 7f02bfe00640 1 -- 172.21.15.32:0/3736527724 >> 172.21.15.32:0/3736527724 conn(0x7f02c00feba0 msgr2=0x7f02c0101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:34.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.145+0000 7f02bfe00640 1 -- 172.21.15.32:0/3736527724 shutdown_connections 2024-04-30T08:31:34.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.145+0000 7f02bfe00640 1 -- 172.21.15.32:0/3736527724 wait complete. 2024-04-30T08:31:34.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.146+0000 7f02bfe00640 1 Processor -- start 2024-04-30T08:31:34.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.146+0000 7f02bfe00640 1 -- start start 2024-04-30T08:31:34.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.147+0000 7f02bfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02c0075010 0x7f02c019c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:34.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.147+0000 7f02bfe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02c0075960 0x7f02c019c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:34.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.147+0000 7f02bfe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f02c010b900 0x7f02c01a37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:34.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.147+0000 7f02bfe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f02c0111710 con 0x7f02c0075010 2024-04-30T08:31:34.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.147+0000 7f02bfe00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f02c0111590 con 0x7f02c010b900 2024-04-30T08:31:34.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.147+0000 7f02bfe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f02c0111890 con 0x7f02c0075960 2024-04-30T08:31:34.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.147+0000 7f02bea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02c0075010 0x7f02c019c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:34.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.147+0000 7f02bea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02c0075010 0x7f02c019c2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42610/0 (socket says 172.21.15.32:42610) 2024-04-30T08:31:34.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.147+0000 7f02bf400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f02c010b900 0x7f02c01a37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:34.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.147+0000 7f02bea00640 1 -- 172.21.15.32:0/2636338907 learned_addr learned my addr 172.21.15.32:0/2636338907 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:31:34.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.147+0000 7f02be000640 1 --2- 172.21.15.32:0/2636338907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02c0075960 0x7f02c019c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:34.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.148+0000 7f02bf400640 1 -- 172.21.15.32:0/2636338907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02c0075960 msgr2=0x7f02c019c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:34.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.148+0000 7f02bf400640 1 --2- 172.21.15.32:0/2636338907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02c0075960 0x7f02c019c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:34.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.148+0000 7f02bf400640 1 -- 172.21.15.32:0/2636338907 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02c0075010 msgr2=0x7f02c019c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:34.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.148+0000 7f02bf400640 1 --2- 172.21.15.32:0/2636338907 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02c0075010 0x7f02c019c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:34.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.148+0000 7f02bf400640 1 -- 172.21.15.32:0/2636338907 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02c01a3e30 con 0x7f02c010b900 2024-04-30T08:31:34.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.148+0000 7f02bea00640 1 --2- 172.21.15.32:0/2636338907 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02c0075010 0x7f02c019c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:31:34.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.148+0000 7f02bf400640 1 --2- 172.21.15.32:0/2636338907 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f02c010b900 0x7f02c01a37c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f02b8005fd0 tx=0x7f02b800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:34.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.148+0000 7f02b7400640 1 -- 172.21.15.32:0/2636338907 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b800cdf0 con 0x7f02c010b900 2024-04-30T08:31:34.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.148+0000 7f02b7400640 1 -- 172.21.15.32:0/2636338907 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f02b8005020 con 0x7f02c010b900 2024-04-30T08:31:34.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.148+0000 7f02b7400640 1 -- 172.21.15.32:0/2636338907 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02b8010690 con 0x7f02c010b900 2024-04-30T08:31:34.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.148+0000 7f02bfe00640 1 -- 172.21.15.32:0/2636338907 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02c01a4120 con 0x7f02c010b900 2024-04-30T08:31:34.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.148+0000 7f02bfe00640 1 -- 172.21.15.32:0/2636338907 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02c01a46c0 con 0x7f02c010b900 2024-04-30T08:31:34.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.150+0000 7f02bfe00640 1 -- 172.21.15.32:0/2636338907 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02c010aac0 con 0x7f02c010b900 2024-04-30T08:31:34.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.154+0000 7f02b7400640 1 -- 172.21.15.32:0/2636338907 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f02b8005460 con 0x7f02c010b900 2024-04-30T08:31:34.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.155+0000 7f02b7400640 1 --2- 172.21.15.32:0/2636338907 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f02a40afb60 0x7f02a40b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:34.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.155+0000 7f02b7400640 1 -- 172.21.15.32:0/2636338907 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f02b80ded40 con 0x7f02c010b900 2024-04-30T08:31:34.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.156+0000 7f02bea00640 1 --2- 172.21.15.32:0/2636338907 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f02a40afb60 0x7f02a40b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:34.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.156+0000 7f02bea00640 1 --2- 172.21.15.32:0/2636338907 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f02a40afb60 0x7f02a40b2070 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f02b00096c0 tx=0x7f02b00314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:34.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.160+0000 7f02b7400640 1 -- 172.21.15.32:0/2636338907 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f02b80a7dc0 con 0x7f02c010b900 2024-04-30T08:31:34.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.402+0000 7f02bfe00640 1 -- 172.21.15.32:0/2636338907 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f02c002dbd0 con 0x7f02c010b900 2024-04-30T08:31:34.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.403+0000 7f02b7400640 1 -- 172.21.15.32:0/2636338907 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+7995 (secure 0 0 0) 0x7f02b80a76c0 con 0x7f02c010b900 2024-04-30T08:31:34.405 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:31:34.405 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2235842676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2235842676},{"type":"v1","addr":"172.21.15.32:6835","nonce":2235842676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/463203038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":463203038},{"type":"v1","addr":"172.21.15.189:6833","nonce":463203038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3635143591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3635143591},{"type":"v1","addr":"172.21.15.71:6833","nonce":3635143591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24484,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2569449994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2569449994},{"type":"v1","addr":"172.21.15.32:6837","nonce":2569449994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/278686565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":278686565},{"type":"v1","addr":"172.21.15.189:6835","nonce":278686565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6835/1558303056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1558303056},{"type":"v1","addr":"172.21.15.71:6835","nonce":1558303056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/686520873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":686520873},{"type":"v1","addr":"172.21.15.189:6837","nonce":686520873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/2324039932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2324039932},{"type":"v1","addr":"172.21.15.71:6837","nonce":2324039932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6839/352970107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":352970107},{"type":"v1","addr":"172.21.15.189:6839","nonce":352970107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-30T08:31:34.405 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 10 2024-04-30T08:31:34.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.406+0000 7f02bfe00640 1 -- 172.21.15.32:0/2636338907 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f02a40afb60 msgr2=0x7f02a40b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:34.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.406+0000 7f02bfe00640 1 --2- 172.21.15.32:0/2636338907 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f02a40afb60 0x7f02a40b2070 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f02b00096c0 tx=0x7f02b00314a0 comp rx=0 tx=0).stop 2024-04-30T08:31:34.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.406+0000 7f02bfe00640 1 -- 172.21.15.32:0/2636338907 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f02c010b900 msgr2=0x7f02c01a37c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:34.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.406+0000 7f02bfe00640 1 --2- 172.21.15.32:0/2636338907 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f02c010b900 0x7f02c01a37c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f02b8005fd0 tx=0x7f02b800ef10 comp rx=0 tx=0).stop 2024-04-30T08:31:34.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.406+0000 7f02bfe00640 1 -- 172.21.15.32:0/2636338907 shutdown_connections 2024-04-30T08:31:34.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.406+0000 7f02bfe00640 1 --2- 172.21.15.32:0/2636338907 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f02a40afb60 0x7f02a40b2070 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:34.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.406+0000 7f02bfe00640 1 --2- 172.21.15.32:0/2636338907 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f02c010b900 0x7f02c01a37c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:34.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.406+0000 7f02bfe00640 1 --2- 172.21.15.32:0/2636338907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f02c0075960 0x7f02c019c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:34.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.406+0000 7f02bfe00640 1 --2- 172.21.15.32:0/2636338907 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f02c0075010 0x7f02c019c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:34.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.406+0000 7f02bfe00640 1 -- 172.21.15.32:0/2636338907 >> 172.21.15.32:0/2636338907 conn(0x7f02c00feba0 msgr2=0x7f02c00ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:34.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.406+0000 7f02bfe00640 1 -- 172.21.15.32:0/2636338907 shutdown_connections 2024-04-30T08:31:34.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:34.406+0000 7f02bfe00640 1 -- 172.21.15.32:0/2636338907 wait complete. 2024-04-30T08:31:35.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:34 smithi189 ceph-mon[33038]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:35.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:34 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] up:boot 2024-04-30T08:31:35.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:34 smithi189 ceph-mon[33038]: fsmap 9 up:standby 2024-04-30T08:31:35.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:34 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-30T08:31:35.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:34 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:35.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:34 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:35.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:34 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:35.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:34 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:35.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:34 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:35.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:34 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:35.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:34 smithi189 ceph-mon[33038]: Deploying daemon mds.g on smithi032 2024-04-30T08:31:35.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:34 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2636338907' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:35.078 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:34 smithi032 ceph-mon[28331]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:35.078 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:34 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] up:boot 2024-04-30T08:31:35.078 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:34 smithi032 ceph-mon[28331]: fsmap 9 up:standby 2024-04-30T08:31:35.078 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:34 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-30T08:31:35.078 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:34 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:35.079 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:34 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:35.079 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:34 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:35.079 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:34 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:35.079 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:34 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:35.079 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:34 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:35.079 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:34 smithi032 ceph-mon[28331]: Deploying daemon mds.g on smithi032 2024-04-30T08:31:35.079 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:34 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2636338907' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:35.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:34 smithi071 ceph-mon[33385]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:35.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:34 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] up:boot 2024-04-30T08:31:35.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:34 smithi071 ceph-mon[33385]: fsmap 9 up:standby 2024-04-30T08:31:35.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:34 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-30T08:31:35.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:34 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:35.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:34 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:35.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:34 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:35.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:34 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:35.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:34 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:35.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:34 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:35.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:34 smithi071 ceph-mon[33385]: Deploying daemon mds.g on smithi032 2024-04-30T08:31:35.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:34 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2636338907' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:36.438 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-30T08:31:36.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.980+0000 7ff19d000640 1 -- 172.21.15.32:0/3695435233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff198103990 msgr2=0x7ff198104690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:36.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.980+0000 7ff19d000640 1 --2- 172.21.15.32:0/3695435233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff198103990 0x7ff198104690 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff188009900 tx=0x7ff18802a3f0 comp rx=0 tx=0).stop 2024-04-30T08:31:36.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.980+0000 7ff19d000640 1 -- 172.21.15.32:0/3695435233 shutdown_connections 2024-04-30T08:31:36.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.980+0000 7ff19d000640 1 --2- 172.21.15.32:0/3695435233 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff1981098d0 0x7ff19810bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:36.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.980+0000 7ff19d000640 1 --2- 172.21.15.32:0/3695435233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff198104bd0 0x7ff198104fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:36.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.980+0000 7ff19d000640 1 --2- 172.21.15.32:0/3695435233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff198103990 0x7ff198104690 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:36.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.980+0000 7ff19d000640 1 -- 172.21.15.32:0/3695435233 >> 172.21.15.32:0/3695435233 conn(0x7ff1980feba0 msgr2=0x7ff198101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:36.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.980+0000 7ff19d000640 1 -- 172.21.15.32:0/3695435233 shutdown_connections 2024-04-30T08:31:36.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.980+0000 7ff19d000640 1 -- 172.21.15.32:0/3695435233 wait complete. 2024-04-30T08:31:36.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.982+0000 7ff19d000640 1 Processor -- start 2024-04-30T08:31:36.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.982+0000 7ff19d000640 1 -- start start 2024-04-30T08:31:36.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.982+0000 7ff19d000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff198103990 0x7ff19819c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:36.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.982+0000 7ff19d000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff198104bd0 0x7ff19819c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:36.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.982+0000 7ff19d000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1981098d0 0x7ff1981a37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:36.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.983+0000 7ff19d000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff19810f4e0 con 0x7ff1981098d0 2024-04-30T08:31:36.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.983+0000 7ff19d000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff19810f360 con 0x7ff198104bd0 2024-04-30T08:31:36.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.983+0000 7ff19d000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff19810f660 con 0x7ff198103990 2024-04-30T08:31:36.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.983+0000 7ff197400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff198103990 0x7ff19819c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:36.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.983+0000 7ff197400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff198103990 0x7ff19819c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:49756/0 (socket says 172.21.15.32:49756) 2024-04-30T08:31:36.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.983+0000 7ff197400640 1 -- 172.21.15.32:0/3371944769 learned_addr learned my addr 172.21.15.32:0/3371944769 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:31:36.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.983+0000 7ff197e00640 1 --2- 172.21.15.32:0/3371944769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1981098d0 0x7ff1981a37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:36.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.983+0000 7ff18fe00640 1 --2- 172.21.15.32:0/3371944769 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff198104bd0 0x7ff19819c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:36.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.983+0000 7ff197400640 1 -- 172.21.15.32:0/3371944769 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff198104bd0 msgr2=0x7ff19819c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:36.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.983+0000 7ff197400640 1 --2- 172.21.15.32:0/3371944769 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff198104bd0 0x7ff19819c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:36.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.983+0000 7ff197400640 1 -- 172.21.15.32:0/3371944769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1981098d0 msgr2=0x7ff1981a37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:36.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.983+0000 7ff197400640 1 --2- 172.21.15.32:0/3371944769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1981098d0 0x7ff1981a37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:36.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.983+0000 7ff197400640 1 -- 172.21.15.32:0/3371944769 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1981a3e10 con 0x7ff198103990 2024-04-30T08:31:36.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.984+0000 7ff197400640 1 --2- 172.21.15.32:0/3371944769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff198103990 0x7ff19819c2d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff188031440 tx=0x7ff188003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:36.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.984+0000 7ff194c00640 1 -- 172.21.15.32:0/3371944769 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff18803f070 con 0x7ff198103990 2024-04-30T08:31:36.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.984+0000 7ff194c00640 1 -- 172.21.15.32:0/3371944769 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff188031c50 con 0x7ff198103990 2024-04-30T08:31:36.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.984+0000 7ff194c00640 1 -- 172.21.15.32:0/3371944769 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff188035030 con 0x7ff198103990 2024-04-30T08:31:36.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.984+0000 7ff19d000640 1 -- 172.21.15.32:0/3371944769 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1981a40a0 con 0x7ff198103990 2024-04-30T08:31:36.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.984+0000 7ff19d000640 1 -- 172.21.15.32:0/3371944769 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1981a4540 con 0x7ff198103990 2024-04-30T08:31:36.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.985+0000 7ff19d000640 1 -- 172.21.15.32:0/3371944769 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1981a47a0 con 0x7ff198103990 2024-04-30T08:31:36.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.990+0000 7ff194c00640 1 -- 172.21.15.32:0/3371944769 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7ff188042070 con 0x7ff198103990 2024-04-30T08:31:36.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.991+0000 7ff194c00640 1 --2- 172.21.15.32:0/3371944769 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff17c0afb60 0x7ff17c0b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:36.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.991+0000 7ff194c00640 1 -- 172.21.15.32:0/3371944769 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7ff1880ff670 con 0x7ff198103990 2024-04-30T08:31:36.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.992+0000 7ff18fe00640 1 --2- 172.21.15.32:0/3371944769 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff17c0afb60 0x7ff17c0b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:36.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.992+0000 7ff194c00640 1 -- 172.21.15.32:0/3371944769 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff1880c8000 con 0x7ff198103990 2024-04-30T08:31:36.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:36.992+0000 7ff18fe00640 1 --2- 172.21.15.32:0/3371944769 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff17c0afb60 0x7ff17c0b2070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff19819d370 tx=0x7ff180005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:37.068 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:36 smithi189 ceph-mon[33038]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:37.084 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:36 smithi032 ceph-mon[28331]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:37.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:36 smithi071 ceph-mon[33385]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:37.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:37.255+0000 7ff19d000640 1 -- 172.21.15.32:0/3371944769 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff1981a4970 con 0x7ff198103990 2024-04-30T08:31:37.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:37.256+0000 7ff194c00640 1 -- 172.21.15.32:0/3371944769 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+7995 (secure 0 0 0) 0x7ff1880cc300 con 0x7ff198103990 2024-04-30T08:31:37.258 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:31:37.258 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2235842676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2235842676},{"type":"v1","addr":"172.21.15.32:6835","nonce":2235842676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/463203038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":463203038},{"type":"v1","addr":"172.21.15.189:6833","nonce":463203038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3635143591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3635143591},{"type":"v1","addr":"172.21.15.71:6833","nonce":3635143591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24484,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2569449994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2569449994},{"type":"v1","addr":"172.21.15.32:6837","nonce":2569449994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/278686565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":278686565},{"type":"v1","addr":"172.21.15.189:6835","nonce":278686565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6835/1558303056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1558303056},{"type":"v1","addr":"172.21.15.71:6835","nonce":1558303056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/686520873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":686520873},{"type":"v1","addr":"172.21.15.189:6837","nonce":686520873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/2324039932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2324039932},{"type":"v1","addr":"172.21.15.71:6837","nonce":2324039932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6839/352970107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":352970107},{"type":"v1","addr":"172.21.15.189:6839","nonce":352970107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-30T08:31:37.258 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 10 2024-04-30T08:31:37.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:37.259+0000 7ff19d000640 1 -- 172.21.15.32:0/3371944769 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff17c0afb60 msgr2=0x7ff17c0b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:37.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:37.259+0000 7ff19d000640 1 --2- 172.21.15.32:0/3371944769 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff17c0afb60 0x7ff17c0b2070 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7ff19819d370 tx=0x7ff180005f90 comp rx=0 tx=0).stop 2024-04-30T08:31:37.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:37.259+0000 7ff19d000640 1 -- 172.21.15.32:0/3371944769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff198103990 msgr2=0x7ff19819c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:37.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:37.259+0000 7ff19d000640 1 --2- 172.21.15.32:0/3371944769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff198103990 0x7ff19819c2d0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff188031440 tx=0x7ff188003c10 comp rx=0 tx=0).stop 2024-04-30T08:31:37.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:37.259+0000 7ff19d000640 1 -- 172.21.15.32:0/3371944769 shutdown_connections 2024-04-30T08:31:37.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:37.259+0000 7ff19d000640 1 --2- 172.21.15.32:0/3371944769 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff17c0afb60 0x7ff17c0b2070 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:37.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:37.259+0000 7ff19d000640 1 --2- 172.21.15.32:0/3371944769 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff1981098d0 0x7ff1981a37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:37.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:37.259+0000 7ff19d000640 1 --2- 172.21.15.32:0/3371944769 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff198104bd0 0x7ff19819c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:37.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:37.259+0000 7ff19d000640 1 --2- 172.21.15.32:0/3371944769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff198103990 0x7ff19819c2d0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:37.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:37.259+0000 7ff19d000640 1 -- 172.21.15.32:0/3371944769 >> 172.21.15.32:0/3371944769 conn(0x7ff1980feba0 msgr2=0x7ff1980ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:37.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:37.259+0000 7ff19d000640 1 -- 172.21.15.32:0/3371944769 shutdown_connections 2024-04-30T08:31:37.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:37.259+0000 7ff19d000640 1 -- 172.21.15.32:0/3371944769 wait complete. 2024-04-30T08:31:37.806 INFO:journalctl@ceph.mds.g.smithi032.stdout:Apr 30 08:31:37 smithi032 podman[70094]: 2024-04-30 08:31:37.525857517 +0000 UTC m=+0.145926207 container create 42dc38ed484fb3750e5dddc1f73258edbee51de89147ecec3dced15d223e9684 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-g, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-30T08:31:38.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:37 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3371944769' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:38.191 INFO:journalctl@ceph.mds.g.smithi032.stdout:Apr 30 08:31:37 smithi032 podman[70094]: 2024-04-30 08:31:37.813483858 +0000 UTC m=+0.433552551 container init 42dc38ed484fb3750e5dddc1f73258edbee51de89147ecec3dced15d223e9684 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-g, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.build-date=20240430, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-30T08:31:38.191 INFO:journalctl@ceph.mds.g.smithi032.stdout:Apr 30 08:31:37 smithi032 podman[70094]: 2024-04-30 08:31:37.824761197 +0000 UTC m=+0.444829890 container start 42dc38ed484fb3750e5dddc1f73258edbee51de89147ecec3dced15d223e9684 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-g, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-30T08:31:38.192 INFO:journalctl@ceph.mds.g.smithi032.stdout:Apr 30 08:31:37 smithi032 bash[70094]: 42dc38ed484fb3750e5dddc1f73258edbee51de89147ecec3dced15d223e9684 2024-04-30T08:31:38.192 INFO:journalctl@ceph.mds.g.smithi032.stdout:Apr 30 08:31:37 smithi032 systemd[1]: Started Ceph mds.g for 98c0a670-06ca-11ef-bc93-c7b262605968. 2024-04-30T08:31:38.192 INFO:journalctl@ceph.mds.g.smithi032.stdout:Apr 30 08:31:37 smithi032 ceph-mds[70113]: -- 172.21.15.32:0/1235121555 <== mon.0 v2:172.21.15.32:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556f47ca1a40 con 0x556f47ce9180 2024-04-30T08:31:38.192 INFO:journalctl@ceph.mds.g.smithi032.stdout:Apr 30 08:31:37 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-g[70109]: starting mds.g at 2024-04-30T08:31:38.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:37 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3371944769' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:38.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:37 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3371944769' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:39.290 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:39 smithi032 ceph-mon[28331]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:39.290 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:39 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:39.290 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:39 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:39.290 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:39 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:39.290 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:39 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:39.291 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:39 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:39.291 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:39 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:39.291 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:39 smithi032 ceph-mon[28331]: Deploying daemon mds.a on smithi032 2024-04-30T08:31:39.291 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:39 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] up:boot 2024-04-30T08:31:39.291 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:39 smithi032 ceph-mon[28331]: fsmap 10 up:standby 2024-04-30T08:31:39.291 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:39 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-30T08:31:39.291 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-30T08:31:39.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:39 smithi189 ceph-mon[33038]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:39.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:39 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:39.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:39 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:39.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:39 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:39.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:39 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:39.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:39 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:39.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:39 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:39.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:39 smithi189 ceph-mon[33038]: Deploying daemon mds.a on smithi032 2024-04-30T08:31:39.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:39 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] up:boot 2024-04-30T08:31:39.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:39 smithi189 ceph-mon[33038]: fsmap 10 up:standby 2024-04-30T08:31:39.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:39 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-30T08:31:39.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:39 smithi071 ceph-mon[33385]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:39.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:39 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:39.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:39 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:39.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:39 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:39.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:39 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:39.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:39 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:39.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:39 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:39.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:39 smithi071 ceph-mon[33385]: Deploying daemon mds.a on smithi032 2024-04-30T08:31:39.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:39 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] up:boot 2024-04-30T08:31:39.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:39 smithi071 ceph-mon[33385]: fsmap 10 up:standby 2024-04-30T08:31:39.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:39 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-30T08:31:39.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.573+0000 7fc826400640 1 -- 172.21.15.32:0/432856976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc82010b920 msgr2=0x7fc82010dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:39.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.573+0000 7fc826400640 1 --2- 172.21.15.32:0/432856976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc82010b920 0x7fc82010dd60 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fc818009be0 tx=0x7fc81802a6c0 comp rx=0 tx=0).stop 2024-04-30T08:31:39.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.573+0000 7fc826400640 1 -- 172.21.15.32:0/432856976 shutdown_connections 2024-04-30T08:31:39.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.573+0000 7fc826400640 1 --2- 172.21.15.32:0/432856976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc82010b920 0x7fc82010dd60 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:39.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.573+0000 7fc826400640 1 --2- 172.21.15.32:0/432856976 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc820075960 0x7fc82010b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:39.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.573+0000 7fc826400640 1 --2- 172.21.15.32:0/432856976 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc820075010 0x7fc820075420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:39.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.573+0000 7fc826400640 1 -- 172.21.15.32:0/432856976 >> 172.21.15.32:0/432856976 conn(0x7fc8200feba0 msgr2=0x7fc820101050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:39.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.573+0000 7fc826400640 1 -- 172.21.15.32:0/432856976 shutdown_connections 2024-04-30T08:31:39.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.573+0000 7fc826400640 1 -- 172.21.15.32:0/432856976 wait complete. 2024-04-30T08:31:39.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.574+0000 7fc826400640 1 Processor -- start 2024-04-30T08:31:39.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.575+0000 7fc826400640 1 -- start start 2024-04-30T08:31:39.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.575+0000 7fc826400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc820075010 0x7fc82019c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:39.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.575+0000 7fc826400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc820075960 0x7fc82019c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:39.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.575+0000 7fc826400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc82010b920 0x7fc8201a3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:39.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.575+0000 7fc826400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc8201116e0 con 0x7fc820075960 2024-04-30T08:31:39.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.575+0000 7fc826400640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc820111560 con 0x7fc82010b920 2024-04-30T08:31:39.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.575+0000 7fc826400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc820111860 con 0x7fc820075010 2024-04-30T08:31:39.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.575+0000 7fc825000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc820075010 0x7fc82019c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:39.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.575+0000 7fc81fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc820075960 0x7fc82019c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:39.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.575+0000 7fc825000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc820075010 0x7fc82019c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:49804/0 (socket says 172.21.15.32:49804) 2024-04-30T08:31:39.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.575+0000 7fc825000640 1 -- 172.21.15.32:0/79072836 learned_addr learned my addr 172.21.15.32:0/79072836 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:31:39.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.576+0000 7fc825a00640 1 --2- 172.21.15.32:0/79072836 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc82010b920 0x7fc8201a3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:39.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.576+0000 7fc825000640 1 -- 172.21.15.32:0/79072836 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc82010b920 msgr2=0x7fc8201a3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:39.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.576+0000 7fc825000640 1 --2- 172.21.15.32:0/79072836 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc82010b920 0x7fc8201a3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:39.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.576+0000 7fc825000640 1 -- 172.21.15.32:0/79072836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc820075960 msgr2=0x7fc82019c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:39.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.576+0000 7fc825000640 1 --2- 172.21.15.32:0/79072836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc820075960 0x7fc82019c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:39.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.576+0000 7fc825000640 1 -- 172.21.15.32:0/79072836 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8201a3e00 con 0x7fc820075010 2024-04-30T08:31:39.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.576+0000 7fc825000640 1 --2- 172.21.15.32:0/79072836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc820075010 0x7fc82019c310 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc81400e9f0 tx=0x7fc81400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:39.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.576+0000 7fc81d600640 1 -- 172.21.15.32:0/79072836 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc81400ccc0 con 0x7fc820075010 2024-04-30T08:31:39.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.576+0000 7fc81d600640 1 -- 172.21.15.32:0/79072836 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc8140054d0 con 0x7fc820075010 2024-04-30T08:31:39.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.576+0000 7fc81d600640 1 -- 172.21.15.32:0/79072836 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc814010430 con 0x7fc820075010 2024-04-30T08:31:39.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.576+0000 7fc826400640 1 -- 172.21.15.32:0/79072836 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc8201a40f0 con 0x7fc820075010 2024-04-30T08:31:39.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.577+0000 7fc826400640 1 -- 172.21.15.32:0/79072836 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8201a45f0 con 0x7fc820075010 2024-04-30T08:31:39.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.578+0000 7fc826400640 1 -- 172.21.15.32:0/79072836 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc82010a1f0 con 0x7fc820075010 2024-04-30T08:31:39.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.580+0000 7fc81d600640 1 -- 172.21.15.32:0/79072836 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fc8140046c0 con 0x7fc820075010 2024-04-30T08:31:39.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.582+0000 7fc81d600640 1 --2- 172.21.15.32:0/79072836 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc8080afae0 0x7fc8080b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:39.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.582+0000 7fc81d600640 1 -- 172.21.15.32:0/79072836 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7fc8140ded60 con 0x7fc820075010 2024-04-30T08:31:39.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.582+0000 7fc81fe00640 1 --2- 172.21.15.32:0/79072836 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc8080afae0 0x7fc8080b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:39.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.583+0000 7fc81fe00640 1 --2- 172.21.15.32:0/79072836 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc8080afae0 0x7fc8080b1ff0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc80c0096c0 tx=0x7fc80c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:39.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.584+0000 7fc81d600640 1 -- 172.21.15.32:0/79072836 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc814053ff0 con 0x7fc820075010 2024-04-30T08:31:39.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.838+0000 7fc826400640 1 -- 172.21.15.32:0/79072836 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc82002dbd0 con 0x7fc820075010 2024-04-30T08:31:39.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.839+0000 7fc81d600640 1 -- 172.21.15.32:0/79072836 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8821 (secure 0 0 0) 0x7fc8140a7d40 con 0x7fc820075010 2024-04-30T08:31:39.841 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:31:39.841 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2235842676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2235842676},{"type":"v1","addr":"172.21.15.32:6835","nonce":2235842676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/463203038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":463203038},{"type":"v1","addr":"172.21.15.189:6833","nonce":463203038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3635143591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3635143591},{"type":"v1","addr":"172.21.15.71:6833","nonce":3635143591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24484,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2569449994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2569449994},{"type":"v1","addr":"172.21.15.32:6837","nonce":2569449994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/278686565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":278686565},{"type":"v1","addr":"172.21.15.189:6835","nonce":278686565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6835/1558303056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1558303056},{"type":"v1","addr":"172.21.15.71:6835","nonce":1558303056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/686520873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":686520873},{"type":"v1","addr":"172.21.15.189:6837","nonce":686520873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/2324039932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2324039932},{"type":"v1","addr":"172.21.15.71:6837","nonce":2324039932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6839/352970107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":352970107},{"type":"v1","addr":"172.21.15.189:6839","nonce":352970107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24560,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/4160386524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4160386524},{"type":"v1","addr":"172.21.15.32:6839","nonce":4160386524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-04-30T08:31:39.842 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 11 2024-04-30T08:31:39.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.842+0000 7fc826400640 1 -- 172.21.15.32:0/79072836 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc8080afae0 msgr2=0x7fc8080b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:39.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.842+0000 7fc826400640 1 --2- 172.21.15.32:0/79072836 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc8080afae0 0x7fc8080b1ff0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc80c0096c0 tx=0x7fc80c009260 comp rx=0 tx=0).stop 2024-04-30T08:31:39.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.842+0000 7fc826400640 1 -- 172.21.15.32:0/79072836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc820075010 msgr2=0x7fc82019c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:39.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.842+0000 7fc826400640 1 --2- 172.21.15.32:0/79072836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc820075010 0x7fc82019c310 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fc81400e9f0 tx=0x7fc81400eeb0 comp rx=0 tx=0).stop 2024-04-30T08:31:39.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.842+0000 7fc826400640 1 -- 172.21.15.32:0/79072836 shutdown_connections 2024-04-30T08:31:39.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.842+0000 7fc826400640 1 --2- 172.21.15.32:0/79072836 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc8080afae0 0x7fc8080b1ff0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:39.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.842+0000 7fc826400640 1 --2- 172.21.15.32:0/79072836 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc82010b920 0x7fc8201a3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:39.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.842+0000 7fc826400640 1 --2- 172.21.15.32:0/79072836 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc820075960 0x7fc82019c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:39.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.842+0000 7fc826400640 1 --2- 172.21.15.32:0/79072836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc820075010 0x7fc82019c310 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:39.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.842+0000 7fc826400640 1 -- 172.21.15.32:0/79072836 >> 172.21.15.32:0/79072836 conn(0x7fc8200feba0 msgr2=0x7fc8201073c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:39.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.842+0000 7fc826400640 1 -- 172.21.15.32:0/79072836 shutdown_connections 2024-04-30T08:31:39.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:39.843+0000 7fc826400640 1 -- 172.21.15.32:0/79072836 wait complete. 2024-04-30T08:31:40.132 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:40 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/79072836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:40.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:40 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/79072836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:40.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:40 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/79072836' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:41.180 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:41 smithi032 ceph-mon[28331]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:41.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:41 smithi189 ceph-mon[33038]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:41.441 INFO:journalctl@ceph.mds.a.smithi032.stdout:Apr 30 08:31:41 smithi032 systemd[1]: Starting Ceph mds.a for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:31:41.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:41 smithi071 ceph-mon[33385]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:41.872 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-30T08:31:41.890 INFO:journalctl@ceph.mds.a.smithi032.stdout:Apr 30 08:31:41 smithi032 podman[70441]: 2024-04-30 08:31:41.461936802 +0000 UTC m=+0.036138772 image pull 16056ace0edc7f73516cf28297f1db04dda872801fb5b17b9157b0d1ccf38f9b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de 2024-04-30T08:31:41.890 INFO:journalctl@ceph.mds.a.smithi032.stdout:Apr 30 08:31:41 smithi032 podman[70441]: 2024-04-30 08:31:41.590018032 +0000 UTC m=+0.164220030 container create 5234a090b9bb0665b92884a9b6a46f6b6c14473dabc3087360376d433fc12e69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-a, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, ceph=True) 2024-04-30T08:31:41.890 INFO:journalctl@ceph.mds.a.smithi032.stdout:Apr 30 08:31:41 smithi032 podman[70441]: 2024-04-30 08:31:41.886901069 +0000 UTC m=+0.461103071 container init 5234a090b9bb0665b92884a9b6a46f6b6c14473dabc3087360376d433fc12e69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-a, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-30T08:31:42.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.172+0000 7f6953200640 1 -- 172.21.15.32:0/3255236745 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6954103ac0 msgr2=0x7f6954103ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:42.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.172+0000 7f6953200640 1 --2- 172.21.15.32:0/3255236745 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6954103ac0 0x7f6954103ed0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f693c009900 tx=0x7f693c02a410 comp rx=0 tx=0).stop 2024-04-30T08:31:42.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.172+0000 7f6953200640 1 -- 172.21.15.32:0/3255236745 shutdown_connections 2024-04-30T08:31:42.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.172+0000 7f6953200640 1 --2- 172.21.15.32:0/3255236745 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f69541087c0 0x7f695410abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:42.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.172+0000 7f6953200640 1 --2- 172.21.15.32:0/3255236745 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6954103ac0 0x7f6954103ed0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:42.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.172+0000 7f6953200640 1 --2- 172.21.15.32:0/3255236745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6954102880 0x7f6954103580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:42.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.172+0000 7f6953200640 1 -- 172.21.15.32:0/3255236745 >> 172.21.15.32:0/3255236745 conn(0x7f69540fda70 msgr2=0x7f69540fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:42.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.172+0000 7f6953200640 1 -- 172.21.15.32:0/3255236745 shutdown_connections 2024-04-30T08:31:42.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.172+0000 7f6953200640 1 -- 172.21.15.32:0/3255236745 wait complete. 2024-04-30T08:31:42.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.173+0000 7f6953200640 1 Processor -- start 2024-04-30T08:31:42.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.173+0000 7f6953200640 1 -- start start 2024-04-30T08:31:42.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.174+0000 7f6953200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6954102880 0x7f695419a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:42.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.174+0000 7f6953200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6954103ac0 0x7f695419a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:42.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.174+0000 7f6953200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69541087c0 0x7f69541a1580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:42.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.174+0000 7f6953200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f695410f4b0 con 0x7f6954103ac0 2024-04-30T08:31:42.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.174+0000 7f6953200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f695410f330 con 0x7f6954102880 2024-04-30T08:31:42.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.174+0000 7f6953200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f695410f630 con 0x7f69541087c0 2024-04-30T08:31:42.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.174+0000 7f6951e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6954102880 0x7f695419a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:42.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.174+0000 7f694be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6954103ac0 0x7f695419a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:42.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.174+0000 7f6951e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6954102880 0x7f695419a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:57518/0 (socket says 172.21.15.32:57518) 2024-04-30T08:31:42.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.174+0000 7f6951e00640 1 -- 172.21.15.32:0/2609574188 learned_addr learned my addr 172.21.15.32:0/2609574188 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:31:42.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.174+0000 7f6952800640 1 --2- 172.21.15.32:0/2609574188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69541087c0 0x7f69541a1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:42.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.175+0000 7f6951e00640 1 -- 172.21.15.32:0/2609574188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69541087c0 msgr2=0x7f69541a1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:42.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.175+0000 7f6951e00640 1 --2- 172.21.15.32:0/2609574188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69541087c0 0x7f69541a1580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:42.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.175+0000 7f6951e00640 1 -- 172.21.15.32:0/2609574188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6954103ac0 msgr2=0x7f695419a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:42.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.175+0000 7f6951e00640 1 --2- 172.21.15.32:0/2609574188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6954103ac0 0x7f695419a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:42.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.175+0000 7f6951e00640 1 -- 172.21.15.32:0/2609574188 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69541a1bf0 con 0x7f6954102880 2024-04-30T08:31:42.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.175+0000 7f6952800640 1 --2- 172.21.15.32:0/2609574188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69541087c0 0x7f69541a1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:31:42.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.175+0000 7f6951e00640 1 --2- 172.21.15.32:0/2609574188 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6954102880 0x7f695419a120 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f694400e9f0 tx=0x7f694400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:42.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.175+0000 7f694aa00640 1 -- 172.21.15.32:0/2609574188 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f694400ccc0 con 0x7f6954102880 2024-04-30T08:31:42.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.175+0000 7f694aa00640 1 -- 172.21.15.32:0/2609574188 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f69440035f0 con 0x7f6954102880 2024-04-30T08:31:42.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.175+0000 7f694aa00640 1 -- 172.21.15.32:0/2609574188 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f694400ce60 con 0x7f6954102880 2024-04-30T08:31:42.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.175+0000 7f6953200640 1 -- 172.21.15.32:0/2609574188 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69541a1ee0 con 0x7f6954102880 2024-04-30T08:31:42.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.175+0000 7f6953200640 1 -- 172.21.15.32:0/2609574188 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69541a22f0 con 0x7f6954102880 2024-04-30T08:31:42.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.177+0000 7f6953200640 1 -- 172.21.15.32:0/2609574188 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6954069ca0 con 0x7f6954102880 2024-04-30T08:31:42.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.179+0000 7f694aa00640 1 -- 172.21.15.32:0/2609574188 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f6944003aa0 con 0x7f6954102880 2024-04-30T08:31:42.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.180+0000 7f694aa00640 1 --2- 172.21.15.32:0/2609574188 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f69380afb60 0x7f69380b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:42.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.181+0000 7f694aa00640 1 -- 172.21.15.32:0/2609574188 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f69440deb50 con 0x7f6954102880 2024-04-30T08:31:42.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.181+0000 7f694be00640 1 --2- 172.21.15.32:0/2609574188 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f69380afb60 0x7f69380b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:42.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.182+0000 7f694be00640 1 --2- 172.21.15.32:0/2609574188 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f69380afb60 0x7f69380b2070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f693c009230 tx=0x7f693c031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:42.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.185+0000 7f694aa00640 1 -- 172.21.15.32:0/2609574188 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f69440a7b30 con 0x7f6954102880 2024-04-30T08:31:42.191 INFO:journalctl@ceph.mds.a.smithi032.stdout:Apr 30 08:31:41 smithi032 podman[70441]: 2024-04-30 08:31:41.899154357 +0000 UTC m=+0.473356322 container start 5234a090b9bb0665b92884a9b6a46f6b6c14473dabc3087360376d433fc12e69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-30T08:31:42.191 INFO:journalctl@ceph.mds.a.smithi032.stdout:Apr 30 08:31:41 smithi032 bash[70441]: 5234a090b9bb0665b92884a9b6a46f6b6c14473dabc3087360376d433fc12e69 2024-04-30T08:31:42.191 INFO:journalctl@ceph.mds.a.smithi032.stdout:Apr 30 08:31:42 smithi032 systemd[1]: Started Ceph mds.a for 98c0a670-06ca-11ef-bc93-c7b262605968. 2024-04-30T08:31:42.191 INFO:journalctl@ceph.mds.a.smithi032.stdout:Apr 30 08:31:42 smithi032 ceph-mds[70465]: -- 172.21.15.32:0/2272387922 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5608267c7a40 con 0x56082680f600 2024-04-30T08:31:42.191 INFO:journalctl@ceph.mds.a.smithi032.stdout:Apr 30 08:31:42 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-a[70456]: starting mds.a at 2024-04-30T08:31:42.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.438+0000 7f6953200640 1 -- 172.21.15.32:0/2609574188 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f69540638c0 con 0x7f6954102880 2024-04-30T08:31:42.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.439+0000 7f694aa00640 1 -- 172.21.15.32:0/2609574188 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8821 (secure 0 0 0) 0x7f69440a7430 con 0x7f6954102880 2024-04-30T08:31:42.441 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:31:42.441 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2235842676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2235842676},{"type":"v1","addr":"172.21.15.32:6835","nonce":2235842676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/463203038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":463203038},{"type":"v1","addr":"172.21.15.189:6833","nonce":463203038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3635143591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3635143591},{"type":"v1","addr":"172.21.15.71:6833","nonce":3635143591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24484,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2569449994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2569449994},{"type":"v1","addr":"172.21.15.32:6837","nonce":2569449994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/278686565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":278686565},{"type":"v1","addr":"172.21.15.189:6835","nonce":278686565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6835/1558303056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1558303056},{"type":"v1","addr":"172.21.15.71:6835","nonce":1558303056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/686520873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":686520873},{"type":"v1","addr":"172.21.15.189:6837","nonce":686520873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/2324039932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2324039932},{"type":"v1","addr":"172.21.15.71:6837","nonce":2324039932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6839/352970107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":352970107},{"type":"v1","addr":"172.21.15.189:6839","nonce":352970107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24560,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/4160386524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4160386524},{"type":"v1","addr":"172.21.15.32:6839","nonce":4160386524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-04-30T08:31:42.441 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 11 2024-04-30T08:31:42.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.442+0000 7f6953200640 1 -- 172.21.15.32:0/2609574188 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f69380afb60 msgr2=0x7f69380b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:42.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.442+0000 7f6953200640 1 --2- 172.21.15.32:0/2609574188 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f69380afb60 0x7f69380b2070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f693c009230 tx=0x7f693c031400 comp rx=0 tx=0).stop 2024-04-30T08:31:42.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.442+0000 7f6953200640 1 -- 172.21.15.32:0/2609574188 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6954102880 msgr2=0x7f695419a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:42.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.442+0000 7f6953200640 1 --2- 172.21.15.32:0/2609574188 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6954102880 0x7f695419a120 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f694400e9f0 tx=0x7f694400eeb0 comp rx=0 tx=0).stop 2024-04-30T08:31:42.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.442+0000 7f6953200640 1 -- 172.21.15.32:0/2609574188 shutdown_connections 2024-04-30T08:31:42.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.442+0000 7f6953200640 1 --2- 172.21.15.32:0/2609574188 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f69380afb60 0x7f69380b2070 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:42.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.442+0000 7f6953200640 1 --2- 172.21.15.32:0/2609574188 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f69541087c0 0x7f69541a1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:42.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.442+0000 7f6953200640 1 --2- 172.21.15.32:0/2609574188 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6954103ac0 0x7f695419a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:42.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.442+0000 7f6953200640 1 --2- 172.21.15.32:0/2609574188 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6954102880 0x7f695419a120 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:42.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.442+0000 7f6953200640 1 -- 172.21.15.32:0/2609574188 >> 172.21.15.32:0/2609574188 conn(0x7f69540fda70 msgr2=0x7f69541047a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:42.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.443+0000 7f6953200640 1 -- 172.21.15.32:0/2609574188 shutdown_connections 2024-04-30T08:31:42.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:42.443+0000 7f6953200640 1 -- 172.21.15.32:0/2609574188 wait complete. 2024-04-30T08:31:43.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:43 smithi032 ceph-mon[28331]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:43.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:43 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:43.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:43 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:43.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:43 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:43.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:43 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:43.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:43 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:43.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:43 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:43.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:43 smithi032 ceph-mon[28331]: Deploying daemon mds.b on smithi071 2024-04-30T08:31:43.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:43 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2609574188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:43.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:43 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.32:6840/3577009355,v1:172.21.15.32:6841/3577009355] up:boot 2024-04-30T08:31:43.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:43 smithi032 ceph-mon[28331]: fsmap 11 up:standby 2024-04-30T08:31:43.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:43 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-30T08:31:43.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:43 smithi071 ceph-mon[33385]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:43.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:43 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:43.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:43 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:43.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:43 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:43.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:43 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:43.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:43 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:43.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:43 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:43.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:43 smithi071 ceph-mon[33385]: Deploying daemon mds.b on smithi071 2024-04-30T08:31:43.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:43 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2609574188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:43.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:43 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.32:6840/3577009355,v1:172.21.15.32:6841/3577009355] up:boot 2024-04-30T08:31:43.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:43 smithi071 ceph-mon[33385]: fsmap 11 up:standby 2024-04-30T08:31:43.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:43 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-30T08:31:43.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:43 smithi189 ceph-mon[33038]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:43.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:43 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:43.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:43 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:43.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:43 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:43.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:43 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-30T08:31:43.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:43 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-30T08:31:43.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:43 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:43.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:43 smithi189 ceph-mon[33038]: Deploying daemon mds.b on smithi071 2024-04-30T08:31:43.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:43 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2609574188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:43.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:43 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.32:6840/3577009355,v1:172.21.15.32:6841/3577009355] up:boot 2024-04-30T08:31:43.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:43 smithi189 ceph-mon[33038]: fsmap 11 up:standby 2024-04-30T08:31:43.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:43 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-30T08:31:44.471 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-30T08:31:44.504 INFO:journalctl@ceph.mds.b.smithi071.stdout:Apr 30 08:31:44 smithi071 systemd[1]: Starting Ceph mds.b for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:31:44.504 INFO:journalctl@ceph.mds.b.smithi071.stdout:Apr 30 08:31:44 smithi071 podman[64012]: 2024-04-30 08:31:44.405092788 +0000 UTC m=+0.017701613 image pull 16056ace0edc7f73516cf28297f1db04dda872801fb5b17b9157b0d1ccf38f9b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de 2024-04-30T08:31:44.753 INFO:journalctl@ceph.mds.b.smithi071.stdout:Apr 30 08:31:44 smithi071 podman[64012]: 2024-04-30 08:31:44.523742752 +0000 UTC m=+0.136351574 container create 562a91b76601bfa2f3e437103889bab3af607fe26861f7b526e61f2dd21912f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-b, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-30T08:31:44.754 INFO:journalctl@ceph.mds.b.smithi071.stdout:Apr 30 08:31:44 smithi071 podman[64012]: 2024-04-30 08:31:44.751489366 +0000 UTC m=+0.364098187 container init 562a91b76601bfa2f3e437103889bab3af607fe26861f7b526e61f2dd21912f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-04-30T08:31:44.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.767+0000 7f1546400640 1 -- 172.21.15.32:0/3022847783 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f154010a160 msgr2=0x7f154010a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:44.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.767+0000 7f1546400640 1 --2- 172.21.15.32:0/3022847783 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f154010a160 0x7f154010a570 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f1530009930 tx=0x7f153002a420 comp rx=0 tx=0).stop 2024-04-30T08:31:44.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.767+0000 7f1546400640 1 -- 172.21.15.32:0/3022847783 shutdown_connections 2024-04-30T08:31:44.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.767+0000 7f1546400640 1 --2- 172.21.15.32:0/3022847783 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f154010bbd0 0x7f15401135e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:44.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.767+0000 7f1546400640 1 --2- 172.21.15.32:0/3022847783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f154010b280 0x7f154010b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:44.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.767+0000 7f1546400640 1 --2- 172.21.15.32:0/3022847783 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f154010a160 0x7f154010a570 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:44.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.767+0000 7f1546400640 1 -- 172.21.15.32:0/3022847783 >> 172.21.15.32:0/3022847783 conn(0x7f15400709a0 msgr2=0x7f1540072e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:44.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.767+0000 7f1546400640 1 -- 172.21.15.32:0/3022847783 shutdown_connections 2024-04-30T08:31:44.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.767+0000 7f1546400640 1 -- 172.21.15.32:0/3022847783 wait complete. 2024-04-30T08:31:44.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.768+0000 7f1546400640 1 Processor -- start 2024-04-30T08:31:44.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.768+0000 7f1546400640 1 -- start start 2024-04-30T08:31:44.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.768+0000 7f1546400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f154010a160 0x7f15401a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:44.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.768+0000 7f1546400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f154010b280 0x7f15401a5280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:44.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.769+0000 7f1546400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f154010bbd0 0x7f15401ac1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:44.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.769+0000 7f1546400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1540116dc0 con 0x7f154010a160 2024-04-30T08:31:44.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.769+0000 7f1546400640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1540116c40 con 0x7f154010b280 2024-04-30T08:31:44.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.769+0000 7f1546400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1540116f40 con 0x7f154010bbd0 2024-04-30T08:31:44.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.769+0000 7f153f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f154010a160 0x7f15401a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:44.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.769+0000 7f153f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f154010a160 0x7f15401a4d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39088/0 (socket says 172.21.15.32:39088) 2024-04-30T08:31:44.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.769+0000 7f153f400640 1 -- 172.21.15.32:0/359533038 learned_addr learned my addr 172.21.15.32:0/359533038 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:31:44.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.769+0000 7f153ea00640 1 --2- 172.21.15.32:0/359533038 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f154010b280 0x7f15401a5280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:44.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.769+0000 7f153fe00640 1 --2- 172.21.15.32:0/359533038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f154010bbd0 0x7f15401ac1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:44.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.769+0000 7f153f400640 1 -- 172.21.15.32:0/359533038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f154010bbd0 msgr2=0x7f15401ac1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:44.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.769+0000 7f153f400640 1 --2- 172.21.15.32:0/359533038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f154010bbd0 0x7f15401ac1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:44.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.769+0000 7f153f400640 1 -- 172.21.15.32:0/359533038 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f154010b280 msgr2=0x7f15401a5280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:44.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.769+0000 7f153f400640 1 --2- 172.21.15.32:0/359533038 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f154010b280 0x7f15401a5280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:44.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.769+0000 7f153f400640 1 -- 172.21.15.32:0/359533038 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15401ac870 con 0x7f154010a160 2024-04-30T08:31:44.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.770+0000 7f153f400640 1 --2- 172.21.15.32:0/359533038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f154010a160 0x7f15401a4d40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1530031440 tx=0x7f1530003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:44.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.770+0000 7f1537e00640 1 -- 172.21.15.32:0/359533038 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f153003f070 con 0x7f154010a160 2024-04-30T08:31:44.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.770+0000 7f1546400640 1 -- 172.21.15.32:0/359533038 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15401acb00 con 0x7f154010a160 2024-04-30T08:31:44.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.770+0000 7f1537e00640 1 -- 172.21.15.32:0/359533038 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1530031c50 con 0x7f154010a160 2024-04-30T08:31:44.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.770+0000 7f1537e00640 1 -- 172.21.15.32:0/359533038 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1530035030 con 0x7f154010a160 2024-04-30T08:31:44.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.770+0000 7f1546400640 1 -- 172.21.15.32:0/359533038 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15401acff0 con 0x7f154010a160 2024-04-30T08:31:44.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.771+0000 7f1546400640 1 -- 172.21.15.32:0/359533038 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15401112e0 con 0x7f154010a160 2024-04-30T08:31:44.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.774+0000 7f1537e00640 1 -- 172.21.15.32:0/359533038 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f1530042070 con 0x7f154010a160 2024-04-30T08:31:44.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.776+0000 7f1537e00640 1 --2- 172.21.15.32:0/359533038 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f15240afaf0 0x7f15240b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:44.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.777+0000 7f1537e00640 1 -- 172.21.15.32:0/359533038 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f15300ff690 con 0x7f154010a160 2024-04-30T08:31:44.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.777+0000 7f153ea00640 1 --2- 172.21.15.32:0/359533038 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f15240afaf0 0x7f15240b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:44.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.777+0000 7f153ea00640 1 --2- 172.21.15.32:0/359533038 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f15240afaf0 0x7f15240b2000 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1528009bd0 tx=0x7f1528005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:44.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:44.779+0000 7f1537e00640 1 -- 172.21.15.32:0/359533038 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f15300c8020 con 0x7f154010a160 2024-04-30T08:31:45.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:45.031+0000 7f1546400640 1 -- 172.21.15.32:0/359533038 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f154002dbd0 con 0x7f154010a160 2024-04-30T08:31:45.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:45.032+0000 7f1537e00640 1 -- 172.21.15.32:0/359533038 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9647 (secure 0 0 0) 0x7f15300cc320 con 0x7f154010a160 2024-04-30T08:31:45.034 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:31:45.034 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2235842676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2235842676},{"type":"v1","addr":"172.21.15.32:6835","nonce":2235842676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/3577009355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":3577009355},{"type":"v1","addr":"172.21.15.32:6841","nonce":3577009355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/463203038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":463203038},{"type":"v1","addr":"172.21.15.189:6833","nonce":463203038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3635143591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3635143591},{"type":"v1","addr":"172.21.15.71:6833","nonce":3635143591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24484,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2569449994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2569449994},{"type":"v1","addr":"172.21.15.32:6837","nonce":2569449994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/278686565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":278686565},{"type":"v1","addr":"172.21.15.189:6835","nonce":278686565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6835/1558303056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1558303056},{"type":"v1","addr":"172.21.15.71:6835","nonce":1558303056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/686520873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":686520873},{"type":"v1","addr":"172.21.15.189:6837","nonce":686520873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/2324039932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2324039932},{"type":"v1","addr":"172.21.15.71:6837","nonce":2324039932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6839/352970107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":352970107},{"type":"v1","addr":"172.21.15.189:6839","nonce":352970107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24560,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/4160386524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4160386524},{"type":"v1","addr":"172.21.15.32:6839","nonce":4160386524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-04-30T08:31:45.034 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 12 2024-04-30T08:31:45.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:45.036+0000 7f1546400640 1 -- 172.21.15.32:0/359533038 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f15240afaf0 msgr2=0x7f15240b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:45.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:45.036+0000 7f1546400640 1 --2- 172.21.15.32:0/359533038 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f15240afaf0 0x7f15240b2000 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1528009bd0 tx=0x7f1528005f90 comp rx=0 tx=0).stop 2024-04-30T08:31:45.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:45.036+0000 7f1546400640 1 -- 172.21.15.32:0/359533038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f154010a160 msgr2=0x7f15401a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:45.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:45.036+0000 7f1546400640 1 --2- 172.21.15.32:0/359533038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f154010a160 0x7f15401a4d40 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1530031440 tx=0x7f1530003c10 comp rx=0 tx=0).stop 2024-04-30T08:31:45.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:45.036+0000 7f1546400640 1 -- 172.21.15.32:0/359533038 shutdown_connections 2024-04-30T08:31:45.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:45.036+0000 7f1546400640 1 --2- 172.21.15.32:0/359533038 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f15240afaf0 0x7f15240b2000 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:45.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:45.036+0000 7f1546400640 1 --2- 172.21.15.32:0/359533038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f154010bbd0 0x7f15401ac1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:45.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:45.036+0000 7f1546400640 1 --2- 172.21.15.32:0/359533038 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f154010b280 0x7f15401a5280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:45.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:45.036+0000 7f1546400640 1 --2- 172.21.15.32:0/359533038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f154010a160 0x7f15401a4d40 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:45.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:45.036+0000 7f1546400640 1 -- 172.21.15.32:0/359533038 >> 172.21.15.32:0/359533038 conn(0x7f15400709a0 msgr2=0x7f154010e4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:45.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:45.036+0000 7f1546400640 1 -- 172.21.15.32:0/359533038 shutdown_connections 2024-04-30T08:31:45.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:45.036+0000 7f1546400640 1 -- 172.21.15.32:0/359533038 wait complete. 2024-04-30T08:31:45.111 INFO:journalctl@ceph.mds.b.smithi071.stdout:Apr 30 08:31:44 smithi071 podman[64012]: 2024-04-30 08:31:44.754184076 +0000 UTC m=+0.366792905 container start 562a91b76601bfa2f3e437103889bab3af607fe26861f7b526e61f2dd21912f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-b, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-30T08:31:45.111 INFO:journalctl@ceph.mds.b.smithi071.stdout:Apr 30 08:31:44 smithi071 ceph-mds[64030]: -- 172.21.15.71:0/2072676158 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d17036ba40 con 0x55d1703b3a80 2024-04-30T08:31:45.111 INFO:journalctl@ceph.mds.b.smithi071.stdout:Apr 30 08:31:44 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-b[64026]: starting mds.b at 2024-04-30T08:31:45.111 INFO:journalctl@ceph.mds.b.smithi071.stdout:Apr 30 08:31:44 smithi071 bash[64012]: 562a91b76601bfa2f3e437103889bab3af607fe26861f7b526e61f2dd21912f9 2024-04-30T08:31:45.111 INFO:journalctl@ceph.mds.b.smithi071.stdout:Apr 30 08:31:44 smithi071 systemd[1]: Started Ceph mds.b for 98c0a670-06ca-11ef-bc93-c7b262605968. 2024-04-30T08:31:45.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:45 smithi032 ceph-mon[28331]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:45.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:45 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:45.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:45 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:45.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:45 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:45.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:45 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:45.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:45 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:45.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:45 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/359533038' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:45.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:45 smithi071 ceph-mon[33385]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:45.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:45 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:45.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:45 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:45.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:45 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:45.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:45 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:45.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:45 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:45.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:45 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/359533038' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:45.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:45 smithi189 ceph-mon[33038]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:45.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:45 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:45.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:45 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:45.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:45 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:45.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:45 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:45.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:45 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:45.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:45 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/359533038' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:46.373 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:46 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:boot 2024-04-30T08:31:46.373 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:46 smithi071 ceph-mon[33385]: fsmap 12 up:standby 2024-04-30T08:31:46.374 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:46 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-30T08:31:46.374 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:46 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:31:46.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:46 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:boot 2024-04-30T08:31:46.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:46 smithi032 ceph-mon[28331]: fsmap 12 up:standby 2024-04-30T08:31:46.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:46 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-30T08:31:46.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:46 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:31:46.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:46 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:boot 2024-04-30T08:31:46.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:46 smithi189 ceph-mon[33038]: fsmap 12 up:standby 2024-04-30T08:31:46.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:46 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-30T08:31:46.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:46 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:31:47.063 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-30T08:31:47.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.368+0000 7ff6dca00640 1 -- 172.21.15.32:0/909344046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6d8075010 msgr2=0x7ff6d8075420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:47.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.368+0000 7ff6dca00640 1 --2- 172.21.15.32:0/909344046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6d8075010 0x7ff6d8075420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff6c8009ca0 tx=0x7ff6c802a8c0 comp rx=0 tx=0).stop 2024-04-30T08:31:47.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.369+0000 7ff6dca00640 1 -- 172.21.15.32:0/909344046 shutdown_connections 2024-04-30T08:31:47.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.369+0000 7ff6dca00640 1 --2- 172.21.15.32:0/909344046 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6d81100f0 0x7ff6d8112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:47.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.369+0000 7ff6dca00640 1 --2- 172.21.15.32:0/909344046 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6d8075960 0x7ff6d810fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:47.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.369+0000 7ff6dca00640 1 --2- 172.21.15.32:0/909344046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6d8075010 0x7ff6d8075420 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:47.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.369+0000 7ff6dca00640 1 -- 172.21.15.32:0/909344046 >> 172.21.15.32:0/909344046 conn(0x7ff6d80feba0 msgr2=0x7ff6d8101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:47.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.369+0000 7ff6dca00640 1 -- 172.21.15.32:0/909344046 shutdown_connections 2024-04-30T08:31:47.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.369+0000 7ff6dca00640 1 -- 172.21.15.32:0/909344046 wait complete. 2024-04-30T08:31:47.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.370+0000 7ff6dca00640 1 Processor -- start 2024-04-30T08:31:47.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.370+0000 7ff6dca00640 1 -- start start 2024-04-30T08:31:47.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.370+0000 7ff6dca00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6d8075010 0x7ff6d81a06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:47.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.370+0000 7ff6dca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6d8075960 0x7ff6d81a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:47.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.370+0000 7ff6dca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6d81100f0 0x7ff6d81a7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:47.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.370+0000 7ff6dca00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d8115b20 con 0x7ff6d8075960 2024-04-30T08:31:47.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.370+0000 7ff6dca00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d81159a0 con 0x7ff6d8075010 2024-04-30T08:31:47.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.370+0000 7ff6dca00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff6d8115ca0 con 0x7ff6d81100f0 2024-04-30T08:31:47.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.371+0000 7ff6d7400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6d8075010 0x7ff6d81a06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:47.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.371+0000 7ff6d7400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6d8075010 0x7ff6d81a06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:47202/0 (socket says 172.21.15.32:47202) 2024-04-30T08:31:47.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.371+0000 7ff6d7400640 1 -- 172.21.15.32:0/613086421 learned_addr learned my addr 172.21.15.32:0/613086421 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:31:47.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.371+0000 7ff6cea00640 1 --2- 172.21.15.32:0/613086421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6d8075960 0x7ff6d81a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:47.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.371+0000 7ff6d7e00640 1 --2- 172.21.15.32:0/613086421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6d81100f0 0x7ff6d81a7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:47.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.371+0000 7ff6d7400640 1 -- 172.21.15.32:0/613086421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6d81100f0 msgr2=0x7ff6d81a7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:47.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.371+0000 7ff6d7400640 1 --2- 172.21.15.32:0/613086421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6d81100f0 0x7ff6d81a7bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:47.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.371+0000 7ff6d7400640 1 -- 172.21.15.32:0/613086421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6d8075960 msgr2=0x7ff6d81a0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:47.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.371+0000 7ff6d7400640 1 --2- 172.21.15.32:0/613086421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6d8075960 0x7ff6d81a0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:47.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.371+0000 7ff6d7400640 1 -- 172.21.15.32:0/613086421 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6d81a8220 con 0x7ff6d8075010 2024-04-30T08:31:47.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.371+0000 7ff6d7e00640 1 --2- 172.21.15.32:0/613086421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6d81100f0 0x7ff6d81a7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:31:47.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.371+0000 7ff6d7400640 1 --2- 172.21.15.32:0/613086421 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6d8075010 0x7ff6d81a06e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff6c8005e30 tx=0x7ff6c80317e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:47.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.372+0000 7ff6d4c00640 1 -- 172.21.15.32:0/613086421 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c8003bf0 con 0x7ff6d8075010 2024-04-30T08:31:47.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.372+0000 7ff6dca00640 1 -- 172.21.15.32:0/613086421 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6d81a84b0 con 0x7ff6d8075010 2024-04-30T08:31:47.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.372+0000 7ff6d4c00640 1 -- 172.21.15.32:0/613086421 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6c8003d90 con 0x7ff6d8075010 2024-04-30T08:31:47.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.372+0000 7ff6d4c00640 1 -- 172.21.15.32:0/613086421 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c802fc10 con 0x7ff6d8075010 2024-04-30T08:31:47.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.372+0000 7ff6dca00640 1 -- 172.21.15.32:0/613086421 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6d81a8a00 con 0x7ff6d8075010 2024-04-30T08:31:47.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.373+0000 7ff6dca00640 1 -- 172.21.15.32:0/613086421 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6d8105d80 con 0x7ff6d8075010 2024-04-30T08:31:47.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.375+0000 7ff6d4c00640 1 -- 172.21.15.32:0/613086421 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7ff6c8004040 con 0x7ff6d8075010 2024-04-30T08:31:47.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.377+0000 7ff6d4c00640 1 --2- 172.21.15.32:0/613086421 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff6bc0afae0 0x7ff6bc0b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:47.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.377+0000 7ff6d4c00640 1 -- 172.21.15.32:0/613086421 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7ff6c80ffca0 con 0x7ff6d8075010 2024-04-30T08:31:47.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.377+0000 7ff6cea00640 1 --2- 172.21.15.32:0/613086421 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff6bc0afae0 0x7ff6bc0b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:47.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.378+0000 7ff6cea00640 1 --2- 172.21.15.32:0/613086421 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff6bc0afae0 0x7ff6bc0b1ff0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff6d81a1780 tx=0x7ff6c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:47.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.382+0000 7ff6d4c00640 1 -- 172.21.15.32:0/613086421 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff6c80c8d10 con 0x7ff6d8075010 2024-04-30T08:31:47.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:47 smithi032 ceph-mon[28331]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:47.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:47 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:47.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:47 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:47.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:47 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:47.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:47 smithi071 ceph-mon[33385]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:47.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:47 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:47.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:47 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:47.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:47 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:47.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:47 smithi189 ceph-mon[33038]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:47.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:47 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:47.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:47 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:47.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:47 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:47.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.671+0000 7ff6dca00640 1 -- 172.21.15.32:0/613086421 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff6d802dbd0 con 0x7ff6d8075010 2024-04-30T08:31:47.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.672+0000 7ff6d4c00640 1 -- 172.21.15.32:0/613086421 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10473 (secure 0 0 0) 0x7ff6c80c8610 con 0x7ff6d8075010 2024-04-30T08:31:47.674 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:31:47.674 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2235842676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2235842676},{"type":"v1","addr":"172.21.15.32:6835","nonce":2235842676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/3577009355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":3577009355},{"type":"v1","addr":"172.21.15.32:6841","nonce":3577009355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/463203038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":463203038},{"type":"v1","addr":"172.21.15.189:6833","nonce":463203038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3635143591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3635143591},{"type":"v1","addr":"172.21.15.71:6833","nonce":3635143591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24484,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2569449994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2569449994},{"type":"v1","addr":"172.21.15.32:6837","nonce":2569449994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/278686565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":278686565},{"type":"v1","addr":"172.21.15.189:6835","nonce":278686565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6835/1558303056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1558303056},{"type":"v1","addr":"172.21.15.71:6835","nonce":1558303056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/686520873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":686520873},{"type":"v1","addr":"172.21.15.189:6837","nonce":686520873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/2324039932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2324039932},{"type":"v1","addr":"172.21.15.71:6837","nonce":2324039932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6839/352970107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":352970107},{"type":"v1","addr":"172.21.15.189:6839","nonce":352970107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24560,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/4160386524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4160386524},{"type":"v1","addr":"172.21.15.32:6839","nonce":4160386524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24590,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6839/3517409993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3517409993},{"type":"v1","addr":"172.21.15.71:6839","nonce":3517409993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-04-30T08:31:47.674 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 13 2024-04-30T08:31:47.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.675+0000 7ff6dca00640 1 -- 172.21.15.32:0/613086421 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff6bc0afae0 msgr2=0x7ff6bc0b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:47.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.675+0000 7ff6dca00640 1 --2- 172.21.15.32:0/613086421 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff6bc0afae0 0x7ff6bc0b1ff0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff6d81a1780 tx=0x7ff6c0005f90 comp rx=0 tx=0).stop 2024-04-30T08:31:47.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.675+0000 7ff6dca00640 1 -- 172.21.15.32:0/613086421 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6d8075010 msgr2=0x7ff6d81a06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:47.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.675+0000 7ff6dca00640 1 --2- 172.21.15.32:0/613086421 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6d8075010 0x7ff6d81a06e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7ff6c8005e30 tx=0x7ff6c80317e0 comp rx=0 tx=0).stop 2024-04-30T08:31:47.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.675+0000 7ff6dca00640 1 -- 172.21.15.32:0/613086421 shutdown_connections 2024-04-30T08:31:47.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.675+0000 7ff6dca00640 1 --2- 172.21.15.32:0/613086421 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff6bc0afae0 0x7ff6bc0b1ff0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:47.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.675+0000 7ff6dca00640 1 --2- 172.21.15.32:0/613086421 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff6d81100f0 0x7ff6d81a7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:47.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.675+0000 7ff6dca00640 1 --2- 172.21.15.32:0/613086421 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6d8075960 0x7ff6d81a0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:47.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.675+0000 7ff6dca00640 1 --2- 172.21.15.32:0/613086421 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff6d8075010 0x7ff6d81a06e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:47.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.675+0000 7ff6dca00640 1 -- 172.21.15.32:0/613086421 >> 172.21.15.32:0/613086421 conn(0x7ff6d80feba0 msgr2=0x7ff6d8102f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:47.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.675+0000 7ff6dca00640 1 -- 172.21.15.32:0/613086421 shutdown_connections 2024-04-30T08:31:47.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.675+0000 7ff6dca00640 1 -- 172.21.15.32:0/613086421 wait complete. 2024-04-30T08:31:47.704 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-30T08:31:47.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.972+0000 7f3265e00640 1 -- 172.21.15.32:0/4236093073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3260104dd0 msgr2=0x7f32601051e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:47.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.972+0000 7f3265e00640 1 --2- 172.21.15.32:0/4236093073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3260104dd0 0x7f32601051e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f324c009860 tx=0x7f324c02a340 comp rx=0 tx=0).stop 2024-04-30T08:31:47.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.972+0000 7f3265e00640 1 -- 172.21.15.32:0/4236093073 shutdown_connections 2024-04-30T08:31:47.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.972+0000 7f3265e00640 1 --2- 172.21.15.32:0/4236093073 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3260109ad0 0x7f326010bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:47.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.972+0000 7f3265e00640 1 --2- 172.21.15.32:0/4236093073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3260104dd0 0x7f32601051e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:47.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.972+0000 7f3265e00640 1 --2- 172.21.15.32:0/4236093073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3260103b90 0x7f3260104890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:47.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.972+0000 7f3265e00640 1 -- 172.21.15.32:0/4236093073 >> 172.21.15.32:0/4236093073 conn(0x7f32600fedc0 msgr2=0x7f3260101230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:47.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.972+0000 7f3265e00640 1 -- 172.21.15.32:0/4236093073 shutdown_connections 2024-04-30T08:31:47.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.972+0000 7f3265e00640 1 -- 172.21.15.32:0/4236093073 wait complete. 2024-04-30T08:31:47.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.973+0000 7f3265e00640 1 Processor -- start 2024-04-30T08:31:47.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.973+0000 7f3265e00640 1 -- start start 2024-04-30T08:31:47.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.974+0000 7f3265e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3260103b90 0x7f32601980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:47.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.974+0000 7f3265e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3260104dd0 0x7f3260198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:47.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.974+0000 7f3265e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3260109ad0 0x7f32601a7da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:47.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.974+0000 7f3265e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f326010f6b0 con 0x7f3260104dd0 2024-04-30T08:31:47.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.974+0000 7f3265e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f326010f530 con 0x7f3260103b90 2024-04-30T08:31:47.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.974+0000 7f3265e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f326010f830 con 0x7f3260109ad0 2024-04-30T08:31:47.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.974+0000 7f325e000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3260104dd0 0x7f3260198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:47.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.974+0000 7f325e000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3260104dd0 0x7f3260198600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43258/0 (socket says 172.21.15.32:43258) 2024-04-30T08:31:47.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.974+0000 7f325e000640 1 -- 172.21.15.32:0/392191197 learned_addr learned my addr 172.21.15.32:0/392191197 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:31:47.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.974+0000 7f325ea00640 1 --2- 172.21.15.32:0/392191197 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3260103b90 0x7f32601980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:47.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.974+0000 7f325f400640 1 --2- 172.21.15.32:0/392191197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3260109ad0 0x7f32601a7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:47.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.975+0000 7f325e000640 1 -- 172.21.15.32:0/392191197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3260109ad0 msgr2=0x7f32601a7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:47.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.975+0000 7f325e000640 1 --2- 172.21.15.32:0/392191197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3260109ad0 0x7f32601a7da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:47.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.975+0000 7f325e000640 1 -- 172.21.15.32:0/392191197 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3260103b90 msgr2=0x7f32601980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:47.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.975+0000 7f325e000640 1 --2- 172.21.15.32:0/392191197 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3260103b90 0x7f32601980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:47.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.975+0000 7f325e000640 1 -- 172.21.15.32:0/392191197 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32601a8410 con 0x7f3260104dd0 2024-04-30T08:31:47.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.975+0000 7f325ea00640 1 --2- 172.21.15.32:0/392191197 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3260103b90 0x7f32601980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:31:47.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.975+0000 7f325e000640 1 --2- 172.21.15.32:0/392191197 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3260104dd0 0x7f3260198600 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f324c009980 tx=0x7f324c032dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:47.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.975+0000 7f3264a00640 1 -- 172.21.15.32:0/392191197 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f324c040020 con 0x7f3260104dd0 2024-04-30T08:31:47.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.975+0000 7f3264a00640 1 -- 172.21.15.32:0/392191197 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f324c030dd0 con 0x7f3260104dd0 2024-04-30T08:31:47.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.975+0000 7f3264a00640 1 -- 172.21.15.32:0/392191197 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f324c037040 con 0x7f3260104dd0 2024-04-30T08:31:47.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.975+0000 7f3265e00640 1 -- 172.21.15.32:0/392191197 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32601a86a0 con 0x7f3260104dd0 2024-04-30T08:31:47.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.976+0000 7f3265e00640 1 -- 172.21.15.32:0/392191197 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32601a8b40 con 0x7f3260104dd0 2024-04-30T08:31:47.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.977+0000 7f3265e00640 1 -- 172.21.15.32:0/392191197 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32601a8e60 con 0x7f3260104dd0 2024-04-30T08:31:47.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.983+0000 7f3264a00640 1 -- 172.21.15.32:0/392191197 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f324c043020 con 0x7f3260104dd0 2024-04-30T08:31:47.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.984+0000 7f3264a00640 1 --2- 172.21.15.32:0/392191197 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f32440afaf0 0x7f32440b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:47.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.984+0000 7f3264a00640 1 -- 172.21.15.32:0/392191197 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f324c0fed60 con 0x7f3260104dd0 2024-04-30T08:31:47.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.984+0000 7f325ea00640 1 --2- 172.21.15.32:0/392191197 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f32440afaf0 0x7f32440b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:47.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.985+0000 7f3264a00640 1 -- 172.21.15.32:0/392191197 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f324c04a020 con 0x7f3260104dd0 2024-04-30T08:31:47.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:47.985+0000 7f325ea00640 1 --2- 172.21.15.32:0/392191197 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f32440afaf0 0x7f32440b2000 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3260104790 tx=0x7f3250009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:48.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.203+0000 7f3265e00640 1 -- 172.21.15.32:0/392191197 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f32601a9030 con 0x7f3260104dd0 2024-04-30T08:31:48.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.205+0000 7f3264a00640 1 -- 172.21.15.32:0/392191197 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) ==== 74+0+18381 (secure 0 0 0) 0x7f324c0c7d40 con 0x7f3260104dd0 2024-04-30T08:31:48.207 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:31:48.207 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":72,"fsid":"98c0a670-06ca-11ef-bc93-c7b262605968","created":"2024-04-30T08:22:11.122875+0000","modified":"2024-04-30T08:30:51.365145+0000","last_up_change":"2024-04-30T08:30:50.350439+0000","last_in_change":"2024-04-30T08:30:30.210620+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-30T08:26:19.728331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"a7e060c8-1cbb-418f-9a03-06693f6b3f52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6803","nonce":2624700402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6805","nonce":2624700402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6809","nonce":2624700402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6807","nonce":2624700402}]},"public_addr":"172.21.15.32:6803/2624700402","cluster_addr":"172.21.15.32:6805/2624700402","heartbeat_back_addr":"172.21.15.32:6809/2624700402","heartbeat_front_addr":"172.21.15.32:6807/2624700402","state":["exists","up"]},{"osd":1,"uuid":"72ef2fb1-e5ca-48ef-aa3a-4796a64b922f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":37,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6801","nonce":444649711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6803","nonce":444649711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6807","nonce":444649711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6805","nonce":444649711}]},"public_addr":"172.21.15.71:6801/444649711","cluster_addr":"172.21.15.71:6803/444649711","heartbeat_back_addr":"172.21.15.71:6807/444649711","heartbeat_front_addr":"172.21.15.71:6805/444649711","state":["exists","up"]},{"osd":2,"uuid":"704e5e8a-935e-47c7-ae28-27119876ac09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6801","nonce":3255528671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6803","nonce":3255528671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6807","nonce":3255528671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6805","nonce":3255528671}]},"public_addr":"172.21.15.189:6801/3255528671","cluster_addr":"172.21.15.189:6803/3255528671","heartbeat_back_addr":"172.21.15.189:6807/3255528671","heartbeat_front_addr":"172.21.15.189:6805/3255528671","state":["exists","up"]},{"osd":3,"uuid":"575004f7-6559-4a62-863e-e33078b0eebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6811","nonce":1384702112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6813","nonce":1384702112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6817","nonce":1384702112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6815","nonce":1384702112}]},"public_addr":"172.21.15.32:6811/1384702112","cluster_addr":"172.21.15.32:6813/1384702112","heartbeat_back_addr":"172.21.15.32:6817/1384702112","heartbeat_front_addr":"172.21.15.32:6815/1384702112","state":["exists","up"]},{"osd":4,"uuid":"e1c1e894-074d-4c78-89ef-9e5a114268bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6809","nonce":4250534574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6811","nonce":4250534574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6815","nonce":4250534574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6813","nonce":4250534574}]},"public_addr":"172.21.15.71:6809/4250534574","cluster_addr":"172.21.15.71:6811/4250534574","heartbeat_back_addr":"172.21.15.71:6815/4250534574","heartbeat_front_addr":"172.21.15.71:6813/4250534574","state":["exists","up"]},{"osd":5,"uuid":"12c022d4-fac6-40c4-ae7a-691571bee5d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":48,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6809","nonce":4245346198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6811","nonce":4245346198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6815","nonce":4245346198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6813","nonce":4245346198}]},"public_addr":"172.21.15.189:6809/4245346198","cluster_addr":"172.21.15.189:6811/4245346198","heartbeat_back_addr":"172.21.15.189:6815/4245346198","heartbeat_front_addr":"172.21.15.189:6813/4245346198","state":["exists","up"]},{"osd":6,"uuid":"b413f8ee-3d4a-415a-a2ba-d68437b7330d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6819","nonce":1910389190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6821","nonce":1910389190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6825","nonce":1910389190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6823","nonce":1910389190}]},"public_addr":"172.21.15.32:6819/1910389190","cluster_addr":"172.21.15.32:6821/1910389190","heartbeat_back_addr":"172.21.15.32:6825/1910389190","heartbeat_front_addr":"172.21.15.32:6823/1910389190","state":["exists","up"]},{"osd":7,"uuid":"aac64fb3-fc13-4126-b842-7b06ad5ce238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":66,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6817","nonce":692607380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6819","nonce":692607380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6823","nonce":692607380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6821","nonce":692607380}]},"public_addr":"172.21.15.71:6817/692607380","cluster_addr":"172.21.15.71:6819/692607380","heartbeat_back_addr":"172.21.15.71:6823/692607380","heartbeat_front_addr":"172.21.15.71:6821/692607380","state":["exists","up"]},{"osd":8,"uuid":"cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6817","nonce":3973697069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6819","nonce":3973697069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6823","nonce":3973697069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6821","nonce":3973697069}]},"public_addr":"172.21.15.189:6817/3973697069","cluster_addr":"172.21.15.189:6819/3973697069","heartbeat_back_addr":"172.21.15.189:6823/3973697069","heartbeat_front_addr":"172.21.15.189:6821/3973697069","state":["exists","up"]},{"osd":9,"uuid":"ef4a1645-ae6e-45c3-82ad-161510e62aa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6827","nonce":46066685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6829","nonce":46066685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6833","nonce":46066685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6831","nonce":46066685}]},"public_addr":"172.21.15.32:6827/46066685","cluster_addr":"172.21.15.32:6829/46066685","heartbeat_back_addr":"172.21.15.32:6833/46066685","heartbeat_front_addr":"172.21.15.32:6831/46066685","state":["exists","up"]},{"osd":10,"uuid":"25b26a38-a644-409f-b941-59efdf04a3ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6825","nonce":1886290293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6827","nonce":1886290293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6831","nonce":1886290293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6829","nonce":1886290293}]},"public_addr":"172.21.15.71:6825/1886290293","cluster_addr":"172.21.15.71:6827/1886290293","heartbeat_back_addr":"172.21.15.71:6831/1886290293","heartbeat_front_addr":"172.21.15.71:6829/1886290293","state":["exists","up"]},{"osd":11,"uuid":"ab96b3ff-3688-455b-af64-60226c5bfbff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6825","nonce":850185001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6827","nonce":850185001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6831","nonce":850185001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6829","nonce":850185001}]},"public_addr":"172.21.15.189:6825/850185001","cluster_addr":"172.21.15.189:6827/850185001","heartbeat_back_addr":"172.21.15.189:6831/850185001","heartbeat_front_addr":"172.21.15.189:6829/850185001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:27.220308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:53.230783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:17.190642+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:52.603799+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:20.568671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:46.158363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:22.706933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:50.665173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:16.602410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:53.814698+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:22.435189+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:48.651949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1266897222":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/2915689533":"2024-05-01T08:23:29.526664+0000","172.21.15.32:6801/4019351306":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/425836193":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/1134075722":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6801/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/21059253":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/3245550894":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/4019351306":"2024-05-01T08:23:29.526664+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-30T08:31:48.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.208+0000 7f3265e00640 1 -- 172.21.15.32:0/392191197 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f32440afaf0 msgr2=0x7f32440b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:48.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.208+0000 7f3265e00640 1 --2- 172.21.15.32:0/392191197 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f32440afaf0 0x7f32440b2000 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3260104790 tx=0x7f3250009260 comp rx=0 tx=0).stop 2024-04-30T08:31:48.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.208+0000 7f3265e00640 1 -- 172.21.15.32:0/392191197 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3260104dd0 msgr2=0x7f3260198600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:48.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.208+0000 7f3265e00640 1 --2- 172.21.15.32:0/392191197 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3260104dd0 0x7f3260198600 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f324c009980 tx=0x7f324c032dc0 comp rx=0 tx=0).stop 2024-04-30T08:31:48.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.208+0000 7f3265e00640 1 -- 172.21.15.32:0/392191197 shutdown_connections 2024-04-30T08:31:48.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.208+0000 7f3265e00640 1 --2- 172.21.15.32:0/392191197 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f32440afaf0 0x7f32440b2000 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:48.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.208+0000 7f3265e00640 1 --2- 172.21.15.32:0/392191197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3260109ad0 0x7f32601a7da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:48.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.208+0000 7f3265e00640 1 --2- 172.21.15.32:0/392191197 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3260104dd0 0x7f3260198600 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:48.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.208+0000 7f3265e00640 1 --2- 172.21.15.32:0/392191197 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3260103b90 0x7f32601980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:48.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.209+0000 7f3265e00640 1 -- 172.21.15.32:0/392191197 >> 172.21.15.32:0/392191197 conn(0x7f32600fedc0 msgr2=0x7f32600ff310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:48.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.209+0000 7f3265e00640 1 -- 172.21.15.32:0/392191197 shutdown_connections 2024-04-30T08:31:48.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.209+0000 7f3265e00640 1 -- 172.21.15.32:0/392191197 wait complete. 2024-04-30T08:31:48.237 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-30T08:26:19.728331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-30T08:31:48.238 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-30T08:31:48.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:48 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:48.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:48 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/613086421' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:48.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.487+0000 7f4f33e00640 1 -- 172.21.15.32:0/3547939411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f34075960 msgr2=0x7f4f3410fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:48.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.487+0000 7f4f33e00640 1 --2- 172.21.15.32:0/3547939411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f34075960 0x7f4f3410fbb0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f4f20009900 tx=0x7f4f2002a3f0 comp rx=0 tx=0).stop 2024-04-30T08:31:48.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.487+0000 7f4f33e00640 1 -- 172.21.15.32:0/3547939411 shutdown_connections 2024-04-30T08:31:48.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.487+0000 7f4f33e00640 1 --2- 172.21.15.32:0/3547939411 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f341100f0 0x7f4f34112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:48.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.487+0000 7f4f33e00640 1 --2- 172.21.15.32:0/3547939411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f34075960 0x7f4f3410fbb0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:48.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.487+0000 7f4f33e00640 1 --2- 172.21.15.32:0/3547939411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f34075010 0x7f4f34075420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:48.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.487+0000 7f4f33e00640 1 -- 172.21.15.32:0/3547939411 >> 172.21.15.32:0/3547939411 conn(0x7f4f340feba0 msgr2=0x7f4f34101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:48.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.487+0000 7f4f33e00640 1 -- 172.21.15.32:0/3547939411 shutdown_connections 2024-04-30T08:31:48.489 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.487+0000 7f4f33e00640 1 -- 172.21.15.32:0/3547939411 wait complete. 2024-04-30T08:31:48.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.488+0000 7f4f33e00640 1 Processor -- start 2024-04-30T08:31:48.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.488+0000 7f4f33e00640 1 -- start start 2024-04-30T08:31:48.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.488+0000 7f4f33e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f34075010 0x7f4f341a0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:48.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.488+0000 7f4f33e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f34075960 0x7f4f341a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:48.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.488+0000 7f4f33e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f341100f0 0x7f4f341a7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:48.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.488+0000 7f4f33e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4f34115b20 con 0x7f4f34075010 2024-04-30T08:31:48.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.488+0000 7f4f33e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4f341159a0 con 0x7f4f341100f0 2024-04-30T08:31:48.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.488+0000 7f4f33e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4f34115ca0 con 0x7f4f34075960 2024-04-30T08:31:48.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.489+0000 7f4f32a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f34075010 0x7f4f341a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:48.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.489+0000 7f4f33400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f341100f0 0x7f4f341a7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:48.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.489+0000 7f4f32000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f34075960 0x7f4f341a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:48.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.489+0000 7f4f33400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f341100f0 0x7f4f341a7be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:47236/0 (socket says 172.21.15.32:47236) 2024-04-30T08:31:48.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.489+0000 7f4f33400640 1 -- 172.21.15.32:0/2670351864 learned_addr learned my addr 172.21.15.32:0/2670351864 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:31:48.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.489+0000 7f4f33400640 1 -- 172.21.15.32:0/2670351864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f34075960 msgr2=0x7f4f341a0c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:48.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.489+0000 7f4f33400640 1 --2- 172.21.15.32:0/2670351864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f34075960 0x7f4f341a0c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:48.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.489+0000 7f4f33400640 1 -- 172.21.15.32:0/2670351864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f34075010 msgr2=0x7f4f341a0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:48.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.489+0000 7f4f33400640 1 --2- 172.21.15.32:0/2670351864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f34075010 0x7f4f341a0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:48.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.489+0000 7f4f33400640 1 -- 172.21.15.32:0/2670351864 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4f341a8250 con 0x7f4f341100f0 2024-04-30T08:31:48.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.489+0000 7f4f33400640 1 --2- 172.21.15.32:0/2670351864 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f341100f0 0x7f4f341a7be0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c00eb40 tx=0x7f4f2c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:48.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.490+0000 7f4f2b400640 1 -- 172.21.15.32:0/2670351864 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c004b20 con 0x7f4f341100f0 2024-04-30T08:31:48.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.490+0000 7f4f2b400640 1 -- 172.21.15.32:0/2670351864 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4f2c004cc0 con 0x7f4f341100f0 2024-04-30T08:31:48.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.490+0000 7f4f2b400640 1 -- 172.21.15.32:0/2670351864 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4f2c010720 con 0x7f4f341100f0 2024-04-30T08:31:48.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.490+0000 7f4f33e00640 1 -- 172.21.15.32:0/2670351864 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4f341a8540 con 0x7f4f341100f0 2024-04-30T08:31:48.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.490+0000 7f4f33e00640 1 -- 172.21.15.32:0/2670351864 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4f341a8a10 con 0x7f4f341100f0 2024-04-30T08:31:48.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.491+0000 7f4f33e00640 1 -- 172.21.15.32:0/2670351864 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4f34105d80 con 0x7f4f341100f0 2024-04-30T08:31:48.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.494+0000 7f4f2b400640 1 -- 172.21.15.32:0/2670351864 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f4f2c005560 con 0x7f4f341100f0 2024-04-30T08:31:48.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.495+0000 7f4f2b400640 1 --2- 172.21.15.32:0/2670351864 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4f180afae0 0x7f4f180b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:48.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.496+0000 7f4f2b400640 1 -- 172.21.15.32:0/2670351864 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f4f2c0defb0 con 0x7f4f341100f0 2024-04-30T08:31:48.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.496+0000 7f4f32a00640 1 --2- 172.21.15.32:0/2670351864 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4f180afae0 0x7f4f180b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:48.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.496+0000 7f4f32a00640 1 --2- 172.21.15.32:0/2670351864 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4f180afae0 0x7f4f180b1ff0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4f34076840 tx=0x7f4f24005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:48.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.499+0000 7f4f2b400640 1 -- 172.21.15.32:0/2670351864 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4f2c0a7f90 con 0x7f4f341100f0 2024-04-30T08:31:48.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:48 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:48.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:48 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/613086421' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:48.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:48 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:48.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:48 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/613086421' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:31:48.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.713+0000 7f4f33e00640 1 -- 172.21.15.32:0/2670351864 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4f34063820 con 0x7f4f341100f0 2024-04-30T08:31:48.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.713+0000 7f4f2b400640 1 -- 172.21.15.32:0/2670351864 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v72) ==== 93+0+10 (secure 0 0 0) 0x7f4f2c0a7890 con 0x7f4f341100f0 2024-04-30T08:31:48.715 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-30T08:31:48.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.718+0000 7f4f33e00640 1 -- 172.21.15.32:0/2670351864 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4f180afae0 msgr2=0x7f4f180b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:48.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.718+0000 7f4f33e00640 1 --2- 172.21.15.32:0/2670351864 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4f180afae0 0x7f4f180b1ff0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4f34076840 tx=0x7f4f24005f90 comp rx=0 tx=0).stop 2024-04-30T08:31:48.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.718+0000 7f4f33e00640 1 -- 172.21.15.32:0/2670351864 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f341100f0 msgr2=0x7f4f341a7be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:48.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.718+0000 7f4f33e00640 1 --2- 172.21.15.32:0/2670351864 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f341100f0 0x7f4f341a7be0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f4f2c00eb40 tx=0x7f4f2c00c2d0 comp rx=0 tx=0).stop 2024-04-30T08:31:48.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.718+0000 7f4f33e00640 1 -- 172.21.15.32:0/2670351864 shutdown_connections 2024-04-30T08:31:48.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.718+0000 7f4f33e00640 1 --2- 172.21.15.32:0/2670351864 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4f180afae0 0x7f4f180b1ff0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:48.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.718+0000 7f4f33e00640 1 --2- 172.21.15.32:0/2670351864 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4f341100f0 0x7f4f341a7be0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:48.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.718+0000 7f4f33e00640 1 --2- 172.21.15.32:0/2670351864 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4f34075960 0x7f4f341a0c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:48.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.718+0000 7f4f33e00640 1 --2- 172.21.15.32:0/2670351864 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4f34075010 0x7f4f341a0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:48.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.718+0000 7f4f33e00640 1 -- 172.21.15.32:0/2670351864 >> 172.21.15.32:0/2670351864 conn(0x7f4f340feba0 msgr2=0x7f4f34102f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:48.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.718+0000 7f4f33e00640 1 -- 172.21.15.32:0/2670351864 shutdown_connections 2024-04-30T08:31:48.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:48.718+0000 7f4f33e00640 1 -- 172.21.15.32:0/2670351864 wait complete. 2024-04-30T08:31:48.767 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-30T08:31:49.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.045+0000 7f3b9d600640 1 -- 172.21.15.32:0/2429442083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b98104470 msgr2=0x7f3b98106890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:49.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.045+0000 7f3b9d600640 1 --2- 172.21.15.32:0/2429442083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b98104470 0x7f3b98106890 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3b88009900 tx=0x7f3b8802a3f0 comp rx=0 tx=0).stop 2024-04-30T08:31:49.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.046+0000 7f3b9d600640 1 -- 172.21.15.32:0/2429442083 shutdown_connections 2024-04-30T08:31:49.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.046+0000 7f3b9d600640 1 --2- 172.21.15.32:0/2429442083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b98109730 0x7f3b9810bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:49.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.046+0000 7f3b9d600640 1 --2- 172.21.15.32:0/2429442083 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b98106dd0 0x7f3b981091f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:49.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.046+0000 7f3b9d600640 1 --2- 172.21.15.32:0/2429442083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b98104470 0x7f3b98106890 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:49.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.046+0000 7f3b9d600640 1 -- 172.21.15.32:0/2429442083 >> 172.21.15.32:0/2429442083 conn(0x7f3b980fda70 msgr2=0x7f3b980fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:49.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.046+0000 7f3b9d600640 1 -- 172.21.15.32:0/2429442083 shutdown_connections 2024-04-30T08:31:49.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.046+0000 7f3b9d600640 1 -- 172.21.15.32:0/2429442083 wait complete. 2024-04-30T08:31:49.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.047+0000 7f3b9d600640 1 Processor -- start 2024-04-30T08:31:49.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.047+0000 7f3b9d600640 1 -- start start 2024-04-30T08:31:49.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.047+0000 7f3b9d600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b98104470 0x7f3b9819c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:49.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.047+0000 7f3b9d600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b98106dd0 0x7f3b9819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:49.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.047+0000 7f3b9d600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b98109730 0x7f3b981a37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:49.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.047+0000 7f3b9d600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9810f4d0 con 0x7f3b98104470 2024-04-30T08:31:49.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.047+0000 7f3b9d600640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9810f350 con 0x7f3b98106dd0 2024-04-30T08:31:49.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.047+0000 7f3b9d600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3b9810f650 con 0x7f3b98109730 2024-04-30T08:31:49.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.047+0000 7f3b97e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b98104470 0x7f3b9819c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:49.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.047+0000 7f3b9cc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b98109730 0x7f3b981a37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:49.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.047+0000 7f3b97e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b98104470 0x7f3b9819c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43284/0 (socket says 172.21.15.32:43284) 2024-04-30T08:31:49.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.047+0000 7f3b97400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b98106dd0 0x7f3b9819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:49.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.047+0000 7f3b9cc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b98109730 0x7f3b981a37b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:49442/0 (socket says 172.21.15.32:49442) 2024-04-30T08:31:49.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.047+0000 7f3b97e00640 1 -- 172.21.15.32:0/1219188988 learned_addr learned my addr 172.21.15.32:0/1219188988 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:31:49.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.048+0000 7f3b9cc00640 1 -- 172.21.15.32:0/1219188988 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b98106dd0 msgr2=0x7f3b9819c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:49.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.048+0000 7f3b9cc00640 1 --2- 172.21.15.32:0/1219188988 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b98106dd0 0x7f3b9819c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:49.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.048+0000 7f3b9cc00640 1 -- 172.21.15.32:0/1219188988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b98104470 msgr2=0x7f3b9819c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:49.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.048+0000 7f3b9cc00640 1 --2- 172.21.15.32:0/1219188988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b98104470 0x7f3b9819c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:49.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.048+0000 7f3b9cc00640 1 -- 172.21.15.32:0/1219188988 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b981a3e20 con 0x7f3b98109730 2024-04-30T08:31:49.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.048+0000 7f3b97e00640 1 --2- 172.21.15.32:0/1219188988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b98104470 0x7f3b9819c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:31:49.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.048+0000 7f3b9cc00640 1 --2- 172.21.15.32:0/1219188988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b98109730 0x7f3b981a37b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3b90005fd0 tx=0x7f3b9000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:49.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.048+0000 7f3b94c00640 1 -- 172.21.15.32:0/1219188988 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b9000cdf0 con 0x7f3b98109730 2024-04-30T08:31:49.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.048+0000 7f3b94c00640 1 -- 172.21.15.32:0/1219188988 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3b90005020 con 0x7f3b98109730 2024-04-30T08:31:49.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.048+0000 7f3b9d600640 1 -- 172.21.15.32:0/1219188988 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b981a4110 con 0x7f3b98109730 2024-04-30T08:31:49.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.048+0000 7f3b94c00640 1 -- 172.21.15.32:0/1219188988 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b90010690 con 0x7f3b98109730 2024-04-30T08:31:49.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.049+0000 7f3b9d600640 1 -- 172.21.15.32:0/1219188988 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b981a4610 con 0x7f3b98109730 2024-04-30T08:31:49.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.050+0000 7f3b9d600640 1 -- 172.21.15.32:0/1219188988 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b981a48d0 con 0x7f3b98109730 2024-04-30T08:31:49.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.052+0000 7f3b94c00640 1 -- 172.21.15.32:0/1219188988 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f3b90005460 con 0x7f3b98109730 2024-04-30T08:31:49.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.053+0000 7f3b94c00640 1 --2- 172.21.15.32:0/1219188988 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3b7c0afae0 0x7f3b7c0b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:49.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.054+0000 7f3b94c00640 1 -- 172.21.15.32:0/1219188988 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f3b900deed0 con 0x7f3b98109730 2024-04-30T08:31:49.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.054+0000 7f3b97e00640 1 --2- 172.21.15.32:0/1219188988 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3b7c0afae0 0x7f3b7c0b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:49.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.054+0000 7f3b97e00640 1 --2- 172.21.15.32:0/1219188988 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3b7c0afae0 0x7f3b7c0b1ff0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3b880096c0 tx=0x7f3b880314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:49.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.056+0000 7f3b94c00640 1 -- 172.21.15.32:0/1219188988 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3b900a7eb0 con 0x7f3b98109730 2024-04-30T08:31:49.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.274+0000 7f3b9d600640 1 -- 172.21.15.32:0/1219188988 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f3b981a4aa0 con 0x7f3b98109730 2024-04-30T08:31:49.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.275+0000 7f3b94c00640 1 -- 172.21.15.32:0/1219188988 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v72) ==== 84+0+65 (secure 0 0 0) 0x7f3b900a77b0 con 0x7f3b98109730 2024-04-30T08:31:49.277 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:31:49.277 INFO:teuthology.orchestra.run.smithi032.stdout:[ 2024-04-30T08:31:49.277 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:31:49.277 INFO:teuthology.orchestra.run.smithi032.stdout: "poolnum": 1, 2024-04-30T08:31:49.277 INFO:teuthology.orchestra.run.smithi032.stdout: "poolname": ".mgr" 2024-04-30T08:31:49.277 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:31:49.277 INFO:teuthology.orchestra.run.smithi032.stdout:] 2024-04-30T08:31:49.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.278+0000 7f3b9d600640 1 -- 172.21.15.32:0/1219188988 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3b7c0afae0 msgr2=0x7f3b7c0b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:49.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.278+0000 7f3b9d600640 1 --2- 172.21.15.32:0/1219188988 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3b7c0afae0 0x7f3b7c0b1ff0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f3b880096c0 tx=0x7f3b880314a0 comp rx=0 tx=0).stop 2024-04-30T08:31:49.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.278+0000 7f3b9d600640 1 -- 172.21.15.32:0/1219188988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b98109730 msgr2=0x7f3b981a37b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:49.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.278+0000 7f3b9d600640 1 --2- 172.21.15.32:0/1219188988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b98109730 0x7f3b981a37b0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f3b90005fd0 tx=0x7f3b9000ef10 comp rx=0 tx=0).stop 2024-04-30T08:31:49.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.278+0000 7f3b9d600640 1 -- 172.21.15.32:0/1219188988 shutdown_connections 2024-04-30T08:31:49.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.278+0000 7f3b9d600640 1 --2- 172.21.15.32:0/1219188988 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3b7c0afae0 0x7f3b7c0b1ff0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:49.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.278+0000 7f3b9d600640 1 --2- 172.21.15.32:0/1219188988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b98109730 0x7f3b981a37b0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:49.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.278+0000 7f3b9d600640 1 --2- 172.21.15.32:0/1219188988 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3b98106dd0 0x7f3b9819c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:49.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.278+0000 7f3b9d600640 1 --2- 172.21.15.32:0/1219188988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3b98104470 0x7f3b9819c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:49.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.278+0000 7f3b9d600640 1 -- 172.21.15.32:0/1219188988 >> 172.21.15.32:0/1219188988 conn(0x7f3b980fda70 msgr2=0x7f3b98101e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:49.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.278+0000 7f3b9d600640 1 -- 172.21.15.32:0/1219188988 shutdown_connections 2024-04-30T08:31:49.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.278+0000 7f3b9d600640 1 -- 172.21.15.32:0/1219188988 wait complete. 2024-04-30T08:31:49.307 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-30T08:31:49.307 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-30T08:31:49.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:49 smithi032 ceph-mon[28331]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:49.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:49 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:49.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:49 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:49.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:49 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:49.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:49 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:31:49.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:49 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:49.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:49 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/392191197' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:31:49.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:49 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:31:49.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:49 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2670351864' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:31:49.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:49 smithi189 ceph-mon[33038]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:49.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:49 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:49.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:49 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:49.446 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:49 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:49.447 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:49 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:31:49.447 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:49 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:49.447 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:49 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/392191197' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:31:49.447 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:49 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:31:49.447 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:49 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2670351864' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:31:49.455 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:49 smithi071 ceph-mon[33385]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:49.455 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:49 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:49.455 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:49 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:49.455 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:49 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:49.455 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:49 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:31:49.456 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:49 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:49.456 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:49 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/392191197' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:31:49.456 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:49 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:31:49.456 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:49 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2670351864' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:31:49.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.563+0000 7f3eb8c00640 1 -- 172.21.15.32:0/955666932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3eb4109fc0 msgr2=0x7f3eb410acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:49.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.563+0000 7f3eb8c00640 1 --2- 172.21.15.32:0/955666932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3eb4109fc0 0x7f3eb410acc0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4009900 tx=0x7f3ea402a410 comp rx=0 tx=0).stop 2024-04-30T08:31:49.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.563+0000 7f3eb8c00640 1 -- 172.21.15.32:0/955666932 shutdown_connections 2024-04-30T08:31:49.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.563+0000 7f3eb8c00640 1 --2- 172.21.15.32:0/955666932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3eb410ff00 0x7f3eb4112320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:49.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.563+0000 7f3eb8c00640 1 --2- 172.21.15.32:0/955666932 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3eb410b200 0x7f3eb410b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:49.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.563+0000 7f3eb8c00640 1 --2- 172.21.15.32:0/955666932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3eb4109fc0 0x7f3eb410acc0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:49.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.563+0000 7f3eb8c00640 1 -- 172.21.15.32:0/955666932 >> 172.21.15.32:0/955666932 conn(0x7f3eb4078bb0 msgr2=0x7f3eb4107a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:49.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.563+0000 7f3eb8c00640 1 -- 172.21.15.32:0/955666932 shutdown_connections 2024-04-30T08:31:49.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.563+0000 7f3eb8c00640 1 -- 172.21.15.32:0/955666932 wait complete. 2024-04-30T08:31:49.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.564+0000 7f3eb8c00640 1 Processor -- start 2024-04-30T08:31:49.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.564+0000 7f3eb8c00640 1 -- start start 2024-04-30T08:31:49.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.564+0000 7f3eb8c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3eb4109fc0 0x7f3eb41a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:49.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.564+0000 7f3eb8c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3eb410b200 0x7f3eb41a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:49.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.564+0000 7f3eb8c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3eb410ff00 0x7f3eb41a7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:49.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.564+0000 7f3eb8c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb4115ae0 con 0x7f3eb4109fc0 2024-04-30T08:31:49.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.564+0000 7f3eb8c00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb4115960 con 0x7f3eb410b200 2024-04-30T08:31:49.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.564+0000 7f3eb8c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb4115c60 con 0x7f3eb410ff00 2024-04-30T08:31:49.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.565+0000 7f3eb3e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3eb410ff00 0x7f3eb41a7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:49.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.565+0000 7f3eb3e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3eb410ff00 0x7f3eb41a7ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:49454/0 (socket says 172.21.15.32:49454) 2024-04-30T08:31:49.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.565+0000 7f3eb3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3eb4109fc0 0x7f3eb41a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:49.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.565+0000 7f3eb3e00640 1 -- 172.21.15.32:0/3336734906 learned_addr learned my addr 172.21.15.32:0/3336734906 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:31:49.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.565+0000 7f3eaaa00640 1 --2- 172.21.15.32:0/3336734906 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3eb410b200 0x7f3eb41a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:49.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.565+0000 7f3eb3e00640 1 -- 172.21.15.32:0/3336734906 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3eb410b200 msgr2=0x7f3eb41a0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:49.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.565+0000 7f3eb3e00640 1 --2- 172.21.15.32:0/3336734906 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3eb410b200 0x7f3eb41a0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:49.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.565+0000 7f3eb3e00640 1 -- 172.21.15.32:0/3336734906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3eb4109fc0 msgr2=0x7f3eb41a06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:49.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.565+0000 7f3eb3e00640 1 --2- 172.21.15.32:0/3336734906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3eb4109fc0 0x7f3eb41a06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:49.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.565+0000 7f3eb3e00640 1 -- 172.21.15.32:0/3336734906 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3eb41a8210 con 0x7f3eb410ff00 2024-04-30T08:31:49.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.565+0000 7f3eb3e00640 1 --2- 172.21.15.32:0/3336734906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3eb410ff00 0x7f3eb41a7ba0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3eac00e970 tx=0x7f3eac00ed70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:49.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.566+0000 7f3eb0c00640 1 -- 172.21.15.32:0/3336734906 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eac00ccc0 con 0x7f3eb410ff00 2024-04-30T08:31:49.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.566+0000 7f3eb8c00640 1 -- 172.21.15.32:0/3336734906 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3eb41a8500 con 0x7f3eb410ff00 2024-04-30T08:31:49.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.566+0000 7f3eb8c00640 1 -- 172.21.15.32:0/3336734906 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3eb41a8a50 con 0x7f3eb410ff00 2024-04-30T08:31:49.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.566+0000 7f3eb0c00640 1 -- 172.21.15.32:0/3336734906 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3eac00ce60 con 0x7f3eb410ff00 2024-04-30T08:31:49.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.566+0000 7f3eb0c00640 1 -- 172.21.15.32:0/3336734906 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eac010430 con 0x7f3eb410ff00 2024-04-30T08:31:49.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.567+0000 7f3eb8c00640 1 -- 172.21.15.32:0/3336734906 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e900052c0 con 0x7f3eb410ff00 2024-04-30T08:31:49.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.569+0000 7f3eb0c00640 1 -- 172.21.15.32:0/3336734906 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f3eac0046c0 con 0x7f3eb410ff00 2024-04-30T08:31:49.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.571+0000 7f3eb0c00640 1 --2- 172.21.15.32:0/3336734906 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3e980afae0 0x7f3e980b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:49.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.571+0000 7f3eb0c00640 1 -- 172.21.15.32:0/3336734906 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(72..72 src has 1..72) ==== 7127+0+0 (secure 0 0 0) 0x7f3eac0deb40 con 0x7f3eb410ff00 2024-04-30T08:31:49.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.571+0000 7f3eb3400640 1 --2- 172.21.15.32:0/3336734906 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3e980afae0 0x7f3e980b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:49.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.572+0000 7f3eb3400640 1 --2- 172.21.15.32:0/3336734906 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3e980afae0 0x7f3e980b1ff0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4005ab0 tx=0x7f3ea4031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:49.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.575+0000 7f3eb0c00640 1 -- 172.21.15.32:0/3336734906 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3eac0a7b20 con 0x7f3eb410ff00 2024-04-30T08:31:49.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:49.777+0000 7f3eb8c00640 1 -- 172.21.15.32:0/3336734906 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f3e900055b0 con 0x7f3eb410ff00 2024-04-30T08:31:50.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:50 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1219188988' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-30T08:31:50.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:50 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:50.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:50 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3336734906' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-30T08:31:50.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:50 smithi032 ceph-mon[28331]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-30T08:31:50.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:50 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1219188988' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-30T08:31:50.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:50 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:50.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:50 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3336734906' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-30T08:31:50.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:50 smithi071 ceph-mon[33385]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-30T08:31:50.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:50 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1219188988' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-30T08:31:50.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:50 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:50.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:50 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3336734906' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-30T08:31:50.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:50 smithi189 ceph-mon[33038]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-30T08:31:50.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.658+0000 7f3eb0c00640 1 -- 172.21.15.32:0/3336734906 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v73) ==== 138+0+0 (secure 0 0 0) 0x7f3eac0a7420 con 0x7f3eb410ff00 2024-04-30T08:31:50.662 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' created 2024-04-30T08:31:50.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.664+0000 7f3eb8c00640 1 -- 172.21.15.32:0/3336734906 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3e980afae0 msgr2=0x7f3e980b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:50.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.664+0000 7f3eb8c00640 1 --2- 172.21.15.32:0/3336734906 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3e980afae0 0x7f3e980b1ff0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4005ab0 tx=0x7f3ea4031400 comp rx=0 tx=0).stop 2024-04-30T08:31:50.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.664+0000 7f3eb8c00640 1 -- 172.21.15.32:0/3336734906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3eb410ff00 msgr2=0x7f3eb41a7ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:50.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.664+0000 7f3eb8c00640 1 --2- 172.21.15.32:0/3336734906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3eb410ff00 0x7f3eb41a7ba0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3eac00e970 tx=0x7f3eac00ed70 comp rx=0 tx=0).stop 2024-04-30T08:31:50.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.664+0000 7f3eb8c00640 1 -- 172.21.15.32:0/3336734906 shutdown_connections 2024-04-30T08:31:50.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.664+0000 7f3eb8c00640 1 --2- 172.21.15.32:0/3336734906 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3e980afae0 0x7f3e980b1ff0 secure :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f3ea4005ab0 tx=0x7f3ea4031400 comp rx=0 tx=0).stop 2024-04-30T08:31:50.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.664+0000 7f3eb8c00640 1 --2- 172.21.15.32:0/3336734906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3eb410ff00 0x7f3eb41a7ba0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:50.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.664+0000 7f3eb8c00640 1 --2- 172.21.15.32:0/3336734906 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3eb410b200 0x7f3eb41a0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:50.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.664+0000 7f3eb8c00640 1 --2- 172.21.15.32:0/3336734906 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3eb4109fc0 0x7f3eb41a06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:50.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.664+0000 7f3eb8c00640 1 -- 172.21.15.32:0/3336734906 >> 172.21.15.32:0/3336734906 conn(0x7f3eb4078bb0 msgr2=0x7f3eb410bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:50.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.666+0000 7f3eb8c00640 1 -- 172.21.15.32:0/3336734906 shutdown_connections 2024-04-30T08:31:50.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.667+0000 7f3eb8c00640 1 -- 172.21.15.32:0/3336734906 wait complete. 2024-04-30T08:31:50.700 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-04-30T08:31:50.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.965+0000 7f8ba3e00640 1 -- 172.21.15.32:0/2086587057 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ba4109fc0 msgr2=0x7f8ba410acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:50.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.965+0000 7f8ba3e00640 1 --2- 172.21.15.32:0/2086587057 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ba4109fc0 0x7f8ba410acc0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8b94009900 tx=0x7f8b9402a3f0 comp rx=0 tx=0).stop 2024-04-30T08:31:50.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.965+0000 7f8ba3e00640 1 -- 172.21.15.32:0/2086587057 shutdown_connections 2024-04-30T08:31:50.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.965+0000 7f8ba3e00640 1 --2- 172.21.15.32:0/2086587057 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ba410ff00 0x7f8ba4112320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:50.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.965+0000 7f8ba3e00640 1 --2- 172.21.15.32:0/2086587057 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ba410b200 0x7f8ba410b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:50.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.965+0000 7f8ba3e00640 1 --2- 172.21.15.32:0/2086587057 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ba4109fc0 0x7f8ba410acc0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:50.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.965+0000 7f8ba3e00640 1 -- 172.21.15.32:0/2086587057 >> 172.21.15.32:0/2086587057 conn(0x7f8ba4078bb0 msgr2=0x7f8ba4107a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:50.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.965+0000 7f8ba3e00640 1 -- 172.21.15.32:0/2086587057 shutdown_connections 2024-04-30T08:31:50.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.965+0000 7f8ba3e00640 1 -- 172.21.15.32:0/2086587057 wait complete. 2024-04-30T08:31:50.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.966+0000 7f8ba3e00640 1 Processor -- start 2024-04-30T08:31:50.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.966+0000 7f8ba3e00640 1 -- start start 2024-04-30T08:31:50.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.966+0000 7f8ba3e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ba4109fc0 0x7f8ba41a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:50.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.966+0000 7f8ba3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ba410b200 0x7f8ba41a0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:50.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.966+0000 7f8ba3e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ba410ff00 0x7f8ba41a7c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:50.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.966+0000 7f8ba3e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba4115ae0 con 0x7f8ba410b200 2024-04-30T08:31:50.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.966+0000 7f8ba3e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba4115960 con 0x7f8ba410ff00 2024-04-30T08:31:50.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.966+0000 7f8ba3e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8ba4115c60 con 0x7f8ba4109fc0 2024-04-30T08:31:50.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.967+0000 7f8ba2000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ba410b200 0x7f8ba41a0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:50.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.967+0000 7f8ba2a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ba4109fc0 0x7f8ba41a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:50.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.967+0000 7f8ba2a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ba4109fc0 0x7f8ba41a0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:49462/0 (socket says 172.21.15.32:49462) 2024-04-30T08:31:50.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.967+0000 7f8ba2a00640 1 -- 172.21.15.32:0/484590844 learned_addr learned my addr 172.21.15.32:0/484590844 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:31:50.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.967+0000 7f8ba3400640 1 --2- 172.21.15.32:0/484590844 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ba410ff00 0x7f8ba41a7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:50.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.967+0000 7f8ba2a00640 1 -- 172.21.15.32:0/484590844 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ba410ff00 msgr2=0x7f8ba41a7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:50.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.967+0000 7f8ba2a00640 1 --2- 172.21.15.32:0/484590844 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ba410ff00 0x7f8ba41a7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:50.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.967+0000 7f8ba2a00640 1 -- 172.21.15.32:0/484590844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ba410b200 msgr2=0x7f8ba41a0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:50.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.967+0000 7f8ba2a00640 1 --2- 172.21.15.32:0/484590844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ba410b200 0x7f8ba41a0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:50.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.967+0000 7f8ba2a00640 1 -- 172.21.15.32:0/484590844 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ba41a82c0 con 0x7f8ba4109fc0 2024-04-30T08:31:50.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.968+0000 7f8ba2a00640 1 --2- 172.21.15.32:0/484590844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ba4109fc0 0x7f8ba41a0780 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8b94006010 tx=0x7f8b94004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:50.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.968+0000 7f8b9b400640 1 -- 172.21.15.32:0/484590844 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b9403f030 con 0x7f8ba4109fc0 2024-04-30T08:31:50.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.968+0000 7f8ba3e00640 1 -- 172.21.15.32:0/484590844 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ba41a8550 con 0x7f8ba4109fc0 2024-04-30T08:31:50.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.968+0000 7f8b9b400640 1 -- 172.21.15.32:0/484590844 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b94031910 con 0x7f8ba4109fc0 2024-04-30T08:31:50.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.968+0000 7f8b9b400640 1 -- 172.21.15.32:0/484590844 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b94035070 con 0x7f8ba4109fc0 2024-04-30T08:31:50.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.968+0000 7f8ba3e00640 1 -- 172.21.15.32:0/484590844 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ba41a8a40 con 0x7f8ba4109fc0 2024-04-30T08:31:50.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.969+0000 7f8ba3e00640 1 -- 172.21.15.32:0/484590844 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ba410ed10 con 0x7f8ba4109fc0 2024-04-30T08:31:50.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.972+0000 7f8b9b400640 1 -- 172.21.15.32:0/484590844 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f8b94042030 con 0x7f8ba4109fc0 2024-04-30T08:31:50.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.973+0000 7f8b9b400640 1 --2- 172.21.15.32:0/484590844 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8b880afae0 0x7f8b880b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:50.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.973+0000 7f8b9b400640 1 -- 172.21.15.32:0/484590844 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(73..73 src has 1..73) ==== 7497+0+0 (secure 0 0 0) 0x7f8b940ff0a0 con 0x7f8ba4109fc0 2024-04-30T08:31:50.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.974+0000 7f8ba2000640 1 --2- 172.21.15.32:0/484590844 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8b880afae0 0x7f8b880b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:50.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.975+0000 7f8ba2000640 1 --2- 172.21.15.32:0/484590844 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8b880afae0 0x7f8b880b1ff0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8ba41a1820 tx=0x7f8b90005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:50.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:50.975+0000 7f8b9b400640 1 -- 172.21.15.32:0/484590844 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8b940c78c0 con 0x7f8ba4109fc0 2024-04-30T08:31:51.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.191+0000 7f8ba3e00640 1 -- 172.21.15.32:0/484590844 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7f8ba402dbd0 con 0x7f8ba4109fc0 2024-04-30T08:31:51.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:51 smithi071 ceph-mon[33385]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:51.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:51 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:51.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:51 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:51.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:51 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:51.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:51 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:51.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:51 smithi071 ceph-mon[33385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-30T08:31:51.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:51 smithi071 ceph-mon[33385]: osdmap e73: 12 total, 12 up, 12 in 2024-04-30T08:31:51.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:51 smithi189 ceph-mon[33038]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:51.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:51 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:51.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:51 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:51.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:51 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:51.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:51 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:51.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:51 smithi189 ceph-mon[33038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-30T08:31:51.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:51 smithi189 ceph-mon[33038]: osdmap e73: 12 total, 12 up, 12 in 2024-04-30T08:31:51.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.663+0000 7f8b9b400640 1 -- 172.21.15.32:0/484590844 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v74) ==== 171+0+0 (secure 0 0 0) 0x7f8b940cbbc0 con 0x7f8ba4109fc0 2024-04-30T08:31:51.665 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' created 2024-04-30T08:31:51.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.669+0000 7f8ba3e00640 1 -- 172.21.15.32:0/484590844 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8b880afae0 msgr2=0x7f8b880b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:51.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.669+0000 7f8ba3e00640 1 --2- 172.21.15.32:0/484590844 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8b880afae0 0x7f8b880b1ff0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8ba41a1820 tx=0x7f8b90005f90 comp rx=0 tx=0).stop 2024-04-30T08:31:51.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.669+0000 7f8ba3e00640 1 -- 172.21.15.32:0/484590844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ba4109fc0 msgr2=0x7f8ba41a0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:51.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.669+0000 7f8ba3e00640 1 --2- 172.21.15.32:0/484590844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ba4109fc0 0x7f8ba41a0780 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8b94006010 tx=0x7f8b94004270 comp rx=0 tx=0).stop 2024-04-30T08:31:51.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.670+0000 7f8ba3e00640 1 -- 172.21.15.32:0/484590844 shutdown_connections 2024-04-30T08:31:51.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.670+0000 7f8ba3e00640 1 --2- 172.21.15.32:0/484590844 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8b880afae0 0x7f8b880b1ff0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:51.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.670+0000 7f8ba3e00640 1 --2- 172.21.15.32:0/484590844 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ba410ff00 0x7f8ba41a7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:51.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.670+0000 7f8ba3e00640 1 --2- 172.21.15.32:0/484590844 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ba410b200 0x7f8ba41a0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:51.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.670+0000 7f8ba3e00640 1 --2- 172.21.15.32:0/484590844 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ba4109fc0 0x7f8ba41a0780 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:51.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.670+0000 7f8ba3e00640 1 -- 172.21.15.32:0/484590844 >> 172.21.15.32:0/484590844 conn(0x7f8ba4078bb0 msgr2=0x7f8ba410bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:51.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.671+0000 7f8ba3e00640 1 -- 172.21.15.32:0/484590844 shutdown_connections 2024-04-30T08:31:51.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.671+0000 7f8ba3e00640 1 -- 172.21.15.32:0/484590844 wait complete. 2024-04-30T08:31:51.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:51 smithi032 ceph-mon[28331]: pgmap v312: 1 pgs: 1 active+clean; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:51.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:51 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:51.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:51 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:51.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:51 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:51.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:51 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:51.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:51 smithi032 ceph-mon[28331]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-30T08:31:51.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:51 smithi032 ceph-mon[28331]: osdmap e73: 12 total, 12 up, 12 in 2024-04-30T08:31:51.716 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-30T08:31:51.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.971+0000 7f444da00640 1 -- 172.21.15.32:0/2772433993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4448109ab0 msgr2=0x7f444810bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:51.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.971+0000 7f444da00640 1 --2- 172.21.15.32:0/2772433993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4448109ab0 0x7f444810bed0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4438009900 tx=0x7f443802a410 comp rx=0 tx=0).stop 2024-04-30T08:31:51.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.971+0000 7f444da00640 1 -- 172.21.15.32:0/2772433993 shutdown_connections 2024-04-30T08:31:51.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.971+0000 7f444da00640 1 --2- 172.21.15.32:0/2772433993 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4448109ab0 0x7f444810bed0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:51.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.971+0000 7f444da00640 1 --2- 172.21.15.32:0/2772433993 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4448104db0 0x7f44481051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:51.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.971+0000 7f444da00640 1 --2- 172.21.15.32:0/2772433993 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4448103b70 0x7f4448104870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:51.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.971+0000 7f444da00640 1 -- 172.21.15.32:0/2772433993 >> 172.21.15.32:0/2772433993 conn(0x7f44480fed60 msgr2=0x7f4448101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:51.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.971+0000 7f444da00640 1 -- 172.21.15.32:0/2772433993 shutdown_connections 2024-04-30T08:31:51.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.971+0000 7f444da00640 1 -- 172.21.15.32:0/2772433993 wait complete. 2024-04-30T08:31:51.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.972+0000 7f444da00640 1 Processor -- start 2024-04-30T08:31:51.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.973+0000 7f444da00640 1 -- start start 2024-04-30T08:31:51.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.973+0000 7f444da00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4448103b70 0x7f44480779e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:51.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.973+0000 7f444da00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4448104db0 0x7f4448076080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:51.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.973+0000 7f444da00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4448109ab0 0x7f44480765c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:51.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.973+0000 7f444da00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f444810f6d0 con 0x7f4448103b70 2024-04-30T08:31:51.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.973+0000 7f444da00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f444810f550 con 0x7f4448109ab0 2024-04-30T08:31:51.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.973+0000 7f444da00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f444810f850 con 0x7f4448104db0 2024-04-30T08:31:51.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.973+0000 7f4446a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4448103b70 0x7f44480779e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:51.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.973+0000 7f4446a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4448103b70 0x7f44480779e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43348/0 (socket says 172.21.15.32:43348) 2024-04-30T08:31:51.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.973+0000 7f4446a00640 1 -- 172.21.15.32:0/597884215 learned_addr learned my addr 172.21.15.32:0/597884215 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:31:51.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.973+0000 7f4446000640 1 --2- 172.21.15.32:0/597884215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4448104db0 0x7f4448076080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:51.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.973+0000 7f4447400640 1 --2- 172.21.15.32:0/597884215 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4448109ab0 0x7f44480765c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:51.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.974+0000 7f4446a00640 1 -- 172.21.15.32:0/597884215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4448104db0 msgr2=0x7f4448076080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:51.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.974+0000 7f4446a00640 1 --2- 172.21.15.32:0/597884215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4448104db0 0x7f4448076080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:51.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.974+0000 7f4446a00640 1 -- 172.21.15.32:0/597884215 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4448109ab0 msgr2=0x7f44480765c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:51.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.974+0000 7f4446a00640 1 --2- 172.21.15.32:0/597884215 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4448109ab0 0x7f44480765c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:51.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.974+0000 7f4446a00640 1 -- 172.21.15.32:0/597884215 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4448076cf0 con 0x7f4448103b70 2024-04-30T08:31:51.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.974+0000 7f4446a00640 1 --2- 172.21.15.32:0/597884215 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4448103b70 0x7f44480779e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f444000ea50 tx=0x7f444000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:51.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.974+0000 7f443f400640 1 -- 172.21.15.32:0/597884215 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f444000cdc0 con 0x7f4448103b70 2024-04-30T08:31:51.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.975+0000 7f443f400640 1 -- 172.21.15.32:0/597884215 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4440011920 con 0x7f4448103b70 2024-04-30T08:31:51.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.975+0000 7f444da00640 1 -- 172.21.15.32:0/597884215 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44481a42d0 con 0x7f4448103b70 2024-04-30T08:31:51.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.975+0000 7f443f400640 1 -- 172.21.15.32:0/597884215 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f444001aae0 con 0x7f4448103b70 2024-04-30T08:31:51.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.975+0000 7f444da00640 1 -- 172.21.15.32:0/597884215 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44481a4790 con 0x7f4448103b70 2024-04-30T08:31:51.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.976+0000 7f444da00640 1 -- 172.21.15.32:0/597884215 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f44481088c0 con 0x7f4448103b70 2024-04-30T08:31:51.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.978+0000 7f443f400640 1 -- 172.21.15.32:0/597884215 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f444001ac80 con 0x7f4448103b70 2024-04-30T08:31:51.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.980+0000 7f443f400640 1 --2- 172.21.15.32:0/597884215 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f442c0afaf0 0x7f442c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:51.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.980+0000 7f443f400640 1 -- 172.21.15.32:0/597884215 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(74..74 src has 1..74) ==== 7879+0+0 (secure 0 0 0) 0x7f44400df170 con 0x7f4448103b70 2024-04-30T08:31:51.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.980+0000 7f4446000640 1 --2- 172.21.15.32:0/597884215 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f442c0afaf0 0x7f442c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:51.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.981+0000 7f4446000640 1 --2- 172.21.15.32:0/597884215 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f442c0afaf0 0x7f442c0b2000 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4448077380 tx=0x7f4430005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:51.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:51.983+0000 7f443f400640 1 -- 172.21.15.32:0/597884215 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4440055ff0 con 0x7f4448103b70 2024-04-30T08:31:52.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:52.234+0000 7f444da00640 1 -- 172.21.15.32:0/597884215 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f444802dbd0 con 0x7f4448103b70 2024-04-30T08:31:52.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:52 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/484590844' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-30T08:31:52.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:52 smithi071 ceph-mon[33385]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-30T08:31:52.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:52 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:52.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:52 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:52.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:52 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:52.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:52 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:31:52.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:52 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:52.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:52 smithi071 ceph-mon[33385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-30T08:31:52.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:52 smithi071 ceph-mon[33385]: osdmap e74: 12 total, 12 up, 12 in 2024-04-30T08:31:52.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:52 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/484590844' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-30T08:31:52.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:52 smithi189 ceph-mon[33038]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-30T08:31:52.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:52 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:52.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:52 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:52.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:52 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:52.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:52 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:31:52.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:52 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:52.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:52 smithi189 ceph-mon[33038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-30T08:31:52.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:52 smithi189 ceph-mon[33038]: osdmap e74: 12 total, 12 up, 12 in 2024-04-30T08:31:52.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:52 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/484590844' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-30T08:31:52.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:52 smithi032 ceph-mon[28331]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-04-30T08:31:52.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:52.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:52.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:31:52.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:31:52.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:52 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:31:52.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:52 smithi032 ceph-mon[28331]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-04-30T08:31:52.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:52 smithi032 ceph-mon[28331]: osdmap e74: 12 total, 12 up, 12 in 2024-04-30T08:31:53.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.229+0000 7f443f400640 1 -- 172.21.15.32:0/597884215 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f4440016030 con 0x7f4448103b70 2024-04-30T08:31:53.231 INFO:teuthology.orchestra.run.smithi032.stderr:new fs with metadata pool 2 and data pool 3 2024-04-30T08:31:53.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.233+0000 7f444da00640 1 -- 172.21.15.32:0/597884215 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f442c0afaf0 msgr2=0x7f442c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:53.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.233+0000 7f444da00640 1 --2- 172.21.15.32:0/597884215 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f442c0afaf0 0x7f442c0b2000 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f4448077380 tx=0x7f4430005f90 comp rx=0 tx=0).stop 2024-04-30T08:31:53.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.233+0000 7f444da00640 1 -- 172.21.15.32:0/597884215 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4448103b70 msgr2=0x7f44480779e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:53.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.233+0000 7f444da00640 1 --2- 172.21.15.32:0/597884215 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4448103b70 0x7f44480779e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f444000ea50 tx=0x7f444000ef10 comp rx=0 tx=0).stop 2024-04-30T08:31:53.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.233+0000 7f444da00640 1 -- 172.21.15.32:0/597884215 shutdown_connections 2024-04-30T08:31:53.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.233+0000 7f444da00640 1 --2- 172.21.15.32:0/597884215 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f442c0afaf0 0x7f442c0b2000 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:53.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.233+0000 7f444da00640 1 --2- 172.21.15.32:0/597884215 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4448109ab0 0x7f44480765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:53.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.233+0000 7f444da00640 1 --2- 172.21.15.32:0/597884215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4448104db0 0x7f4448076080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:53.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.233+0000 7f444da00640 1 --2- 172.21.15.32:0/597884215 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4448103b70 0x7f44480779e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:53.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.233+0000 7f444da00640 1 -- 172.21.15.32:0/597884215 >> 172.21.15.32:0/597884215 conn(0x7f44480fed60 msgr2=0x7f4448105a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:53.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.234+0000 7f444da00640 1 -- 172.21.15.32:0/597884215 shutdown_connections 2024-04-30T08:31:53.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.234+0000 7f444da00640 1 -- 172.21.15.32:0/597884215 wait complete. 2024-04-30T08:31:53.271 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-04-30T08:31:53.271 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-04-30T08:31:53.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:53 smithi071 ceph-mon[33385]: pgmap v315: 97 pgs: 4 creating+peering, 1 active+clean, 92 unknown; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:53.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:53 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/597884215' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-30T08:31:53.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:53 smithi071 ceph-mon[33385]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-30T08:31:53.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:53 smithi071 ceph-mon[33385]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-30T08:31:53.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:53 smithi071 ceph-mon[33385]: osdmap e75: 12 total, 12 up, 12 in 2024-04-30T08:31:53.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.547+0000 7f2743000640 1 -- 172.21.15.32:0/54315019 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f273c075010 msgr2=0x7f273c075420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:53.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.547+0000 7f2743000640 1 --2- 172.21.15.32:0/54315019 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f273c075010 0x7f273c075420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2730009900 tx=0x7f273002a410 comp rx=0 tx=0).stop 2024-04-30T08:31:53.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.548+0000 7f2743000640 1 -- 172.21.15.32:0/54315019 shutdown_connections 2024-04-30T08:31:53.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.548+0000 7f2743000640 1 --2- 172.21.15.32:0/54315019 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f273c10b920 0x7f273c10dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:53.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.548+0000 7f2743000640 1 --2- 172.21.15.32:0/54315019 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f273c075960 0x7f273c10b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:53.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.548+0000 7f2743000640 1 --2- 172.21.15.32:0/54315019 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f273c075010 0x7f273c075420 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:53.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.548+0000 7f2743000640 1 -- 172.21.15.32:0/54315019 >> 172.21.15.32:0/54315019 conn(0x7f273c0feba0 msgr2=0x7f273c101050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:53.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.548+0000 7f2743000640 1 -- 172.21.15.32:0/54315019 shutdown_connections 2024-04-30T08:31:53.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.548+0000 7f2743000640 1 -- 172.21.15.32:0/54315019 wait complete. 2024-04-30T08:31:53.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:53 smithi189 ceph-mon[33038]: pgmap v315: 97 pgs: 4 creating+peering, 1 active+clean, 92 unknown; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:53.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:53 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/597884215' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-30T08:31:53.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:53 smithi189 ceph-mon[33038]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-30T08:31:53.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:53 smithi189 ceph-mon[33038]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-30T08:31:53.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:53 smithi189 ceph-mon[33038]: osdmap e75: 12 total, 12 up, 12 in 2024-04-30T08:31:53.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.549+0000 7f2743000640 1 Processor -- start 2024-04-30T08:31:53.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.549+0000 7f2743000640 1 -- start start 2024-04-30T08:31:53.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.549+0000 7f2743000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f273c075010 0x7f273c19c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:53.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.549+0000 7f2743000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f273c075960 0x7f273c19c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:53.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.549+0000 7f2743000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f273c10b920 0x7f273c1a37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:53.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.549+0000 7f2743000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f273c111730 con 0x7f273c10b920 2024-04-30T08:31:53.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.549+0000 7f2743000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f273c1115b0 con 0x7f273c075960 2024-04-30T08:31:53.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.549+0000 7f2743000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f273c1118b0 con 0x7f273c075010 2024-04-30T08:31:53.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.550+0000 7f2742600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f273c10b920 0x7f273c1a37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:53.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.550+0000 7f2741c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f273c075010 0x7f273c19c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:53.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.550+0000 7f2741c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f273c075010 0x7f273c19c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:49482/0 (socket says 172.21.15.32:49482) 2024-04-30T08:31:53.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.550+0000 7f2741c00640 1 -- 172.21.15.32:0/1777503023 learned_addr learned my addr 172.21.15.32:0/1777503023 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:31:53.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.550+0000 7f2741200640 1 --2- 172.21.15.32:0/1777503023 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f273c075960 0x7f273c19c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:53.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.550+0000 7f2742600640 1 -- 172.21.15.32:0/1777503023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f273c075010 msgr2=0x7f273c19c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:53.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.550+0000 7f2742600640 1 --2- 172.21.15.32:0/1777503023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f273c075010 0x7f273c19c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:53.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.550+0000 7f2742600640 1 -- 172.21.15.32:0/1777503023 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f273c075960 msgr2=0x7f273c19c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:53.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.550+0000 7f2742600640 1 --2- 172.21.15.32:0/1777503023 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f273c075960 0x7f273c19c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:53.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.550+0000 7f2742600640 1 -- 172.21.15.32:0/1777503023 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f273c1a3e10 con 0x7f273c10b920 2024-04-30T08:31:53.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.550+0000 7f2741c00640 1 --2- 172.21.15.32:0/1777503023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f273c075010 0x7f273c19c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:31:53.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.551+0000 7f2742600640 1 --2- 172.21.15.32:0/1777503023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f273c10b920 0x7f273c1a37a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2734005fd0 tx=0x7f273400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:53.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.551+0000 7f273a000640 1 -- 172.21.15.32:0/1777503023 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f273400cdf0 con 0x7f273c10b920 2024-04-30T08:31:53.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.551+0000 7f273a000640 1 -- 172.21.15.32:0/1777503023 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2734005020 con 0x7f273c10b920 2024-04-30T08:31:53.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.551+0000 7f2743000640 1 -- 172.21.15.32:0/1777503023 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f273c1a4100 con 0x7f273c10b920 2024-04-30T08:31:53.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.551+0000 7f273a000640 1 -- 172.21.15.32:0/1777503023 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2734010690 con 0x7f273c10b920 2024-04-30T08:31:53.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.551+0000 7f2743000640 1 -- 172.21.15.32:0/1777503023 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f273c1a4650 con 0x7f273c10b920 2024-04-30T08:31:53.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.553+0000 7f2743000640 1 -- 172.21.15.32:0/1777503023 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f273c10a1f0 con 0x7f273c10b920 2024-04-30T08:31:53.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.555+0000 7f273a000640 1 -- 172.21.15.32:0/1777503023 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f2734005460 con 0x7f273c10b920 2024-04-30T08:31:53.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.558+0000 7f273a000640 1 --2- 172.21.15.32:0/1777503023 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f27240afae0 0x7f27240b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:53.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.558+0000 7f273a000640 1 -- 172.21.15.32:0/1777503023 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 7979+0+0 (secure 0 0 0) 0x7f27340df2a0 con 0x7f273c10b920 2024-04-30T08:31:53.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.558+0000 7f2741c00640 1 --2- 172.21.15.32:0/1777503023 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f27240afae0 0x7f27240b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:53.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.559+0000 7f2741c00640 1 --2- 172.21.15.32:0/1777503023 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f27240afae0 0x7f27240b1ff0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f27300096c0 tx=0x7f27300314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:53.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.561+0000 7f273a000640 1 -- 172.21.15.32:0/1777503023 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f27340a7fc0 con 0x7f273c10b920 2024-04-30T08:31:53.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:53 smithi032 ceph-mon[28331]: pgmap v315: 97 pgs: 4 creating+peering, 1 active+clean, 92 unknown; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:31:53.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:53 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/597884215' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-30T08:31:53.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:53 smithi032 ceph-mon[28331]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-04-30T08:31:53.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:53 smithi032 ceph-mon[28331]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-30T08:31:53.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:53 smithi032 ceph-mon[28331]: osdmap e75: 12 total, 12 up, 12 in 2024-04-30T08:31:53.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:53.781+0000 7f2743000640 1 -- 172.21.15.32:0/1777503023 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f273c02dbd0 con 0x7f273c10b920 2024-04-30T08:31:54.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:54 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/597884215' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-30T08:31:54.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:54 smithi071 ceph-mon[33385]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-30T08:31:54.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:54 smithi071 ceph-mon[33385]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-30T08:31:54.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:54 smithi071 ceph-mon[33385]: osdmap e76: 12 total, 12 up, 12 in 2024-04-30T08:31:54.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:54 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1777503023' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-30T08:31:54.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:54 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/597884215' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-30T08:31:54.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:54 smithi189 ceph-mon[33038]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-30T08:31:54.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:54 smithi189 ceph-mon[33038]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-30T08:31:54.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:54 smithi189 ceph-mon[33038]: osdmap e76: 12 total, 12 up, 12 in 2024-04-30T08:31:54.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:54 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1777503023' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-30T08:31:54.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:54.666+0000 7f273a000640 1 -- 172.21.15.32:0/1777503023 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v77) ==== 157+0+0 (secure 0 0 0) 0x7f27340a78c0 con 0x7f273c10b920 2024-04-30T08:31:54.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:54.670+0000 7f2743000640 1 -- 172.21.15.32:0/1777503023 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f27240afae0 msgr2=0x7f27240b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:54.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:54.670+0000 7f2743000640 1 --2- 172.21.15.32:0/1777503023 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f27240afae0 0x7f27240b1ff0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f27300096c0 tx=0x7f27300314a0 comp rx=0 tx=0).stop 2024-04-30T08:31:54.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:54.670+0000 7f2743000640 1 -- 172.21.15.32:0/1777503023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f273c10b920 msgr2=0x7f273c1a37a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:54.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:54.670+0000 7f2743000640 1 --2- 172.21.15.32:0/1777503023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f273c10b920 0x7f273c1a37a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f2734005fd0 tx=0x7f273400ef10 comp rx=0 tx=0).stop 2024-04-30T08:31:54.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:54.671+0000 7f2743000640 1 -- 172.21.15.32:0/1777503023 shutdown_connections 2024-04-30T08:31:54.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:54.671+0000 7f2743000640 1 --2- 172.21.15.32:0/1777503023 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f27240afae0 0x7f27240b1ff0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:54.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:54.671+0000 7f2743000640 1 --2- 172.21.15.32:0/1777503023 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f273c10b920 0x7f273c1a37a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:54.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:54.671+0000 7f2743000640 1 --2- 172.21.15.32:0/1777503023 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f273c075960 0x7f273c19c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:54.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:54.671+0000 7f2743000640 1 --2- 172.21.15.32:0/1777503023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f273c075010 0x7f273c19c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:54.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:54.671+0000 7f2743000640 1 -- 172.21.15.32:0/1777503023 >> 172.21.15.32:0/1777503023 conn(0x7f273c0feba0 msgr2=0x7f273c1073c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:54.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:54.671+0000 7f2743000640 1 -- 172.21.15.32:0/1777503023 shutdown_connections 2024-04-30T08:31:54.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:54.671+0000 7f2743000640 1 -- 172.21.15.32:0/1777503023 wait complete. 2024-04-30T08:31:54.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:54 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/597884215' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-30T08:31:54.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:54 smithi032 ceph-mon[28331]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-04-30T08:31:54.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:54 smithi032 ceph-mon[28331]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-04-30T08:31:54.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:54 smithi032 ceph-mon[28331]: osdmap e76: 12 total, 12 up, 12 in 2024-04-30T08:31:54.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:54 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1777503023' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-04-30T08:31:54.703 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-04-30T08:31:55.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.001+0000 7f5740a00640 1 -- 172.21.15.32:0/1252499867 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f573c106fd0 msgr2=0x7f573c1073e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:55.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.001+0000 7f5740a00640 1 --2- 172.21.15.32:0/1252499867 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f573c106fd0 0x7f573c1073e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5724009900 tx=0x7f572402a3f0 comp rx=0 tx=0).stop 2024-04-30T08:31:55.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.001+0000 7f5740a00640 1 -- 172.21.15.32:0/1252499867 shutdown_connections 2024-04-30T08:31:55.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.001+0000 7f5740a00640 1 --2- 172.21.15.32:0/1252499867 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f573c10bcd0 0x7f573c10e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:55.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.001+0000 7f5740a00640 1 --2- 172.21.15.32:0/1252499867 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f573c106fd0 0x7f573c1073e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:55.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.001+0000 7f5740a00640 1 --2- 172.21.15.32:0/1252499867 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f573c105d90 0x7f573c106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:55.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.001+0000 7f5740a00640 1 -- 172.21.15.32:0/1252499867 >> 172.21.15.32:0/1252499867 conn(0x7f573c100fa0 msgr2=0x7f573c103430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:55.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.002+0000 7f5740a00640 1 -- 172.21.15.32:0/1252499867 shutdown_connections 2024-04-30T08:31:55.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.002+0000 7f5740a00640 1 -- 172.21.15.32:0/1252499867 wait complete. 2024-04-30T08:31:55.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.003+0000 7f5740a00640 1 Processor -- start 2024-04-30T08:31:55.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.003+0000 7f5740a00640 1 -- start start 2024-04-30T08:31:55.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.004+0000 7f5740a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f573c105d90 0x7f573c19c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:55.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.004+0000 7f5740a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f573c106fd0 0x7f573c19c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:55.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.004+0000 7f5740a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f573c10bcd0 0x7f573c1a3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:55.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.004+0000 7f5740a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f573c111890 con 0x7f573c105d90 2024-04-30T08:31:55.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.004+0000 7f5740a00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f573c111710 con 0x7f573c106fd0 2024-04-30T08:31:55.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.004+0000 7f5740a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f573c111a10 con 0x7f573c10bcd0 2024-04-30T08:31:55.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.004+0000 7f5739e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f573c105d90 0x7f573c19c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:55.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.004+0000 7f5733e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f573c106fd0 0x7f573c19c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:55.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.004+0000 7f5739e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f573c105d90 0x7f573c19c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43390/0 (socket says 172.21.15.32:43390) 2024-04-30T08:31:55.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.004+0000 7f5733e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f573c106fd0 0x7f573c19c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:47344/0 (socket says 172.21.15.32:47344) 2024-04-30T08:31:55.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.004+0000 7f5739e00640 1 -- 172.21.15.32:0/519933124 learned_addr learned my addr 172.21.15.32:0/519933124 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:31:55.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.004+0000 7f573a800640 1 --2- 172.21.15.32:0/519933124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f573c10bcd0 0x7f573c1a3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:55.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.005+0000 7f5739e00640 1 -- 172.21.15.32:0/519933124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f573c10bcd0 msgr2=0x7f573c1a3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:55.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.005+0000 7f5739e00640 1 --2- 172.21.15.32:0/519933124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f573c10bcd0 0x7f573c1a3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:55.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.005+0000 7f5739e00640 1 -- 172.21.15.32:0/519933124 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f573c106fd0 msgr2=0x7f573c19c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:55.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.005+0000 7f5739e00640 1 --2- 172.21.15.32:0/519933124 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f573c106fd0 0x7f573c19c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:55.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.005+0000 7f5739e00640 1 -- 172.21.15.32:0/519933124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f573c1a3fa0 con 0x7f573c105d90 2024-04-30T08:31:55.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.005+0000 7f5739e00640 1 --2- 172.21.15.32:0/519933124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f573c105d90 0x7f573c19c460 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f572c00e9f0 tx=0x7f572c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:55.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.005+0000 7f5732a00640 1 -- 172.21.15.32:0/519933124 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f572c00ccc0 con 0x7f573c105d90 2024-04-30T08:31:55.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.005+0000 7f5740a00640 1 -- 172.21.15.32:0/519933124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f573c1a4290 con 0x7f573c105d90 2024-04-30T08:31:55.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.005+0000 7f5732a00640 1 -- 172.21.15.32:0/519933124 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f572c0035f0 con 0x7f573c105d90 2024-04-30T08:31:55.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.005+0000 7f5732a00640 1 -- 172.21.15.32:0/519933124 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f572c00ce60 con 0x7f573c105d90 2024-04-30T08:31:55.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.006+0000 7f5740a00640 1 -- 172.21.15.32:0/519933124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f573c1a47e0 con 0x7f573c105d90 2024-04-30T08:31:55.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.007+0000 7f5740a00640 1 -- 172.21.15.32:0/519933124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f573c10aae0 con 0x7f573c105d90 2024-04-30T08:31:55.011 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.009+0000 7f5732a00640 1 -- 172.21.15.32:0/519933124 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f572c003aa0 con 0x7f573c105d90 2024-04-30T08:31:55.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.011+0000 7f5732a00640 1 --2- 172.21.15.32:0/519933124 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f571c0afaf0 0x7f571c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:55.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.011+0000 7f5732a00640 1 -- 172.21.15.32:0/519933124 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(77..77 src has 1..77) ==== 8277+0+0 (secure 0 0 0) 0x7f572c0df090 con 0x7f573c105d90 2024-04-30T08:31:55.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.011+0000 7f5733e00640 1 --2- 172.21.15.32:0/519933124 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f571c0afaf0 0x7f571c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:55.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.012+0000 7f5733e00640 1 --2- 172.21.15.32:0/519933124 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f571c0afaf0 0x7f571c0b2000 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5724009230 tx=0x7f5724031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:55.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.013+0000 7f5732a00640 1 -- 172.21.15.32:0/519933124 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f572c0a7be0 con 0x7f573c105d90 2024-04-30T08:31:55.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:55.224+0000 7f5740a00640 1 -- 172.21.15.32:0/519933124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f573c02dbd0 con 0x7f573c105d90 2024-04-30T08:31:55.405 INFO:journalctl@ceph.mds.b.smithi071.stdout:Apr 30 08:31:55 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-b[64026]: 2024-04-30T08:31:55.111+0000 7f6ea3abc700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-30T08:31:55.406 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:55 smithi071 ceph-mon[33385]: pgmap v318: 97 pgs: 27 active+clean, 4 creating+peering, 66 unknown; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-04-30T08:31:55.406 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:55 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:31:55.406 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:55 smithi071 ceph-mon[33385]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-30T08:31:55.406 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:55 smithi071 ceph-mon[33385]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-30T08:31:55.406 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:55 smithi071 ceph-mon[33385]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-30T08:31:55.407 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:55 smithi071 ceph-mon[33385]: Cluster is now healthy 2024-04-30T08:31:55.407 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:55 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1777503023' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-30T08:31:55.407 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:55 smithi071 ceph-mon[33385]: osdmap e77: 12 total, 12 up, 12 in 2024-04-30T08:31:55.407 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:55 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/519933124' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-30T08:31:55.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:55 smithi189 ceph-mon[33038]: pgmap v318: 97 pgs: 27 active+clean, 4 creating+peering, 66 unknown; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-04-30T08:31:55.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:55 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:31:55.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:55 smithi189 ceph-mon[33038]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-30T08:31:55.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:55 smithi189 ceph-mon[33038]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-30T08:31:55.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:55 smithi189 ceph-mon[33038]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-30T08:31:55.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:55 smithi189 ceph-mon[33038]: Cluster is now healthy 2024-04-30T08:31:55.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:55 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1777503023' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-30T08:31:55.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:55 smithi189 ceph-mon[33038]: osdmap e77: 12 total, 12 up, 12 in 2024-04-30T08:31:55.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:55 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/519933124' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-30T08:31:55.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:55 smithi032 ceph-mon[28331]: pgmap v318: 97 pgs: 27 active+clean, 4 creating+peering, 66 unknown; 577 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-04-30T08:31:55.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:55 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:31:55.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:55 smithi032 ceph-mon[28331]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-30T08:31:55.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:55 smithi032 ceph-mon[28331]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-30T08:31:55.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:55 smithi032 ceph-mon[28331]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-04-30T08:31:55.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:55 smithi032 ceph-mon[28331]: Cluster is now healthy 2024-04-30T08:31:55.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:55 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1777503023' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-04-30T08:31:55.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:55 smithi032 ceph-mon[28331]: osdmap e77: 12 total, 12 up, 12 in 2024-04-30T08:31:55.694 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:55 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/519933124' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-30T08:31:56.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:56.700+0000 7f5732a00640 1 -- 172.21.15.32:0/519933124 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v79) ==== 228+0+0 (secure 0 0 0) 0x7f572c0a74e0 con 0x7f573c105d90 2024-04-30T08:31:56.701 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data_ec' created 2024-04-30T08:31:56.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:56.706+0000 7f5740a00640 1 -- 172.21.15.32:0/519933124 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f571c0afaf0 msgr2=0x7f571c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:56.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:56.706+0000 7f5740a00640 1 --2- 172.21.15.32:0/519933124 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f571c0afaf0 0x7f571c0b2000 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f5724009230 tx=0x7f5724031400 comp rx=0 tx=0).stop 2024-04-30T08:31:56.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:56.706+0000 7f5740a00640 1 -- 172.21.15.32:0/519933124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f573c105d90 msgr2=0x7f573c19c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:56.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:56.706+0000 7f5740a00640 1 --2- 172.21.15.32:0/519933124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f573c105d90 0x7f573c19c460 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f572c00e9f0 tx=0x7f572c00eeb0 comp rx=0 tx=0).stop 2024-04-30T08:31:56.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:56.709+0000 7f5740a00640 1 -- 172.21.15.32:0/519933124 shutdown_connections 2024-04-30T08:31:56.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:56.709+0000 7f5740a00640 1 --2- 172.21.15.32:0/519933124 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f571c0afaf0 0x7f571c0b2000 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:56.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:56.709+0000 7f5740a00640 1 --2- 172.21.15.32:0/519933124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f573c10bcd0 0x7f573c1a3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:56.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:56.709+0000 7f5740a00640 1 --2- 172.21.15.32:0/519933124 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f573c106fd0 0x7f573c19c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:56.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:56.709+0000 7f5740a00640 1 --2- 172.21.15.32:0/519933124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f573c105d90 0x7f573c19c460 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:56.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:56.709+0000 7f5740a00640 1 -- 172.21.15.32:0/519933124 >> 172.21.15.32:0/519933124 conn(0x7f573c100fa0 msgr2=0x7f573c107cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:56.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:56.709+0000 7f5740a00640 1 -- 172.21.15.32:0/519933124 shutdown_connections 2024-04-30T08:31:56.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:56.709+0000 7f5740a00640 1 -- 172.21.15.32:0/519933124 wait complete. 2024-04-30T08:31:56.748 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-04-30T08:31:57.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:56 smithi071 ceph-mon[33385]: pgmap v321: 97 pgs: 78 active+clean, 4 creating+peering, 15 unknown; 578 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 8 op/s 2024-04-30T08:31:57.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:56 smithi071 ceph-mon[33385]: osdmap e78: 12 total, 12 up, 12 in 2024-04-30T08:31:57.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.014+0000 7ffa1d000640 1 -- 172.21.15.32:0/802860324 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa18106d90 msgr2=0x7ffa181091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:57.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.014+0000 7ffa1d000640 1 --2- 172.21.15.32:0/802860324 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa18106d90 0x7ffa181091b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7ffa04009900 tx=0x7ffa0402a3f0 comp rx=0 tx=0).stop 2024-04-30T08:31:57.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.014+0000 7ffa1d000640 1 -- 172.21.15.32:0/802860324 shutdown_connections 2024-04-30T08:31:57.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.014+0000 7ffa1d000640 1 --2- 172.21.15.32:0/802860324 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa181096f0 0x7ffa1810bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:57.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.014+0000 7ffa1d000640 1 --2- 172.21.15.32:0/802860324 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa18106d90 0x7ffa181091b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:57.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.014+0000 7ffa1d000640 1 --2- 172.21.15.32:0/802860324 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa181015f0 0x7ffa18101a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:57.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.014+0000 7ffa1d000640 1 -- 172.21.15.32:0/802860324 >> 172.21.15.32:0/802860324 conn(0x7ffa180fcdc0 msgr2=0x7ffa180ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:57.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.014+0000 7ffa1d000640 1 -- 172.21.15.32:0/802860324 shutdown_connections 2024-04-30T08:31:57.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.014+0000 7ffa1d000640 1 -- 172.21.15.32:0/802860324 wait complete. 2024-04-30T08:31:57.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.015+0000 7ffa1d000640 1 Processor -- start 2024-04-30T08:31:57.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.016+0000 7ffa1d000640 1 -- start start 2024-04-30T08:31:57.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.016+0000 7ffa1d000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa181015f0 0x7ffa1819c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:57.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.016+0000 7ffa1d000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa18106d90 0x7ffa1819c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:57.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.016+0000 7ffa1d000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa181096f0 0x7ffa181a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:57.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.016+0000 7ffa1d000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1810f520 con 0x7ffa181096f0 2024-04-30T08:31:57.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.016+0000 7ffa1d000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1810f3a0 con 0x7ffa18106d90 2024-04-30T08:31:57.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.016+0000 7ffa1d000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ffa1810f6a0 con 0x7ffa181015f0 2024-04-30T08:31:57.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.016+0000 7ffa16a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa18106d90 0x7ffa1819c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:57.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.016+0000 7ffa17400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa181015f0 0x7ffa1819c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:57.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.016+0000 7ffa16a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa18106d90 0x7ffa1819c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:45168/0 (socket says 172.21.15.32:45168) 2024-04-30T08:31:57.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.016+0000 7ffa16a00640 1 -- 172.21.15.32:0/606488303 learned_addr learned my addr 172.21.15.32:0/606488303 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:31:57.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.016+0000 7ffa17e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa181096f0 0x7ffa181a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:57.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.017+0000 7ffa17400640 1 -- 172.21.15.32:0/606488303 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa18106d90 msgr2=0x7ffa1819c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:57.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.017+0000 7ffa17400640 1 --2- 172.21.15.32:0/606488303 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa18106d90 0x7ffa1819c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:57.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.017+0000 7ffa17400640 1 -- 172.21.15.32:0/606488303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa181096f0 msgr2=0x7ffa181a37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:57.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.017+0000 7ffa17400640 1 --2- 172.21.15.32:0/606488303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa181096f0 0x7ffa181a37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:57.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.017+0000 7ffa17400640 1 -- 172.21.15.32:0/606488303 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa181a3e50 con 0x7ffa181015f0 2024-04-30T08:31:57.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.017+0000 7ffa17400640 1 --2- 172.21.15.32:0/606488303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa181015f0 0x7ffa1819c310 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ffa180695f0 tx=0x7ffa0800ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:57.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.017+0000 7ffa0fe00640 1 -- 172.21.15.32:0/606488303 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa0800cc40 con 0x7ffa181015f0 2024-04-30T08:31:57.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.017+0000 7ffa1d000640 1 -- 172.21.15.32:0/606488303 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa181a4140 con 0x7ffa181015f0 2024-04-30T08:31:57.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.017+0000 7ffa0fe00640 1 -- 172.21.15.32:0/606488303 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa0800cde0 con 0x7ffa181015f0 2024-04-30T08:31:57.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.017+0000 7ffa0fe00640 1 -- 172.21.15.32:0/606488303 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa08004a00 con 0x7ffa181015f0 2024-04-30T08:31:57.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.018+0000 7ffa1d000640 1 -- 172.21.15.32:0/606488303 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa181a4640 con 0x7ffa181015f0 2024-04-30T08:31:57.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.019+0000 7ffa1d000640 1 -- 172.21.15.32:0/606488303 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa1810b210 con 0x7ffa181015f0 2024-04-30T08:31:57.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.021+0000 7ffa0fe00640 1 -- 172.21.15.32:0/606488303 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7ffa08004ba0 con 0x7ffa181015f0 2024-04-30T08:31:57.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.023+0000 7ffa0fe00640 1 --2- 172.21.15.32:0/606488303 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff9fc0afae0 0x7ff9fc0b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:57.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.023+0000 7ffa0fe00640 1 -- 172.21.15.32:0/606488303 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(79..79 src has 1..79) ==== 8770+0+0 (secure 0 0 0) 0x7ffa080def00 con 0x7ffa181015f0 2024-04-30T08:31:57.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.023+0000 7ffa16a00640 1 --2- 172.21.15.32:0/606488303 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff9fc0afae0 0x7ff9fc0b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:57.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.024+0000 7ffa16a00640 1 --2- 172.21.15.32:0/606488303 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff9fc0afae0 0x7ff9fc0b1ff0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ffa04005f70 tx=0x7ffa040314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:57.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.025+0000 7ffa0fe00640 1 -- 172.21.15.32:0/606488303 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffa080a78f0 con 0x7ffa181015f0 2024-04-30T08:31:57.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:56 smithi189 ceph-mon[33038]: pgmap v321: 97 pgs: 78 active+clean, 4 creating+peering, 15 unknown; 578 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 8 op/s 2024-04-30T08:31:57.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:56 smithi189 ceph-mon[33038]: osdmap e78: 12 total, 12 up, 12 in 2024-04-30T08:31:57.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:56 smithi032 ceph-mon[28331]: pgmap v321: 97 pgs: 78 active+clean, 4 creating+peering, 15 unknown; 578 KiB data, 354 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s wr, 8 op/s 2024-04-30T08:31:57.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:56 smithi032 ceph-mon[28331]: osdmap e78: 12 total, 12 up, 12 in 2024-04-30T08:31:57.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.231+0000 7ffa1d000640 1 -- 172.21.15.32:0/606488303 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7ffa1802dbd0 con 0x7ffa181015f0 2024-04-30T08:31:57.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.722+0000 7ffa0fe00640 1 -- 172.21.15.32:0/606488303 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v80) ==== 169+0+0 (secure 0 0 0) 0x7ffa080a71f0 con 0x7ffa181015f0 2024-04-30T08:31:57.723 INFO:teuthology.orchestra.run.smithi032.stderr:set pool 4 allow_ec_overwrites to true 2024-04-30T08:31:57.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.729+0000 7ffa1d000640 1 -- 172.21.15.32:0/606488303 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff9fc0afae0 msgr2=0x7ff9fc0b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:57.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.729+0000 7ffa1d000640 1 --2- 172.21.15.32:0/606488303 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff9fc0afae0 0x7ff9fc0b1ff0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ffa04005f70 tx=0x7ffa040314a0 comp rx=0 tx=0).stop 2024-04-30T08:31:57.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.729+0000 7ffa1d000640 1 -- 172.21.15.32:0/606488303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa181015f0 msgr2=0x7ffa1819c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:57.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.729+0000 7ffa1d000640 1 --2- 172.21.15.32:0/606488303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa181015f0 0x7ffa1819c310 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ffa180695f0 tx=0x7ffa0800ee50 comp rx=0 tx=0).stop 2024-04-30T08:31:57.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.731+0000 7ffa1d000640 1 -- 172.21.15.32:0/606488303 shutdown_connections 2024-04-30T08:31:57.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.731+0000 7ffa1d000640 1 --2- 172.21.15.32:0/606488303 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff9fc0afae0 0x7ff9fc0b1ff0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:57.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.731+0000 7ffa1d000640 1 --2- 172.21.15.32:0/606488303 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa181096f0 0x7ffa181a37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:57.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.731+0000 7ffa1d000640 1 --2- 172.21.15.32:0/606488303 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa18106d90 0x7ffa1819c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:57.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.731+0000 7ffa1d000640 1 --2- 172.21.15.32:0/606488303 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa181015f0 0x7ffa1819c310 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:57.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.731+0000 7ffa1d000640 1 -- 172.21.15.32:0/606488303 >> 172.21.15.32:0/606488303 conn(0x7ffa180fcdc0 msgr2=0x7ffa18107d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:57.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.732+0000 7ffa1d000640 1 -- 172.21.15.32:0/606488303 shutdown_connections 2024-04-30T08:31:57.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:57.732+0000 7ffa1d000640 1 -- 172.21.15.32:0/606488303 wait complete. 2024-04-30T08:31:57.776 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-04-30T08:31:58.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:57 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/519933124' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-30T08:31:58.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:57 smithi071 ceph-mon[33385]: osdmap e79: 12 total, 12 up, 12 in 2024-04-30T08:31:58.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:57 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/606488303' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-30T08:31:58.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:57 smithi071 ceph-mon[33385]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-30T08:31:58.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.035+0000 7f5d5be00640 1 -- 172.21.15.32:0/3497425709 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d5c075010 msgr2=0x7f5d5c075420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:58.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.035+0000 7f5d5be00640 1 --2- 172.21.15.32:0/3497425709 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d5c075010 0x7f5d5c075420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c009900 tx=0x7f5d4c02a410 comp rx=0 tx=0).stop 2024-04-30T08:31:58.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.035+0000 7f5d5be00640 1 -- 172.21.15.32:0/3497425709 shutdown_connections 2024-04-30T08:31:58.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.035+0000 7f5d5be00640 1 --2- 172.21.15.32:0/3497425709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d5c1100f0 0x7f5d5c112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:58.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.035+0000 7f5d5be00640 1 --2- 172.21.15.32:0/3497425709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d5c075960 0x7f5d5c10fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:58.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.035+0000 7f5d5be00640 1 --2- 172.21.15.32:0/3497425709 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d5c075010 0x7f5d5c075420 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:58.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.035+0000 7f5d5be00640 1 -- 172.21.15.32:0/3497425709 >> 172.21.15.32:0/3497425709 conn(0x7f5d5c0feba0 msgr2=0x7f5d5c101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:58.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.035+0000 7f5d5be00640 1 -- 172.21.15.32:0/3497425709 shutdown_connections 2024-04-30T08:31:58.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.035+0000 7f5d5be00640 1 -- 172.21.15.32:0/3497425709 wait complete. 2024-04-30T08:31:58.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.036+0000 7f5d5be00640 1 Processor -- start 2024-04-30T08:31:58.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.036+0000 7f5d5be00640 1 -- start start 2024-04-30T08:31:58.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.037+0000 7f5d5be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d5c075010 0x7f5d5c1a0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:58.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.037+0000 7f5d5be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d5c075960 0x7f5d5c1a0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:58.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.037+0000 7f5d5be00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d5c1100f0 0x7f5d5c1a7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:58.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.037+0000 7f5d5be00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5d5c115ad0 con 0x7f5d5c075010 2024-04-30T08:31:58.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.037+0000 7f5d5be00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5d5c115950 con 0x7f5d5c1100f0 2024-04-30T08:31:58.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.037+0000 7f5d5be00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5d5c115c50 con 0x7f5d5c075960 2024-04-30T08:31:58.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.037+0000 7f5d5aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d5c075010 0x7f5d5c1a0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:58.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.037+0000 7f5d53e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d5c075960 0x7f5d5c1a0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:58.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.037+0000 7f5d5aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d5c075010 0x7f5d5c1a0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44568/0 (socket says 172.21.15.32:44568) 2024-04-30T08:31:58.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.037+0000 7f5d5aa00640 1 -- 172.21.15.32:0/4236325512 learned_addr learned my addr 172.21.15.32:0/4236325512 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:31:58.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.037+0000 7f5d5b400640 1 --2- 172.21.15.32:0/4236325512 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d5c1100f0 0x7f5d5c1a7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:58.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.038+0000 7f5d53e00640 1 -- 172.21.15.32:0/4236325512 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d5c1100f0 msgr2=0x7f5d5c1a7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:58.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.038+0000 7f5d53e00640 1 --2- 172.21.15.32:0/4236325512 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d5c1100f0 0x7f5d5c1a7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:58.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.038+0000 7f5d53e00640 1 -- 172.21.15.32:0/4236325512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d5c075010 msgr2=0x7f5d5c1a0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:58.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.038+0000 7f5d53e00640 1 --2- 172.21.15.32:0/4236325512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d5c075010 0x7f5d5c1a0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:58.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.038+0000 7f5d53e00640 1 -- 172.21.15.32:0/4236325512 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d5c1a8260 con 0x7f5d5c075960 2024-04-30T08:31:58.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.038+0000 7f5d5aa00640 1 --2- 172.21.15.32:0/4236325512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d5c075010 0x7f5d5c1a0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:31:58.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.038+0000 7f5d53e00640 1 --2- 172.21.15.32:0/4236325512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d5c075960 0x7f5d5c1a0c60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5d4400e9f0 tx=0x7f5d4400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:58.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.038+0000 7f5d60a00640 1 -- 172.21.15.32:0/4236325512 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d4400ccc0 con 0x7f5d5c075960 2024-04-30T08:31:58.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.039+0000 7f5d5be00640 1 -- 172.21.15.32:0/4236325512 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d5c1a8550 con 0x7f5d5c075960 2024-04-30T08:31:58.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.039+0000 7f5d60a00640 1 -- 172.21.15.32:0/4236325512 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d440108e0 con 0x7f5d5c075960 2024-04-30T08:31:58.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.039+0000 7f5d60a00640 1 -- 172.21.15.32:0/4236325512 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d44018880 con 0x7f5d5c075960 2024-04-30T08:31:58.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.039+0000 7f5d5be00640 1 -- 172.21.15.32:0/4236325512 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d5c1a8af0 con 0x7f5d5c075960 2024-04-30T08:31:58.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.040+0000 7f5d5be00640 1 -- 172.21.15.32:0/4236325512 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d5c069c60 con 0x7f5d5c075960 2024-04-30T08:31:58.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.042+0000 7f5d60a00640 1 -- 172.21.15.32:0/4236325512 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f5d44010430 con 0x7f5d5c075960 2024-04-30T08:31:58.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.044+0000 7f5d60a00640 1 --2- 172.21.15.32:0/4236325512 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f5d400afae0 0x7f5d400b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:58.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.044+0000 7f5d60a00640 1 -- 172.21.15.32:0/4236325512 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 8770+0+0 (secure 0 0 0) 0x7f5d440df630 con 0x7f5d5c075960 2024-04-30T08:31:58.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.044+0000 7f5d5aa00640 1 --2- 172.21.15.32:0/4236325512 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f5d400afae0 0x7f5d400b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:58.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.045+0000 7f5d5aa00640 1 --2- 172.21.15.32:0/4236325512 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f5d400afae0 0x7f5d400b1ff0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c004730 tx=0x7f5d4c0046c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:58.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.046+0000 7f5d60a00640 1 -- 172.21.15.32:0/4236325512 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5d440a7940 con 0x7f5d5c075960 2024-04-30T08:31:58.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:57 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/519933124' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-30T08:31:58.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:57 smithi189 ceph-mon[33038]: osdmap e79: 12 total, 12 up, 12 in 2024-04-30T08:31:58.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:57 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/606488303' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-30T08:31:58.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:57 smithi189 ceph-mon[33038]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-30T08:31:58.154 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:57 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/519933124' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-30T08:31:58.154 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:57 smithi032 ceph-mon[28331]: osdmap e79: 12 total, 12 up, 12 in 2024-04-30T08:31:58.154 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:57 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/606488303' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-30T08:31:58.154 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:57 smithi032 ceph-mon[28331]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-04-30T08:31:58.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.297+0000 7f5d5be00640 1 -- 172.21.15.32:0/4236325512 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f5d5c105d80 con 0x7f5d5c075960 2024-04-30T08:31:58.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.726+0000 7f5d60a00640 1 -- 172.21.15.32:0/4236325512 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f5d440abc40 con 0x7f5d5c075960 2024-04-30T08:31:58.727 INFO:teuthology.orchestra.run.smithi032.stderr:added data pool 4 to fsmap 2024-04-30T08:31:58.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.728+0000 7f5d5be00640 1 -- 172.21.15.32:0/4236325512 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f5d400afae0 msgr2=0x7f5d400b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:58.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.728+0000 7f5d5be00640 1 --2- 172.21.15.32:0/4236325512 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f5d400afae0 0x7f5d400b1ff0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5d4c004730 tx=0x7f5d4c0046c0 comp rx=0 tx=0).stop 2024-04-30T08:31:58.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.728+0000 7f5d5be00640 1 -- 172.21.15.32:0/4236325512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d5c075960 msgr2=0x7f5d5c1a0c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:58.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.728+0000 7f5d5be00640 1 --2- 172.21.15.32:0/4236325512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d5c075960 0x7f5d5c1a0c60 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5d4400e9f0 tx=0x7f5d4400eeb0 comp rx=0 tx=0).stop 2024-04-30T08:31:58.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.729+0000 7f5d5be00640 1 -- 172.21.15.32:0/4236325512 shutdown_connections 2024-04-30T08:31:58.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.729+0000 7f5d5be00640 1 --2- 172.21.15.32:0/4236325512 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f5d400afae0 0x7f5d400b1ff0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:58.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.729+0000 7f5d5be00640 1 --2- 172.21.15.32:0/4236325512 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d5c1100f0 0x7f5d5c1a7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:58.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.729+0000 7f5d5be00640 1 --2- 172.21.15.32:0/4236325512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d5c075960 0x7f5d5c1a0c60 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:58.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.729+0000 7f5d5be00640 1 --2- 172.21.15.32:0/4236325512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d5c075010 0x7f5d5c1a0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:58.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.729+0000 7f5d5be00640 1 -- 172.21.15.32:0/4236325512 >> 172.21.15.32:0/4236325512 conn(0x7f5d5c0feba0 msgr2=0x7f5d5c102f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:58.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.729+0000 7f5d5be00640 1 -- 172.21.15.32:0/4236325512 shutdown_connections 2024-04-30T08:31:58.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:58.729+0000 7f5d5be00640 1 -- 172.21.15.32:0/4236325512 wait complete. 2024-04-30T08:31:58.754 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:58 smithi032 ceph-mon[28331]: pgmap v323: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 9 op/s 2024-04-30T08:31:58.754 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:58 smithi032 ceph-mon[28331]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-30T08:31:58.755 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:58 smithi032 ceph-mon[28331]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-30T08:31:58.755 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:58 smithi032 ceph-mon[28331]: osdmap e80: 12 total, 12 up, 12 in 2024-04-30T08:31:58.755 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:58 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/4236325512' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-30T08:31:58.755 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:58 smithi032 ceph-mon[28331]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-30T08:31:58.757 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-30T08:31:59.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:58 smithi071 ceph-mon[33385]: pgmap v323: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 9 op/s 2024-04-30T08:31:59.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:58 smithi071 ceph-mon[33385]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-30T08:31:59.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:58 smithi071 ceph-mon[33385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-30T08:31:59.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:58 smithi071 ceph-mon[33385]: osdmap e80: 12 total, 12 up, 12 in 2024-04-30T08:31:59.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:58 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/4236325512' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-30T08:31:59.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:58 smithi071 ceph-mon[33385]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-30T08:31:59.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.009+0000 7f95cfc00640 1 -- 172.21.15.32:0/1693428758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95c81042b0 msgr2=0x7f95c81066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:59.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.009+0000 7f95cfc00640 1 --2- 172.21.15.32:0/1693428758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95c81042b0 0x7f95c81066d0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f95bc009900 tx=0x7f95bc02a400 comp rx=0 tx=0).stop 2024-04-30T08:31:59.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.009+0000 7f95cfc00640 1 -- 172.21.15.32:0/1693428758 shutdown_connections 2024-04-30T08:31:59.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.009+0000 7f95cfc00640 1 --2- 172.21.15.32:0/1693428758 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95c8109810 0x7f95c810bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:59.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.009+0000 7f95cfc00640 1 --2- 172.21.15.32:0/1693428758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95c8106ca0 0x7f95c8109180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:59.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.009+0000 7f95cfc00640 1 --2- 172.21.15.32:0/1693428758 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95c81042b0 0x7f95c81066d0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:59.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.009+0000 7f95cfc00640 1 -- 172.21.15.32:0/1693428758 >> 172.21.15.32:0/1693428758 conn(0x7f95c80fdc50 msgr2=0x7f95c8100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:59.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.009+0000 7f95cfc00640 1 -- 172.21.15.32:0/1693428758 shutdown_connections 2024-04-30T08:31:59.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.009+0000 7f95cfc00640 1 -- 172.21.15.32:0/1693428758 wait complete. 2024-04-30T08:31:59.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.010+0000 7f95cfc00640 1 Processor -- start 2024-04-30T08:31:59.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.010+0000 7f95cfc00640 1 -- start start 2024-04-30T08:31:59.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.011+0000 7f95cfc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95c81042b0 0x7f95c8103b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:59.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.011+0000 7f95cfc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95c8106ca0 0x7f95c81021c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:59.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.011+0000 7f95cfc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95c8109810 0x7f95c8102700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:59.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.011+0000 7f95cfc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f95c810f620 con 0x7f95c8109810 2024-04-30T08:31:59.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.011+0000 7f95cfc00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f95c810f4a0 con 0x7f95c8106ca0 2024-04-30T08:31:59.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.011+0000 7f95cfc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f95c810f7a0 con 0x7f95c81042b0 2024-04-30T08:31:59.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.011+0000 7f95cdc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95c8109810 0x7f95c8102700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:59.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.011+0000 7f95cdc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95c8109810 0x7f95c8102700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44586/0 (socket says 172.21.15.32:44586) 2024-04-30T08:31:59.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.011+0000 7f95cdc00640 1 -- 172.21.15.32:0/3751378488 learned_addr learned my addr 172.21.15.32:0/3751378488 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:31:59.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.011+0000 7f95cd200640 1 --2- 172.21.15.32:0/3751378488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95c81042b0 0x7f95c8103b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:59.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.011+0000 7f95c7e00640 1 --2- 172.21.15.32:0/3751378488 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95c8106ca0 0x7f95c81021c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:59.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.012+0000 7f95cdc00640 1 -- 172.21.15.32:0/3751378488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95c81042b0 msgr2=0x7f95c8103b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:59.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.012+0000 7f95cdc00640 1 --2- 172.21.15.32:0/3751378488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95c81042b0 0x7f95c8103b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:59.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.012+0000 7f95cdc00640 1 -- 172.21.15.32:0/3751378488 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95c8106ca0 msgr2=0x7f95c81021c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:59.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.012+0000 7f95cdc00640 1 --2- 172.21.15.32:0/3751378488 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95c8106ca0 0x7f95c81021c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:59.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.012+0000 7f95cdc00640 1 -- 172.21.15.32:0/3751378488 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95c8102e30 con 0x7f95c8109810 2024-04-30T08:31:59.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.012+0000 7f95cdc00640 1 --2- 172.21.15.32:0/3751378488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95c8109810 0x7f95c8102700 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f95c0009fd0 tx=0x7f95c000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:59.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.012+0000 7f95c5600640 1 -- 172.21.15.32:0/3751378488 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c0017070 con 0x7f95c8109810 2024-04-30T08:31:59.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.012+0000 7f95c5600640 1 -- 172.21.15.32:0/3751378488 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f95c0003e20 con 0x7f95c8109810 2024-04-30T08:31:59.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.012+0000 7f95cfc00640 1 -- 172.21.15.32:0/3751378488 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95c81a4340 con 0x7f95c8109810 2024-04-30T08:31:59.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.012+0000 7f95c5600640 1 -- 172.21.15.32:0/3751378488 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95c0010460 con 0x7f95c8109810 2024-04-30T08:31:59.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.012+0000 7f95cfc00640 1 -- 172.21.15.32:0/3751378488 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95c81a4800 con 0x7f95c8109810 2024-04-30T08:31:59.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.014+0000 7f95cfc00640 1 -- 172.21.15.32:0/3751378488 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95c80ff770 con 0x7f95c8109810 2024-04-30T08:31:59.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.016+0000 7f95c5600640 1 -- 172.21.15.32:0/3751378488 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f95c0003970 con 0x7f95c8109810 2024-04-30T08:31:59.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.019+0000 7f95c5600640 1 --2- 172.21.15.32:0/3751378488 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f95b00afb70 0x7f95b00b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:59.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.019+0000 7f95c5600640 1 -- 172.21.15.32:0/3751378488 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f95c00deb30 con 0x7f95c8109810 2024-04-30T08:31:59.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.019+0000 7f95cd200640 1 --2- 172.21.15.32:0/3751378488 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f95b00afb70 0x7f95b00b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:59.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.020+0000 7f95cd200640 1 --2- 172.21.15.32:0/3751378488 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f95b00afb70 0x7f95b00b2080 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f95bc000c00 tx=0x7f95bc031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:59.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.023+0000 7f95c5600640 1 -- 172.21.15.32:0/3751378488 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f95c00a6e20 con 0x7f95c8109810 2024-04-30T08:31:59.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:58 smithi189 ceph-mon[33038]: pgmap v323: 129 pgs: 32 unknown, 97 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s wr, 9 op/s 2024-04-30T08:31:59.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:58 smithi189 ceph-mon[33038]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-30T08:31:59.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:58 smithi189 ceph-mon[33038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-04-30T08:31:59.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:58 smithi189 ceph-mon[33038]: osdmap e80: 12 total, 12 up, 12 in 2024-04-30T08:31:59.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:58 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/4236325512' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-30T08:31:59.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:58 smithi189 ceph-mon[33038]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-04-30T08:31:59.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.268+0000 7f95cfc00640 1 -- 172.21.15.32:0/3751378488 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f95c802dbd0 con 0x7f95c8109810 2024-04-30T08:31:59.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.269+0000 7f95c5600640 1 -- 172.21.15.32:0/3751378488 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11791 (secure 0 0 0) 0x7f95c00ab120 con 0x7f95c8109810 2024-04-30T08:31:59.270 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:31:59.270 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2235842676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2235842676},{"type":"v1","addr":"172.21.15.32:6835","nonce":2235842676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/3577009355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":3577009355},{"type":"v1","addr":"172.21.15.32:6841","nonce":3577009355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/463203038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":463203038},{"type":"v1","addr":"172.21.15.189:6833","nonce":463203038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3635143591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3635143591},{"type":"v1","addr":"172.21.15.71:6833","nonce":3635143591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24484,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2569449994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2569449994},{"type":"v1","addr":"172.21.15.32:6837","nonce":2569449994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/278686565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":278686565},{"type":"v1","addr":"172.21.15.189:6835","nonce":278686565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6835/1558303056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1558303056},{"type":"v1","addr":"172.21.15.71:6835","nonce":1558303056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/686520873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":686520873},{"type":"v1","addr":"172.21.15.189:6837","nonce":686520873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/2324039932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2324039932},{"type":"v1","addr":"172.21.15.71:6837","nonce":2324039932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6839/352970107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":352970107},{"type":"v1","addr":"172.21.15.189:6839","nonce":352970107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24560,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/4160386524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4160386524},{"type":"v1","addr":"172.21.15.32:6839","nonce":4160386524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-30T08:31:52.236668+0000","modified":"2024-04-30T08:31:58.299202+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.71:6839/3517409993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3517409993},{"type":"v1","addr":"172.21.15.71:6839","nonce":3517409993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24590,"qdb_cluster":[24590]},"id":1}]} 2024-04-30T08:31:59.271 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 17 2024-04-30T08:31:59.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.272+0000 7f95cfc00640 1 -- 172.21.15.32:0/3751378488 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f95b00afb70 msgr2=0x7f95b00b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:59.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.272+0000 7f95cfc00640 1 --2- 172.21.15.32:0/3751378488 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f95b00afb70 0x7f95b00b2080 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f95bc000c00 tx=0x7f95bc031400 comp rx=0 tx=0).stop 2024-04-30T08:31:59.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.272+0000 7f95cfc00640 1 -- 172.21.15.32:0/3751378488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95c8109810 msgr2=0x7f95c8102700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:59.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.272+0000 7f95cfc00640 1 --2- 172.21.15.32:0/3751378488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95c8109810 0x7f95c8102700 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f95c0009fd0 tx=0x7f95c000b260 comp rx=0 tx=0).stop 2024-04-30T08:31:59.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.272+0000 7f95cfc00640 1 -- 172.21.15.32:0/3751378488 shutdown_connections 2024-04-30T08:31:59.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.272+0000 7f95cfc00640 1 --2- 172.21.15.32:0/3751378488 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f95b00afb70 0x7f95b00b2080 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:59.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.272+0000 7f95cfc00640 1 --2- 172.21.15.32:0/3751378488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95c8109810 0x7f95c8102700 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:59.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.272+0000 7f95cfc00640 1 --2- 172.21.15.32:0/3751378488 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f95c8106ca0 0x7f95c81021c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:59.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.273+0000 7f95cfc00640 1 --2- 172.21.15.32:0/3751378488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95c81042b0 0x7f95c8103b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:59.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.273+0000 7f95cfc00640 1 -- 172.21.15.32:0/3751378488 >> 172.21.15.32:0/3751378488 conn(0x7f95c80fdc50 msgr2=0x7f95c8104e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:59.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.273+0000 7f95cfc00640 1 -- 172.21.15.32:0/3751378488 shutdown_connections 2024-04-30T08:31:59.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.273+0000 7f95cfc00640 1 -- 172.21.15.32:0/3751378488 wait complete. 2024-04-30T08:31:59.297 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-30T08:31:59.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.564+0000 7fb292e00640 1 -- 172.21.15.32:0/3224951483 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb28c104dd0 msgr2=0x7fb28c1051e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:59.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.564+0000 7fb292e00640 1 --2- 172.21.15.32:0/3224951483 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb28c104dd0 0x7fb28c1051e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb274009be0 tx=0x7fb27402a6c0 comp rx=0 tx=0).stop 2024-04-30T08:31:59.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.564+0000 7fb292e00640 1 -- 172.21.15.32:0/3224951483 shutdown_connections 2024-04-30T08:31:59.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.564+0000 7fb292e00640 1 --2- 172.21.15.32:0/3224951483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb28c109ad0 0x7fb28c10bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:59.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.564+0000 7fb292e00640 1 --2- 172.21.15.32:0/3224951483 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb28c104dd0 0x7fb28c1051e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:59.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.564+0000 7fb292e00640 1 --2- 172.21.15.32:0/3224951483 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb28c103b90 0x7fb28c104890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:59.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.564+0000 7fb292e00640 1 -- 172.21.15.32:0/3224951483 >> 172.21.15.32:0/3224951483 conn(0x7fb28c0fedc0 msgr2=0x7fb28c101230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:59.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.565+0000 7fb292e00640 1 -- 172.21.15.32:0/3224951483 shutdown_connections 2024-04-30T08:31:59.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.565+0000 7fb292e00640 1 -- 172.21.15.32:0/3224951483 wait complete. 2024-04-30T08:31:59.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.566+0000 7fb292e00640 1 Processor -- start 2024-04-30T08:31:59.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.566+0000 7fb292e00640 1 -- start start 2024-04-30T08:31:59.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.567+0000 7fb292e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb28c103b90 0x7fb28c19c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:59.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.567+0000 7fb292e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb28c104dd0 0x7fb28c19ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:59.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.567+0000 7fb292e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb28c109ad0 0x7fb28c1a3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:59.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.567+0000 7fb292e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb28c10f690 con 0x7fb28c103b90 2024-04-30T08:31:59.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.567+0000 7fb292e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb28c10f510 con 0x7fb28c109ad0 2024-04-30T08:31:59.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.567+0000 7fb292e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb28c10f810 con 0x7fb28c104dd0 2024-04-30T08:31:59.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.567+0000 7fb28be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb28c103b90 0x7fb28c19c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:59.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.567+0000 7fb28be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb28c103b90 0x7fb28c19c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44608/0 (socket says 172.21.15.32:44608) 2024-04-30T08:31:59.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.567+0000 7fb290e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb28c109ad0 0x7fb28c1a3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:59.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.567+0000 7fb28be00640 1 -- 172.21.15.32:0/2284722201 learned_addr learned my addr 172.21.15.32:0/2284722201 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:31:59.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.567+0000 7fb28b400640 1 --2- 172.21.15.32:0/2284722201 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb28c104dd0 0x7fb28c19ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:59.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.567+0000 7fb290e00640 1 -- 172.21.15.32:0/2284722201 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb28c104dd0 msgr2=0x7fb28c19ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:59.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.568+0000 7fb290e00640 1 --2- 172.21.15.32:0/2284722201 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb28c104dd0 0x7fb28c19ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:59.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.568+0000 7fb290e00640 1 -- 172.21.15.32:0/2284722201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb28c103b90 msgr2=0x7fb28c19c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:59.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.568+0000 7fb290e00640 1 --2- 172.21.15.32:0/2284722201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb28c103b90 0x7fb28c19c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:59.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.568+0000 7fb290e00640 1 -- 172.21.15.32:0/2284722201 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb28c1a4000 con 0x7fb28c109ad0 2024-04-30T08:31:59.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.568+0000 7fb28be00640 1 --2- 172.21.15.32:0/2284722201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb28c103b90 0x7fb28c19c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:31:59.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.568+0000 7fb290e00640 1 --2- 172.21.15.32:0/2284722201 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb28c109ad0 0x7fb28c1a3990 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb28400e970 tx=0x7fb28400ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:59.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.568+0000 7fb288c00640 1 -- 172.21.15.32:0/2284722201 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28400ccc0 con 0x7fb28c109ad0 2024-04-30T08:31:59.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.568+0000 7fb292e00640 1 -- 172.21.15.32:0/2284722201 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb28c1a42f0 con 0x7fb28c109ad0 2024-04-30T08:31:59.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.568+0000 7fb288c00640 1 -- 172.21.15.32:0/2284722201 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb284009e40 con 0x7fb28c109ad0 2024-04-30T08:31:59.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.568+0000 7fb288c00640 1 -- 172.21.15.32:0/2284722201 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb284004a00 con 0x7fb28c109ad0 2024-04-30T08:31:59.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.568+0000 7fb292e00640 1 -- 172.21.15.32:0/2284722201 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb28c1a4840 con 0x7fb28c109ad0 2024-04-30T08:31:59.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.569+0000 7fb292e00640 1 -- 172.21.15.32:0/2284722201 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb28c1a4b00 con 0x7fb28c109ad0 2024-04-30T08:31:59.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.572+0000 7fb288c00640 1 -- 172.21.15.32:0/2284722201 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fb284004ba0 con 0x7fb28c109ad0 2024-04-30T08:31:59.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.573+0000 7fb288c00640 1 --2- 172.21.15.32:0/2284722201 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb26c0afaf0 0x7fb26c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:31:59.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.574+0000 7fb288c00640 1 -- 172.21.15.32:0/2284722201 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7fb2840def00 con 0x7fb28c109ad0 2024-04-30T08:31:59.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.574+0000 7fb28be00640 1 --2- 172.21.15.32:0/2284722201 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb26c0afaf0 0x7fb26c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:31:59.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.574+0000 7fb28be00640 1 --2- 172.21.15.32:0/2284722201 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb26c0afaf0 0x7fb26c0b2000 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb27c011490 tx=0x7fb27c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:31:59.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.576+0000 7fb288c00640 1 -- 172.21.15.32:0/2284722201 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb2840a78d0 con 0x7fb28c109ad0 2024-04-30T08:31:59.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.783+0000 7fb292e00640 1 -- 172.21.15.32:0/2284722201 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb28c1a4cd0 con 0x7fb28c109ad0 2024-04-30T08:31:59.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.786+0000 7fb288c00640 1 -- 172.21.15.32:0/2284722201 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+24053 (secure 0 0 0) 0x7fb2840a71d0 con 0x7fb28c109ad0 2024-04-30T08:31:59.787 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:31:59.787 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":81,"fsid":"98c0a670-06ca-11ef-bc93-c7b262605968","created":"2024-04-30T08:22:11.122875+0000","modified":"2024-04-30T08:31:58.298969+0000","last_up_change":"2024-04-30T08:30:50.350439+0000","last_in_change":"2024-04-30T08:30:30.210620+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-30T08:26:19.728331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-30T08:31:49.800542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-30T08:31:51.211320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-30T08:31:55.716625+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a7e060c8-1cbb-418f-9a03-06693f6b3f52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6803","nonce":2624700402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6805","nonce":2624700402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6809","nonce":2624700402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6807","nonce":2624700402}]},"public_addr":"172.21.15.32:6803/2624700402","cluster_addr":"172.21.15.32:6805/2624700402","heartbeat_back_addr":"172.21.15.32:6809/2624700402","heartbeat_front_addr":"172.21.15.32:6807/2624700402","state":["exists","up"]},{"osd":1,"uuid":"72ef2fb1-e5ca-48ef-aa3a-4796a64b922f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6801","nonce":444649711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6803","nonce":444649711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6807","nonce":444649711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6805","nonce":444649711}]},"public_addr":"172.21.15.71:6801/444649711","cluster_addr":"172.21.15.71:6803/444649711","heartbeat_back_addr":"172.21.15.71:6807/444649711","heartbeat_front_addr":"172.21.15.71:6805/444649711","state":["exists","up"]},{"osd":2,"uuid":"704e5e8a-935e-47c7-ae28-27119876ac09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6801","nonce":3255528671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6803","nonce":3255528671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6807","nonce":3255528671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6805","nonce":3255528671}]},"public_addr":"172.21.15.189:6801/3255528671","cluster_addr":"172.21.15.189:6803/3255528671","heartbeat_back_addr":"172.21.15.189:6807/3255528671","heartbeat_front_addr":"172.21.15.189:6805/3255528671","state":["exists","up"]},{"osd":3,"uuid":"575004f7-6559-4a62-863e-e33078b0eebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6811","nonce":1384702112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6813","nonce":1384702112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6817","nonce":1384702112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6815","nonce":1384702112}]},"public_addr":"172.21.15.32:6811/1384702112","cluster_addr":"172.21.15.32:6813/1384702112","heartbeat_back_addr":"172.21.15.32:6817/1384702112","heartbeat_front_addr":"172.21.15.32:6815/1384702112","state":["exists","up"]},{"osd":4,"uuid":"e1c1e894-074d-4c78-89ef-9e5a114268bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6809","nonce":4250534574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6811","nonce":4250534574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6815","nonce":4250534574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6813","nonce":4250534574}]},"public_addr":"172.21.15.71:6809/4250534574","cluster_addr":"172.21.15.71:6811/4250534574","heartbeat_back_addr":"172.21.15.71:6815/4250534574","heartbeat_front_addr":"172.21.15.71:6813/4250534574","state":["exists","up"]},{"osd":5,"uuid":"12c022d4-fac6-40c4-ae7a-691571bee5d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6809","nonce":4245346198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6811","nonce":4245346198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6815","nonce":4245346198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6813","nonce":4245346198}]},"public_addr":"172.21.15.189:6809/4245346198","cluster_addr":"172.21.15.189:6811/4245346198","heartbeat_back_addr":"172.21.15.189:6815/4245346198","heartbeat_front_addr":"172.21.15.189:6813/4245346198","state":["exists","up"]},{"osd":6,"uuid":"b413f8ee-3d4a-415a-a2ba-d68437b7330d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6819","nonce":1910389190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6821","nonce":1910389190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6825","nonce":1910389190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6823","nonce":1910389190}]},"public_addr":"172.21.15.32:6819/1910389190","cluster_addr":"172.21.15.32:6821/1910389190","heartbeat_back_addr":"172.21.15.32:6825/1910389190","heartbeat_front_addr":"172.21.15.32:6823/1910389190","state":["exists","up"]},{"osd":7,"uuid":"aac64fb3-fc13-4126-b842-7b06ad5ce238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6817","nonce":692607380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6819","nonce":692607380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6823","nonce":692607380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6821","nonce":692607380}]},"public_addr":"172.21.15.71:6817/692607380","cluster_addr":"172.21.15.71:6819/692607380","heartbeat_back_addr":"172.21.15.71:6823/692607380","heartbeat_front_addr":"172.21.15.71:6821/692607380","state":["exists","up"]},{"osd":8,"uuid":"cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6817","nonce":3973697069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6819","nonce":3973697069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6823","nonce":3973697069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6821","nonce":3973697069}]},"public_addr":"172.21.15.189:6817/3973697069","cluster_addr":"172.21.15.189:6819/3973697069","heartbeat_back_addr":"172.21.15.189:6823/3973697069","heartbeat_front_addr":"172.21.15.189:6821/3973697069","state":["exists","up"]},{"osd":9,"uuid":"ef4a1645-ae6e-45c3-82ad-161510e62aa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6827","nonce":46066685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6829","nonce":46066685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6833","nonce":46066685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6831","nonce":46066685}]},"public_addr":"172.21.15.32:6827/46066685","cluster_addr":"172.21.15.32:6829/46066685","heartbeat_back_addr":"172.21.15.32:6833/46066685","heartbeat_front_addr":"172.21.15.32:6831/46066685","state":["exists","up"]},{"osd":10,"uuid":"25b26a38-a644-409f-b941-59efdf04a3ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6825","nonce":1886290293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6827","nonce":1886290293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6831","nonce":1886290293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6829","nonce":1886290293}]},"public_addr":"172.21.15.71:6825/1886290293","cluster_addr":"172.21.15.71:6827/1886290293","heartbeat_back_addr":"172.21.15.71:6831/1886290293","heartbeat_front_addr":"172.21.15.71:6829/1886290293","state":["exists","up"]},{"osd":11,"uuid":"ab96b3ff-3688-455b-af64-60226c5bfbff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6825","nonce":850185001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6827","nonce":850185001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6831","nonce":850185001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6829","nonce":850185001}]},"public_addr":"172.21.15.189:6825/850185001","cluster_addr":"172.21.15.189:6827/850185001","heartbeat_back_addr":"172.21.15.189:6831/850185001","heartbeat_front_addr":"172.21.15.189:6829/850185001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:27.220308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:53.230783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:17.190642+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:52.603799+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:20.568671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:46.158363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:22.706933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:50.665173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:16.602410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:53.814698+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:22.435189+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:48.651949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1266897222":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/2915689533":"2024-05-01T08:23:29.526664+0000","172.21.15.32:6801/4019351306":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/425836193":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/1134075722":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6801/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/21059253":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/3245550894":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/4019351306":"2024-05-01T08:23:29.526664+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-30T08:31:59.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.789+0000 7fb292e00640 1 -- 172.21.15.32:0/2284722201 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb26c0afaf0 msgr2=0x7fb26c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:59.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.789+0000 7fb292e00640 1 --2- 172.21.15.32:0/2284722201 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb26c0afaf0 0x7fb26c0b2000 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fb27c011490 tx=0x7fb27c009260 comp rx=0 tx=0).stop 2024-04-30T08:31:59.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.789+0000 7fb292e00640 1 -- 172.21.15.32:0/2284722201 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb28c109ad0 msgr2=0x7fb28c1a3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:31:59.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.789+0000 7fb292e00640 1 --2- 172.21.15.32:0/2284722201 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb28c109ad0 0x7fb28c1a3990 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb28400e970 tx=0x7fb28400ed70 comp rx=0 tx=0).stop 2024-04-30T08:31:59.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.789+0000 7fb292e00640 1 -- 172.21.15.32:0/2284722201 shutdown_connections 2024-04-30T08:31:59.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.789+0000 7fb292e00640 1 --2- 172.21.15.32:0/2284722201 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb26c0afaf0 0x7fb26c0b2000 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:59.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.789+0000 7fb292e00640 1 --2- 172.21.15.32:0/2284722201 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb28c109ad0 0x7fb28c1a3990 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:59.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.789+0000 7fb292e00640 1 --2- 172.21.15.32:0/2284722201 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb28c104dd0 0x7fb28c19ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:59.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.789+0000 7fb292e00640 1 --2- 172.21.15.32:0/2284722201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb28c103b90 0x7fb28c19c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:31:59.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.789+0000 7fb292e00640 1 -- 172.21.15.32:0/2284722201 >> 172.21.15.32:0/2284722201 conn(0x7fb28c0fedc0 msgr2=0x7fb28c0ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:31:59.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.789+0000 7fb292e00640 1 -- 172.21.15.32:0/2284722201 shutdown_connections 2024-04-30T08:31:59.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:31:59.789+0000 7fb292e00640 1 -- 172.21.15.32:0/2284722201 wait complete. 2024-04-30T08:31:59.814 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-30T08:32:00.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:59 smithi071 ceph-mon[33385]: osdmap e81: 12 total, 12 up, 12 in 2024-04-30T08:32:00.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:59 smithi071 ceph-mon[33385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-30T08:32:00.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:59 smithi071 ceph-mon[33385]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-30T08:32:00.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:31:59 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3751378488' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:32:00.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:59 smithi189 ceph-mon[33038]: osdmap e81: 12 total, 12 up, 12 in 2024-04-30T08:32:00.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:59 smithi189 ceph-mon[33038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-30T08:32:00.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:59 smithi189 ceph-mon[33038]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-30T08:32:00.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:31:59 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3751378488' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:32:00.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.083+0000 7f9e12400640 1 -- 172.21.15.32:0/1783426858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e0c10baf0 msgr2=0x7f9e0c10df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:00.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.083+0000 7f9e12400640 1 --2- 172.21.15.32:0/1783426858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e0c10baf0 0x7f9e0c10df10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9e04009ca0 tx=0x7f9e0402a8c0 comp rx=0 tx=0).stop 2024-04-30T08:32:00.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.083+0000 7f9e12400640 1 -- 172.21.15.32:0/1783426858 shutdown_connections 2024-04-30T08:32:00.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.083+0000 7f9e12400640 1 --2- 172.21.15.32:0/1783426858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e0c10baf0 0x7f9e0c10df10 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:00.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.083+0000 7f9e12400640 1 --2- 172.21.15.32:0/1783426858 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e0c106df0 0x7f9e0c107200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:00.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.083+0000 7f9e12400640 1 --2- 172.21.15.32:0/1783426858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e0c105bb0 0x7f9e0c1068b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:00.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.083+0000 7f9e12400640 1 -- 172.21.15.32:0/1783426858 >> 172.21.15.32:0/1783426858 conn(0x7f9e0c100da0 msgr2=0x7f9e0c103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:00.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.083+0000 7f9e12400640 1 -- 172.21.15.32:0/1783426858 shutdown_connections 2024-04-30T08:32:00.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.083+0000 7f9e12400640 1 -- 172.21.15.32:0/1783426858 wait complete. 2024-04-30T08:32:00.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.084+0000 7f9e12400640 1 Processor -- start 2024-04-30T08:32:00.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.085+0000 7f9e12400640 1 -- start start 2024-04-30T08:32:00.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.085+0000 7f9e12400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e0c105bb0 0x7f9e0c19c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:00.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.085+0000 7f9e12400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e0c106df0 0x7f9e0c19c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:00.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.085+0000 7f9e12400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e0c10baf0 0x7f9e0c1a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:00.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.085+0000 7f9e12400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9e0c111710 con 0x7f9e0c105bb0 2024-04-30T08:32:00.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.085+0000 7f9e12400640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f9e0c111590 con 0x7f9e0c106df0 2024-04-30T08:32:00.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.085+0000 7f9e12400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9e0c111890 con 0x7f9e0c10baf0 2024-04-30T08:32:00.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.085+0000 7f9e11000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e0c105bb0 0x7f9e0c19c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:00.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.085+0000 7f9e0be00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e0c106df0 0x7f9e0c19c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:00.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.085+0000 7f9e11000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e0c105bb0 0x7f9e0c19c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44624/0 (socket says 172.21.15.32:44624) 2024-04-30T08:32:00.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.085+0000 7f9e11000640 1 -- 172.21.15.32:0/1873334093 learned_addr learned my addr 172.21.15.32:0/1873334093 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:00.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.085+0000 7f9e11a00640 1 --2- 172.21.15.32:0/1873334093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e0c10baf0 0x7f9e0c1a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:00.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.086+0000 7f9e0be00640 1 -- 172.21.15.32:0/1873334093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e0c10baf0 msgr2=0x7f9e0c1a3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:00.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.086+0000 7f9e0be00640 1 --2- 172.21.15.32:0/1873334093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e0c10baf0 0x7f9e0c1a3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:00.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.086+0000 7f9e0be00640 1 -- 172.21.15.32:0/1873334093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e0c105bb0 msgr2=0x7f9e0c19c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:00.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.086+0000 7f9e0be00640 1 --2- 172.21.15.32:0/1873334093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e0c105bb0 0x7f9e0c19c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:00.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.086+0000 7f9e0be00640 1 -- 172.21.15.32:0/1873334093 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e0c1a3dc0 con 0x7f9e0c106df0 2024-04-30T08:32:00.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.086+0000 7f9e0be00640 1 --2- 172.21.15.32:0/1873334093 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e0c106df0 0x7f9e0c19c7c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc00e9f0 tx=0x7f9dfc00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:00.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.086+0000 7f9e09600640 1 -- 172.21.15.32:0/1873334093 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc00cd80 con 0x7f9e0c106df0 2024-04-30T08:32:00.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.086+0000 7f9e12400640 1 -- 172.21.15.32:0/1873334093 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e0c1a40b0 con 0x7f9e0c106df0 2024-04-30T08:32:00.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.087+0000 7f9e09600640 1 -- 172.21.15.32:0/1873334093 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9dfc0035f0 con 0x7f9e0c106df0 2024-04-30T08:32:00.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.087+0000 7f9e09600640 1 -- 172.21.15.32:0/1873334093 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9dfc010690 con 0x7f9e0c106df0 2024-04-30T08:32:00.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.087+0000 7f9e12400640 1 -- 172.21.15.32:0/1873334093 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e0c1a4600 con 0x7f9e0c106df0 2024-04-30T08:32:00.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.088+0000 7f9e12400640 1 -- 172.21.15.32:0/1873334093 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e0c10a900 con 0x7f9e0c106df0 2024-04-30T08:32:00.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.090+0000 7f9e09600640 1 -- 172.21.15.32:0/1873334093 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f9dfc0037a0 con 0x7f9e0c106df0 2024-04-30T08:32:00.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.092+0000 7f9e09600640 1 --2- 172.21.15.32:0/1873334093 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f9df40afae0 0x7f9df40b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:00.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.092+0000 7f9e09600640 1 -- 172.21.15.32:0/1873334093 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f9dfc0df180 con 0x7f9e0c106df0 2024-04-30T08:32:00.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.092+0000 7f9e11000640 1 --2- 172.21.15.32:0/1873334093 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f9df40afae0 0x7f9df40b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:00.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.093+0000 7f9e11000640 1 --2- 172.21.15.32:0/1873334093 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f9df40afae0 0x7f9df40b1ff0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9e00009c30 tx=0x7f9e00005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:00.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.094+0000 7f9e09600640 1 -- 172.21.15.32:0/1873334093 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9dfc0a7b50 con 0x7f9e0c106df0 2024-04-30T08:32:00.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:59 smithi032 ceph-mon[28331]: osdmap e81: 12 total, 12 up, 12 in 2024-04-30T08:32:00.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:59 smithi032 ceph-mon[28331]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-04-30T08:32:00.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:59 smithi032 ceph-mon[28331]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-30T08:32:00.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:31:59 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3751378488' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:32:00.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.304+0000 7f9e12400640 1 -- 172.21.15.32:0/1873334093 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9e0c02dda0 con 0x7f9e0c106df0 2024-04-30T08:32:00.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.306+0000 7f9e09600640 1 -- 172.21.15.32:0/1873334093 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+24053 (secure 0 0 0) 0x7f9dfc0a7450 con 0x7f9e0c106df0 2024-04-30T08:32:00.307 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:32:00.307 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":81,"fsid":"98c0a670-06ca-11ef-bc93-c7b262605968","created":"2024-04-30T08:22:11.122875+0000","modified":"2024-04-30T08:31:58.298969+0000","last_up_change":"2024-04-30T08:30:50.350439+0000","last_in_change":"2024-04-30T08:30:30.210620+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-30T08:26:19.728331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-30T08:31:49.800542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-30T08:31:51.211320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-30T08:31:55.716625+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a7e060c8-1cbb-418f-9a03-06693f6b3f52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6803","nonce":2624700402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6805","nonce":2624700402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6809","nonce":2624700402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6807","nonce":2624700402}]},"public_addr":"172.21.15.32:6803/2624700402","cluster_addr":"172.21.15.32:6805/2624700402","heartbeat_back_addr":"172.21.15.32:6809/2624700402","heartbeat_front_addr":"172.21.15.32:6807/2624700402","state":["exists","up"]},{"osd":1,"uuid":"72ef2fb1-e5ca-48ef-aa3a-4796a64b922f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6801","nonce":444649711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6803","nonce":444649711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6807","nonce":444649711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6805","nonce":444649711}]},"public_addr":"172.21.15.71:6801/444649711","cluster_addr":"172.21.15.71:6803/444649711","heartbeat_back_addr":"172.21.15.71:6807/444649711","heartbeat_front_addr":"172.21.15.71:6805/444649711","state":["exists","up"]},{"osd":2,"uuid":"704e5e8a-935e-47c7-ae28-27119876ac09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6801","nonce":3255528671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6803","nonce":3255528671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6807","nonce":3255528671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6805","nonce":3255528671}]},"public_addr":"172.21.15.189:6801/3255528671","cluster_addr":"172.21.15.189:6803/3255528671","heartbeat_back_addr":"172.21.15.189:6807/3255528671","heartbeat_front_addr":"172.21.15.189:6805/3255528671","state":["exists","up"]},{"osd":3,"uuid":"575004f7-6559-4a62-863e-e33078b0eebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6811","nonce":1384702112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6813","nonce":1384702112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6817","nonce":1384702112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6815","nonce":1384702112}]},"public_addr":"172.21.15.32:6811/1384702112","cluster_addr":"172.21.15.32:6813/1384702112","heartbeat_back_addr":"172.21.15.32:6817/1384702112","heartbeat_front_addr":"172.21.15.32:6815/1384702112","state":["exists","up"]},{"osd":4,"uuid":"e1c1e894-074d-4c78-89ef-9e5a114268bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6809","nonce":4250534574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6811","nonce":4250534574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6815","nonce":4250534574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6813","nonce":4250534574}]},"public_addr":"172.21.15.71:6809/4250534574","cluster_addr":"172.21.15.71:6811/4250534574","heartbeat_back_addr":"172.21.15.71:6815/4250534574","heartbeat_front_addr":"172.21.15.71:6813/4250534574","state":["exists","up"]},{"osd":5,"uuid":"12c022d4-fac6-40c4-ae7a-691571bee5d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6809","nonce":4245346198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6811","nonce":4245346198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6815","nonce":4245346198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6813","nonce":4245346198}]},"public_addr":"172.21.15.189:6809/4245346198","cluster_addr":"172.21.15.189:6811/4245346198","heartbeat_back_addr":"172.21.15.189:6815/4245346198","heartbeat_front_addr":"172.21.15.189:6813/4245346198","state":["exists","up"]},{"osd":6,"uuid":"b413f8ee-3d4a-415a-a2ba-d68437b7330d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6819","nonce":1910389190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6821","nonce":1910389190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6825","nonce":1910389190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6823","nonce":1910389190}]},"public_addr":"172.21.15.32:6819/1910389190","cluster_addr":"172.21.15.32:6821/1910389190","heartbeat_back_addr":"172.21.15.32:6825/1910389190","heartbeat_front_addr":"172.21.15.32:6823/1910389190","state":["exists","up"]},{"osd":7,"uuid":"aac64fb3-fc13-4126-b842-7b06ad5ce238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6817","nonce":692607380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6819","nonce":692607380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6823","nonce":692607380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6821","nonce":692607380}]},"public_addr":"172.21.15.71:6817/692607380","cluster_addr":"172.21.15.71:6819/692607380","heartbeat_back_addr":"172.21.15.71:6823/692607380","heartbeat_front_addr":"172.21.15.71:6821/692607380","state":["exists","up"]},{"osd":8,"uuid":"cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6817","nonce":3973697069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6819","nonce":3973697069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6823","nonce":3973697069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6821","nonce":3973697069}]},"public_addr":"172.21.15.189:6817/3973697069","cluster_addr":"172.21.15.189:6819/3973697069","heartbeat_back_addr":"172.21.15.189:6823/3973697069","heartbeat_front_addr":"172.21.15.189:6821/3973697069","state":["exists","up"]},{"osd":9,"uuid":"ef4a1645-ae6e-45c3-82ad-161510e62aa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6827","nonce":46066685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6829","nonce":46066685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6833","nonce":46066685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6831","nonce":46066685}]},"public_addr":"172.21.15.32:6827/46066685","cluster_addr":"172.21.15.32:6829/46066685","heartbeat_back_addr":"172.21.15.32:6833/46066685","heartbeat_front_addr":"172.21.15.32:6831/46066685","state":["exists","up"]},{"osd":10,"uuid":"25b26a38-a644-409f-b941-59efdf04a3ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6825","nonce":1886290293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6827","nonce":1886290293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6831","nonce":1886290293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6829","nonce":1886290293}]},"public_addr":"172.21.15.71:6825/1886290293","cluster_addr":"172.21.15.71:6827/1886290293","heartbeat_back_addr":"172.21.15.71:6831/1886290293","heartbeat_front_addr":"172.21.15.71:6829/1886290293","state":["exists","up"]},{"osd":11,"uuid":"ab96b3ff-3688-455b-af64-60226c5bfbff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6825","nonce":850185001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6827","nonce":850185001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6831","nonce":850185001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6829","nonce":850185001}]},"public_addr":"172.21.15.189:6825/850185001","cluster_addr":"172.21.15.189:6827/850185001","heartbeat_back_addr":"172.21.15.189:6831/850185001","heartbeat_front_addr":"172.21.15.189:6829/850185001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:27.220308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:53.230783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:17.190642+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:52.603799+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:20.568671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:46.158363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:22.706933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:50.665173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:16.602410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:53.814698+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:22.435189+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:48.651949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1266897222":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/2915689533":"2024-05-01T08:23:29.526664+0000","172.21.15.32:6801/4019351306":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/425836193":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/1134075722":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6801/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/21059253":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/3245550894":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/4019351306":"2024-05-01T08:23:29.526664+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-30T08:32:00.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.309+0000 7f9e12400640 1 -- 172.21.15.32:0/1873334093 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f9df40afae0 msgr2=0x7f9df40b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:00.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.309+0000 7f9e12400640 1 --2- 172.21.15.32:0/1873334093 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f9df40afae0 0x7f9df40b1ff0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9e00009c30 tx=0x7f9e00005f90 comp rx=0 tx=0).stop 2024-04-30T08:32:00.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.309+0000 7f9e12400640 1 -- 172.21.15.32:0/1873334093 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e0c106df0 msgr2=0x7f9e0c19c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:00.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.309+0000 7f9e12400640 1 --2- 172.21.15.32:0/1873334093 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e0c106df0 0x7f9e0c19c7c0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9dfc00e9f0 tx=0x7f9dfc00eeb0 comp rx=0 tx=0).stop 2024-04-30T08:32:00.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.309+0000 7f9e12400640 1 -- 172.21.15.32:0/1873334093 shutdown_connections 2024-04-30T08:32:00.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.309+0000 7f9e12400640 1 --2- 172.21.15.32:0/1873334093 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f9df40afae0 0x7f9df40b1ff0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:00.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.309+0000 7f9e12400640 1 --2- 172.21.15.32:0/1873334093 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e0c10baf0 0x7f9e0c1a3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:00.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.309+0000 7f9e12400640 1 --2- 172.21.15.32:0/1873334093 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9e0c106df0 0x7f9e0c19c7c0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:00.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.309+0000 7f9e12400640 1 --2- 172.21.15.32:0/1873334093 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9e0c105bb0 0x7f9e0c19c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:00.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.309+0000 7f9e12400640 1 -- 172.21.15.32:0/1873334093 >> 172.21.15.32:0/1873334093 conn(0x7f9e0c100da0 msgr2=0x7f9e0c107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:00.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.309+0000 7f9e12400640 1 -- 172.21.15.32:0/1873334093 shutdown_connections 2024-04-30T08:32:00.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.310+0000 7f9e12400640 1 -- 172.21.15.32:0/1873334093 wait complete. 2024-04-30T08:32:00.351 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-30T08:32:00.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.638+0000 7fc643e00640 1 -- 172.21.15.32:0/3756318900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6441028b0 msgr2=0x7fc6441035b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:00.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.638+0000 7fc643e00640 1 --2- 172.21.15.32:0/3756318900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6441028b0 0x7fc6441035b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc634009900 tx=0x7fc63402a3f0 comp rx=0 tx=0).stop 2024-04-30T08:32:00.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.638+0000 7fc643e00640 1 -- 172.21.15.32:0/3756318900 shutdown_connections 2024-04-30T08:32:00.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.638+0000 7fc643e00640 1 --2- 172.21.15.32:0/3756318900 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6441087f0 0x7fc64410ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:00.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.638+0000 7fc643e00640 1 --2- 172.21.15.32:0/3756318900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc644103af0 0x7fc644103f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:00.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.638+0000 7fc643e00640 1 --2- 172.21.15.32:0/3756318900 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc6441028b0 0x7fc6441035b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:00.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.638+0000 7fc643e00640 1 -- 172.21.15.32:0/3756318900 >> 172.21.15.32:0/3756318900 conn(0x7fc6440fdaa0 msgr2=0x7fc6440fff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:00.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.638+0000 7fc643e00640 1 -- 172.21.15.32:0/3756318900 shutdown_connections 2024-04-30T08:32:00.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.638+0000 7fc643e00640 1 -- 172.21.15.32:0/3756318900 wait complete. 2024-04-30T08:32:00.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.639+0000 7fc643e00640 1 Processor -- start 2024-04-30T08:32:00.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.639+0000 7fc643e00640 1 -- start start 2024-04-30T08:32:00.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.639+0000 7fc643e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6441028b0 0x7fc64419c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:00.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.639+0000 7fc643e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc644103af0 0x7fc64419c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:00.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.639+0000 7fc643e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6441087f0 0x7fc6441a37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:00.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.639+0000 7fc643e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc64410f4e0 con 0x7fc644103af0 2024-04-30T08:32:00.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.639+0000 7fc643e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc64410f360 con 0x7fc6441087f0 2024-04-30T08:32:00.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.639+0000 7fc643e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc64410f660 con 0x7fc6441028b0 2024-04-30T08:32:00.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.640+0000 7fc63a000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc644103af0 0x7fc64419c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:00.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.640+0000 7fc63a000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc644103af0 0x7fc64419c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44636/0 (socket says 172.21.15.32:44636) 2024-04-30T08:32:00.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.640+0000 7fc63a000640 1 -- 172.21.15.32:0/1458939219 learned_addr learned my addr 172.21.15.32:0/1458939219 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:00.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.640+0000 7fc643400640 1 --2- 172.21.15.32:0/1458939219 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6441087f0 0x7fc6441a37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:00.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.640+0000 7fc642a00640 1 --2- 172.21.15.32:0/1458939219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6441028b0 0x7fc64419c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:00.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.640+0000 7fc63a000640 1 -- 172.21.15.32:0/1458939219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6441028b0 msgr2=0x7fc64419c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:00.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.640+0000 7fc63a000640 1 --2- 172.21.15.32:0/1458939219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6441028b0 0x7fc64419c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:00.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.640+0000 7fc63a000640 1 -- 172.21.15.32:0/1458939219 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6441087f0 msgr2=0x7fc6441a37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:00.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.640+0000 7fc63a000640 1 --2- 172.21.15.32:0/1458939219 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6441087f0 0x7fc6441a37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:00.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.640+0000 7fc63a000640 1 -- 172.21.15.32:0/1458939219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6441a3e30 con 0x7fc644103af0 2024-04-30T08:32:00.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.640+0000 7fc643400640 1 --2- 172.21.15.32:0/1458939219 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6441087f0 0x7fc6441a37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:32:00.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.641+0000 7fc63a000640 1 --2- 172.21.15.32:0/1458939219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc644103af0 0x7fc64419c830 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc62c00e9f0 tx=0x7fc62c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:00.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.641+0000 7fc63be00640 1 -- 172.21.15.32:0/1458939219 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62c00cd80 con 0x7fc644103af0 2024-04-30T08:32:00.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.641+0000 7fc63be00640 1 -- 172.21.15.32:0/1458939219 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc62c0054d0 con 0x7fc644103af0 2024-04-30T08:32:00.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.641+0000 7fc63be00640 1 -- 172.21.15.32:0/1458939219 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62c010690 con 0x7fc644103af0 2024-04-30T08:32:00.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.641+0000 7fc643e00640 1 -- 172.21.15.32:0/1458939219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6441a4120 con 0x7fc644103af0 2024-04-30T08:32:00.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.641+0000 7fc643e00640 1 -- 172.21.15.32:0/1458939219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6441a4670 con 0x7fc644103af0 2024-04-30T08:32:00.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.642+0000 7fc643e00640 1 -- 172.21.15.32:0/1458939219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6441a4930 con 0x7fc644103af0 2024-04-30T08:32:00.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.646+0000 7fc63be00640 1 -- 172.21.15.32:0/1458939219 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fc62c004e30 con 0x7fc644103af0 2024-04-30T08:32:00.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.648+0000 7fc63be00640 1 --2- 172.21.15.32:0/1458939219 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc6280afb60 0x7fc6280b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:00.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.648+0000 7fc63be00640 1 -- 172.21.15.32:0/1458939219 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7fc62c0df4f0 con 0x7fc644103af0 2024-04-30T08:32:00.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.648+0000 7fc642a00640 1 --2- 172.21.15.32:0/1458939219 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc6280afb60 0x7fc6280b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:00.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.649+0000 7fc642a00640 1 --2- 172.21.15.32:0/1458939219 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc6280afb60 0x7fc6280b2070 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc634005ab0 tx=0x7fc634031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:00.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.650+0000 7fc63be00640 1 -- 172.21.15.32:0/1458939219 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc62c053ff0 con 0x7fc644103af0 2024-04-30T08:32:00.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.900+0000 7fc643e00640 1 -- 172.21.15.32:0/1458939219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc6441a4b00 con 0x7fc644103af0 2024-04-30T08:32:00.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.902+0000 7fc63be00640 1 -- 172.21.15.32:0/1458939219 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11791 (secure 0 0 0) 0x7fc62c0a7e30 con 0x7fc644103af0 2024-04-30T08:32:00.903 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:32:00.903 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2235842676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2235842676},{"type":"v1","addr":"172.21.15.32:6835","nonce":2235842676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/3577009355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":3577009355},{"type":"v1","addr":"172.21.15.32:6841","nonce":3577009355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24442,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/463203038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":463203038},{"type":"v1","addr":"172.21.15.189:6833","nonce":463203038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24458,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6833/3635143591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3635143591},{"type":"v1","addr":"172.21.15.71:6833","nonce":3635143591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24484,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6837/2569449994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2569449994},{"type":"v1","addr":"172.21.15.32:6837","nonce":2569449994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24505,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/278686565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":278686565},{"type":"v1","addr":"172.21.15.189:6835","nonce":278686565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24506,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6835/1558303056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1558303056},{"type":"v1","addr":"172.21.15.71:6835","nonce":1558303056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24517,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/686520873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":686520873},{"type":"v1","addr":"172.21.15.189:6837","nonce":686520873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24533,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.71:6837/2324039932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2324039932},{"type":"v1","addr":"172.21.15.71:6837","nonce":2324039932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24541,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6839/352970107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":352970107},{"type":"v1","addr":"172.21.15.189:6839","nonce":352970107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24560,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6839/4160386524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4160386524},{"type":"v1","addr":"172.21.15.32:6839","nonce":4160386524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-04-30T08:31:52.236668+0000","modified":"2024-04-30T08:31:58.299202+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24590},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.71:6839/3517409993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3517409993},{"type":"v1","addr":"172.21.15.71:6839","nonce":3517409993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24590,"qdb_cluster":[24590]},"id":1}]} 2024-04-30T08:32:00.903 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 17 2024-04-30T08:32:00.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.905+0000 7fc643e00640 1 -- 172.21.15.32:0/1458939219 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc6280afb60 msgr2=0x7fc6280b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:00.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.905+0000 7fc643e00640 1 --2- 172.21.15.32:0/1458939219 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc6280afb60 0x7fc6280b2070 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fc634005ab0 tx=0x7fc634031400 comp rx=0 tx=0).stop 2024-04-30T08:32:00.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.905+0000 7fc643e00640 1 -- 172.21.15.32:0/1458939219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc644103af0 msgr2=0x7fc64419c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:00.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.905+0000 7fc643e00640 1 --2- 172.21.15.32:0/1458939219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc644103af0 0x7fc64419c830 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc62c00e9f0 tx=0x7fc62c00edf0 comp rx=0 tx=0).stop 2024-04-30T08:32:00.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.905+0000 7fc643e00640 1 -- 172.21.15.32:0/1458939219 shutdown_connections 2024-04-30T08:32:00.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.905+0000 7fc643e00640 1 --2- 172.21.15.32:0/1458939219 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc6280afb60 0x7fc6280b2070 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:00.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.905+0000 7fc643e00640 1 --2- 172.21.15.32:0/1458939219 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc6441087f0 0x7fc6441a37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:00.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.905+0000 7fc643e00640 1 --2- 172.21.15.32:0/1458939219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc644103af0 0x7fc64419c830 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:00.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.905+0000 7fc643e00640 1 --2- 172.21.15.32:0/1458939219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc6441028b0 0x7fc64419c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:00.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.905+0000 7fc643e00640 1 -- 172.21.15.32:0/1458939219 >> 172.21.15.32:0/1458939219 conn(0x7fc6440fdaa0 msgr2=0x7fc6440fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:00.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.905+0000 7fc643e00640 1 -- 172.21.15.32:0/1458939219 shutdown_connections 2024-04-30T08:32:00.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:00.905+0000 7fc643e00640 1 -- 172.21.15.32:0/1458939219 wait complete. 2024-04-30T08:32:00.932 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-30T08:32:01.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:00 smithi071 ceph-mon[33385]: pgmap v326: 129 pgs: 19 unknown, 110 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 766 B/s wr, 2 op/s 2024-04-30T08:32:01.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:00 smithi071 ceph-mon[33385]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-30T08:32:01.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:00 smithi071 ceph-mon[33385]: Cluster is now healthy 2024-04-30T08:32:01.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:00 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2284722201' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:32:01.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:00 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1873334093' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:32:01.004 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:00 smithi032 ceph-mon[28331]: pgmap v326: 129 pgs: 19 unknown, 110 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 766 B/s wr, 2 op/s 2024-04-30T08:32:01.004 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:00 smithi032 ceph-mon[28331]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-30T08:32:01.004 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:00 smithi032 ceph-mon[28331]: Cluster is now healthy 2024-04-30T08:32:01.004 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:00 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2284722201' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:32:01.005 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:00 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1873334093' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:32:01.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:00 smithi189 ceph-mon[33038]: pgmap v326: 129 pgs: 19 unknown, 110 active+clean; 579 KiB data, 355 MiB used, 1.0 TiB / 1.0 TiB avail; 766 B/s wr, 2 op/s 2024-04-30T08:32:01.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:00 smithi189 ceph-mon[33038]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-30T08:32:01.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:00 smithi189 ceph-mon[33038]: Cluster is now healthy 2024-04-30T08:32:01.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:00 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2284722201' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:32:01.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:00 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1873334093' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:32:01.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.219+0000 7f8b82e00640 1 -- 172.21.15.32:0/3048070688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b7c10baf0 msgr2=0x7f8b7c10df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:01.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.219+0000 7f8b82e00640 1 --2- 172.21.15.32:0/3048070688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b7c10baf0 0x7f8b7c10df10 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8b74009ca0 tx=0x7f8b7402a8c0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.219+0000 7f8b82e00640 1 -- 172.21.15.32:0/3048070688 shutdown_connections 2024-04-30T08:32:01.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.219+0000 7f8b82e00640 1 --2- 172.21.15.32:0/3048070688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b7c10baf0 0x7f8b7c10df10 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.219+0000 7f8b82e00640 1 --2- 172.21.15.32:0/3048070688 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8b7c106df0 0x7f8b7c107200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.219+0000 7f8b82e00640 1 --2- 172.21.15.32:0/3048070688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b7c105bb0 0x7f8b7c1068b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.219+0000 7f8b82e00640 1 -- 172.21.15.32:0/3048070688 >> 172.21.15.32:0/3048070688 conn(0x7f8b7c100da0 msgr2=0x7f8b7c103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:01.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.219+0000 7f8b82e00640 1 -- 172.21.15.32:0/3048070688 shutdown_connections 2024-04-30T08:32:01.221 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.219+0000 7f8b82e00640 1 -- 172.21.15.32:0/3048070688 wait complete. 2024-04-30T08:32:01.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.220+0000 7f8b82e00640 1 Processor -- start 2024-04-30T08:32:01.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.220+0000 7f8b82e00640 1 -- start start 2024-04-30T08:32:01.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.221+0000 7f8b82e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b7c105bb0 0x7f8b7c19c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:01.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.221+0000 7f8b82e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b7c106df0 0x7f8b7c19c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:01.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.221+0000 7f8b82e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8b7c10baf0 0x7f8b7c1a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:01.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.221+0000 7f8b82e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8b7c1116e0 con 0x7f8b7c105bb0 2024-04-30T08:32:01.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.221+0000 7f8b82e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8b7c111560 con 0x7f8b7c10baf0 2024-04-30T08:32:01.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.221+0000 7f8b82e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8b7c111860 con 0x7f8b7c106df0 2024-04-30T08:32:01.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.221+0000 7f8b81a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b7c105bb0 0x7f8b7c19c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:01.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.221+0000 7f8b81a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b7c105bb0 0x7f8b7c19c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44658/0 (socket says 172.21.15.32:44658) 2024-04-30T08:32:01.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.221+0000 7f8b81a00640 1 -- 172.21.15.32:0/3339546680 learned_addr learned my addr 172.21.15.32:0/3339546680 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:01.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.221+0000 7f8b82400640 1 --2- 172.21.15.32:0/3339546680 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8b7c10baf0 0x7f8b7c1a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:01.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.221+0000 7f8b81000640 1 --2- 172.21.15.32:0/3339546680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b7c106df0 0x7f8b7c19c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:01.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.221+0000 7f8b81a00640 1 -- 172.21.15.32:0/3339546680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b7c106df0 msgr2=0x7f8b7c19c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:01.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.221+0000 7f8b81a00640 1 --2- 172.21.15.32:0/3339546680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b7c106df0 0x7f8b7c19c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.221+0000 7f8b81a00640 1 -- 172.21.15.32:0/3339546680 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8b7c10baf0 msgr2=0x7f8b7c1a3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:01.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.221+0000 7f8b81a00640 1 --2- 172.21.15.32:0/3339546680 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8b7c10baf0 0x7f8b7c1a3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.222+0000 7f8b81a00640 1 -- 172.21.15.32:0/3339546680 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b7c1a3dc0 con 0x7f8b7c105bb0 2024-04-30T08:32:01.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.222+0000 7f8b81a00640 1 --2- 172.21.15.32:0/3339546680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b7c105bb0 0x7f8b7c19c280 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8b7000e9f0 tx=0x7f8b7000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:01.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.222+0000 7f8b7a000640 1 -- 172.21.15.32:0/3339546680 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b7000ccc0 con 0x7f8b7c105bb0 2024-04-30T08:32:01.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.222+0000 7f8b7a000640 1 -- 172.21.15.32:0/3339546680 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8b700035f0 con 0x7f8b7c105bb0 2024-04-30T08:32:01.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.222+0000 7f8b82e00640 1 -- 172.21.15.32:0/3339546680 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b7c1a40b0 con 0x7f8b7c105bb0 2024-04-30T08:32:01.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.222+0000 7f8b7a000640 1 -- 172.21.15.32:0/3339546680 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b7000ce60 con 0x7f8b7c105bb0 2024-04-30T08:32:01.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.222+0000 7f8b82e00640 1 -- 172.21.15.32:0/3339546680 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b7c1a45b0 con 0x7f8b7c105bb0 2024-04-30T08:32:01.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.223+0000 7f8b82e00640 1 -- 172.21.15.32:0/3339546680 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b7c10a900 con 0x7f8b7c105bb0 2024-04-30T08:32:01.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.226+0000 7f8b7a000640 1 -- 172.21.15.32:0/3339546680 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f8b70003aa0 con 0x7f8b7c105bb0 2024-04-30T08:32:01.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.228+0000 7f8b7a000640 1 --2- 172.21.15.32:0/3339546680 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8b640afae0 0x7f8b640b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:01.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.228+0000 7f8b7a000640 1 -- 172.21.15.32:0/3339546680 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f8b700df210 con 0x7f8b7c105bb0 2024-04-30T08:32:01.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.228+0000 7f8b81000640 1 --2- 172.21.15.32:0/3339546680 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8b640afae0 0x7f8b640b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:01.229 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.229+0000 7f8b81000640 1 --2- 172.21.15.32:0/3339546680 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8b640afae0 0x7f8b640b1ff0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c19d320 tx=0x7f8b6c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:01.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.230+0000 7f8b7a000640 1 -- 172.21.15.32:0/3339546680 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8b700a7c50 con 0x7f8b7c105bb0 2024-04-30T08:32:01.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.437+0000 7f8b82e00640 1 -- 172.21.15.32:0/3339546680 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8b7c02dda0 con 0x7f8b7c105bb0 2024-04-30T08:32:01.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.442+0000 7f8b7a000640 1 -- 172.21.15.32:0/3339546680 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+24053 (secure 0 0 0) 0x7f8b700a7550 con 0x7f8b7c105bb0 2024-04-30T08:32:01.443 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:32:01.443 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":81,"fsid":"98c0a670-06ca-11ef-bc93-c7b262605968","created":"2024-04-30T08:22:11.122875+0000","modified":"2024-04-30T08:31:58.298969+0000","last_up_change":"2024-04-30T08:30:50.350439+0000","last_in_change":"2024-04-30T08:30:30.210620+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-30T08:26:19.728331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-30T08:31:49.800542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-30T08:31:51.211320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-30T08:31:55.716625+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a7e060c8-1cbb-418f-9a03-06693f6b3f52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6803","nonce":2624700402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6805","nonce":2624700402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6809","nonce":2624700402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6807","nonce":2624700402}]},"public_addr":"172.21.15.32:6803/2624700402","cluster_addr":"172.21.15.32:6805/2624700402","heartbeat_back_addr":"172.21.15.32:6809/2624700402","heartbeat_front_addr":"172.21.15.32:6807/2624700402","state":["exists","up"]},{"osd":1,"uuid":"72ef2fb1-e5ca-48ef-aa3a-4796a64b922f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6801","nonce":444649711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6803","nonce":444649711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6807","nonce":444649711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6805","nonce":444649711}]},"public_addr":"172.21.15.71:6801/444649711","cluster_addr":"172.21.15.71:6803/444649711","heartbeat_back_addr":"172.21.15.71:6807/444649711","heartbeat_front_addr":"172.21.15.71:6805/444649711","state":["exists","up"]},{"osd":2,"uuid":"704e5e8a-935e-47c7-ae28-27119876ac09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6801","nonce":3255528671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6803","nonce":3255528671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6807","nonce":3255528671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6805","nonce":3255528671}]},"public_addr":"172.21.15.189:6801/3255528671","cluster_addr":"172.21.15.189:6803/3255528671","heartbeat_back_addr":"172.21.15.189:6807/3255528671","heartbeat_front_addr":"172.21.15.189:6805/3255528671","state":["exists","up"]},{"osd":3,"uuid":"575004f7-6559-4a62-863e-e33078b0eebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6811","nonce":1384702112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6813","nonce":1384702112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6817","nonce":1384702112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6815","nonce":1384702112}]},"public_addr":"172.21.15.32:6811/1384702112","cluster_addr":"172.21.15.32:6813/1384702112","heartbeat_back_addr":"172.21.15.32:6817/1384702112","heartbeat_front_addr":"172.21.15.32:6815/1384702112","state":["exists","up"]},{"osd":4,"uuid":"e1c1e894-074d-4c78-89ef-9e5a114268bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6809","nonce":4250534574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6811","nonce":4250534574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6815","nonce":4250534574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6813","nonce":4250534574}]},"public_addr":"172.21.15.71:6809/4250534574","cluster_addr":"172.21.15.71:6811/4250534574","heartbeat_back_addr":"172.21.15.71:6815/4250534574","heartbeat_front_addr":"172.21.15.71:6813/4250534574","state":["exists","up"]},{"osd":5,"uuid":"12c022d4-fac6-40c4-ae7a-691571bee5d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6809","nonce":4245346198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6811","nonce":4245346198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6815","nonce":4245346198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6813","nonce":4245346198}]},"public_addr":"172.21.15.189:6809/4245346198","cluster_addr":"172.21.15.189:6811/4245346198","heartbeat_back_addr":"172.21.15.189:6815/4245346198","heartbeat_front_addr":"172.21.15.189:6813/4245346198","state":["exists","up"]},{"osd":6,"uuid":"b413f8ee-3d4a-415a-a2ba-d68437b7330d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6819","nonce":1910389190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6821","nonce":1910389190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6825","nonce":1910389190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6823","nonce":1910389190}]},"public_addr":"172.21.15.32:6819/1910389190","cluster_addr":"172.21.15.32:6821/1910389190","heartbeat_back_addr":"172.21.15.32:6825/1910389190","heartbeat_front_addr":"172.21.15.32:6823/1910389190","state":["exists","up"]},{"osd":7,"uuid":"aac64fb3-fc13-4126-b842-7b06ad5ce238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6817","nonce":692607380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6819","nonce":692607380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6823","nonce":692607380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6821","nonce":692607380}]},"public_addr":"172.21.15.71:6817/692607380","cluster_addr":"172.21.15.71:6819/692607380","heartbeat_back_addr":"172.21.15.71:6823/692607380","heartbeat_front_addr":"172.21.15.71:6821/692607380","state":["exists","up"]},{"osd":8,"uuid":"cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6817","nonce":3973697069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6819","nonce":3973697069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6823","nonce":3973697069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6821","nonce":3973697069}]},"public_addr":"172.21.15.189:6817/3973697069","cluster_addr":"172.21.15.189:6819/3973697069","heartbeat_back_addr":"172.21.15.189:6823/3973697069","heartbeat_front_addr":"172.21.15.189:6821/3973697069","state":["exists","up"]},{"osd":9,"uuid":"ef4a1645-ae6e-45c3-82ad-161510e62aa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6827","nonce":46066685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6829","nonce":46066685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6833","nonce":46066685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6831","nonce":46066685}]},"public_addr":"172.21.15.32:6827/46066685","cluster_addr":"172.21.15.32:6829/46066685","heartbeat_back_addr":"172.21.15.32:6833/46066685","heartbeat_front_addr":"172.21.15.32:6831/46066685","state":["exists","up"]},{"osd":10,"uuid":"25b26a38-a644-409f-b941-59efdf04a3ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6825","nonce":1886290293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6827","nonce":1886290293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6831","nonce":1886290293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6829","nonce":1886290293}]},"public_addr":"172.21.15.71:6825/1886290293","cluster_addr":"172.21.15.71:6827/1886290293","heartbeat_back_addr":"172.21.15.71:6831/1886290293","heartbeat_front_addr":"172.21.15.71:6829/1886290293","state":["exists","up"]},{"osd":11,"uuid":"ab96b3ff-3688-455b-af64-60226c5bfbff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6825","nonce":850185001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6827","nonce":850185001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6831","nonce":850185001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6829","nonce":850185001}]},"public_addr":"172.21.15.189:6825/850185001","cluster_addr":"172.21.15.189:6827/850185001","heartbeat_back_addr":"172.21.15.189:6831/850185001","heartbeat_front_addr":"172.21.15.189:6829/850185001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:27.220308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:53.230783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:17.190642+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:52.603799+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:20.568671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:46.158363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:22.706933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:50.665173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:16.602410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:53.814698+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:22.435189+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:48.651949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1266897222":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/2915689533":"2024-05-01T08:23:29.526664+0000","172.21.15.32:6801/4019351306":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/425836193":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/1134075722":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6801/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/21059253":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/3245550894":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/4019351306":"2024-05-01T08:23:29.526664+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-30T08:32:01.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.445+0000 7f8b82e00640 1 -- 172.21.15.32:0/3339546680 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8b640afae0 msgr2=0x7f8b640b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:01.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.445+0000 7f8b82e00640 1 --2- 172.21.15.32:0/3339546680 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8b640afae0 0x7f8b640b1ff0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c19d320 tx=0x7f8b6c005f90 comp rx=0 tx=0).stop 2024-04-30T08:32:01.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.445+0000 7f8b82e00640 1 -- 172.21.15.32:0/3339546680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b7c105bb0 msgr2=0x7f8b7c19c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:01.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.445+0000 7f8b82e00640 1 --2- 172.21.15.32:0/3339546680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b7c105bb0 0x7f8b7c19c280 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8b7000e9f0 tx=0x7f8b7000eeb0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.445+0000 7f8b82e00640 1 -- 172.21.15.32:0/3339546680 shutdown_connections 2024-04-30T08:32:01.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.445+0000 7f8b82e00640 1 --2- 172.21.15.32:0/3339546680 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8b640afae0 0x7f8b640b1ff0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.445+0000 7f8b82e00640 1 --2- 172.21.15.32:0/3339546680 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8b7c10baf0 0x7f8b7c1a3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.445+0000 7f8b82e00640 1 --2- 172.21.15.32:0/3339546680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8b7c106df0 0x7f8b7c19c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.445+0000 7f8b82e00640 1 --2- 172.21.15.32:0/3339546680 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8b7c105bb0 0x7f8b7c19c280 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.445+0000 7f8b82e00640 1 -- 172.21.15.32:0/3339546680 >> 172.21.15.32:0/3339546680 conn(0x7f8b7c100da0 msgr2=0x7f8b7c107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:01.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.445+0000 7f8b82e00640 1 -- 172.21.15.32:0/3339546680 shutdown_connections 2024-04-30T08:32:01.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.445+0000 7f8b82e00640 1 -- 172.21.15.32:0/3339546680 wait complete. 2024-04-30T08:32:01.472 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-04-30T08:31:52.236668+0000', 'modified': '2024-04-30T08:31:58.299202+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24590}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24590': {'gid': 24590, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.71:6839/3517409993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6838', 'nonce': 3517409993}, {'type': 'v1', 'addr': '172.21.15.71:6839', 'nonce': 3517409993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24590, 'qdb_cluster': [24590]} 2024-04-30T08:32:01.472 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-04-30T08:32:01.473 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-30T08:32:01.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.766+0000 7f1301a00640 1 -- 172.21.15.32:0/2454060822 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f12fc105bb0 msgr2=0x7f12fc1068b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:01.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.766+0000 7f1301a00640 1 --2- 172.21.15.32:0/2454060822 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f12fc105bb0 0x7f12fc1068b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f12ec009900 tx=0x7f12ec02a3f0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.767+0000 7f1301a00640 1 -- 172.21.15.32:0/2454060822 shutdown_connections 2024-04-30T08:32:01.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.767+0000 7f1301a00640 1 --2- 172.21.15.32:0/2454060822 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12fc10baf0 0x7f12fc10df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.767+0000 7f1301a00640 1 --2- 172.21.15.32:0/2454060822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12fc106df0 0x7f12fc107200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.767+0000 7f1301a00640 1 --2- 172.21.15.32:0/2454060822 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f12fc105bb0 0x7f12fc1068b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.767+0000 7f1301a00640 1 -- 172.21.15.32:0/2454060822 >> 172.21.15.32:0/2454060822 conn(0x7f12fc100da0 msgr2=0x7f12fc103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:01.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.767+0000 7f1301a00640 1 -- 172.21.15.32:0/2454060822 shutdown_connections 2024-04-30T08:32:01.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.767+0000 7f1301a00640 1 -- 172.21.15.32:0/2454060822 wait complete. 2024-04-30T08:32:01.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.768+0000 7f1301a00640 1 Processor -- start 2024-04-30T08:32:01.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.768+0000 7f1301a00640 1 -- start start 2024-04-30T08:32:01.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.769+0000 7f1301a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12fc105bb0 0x7f12fc19c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:01.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.769+0000 7f1301a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12fc106df0 0x7f12fc19c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:01.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.769+0000 7f1301a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f12fc10baf0 0x7f12fc1a3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:01.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.769+0000 7f1301a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f12fc111700 con 0x7f12fc106df0 2024-04-30T08:32:01.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.769+0000 7f1301a00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f12fc111580 con 0x7f12fc10baf0 2024-04-30T08:32:01.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.769+0000 7f1301a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f12fc111880 con 0x7f12fc105bb0 2024-04-30T08:32:01.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.769+0000 7f12fb400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12fc106df0 0x7f12fc19c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:01.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.769+0000 7f12fbe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12fc105bb0 0x7f12fc19c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:01.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.769+0000 7f12fb400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12fc106df0 0x7f12fc19c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44678/0 (socket says 172.21.15.32:44678) 2024-04-30T08:32:01.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.769+0000 7f12fb400640 1 -- 172.21.15.32:0/1630881232 learned_addr learned my addr 172.21.15.32:0/1630881232 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:01.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.769+0000 7f1301000640 1 --2- 172.21.15.32:0/1630881232 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f12fc10baf0 0x7f12fc1a3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:01.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.770+0000 7f12fbe00640 1 -- 172.21.15.32:0/1630881232 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f12fc10baf0 msgr2=0x7f12fc1a3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:01.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.770+0000 7f12fbe00640 1 --2- 172.21.15.32:0/1630881232 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f12fc10baf0 0x7f12fc1a3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.770+0000 7f12fbe00640 1 -- 172.21.15.32:0/1630881232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12fc106df0 msgr2=0x7f12fc19c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:01.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.770+0000 7f12fbe00640 1 --2- 172.21.15.32:0/1630881232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12fc106df0 0x7f12fc19c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.770+0000 7f12fbe00640 1 -- 172.21.15.32:0/1630881232 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12fc1a3e70 con 0x7f12fc105bb0 2024-04-30T08:32:01.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.770+0000 7f12fbe00640 1 --2- 172.21.15.32:0/1630881232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12fc105bb0 0x7f12fc19c330 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f12ec009230 tx=0x7f12ec003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:01.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.770+0000 7f12f8c00640 1 -- 172.21.15.32:0/1630881232 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12ec03f070 con 0x7f12fc105bb0 2024-04-30T08:32:01.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.770+0000 7f12f8c00640 1 -- 172.21.15.32:0/1630881232 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12ec031c50 con 0x7f12fc105bb0 2024-04-30T08:32:01.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.770+0000 7f12f8c00640 1 -- 172.21.15.32:0/1630881232 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12ec035030 con 0x7f12fc105bb0 2024-04-30T08:32:01.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.770+0000 7f1301a00640 1 -- 172.21.15.32:0/1630881232 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12fc1a4100 con 0x7f12fc105bb0 2024-04-30T08:32:01.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.771+0000 7f1301a00640 1 -- 172.21.15.32:0/1630881232 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12fc1a4570 con 0x7f12fc105bb0 2024-04-30T08:32:01.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.775+0000 7f1309ff3740 1 Processor -- start 2024-04-30T08:32:01.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.775+0000 7f1309ff3740 1 -- start start 2024-04-30T08:32:01.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.776+0000 7f1309ff3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55914f3bf270 0x55914f761530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:01.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.776+0000 7f1309ff3740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55914f3963b0 0x55914f761ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:01.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.776+0000 7f1309ff3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55914f39e590 0x55914f762740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:01.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.776+0000 7f1309ff3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55914f6c5790 con 0x55914f3bf270 2024-04-30T08:32:01.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.776+0000 7f1309ff3740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55914f66af20 con 0x55914f3963b0 2024-04-30T08:32:01.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.776+0000 7f1309ff3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55914f6c1750 con 0x55914f39e590 2024-04-30T08:32:01.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.776+0000 7f12f8c00640 1 -- 172.21.15.32:0/1630881232 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f12ec042070 con 0x7f12fc105bb0 2024-04-30T08:32:01.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.776+0000 7f1301000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55914f3bf270 0x55914f761530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:01.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.776+0000 7f1301000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55914f3bf270 0x55914f761530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44690/0 (socket says 172.21.15.32:44690) 2024-04-30T08:32:01.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.776+0000 7f1301000640 1 -- 172.21.15.32:0/2406637603 learned_addr learned my addr 172.21.15.32:0/2406637603 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:01.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.776+0000 7f12fb400640 1 --2- 172.21.15.32:0/2406637603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55914f39e590 0x55914f762740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:01.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.776+0000 7f12fbe00640 1 --2- 172.21.15.32:0/2406637603 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55914f3963b0 0x55914f761ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:01.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.777+0000 7f1301000640 1 -- 172.21.15.32:0/2406637603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55914f39e590 msgr2=0x55914f762740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:32:01.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.777+0000 7f1301000640 1 --2- 172.21.15.32:0/2406637603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55914f39e590 0x55914f762740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.777+0000 7f1301000640 1 -- 172.21.15.32:0/2406637603 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55914f3963b0 msgr2=0x55914f761ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:32:01.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.777+0000 7f1301000640 1 --2- 172.21.15.32:0/2406637603 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55914f3963b0 0x55914f761ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.777+0000 7f1301000640 1 -- 172.21.15.32:0/2406637603 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55914f3d72d0 con 0x55914f3bf270 2024-04-30T08:32:01.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.777+0000 7f12fb400640 1 --2- 172.21.15.32:0/2406637603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55914f39e590 0x55914f762740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:32:01.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.777+0000 7f12fbe00640 1 --2- 172.21.15.32:0/2406637603 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55914f3963b0 0x55914f761ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:32:01.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.777+0000 7f1301000640 1 --2- 172.21.15.32:0/2406637603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55914f3bf270 0x55914f761530 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f12f4011440 tx=0x7f12f4011900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c61862643d68f897 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:01.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.777+0000 7f12f1600640 1 -- 172.21.15.32:0/2406637603 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f4003c30 con 0x55914f3bf270 2024-04-30T08:32:01.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.777+0000 7f12f1600640 1 -- 172.21.15.32:0/2406637603 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12f4003dd0 con 0x55914f3bf270 2024-04-30T08:32:01.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.778+0000 7f12f1600640 1 -- 172.21.15.32:0/2406637603 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12f400fae0 con 0x55914f3bf270 2024-04-30T08:32:01.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.778+0000 7f12f8c00640 1 --2- 172.21.15.32:0/1630881232 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f12e00afae0 0x7f12e00b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:01.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.778+0000 7f12f8c00640 1 -- 172.21.15.32:0/1630881232 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f12ec0ff4f0 con 0x7f12fc105bb0 2024-04-30T08:32:01.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.778+0000 7f12fbe00640 1 --2- 172.21.15.32:0/1630881232 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f12e00afae0 0x7f12e00b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:01.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.778+0000 7f1309ff3740 1 -- 172.21.15.32:0/2406637603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55914f3bf270 msgr2=0x55914f761530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:01.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.778+0000 7f1309ff3740 1 --2- 172.21.15.32:0/2406637603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55914f3bf270 0x55914f761530 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f12f4011440 tx=0x7f12f4011900 comp rx=0 tx=0).stop 2024-04-30T08:32:01.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.779+0000 7f1309ff3740 1 -- 172.21.15.32:0/2406637603 shutdown_connections 2024-04-30T08:32:01.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.779+0000 7f1309ff3740 1 --2- 172.21.15.32:0/2406637603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55914f39e590 0x55914f762740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.779+0000 7f1309ff3740 1 --2- 172.21.15.32:0/2406637603 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55914f3963b0 0x55914f761ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.779+0000 7f1309ff3740 1 --2- 172.21.15.32:0/2406637603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55914f3bf270 0x55914f761530 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.779+0000 7f1309ff3740 1 -- 172.21.15.32:0/2406637603 >> 172.21.15.32:0/2406637603 conn(0x55914f3b33c0 msgr2=0x55914f75cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:01.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.779+0000 7f1309ff3740 1 -- 172.21.15.32:0/2406637603 shutdown_connections 2024-04-30T08:32:01.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.779+0000 7f1309ff3740 1 -- 172.21.15.32:0/2406637603 wait complete. 2024-04-30T08:32:01.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.779+0000 7f12fbe00640 1 --2- 172.21.15.32:0/1630881232 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f12e00afae0 0x7f12e00b1ff0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f12ec000c00 tx=0x7f12ec031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:01.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.779+0000 7f1309ff3740 1 -- start start 2024-04-30T08:32:01.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.780+0000 7f1309ff3740 1 Processor -- start 2024-04-30T08:32:01.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.780+0000 7f1309ff3740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55914f3963b0 0x55914f76b0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:01.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.780+0000 7f1309ff3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55914f39e590 0x55914f76b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:01.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.780+0000 7f1309ff3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55914f3bf270 0x55914f772550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:01.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.780+0000 7f1309ff3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55914f763250 con 0x55914f39e590 2024-04-30T08:32:01.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.780+0000 7f1309ff3740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55914f67e9c0 con 0x55914f3963b0 2024-04-30T08:32:01.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.780+0000 7f1309ff3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55914f7633d0 con 0x55914f3bf270 2024-04-30T08:32:01.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.780+0000 7f1309ff3740 10 client.0 _ll_register_callbacks cb 0x55914f64b120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:32:01.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.780+0000 7f12fb400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55914f39e590 0x55914f76b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:01.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.781+0000 7f12fb400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55914f39e590 0x55914f76b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44706/0 (socket says 172.21.15.32:44706) 2024-04-30T08:32:01.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.781+0000 7f12fb400640 1 -- 172.21.15.32:0/332176002 learned_addr learned my addr 172.21.15.32:0/332176002 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:01.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.781+0000 7f1301000640 1 --2- 172.21.15.32:0/332176002 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55914f3963b0 0x55914f76b0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:01.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.781+0000 7f1301000640 1 --2- 172.21.15.32:0/332176002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55914f3bf270 0x55914f772550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:01.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.781+0000 7f12fb400640 1 -- 172.21.15.32:0/332176002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55914f3bf270 msgr2=0x55914f772550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:32:01.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.781+0000 7f12fb400640 1 --2- 172.21.15.32:0/332176002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55914f3bf270 0x55914f772550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.781+0000 7f12fb400640 1 -- 172.21.15.32:0/332176002 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55914f3963b0 msgr2=0x55914f76b0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:32:01.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.781+0000 7f12fb400640 1 --2- 172.21.15.32:0/332176002 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55914f3963b0 0x55914f76b0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.781+0000 7f12fb400640 1 -- 172.21.15.32:0/332176002 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55914f4a16b0 con 0x55914f39e590 2024-04-30T08:32:01.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.781+0000 7f1301000640 1 --2- 172.21.15.32:0/332176002 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55914f3963b0 0x55914f76b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:32:01.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.781+0000 7f12fb400640 1 --2- 172.21.15.32:0/332176002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55914f39e590 0x55914f76b5f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f12e4012790 tx=0x7f12e4012c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2172d3b76d887d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:01.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.781+0000 7f12ea000640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:32:01.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.781+0000 7f12ea000640 1 -- 172.21.15.32:0/332176002 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e400f920 con 0x55914f39e590 2024-04-30T08:32:01.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.782+0000 7f1309ff3740 1 -- 172.21.15.32:0/332176002 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12f4003dd0 con 0x55914f39e590 2024-04-30T08:32:01.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.782+0000 7f12ea000640 1 -- 172.21.15.32:0/332176002 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f12e400fac0 con 0x55914f39e590 2024-04-30T08:32:01.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.782+0000 7f12ea000640 1 -- 172.21.15.32:0/332176002 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f12e400fcc0 con 0x55914f39e590 2024-04-30T08:32:01.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.782+0000 7f1309ff3740 10 client.14577 fetch_fsmap learned FSMap version 17 2024-04-30T08:32:01.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.782+0000 7f1309ff3740 1 -- 172.21.15.32:0/332176002 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55914f775550 con 0x55914f39e590 2024-04-30T08:32:01.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.782+0000 7f12ea000640 1 -- 172.21.15.32:0/332176002 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12f4003dd0 con 0x55914f39e590 2024-04-30T08:32:01.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.782+0000 7f12ea000640 1 -- 172.21.15.32:0/332176002 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f12e4010430 con 0x55914f39e590 2024-04-30T08:32:01.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.782+0000 7f1309ff3740 10 client.14577 fetch_fsmap finished waiting for FSMap version 17 2024-04-30T08:32:01.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.783+0000 7f1309ff3740 10 client.14577 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-30T08:32:01.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.783+0000 7f1309ff3740 20 client.14577 populate_metadata read hostname 'smithi032' 2024-04-30T08:32:01.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.783+0000 7f1309ff3740 1 --2- 172.21.15.32:0/332176002 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55914f37de40 0x55914f778fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:01.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.783+0000 7f1309ff3740 4 client.14577 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:32:01.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.783+0000 7f1309ff3740 1 -- 172.21.15.32:0/332176002 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55914f6d33a0 con 0x55914f37de40 2024-04-30T08:32:01.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.783+0000 7f1301000640 1 --2- 172.21.15.32:0/332176002 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55914f37de40 0x55914f778fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:01.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.784+0000 7f1301000640 1 --2- 172.21.15.32:0/332176002 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55914f37de40 0x55914f778fd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db8783274d4c9197 server_cookie=141afcb71f1f082c in_seq=0 out_seq=0 2024-04-30T08:32:01.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.784+0000 7f12ea000640 10 client.14577 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:32:01.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.784+0000 7f12ea000640 1 -- 172.21.15.32:0/332176002 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55914f6d33a0 con 0x55914f37de40 2024-04-30T08:32:01.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.784+0000 7f12ea000640 10 client.14577 handle_command_reply: tid=0 2024-04-30T08:32:01.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.785+0000 7f1309ff3740 1 client.14577 shutdown 2024-04-30T08:32:01.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.785+0000 7f1309ff3740 1 -- 172.21.15.32:0/332176002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55914f39e590 msgr2=0x55914f76b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:01.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.785+0000 7f1309ff3740 1 --2- 172.21.15.32:0/332176002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55914f39e590 0x55914f76b5f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f12e4012790 tx=0x7f12e4012c50 comp rx=0 tx=0).stop 2024-04-30T08:32:01.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.785+0000 7f1309ff3740 1 -- 172.21.15.32:0/332176002 shutdown_connections 2024-04-30T08:32:01.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.785+0000 7f1309ff3740 1 --2- 172.21.15.32:0/332176002 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55914f37de40 0x55914f778fd0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.785+0000 7f1309ff3740 1 --2- 172.21.15.32:0/332176002 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55914f3bf270 0x55914f772550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.785+0000 7f1309ff3740 1 --2- 172.21.15.32:0/332176002 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55914f39e590 0x55914f76b5f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.785+0000 7f1309ff3740 1 --2- 172.21.15.32:0/332176002 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55914f3963b0 0x55914f76b0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.785+0000 7f12ea000640 0 client.14577 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:32:01.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.785+0000 7f1309ff3740 1 -- 172.21.15.32:0/332176002 >> 172.21.15.32:0/332176002 conn(0x55914f3b33c0 msgr2=0x55914f75bd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:01.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.785+0000 7f1309ff3740 1 -- 172.21.15.32:0/332176002 shutdown_connections 2024-04-30T08:32:01.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.785+0000 7f1309ff3740 1 -- 172.21.15.32:0/332176002 wait complete. 2024-04-30T08:32:01.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.785+0000 7f1309ff3740 20 client.14577 trim_cache size 0 max 16384 2024-04-30T08:32:01.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.806+0000 7f1309ff3740 1 Processor -- start 2024-04-30T08:32:01.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.807+0000 7f1309ff3740 1 -- start start 2024-04-30T08:32:01.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.807+0000 7f1309ff3740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55914f3963b0 0x55914f7614b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:01.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.807+0000 7f1309ff3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55914f39e590 0x55914f772470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:01.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.807+0000 7f1309ff3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55914f3bf270 0x55914f76b050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:01.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.807+0000 7f1309ff3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55914f777310 con 0x55914f3bf270 2024-04-30T08:32:01.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.807+0000 7f1309ff3740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55914f7779e0 con 0x55914f3963b0 2024-04-30T08:32:01.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.807+0000 7f1309ff3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55914f7780b0 con 0x55914f39e590 2024-04-30T08:32:01.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.807+0000 7f12fb400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55914f39e590 0x55914f772470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:01.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.807+0000 7f1301000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55914f3963b0 0x55914f7614b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:01.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.807+0000 7f12fb400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55914f39e590 0x55914f772470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:45028/0 (socket says 172.21.15.32:45028) 2024-04-30T08:32:01.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.807+0000 7f12fb400640 1 -- 172.21.15.32:0/1994262071 learned_addr learned my addr 172.21.15.32:0/1994262071 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:01.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.807+0000 7f1301000640 1 --2- 172.21.15.32:0/1994262071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55914f3bf270 0x55914f76b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:01.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.808+0000 7f12fb400640 1 -- 172.21.15.32:0/1994262071 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55914f3963b0 msgr2=0x55914f7614b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:32:01.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.808+0000 7f12fb400640 1 --2- 172.21.15.32:0/1994262071 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55914f3963b0 0x55914f7614b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.808+0000 7f12fb400640 1 -- 172.21.15.32:0/1994262071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55914f3bf270 msgr2=0x55914f76b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:32:01.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.808+0000 7f12fb400640 1 --2- 172.21.15.32:0/1994262071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55914f3bf270 0x55914f76b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.808+0000 7f12fb400640 1 -- 172.21.15.32:0/1994262071 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55914f769770 con 0x55914f39e590 2024-04-30T08:32:01.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.808+0000 7f1301000640 1 --2- 172.21.15.32:0/1994262071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55914f3bf270 0x55914f76b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:32:01.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.808+0000 7f12fb400640 1 --2- 172.21.15.32:0/1994262071 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55914f39e590 0x55914f772470 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f12e4010800 tx=0x7f12e4009720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=707a300689e6f0ff server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:01.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.808+0000 7f12f0c00640 1 -- 172.21.15.32:0/1994262071 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e40104f0 con 0x55914f39e590 2024-04-30T08:32:01.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.808+0000 7f12f0c00640 1 -- 172.21.15.32:0/1994262071 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12e400f920 con 0x55914f39e590 2024-04-30T08:32:01.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.809+0000 7f12f0c00640 1 -- 172.21.15.32:0/1994262071 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e4004680 con 0x55914f39e590 2024-04-30T08:32:01.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.809+0000 7f1309ff3740 1 -- 172.21.15.32:0/1994262071 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55914f39e590 msgr2=0x55914f772470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:01.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.809+0000 7f1309ff3740 1 --2- 172.21.15.32:0/1994262071 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55914f39e590 0x55914f772470 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f12e4010800 tx=0x7f12e4009720 comp rx=0 tx=0).stop 2024-04-30T08:32:01.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.809+0000 7f1309ff3740 1 -- 172.21.15.32:0/1994262071 shutdown_connections 2024-04-30T08:32:01.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.809+0000 7f1309ff3740 1 --2- 172.21.15.32:0/1994262071 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55914f3bf270 0x55914f76b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.809+0000 7f1309ff3740 1 --2- 172.21.15.32:0/1994262071 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55914f39e590 0x55914f772470 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.809+0000 7f1309ff3740 1 --2- 172.21.15.32:0/1994262071 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55914f3963b0 0x55914f7614b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.809+0000 7f1309ff3740 1 -- 172.21.15.32:0/1994262071 >> 172.21.15.32:0/1994262071 conn(0x55914f3b33c0 msgr2=0x55914f76a690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:01.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.809+0000 7f1309ff3740 1 -- 172.21.15.32:0/1994262071 shutdown_connections 2024-04-30T08:32:01.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.809+0000 7f1309ff3740 1 -- 172.21.15.32:0/1994262071 wait complete. 2024-04-30T08:32:01.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.810+0000 7f1309ff3740 1 -- start start 2024-04-30T08:32:01.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.810+0000 7f1309ff3740 1 Processor -- start 2024-04-30T08:32:01.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.811+0000 7f1309ff3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55914f3963b0 0x55914f76d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:01.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.811+0000 7f1309ff3740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55914f39e590 0x55914f76dd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:01.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.811+0000 7f1309ff3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55914f3bf270 0x55914f770c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:01.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.811+0000 7f1309ff3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55914f778e50 con 0x55914f3bf270 2024-04-30T08:32:01.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.811+0000 7f1309ff3740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55914f778780 con 0x55914f39e590 2024-04-30T08:32:01.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.811+0000 7f1309ff3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55914f779520 con 0x55914f3963b0 2024-04-30T08:32:01.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.811+0000 7f1309ff3740 10 client.0 _ll_register_callbacks cb 0x7f12e4009960 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:32:01.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.811+0000 7f1301000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55914f3bf270 0x55914f770c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:01.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.811+0000 7f1301000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55914f3bf270 0x55914f770c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44716/0 (socket says 172.21.15.32:44716) 2024-04-30T08:32:01.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.811+0000 7f1301000640 1 -- 172.21.15.32:0/2931764856 learned_addr learned my addr 172.21.15.32:0/2931764856 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:01.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.811+0000 7f12fb400640 1 --2- 172.21.15.32:0/2931764856 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55914f39e590 0x55914f76dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:01.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.811+0000 7f1301000640 1 --2- 172.21.15.32:0/2931764856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55914f3963b0 0x55914f76d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:01.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.811+0000 7f12fb400640 1 -- 172.21.15.32:0/2931764856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55914f3963b0 msgr2=0x55914f76d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:32:01.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.811+0000 7f12fb400640 1 --2- 172.21.15.32:0/2931764856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55914f3963b0 0x55914f76d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.811+0000 7f12fb400640 1 -- 172.21.15.32:0/2931764856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55914f3bf270 msgr2=0x55914f770c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:32:01.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.811+0000 7f12fb400640 1 --2- 172.21.15.32:0/2931764856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55914f3bf270 0x55914f770c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.811+0000 7f12fb400640 1 -- 172.21.15.32:0/2931764856 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55914f771250 con 0x55914f39e590 2024-04-30T08:32:01.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.812+0000 7f12fb400640 1 --2- 172.21.15.32:0/2931764856 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55914f39e590 0x55914f76dd50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f12e4010920 tx=0x7f12e4005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2133f1832d9a7e39 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:01.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.812+0000 7f12ea000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:32:01.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.812+0000 7f12ea000640 1 -- 172.21.15.32:0/2931764856 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12e4031070 con 0x55914f39e590 2024-04-30T08:32:01.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.812+0000 7f12ea000640 1 -- 172.21.15.32:0/2931764856 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f12e4023480 con 0x55914f39e590 2024-04-30T08:32:01.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.812+0000 7f1309ff3740 1 -- 172.21.15.32:0/2931764856 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12e400f920 con 0x55914f39e590 2024-04-30T08:32:01.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.812+0000 7f12ea000640 1 -- 172.21.15.32:0/2931764856 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f12e403b050 con 0x55914f39e590 2024-04-30T08:32:01.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.812+0000 7f1309ff3740 10 client.24704 fetch_fsmap learned FSMap version 17 2024-04-30T08:32:01.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.812+0000 7f1309ff3740 1 -- 172.21.15.32:0/2931764856 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x55914f7724d0 con 0x55914f39e590 2024-04-30T08:32:01.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.813+0000 7f12ea000640 1 -- 172.21.15.32:0/2931764856 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f12e400f920 con 0x55914f39e590 2024-04-30T08:32:01.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.813+0000 7f12ea000640 1 -- 172.21.15.32:0/2931764856 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f12e402b020 con 0x55914f39e590 2024-04-30T08:32:01.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.813+0000 7f1309ff3740 10 client.24704 fetch_fsmap finished waiting for FSMap version 17 2024-04-30T08:32:01.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.813+0000 7f1309ff3740 10 client.24704 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-30T08:32:01.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.813+0000 7f1309ff3740 20 client.24704 populate_metadata read hostname 'smithi032' 2024-04-30T08:32:01.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.813+0000 7f1309ff3740 1 --2- 172.21.15.32:0/2931764856 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55914f353280 0x55914f7e4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:01.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.813+0000 7f1309ff3740 4 client.24704 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-30T08:32:01.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.813+0000 7f1309ff3740 1 -- 172.21.15.32:0/2931764856 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "status"}) -- 0x55914f6cd210 con 0x55914f353280 2024-04-30T08:32:01.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.813+0000 7f1301000640 1 --2- 172.21.15.32:0/2931764856 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55914f353280 0x55914f7e4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:01.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.814+0000 7f1301000640 1 --2- 172.21.15.32:0/2931764856 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55914f353280 0x55914f7e4ef0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b35ce524623f40a server_cookie=330918cf68541221 in_seq=0 out_seq=0 2024-04-30T08:32:01.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.814+0000 7f12ea000640 10 client.24704 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:32:01.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.814+0000 7f12ea000640 1 -- 172.21.15.32:0/2931764856 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55914f6cd210 con 0x55914f353280 2024-04-30T08:32:01.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.814+0000 7f12ea000640 10 client.24704 handle_command_reply: tid=0 2024-04-30T08:32:01.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.814+0000 7f1309ff3740 1 client.24704 shutdown 2024-04-30T08:32:01.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.814+0000 7f1309ff3740 1 -- 172.21.15.32:0/2931764856 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55914f39e590 msgr2=0x55914f76dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:01.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.814+0000 7f1309ff3740 1 --2- 172.21.15.32:0/2931764856 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55914f39e590 0x55914f76dd50 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f12e4010920 tx=0x7f12e4005f90 comp rx=0 tx=0).stop 2024-04-30T08:32:01.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.814+0000 7f1309ff3740 1 -- 172.21.15.32:0/2931764856 shutdown_connections 2024-04-30T08:32:01.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.814+0000 7f1309ff3740 1 --2- 172.21.15.32:0/2931764856 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55914f353280 0x55914f7e4ef0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.814+0000 7f1309ff3740 1 --2- 172.21.15.32:0/2931764856 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55914f3bf270 0x55914f770c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.814+0000 7f1309ff3740 1 --2- 172.21.15.32:0/2931764856 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55914f39e590 0x55914f76dd50 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.814+0000 7f1309ff3740 1 --2- 172.21.15.32:0/2931764856 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55914f3963b0 0x55914f76d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.814+0000 7f12ea000640 0 client.24704 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:32:01.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.814+0000 7f1309ff3740 1 -- 172.21.15.32:0/2931764856 >> 172.21.15.32:0/2931764856 conn(0x55914f3b33c0 msgr2=0x55914f76b050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:01.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.814+0000 7f1309ff3740 1 -- 172.21.15.32:0/2931764856 shutdown_connections 2024-04-30T08:32:01.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.814+0000 7f1309ff3740 1 -- 172.21.15.32:0/2931764856 wait complete. 2024-04-30T08:32:01.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.815+0000 7f1309ff3740 20 client.24704 trim_cache size 0 max 16384 2024-04-30T08:32:01.822 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-30T08:32:01.822 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "98c0a670-06ca-11ef-bc93-c7b262605968", 2024-04-30T08:32:01.822 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-04-30T08:32:01.822 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 24590, 2024-04-30T08:32:01.823 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-04-30T08:32:01.823 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-04-30T08:32:01.823 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-04-30T08:32:01.823 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 8.5822863940000005, 2024-04-30T08:32:01.823 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 17, 2024-04-30T08:32:01.823 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 81, 2024-04-30T08:32:01.823 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 81, 2024-04-30T08:32:01.823 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 17.024920760000001 2024-04-30T08:32:01.824 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-30T08:32:01.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.815+0000 7f1301a00640 1 -- 172.21.15.32:0/1630881232 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f12e00afae0 msgr2=0x7f12e00b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:01.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.815+0000 7f1301a00640 1 --2- 172.21.15.32:0/1630881232 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f12e00afae0 0x7f12e00b1ff0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f12ec000c00 tx=0x7f12ec031400 comp rx=0 tx=0).stop 2024-04-30T08:32:01.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.815+0000 7f1301a00640 1 -- 172.21.15.32:0/1630881232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12fc105bb0 msgr2=0x7f12fc19c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:01.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.815+0000 7f1301a00640 1 --2- 172.21.15.32:0/1630881232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12fc105bb0 0x7f12fc19c330 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f12ec009230 tx=0x7f12ec003c10 comp rx=0 tx=0).stop 2024-04-30T08:32:01.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.815+0000 7f1301a00640 1 -- 172.21.15.32:0/1630881232 shutdown_connections 2024-04-30T08:32:01.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.815+0000 7f1301a00640 1 --2- 172.21.15.32:0/1630881232 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f12e00afae0 0x7f12e00b1ff0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.815+0000 7f1301a00640 1 --2- 172.21.15.32:0/1630881232 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f12fc10baf0 0x7f12fc1a3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.815+0000 7f1301a00640 1 --2- 172.21.15.32:0/1630881232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f12fc106df0 0x7f12fc19c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.815+0000 7f1301a00640 1 --2- 172.21.15.32:0/1630881232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f12fc105bb0 0x7f12fc19c330 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:01.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.815+0000 7f1301a00640 1 -- 172.21.15.32:0/1630881232 >> 172.21.15.32:0/1630881232 conn(0x7f12fc100da0 msgr2=0x7f12fc107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:01.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.816+0000 7f1301a00640 1 -- 172.21.15.32:0/1630881232 shutdown_connections 2024-04-30T08:32:01.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:01.816+0000 7f1301a00640 1 -- 172.21.15.32:0/1630881232 wait complete. 2024-04-30T08:32:01.843 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-30T08:32:01.843 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-30T08:32:01.843 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-30T08:32:02.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:01 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1458939219' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:32:02.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:01 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3339546680' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:32:02.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:01 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1458939219' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:32:02.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:01 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3339546680' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:32:02.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:01 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1458939219' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:32:02.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:01 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3339546680' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:32:02.309 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:32:03.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:02 smithi071 ceph-mon[33385]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-04-30T08:32:03.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:02 smithi189 ceph-mon[33038]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-04-30T08:32:03.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:02 smithi032 ceph-mon[28331]: pgmap v327: 129 pgs: 129 active+clean; 579 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-04-30T08:32:04.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.804+0000 7fc002c00700 1 -- 172.21.15.32:0/3283977609 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbffc106e20 msgr2=0x7fbffc1132b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:04.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.804+0000 7fc002c00700 1 --2- 172.21.15.32:0/3283977609 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbffc106e20 0x7fbffc1132b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbff400b0a0 tx=0x7fbff400b3b0 comp rx=0 tx=0).stop 2024-04-30T08:32:04.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.804+0000 7fc002c00700 1 -- 172.21.15.32:0/3283977609 shutdown_connections 2024-04-30T08:32:04.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.804+0000 7fc002c00700 1 --2- 172.21.15.32:0/3283977609 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbffc106e20 0x7fbffc1132b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:04.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.804+0000 7fc002c00700 1 --2- 172.21.15.32:0/3283977609 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbffc10c510 0x7fbffc10c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:04.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.804+0000 7fc002c00700 1 --2- 172.21.15.32:0/3283977609 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbffc1064d0 0x7fbffc1068e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:04.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.804+0000 7fc002c00700 1 -- 172.21.15.32:0/3283977609 >> 172.21.15.32:0/3283977609 conn(0x7fbffc101fd0 msgr2=0x7fbffc104420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:04.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.805+0000 7fc002c00700 1 -- 172.21.15.32:0/3283977609 shutdown_connections 2024-04-30T08:32:04.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.805+0000 7fc002c00700 1 -- 172.21.15.32:0/3283977609 wait complete. 2024-04-30T08:32:04.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.806+0000 7fc002c00700 1 Processor -- start 2024-04-30T08:32:04.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.806+0000 7fc002c00700 1 -- start start 2024-04-30T08:32:04.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.807+0000 7fc002c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbffc1064d0 0x7fbffc19cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:04.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.807+0000 7fc002c00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbffc106e20 0x7fbffc19d540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:04.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.807+0000 7fc002c00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbffc10c510 0x7fbffc1a17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:04.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.807+0000 7fc002c00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbffc116690 con 0x7fbffc106e20 2024-04-30T08:32:04.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.807+0000 7fc002c00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fbffc116510 con 0x7fbffc10c510 2024-04-30T08:32:04.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.807+0000 7fc002c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbffc116810 con 0x7fbffc1064d0 2024-04-30T08:32:04.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.807+0000 7fbffb400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbffc106e20 0x7fbffc19d540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:04.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.807+0000 7fc000c00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbffc10c510 0x7fbffc1a17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:04.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.807+0000 7fbffbe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbffc1064d0 0x7fbffc19cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:04.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.807+0000 7fbffb400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbffc106e20 0x7fbffc19d540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:44734/0 (socket says 172.21.15.32:44734) 2024-04-30T08:32:04.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.807+0000 7fc000c00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbffc10c510 0x7fbffc1a17d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:45358/0 (socket says 172.21.15.32:45358) 2024-04-30T08:32:04.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.807+0000 7fbffb400700 1 -- 172.21.15.32:0/2234419357 learned_addr learned my addr 172.21.15.32:0/2234419357 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:04.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.808+0000 7fc000c00700 1 -- 172.21.15.32:0/2234419357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbffc1064d0 msgr2=0x7fbffc19cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:04.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.808+0000 7fc000c00700 1 --2- 172.21.15.32:0/2234419357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbffc1064d0 0x7fbffc19cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:04.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.808+0000 7fc000c00700 1 -- 172.21.15.32:0/2234419357 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbffc106e20 msgr2=0x7fbffc19d540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:04.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.808+0000 7fc000c00700 1 --2- 172.21.15.32:0/2234419357 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbffc106e20 0x7fbffc19d540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:04.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.808+0000 7fc000c00700 1 -- 172.21.15.32:0/2234419357 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbffc1a1e70 con 0x7fbffc10c510 2024-04-30T08:32:04.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.808+0000 7fbffbe00700 1 --2- 172.21.15.32:0/2234419357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbffc1064d0 0x7fbffc19cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:32:04.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.808+0000 7fbffb400700 1 --2- 172.21.15.32:0/2234419357 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbffc106e20 0x7fbffc19d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:32:04.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.808+0000 7fc000c00700 1 --2- 172.21.15.32:0/2234419357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbffc10c510 0x7fbffc1a17d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbff400b070 tx=0x7fbff4004c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:04.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.809+0000 7fbff8c00700 1 -- 172.21.15.32:0/2234419357 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff4021070 con 0x7fbffc10c510 2024-04-30T08:32:04.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.809+0000 7fc002c00700 1 -- 172.21.15.32:0/2234419357 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbffc1a2100 con 0x7fbffc10c510 2024-04-30T08:32:04.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.809+0000 7fbff8c00700 1 -- 172.21.15.32:0/2234419357 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbff4007e30 con 0x7fbffc10c510 2024-04-30T08:32:04.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.809+0000 7fbff8c00700 1 -- 172.21.15.32:0/2234419357 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbff40045f0 con 0x7fbffc10c510 2024-04-30T08:32:04.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.809+0000 7fc002c00700 1 -- 172.21.15.32:0/2234419357 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbffc1a9a80 con 0x7fbffc10c510 2024-04-30T08:32:04.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.810+0000 7fc002c00700 1 -- 172.21.15.32:0/2234419357 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbffc10e8d0 con 0x7fbffc10c510 2024-04-30T08:32:04.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.812+0000 7fbff8c00700 1 -- 172.21.15.32:0/2234419357 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fbff4007980 con 0x7fbffc10c510 2024-04-30T08:32:04.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.814+0000 7fbff8c00700 1 --2- 172.21.15.32:0/2234419357 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbfe00afb80 0x7fbfe00b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:04.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.814+0000 7fbff8c00700 1 -- 172.21.15.32:0/2234419357 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7fbff400f020 con 0x7fbffc10c510 2024-04-30T08:32:04.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.814+0000 7fbffbe00700 1 --2- 172.21.15.32:0/2234419357 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbfe00afb80 0x7fbfe00b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:04.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.815+0000 7fbffbe00700 1 --2- 172.21.15.32:0/2234419357 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbfe00afb80 0x7fbfe00b2090 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbfec009680 tx=0x7fbfec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:04.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:04.816+0000 7fbff8c00700 1 -- 172.21.15.32:0/2234419357 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbff402a020 con 0x7fbffc10c510 2024-04-30T08:32:05.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:04 smithi071 ceph-mon[33385]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 292 B/s wr, 0 op/s 2024-04-30T08:32:05.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:04 smithi189 ceph-mon[33038]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 292 B/s wr, 0 op/s 2024-04-30T08:32:05.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:05.101+0000 7fc002c00700 1 -- 172.21.15.32:0/2234419357 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fbffc19e300 con 0x7fbffc10c510 2024-04-30T08:32:05.102 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:04 smithi032 ceph-mon[28331]: pgmap v328: 129 pgs: 129 active+clean; 579 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 292 B/s wr, 0 op/s 2024-04-30T08:32:05.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:05.102+0000 7fbff8c00700 1 -- 172.21.15.32:0/2234419357 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7fbff40a8b70 con 0x7fbffc10c510 2024-04-30T08:32:05.103 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-30T08:32:05.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:05.106+0000 7fc002c00700 1 -- 172.21.15.32:0/2234419357 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbfe00afb80 msgr2=0x7fbfe00b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:05.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:05.106+0000 7fc002c00700 1 --2- 172.21.15.32:0/2234419357 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbfe00afb80 0x7fbfe00b2090 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbfec009680 tx=0x7fbfec009260 comp rx=0 tx=0).stop 2024-04-30T08:32:05.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:05.106+0000 7fc002c00700 1 -- 172.21.15.32:0/2234419357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbffc10c510 msgr2=0x7fbffc1a17d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:05.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:05.106+0000 7fc002c00700 1 --2- 172.21.15.32:0/2234419357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbffc10c510 0x7fbffc1a17d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbff400b070 tx=0x7fbff4004c80 comp rx=0 tx=0).stop 2024-04-30T08:32:05.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:05.106+0000 7fc002c00700 1 -- 172.21.15.32:0/2234419357 shutdown_connections 2024-04-30T08:32:05.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:05.106+0000 7fc002c00700 1 --2- 172.21.15.32:0/2234419357 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbfe00afb80 0x7fbfe00b2090 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:05.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:05.106+0000 7fc002c00700 1 --2- 172.21.15.32:0/2234419357 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbffc10c510 0x7fbffc1a17d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:05.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:05.106+0000 7fc002c00700 1 --2- 172.21.15.32:0/2234419357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbffc1064d0 0x7fbffc19cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:05.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:05.106+0000 7fc002c00700 1 --2- 172.21.15.32:0/2234419357 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbffc106e20 0x7fbffc19d540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:05.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:05.106+0000 7fc002c00700 1 -- 172.21.15.32:0/2234419357 >> 172.21.15.32:0/2234419357 conn(0x7fbffc101fd0 msgr2=0x7fbffc104610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:05.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:05.107+0000 7fc002c00700 1 -- 172.21.15.32:0/2234419357 shutdown_connections 2024-04-30T08:32:05.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:05.107+0000 7fc002c00700 1 -- 172.21.15.32:0/2234419357 wait complete. 2024-04-30T08:32:05.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:05 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2234419357' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-30T08:32:06.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:05 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2234419357' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-30T08:32:06.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:05 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2234419357' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-30T08:32:06.057 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-04-30T08:32:06.057 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-30T08:32:06.058 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-04-30T08:32:06.058 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-04-30T08:32:06.058 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-04-30T08:32:06.058 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-04-30T08:32:06.058 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-30T08:32:06.058 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-30T08:32:06.058 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-04-30T08:32:06.058 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-30T08:32:06.058 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-04-30T08:32:06.120 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-30T08:32:06.120 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-30T08:32:06.120 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-04-30T08:32:06.120 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-04-30T08:32:06.120 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-04-30T08:32:06.120 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-04-30T08:32:06.120 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-30T08:32:06.121 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-04-30T08:32:06.121 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-04-30T08:32:06.121 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-30T08:32:06.121 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-04-30T08:32:06.121 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-04-30T08:32:06.121 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-30T08:32:06.121 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-04-30T08:32:06.121 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-04-30T08:32:06.121 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-30T08:32:06.121 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 42019sec preferred_lft 42019sec 2024-04-30T08:32:06.121 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-04-30T08:32:06.121 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-04-30T08:32:06.121 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-30T08:32:06.122 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-04-30T08:32:06.122 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-04-30T08:32:06.123 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-30T08:32:06.123 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-30T08:32:06.123 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-04-30T08:32:06.123 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link add name ceph-brx type bridge 2024-04-30T08:32:06.123 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip addr flush dev ceph-brx 2024-04-30T08:32:06.123 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set ceph-brx up 2024-04-30T08:32:06.123 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-30T08:32:06.124 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-04-30T08:32:06.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:06 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-30T08:32:06.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:06 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-30T08:32:06.642 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-30T08:32:06.643 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-30T08:32:06.725 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-04-30T08:32:06.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-30T08:32:06.730 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-04-30T08:32:06.788 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-04-30T08:32:06.789 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-30T08:32:06.789 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-30T08:32:06.789 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-30T08:32:06.789 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-30T08:32:06.789 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-30T08:32:06.789 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-04-30T08:32:06.789 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-30T08:32:06.790 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-30T08:32:06.790 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-30T08:32:06.790 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-04-30T08:32:06.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:06 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-30T08:32:06.914 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:06 smithi032 ceph-mon[28331]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-30T08:32:07.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:06 smithi071 ceph-mon[33385]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-30T08:32:07.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:06 smithi189 ceph-mon[33038]: pgmap v329: 129 pgs: 129 active+clean; 579 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-30T08:32:07.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-30T08:32:07.635 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-30T08:32:07.635 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-04-30T08:32:07.693 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-30T08:32:07.693 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list-id 2024-04-30T08:32:07.744 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-30T08:32:07.744 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-04-30T08:32:07.744 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-30T08:32:07.744 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-04-30T08:32:07.744 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-04-30T08:32:07.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:07 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-30T08:32:07.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:07 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-30T08:32:07.933 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-04-30T08:32:07.933 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-30T08:32:07.933 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-04-30T08:32:07.934 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-04-30T08:32:07.934 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-30T08:32:07.934 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-04-30T08:32:07.934 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-04-30T08:32:07.934 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-04-30T08:32:07.934 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-04-30T08:32:08.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-30T08:32:08.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-30T08:32:08.241 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-30T08:32:08.242 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-04-30T08:32:08.242 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-04-30T08:32:08.242 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set dev brx.0 master ceph-brx 2024-04-30T08:32:08.242 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-04-30T08:32:08.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:08 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-30T08:32:08.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:08 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-30T08:32:08.447 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-04-30T08:32:08.447 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-30T08:32:08.447 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-04-30T08:32:08.490 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-04-30T08:32:08.490 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-30T08:32:08.490 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-04-30T08:32:08.585 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2024-04-30T08:32:08.658 DEBUG:teuthology.orchestra.run.smithi032:> cat /proc/self/mounts | awk '{print $2}' 2024-04-30T08:32:08.712 INFO:teuthology.orchestra.run.smithi032.stdout:/proc 2024-04-30T08:32:08.713 INFO:teuthology.orchestra.run.smithi032.stdout:/sys 2024-04-30T08:32:08.713 INFO:teuthology.orchestra.run.smithi032.stdout:/dev 2024-04-30T08:32:08.713 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/kernel/security 2024-04-30T08:32:08.713 INFO:teuthology.orchestra.run.smithi032.stdout:/dev/shm 2024-04-30T08:32:08.713 INFO:teuthology.orchestra.run.smithi032.stdout:/dev/pts 2024-04-30T08:32:08.713 INFO:teuthology.orchestra.run.smithi032.stdout:/run 2024-04-30T08:32:08.713 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/fs/cgroup 2024-04-30T08:32:08.713 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/fs/pstore 2024-04-30T08:32:08.713 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/fs/bpf 2024-04-30T08:32:08.713 INFO:teuthology.orchestra.run.smithi032.stdout:/ 2024-04-30T08:32:08.713 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/fs/selinux 2024-04-30T08:32:08.714 INFO:teuthology.orchestra.run.smithi032.stdout:/proc/sys/fs/binfmt_misc 2024-04-30T08:32:08.714 INFO:teuthology.orchestra.run.smithi032.stdout:/dev/mqueue 2024-04-30T08:32:08.714 INFO:teuthology.orchestra.run.smithi032.stdout:/dev/hugepages 2024-04-30T08:32:08.714 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/kernel/debug 2024-04-30T08:32:08.714 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/kernel/tracing 2024-04-30T08:32:08.714 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/fs/fuse/connections 2024-04-30T08:32:08.714 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/kernel/config 2024-04-30T08:32:08.714 INFO:teuthology.orchestra.run.smithi032.stdout:/run/credentials/systemd-sysctl.service 2024-04-30T08:32:08.714 INFO:teuthology.orchestra.run.smithi032.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-30T08:32:08.714 INFO:teuthology.orchestra.run.smithi032.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-30T08:32:08.714 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/nfs/rpc_pipefs 2024-04-30T08:32:08.714 INFO:teuthology.orchestra.run.smithi032.stdout:/run/user/1000 2024-04-30T08:32:08.715 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/ceph 2024-04-30T08:32:08.715 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/containers/storage/overlay 2024-04-30T08:32:08.715 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/containers/storage/overlay/655dcfd70762c28e0fd55f22e9899475aa4c44a6e171d6b217b6e4c45cf4ed53/merged 2024-04-30T08:32:08.715 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/containers/storage/overlay/305ef99cd576959b8c7a33096bd1a308140c966128fcbcf15325a10b2abecca6/merged 2024-04-30T08:32:08.715 INFO:teuthology.orchestra.run.smithi032.stdout:/run/user/0 2024-04-30T08:32:08.715 INFO:teuthology.orchestra.run.smithi032.stdout:/proc/sys/fs/binfmt_misc 2024-04-30T08:32:08.715 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/containers/storage/overlay/f7cbdd9817d386562affbbdb7b6d2f49f3e3cfa6ebbbe83e6d82362e9040335b/merged 2024-04-30T08:32:08.716 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/containers/storage/overlay/186456821c6aaa2f0e77ce5d1c9818097ce5136349e14b20d9cac22a1daab496/merged 2024-04-30T08:32:08.716 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/containers/storage/overlay/313a2d52f90bf15f3ed13d29a73951681f19e53a1422fe4fec9cf39fa7969932/merged 2024-04-30T08:32:08.716 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/containers/storage/overlay/33f1f2d434abd75169f58761b6cf744f11a86f83280e9e14e73e1bb1bbe8d9dc/merged 2024-04-30T08:32:08.716 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/containers/storage/overlay/b5c232a13935efa166a37f0c9facf199401b43a530f22deb4afcc90b4a0eb92a/merged 2024-04-30T08:32:08.716 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/containers/storage/overlay/6e1c00c34cdc09db8646deace3032d23b037f1f07560cd583d5ab75f540a6de0/merged 2024-04-30T08:32:08.716 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/containers/storage/overlay/a87b50f7886eee3f513277639787ac14c794596e6271165b8037b96873500b3a/merged 2024-04-30T08:32:08.716 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/containers/storage/overlay/6e69d365f1e7755d98afe0dc54d8db09fcefb0cdc0909acc7c74b9c5ca79b1a3/merged 2024-04-30T08:32:08.716 INFO:teuthology.orchestra.run.smithi032.stdout:/run/netns 2024-04-30T08:32:08.716 INFO:teuthology.orchestra.run.smithi032.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-30T08:32:08.716 INFO:teuthology.orchestra.run.smithi032.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-30T08:32:08.718 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-30T08:32:08.718 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2024-04-30T08:32:08.772 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-04-30T08:32:08.773 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-04-30T08:32:08.778 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2024-04-30T08:32:08.847 DEBUG:teuthology.orchestra.run.smithi032:> cat /proc/self/mounts | awk '{print $2}' 2024-04-30T08:32:08.893 INFO:teuthology.orchestra.run.smithi032.stdout:/proc 2024-04-30T08:32:08.894 INFO:teuthology.orchestra.run.smithi032.stdout:/sys 2024-04-30T08:32:08.894 INFO:teuthology.orchestra.run.smithi032.stdout:/dev 2024-04-30T08:32:08.894 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/kernel/security 2024-04-30T08:32:08.894 INFO:teuthology.orchestra.run.smithi032.stdout:/dev/shm 2024-04-30T08:32:08.894 INFO:teuthology.orchestra.run.smithi032.stdout:/dev/pts 2024-04-30T08:32:08.894 INFO:teuthology.orchestra.run.smithi032.stdout:/run 2024-04-30T08:32:08.894 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/fs/cgroup 2024-04-30T08:32:08.894 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/fs/pstore 2024-04-30T08:32:08.894 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/fs/bpf 2024-04-30T08:32:08.894 INFO:teuthology.orchestra.run.smithi032.stdout:/ 2024-04-30T08:32:08.894 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/fs/selinux 2024-04-30T08:32:08.895 INFO:teuthology.orchestra.run.smithi032.stdout:/proc/sys/fs/binfmt_misc 2024-04-30T08:32:08.895 INFO:teuthology.orchestra.run.smithi032.stdout:/dev/mqueue 2024-04-30T08:32:08.895 INFO:teuthology.orchestra.run.smithi032.stdout:/dev/hugepages 2024-04-30T08:32:08.895 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/kernel/debug 2024-04-30T08:32:08.895 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/kernel/tracing 2024-04-30T08:32:08.895 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/fs/fuse/connections 2024-04-30T08:32:08.895 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/kernel/config 2024-04-30T08:32:08.895 INFO:teuthology.orchestra.run.smithi032.stdout:/run/credentials/systemd-sysctl.service 2024-04-30T08:32:08.895 INFO:teuthology.orchestra.run.smithi032.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-30T08:32:08.895 INFO:teuthology.orchestra.run.smithi032.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-30T08:32:08.895 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/nfs/rpc_pipefs 2024-04-30T08:32:08.896 INFO:teuthology.orchestra.run.smithi032.stdout:/run/user/1000 2024-04-30T08:32:08.896 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/ceph 2024-04-30T08:32:08.896 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/containers/storage/overlay 2024-04-30T08:32:08.896 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/containers/storage/overlay/655dcfd70762c28e0fd55f22e9899475aa4c44a6e171d6b217b6e4c45cf4ed53/merged 2024-04-30T08:32:08.896 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/containers/storage/overlay/305ef99cd576959b8c7a33096bd1a308140c966128fcbcf15325a10b2abecca6/merged 2024-04-30T08:32:08.896 INFO:teuthology.orchestra.run.smithi032.stdout:/run/user/0 2024-04-30T08:32:08.896 INFO:teuthology.orchestra.run.smithi032.stdout:/proc/sys/fs/binfmt_misc 2024-04-30T08:32:08.896 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/containers/storage/overlay/f7cbdd9817d386562affbbdb7b6d2f49f3e3cfa6ebbbe83e6d82362e9040335b/merged 2024-04-30T08:32:08.896 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/containers/storage/overlay/186456821c6aaa2f0e77ce5d1c9818097ce5136349e14b20d9cac22a1daab496/merged 2024-04-30T08:32:08.896 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/containers/storage/overlay/313a2d52f90bf15f3ed13d29a73951681f19e53a1422fe4fec9cf39fa7969932/merged 2024-04-30T08:32:08.896 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/containers/storage/overlay/33f1f2d434abd75169f58761b6cf744f11a86f83280e9e14e73e1bb1bbe8d9dc/merged 2024-04-30T08:32:08.896 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/containers/storage/overlay/b5c232a13935efa166a37f0c9facf199401b43a530f22deb4afcc90b4a0eb92a/merged 2024-04-30T08:32:08.897 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/containers/storage/overlay/6e1c00c34cdc09db8646deace3032d23b037f1f07560cd583d5ab75f540a6de0/merged 2024-04-30T08:32:08.897 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/containers/storage/overlay/a87b50f7886eee3f513277639787ac14c794596e6271165b8037b96873500b3a/merged 2024-04-30T08:32:08.897 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/containers/storage/overlay/6e69d365f1e7755d98afe0dc54d8db09fcefb0cdc0909acc7c74b9c5ca79b1a3/merged 2024-04-30T08:32:08.897 INFO:teuthology.orchestra.run.smithi032.stdout:/run/netns 2024-04-30T08:32:08.897 INFO:teuthology.orchestra.run.smithi032.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-30T08:32:08.897 INFO:teuthology.orchestra.run.smithi032.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-30T08:32:08.899 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-30T08:32:08.899 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2024-04-30T08:32:08.940 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:08 smithi032 ceph-mon[28331]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 205 B/s wr, 0 op/s 2024-04-30T08:32:09.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:08 smithi071 ceph-mon[33385]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 205 B/s wr, 0 op/s 2024-04-30T08:32:09.036 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi032.stderr:2024-04-30T08:32:09.036+0000 7f65c631b480 -1 init, newargv = 0x7f65a8004d20 newargc=15 2024-04-30T08:32:09.037 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi032.stderr:ceph-fuse[72631]: starting ceph client 2024-04-30T08:32:09.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:08 smithi189 ceph-mon[33038]: pgmap v330: 129 pgs: 129 active+clean; 579 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 205 B/s wr, 0 op/s 2024-04-30T08:32:09.051 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi032.stderr:ceph-fuse[72631]: starting fuse 2024-04-30T08:32:10.041 DEBUG:teuthology.orchestra.run.smithi032:> sudo modprobe fuse 2024-04-30T08:32:10.135 DEBUG:teuthology.orchestra.run.smithi032:> cat /proc/self/mounts | awk '{print $2}' 2024-04-30T08:32:10.191 INFO:teuthology.orchestra.run.smithi032.stdout:/proc 2024-04-30T08:32:10.191 INFO:teuthology.orchestra.run.smithi032.stdout:/sys 2024-04-30T08:32:10.192 INFO:teuthology.orchestra.run.smithi032.stdout:/dev 2024-04-30T08:32:10.192 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/kernel/security 2024-04-30T08:32:10.192 INFO:teuthology.orchestra.run.smithi032.stdout:/dev/shm 2024-04-30T08:32:10.192 INFO:teuthology.orchestra.run.smithi032.stdout:/dev/pts 2024-04-30T08:32:10.192 INFO:teuthology.orchestra.run.smithi032.stdout:/run 2024-04-30T08:32:10.192 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/fs/cgroup 2024-04-30T08:32:10.192 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/fs/pstore 2024-04-30T08:32:10.192 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/fs/bpf 2024-04-30T08:32:10.192 INFO:teuthology.orchestra.run.smithi032.stdout:/ 2024-04-30T08:32:10.192 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/fs/selinux 2024-04-30T08:32:10.192 INFO:teuthology.orchestra.run.smithi032.stdout:/proc/sys/fs/binfmt_misc 2024-04-30T08:32:10.192 INFO:teuthology.orchestra.run.smithi032.stdout:/dev/mqueue 2024-04-30T08:32:10.192 INFO:teuthology.orchestra.run.smithi032.stdout:/dev/hugepages 2024-04-30T08:32:10.192 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/kernel/debug 2024-04-30T08:32:10.193 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/kernel/tracing 2024-04-30T08:32:10.193 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/fs/fuse/connections 2024-04-30T08:32:10.193 INFO:teuthology.orchestra.run.smithi032.stdout:/sys/kernel/config 2024-04-30T08:32:10.193 INFO:teuthology.orchestra.run.smithi032.stdout:/run/credentials/systemd-sysctl.service 2024-04-30T08:32:10.193 INFO:teuthology.orchestra.run.smithi032.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-04-30T08:32:10.193 INFO:teuthology.orchestra.run.smithi032.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-04-30T08:32:10.193 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/nfs/rpc_pipefs 2024-04-30T08:32:10.193 INFO:teuthology.orchestra.run.smithi032.stdout:/run/user/1000 2024-04-30T08:32:10.193 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/ceph 2024-04-30T08:32:10.193 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/containers/storage/overlay 2024-04-30T08:32:10.193 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/containers/storage/overlay/655dcfd70762c28e0fd55f22e9899475aa4c44a6e171d6b217b6e4c45cf4ed53/merged 2024-04-30T08:32:10.193 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/containers/storage/overlay/305ef99cd576959b8c7a33096bd1a308140c966128fcbcf15325a10b2abecca6/merged 2024-04-30T08:32:10.193 INFO:teuthology.orchestra.run.smithi032.stdout:/run/user/0 2024-04-30T08:32:10.193 INFO:teuthology.orchestra.run.smithi032.stdout:/proc/sys/fs/binfmt_misc 2024-04-30T08:32:10.194 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/containers/storage/overlay/f7cbdd9817d386562affbbdb7b6d2f49f3e3cfa6ebbbe83e6d82362e9040335b/merged 2024-04-30T08:32:10.194 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/containers/storage/overlay/186456821c6aaa2f0e77ce5d1c9818097ce5136349e14b20d9cac22a1daab496/merged 2024-04-30T08:32:10.194 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/containers/storage/overlay/313a2d52f90bf15f3ed13d29a73951681f19e53a1422fe4fec9cf39fa7969932/merged 2024-04-30T08:32:10.194 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/containers/storage/overlay/33f1f2d434abd75169f58761b6cf744f11a86f83280e9e14e73e1bb1bbe8d9dc/merged 2024-04-30T08:32:10.194 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/containers/storage/overlay/b5c232a13935efa166a37f0c9facf199401b43a530f22deb4afcc90b4a0eb92a/merged 2024-04-30T08:32:10.194 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/containers/storage/overlay/6e1c00c34cdc09db8646deace3032d23b037f1f07560cd583d5ab75f540a6de0/merged 2024-04-30T08:32:10.194 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/containers/storage/overlay/a87b50f7886eee3f513277639787ac14c794596e6271165b8037b96873500b3a/merged 2024-04-30T08:32:10.194 INFO:teuthology.orchestra.run.smithi032.stdout:/var/lib/containers/storage/overlay/6e69d365f1e7755d98afe0dc54d8db09fcefb0cdc0909acc7c74b9c5ca79b1a3/merged 2024-04-30T08:32:10.194 INFO:teuthology.orchestra.run.smithi032.stdout:/run/netns 2024-04-30T08:32:10.194 INFO:teuthology.orchestra.run.smithi032.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-30T08:32:10.194 INFO:teuthology.orchestra.run.smithi032.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-30T08:32:10.194 INFO:teuthology.orchestra.run.smithi032.stdout:/home/ubuntu/cephtest/mnt.admin 2024-04-30T08:32:10.195 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-30T08:32:10.195 DEBUG:teuthology.orchestra.run.smithi032:> ls /sys/fs/fuse/connections 2024-04-30T08:32:10.250 INFO:teuthology.orchestra.run.smithi032.stdout:114 2024-04-30T08:32:10.250 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [114] 2024-04-30T08:32:10.250 DEBUG:teuthology.orchestra.run.smithi032:> sudo stdin-killer -- python3 -c ' 2024-04-30T08:32:10.250 DEBUG:teuthology.orchestra.run.smithi032:> import glob 2024-04-30T08:32:10.250 DEBUG:teuthology.orchestra.run.smithi032:> import re 2024-04-30T08:32:10.251 DEBUG:teuthology.orchestra.run.smithi032:> import os 2024-04-30T08:32:10.251 DEBUG:teuthology.orchestra.run.smithi032:> import subprocess 2024-04-30T08:32:10.251 DEBUG:teuthology.orchestra.run.smithi032:> 2024-04-30T08:32:10.251 DEBUG:teuthology.orchestra.run.smithi032:> def _find_admin_socket(client_name): 2024-04-30T08:32:10.251 DEBUG:teuthology.orchestra.run.smithi032:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-04-30T08:32:10.251 DEBUG:teuthology.orchestra.run.smithi032:> files = glob.glob(asok_path) 2024-04-30T08:32:10.251 DEBUG:teuthology.orchestra.run.smithi032:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-04-30T08:32:10.251 DEBUG:teuthology.orchestra.run.smithi032:> 2024-04-30T08:32:10.251 DEBUG:teuthology.orchestra.run.smithi032:> # Given a non-glob path, it better be there 2024-04-30T08:32:10.251 DEBUG:teuthology.orchestra.run.smithi032:> if "*" not in asok_path: 2024-04-30T08:32:10.251 DEBUG:teuthology.orchestra.run.smithi032:> assert(len(files) == 1) 2024-04-30T08:32:10.251 DEBUG:teuthology.orchestra.run.smithi032:> return files[0] 2024-04-30T08:32:10.251 DEBUG:teuthology.orchestra.run.smithi032:> 2024-04-30T08:32:10.251 DEBUG:teuthology.orchestra.run.smithi032:> for f in files: 2024-04-30T08:32:10.251 DEBUG:teuthology.orchestra.run.smithi032:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-04-30T08:32:10.252 DEBUG:teuthology.orchestra.run.smithi032:> if os.path.exists("/proc/{0}".format(pid)): 2024-04-30T08:32:10.252 DEBUG:teuthology.orchestra.run.smithi032:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-04-30T08:32:10.252 DEBUG:teuthology.orchestra.run.smithi032:> contents = proc_f.read() 2024-04-30T08:32:10.252 DEBUG:teuthology.orchestra.run.smithi032:> if mountpoint in contents: 2024-04-30T08:32:10.252 DEBUG:teuthology.orchestra.run.smithi032:> return f 2024-04-30T08:32:10.252 DEBUG:teuthology.orchestra.run.smithi032:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-04-30T08:32:10.252 DEBUG:teuthology.orchestra.run.smithi032:> 2024-04-30T08:32:10.252 DEBUG:teuthology.orchestra.run.smithi032:> print(_find_admin_socket("client.admin")) 2024-04-30T08:32:10.252 DEBUG:teuthology.orchestra.run.smithi032:> ' 2024-04-30T08:32:10.422 INFO:teuthology.orchestra.run.smithi032.stdout:/var/run/ceph/ceph-client.admin.72631.asok 2024-04-30T08:32:10.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:10 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-30T08:32:10.441 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.72631.asok 2024-04-30T08:32:10.441 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-30T08:32:10.441 DEBUG:teuthology.orchestra.run.smithi032:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.72631.asok status 2024-04-30T08:32:10.761 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-30T08:32:10.761 INFO:teuthology.orchestra.run.smithi032.stdout: "metadata": { 2024-04-30T08:32:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_sha1": "8feaf7d9864e980041b042fd5d964ac18adfd20f", 2024-04-30T08:32:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_version": "ceph version 19.0.0-3348-g8feaf7d9 (8feaf7d9864e980041b042fd5d964ac18adfd20f) squid (dev)", 2024-04-30T08:32:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: "entity_id": "admin", 2024-04-30T08:32:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: "hostname": "smithi032", 2024-04-30T08:32:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-04-30T08:32:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: "pid": "72631", 2024-04-30T08:32:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: "root": "/" 2024-04-30T08:32:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:32:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_count": 0, 2024-04-30T08:32:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: "dentry_pinned_count": 0, 2024-04-30T08:32:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 24716, 2024-04-30T08:32:10.762 INFO:teuthology.orchestra.run.smithi032.stdout: "inst": { 2024-04-30T08:32:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: "name": { 2024-04-30T08:32:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "client", 2024-04-30T08:32:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: "num": 24716 2024-04-30T08:32:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:32:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2024-04-30T08:32:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-30T08:32:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2024-04-30T08:32:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1508818458 2024-04-30T08:32:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:32:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:32:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": { 2024-04-30T08:32:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-30T08:32:10.763 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2024-04-30T08:32:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1508818458 2024-04-30T08:32:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:32:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: "inst_str": "client.24716 172.21.15.32:0/1508818458", 2024-04-30T08:32:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: "addr_str": "172.21.15.32:0/1508818458", 2024-04-30T08:32:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: "inode_count": 1, 2024-04-30T08:32:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_epoch": 17, 2024-04-30T08:32:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch": 81, 2024-04-30T08:32:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_epoch_barrier": 0, 2024-04-30T08:32:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: "blocklisted": false, 2024-04-30T08:32:10.764 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs" 2024-04-30T08:32:10.764 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-30T08:32:10.784 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-30T08:32:10.784 DEBUG:teuthology.orchestra.run.smithi032:> stat --file-system '--printf=%T 2024-04-30T08:32:10.785 DEBUG:teuthology.orchestra.run.smithi032:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-04-30T08:32:10.838 INFO:teuthology.orchestra.run.smithi032.stdout:fuseblk 2024-04-30T08:32:10.838 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-04-30T08:32:10.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-30T08:32:10.838 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-04-30T08:32:10.932 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-04-30T08:32:11.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:10 smithi071 ceph-mon[33385]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 186 B/s wr, 0 op/s 2024-04-30T08:32:11.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-30T08:32:11.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:10 smithi189 ceph-mon[33038]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 186 B/s wr, 0 op/s 2024-04-30T08:32:11.078 INFO:teuthology.orchestra.run.smithi032.stdout:# file: . 2024-04-30T08:32:11.079 INFO:teuthology.orchestra.run.smithi032.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-04-30T08:32:11.079 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:32:11.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-30T08:32:11.089 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-30T08:32:11.089 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-30T08:32:11.143 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-30T08:32:11.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:10 smithi032 ceph-mon[28331]: pgmap v331: 129 pgs: 129 active+clean; 579 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 186 B/s wr, 0 op/s 2024-04-30T08:32:11.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.430+0000 7effc2800640 1 -- 172.21.15.32:0/2837129890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effbc1100f0 msgr2=0x7effbc112510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:11.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.430+0000 7effc2800640 1 --2- 172.21.15.32:0/2837129890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effbc1100f0 0x7effbc112510 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7effb4009be0 tx=0x7effb402a6c0 comp rx=0 tx=0).stop 2024-04-30T08:32:11.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.430+0000 7effc2800640 1 -- 172.21.15.32:0/2837129890 shutdown_connections 2024-04-30T08:32:11.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.430+0000 7effc2800640 1 --2- 172.21.15.32:0/2837129890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effbc1100f0 0x7effbc112510 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:11.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.430+0000 7effc2800640 1 --2- 172.21.15.32:0/2837129890 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7effbc075960 0x7effbc10fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:11.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.430+0000 7effc2800640 1 --2- 172.21.15.32:0/2837129890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effbc075010 0x7effbc075420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:11.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.430+0000 7effc2800640 1 -- 172.21.15.32:0/2837129890 >> 172.21.15.32:0/2837129890 conn(0x7effbc0feba0 msgr2=0x7effbc101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:11.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.430+0000 7effc2800640 1 -- 172.21.15.32:0/2837129890 shutdown_connections 2024-04-30T08:32:11.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.430+0000 7effc2800640 1 -- 172.21.15.32:0/2837129890 wait complete. 2024-04-30T08:32:11.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.431+0000 7effc2800640 1 Processor -- start 2024-04-30T08:32:11.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.432+0000 7effc2800640 1 -- start start 2024-04-30T08:32:11.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.432+0000 7effc2800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7effbc075010 0x7effbc1a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:11.434 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.432+0000 7effc2800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effbc075960 0x7effbc1a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:11.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.432+0000 7effc2800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effbc1100f0 0x7effbc1a7b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:11.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.432+0000 7effc2800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7effbc115ad0 con 0x7effbc1100f0 2024-04-30T08:32:11.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.432+0000 7effc2800640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7effbc115950 con 0x7effbc075010 2024-04-30T08:32:11.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.432+0000 7effc2800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7effbc115c50 con 0x7effbc075960 2024-04-30T08:32:11.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.432+0000 7effc1400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7effbc075010 0x7effbc1a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:11.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.432+0000 7effc1400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7effbc075010 0x7effbc1a0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:38556/0 (socket says 172.21.15.32:38556) 2024-04-30T08:32:11.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.432+0000 7effc1400640 1 -- 172.21.15.32:0/2810045377 learned_addr learned my addr 172.21.15.32:0/2810045377 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:11.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.432+0000 7effc0a00640 1 --2- 172.21.15.32:0/2810045377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effbc075960 0x7effbc1a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:11.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.432+0000 7effc1e00640 1 --2- 172.21.15.32:0/2810045377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effbc1100f0 0x7effbc1a7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:11.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.433+0000 7effc0a00640 1 -- 172.21.15.32:0/2810045377 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7effbc075010 msgr2=0x7effbc1a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:11.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.433+0000 7effc0a00640 1 --2- 172.21.15.32:0/2810045377 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7effbc075010 0x7effbc1a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:11.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.433+0000 7effc0a00640 1 -- 172.21.15.32:0/2810045377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effbc1100f0 msgr2=0x7effbc1a7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:11.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.433+0000 7effc0a00640 1 --2- 172.21.15.32:0/2810045377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effbc1100f0 0x7effbc1a7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:11.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.433+0000 7effc0a00640 1 -- 172.21.15.32:0/2810045377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effbc1a81a0 con 0x7effbc075960 2024-04-30T08:32:11.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.433+0000 7effc0a00640 1 --2- 172.21.15.32:0/2810045377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effbc075960 0x7effbc1a0ba0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7effa8005fd0 tx=0x7effa800ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:11.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.433+0000 7effba000640 1 -- 172.21.15.32:0/2810045377 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa800cd00 con 0x7effbc075960 2024-04-30T08:32:11.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.433+0000 7effc2800640 1 -- 172.21.15.32:0/2810045377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effbc1a8490 con 0x7effbc075960 2024-04-30T08:32:11.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.433+0000 7effba000640 1 -- 172.21.15.32:0/2810045377 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7effa80035f0 con 0x7effbc075960 2024-04-30T08:32:11.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.434+0000 7effba000640 1 -- 172.21.15.32:0/2810045377 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa8010670 con 0x7effbc075960 2024-04-30T08:32:11.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.434+0000 7effc2800640 1 -- 172.21.15.32:0/2810045377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effbc1a89e0 con 0x7effbc075960 2024-04-30T08:32:11.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.435+0000 7effc2800640 1 -- 172.21.15.32:0/2810045377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effbc105d80 con 0x7effbc075960 2024-04-30T08:32:11.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.437+0000 7effba000640 1 -- 172.21.15.32:0/2810045377 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7effa80037a0 con 0x7effbc075960 2024-04-30T08:32:11.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.439+0000 7effba000640 1 --2- 172.21.15.32:0/2810045377 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7effa40afae0 0x7effa40b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:11.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.439+0000 7effba000640 1 -- 172.21.15.32:0/2810045377 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7effa80df150 con 0x7effbc075960 2024-04-30T08:32:11.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.439+0000 7effc1400640 1 --2- 172.21.15.32:0/2810045377 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7effa40afae0 0x7effa40b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:11.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.440+0000 7effc1400640 1 --2- 172.21.15.32:0/2810045377 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7effa40afae0 0x7effa40b1ff0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7effb00058c0 tx=0x7effb0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:11.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.441+0000 7effba000640 1 -- 172.21.15.32:0/2810045377 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7effa80a7b20 con 0x7effbc075960 2024-04-30T08:32:11.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.652+0000 7effc2800640 1 -- 172.21.15.32:0/2810045377 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7effbc063820 con 0x7effbc075960 2024-04-30T08:32:11.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.654+0000 7effba000640 1 -- 172.21.15.32:0/2810045377 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+24053 (secure 0 0 0) 0x7effa8009510 con 0x7effbc075960 2024-04-30T08:32:11.655 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:32:11.655 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":81,"fsid":"98c0a670-06ca-11ef-bc93-c7b262605968","created":"2024-04-30T08:22:11.122875+0000","modified":"2024-04-30T08:31:58.298969+0000","last_up_change":"2024-04-30T08:30:50.350439+0000","last_in_change":"2024-04-30T08:30:30.210620+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-30T08:26:19.728331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-30T08:31:49.800542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-30T08:31:51.211320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-30T08:31:55.716625+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a7e060c8-1cbb-418f-9a03-06693f6b3f52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6803","nonce":2624700402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6805","nonce":2624700402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6809","nonce":2624700402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6807","nonce":2624700402}]},"public_addr":"172.21.15.32:6803/2624700402","cluster_addr":"172.21.15.32:6805/2624700402","heartbeat_back_addr":"172.21.15.32:6809/2624700402","heartbeat_front_addr":"172.21.15.32:6807/2624700402","state":["exists","up"]},{"osd":1,"uuid":"72ef2fb1-e5ca-48ef-aa3a-4796a64b922f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6801","nonce":444649711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6803","nonce":444649711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6807","nonce":444649711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6805","nonce":444649711}]},"public_addr":"172.21.15.71:6801/444649711","cluster_addr":"172.21.15.71:6803/444649711","heartbeat_back_addr":"172.21.15.71:6807/444649711","heartbeat_front_addr":"172.21.15.71:6805/444649711","state":["exists","up"]},{"osd":2,"uuid":"704e5e8a-935e-47c7-ae28-27119876ac09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6801","nonce":3255528671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6803","nonce":3255528671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6807","nonce":3255528671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6805","nonce":3255528671}]},"public_addr":"172.21.15.189:6801/3255528671","cluster_addr":"172.21.15.189:6803/3255528671","heartbeat_back_addr":"172.21.15.189:6807/3255528671","heartbeat_front_addr":"172.21.15.189:6805/3255528671","state":["exists","up"]},{"osd":3,"uuid":"575004f7-6559-4a62-863e-e33078b0eebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6811","nonce":1384702112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6813","nonce":1384702112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6817","nonce":1384702112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6815","nonce":1384702112}]},"public_addr":"172.21.15.32:6811/1384702112","cluster_addr":"172.21.15.32:6813/1384702112","heartbeat_back_addr":"172.21.15.32:6817/1384702112","heartbeat_front_addr":"172.21.15.32:6815/1384702112","state":["exists","up"]},{"osd":4,"uuid":"e1c1e894-074d-4c78-89ef-9e5a114268bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6809","nonce":4250534574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6811","nonce":4250534574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6815","nonce":4250534574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6813","nonce":4250534574}]},"public_addr":"172.21.15.71:6809/4250534574","cluster_addr":"172.21.15.71:6811/4250534574","heartbeat_back_addr":"172.21.15.71:6815/4250534574","heartbeat_front_addr":"172.21.15.71:6813/4250534574","state":["exists","up"]},{"osd":5,"uuid":"12c022d4-fac6-40c4-ae7a-691571bee5d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6809","nonce":4245346198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6811","nonce":4245346198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6815","nonce":4245346198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6813","nonce":4245346198}]},"public_addr":"172.21.15.189:6809/4245346198","cluster_addr":"172.21.15.189:6811/4245346198","heartbeat_back_addr":"172.21.15.189:6815/4245346198","heartbeat_front_addr":"172.21.15.189:6813/4245346198","state":["exists","up"]},{"osd":6,"uuid":"b413f8ee-3d4a-415a-a2ba-d68437b7330d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6819","nonce":1910389190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6821","nonce":1910389190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6825","nonce":1910389190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6823","nonce":1910389190}]},"public_addr":"172.21.15.32:6819/1910389190","cluster_addr":"172.21.15.32:6821/1910389190","heartbeat_back_addr":"172.21.15.32:6825/1910389190","heartbeat_front_addr":"172.21.15.32:6823/1910389190","state":["exists","up"]},{"osd":7,"uuid":"aac64fb3-fc13-4126-b842-7b06ad5ce238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6817","nonce":692607380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6819","nonce":692607380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6823","nonce":692607380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6821","nonce":692607380}]},"public_addr":"172.21.15.71:6817/692607380","cluster_addr":"172.21.15.71:6819/692607380","heartbeat_back_addr":"172.21.15.71:6823/692607380","heartbeat_front_addr":"172.21.15.71:6821/692607380","state":["exists","up"]},{"osd":8,"uuid":"cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6817","nonce":3973697069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6819","nonce":3973697069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6823","nonce":3973697069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6821","nonce":3973697069}]},"public_addr":"172.21.15.189:6817/3973697069","cluster_addr":"172.21.15.189:6819/3973697069","heartbeat_back_addr":"172.21.15.189:6823/3973697069","heartbeat_front_addr":"172.21.15.189:6821/3973697069","state":["exists","up"]},{"osd":9,"uuid":"ef4a1645-ae6e-45c3-82ad-161510e62aa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6827","nonce":46066685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6829","nonce":46066685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6833","nonce":46066685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6831","nonce":46066685}]},"public_addr":"172.21.15.32:6827/46066685","cluster_addr":"172.21.15.32:6829/46066685","heartbeat_back_addr":"172.21.15.32:6833/46066685","heartbeat_front_addr":"172.21.15.32:6831/46066685","state":["exists","up"]},{"osd":10,"uuid":"25b26a38-a644-409f-b941-59efdf04a3ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6825","nonce":1886290293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6827","nonce":1886290293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6831","nonce":1886290293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6829","nonce":1886290293}]},"public_addr":"172.21.15.71:6825/1886290293","cluster_addr":"172.21.15.71:6827/1886290293","heartbeat_back_addr":"172.21.15.71:6831/1886290293","heartbeat_front_addr":"172.21.15.71:6829/1886290293","state":["exists","up"]},{"osd":11,"uuid":"ab96b3ff-3688-455b-af64-60226c5bfbff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6825","nonce":850185001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6827","nonce":850185001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6831","nonce":850185001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6829","nonce":850185001}]},"public_addr":"172.21.15.189:6825/850185001","cluster_addr":"172.21.15.189:6827/850185001","heartbeat_back_addr":"172.21.15.189:6831/850185001","heartbeat_front_addr":"172.21.15.189:6829/850185001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:27.220308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:53.230783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:17.190642+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:52.603799+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:20.568671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:46.158363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:22.706933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:50.665173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:16.602410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:53.814698+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:22.435189+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:48.651949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1266897222":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/2915689533":"2024-05-01T08:23:29.526664+0000","172.21.15.32:6801/4019351306":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/425836193":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/1134075722":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6801/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/21059253":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/3245550894":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/4019351306":"2024-05-01T08:23:29.526664+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-30T08:32:11.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.657+0000 7effc2800640 1 -- 172.21.15.32:0/2810045377 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7effa40afae0 msgr2=0x7effa40b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:11.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.657+0000 7effc2800640 1 --2- 172.21.15.32:0/2810045377 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7effa40afae0 0x7effa40b1ff0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7effb00058c0 tx=0x7effb0005f90 comp rx=0 tx=0).stop 2024-04-30T08:32:11.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.657+0000 7effc2800640 1 -- 172.21.15.32:0/2810045377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effbc075960 msgr2=0x7effbc1a0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:11.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.657+0000 7effc2800640 1 --2- 172.21.15.32:0/2810045377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effbc075960 0x7effbc1a0ba0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7effa8005fd0 tx=0x7effa800ee50 comp rx=0 tx=0).stop 2024-04-30T08:32:11.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.657+0000 7effc2800640 1 -- 172.21.15.32:0/2810045377 shutdown_connections 2024-04-30T08:32:11.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.657+0000 7effc2800640 1 --2- 172.21.15.32:0/2810045377 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7effa40afae0 0x7effa40b1ff0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:11.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.657+0000 7effc2800640 1 --2- 172.21.15.32:0/2810045377 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7effbc1100f0 0x7effbc1a7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:11.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.657+0000 7effc2800640 1 --2- 172.21.15.32:0/2810045377 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7effbc075960 0x7effbc1a0ba0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:11.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.657+0000 7effc2800640 1 --2- 172.21.15.32:0/2810045377 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7effbc075010 0x7effbc1a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:11.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.657+0000 7effc2800640 1 -- 172.21.15.32:0/2810045377 >> 172.21.15.32:0/2810045377 conn(0x7effbc0feba0 msgr2=0x7effbc102f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:11.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.657+0000 7effc2800640 1 -- 172.21.15.32:0/2810045377 shutdown_connections 2024-04-30T08:32:11.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:11.657+0000 7effc2800640 1 -- 172.21.15.32:0/2810045377 wait complete. 2024-04-30T08:32:11.687 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-30T08:26:19.728331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-30T08:31:49.800542+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-30T08:31:51.211320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-30T08:31:55.716625+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-30T08:32:11.688 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-30T08:32:11.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:11 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2810045377' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:32:12.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.015+0000 7f03bee00640 1 -- 172.21.15.32:0/3148297433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03b8104bd0 msgr2=0x7f03b8104fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:12.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.015+0000 7f03bee00640 1 --2- 172.21.15.32:0/3148297433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03b8104bd0 0x7f03b8104fe0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f03a4009900 tx=0x7f03a402a410 comp rx=0 tx=0).stop 2024-04-30T08:32:12.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.016+0000 7f03bee00640 1 -- 172.21.15.32:0/3148297433 shutdown_connections 2024-04-30T08:32:12.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.016+0000 7f03bee00640 1 --2- 172.21.15.32:0/3148297433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03b81098d0 0x7f03b810bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:12.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.016+0000 7f03bee00640 1 --2- 172.21.15.32:0/3148297433 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03b8104bd0 0x7f03b8104fe0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:12.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.016+0000 7f03bee00640 1 --2- 172.21.15.32:0/3148297433 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03b8103990 0x7f03b8104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:12.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.016+0000 7f03bee00640 1 -- 172.21.15.32:0/3148297433 >> 172.21.15.32:0/3148297433 conn(0x7f03b80feba0 msgr2=0x7f03b8101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:12.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.016+0000 7f03bee00640 1 -- 172.21.15.32:0/3148297433 shutdown_connections 2024-04-30T08:32:12.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.016+0000 7f03bee00640 1 -- 172.21.15.32:0/3148297433 wait complete. 2024-04-30T08:32:12.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.017+0000 7f03bee00640 1 Processor -- start 2024-04-30T08:32:12.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.017+0000 7f03bee00640 1 -- start start 2024-04-30T08:32:12.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.017+0000 7f03bee00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03b8103990 0x7f03b819c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:12.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.017+0000 7f03bee00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03b8104bd0 0x7f03b819c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:12.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.017+0000 7f03bee00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03b81098d0 0x7f03b81a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:12.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.017+0000 7f03bee00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f03b810f510 con 0x7f03b8103990 2024-04-30T08:32:12.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.017+0000 7f03bee00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f03b810f390 con 0x7f03b8104bd0 2024-04-30T08:32:12.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.017+0000 7f03bee00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f03b810f690 con 0x7f03b81098d0 2024-04-30T08:32:12.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.018+0000 7f03bda00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03b8103990 0x7f03b819c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:12.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.018+0000 7f03bd000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03b8104bd0 0x7f03b819c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:12.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.018+0000 7f03bda00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03b8103990 0x7f03b819c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42430/0 (socket says 172.21.15.32:42430) 2024-04-30T08:32:12.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.018+0000 7f03bda00640 1 -- 172.21.15.32:0/2540298053 learned_addr learned my addr 172.21.15.32:0/2540298053 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:12.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.018+0000 7f03be400640 1 --2- 172.21.15.32:0/2540298053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03b81098d0 0x7f03b81a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:12.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.018+0000 7f03bd000640 1 -- 172.21.15.32:0/2540298053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03b81098d0 msgr2=0x7f03b81a3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:12.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.018+0000 7f03bd000640 1 --2- 172.21.15.32:0/2540298053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03b81098d0 0x7f03b81a3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:12.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.018+0000 7f03bd000640 1 -- 172.21.15.32:0/2540298053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03b8103990 msgr2=0x7f03b819c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:12.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.018+0000 7f03bd000640 1 --2- 172.21.15.32:0/2540298053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03b8103990 0x7f03b819c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:12.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.018+0000 7f03bd000640 1 -- 172.21.15.32:0/2540298053 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03b81a3de0 con 0x7f03b8104bd0 2024-04-30T08:32:12.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.019+0000 7f03bd000640 1 --2- 172.21.15.32:0/2540298053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03b8104bd0 0x7f03b819c7e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f03a402aa00 tx=0x7f03a4003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:12.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.019+0000 7f03b6000640 1 -- 172.21.15.32:0/2540298053 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a403f070 con 0x7f03b8104bd0 2024-04-30T08:32:12.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.019+0000 7f03bee00640 1 -- 172.21.15.32:0/2540298053 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03b81a4070 con 0x7f03b8104bd0 2024-04-30T08:32:12.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.019+0000 7f03b6000640 1 -- 172.21.15.32:0/2540298053 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f03a4031c50 con 0x7f03b8104bd0 2024-04-30T08:32:12.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.019+0000 7f03b6000640 1 -- 172.21.15.32:0/2540298053 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a4035030 con 0x7f03b8104bd0 2024-04-30T08:32:12.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.019+0000 7f03bee00640 1 -- 172.21.15.32:0/2540298053 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03b81a4530 con 0x7f03b8104bd0 2024-04-30T08:32:12.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.020+0000 7f03bee00640 1 -- 172.21.15.32:0/2540298053 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03b81a4790 con 0x7f03b8104bd0 2024-04-30T08:32:12.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.023+0000 7f03b6000640 1 -- 172.21.15.32:0/2540298053 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f03a4042070 con 0x7f03b8104bd0 2024-04-30T08:32:12.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.024+0000 7f03b6000640 1 --2- 172.21.15.32:0/2540298053 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f03a00afae0 0x7f03a00b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:12.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.025+0000 7f03b6000640 1 -- 172.21.15.32:0/2540298053 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f03a40ffb60 con 0x7f03b8104bd0 2024-04-30T08:32:12.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.025+0000 7f03bda00640 1 --2- 172.21.15.32:0/2540298053 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f03a00afae0 0x7f03a00b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:12.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.025+0000 7f03bda00640 1 --2- 172.21.15.32:0/2540298053 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f03a00afae0 0x7f03a00b1ff0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f03ac009c30 tx=0x7f03ac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:12.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.027+0000 7f03b6000640 1 -- 172.21.15.32:0/2540298053 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f03a40c7e50 con 0x7f03b8104bd0 2024-04-30T08:32:12.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:11 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2810045377' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:32:12.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.234+0000 7f03bee00640 1 -- 172.21.15.32:0/2540298053 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f03b81a4960 con 0x7f03b8104bd0 2024-04-30T08:32:12.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.235+0000 7f03b6000640 1 -- 172.21.15.32:0/2540298053 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f03a40cc150 con 0x7f03b8104bd0 2024-04-30T08:32:12.235 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-30T08:32:12.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.238+0000 7f03bee00640 1 -- 172.21.15.32:0/2540298053 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f03a00afae0 msgr2=0x7f03a00b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:12.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.238+0000 7f03bee00640 1 --2- 172.21.15.32:0/2540298053 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f03a00afae0 0x7f03a00b1ff0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f03ac009c30 tx=0x7f03ac005f90 comp rx=0 tx=0).stop 2024-04-30T08:32:12.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.238+0000 7f03bee00640 1 -- 172.21.15.32:0/2540298053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03b8104bd0 msgr2=0x7f03b819c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:12.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.238+0000 7f03bee00640 1 --2- 172.21.15.32:0/2540298053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03b8104bd0 0x7f03b819c7e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f03a402aa00 tx=0x7f03a4003c10 comp rx=0 tx=0).stop 2024-04-30T08:32:12.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.238+0000 7f03bee00640 1 -- 172.21.15.32:0/2540298053 shutdown_connections 2024-04-30T08:32:12.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.238+0000 7f03bee00640 1 --2- 172.21.15.32:0/2540298053 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f03a00afae0 0x7f03a00b1ff0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:12.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.238+0000 7f03bee00640 1 --2- 172.21.15.32:0/2540298053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03b81098d0 0x7f03b81a3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:12.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.238+0000 7f03bee00640 1 --2- 172.21.15.32:0/2540298053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f03b8104bd0 0x7f03b819c7e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:12.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.238+0000 7f03bee00640 1 --2- 172.21.15.32:0/2540298053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f03b8103990 0x7f03b819c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:12.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.238+0000 7f03bee00640 1 -- 172.21.15.32:0/2540298053 >> 172.21.15.32:0/2540298053 conn(0x7f03b80feba0 msgr2=0x7f03b80ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:12.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.238+0000 7f03bee00640 1 -- 172.21.15.32:0/2540298053 shutdown_connections 2024-04-30T08:32:12.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.238+0000 7f03bee00640 1 -- 172.21.15.32:0/2540298053 wait complete. 2024-04-30T08:32:12.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:11 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2810045377' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:32:12.265 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-30T08:32:12.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.556+0000 7fafe2000640 1 -- 172.21.15.32:0/972470183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafdc103990 msgr2=0x7fafdc104690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:12.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.556+0000 7fafe2000640 1 --2- 172.21.15.32:0/972470183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafdc103990 0x7fafdc104690 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fafcc009900 tx=0x7fafcc02a400 comp rx=0 tx=0).stop 2024-04-30T08:32:12.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.556+0000 7fafe2000640 1 -- 172.21.15.32:0/972470183 shutdown_connections 2024-04-30T08:32:12.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.556+0000 7fafe2000640 1 --2- 172.21.15.32:0/972470183 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fafdc1098d0 0x7fafdc10bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:12.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.556+0000 7fafe2000640 1 --2- 172.21.15.32:0/972470183 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafdc104bd0 0x7fafdc104fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:12.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.556+0000 7fafe2000640 1 --2- 172.21.15.32:0/972470183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafdc103990 0x7fafdc104690 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:12.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.556+0000 7fafe2000640 1 -- 172.21.15.32:0/972470183 >> 172.21.15.32:0/972470183 conn(0x7fafdc0feba0 msgr2=0x7fafdc101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:12.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.556+0000 7fafe2000640 1 -- 172.21.15.32:0/972470183 shutdown_connections 2024-04-30T08:32:12.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.556+0000 7fafe2000640 1 -- 172.21.15.32:0/972470183 wait complete. 2024-04-30T08:32:12.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.557+0000 7fafe2000640 1 Processor -- start 2024-04-30T08:32:12.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.557+0000 7fafe2000640 1 -- start start 2024-04-30T08:32:12.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.558+0000 7fafe2000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafdc103990 0x7fafdc0768d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:12.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.558+0000 7fafe2000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fafdc104bd0 0x7fafdc074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:12.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.558+0000 7fafe2000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafdc1098d0 0x7fafdc0754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:12.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.558+0000 7fafe2000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fafdc10f4e0 con 0x7fafdc1098d0 2024-04-30T08:32:12.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.558+0000 7fafe2000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fafdc10f360 con 0x7fafdc104bd0 2024-04-30T08:32:12.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.558+0000 7fafe2000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fafdc10f660 con 0x7fafdc103990 2024-04-30T08:32:12.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.558+0000 7fafe0c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafdc103990 0x7fafdc0768d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:12.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.558+0000 7fafd3e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fafdc104bd0 0x7fafdc074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:12.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.558+0000 7fafe1600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafdc1098d0 0x7fafdc0754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:12.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.558+0000 7fafe0c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafdc103990 0x7fafdc0768d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:37616/0 (socket says 172.21.15.32:37616) 2024-04-30T08:32:12.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.558+0000 7fafe0c00640 1 -- 172.21.15.32:0/2946999154 learned_addr learned my addr 172.21.15.32:0/2946999154 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:12.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.559+0000 7fafd3e00640 1 -- 172.21.15.32:0/2946999154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafdc103990 msgr2=0x7fafdc0768d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:12.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.559+0000 7fafd3e00640 1 --2- 172.21.15.32:0/2946999154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafdc103990 0x7fafdc0768d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:12.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.559+0000 7fafd3e00640 1 -- 172.21.15.32:0/2946999154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafdc1098d0 msgr2=0x7fafdc0754b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:12.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.559+0000 7fafd3e00640 1 --2- 172.21.15.32:0/2946999154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafdc1098d0 0x7fafdc0754b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:12.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.559+0000 7fafd3e00640 1 -- 172.21.15.32:0/2946999154 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafdc075be0 con 0x7fafdc104bd0 2024-04-30T08:32:12.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.559+0000 7fafe0c00640 1 --2- 172.21.15.32:0/2946999154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafdc103990 0x7fafdc0768d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:32:12.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.559+0000 7fafd3e00640 1 --2- 172.21.15.32:0/2946999154 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fafdc104bd0 0x7fafdc074f70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fafc400e9f0 tx=0x7fafc400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:12.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.559+0000 7fafda000640 1 -- 172.21.15.32:0/2946999154 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc400ccc0 con 0x7fafdc104bd0 2024-04-30T08:32:12.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.559+0000 7fafda000640 1 -- 172.21.15.32:0/2946999154 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fafc40108e0 con 0x7fafdc104bd0 2024-04-30T08:32:12.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.559+0000 7fafe2000640 1 -- 172.21.15.32:0/2946999154 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafdc1a41a0 con 0x7fafdc104bd0 2024-04-30T08:32:12.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.559+0000 7fafda000640 1 -- 172.21.15.32:0/2946999154 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc4018880 con 0x7fafdc104bd0 2024-04-30T08:32:12.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.560+0000 7fafe2000640 1 -- 172.21.15.32:0/2946999154 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafdc1a4660 con 0x7fafdc104bd0 2024-04-30T08:32:12.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.561+0000 7fafe2000640 1 -- 172.21.15.32:0/2946999154 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafdc1086e0 con 0x7fafdc104bd0 2024-04-30T08:32:12.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.563+0000 7fafda000640 1 -- 172.21.15.32:0/2946999154 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fafc4010430 con 0x7fafdc104bd0 2024-04-30T08:32:12.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.565+0000 7fafda000640 1 --2- 172.21.15.32:0/2946999154 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fafc00afae0 0x7fafc00b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:12.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.565+0000 7fafda000640 1 -- 172.21.15.32:0/2946999154 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7fafc40df650 con 0x7fafdc104bd0 2024-04-30T08:32:12.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.565+0000 7fafe0c00640 1 --2- 172.21.15.32:0/2946999154 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fafc00afae0 0x7fafc00b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:12.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.566+0000 7fafe0c00640 1 --2- 172.21.15.32:0/2946999154 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fafc00afae0 0x7fafc00b1ff0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fafcc000c00 tx=0x7fafcc0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:12.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.566+0000 7fafda000640 1 -- 172.21.15.32:0/2946999154 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fafc40a7940 con 0x7fafdc104bd0 2024-04-30T08:32:12.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.781+0000 7fafe2000640 1 -- 172.21.15.32:0/2946999154 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fafdc063820 con 0x7fafdc104bd0 2024-04-30T08:32:12.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.782+0000 7fafda000640 1 -- 172.21.15.32:0/2946999154 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) ==== 104+0+11 (secure 0 0 0) 0x7fafc40abc40 con 0x7fafdc104bd0 2024-04-30T08:32:12.782 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-30T08:32:12.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.784+0000 7fafe2000640 1 -- 172.21.15.32:0/2946999154 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fafc00afae0 msgr2=0x7fafc00b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:12.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.784+0000 7fafe2000640 1 --2- 172.21.15.32:0/2946999154 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fafc00afae0 0x7fafc00b1ff0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fafcc000c00 tx=0x7fafcc0314a0 comp rx=0 tx=0).stop 2024-04-30T08:32:12.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.784+0000 7fafe2000640 1 -- 172.21.15.32:0/2946999154 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fafdc104bd0 msgr2=0x7fafdc074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:12.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.784+0000 7fafe2000640 1 --2- 172.21.15.32:0/2946999154 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fafdc104bd0 0x7fafdc074f70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fafc400e9f0 tx=0x7fafc400eeb0 comp rx=0 tx=0).stop 2024-04-30T08:32:12.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.785+0000 7fafe2000640 1 -- 172.21.15.32:0/2946999154 shutdown_connections 2024-04-30T08:32:12.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.785+0000 7fafe2000640 1 --2- 172.21.15.32:0/2946999154 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fafc00afae0 0x7fafc00b1ff0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:12.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.785+0000 7fafe2000640 1 --2- 172.21.15.32:0/2946999154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fafdc1098d0 0x7fafdc0754b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:12.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.785+0000 7fafe2000640 1 --2- 172.21.15.32:0/2946999154 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fafdc104bd0 0x7fafdc074f70 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:12.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.785+0000 7fafe2000640 1 --2- 172.21.15.32:0/2946999154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafdc103990 0x7fafdc0768d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:12.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.785+0000 7fafe2000640 1 -- 172.21.15.32:0/2946999154 >> 172.21.15.32:0/2946999154 conn(0x7fafdc0feba0 msgr2=0x7fafdc1058b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:12.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.785+0000 7fafe2000640 1 -- 172.21.15.32:0/2946999154 shutdown_connections 2024-04-30T08:32:12.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:12.785+0000 7fafe2000640 1 -- 172.21.15.32:0/2946999154 wait complete. 2024-04-30T08:32:12.814 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-30T08:32:13.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:12 smithi189 ceph-mon[33038]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-04-30T08:32:13.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:12 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2540298053' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:32:13.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.103+0000 7fbef2200640 1 -- 172.21.15.32:0/95039803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbeec111230 msgr2=0x7fbeec113650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:13.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.103+0000 7fbef2200640 1 --2- 172.21.15.32:0/95039803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbeec111230 0x7fbeec113650 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbee4009900 tx=0x7fbee402a410 comp rx=0 tx=0).stop 2024-04-30T08:32:13.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.103+0000 7fbef2200640 1 -- 172.21.15.32:0/95039803 shutdown_connections 2024-04-30T08:32:13.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.103+0000 7fbef2200640 1 --2- 172.21.15.32:0/95039803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbeec111230 0x7fbeec113650 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:13.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.103+0000 7fbef2200640 1 --2- 172.21.15.32:0/95039803 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbeec10cb20 0x7fbeec10b320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:13.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.103+0000 7fbef2200640 1 --2- 172.21.15.32:0/95039803 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbeec10c620 0x7fbeec10ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:13.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.103+0000 7fbef2200640 1 -- 172.21.15.32:0/95039803 >> 172.21.15.32:0/95039803 conn(0x7fbeec071680 msgr2=0x7fbeec073b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:13.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.103+0000 7fbef2200640 1 -- 172.21.15.32:0/95039803 shutdown_connections 2024-04-30T08:32:13.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.103+0000 7fbef2200640 1 -- 172.21.15.32:0/95039803 wait complete. 2024-04-30T08:32:13.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.104+0000 7fbef2200640 1 Processor -- start 2024-04-30T08:32:13.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.104+0000 7fbef2200640 1 -- start start 2024-04-30T08:32:13.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.105+0000 7fbef2200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbeec10c620 0x7fbeec1a4c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:13.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.105+0000 7fbef2200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbeec10cb20 0x7fbeec1a51a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:13.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.105+0000 7fbef2200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbeec111230 0x7fbeec1ac0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:13.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.105+0000 7fbef2200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbeec116dd0 con 0x7fbeec10c620 2024-04-30T08:32:13.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.105+0000 7fbef2200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fbeec116c50 con 0x7fbeec10cb20 2024-04-30T08:32:13.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.105+0000 7fbef2200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbeec116f50 con 0x7fbeec111230 2024-04-30T08:32:13.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.105+0000 7fbeeb400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbeec10c620 0x7fbeec1a4c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:13.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.105+0000 7fbeebe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbeec111230 0x7fbeec1ac0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:13.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.105+0000 7fbeeb400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbeec10c620 0x7fbeec1a4c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42454/0 (socket says 172.21.15.32:42454) 2024-04-30T08:32:13.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.105+0000 7fbeeb400640 1 -- 172.21.15.32:0/3078494158 learned_addr learned my addr 172.21.15.32:0/3078494158 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:13.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.105+0000 7fbeeaa00640 1 --2- 172.21.15.32:0/3078494158 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbeec10cb20 0x7fbeec1a51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:13.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.105+0000 7fbeeb400640 1 -- 172.21.15.32:0/3078494158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbeec111230 msgr2=0x7fbeec1ac0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:13.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.105+0000 7fbeeb400640 1 --2- 172.21.15.32:0/3078494158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbeec111230 0x7fbeec1ac0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:13.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.105+0000 7fbeeb400640 1 -- 172.21.15.32:0/3078494158 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbeec10cb20 msgr2=0x7fbeec1a51a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:13.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.105+0000 7fbeeb400640 1 --2- 172.21.15.32:0/3078494158 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbeec10cb20 0x7fbeec1a51a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:13.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.106+0000 7fbeeb400640 1 -- 172.21.15.32:0/3078494158 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeec1ac790 con 0x7fbeec10c620 2024-04-30T08:32:13.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.106+0000 7fbeebe00640 1 --2- 172.21.15.32:0/3078494158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbeec111230 0x7fbeec1ac0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:32:13.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.106+0000 7fbeeb400640 1 --2- 172.21.15.32:0/3078494158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbeec10c620 0x7fbeec1a4c60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbedc00e9c0 tx=0x7fbedc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:13.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.106+0000 7fbee3e00640 1 -- 172.21.15.32:0/3078494158 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbedc00cd80 con 0x7fbeec10c620 2024-04-30T08:32:13.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.106+0000 7fbee3e00640 1 -- 172.21.15.32:0/3078494158 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbedc0054d0 con 0x7fbeec10c620 2024-04-30T08:32:13.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.106+0000 7fbef2200640 1 -- 172.21.15.32:0/3078494158 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbeec1aca80 con 0x7fbeec10c620 2024-04-30T08:32:13.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.106+0000 7fbee3e00640 1 -- 172.21.15.32:0/3078494158 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbedc010430 con 0x7fbeec10c620 2024-04-30T08:32:13.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.106+0000 7fbef2200640 1 -- 172.21.15.32:0/3078494158 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbeec1acfd0 con 0x7fbeec10c620 2024-04-30T08:32:13.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.107+0000 7fbef2200640 1 -- 172.21.15.32:0/3078494158 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbeec1ad260 con 0x7fbeec10c620 2024-04-30T08:32:13.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.110+0000 7fbee3e00640 1 -- 172.21.15.32:0/3078494158 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fbedc004fb0 con 0x7fbeec10c620 2024-04-30T08:32:13.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.112+0000 7fbee3e00640 1 --2- 172.21.15.32:0/3078494158 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbed00afaf0 0x7fbed00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:13.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.112+0000 7fbee3e00640 1 -- 172.21.15.32:0/3078494158 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7fbedc0df480 con 0x7fbeec10c620 2024-04-30T08:32:13.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.112+0000 7fbeeaa00640 1 --2- 172.21.15.32:0/3078494158 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbed00afaf0 0x7fbed00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:13.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.113+0000 7fbeeaa00640 1 --2- 172.21.15.32:0/3078494158 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbed00afaf0 0x7fbed00b2000 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbed8005fd0 tx=0x7fbed8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:13.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.115+0000 7fbee3e00640 1 -- 172.21.15.32:0/3078494158 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbedc053ff0 con 0x7fbeec10c620 2024-04-30T08:32:13.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:12 smithi032 ceph-mon[28331]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-04-30T08:32:13.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:12 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2540298053' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:32:13.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:12 smithi071 ceph-mon[33385]: pgmap v332: 129 pgs: 129 active+clean; 579 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-04-30T08:32:13.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:12 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2540298053' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:32:13.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.321+0000 7fbef2200640 1 -- 172.21.15.32:0/3078494158 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fbeec1ad430 con 0x7fbeec10c620 2024-04-30T08:32:13.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.322+0000 7fbee3e00640 1 -- 172.21.15.32:0/3078494158 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) ==== 100+0+11 (secure 0 0 0) 0x7fbedc0a7dc0 con 0x7fbeec10c620 2024-04-30T08:32:13.323 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-04-30T08:32:13.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.325+0000 7fbef2200640 1 -- 172.21.15.32:0/3078494158 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbed00afaf0 msgr2=0x7fbed00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:13.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.325+0000 7fbef2200640 1 --2- 172.21.15.32:0/3078494158 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbed00afaf0 0x7fbed00b2000 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbed8005fd0 tx=0x7fbed8009500 comp rx=0 tx=0).stop 2024-04-30T08:32:13.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.325+0000 7fbef2200640 1 -- 172.21.15.32:0/3078494158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbeec10c620 msgr2=0x7fbeec1a4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:13.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.325+0000 7fbef2200640 1 --2- 172.21.15.32:0/3078494158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbeec10c620 0x7fbeec1a4c60 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fbedc00e9c0 tx=0x7fbedc00ee80 comp rx=0 tx=0).stop 2024-04-30T08:32:13.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.325+0000 7fbef2200640 1 -- 172.21.15.32:0/3078494158 shutdown_connections 2024-04-30T08:32:13.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.325+0000 7fbef2200640 1 --2- 172.21.15.32:0/3078494158 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbed00afaf0 0x7fbed00b2000 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:13.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.325+0000 7fbef2200640 1 --2- 172.21.15.32:0/3078494158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbeec111230 0x7fbeec1ac0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:13.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.325+0000 7fbef2200640 1 --2- 172.21.15.32:0/3078494158 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbeec10cb20 0x7fbeec1a51a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:13.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.325+0000 7fbef2200640 1 --2- 172.21.15.32:0/3078494158 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbeec10c620 0x7fbeec1a4c60 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:13.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.325+0000 7fbef2200640 1 -- 172.21.15.32:0/3078494158 >> 172.21.15.32:0/3078494158 conn(0x7fbeec071680 msgr2=0x7fbeec112610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:13.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.325+0000 7fbef2200640 1 -- 172.21.15.32:0/3078494158 shutdown_connections 2024-04-30T08:32:13.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.325+0000 7fbef2200640 1 -- 172.21.15.32:0/3078494158 wait complete. 2024-04-30T08:32:13.353 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-30T08:32:13.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.649+0000 7f23e9000640 1 -- 172.21.15.32:0/285511894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23e4075010 msgr2=0x7f23e4075420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:13.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.649+0000 7f23e9000640 1 --2- 172.21.15.32:0/285511894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23e4075010 0x7f23e4075420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f23d4009be0 tx=0x7f23d402a6c0 comp rx=0 tx=0).stop 2024-04-30T08:32:13.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.649+0000 7f23e9000640 1 -- 172.21.15.32:0/285511894 shutdown_connections 2024-04-30T08:32:13.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.649+0000 7f23e9000640 1 --2- 172.21.15.32:0/285511894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23e4110110 0x7f23e4112530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:13.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.649+0000 7f23e9000640 1 --2- 172.21.15.32:0/285511894 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f23e4075960 0x7f23e410fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:13.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.649+0000 7f23e9000640 1 --2- 172.21.15.32:0/285511894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23e4075010 0x7f23e4075420 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:13.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.649+0000 7f23e9000640 1 -- 172.21.15.32:0/285511894 >> 172.21.15.32:0/285511894 conn(0x7f23e40feba0 msgr2=0x7f23e4101050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:13.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.649+0000 7f23e9000640 1 -- 172.21.15.32:0/285511894 shutdown_connections 2024-04-30T08:32:13.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.649+0000 7f23e9000640 1 -- 172.21.15.32:0/285511894 wait complete. 2024-04-30T08:32:13.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.651+0000 7f23e9000640 1 Processor -- start 2024-04-30T08:32:13.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.651+0000 7f23e9000640 1 -- start start 2024-04-30T08:32:13.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.651+0000 7f23e9000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23e4075010 0x7f23e41a06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:13.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.651+0000 7f23e9000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f23e4075960 0x7f23e41a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:13.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.651+0000 7f23e9000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23e4110110 0x7f23e41a7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:13.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.651+0000 7f23e9000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f23e4115af0 con 0x7f23e4075010 2024-04-30T08:32:13.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.651+0000 7f23e9000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f23e4115970 con 0x7f23e4075960 2024-04-30T08:32:13.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.651+0000 7f23e9000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f23e4115c70 con 0x7f23e4110110 2024-04-30T08:32:13.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.652+0000 7f23e3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23e4075010 0x7f23e41a06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:13.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.652+0000 7f23e3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23e4075010 0x7f23e41a06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42476/0 (socket says 172.21.15.32:42476) 2024-04-30T08:32:13.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.652+0000 7f23e3400640 1 -- 172.21.15.32:0/2459241435 learned_addr learned my addr 172.21.15.32:0/2459241435 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:13.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.652+0000 7f23e3e00640 1 --2- 172.21.15.32:0/2459241435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23e4110110 0x7f23e41a7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:13.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.652+0000 7f23e2a00640 1 --2- 172.21.15.32:0/2459241435 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f23e4075960 0x7f23e41a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:13.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.652+0000 7f23e3400640 1 -- 172.21.15.32:0/2459241435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23e4110110 msgr2=0x7f23e41a7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:13.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.652+0000 7f23e3400640 1 --2- 172.21.15.32:0/2459241435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23e4110110 0x7f23e41a7bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:13.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.652+0000 7f23e3400640 1 -- 172.21.15.32:0/2459241435 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f23e4075960 msgr2=0x7f23e41a0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:13.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.652+0000 7f23e3400640 1 --2- 172.21.15.32:0/2459241435 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f23e4075960 0x7f23e41a0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:13.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.652+0000 7f23e3400640 1 -- 172.21.15.32:0/2459241435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23e41a8220 con 0x7f23e4075010 2024-04-30T08:32:13.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.652+0000 7f23e2a00640 1 --2- 172.21.15.32:0/2459241435 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f23e4075960 0x7f23e41a0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:32:13.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.653+0000 7f23e3400640 1 --2- 172.21.15.32:0/2459241435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23e4075010 0x7f23e41a06e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f23d4000f80 tx=0x7f23d4030ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:13.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.653+0000 7f23dbe00640 1 -- 172.21.15.32:0/2459241435 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d4041020 con 0x7f23e4075010 2024-04-30T08:32:13.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.653+0000 7f23dbe00640 1 -- 172.21.15.32:0/2459241435 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f23d4030e90 con 0x7f23e4075010 2024-04-30T08:32:13.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.653+0000 7f23dbe00640 1 -- 172.21.15.32:0/2459241435 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d4036040 con 0x7f23e4075010 2024-04-30T08:32:13.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.653+0000 7f23e9000640 1 -- 172.21.15.32:0/2459241435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23e41a84b0 con 0x7f23e4075010 2024-04-30T08:32:13.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.653+0000 7f23e9000640 1 -- 172.21.15.32:0/2459241435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23e41a89a0 con 0x7f23e4075010 2024-04-30T08:32:13.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.654+0000 7f23e9000640 1 -- 172.21.15.32:0/2459241435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23e4105da0 con 0x7f23e4075010 2024-04-30T08:32:13.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.657+0000 7f23dbe00640 1 -- 172.21.15.32:0/2459241435 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f23d4044020 con 0x7f23e4075010 2024-04-30T08:32:13.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.658+0000 7f23dbe00640 1 --2- 172.21.15.32:0/2459241435 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f23c80afae0 0x7f23c80b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:13.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.659+0000 7f23dbe00640 1 -- 172.21.15.32:0/2459241435 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f23d40fffe0 con 0x7f23e4075010 2024-04-30T08:32:13.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.659+0000 7f23e2a00640 1 --2- 172.21.15.32:0/2459241435 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f23c80afae0 0x7f23c80b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:13.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.660+0000 7f23e2a00640 1 --2- 172.21.15.32:0/2459241435 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f23c80afae0 0x7f23c80b1ff0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f23e41a1780 tx=0x7f23cc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:13.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.661+0000 7f23dbe00640 1 -- 172.21.15.32:0/2459241435 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f23d404b030 con 0x7f23e4075010 2024-04-30T08:32:13.871 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.870+0000 7f23e9000640 1 -- 172.21.15.32:0/2459241435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f23e4063820 con 0x7f23e4075010 2024-04-30T08:32:13.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.872+0000 7f23dbe00640 1 -- 172.21.15.32:0/2459241435 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v81) ==== 103+0+11 (secure 0 0 0) 0x7f23d40c8960 con 0x7f23e4075010 2024-04-30T08:32:13.872 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-30T08:32:13.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.875+0000 7f23e9000640 1 -- 172.21.15.32:0/2459241435 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f23c80afae0 msgr2=0x7f23c80b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:13.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.875+0000 7f23e9000640 1 --2- 172.21.15.32:0/2459241435 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f23c80afae0 0x7f23c80b1ff0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f23e41a1780 tx=0x7f23cc009260 comp rx=0 tx=0).stop 2024-04-30T08:32:13.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.875+0000 7f23e9000640 1 -- 172.21.15.32:0/2459241435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23e4075010 msgr2=0x7f23e41a06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:13.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.875+0000 7f23e9000640 1 --2- 172.21.15.32:0/2459241435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23e4075010 0x7f23e41a06e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f23d4000f80 tx=0x7f23d4030ca0 comp rx=0 tx=0).stop 2024-04-30T08:32:13.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.875+0000 7f23e9000640 1 -- 172.21.15.32:0/2459241435 shutdown_connections 2024-04-30T08:32:13.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.875+0000 7f23e9000640 1 --2- 172.21.15.32:0/2459241435 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f23c80afae0 0x7f23c80b1ff0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:13.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.875+0000 7f23e9000640 1 --2- 172.21.15.32:0/2459241435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f23e4110110 0x7f23e41a7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:13.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.875+0000 7f23e9000640 1 --2- 172.21.15.32:0/2459241435 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f23e4075960 0x7f23e41a0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:13.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.875+0000 7f23e9000640 1 --2- 172.21.15.32:0/2459241435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f23e4075010 0x7f23e41a06e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:13.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.875+0000 7f23e9000640 1 -- 172.21.15.32:0/2459241435 >> 172.21.15.32:0/2459241435 conn(0x7f23e40feba0 msgr2=0x7f23e4102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:13.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.875+0000 7f23e9000640 1 -- 172.21.15.32:0/2459241435 shutdown_connections 2024-04-30T08:32:13.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:13.875+0000 7f23e9000640 1 -- 172.21.15.32:0/2459241435 wait complete. 2024-04-30T08:32:13.903 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-30T08:32:14.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:13 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2946999154' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-30T08:32:14.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:13 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3078494158' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-30T08:32:14.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:13 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2946999154' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-30T08:32:14.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:13 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3078494158' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-30T08:32:14.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.193+0000 7f65fd400640 1 -- 172.21.15.32:0/1832436403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65f8103990 msgr2=0x7f65f8104690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:14.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.193+0000 7f65fd400640 1 --2- 172.21.15.32:0/1832436403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65f8103990 0x7f65f8104690 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f65e8009900 tx=0x7f65e802a400 comp rx=0 tx=0).stop 2024-04-30T08:32:14.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.193+0000 7f65fd400640 1 -- 172.21.15.32:0/1832436403 shutdown_connections 2024-04-30T08:32:14.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.193+0000 7f65fd400640 1 --2- 172.21.15.32:0/1832436403 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65f81098d0 0x7f65f810bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:14.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.193+0000 7f65fd400640 1 --2- 172.21.15.32:0/1832436403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65f8104bd0 0x7f65f8104fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:14.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.193+0000 7f65fd400640 1 --2- 172.21.15.32:0/1832436403 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65f8103990 0x7f65f8104690 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:14.194 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.193+0000 7f65fd400640 1 -- 172.21.15.32:0/1832436403 >> 172.21.15.32:0/1832436403 conn(0x7f65f80feba0 msgr2=0x7f65f8101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:14.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.194+0000 7f65fd400640 1 -- 172.21.15.32:0/1832436403 shutdown_connections 2024-04-30T08:32:14.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.194+0000 7f65fd400640 1 -- 172.21.15.32:0/1832436403 wait complete. 2024-04-30T08:32:14.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.195+0000 7f65fd400640 1 Processor -- start 2024-04-30T08:32:14.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.195+0000 7f65fd400640 1 -- start start 2024-04-30T08:32:14.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.195+0000 7f65fd400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65f8103990 0x7f65f819a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:14.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.195+0000 7f65fd400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65f8104bd0 0x7f65f819a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:14.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.195+0000 7f65fd400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65f81098d0 0x7f65f81a15f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:14.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.195+0000 7f65fd400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f65f810f4c0 con 0x7f65f81098d0 2024-04-30T08:32:14.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.195+0000 7f65fd400640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f65f810f340 con 0x7f65f8104bd0 2024-04-30T08:32:14.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.195+0000 7f65fd400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f65f810f640 con 0x7f65f8103990 2024-04-30T08:32:14.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.195+0000 7f65fca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65f81098d0 0x7f65f81a15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:14.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.195+0000 7f65f7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65f8103990 0x7f65f819a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:14.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.195+0000 7f65f7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65f8103990 0x7f65f819a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:37650/0 (socket says 172.21.15.32:37650) 2024-04-30T08:32:14.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.195+0000 7f65f7e00640 1 -- 172.21.15.32:0/241051266 learned_addr learned my addr 172.21.15.32:0/241051266 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:14.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.196+0000 7f65f7400640 1 --2- 172.21.15.32:0/241051266 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65f8104bd0 0x7f65f819a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:14.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.196+0000 7f65fca00640 1 -- 172.21.15.32:0/241051266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65f8103990 msgr2=0x7f65f819a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:14.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.196+0000 7f65fca00640 1 --2- 172.21.15.32:0/241051266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65f8103990 0x7f65f819a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:14.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.196+0000 7f65fca00640 1 -- 172.21.15.32:0/241051266 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65f8104bd0 msgr2=0x7f65f819a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:14.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.196+0000 7f65fca00640 1 --2- 172.21.15.32:0/241051266 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65f8104bd0 0x7f65f819a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:14.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.196+0000 7f65fca00640 1 -- 172.21.15.32:0/241051266 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65f81a1c60 con 0x7f65f81098d0 2024-04-30T08:32:14.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.196+0000 7f65f7e00640 1 --2- 172.21.15.32:0/241051266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65f8103990 0x7f65f819a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:32:14.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.196+0000 7f65fca00640 1 --2- 172.21.15.32:0/241051266 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65f81098d0 0x7f65f81a15f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f65f000da10 tx=0x7f65f000ded0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:14.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.197+0000 7f65f4c00640 1 -- 172.21.15.32:0/241051266 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f000bd70 con 0x7f65f81098d0 2024-04-30T08:32:14.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.197+0000 7f65f4c00640 1 -- 172.21.15.32:0/241051266 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65f0003e20 con 0x7f65f81098d0 2024-04-30T08:32:14.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.197+0000 7f65fd400640 1 -- 172.21.15.32:0/241051266 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65f81a1f50 con 0x7f65f81098d0 2024-04-30T08:32:14.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.197+0000 7f65f4c00640 1 -- 172.21.15.32:0/241051266 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f00106c0 con 0x7f65f81098d0 2024-04-30T08:32:14.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.197+0000 7f65fd400640 1 -- 172.21.15.32:0/241051266 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65f81a23b0 con 0x7f65f81098d0 2024-04-30T08:32:14.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.198+0000 7f65fd400640 1 -- 172.21.15.32:0/241051266 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65f81086e0 con 0x7f65f81098d0 2024-04-30T08:32:14.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.201+0000 7f65f4c00640 1 -- 172.21.15.32:0/241051266 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f65f0004270 con 0x7f65f81098d0 2024-04-30T08:32:14.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.202+0000 7f65f4c00640 1 --2- 172.21.15.32:0/241051266 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f65dc0afb60 0x7f65dc0b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:14.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.202+0000 7f65f4c00640 1 -- 172.21.15.32:0/241051266 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f65f00df4d0 con 0x7f65f81098d0 2024-04-30T08:32:14.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.202+0000 7f65f7e00640 1 --2- 172.21.15.32:0/241051266 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f65dc0afb60 0x7f65dc0b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:14.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.203+0000 7f65f7e00640 1 --2- 172.21.15.32:0/241051266 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f65dc0afb60 0x7f65dc0b2070 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f65e8000c00 tx=0x7f65e80314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:14.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.205+0000 7f65f4c00640 1 -- 172.21.15.32:0/241051266 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f65f00a7e90 con 0x7f65f81098d0 2024-04-30T08:32:14.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:13 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2946999154' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-30T08:32:14.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:13 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3078494158' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-30T08:32:14.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.426+0000 7f65fd400640 1 -- 172.21.15.32:0/241051266 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f65f802de20 con 0x7f65f81098d0 2024-04-30T08:32:14.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.427+0000 7f65f4c00640 1 -- 172.21.15.32:0/241051266 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v81) ==== 81+0+587 (secure 0 0 0) 0x7f65f00a7790 con 0x7f65f81098d0 2024-04-30T08:32:14.428 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/1266897222 2024-05-01T08:23:29.526664+0000 2024-04-30T08:32:14.428 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/2915689533 2024-05-01T08:23:29.526664+0000 2024-04-30T08:32:14.428 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6801/4019351306 2024-05-01T08:23:29.526664+0000 2024-04-30T08:32:14.428 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/425836193 2024-05-01T08:23:29.526664+0000 2024-04-30T08:32:14.428 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/1134075722 2024-05-01T08:22:53.639893+0000 2024-04-30T08:32:14.428 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6800/464872591 2024-05-01T08:22:53.639893+0000 2024-04-30T08:32:14.428 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6801/464872591 2024-05-01T08:22:53.639893+0000 2024-04-30T08:32:14.429 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/21059253 2024-05-01T08:22:53.639893+0000 2024-04-30T08:32:14.429 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/3245550894 2024-05-01T08:22:53.639893+0000 2024-04-30T08:32:14.429 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6800/4019351306 2024-05-01T08:23:29.526664+0000 2024-04-30T08:32:14.429 INFO:teuthology.orchestra.run.smithi032.stderr:listed 10 entries 2024-04-30T08:32:14.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.430+0000 7f65fd400640 1 -- 172.21.15.32:0/241051266 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f65dc0afb60 msgr2=0x7f65dc0b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:14.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.430+0000 7f65fd400640 1 --2- 172.21.15.32:0/241051266 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f65dc0afb60 0x7f65dc0b2070 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f65e8000c00 tx=0x7f65e80314a0 comp rx=0 tx=0).stop 2024-04-30T08:32:14.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.430+0000 7f65fd400640 1 -- 172.21.15.32:0/241051266 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65f81098d0 msgr2=0x7f65f81a15f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:14.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.430+0000 7f65fd400640 1 --2- 172.21.15.32:0/241051266 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65f81098d0 0x7f65f81a15f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f65f000da10 tx=0x7f65f000ded0 comp rx=0 tx=0).stop 2024-04-30T08:32:14.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.430+0000 7f65fd400640 1 -- 172.21.15.32:0/241051266 shutdown_connections 2024-04-30T08:32:14.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.430+0000 7f65fd400640 1 --2- 172.21.15.32:0/241051266 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f65dc0afb60 0x7f65dc0b2070 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:14.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.430+0000 7f65fd400640 1 --2- 172.21.15.32:0/241051266 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f65f81098d0 0x7f65f81a15f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:14.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.430+0000 7f65fd400640 1 --2- 172.21.15.32:0/241051266 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f65f8104bd0 0x7f65f819a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:14.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.430+0000 7f65fd400640 1 --2- 172.21.15.32:0/241051266 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f65f8103990 0x7f65f819a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:14.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.430+0000 7f65fd400640 1 -- 172.21.15.32:0/241051266 >> 172.21.15.32:0/241051266 conn(0x7f65f80feba0 msgr2=0x7f65f81058b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:14.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.431+0000 7f65fd400640 1 -- 172.21.15.32:0/241051266 shutdown_connections 2024-04-30T08:32:14.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.431+0000 7f65fd400640 1 -- 172.21.15.32:0/241051266 wait complete. 2024-04-30T08:32:14.458 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-30T08:32:14.458 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-30T08:32:14.508 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-30T08:32:14.777 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:14 smithi032 ceph-mon[28331]: pgmap v333: 129 pgs: 129 active+clean; 579 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-04-30T08:32:14.777 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:14 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2459241435' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-30T08:32:14.777 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:14 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/241051266' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-30T08:32:14.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.796+0000 7f885dc00640 1 -- 172.21.15.32:0/2132291852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8858105b90 msgr2=0x7f8858106890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:14.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.796+0000 7f885dc00640 1 --2- 172.21.15.32:0/2132291852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8858105b90 0x7f8858106890 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8848009900 tx=0x7f884802a3f0 comp rx=0 tx=0).stop 2024-04-30T08:32:14.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.797+0000 7f885dc00640 1 -- 172.21.15.32:0/2132291852 shutdown_connections 2024-04-30T08:32:14.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.797+0000 7f885dc00640 1 --2- 172.21.15.32:0/2132291852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f885810bad0 0x7f885810def0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:14.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.797+0000 7f885dc00640 1 --2- 172.21.15.32:0/2132291852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8858106dd0 0x7f88581071e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:14.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.797+0000 7f885dc00640 1 --2- 172.21.15.32:0/2132291852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8858105b90 0x7f8858106890 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:14.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.797+0000 7f885dc00640 1 -- 172.21.15.32:0/2132291852 >> 172.21.15.32:0/2132291852 conn(0x7f8858100da0 msgr2=0x7f8858103230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:14.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.797+0000 7f885dc00640 1 -- 172.21.15.32:0/2132291852 shutdown_connections 2024-04-30T08:32:14.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.797+0000 7f885dc00640 1 -- 172.21.15.32:0/2132291852 wait complete. 2024-04-30T08:32:14.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.798+0000 7f885dc00640 1 Processor -- start 2024-04-30T08:32:14.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.798+0000 7f885dc00640 1 -- start start 2024-04-30T08:32:14.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.798+0000 7f885dc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8858105b90 0x7f885819c260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:14.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.798+0000 7f885dc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8858106dd0 0x7f885819c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:14.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.798+0000 7f885dc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f885810bad0 0x7f88581a3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:14.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.798+0000 7f885dc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f88581116c0 con 0x7f8858106dd0 2024-04-30T08:32:14.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.798+0000 7f885dc00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8858111540 con 0x7f885810bad0 2024-04-30T08:32:14.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.798+0000 7f885dc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8858111840 con 0x7f8858105b90 2024-04-30T08:32:14.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.799+0000 7f884fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8858106dd0 0x7f885819c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:14.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.799+0000 7f884fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8858106dd0 0x7f885819c7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42522/0 (socket says 172.21.15.32:42522) 2024-04-30T08:32:14.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.799+0000 7f884fe00640 1 -- 172.21.15.32:0/1095698282 learned_addr learned my addr 172.21.15.32:0/1095698282 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:14.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.799+0000 7f8857e00640 1 --2- 172.21.15.32:0/1095698282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8858105b90 0x7f885819c260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:14.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.799+0000 7f885d200640 1 --2- 172.21.15.32:0/1095698282 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f885810bad0 0x7f88581a3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:14.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.799+0000 7f884fe00640 1 -- 172.21.15.32:0/1095698282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8858105b90 msgr2=0x7f885819c260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:14.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.799+0000 7f884fe00640 1 --2- 172.21.15.32:0/1095698282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8858105b90 0x7f885819c260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:14.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.799+0000 7f884fe00640 1 -- 172.21.15.32:0/1095698282 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f885810bad0 msgr2=0x7f88581a3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:14.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.799+0000 7f884fe00640 1 --2- 172.21.15.32:0/1095698282 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f885810bad0 0x7f88581a3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:14.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.799+0000 7f884fe00640 1 -- 172.21.15.32:0/1095698282 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88581a3da0 con 0x7f8858106dd0 2024-04-30T08:32:14.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.799+0000 7f884fe00640 1 --2- 172.21.15.32:0/1095698282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8858106dd0 0x7f885819c7a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f884000d950 tx=0x7f884000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:14.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.800+0000 7f8855600640 1 -- 172.21.15.32:0/1095698282 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8840017030 con 0x7f8858106dd0 2024-04-30T08:32:14.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.800+0000 7f8855600640 1 -- 172.21.15.32:0/1095698282 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8840004020 con 0x7f8858106dd0 2024-04-30T08:32:14.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.800+0000 7f885dc00640 1 -- 172.21.15.32:0/1095698282 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88581a4090 con 0x7f8858106dd0 2024-04-30T08:32:14.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.800+0000 7f8855600640 1 -- 172.21.15.32:0/1095698282 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88400047a0 con 0x7f8858106dd0 2024-04-30T08:32:14.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.800+0000 7f885dc00640 1 -- 172.21.15.32:0/1095698282 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88581a45e0 con 0x7f8858106dd0 2024-04-30T08:32:14.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.801+0000 7f885dc00640 1 -- 172.21.15.32:0/1095698282 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f885810a8e0 con 0x7f8858106dd0 2024-04-30T08:32:14.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.804+0000 7f8855600640 1 -- 172.21.15.32:0/1095698282 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f8840009a30 con 0x7f8858106dd0 2024-04-30T08:32:14.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.805+0000 7f8855600640 1 --2- 172.21.15.32:0/1095698282 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f883c0afae0 0x7f883c0b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:14.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.805+0000 7f8855600640 1 -- 172.21.15.32:0/1095698282 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f88400deff0 con 0x7f8858106dd0 2024-04-30T08:32:14.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.805+0000 7f8857e00640 1 --2- 172.21.15.32:0/1095698282 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f883c0afae0 0x7f883c0b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:14.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.806+0000 7f8857e00640 1 --2- 172.21.15.32:0/1095698282 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f883c0afae0 0x7f883c0b1ff0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f884802a9e0 tx=0x7f88480314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:14.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:14.807+0000 7f8855600640 1 -- 172.21.15.32:0/1095698282 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f88400a7930 con 0x7f8858106dd0 2024-04-30T08:32:15.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.013+0000 7f885dc00640 1 -- 172.21.15.32:0/1095698282 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8858063820 con 0x7f8858106dd0 2024-04-30T08:32:15.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.015+0000 7f8855600640 1 -- 172.21.15.32:0/1095698282 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+24053 (secure 0 0 0) 0x7f88400a7230 con 0x7f8858106dd0 2024-04-30T08:32:15.016 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:32:15.016 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":81,"fsid":"98c0a670-06ca-11ef-bc93-c7b262605968","created":"2024-04-30T08:22:11.122875+0000","modified":"2024-04-30T08:31:58.298969+0000","last_up_change":"2024-04-30T08:30:50.350439+0000","last_in_change":"2024-04-30T08:30:30.210620+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-30T08:26:19.728331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-30T08:31:49.800542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-30T08:31:51.211320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-30T08:31:55.716625+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a7e060c8-1cbb-418f-9a03-06693f6b3f52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6803","nonce":2624700402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6805","nonce":2624700402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6809","nonce":2624700402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6807","nonce":2624700402}]},"public_addr":"172.21.15.32:6803/2624700402","cluster_addr":"172.21.15.32:6805/2624700402","heartbeat_back_addr":"172.21.15.32:6809/2624700402","heartbeat_front_addr":"172.21.15.32:6807/2624700402","state":["exists","up"]},{"osd":1,"uuid":"72ef2fb1-e5ca-48ef-aa3a-4796a64b922f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6801","nonce":444649711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6803","nonce":444649711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6807","nonce":444649711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6805","nonce":444649711}]},"public_addr":"172.21.15.71:6801/444649711","cluster_addr":"172.21.15.71:6803/444649711","heartbeat_back_addr":"172.21.15.71:6807/444649711","heartbeat_front_addr":"172.21.15.71:6805/444649711","state":["exists","up"]},{"osd":2,"uuid":"704e5e8a-935e-47c7-ae28-27119876ac09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6801","nonce":3255528671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6803","nonce":3255528671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6807","nonce":3255528671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6805","nonce":3255528671}]},"public_addr":"172.21.15.189:6801/3255528671","cluster_addr":"172.21.15.189:6803/3255528671","heartbeat_back_addr":"172.21.15.189:6807/3255528671","heartbeat_front_addr":"172.21.15.189:6805/3255528671","state":["exists","up"]},{"osd":3,"uuid":"575004f7-6559-4a62-863e-e33078b0eebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6811","nonce":1384702112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6813","nonce":1384702112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6817","nonce":1384702112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6815","nonce":1384702112}]},"public_addr":"172.21.15.32:6811/1384702112","cluster_addr":"172.21.15.32:6813/1384702112","heartbeat_back_addr":"172.21.15.32:6817/1384702112","heartbeat_front_addr":"172.21.15.32:6815/1384702112","state":["exists","up"]},{"osd":4,"uuid":"e1c1e894-074d-4c78-89ef-9e5a114268bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6809","nonce":4250534574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6811","nonce":4250534574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6815","nonce":4250534574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6813","nonce":4250534574}]},"public_addr":"172.21.15.71:6809/4250534574","cluster_addr":"172.21.15.71:6811/4250534574","heartbeat_back_addr":"172.21.15.71:6815/4250534574","heartbeat_front_addr":"172.21.15.71:6813/4250534574","state":["exists","up"]},{"osd":5,"uuid":"12c022d4-fac6-40c4-ae7a-691571bee5d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6809","nonce":4245346198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6811","nonce":4245346198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6815","nonce":4245346198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6813","nonce":4245346198}]},"public_addr":"172.21.15.189:6809/4245346198","cluster_addr":"172.21.15.189:6811/4245346198","heartbeat_back_addr":"172.21.15.189:6815/4245346198","heartbeat_front_addr":"172.21.15.189:6813/4245346198","state":["exists","up"]},{"osd":6,"uuid":"b413f8ee-3d4a-415a-a2ba-d68437b7330d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6819","nonce":1910389190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6821","nonce":1910389190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6825","nonce":1910389190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6823","nonce":1910389190}]},"public_addr":"172.21.15.32:6819/1910389190","cluster_addr":"172.21.15.32:6821/1910389190","heartbeat_back_addr":"172.21.15.32:6825/1910389190","heartbeat_front_addr":"172.21.15.32:6823/1910389190","state":["exists","up"]},{"osd":7,"uuid":"aac64fb3-fc13-4126-b842-7b06ad5ce238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6817","nonce":692607380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6819","nonce":692607380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6823","nonce":692607380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6821","nonce":692607380}]},"public_addr":"172.21.15.71:6817/692607380","cluster_addr":"172.21.15.71:6819/692607380","heartbeat_back_addr":"172.21.15.71:6823/692607380","heartbeat_front_addr":"172.21.15.71:6821/692607380","state":["exists","up"]},{"osd":8,"uuid":"cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6817","nonce":3973697069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6819","nonce":3973697069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6823","nonce":3973697069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6821","nonce":3973697069}]},"public_addr":"172.21.15.189:6817/3973697069","cluster_addr":"172.21.15.189:6819/3973697069","heartbeat_back_addr":"172.21.15.189:6823/3973697069","heartbeat_front_addr":"172.21.15.189:6821/3973697069","state":["exists","up"]},{"osd":9,"uuid":"ef4a1645-ae6e-45c3-82ad-161510e62aa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6827","nonce":46066685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6829","nonce":46066685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6833","nonce":46066685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6831","nonce":46066685}]},"public_addr":"172.21.15.32:6827/46066685","cluster_addr":"172.21.15.32:6829/46066685","heartbeat_back_addr":"172.21.15.32:6833/46066685","heartbeat_front_addr":"172.21.15.32:6831/46066685","state":["exists","up"]},{"osd":10,"uuid":"25b26a38-a644-409f-b941-59efdf04a3ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6825","nonce":1886290293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6827","nonce":1886290293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6831","nonce":1886290293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6829","nonce":1886290293}]},"public_addr":"172.21.15.71:6825/1886290293","cluster_addr":"172.21.15.71:6827/1886290293","heartbeat_back_addr":"172.21.15.71:6831/1886290293","heartbeat_front_addr":"172.21.15.71:6829/1886290293","state":["exists","up"]},{"osd":11,"uuid":"ab96b3ff-3688-455b-af64-60226c5bfbff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6825","nonce":850185001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6827","nonce":850185001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6831","nonce":850185001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6829","nonce":850185001}]},"public_addr":"172.21.15.189:6825/850185001","cluster_addr":"172.21.15.189:6827/850185001","heartbeat_back_addr":"172.21.15.189:6831/850185001","heartbeat_front_addr":"172.21.15.189:6829/850185001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:27.220308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:53.230783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:17.190642+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:52.603799+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:20.568671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:46.158363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:22.706933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:50.665173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:16.602410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:53.814698+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:22.435189+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:48.651949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1266897222":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/2915689533":"2024-05-01T08:23:29.526664+0000","172.21.15.32:6801/4019351306":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/425836193":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/1134075722":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6801/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/21059253":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/3245550894":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/4019351306":"2024-05-01T08:23:29.526664+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-30T08:32:15.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.018+0000 7f885dc00640 1 -- 172.21.15.32:0/1095698282 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f883c0afae0 msgr2=0x7f883c0b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:15.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.018+0000 7f885dc00640 1 --2- 172.21.15.32:0/1095698282 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f883c0afae0 0x7f883c0b1ff0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f884802a9e0 tx=0x7f88480314a0 comp rx=0 tx=0).stop 2024-04-30T08:32:15.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.018+0000 7f885dc00640 1 -- 172.21.15.32:0/1095698282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8858106dd0 msgr2=0x7f885819c7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:15.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.018+0000 7f885dc00640 1 --2- 172.21.15.32:0/1095698282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8858106dd0 0x7f885819c7a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f884000d950 tx=0x7f884000de10 comp rx=0 tx=0).stop 2024-04-30T08:32:15.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.018+0000 7f885dc00640 1 -- 172.21.15.32:0/1095698282 shutdown_connections 2024-04-30T08:32:15.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.018+0000 7f885dc00640 1 --2- 172.21.15.32:0/1095698282 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f883c0afae0 0x7f883c0b1ff0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:15.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.018+0000 7f885dc00640 1 --2- 172.21.15.32:0/1095698282 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f885810bad0 0x7f88581a3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:15.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.018+0000 7f885dc00640 1 --2- 172.21.15.32:0/1095698282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8858106dd0 0x7f885819c7a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:15.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.018+0000 7f885dc00640 1 --2- 172.21.15.32:0/1095698282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8858105b90 0x7f885819c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:15.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.018+0000 7f885dc00640 1 -- 172.21.15.32:0/1095698282 >> 172.21.15.32:0/1095698282 conn(0x7f8858100da0 msgr2=0x7f8858107ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:15.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.018+0000 7f885dc00640 1 -- 172.21.15.32:0/1095698282 shutdown_connections 2024-04-30T08:32:15.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.018+0000 7f885dc00640 1 -- 172.21.15.32:0/1095698282 wait complete. 2024-04-30T08:32:15.045 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-30T08:26:19.728331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-30T08:31:49.800542+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-30T08:31:51.211320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-30T08:31:55.716625+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-30T08:32:15.046 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-30T08:32:15.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:14 smithi189 ceph-mon[33038]: pgmap v333: 129 pgs: 129 active+clean; 579 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-04-30T08:32:15.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:14 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2459241435' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-30T08:32:15.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:14 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/241051266' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-30T08:32:15.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:14 smithi071 ceph-mon[33385]: pgmap v333: 129 pgs: 129 active+clean; 579 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 170 B/s wr, 0 op/s 2024-04-30T08:32:15.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:14 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2459241435' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-30T08:32:15.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:14 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/241051266' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-30T08:32:15.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.337+0000 7f8ee5000640 1 -- 172.21.15.32:0/974930521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ee0103990 msgr2=0x7f8ee0104690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:15.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.337+0000 7f8ee5000640 1 --2- 172.21.15.32:0/974930521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ee0103990 0x7f8ee0104690 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0009ca0 tx=0x7f8ed002a8c0 comp rx=0 tx=0).stop 2024-04-30T08:32:15.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.337+0000 7f8ee5000640 1 -- 172.21.15.32:0/974930521 shutdown_connections 2024-04-30T08:32:15.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.337+0000 7f8ee5000640 1 --2- 172.21.15.32:0/974930521 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ee01098d0 0x7f8ee010bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:15.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.337+0000 7f8ee5000640 1 --2- 172.21.15.32:0/974930521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ee0104bd0 0x7f8ee0104fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:15.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.337+0000 7f8ee5000640 1 --2- 172.21.15.32:0/974930521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ee0103990 0x7f8ee0104690 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:15.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.337+0000 7f8ee5000640 1 -- 172.21.15.32:0/974930521 >> 172.21.15.32:0/974930521 conn(0x7f8ee00feba0 msgr2=0x7f8ee0101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:15.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.337+0000 7f8ee5000640 1 -- 172.21.15.32:0/974930521 shutdown_connections 2024-04-30T08:32:15.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.337+0000 7f8ee5000640 1 -- 172.21.15.32:0/974930521 wait complete. 2024-04-30T08:32:15.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.338+0000 7f8ee5000640 1 Processor -- start 2024-04-30T08:32:15.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.338+0000 7f8ee5000640 1 -- start start 2024-04-30T08:32:15.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.338+0000 7f8ee5000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ee0103990 0x7f8ee019c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:15.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.339+0000 7f8ee5000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ee0104bd0 0x7f8ee019c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:15.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.339+0000 7f8ee5000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ee01098d0 0x7f8ee01a3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:15.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.339+0000 7f8ee5000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee010f4c0 con 0x7f8ee0104bd0 2024-04-30T08:32:15.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.339+0000 7f8ee5000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee010f340 con 0x7f8ee0103990 2024-04-30T08:32:15.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.339+0000 7f8ee5000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee010f640 con 0x7f8ee01098d0 2024-04-30T08:32:15.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.339+0000 7f8ed6a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ee0104bd0 0x7f8ee019c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:15.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.339+0000 7f8ed6a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ee0104bd0 0x7f8ee019c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42532/0 (socket says 172.21.15.32:42532) 2024-04-30T08:32:15.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.339+0000 7f8ed6a00640 1 -- 172.21.15.32:0/3744513808 learned_addr learned my addr 172.21.15.32:0/3744513808 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:15.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.339+0000 7f8edf400640 1 --2- 172.21.15.32:0/3744513808 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ee0103990 0x7f8ee019c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:15.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.339+0000 7f8edfe00640 1 --2- 172.21.15.32:0/3744513808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ee01098d0 0x7f8ee01a3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:15.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.339+0000 7f8ed6a00640 1 -- 172.21.15.32:0/3744513808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ee01098d0 msgr2=0x7f8ee01a3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:15.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.339+0000 7f8ed6a00640 1 --2- 172.21.15.32:0/3744513808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ee01098d0 0x7f8ee01a3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:15.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.339+0000 7f8ed6a00640 1 -- 172.21.15.32:0/3744513808 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ee0103990 msgr2=0x7f8ee019c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:15.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.339+0000 7f8ed6a00640 1 --2- 172.21.15.32:0/3744513808 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ee0103990 0x7f8ee019c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:15.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.339+0000 7f8ed6a00640 1 -- 172.21.15.32:0/3744513808 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ee01a3e70 con 0x7f8ee0104bd0 2024-04-30T08:32:15.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.340+0000 7f8ed6a00640 1 --2- 172.21.15.32:0/3744513808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ee0104bd0 0x7f8ee019c870 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8ec800e9f0 tx=0x7f8ec800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:15.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.340+0000 7f8edcc00640 1 -- 172.21.15.32:0/3744513808 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec800cd80 con 0x7f8ee0104bd0 2024-04-30T08:32:15.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.340+0000 7f8edcc00640 1 -- 172.21.15.32:0/3744513808 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ec80108e0 con 0x7f8ee0104bd0 2024-04-30T08:32:15.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.340+0000 7f8edcc00640 1 -- 172.21.15.32:0/3744513808 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec8018980 con 0x7f8ee0104bd0 2024-04-30T08:32:15.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.340+0000 7f8ee5000640 1 -- 172.21.15.32:0/3744513808 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ee01a4160 con 0x7f8ee0104bd0 2024-04-30T08:32:15.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.340+0000 7f8ee5000640 1 -- 172.21.15.32:0/3744513808 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ee01a4700 con 0x7f8ee0104bd0 2024-04-30T08:32:15.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.341+0000 7f8ee5000640 1 -- 172.21.15.32:0/3744513808 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ee01a4960 con 0x7f8ee0104bd0 2024-04-30T08:32:15.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.344+0000 7f8edcc00640 1 -- 172.21.15.32:0/3744513808 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f8ec8010430 con 0x7f8ee0104bd0 2024-04-30T08:32:15.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.346+0000 7f8edcc00640 1 --2- 172.21.15.32:0/3744513808 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8ec40afb60 0x7f8ec40b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:15.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.347+0000 7f8edcc00640 1 -- 172.21.15.32:0/3744513808 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f8ec80df5a0 con 0x7f8ee0104bd0 2024-04-30T08:32:15.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.347+0000 7f8edf400640 1 --2- 172.21.15.32:0/3744513808 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8ec40afb60 0x7f8ec40b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:15.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.348+0000 7f8edf400640 1 --2- 172.21.15.32:0/3744513808 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8ec40afb60 0x7f8ec40b2070 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0000f80 tx=0x7f8ed0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:15.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.349+0000 7f8edcc00640 1 -- 172.21.15.32:0/3744513808 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8ec80a78f0 con 0x7f8ee0104bd0 2024-04-30T08:32:15.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.558+0000 7f8ee5000640 1 -- 172.21.15.32:0/3744513808 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8ee01a4f70 con 0x7f8ee0104bd0 2024-04-30T08:32:15.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.559+0000 7f8edcc00640 1 -- 172.21.15.32:0/3744513808 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f8ec80abbf0 con 0x7f8ee0104bd0 2024-04-30T08:32:15.560 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-30T08:32:15.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.562+0000 7f8ee5000640 1 -- 172.21.15.32:0/3744513808 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8ec40afb60 msgr2=0x7f8ec40b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:15.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.562+0000 7f8ee5000640 1 --2- 172.21.15.32:0/3744513808 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8ec40afb60 0x7f8ec40b2070 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f8ed0000f80 tx=0x7f8ed0005fd0 comp rx=0 tx=0).stop 2024-04-30T08:32:15.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.562+0000 7f8ee5000640 1 -- 172.21.15.32:0/3744513808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ee0104bd0 msgr2=0x7f8ee019c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:15.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.562+0000 7f8ee5000640 1 --2- 172.21.15.32:0/3744513808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ee0104bd0 0x7f8ee019c870 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8ec800e9f0 tx=0x7f8ec800eeb0 comp rx=0 tx=0).stop 2024-04-30T08:32:15.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.562+0000 7f8ee5000640 1 -- 172.21.15.32:0/3744513808 shutdown_connections 2024-04-30T08:32:15.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.562+0000 7f8ee5000640 1 --2- 172.21.15.32:0/3744513808 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8ec40afb60 0x7f8ec40b2070 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:15.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.562+0000 7f8ee5000640 1 --2- 172.21.15.32:0/3744513808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ee01098d0 0x7f8ee01a3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:15.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.562+0000 7f8ee5000640 1 --2- 172.21.15.32:0/3744513808 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ee0104bd0 0x7f8ee019c870 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:15.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.562+0000 7f8ee5000640 1 --2- 172.21.15.32:0/3744513808 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ee0103990 0x7f8ee019c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:15.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.562+0000 7f8ee5000640 1 -- 172.21.15.32:0/3744513808 >> 172.21.15.32:0/3744513808 conn(0x7f8ee00feba0 msgr2=0x7f8ee00fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:15.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.563+0000 7f8ee5000640 1 -- 172.21.15.32:0/3744513808 shutdown_connections 2024-04-30T08:32:15.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.563+0000 7f8ee5000640 1 -- 172.21.15.32:0/3744513808 wait complete. 2024-04-30T08:32:15.591 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-30T08:32:15.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.893+0000 7f45e1800640 1 -- 172.21.15.32:0/2429194116 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f45dc10baf0 msgr2=0x7f45dc10df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:15.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.893+0000 7f45e1800640 1 --2- 172.21.15.32:0/2429194116 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f45dc10baf0 0x7f45dc10df10 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f45d4009900 tx=0x7f45d402a3f0 comp rx=0 tx=0).stop 2024-04-30T08:32:15.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.893+0000 7f45e1800640 1 -- 172.21.15.32:0/2429194116 shutdown_connections 2024-04-30T08:32:15.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.893+0000 7f45e1800640 1 --2- 172.21.15.32:0/2429194116 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f45dc10baf0 0x7f45dc10df10 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:15.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.893+0000 7f45e1800640 1 --2- 172.21.15.32:0/2429194116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45dc106df0 0x7f45dc107200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:15.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.893+0000 7f45e1800640 1 --2- 172.21.15.32:0/2429194116 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45dc105bb0 0x7f45dc1068b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:15.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.893+0000 7f45e1800640 1 -- 172.21.15.32:0/2429194116 >> 172.21.15.32:0/2429194116 conn(0x7f45dc100da0 msgr2=0x7f45dc103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:15.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.894+0000 7f45e1800640 1 -- 172.21.15.32:0/2429194116 shutdown_connections 2024-04-30T08:32:15.897 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.894+0000 7f45e1800640 1 -- 172.21.15.32:0/2429194116 wait complete. 2024-04-30T08:32:15.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.895+0000 7f45e1800640 1 Processor -- start 2024-04-30T08:32:15.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.895+0000 7f45e1800640 1 -- start start 2024-04-30T08:32:15.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.895+0000 7f45e1800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f45dc105bb0 0x7f45dc19c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:15.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.895+0000 7f45e1800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45dc106df0 0x7f45dc19c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:15.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.895+0000 7f45e1800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45dc10baf0 0x7f45dc1a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:15.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.895+0000 7f45e1800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f45dc111710 con 0x7f45dc106df0 2024-04-30T08:32:15.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.895+0000 7f45e1800640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f45dc111590 con 0x7f45dc105bb0 2024-04-30T08:32:15.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.895+0000 7f45e1800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f45dc111890 con 0x7f45dc10baf0 2024-04-30T08:32:15.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.896+0000 7f45db400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45dc106df0 0x7f45dc19c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:15.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.896+0000 7f45dbe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f45dc105bb0 0x7f45dc19c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:15.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.896+0000 7f45db400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45dc106df0 0x7f45dc19c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42536/0 (socket says 172.21.15.32:42536) 2024-04-30T08:32:15.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.896+0000 7f45dbe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f45dc105bb0 0x7f45dc19c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:38704/0 (socket says 172.21.15.32:38704) 2024-04-30T08:32:15.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.896+0000 7f45db400640 1 -- 172.21.15.32:0/1241534051 learned_addr learned my addr 172.21.15.32:0/1241534051 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:15.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.896+0000 7f45e0e00640 1 --2- 172.21.15.32:0/1241534051 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45dc10baf0 0x7f45dc1a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:15.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.896+0000 7f45dbe00640 1 -- 172.21.15.32:0/1241534051 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45dc10baf0 msgr2=0x7f45dc1a3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:15.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.896+0000 7f45dbe00640 1 --2- 172.21.15.32:0/1241534051 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45dc10baf0 0x7f45dc1a3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:15.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.896+0000 7f45dbe00640 1 -- 172.21.15.32:0/1241534051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45dc106df0 msgr2=0x7f45dc19c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:15.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.896+0000 7f45dbe00640 1 --2- 172.21.15.32:0/1241534051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45dc106df0 0x7f45dc19c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:15.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.896+0000 7f45dbe00640 1 -- 172.21.15.32:0/1241534051 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45dc1a3dc0 con 0x7f45dc105bb0 2024-04-30T08:32:15.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.896+0000 7f45dbe00640 1 --2- 172.21.15.32:0/1241534051 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f45dc105bb0 0x7f45dc19c280 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f45dc105480 tx=0x7f45cc00ef90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:15.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.897+0000 7f45d8c00640 1 -- 172.21.15.32:0/1241534051 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc003970 con 0x7f45dc105bb0 2024-04-30T08:32:15.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.897+0000 7f45d8c00640 1 -- 172.21.15.32:0/1241534051 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f45cc004030 con 0x7f45dc105bb0 2024-04-30T08:32:15.901 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.897+0000 7f45d8c00640 1 -- 172.21.15.32:0/1241534051 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45cc010690 con 0x7f45dc105bb0 2024-04-30T08:32:15.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.897+0000 7f45e1800640 1 -- 172.21.15.32:0/1241534051 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45dc1a40b0 con 0x7f45dc105bb0 2024-04-30T08:32:15.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.897+0000 7f45e1800640 1 -- 172.21.15.32:0/1241534051 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45dc1a4650 con 0x7f45dc105bb0 2024-04-30T08:32:15.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.898+0000 7f45e1800640 1 -- 172.21.15.32:0/1241534051 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45dc10a900 con 0x7f45dc105bb0 2024-04-30T08:32:15.903 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.901+0000 7f45d8c00640 1 -- 172.21.15.32:0/1241534051 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f45cc003b10 con 0x7f45dc105bb0 2024-04-30T08:32:15.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.902+0000 7f45d8c00640 1 --2- 172.21.15.32:0/1241534051 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f45c00afb60 0x7f45c00b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:15.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.902+0000 7f45d8c00640 1 -- 172.21.15.32:0/1241534051 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f45cc0df640 con 0x7f45dc105bb0 2024-04-30T08:32:15.904 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.903+0000 7f45db400640 1 --2- 172.21.15.32:0/1241534051 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f45c00afb60 0x7f45c00b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:15.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.903+0000 7f45db400640 1 --2- 172.21.15.32:0/1241534051 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f45c00afb60 0x7f45c00b2070 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f45dc19d320 tx=0x7f45c8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:15.905 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:15.904+0000 7f45d8c00640 1 -- 172.21.15.32:0/1241534051 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f45cc0a7f80 con 0x7f45dc105bb0 2024-04-30T08:32:15.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:15 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1095698282' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:32:15.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:15 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3744513808' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:32:16.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:15 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1095698282' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:32:16.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:15 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3744513808' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:32:16.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.111+0000 7f45e1800640 1 -- 172.21.15.32:0/1241534051 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f45dc063820 con 0x7f45dc105bb0 2024-04-30T08:32:16.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.112+0000 7f45d8c00640 1 -- 172.21.15.32:0/1241534051 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) ==== 104+0+11 (secure 0 0 0) 0x7f45cc0a7880 con 0x7f45dc105bb0 2024-04-30T08:32:16.113 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-30T08:32:16.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.114+0000 7f45e1800640 1 -- 172.21.15.32:0/1241534051 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f45c00afb60 msgr2=0x7f45c00b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:16.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.114+0000 7f45e1800640 1 --2- 172.21.15.32:0/1241534051 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f45c00afb60 0x7f45c00b2070 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f45dc19d320 tx=0x7f45c8005f90 comp rx=0 tx=0).stop 2024-04-30T08:32:16.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.114+0000 7f45e1800640 1 -- 172.21.15.32:0/1241534051 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f45dc105bb0 msgr2=0x7f45dc19c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:16.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.114+0000 7f45e1800640 1 --2- 172.21.15.32:0/1241534051 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f45dc105bb0 0x7f45dc19c280 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f45dc105480 tx=0x7f45cc00ef90 comp rx=0 tx=0).stop 2024-04-30T08:32:16.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.114+0000 7f45e1800640 1 -- 172.21.15.32:0/1241534051 shutdown_connections 2024-04-30T08:32:16.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.114+0000 7f45e1800640 1 --2- 172.21.15.32:0/1241534051 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f45c00afb60 0x7f45c00b2070 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:16.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.114+0000 7f45e1800640 1 --2- 172.21.15.32:0/1241534051 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45dc10baf0 0x7f45dc1a3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:16.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.114+0000 7f45e1800640 1 --2- 172.21.15.32:0/1241534051 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f45dc106df0 0x7f45dc19c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:16.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.114+0000 7f45e1800640 1 --2- 172.21.15.32:0/1241534051 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f45dc105bb0 0x7f45dc19c280 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:16.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.114+0000 7f45e1800640 1 -- 172.21.15.32:0/1241534051 >> 172.21.15.32:0/1241534051 conn(0x7f45dc100da0 msgr2=0x7f45dc107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:16.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.115+0000 7f45e1800640 1 -- 172.21.15.32:0/1241534051 shutdown_connections 2024-04-30T08:32:16.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.115+0000 7f45e1800640 1 -- 172.21.15.32:0/1241534051 wait complete. 2024-04-30T08:32:16.145 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-30T08:32:16.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:15 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1095698282' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:32:16.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:15 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3744513808' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:32:16.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.439+0000 7f3ce0c00640 1 -- 172.21.15.32:0/2140810788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cdc103990 msgr2=0x7f3cdc104690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:16.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.440+0000 7f3ce0c00640 1 --2- 172.21.15.32:0/2140810788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cdc103990 0x7f3cdc104690 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc009860 tx=0x7f3ccc02a340 comp rx=0 tx=0).stop 2024-04-30T08:32:16.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.440+0000 7f3ce0c00640 1 -- 172.21.15.32:0/2140810788 shutdown_connections 2024-04-30T08:32:16.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.440+0000 7f3ce0c00640 1 --2- 172.21.15.32:0/2140810788 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cdc1098d0 0x7f3cdc10bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:16.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.440+0000 7f3ce0c00640 1 --2- 172.21.15.32:0/2140810788 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cdc104bd0 0x7f3cdc104fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:16.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.440+0000 7f3ce0c00640 1 --2- 172.21.15.32:0/2140810788 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cdc103990 0x7f3cdc104690 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:16.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.440+0000 7f3ce0c00640 1 -- 172.21.15.32:0/2140810788 >> 172.21.15.32:0/2140810788 conn(0x7f3cdc0feba0 msgr2=0x7f3cdc101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:16.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.440+0000 7f3ce0c00640 1 -- 172.21.15.32:0/2140810788 shutdown_connections 2024-04-30T08:32:16.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.440+0000 7f3ce0c00640 1 -- 172.21.15.32:0/2140810788 wait complete. 2024-04-30T08:32:16.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.441+0000 7f3ce0c00640 1 Processor -- start 2024-04-30T08:32:16.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.441+0000 7f3ce0c00640 1 -- start start 2024-04-30T08:32:16.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.442+0000 7f3ce0c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cdc103990 0x7f3cdc19a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:16.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.442+0000 7f3ce0c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cdc104bd0 0x7f3cdc19a630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:16.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.442+0000 7f3ce0c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cdc1098d0 0x7f3cdc1a15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:16.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.442+0000 7f3ce0c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3cdc10f490 con 0x7f3cdc1098d0 2024-04-30T08:32:16.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.442+0000 7f3ce0c00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f3cdc10f310 con 0x7f3cdc103990 2024-04-30T08:32:16.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.442+0000 7f3ce0c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3cdc10f610 con 0x7f3cdc104bd0 2024-04-30T08:32:16.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.442+0000 7f3cdaa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cdc104bd0 0x7f3cdc19a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:16.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.442+0000 7f3cdaa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cdc104bd0 0x7f3cdc19a630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:37704/0 (socket says 172.21.15.32:37704) 2024-04-30T08:32:16.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.442+0000 7f3cdaa00640 1 -- 172.21.15.32:0/1913973029 learned_addr learned my addr 172.21.15.32:0/1913973029 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:16.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.442+0000 7f3cdb400640 1 --2- 172.21.15.32:0/1913973029 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cdc103990 0x7f3cdc19a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:16.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.442+0000 7f3cdbe00640 1 --2- 172.21.15.32:0/1913973029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cdc1098d0 0x7f3cdc1a15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:16.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.443+0000 7f3cdaa00640 1 -- 172.21.15.32:0/1913973029 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cdc103990 msgr2=0x7f3cdc19a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:16.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.443+0000 7f3cdaa00640 1 --2- 172.21.15.32:0/1913973029 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cdc103990 0x7f3cdc19a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:16.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.443+0000 7f3cdaa00640 1 -- 172.21.15.32:0/1913973029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cdc1098d0 msgr2=0x7f3cdc1a15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:16.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.443+0000 7f3cdaa00640 1 --2- 172.21.15.32:0/1913973029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cdc1098d0 0x7f3cdc1a15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:16.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.443+0000 7f3cdaa00640 1 -- 172.21.15.32:0/1913973029 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cdc1a1c30 con 0x7f3cdc104bd0 2024-04-30T08:32:16.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.443+0000 7f3cdaa00640 1 --2- 172.21.15.32:0/1913973029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cdc104bd0 0x7f3cdc19a630 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3cc800e9f0 tx=0x7f3cc800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:16.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.443+0000 7f3cd3e00640 1 -- 172.21.15.32:0/1913973029 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc800cd80 con 0x7f3cdc104bd0 2024-04-30T08:32:16.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.443+0000 7f3cd3e00640 1 -- 172.21.15.32:0/1913973029 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3cc80054d0 con 0x7f3cdc104bd0 2024-04-30T08:32:16.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.443+0000 7f3cd3e00640 1 -- 172.21.15.32:0/1913973029 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3cc8010690 con 0x7f3cdc104bd0 2024-04-30T08:32:16.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.443+0000 7f3ce0c00640 1 -- 172.21.15.32:0/1913973029 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cdc1a1f20 con 0x7f3cdc104bd0 2024-04-30T08:32:16.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.444+0000 7f3ce0c00640 1 -- 172.21.15.32:0/1913973029 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cdc1a2330 con 0x7f3cdc104bd0 2024-04-30T08:32:16.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.445+0000 7f3ce0c00640 1 -- 172.21.15.32:0/1913973029 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cdc1086e0 con 0x7f3cdc104bd0 2024-04-30T08:32:16.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.448+0000 7f3cd3e00640 1 -- 172.21.15.32:0/1913973029 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f3cc8004e30 con 0x7f3cdc104bd0 2024-04-30T08:32:16.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.450+0000 7f3cd3e00640 1 --2- 172.21.15.32:0/1913973029 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3cc00afae0 0x7f3cc00b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:16.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.450+0000 7f3cd3e00640 1 -- 172.21.15.32:0/1913973029 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f3cc80df520 con 0x7f3cdc104bd0 2024-04-30T08:32:16.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.450+0000 7f3cdb400640 1 --2- 172.21.15.32:0/1913973029 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3cc00afae0 0x7f3cc00b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:16.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.451+0000 7f3cdb400640 1 --2- 172.21.15.32:0/1913973029 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3cc00afae0 0x7f3cc00b1ff0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc009830 tx=0x7f3ccc030450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:16.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.451+0000 7f3cd3e00640 1 -- 172.21.15.32:0/1913973029 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3cc8053ff0 con 0x7f3cdc104bd0 2024-04-30T08:32:16.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.657+0000 7f3ce0c00640 1 -- 172.21.15.32:0/1913973029 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3cdc063850 con 0x7f3cdc104bd0 2024-04-30T08:32:16.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.658+0000 7f3cd3e00640 1 -- 172.21.15.32:0/1913973029 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) ==== 100+0+11 (secure 0 0 0) 0x7f3cc80a7e60 con 0x7f3cdc104bd0 2024-04-30T08:32:16.660 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-04-30T08:32:16.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.661+0000 7f3ce0c00640 1 -- 172.21.15.32:0/1913973029 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3cc00afae0 msgr2=0x7f3cc00b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:16.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.661+0000 7f3ce0c00640 1 --2- 172.21.15.32:0/1913973029 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3cc00afae0 0x7f3cc00b1ff0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3ccc009830 tx=0x7f3ccc030450 comp rx=0 tx=0).stop 2024-04-30T08:32:16.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.661+0000 7f3ce0c00640 1 -- 172.21.15.32:0/1913973029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cdc104bd0 msgr2=0x7f3cdc19a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:16.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.661+0000 7f3ce0c00640 1 --2- 172.21.15.32:0/1913973029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cdc104bd0 0x7f3cdc19a630 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3cc800e9f0 tx=0x7f3cc800edf0 comp rx=0 tx=0).stop 2024-04-30T08:32:16.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.661+0000 7f3ce0c00640 1 -- 172.21.15.32:0/1913973029 shutdown_connections 2024-04-30T08:32:16.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.661+0000 7f3ce0c00640 1 --2- 172.21.15.32:0/1913973029 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3cc00afae0 0x7f3cc00b1ff0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:16.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.661+0000 7f3ce0c00640 1 --2- 172.21.15.32:0/1913973029 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3cdc1098d0 0x7f3cdc1a15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:16.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.661+0000 7f3ce0c00640 1 --2- 172.21.15.32:0/1913973029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3cdc104bd0 0x7f3cdc19a630 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:16.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.661+0000 7f3ce0c00640 1 --2- 172.21.15.32:0/1913973029 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3cdc103990 0x7f3cdc19a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:16.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.661+0000 7f3ce0c00640 1 -- 172.21.15.32:0/1913973029 >> 172.21.15.32:0/1913973029 conn(0x7f3cdc0feba0 msgr2=0x7f3cdc1058b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:16.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.662+0000 7f3ce0c00640 1 -- 172.21.15.32:0/1913973029 shutdown_connections 2024-04-30T08:32:16.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.662+0000 7f3ce0c00640 1 -- 172.21.15.32:0/1913973029 wait complete. 2024-04-30T08:32:16.690 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-30T08:32:16.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:16 smithi032 ceph-mon[28331]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-30T08:32:16.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:16 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1241534051' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-30T08:32:16.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:16 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1913973029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-30T08:32:16.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.979+0000 7f4211800640 1 -- 172.21.15.32:0/2425388378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f420c10c700 msgr2=0x7f420c10d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:16.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.979+0000 7f4211800640 1 --2- 172.21.15.32:0/2425388378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f420c10c700 0x7f420c10d5f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f41f8009ca0 tx=0x7f41f802a8c0 comp rx=0 tx=0).stop 2024-04-30T08:32:16.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.979+0000 7f4211800640 1 -- 172.21.15.32:0/2425388378 shutdown_connections 2024-04-30T08:32:16.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.979+0000 7f4211800640 1 --2- 172.21.15.32:0/2425388378 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f420c1122c0 0x7f420c1146e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:16.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.979+0000 7f4211800640 1 --2- 172.21.15.32:0/2425388378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f420c10c700 0x7f420c10d5f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:16.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.979+0000 7f4211800640 1 --2- 172.21.15.32:0/2425388378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f420c10c240 0x7f420c10cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:16.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.979+0000 7f4211800640 1 -- 172.21.15.32:0/2425388378 >> 172.21.15.32:0/2425388378 conn(0x7f420c072770 msgr2=0x7f420c109c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:16.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.979+0000 7f4211800640 1 -- 172.21.15.32:0/2425388378 shutdown_connections 2024-04-30T08:32:16.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.980+0000 7f4211800640 1 -- 172.21.15.32:0/2425388378 wait complete. 2024-04-30T08:32:16.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.981+0000 7f4211800640 1 Processor -- start 2024-04-30T08:32:16.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.981+0000 7f4211800640 1 -- start start 2024-04-30T08:32:16.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.981+0000 7f4211800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f420c10c240 0x7f420c06e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:16.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.981+0000 7f4211800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f420c1122c0 0x7f420c06c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:16.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.981+0000 7f4211800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f420c06cda0 0x7f420c06d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:16.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.981+0000 7f4211800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f420c117f00 con 0x7f420c1122c0 2024-04-30T08:32:16.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.981+0000 7f4211800640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f420c117d80 con 0x7f420c06cda0 2024-04-30T08:32:16.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.981+0000 7f4211800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f420c118080 con 0x7f420c10c240 2024-04-30T08:32:16.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.981+0000 7f420a000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f420c1122c0 0x7f420c06c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:16.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.981+0000 7f420aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f420c10c240 0x7f420c06e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:16.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.982+0000 7f420a000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f420c1122c0 0x7f420c06c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41992/0 (socket says 172.21.15.32:41992) 2024-04-30T08:32:16.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.982+0000 7f420a000640 1 -- 172.21.15.32:0/1159561874 learned_addr learned my addr 172.21.15.32:0/1159561874 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:16.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.982+0000 7f420b400640 1 --2- 172.21.15.32:0/1159561874 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f420c06cda0 0x7f420c06d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:16.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.982+0000 7f420aa00640 1 -- 172.21.15.32:0/1159561874 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f420c06cda0 msgr2=0x7f420c06d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:16.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.982+0000 7f420aa00640 1 --2- 172.21.15.32:0/1159561874 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f420c06cda0 0x7f420c06d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:16.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.982+0000 7f420aa00640 1 -- 172.21.15.32:0/1159561874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f420c1122c0 msgr2=0x7f420c06c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:16.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.982+0000 7f420aa00640 1 --2- 172.21.15.32:0/1159561874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f420c1122c0 0x7f420c06c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:16.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.982+0000 7f420aa00640 1 -- 172.21.15.32:0/1159561874 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f420c1ac7f0 con 0x7f420c10c240 2024-04-30T08:32:16.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.982+0000 7f420aa00640 1 --2- 172.21.15.32:0/1159561874 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f420c10c240 0x7f420c06e1c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f420c10cc00 tx=0x7f41fc00ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:16.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.983+0000 7f4203400640 1 -- 172.21.15.32:0/1159561874 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41fc00cc40 con 0x7f420c10c240 2024-04-30T08:32:16.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.983+0000 7f4203400640 1 -- 172.21.15.32:0/1159561874 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41fc00cde0 con 0x7f420c10c240 2024-04-30T08:32:16.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.983+0000 7f4203400640 1 -- 172.21.15.32:0/1159561874 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41fc018780 con 0x7f420c10c240 2024-04-30T08:32:16.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.983+0000 7f4211800640 1 -- 172.21.15.32:0/1159561874 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f420c1acae0 con 0x7f420c10c240 2024-04-30T08:32:16.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.983+0000 7f4211800640 1 -- 172.21.15.32:0/1159561874 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f420c1ad080 con 0x7f420c10c240 2024-04-30T08:32:16.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.984+0000 7f4211800640 1 -- 172.21.15.32:0/1159561874 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f420c1110d0 con 0x7f420c10c240 2024-04-30T08:32:16.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.986+0000 7f4203400640 1 -- 172.21.15.32:0/1159561874 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f41fc010a90 con 0x7f420c10c240 2024-04-30T08:32:16.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.988+0000 7f4203400640 1 --2- 172.21.15.32:0/1159561874 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f41f00afb70 0x7f41f00b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:16.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.988+0000 7f4203400640 1 -- 172.21.15.32:0/1159561874 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f41fc0df5e0 con 0x7f420c10c240 2024-04-30T08:32:16.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.988+0000 7f420a000640 1 --2- 172.21.15.32:0/1159561874 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f41f00afb70 0x7f41f00b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:16.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.989+0000 7f420a000640 1 --2- 172.21.15.32:0/1159561874 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f41f00afb70 0x7f41f00b2080 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f41f8000f80 tx=0x7f41f8031af0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:16.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:16.990+0000 7f4203400640 1 -- 172.21.15.32:0/1159561874 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f41fc0a78d0 con 0x7f420c10c240 2024-04-30T08:32:17.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:16 smithi189 ceph-mon[33038]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-30T08:32:17.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:16 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1241534051' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-30T08:32:17.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:16 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1913973029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-30T08:32:17.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.217+0000 7f4211800640 1 -- 172.21.15.32:0/1159561874 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f420c02de20 con 0x7f420c10c240 2024-04-30T08:32:17.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.218+0000 7f4203400640 1 -- 172.21.15.32:0/1159561874 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v81) ==== 103+0+11 (secure 0 0 0) 0x7f41fc0abbd0 con 0x7f420c10c240 2024-04-30T08:32:17.220 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-30T08:32:17.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.221+0000 7f4211800640 1 -- 172.21.15.32:0/1159561874 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f41f00afb70 msgr2=0x7f41f00b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:17.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.221+0000 7f4211800640 1 --2- 172.21.15.32:0/1159561874 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f41f00afb70 0x7f41f00b2080 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f41f8000f80 tx=0x7f41f8031af0 comp rx=0 tx=0).stop 2024-04-30T08:32:17.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.221+0000 7f4211800640 1 -- 172.21.15.32:0/1159561874 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f420c10c240 msgr2=0x7f420c06e1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:17.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.221+0000 7f4211800640 1 --2- 172.21.15.32:0/1159561874 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f420c10c240 0x7f420c06e1c0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f420c10cc00 tx=0x7f41fc00ee50 comp rx=0 tx=0).stop 2024-04-30T08:32:17.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.222+0000 7f4211800640 1 -- 172.21.15.32:0/1159561874 shutdown_connections 2024-04-30T08:32:17.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.222+0000 7f4211800640 1 --2- 172.21.15.32:0/1159561874 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f41f00afb70 0x7f41f00b2080 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:17.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.222+0000 7f4211800640 1 --2- 172.21.15.32:0/1159561874 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f420c06cda0 0x7f420c06d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:17.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.222+0000 7f4211800640 1 --2- 172.21.15.32:0/1159561874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f420c1122c0 0x7f420c06c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:17.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.222+0000 7f4211800640 1 --2- 172.21.15.32:0/1159561874 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f420c10c240 0x7f420c06e1c0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:17.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.222+0000 7f4211800640 1 -- 172.21.15.32:0/1159561874 >> 172.21.15.32:0/1159561874 conn(0x7f420c072770 msgr2=0x7f420c10e2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:17.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.222+0000 7f4211800640 1 -- 172.21.15.32:0/1159561874 shutdown_connections 2024-04-30T08:32:17.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.222+0000 7f4211800640 1 -- 172.21.15.32:0/1159561874 wait complete. 2024-04-30T08:32:17.248 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-30T08:32:17.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:16 smithi071 ceph-mon[33385]: pgmap v334: 129 pgs: 129 active+clean; 581 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-30T08:32:17.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:16 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1241534051' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-30T08:32:17.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:16 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1913973029' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-30T08:32:17.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.515+0000 7f376f200640 1 -- 172.21.15.32:0/3934388207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3770103990 msgr2=0x7f3770104690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:17.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.515+0000 7f376f200640 1 --2- 172.21.15.32:0/3934388207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3770103990 0x7f3770104690 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f3760009900 tx=0x7f376002a3f0 comp rx=0 tx=0).stop 2024-04-30T08:32:17.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.515+0000 7f376f200640 1 -- 172.21.15.32:0/3934388207 shutdown_connections 2024-04-30T08:32:17.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.515+0000 7f376f200640 1 --2- 172.21.15.32:0/3934388207 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37701098d0 0x7f377010bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:17.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.515+0000 7f376f200640 1 --2- 172.21.15.32:0/3934388207 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3770104bd0 0x7f3770104fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:17.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.515+0000 7f376f200640 1 --2- 172.21.15.32:0/3934388207 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3770103990 0x7f3770104690 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:17.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.515+0000 7f376f200640 1 -- 172.21.15.32:0/3934388207 >> 172.21.15.32:0/3934388207 conn(0x7f37700feba0 msgr2=0x7f3770101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:17.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.516+0000 7f376f200640 1 -- 172.21.15.32:0/3934388207 shutdown_connections 2024-04-30T08:32:17.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.516+0000 7f376f200640 1 -- 172.21.15.32:0/3934388207 wait complete. 2024-04-30T08:32:17.518 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.517+0000 7f376f200640 1 Processor -- start 2024-04-30T08:32:17.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.517+0000 7f376f200640 1 -- start start 2024-04-30T08:32:17.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.518+0000 7f376f200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3770103990 0x7f37700768d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:17.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.518+0000 7f376f200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3770104bd0 0x7f3770074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:17.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.518+0000 7f376f200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37701098d0 0x7f37700754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:17.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.518+0000 7f376f200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f377010f470 con 0x7f37701098d0 2024-04-30T08:32:17.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.518+0000 7f376f200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f377010f2f0 con 0x7f3770103990 2024-04-30T08:32:17.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.518+0000 7f376f200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f377010f5f0 con 0x7f3770104bd0 2024-04-30T08:32:17.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.518+0000 7f376e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37701098d0 0x7f37700754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:17.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.518+0000 7f376de00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3770103990 0x7f37700768d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:17.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.518+0000 7f376de00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3770103990 0x7f37700768d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:41224/0 (socket says 172.21.15.32:41224) 2024-04-30T08:32:17.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.518+0000 7f376de00640 1 -- 172.21.15.32:0/1553296871 learned_addr learned my addr 172.21.15.32:0/1553296871 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:17.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.518+0000 7f376d400640 1 --2- 172.21.15.32:0/1553296871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3770104bd0 0x7f3770074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:17.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.519+0000 7f376de00640 1 -- 172.21.15.32:0/1553296871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3770104bd0 msgr2=0x7f3770074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:17.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.519+0000 7f376de00640 1 --2- 172.21.15.32:0/1553296871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3770104bd0 0x7f3770074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:17.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.519+0000 7f376de00640 1 -- 172.21.15.32:0/1553296871 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37701098d0 msgr2=0x7f37700754b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:17.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.519+0000 7f376de00640 1 --2- 172.21.15.32:0/1553296871 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37701098d0 0x7f37700754b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:17.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.519+0000 7f376de00640 1 -- 172.21.15.32:0/1553296871 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3770075be0 con 0x7f3770103990 2024-04-30T08:32:17.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.519+0000 7f376de00640 1 --2- 172.21.15.32:0/1553296871 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3770103990 0x7f37700768d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3760006010 tx=0x7f3760004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:17.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.520+0000 7f3766a00640 1 -- 172.21.15.32:0/1553296871 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f376003f030 con 0x7f3770103990 2024-04-30T08:32:17.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.520+0000 7f376f200640 1 -- 172.21.15.32:0/1553296871 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37701a4190 con 0x7f3770103990 2024-04-30T08:32:17.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.520+0000 7f3766a00640 1 -- 172.21.15.32:0/1553296871 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3760003aa0 con 0x7f3770103990 2024-04-30T08:32:17.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.520+0000 7f3766a00640 1 -- 172.21.15.32:0/1553296871 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3760035070 con 0x7f3770103990 2024-04-30T08:32:17.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.520+0000 7f376f200640 1 -- 172.21.15.32:0/1553296871 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37701a45f0 con 0x7f3770103990 2024-04-30T08:32:17.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.521+0000 7f376f200640 1 -- 172.21.15.32:0/1553296871 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37701086e0 con 0x7f3770103990 2024-04-30T08:32:17.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.523+0000 7f3766a00640 1 -- 172.21.15.32:0/1553296871 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f3760042030 con 0x7f3770103990 2024-04-30T08:32:17.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.525+0000 7f3766a00640 1 --2- 172.21.15.32:0/1553296871 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f37540afae0 0x7f37540b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:17.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.525+0000 7f3766a00640 1 -- 172.21.15.32:0/1553296871 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f37600ff8a0 con 0x7f3770103990 2024-04-30T08:32:17.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.525+0000 7f376d400640 1 --2- 172.21.15.32:0/1553296871 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f37540afae0 0x7f37540b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:17.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.526+0000 7f376d400640 1 --2- 172.21.15.32:0/1553296871 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f37540afae0 0x7f37540b1ff0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3770076270 tx=0x7f375c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:17.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.527+0000 7f3766a00640 1 -- 172.21.15.32:0/1553296871 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f37600c7b90 con 0x7f3770103990 2024-04-30T08:32:17.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.740+0000 7f376f200640 1 -- 172.21.15.32:0/1553296871 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f377002dbd0 con 0x7f3770103990 2024-04-30T08:32:17.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.741+0000 7f3766a00640 1 -- 172.21.15.32:0/1553296871 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v81) ==== 81+0+587 (secure 0 0 0) 0x7f37600cbe90 con 0x7f3770103990 2024-04-30T08:32:17.743 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/1266897222 2024-05-01T08:23:29.526664+0000 2024-04-30T08:32:17.743 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/2915689533 2024-05-01T08:23:29.526664+0000 2024-04-30T08:32:17.743 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6801/4019351306 2024-05-01T08:23:29.526664+0000 2024-04-30T08:32:17.743 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/425836193 2024-05-01T08:23:29.526664+0000 2024-04-30T08:32:17.743 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/1134075722 2024-05-01T08:22:53.639893+0000 2024-04-30T08:32:17.743 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6800/464872591 2024-05-01T08:22:53.639893+0000 2024-04-30T08:32:17.743 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6801/464872591 2024-05-01T08:22:53.639893+0000 2024-04-30T08:32:17.743 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/21059253 2024-05-01T08:22:53.639893+0000 2024-04-30T08:32:17.743 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/3245550894 2024-05-01T08:22:53.639893+0000 2024-04-30T08:32:17.743 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6800/4019351306 2024-05-01T08:23:29.526664+0000 2024-04-30T08:32:17.743 INFO:teuthology.orchestra.run.smithi032.stderr:listed 10 entries 2024-04-30T08:32:17.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.743+0000 7f376f200640 1 -- 172.21.15.32:0/1553296871 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f37540afae0 msgr2=0x7f37540b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:17.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.743+0000 7f376f200640 1 --2- 172.21.15.32:0/1553296871 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f37540afae0 0x7f37540b1ff0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3770076270 tx=0x7f375c00b390 comp rx=0 tx=0).stop 2024-04-30T08:32:17.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.743+0000 7f376f200640 1 -- 172.21.15.32:0/1553296871 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3770103990 msgr2=0x7f37700768d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:17.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.743+0000 7f376f200640 1 --2- 172.21.15.32:0/1553296871 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3770103990 0x7f37700768d0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3760006010 tx=0x7f3760004270 comp rx=0 tx=0).stop 2024-04-30T08:32:17.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.743+0000 7f376f200640 1 -- 172.21.15.32:0/1553296871 shutdown_connections 2024-04-30T08:32:17.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.743+0000 7f376f200640 1 --2- 172.21.15.32:0/1553296871 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f37540afae0 0x7f37540b1ff0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:17.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.743+0000 7f376f200640 1 --2- 172.21.15.32:0/1553296871 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f37701098d0 0x7f37700754b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:17.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.743+0000 7f376f200640 1 --2- 172.21.15.32:0/1553296871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3770104bd0 0x7f3770074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:17.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.743+0000 7f376f200640 1 --2- 172.21.15.32:0/1553296871 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3770103990 0x7f37700768d0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:17.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.744+0000 7f376f200640 1 -- 172.21.15.32:0/1553296871 >> 172.21.15.32:0/1553296871 conn(0x7f37700feba0 msgr2=0x7f37701058b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:17.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.744+0000 7f376f200640 1 -- 172.21.15.32:0/1553296871 shutdown_connections 2024-04-30T08:32:17.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:17.744+0000 7f376f200640 1 -- 172.21.15.32:0/1553296871 wait complete. 2024-04-30T08:32:17.769 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi032.front.sepia.ceph.com... 2024-04-30T08:32:17.769 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-30T08:32:17.769 DEBUG:teuthology.orchestra.run.smithi032:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-04-30T08:32:17.852 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi032.stderr:ceph-fuse[72631]: fuse finished with error 0 and tester_r 0 2024-04-30T08:32:17.859 INFO:teuthology.orchestra.run:waiting for 300 2024-04-30T08:32:18.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:17 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1159561874' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-30T08:32:18.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:17 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1553296871' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-30T08:32:18.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:17 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1159561874' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-30T08:32:18.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:17 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1553296871' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-30T08:32:18.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:17 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1159561874' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-30T08:32:18.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:17 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1553296871' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-30T08:32:19.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:18 smithi189 ceph-mon[33038]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-30T08:32:19.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:18 smithi032 ceph-mon[28331]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-30T08:32:19.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:18 smithi071 ceph-mon[33385]: pgmap v335: 129 pgs: 129 active+clean; 581 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 341 B/s wr, 0 op/s 2024-04-30T08:32:21.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:20 smithi032 ceph-mon[28331]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-30T08:32:21.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:20 smithi071 ceph-mon[33385]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-30T08:32:21.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:20 smithi189 ceph-mon[33038]: pgmap v336: 129 pgs: 129 active+clean; 581 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-04-30T08:32:22.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:22 smithi071 ceph-mon[33385]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-30T08:32:22.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:22 smithi189 ceph-mon[33038]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-30T08:32:22.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:22 smithi032 ceph-mon[28331]: pgmap v337: 129 pgs: 129 active+clean; 581 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-30T08:32:23.961 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-04-30T08:32:23.961 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-30T08:32:23.961 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-04-30T08:32:24.034 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-30T08:32:24.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.327+0000 7fc42a600640 1 -- 172.21.15.32:0/3528586669 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc42407aa30 msgr2=0x7fc42407ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:24.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.327+0000 7fc42a600640 1 --2- 172.21.15.32:0/3528586669 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc42407aa30 0x7fc42407ae40 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc41c009900 tx=0x7fc41c02a410 comp rx=0 tx=0).stop 2024-04-30T08:32:24.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.328+0000 7fc42a600640 1 -- 172.21.15.32:0/3528586669 shutdown_connections 2024-04-30T08:32:24.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.328+0000 7fc42a600640 1 --2- 172.21.15.32:0/3528586669 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc42407aa30 0x7fc42407ae40 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:24.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.328+0000 7fc42a600640 1 --2- 172.21.15.32:0/3528586669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc424079d30 0x7fc42407a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:24.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.328+0000 7fc42a600640 1 --2- 172.21.15.32:0/3528586669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc424078aa0 0x7fc4240797f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:24.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.328+0000 7fc42a600640 1 -- 172.21.15.32:0/3528586669 >> 172.21.15.32:0/3528586669 conn(0x7fc424100f90 msgr2=0x7fc424103440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:24.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.328+0000 7fc42a600640 1 -- 172.21.15.32:0/3528586669 shutdown_connections 2024-04-30T08:32:24.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.328+0000 7fc42a600640 1 -- 172.21.15.32:0/3528586669 wait complete. 2024-04-30T08:32:24.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.329+0000 7fc42a600640 1 Processor -- start 2024-04-30T08:32:24.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.329+0000 7fc42a600640 1 -- start start 2024-04-30T08:32:24.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.329+0000 7fc42a600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc424078aa0 0x7fc4241a08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:24.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.329+0000 7fc42a600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc424079d30 0x7fc4241a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:24.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.329+0000 7fc42a600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc42407aa30 0x7fc4241a7d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:24.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.329+0000 7fc42a600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc424115d00 con 0x7fc424078aa0 2024-04-30T08:32:24.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.329+0000 7fc42a600640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc424115b80 con 0x7fc42407aa30 2024-04-30T08:32:24.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.329+0000 7fc42a600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc424115e80 con 0x7fc424079d30 2024-04-30T08:32:24.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.330+0000 7fc41aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc424079d30 0x7fc4241a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:24.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.330+0000 7fc423400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc424078aa0 0x7fc4241a08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:24.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.330+0000 7fc41aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc424079d30 0x7fc4241a0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:44492/0 (socket says 172.21.15.32:44492) 2024-04-30T08:32:24.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.330+0000 7fc41aa00640 1 -- 172.21.15.32:0/3411845220 learned_addr learned my addr 172.21.15.32:0/3411845220 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:24.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.330+0000 7fc423400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc424078aa0 0x7fc4241a08a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42032/0 (socket says 172.21.15.32:42032) 2024-04-30T08:32:24.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.330+0000 7fc423e00640 1 --2- 172.21.15.32:0/3411845220 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc42407aa30 0x7fc4241a7d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:24.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.330+0000 7fc41aa00640 1 -- 172.21.15.32:0/3411845220 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc42407aa30 msgr2=0x7fc4241a7d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:24.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.330+0000 7fc41aa00640 1 --2- 172.21.15.32:0/3411845220 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc42407aa30 0x7fc4241a7d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:24.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.330+0000 7fc41aa00640 1 -- 172.21.15.32:0/3411845220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc424078aa0 msgr2=0x7fc4241a08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:24.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.330+0000 7fc41aa00640 1 --2- 172.21.15.32:0/3411845220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc424078aa0 0x7fc4241a08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:24.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.330+0000 7fc41aa00640 1 -- 172.21.15.32:0/3411845220 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4241a83e0 con 0x7fc424079d30 2024-04-30T08:32:24.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.331+0000 7fc41aa00640 1 --2- 172.21.15.32:0/3411845220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc424079d30 0x7fc4241a0de0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc40c00e9f0 tx=0x7fc40c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:24.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.331+0000 7fc420c00640 1 -- 172.21.15.32:0/3411845220 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc40c00cd80 con 0x7fc424079d30 2024-04-30T08:32:24.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.331+0000 7fc42a600640 1 -- 172.21.15.32:0/3411845220 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4241a86d0 con 0x7fc424079d30 2024-04-30T08:32:24.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.331+0000 7fc420c00640 1 -- 172.21.15.32:0/3411845220 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc40c0108e0 con 0x7fc424079d30 2024-04-30T08:32:24.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.331+0000 7fc420c00640 1 -- 172.21.15.32:0/3411845220 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc40c018980 con 0x7fc424079d30 2024-04-30T08:32:24.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.331+0000 7fc42a600640 1 -- 172.21.15.32:0/3411845220 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4241a8c20 con 0x7fc424079d30 2024-04-30T08:32:24.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.332+0000 7fc42a600640 1 -- 172.21.15.32:0/3411845220 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc424112a70 con 0x7fc424079d30 2024-04-30T08:32:24.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.335+0000 7fc420c00640 1 -- 172.21.15.32:0/3411845220 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fc40c010430 con 0x7fc424079d30 2024-04-30T08:32:24.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.336+0000 7fc420c00640 1 --2- 172.21.15.32:0/3411845220 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc4080afaf0 0x7fc4080b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:24.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.336+0000 7fc420c00640 1 -- 172.21.15.32:0/3411845220 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7fc40c0df670 con 0x7fc424079d30 2024-04-30T08:32:24.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.337+0000 7fc423400640 1 --2- 172.21.15.32:0/3411845220 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc4080afaf0 0x7fc4080b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:24.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.337+0000 7fc423400640 1 --2- 172.21.15.32:0/3411845220 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc4080afaf0 0x7fc4080b2000 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc424078370 tx=0x7fc414009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:24.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.338+0000 7fc420c00640 1 -- 172.21.15.32:0/3411845220 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc40c0a7960 con 0x7fc424079d30 2024-04-30T08:32:24.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.546+0000 7fc42a600640 1 -- 172.21.15.32:0/3411845220 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc424063820 con 0x7fc424079d30 2024-04-30T08:32:24.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.548+0000 7fc420c00640 1 -- 172.21.15.32:0/3411845220 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+24053 (secure 0 0 0) 0x7fc40c0abc60 con 0x7fc424079d30 2024-04-30T08:32:24.551 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:32:24.551 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":81,"fsid":"98c0a670-06ca-11ef-bc93-c7b262605968","created":"2024-04-30T08:22:11.122875+0000","modified":"2024-04-30T08:31:58.298969+0000","last_up_change":"2024-04-30T08:30:50.350439+0000","last_in_change":"2024-04-30T08:30:30.210620+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-30T08:26:19.728331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-30T08:31:49.800542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-30T08:31:51.211320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-30T08:31:55.716625+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a7e060c8-1cbb-418f-9a03-06693f6b3f52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6803","nonce":2624700402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6805","nonce":2624700402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6809","nonce":2624700402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6807","nonce":2624700402}]},"public_addr":"172.21.15.32:6803/2624700402","cluster_addr":"172.21.15.32:6805/2624700402","heartbeat_back_addr":"172.21.15.32:6809/2624700402","heartbeat_front_addr":"172.21.15.32:6807/2624700402","state":["exists","up"]},{"osd":1,"uuid":"72ef2fb1-e5ca-48ef-aa3a-4796a64b922f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6801","nonce":444649711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6803","nonce":444649711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6807","nonce":444649711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6805","nonce":444649711}]},"public_addr":"172.21.15.71:6801/444649711","cluster_addr":"172.21.15.71:6803/444649711","heartbeat_back_addr":"172.21.15.71:6807/444649711","heartbeat_front_addr":"172.21.15.71:6805/444649711","state":["exists","up"]},{"osd":2,"uuid":"704e5e8a-935e-47c7-ae28-27119876ac09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6801","nonce":3255528671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6803","nonce":3255528671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6807","nonce":3255528671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6805","nonce":3255528671}]},"public_addr":"172.21.15.189:6801/3255528671","cluster_addr":"172.21.15.189:6803/3255528671","heartbeat_back_addr":"172.21.15.189:6807/3255528671","heartbeat_front_addr":"172.21.15.189:6805/3255528671","state":["exists","up"]},{"osd":3,"uuid":"575004f7-6559-4a62-863e-e33078b0eebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6811","nonce":1384702112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6813","nonce":1384702112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6817","nonce":1384702112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6815","nonce":1384702112}]},"public_addr":"172.21.15.32:6811/1384702112","cluster_addr":"172.21.15.32:6813/1384702112","heartbeat_back_addr":"172.21.15.32:6817/1384702112","heartbeat_front_addr":"172.21.15.32:6815/1384702112","state":["exists","up"]},{"osd":4,"uuid":"e1c1e894-074d-4c78-89ef-9e5a114268bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6809","nonce":4250534574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6811","nonce":4250534574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6815","nonce":4250534574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6813","nonce":4250534574}]},"public_addr":"172.21.15.71:6809/4250534574","cluster_addr":"172.21.15.71:6811/4250534574","heartbeat_back_addr":"172.21.15.71:6815/4250534574","heartbeat_front_addr":"172.21.15.71:6813/4250534574","state":["exists","up"]},{"osd":5,"uuid":"12c022d4-fac6-40c4-ae7a-691571bee5d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6809","nonce":4245346198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6811","nonce":4245346198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6815","nonce":4245346198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6813","nonce":4245346198}]},"public_addr":"172.21.15.189:6809/4245346198","cluster_addr":"172.21.15.189:6811/4245346198","heartbeat_back_addr":"172.21.15.189:6815/4245346198","heartbeat_front_addr":"172.21.15.189:6813/4245346198","state":["exists","up"]},{"osd":6,"uuid":"b413f8ee-3d4a-415a-a2ba-d68437b7330d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6819","nonce":1910389190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6821","nonce":1910389190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6825","nonce":1910389190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6823","nonce":1910389190}]},"public_addr":"172.21.15.32:6819/1910389190","cluster_addr":"172.21.15.32:6821/1910389190","heartbeat_back_addr":"172.21.15.32:6825/1910389190","heartbeat_front_addr":"172.21.15.32:6823/1910389190","state":["exists","up"]},{"osd":7,"uuid":"aac64fb3-fc13-4126-b842-7b06ad5ce238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6817","nonce":692607380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6819","nonce":692607380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6823","nonce":692607380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6821","nonce":692607380}]},"public_addr":"172.21.15.71:6817/692607380","cluster_addr":"172.21.15.71:6819/692607380","heartbeat_back_addr":"172.21.15.71:6823/692607380","heartbeat_front_addr":"172.21.15.71:6821/692607380","state":["exists","up"]},{"osd":8,"uuid":"cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6817","nonce":3973697069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6819","nonce":3973697069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6823","nonce":3973697069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6821","nonce":3973697069}]},"public_addr":"172.21.15.189:6817/3973697069","cluster_addr":"172.21.15.189:6819/3973697069","heartbeat_back_addr":"172.21.15.189:6823/3973697069","heartbeat_front_addr":"172.21.15.189:6821/3973697069","state":["exists","up"]},{"osd":9,"uuid":"ef4a1645-ae6e-45c3-82ad-161510e62aa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6827","nonce":46066685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6829","nonce":46066685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6833","nonce":46066685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6831","nonce":46066685}]},"public_addr":"172.21.15.32:6827/46066685","cluster_addr":"172.21.15.32:6829/46066685","heartbeat_back_addr":"172.21.15.32:6833/46066685","heartbeat_front_addr":"172.21.15.32:6831/46066685","state":["exists","up"]},{"osd":10,"uuid":"25b26a38-a644-409f-b941-59efdf04a3ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6825","nonce":1886290293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6827","nonce":1886290293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6831","nonce":1886290293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6829","nonce":1886290293}]},"public_addr":"172.21.15.71:6825/1886290293","cluster_addr":"172.21.15.71:6827/1886290293","heartbeat_back_addr":"172.21.15.71:6831/1886290293","heartbeat_front_addr":"172.21.15.71:6829/1886290293","state":["exists","up"]},{"osd":11,"uuid":"ab96b3ff-3688-455b-af64-60226c5bfbff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6825","nonce":850185001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6827","nonce":850185001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6831","nonce":850185001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6829","nonce":850185001}]},"public_addr":"172.21.15.189:6825/850185001","cluster_addr":"172.21.15.189:6827/850185001","heartbeat_back_addr":"172.21.15.189:6831/850185001","heartbeat_front_addr":"172.21.15.189:6829/850185001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:27.220308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:53.230783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:17.190642+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:52.603799+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:20.568671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:46.158363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:22.706933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:50.665173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:16.602410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:53.814698+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:22.435189+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:48.651949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1266897222":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/2915689533":"2024-05-01T08:23:29.526664+0000","172.21.15.32:6801/4019351306":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/425836193":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/1134075722":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6801/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/21059253":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/3245550894":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/4019351306":"2024-05-01T08:23:29.526664+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-30T08:32:24.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.552+0000 7fc42a600640 1 -- 172.21.15.32:0/3411845220 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc4080afaf0 msgr2=0x7fc4080b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:24.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.552+0000 7fc42a600640 1 --2- 172.21.15.32:0/3411845220 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc4080afaf0 0x7fc4080b2000 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc424078370 tx=0x7fc414009500 comp rx=0 tx=0).stop 2024-04-30T08:32:24.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.552+0000 7fc42a600640 1 -- 172.21.15.32:0/3411845220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc424079d30 msgr2=0x7fc4241a0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:24.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.552+0000 7fc42a600640 1 --2- 172.21.15.32:0/3411845220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc424079d30 0x7fc4241a0de0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fc40c00e9f0 tx=0x7fc40c00eeb0 comp rx=0 tx=0).stop 2024-04-30T08:32:24.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.552+0000 7fc42a600640 1 -- 172.21.15.32:0/3411845220 shutdown_connections 2024-04-30T08:32:24.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.552+0000 7fc42a600640 1 --2- 172.21.15.32:0/3411845220 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc4080afaf0 0x7fc4080b2000 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:24.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.552+0000 7fc42a600640 1 --2- 172.21.15.32:0/3411845220 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc42407aa30 0x7fc4241a7d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:24.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.552+0000 7fc42a600640 1 --2- 172.21.15.32:0/3411845220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc424079d30 0x7fc4241a0de0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:24.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.552+0000 7fc42a600640 1 --2- 172.21.15.32:0/3411845220 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc424078aa0 0x7fc4241a08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:24.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.552+0000 7fc42a600640 1 -- 172.21.15.32:0/3411845220 >> 172.21.15.32:0/3411845220 conn(0x7fc424100f90 msgr2=0x7fc42410fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:24.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.552+0000 7fc42a600640 1 -- 172.21.15.32:0/3411845220 shutdown_connections 2024-04-30T08:32:24.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.552+0000 7fc42a600640 1 -- 172.21.15.32:0/3411845220 wait complete. 2024-04-30T08:32:24.581 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-30T08:32:24.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.857+0000 7f6449200640 1 -- 172.21.15.32:0/3292093018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6444105bb0 msgr2=0x7f64441068b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:24.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.857+0000 7f6449200640 1 --2- 172.21.15.32:0/3292093018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6444105bb0 0x7f64441068b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f6434009ca0 tx=0x7f643402a8b0 comp rx=0 tx=0).stop 2024-04-30T08:32:24.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.857+0000 7f6449200640 1 -- 172.21.15.32:0/3292093018 shutdown_connections 2024-04-30T08:32:24.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.857+0000 7f6449200640 1 --2- 172.21.15.32:0/3292093018 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f644410baf0 0x7f644410df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:24.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.857+0000 7f6449200640 1 --2- 172.21.15.32:0/3292093018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6444106df0 0x7f6444107200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:24.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.857+0000 7f6449200640 1 --2- 172.21.15.32:0/3292093018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6444105bb0 0x7f64441068b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:24.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.857+0000 7f6449200640 1 -- 172.21.15.32:0/3292093018 >> 172.21.15.32:0/3292093018 conn(0x7f6444100da0 msgr2=0x7f6444103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:24.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.857+0000 7f6449200640 1 -- 172.21.15.32:0/3292093018 shutdown_connections 2024-04-30T08:32:24.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.857+0000 7f6449200640 1 -- 172.21.15.32:0/3292093018 wait complete. 2024-04-30T08:32:24.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.859+0000 7f6449200640 1 Processor -- start 2024-04-30T08:32:24.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.859+0000 7f6449200640 1 -- start start 2024-04-30T08:32:24.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.859+0000 7f6449200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6444105bb0 0x7f644419c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:24.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.859+0000 7f6449200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6444106df0 0x7f644419c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:24.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.859+0000 7f6449200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f644410baf0 0x7f64441a37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:24.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.859+0000 7f6449200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f64441116e0 con 0x7f644410baf0 2024-04-30T08:32:24.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.859+0000 7f6449200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6444111560 con 0x7f6444106df0 2024-04-30T08:32:24.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.859+0000 7f6449200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6444111860 con 0x7f6444105bb0 2024-04-30T08:32:24.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.859+0000 7f6443400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6444105bb0 0x7f644419c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:24.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.860+0000 7f6443400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6444105bb0 0x7f644419c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:44520/0 (socket says 172.21.15.32:44520) 2024-04-30T08:32:24.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.860+0000 7f6443400640 1 -- 172.21.15.32:0/12575880 learned_addr learned my addr 172.21.15.32:0/12575880 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:24.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.860+0000 7f643aa00640 1 --2- 172.21.15.32:0/12575880 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6444106df0 0x7f644419c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:24.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.860+0000 7f6443e00640 1 --2- 172.21.15.32:0/12575880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f644410baf0 0x7f64441a37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:24.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.860+0000 7f6443400640 1 -- 172.21.15.32:0/12575880 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6444106df0 msgr2=0x7f644419c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:24.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.860+0000 7f6443400640 1 --2- 172.21.15.32:0/12575880 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6444106df0 0x7f644419c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:24.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.860+0000 7f6443400640 1 -- 172.21.15.32:0/12575880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f644410baf0 msgr2=0x7f64441a37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:24.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.860+0000 7f6443400640 1 --2- 172.21.15.32:0/12575880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f644410baf0 0x7f64441a37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:24.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.860+0000 7f6443400640 1 -- 172.21.15.32:0/12575880 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64441a3e20 con 0x7f6444105bb0 2024-04-30T08:32:24.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.860+0000 7f6443400640 1 --2- 172.21.15.32:0/12575880 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6444105bb0 0x7f644419c2e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f643402aea0 tx=0x7f6434003c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:24.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.861+0000 7f6440c00640 1 -- 172.21.15.32:0/12575880 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6434004450 con 0x7f6444105bb0 2024-04-30T08:32:24.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.861+0000 7f6449200640 1 -- 172.21.15.32:0/12575880 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64441a40b0 con 0x7f6444105bb0 2024-04-30T08:32:24.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.861+0000 7f6440c00640 1 -- 172.21.15.32:0/12575880 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6434040070 con 0x7f6444105bb0 2024-04-30T08:32:24.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.861+0000 7f6440c00640 1 -- 172.21.15.32:0/12575880 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6434045b80 con 0x7f6444105bb0 2024-04-30T08:32:24.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.861+0000 7f6449200640 1 -- 172.21.15.32:0/12575880 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64441a4600 con 0x7f6444105bb0 2024-04-30T08:32:24.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.862+0000 7f6449200640 1 -- 172.21.15.32:0/12575880 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f644410a900 con 0x7f6444105bb0 2024-04-30T08:32:24.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.865+0000 7f6440c00640 1 -- 172.21.15.32:0/12575880 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f6434045470 con 0x7f6444105bb0 2024-04-30T08:32:24.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.866+0000 7f6440c00640 1 --2- 172.21.15.32:0/12575880 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f64280afae0 0x7f64280b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:24.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.866+0000 7f6440c00640 1 -- 172.21.15.32:0/12575880 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f64341002c0 con 0x7f6444105bb0 2024-04-30T08:32:24.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.867+0000 7f643aa00640 1 --2- 172.21.15.32:0/12575880 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f64280afae0 0x7f64280b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:24.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.867+0000 7f643aa00640 1 --2- 172.21.15.32:0/12575880 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f64280afae0 0x7f64280b1ff0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f644419d380 tx=0x7f642c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:24.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:24.868+0000 7f6440c00640 1 -- 172.21.15.32:0/12575880 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f64340c8c00 con 0x7f6444105bb0 2024-04-30T08:32:24.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:24 smithi032 ceph-mon[28331]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-30T08:32:24.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:24 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3411845220' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:32:25.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:24 smithi189 ceph-mon[33038]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-30T08:32:25.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:24 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3411845220' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:32:25.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.075+0000 7f6449200640 1 -- 172.21.15.32:0/12575880 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6444063820 con 0x7f6444105bb0 2024-04-30T08:32:25.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.078+0000 7f6440c00640 1 -- 172.21.15.32:0/12575880 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+24053 (secure 0 0 0) 0x7f64340c8500 con 0x7f6444105bb0 2024-04-30T08:32:25.079 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:32:25.080 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":81,"fsid":"98c0a670-06ca-11ef-bc93-c7b262605968","created":"2024-04-30T08:22:11.122875+0000","modified":"2024-04-30T08:31:58.298969+0000","last_up_change":"2024-04-30T08:30:50.350439+0000","last_in_change":"2024-04-30T08:30:30.210620+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-30T08:26:19.728331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-30T08:31:49.800542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-30T08:31:51.211320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-30T08:31:55.716625+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a7e060c8-1cbb-418f-9a03-06693f6b3f52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6803","nonce":2624700402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6805","nonce":2624700402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6809","nonce":2624700402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6807","nonce":2624700402}]},"public_addr":"172.21.15.32:6803/2624700402","cluster_addr":"172.21.15.32:6805/2624700402","heartbeat_back_addr":"172.21.15.32:6809/2624700402","heartbeat_front_addr":"172.21.15.32:6807/2624700402","state":["exists","up"]},{"osd":1,"uuid":"72ef2fb1-e5ca-48ef-aa3a-4796a64b922f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6801","nonce":444649711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6803","nonce":444649711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6807","nonce":444649711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6805","nonce":444649711}]},"public_addr":"172.21.15.71:6801/444649711","cluster_addr":"172.21.15.71:6803/444649711","heartbeat_back_addr":"172.21.15.71:6807/444649711","heartbeat_front_addr":"172.21.15.71:6805/444649711","state":["exists","up"]},{"osd":2,"uuid":"704e5e8a-935e-47c7-ae28-27119876ac09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6801","nonce":3255528671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6803","nonce":3255528671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6807","nonce":3255528671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6805","nonce":3255528671}]},"public_addr":"172.21.15.189:6801/3255528671","cluster_addr":"172.21.15.189:6803/3255528671","heartbeat_back_addr":"172.21.15.189:6807/3255528671","heartbeat_front_addr":"172.21.15.189:6805/3255528671","state":["exists","up"]},{"osd":3,"uuid":"575004f7-6559-4a62-863e-e33078b0eebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6811","nonce":1384702112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6813","nonce":1384702112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6817","nonce":1384702112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6815","nonce":1384702112}]},"public_addr":"172.21.15.32:6811/1384702112","cluster_addr":"172.21.15.32:6813/1384702112","heartbeat_back_addr":"172.21.15.32:6817/1384702112","heartbeat_front_addr":"172.21.15.32:6815/1384702112","state":["exists","up"]},{"osd":4,"uuid":"e1c1e894-074d-4c78-89ef-9e5a114268bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6809","nonce":4250534574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6811","nonce":4250534574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6815","nonce":4250534574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6813","nonce":4250534574}]},"public_addr":"172.21.15.71:6809/4250534574","cluster_addr":"172.21.15.71:6811/4250534574","heartbeat_back_addr":"172.21.15.71:6815/4250534574","heartbeat_front_addr":"172.21.15.71:6813/4250534574","state":["exists","up"]},{"osd":5,"uuid":"12c022d4-fac6-40c4-ae7a-691571bee5d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6809","nonce":4245346198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6811","nonce":4245346198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6815","nonce":4245346198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6813","nonce":4245346198}]},"public_addr":"172.21.15.189:6809/4245346198","cluster_addr":"172.21.15.189:6811/4245346198","heartbeat_back_addr":"172.21.15.189:6815/4245346198","heartbeat_front_addr":"172.21.15.189:6813/4245346198","state":["exists","up"]},{"osd":6,"uuid":"b413f8ee-3d4a-415a-a2ba-d68437b7330d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6819","nonce":1910389190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6821","nonce":1910389190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6825","nonce":1910389190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6823","nonce":1910389190}]},"public_addr":"172.21.15.32:6819/1910389190","cluster_addr":"172.21.15.32:6821/1910389190","heartbeat_back_addr":"172.21.15.32:6825/1910389190","heartbeat_front_addr":"172.21.15.32:6823/1910389190","state":["exists","up"]},{"osd":7,"uuid":"aac64fb3-fc13-4126-b842-7b06ad5ce238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6817","nonce":692607380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6819","nonce":692607380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6823","nonce":692607380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6821","nonce":692607380}]},"public_addr":"172.21.15.71:6817/692607380","cluster_addr":"172.21.15.71:6819/692607380","heartbeat_back_addr":"172.21.15.71:6823/692607380","heartbeat_front_addr":"172.21.15.71:6821/692607380","state":["exists","up"]},{"osd":8,"uuid":"cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6817","nonce":3973697069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6819","nonce":3973697069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6823","nonce":3973697069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6821","nonce":3973697069}]},"public_addr":"172.21.15.189:6817/3973697069","cluster_addr":"172.21.15.189:6819/3973697069","heartbeat_back_addr":"172.21.15.189:6823/3973697069","heartbeat_front_addr":"172.21.15.189:6821/3973697069","state":["exists","up"]},{"osd":9,"uuid":"ef4a1645-ae6e-45c3-82ad-161510e62aa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6827","nonce":46066685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6829","nonce":46066685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6833","nonce":46066685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6831","nonce":46066685}]},"public_addr":"172.21.15.32:6827/46066685","cluster_addr":"172.21.15.32:6829/46066685","heartbeat_back_addr":"172.21.15.32:6833/46066685","heartbeat_front_addr":"172.21.15.32:6831/46066685","state":["exists","up"]},{"osd":10,"uuid":"25b26a38-a644-409f-b941-59efdf04a3ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":74,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6825","nonce":1886290293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6827","nonce":1886290293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6831","nonce":1886290293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6829","nonce":1886290293}]},"public_addr":"172.21.15.71:6825/1886290293","cluster_addr":"172.21.15.71:6827/1886290293","heartbeat_back_addr":"172.21.15.71:6831/1886290293","heartbeat_front_addr":"172.21.15.71:6829/1886290293","state":["exists","up"]},{"osd":11,"uuid":"ab96b3ff-3688-455b-af64-60226c5bfbff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6825","nonce":850185001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6827","nonce":850185001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6831","nonce":850185001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6829","nonce":850185001}]},"public_addr":"172.21.15.189:6825/850185001","cluster_addr":"172.21.15.189:6827/850185001","heartbeat_back_addr":"172.21.15.189:6831/850185001","heartbeat_front_addr":"172.21.15.189:6829/850185001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:27.220308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:53.230783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:17.190642+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:52.603799+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:20.568671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:46.158363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:22.706933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:50.665173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:16.602410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:53.814698+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:22.435189+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:48.651949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1266897222":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/2915689533":"2024-05-01T08:23:29.526664+0000","172.21.15.32:6801/4019351306":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/425836193":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/1134075722":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6801/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/21059253":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/3245550894":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/4019351306":"2024-05-01T08:23:29.526664+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-30T08:32:25.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.080+0000 7f6449200640 1 -- 172.21.15.32:0/12575880 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f64280afae0 msgr2=0x7f64280b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:25.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.080+0000 7f6449200640 1 --2- 172.21.15.32:0/12575880 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f64280afae0 0x7f64280b1ff0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f644419d380 tx=0x7f642c005f90 comp rx=0 tx=0).stop 2024-04-30T08:32:25.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.080+0000 7f6449200640 1 -- 172.21.15.32:0/12575880 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6444105bb0 msgr2=0x7f644419c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:25.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.080+0000 7f6449200640 1 --2- 172.21.15.32:0/12575880 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6444105bb0 0x7f644419c2e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f643402aea0 tx=0x7f6434003c20 comp rx=0 tx=0).stop 2024-04-30T08:32:25.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.080+0000 7f6449200640 1 -- 172.21.15.32:0/12575880 shutdown_connections 2024-04-30T08:32:25.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.080+0000 7f6449200640 1 --2- 172.21.15.32:0/12575880 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f64280afae0 0x7f64280b1ff0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:25.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.081+0000 7f6449200640 1 --2- 172.21.15.32:0/12575880 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f644410baf0 0x7f64441a37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:25.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.081+0000 7f6449200640 1 --2- 172.21.15.32:0/12575880 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6444106df0 0x7f644419c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:25.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.081+0000 7f6449200640 1 --2- 172.21.15.32:0/12575880 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6444105bb0 0x7f644419c2e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:25.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.081+0000 7f6449200640 1 -- 172.21.15.32:0/12575880 >> 172.21.15.32:0/12575880 conn(0x7f6444100da0 msgr2=0x7f6444107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:25.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.081+0000 7f6449200640 1 -- 172.21.15.32:0/12575880 shutdown_connections 2024-04-30T08:32:25.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.081+0000 7f6449200640 1 -- 172.21.15.32:0/12575880 wait complete. 2024-04-30T08:32:25.111 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-30T08:32:25.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:24 smithi071 ceph-mon[33385]: pgmap v338: 129 pgs: 129 active+clean; 581 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-30T08:32:25.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:24 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3411845220' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:32:25.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.403+0000 7f4d43000640 1 -- 172.21.15.32:0/2578933405 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d3c109750 msgr2=0x7f4d3c10bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:25.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.403+0000 7f4d43000640 1 --2- 172.21.15.32:0/2578933405 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d3c109750 0x7f4d3c10bb70 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4d34009860 tx=0x7f4d3402a340 comp rx=0 tx=0).stop 2024-04-30T08:32:25.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.403+0000 7f4d43000640 1 -- 172.21.15.32:0/2578933405 shutdown_connections 2024-04-30T08:32:25.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.403+0000 7f4d43000640 1 --2- 172.21.15.32:0/2578933405 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d3c109750 0x7f4d3c10bb70 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:25.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.403+0000 7f4d43000640 1 --2- 172.21.15.32:0/2578933405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d3c106df0 0x7f4d3c109210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:25.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.403+0000 7f4d43000640 1 --2- 172.21.15.32:0/2578933405 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d3c104490 0x7f4d3c1068b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:25.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.403+0000 7f4d43000640 1 -- 172.21.15.32:0/2578933405 >> 172.21.15.32:0/2578933405 conn(0x7f4d3c0fda90 msgr2=0x7f4d3c0fff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:25.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.403+0000 7f4d43000640 1 -- 172.21.15.32:0/2578933405 shutdown_connections 2024-04-30T08:32:25.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.403+0000 7f4d43000640 1 -- 172.21.15.32:0/2578933405 wait complete. 2024-04-30T08:32:25.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.404+0000 7f4d43000640 1 Processor -- start 2024-04-30T08:32:25.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.405+0000 7f4d43000640 1 -- start start 2024-04-30T08:32:25.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.405+0000 7f4d43000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d3c104490 0x7f4d3c1038e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:25.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.405+0000 7f4d43000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d3c106df0 0x7f4d3c101f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:25.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.405+0000 7f4d43000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d3c109750 0x7f4d3c1024c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:25.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.405+0000 7f4d43000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3c10f460 con 0x7f4d3c106df0 2024-04-30T08:32:25.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.405+0000 7f4d43000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3c10f2e0 con 0x7f4d3c109750 2024-04-30T08:32:25.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.405+0000 7f4d43000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4d3c10f5e0 con 0x7f4d3c104490 2024-04-30T08:32:25.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.405+0000 7f4d41c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d3c104490 0x7f4d3c1038e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:25.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.405+0000 7f4d41200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d3c106df0 0x7f4d3c101f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:25.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.405+0000 7f4d42600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d3c109750 0x7f4d3c1024c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:25.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.405+0000 7f4d41200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d3c106df0 0x7f4d3c101f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42066/0 (socket says 172.21.15.32:42066) 2024-04-30T08:32:25.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.405+0000 7f4d41c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d3c104490 0x7f4d3c1038e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:44524/0 (socket says 172.21.15.32:44524) 2024-04-30T08:32:25.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.405+0000 7f4d41200640 1 -- 172.21.15.32:0/1870301895 learned_addr learned my addr 172.21.15.32:0/1870301895 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:25.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.406+0000 7f4d42600640 1 -- 172.21.15.32:0/1870301895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d3c104490 msgr2=0x7f4d3c1038e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:25.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.406+0000 7f4d42600640 1 --2- 172.21.15.32:0/1870301895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d3c104490 0x7f4d3c1038e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:25.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.406+0000 7f4d42600640 1 -- 172.21.15.32:0/1870301895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d3c106df0 msgr2=0x7f4d3c101f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:25.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.406+0000 7f4d42600640 1 --2- 172.21.15.32:0/1870301895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d3c106df0 0x7f4d3c101f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:25.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.406+0000 7f4d42600640 1 -- 172.21.15.32:0/1870301895 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d3c102bf0 con 0x7f4d3c109750 2024-04-30T08:32:25.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.406+0000 7f4d41c00640 1 --2- 172.21.15.32:0/1870301895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d3c104490 0x7f4d3c1038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:32:25.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.406+0000 7f4d42600640 1 --2- 172.21.15.32:0/1870301895 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d3c109750 0x7f4d3c1024c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4d3402a860 tx=0x7f4d34032d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:25.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.406+0000 7f4d3a000640 1 -- 172.21.15.32:0/1870301895 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d34040020 con 0x7f4d3c109750 2024-04-30T08:32:25.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.407+0000 7f4d43000640 1 -- 172.21.15.32:0/1870301895 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d3c1a40c0 con 0x7f4d3c109750 2024-04-30T08:32:25.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.407+0000 7f4d3a000640 1 -- 172.21.15.32:0/1870301895 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d34030e10 con 0x7f4d3c109750 2024-04-30T08:32:25.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.407+0000 7f4d3a000640 1 -- 172.21.15.32:0/1870301895 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d34037040 con 0x7f4d3c109750 2024-04-30T08:32:25.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.407+0000 7f4d43000640 1 -- 172.21.15.32:0/1870301895 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d3c1a4520 con 0x7f4d3c109750 2024-04-30T08:32:25.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.408+0000 7f4d43000640 1 -- 172.21.15.32:0/1870301895 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d3c0ff5b0 con 0x7f4d3c109750 2024-04-30T08:32:25.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.410+0000 7f4d3a000640 1 -- 172.21.15.32:0/1870301895 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f4d34043020 con 0x7f4d3c109750 2024-04-30T08:32:25.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.412+0000 7f4d3a000640 1 --2- 172.21.15.32:0/1870301895 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4d200afb70 0x7f4d200b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:25.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.412+0000 7f4d3a000640 1 -- 172.21.15.32:0/1870301895 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f4d340ff2a0 con 0x7f4d3c109750 2024-04-30T08:32:25.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.412+0000 7f4d41c00640 1 --2- 172.21.15.32:0/1870301895 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4d200afb70 0x7f4d200b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:25.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.413+0000 7f4d41c00640 1 --2- 172.21.15.32:0/1870301895 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4d200afb70 0x7f4d200b2080 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c06a930 tx=0x7f4d2c0095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:25.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.414+0000 7f4d3a000640 1 -- 172.21.15.32:0/1870301895 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4d3404a030 con 0x7f4d3c109750 2024-04-30T08:32:25.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.663+0000 7f4d43000640 1 -- 172.21.15.32:0/1870301895 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f4d3c02dbd0 con 0x7f4d3c109750 2024-04-30T08:32:25.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.783+0000 7f4d3a000640 1 -- 172.21.15.32:0/1870301895 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7f4d340c7ce0 con 0x7f4d3c109750 2024-04-30T08:32:25.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.786+0000 7f4d43000640 1 -- 172.21.15.32:0/1870301895 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4d200afb70 msgr2=0x7f4d200b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:25.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.786+0000 7f4d43000640 1 --2- 172.21.15.32:0/1870301895 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4d200afb70 0x7f4d200b2080 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c06a930 tx=0x7f4d2c0095c0 comp rx=0 tx=0).stop 2024-04-30T08:32:25.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.786+0000 7f4d43000640 1 -- 172.21.15.32:0/1870301895 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d3c109750 msgr2=0x7f4d3c1024c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:25.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.786+0000 7f4d43000640 1 --2- 172.21.15.32:0/1870301895 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d3c109750 0x7f4d3c1024c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f4d3402a860 tx=0x7f4d34032d50 comp rx=0 tx=0).stop 2024-04-30T08:32:25.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.787+0000 7f4d43000640 1 -- 172.21.15.32:0/1870301895 shutdown_connections 2024-04-30T08:32:25.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.787+0000 7f4d43000640 1 --2- 172.21.15.32:0/1870301895 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4d200afb70 0x7f4d200b2080 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:25.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.787+0000 7f4d43000640 1 --2- 172.21.15.32:0/1870301895 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d3c109750 0x7f4d3c1024c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:25.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.787+0000 7f4d43000640 1 --2- 172.21.15.32:0/1870301895 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d3c106df0 0x7f4d3c101f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:25.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.787+0000 7f4d43000640 1 --2- 172.21.15.32:0/1870301895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d3c104490 0x7f4d3c1038e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:25.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.787+0000 7f4d43000640 1 -- 172.21.15.32:0/1870301895 >> 172.21.15.32:0/1870301895 conn(0x7f4d3c0fda90 msgr2=0x7f4d3c105330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:25.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.787+0000 7f4d43000640 1 -- 172.21.15.32:0/1870301895 shutdown_connections 2024-04-30T08:32:25.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:25.787+0000 7f4d43000640 1 -- 172.21.15.32:0/1870301895 wait complete. 2024-04-30T08:32:25.822 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-04-30T08:32:25.822 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-04-30T08:32:26.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:25 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/12575880' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:32:26.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:25 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1870301895' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-30T08:32:26.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:25 smithi189 ceph-mon[33038]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-30T08:32:26.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.108+0000 7f9a0ec00640 1 -- 172.21.15.32:0/668851607 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a08103b80 msgr2=0x7f9a08104880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:26.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.108+0000 7f9a0ec00640 1 --2- 172.21.15.32:0/668851607 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a08103b80 0x7f9a08104880 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f99f8009900 tx=0x7f99f802a3f0 comp rx=0 tx=0).stop 2024-04-30T08:32:26.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.108+0000 7f9a0ec00640 1 -- 172.21.15.32:0/668851607 shutdown_connections 2024-04-30T08:32:26.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.108+0000 7f9a0ec00640 1 --2- 172.21.15.32:0/668851607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a08109ac0 0x7f9a0810bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:26.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.108+0000 7f9a0ec00640 1 --2- 172.21.15.32:0/668851607 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a08104dc0 0x7f9a081051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:26.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.108+0000 7f9a0ec00640 1 --2- 172.21.15.32:0/668851607 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a08103b80 0x7f9a08104880 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:26.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.108+0000 7f9a0ec00640 1 -- 172.21.15.32:0/668851607 >> 172.21.15.32:0/668851607 conn(0x7f9a080fed90 msgr2=0x7f9a08101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:26.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.108+0000 7f9a0ec00640 1 -- 172.21.15.32:0/668851607 shutdown_connections 2024-04-30T08:32:26.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.108+0000 7f9a0ec00640 1 -- 172.21.15.32:0/668851607 wait complete. 2024-04-30T08:32:26.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.109+0000 7f9a0ec00640 1 Processor -- start 2024-04-30T08:32:26.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.110+0000 7f9a0ec00640 1 -- start start 2024-04-30T08:32:26.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.110+0000 7f9a0ec00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a08103b80 0x7f9a0819c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:26.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.110+0000 7f9a0ec00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a08104dc0 0x7f9a0819ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:26.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.110+0000 7f9a0ec00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a08109ac0 0x7f9a081a39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:26.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.110+0000 7f9a0ec00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9a0810f680 con 0x7f9a08104dc0 2024-04-30T08:32:26.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.110+0000 7f9a0ec00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f9a0810f500 con 0x7f9a08109ac0 2024-04-30T08:32:26.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.110+0000 7f9a0ec00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9a0810f800 con 0x7f9a08103b80 2024-04-30T08:32:26.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.110+0000 7f9a07400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a08104dc0 0x7f9a0819ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:26.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.110+0000 7f9a07e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a08103b80 0x7f9a0819c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:26.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.110+0000 7f9a07400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a08104dc0 0x7f9a0819ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42074/0 (socket says 172.21.15.32:42074) 2024-04-30T08:32:26.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.110+0000 7f9a0cc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a08109ac0 0x7f9a081a39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:26.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.110+0000 7f9a07400640 1 -- 172.21.15.32:0/3165968160 learned_addr learned my addr 172.21.15.32:0/3165968160 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:26.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.111+0000 7f9a07e00640 1 -- 172.21.15.32:0/3165968160 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a08109ac0 msgr2=0x7f9a081a39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:26.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.111+0000 7f9a07e00640 1 --2- 172.21.15.32:0/3165968160 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a08109ac0 0x7f9a081a39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:26.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.111+0000 7f9a07e00640 1 -- 172.21.15.32:0/3165968160 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a08104dc0 msgr2=0x7f9a0819ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:26.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.111+0000 7f9a07e00640 1 --2- 172.21.15.32:0/3165968160 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a08104dc0 0x7f9a0819ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:26.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.111+0000 7f9a07e00640 1 -- 172.21.15.32:0/3165968160 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a081a4010 con 0x7f9a08103b80 2024-04-30T08:32:26.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.111+0000 7f9a07400640 1 --2- 172.21.15.32:0/3165968160 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a08104dc0 0x7f9a0819ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:32:26.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.111+0000 7f9a07e00640 1 --2- 172.21.15.32:0/3165968160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a08103b80 0x7f9a0819c4d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f99f802a910 tx=0x7f99f8004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:26.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.112+0000 7f9a04c00640 1 -- 172.21.15.32:0/3165968160 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f803f070 con 0x7f9a08103b80 2024-04-30T08:32:26.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.112+0000 7f9a0ec00640 1 -- 172.21.15.32:0/3165968160 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a081a42a0 con 0x7f9a08103b80 2024-04-30T08:32:26.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.112+0000 7f9a0ec00640 1 -- 172.21.15.32:0/3165968160 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a081a47b0 con 0x7f9a08103b80 2024-04-30T08:32:26.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.112+0000 7f9a04c00640 1 -- 172.21.15.32:0/3165968160 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99f8031910 con 0x7f9a08103b80 2024-04-30T08:32:26.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.112+0000 7f9a04c00640 1 -- 172.21.15.32:0/3165968160 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99f8048060 con 0x7f9a08103b80 2024-04-30T08:32:26.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.113+0000 7f9a0ec00640 1 -- 172.21.15.32:0/3165968160 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a081a4a30 con 0x7f9a08103b80 2024-04-30T08:32:26.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.115+0000 7f9a04c00640 1 -- 172.21.15.32:0/3165968160 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f99f80037f0 con 0x7f9a08103b80 2024-04-30T08:32:26.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.117+0000 7f9a04c00640 1 --2- 172.21.15.32:0/3165968160 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f99ec0afb70 0x7f99ec0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:26.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.117+0000 7f9a04c00640 1 -- 172.21.15.32:0/3165968160 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f99f80ff820 con 0x7f9a08103b80 2024-04-30T08:32:26.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.117+0000 7f9a07400640 1 --2- 172.21.15.32:0/3165968160 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f99ec0afb70 0x7f99ec0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:26.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.118+0000 7f9a07400640 1 --2- 172.21.15.32:0/3165968160 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f99ec0afb70 0x7f99ec0b2080 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9a0819d570 tx=0x7f99f4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:26.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.119+0000 7f9a04c00640 1 -- 172.21.15.32:0/3165968160 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f99f80c7b10 con 0x7f9a08103b80 2024-04-30T08:32:26.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:25 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/12575880' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:32:26.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:25 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1870301895' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-30T08:32:26.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:25 smithi032 ceph-mon[28331]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-30T08:32:26.260 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:25 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/12575880' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:32:26.260 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:25 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1870301895' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-30T08:32:26.260 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:25 smithi071 ceph-mon[33385]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-30T08:32:26.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.369+0000 7f9a0ec00640 1 -- 172.21.15.32:0/3165968160 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f9a081a4c00 con 0x7f9a08103b80 2024-04-30T08:32:26.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.493+0000 7f9a04c00640 1 -- 172.21.15.32:0/3165968160 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7f99f80cbe10 con 0x7f9a08103b80 2024-04-30T08:32:26.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.496+0000 7f9a0ec00640 1 -- 172.21.15.32:0/3165968160 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f99ec0afb70 msgr2=0x7f99ec0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:26.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.496+0000 7f9a0ec00640 1 --2- 172.21.15.32:0/3165968160 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f99ec0afb70 0x7f99ec0b2080 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f9a0819d570 tx=0x7f99f4005f90 comp rx=0 tx=0).stop 2024-04-30T08:32:26.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.496+0000 7f9a0ec00640 1 -- 172.21.15.32:0/3165968160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a08103b80 msgr2=0x7f9a0819c4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:26.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.496+0000 7f9a0ec00640 1 --2- 172.21.15.32:0/3165968160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a08103b80 0x7f9a0819c4d0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f99f802a910 tx=0x7f99f8004270 comp rx=0 tx=0).stop 2024-04-30T08:32:26.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.497+0000 7f9a0ec00640 1 -- 172.21.15.32:0/3165968160 shutdown_connections 2024-04-30T08:32:26.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.497+0000 7f9a0ec00640 1 --2- 172.21.15.32:0/3165968160 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f99ec0afb70 0x7f99ec0b2080 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:26.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.497+0000 7f9a0ec00640 1 --2- 172.21.15.32:0/3165968160 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9a08109ac0 0x7f9a081a39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:26.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.497+0000 7f9a0ec00640 1 --2- 172.21.15.32:0/3165968160 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9a08104dc0 0x7f9a0819ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:26.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.497+0000 7f9a0ec00640 1 --2- 172.21.15.32:0/3165968160 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9a08103b80 0x7f9a0819c4d0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:26.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.497+0000 7f9a0ec00640 1 -- 172.21.15.32:0/3165968160 >> 172.21.15.32:0/3165968160 conn(0x7f9a080fed90 msgr2=0x7f9a080ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:26.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.497+0000 7f9a0ec00640 1 -- 172.21.15.32:0/3165968160 shutdown_connections 2024-04-30T08:32:26.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.497+0000 7f9a0ec00640 1 -- 172.21.15.32:0/3165968160 wait complete. 2024-04-30T08:32:26.536 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-30T08:32:26.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.801+0000 7fd02ac00640 1 -- 172.21.15.32:0/3368770127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd024104dc0 msgr2=0x7fd0241051d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:26.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.801+0000 7fd02ac00640 1 --2- 172.21.15.32:0/3368770127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd024104dc0 0x7fd0241051d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fd00c009900 tx=0x7fd00c02a3f0 comp rx=0 tx=0).stop 2024-04-30T08:32:26.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.801+0000 7fd02ac00640 1 -- 172.21.15.32:0/3368770127 shutdown_connections 2024-04-30T08:32:26.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.801+0000 7fd02ac00640 1 --2- 172.21.15.32:0/3368770127 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd024109ac0 0x7fd02410bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:26.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.801+0000 7fd02ac00640 1 --2- 172.21.15.32:0/3368770127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd024104dc0 0x7fd0241051d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:26.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.801+0000 7fd02ac00640 1 --2- 172.21.15.32:0/3368770127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd024103b80 0x7fd024104880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:26.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.801+0000 7fd02ac00640 1 -- 172.21.15.32:0/3368770127 >> 172.21.15.32:0/3368770127 conn(0x7fd0240fed90 msgr2=0x7fd024101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:26.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.801+0000 7fd02ac00640 1 -- 172.21.15.32:0/3368770127 shutdown_connections 2024-04-30T08:32:26.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.801+0000 7fd02ac00640 1 -- 172.21.15.32:0/3368770127 wait complete. 2024-04-30T08:32:26.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.802+0000 7fd02ac00640 1 Processor -- start 2024-04-30T08:32:26.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.802+0000 7fd02ac00640 1 -- start start 2024-04-30T08:32:26.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.803+0000 7fd02ac00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd024103b80 0x7fd0240779e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:26.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.803+0000 7fd02ac00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd024104dc0 0x7fd024076080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:26.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.803+0000 7fd02ac00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd024109ac0 0x7fd0240765c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:26.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.803+0000 7fd02ac00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd02410f6f0 con 0x7fd024109ac0 2024-04-30T08:32:26.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.803+0000 7fd02ac00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd02410f570 con 0x7fd024103b80 2024-04-30T08:32:26.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.803+0000 7fd02ac00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd02410f870 con 0x7fd024104dc0 2024-04-30T08:32:26.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.803+0000 7fd028c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd024109ac0 0x7fd0240765c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:26.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.803+0000 7fd028c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd024109ac0 0x7fd0240765c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:42096/0 (socket says 172.21.15.32:42096) 2024-04-30T08:32:26.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.803+0000 7fd028c00640 1 -- 172.21.15.32:0/3955297738 learned_addr learned my addr 172.21.15.32:0/3955297738 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:26.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.803+0000 7fd023e00640 1 --2- 172.21.15.32:0/3955297738 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd024103b80 0x7fd0240779e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:26.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.803+0000 7fd01be00640 1 --2- 172.21.15.32:0/3955297738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd024104dc0 0x7fd024076080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:26.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.804+0000 7fd023e00640 1 -- 172.21.15.32:0/3955297738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd024104dc0 msgr2=0x7fd024076080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:26.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.804+0000 7fd023e00640 1 --2- 172.21.15.32:0/3955297738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd024104dc0 0x7fd024076080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:26.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.804+0000 7fd023e00640 1 -- 172.21.15.32:0/3955297738 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd024109ac0 msgr2=0x7fd0240765c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:26.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.804+0000 7fd023e00640 1 --2- 172.21.15.32:0/3955297738 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd024109ac0 0x7fd0240765c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:26.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.804+0000 7fd023e00640 1 -- 172.21.15.32:0/3955297738 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd024076cf0 con 0x7fd024103b80 2024-04-30T08:32:26.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.804+0000 7fd028c00640 1 --2- 172.21.15.32:0/3955297738 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd024109ac0 0x7fd0240765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:32:26.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.804+0000 7fd023e00640 1 --2- 172.21.15.32:0/3955297738 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd024103b80 0x7fd0240779e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd024103450 tx=0x7fd01400ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:26.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.804+0000 7fd021600640 1 -- 172.21.15.32:0/3955297738 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd01400cc40 con 0x7fd024103b80 2024-04-30T08:32:26.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.804+0000 7fd021600640 1 -- 172.21.15.32:0/3955297738 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd01400cde0 con 0x7fd024103b80 2024-04-30T08:32:26.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.804+0000 7fd02ac00640 1 -- 172.21.15.32:0/3955297738 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0241a4350 con 0x7fd024103b80 2024-04-30T08:32:26.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.804+0000 7fd021600640 1 -- 172.21.15.32:0/3955297738 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd014010430 con 0x7fd024103b80 2024-04-30T08:32:26.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.804+0000 7fd02ac00640 1 -- 172.21.15.32:0/3955297738 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0241a47c0 con 0x7fd024103b80 2024-04-30T08:32:26.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.805+0000 7fd02ac00640 1 -- 172.21.15.32:0/3955297738 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0241088d0 con 0x7fd024103b80 2024-04-30T08:32:26.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.808+0000 7fd021600640 1 -- 172.21.15.32:0/3955297738 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fd014003680 con 0x7fd024103b80 2024-04-30T08:32:26.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.809+0000 7fd021600640 1 --2- 172.21.15.32:0/3955297738 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd0080afaf0 0x7fd0080b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:26.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.809+0000 7fd021600640 1 -- 172.21.15.32:0/3955297738 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7fd0140df1c0 con 0x7fd024103b80 2024-04-30T08:32:26.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.809+0000 7fd01be00640 1 --2- 172.21.15.32:0/3955297738 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd0080afaf0 0x7fd0080b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:26.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.810+0000 7fd01be00640 1 --2- 172.21.15.32:0/3955297738 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd0080afaf0 0x7fd0080b2000 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd00c004770 tx=0x7fd00c0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:26.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:26.814+0000 7fd021600640 1 -- 172.21.15.32:0/3955297738 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd0140a7b00 con 0x7fd024103b80 2024-04-30T08:32:26.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:26 smithi032 ceph-mon[28331]: pgmap v339: 129 pgs: 129 active+clean; 581 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-30T08:32:26.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:26 smithi032 ceph-mon[28331]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-30T08:32:26.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:26 smithi032 ceph-mon[28331]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-30T08:32:26.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:26 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3165968160' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-30T08:32:26.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:26 smithi032 ceph-mon[28331]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-30T08:32:26.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:26 smithi032 ceph-mon[28331]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-30T08:32:26.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:26 smithi032 ceph-mon[28331]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-30T08:32:26.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:26 smithi032 ceph-mon[28331]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-30T08:32:26.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:26 smithi032 ceph-mon[28331]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2024-04-30T08:32:26.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:26 smithi032 ceph-mon[28331]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-30T08:32:27.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:26 smithi189 ceph-mon[33038]: pgmap v339: 129 pgs: 129 active+clean; 581 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-30T08:32:27.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:26 smithi189 ceph-mon[33038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-30T08:32:27.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:26 smithi189 ceph-mon[33038]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-30T08:32:27.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:26 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3165968160' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-30T08:32:27.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:26 smithi189 ceph-mon[33038]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-30T08:32:27.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:26 smithi189 ceph-mon[33038]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-30T08:32:27.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:26 smithi189 ceph-mon[33038]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-30T08:32:27.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:26 smithi189 ceph-mon[33038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-30T08:32:27.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:26 smithi189 ceph-mon[33038]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2024-04-30T08:32:27.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:26 smithi189 ceph-mon[33038]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-30T08:32:27.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.061+0000 7fd02ac00640 1 -- 172.21.15.32:0/3955297738 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7fd02402dbd0 con 0x7fd024103b80 2024-04-30T08:32:27.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:26 smithi071 ceph-mon[33385]: pgmap v339: 129 pgs: 129 active+clean; 581 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-04-30T08:32:27.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:26 smithi071 ceph-mon[33385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-30T08:32:27.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:26 smithi071 ceph-mon[33385]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-04-30T08:32:27.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:26 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3165968160' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-30T08:32:27.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:26 smithi071 ceph-mon[33385]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-04-30T08:32:27.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:26 smithi071 ceph-mon[33385]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-30T08:32:27.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:26 smithi071 ceph-mon[33385]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-30T08:32:27.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:26 smithi071 ceph-mon[33385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-04-30T08:32:27.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:26 smithi071 ceph-mon[33385]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2024-04-30T08:32:27.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:26 smithi071 ceph-mon[33385]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-04-30T08:32:27.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.497+0000 7fd021600640 1 -- 172.21.15.32:0/3955297738 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) ==== 121+0+0 (secure 0 0 0) 0x7fd0140a7400 con 0x7fd024103b80 2024-04-30T08:32:27.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.501+0000 7fd02ac00640 1 -- 172.21.15.32:0/3955297738 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd0080afaf0 msgr2=0x7fd0080b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:27.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.501+0000 7fd02ac00640 1 --2- 172.21.15.32:0/3955297738 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd0080afaf0 0x7fd0080b2000 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd00c004770 tx=0x7fd00c0314a0 comp rx=0 tx=0).stop 2024-04-30T08:32:27.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.501+0000 7fd02ac00640 1 -- 172.21.15.32:0/3955297738 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd024103b80 msgr2=0x7fd0240779e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:27.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.501+0000 7fd02ac00640 1 --2- 172.21.15.32:0/3955297738 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd024103b80 0x7fd0240779e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd024103450 tx=0x7fd01400ee50 comp rx=0 tx=0).stop 2024-04-30T08:32:27.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.501+0000 7fd02ac00640 1 -- 172.21.15.32:0/3955297738 shutdown_connections 2024-04-30T08:32:27.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.501+0000 7fd02ac00640 1 --2- 172.21.15.32:0/3955297738 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd0080afaf0 0x7fd0080b2000 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:27.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.501+0000 7fd02ac00640 1 --2- 172.21.15.32:0/3955297738 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd024109ac0 0x7fd0240765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:27.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.501+0000 7fd02ac00640 1 --2- 172.21.15.32:0/3955297738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd024104dc0 0x7fd024076080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:27.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.501+0000 7fd02ac00640 1 --2- 172.21.15.32:0/3955297738 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd024103b80 0x7fd0240779e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:27.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.501+0000 7fd02ac00640 1 -- 172.21.15.32:0/3955297738 >> 172.21.15.32:0/3955297738 conn(0x7fd0240fed90 msgr2=0x7fd024105aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:27.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.501+0000 7fd02ac00640 1 -- 172.21.15.32:0/3955297738 shutdown_connections 2024-04-30T08:32:27.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.501+0000 7fd02ac00640 1 -- 172.21.15.32:0/3955297738 wait complete. 2024-04-30T08:32:27.546 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-30T08:32:27.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.814+0000 7f3ee6400640 1 -- 172.21.15.32:0/2030452635 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ee010baf0 msgr2=0x7f3ee010df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:27.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.814+0000 7f3ee6400640 1 --2- 172.21.15.32:0/2030452635 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ee010baf0 0x7f3ee010df10 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f3ed8009be0 tx=0x7f3ed802a6c0 comp rx=0 tx=0).stop 2024-04-30T08:32:27.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.815+0000 7f3ee6400640 1 -- 172.21.15.32:0/2030452635 shutdown_connections 2024-04-30T08:32:27.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.815+0000 7f3ee6400640 1 --2- 172.21.15.32:0/2030452635 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ee010baf0 0x7f3ee010df10 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:27.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.815+0000 7f3ee6400640 1 --2- 172.21.15.32:0/2030452635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ee0106df0 0x7f3ee0107200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:27.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.815+0000 7f3ee6400640 1 --2- 172.21.15.32:0/2030452635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ee0105bb0 0x7f3ee01068b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:27.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.815+0000 7f3ee6400640 1 -- 172.21.15.32:0/2030452635 >> 172.21.15.32:0/2030452635 conn(0x7f3ee0100da0 msgr2=0x7f3ee0103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:27.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.815+0000 7f3ee6400640 1 -- 172.21.15.32:0/2030452635 shutdown_connections 2024-04-30T08:32:27.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.815+0000 7f3ee6400640 1 -- 172.21.15.32:0/2030452635 wait complete. 2024-04-30T08:32:27.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.816+0000 7f3ee6400640 1 Processor -- start 2024-04-30T08:32:27.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.816+0000 7f3ee6400640 1 -- start start 2024-04-30T08:32:27.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.816+0000 7f3ee6400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ee0105bb0 0x7f3ee019c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:27.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.816+0000 7f3ee6400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ee0106df0 0x7f3ee019c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:27.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.816+0000 7f3ee6400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ee010baf0 0x7f3ee01a3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:27.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.816+0000 7f3ee6400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3ee01116e0 con 0x7f3ee010baf0 2024-04-30T08:32:27.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.816+0000 7f3ee6400640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f3ee0111560 con 0x7f3ee0105bb0 2024-04-30T08:32:27.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.816+0000 7f3ee6400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3ee0111860 con 0x7f3ee0106df0 2024-04-30T08:32:27.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.817+0000 7f3ee5000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ee0105bb0 0x7f3ee019c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:27.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.817+0000 7f3ee5000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ee0105bb0 0x7f3ee019c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:59538/0 (socket says 172.21.15.32:59538) 2024-04-30T08:32:27.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.817+0000 7f3ee5000640 1 -- 172.21.15.32:0/2537664133 learned_addr learned my addr 172.21.15.32:0/2537664133 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:27.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.817+0000 7f3edfe00640 1 --2- 172.21.15.32:0/2537664133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ee0106df0 0x7f3ee019c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:27.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.817+0000 7f3ee5a00640 1 --2- 172.21.15.32:0/2537664133 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ee010baf0 0x7f3ee01a3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:27.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.817+0000 7f3ee5000640 1 -- 172.21.15.32:0/2537664133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ee0106df0 msgr2=0x7f3ee019c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:27.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.817+0000 7f3ee5000640 1 --2- 172.21.15.32:0/2537664133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ee0106df0 0x7f3ee019c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:27.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.817+0000 7f3ee5000640 1 -- 172.21.15.32:0/2537664133 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ee010baf0 msgr2=0x7f3ee01a3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:27.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.817+0000 7f3ee5000640 1 --2- 172.21.15.32:0/2537664133 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ee010baf0 0x7f3ee01a3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:27.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.817+0000 7f3ee5000640 1 -- 172.21.15.32:0/2537664133 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ee01a3d90 con 0x7f3ee0105bb0 2024-04-30T08:32:27.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.817+0000 7f3ee5000640 1 --2- 172.21.15.32:0/2537664133 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ee0105bb0 0x7f3ee019c250 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3ed400e9f0 tx=0x7f3ed400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:27.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.817+0000 7f3edd600640 1 -- 172.21.15.32:0/2537664133 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed400ccc0 con 0x7f3ee0105bb0 2024-04-30T08:32:27.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.818+0000 7f3edd600640 1 -- 172.21.15.32:0/2537664133 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ed40035f0 con 0x7f3ee0105bb0 2024-04-30T08:32:27.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.818+0000 7f3ee6400640 1 -- 172.21.15.32:0/2537664133 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ee01a4080 con 0x7f3ee0105bb0 2024-04-30T08:32:27.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.818+0000 7f3edd600640 1 -- 172.21.15.32:0/2537664133 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ed400ce60 con 0x7f3ee0105bb0 2024-04-30T08:32:27.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.818+0000 7f3ee6400640 1 -- 172.21.15.32:0/2537664133 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ee01a45d0 con 0x7f3ee0105bb0 2024-04-30T08:32:27.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.819+0000 7f3ee6400640 1 -- 172.21.15.32:0/2537664133 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ee010a900 con 0x7f3ee0105bb0 2024-04-30T08:32:27.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.821+0000 7f3edd600640 1 -- 172.21.15.32:0/2537664133 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f3ed4004030 con 0x7f3ee0105bb0 2024-04-30T08:32:27.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.822+0000 7f3edd600640 1 --2- 172.21.15.32:0/2537664133 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3ec80afb60 0x7f3ec80b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:27.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.822+0000 7f3edd600640 1 -- 172.21.15.32:0/2537664133 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f3ed40df290 con 0x7f3ee0105bb0 2024-04-30T08:32:27.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.823+0000 7f3edfe00640 1 --2- 172.21.15.32:0/2537664133 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3ec80afb60 0x7f3ec80b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:27.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.823+0000 7f3edfe00640 1 --2- 172.21.15.32:0/2537664133 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3ec80afb60 0x7f3ec80b2070 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc0096c0 tx=0x7f3ecc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:27.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:27.826+0000 7f3edd600640 1 -- 172.21.15.32:0/2537664133 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3ed40a7bd0 con 0x7f3ee0105bb0 2024-04-30T08:32:27.846 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:27 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3955297738' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-30T08:32:27.846 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:27 smithi032 ceph-mon[28331]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-30T08:32:27.846 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:27 smithi032 ceph-mon[28331]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-30T08:32:27.846 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:27 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] up:active 2024-04-30T08:32:27.847 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:27 smithi032 ceph-mon[28331]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-30T08:32:27.847 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:27 smithi032 ceph-mon[28331]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 10 up:standby 2024-04-30T08:32:27.847 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:27 smithi032 ceph-mon[28331]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:creating} 9 up:standby 2024-04-30T08:32:27.847 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:27 smithi032 ceph-mon[28331]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-30T08:32:28.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:27 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3955297738' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-30T08:32:28.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:27 smithi189 ceph-mon[33038]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-30T08:32:28.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:27 smithi189 ceph-mon[33038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-30T08:32:28.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:27 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] up:active 2024-04-30T08:32:28.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:27 smithi189 ceph-mon[33038]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-30T08:32:28.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:27 smithi189 ceph-mon[33038]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 10 up:standby 2024-04-30T08:32:28.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:27 smithi189 ceph-mon[33038]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:creating} 9 up:standby 2024-04-30T08:32:28.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:27 smithi189 ceph-mon[33038]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-30T08:32:28.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.086+0000 7f3ee6400640 1 -- 172.21.15.32:0/2537664133 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f3ee0063820 con 0x7f3ee0105bb0 2024-04-30T08:32:28.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:27 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3955297738' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-30T08:32:28.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:27 smithi071 ceph-mon[33385]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-30T08:32:28.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:27 smithi071 ceph-mon[33385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-30T08:32:28.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:27 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] up:active 2024-04-30T08:32:28.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:27 smithi071 ceph-mon[33385]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-30T08:32:28.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:27 smithi071 ceph-mon[33385]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 10 up:standby 2024-04-30T08:32:28.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:27 smithi071 ceph-mon[33385]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:creating} 9 up:standby 2024-04-30T08:32:28.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:27 smithi071 ceph-mon[33385]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-30T08:32:28.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.511+0000 7f3edd600640 1 -- 172.21.15.32:0/2537664133 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) ==== 115+0+0 (secure 0 0 0) 0x7f3ed40a74d0 con 0x7f3ee0105bb0 2024-04-30T08:32:28.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.514+0000 7f3ee6400640 1 -- 172.21.15.32:0/2537664133 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3ec80afb60 msgr2=0x7f3ec80b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:28.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.514+0000 7f3ee6400640 1 --2- 172.21.15.32:0/2537664133 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3ec80afb60 0x7f3ec80b2070 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3ecc0096c0 tx=0x7f3ecc009260 comp rx=0 tx=0).stop 2024-04-30T08:32:28.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.514+0000 7f3ee6400640 1 -- 172.21.15.32:0/2537664133 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ee0105bb0 msgr2=0x7f3ee019c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:28.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.514+0000 7f3ee6400640 1 --2- 172.21.15.32:0/2537664133 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ee0105bb0 0x7f3ee019c250 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3ed400e9f0 tx=0x7f3ed400eeb0 comp rx=0 tx=0).stop 2024-04-30T08:32:28.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.514+0000 7f3ee6400640 1 -- 172.21.15.32:0/2537664133 shutdown_connections 2024-04-30T08:32:28.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.514+0000 7f3ee6400640 1 --2- 172.21.15.32:0/2537664133 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3ec80afb60 0x7f3ec80b2070 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:28.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.515+0000 7f3ee6400640 1 --2- 172.21.15.32:0/2537664133 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ee010baf0 0x7f3ee01a3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:28.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.515+0000 7f3ee6400640 1 --2- 172.21.15.32:0/2537664133 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ee0106df0 0x7f3ee019c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:28.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.515+0000 7f3ee6400640 1 --2- 172.21.15.32:0/2537664133 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ee0105bb0 0x7f3ee019c250 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:28.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.515+0000 7f3ee6400640 1 -- 172.21.15.32:0/2537664133 >> 172.21.15.32:0/2537664133 conn(0x7f3ee0100da0 msgr2=0x7f3ee0107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:28.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.515+0000 7f3ee6400640 1 -- 172.21.15.32:0/2537664133 shutdown_connections 2024-04-30T08:32:28.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.515+0000 7f3ee6400640 1 -- 172.21.15.32:0/2537664133 wait complete. 2024-04-30T08:32:28.551 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-04-30T08:32:28.552 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-30T08:32:28.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.803+0000 7f9669400640 1 -- 172.21.15.32:0/757327124 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9664104bd0 msgr2=0x7f9664104fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:28.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.803+0000 7f9669400640 1 --2- 172.21.15.32:0/757327124 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9664104bd0 0x7f9664104fe0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f9650009900 tx=0x7f965002a410 comp rx=0 tx=0).stop 2024-04-30T08:32:28.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.803+0000 7f9669400640 1 -- 172.21.15.32:0/757327124 shutdown_connections 2024-04-30T08:32:28.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.803+0000 7f9669400640 1 --2- 172.21.15.32:0/757327124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96641098d0 0x7f966410bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:28.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.803+0000 7f9669400640 1 --2- 172.21.15.32:0/757327124 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9664104bd0 0x7f9664104fe0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:28.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.803+0000 7f9669400640 1 --2- 172.21.15.32:0/757327124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9664103990 0x7f9664104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:28.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.803+0000 7f9669400640 1 -- 172.21.15.32:0/757327124 >> 172.21.15.32:0/757327124 conn(0x7f96640feba0 msgr2=0x7f9664101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:28.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.804+0000 7f9669400640 1 -- 172.21.15.32:0/757327124 shutdown_connections 2024-04-30T08:32:28.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.804+0000 7f9669400640 1 -- 172.21.15.32:0/757327124 wait complete. 2024-04-30T08:32:28.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.805+0000 7f9669400640 1 Processor -- start 2024-04-30T08:32:28.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.805+0000 7f9669400640 1 -- start start 2024-04-30T08:32:28.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.805+0000 7f9669400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9664103990 0x7f966419a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:28.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.805+0000 7f9669400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9664104bd0 0x7f966419a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:28.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.805+0000 7f9669400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96641098d0 0x7f96641a15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:28.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.805+0000 7f9669400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f966410f510 con 0x7f96641098d0 2024-04-30T08:32:28.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.805+0000 7f9669400640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f966410f390 con 0x7f9664104bd0 2024-04-30T08:32:28.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.805+0000 7f9669400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f966410f690 con 0x7f9664103990 2024-04-30T08:32:28.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.805+0000 7f9663400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9664104bd0 0x7f966419a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:28.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.805+0000 7f9663400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9664104bd0 0x7f966419a620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:59546/0 (socket says 172.21.15.32:59546) 2024-04-30T08:32:28.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.806+0000 7f9663400640 1 -- 172.21.15.32:0/1018293583 learned_addr learned my addr 172.21.15.32:0/1018293583 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:28.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.806+0000 7f9668a00640 1 --2- 172.21.15.32:0/1018293583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96641098d0 0x7f96641a15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:28.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.806+0000 7f9663e00640 1 --2- 172.21.15.32:0/1018293583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9664103990 0x7f966419a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:28.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.806+0000 7f9663400640 1 -- 172.21.15.32:0/1018293583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9664103990 msgr2=0x7f966419a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:28.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.806+0000 7f9663400640 1 --2- 172.21.15.32:0/1018293583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9664103990 0x7f966419a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:28.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.806+0000 7f9663400640 1 -- 172.21.15.32:0/1018293583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96641098d0 msgr2=0x7f96641a15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:28.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.806+0000 7f9663400640 1 --2- 172.21.15.32:0/1018293583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96641098d0 0x7f96641a15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:28.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.806+0000 7f9663400640 1 -- 172.21.15.32:0/1018293583 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96641a1c20 con 0x7f9664104bd0 2024-04-30T08:32:28.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.806+0000 7f9663400640 1 --2- 172.21.15.32:0/1018293583 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9664104bd0 0x7f966419a620 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9650031440 tx=0x7f9650003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:28.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.807+0000 7f9660c00640 1 -- 172.21.15.32:0/1018293583 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f965003f070 con 0x7f9664104bd0 2024-04-30T08:32:28.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.807+0000 7f9660c00640 1 -- 172.21.15.32:0/1018293583 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9650031c50 con 0x7f9664104bd0 2024-04-30T08:32:28.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.807+0000 7f9660c00640 1 -- 172.21.15.32:0/1018293583 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9650035030 con 0x7f9664104bd0 2024-04-30T08:32:28.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.807+0000 7f9669400640 1 -- 172.21.15.32:0/1018293583 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96641a1eb0 con 0x7f9664104bd0 2024-04-30T08:32:28.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.807+0000 7f9669400640 1 -- 172.21.15.32:0/1018293583 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96641a22c0 con 0x7f9664104bd0 2024-04-30T08:32:28.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.808+0000 7f9669400640 1 -- 172.21.15.32:0/1018293583 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f96641086e0 con 0x7f9664104bd0 2024-04-30T08:32:28.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.810+0000 7f9660c00640 1 -- 172.21.15.32:0/1018293583 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f9650042070 con 0x7f9664104bd0 2024-04-30T08:32:28.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.811+0000 7f9660c00640 1 --2- 172.21.15.32:0/1018293583 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f96480afae0 0x7f96480b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:28.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.811+0000 7f9660c00640 1 -- 172.21.15.32:0/1018293583 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f96500ff650 con 0x7f9664104bd0 2024-04-30T08:32:28.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.812+0000 7f9663e00640 1 --2- 172.21.15.32:0/1018293583 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f96480afae0 0x7f96480b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:28.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.812+0000 7f9663e00640 1 --2- 172.21.15.32:0/1018293583 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f96480afae0 0x7f96480b1ff0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9664104590 tx=0x7f9654005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:28.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:28.815+0000 7f9660c00640 1 -- 172.21.15.32:0/1018293583 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f96500c7940 con 0x7f9664104bd0 2024-04-30T08:32:28.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:28 smithi032 ceph-mon[28331]: pgmap v340: 129 pgs: 129 active+clean; 581 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-04-30T08:32:28.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:28 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2537664133' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-30T08:32:28.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:28 smithi032 ceph-mon[28331]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-30T08:32:28.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:28 smithi032 ceph-mon[28331]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-30T08:32:28.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:28 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] up:active 2024-04-30T08:32:28.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:28 smithi032 ceph-mon[28331]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-30T08:32:28.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:28 smithi032 ceph-mon[28331]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 9 up:standby 2024-04-30T08:32:28.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:28 smithi032 ceph-mon[28331]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=h=up:creating} 8 up:standby 2024-04-30T08:32:28.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:28 smithi032 ceph-mon[28331]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-04-30T08:32:29.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.044+0000 7f9669400640 1 -- 172.21.15.32:0/1018293583 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f9664063060 con 0x7f96480afae0 2024-04-30T08:32:29.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:28 smithi189 ceph-mon[33038]: pgmap v340: 129 pgs: 129 active+clean; 581 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-04-30T08:32:29.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:28 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2537664133' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-30T08:32:29.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:28 smithi189 ceph-mon[33038]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-30T08:32:29.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:28 smithi189 ceph-mon[33038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-30T08:32:29.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:28 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] up:active 2024-04-30T08:32:29.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:28 smithi189 ceph-mon[33038]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-30T08:32:29.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:28 smithi189 ceph-mon[33038]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 9 up:standby 2024-04-30T08:32:29.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:28 smithi189 ceph-mon[33038]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=h=up:creating} 8 up:standby 2024-04-30T08:32:29.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:28 smithi189 ceph-mon[33038]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-04-30T08:32:29.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:28 smithi071 ceph-mon[33385]: pgmap v340: 129 pgs: 129 active+clean; 581 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 85 B/s wr, 0 op/s 2024-04-30T08:32:29.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:28 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2537664133' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-30T08:32:29.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:28 smithi071 ceph-mon[33385]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-30T08:32:29.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:28 smithi071 ceph-mon[33385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-30T08:32:29.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:28 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] up:active 2024-04-30T08:32:29.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:28 smithi071 ceph-mon[33385]: daemon mds.h assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-04-30T08:32:29.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:28 smithi071 ceph-mon[33385]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=i=up:active} 9 up:standby 2024-04-30T08:32:29.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:28 smithi071 ceph-mon[33385]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=h=up:creating} 8 up:standby 2024-04-30T08:32:29.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:28 smithi071 ceph-mon[33385]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-04-30T08:32:29.442 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:32:29 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:32:29.054+0000 7fbb1a000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-30T08:32:29.442 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:32:29 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:32:29.054+0000 7fbb1a000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-30T08:32:29.442 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:32:29 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:32:29.054+0000 7fbb1a000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-30T08:32:29.442 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:32:29 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:32:29.054+0000 7fbb1a000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-30T08:32:29.442 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:32:29 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:32:29.054+0000 7fbb1a000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-30T08:32:29.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.563+0000 7f9660c00640 1 -- 172.21.15.32:0/1018293583 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f9664063060 con 0x7f96480afae0 2024-04-30T08:32:29.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.566+0000 7f9669400640 1 -- 172.21.15.32:0/1018293583 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f96480afae0 msgr2=0x7f96480b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:29.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.566+0000 7f9669400640 1 --2- 172.21.15.32:0/1018293583 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f96480afae0 0x7f96480b1ff0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9664104590 tx=0x7f9654005f90 comp rx=0 tx=0).stop 2024-04-30T08:32:29.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.566+0000 7f9669400640 1 -- 172.21.15.32:0/1018293583 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9664104bd0 msgr2=0x7f966419a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:29.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.566+0000 7f9669400640 1 --2- 172.21.15.32:0/1018293583 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9664104bd0 0x7f966419a620 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9650031440 tx=0x7f9650003c10 comp rx=0 tx=0).stop 2024-04-30T08:32:29.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.566+0000 7f9669400640 1 -- 172.21.15.32:0/1018293583 shutdown_connections 2024-04-30T08:32:29.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.566+0000 7f9669400640 1 --2- 172.21.15.32:0/1018293583 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f96480afae0 0x7f96480b1ff0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:29.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.566+0000 7f9669400640 1 --2- 172.21.15.32:0/1018293583 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f96641098d0 0x7f96641a15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:29.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.566+0000 7f9669400640 1 --2- 172.21.15.32:0/1018293583 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9664104bd0 0x7f966419a620 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:29.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.566+0000 7f9669400640 1 --2- 172.21.15.32:0/1018293583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9664103990 0x7f966419a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:29.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.566+0000 7f9669400640 1 -- 172.21.15.32:0/1018293583 >> 172.21.15.32:0/1018293583 conn(0x7f96640feba0 msgr2=0x7f96641058b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:29.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.566+0000 7f9669400640 1 -- 172.21.15.32:0/1018293583 shutdown_connections 2024-04-30T08:32:29.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.566+0000 7f9669400640 1 -- 172.21.15.32:0/1018293583 wait complete. 2024-04-30T08:32:29.595 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-30T08:32:29.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.871+0000 7f5451000640 1 -- 172.21.15.32:0/2672459278 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f544c103990 msgr2=0x7f544c104690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:29.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.871+0000 7f5451000640 1 --2- 172.21.15.32:0/2672459278 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f544c103990 0x7f544c104690 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f543c009ca0 tx=0x7f543c02a8c0 comp rx=0 tx=0).stop 2024-04-30T08:32:29.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.871+0000 7f5451000640 1 -- 172.21.15.32:0/2672459278 shutdown_connections 2024-04-30T08:32:29.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.871+0000 7f5451000640 1 --2- 172.21.15.32:0/2672459278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f544c1098d0 0x7f544c10bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:29.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.871+0000 7f5451000640 1 --2- 172.21.15.32:0/2672459278 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f544c104bd0 0x7f544c104fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:29.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.871+0000 7f5451000640 1 --2- 172.21.15.32:0/2672459278 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f544c103990 0x7f544c104690 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:29.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.871+0000 7f5451000640 1 -- 172.21.15.32:0/2672459278 >> 172.21.15.32:0/2672459278 conn(0x7f544c0feba0 msgr2=0x7f544c101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:29.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.871+0000 7f5451000640 1 -- 172.21.15.32:0/2672459278 shutdown_connections 2024-04-30T08:32:29.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.871+0000 7f5451000640 1 -- 172.21.15.32:0/2672459278 wait complete. 2024-04-30T08:32:29.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.872+0000 7f5451000640 1 Processor -- start 2024-04-30T08:32:29.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.873+0000 7f5451000640 1 -- start start 2024-04-30T08:32:29.875 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.873+0000 7f5451000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f544c103990 0x7f544c19a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:29.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.873+0000 7f5451000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f544c104bd0 0x7f544c19a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:29.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.873+0000 7f5451000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f544c1098d0 0x7f544c1a15b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:29.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.873+0000 7f5451000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f544c10f4e0 con 0x7f544c103990 2024-04-30T08:32:29.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.873+0000 7f5451000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f544c10f360 con 0x7f544c1098d0 2024-04-30T08:32:29.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.873+0000 7f5451000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f544c10f660 con 0x7f544c104bd0 2024-04-30T08:32:29.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.873+0000 7f544b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f544c103990 0x7f544c19a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:29.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.873+0000 7f544b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f544c103990 0x7f544c19a0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45142/0 (socket says 172.21.15.32:45142) 2024-04-30T08:32:29.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.873+0000 7f544b400640 1 -- 172.21.15.32:0/3306007932 learned_addr learned my addr 172.21.15.32:0/3306007932 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:29.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.873+0000 7f544be00640 1 --2- 172.21.15.32:0/3306007932 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f544c1098d0 0x7f544c1a15b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:29.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.874+0000 7f544aa00640 1 --2- 172.21.15.32:0/3306007932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f544c104bd0 0x7f544c19a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:29.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.874+0000 7f544b400640 1 -- 172.21.15.32:0/3306007932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f544c104bd0 msgr2=0x7f544c19a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:29.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.874+0000 7f544b400640 1 --2- 172.21.15.32:0/3306007932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f544c104bd0 0x7f544c19a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:29.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.874+0000 7f544b400640 1 -- 172.21.15.32:0/3306007932 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f544c1098d0 msgr2=0x7f544c1a15b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:29.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.874+0000 7f544b400640 1 --2- 172.21.15.32:0/3306007932 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f544c1098d0 0x7f544c1a15b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:29.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.874+0000 7f544b400640 1 -- 172.21.15.32:0/3306007932 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f544c1a1c20 con 0x7f544c103990 2024-04-30T08:32:29.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.874+0000 7f544be00640 1 --2- 172.21.15.32:0/3306007932 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f544c1098d0 0x7f544c1a15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:32:29.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.874+0000 7f544b400640 1 --2- 172.21.15.32:0/3306007932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f544c103990 0x7f544c19a0e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f543c000f80 tx=0x7f543c003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:29.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.875+0000 7f5443e00640 1 -- 172.21.15.32:0/3306007932 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f543c0317c0 con 0x7f544c103990 2024-04-30T08:32:29.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.875+0000 7f5451000640 1 -- 172.21.15.32:0/3306007932 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f544c1a1eb0 con 0x7f544c103990 2024-04-30T08:32:29.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.875+0000 7f5443e00640 1 -- 172.21.15.32:0/3306007932 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f543c0044f0 con 0x7f544c103990 2024-04-30T08:32:29.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.875+0000 7f5443e00640 1 -- 172.21.15.32:0/3306007932 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f543c02fd60 con 0x7f544c103990 2024-04-30T08:32:29.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.875+0000 7f5451000640 1 -- 172.21.15.32:0/3306007932 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f544c1a2320 con 0x7f544c103990 2024-04-30T08:32:29.878 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.876+0000 7f5451000640 1 -- 172.21.15.32:0/3306007932 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f544c1086e0 con 0x7f544c103990 2024-04-30T08:32:29.880 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.879+0000 7f5443e00640 1 -- 172.21.15.32:0/3306007932 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7f543c004040 con 0x7f544c103990 2024-04-30T08:32:29.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.881+0000 7f5443e00640 1 --2- 172.21.15.32:0/3306007932 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f54300afae0 0x7f54300b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:29.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.881+0000 7f5443e00640 1 -- 172.21.15.32:0/3306007932 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7f543c100590 con 0x7f544c103990 2024-04-30T08:32:29.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.881+0000 7f544aa00640 1 --2- 172.21.15.32:0/3306007932 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f54300afae0 0x7f54300b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:29.883 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.882+0000 7f544aa00640 1 --2- 172.21.15.32:0/3306007932 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f54300afae0 0x7f54300b1ff0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f544c19b180 tx=0x7f543400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:29.884 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:29.883+0000 7f5443e00640 1 -- 172.21.15.32:0/3306007932 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f543c0c8ed0 con 0x7f544c103990 2024-04-30T08:32:30.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:29 smithi189 ceph-mon[33038]: from='client.24830 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:32:30.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:29 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] up:active 2024-04-30T08:32:30.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:29 smithi189 ceph-mon[33038]: daemon mds.c assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-30T08:32:30.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:29 smithi189 ceph-mon[33038]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-30T08:32:30.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:29 smithi189 ceph-mon[33038]: Cluster is now healthy 2024-04-30T08:32:30.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:29 smithi189 ceph-mon[33038]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=h=up:active} 8 up:standby 2024-04-30T08:32:30.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:29 smithi189 ceph-mon[33038]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-04-30T08:32:30.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:29 smithi189 ceph-mon[33038]: daemon mds.c is now active in filesystem cephfs as rank 4 2024-04-30T08:32:30.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:29 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-30T08:32:30.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:29 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-04-30T08:32:30.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:29 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2024-04-30T08:32:30.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:29 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]: dispatch 2024-04-30T08:32:30.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:29 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]: dispatch 2024-04-30T08:32:30.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:29 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-04-30T08:32:30.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:29 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]: dispatch 2024-04-30T08:32:30.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:29 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-04-30T08:32:30.052 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:29 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-04-30T08:32:30.052 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:29 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2024-04-30T08:32:30.115 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:29 smithi071 ceph-mon[33385]: from='client.24830 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:32:30.116 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:29 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] up:active 2024-04-30T08:32:30.116 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:29 smithi071 ceph-mon[33385]: daemon mds.c assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-30T08:32:30.116 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:29 smithi071 ceph-mon[33385]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-30T08:32:30.116 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:29 smithi071 ceph-mon[33385]: Cluster is now healthy 2024-04-30T08:32:30.116 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:29 smithi071 ceph-mon[33385]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=h=up:active} 8 up:standby 2024-04-30T08:32:30.116 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:29 smithi071 ceph-mon[33385]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-04-30T08:32:30.116 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:29 smithi071 ceph-mon[33385]: daemon mds.c is now active in filesystem cephfs as rank 4 2024-04-30T08:32:30.116 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:29 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-30T08:32:30.116 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:29 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-04-30T08:32:30.116 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:29 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2024-04-30T08:32:30.117 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:29 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]: dispatch 2024-04-30T08:32:30.117 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:29 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]: dispatch 2024-04-30T08:32:30.117 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:29 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-04-30T08:32:30.117 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:29 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]: dispatch 2024-04-30T08:32:30.117 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:29 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-04-30T08:32:30.117 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:29 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-04-30T08:32:30.117 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:29 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2024-04-30T08:32:30.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.114+0000 7f5451000640 1 -- 172.21.15.32:0/3306007932 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f544c063060 con 0x7f54300afae0 2024-04-30T08:32:30.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.153+0000 7f5443e00640 1 -- 172.21.15.32:0/3306007932 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f544c063060 con 0x7f54300afae0 2024-04-30T08:32:30.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.155+0000 7f5451000640 1 -- 172.21.15.32:0/3306007932 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f54300afae0 msgr2=0x7f54300b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:30.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.155+0000 7f5451000640 1 --2- 172.21.15.32:0/3306007932 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f54300afae0 0x7f54300b1ff0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f544c19b180 tx=0x7f543400b410 comp rx=0 tx=0).stop 2024-04-30T08:32:30.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.155+0000 7f5451000640 1 -- 172.21.15.32:0/3306007932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f544c103990 msgr2=0x7f544c19a0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:30.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.155+0000 7f5451000640 1 --2- 172.21.15.32:0/3306007932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f544c103990 0x7f544c19a0e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f543c000f80 tx=0x7f543c003ba0 comp rx=0 tx=0).stop 2024-04-30T08:32:30.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.155+0000 7f5451000640 1 -- 172.21.15.32:0/3306007932 shutdown_connections 2024-04-30T08:32:30.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.155+0000 7f5451000640 1 --2- 172.21.15.32:0/3306007932 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f54300afae0 0x7f54300b1ff0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:30.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.155+0000 7f5451000640 1 --2- 172.21.15.32:0/3306007932 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f544c1098d0 0x7f544c1a15b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:30.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.155+0000 7f5451000640 1 --2- 172.21.15.32:0/3306007932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f544c104bd0 0x7f544c19a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:30.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.155+0000 7f5451000640 1 --2- 172.21.15.32:0/3306007932 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f544c103990 0x7f544c19a0e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:30.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.155+0000 7f5451000640 1 -- 172.21.15.32:0/3306007932 >> 172.21.15.32:0/3306007932 conn(0x7f544c0feba0 msgr2=0x7f544c1058b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:30.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.155+0000 7f5451000640 1 -- 172.21.15.32:0/3306007932 shutdown_connections 2024-04-30T08:32:30.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.156+0000 7f5451000640 1 -- 172.21.15.32:0/3306007932 wait complete. 2024-04-30T08:32:30.180 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:29 smithi032 ceph-mon[28331]: from='client.24830 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:32:30.180 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:29 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] up:active 2024-04-30T08:32:30.180 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:29 smithi032 ceph-mon[28331]: daemon mds.c assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-04-30T08:32:30.180 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:29 smithi032 ceph-mon[28331]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-30T08:32:30.180 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:29 smithi032 ceph-mon[28331]: Cluster is now healthy 2024-04-30T08:32:30.180 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:29 smithi032 ceph-mon[28331]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=i=up:active,3=h=up:active} 8 up:standby 2024-04-30T08:32:30.180 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:29 smithi032 ceph-mon[28331]: fsmap cephfs:5 1 up:creating 7 up:standby 4 up:active 2024-04-30T08:32:30.180 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:29 smithi032 ceph-mon[28331]: daemon mds.c is now active in filesystem cephfs as rank 4 2024-04-30T08:32:30.180 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:29 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-04-30T08:32:30.181 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:29 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-04-30T08:32:30.181 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:29 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]: dispatch 2024-04-30T08:32:30.181 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:29 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]: dispatch 2024-04-30T08:32:30.181 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:29 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]: dispatch 2024-04-30T08:32:30.181 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:29 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]: dispatch 2024-04-30T08:32:30.181 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:29 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]: dispatch 2024-04-30T08:32:30.181 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:29 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-04-30T08:32:30.181 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:29 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]: dispatch 2024-04-30T08:32:30.181 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:29 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]: dispatch 2024-04-30T08:32:30.182 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-30T08:32:30.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.452+0000 7fdfbd600640 1 -- 172.21.15.32:0/1803822263 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdfb8076a70 msgr2=0x7fdfb810b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:30.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.452+0000 7fdfbd600640 1 --2- 172.21.15.32:0/1803822263 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdfb8076a70 0x7fdfb810b5a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4009900 tx=0x7fdfa402a3f0 comp rx=0 tx=0).stop 2024-04-30T08:32:30.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.452+0000 7fdfbd600640 1 -- 172.21.15.32:0/1803822263 shutdown_connections 2024-04-30T08:32:30.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.452+0000 7fdfbd600640 1 --2- 172.21.15.32:0/1803822263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfb810bae0 0x7fdfb810df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:30.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.452+0000 7fdfbd600640 1 --2- 172.21.15.32:0/1803822263 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdfb8076a70 0x7fdfb810b5a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:30.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.452+0000 7fdfbd600640 1 --2- 172.21.15.32:0/1803822263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfb8076120 0x7fdfb8076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:30.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.452+0000 7fdfbd600640 1 -- 172.21.15.32:0/1803822263 >> 172.21.15.32:0/1803822263 conn(0x7fdfb80fed60 msgr2=0x7fdfb8101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:30.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.452+0000 7fdfbd600640 1 -- 172.21.15.32:0/1803822263 shutdown_connections 2024-04-30T08:32:30.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.452+0000 7fdfbd600640 1 -- 172.21.15.32:0/1803822263 wait complete. 2024-04-30T08:32:30.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.453+0000 7fdfbd600640 1 Processor -- start 2024-04-30T08:32:30.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.453+0000 7fdfbd600640 1 -- start start 2024-04-30T08:32:30.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.454+0000 7fdfbd600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfb8076120 0x7fdfb819c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:30.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.454+0000 7fdfbd600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdfb8076a70 0x7fdfb819ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:30.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.454+0000 7fdfbd600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfb810bae0 0x7fdfb81a39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:30.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.454+0000 7fdfbd600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb81118c0 con 0x7fdfb8076120 2024-04-30T08:32:30.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.454+0000 7fdfbd600640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb8111740 con 0x7fdfb8076a70 2024-04-30T08:32:30.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.454+0000 7fdfbd600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fdfb8111a40 con 0x7fdfb810bae0 2024-04-30T08:32:30.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.454+0000 7fdfb6a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfb8076120 0x7fdfb819c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:30.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.454+0000 7fdfb6000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdfb8076a70 0x7fdfb819ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:30.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.454+0000 7fdfb6000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdfb8076a70 0x7fdfb819ca60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:59596/0 (socket says 172.21.15.32:59596) 2024-04-30T08:32:30.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.454+0000 7fdfb6a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfb8076120 0x7fdfb819c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45160/0 (socket says 172.21.15.32:45160) 2024-04-30T08:32:30.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.454+0000 7fdfb6000640 1 -- 172.21.15.32:0/1418704466 learned_addr learned my addr 172.21.15.32:0/1418704466 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:30.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.454+0000 7fdfb7400640 1 --2- 172.21.15.32:0/1418704466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfb810bae0 0x7fdfb81a39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:30.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.454+0000 7fdfb6a00640 1 -- 172.21.15.32:0/1418704466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfb810bae0 msgr2=0x7fdfb81a39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:30.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.454+0000 7fdfb6a00640 1 --2- 172.21.15.32:0/1418704466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfb810bae0 0x7fdfb81a39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:30.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.454+0000 7fdfb6a00640 1 -- 172.21.15.32:0/1418704466 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdfb8076a70 msgr2=0x7fdfb819ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:30.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.454+0000 7fdfb6a00640 1 --2- 172.21.15.32:0/1418704466 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdfb8076a70 0x7fdfb819ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:30.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.454+0000 7fdfb6a00640 1 -- 172.21.15.32:0/1418704466 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfb81a4060 con 0x7fdfb8076120 2024-04-30T08:32:30.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.455+0000 7fdfb6a00640 1 --2- 172.21.15.32:0/1418704466 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfb8076120 0x7fdfb819c520 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fdfa800e9c0 tx=0x7fdfa800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:30.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.455+0000 7fdfaf400640 1 -- 172.21.15.32:0/1418704466 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa800ccc0 con 0x7fdfb8076120 2024-04-30T08:32:30.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.455+0000 7fdfaf400640 1 -- 172.21.15.32:0/1418704466 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdfa8005020 con 0x7fdfb8076120 2024-04-30T08:32:30.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.455+0000 7fdfbd600640 1 -- 172.21.15.32:0/1418704466 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfb81a4350 con 0x7fdfb8076120 2024-04-30T08:32:30.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.455+0000 7fdfaf400640 1 -- 172.21.15.32:0/1418704466 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfa8004a00 con 0x7fdfb8076120 2024-04-30T08:32:30.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.455+0000 7fdfbd600640 1 -- 172.21.15.32:0/1418704466 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfb81a4850 con 0x7fdfb8076120 2024-04-30T08:32:30.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.456+0000 7fdfbd600640 1 -- 172.21.15.32:0/1418704466 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfb810aca0 con 0x7fdfb8076120 2024-04-30T08:32:30.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.459+0000 7fdfaf400640 1 -- 172.21.15.32:0/1418704466 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 16) ==== 147520+0+0 (secure 0 0 0) 0x7fdfa80051d0 con 0x7fdfb8076120 2024-04-30T08:32:30.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.460+0000 7fdfaf400640 1 --2- 172.21.15.32:0/1418704466 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fdf9c0afaf0 0x7fdf9c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:30.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.460+0000 7fdfaf400640 1 -- 172.21.15.32:0/1418704466 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 8802+0+0 (secure 0 0 0) 0x7fdfa80def00 con 0x7fdfb8076120 2024-04-30T08:32:30.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.461+0000 7fdfb6000640 1 --2- 172.21.15.32:0/1418704466 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fdf9c0afaf0 0x7fdf9c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:30.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.461+0000 7fdfb6000640 1 --2- 172.21.15.32:0/1418704466 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fdf9c0afaf0 0x7fdf9c0b2000 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4031440 tx=0x7fdfa40314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:30.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.463+0000 7fdfaf400640 1 -- 172.21.15.32:0/1418704466 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdfa80a7930 con 0x7fdfb8076120 2024-04-30T08:32:30.503 INFO:journalctl@ceph.mds.b.smithi071.stdout:Apr 30 08:32:30 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-b[64026]: 2024-04-30T08:32:30.114+0000 7f6ea3abc700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-04-30T08:32:30.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.711+0000 7fdfbd600640 1 -- 172.21.15.32:0/1418704466 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7fdfb8063060 con 0x7fdf9c0afaf0 2024-04-30T08:32:30.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.726+0000 7fdfaf400640 1 -- 172.21.15.32:0/1418704466 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7fdfb8063060 con 0x7fdf9c0afaf0 2024-04-30T08:32:30.727 INFO:teuthology.orchestra.run.smithi032.stdout:/volumes/qa/sv_0/acaaf4c0-9638-419b-8f8d-d5a9451751de 2024-04-30T08:32:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.729+0000 7fdfbd600640 1 -- 172.21.15.32:0/1418704466 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fdf9c0afaf0 msgr2=0x7fdf9c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.729+0000 7fdfbd600640 1 --2- 172.21.15.32:0/1418704466 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fdf9c0afaf0 0x7fdf9c0b2000 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdfa4031440 tx=0x7fdfa40314a0 comp rx=0 tx=0).stop 2024-04-30T08:32:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.729+0000 7fdfbd600640 1 -- 172.21.15.32:0/1418704466 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfb8076120 msgr2=0x7fdfb819c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:30.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.729+0000 7fdfbd600640 1 --2- 172.21.15.32:0/1418704466 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfb8076120 0x7fdfb819c520 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fdfa800e9c0 tx=0x7fdfa800ee80 comp rx=0 tx=0).stop 2024-04-30T08:32:30.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.729+0000 7fdfbd600640 1 -- 172.21.15.32:0/1418704466 shutdown_connections 2024-04-30T08:32:30.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.729+0000 7fdfbd600640 1 --2- 172.21.15.32:0/1418704466 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fdf9c0afaf0 0x7fdf9c0b2000 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:30.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.729+0000 7fdfbd600640 1 --2- 172.21.15.32:0/1418704466 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdfb810bae0 0x7fdfb81a39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:30.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.729+0000 7fdfbd600640 1 --2- 172.21.15.32:0/1418704466 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdfb8076a70 0x7fdfb819ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:30.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.729+0000 7fdfbd600640 1 --2- 172.21.15.32:0/1418704466 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdfb8076120 0x7fdfb819c520 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:30.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.729+0000 7fdfbd600640 1 -- 172.21.15.32:0/1418704466 >> 172.21.15.32:0/1418704466 conn(0x7fdfb80fed60 msgr2=0x7fdfb80ffba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:30.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.729+0000 7fdfbd600640 1 -- 172.21.15.32:0/1418704466 shutdown_connections 2024-04-30T08:32:30.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:30.729+0000 7fdfbd600640 1 -- 172.21.15.32:0/1418704466 wait complete. 2024-04-30T08:32:30.757 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-30T08:32:31.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.040+0000 7fd39a200640 1 -- 172.21.15.32:0/3704933789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd394105d80 msgr2=0x7fd394106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:31.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.040+0000 7fd39a200640 1 --2- 172.21.15.32:0/3704933789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd394105d80 0x7fd394106a80 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd384009900 tx=0x7fd38402a3f0 comp rx=0 tx=0).stop 2024-04-30T08:32:31.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.040+0000 7fd39a200640 1 -- 172.21.15.32:0/3704933789 shutdown_connections 2024-04-30T08:32:31.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.040+0000 7fd39a200640 1 --2- 172.21.15.32:0/3704933789 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd39410bcc0 0x7fd39410e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:31.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.040+0000 7fd39a200640 1 --2- 172.21.15.32:0/3704933789 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd394106fc0 0x7fd3941073d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:31.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.040+0000 7fd39a200640 1 --2- 172.21.15.32:0/3704933789 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd394105d80 0x7fd394106a80 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:31.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.040+0000 7fd39a200640 1 -- 172.21.15.32:0/3704933789 >> 172.21.15.32:0/3704933789 conn(0x7fd394100f70 msgr2=0x7fd394103420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:31.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.040+0000 7fd39a200640 1 -- 172.21.15.32:0/3704933789 shutdown_connections 2024-04-30T08:32:31.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.040+0000 7fd39a200640 1 -- 172.21.15.32:0/3704933789 wait complete. 2024-04-30T08:32:31.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.041+0000 7fd39a200640 1 Processor -- start 2024-04-30T08:32:31.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.042+0000 7fd39a200640 1 -- start start 2024-04-30T08:32:31.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.042+0000 7fd39a200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd394105d80 0x7fd39419c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:31.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.042+0000 7fd39a200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd394106fc0 0x7fd39419ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:31.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.042+0000 7fd39a200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd39410bcc0 0x7fd3941a3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:31.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.042+0000 7fd39a200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd3941118b0 con 0x7fd394105d80 2024-04-30T08:32:31.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.042+0000 7fd39a200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd394111730 con 0x7fd39410bcc0 2024-04-30T08:32:31.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.042+0000 7fd39a200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd394111a30 con 0x7fd394106fc0 2024-04-30T08:32:31.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.042+0000 7fd393400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd394105d80 0x7fd39419c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:31.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.042+0000 7fd393e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd39410bcc0 0x7fd3941a3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:31.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.042+0000 7fd393400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd394105d80 0x7fd39419c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45178/0 (socket says 172.21.15.32:45178) 2024-04-30T08:32:31.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.042+0000 7fd393e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd39410bcc0 0x7fd3941a3990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:59618/0 (socket says 172.21.15.32:59618) 2024-04-30T08:32:31.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.043+0000 7fd393400640 1 -- 172.21.15.32:0/2751670411 learned_addr learned my addr 172.21.15.32:0/2751670411 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:31.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.043+0000 7fd392a00640 1 --2- 172.21.15.32:0/2751670411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd394106fc0 0x7fd39419ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:31.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.043+0000 7fd393e00640 1 -- 172.21.15.32:0/2751670411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd394106fc0 msgr2=0x7fd39419ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:31.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.043+0000 7fd393e00640 1 --2- 172.21.15.32:0/2751670411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd394106fc0 0x7fd39419ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:31.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.043+0000 7fd393e00640 1 -- 172.21.15.32:0/2751670411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd394105d80 msgr2=0x7fd39419c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:31.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.043+0000 7fd393e00640 1 --2- 172.21.15.32:0/2751670411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd394105d80 0x7fd39419c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:31.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.043+0000 7fd393e00640 1 -- 172.21.15.32:0/2751670411 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3941a4000 con 0x7fd39410bcc0 2024-04-30T08:32:31.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.043+0000 7fd393e00640 1 --2- 172.21.15.32:0/2751670411 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd39410bcc0 0x7fd3941a3990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd38c009fd0 tx=0x7fd38c00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:31.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.043+0000 7fd38be00640 1 -- 172.21.15.32:0/2751670411 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd38c00ce20 con 0x7fd39410bcc0 2024-04-30T08:32:31.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.044+0000 7fd38be00640 1 -- 172.21.15.32:0/2751670411 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd38c005020 con 0x7fd39410bcc0 2024-04-30T08:32:31.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.044+0000 7fd39a200640 1 -- 172.21.15.32:0/2751670411 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3941a42f0 con 0x7fd39410bcc0 2024-04-30T08:32:31.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.044+0000 7fd38be00640 1 -- 172.21.15.32:0/2751670411 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd38c010690 con 0x7fd39410bcc0 2024-04-30T08:32:31.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.044+0000 7fd39a200640 1 -- 172.21.15.32:0/2751670411 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3941a4840 con 0x7fd39410bcc0 2024-04-30T08:32:31.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.045+0000 7fd39a200640 1 -- 172.21.15.32:0/2751670411 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd39410aad0 con 0x7fd39410bcc0 2024-04-30T08:32:31.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.047+0000 7fd38be00640 1 -- 172.21.15.32:0/2751670411 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fd38c0054d0 con 0x7fd39410bcc0 2024-04-30T08:32:31.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.049+0000 7fd38be00640 1 --2- 172.21.15.32:0/2751670411 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd3780afc00 0x7fd3780b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:31.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.049+0000 7fd38be00640 1 -- 172.21.15.32:0/2751670411 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 9092+0+0 (secure 0 0 0) 0x7fd38c0df7e0 con 0x7fd39410bcc0 2024-04-30T08:32:31.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.049+0000 7fd393400640 1 --2- 172.21.15.32:0/2751670411 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd3780afc00 0x7fd3780b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:31.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.050+0000 7fd393400640 1 --2- 172.21.15.32:0/2751670411 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd3780afc00 0x7fd3780b2110 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd384000c00 tx=0x7fd3840314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:31.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.053+0000 7fd38be00640 1 -- 172.21.15.32:0/2751670411 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd38c0a7ea0 con 0x7fd39410bcc0 2024-04-30T08:32:31.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:30 smithi032 ceph-mon[28331]: pgmap v341: 129 pgs: 129 active+clean; 583 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 5 op/s 2024-04-30T08:32:31.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:30 smithi032 ceph-mon[28331]: from='client.14718 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:32:31.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:30 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] up:active 2024-04-30T08:32:31.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:30 smithi032 ceph-mon[28331]: fsmap cephfs:5 7 up:standby 5 up:active 2024-04-30T08:32:31.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:30 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-30T08:32:31.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:30 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-04-30T08:32:31.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:30 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-04-30T08:32:31.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:30 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]': finished 2024-04-30T08:32:31.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:30 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]': finished 2024-04-30T08:32:31.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:30 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-04-30T08:32:31.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:30 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]': finished 2024-04-30T08:32:31.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:30 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-04-30T08:32:31.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:30 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-04-30T08:32:31.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:30 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2024-04-30T08:32:31.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:30 smithi032 ceph-mon[28331]: osdmap e82: 12 total, 12 up, 12 in 2024-04-30T08:32:31.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:30 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:32:31.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:30 smithi071 ceph-mon[33385]: pgmap v341: 129 pgs: 129 active+clean; 583 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 5 op/s 2024-04-30T08:32:31.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:30 smithi071 ceph-mon[33385]: from='client.14718 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:32:31.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:30 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] up:active 2024-04-30T08:32:31.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:30 smithi071 ceph-mon[33385]: fsmap cephfs:5 7 up:standby 5 up:active 2024-04-30T08:32:31.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:30 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-30T08:32:31.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:30 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-04-30T08:32:31.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:30 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-04-30T08:32:31.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:30 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]': finished 2024-04-30T08:32:31.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:30 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]': finished 2024-04-30T08:32:31.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:30 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-04-30T08:32:31.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:30 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]': finished 2024-04-30T08:32:31.255 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:30 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-04-30T08:32:31.255 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:30 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-04-30T08:32:31.255 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:30 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2024-04-30T08:32:31.255 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:30 smithi071 ceph-mon[33385]: osdmap e82: 12 total, 12 up, 12 in 2024-04-30T08:32:31.255 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:30 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:32:31.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.291+0000 7fd39a200640 1 -- 172.21.15.32:0/2751670411 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7fd394063060 con 0x7fd3780afc00 2024-04-30T08:32:31.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:30 smithi189 ceph-mon[33038]: pgmap v341: 129 pgs: 129 active+clean; 583 KiB data, 359 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s wr, 5 op/s 2024-04-30T08:32:31.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:30 smithi189 ceph-mon[33038]: from='client.14718 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:32:31.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:30 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] up:active 2024-04-30T08:32:31.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:30 smithi189 ceph-mon[33038]: fsmap cephfs:5 7 up:standby 5 up:active 2024-04-30T08:32:31.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:30 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-04-30T08:32:31.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:30 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-04-30T08:32:31.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:30 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [4, 8]}]': finished 2024-04-30T08:32:31.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:30 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.17", "id": [4, 6]}]': finished 2024-04-30T08:32:31.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:30 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1a", "id": [5, 7]}]': finished 2024-04-30T08:32:31.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:30 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.36", "id": [4, 9]}]': finished 2024-04-30T08:32:31.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:30 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.8", "id": [5, 8]}]': finished 2024-04-30T08:32:31.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:30 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-04-30T08:32:31.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:30 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.13", "id": [9, 8]}]': finished 2024-04-30T08:32:31.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:30 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.14", "id": [9, 8]}]': finished 2024-04-30T08:32:31.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:30 smithi189 ceph-mon[33038]: osdmap e82: 12 total, 12 up, 12 in 2024-04-30T08:32:31.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:30 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:32:31.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.334+0000 7fd38be00640 1 -- 172.21.15.32:0/2751670411 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fd394063060 con 0x7fd3780afc00 2024-04-30T08:32:31.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.337+0000 7fd39a200640 1 -- 172.21.15.32:0/2751670411 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd3780afc00 msgr2=0x7fd3780b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:31.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.337+0000 7fd39a200640 1 --2- 172.21.15.32:0/2751670411 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd3780afc00 0x7fd3780b2110 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd384000c00 tx=0x7fd3840314a0 comp rx=0 tx=0).stop 2024-04-30T08:32:31.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.337+0000 7fd39a200640 1 -- 172.21.15.32:0/2751670411 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd39410bcc0 msgr2=0x7fd3941a3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:31.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.337+0000 7fd39a200640 1 --2- 172.21.15.32:0/2751670411 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd39410bcc0 0x7fd3941a3990 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fd38c009fd0 tx=0x7fd38c00ef10 comp rx=0 tx=0).stop 2024-04-30T08:32:31.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.337+0000 7fd39a200640 1 -- 172.21.15.32:0/2751670411 shutdown_connections 2024-04-30T08:32:31.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.337+0000 7fd39a200640 1 --2- 172.21.15.32:0/2751670411 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd3780afc00 0x7fd3780b2110 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:31.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.337+0000 7fd39a200640 1 --2- 172.21.15.32:0/2751670411 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd39410bcc0 0x7fd3941a3990 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:31.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.337+0000 7fd39a200640 1 --2- 172.21.15.32:0/2751670411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd394106fc0 0x7fd39419ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:31.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.337+0000 7fd39a200640 1 --2- 172.21.15.32:0/2751670411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd394105d80 0x7fd39419c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:31.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.337+0000 7fd39a200640 1 -- 172.21.15.32:0/2751670411 >> 172.21.15.32:0/2751670411 conn(0x7fd394100f70 msgr2=0x7fd394107ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:31.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.337+0000 7fd39a200640 1 -- 172.21.15.32:0/2751670411 shutdown_connections 2024-04-30T08:32:31.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.337+0000 7fd39a200640 1 -- 172.21.15.32:0/2751670411 wait complete. 2024-04-30T08:32:31.366 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-30T08:32:31.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.671+0000 7f94aa200640 1 -- 172.21.15.32:0/2449505154 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f94a4103990 msgr2=0x7f94a4104690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:31.673 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.671+0000 7f94aa200640 1 --2- 172.21.15.32:0/2449505154 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f94a4103990 0x7f94a4104690 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9498009860 tx=0x7f949802a340 comp rx=0 tx=0).stop 2024-04-30T08:32:31.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.672+0000 7f94aa200640 1 -- 172.21.15.32:0/2449505154 shutdown_connections 2024-04-30T08:32:31.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.672+0000 7f94aa200640 1 --2- 172.21.15.32:0/2449505154 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94a41098d0 0x7f94a410bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:31.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.672+0000 7f94aa200640 1 --2- 172.21.15.32:0/2449505154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94a4104bd0 0x7f94a4104fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:31.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.672+0000 7f94aa200640 1 --2- 172.21.15.32:0/2449505154 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f94a4103990 0x7f94a4104690 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:31.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.672+0000 7f94aa200640 1 -- 172.21.15.32:0/2449505154 >> 172.21.15.32:0/2449505154 conn(0x7f94a40feba0 msgr2=0x7f94a4101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:31.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.672+0000 7f94aa200640 1 -- 172.21.15.32:0/2449505154 shutdown_connections 2024-04-30T08:32:31.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.672+0000 7f94aa200640 1 -- 172.21.15.32:0/2449505154 wait complete. 2024-04-30T08:32:31.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.673+0000 7f94aa200640 1 Processor -- start 2024-04-30T08:32:31.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.673+0000 7f94aa200640 1 -- start start 2024-04-30T08:32:31.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.674+0000 7f94aa200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94a4103990 0x7f94a419c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:31.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.674+0000 7f94aa200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f94a4104bd0 0x7f94a419c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:31.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.674+0000 7f94aa200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94a41098d0 0x7f94a41a3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:31.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.674+0000 7f94aa200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f94a410f490 con 0x7f94a41098d0 2024-04-30T08:32:31.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.674+0000 7f94aa200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f94a410f310 con 0x7f94a4104bd0 2024-04-30T08:32:31.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.674+0000 7f94aa200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f94a410f610 con 0x7f94a4103990 2024-04-30T08:32:31.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.674+0000 7f94a9800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94a41098d0 0x7f94a41a3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:31.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.674+0000 7f94a8e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94a4103990 0x7f94a419c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:31.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.674+0000 7f94a3e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f94a4104bd0 0x7f94a419c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:31.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.674+0000 7f94a8e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94a4103990 0x7f94a419c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:40484/0 (socket says 172.21.15.32:40484) 2024-04-30T08:32:31.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.674+0000 7f94a8e00640 1 -- 172.21.15.32:0/3806381811 learned_addr learned my addr 172.21.15.32:0/3806381811 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:31.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.674+0000 7f94a9800640 1 --2- 172.21.15.32:0/3806381811 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94a41098d0 0x7f94a41a3850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45194/0 (socket says 172.21.15.32:45194) 2024-04-30T08:32:31.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.675+0000 7f94a3e00640 1 -- 172.21.15.32:0/3806381811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94a4103990 msgr2=0x7f94a419c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:31.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.675+0000 7f94a3e00640 1 --2- 172.21.15.32:0/3806381811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94a4103990 0x7f94a419c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:31.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.675+0000 7f94a3e00640 1 -- 172.21.15.32:0/3806381811 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94a41098d0 msgr2=0x7f94a41a3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:31.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.675+0000 7f94a3e00640 1 --2- 172.21.15.32:0/3806381811 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94a41098d0 0x7f94a41a3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:31.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.675+0000 7f94a3e00640 1 -- 172.21.15.32:0/3806381811 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94a41a3ec0 con 0x7f94a4104bd0 2024-04-30T08:32:31.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.675+0000 7f94a8e00640 1 --2- 172.21.15.32:0/3806381811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94a4103990 0x7f94a419c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:32:31.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.675+0000 7f94a3e00640 1 --2- 172.21.15.32:0/3806381811 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f94a4104bd0 0x7f94a419c8c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f949400e9f0 tx=0x7f949400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:31.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.675+0000 7f94a1600640 1 -- 172.21.15.32:0/3806381811 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f949400ccc0 con 0x7f94a4104bd0 2024-04-30T08:32:31.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.675+0000 7f94a1600640 1 -- 172.21.15.32:0/3806381811 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9494005020 con 0x7f94a4104bd0 2024-04-30T08:32:31.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.675+0000 7f94a1600640 1 -- 172.21.15.32:0/3806381811 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9494004a00 con 0x7f94a4104bd0 2024-04-30T08:32:31.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.675+0000 7f94aa200640 1 -- 172.21.15.32:0/3806381811 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94a41a41b0 con 0x7f94a4104bd0 2024-04-30T08:32:31.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.676+0000 7f94aa200640 1 -- 172.21.15.32:0/3806381811 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94a41a4700 con 0x7f94a4104bd0 2024-04-30T08:32:31.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.677+0000 7f94aa200640 1 -- 172.21.15.32:0/3806381811 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94a41a49c0 con 0x7f94a4104bd0 2024-04-30T08:32:31.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.680+0000 7f94a1600640 1 -- 172.21.15.32:0/3806381811 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f9494004ba0 con 0x7f94a4104bd0 2024-04-30T08:32:31.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.681+0000 7f94a1600640 1 --2- 172.21.15.32:0/3806381811 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f948c0afbf0 0x7f948c0b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:31.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.681+0000 7f94a1600640 1 -- 172.21.15.32:0/3806381811 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f94940df0c0 con 0x7f94a4104bd0 2024-04-30T08:32:31.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.681+0000 7f94a8e00640 1 --2- 172.21.15.32:0/3806381811 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f948c0afbf0 0x7f948c0b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:31.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.682+0000 7f94a8e00640 1 --2- 172.21.15.32:0/3806381811 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f948c0afbf0 0x7f948c0b2100 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f949802a930 tx=0x7f9498005780 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:31.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.685+0000 7f94a1600640 1 -- 172.21.15.32:0/3806381811 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f94940a7780 con 0x7f94a4104bd0 2024-04-30T08:32:31.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.930+0000 7f94aa200640 1 -- 172.21.15.32:0/3806381811 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f94a41006d0 con 0x7f948c0afbf0 2024-04-30T08:32:31.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.940+0000 7f94a1600640 1 -- 172.21.15.32:0/3806381811 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f94a41006d0 con 0x7f948c0afbf0 2024-04-30T08:32:31.941 INFO:teuthology.orchestra.run.smithi032.stdout:/volumes/qa/sv_1/201e6199-10b1-41e5-a912-eef3ace842a3 2024-04-30T08:32:31.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.942+0000 7f94aa200640 1 -- 172.21.15.32:0/3806381811 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f948c0afbf0 msgr2=0x7f948c0b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:31.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.942+0000 7f94aa200640 1 --2- 172.21.15.32:0/3806381811 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f948c0afbf0 0x7f948c0b2100 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f949802a930 tx=0x7f9498005780 comp rx=0 tx=0).stop 2024-04-30T08:32:31.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.942+0000 7f94aa200640 1 -- 172.21.15.32:0/3806381811 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f94a4104bd0 msgr2=0x7f94a419c8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:31.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.942+0000 7f94aa200640 1 --2- 172.21.15.32:0/3806381811 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f94a4104bd0 0x7f94a419c8c0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f949400e9f0 tx=0x7f949400eeb0 comp rx=0 tx=0).stop 2024-04-30T08:32:31.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.943+0000 7f94aa200640 1 -- 172.21.15.32:0/3806381811 shutdown_connections 2024-04-30T08:32:31.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.943+0000 7f94aa200640 1 --2- 172.21.15.32:0/3806381811 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f948c0afbf0 0x7f948c0b2100 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:31.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.943+0000 7f94aa200640 1 --2- 172.21.15.32:0/3806381811 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94a41098d0 0x7f94a41a3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:31.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.943+0000 7f94aa200640 1 --2- 172.21.15.32:0/3806381811 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f94a4104bd0 0x7f94a419c8c0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:31.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.943+0000 7f94aa200640 1 --2- 172.21.15.32:0/3806381811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94a4103990 0x7f94a419c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:31.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.943+0000 7f94aa200640 1 -- 172.21.15.32:0/3806381811 >> 172.21.15.32:0/3806381811 conn(0x7f94a40feba0 msgr2=0x7f94a40fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:31.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.943+0000 7f94aa200640 1 -- 172.21.15.32:0/3806381811 shutdown_connections 2024-04-30T08:32:31.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:31.943+0000 7f94aa200640 1 -- 172.21.15.32:0/3806381811 wait complete. 2024-04-30T08:32:31.972 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-30T08:32:32.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:31 smithi032 ceph-mon[28331]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:32:32.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:31 smithi032 ceph-mon[28331]: mgrmap e17: x(active, since 9m), standbys: z, y 2024-04-30T08:32:32.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:31 smithi032 ceph-mon[28331]: from='client.24854 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:32:32.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:31 smithi032 ceph-mon[28331]: osdmap e83: 12 total, 12 up, 12 in 2024-04-30T08:32:32.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:31 smithi071 ceph-mon[33385]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:32:32.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:31 smithi071 ceph-mon[33385]: mgrmap e17: x(active, since 9m), standbys: z, y 2024-04-30T08:32:32.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:31 smithi071 ceph-mon[33385]: from='client.24854 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:32:32.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:31 smithi071 ceph-mon[33385]: osdmap e83: 12 total, 12 up, 12 in 2024-04-30T08:32:32.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.254+0000 7ffa2aa00640 1 -- 172.21.15.32:0/1794051772 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa24079920 msgr2=0x7ffa24079d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:32.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.254+0000 7ffa2aa00640 1 --2- 172.21.15.32:0/1794051772 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa24079920 0x7ffa24079d30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c009be0 tx=0x7ffa1c02a6d0 comp rx=0 tx=0).stop 2024-04-30T08:32:32.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.254+0000 7ffa2aa00640 1 -- 172.21.15.32:0/1794051772 shutdown_connections 2024-04-30T08:32:32.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.254+0000 7ffa2aa00640 1 --2- 172.21.15.32:0/1794051772 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa24079920 0x7ffa24079d30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:32.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.254+0000 7ffa2aa00640 1 --2- 172.21.15.32:0/1794051772 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa24078c20 0x7ffa24079030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:32.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.254+0000 7ffa2aa00640 1 --2- 172.21.15.32:0/1794051772 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa24077990 0x7ffa240786e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:32.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.254+0000 7ffa2aa00640 1 -- 172.21.15.32:0/1794051772 >> 172.21.15.32:0/1794051772 conn(0x7ffa24100da0 msgr2=0x7ffa24103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:32.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.254+0000 7ffa2aa00640 1 -- 172.21.15.32:0/1794051772 shutdown_connections 2024-04-30T08:32:32.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.254+0000 7ffa2aa00640 1 -- 172.21.15.32:0/1794051772 wait complete. 2024-04-30T08:32:32.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.255+0000 7ffa2aa00640 1 Processor -- start 2024-04-30T08:32:32.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.256+0000 7ffa2aa00640 1 -- start start 2024-04-30T08:32:32.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.256+0000 7ffa2aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa24077990 0x7ffa241a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:32.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.256+0000 7ffa2aa00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa24078c20 0x7ffa241a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:32.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.256+0000 7ffa2aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa24079920 0x7ffa241a7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:32.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.256+0000 7ffa2aa00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ffa24115ac0 con 0x7ffa24079920 2024-04-30T08:32:32.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.256+0000 7ffa2aa00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ffa24115940 con 0x7ffa24078c20 2024-04-30T08:32:32.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.256+0000 7ffa2aa00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ffa24115c40 con 0x7ffa24077990 2024-04-30T08:32:32.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.256+0000 7ffa29600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa24077990 0x7ffa241a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:32.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.256+0000 7ffa29600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa24077990 0x7ffa241a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:40490/0 (socket says 172.21.15.32:40490) 2024-04-30T08:32:32.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.256+0000 7ffa29600640 1 -- 172.21.15.32:0/1370429111 learned_addr learned my addr 172.21.15.32:0/1370429111 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:32.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.256+0000 7ffa2a000640 1 --2- 172.21.15.32:0/1370429111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa24079920 0x7ffa241a7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:32.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.256+0000 7ffa28c00640 1 --2- 172.21.15.32:0/1370429111 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa24078c20 0x7ffa241a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:32.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.257+0000 7ffa29600640 1 -- 172.21.15.32:0/1370429111 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa24078c20 msgr2=0x7ffa241a0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:32.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.257+0000 7ffa29600640 1 --2- 172.21.15.32:0/1370429111 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa24078c20 0x7ffa241a0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:32.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.257+0000 7ffa29600640 1 -- 172.21.15.32:0/1370429111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa24079920 msgr2=0x7ffa241a7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:32.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.257+0000 7ffa29600640 1 --2- 172.21.15.32:0/1370429111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa24079920 0x7ffa241a7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:32.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.257+0000 7ffa29600640 1 -- 172.21.15.32:0/1370429111 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa241a8180 con 0x7ffa24077990 2024-04-30T08:32:32.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.257+0000 7ffa28c00640 1 --2- 172.21.15.32:0/1370429111 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa24078c20 0x7ffa241a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:32:32.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.257+0000 7ffa29600640 1 --2- 172.21.15.32:0/1370429111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa24077990 0x7ffa241a0640 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ffa1800e9a0 tx=0x7ffa1800ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:32.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.258+0000 7ffa22000640 1 -- 172.21.15.32:0/1370429111 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1800cd80 con 0x7ffa24077990 2024-04-30T08:32:32.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.258+0000 7ffa22000640 1 -- 172.21.15.32:0/1370429111 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa180035f0 con 0x7ffa24077990 2024-04-30T08:32:32.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.258+0000 7ffa2aa00640 1 -- 172.21.15.32:0/1370429111 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa241a8470 con 0x7ffa24077990 2024-04-30T08:32:32.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.258+0000 7ffa22000640 1 -- 172.21.15.32:0/1370429111 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa18010690 con 0x7ffa24077990 2024-04-30T08:32:32.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.258+0000 7ffa2aa00640 1 -- 172.21.15.32:0/1370429111 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa241a8a10 con 0x7ffa24077990 2024-04-30T08:32:32.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.259+0000 7ffa2aa00640 1 -- 172.21.15.32:0/1370429111 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa24112170 con 0x7ffa24077990 2024-04-30T08:32:32.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.262+0000 7ffa22000640 1 -- 172.21.15.32:0/1370429111 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7ffa18004000 con 0x7ffa24077990 2024-04-30T08:32:32.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.263+0000 7ffa22000640 1 --2- 172.21.15.32:0/1370429111 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ffa0c0afbf0 0x7ffa0c0b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:32.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.263+0000 7ffa22000640 1 -- 172.21.15.32:0/1370429111 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7ffa180df480 con 0x7ffa24077990 2024-04-30T08:32:32.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.263+0000 7ffa28c00640 1 --2- 172.21.15.32:0/1370429111 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ffa0c0afbf0 0x7ffa0c0b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:32.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.264+0000 7ffa28c00640 1 --2- 172.21.15.32:0/1370429111 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ffa0c0afbf0 0x7ffa0c0b2100 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ffa241a16e0 tx=0x7ffa10005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:32.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.265+0000 7ffa22000640 1 -- 172.21.15.32:0/1370429111 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffa180042f0 con 0x7ffa24077990 2024-04-30T08:32:32.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:31 smithi189 ceph-mon[33038]: from='client.14724 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:32:32.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:31 smithi189 ceph-mon[33038]: mgrmap e17: x(active, since 9m), standbys: z, y 2024-04-30T08:32:32.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:31 smithi189 ceph-mon[33038]: from='client.24854 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:32:32.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:31 smithi189 ceph-mon[33038]: osdmap e83: 12 total, 12 up, 12 in 2024-04-30T08:32:32.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.516+0000 7ffa2aa00640 1 -- 172.21.15.32:0/1370429111 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa2402dbd0 con 0x7ffa24077990 2024-04-30T08:32:32.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.517+0000 7ffa22000640 1 -- 172.21.15.32:0/1370429111 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7ffa180a7bd0 con 0x7ffa24077990 2024-04-30T08:32:32.519 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:32:32.519 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2235842676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2235842676},{"type":"v1","addr":"172.21.15.32:6835","nonce":2235842676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/3577009355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":3577009355},{"type":"v1","addr":"172.21.15.32:6841","nonce":3577009355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-30T08:31:52.236668+0000","modified":"2024-04-30T08:32:30.541714+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24541,"mds_3":24533,"mds_4":24517},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/463203038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":463203038},{"type":"v1","addr":"172.21.15.189:6833","nonce":463203038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6833/3635143591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3635143591},{"type":"v1","addr":"172.21.15.71:6833","nonce":3635143591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6837/2569449994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2569449994},{"type":"v1","addr":"172.21.15.32:6837","nonce":2569449994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/278686565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":278686565},{"type":"v1","addr":"172.21.15.189:6835","nonce":278686565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6835/1558303056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1558303056},{"type":"v1","addr":"172.21.15.71:6835","nonce":1558303056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.189:6837/686520873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":686520873},{"type":"v1","addr":"172.21.15.189:6837","nonce":686520873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.71:6837/2324039932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2324039932},{"type":"v1","addr":"172.21.15.71:6837","nonce":2324039932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6839/352970107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":352970107},{"type":"v1","addr":"172.21.15.189:6839","nonce":352970107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.32:6839/4160386524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4160386524},{"type":"v1","addr":"172.21.15.32:6839","nonce":4160386524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.71:6839/3517409993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3517409993},{"type":"v1","addr":"172.21.15.71:6839","nonce":3517409993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24590,24541,24560,24533,24517]},"id":1}]} 2024-04-30T08:32:32.520 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 27 2024-04-30T08:32:32.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.521+0000 7ffa2aa00640 1 -- 172.21.15.32:0/1370429111 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ffa0c0afbf0 msgr2=0x7ffa0c0b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:32.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.521+0000 7ffa2aa00640 1 --2- 172.21.15.32:0/1370429111 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ffa0c0afbf0 0x7ffa0c0b2100 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7ffa241a16e0 tx=0x7ffa10005f90 comp rx=0 tx=0).stop 2024-04-30T08:32:32.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.521+0000 7ffa2aa00640 1 -- 172.21.15.32:0/1370429111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa24077990 msgr2=0x7ffa241a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:32.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.521+0000 7ffa2aa00640 1 --2- 172.21.15.32:0/1370429111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa24077990 0x7ffa241a0640 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ffa1800e9a0 tx=0x7ffa1800ee60 comp rx=0 tx=0).stop 2024-04-30T08:32:32.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.521+0000 7ffa2aa00640 1 -- 172.21.15.32:0/1370429111 shutdown_connections 2024-04-30T08:32:32.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.521+0000 7ffa2aa00640 1 --2- 172.21.15.32:0/1370429111 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ffa0c0afbf0 0x7ffa0c0b2100 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:32.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.521+0000 7ffa2aa00640 1 --2- 172.21.15.32:0/1370429111 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa24079920 0x7ffa241a7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:32.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.521+0000 7ffa2aa00640 1 --2- 172.21.15.32:0/1370429111 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa24078c20 0x7ffa241a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:32.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.521+0000 7ffa2aa00640 1 --2- 172.21.15.32:0/1370429111 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa24077990 0x7ffa241a0640 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:32.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.521+0000 7ffa2aa00640 1 -- 172.21.15.32:0/1370429111 >> 172.21.15.32:0/1370429111 conn(0x7ffa24100da0 msgr2=0x7ffa241013f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:32.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.521+0000 7ffa2aa00640 1 -- 172.21.15.32:0/1370429111 shutdown_connections 2024-04-30T08:32:32.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.521+0000 7ffa2aa00640 1 -- 172.21.15.32:0/1370429111 wait complete. 2024-04-30T08:32:32.549 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-30T08:32:32.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.830+0000 7fbee7000640 1 -- 172.21.15.32:0/1145613360 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbee010f790 msgr2=0x7fbee010fba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:32.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.830+0000 7fbee7000640 1 --2- 172.21.15.32:0/1145613360 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbee010f790 0x7fbee010fba0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fbec8009900 tx=0x7fbec802a3f0 comp rx=0 tx=0).stop 2024-04-30T08:32:32.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.830+0000 7fbee7000640 1 -- 172.21.15.32:0/1145613360 shutdown_connections 2024-04-30T08:32:32.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.830+0000 7fbee7000640 1 --2- 172.21.15.32:0/1145613360 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbee0114490 0x7fbee01168b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:32.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.830+0000 7fbee7000640 1 --2- 172.21.15.32:0/1145613360 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbee010f790 0x7fbee010fba0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:32.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.830+0000 7fbee7000640 1 --2- 172.21.15.32:0/1145613360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbee010e550 0x7fbee010f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:32.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.830+0000 7fbee7000640 1 -- 172.21.15.32:0/1145613360 >> 172.21.15.32:0/1145613360 conn(0x7fbee0074a00 msgr2=0x7fbee010bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:32.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.831+0000 7fbee7000640 1 -- 172.21.15.32:0/1145613360 shutdown_connections 2024-04-30T08:32:32.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.831+0000 7fbee7000640 1 -- 172.21.15.32:0/1145613360 wait complete. 2024-04-30T08:32:32.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.832+0000 7fbee7000640 1 Processor -- start 2024-04-30T08:32:32.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.832+0000 7fbee7000640 1 -- start start 2024-04-30T08:32:32.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.832+0000 7fbee7000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbee010e550 0x7fbee01a4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:32.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.832+0000 7fbee7000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbee010f790 0x7fbee01a5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:32.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.832+0000 7fbee7000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbee0114490 0x7fbee01ac110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:32.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.832+0000 7fbee7000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbee011a0b0 con 0x7fbee010f790 2024-04-30T08:32:32.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.832+0000 7fbee7000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fbee0119f30 con 0x7fbee010e550 2024-04-30T08:32:32.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.832+0000 7fbee7000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbee011a230 con 0x7fbee0114490 2024-04-30T08:32:32.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.832+0000 7fbedf400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbee010f790 0x7fbee01a5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:32.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.832+0000 7fbee5000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbee0114490 0x7fbee01ac110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:32.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.832+0000 7fbedf400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbee010f790 0x7fbee01a5210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45232/0 (socket says 172.21.15.32:45232) 2024-04-30T08:32:32.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.833+0000 7fbedf400640 1 -- 172.21.15.32:0/1036212298 learned_addr learned my addr 172.21.15.32:0/1036212298 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:32.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.833+0000 7fbedfe00640 1 --2- 172.21.15.32:0/1036212298 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbee010e550 0x7fbee01a4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:32.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.833+0000 7fbee5000640 1 -- 172.21.15.32:0/1036212298 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbee010e550 msgr2=0x7fbee01a4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:32.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.833+0000 7fbee5000640 1 --2- 172.21.15.32:0/1036212298 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbee010e550 0x7fbee01a4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:32.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.833+0000 7fbee5000640 1 -- 172.21.15.32:0/1036212298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbee010f790 msgr2=0x7fbee01a5210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:32.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.833+0000 7fbee5000640 1 --2- 172.21.15.32:0/1036212298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbee010f790 0x7fbee01a5210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:32.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.833+0000 7fbee5000640 1 -- 172.21.15.32:0/1036212298 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbee01ac7b0 con 0x7fbee0114490 2024-04-30T08:32:32.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.833+0000 7fbedf400640 1 --2- 172.21.15.32:0/1036212298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbee010f790 0x7fbee01a5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:32:32.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.833+0000 7fbee5000640 1 --2- 172.21.15.32:0/1036212298 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbee0114490 0x7fbee01ac110 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbed800d680 tx=0x7fbed800db40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:32.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.834+0000 7fbedcc00640 1 -- 172.21.15.32:0/1036212298 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed8017070 con 0x7fbee0114490 2024-04-30T08:32:32.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.834+0000 7fbee7000640 1 -- 172.21.15.32:0/1036212298 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbee01acaa0 con 0x7fbee0114490 2024-04-30T08:32:32.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.834+0000 7fbedcc00640 1 -- 172.21.15.32:0/1036212298 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbed8009820 con 0x7fbee0114490 2024-04-30T08:32:32.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.834+0000 7fbedcc00640 1 -- 172.21.15.32:0/1036212298 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbed8004ce0 con 0x7fbee0114490 2024-04-30T08:32:32.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.834+0000 7fbee7000640 1 -- 172.21.15.32:0/1036212298 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbee01acff0 con 0x7fbee0114490 2024-04-30T08:32:32.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.835+0000 7fbee7000640 1 -- 172.21.15.32:0/1036212298 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbee01132a0 con 0x7fbee0114490 2024-04-30T08:32:32.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.837+0000 7fbedcc00640 1 -- 172.21.15.32:0/1036212298 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fbed80099d0 con 0x7fbee0114490 2024-04-30T08:32:32.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.839+0000 7fbedcc00640 1 --2- 172.21.15.32:0/1036212298 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbec40afb80 0x7fbec40b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:32.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.839+0000 7fbedcc00640 1 -- 172.21.15.32:0/1036212298 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fbed80de9d0 con 0x7fbee0114490 2024-04-30T08:32:32.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.839+0000 7fbedfe00640 1 --2- 172.21.15.32:0/1036212298 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbec40afb80 0x7fbec40b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:32.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.840+0000 7fbedfe00640 1 --2- 172.21.15.32:0/1036212298 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbec40afb80 0x7fbec40b2090 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbee01a6070 tx=0x7fbed0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:32.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:32.844+0000 7fbedcc00640 1 -- 172.21.15.32:0/1036212298 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbed80a6a40 con 0x7fbee0114490 2024-04-30T08:32:33.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.055+0000 7fbee7000640 1 -- 172.21.15.32:0/1036212298 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbee0063820 con 0x7fbee0114490 2024-04-30T08:32:33.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.057+0000 7fbedcc00640 1 -- 172.21.15.32:0/1036212298 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+24525 (secure 0 0 0) 0x7fbed80aad40 con 0x7fbee0114490 2024-04-30T08:32:33.060 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:32:33.060 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":83,"fsid":"98c0a670-06ca-11ef-bc93-c7b262605968","created":"2024-04-30T08:22:11.122875+0000","modified":"2024-04-30T08:32:31.544477+0000","last_up_change":"2024-04-30T08:30:50.350439+0000","last_in_change":"2024-04-30T08:30:30.210620+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-30T08:26:19.728331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-30T08:31:49.800542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-30T08:31:51.211320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-30T08:31:55.716625+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a7e060c8-1cbb-418f-9a03-06693f6b3f52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6803","nonce":2624700402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6805","nonce":2624700402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6809","nonce":2624700402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6807","nonce":2624700402}]},"public_addr":"172.21.15.32:6803/2624700402","cluster_addr":"172.21.15.32:6805/2624700402","heartbeat_back_addr":"172.21.15.32:6809/2624700402","heartbeat_front_addr":"172.21.15.32:6807/2624700402","state":["exists","up"]},{"osd":1,"uuid":"72ef2fb1-e5ca-48ef-aa3a-4796a64b922f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6801","nonce":444649711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6803","nonce":444649711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6807","nonce":444649711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6805","nonce":444649711}]},"public_addr":"172.21.15.71:6801/444649711","cluster_addr":"172.21.15.71:6803/444649711","heartbeat_back_addr":"172.21.15.71:6807/444649711","heartbeat_front_addr":"172.21.15.71:6805/444649711","state":["exists","up"]},{"osd":2,"uuid":"704e5e8a-935e-47c7-ae28-27119876ac09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6801","nonce":3255528671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6803","nonce":3255528671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6807","nonce":3255528671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6805","nonce":3255528671}]},"public_addr":"172.21.15.189:6801/3255528671","cluster_addr":"172.21.15.189:6803/3255528671","heartbeat_back_addr":"172.21.15.189:6807/3255528671","heartbeat_front_addr":"172.21.15.189:6805/3255528671","state":["exists","up"]},{"osd":3,"uuid":"575004f7-6559-4a62-863e-e33078b0eebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6811","nonce":1384702112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6813","nonce":1384702112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6817","nonce":1384702112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6815","nonce":1384702112}]},"public_addr":"172.21.15.32:6811/1384702112","cluster_addr":"172.21.15.32:6813/1384702112","heartbeat_back_addr":"172.21.15.32:6817/1384702112","heartbeat_front_addr":"172.21.15.32:6815/1384702112","state":["exists","up"]},{"osd":4,"uuid":"e1c1e894-074d-4c78-89ef-9e5a114268bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6809","nonce":4250534574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6811","nonce":4250534574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6815","nonce":4250534574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6813","nonce":4250534574}]},"public_addr":"172.21.15.71:6809/4250534574","cluster_addr":"172.21.15.71:6811/4250534574","heartbeat_back_addr":"172.21.15.71:6815/4250534574","heartbeat_front_addr":"172.21.15.71:6813/4250534574","state":["exists","up"]},{"osd":5,"uuid":"12c022d4-fac6-40c4-ae7a-691571bee5d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6809","nonce":4245346198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6811","nonce":4245346198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6815","nonce":4245346198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6813","nonce":4245346198}]},"public_addr":"172.21.15.189:6809/4245346198","cluster_addr":"172.21.15.189:6811/4245346198","heartbeat_back_addr":"172.21.15.189:6815/4245346198","heartbeat_front_addr":"172.21.15.189:6813/4245346198","state":["exists","up"]},{"osd":6,"uuid":"b413f8ee-3d4a-415a-a2ba-d68437b7330d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6819","nonce":1910389190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6821","nonce":1910389190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6825","nonce":1910389190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6823","nonce":1910389190}]},"public_addr":"172.21.15.32:6819/1910389190","cluster_addr":"172.21.15.32:6821/1910389190","heartbeat_back_addr":"172.21.15.32:6825/1910389190","heartbeat_front_addr":"172.21.15.32:6823/1910389190","state":["exists","up"]},{"osd":7,"uuid":"aac64fb3-fc13-4126-b842-7b06ad5ce238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6817","nonce":692607380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6819","nonce":692607380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6823","nonce":692607380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6821","nonce":692607380}]},"public_addr":"172.21.15.71:6817/692607380","cluster_addr":"172.21.15.71:6819/692607380","heartbeat_back_addr":"172.21.15.71:6823/692607380","heartbeat_front_addr":"172.21.15.71:6821/692607380","state":["exists","up"]},{"osd":8,"uuid":"cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6817","nonce":3973697069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6819","nonce":3973697069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6823","nonce":3973697069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6821","nonce":3973697069}]},"public_addr":"172.21.15.189:6817/3973697069","cluster_addr":"172.21.15.189:6819/3973697069","heartbeat_back_addr":"172.21.15.189:6823/3973697069","heartbeat_front_addr":"172.21.15.189:6821/3973697069","state":["exists","up"]},{"osd":9,"uuid":"ef4a1645-ae6e-45c3-82ad-161510e62aa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6827","nonce":46066685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6829","nonce":46066685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6833","nonce":46066685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6831","nonce":46066685}]},"public_addr":"172.21.15.32:6827/46066685","cluster_addr":"172.21.15.32:6829/46066685","heartbeat_back_addr":"172.21.15.32:6833/46066685","heartbeat_front_addr":"172.21.15.32:6831/46066685","state":["exists","up"]},{"osd":10,"uuid":"25b26a38-a644-409f-b941-59efdf04a3ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6825","nonce":1886290293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6827","nonce":1886290293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6831","nonce":1886290293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6829","nonce":1886290293}]},"public_addr":"172.21.15.71:6825/1886290293","cluster_addr":"172.21.15.71:6827/1886290293","heartbeat_back_addr":"172.21.15.71:6831/1886290293","heartbeat_front_addr":"172.21.15.71:6829/1886290293","state":["exists","up"]},{"osd":11,"uuid":"ab96b3ff-3688-455b-af64-60226c5bfbff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6825","nonce":850185001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6827","nonce":850185001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6831","nonce":850185001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6829","nonce":850185001}]},"public_addr":"172.21.15.189:6825/850185001","cluster_addr":"172.21.15.189:6827/850185001","heartbeat_back_addr":"172.21.15.189:6831/850185001","heartbeat_front_addr":"172.21.15.189:6829/850185001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:27.220308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:53.230783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:17.190642+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:52.603799+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:20.568671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:46.158363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:22.706933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:50.665173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:16.602410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:53.814698+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:22.435189+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:48.651949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1266897222":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/2915689533":"2024-05-01T08:23:29.526664+0000","172.21.15.32:6801/4019351306":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/425836193":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/1134075722":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6801/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/21059253":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/3245550894":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/4019351306":"2024-05-01T08:23:29.526664+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-30T08:32:33.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.060+0000 7fbee7000640 1 -- 172.21.15.32:0/1036212298 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbec40afb80 msgr2=0x7fbec40b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:33.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.060+0000 7fbee7000640 1 --2- 172.21.15.32:0/1036212298 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbec40afb80 0x7fbec40b2090 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbee01a6070 tx=0x7fbed0009500 comp rx=0 tx=0).stop 2024-04-30T08:32:33.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.061+0000 7fbee7000640 1 -- 172.21.15.32:0/1036212298 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbee0114490 msgr2=0x7fbee01ac110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:33.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.061+0000 7fbee7000640 1 --2- 172.21.15.32:0/1036212298 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbee0114490 0x7fbee01ac110 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbed800d680 tx=0x7fbed800db40 comp rx=0 tx=0).stop 2024-04-30T08:32:33.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.061+0000 7fbee7000640 1 -- 172.21.15.32:0/1036212298 shutdown_connections 2024-04-30T08:32:33.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.061+0000 7fbee7000640 1 --2- 172.21.15.32:0/1036212298 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbec40afb80 0x7fbec40b2090 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:33.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.061+0000 7fbee7000640 1 --2- 172.21.15.32:0/1036212298 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbee0114490 0x7fbee01ac110 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:33.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.061+0000 7fbee7000640 1 --2- 172.21.15.32:0/1036212298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbee010f790 0x7fbee01a5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:33.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.061+0000 7fbee7000640 1 --2- 172.21.15.32:0/1036212298 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbee010e550 0x7fbee01a4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:33.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.061+0000 7fbee7000640 1 -- 172.21.15.32:0/1036212298 >> 172.21.15.32:0/1036212298 conn(0x7fbee0074a00 msgr2=0x7fbee0110470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:33.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.061+0000 7fbee7000640 1 -- 172.21.15.32:0/1036212298 shutdown_connections 2024-04-30T08:32:33.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.061+0000 7fbee7000640 1 -- 172.21.15.32:0/1036212298 wait complete. 2024-04-30T08:32:33.088 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:32 smithi032 ceph-mon[28331]: pgmap v344: 129 pgs: 3 peering, 126 active+clean; 585 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 383 B/s rd, 4.6 KiB/s wr, 16 op/s 2024-04-30T08:32:33.088 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:32 smithi032 ceph-mon[28331]: from='client.24860 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:32:33.088 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:32 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1370429111' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:32:33.091 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-30T08:32:33.091 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-30T08:32:33.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:32 smithi071 ceph-mon[33385]: pgmap v344: 129 pgs: 3 peering, 126 active+clean; 585 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 383 B/s rd, 4.6 KiB/s wr, 16 op/s 2024-04-30T08:32:33.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:32 smithi071 ceph-mon[33385]: from='client.24860 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:32:33.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:32 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1370429111' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:32:33.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:32 smithi189 ceph-mon[33038]: pgmap v344: 129 pgs: 3 peering, 126 active+clean; 585 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 383 B/s rd, 4.6 KiB/s wr, 16 op/s 2024-04-30T08:32:33.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:32 smithi189 ceph-mon[33038]: from='client.24860 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:32:33.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:32 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1370429111' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:32:33.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.380+0000 7f6b23400640 1 -- 172.21.15.32:0/1458592537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b1c10bb50 msgr2=0x7f6b1c10df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:33.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.380+0000 7f6b23400640 1 --2- 172.21.15.32:0/1458592537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b1c10bb50 0x7f6b1c10df70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6b14009900 tx=0x7f6b1402a410 comp rx=0 tx=0).stop 2024-04-30T08:32:33.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.380+0000 7f6b23400640 1 -- 172.21.15.32:0/1458592537 shutdown_connections 2024-04-30T08:32:33.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.380+0000 7f6b23400640 1 --2- 172.21.15.32:0/1458592537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b1c10bb50 0x7f6b1c10df70 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:33.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.380+0000 7f6b23400640 1 --2- 172.21.15.32:0/1458592537 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b1c076a10 0x7f6b1c10b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:33.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.380+0000 7f6b23400640 1 --2- 172.21.15.32:0/1458592537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b1c0760c0 0x7f6b1c0764d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:33.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.380+0000 7f6b23400640 1 -- 172.21.15.32:0/1458592537 >> 172.21.15.32:0/1458592537 conn(0x7f6b1c0fed90 msgr2=0x7f6b1c101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:33.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.380+0000 7f6b23400640 1 -- 172.21.15.32:0/1458592537 shutdown_connections 2024-04-30T08:32:33.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.380+0000 7f6b23400640 1 -- 172.21.15.32:0/1458592537 wait complete. 2024-04-30T08:32:33.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.381+0000 7f6b23400640 1 Processor -- start 2024-04-30T08:32:33.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.381+0000 7f6b23400640 1 -- start start 2024-04-30T08:32:33.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.382+0000 7f6b23400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b1c0760c0 0x7f6b1c19c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:33.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.382+0000 7f6b23400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b1c076a10 0x7f6b1c19c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:33.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.382+0000 7f6b23400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b1c10bb50 0x7f6b1c1a38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:33.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.382+0000 7f6b23400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6b1c1118f0 con 0x7f6b1c076a10 2024-04-30T08:32:33.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.382+0000 7f6b23400640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6b1c111770 con 0x7f6b1c0760c0 2024-04-30T08:32:33.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.382+0000 7f6b23400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6b1c111a70 con 0x7f6b1c10bb50 2024-04-30T08:32:33.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.382+0000 7f6b1be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b1c076a10 0x7f6b1c19c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:33.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.382+0000 7f6b20a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b1c0760c0 0x7f6b1c19c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:33.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.382+0000 7f6b20a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b1c0760c0 0x7f6b1c19c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:59680/0 (socket says 172.21.15.32:59680) 2024-04-30T08:32:33.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.382+0000 7f6b20a00640 1 -- 172.21.15.32:0/2299327983 learned_addr learned my addr 172.21.15.32:0/2299327983 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:33.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.382+0000 7f6b21400640 1 --2- 172.21.15.32:0/2299327983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b1c10bb50 0x7f6b1c1a38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:33.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.383+0000 7f6b20a00640 1 -- 172.21.15.32:0/2299327983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b1c10bb50 msgr2=0x7f6b1c1a38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:33.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.383+0000 7f6b20a00640 1 --2- 172.21.15.32:0/2299327983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b1c10bb50 0x7f6b1c1a38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:33.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.383+0000 7f6b20a00640 1 -- 172.21.15.32:0/2299327983 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b1c076a10 msgr2=0x7f6b1c19c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:33.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.383+0000 7f6b20a00640 1 --2- 172.21.15.32:0/2299327983 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b1c076a10 0x7f6b1c19c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:33.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.383+0000 7f6b20a00640 1 -- 172.21.15.32:0/2299327983 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b1c1a3f50 con 0x7f6b1c0760c0 2024-04-30T08:32:33.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.383+0000 7f6b1be00640 1 --2- 172.21.15.32:0/2299327983 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b1c076a10 0x7f6b1c19c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:32:33.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.383+0000 7f6b20a00640 1 --2- 172.21.15.32:0/2299327983 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b1c0760c0 0x7f6b1c19c410 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c00dbf0 tx=0x7f6b0c00b630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:33.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.383+0000 7f6b19600640 1 -- 172.21.15.32:0/2299327983 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b0c017070 con 0x7f6b1c0760c0 2024-04-30T08:32:33.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.383+0000 7f6b19600640 1 -- 172.21.15.32:0/2299327983 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6b0c003e20 con 0x7f6b1c0760c0 2024-04-30T08:32:33.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.383+0000 7f6b23400640 1 -- 172.21.15.32:0/2299327983 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b1c1a4240 con 0x7f6b1c0760c0 2024-04-30T08:32:33.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.383+0000 7f6b19600640 1 -- 172.21.15.32:0/2299327983 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b0c010460 con 0x7f6b1c0760c0 2024-04-30T08:32:33.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.383+0000 7f6b23400640 1 -- 172.21.15.32:0/2299327983 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b1c1a4790 con 0x7f6b1c0760c0 2024-04-30T08:32:33.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.384+0000 7f6b23400640 1 -- 172.21.15.32:0/2299327983 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b1c10ad10 con 0x7f6b1c0760c0 2024-04-30T08:32:33.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.387+0000 7f6b19600640 1 -- 172.21.15.32:0/2299327983 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f6b0c0040b0 con 0x7f6b1c0760c0 2024-04-30T08:32:33.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.389+0000 7f6b19600640 1 --2- 172.21.15.32:0/2299327983 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f6b000afb80 0x7f6b000b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:33.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.389+0000 7f6b19600640 1 -- 172.21.15.32:0/2299327983 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f6b0c0df340 con 0x7f6b1c0760c0 2024-04-30T08:32:33.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.389+0000 7f6b1be00640 1 --2- 172.21.15.32:0/2299327983 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f6b000afb80 0x7f6b000b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:33.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.390+0000 7f6b1be00640 1 --2- 172.21.15.32:0/2299327983 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f6b000afb80 0x7f6b000b2090 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c19d4b0 tx=0x7f6b04005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:33.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.390+0000 7f6b19600640 1 -- 172.21.15.32:0/2299327983 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6b0c0a73b0 con 0x7f6b1c0760c0 2024-04-30T08:32:33.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.602+0000 7f6b23400640 1 -- 172.21.15.32:0/2299327983 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6b1c063060 con 0x7f6b000afb80 2024-04-30T08:32:33.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.624+0000 7f6b19600640 1 -- 172.21.15.32:0/2299327983 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340554 (secure 0 0 0) 0x7f6b1c063060 con 0x7f6b000afb80 2024-04-30T08:32:33.632 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:32:33.634 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-30T08:32:33.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.633+0000 7f6b23400640 1 -- 172.21.15.32:0/2299327983 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f6b000afb80 msgr2=0x7f6b000b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:33.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.633+0000 7f6b23400640 1 --2- 172.21.15.32:0/2299327983 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f6b000afb80 0x7f6b000b2090 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f6b1c19d4b0 tx=0x7f6b04005f90 comp rx=0 tx=0).stop 2024-04-30T08:32:33.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.633+0000 7f6b23400640 1 -- 172.21.15.32:0/2299327983 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b1c0760c0 msgr2=0x7f6b1c19c410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:33.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.633+0000 7f6b23400640 1 --2- 172.21.15.32:0/2299327983 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b1c0760c0 0x7f6b1c19c410 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6b0c00dbf0 tx=0x7f6b0c00b630 comp rx=0 tx=0).stop 2024-04-30T08:32:33.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.633+0000 7f6b23400640 1 -- 172.21.15.32:0/2299327983 shutdown_connections 2024-04-30T08:32:33.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.633+0000 7f6b23400640 1 --2- 172.21.15.32:0/2299327983 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f6b000afb80 0x7f6b000b2090 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:33.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.633+0000 7f6b23400640 1 --2- 172.21.15.32:0/2299327983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b1c10bb50 0x7f6b1c1a38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:33.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.633+0000 7f6b23400640 1 --2- 172.21.15.32:0/2299327983 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b1c076a10 0x7f6b1c19c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:33.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.633+0000 7f6b23400640 1 --2- 172.21.15.32:0/2299327983 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b1c0760c0 0x7f6b1c19c410 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:33.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.633+0000 7f6b23400640 1 -- 172.21.15.32:0/2299327983 >> 172.21.15.32:0/2299327983 conn(0x7f6b1c0fed90 msgr2=0x7f6b1c0ffbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:33.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.633+0000 7f6b23400640 1 -- 172.21.15.32:0/2299327983 shutdown_connections 2024-04-30T08:32:33.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.633+0000 7f6b23400640 1 -- 172.21.15.32:0/2299327983 wait complete. 2024-04-30T08:32:33.664 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":344,"stamp":"2024-04-30T08:32:31.713519+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598666,"num_objects":84,"num_object_clones":0,"num_object_copies":252,"num_objects_missing_on_primary":0,"num_objects_missing":8,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":84,"num_whiteouts":0,"num_read":79,"num_read_kb":67,"num_write":303,"num_write_kb":2206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":56,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":247,"ondisk_log_size":247,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":368236,"kb_used_data":10752,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124573588,"statfs":{"total":1151940427776,"available":1151563354112,"internally_reserved":0,"allocated":11010048,"data_stored":7086284,"data_compressed":294087,"data_compressed_allocated":393216,"data_compressed_original":2285568,"omap_allocated":19141,"internal_metadata":365933883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3658,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":8,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":131,"num_write_kb":37,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":42,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004592"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548872+0000","last_change":"2024-04-30T08:31:52.711443+0000","last_active":"2024-04-30T08:32:30.548872+0000","last_peered":"2024-04-30T08:32:30.548872+0000","last_clean":"2024-04-30T08:32:30.548872+0000","last_became_active":"2024-04-30T08:31:52.711029+0000","last_became_peered":"2024-04-30T08:31:52.711029+0000","last_unstale":"2024-04-30T08:32:30.548872+0000","last_undegraded":"2024-04-30T08:32:30.548872+0000","last_fullsized":"2024-04-30T08:32:30.548872+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:51:15.208528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834854+0000","last_change":"2024-04-30T08:31:52.712324+0000","last_active":"2024-04-30T08:31:59.834854+0000","last_peered":"2024-04-30T08:31:59.834854+0000","last_clean":"2024-04-30T08:31:59.834854+0000","last_became_active":"2024-04-30T08:31:52.711923+0000","last_became_peered":"2024-04-30T08:31:52.711923+0000","last_unstale":"2024-04-30T08:31:59.834854+0000","last_undegraded":"2024-04-30T08:31:59.834854+0000","last_fullsized":"2024-04-30T08:31:59.834854+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:29:55.071459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834894+0000","last_change":"2024-04-30T08:31:52.712936+0000","last_active":"2024-04-30T08:31:59.834894+0000","last_peered":"2024-04-30T08:31:59.834894+0000","last_clean":"2024-04-30T08:31:59.834894+0000","last_became_active":"2024-04-30T08:31:52.712585+0000","last_became_peered":"2024-04-30T08:31:52.712585+0000","last_unstale":"2024-04-30T08:31:59.834894+0000","last_undegraded":"2024-04-30T08:31:59.834894+0000","last_fullsized":"2024-04-30T08:31:59.834894+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:13:43.745561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.835699+0000","last_change":"2024-04-30T08:31:52.686880+0000","last_active":"2024-04-30T08:31:59.835699+0000","last_peered":"2024-04-30T08:31:59.835699+0000","last_clean":"2024-04-30T08:31:59.835699+0000","last_became_active":"2024-04-30T08:31:52.686506+0000","last_became_peered":"2024-04-30T08:31:52.686506+0000","last_unstale":"2024-04-30T08:31:59.835699+0000","last_undegraded":"2024-04-30T08:31:59.835699+0000","last_fullsized":"2024-04-30T08:31:59.835699+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:16:33.390951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.357817+0000","last_change":"2024-04-30T08:31:52.706014+0000","last_active":"2024-04-30T08:31:59.357817+0000","last_peered":"2024-04-30T08:31:59.357817+0000","last_clean":"2024-04-30T08:31:59.357817+0000","last_became_active":"2024-04-30T08:31:52.705582+0000","last_became_peered":"2024-04-30T08:31:52.705582+0000","last_unstale":"2024-04-30T08:31:59.357817+0000","last_undegraded":"2024-04-30T08:31:59.357817+0000","last_fullsized":"2024-04-30T08:31:59.357817+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:24:49.804592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:58.709591+0000","last_change":"2024-04-30T08:31:52.690462+0000","last_active":"2024-04-30T08:31:58.709591+0000","last_peered":"2024-04-30T08:31:58.709591+0000","last_clean":"2024-04-30T08:31:58.709591+0000","last_became_active":"2024-04-30T08:31:52.690092+0000","last_became_peered":"2024-04-30T08:31:52.690092+0000","last_unstale":"2024-04-30T08:31:58.709591+0000","last_undegraded":"2024-04-30T08:31:58.709591+0000","last_fullsized":"2024-04-30T08:31:58.709591+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:32:49.769383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.835448+0000","last_change":"2024-04-30T08:31:52.701786+0000","last_active":"2024-04-30T08:31:59.835448+0000","last_peered":"2024-04-30T08:31:59.835448+0000","last_clean":"2024-04-30T08:31:59.835448+0000","last_became_active":"2024-04-30T08:31:52.701402+0000","last_became_peered":"2024-04-30T08:31:52.701402+0000","last_unstale":"2024-04-30T08:31:59.835448+0000","last_undegraded":"2024-04-30T08:31:59.835448+0000","last_fullsized":"2024-04-30T08:31:59.835448+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:58:06.153479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.669311+0000","last_change":"2024-04-30T08:31:52.698037+0000","last_active":"2024-04-30T08:31:59.669311+0000","last_peered":"2024-04-30T08:31:59.669311+0000","last_clean":"2024-04-30T08:31:59.669311+0000","last_became_active":"2024-04-30T08:31:52.697646+0000","last_became_peered":"2024-04-30T08:31:52.697646+0000","last_unstale":"2024-04-30T08:31:59.669311+0000","last_undegraded":"2024-04-30T08:31:59.669311+0000","last_fullsized":"2024-04-30T08:31:59.669311+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:28:59.520091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548406+0000","last_change":"2024-04-30T08:31:52.680377+0000","last_active":"2024-04-30T08:32:30.548406+0000","last_peered":"2024-04-30T08:32:30.548406+0000","last_clean":"2024-04-30T08:32:30.548406+0000","last_became_active":"2024-04-30T08:31:52.679977+0000","last_became_peered":"2024-04-30T08:31:52.679977+0000","last_unstale":"2024-04-30T08:32:30.548406+0000","last_undegraded":"2024-04-30T08:32:30.548406+0000","last_fullsized":"2024-04-30T08:32:30.548406+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:35:22.488450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:58.710009+0000","last_change":"2024-04-30T08:31:52.686488+0000","last_active":"2024-04-30T08:31:58.710009+0000","last_peered":"2024-04-30T08:31:58.710009+0000","last_clean":"2024-04-30T08:31:58.710009+0000","last_became_active":"2024-04-30T08:31:52.686057+0000","last_became_peered":"2024-04-30T08:31:52.686057+0000","last_unstale":"2024-04-30T08:31:58.710009+0000","last_undegraded":"2024-04-30T08:31:58.710009+0000","last_fullsized":"2024-04-30T08:31:58.710009+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:51:47.170678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.836003+0000","last_change":"2024-04-30T08:31:52.690344+0000","last_active":"2024-04-30T08:31:59.836003+0000","last_peered":"2024-04-30T08:31:59.836003+0000","last_clean":"2024-04-30T08:31:59.836003+0000","last_became_active":"2024-04-30T08:31:52.689857+0000","last_became_peered":"2024-04-30T08:31:52.689857+0000","last_unstale":"2024-04-30T08:31:59.836003+0000","last_undegraded":"2024-04-30T08:31:59.836003+0000","last_fullsized":"2024-04-30T08:31:59.836003+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:45:13.395589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.835553+0000","last_change":"2024-04-30T08:31:52.685151+0000","last_active":"2024-04-30T08:31:59.835553+0000","last_peered":"2024-04-30T08:31:59.835553+0000","last_clean":"2024-04-30T08:31:59.835553+0000","last_became_active":"2024-04-30T08:31:52.684722+0000","last_became_peered":"2024-04-30T08:31:52.684722+0000","last_unstale":"2024-04-30T08:31:59.835553+0000","last_undegraded":"2024-04-30T08:31:59.835553+0000","last_fullsized":"2024-04-30T08:31:59.835553+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:14:00.660478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.553484+0000","last_change":"2024-04-30T08:31:52.714294+0000","last_active":"2024-04-30T08:32:30.553484+0000","last_peered":"2024-04-30T08:32:30.553484+0000","last_clean":"2024-04-30T08:32:30.553484+0000","last_became_active":"2024-04-30T08:31:52.710094+0000","last_became_peered":"2024-04-30T08:31:52.710094+0000","last_unstale":"2024-04-30T08:32:30.553484+0000","last_undegraded":"2024-04-30T08:32:30.553484+0000","last_fullsized":"2024-04-30T08:32:30.553484+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:34:44.203093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834299+0000","last_change":"2024-04-30T08:31:52.707992+0000","last_active":"2024-04-30T08:31:59.834299+0000","last_peered":"2024-04-30T08:31:59.834299+0000","last_clean":"2024-04-30T08:31:59.834299+0000","last_became_active":"2024-04-30T08:31:52.706456+0000","last_became_peered":"2024-04-30T08:31:52.706456+0000","last_unstale":"2024-04-30T08:31:59.834299+0000","last_undegraded":"2024-04-30T08:31:59.834299+0000","last_fullsized":"2024-04-30T08:31:59.834299+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:45:32.717842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834299+0000","last_change":"2024-04-30T08:31:52.695605+0000","last_active":"2024-04-30T08:31:59.834299+0000","last_peered":"2024-04-30T08:31:59.834299+0000","last_clean":"2024-04-30T08:31:59.834299+0000","last_became_active":"2024-04-30T08:31:52.692439+0000","last_became_peered":"2024-04-30T08:31:52.692439+0000","last_unstale":"2024-04-30T08:31:59.834299+0000","last_undegraded":"2024-04-30T08:31:59.834299+0000","last_fullsized":"2024-04-30T08:31:59.834299+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:39:38.746567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.547727+0000","last_change":"2024-04-30T08:31:52.707011+0000","last_active":"2024-04-30T08:32:30.547727+0000","last_peered":"2024-04-30T08:32:30.547727+0000","last_clean":"2024-04-30T08:32:30.547727+0000","last_became_active":"2024-04-30T08:31:52.706602+0000","last_became_peered":"2024-04-30T08:31:52.706602+0000","last_unstale":"2024-04-30T08:32:30.547727+0000","last_undegraded":"2024-04-30T08:32:30.547727+0000","last_fullsized":"2024-04-30T08:32:30.547727+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:43:13.922905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548723+0000","last_change":"2024-04-30T08:31:52.711439+0000","last_active":"2024-04-30T08:32:30.548723+0000","last_peered":"2024-04-30T08:32:30.548723+0000","last_clean":"2024-04-30T08:32:30.548723+0000","last_became_active":"2024-04-30T08:31:52.711023+0000","last_became_peered":"2024-04-30T08:31:52.711023+0000","last_unstale":"2024-04-30T08:32:30.548723+0000","last_undegraded":"2024-04-30T08:32:30.548723+0000","last_fullsized":"2024-04-30T08:32:30.548723+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:31:00.654348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.558525+0000","last_change":"2024-04-30T08:31:52.708476+0000","last_active":"2024-04-30T08:32:30.558525+0000","last_peered":"2024-04-30T08:32:30.558525+0000","last_clean":"2024-04-30T08:32:30.558525+0000","last_became_active":"2024-04-30T08:31:52.704735+0000","last_became_peered":"2024-04-30T08:31:52.704735+0000","last_unstale":"2024-04-30T08:32:30.558525+0000","last_undegraded":"2024-04-30T08:32:30.558525+0000","last_fullsized":"2024-04-30T08:32:30.558525+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:40:00.011783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834768+0000","last_change":"2024-04-30T08:31:52.704901+0000","last_active":"2024-04-30T08:31:59.834768+0000","last_peered":"2024-04-30T08:31:59.834768+0000","last_clean":"2024-04-30T08:31:59.834768+0000","last_became_active":"2024-04-30T08:31:52.704636+0000","last_became_peered":"2024-04-30T08:31:52.704636+0000","last_unstale":"2024-04-30T08:31:59.834768+0000","last_undegraded":"2024-04-30T08:31:59.834768+0000","last_fullsized":"2024-04-30T08:31:59.834768+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:08:53.580809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834299+0000","last_change":"2024-04-30T08:31:57.743342+0000","last_active":"2024-04-30T08:31:59.834299+0000","last_peered":"2024-04-30T08:31:59.834299+0000","last_clean":"2024-04-30T08:31:59.834299+0000","last_became_active":"2024-04-30T08:31:57.742850+0000","last_became_peered":"2024-04-30T08:31:57.742850+0000","last_unstale":"2024-04-30T08:31:59.834299+0000","last_undegraded":"2024-04-30T08:31:59.834299+0000","last_fullsized":"2024-04-30T08:31:59.834299+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:59:10.628822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"81'13","reported_seq":40,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.558153+0000","last_change":"2024-04-30T08:31:51.694984+0000","last_active":"2024-04-30T08:32:30.558153+0000","last_peered":"2024-04-30T08:32:30.558153+0000","last_clean":"2024-04-30T08:32:30.558153+0000","last_became_active":"2024-04-30T08:31:51.694514+0000","last_became_peered":"2024-04-30T08:31:51.694514+0000","last_unstale":"2024-04-30T08:32:30.558153+0000","last_undegraded":"2024-04-30T08:32:30.558153+0000","last_fullsized":"2024-04-30T08:32:30.558153+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:17:03.873090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.670928+0000","last_change":"2024-04-30T08:31:52.694276+0000","last_active":"2024-04-30T08:31:59.670928+0000","last_peered":"2024-04-30T08:31:59.670928+0000","last_clean":"2024-04-30T08:31:59.670928+0000","last_became_active":"2024-04-30T08:31:52.693839+0000","last_became_peered":"2024-04-30T08:31:52.693839+0000","last_unstale":"2024-04-30T08:31:59.670928+0000","last_undegraded":"2024-04-30T08:31:59.670928+0000","last_fullsized":"2024-04-30T08:31:59.670928+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:29:21.271040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834679+0000","last_change":"2024-04-30T08:31:57.754828+0000","last_active":"2024-04-30T08:31:59.834679+0000","last_peered":"2024-04-30T08:31:59.834679+0000","last_clean":"2024-04-30T08:31:59.834679+0000","last_became_active":"2024-04-30T08:31:57.754322+0000","last_became_peered":"2024-04-30T08:31:57.754322+0000","last_unstale":"2024-04-30T08:31:59.834679+0000","last_undegraded":"2024-04-30T08:31:59.834679+0000","last_fullsized":"2024-04-30T08:31:59.834679+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:36:02.560134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"81'2","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.671373+0000","last_change":"2024-04-30T08:31:51.707637+0000","last_active":"2024-04-30T08:31:59.671373+0000","last_peered":"2024-04-30T08:31:59.671373+0000","last_clean":"2024-04-30T08:31:59.671373+0000","last_became_active":"2024-04-30T08:31:51.707233+0000","last_became_peered":"2024-04-30T08:31:51.707233+0000","last_unstale":"2024-04-30T08:31:59.671373+0000","last_undegraded":"2024-04-30T08:31:59.671373+0000","last_fullsized":"2024-04-30T08:31:59.671373+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:14:46.041169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.835049+0000","last_change":"2024-04-30T08:31:52.711442+0000","last_active":"2024-04-30T08:31:59.835049+0000","last_peered":"2024-04-30T08:31:59.835049+0000","last_clean":"2024-04-30T08:31:59.835049+0000","last_became_active":"2024-04-30T08:31:52.710942+0000","last_became_peered":"2024-04-30T08:31:52.710942+0000","last_unstale":"2024-04-30T08:31:59.835049+0000","last_undegraded":"2024-04-30T08:31:59.835049+0000","last_fullsized":"2024-04-30T08:31:59.835049+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:26:56.428770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834859+0000","last_change":"2024-04-30T08:31:57.760535+0000","last_active":"2024-04-30T08:31:59.834859+0000","last_peered":"2024-04-30T08:31:59.834859+0000","last_clean":"2024-04-30T08:31:59.834859+0000","last_became_active":"2024-04-30T08:31:57.760073+0000","last_became_peered":"2024-04-30T08:31:57.760073+0000","last_unstale":"2024-04-30T08:31:59.834859+0000","last_undegraded":"2024-04-30T08:31:59.834859+0000","last_fullsized":"2024-04-30T08:31:59.834859+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:03:51.359474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"81'7","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:32:28.528438+0000","last_change":"2024-04-30T08:31:51.678575+0000","last_active":"2024-04-30T08:32:28.528438+0000","last_peered":"2024-04-30T08:32:28.528438+0000","last_clean":"2024-04-30T08:32:28.528438+0000","last_became_active":"2024-04-30T08:31:51.678114+0000","last_became_peered":"2024-04-30T08:31:51.678114+0000","last_unstale":"2024-04-30T08:32:28.528438+0000","last_undegraded":"2024-04-30T08:32:28.528438+0000","last_fullsized":"2024-04-30T08:32:28.528438+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:10:35.517183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548286+0000","last_change":"2024-04-30T08:31:52.704137+0000","last_active":"2024-04-30T08:32:30.548286+0000","last_peered":"2024-04-30T08:32:30.548286+0000","last_clean":"2024-04-30T08:32:30.548286+0000","last_became_active":"2024-04-30T08:31:52.703695+0000","last_became_peered":"2024-04-30T08:31:52.703695+0000","last_unstale":"2024-04-30T08:32:30.548286+0000","last_undegraded":"2024-04-30T08:32:30.548286+0000","last_fullsized":"2024-04-30T08:32:30.548286+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:31:05.461106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834972+0000","last_change":"2024-04-30T08:31:57.755237+0000","last_active":"2024-04-30T08:31:59.834972+0000","last_peered":"2024-04-30T08:31:59.834972+0000","last_clean":"2024-04-30T08:31:59.834972+0000","last_became_active":"2024-04-30T08:31:57.754438+0000","last_became_peered":"2024-04-30T08:31:57.754438+0000","last_unstale":"2024-04-30T08:31:59.834972+0000","last_undegraded":"2024-04-30T08:31:59.834972+0000","last_fullsized":"2024-04-30T08:31:59.834972+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:20:59.035110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"75'2","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.357899+0000","last_change":"2024-04-30T08:31:51.673793+0000","last_active":"2024-04-30T08:31:59.357899+0000","last_peered":"2024-04-30T08:31:59.357899+0000","last_clean":"2024-04-30T08:31:59.357899+0000","last_became_active":"2024-04-30T08:31:51.672703+0000","last_became_peered":"2024-04-30T08:31:51.672703+0000","last_unstale":"2024-04-30T08:31:59.357899+0000","last_undegraded":"2024-04-30T08:31:59.357899+0000","last_fullsized":"2024-04-30T08:31:59.357899+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:13:18.959542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834819+0000","last_change":"2024-04-30T08:31:52.694320+0000","last_active":"2024-04-30T08:31:59.834819+0000","last_peered":"2024-04-30T08:31:59.834819+0000","last_clean":"2024-04-30T08:31:59.834819+0000","last_became_active":"2024-04-30T08:31:52.694032+0000","last_became_peered":"2024-04-30T08:31:52.694032+0000","last_unstale":"2024-04-30T08:31:59.834819+0000","last_undegraded":"2024-04-30T08:31:59.834819+0000","last_fullsized":"2024-04-30T08:31:59.834819+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:33:12.347369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834607+0000","last_change":"2024-04-30T08:31:57.739984+0000","last_active":"2024-04-30T08:31:59.834607+0000","last_peered":"2024-04-30T08:31:59.834607+0000","last_clean":"2024-04-30T08:31:59.834607+0000","last_became_active":"2024-04-30T08:31:57.739477+0000","last_became_peered":"2024-04-30T08:31:57.739477+0000","last_unstale":"2024-04-30T08:31:59.834607+0000","last_undegraded":"2024-04-30T08:31:59.834607+0000","last_fullsized":"2024-04-30T08:31:59.834607+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:10:11.325419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"81'5","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:32:26.512680+0000","last_change":"2024-04-30T08:31:51.678792+0000","last_active":"2024-04-30T08:32:26.512680+0000","last_peered":"2024-04-30T08:32:26.512680+0000","last_clean":"2024-04-30T08:32:26.512680+0000","last_became_active":"2024-04-30T08:31:51.678094+0000","last_became_peered":"2024-04-30T08:31:51.678094+0000","last_unstale":"2024-04-30T08:32:26.512680+0000","last_undegraded":"2024-04-30T08:32:26.512680+0000","last_fullsized":"2024-04-30T08:32:26.512680+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:03:17.526016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.357855+0000","last_change":"2024-04-30T08:31:52.701925+0000","last_active":"2024-04-30T08:31:59.357855+0000","last_peered":"2024-04-30T08:31:59.357855+0000","last_clean":"2024-04-30T08:31:59.357855+0000","last_became_active":"2024-04-30T08:31:52.701425+0000","last_became_peered":"2024-04-30T08:31:52.701425+0000","last_unstale":"2024-04-30T08:31:59.357855+0000","last_undegraded":"2024-04-30T08:31:59.357855+0000","last_fullsized":"2024-04-30T08:31:59.357855+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:20:35.767086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834905+0000","last_change":"2024-04-30T08:31:57.765047+0000","last_active":"2024-04-30T08:31:59.834905+0000","last_peered":"2024-04-30T08:31:59.834905+0000","last_clean":"2024-04-30T08:31:59.834905+0000","last_became_active":"2024-04-30T08:31:57.764519+0000","last_became_peered":"2024-04-30T08:31:57.764519+0000","last_unstale":"2024-04-30T08:31:59.834905+0000","last_undegraded":"2024-04-30T08:31:59.834905+0000","last_fullsized":"2024-04-30T08:31:59.834905+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:06:16.126396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"81'7","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:32:28.533731+0000","last_change":"2024-04-30T08:31:51.686403+0000","last_active":"2024-04-30T08:32:28.533731+0000","last_peered":"2024-04-30T08:32:28.533731+0000","last_clean":"2024-04-30T08:32:28.533731+0000","last_became_active":"2024-04-30T08:31:51.686096+0000","last_became_peered":"2024-04-30T08:31:51.686096+0000","last_unstale":"2024-04-30T08:32:28.533731+0000","last_undegraded":"2024-04-30T08:32:28.533731+0000","last_fullsized":"2024-04-30T08:32:28.533731+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:47:05.961930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.835635+0000","last_change":"2024-04-30T08:31:52.698512+0000","last_active":"2024-04-30T08:31:59.835635+0000","last_peered":"2024-04-30T08:31:59.835635+0000","last_clean":"2024-04-30T08:31:59.835635+0000","last_became_active":"2024-04-30T08:31:52.698168+0000","last_became_peered":"2024-04-30T08:31:52.698168+0000","last_unstale":"2024-04-30T08:31:59.835635+0000","last_undegraded":"2024-04-30T08:31:59.835635+0000","last_fullsized":"2024-04-30T08:31:59.835635+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:22:45.094365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548550+0000","last_change":"2024-04-30T08:31:57.761430+0000","last_active":"2024-04-30T08:32:30.548550+0000","last_peered":"2024-04-30T08:32:30.548550+0000","last_clean":"2024-04-30T08:32:30.548550+0000","last_became_active":"2024-04-30T08:31:57.761062+0000","last_became_peered":"2024-04-30T08:31:57.761062+0000","last_unstale":"2024-04-30T08:32:30.548550+0000","last_undegraded":"2024-04-30T08:32:30.548550+0000","last_fullsized":"2024-04-30T08:32:30.548550+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:32:05.403786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"81'3","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.552212+0000","last_change":"2024-04-30T08:31:51.686694+0000","last_active":"2024-04-30T08:32:30.552212+0000","last_peered":"2024-04-30T08:32:30.552212+0000","last_clean":"2024-04-30T08:32:30.552212+0000","last_became_active":"2024-04-30T08:31:51.685773+0000","last_became_peered":"2024-04-30T08:31:51.685773+0000","last_unstale":"2024-04-30T08:32:30.552212+0000","last_undegraded":"2024-04-30T08:32:30.552212+0000","last_fullsized":"2024-04-30T08:32:30.552212+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:44:25.327977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.669796+0000","last_change":"2024-04-30T08:31:52.709794+0000","last_active":"2024-04-30T08:31:59.669796+0000","last_peered":"2024-04-30T08:31:59.669796+0000","last_clean":"2024-04-30T08:31:59.669796+0000","last_became_active":"2024-04-30T08:31:52.709371+0000","last_became_peered":"2024-04-30T08:31:52.709371+0000","last_unstale":"2024-04-30T08:31:59.669796+0000","last_undegraded":"2024-04-30T08:31:59.669796+0000","last_fullsized":"2024-04-30T08:31:59.669796+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:36:59.892551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834943+0000","last_change":"2024-04-30T08:31:57.757224+0000","last_active":"2024-04-30T08:31:59.834943+0000","last_peered":"2024-04-30T08:31:59.834943+0000","last_clean":"2024-04-30T08:31:59.834943+0000","last_became_active":"2024-04-30T08:31:57.756658+0000","last_became_peered":"2024-04-30T08:31:57.756658+0000","last_unstale":"2024-04-30T08:31:59.834943+0000","last_undegraded":"2024-04-30T08:31:59.834943+0000","last_fullsized":"2024-04-30T08:31:59.834943+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:01:21.309322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"81'1","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834639+0000","last_change":"2024-04-30T08:31:51.678944+0000","last_active":"2024-04-30T08:31:59.834639+0000","last_peered":"2024-04-30T08:31:59.834639+0000","last_clean":"2024-04-30T08:31:59.834639+0000","last_became_active":"2024-04-30T08:31:51.678493+0000","last_became_peered":"2024-04-30T08:31:51.678493+0000","last_unstale":"2024-04-30T08:31:59.834639+0000","last_undegraded":"2024-04-30T08:31:59.834639+0000","last_fullsized":"2024-04-30T08:31:59.834639+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:48:11.111466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.555177+0000","last_change":"2024-04-30T08:31:52.707627+0000","last_active":"2024-04-30T08:32:30.555177+0000","last_peered":"2024-04-30T08:32:30.555177+0000","last_clean":"2024-04-30T08:32:30.555177+0000","last_became_active":"2024-04-30T08:31:52.706441+0000","last_became_peered":"2024-04-30T08:31:52.706441+0000","last_unstale":"2024-04-30T08:32:30.555177+0000","last_undegraded":"2024-04-30T08:32:30.555177+0000","last_fullsized":"2024-04-30T08:32:30.555177+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:15:45.768927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.669809+0000","last_change":"2024-04-30T08:31:57.740324+0000","last_active":"2024-04-30T08:31:59.669809+0000","last_peered":"2024-04-30T08:31:59.669809+0000","last_clean":"2024-04-30T08:31:59.669809+0000","last_became_active":"2024-04-30T08:31:57.739464+0000","last_became_peered":"2024-04-30T08:31:57.739464+0000","last_unstale":"2024-04-30T08:31:59.669809+0000","last_undegraded":"2024-04-30T08:31:59.669809+0000","last_fullsized":"2024-04-30T08:31:59.669809+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:11:37.395545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"81'4","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:32:26.513947+0000","last_change":"2024-04-30T08:31:51.695283+0000","last_active":"2024-04-30T08:32:26.513947+0000","last_peered":"2024-04-30T08:32:26.513947+0000","last_clean":"2024-04-30T08:32:26.513947+0000","last_became_active":"2024-04-30T08:31:51.694848+0000","last_became_peered":"2024-04-30T08:31:51.694848+0000","last_unstale":"2024-04-30T08:32:26.513947+0000","last_undegraded":"2024-04-30T08:32:26.513947+0000","last_fullsized":"2024-04-30T08:32:26.513947+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:41:38.638034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548438+0000","last_change":"2024-04-30T08:31:52.693594+0000","last_active":"2024-04-30T08:32:30.548438+0000","last_peered":"2024-04-30T08:32:30.548438+0000","last_clean":"2024-04-30T08:32:30.548438+0000","last_became_active":"2024-04-30T08:31:52.693199+0000","last_became_peered":"2024-04-30T08:31:52.693199+0000","last_unstale":"2024-04-30T08:32:30.548438+0000","last_undegraded":"2024-04-30T08:32:30.548438+0000","last_fullsized":"2024-04-30T08:32:30.548438+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:57:49.908617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.553991+0000","last_change":"2024-04-30T08:31:57.749727+0000","last_active":"2024-04-30T08:32:30.553991+0000","last_peered":"2024-04-30T08:32:30.553991+0000","last_clean":"2024-04-30T08:32:30.553991+0000","last_became_active":"2024-04-30T08:31:57.749267+0000","last_became_peered":"2024-04-30T08:31:57.749267+0000","last_unstale":"2024-04-30T08:32:30.553991+0000","last_undegraded":"2024-04-30T08:32:30.553991+0000","last_fullsized":"2024-04-30T08:32:30.553991+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:26:12.683008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"81'2","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:32:26.511058+0000","last_change":"2024-04-30T08:31:51.677398+0000","last_active":"2024-04-30T08:32:26.511058+0000","last_peered":"2024-04-30T08:32:26.511058+0000","last_clean":"2024-04-30T08:32:26.511058+0000","last_became_active":"2024-04-30T08:31:51.677031+0000","last_became_peered":"2024-04-30T08:31:51.677031+0000","last_unstale":"2024-04-30T08:32:26.511058+0000","last_undegraded":"2024-04-30T08:32:26.511058+0000","last_fullsized":"2024-04-30T08:32:26.511058+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:24:25.154229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.835058+0000","last_change":"2024-04-30T08:31:52.717016+0000","last_active":"2024-04-30T08:31:59.835058+0000","last_peered":"2024-04-30T08:31:59.835058+0000","last_clean":"2024-04-30T08:31:59.835058+0000","last_became_active":"2024-04-30T08:31:52.714622+0000","last_became_peered":"2024-04-30T08:31:52.714622+0000","last_unstale":"2024-04-30T08:31:59.835058+0000","last_undegraded":"2024-04-30T08:31:59.835058+0000","last_fullsized":"2024-04-30T08:31:59.835058+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:56:44.550915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.835241+0000","last_change":"2024-04-30T08:31:57.755026+0000","last_active":"2024-04-30T08:31:59.835241+0000","last_peered":"2024-04-30T08:31:59.835241+0000","last_clean":"2024-04-30T08:31:59.835241+0000","last_became_active":"2024-04-30T08:31:57.754139+0000","last_became_peered":"2024-04-30T08:31:57.754139+0000","last_unstale":"2024-04-30T08:31:59.835241+0000","last_undegraded":"2024-04-30T08:31:59.835241+0000","last_fullsized":"2024-04-30T08:31:59.835241+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:38:33.918185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"81'4","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834746+0000","last_change":"2024-04-30T08:31:51.694424+0000","last_active":"2024-04-30T08:31:59.834746+0000","last_peered":"2024-04-30T08:31:59.834746+0000","last_clean":"2024-04-30T08:31:59.834746+0000","last_became_active":"2024-04-30T08:31:51.694081+0000","last_became_peered":"2024-04-30T08:31:51.694081+0000","last_unstale":"2024-04-30T08:31:59.834746+0000","last_undegraded":"2024-04-30T08:31:59.834746+0000","last_fullsized":"2024-04-30T08:31:59.834746+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:27:33.328477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548544+0000","last_change":"2024-04-30T08:31:52.684739+0000","last_active":"2024-04-30T08:32:30.548544+0000","last_peered":"2024-04-30T08:32:30.548544+0000","last_clean":"2024-04-30T08:32:30.548544+0000","last_became_active":"2024-04-30T08:31:52.684316+0000","last_became_peered":"2024-04-30T08:31:52.684316+0000","last_unstale":"2024-04-30T08:32:30.548544+0000","last_undegraded":"2024-04-30T08:32:30.548544+0000","last_fullsized":"2024-04-30T08:32:30.548544+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:12:05.437493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.615550+0000","last_change":"2024-04-30T08:31:57.760643+0000","last_active":"2024-04-30T08:31:59.615550+0000","last_peered":"2024-04-30T08:31:59.615550+0000","last_clean":"2024-04-30T08:31:59.615550+0000","last_became_active":"2024-04-30T08:31:57.760053+0000","last_became_peered":"2024-04-30T08:31:57.760053+0000","last_unstale":"2024-04-30T08:31:59.615550+0000","last_undegraded":"2024-04-30T08:31:59.615550+0000","last_fullsized":"2024-04-30T08:31:59.615550+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:29:12.748550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"81'12","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:32:26.511027+0000","last_change":"2024-04-30T08:31:51.676223+0000","last_active":"2024-04-30T08:32:26.511027+0000","last_peered":"2024-04-30T08:32:26.511027+0000","last_clean":"2024-04-30T08:32:26.511027+0000","last_became_active":"2024-04-30T08:31:51.675830+0000","last_became_peered":"2024-04-30T08:31:51.675830+0000","last_unstale":"2024-04-30T08:32:26.511027+0000","last_undegraded":"2024-04-30T08:32:26.511027+0000","last_fullsized":"2024-04-30T08:32:26.511027+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:28:49.497719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.358155+0000","last_change":"2024-04-30T08:31:52.686892+0000","last_active":"2024-04-30T08:31:59.358155+0000","last_peered":"2024-04-30T08:31:59.358155+0000","last_clean":"2024-04-30T08:31:59.358155+0000","last_became_active":"2024-04-30T08:31:52.686533+0000","last_became_peered":"2024-04-30T08:31:52.686533+0000","last_unstale":"2024-04-30T08:31:59.358155+0000","last_undegraded":"2024-04-30T08:31:59.358155+0000","last_fullsized":"2024-04-30T08:31:59.358155+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:50:18.836161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.835167+0000","last_change":"2024-04-30T08:31:57.752865+0000","last_active":"2024-04-30T08:31:59.835167+0000","last_peered":"2024-04-30T08:31:59.835167+0000","last_clean":"2024-04-30T08:31:59.835167+0000","last_became_active":"2024-04-30T08:31:57.752385+0000","last_became_peered":"2024-04-30T08:31:57.752385+0000","last_unstale":"2024-04-30T08:31:59.835167+0000","last_undegraded":"2024-04-30T08:31:59.835167+0000","last_fullsized":"2024-04-30T08:31:59.835167+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:15:15.315580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"81'9","reported_seq":2,"reported_epoch":82,"state":"peering","last_fresh":"2024-04-30T08:32:29.552341+0000","last_change":"2024-04-30T08:32:30.944941+0000","last_active":"2024-04-30T08:32:30.944941+0000","last_peered":"2024-04-30T08:32:29.552341+0000","last_clean":"2024-04-30T08:32:29.552341+0000","last_became_active":"2024-04-30T08:31:51.702122+0000","last_became_peered":"2024-04-30T08:31:51.702122+0000","last_unstale":"2024-04-30T08:32:29.552341+0000","last_undegraded":"2024-04-30T08:32:29.552341+0000","last_fullsized":"2024-04-30T08:32:29.552341+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":6,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.553023+0000","last_change":"2024-04-30T08:31:52.706259+0000","last_active":"2024-04-30T08:32:30.553023+0000","last_peered":"2024-04-30T08:32:30.553023+0000","last_clean":"2024-04-30T08:32:30.553023+0000","last_became_active":"2024-04-30T08:31:52.705620+0000","last_became_peered":"2024-04-30T08:31:52.705620+0000","last_unstale":"2024-04-30T08:32:30.553023+0000","last_undegraded":"2024-04-30T08:32:30.553023+0000","last_fullsized":"2024-04-30T08:32:30.553023+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:22:29.816950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"peering","last_fresh":"2024-04-30T08:32:00.941417+0000","last_change":"2024-04-30T08:32:30.548960+0000","last_active":"2024-04-30T08:32:30.548960+0000","last_peered":"2024-04-30T08:32:00.941417+0000","last_clean":"2024-04-30T08:32:00.941417+0000","last_became_active":"2024-04-30T08:31:57.762531+0000","last_became_peered":"2024-04-30T08:31:57.762531+0000","last_unstale":"2024-04-30T08:32:00.941417+0000","last_undegraded":"2024-04-30T08:32:00.941417+0000","last_fullsized":"2024-04-30T08:32:00.941417+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7,8,9],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"81'3","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:32:26.512468+0000","last_change":"2024-04-30T08:31:51.701122+0000","last_active":"2024-04-30T08:32:26.512468+0000","last_peered":"2024-04-30T08:32:26.512468+0000","last_clean":"2024-04-30T08:32:26.512468+0000","last_became_active":"2024-04-30T08:31:51.700815+0000","last_became_peered":"2024-04-30T08:31:51.700815+0000","last_unstale":"2024-04-30T08:32:26.512468+0000","last_undegraded":"2024-04-30T08:32:26.512468+0000","last_fullsized":"2024-04-30T08:32:26.512468+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:33:12.714979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.836066+0000","last_change":"2024-04-30T08:31:52.716043+0000","last_active":"2024-04-30T08:31:59.836066+0000","last_peered":"2024-04-30T08:31:59.836066+0000","last_clean":"2024-04-30T08:31:59.836066+0000","last_became_active":"2024-04-30T08:31:52.715284+0000","last_became_peered":"2024-04-30T08:31:52.715284+0000","last_unstale":"2024-04-30T08:31:59.836066+0000","last_undegraded":"2024-04-30T08:31:59.836066+0000","last_fullsized":"2024-04-30T08:31:59.836066+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:00:01.863606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.557749+0000","last_change":"2024-04-30T08:31:57.765008+0000","last_active":"2024-04-30T08:32:30.557749+0000","last_peered":"2024-04-30T08:32:30.557749+0000","last_clean":"2024-04-30T08:32:30.557749+0000","last_became_active":"2024-04-30T08:31:57.762958+0000","last_became_peered":"2024-04-30T08:31:57.762958+0000","last_unstale":"2024-04-30T08:32:30.557749+0000","last_undegraded":"2024-04-30T08:32:30.557749+0000","last_fullsized":"2024-04-30T08:32:30.557749+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:48:47.116004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"81'5","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:32:28.526616+0000","last_change":"2024-04-30T08:31:51.676681+0000","last_active":"2024-04-30T08:32:28.526616+0000","last_peered":"2024-04-30T08:32:28.526616+0000","last_clean":"2024-04-30T08:32:28.526616+0000","last_became_active":"2024-04-30T08:31:51.676311+0000","last_became_peered":"2024-04-30T08:31:51.676311+0000","last_unstale":"2024-04-30T08:32:28.526616+0000","last_undegraded":"2024-04-30T08:32:28.526616+0000","last_fullsized":"2024-04-30T08:32:28.526616+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:24:49.757427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834789+0000","last_change":"2024-04-30T08:31:52.689418+0000","last_active":"2024-04-30T08:31:59.834789+0000","last_peered":"2024-04-30T08:31:59.834789+0000","last_clean":"2024-04-30T08:31:59.834789+0000","last_became_active":"2024-04-30T08:31:52.688840+0000","last_became_peered":"2024-04-30T08:31:52.688840+0000","last_unstale":"2024-04-30T08:31:59.834789+0000","last_undegraded":"2024-04-30T08:31:59.834789+0000","last_fullsized":"2024-04-30T08:31:59.834789+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:40:05.781137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.554105+0000","last_change":"2024-04-30T08:31:57.765875+0000","last_active":"2024-04-30T08:32:30.554105+0000","last_peered":"2024-04-30T08:32:30.554105+0000","last_clean":"2024-04-30T08:32:30.554105+0000","last_became_active":"2024-04-30T08:31:57.765432+0000","last_became_peered":"2024-04-30T08:31:57.765432+0000","last_unstale":"2024-04-30T08:32:30.554105+0000","last_undegraded":"2024-04-30T08:32:30.554105+0000","last_fullsized":"2024-04-30T08:32:30.554105+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:24:44.329532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"81'16","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:32:27.524097+0000","last_change":"2024-04-30T08:31:51.695007+0000","last_active":"2024-04-30T08:32:27.524097+0000","last_peered":"2024-04-30T08:32:27.524097+0000","last_clean":"2024-04-30T08:32:27.524097+0000","last_became_active":"2024-04-30T08:31:51.694533+0000","last_became_peered":"2024-04-30T08:31:51.694533+0000","last_unstale":"2024-04-30T08:32:27.524097+0000","last_undegraded":"2024-04-30T08:32:27.524097+0000","last_fullsized":"2024-04-30T08:32:27.524097+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:47:15.104172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.835658+0000","last_change":"2024-04-30T08:31:52.692274+0000","last_active":"2024-04-30T08:31:59.835658+0000","last_peered":"2024-04-30T08:31:59.835658+0000","last_clean":"2024-04-30T08:31:59.835658+0000","last_became_active":"2024-04-30T08:31:52.691809+0000","last_became_peered":"2024-04-30T08:31:52.691809+0000","last_unstale":"2024-04-30T08:31:59.835658+0000","last_undegraded":"2024-04-30T08:31:59.835658+0000","last_fullsized":"2024-04-30T08:31:59.835658+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:50:00.201782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.835698+0000","last_change":"2024-04-30T08:31:57.732383+0000","last_active":"2024-04-30T08:31:59.835698+0000","last_peered":"2024-04-30T08:31:59.835698+0000","last_clean":"2024-04-30T08:31:59.835698+0000","last_became_active":"2024-04-30T08:31:57.731893+0000","last_became_peered":"2024-04-30T08:31:57.731893+0000","last_unstale":"2024-04-30T08:31:59.835698+0000","last_undegraded":"2024-04-30T08:31:59.835698+0000","last_fullsized":"2024-04-30T08:31:59.835698+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:08:55.243526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"81'2","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.358322+0000","last_change":"2024-04-30T08:31:51.671991+0000","last_active":"2024-04-30T08:31:59.358322+0000","last_peered":"2024-04-30T08:31:59.358322+0000","last_clean":"2024-04-30T08:31:59.358322+0000","last_became_active":"2024-04-30T08:31:51.671508+0000","last_became_peered":"2024-04-30T08:31:51.671508+0000","last_unstale":"2024-04-30T08:31:59.358322+0000","last_undegraded":"2024-04-30T08:31:59.358322+0000","last_fullsized":"2024-04-30T08:31:59.358322+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:54:30.106748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548288+0000","last_change":"2024-04-30T08:31:52.687555+0000","last_active":"2024-04-30T08:32:30.548288+0000","last_peered":"2024-04-30T08:32:30.548288+0000","last_clean":"2024-04-30T08:32:30.548288+0000","last_became_active":"2024-04-30T08:31:52.686000+0000","last_became_peered":"2024-04-30T08:31:52.686000+0000","last_unstale":"2024-04-30T08:32:30.548288+0000","last_undegraded":"2024-04-30T08:32:30.548288+0000","last_fullsized":"2024-04-30T08:32:30.548288+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:37:03.529774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.357960+0000","last_change":"2024-04-30T08:31:58.061077+0000","last_active":"2024-04-30T08:31:59.357960+0000","last_peered":"2024-04-30T08:31:59.357960+0000","last_clean":"2024-04-30T08:31:59.357960+0000","last_became_active":"2024-04-30T08:31:58.060486+0000","last_became_peered":"2024-04-30T08:31:58.060486+0000","last_unstale":"2024-04-30T08:31:59.357960+0000","last_undegraded":"2024-04-30T08:31:59.357960+0000","last_fullsized":"2024-04-30T08:31:59.357960+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:31:50.936560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"81'1","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.557573+0000","last_change":"2024-04-30T08:31:51.702869+0000","last_active":"2024-04-30T08:32:30.557573+0000","last_peered":"2024-04-30T08:32:30.557573+0000","last_clean":"2024-04-30T08:32:30.557573+0000","last_became_active":"2024-04-30T08:31:51.701150+0000","last_became_peered":"2024-04-30T08:31:51.701150+0000","last_unstale":"2024-04-30T08:32:30.557573+0000","last_undegraded":"2024-04-30T08:32:30.557573+0000","last_fullsized":"2024-04-30T08:32:30.557573+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:47:11.010310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:58.709852+0000","last_change":"2024-04-30T08:31:52.709159+0000","last_active":"2024-04-30T08:31:58.709852+0000","last_peered":"2024-04-30T08:31:58.709852+0000","last_clean":"2024-04-30T08:31:58.709852+0000","last_became_active":"2024-04-30T08:31:52.708799+0000","last_became_peered":"2024-04-30T08:31:52.708799+0000","last_unstale":"2024-04-30T08:31:58.709852+0000","last_undegraded":"2024-04-30T08:31:58.709852+0000","last_fullsized":"2024-04-30T08:31:58.709852+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:51:10.588599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.554141+0000","last_change":"2024-04-30T08:31:57.751652+0000","last_active":"2024-04-30T08:32:30.554141+0000","last_peered":"2024-04-30T08:32:30.554141+0000","last_clean":"2024-04-30T08:32:30.554141+0000","last_became_active":"2024-04-30T08:31:57.751176+0000","last_became_peered":"2024-04-30T08:31:57.751176+0000","last_unstale":"2024-04-30T08:32:30.554141+0000","last_undegraded":"2024-04-30T08:32:30.554141+0000","last_fullsized":"2024-04-30T08:32:30.554141+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:13:24.528842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"81'1","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.835190+0000","last_change":"2024-04-30T08:31:51.700816+0000","last_active":"2024-04-30T08:31:59.835190+0000","last_peered":"2024-04-30T08:31:59.835190+0000","last_clean":"2024-04-30T08:31:59.835190+0000","last_became_active":"2024-04-30T08:31:51.700470+0000","last_became_peered":"2024-04-30T08:31:51.700470+0000","last_unstale":"2024-04-30T08:31:59.835190+0000","last_undegraded":"2024-04-30T08:31:59.835190+0000","last_fullsized":"2024-04-30T08:31:59.835190+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:49:11.793412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:58.709513+0000","last_change":"2024-04-30T08:31:52.715236+0000","last_active":"2024-04-30T08:31:58.709513+0000","last_peered":"2024-04-30T08:31:58.709513+0000","last_clean":"2024-04-30T08:31:58.709513+0000","last_became_active":"2024-04-30T08:31:52.714801+0000","last_became_peered":"2024-04-30T08:31:52.714801+0000","last_unstale":"2024-04-30T08:31:58.709513+0000","last_undegraded":"2024-04-30T08:31:58.709513+0000","last_fullsized":"2024-04-30T08:31:58.709513+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:43:09.650002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.552783+0000","last_change":"2024-04-30T08:31:58.060927+0000","last_active":"2024-04-30T08:32:30.552783+0000","last_peered":"2024-04-30T08:32:30.552783+0000","last_clean":"2024-04-30T08:32:30.552783+0000","last_became_active":"2024-04-30T08:31:58.060182+0000","last_became_peered":"2024-04-30T08:31:58.060182+0000","last_unstale":"2024-04-30T08:32:30.552783+0000","last_undegraded":"2024-04-30T08:32:30.552783+0000","last_fullsized":"2024-04-30T08:32:30.552783+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:21:00.228064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834850+0000","last_change":"2024-04-30T08:31:52.692774+0000","last_active":"2024-04-30T08:31:59.834850+0000","last_peered":"2024-04-30T08:31:59.834850+0000","last_clean":"2024-04-30T08:31:59.834850+0000","last_became_active":"2024-04-30T08:31:52.692321+0000","last_became_peered":"2024-04-30T08:31:52.692321+0000","last_unstale":"2024-04-30T08:31:59.834850+0000","last_undegraded":"2024-04-30T08:31:59.834850+0000","last_fullsized":"2024-04-30T08:31:59.834850+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:02:54.217333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"81'5","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.557878+0000","last_change":"2024-04-30T08:31:51.702808+0000","last_active":"2024-04-30T08:32:30.557878+0000","last_peered":"2024-04-30T08:32:30.557878+0000","last_clean":"2024-04-30T08:32:30.557878+0000","last_became_active":"2024-04-30T08:31:51.702326+0000","last_became_peered":"2024-04-30T08:31:51.702326+0000","last_unstale":"2024-04-30T08:32:30.557878+0000","last_undegraded":"2024-04-30T08:32:30.557878+0000","last_fullsized":"2024-04-30T08:32:30.557878+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:11:35.552670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.836030+0000","last_change":"2024-04-30T08:31:57.760359+0000","last_active":"2024-04-30T08:31:59.836030+0000","last_peered":"2024-04-30T08:31:59.836030+0000","last_clean":"2024-04-30T08:31:59.836030+0000","last_became_active":"2024-04-30T08:31:57.759821+0000","last_became_peered":"2024-04-30T08:31:57.759821+0000","last_unstale":"2024-04-30T08:31:59.836030+0000","last_undegraded":"2024-04-30T08:31:59.836030+0000","last_fullsized":"2024-04-30T08:31:59.836030+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:50:17.853791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.669652+0000","last_change":"2024-04-30T08:31:52.689281+0000","last_active":"2024-04-30T08:31:59.669652+0000","last_peered":"2024-04-30T08:31:59.669652+0000","last_clean":"2024-04-30T08:31:59.669652+0000","last_became_active":"2024-04-30T08:31:52.688834+0000","last_became_peered":"2024-04-30T08:31:52.688834+0000","last_unstale":"2024-04-30T08:31:59.669652+0000","last_undegraded":"2024-04-30T08:31:59.669652+0000","last_fullsized":"2024-04-30T08:31:59.669652+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:42:12.836649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'5","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:32:28.529946+0000","last_change":"2024-04-30T08:31:51.683712+0000","last_active":"2024-04-30T08:32:28.529946+0000","last_peered":"2024-04-30T08:32:28.529946+0000","last_clean":"2024-04-30T08:32:28.529946+0000","last_became_active":"2024-04-30T08:31:51.683298+0000","last_became_peered":"2024-04-30T08:31:51.683298+0000","last_unstale":"2024-04-30T08:32:28.529946+0000","last_undegraded":"2024-04-30T08:32:28.529946+0000","last_fullsized":"2024-04-30T08:32:28.529946+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:43:08.832104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":99,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834654+0000","last_change":"2024-04-30T08:30:26.503221+0000","last_active":"2024-04-30T08:31:59.834654+0000","last_peered":"2024-04-30T08:31:59.834654+0000","last_clean":"2024-04-30T08:31:59.834654+0000","last_became_active":"2024-04-30T08:30:26.498399+0000","last_became_peered":"2024-04-30T08:30:26.498399+0000","last_unstale":"2024-04-30T08:31:59.834654+0000","last_undegraded":"2024-04-30T08:31:59.834654+0000","last_fullsized":"2024-04-30T08:31:59.834654+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:26:20.170075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:26:20.170075+0000","last_clean_scrub_stamp":"2024-04-30T08:26:20.170075+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:34:44.067560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.835159+0000","last_change":"2024-04-30T08:31:58.061283+0000","last_active":"2024-04-30T08:31:59.835159+0000","last_peered":"2024-04-30T08:31:59.835159+0000","last_clean":"2024-04-30T08:31:59.835159+0000","last_became_active":"2024-04-30T08:31:58.060650+0000","last_became_peered":"2024-04-30T08:31:58.060650+0000","last_unstale":"2024-04-30T08:31:59.835159+0000","last_undegraded":"2024-04-30T08:31:59.835159+0000","last_fullsized":"2024-04-30T08:31:59.835159+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:13:01.811462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.551989+0000","last_change":"2024-04-30T08:31:52.699639+0000","last_active":"2024-04-30T08:32:30.551989+0000","last_peered":"2024-04-30T08:32:30.551989+0000","last_clean":"2024-04-30T08:32:30.551989+0000","last_became_active":"2024-04-30T08:31:52.699164+0000","last_became_peered":"2024-04-30T08:31:52.699164+0000","last_unstale":"2024-04-30T08:32:30.551989+0000","last_undegraded":"2024-04-30T08:32:30.551989+0000","last_fullsized":"2024-04-30T08:32:30.551989+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:22:18.873504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"81'3","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:32:27.517971+0000","last_change":"2024-04-30T08:31:51.679153+0000","last_active":"2024-04-30T08:32:27.517971+0000","last_peered":"2024-04-30T08:32:27.517971+0000","last_clean":"2024-04-30T08:32:27.517971+0000","last_became_active":"2024-04-30T08:31:51.678348+0000","last_became_peered":"2024-04-30T08:31:51.678348+0000","last_unstale":"2024-04-30T08:32:27.517971+0000","last_undegraded":"2024-04-30T08:32:27.517971+0000","last_fullsized":"2024-04-30T08:32:27.517971+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:02:56.240218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548074+0000","last_change":"2024-04-30T08:31:57.755423+0000","last_active":"2024-04-30T08:32:30.548074+0000","last_peered":"2024-04-30T08:32:30.548074+0000","last_clean":"2024-04-30T08:32:30.548074+0000","last_became_active":"2024-04-30T08:31:57.754835+0000","last_became_peered":"2024-04-30T08:31:57.754835+0000","last_unstale":"2024-04-30T08:32:30.548074+0000","last_undegraded":"2024-04-30T08:32:30.548074+0000","last_fullsized":"2024-04-30T08:32:30.548074+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:19:25.168920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"81'3","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:32:28.529739+0000","last_change":"2024-04-30T08:31:51.693839+0000","last_active":"2024-04-30T08:32:28.529739+0000","last_peered":"2024-04-30T08:32:28.529739+0000","last_clean":"2024-04-30T08:32:28.529739+0000","last_became_active":"2024-04-30T08:31:51.693349+0000","last_became_peered":"2024-04-30T08:31:51.693349+0000","last_unstale":"2024-04-30T08:32:28.529739+0000","last_undegraded":"2024-04-30T08:32:28.529739+0000","last_fullsized":"2024-04-30T08:32:28.529739+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:41:51.330901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.554333+0000","last_change":"2024-04-30T08:31:52.712360+0000","last_active":"2024-04-30T08:32:30.554333+0000","last_peered":"2024-04-30T08:32:30.554333+0000","last_clean":"2024-04-30T08:32:30.554333+0000","last_became_active":"2024-04-30T08:31:52.711652+0000","last_became_peered":"2024-04-30T08:31:52.711652+0000","last_unstale":"2024-04-30T08:32:30.554333+0000","last_undegraded":"2024-04-30T08:32:30.554333+0000","last_fullsized":"2024-04-30T08:32:30.554333+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:21:10.285386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.669947+0000","last_change":"2024-04-30T08:31:58.061654+0000","last_active":"2024-04-30T08:31:59.669947+0000","last_peered":"2024-04-30T08:31:59.669947+0000","last_clean":"2024-04-30T08:31:59.669947+0000","last_became_active":"2024-04-30T08:31:58.060181+0000","last_became_peered":"2024-04-30T08:31:58.060181+0000","last_unstale":"2024-04-30T08:31:59.669947+0000","last_undegraded":"2024-04-30T08:31:59.669947+0000","last_fullsized":"2024-04-30T08:31:59.669947+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:46:03.230626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"81'8","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.558636+0000","last_change":"2024-04-30T08:31:51.706940+0000","last_active":"2024-04-30T08:32:30.558636+0000","last_peered":"2024-04-30T08:32:30.558636+0000","last_clean":"2024-04-30T08:32:30.558636+0000","last_became_active":"2024-04-30T08:31:51.706481+0000","last_became_peered":"2024-04-30T08:31:51.706481+0000","last_unstale":"2024-04-30T08:32:30.558636+0000","last_undegraded":"2024-04-30T08:32:30.558636+0000","last_fullsized":"2024-04-30T08:32:30.558636+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:53:31.955595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.670853+0000","last_change":"2024-04-30T08:31:52.696625+0000","last_active":"2024-04-30T08:31:59.670853+0000","last_peered":"2024-04-30T08:31:59.670853+0000","last_clean":"2024-04-30T08:31:59.670853+0000","last_became_active":"2024-04-30T08:31:52.696230+0000","last_became_peered":"2024-04-30T08:31:52.696230+0000","last_unstale":"2024-04-30T08:31:59.670853+0000","last_undegraded":"2024-04-30T08:31:59.670853+0000","last_fullsized":"2024-04-30T08:31:59.670853+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:11:46.404251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834193+0000","last_change":"2024-04-30T08:31:57.738329+0000","last_active":"2024-04-30T08:31:59.834193+0000","last_peered":"2024-04-30T08:31:59.834193+0000","last_clean":"2024-04-30T08:31:59.834193+0000","last_became_active":"2024-04-30T08:31:57.737897+0000","last_became_peered":"2024-04-30T08:31:57.737897+0000","last_unstale":"2024-04-30T08:31:59.834193+0000","last_undegraded":"2024-04-30T08:31:59.834193+0000","last_fullsized":"2024-04-30T08:31:59.834193+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:02:43.074578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"81'5","reported_seq":30,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:32:26.512280+0000","last_change":"2024-04-30T08:31:51.684066+0000","last_active":"2024-04-30T08:32:26.512280+0000","last_peered":"2024-04-30T08:32:26.512280+0000","last_clean":"2024-04-30T08:32:26.512280+0000","last_became_active":"2024-04-30T08:31:51.683699+0000","last_became_peered":"2024-04-30T08:31:51.683699+0000","last_unstale":"2024-04-30T08:32:26.512280+0000","last_undegraded":"2024-04-30T08:32:26.512280+0000","last_fullsized":"2024-04-30T08:32:26.512280+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:36:20.900733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.835032+0000","last_change":"2024-04-30T08:31:52.715219+0000","last_active":"2024-04-30T08:31:59.835032+0000","last_peered":"2024-04-30T08:31:59.835032+0000","last_clean":"2024-04-30T08:31:59.835032+0000","last_became_active":"2024-04-30T08:31:52.714774+0000","last_became_peered":"2024-04-30T08:31:52.714774+0000","last_unstale":"2024-04-30T08:31:59.835032+0000","last_undegraded":"2024-04-30T08:31:59.835032+0000","last_fullsized":"2024-04-30T08:31:59.835032+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:42:41.082610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.835151+0000","last_change":"2024-04-30T08:31:58.060845+0000","last_active":"2024-04-30T08:31:59.835151+0000","last_peered":"2024-04-30T08:31:59.835151+0000","last_clean":"2024-04-30T08:31:59.835151+0000","last_became_active":"2024-04-30T08:31:58.060365+0000","last_became_peered":"2024-04-30T08:31:58.060365+0000","last_unstale":"2024-04-30T08:31:59.835151+0000","last_undegraded":"2024-04-30T08:31:59.835151+0000","last_fullsized":"2024-04-30T08:31:59.835151+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:21:39.583243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"81'2","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.547845+0000","last_change":"2024-04-30T08:31:51.694833+0000","last_active":"2024-04-30T08:32:30.547845+0000","last_peered":"2024-04-30T08:32:30.547845+0000","last_clean":"2024-04-30T08:32:30.547845+0000","last_became_active":"2024-04-30T08:31:51.694401+0000","last_became_peered":"2024-04-30T08:31:51.694401+0000","last_unstale":"2024-04-30T08:32:30.547845+0000","last_undegraded":"2024-04-30T08:32:30.547845+0000","last_fullsized":"2024-04-30T08:32:30.547845+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:38:52.605625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.357703+0000","last_change":"2024-04-30T08:31:52.686444+0000","last_active":"2024-04-30T08:31:59.357703+0000","last_peered":"2024-04-30T08:31:59.357703+0000","last_clean":"2024-04-30T08:31:59.357703+0000","last_became_active":"2024-04-30T08:31:52.686057+0000","last_became_peered":"2024-04-30T08:31:52.686057+0000","last_unstale":"2024-04-30T08:31:59.357703+0000","last_undegraded":"2024-04-30T08:31:59.357703+0000","last_fullsized":"2024-04-30T08:31:59.357703+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:54:29.824060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.557873+0000","last_change":"2024-04-30T08:31:57.766193+0000","last_active":"2024-04-30T08:32:30.557873+0000","last_peered":"2024-04-30T08:32:30.557873+0000","last_clean":"2024-04-30T08:32:30.557873+0000","last_became_active":"2024-04-30T08:31:57.765758+0000","last_became_peered":"2024-04-30T08:31:57.765758+0000","last_unstale":"2024-04-30T08:32:30.557873+0000","last_undegraded":"2024-04-30T08:32:30.557873+0000","last_fullsized":"2024-04-30T08:32:30.557873+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:59:57.282795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"81'7","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:32:29.554238+0000","last_change":"2024-04-30T08:31:51.704565+0000","last_active":"2024-04-30T08:32:29.554238+0000","last_peered":"2024-04-30T08:32:29.554238+0000","last_clean":"2024-04-30T08:32:29.554238+0000","last_became_active":"2024-04-30T08:31:51.703954+0000","last_became_peered":"2024-04-30T08:31:51.703954+0000","last_unstale":"2024-04-30T08:32:29.554238+0000","last_undegraded":"2024-04-30T08:32:29.554238+0000","last_fullsized":"2024-04-30T08:32:29.554238+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:03:00.517867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548041+0000","last_change":"2024-04-30T08:31:52.706043+0000","last_active":"2024-04-30T08:32:30.548041+0000","last_peered":"2024-04-30T08:32:30.548041+0000","last_clean":"2024-04-30T08:32:30.548041+0000","last_became_active":"2024-04-30T08:31:52.705678+0000","last_became_peered":"2024-04-30T08:31:52.705678+0000","last_unstale":"2024-04-30T08:32:30.548041+0000","last_undegraded":"2024-04-30T08:32:30.548041+0000","last_fullsized":"2024-04-30T08:32:30.548041+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:44:56.787835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.835420+0000","last_change":"2024-04-30T08:31:57.764791+0000","last_active":"2024-04-30T08:31:59.835420+0000","last_peered":"2024-04-30T08:31:59.835420+0000","last_clean":"2024-04-30T08:31:59.835420+0000","last_became_active":"2024-04-30T08:31:57.764368+0000","last_became_peered":"2024-04-30T08:31:57.764368+0000","last_unstale":"2024-04-30T08:31:59.835420+0000","last_undegraded":"2024-04-30T08:31:59.835420+0000","last_fullsized":"2024-04-30T08:31:59.835420+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:19:28.545483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"81'3","reported_seq":2,"reported_epoch":82,"state":"peering","last_fresh":"2024-04-30T08:32:30.524198+0000","last_change":"2024-04-30T08:32:30.944913+0000","last_active":"2024-04-30T08:32:30.944913+0000","last_peered":"2024-04-30T08:32:30.524198+0000","last_clean":"2024-04-30T08:32:30.524198+0000","last_became_active":"2024-04-30T08:31:51.702141+0000","last_became_peered":"2024-04-30T08:31:51.702141+0000","last_unstale":"2024-04-30T08:32:30.524198+0000","last_undegraded":"2024-04-30T08:32:30.524198+0000","last_fullsized":"2024-04-30T08:32:30.524198+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.670593+0000","last_change":"2024-04-30T08:31:52.716707+0000","last_active":"2024-04-30T08:31:59.670593+0000","last_peered":"2024-04-30T08:31:59.670593+0000","last_clean":"2024-04-30T08:31:59.670593+0000","last_became_active":"2024-04-30T08:31:52.716293+0000","last_became_peered":"2024-04-30T08:31:52.716293+0000","last_unstale":"2024-04-30T08:31:59.670593+0000","last_undegraded":"2024-04-30T08:31:59.670593+0000","last_fullsized":"2024-04-30T08:31:59.670593+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:29:54.223472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.835458+0000","last_change":"2024-04-30T08:31:58.060640+0000","last_active":"2024-04-30T08:31:59.835458+0000","last_peered":"2024-04-30T08:31:59.835458+0000","last_clean":"2024-04-30T08:31:59.835458+0000","last_became_active":"2024-04-30T08:31:58.060061+0000","last_became_peered":"2024-04-30T08:31:58.060061+0000","last_unstale":"2024-04-30T08:31:59.835458+0000","last_undegraded":"2024-04-30T08:31:59.835458+0000","last_fullsized":"2024-04-30T08:31:59.835458+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:46:00.844191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"81'4","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:32:26.512050+0000","last_change":"2024-04-30T08:31:51.679532+0000","last_active":"2024-04-30T08:32:26.512050+0000","last_peered":"2024-04-30T08:32:26.512050+0000","last_clean":"2024-04-30T08:32:26.512050+0000","last_became_active":"2024-04-30T08:31:51.679141+0000","last_became_peered":"2024-04-30T08:31:51.679141+0000","last_unstale":"2024-04-30T08:32:26.512050+0000","last_undegraded":"2024-04-30T08:32:26.512050+0000","last_fullsized":"2024-04-30T08:32:26.512050+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:19:50.526563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.835256+0000","last_change":"2024-04-30T08:31:52.703613+0000","last_active":"2024-04-30T08:31:59.835256+0000","last_peered":"2024-04-30T08:31:59.835256+0000","last_clean":"2024-04-30T08:31:59.835256+0000","last_became_active":"2024-04-30T08:31:52.703196+0000","last_became_peered":"2024-04-30T08:31:52.703196+0000","last_unstale":"2024-04-30T08:31:59.835256+0000","last_undegraded":"2024-04-30T08:31:59.835256+0000","last_fullsized":"2024-04-30T08:31:59.835256+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:33:11.136313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.547878+0000","last_change":"2024-04-30T08:31:57.755286+0000","last_active":"2024-04-30T08:32:30.547878+0000","last_peered":"2024-04-30T08:32:30.547878+0000","last_clean":"2024-04-30T08:32:30.547878+0000","last_became_active":"2024-04-30T08:31:57.754786+0000","last_became_peered":"2024-04-30T08:31:57.754786+0000","last_unstale":"2024-04-30T08:32:30.547878+0000","last_undegraded":"2024-04-30T08:32:30.547878+0000","last_fullsized":"2024-04-30T08:32:30.547878+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:02:50.931553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"81'5","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:32:28.529719+0000","last_change":"2024-04-30T08:31:51.693317+0000","last_active":"2024-04-30T08:32:28.529719+0000","last_peered":"2024-04-30T08:32:28.529719+0000","last_clean":"2024-04-30T08:32:28.529719+0000","last_became_active":"2024-04-30T08:31:51.692928+0000","last_became_peered":"2024-04-30T08:31:51.692928+0000","last_unstale":"2024-04-30T08:32:28.529719+0000","last_undegraded":"2024-04-30T08:32:28.529719+0000","last_fullsized":"2024-04-30T08:32:28.529719+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:35:09.851735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.836183+0000","last_change":"2024-04-30T08:31:52.694304+0000","last_active":"2024-04-30T08:31:59.836183+0000","last_peered":"2024-04-30T08:31:59.836183+0000","last_clean":"2024-04-30T08:31:59.836183+0000","last_became_active":"2024-04-30T08:31:52.693317+0000","last_became_peered":"2024-04-30T08:31:52.693317+0000","last_unstale":"2024-04-30T08:31:59.836183+0000","last_undegraded":"2024-04-30T08:31:59.836183+0000","last_fullsized":"2024-04-30T08:31:59.836183+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:31:39.294864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834714+0000","last_change":"2024-04-30T08:31:57.763598+0000","last_active":"2024-04-30T08:31:59.834714+0000","last_peered":"2024-04-30T08:31:59.834714+0000","last_clean":"2024-04-30T08:31:59.834714+0000","last_became_active":"2024-04-30T08:31:57.762307+0000","last_became_peered":"2024-04-30T08:31:57.762307+0000","last_unstale":"2024-04-30T08:31:59.834714+0000","last_undegraded":"2024-04-30T08:31:59.834714+0000","last_fullsized":"2024-04-30T08:31:59.834714+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:24:13.390595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"75'1","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834820+0000","last_change":"2024-04-30T08:31:51.703476+0000","last_active":"2024-04-30T08:31:59.834820+0000","last_peered":"2024-04-30T08:31:59.834820+0000","last_clean":"2024-04-30T08:31:59.834820+0000","last_became_active":"2024-04-30T08:31:51.703072+0000","last_became_peered":"2024-04-30T08:31:51.703072+0000","last_unstale":"2024-04-30T08:31:59.834820+0000","last_undegraded":"2024-04-30T08:31:59.834820+0000","last_fullsized":"2024-04-30T08:31:59.834820+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:07:21.098223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.835688+0000","last_change":"2024-04-30T08:31:52.699223+0000","last_active":"2024-04-30T08:31:59.835688+0000","last_peered":"2024-04-30T08:31:59.835688+0000","last_clean":"2024-04-30T08:31:59.835688+0000","last_became_active":"2024-04-30T08:31:52.698837+0000","last_became_peered":"2024-04-30T08:31:52.698837+0000","last_unstale":"2024-04-30T08:31:59.835688+0000","last_undegraded":"2024-04-30T08:31:59.835688+0000","last_fullsized":"2024-04-30T08:31:59.835688+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:19:11.826470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.835310+0000","last_change":"2024-04-30T08:31:57.737438+0000","last_active":"2024-04-30T08:31:59.835310+0000","last_peered":"2024-04-30T08:31:59.835310+0000","last_clean":"2024-04-30T08:31:59.835310+0000","last_became_active":"2024-04-30T08:31:57.736949+0000","last_became_peered":"2024-04-30T08:31:57.736949+0000","last_unstale":"2024-04-30T08:31:59.835310+0000","last_undegraded":"2024-04-30T08:31:59.835310+0000","last_fullsized":"2024-04-30T08:31:59.835310+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:00:10.921846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"81'5","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.554486+0000","last_change":"2024-04-30T08:31:51.708943+0000","last_active":"2024-04-30T08:32:30.554486+0000","last_peered":"2024-04-30T08:32:30.554486+0000","last_clean":"2024-04-30T08:32:30.554486+0000","last_became_active":"2024-04-30T08:31:51.708376+0000","last_became_peered":"2024-04-30T08:31:51.708376+0000","last_unstale":"2024-04-30T08:32:30.554486+0000","last_undegraded":"2024-04-30T08:32:30.554486+0000","last_fullsized":"2024-04-30T08:32:30.554486+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:14:46.692192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.835847+0000","last_change":"2024-04-30T08:31:52.686634+0000","last_active":"2024-04-30T08:31:59.835847+0000","last_peered":"2024-04-30T08:31:59.835847+0000","last_clean":"2024-04-30T08:31:59.835847+0000","last_became_active":"2024-04-30T08:31:52.686231+0000","last_became_peered":"2024-04-30T08:31:52.686231+0000","last_unstale":"2024-04-30T08:31:59.835847+0000","last_undegraded":"2024-04-30T08:31:59.835847+0000","last_fullsized":"2024-04-30T08:31:59.835847+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:27:31.073041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548133+0000","last_change":"2024-04-30T08:31:52.688695+0000","last_active":"2024-04-30T08:32:30.548133+0000","last_peered":"2024-04-30T08:32:30.548133+0000","last_clean":"2024-04-30T08:32:30.548133+0000","last_became_active":"2024-04-30T08:31:52.686556+0000","last_became_peered":"2024-04-30T08:31:52.686556+0000","last_unstale":"2024-04-30T08:32:30.548133+0000","last_undegraded":"2024-04-30T08:32:30.548133+0000","last_fullsized":"2024-04-30T08:32:30.548133+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:22:48.976440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.835365+0000","last_change":"2024-04-30T08:31:52.688324+0000","last_active":"2024-04-30T08:31:59.835365+0000","last_peered":"2024-04-30T08:31:59.835365+0000","last_clean":"2024-04-30T08:31:59.835365+0000","last_became_active":"2024-04-30T08:31:52.687966+0000","last_became_peered":"2024-04-30T08:31:52.687966+0000","last_unstale":"2024-04-30T08:31:59.835365+0000","last_undegraded":"2024-04-30T08:31:59.835365+0000","last_fullsized":"2024-04-30T08:31:59.835365+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:35:38.237750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834334+0000","last_change":"2024-04-30T08:31:52.695638+0000","last_active":"2024-04-30T08:31:59.834334+0000","last_peered":"2024-04-30T08:31:59.834334+0000","last_clean":"2024-04-30T08:31:59.834334+0000","last_became_active":"2024-04-30T08:31:52.695283+0000","last_became_peered":"2024-04-30T08:31:52.695283+0000","last_unstale":"2024-04-30T08:31:59.834334+0000","last_undegraded":"2024-04-30T08:31:59.834334+0000","last_fullsized":"2024-04-30T08:31:59.834334+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:00:14.510381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834598+0000","last_change":"2024-04-30T08:31:52.719112+0000","last_active":"2024-04-30T08:31:59.834598+0000","last_peered":"2024-04-30T08:31:59.834598+0000","last_clean":"2024-04-30T08:31:59.834598+0000","last_became_active":"2024-04-30T08:31:52.718207+0000","last_became_peered":"2024-04-30T08:31:52.718207+0000","last_unstale":"2024-04-30T08:31:59.834598+0000","last_undegraded":"2024-04-30T08:31:59.834598+0000","last_fullsized":"2024-04-30T08:31:59.834598+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:27:29.263962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.553965+0000","last_change":"2024-04-30T08:31:52.712008+0000","last_active":"2024-04-30T08:32:30.553965+0000","last_peered":"2024-04-30T08:32:30.553965+0000","last_clean":"2024-04-30T08:32:30.553965+0000","last_became_active":"2024-04-30T08:31:52.711064+0000","last_became_peered":"2024-04-30T08:31:52.711064+0000","last_unstale":"2024-04-30T08:32:30.553965+0000","last_undegraded":"2024-04-30T08:32:30.553965+0000","last_fullsized":"2024-04-30T08:32:30.553965+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:57:27.400602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548539+0000","last_change":"2024-04-30T08:31:52.691210+0000","last_active":"2024-04-30T08:32:30.548539+0000","last_peered":"2024-04-30T08:32:30.548539+0000","last_clean":"2024-04-30T08:32:30.548539+0000","last_became_active":"2024-04-30T08:31:52.688852+0000","last_became_peered":"2024-04-30T08:31:52.688852+0000","last_unstale":"2024-04-30T08:32:30.548539+0000","last_undegraded":"2024-04-30T08:32:30.548539+0000","last_fullsized":"2024-04-30T08:32:30.548539+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:04:46.502743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.835158+0000","last_change":"2024-04-30T08:31:52.684501+0000","last_active":"2024-04-30T08:31:59.835158+0000","last_peered":"2024-04-30T08:31:59.835158+0000","last_clean":"2024-04-30T08:31:59.835158+0000","last_became_active":"2024-04-30T08:31:52.684122+0000","last_became_peered":"2024-04-30T08:31:52.684122+0000","last_unstale":"2024-04-30T08:31:59.835158+0000","last_undegraded":"2024-04-30T08:31:59.835158+0000","last_fullsized":"2024-04-30T08:31:59.835158+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:55:58.093269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.671093+0000","last_change":"2024-04-30T08:31:52.698016+0000","last_active":"2024-04-30T08:31:59.671093+0000","last_peered":"2024-04-30T08:31:59.671093+0000","last_clean":"2024-04-30T08:31:59.671093+0000","last_became_active":"2024-04-30T08:31:52.695037+0000","last_became_peered":"2024-04-30T08:31:52.695037+0000","last_unstale":"2024-04-30T08:31:59.671093+0000","last_undegraded":"2024-04-30T08:31:59.671093+0000","last_fullsized":"2024-04-30T08:31:59.671093+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:06:01.360947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834710+0000","last_change":"2024-04-30T08:31:52.698415+0000","last_active":"2024-04-30T08:31:59.834710+0000","last_peered":"2024-04-30T08:31:59.834710+0000","last_clean":"2024-04-30T08:31:59.834710+0000","last_became_active":"2024-04-30T08:31:52.697912+0000","last_became_peered":"2024-04-30T08:31:52.697912+0000","last_unstale":"2024-04-30T08:31:59.834710+0000","last_undegraded":"2024-04-30T08:31:59.834710+0000","last_fullsized":"2024-04-30T08:31:59.834710+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:05:44.420331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.670020+0000","last_change":"2024-04-30T08:31:52.694776+0000","last_active":"2024-04-30T08:31:59.670020+0000","last_peered":"2024-04-30T08:31:59.670020+0000","last_clean":"2024-04-30T08:31:59.670020+0000","last_became_active":"2024-04-30T08:31:52.693532+0000","last_became_peered":"2024-04-30T08:31:52.693532+0000","last_unstale":"2024-04-30T08:31:59.670020+0000","last_undegraded":"2024-04-30T08:31:59.670020+0000","last_fullsized":"2024-04-30T08:31:59.670020+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:31:10.186667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548026+0000","last_change":"2024-04-30T08:31:52.688477+0000","last_active":"2024-04-30T08:32:30.548026+0000","last_peered":"2024-04-30T08:32:30.548026+0000","last_clean":"2024-04-30T08:32:30.548026+0000","last_became_active":"2024-04-30T08:31:52.688072+0000","last_became_peered":"2024-04-30T08:31:52.688072+0000","last_unstale":"2024-04-30T08:32:30.548026+0000","last_undegraded":"2024-04-30T08:32:30.548026+0000","last_fullsized":"2024-04-30T08:32:30.548026+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:35:42.659707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.553593+0000","last_change":"2024-04-30T08:31:52.716788+0000","last_active":"2024-04-30T08:32:30.553593+0000","last_peered":"2024-04-30T08:32:30.553593+0000","last_clean":"2024-04-30T08:32:30.553593+0000","last_became_active":"2024-04-30T08:31:52.716336+0000","last_became_peered":"2024-04-30T08:31:52.716336+0000","last_unstale":"2024-04-30T08:32:30.553593+0000","last_undegraded":"2024-04-30T08:32:30.553593+0000","last_fullsized":"2024-04-30T08:32:30.553593+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:23:23.477968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548294+0000","last_change":"2024-04-30T08:31:52.684571+0000","last_active":"2024-04-30T08:32:30.548294+0000","last_peered":"2024-04-30T08:32:30.548294+0000","last_clean":"2024-04-30T08:32:30.548294+0000","last_became_active":"2024-04-30T08:31:52.684192+0000","last_became_peered":"2024-04-30T08:31:52.684192+0000","last_unstale":"2024-04-30T08:32:30.548294+0000","last_undegraded":"2024-04-30T08:32:30.548294+0000","last_fullsized":"2024-04-30T08:32:30.548294+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:05:57.520133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":12288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284927,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8298,"num_objects":82,"num_object_clones":0,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":8,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":180,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":56,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":380928,"data_stored":68224,"data_compressed":1841,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},"log_size":155,"ondisk_log_size":155,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678038,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32060,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713092,"statfs":{"total":95995035648,"available":95962206208,"internally_reserved":0,"allocated":901120,"data_stored":450863,"data_compressed":17691,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874266,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":1031905,"data_compressed":50930,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037793,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":925696,"data_stored":450793,"data_compressed":17691,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234025,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32048,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713104,"statfs":{"total":95995035648,"available":95962218496,"internally_reserved":0,"allocated":892928,"data_stored":436731,"data_compressed":14704,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430253,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32088,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713064,"statfs":{"total":95995035648,"available":95962177536,"internally_reserved":0,"allocated":933888,"data_stored":1027565,"data_compressed":50930,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626482,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32044,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713108,"statfs":{"total":95995035648,"available":95962222592,"internally_reserved":0,"allocated":888832,"data_stored":436715,"data_compressed":14704,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574929,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36248,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708904,"statfs":{"total":95995035648,"available":95957917696,"internally_reserved":0,"allocated":929792,"data_stored":470468,"data_compressed":17691,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738453,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32068,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713084,"statfs":{"total":95995035648,"available":95962198016,"internally_reserved":0,"allocated":913408,"data_stored":438171,"data_compressed":14704,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411404,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":917504,"data_stored":442198,"data_compressed":14704,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182469,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":897024,"data_stored":436401,"data_compressed":14704,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986239,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27880,"kb_used_data":864,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717272,"statfs":{"total":95995035648,"available":95966486528,"internally_reserved":0,"allocated":884736,"data_stored":436637,"data_compressed":14704,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822713,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":1027837,"data_compressed":50930,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34599,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34599,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34599,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":39559,"data_compressed":1841,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":12805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":666,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":5544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-30T08:32:33.673 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-30T08:32:33.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:33 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1036212298' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:32:33.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.996+0000 7fd060e00640 1 -- 172.21.15.32:0/1685527043 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd05c105bb0 msgr2=0x7fd05c1068b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:33.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.996+0000 7fd060e00640 1 --2- 172.21.15.32:0/1685527043 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd05c105bb0 0x7fd05c1068b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fd04c009900 tx=0x7fd04c02a3f0 comp rx=0 tx=0).stop 2024-04-30T08:32:33.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.996+0000 7fd060e00640 1 -- 172.21.15.32:0/1685527043 shutdown_connections 2024-04-30T08:32:33.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.996+0000 7fd060e00640 1 --2- 172.21.15.32:0/1685527043 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd05c10baf0 0x7fd05c10df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:33.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.996+0000 7fd060e00640 1 --2- 172.21.15.32:0/1685527043 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd05c106df0 0x7fd05c107200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:33.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.996+0000 7fd060e00640 1 --2- 172.21.15.32:0/1685527043 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd05c105bb0 0x7fd05c1068b0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:33.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.996+0000 7fd060e00640 1 -- 172.21.15.32:0/1685527043 >> 172.21.15.32:0/1685527043 conn(0x7fd05c100da0 msgr2=0x7fd05c103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:33.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.996+0000 7fd060e00640 1 -- 172.21.15.32:0/1685527043 shutdown_connections 2024-04-30T08:32:33.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.996+0000 7fd060e00640 1 -- 172.21.15.32:0/1685527043 wait complete. 2024-04-30T08:32:34.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.997+0000 7fd060e00640 1 Processor -- start 2024-04-30T08:32:34.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.997+0000 7fd060e00640 1 -- start start 2024-04-30T08:32:34.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.997+0000 7fd060e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd05c105bb0 0x7fd05c19c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:34.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.997+0000 7fd060e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd05c106df0 0x7fd05c19c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:34.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.997+0000 7fd060e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd05c10baf0 0x7fd05c1a37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:34.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.997+0000 7fd060e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd05c111700 con 0x7fd05c106df0 2024-04-30T08:32:34.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.997+0000 7fd060e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd05c111580 con 0x7fd05c105bb0 2024-04-30T08:32:34.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.997+0000 7fd060e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd05c111880 con 0x7fd05c10baf0 2024-04-30T08:32:34.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.998+0000 7fd05b400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd05c105bb0 0x7fd05c19c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:34.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.998+0000 7fd05b400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd05c105bb0 0x7fd05c19c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:59686/0 (socket says 172.21.15.32:59686) 2024-04-30T08:32:34.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.998+0000 7fd05b400640 1 -- 172.21.15.32:0/3423107682 learned_addr learned my addr 172.21.15.32:0/3423107682 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:34.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.998+0000 7fd05aa00640 1 --2- 172.21.15.32:0/3423107682 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd05c106df0 0x7fd05c19c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:34.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.998+0000 7fd05be00640 1 --2- 172.21.15.32:0/3423107682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd05c10baf0 0x7fd05c1a37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:34.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.998+0000 7fd05b400640 1 -- 172.21.15.32:0/3423107682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd05c10baf0 msgr2=0x7fd05c1a37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:34.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.998+0000 7fd05b400640 1 --2- 172.21.15.32:0/3423107682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd05c10baf0 0x7fd05c1a37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:34.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.998+0000 7fd05b400640 1 -- 172.21.15.32:0/3423107682 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd05c106df0 msgr2=0x7fd05c19c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:34.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.998+0000 7fd05b400640 1 --2- 172.21.15.32:0/3423107682 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd05c106df0 0x7fd05c19c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:34.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.998+0000 7fd05b400640 1 -- 172.21.15.32:0/3423107682 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd05c1a3e10 con 0x7fd05c105bb0 2024-04-30T08:32:34.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.999+0000 7fd05b400640 1 --2- 172.21.15.32:0/3423107682 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd05c105bb0 0x7fd05c19c2d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd04c031440 tx=0x7fd04c003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:34.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.999+0000 7fd053e00640 1 -- 172.21.15.32:0/3423107682 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd04c03f070 con 0x7fd05c105bb0 2024-04-30T08:32:34.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.999+0000 7fd060e00640 1 -- 172.21.15.32:0/3423107682 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd05c1a40a0 con 0x7fd05c105bb0 2024-04-30T08:32:34.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.999+0000 7fd053e00640 1 -- 172.21.15.32:0/3423107682 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd04c031c50 con 0x7fd05c105bb0 2024-04-30T08:32:34.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.999+0000 7fd053e00640 1 -- 172.21.15.32:0/3423107682 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd04c035030 con 0x7fd05c105bb0 2024-04-30T08:32:34.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:33.999+0000 7fd060e00640 1 -- 172.21.15.32:0/3423107682 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd05c1a4590 con 0x7fd05c105bb0 2024-04-30T08:32:34.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.001+0000 7fd060e00640 1 -- 172.21.15.32:0/3423107682 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd05c10a900 con 0x7fd05c105bb0 2024-04-30T08:32:34.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.003+0000 7fd053e00640 1 -- 172.21.15.32:0/3423107682 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fd04c042070 con 0x7fd05c105bb0 2024-04-30T08:32:34.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.004+0000 7fd053e00640 1 --2- 172.21.15.32:0/3423107682 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd0400afb70 0x7fd0400b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:34.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.005+0000 7fd053e00640 1 -- 172.21.15.32:0/3423107682 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fd04c0ff8a0 con 0x7fd05c105bb0 2024-04-30T08:32:34.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.005+0000 7fd05aa00640 1 --2- 172.21.15.32:0/3423107682 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd0400afb70 0x7fd0400b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:34.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.005+0000 7fd05aa00640 1 --2- 172.21.15.32:0/3423107682 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd0400afb70 0x7fd0400b2080 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd05c19d370 tx=0x7fd044005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:34.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.008+0000 7fd053e00640 1 -- 172.21.15.32:0/3423107682 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd04c0c7910 con 0x7fd05c105bb0 2024-04-30T08:32:34.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.213+0000 7fd060e00640 1 -- 172.21.15.32:0/3423107682 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd05c063060 con 0x7fd0400afb70 2024-04-30T08:32:34.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.225+0000 7fd053e00640 1 -- 172.21.15.32:0/3423107682 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340581 (secure 0 0 0) 0x7fd05c063060 con 0x7fd0400afb70 2024-04-30T08:32:34.231 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:32:34.233 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-30T08:32:34.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.231+0000 7fd060e00640 1 -- 172.21.15.32:0/3423107682 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd0400afb70 msgr2=0x7fd0400b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:34.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.231+0000 7fd060e00640 1 --2- 172.21.15.32:0/3423107682 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd0400afb70 0x7fd0400b2080 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fd05c19d370 tx=0x7fd044005f90 comp rx=0 tx=0).stop 2024-04-30T08:32:34.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.231+0000 7fd060e00640 1 -- 172.21.15.32:0/3423107682 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd05c105bb0 msgr2=0x7fd05c19c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:34.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.231+0000 7fd060e00640 1 --2- 172.21.15.32:0/3423107682 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd05c105bb0 0x7fd05c19c2d0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fd04c031440 tx=0x7fd04c003c10 comp rx=0 tx=0).stop 2024-04-30T08:32:34.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.232+0000 7fd060e00640 1 -- 172.21.15.32:0/3423107682 shutdown_connections 2024-04-30T08:32:34.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.232+0000 7fd060e00640 1 --2- 172.21.15.32:0/3423107682 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd0400afb70 0x7fd0400b2080 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:34.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.232+0000 7fd060e00640 1 --2- 172.21.15.32:0/3423107682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd05c10baf0 0x7fd05c1a37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:34.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.232+0000 7fd060e00640 1 --2- 172.21.15.32:0/3423107682 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd05c106df0 0x7fd05c19c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:34.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.232+0000 7fd060e00640 1 --2- 172.21.15.32:0/3423107682 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd05c105bb0 0x7fd05c19c2d0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:34.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.232+0000 7fd060e00640 1 -- 172.21.15.32:0/3423107682 >> 172.21.15.32:0/3423107682 conn(0x7fd05c100da0 msgr2=0x7fd05c107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:34.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.232+0000 7fd060e00640 1 -- 172.21.15.32:0/3423107682 shutdown_connections 2024-04-30T08:32:34.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.232+0000 7fd060e00640 1 -- 172.21.15.32:0/3423107682 wait complete. 2024-04-30T08:32:34.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:33 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1036212298' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:32:34.260 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-04-30T08:32:33.714194+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600976,"num_objects":95,"num_object_clones":0,"num_object_copies":287,"num_objects_missing_on_primary":0,"num_objects_missing":8,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":0,"num_read":105,"num_read_kb":83,"num_write":323,"num_write_kb":2220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":61,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":268,"ondisk_log_size":268,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":368384,"kb_used_data":10884,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124573440,"statfs":{"total":1151940427776,"available":1151563202560,"internally_reserved":0,"allocated":11145216,"data_stored":7164336,"data_compressed":312009,"data_compressed_allocated":417792,"data_compressed_original":2334720,"omap_allocated":19141,"internal_metadata":365933883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5968,"num_objects":71,"num_object_clones":0,"num_object_copies":215,"num_objects_missing_on_primary":0,"num_objects_missing":8,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":151,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004136"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548872+0000","last_change":"2024-04-30T08:31:52.711443+0000","last_active":"2024-04-30T08:32:30.548872+0000","last_peered":"2024-04-30T08:32:30.548872+0000","last_clean":"2024-04-30T08:32:30.548872+0000","last_became_active":"2024-04-30T08:31:52.711029+0000","last_became_peered":"2024-04-30T08:31:52.711029+0000","last_unstale":"2024-04-30T08:32:30.548872+0000","last_undegraded":"2024-04-30T08:32:30.548872+0000","last_fullsized":"2024-04-30T08:32:30.548872+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:51:15.208528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834854+0000","last_change":"2024-04-30T08:31:52.712324+0000","last_active":"2024-04-30T08:31:59.834854+0000","last_peered":"2024-04-30T08:31:59.834854+0000","last_clean":"2024-04-30T08:31:59.834854+0000","last_became_active":"2024-04-30T08:31:52.711923+0000","last_became_peered":"2024-04-30T08:31:52.711923+0000","last_unstale":"2024-04-30T08:31:59.834854+0000","last_undegraded":"2024-04-30T08:31:59.834854+0000","last_fullsized":"2024-04-30T08:31:59.834854+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:29:55.071459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834894+0000","last_change":"2024-04-30T08:31:52.712936+0000","last_active":"2024-04-30T08:31:59.834894+0000","last_peered":"2024-04-30T08:31:59.834894+0000","last_clean":"2024-04-30T08:31:59.834894+0000","last_became_active":"2024-04-30T08:31:52.712585+0000","last_became_peered":"2024-04-30T08:31:52.712585+0000","last_unstale":"2024-04-30T08:31:59.834894+0000","last_undegraded":"2024-04-30T08:31:59.834894+0000","last_fullsized":"2024-04-30T08:31:59.834894+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:13:43.745561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550312+0000","last_change":"2024-04-30T08:31:52.686880+0000","last_active":"2024-04-30T08:32:31.550312+0000","last_peered":"2024-04-30T08:32:31.550312+0000","last_clean":"2024-04-30T08:32:31.550312+0000","last_became_active":"2024-04-30T08:31:52.686506+0000","last_became_peered":"2024-04-30T08:31:52.686506+0000","last_unstale":"2024-04-30T08:32:31.550312+0000","last_undegraded":"2024-04-30T08:32:31.550312+0000","last_fullsized":"2024-04-30T08:32:31.550312+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:16:33.390951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.357817+0000","last_change":"2024-04-30T08:31:52.706014+0000","last_active":"2024-04-30T08:31:59.357817+0000","last_peered":"2024-04-30T08:31:59.357817+0000","last_clean":"2024-04-30T08:31:59.357817+0000","last_became_active":"2024-04-30T08:31:52.705582+0000","last_became_peered":"2024-04-30T08:31:52.705582+0000","last_unstale":"2024-04-30T08:31:59.357817+0000","last_undegraded":"2024-04-30T08:31:59.357817+0000","last_fullsized":"2024-04-30T08:31:59.357817+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:24:49.804592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:58.709591+0000","last_change":"2024-04-30T08:31:52.690462+0000","last_active":"2024-04-30T08:31:58.709591+0000","last_peered":"2024-04-30T08:31:58.709591+0000","last_clean":"2024-04-30T08:31:58.709591+0000","last_became_active":"2024-04-30T08:31:52.690092+0000","last_became_peered":"2024-04-30T08:31:52.690092+0000","last_unstale":"2024-04-30T08:31:58.709591+0000","last_undegraded":"2024-04-30T08:31:58.709591+0000","last_fullsized":"2024-04-30T08:31:58.709591+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:32:49.769383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.557763+0000","last_change":"2024-04-30T08:31:52.701786+0000","last_active":"2024-04-30T08:32:31.557763+0000","last_peered":"2024-04-30T08:32:31.557763+0000","last_clean":"2024-04-30T08:32:31.557763+0000","last_became_active":"2024-04-30T08:31:52.701402+0000","last_became_peered":"2024-04-30T08:31:52.701402+0000","last_unstale":"2024-04-30T08:32:31.557763+0000","last_undegraded":"2024-04-30T08:32:31.557763+0000","last_fullsized":"2024-04-30T08:32:31.557763+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:58:06.153479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.669311+0000","last_change":"2024-04-30T08:31:52.698037+0000","last_active":"2024-04-30T08:31:59.669311+0000","last_peered":"2024-04-30T08:31:59.669311+0000","last_clean":"2024-04-30T08:31:59.669311+0000","last_became_active":"2024-04-30T08:31:52.697646+0000","last_became_peered":"2024-04-30T08:31:52.697646+0000","last_unstale":"2024-04-30T08:31:59.669311+0000","last_undegraded":"2024-04-30T08:31:59.669311+0000","last_fullsized":"2024-04-30T08:31:59.669311+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:28:59.520091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548406+0000","last_change":"2024-04-30T08:31:52.680377+0000","last_active":"2024-04-30T08:32:30.548406+0000","last_peered":"2024-04-30T08:32:30.548406+0000","last_clean":"2024-04-30T08:32:30.548406+0000","last_became_active":"2024-04-30T08:31:52.679977+0000","last_became_peered":"2024-04-30T08:31:52.679977+0000","last_unstale":"2024-04-30T08:32:30.548406+0000","last_undegraded":"2024-04-30T08:32:30.548406+0000","last_fullsized":"2024-04-30T08:32:30.548406+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:35:22.488450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:58.710009+0000","last_change":"2024-04-30T08:31:52.686488+0000","last_active":"2024-04-30T08:31:58.710009+0000","last_peered":"2024-04-30T08:31:58.710009+0000","last_clean":"2024-04-30T08:31:58.710009+0000","last_became_active":"2024-04-30T08:31:52.686057+0000","last_became_peered":"2024-04-30T08:31:52.686057+0000","last_unstale":"2024-04-30T08:31:58.710009+0000","last_undegraded":"2024-04-30T08:31:58.710009+0000","last_fullsized":"2024-04-30T08:31:58.710009+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:51:47.170678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.558068+0000","last_change":"2024-04-30T08:31:52.690344+0000","last_active":"2024-04-30T08:32:31.558068+0000","last_peered":"2024-04-30T08:32:31.558068+0000","last_clean":"2024-04-30T08:32:31.558068+0000","last_became_active":"2024-04-30T08:31:52.689857+0000","last_became_peered":"2024-04-30T08:31:52.689857+0000","last_unstale":"2024-04-30T08:32:31.558068+0000","last_undegraded":"2024-04-30T08:32:31.558068+0000","last_fullsized":"2024-04-30T08:32:31.558068+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:45:13.395589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550256+0000","last_change":"2024-04-30T08:31:52.685151+0000","last_active":"2024-04-30T08:32:31.550256+0000","last_peered":"2024-04-30T08:32:31.550256+0000","last_clean":"2024-04-30T08:32:31.550256+0000","last_became_active":"2024-04-30T08:31:52.684722+0000","last_became_peered":"2024-04-30T08:31:52.684722+0000","last_unstale":"2024-04-30T08:32:31.550256+0000","last_undegraded":"2024-04-30T08:32:31.550256+0000","last_fullsized":"2024-04-30T08:32:31.550256+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:14:00.660478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.553484+0000","last_change":"2024-04-30T08:31:52.714294+0000","last_active":"2024-04-30T08:32:30.553484+0000","last_peered":"2024-04-30T08:32:30.553484+0000","last_clean":"2024-04-30T08:32:30.553484+0000","last_became_active":"2024-04-30T08:31:52.710094+0000","last_became_peered":"2024-04-30T08:31:52.710094+0000","last_unstale":"2024-04-30T08:32:30.553484+0000","last_undegraded":"2024-04-30T08:32:30.553484+0000","last_fullsized":"2024-04-30T08:32:30.553484+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:34:44.203093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834299+0000","last_change":"2024-04-30T08:31:52.707992+0000","last_active":"2024-04-30T08:31:59.834299+0000","last_peered":"2024-04-30T08:31:59.834299+0000","last_clean":"2024-04-30T08:31:59.834299+0000","last_became_active":"2024-04-30T08:31:52.706456+0000","last_became_peered":"2024-04-30T08:31:52.706456+0000","last_unstale":"2024-04-30T08:31:59.834299+0000","last_undegraded":"2024-04-30T08:31:59.834299+0000","last_fullsized":"2024-04-30T08:31:59.834299+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:45:32.717842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834299+0000","last_change":"2024-04-30T08:31:52.695605+0000","last_active":"2024-04-30T08:31:59.834299+0000","last_peered":"2024-04-30T08:31:59.834299+0000","last_clean":"2024-04-30T08:31:59.834299+0000","last_became_active":"2024-04-30T08:31:52.692439+0000","last_became_peered":"2024-04-30T08:31:52.692439+0000","last_unstale":"2024-04-30T08:31:59.834299+0000","last_undegraded":"2024-04-30T08:31:59.834299+0000","last_fullsized":"2024-04-30T08:31:59.834299+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:39:38.746567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.547727+0000","last_change":"2024-04-30T08:31:52.707011+0000","last_active":"2024-04-30T08:32:30.547727+0000","last_peered":"2024-04-30T08:32:30.547727+0000","last_clean":"2024-04-30T08:32:30.547727+0000","last_became_active":"2024-04-30T08:31:52.706602+0000","last_became_peered":"2024-04-30T08:31:52.706602+0000","last_unstale":"2024-04-30T08:32:30.547727+0000","last_undegraded":"2024-04-30T08:32:30.547727+0000","last_fullsized":"2024-04-30T08:32:30.547727+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:43:13.922905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548723+0000","last_change":"2024-04-30T08:31:52.711439+0000","last_active":"2024-04-30T08:32:30.548723+0000","last_peered":"2024-04-30T08:32:30.548723+0000","last_clean":"2024-04-30T08:32:30.548723+0000","last_became_active":"2024-04-30T08:31:52.711023+0000","last_became_peered":"2024-04-30T08:31:52.711023+0000","last_unstale":"2024-04-30T08:32:30.548723+0000","last_undegraded":"2024-04-30T08:32:30.548723+0000","last_fullsized":"2024-04-30T08:32:30.548723+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:31:00.654348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.558525+0000","last_change":"2024-04-30T08:31:52.708476+0000","last_active":"2024-04-30T08:32:30.558525+0000","last_peered":"2024-04-30T08:32:30.558525+0000","last_clean":"2024-04-30T08:32:30.558525+0000","last_became_active":"2024-04-30T08:31:52.704735+0000","last_became_peered":"2024-04-30T08:31:52.704735+0000","last_unstale":"2024-04-30T08:32:30.558525+0000","last_undegraded":"2024-04-30T08:32:30.558525+0000","last_fullsized":"2024-04-30T08:32:30.558525+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:40:00.011783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834768+0000","last_change":"2024-04-30T08:31:52.704901+0000","last_active":"2024-04-30T08:31:59.834768+0000","last_peered":"2024-04-30T08:31:59.834768+0000","last_clean":"2024-04-30T08:31:59.834768+0000","last_became_active":"2024-04-30T08:31:52.704636+0000","last_became_peered":"2024-04-30T08:31:52.704636+0000","last_unstale":"2024-04-30T08:31:59.834768+0000","last_undegraded":"2024-04-30T08:31:59.834768+0000","last_fullsized":"2024-04-30T08:31:59.834768+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:08:53.580809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834299+0000","last_change":"2024-04-30T08:31:57.743342+0000","last_active":"2024-04-30T08:31:59.834299+0000","last_peered":"2024-04-30T08:31:59.834299+0000","last_clean":"2024-04-30T08:31:59.834299+0000","last_became_active":"2024-04-30T08:31:57.742850+0000","last_became_peered":"2024-04-30T08:31:57.742850+0000","last_unstale":"2024-04-30T08:31:59.834299+0000","last_undegraded":"2024-04-30T08:31:59.834299+0000","last_fullsized":"2024-04-30T08:31:59.834299+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:59:10.628822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"81'13","reported_seq":40,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.558153+0000","last_change":"2024-04-30T08:31:51.694984+0000","last_active":"2024-04-30T08:32:30.558153+0000","last_peered":"2024-04-30T08:32:30.558153+0000","last_clean":"2024-04-30T08:32:30.558153+0000","last_became_active":"2024-04-30T08:31:51.694514+0000","last_became_peered":"2024-04-30T08:31:51.694514+0000","last_unstale":"2024-04-30T08:32:30.558153+0000","last_undegraded":"2024-04-30T08:32:30.558153+0000","last_fullsized":"2024-04-30T08:32:30.558153+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:17:03.873090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.670928+0000","last_change":"2024-04-30T08:31:52.694276+0000","last_active":"2024-04-30T08:31:59.670928+0000","last_peered":"2024-04-30T08:31:59.670928+0000","last_clean":"2024-04-30T08:31:59.670928+0000","last_became_active":"2024-04-30T08:31:52.693839+0000","last_became_peered":"2024-04-30T08:31:52.693839+0000","last_unstale":"2024-04-30T08:31:59.670928+0000","last_undegraded":"2024-04-30T08:31:59.670928+0000","last_fullsized":"2024-04-30T08:31:59.670928+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:29:21.271040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.549942+0000","last_change":"2024-04-30T08:31:57.754828+0000","last_active":"2024-04-30T08:32:31.549942+0000","last_peered":"2024-04-30T08:32:31.549942+0000","last_clean":"2024-04-30T08:32:31.549942+0000","last_became_active":"2024-04-30T08:31:57.754322+0000","last_became_peered":"2024-04-30T08:31:57.754322+0000","last_unstale":"2024-04-30T08:32:31.549942+0000","last_undegraded":"2024-04-30T08:32:31.549942+0000","last_fullsized":"2024-04-30T08:32:31.549942+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:36:02.560134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"81'2","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.671373+0000","last_change":"2024-04-30T08:31:51.707637+0000","last_active":"2024-04-30T08:31:59.671373+0000","last_peered":"2024-04-30T08:31:59.671373+0000","last_clean":"2024-04-30T08:31:59.671373+0000","last_became_active":"2024-04-30T08:31:51.707233+0000","last_became_peered":"2024-04-30T08:31:51.707233+0000","last_unstale":"2024-04-30T08:31:59.671373+0000","last_undegraded":"2024-04-30T08:31:59.671373+0000","last_fullsized":"2024-04-30T08:31:59.671373+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:14:46.041169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.835049+0000","last_change":"2024-04-30T08:31:52.711442+0000","last_active":"2024-04-30T08:31:59.835049+0000","last_peered":"2024-04-30T08:31:59.835049+0000","last_clean":"2024-04-30T08:31:59.835049+0000","last_became_active":"2024-04-30T08:31:52.710942+0000","last_became_peered":"2024-04-30T08:31:52.710942+0000","last_unstale":"2024-04-30T08:31:59.835049+0000","last_undegraded":"2024-04-30T08:31:59.835049+0000","last_fullsized":"2024-04-30T08:31:59.835049+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:26:56.428770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550129+0000","last_change":"2024-04-30T08:31:57.760535+0000","last_active":"2024-04-30T08:32:31.550129+0000","last_peered":"2024-04-30T08:32:31.550129+0000","last_clean":"2024-04-30T08:32:31.550129+0000","last_became_active":"2024-04-30T08:31:57.760073+0000","last_became_peered":"2024-04-30T08:31:57.760073+0000","last_unstale":"2024-04-30T08:32:31.550129+0000","last_undegraded":"2024-04-30T08:32:31.550129+0000","last_fullsized":"2024-04-30T08:32:31.550129+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:03:51.359474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"81'7","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:32:28.528438+0000","last_change":"2024-04-30T08:31:51.678575+0000","last_active":"2024-04-30T08:32:28.528438+0000","last_peered":"2024-04-30T08:32:28.528438+0000","last_clean":"2024-04-30T08:32:28.528438+0000","last_became_active":"2024-04-30T08:31:51.678114+0000","last_became_peered":"2024-04-30T08:31:51.678114+0000","last_unstale":"2024-04-30T08:32:28.528438+0000","last_undegraded":"2024-04-30T08:32:28.528438+0000","last_fullsized":"2024-04-30T08:32:28.528438+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:10:35.517183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548286+0000","last_change":"2024-04-30T08:31:52.704137+0000","last_active":"2024-04-30T08:32:30.548286+0000","last_peered":"2024-04-30T08:32:30.548286+0000","last_clean":"2024-04-30T08:32:30.548286+0000","last_became_active":"2024-04-30T08:31:52.703695+0000","last_became_peered":"2024-04-30T08:31:52.703695+0000","last_unstale":"2024-04-30T08:32:30.548286+0000","last_undegraded":"2024-04-30T08:32:30.548286+0000","last_fullsized":"2024-04-30T08:32:30.548286+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:31:05.461106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.549998+0000","last_change":"2024-04-30T08:31:57.755237+0000","last_active":"2024-04-30T08:32:31.549998+0000","last_peered":"2024-04-30T08:32:31.549998+0000","last_clean":"2024-04-30T08:32:31.549998+0000","last_became_active":"2024-04-30T08:31:57.754438+0000","last_became_peered":"2024-04-30T08:31:57.754438+0000","last_unstale":"2024-04-30T08:32:31.549998+0000","last_undegraded":"2024-04-30T08:32:31.549998+0000","last_fullsized":"2024-04-30T08:32:31.549998+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:20:59.035110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"75'2","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.357899+0000","last_change":"2024-04-30T08:31:51.673793+0000","last_active":"2024-04-30T08:31:59.357899+0000","last_peered":"2024-04-30T08:31:59.357899+0000","last_clean":"2024-04-30T08:31:59.357899+0000","last_became_active":"2024-04-30T08:31:51.672703+0000","last_became_peered":"2024-04-30T08:31:51.672703+0000","last_unstale":"2024-04-30T08:31:59.357899+0000","last_undegraded":"2024-04-30T08:31:59.357899+0000","last_fullsized":"2024-04-30T08:31:59.357899+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:13:18.959542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.553233+0000","last_change":"2024-04-30T08:31:52.694320+0000","last_active":"2024-04-30T08:32:31.553233+0000","last_peered":"2024-04-30T08:32:31.553233+0000","last_clean":"2024-04-30T08:32:31.553233+0000","last_became_active":"2024-04-30T08:31:52.694032+0000","last_became_peered":"2024-04-30T08:31:52.694032+0000","last_unstale":"2024-04-30T08:32:31.553233+0000","last_undegraded":"2024-04-30T08:32:31.553233+0000","last_fullsized":"2024-04-30T08:32:31.553233+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:33:12.347369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834607+0000","last_change":"2024-04-30T08:31:57.739984+0000","last_active":"2024-04-30T08:31:59.834607+0000","last_peered":"2024-04-30T08:31:59.834607+0000","last_clean":"2024-04-30T08:31:59.834607+0000","last_became_active":"2024-04-30T08:31:57.739477+0000","last_became_peered":"2024-04-30T08:31:57.739477+0000","last_unstale":"2024-04-30T08:31:59.834607+0000","last_undegraded":"2024-04-30T08:31:59.834607+0000","last_fullsized":"2024-04-30T08:31:59.834607+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:10:11.325419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"81'6","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.564820+0000","last_change":"2024-04-30T08:31:51.678792+0000","last_active":"2024-04-30T08:32:31.564820+0000","last_peered":"2024-04-30T08:32:31.564820+0000","last_clean":"2024-04-30T08:32:31.564820+0000","last_became_active":"2024-04-30T08:31:51.678094+0000","last_became_peered":"2024-04-30T08:31:51.678094+0000","last_unstale":"2024-04-30T08:32:31.564820+0000","last_undegraded":"2024-04-30T08:32:31.564820+0000","last_fullsized":"2024-04-30T08:32:31.564820+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:03:17.526016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.357855+0000","last_change":"2024-04-30T08:31:52.701925+0000","last_active":"2024-04-30T08:31:59.357855+0000","last_peered":"2024-04-30T08:31:59.357855+0000","last_clean":"2024-04-30T08:31:59.357855+0000","last_became_active":"2024-04-30T08:31:52.701425+0000","last_became_peered":"2024-04-30T08:31:52.701425+0000","last_unstale":"2024-04-30T08:31:59.357855+0000","last_undegraded":"2024-04-30T08:31:59.357855+0000","last_fullsized":"2024-04-30T08:31:59.357855+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:20:35.767086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834905+0000","last_change":"2024-04-30T08:31:57.765047+0000","last_active":"2024-04-30T08:31:59.834905+0000","last_peered":"2024-04-30T08:31:59.834905+0000","last_clean":"2024-04-30T08:31:59.834905+0000","last_became_active":"2024-04-30T08:31:57.764519+0000","last_became_peered":"2024-04-30T08:31:57.764519+0000","last_unstale":"2024-04-30T08:31:59.834905+0000","last_undegraded":"2024-04-30T08:31:59.834905+0000","last_fullsized":"2024-04-30T08:31:59.834905+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:06:16.126396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"81'7","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:32:28.533731+0000","last_change":"2024-04-30T08:31:51.686403+0000","last_active":"2024-04-30T08:32:28.533731+0000","last_peered":"2024-04-30T08:32:28.533731+0000","last_clean":"2024-04-30T08:32:28.533731+0000","last_became_active":"2024-04-30T08:31:51.686096+0000","last_became_peered":"2024-04-30T08:31:51.686096+0000","last_unstale":"2024-04-30T08:32:28.533731+0000","last_undegraded":"2024-04-30T08:32:28.533731+0000","last_fullsized":"2024-04-30T08:32:28.533731+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:47:05.961930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.554940+0000","last_change":"2024-04-30T08:31:52.698512+0000","last_active":"2024-04-30T08:32:31.554940+0000","last_peered":"2024-04-30T08:32:31.554940+0000","last_clean":"2024-04-30T08:32:31.554940+0000","last_became_active":"2024-04-30T08:31:52.698168+0000","last_became_peered":"2024-04-30T08:31:52.698168+0000","last_unstale":"2024-04-30T08:32:31.554940+0000","last_undegraded":"2024-04-30T08:32:31.554940+0000","last_fullsized":"2024-04-30T08:32:31.554940+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:22:45.094365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548550+0000","last_change":"2024-04-30T08:31:57.761430+0000","last_active":"2024-04-30T08:32:30.548550+0000","last_peered":"2024-04-30T08:32:30.548550+0000","last_clean":"2024-04-30T08:32:30.548550+0000","last_became_active":"2024-04-30T08:31:57.761062+0000","last_became_peered":"2024-04-30T08:31:57.761062+0000","last_unstale":"2024-04-30T08:32:30.548550+0000","last_undegraded":"2024-04-30T08:32:30.548550+0000","last_fullsized":"2024-04-30T08:32:30.548550+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:32:05.403786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"81'3","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.552212+0000","last_change":"2024-04-30T08:31:51.686694+0000","last_active":"2024-04-30T08:32:30.552212+0000","last_peered":"2024-04-30T08:32:30.552212+0000","last_clean":"2024-04-30T08:32:30.552212+0000","last_became_active":"2024-04-30T08:31:51.685773+0000","last_became_peered":"2024-04-30T08:31:51.685773+0000","last_unstale":"2024-04-30T08:32:30.552212+0000","last_undegraded":"2024-04-30T08:32:30.552212+0000","last_fullsized":"2024-04-30T08:32:30.552212+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:44:25.327977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.669796+0000","last_change":"2024-04-30T08:31:52.709794+0000","last_active":"2024-04-30T08:31:59.669796+0000","last_peered":"2024-04-30T08:31:59.669796+0000","last_clean":"2024-04-30T08:31:59.669796+0000","last_became_active":"2024-04-30T08:31:52.709371+0000","last_became_peered":"2024-04-30T08:31:52.709371+0000","last_unstale":"2024-04-30T08:31:59.669796+0000","last_undegraded":"2024-04-30T08:31:59.669796+0000","last_fullsized":"2024-04-30T08:31:59.669796+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:36:59.892551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834943+0000","last_change":"2024-04-30T08:31:57.757224+0000","last_active":"2024-04-30T08:31:59.834943+0000","last_peered":"2024-04-30T08:31:59.834943+0000","last_clean":"2024-04-30T08:31:59.834943+0000","last_became_active":"2024-04-30T08:31:57.756658+0000","last_became_peered":"2024-04-30T08:31:57.756658+0000","last_unstale":"2024-04-30T08:31:59.834943+0000","last_undegraded":"2024-04-30T08:31:59.834943+0000","last_fullsized":"2024-04-30T08:31:59.834943+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:01:21.309322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"81'3","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.555826+0000","last_change":"2024-04-30T08:31:51.678944+0000","last_active":"2024-04-30T08:32:31.555826+0000","last_peered":"2024-04-30T08:32:31.555826+0000","last_clean":"2024-04-30T08:32:31.555826+0000","last_became_active":"2024-04-30T08:31:51.678493+0000","last_became_peered":"2024-04-30T08:31:51.678493+0000","last_unstale":"2024-04-30T08:32:31.555826+0000","last_undegraded":"2024-04-30T08:32:31.555826+0000","last_fullsized":"2024-04-30T08:32:31.555826+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:48:11.111466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.555177+0000","last_change":"2024-04-30T08:31:52.707627+0000","last_active":"2024-04-30T08:32:30.555177+0000","last_peered":"2024-04-30T08:32:30.555177+0000","last_clean":"2024-04-30T08:32:30.555177+0000","last_became_active":"2024-04-30T08:31:52.706441+0000","last_became_peered":"2024-04-30T08:31:52.706441+0000","last_unstale":"2024-04-30T08:32:30.555177+0000","last_undegraded":"2024-04-30T08:32:30.555177+0000","last_fullsized":"2024-04-30T08:32:30.555177+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:15:45.768927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.669809+0000","last_change":"2024-04-30T08:31:57.740324+0000","last_active":"2024-04-30T08:31:59.669809+0000","last_peered":"2024-04-30T08:31:59.669809+0000","last_clean":"2024-04-30T08:31:59.669809+0000","last_became_active":"2024-04-30T08:31:57.739464+0000","last_became_peered":"2024-04-30T08:31:57.739464+0000","last_unstale":"2024-04-30T08:31:59.669809+0000","last_undegraded":"2024-04-30T08:31:59.669809+0000","last_fullsized":"2024-04-30T08:31:59.669809+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:11:37.395545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"81'4","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:32:26.513947+0000","last_change":"2024-04-30T08:31:51.695283+0000","last_active":"2024-04-30T08:32:26.513947+0000","last_peered":"2024-04-30T08:32:26.513947+0000","last_clean":"2024-04-30T08:32:26.513947+0000","last_became_active":"2024-04-30T08:31:51.694848+0000","last_became_peered":"2024-04-30T08:31:51.694848+0000","last_unstale":"2024-04-30T08:32:26.513947+0000","last_undegraded":"2024-04-30T08:32:26.513947+0000","last_fullsized":"2024-04-30T08:32:26.513947+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:41:38.638034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548438+0000","last_change":"2024-04-30T08:31:52.693594+0000","last_active":"2024-04-30T08:32:30.548438+0000","last_peered":"2024-04-30T08:32:30.548438+0000","last_clean":"2024-04-30T08:32:30.548438+0000","last_became_active":"2024-04-30T08:31:52.693199+0000","last_became_peered":"2024-04-30T08:31:52.693199+0000","last_unstale":"2024-04-30T08:32:30.548438+0000","last_undegraded":"2024-04-30T08:32:30.548438+0000","last_fullsized":"2024-04-30T08:32:30.548438+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:57:49.908617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.553991+0000","last_change":"2024-04-30T08:31:57.749727+0000","last_active":"2024-04-30T08:32:30.553991+0000","last_peered":"2024-04-30T08:32:30.553991+0000","last_clean":"2024-04-30T08:32:30.553991+0000","last_became_active":"2024-04-30T08:31:57.749267+0000","last_became_peered":"2024-04-30T08:31:57.749267+0000","last_unstale":"2024-04-30T08:32:30.553991+0000","last_undegraded":"2024-04-30T08:32:30.553991+0000","last_fullsized":"2024-04-30T08:32:30.553991+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:26:12.683008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"81'4","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.557872+0000","last_change":"2024-04-30T08:31:51.677398+0000","last_active":"2024-04-30T08:32:31.557872+0000","last_peered":"2024-04-30T08:32:31.557872+0000","last_clean":"2024-04-30T08:32:31.557872+0000","last_became_active":"2024-04-30T08:31:51.677031+0000","last_became_peered":"2024-04-30T08:31:51.677031+0000","last_unstale":"2024-04-30T08:32:31.557872+0000","last_undegraded":"2024-04-30T08:32:31.557872+0000","last_fullsized":"2024-04-30T08:32:31.557872+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:24:25.154229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.835058+0000","last_change":"2024-04-30T08:31:52.717016+0000","last_active":"2024-04-30T08:31:59.835058+0000","last_peered":"2024-04-30T08:31:59.835058+0000","last_clean":"2024-04-30T08:31:59.835058+0000","last_became_active":"2024-04-30T08:31:52.714622+0000","last_became_peered":"2024-04-30T08:31:52.714622+0000","last_unstale":"2024-04-30T08:31:59.835058+0000","last_undegraded":"2024-04-30T08:31:59.835058+0000","last_fullsized":"2024-04-30T08:31:59.835058+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:56:44.550915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.586746+0000","last_change":"2024-04-30T08:32:31.586746+0000","last_active":"2024-04-30T08:32:31.586746+0000","last_peered":"2024-04-30T08:32:31.586746+0000","last_clean":"2024-04-30T08:32:31.586746+0000","last_became_active":"2024-04-30T08:32:31.586356+0000","last_became_peered":"2024-04-30T08:32:31.586356+0000","last_unstale":"2024-04-30T08:32:31.586746+0000","last_undegraded":"2024-04-30T08:32:31.586746+0000","last_fullsized":"2024-04-30T08:32:31.586746+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:22:39.371508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"81'5","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.553126+0000","last_change":"2024-04-30T08:31:51.694424+0000","last_active":"2024-04-30T08:32:31.553126+0000","last_peered":"2024-04-30T08:32:31.553126+0000","last_clean":"2024-04-30T08:32:31.553126+0000","last_became_active":"2024-04-30T08:31:51.694081+0000","last_became_peered":"2024-04-30T08:31:51.694081+0000","last_unstale":"2024-04-30T08:32:31.553126+0000","last_undegraded":"2024-04-30T08:32:31.553126+0000","last_fullsized":"2024-04-30T08:32:31.553126+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:27:33.328477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548544+0000","last_change":"2024-04-30T08:31:52.684739+0000","last_active":"2024-04-30T08:32:30.548544+0000","last_peered":"2024-04-30T08:32:30.548544+0000","last_clean":"2024-04-30T08:32:30.548544+0000","last_became_active":"2024-04-30T08:31:52.684316+0000","last_became_peered":"2024-04-30T08:31:52.684316+0000","last_unstale":"2024-04-30T08:32:30.548544+0000","last_undegraded":"2024-04-30T08:32:30.548544+0000","last_fullsized":"2024-04-30T08:32:30.548544+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:12:05.437493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.615550+0000","last_change":"2024-04-30T08:31:57.760643+0000","last_active":"2024-04-30T08:31:59.615550+0000","last_peered":"2024-04-30T08:31:59.615550+0000","last_clean":"2024-04-30T08:31:59.615550+0000","last_became_active":"2024-04-30T08:31:57.760053+0000","last_became_peered":"2024-04-30T08:31:57.760053+0000","last_unstale":"2024-04-30T08:31:59.615550+0000","last_undegraded":"2024-04-30T08:31:59.615550+0000","last_fullsized":"2024-04-30T08:31:59.615550+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:29:12.748550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"81'12","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:32:26.511027+0000","last_change":"2024-04-30T08:31:51.676223+0000","last_active":"2024-04-30T08:32:26.511027+0000","last_peered":"2024-04-30T08:32:26.511027+0000","last_clean":"2024-04-30T08:32:26.511027+0000","last_became_active":"2024-04-30T08:31:51.675830+0000","last_became_peered":"2024-04-30T08:31:51.675830+0000","last_unstale":"2024-04-30T08:32:26.511027+0000","last_undegraded":"2024-04-30T08:32:26.511027+0000","last_fullsized":"2024-04-30T08:32:26.511027+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:28:49.497719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.358155+0000","last_change":"2024-04-30T08:31:52.686892+0000","last_active":"2024-04-30T08:31:59.358155+0000","last_peered":"2024-04-30T08:31:59.358155+0000","last_clean":"2024-04-30T08:31:59.358155+0000","last_became_active":"2024-04-30T08:31:52.686533+0000","last_became_peered":"2024-04-30T08:31:52.686533+0000","last_unstale":"2024-04-30T08:31:59.358155+0000","last_undegraded":"2024-04-30T08:31:59.358155+0000","last_fullsized":"2024-04-30T08:31:59.358155+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:50:18.836161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.553606+0000","last_change":"2024-04-30T08:31:57.752865+0000","last_active":"2024-04-30T08:32:31.553606+0000","last_peered":"2024-04-30T08:32:31.553606+0000","last_clean":"2024-04-30T08:32:31.553606+0000","last_became_active":"2024-04-30T08:31:57.752385+0000","last_became_peered":"2024-04-30T08:31:57.752385+0000","last_unstale":"2024-04-30T08:32:31.553606+0000","last_undegraded":"2024-04-30T08:32:31.553606+0000","last_fullsized":"2024-04-30T08:32:31.553606+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:15:15.315580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"81'9","reported_seq":2,"reported_epoch":82,"state":"peering","last_fresh":"2024-04-30T08:32:29.552341+0000","last_change":"2024-04-30T08:32:30.944941+0000","last_active":"2024-04-30T08:32:30.944941+0000","last_peered":"2024-04-30T08:32:29.552341+0000","last_clean":"2024-04-30T08:32:29.552341+0000","last_became_active":"2024-04-30T08:31:51.702122+0000","last_became_peered":"2024-04-30T08:31:51.702122+0000","last_unstale":"2024-04-30T08:32:29.552341+0000","last_undegraded":"2024-04-30T08:32:29.552341+0000","last_fullsized":"2024-04-30T08:32:29.552341+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":6,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.553023+0000","last_change":"2024-04-30T08:31:52.706259+0000","last_active":"2024-04-30T08:32:30.553023+0000","last_peered":"2024-04-30T08:32:30.553023+0000","last_clean":"2024-04-30T08:32:30.553023+0000","last_became_active":"2024-04-30T08:31:52.705620+0000","last_became_peered":"2024-04-30T08:31:52.705620+0000","last_unstale":"2024-04-30T08:32:30.553023+0000","last_undegraded":"2024-04-30T08:32:30.553023+0000","last_fullsized":"2024-04-30T08:32:30.553023+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:22:29.816950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"peering","last_fresh":"2024-04-30T08:32:00.941417+0000","last_change":"2024-04-30T08:32:30.548960+0000","last_active":"2024-04-30T08:32:30.548960+0000","last_peered":"2024-04-30T08:32:00.941417+0000","last_clean":"2024-04-30T08:32:00.941417+0000","last_became_active":"2024-04-30T08:31:57.762531+0000","last_became_peered":"2024-04-30T08:31:57.762531+0000","last_unstale":"2024-04-30T08:32:00.941417+0000","last_undegraded":"2024-04-30T08:32:00.941417+0000","last_fullsized":"2024-04-30T08:32:00.941417+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7,8,9],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"81'5","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.554892+0000","last_change":"2024-04-30T08:31:51.701122+0000","last_active":"2024-04-30T08:32:31.554892+0000","last_peered":"2024-04-30T08:32:31.554892+0000","last_clean":"2024-04-30T08:32:31.554892+0000","last_became_active":"2024-04-30T08:31:51.700815+0000","last_became_peered":"2024-04-30T08:31:51.700815+0000","last_unstale":"2024-04-30T08:32:31.554892+0000","last_undegraded":"2024-04-30T08:32:31.554892+0000","last_fullsized":"2024-04-30T08:32:31.554892+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:33:12.714979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.558463+0000","last_change":"2024-04-30T08:31:52.716043+0000","last_active":"2024-04-30T08:32:31.558463+0000","last_peered":"2024-04-30T08:32:31.558463+0000","last_clean":"2024-04-30T08:32:31.558463+0000","last_became_active":"2024-04-30T08:31:52.715284+0000","last_became_peered":"2024-04-30T08:31:52.715284+0000","last_unstale":"2024-04-30T08:32:31.558463+0000","last_undegraded":"2024-04-30T08:32:31.558463+0000","last_fullsized":"2024-04-30T08:32:31.558463+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:00:01.863606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.557749+0000","last_change":"2024-04-30T08:31:57.765008+0000","last_active":"2024-04-30T08:32:30.557749+0000","last_peered":"2024-04-30T08:32:30.557749+0000","last_clean":"2024-04-30T08:32:30.557749+0000","last_became_active":"2024-04-30T08:31:57.762958+0000","last_became_peered":"2024-04-30T08:31:57.762958+0000","last_unstale":"2024-04-30T08:32:30.557749+0000","last_undegraded":"2024-04-30T08:32:30.557749+0000","last_fullsized":"2024-04-30T08:32:30.557749+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:48:47.116004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"81'6","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:32.572977+0000","last_change":"2024-04-30T08:31:51.676681+0000","last_active":"2024-04-30T08:32:32.572977+0000","last_peered":"2024-04-30T08:32:32.572977+0000","last_clean":"2024-04-30T08:32:32.572977+0000","last_became_active":"2024-04-30T08:31:51.676311+0000","last_became_peered":"2024-04-30T08:31:51.676311+0000","last_unstale":"2024-04-30T08:32:32.572977+0000","last_undegraded":"2024-04-30T08:32:32.572977+0000","last_fullsized":"2024-04-30T08:32:32.572977+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:24:49.757427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.558213+0000","last_change":"2024-04-30T08:31:52.689418+0000","last_active":"2024-04-30T08:32:31.558213+0000","last_peered":"2024-04-30T08:32:31.558213+0000","last_clean":"2024-04-30T08:32:31.558213+0000","last_became_active":"2024-04-30T08:31:52.688840+0000","last_became_peered":"2024-04-30T08:31:52.688840+0000","last_unstale":"2024-04-30T08:32:31.558213+0000","last_undegraded":"2024-04-30T08:32:31.558213+0000","last_fullsized":"2024-04-30T08:32:31.558213+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:40:05.781137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.554105+0000","last_change":"2024-04-30T08:31:57.765875+0000","last_active":"2024-04-30T08:32:30.554105+0000","last_peered":"2024-04-30T08:32:30.554105+0000","last_clean":"2024-04-30T08:32:30.554105+0000","last_became_active":"2024-04-30T08:31:57.765432+0000","last_became_peered":"2024-04-30T08:31:57.765432+0000","last_unstale":"2024-04-30T08:32:30.554105+0000","last_undegraded":"2024-04-30T08:32:30.554105+0000","last_fullsized":"2024-04-30T08:32:30.554105+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:24:44.329532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"81'16","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:32:27.524097+0000","last_change":"2024-04-30T08:31:51.695007+0000","last_active":"2024-04-30T08:32:27.524097+0000","last_peered":"2024-04-30T08:32:27.524097+0000","last_clean":"2024-04-30T08:32:27.524097+0000","last_became_active":"2024-04-30T08:31:51.694533+0000","last_became_peered":"2024-04-30T08:31:51.694533+0000","last_unstale":"2024-04-30T08:32:27.524097+0000","last_undegraded":"2024-04-30T08:32:27.524097+0000","last_fullsized":"2024-04-30T08:32:27.524097+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:47:15.104172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.553520+0000","last_change":"2024-04-30T08:31:52.692274+0000","last_active":"2024-04-30T08:32:31.553520+0000","last_peered":"2024-04-30T08:32:31.553520+0000","last_clean":"2024-04-30T08:32:31.553520+0000","last_became_active":"2024-04-30T08:31:52.691809+0000","last_became_peered":"2024-04-30T08:31:52.691809+0000","last_unstale":"2024-04-30T08:32:31.553520+0000","last_undegraded":"2024-04-30T08:32:31.553520+0000","last_fullsized":"2024-04-30T08:32:31.553520+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:50:00.201782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.555077+0000","last_change":"2024-04-30T08:31:57.732383+0000","last_active":"2024-04-30T08:32:31.555077+0000","last_peered":"2024-04-30T08:32:31.555077+0000","last_clean":"2024-04-30T08:32:31.555077+0000","last_became_active":"2024-04-30T08:31:57.731893+0000","last_became_peered":"2024-04-30T08:31:57.731893+0000","last_unstale":"2024-04-30T08:32:31.555077+0000","last_undegraded":"2024-04-30T08:32:31.555077+0000","last_fullsized":"2024-04-30T08:32:31.555077+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:08:55.243526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"81'2","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.358322+0000","last_change":"2024-04-30T08:31:51.671991+0000","last_active":"2024-04-30T08:31:59.358322+0000","last_peered":"2024-04-30T08:31:59.358322+0000","last_clean":"2024-04-30T08:31:59.358322+0000","last_became_active":"2024-04-30T08:31:51.671508+0000","last_became_peered":"2024-04-30T08:31:51.671508+0000","last_unstale":"2024-04-30T08:31:59.358322+0000","last_undegraded":"2024-04-30T08:31:59.358322+0000","last_fullsized":"2024-04-30T08:31:59.358322+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:54:30.106748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548288+0000","last_change":"2024-04-30T08:31:52.687555+0000","last_active":"2024-04-30T08:32:30.548288+0000","last_peered":"2024-04-30T08:32:30.548288+0000","last_clean":"2024-04-30T08:32:30.548288+0000","last_became_active":"2024-04-30T08:31:52.686000+0000","last_became_peered":"2024-04-30T08:31:52.686000+0000","last_unstale":"2024-04-30T08:32:30.548288+0000","last_undegraded":"2024-04-30T08:32:30.548288+0000","last_fullsized":"2024-04-30T08:32:30.548288+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:37:03.529774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.357960+0000","last_change":"2024-04-30T08:31:58.061077+0000","last_active":"2024-04-30T08:31:59.357960+0000","last_peered":"2024-04-30T08:31:59.357960+0000","last_clean":"2024-04-30T08:31:59.357960+0000","last_became_active":"2024-04-30T08:31:58.060486+0000","last_became_peered":"2024-04-30T08:31:58.060486+0000","last_unstale":"2024-04-30T08:31:59.357960+0000","last_undegraded":"2024-04-30T08:31:59.357960+0000","last_fullsized":"2024-04-30T08:31:59.357960+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:31:50.936560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"81'1","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.557573+0000","last_change":"2024-04-30T08:31:51.702869+0000","last_active":"2024-04-30T08:32:30.557573+0000","last_peered":"2024-04-30T08:32:30.557573+0000","last_clean":"2024-04-30T08:32:30.557573+0000","last_became_active":"2024-04-30T08:31:51.701150+0000","last_became_peered":"2024-04-30T08:31:51.701150+0000","last_unstale":"2024-04-30T08:32:30.557573+0000","last_undegraded":"2024-04-30T08:32:30.557573+0000","last_fullsized":"2024-04-30T08:32:30.557573+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:47:11.010310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:58.709852+0000","last_change":"2024-04-30T08:31:52.709159+0000","last_active":"2024-04-30T08:31:58.709852+0000","last_peered":"2024-04-30T08:31:58.709852+0000","last_clean":"2024-04-30T08:31:58.709852+0000","last_became_active":"2024-04-30T08:31:52.708799+0000","last_became_peered":"2024-04-30T08:31:52.708799+0000","last_unstale":"2024-04-30T08:31:58.709852+0000","last_undegraded":"2024-04-30T08:31:58.709852+0000","last_fullsized":"2024-04-30T08:31:58.709852+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:51:10.588599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.554141+0000","last_change":"2024-04-30T08:31:57.751652+0000","last_active":"2024-04-30T08:32:30.554141+0000","last_peered":"2024-04-30T08:32:30.554141+0000","last_clean":"2024-04-30T08:32:30.554141+0000","last_became_active":"2024-04-30T08:31:57.751176+0000","last_became_peered":"2024-04-30T08:31:57.751176+0000","last_unstale":"2024-04-30T08:32:30.554141+0000","last_undegraded":"2024-04-30T08:32:30.554141+0000","last_fullsized":"2024-04-30T08:32:30.554141+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:13:24.528842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"81'5","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.571485+0000","last_change":"2024-04-30T08:31:51.700816+0000","last_active":"2024-04-30T08:32:31.571485+0000","last_peered":"2024-04-30T08:32:31.571485+0000","last_clean":"2024-04-30T08:32:31.571485+0000","last_became_active":"2024-04-30T08:31:51.700470+0000","last_became_peered":"2024-04-30T08:31:51.700470+0000","last_unstale":"2024-04-30T08:32:31.571485+0000","last_undegraded":"2024-04-30T08:32:31.571485+0000","last_fullsized":"2024-04-30T08:32:31.571485+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:49:11.793412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:58.709513+0000","last_change":"2024-04-30T08:31:52.715236+0000","last_active":"2024-04-30T08:31:58.709513+0000","last_peered":"2024-04-30T08:31:58.709513+0000","last_clean":"2024-04-30T08:31:58.709513+0000","last_became_active":"2024-04-30T08:31:52.714801+0000","last_became_peered":"2024-04-30T08:31:52.714801+0000","last_unstale":"2024-04-30T08:31:58.709513+0000","last_undegraded":"2024-04-30T08:31:58.709513+0000","last_fullsized":"2024-04-30T08:31:58.709513+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:43:09.650002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.552783+0000","last_change":"2024-04-30T08:31:58.060927+0000","last_active":"2024-04-30T08:32:30.552783+0000","last_peered":"2024-04-30T08:32:30.552783+0000","last_clean":"2024-04-30T08:32:30.552783+0000","last_became_active":"2024-04-30T08:31:58.060182+0000","last_became_peered":"2024-04-30T08:31:58.060182+0000","last_unstale":"2024-04-30T08:32:30.552783+0000","last_undegraded":"2024-04-30T08:32:30.552783+0000","last_fullsized":"2024-04-30T08:32:30.552783+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:21:00.228064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.560485+0000","last_change":"2024-04-30T08:32:31.560485+0000","last_active":"2024-04-30T08:32:31.560485+0000","last_peered":"2024-04-30T08:32:31.560485+0000","last_clean":"2024-04-30T08:32:31.560485+0000","last_became_active":"2024-04-30T08:32:31.560066+0000","last_became_peered":"2024-04-30T08:32:31.560066+0000","last_unstale":"2024-04-30T08:32:31.560485+0000","last_undegraded":"2024-04-30T08:32:31.560485+0000","last_fullsized":"2024-04-30T08:32:31.560485+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:13:40.769761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"81'5","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.557878+0000","last_change":"2024-04-30T08:31:51.702808+0000","last_active":"2024-04-30T08:32:30.557878+0000","last_peered":"2024-04-30T08:32:30.557878+0000","last_clean":"2024-04-30T08:32:30.557878+0000","last_became_active":"2024-04-30T08:31:51.702326+0000","last_became_peered":"2024-04-30T08:31:51.702326+0000","last_unstale":"2024-04-30T08:32:30.557878+0000","last_undegraded":"2024-04-30T08:32:30.557878+0000","last_fullsized":"2024-04-30T08:32:30.557878+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:11:35.552670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"81'1","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.557866+0000","last_change":"2024-04-30T08:31:57.760359+0000","last_active":"2024-04-30T08:32:31.557866+0000","last_peered":"2024-04-30T08:32:31.557866+0000","last_clean":"2024-04-30T08:32:31.557866+0000","last_became_active":"2024-04-30T08:31:57.759821+0000","last_became_peered":"2024-04-30T08:31:57.759821+0000","last_unstale":"2024-04-30T08:32:31.557866+0000","last_undegraded":"2024-04-30T08:32:31.557866+0000","last_fullsized":"2024-04-30T08:32:31.557866+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:50:17.853791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.669652+0000","last_change":"2024-04-30T08:31:52.689281+0000","last_active":"2024-04-30T08:31:59.669652+0000","last_peered":"2024-04-30T08:31:59.669652+0000","last_clean":"2024-04-30T08:31:59.669652+0000","last_became_active":"2024-04-30T08:31:52.688834+0000","last_became_peered":"2024-04-30T08:31:52.688834+0000","last_unstale":"2024-04-30T08:31:59.669652+0000","last_undegraded":"2024-04-30T08:31:59.669652+0000","last_fullsized":"2024-04-30T08:31:59.669652+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:42:12.836649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'6","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550187+0000","last_change":"2024-04-30T08:31:51.683712+0000","last_active":"2024-04-30T08:32:31.550187+0000","last_peered":"2024-04-30T08:32:31.550187+0000","last_clean":"2024-04-30T08:32:31.550187+0000","last_became_active":"2024-04-30T08:31:51.683298+0000","last_became_peered":"2024-04-30T08:31:51.683298+0000","last_unstale":"2024-04-30T08:32:31.550187+0000","last_undegraded":"2024-04-30T08:32:31.550187+0000","last_fullsized":"2024-04-30T08:32:31.550187+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:43:08.832104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":103,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.557511+0000","last_change":"2024-04-30T08:30:26.503221+0000","last_active":"2024-04-30T08:32:31.557511+0000","last_peered":"2024-04-30T08:32:31.557511+0000","last_clean":"2024-04-30T08:32:31.557511+0000","last_became_active":"2024-04-30T08:30:26.498399+0000","last_became_peered":"2024-04-30T08:30:26.498399+0000","last_unstale":"2024-04-30T08:32:31.557511+0000","last_undegraded":"2024-04-30T08:32:31.557511+0000","last_fullsized":"2024-04-30T08:32:31.557511+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:26:20.170075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:26:20.170075+0000","last_clean_scrub_stamp":"2024-04-30T08:26:20.170075+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:34:44.067560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.552597+0000","last_change":"2024-04-30T08:31:58.061283+0000","last_active":"2024-04-30T08:32:31.552597+0000","last_peered":"2024-04-30T08:32:31.552597+0000","last_clean":"2024-04-30T08:32:31.552597+0000","last_became_active":"2024-04-30T08:31:58.060650+0000","last_became_peered":"2024-04-30T08:31:58.060650+0000","last_unstale":"2024-04-30T08:32:31.552597+0000","last_undegraded":"2024-04-30T08:32:31.552597+0000","last_fullsized":"2024-04-30T08:32:31.552597+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:13:01.811462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.551989+0000","last_change":"2024-04-30T08:31:52.699639+0000","last_active":"2024-04-30T08:32:30.551989+0000","last_peered":"2024-04-30T08:32:30.551989+0000","last_clean":"2024-04-30T08:32:30.551989+0000","last_became_active":"2024-04-30T08:31:52.699164+0000","last_became_peered":"2024-04-30T08:31:52.699164+0000","last_unstale":"2024-04-30T08:32:30.551989+0000","last_undegraded":"2024-04-30T08:32:30.551989+0000","last_fullsized":"2024-04-30T08:32:30.551989+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:22:18.873504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"81'4","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.558080+0000","last_change":"2024-04-30T08:31:51.679153+0000","last_active":"2024-04-30T08:32:31.558080+0000","last_peered":"2024-04-30T08:32:31.558080+0000","last_clean":"2024-04-30T08:32:31.558080+0000","last_became_active":"2024-04-30T08:31:51.678348+0000","last_became_peered":"2024-04-30T08:31:51.678348+0000","last_unstale":"2024-04-30T08:32:31.558080+0000","last_undegraded":"2024-04-30T08:32:31.558080+0000","last_fullsized":"2024-04-30T08:32:31.558080+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:02:56.240218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548074+0000","last_change":"2024-04-30T08:31:57.755423+0000","last_active":"2024-04-30T08:32:30.548074+0000","last_peered":"2024-04-30T08:32:30.548074+0000","last_clean":"2024-04-30T08:32:30.548074+0000","last_became_active":"2024-04-30T08:31:57.754835+0000","last_became_peered":"2024-04-30T08:31:57.754835+0000","last_unstale":"2024-04-30T08:32:30.548074+0000","last_undegraded":"2024-04-30T08:32:30.548074+0000","last_fullsized":"2024-04-30T08:32:30.548074+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:19:25.168920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"81'3","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:32:28.529739+0000","last_change":"2024-04-30T08:31:51.693839+0000","last_active":"2024-04-30T08:32:28.529739+0000","last_peered":"2024-04-30T08:32:28.529739+0000","last_clean":"2024-04-30T08:32:28.529739+0000","last_became_active":"2024-04-30T08:31:51.693349+0000","last_became_peered":"2024-04-30T08:31:51.693349+0000","last_unstale":"2024-04-30T08:32:28.529739+0000","last_undegraded":"2024-04-30T08:32:28.529739+0000","last_fullsized":"2024-04-30T08:32:28.529739+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:41:51.330901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.554333+0000","last_change":"2024-04-30T08:31:52.712360+0000","last_active":"2024-04-30T08:32:30.554333+0000","last_peered":"2024-04-30T08:32:30.554333+0000","last_clean":"2024-04-30T08:32:30.554333+0000","last_became_active":"2024-04-30T08:31:52.711652+0000","last_became_peered":"2024-04-30T08:31:52.711652+0000","last_unstale":"2024-04-30T08:32:30.554333+0000","last_undegraded":"2024-04-30T08:32:30.554333+0000","last_fullsized":"2024-04-30T08:32:30.554333+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:21:10.285386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.669947+0000","last_change":"2024-04-30T08:31:58.061654+0000","last_active":"2024-04-30T08:31:59.669947+0000","last_peered":"2024-04-30T08:31:59.669947+0000","last_clean":"2024-04-30T08:31:59.669947+0000","last_became_active":"2024-04-30T08:31:58.060181+0000","last_became_peered":"2024-04-30T08:31:58.060181+0000","last_unstale":"2024-04-30T08:31:59.669947+0000","last_undegraded":"2024-04-30T08:31:59.669947+0000","last_fullsized":"2024-04-30T08:31:59.669947+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:46:03.230626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"81'8","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.558636+0000","last_change":"2024-04-30T08:31:51.706940+0000","last_active":"2024-04-30T08:32:30.558636+0000","last_peered":"2024-04-30T08:32:30.558636+0000","last_clean":"2024-04-30T08:32:30.558636+0000","last_became_active":"2024-04-30T08:31:51.706481+0000","last_became_peered":"2024-04-30T08:31:51.706481+0000","last_unstale":"2024-04-30T08:32:30.558636+0000","last_undegraded":"2024-04-30T08:32:30.558636+0000","last_fullsized":"2024-04-30T08:32:30.558636+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:53:31.955595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.670853+0000","last_change":"2024-04-30T08:31:52.696625+0000","last_active":"2024-04-30T08:31:59.670853+0000","last_peered":"2024-04-30T08:31:59.670853+0000","last_clean":"2024-04-30T08:31:59.670853+0000","last_became_active":"2024-04-30T08:31:52.696230+0000","last_became_peered":"2024-04-30T08:31:52.696230+0000","last_unstale":"2024-04-30T08:31:59.670853+0000","last_undegraded":"2024-04-30T08:31:59.670853+0000","last_fullsized":"2024-04-30T08:31:59.670853+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:11:46.404251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834193+0000","last_change":"2024-04-30T08:31:57.738329+0000","last_active":"2024-04-30T08:31:59.834193+0000","last_peered":"2024-04-30T08:31:59.834193+0000","last_clean":"2024-04-30T08:31:59.834193+0000","last_became_active":"2024-04-30T08:31:57.737897+0000","last_became_peered":"2024-04-30T08:31:57.737897+0000","last_unstale":"2024-04-30T08:31:59.834193+0000","last_undegraded":"2024-04-30T08:31:59.834193+0000","last_fullsized":"2024-04-30T08:31:59.834193+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:02:43.074578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"81'7","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550515+0000","last_change":"2024-04-30T08:31:51.684066+0000","last_active":"2024-04-30T08:32:31.550515+0000","last_peered":"2024-04-30T08:32:31.550515+0000","last_clean":"2024-04-30T08:32:31.550515+0000","last_became_active":"2024-04-30T08:31:51.683699+0000","last_became_peered":"2024-04-30T08:31:51.683699+0000","last_unstale":"2024-04-30T08:32:31.550515+0000","last_undegraded":"2024-04-30T08:32:31.550515+0000","last_fullsized":"2024-04-30T08:32:31.550515+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:36:20.900733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.835032+0000","last_change":"2024-04-30T08:31:52.715219+0000","last_active":"2024-04-30T08:31:59.835032+0000","last_peered":"2024-04-30T08:31:59.835032+0000","last_clean":"2024-04-30T08:31:59.835032+0000","last_became_active":"2024-04-30T08:31:52.714774+0000","last_became_peered":"2024-04-30T08:31:52.714774+0000","last_unstale":"2024-04-30T08:31:59.835032+0000","last_undegraded":"2024-04-30T08:31:59.835032+0000","last_fullsized":"2024-04-30T08:31:59.835032+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:42:41.082610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550306+0000","last_change":"2024-04-30T08:31:58.060845+0000","last_active":"2024-04-30T08:32:31.550306+0000","last_peered":"2024-04-30T08:32:31.550306+0000","last_clean":"2024-04-30T08:32:31.550306+0000","last_became_active":"2024-04-30T08:31:58.060365+0000","last_became_peered":"2024-04-30T08:31:58.060365+0000","last_unstale":"2024-04-30T08:32:31.550306+0000","last_undegraded":"2024-04-30T08:32:31.550306+0000","last_fullsized":"2024-04-30T08:32:31.550306+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:21:39.583243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"81'2","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.547845+0000","last_change":"2024-04-30T08:31:51.694833+0000","last_active":"2024-04-30T08:32:30.547845+0000","last_peered":"2024-04-30T08:32:30.547845+0000","last_clean":"2024-04-30T08:32:30.547845+0000","last_became_active":"2024-04-30T08:31:51.694401+0000","last_became_peered":"2024-04-30T08:31:51.694401+0000","last_unstale":"2024-04-30T08:32:30.547845+0000","last_undegraded":"2024-04-30T08:32:30.547845+0000","last_fullsized":"2024-04-30T08:32:30.547845+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:38:52.605625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.357703+0000","last_change":"2024-04-30T08:31:52.686444+0000","last_active":"2024-04-30T08:31:59.357703+0000","last_peered":"2024-04-30T08:31:59.357703+0000","last_clean":"2024-04-30T08:31:59.357703+0000","last_became_active":"2024-04-30T08:31:52.686057+0000","last_became_peered":"2024-04-30T08:31:52.686057+0000","last_unstale":"2024-04-30T08:31:59.357703+0000","last_undegraded":"2024-04-30T08:31:59.357703+0000","last_fullsized":"2024-04-30T08:31:59.357703+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:54:29.824060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.557873+0000","last_change":"2024-04-30T08:31:57.766193+0000","last_active":"2024-04-30T08:32:30.557873+0000","last_peered":"2024-04-30T08:32:30.557873+0000","last_clean":"2024-04-30T08:32:30.557873+0000","last_became_active":"2024-04-30T08:31:57.765758+0000","last_became_peered":"2024-04-30T08:31:57.765758+0000","last_unstale":"2024-04-30T08:32:30.557873+0000","last_undegraded":"2024-04-30T08:32:30.557873+0000","last_fullsized":"2024-04-30T08:32:30.557873+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:59:57.282795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"81'7","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:32:29.554238+0000","last_change":"2024-04-30T08:31:51.704565+0000","last_active":"2024-04-30T08:32:29.554238+0000","last_peered":"2024-04-30T08:32:29.554238+0000","last_clean":"2024-04-30T08:32:29.554238+0000","last_became_active":"2024-04-30T08:31:51.703954+0000","last_became_peered":"2024-04-30T08:31:51.703954+0000","last_unstale":"2024-04-30T08:32:29.554238+0000","last_undegraded":"2024-04-30T08:32:29.554238+0000","last_fullsized":"2024-04-30T08:32:29.554238+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:03:00.517867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548041+0000","last_change":"2024-04-30T08:31:52.706043+0000","last_active":"2024-04-30T08:32:30.548041+0000","last_peered":"2024-04-30T08:32:30.548041+0000","last_clean":"2024-04-30T08:32:30.548041+0000","last_became_active":"2024-04-30T08:31:52.705678+0000","last_became_peered":"2024-04-30T08:31:52.705678+0000","last_unstale":"2024-04-30T08:32:30.548041+0000","last_undegraded":"2024-04-30T08:32:30.548041+0000","last_fullsized":"2024-04-30T08:32:30.548041+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:44:56.787835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"82'1","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550147+0000","last_change":"2024-04-30T08:31:57.764791+0000","last_active":"2024-04-30T08:32:31.550147+0000","last_peered":"2024-04-30T08:32:31.550147+0000","last_clean":"2024-04-30T08:32:31.550147+0000","last_became_active":"2024-04-30T08:31:57.764368+0000","last_became_peered":"2024-04-30T08:31:57.764368+0000","last_unstale":"2024-04-30T08:32:31.550147+0000","last_undegraded":"2024-04-30T08:32:31.550147+0000","last_fullsized":"2024-04-30T08:32:31.550147+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:19:28.545483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"81'3","reported_seq":2,"reported_epoch":82,"state":"peering","last_fresh":"2024-04-30T08:32:30.524198+0000","last_change":"2024-04-30T08:32:30.944913+0000","last_active":"2024-04-30T08:32:30.944913+0000","last_peered":"2024-04-30T08:32:30.524198+0000","last_clean":"2024-04-30T08:32:30.524198+0000","last_became_active":"2024-04-30T08:31:51.702141+0000","last_became_peered":"2024-04-30T08:31:51.702141+0000","last_unstale":"2024-04-30T08:32:30.524198+0000","last_undegraded":"2024-04-30T08:32:30.524198+0000","last_fullsized":"2024-04-30T08:32:30.524198+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.670593+0000","last_change":"2024-04-30T08:31:52.716707+0000","last_active":"2024-04-30T08:31:59.670593+0000","last_peered":"2024-04-30T08:31:59.670593+0000","last_clean":"2024-04-30T08:31:59.670593+0000","last_became_active":"2024-04-30T08:31:52.716293+0000","last_became_peered":"2024-04-30T08:31:52.716293+0000","last_unstale":"2024-04-30T08:31:59.670593+0000","last_undegraded":"2024-04-30T08:31:59.670593+0000","last_fullsized":"2024-04-30T08:31:59.670593+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:29:54.223472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550379+0000","last_change":"2024-04-30T08:31:58.060640+0000","last_active":"2024-04-30T08:32:31.550379+0000","last_peered":"2024-04-30T08:32:31.550379+0000","last_clean":"2024-04-30T08:32:31.550379+0000","last_became_active":"2024-04-30T08:31:58.060061+0000","last_became_peered":"2024-04-30T08:31:58.060061+0000","last_unstale":"2024-04-30T08:32:31.550379+0000","last_undegraded":"2024-04-30T08:32:31.550379+0000","last_fullsized":"2024-04-30T08:32:31.550379+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:46:00.844191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"81'4","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:32:26.512050+0000","last_change":"2024-04-30T08:31:51.679532+0000","last_active":"2024-04-30T08:32:26.512050+0000","last_peered":"2024-04-30T08:32:26.512050+0000","last_clean":"2024-04-30T08:32:26.512050+0000","last_became_active":"2024-04-30T08:31:51.679141+0000","last_became_peered":"2024-04-30T08:31:51.679141+0000","last_unstale":"2024-04-30T08:32:26.512050+0000","last_undegraded":"2024-04-30T08:32:26.512050+0000","last_fullsized":"2024-04-30T08:32:26.512050+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:19:50.526563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.557596+0000","last_change":"2024-04-30T08:31:52.703613+0000","last_active":"2024-04-30T08:32:31.557596+0000","last_peered":"2024-04-30T08:32:31.557596+0000","last_clean":"2024-04-30T08:32:31.557596+0000","last_became_active":"2024-04-30T08:31:52.703196+0000","last_became_peered":"2024-04-30T08:31:52.703196+0000","last_unstale":"2024-04-30T08:32:31.557596+0000","last_undegraded":"2024-04-30T08:32:31.557596+0000","last_fullsized":"2024-04-30T08:32:31.557596+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:33:11.136313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.547878+0000","last_change":"2024-04-30T08:31:57.755286+0000","last_active":"2024-04-30T08:32:30.547878+0000","last_peered":"2024-04-30T08:32:30.547878+0000","last_clean":"2024-04-30T08:32:30.547878+0000","last_became_active":"2024-04-30T08:31:57.754786+0000","last_became_peered":"2024-04-30T08:31:57.754786+0000","last_unstale":"2024-04-30T08:32:30.547878+0000","last_undegraded":"2024-04-30T08:32:30.547878+0000","last_fullsized":"2024-04-30T08:32:30.547878+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:02:50.931553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"81'5","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:32:28.529719+0000","last_change":"2024-04-30T08:31:51.693317+0000","last_active":"2024-04-30T08:32:28.529719+0000","last_peered":"2024-04-30T08:32:28.529719+0000","last_clean":"2024-04-30T08:32:28.529719+0000","last_became_active":"2024-04-30T08:31:51.692928+0000","last_became_peered":"2024-04-30T08:31:51.692928+0000","last_unstale":"2024-04-30T08:32:28.529719+0000","last_undegraded":"2024-04-30T08:32:28.529719+0000","last_fullsized":"2024-04-30T08:32:28.529719+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:35:09.851735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.558242+0000","last_change":"2024-04-30T08:31:52.694304+0000","last_active":"2024-04-30T08:32:31.558242+0000","last_peered":"2024-04-30T08:32:31.558242+0000","last_clean":"2024-04-30T08:32:31.558242+0000","last_became_active":"2024-04-30T08:31:52.693317+0000","last_became_peered":"2024-04-30T08:31:52.693317+0000","last_unstale":"2024-04-30T08:32:31.558242+0000","last_undegraded":"2024-04-30T08:32:31.558242+0000","last_fullsized":"2024-04-30T08:32:31.558242+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:31:39.294864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834714+0000","last_change":"2024-04-30T08:31:57.763598+0000","last_active":"2024-04-30T08:31:59.834714+0000","last_peered":"2024-04-30T08:31:59.834714+0000","last_clean":"2024-04-30T08:31:59.834714+0000","last_became_active":"2024-04-30T08:31:57.762307+0000","last_became_peered":"2024-04-30T08:31:57.762307+0000","last_unstale":"2024-04-30T08:31:59.834714+0000","last_undegraded":"2024-04-30T08:31:59.834714+0000","last_fullsized":"2024-04-30T08:31:59.834714+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:24:13.390595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"81'3","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.553337+0000","last_change":"2024-04-30T08:31:51.703476+0000","last_active":"2024-04-30T08:32:31.553337+0000","last_peered":"2024-04-30T08:32:31.553337+0000","last_clean":"2024-04-30T08:32:31.553337+0000","last_became_active":"2024-04-30T08:31:51.703072+0000","last_became_peered":"2024-04-30T08:31:51.703072+0000","last_unstale":"2024-04-30T08:32:31.553337+0000","last_undegraded":"2024-04-30T08:32:31.553337+0000","last_fullsized":"2024-04-30T08:32:31.553337+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:07:21.098223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.557803+0000","last_change":"2024-04-30T08:31:52.699223+0000","last_active":"2024-04-30T08:32:31.557803+0000","last_peered":"2024-04-30T08:32:31.557803+0000","last_clean":"2024-04-30T08:32:31.557803+0000","last_became_active":"2024-04-30T08:31:52.698837+0000","last_became_peered":"2024-04-30T08:31:52.698837+0000","last_unstale":"2024-04-30T08:32:31.557803+0000","last_undegraded":"2024-04-30T08:32:31.557803+0000","last_fullsized":"2024-04-30T08:32:31.557803+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:19:11.826470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.835310+0000","last_change":"2024-04-30T08:31:57.737438+0000","last_active":"2024-04-30T08:31:59.835310+0000","last_peered":"2024-04-30T08:31:59.835310+0000","last_clean":"2024-04-30T08:31:59.835310+0000","last_became_active":"2024-04-30T08:31:57.736949+0000","last_became_peered":"2024-04-30T08:31:57.736949+0000","last_unstale":"2024-04-30T08:31:59.835310+0000","last_undegraded":"2024-04-30T08:31:59.835310+0000","last_fullsized":"2024-04-30T08:31:59.835310+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:00:10.921846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"81'5","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.554486+0000","last_change":"2024-04-30T08:31:51.708943+0000","last_active":"2024-04-30T08:32:30.554486+0000","last_peered":"2024-04-30T08:32:30.554486+0000","last_clean":"2024-04-30T08:32:30.554486+0000","last_became_active":"2024-04-30T08:31:51.708376+0000","last_became_peered":"2024-04-30T08:31:51.708376+0000","last_unstale":"2024-04-30T08:32:30.554486+0000","last_undegraded":"2024-04-30T08:32:30.554486+0000","last_fullsized":"2024-04-30T08:32:30.554486+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:14:46.692192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.558028+0000","last_change":"2024-04-30T08:31:52.686634+0000","last_active":"2024-04-30T08:32:31.558028+0000","last_peered":"2024-04-30T08:32:31.558028+0000","last_clean":"2024-04-30T08:32:31.558028+0000","last_became_active":"2024-04-30T08:31:52.686231+0000","last_became_peered":"2024-04-30T08:31:52.686231+0000","last_unstale":"2024-04-30T08:32:31.558028+0000","last_undegraded":"2024-04-30T08:32:31.558028+0000","last_fullsized":"2024-04-30T08:32:31.558028+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:27:31.073041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548133+0000","last_change":"2024-04-30T08:31:52.688695+0000","last_active":"2024-04-30T08:32:30.548133+0000","last_peered":"2024-04-30T08:32:30.548133+0000","last_clean":"2024-04-30T08:32:30.548133+0000","last_became_active":"2024-04-30T08:31:52.686556+0000","last_became_peered":"2024-04-30T08:31:52.686556+0000","last_unstale":"2024-04-30T08:32:30.548133+0000","last_undegraded":"2024-04-30T08:32:30.548133+0000","last_fullsized":"2024-04-30T08:32:30.548133+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:22:48.976440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.835365+0000","last_change":"2024-04-30T08:31:52.688324+0000","last_active":"2024-04-30T08:31:59.835365+0000","last_peered":"2024-04-30T08:31:59.835365+0000","last_clean":"2024-04-30T08:31:59.835365+0000","last_became_active":"2024-04-30T08:31:52.687966+0000","last_became_peered":"2024-04-30T08:31:52.687966+0000","last_unstale":"2024-04-30T08:31:59.835365+0000","last_undegraded":"2024-04-30T08:31:59.835365+0000","last_fullsized":"2024-04-30T08:31:59.835365+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:35:38.237750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834334+0000","last_change":"2024-04-30T08:31:52.695638+0000","last_active":"2024-04-30T08:31:59.834334+0000","last_peered":"2024-04-30T08:31:59.834334+0000","last_clean":"2024-04-30T08:31:59.834334+0000","last_became_active":"2024-04-30T08:31:52.695283+0000","last_became_peered":"2024-04-30T08:31:52.695283+0000","last_unstale":"2024-04-30T08:31:59.834334+0000","last_undegraded":"2024-04-30T08:31:59.834334+0000","last_fullsized":"2024-04-30T08:31:59.834334+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:00:14.510381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834598+0000","last_change":"2024-04-30T08:31:52.719112+0000","last_active":"2024-04-30T08:31:59.834598+0000","last_peered":"2024-04-30T08:31:59.834598+0000","last_clean":"2024-04-30T08:31:59.834598+0000","last_became_active":"2024-04-30T08:31:52.718207+0000","last_became_peered":"2024-04-30T08:31:52.718207+0000","last_unstale":"2024-04-30T08:31:59.834598+0000","last_undegraded":"2024-04-30T08:31:59.834598+0000","last_fullsized":"2024-04-30T08:31:59.834598+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:27:29.263962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.553965+0000","last_change":"2024-04-30T08:31:52.712008+0000","last_active":"2024-04-30T08:32:30.553965+0000","last_peered":"2024-04-30T08:32:30.553965+0000","last_clean":"2024-04-30T08:32:30.553965+0000","last_became_active":"2024-04-30T08:31:52.711064+0000","last_became_peered":"2024-04-30T08:31:52.711064+0000","last_unstale":"2024-04-30T08:32:30.553965+0000","last_undegraded":"2024-04-30T08:32:30.553965+0000","last_fullsized":"2024-04-30T08:32:30.553965+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:57:27.400602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548539+0000","last_change":"2024-04-30T08:31:52.691210+0000","last_active":"2024-04-30T08:32:30.548539+0000","last_peered":"2024-04-30T08:32:30.548539+0000","last_clean":"2024-04-30T08:32:30.548539+0000","last_became_active":"2024-04-30T08:31:52.688852+0000","last_became_peered":"2024-04-30T08:31:52.688852+0000","last_unstale":"2024-04-30T08:32:30.548539+0000","last_undegraded":"2024-04-30T08:32:30.548539+0000","last_fullsized":"2024-04-30T08:32:30.548539+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:04:46.502743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.835158+0000","last_change":"2024-04-30T08:31:52.684501+0000","last_active":"2024-04-30T08:31:59.835158+0000","last_peered":"2024-04-30T08:31:59.835158+0000","last_clean":"2024-04-30T08:31:59.835158+0000","last_became_active":"2024-04-30T08:31:52.684122+0000","last_became_peered":"2024-04-30T08:31:52.684122+0000","last_unstale":"2024-04-30T08:31:59.835158+0000","last_undegraded":"2024-04-30T08:31:59.835158+0000","last_fullsized":"2024-04-30T08:31:59.835158+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:55:58.093269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.671093+0000","last_change":"2024-04-30T08:31:52.698016+0000","last_active":"2024-04-30T08:31:59.671093+0000","last_peered":"2024-04-30T08:31:59.671093+0000","last_clean":"2024-04-30T08:31:59.671093+0000","last_became_active":"2024-04-30T08:31:52.695037+0000","last_became_peered":"2024-04-30T08:31:52.695037+0000","last_unstale":"2024-04-30T08:31:59.671093+0000","last_undegraded":"2024-04-30T08:31:59.671093+0000","last_fullsized":"2024-04-30T08:31:59.671093+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:06:01.360947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.555895+0000","last_change":"2024-04-30T08:31:52.698415+0000","last_active":"2024-04-30T08:32:31.555895+0000","last_peered":"2024-04-30T08:32:31.555895+0000","last_clean":"2024-04-30T08:32:31.555895+0000","last_became_active":"2024-04-30T08:31:52.697912+0000","last_became_peered":"2024-04-30T08:31:52.697912+0000","last_unstale":"2024-04-30T08:32:31.555895+0000","last_undegraded":"2024-04-30T08:32:31.555895+0000","last_fullsized":"2024-04-30T08:32:31.555895+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:05:44.420331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.670020+0000","last_change":"2024-04-30T08:31:52.694776+0000","last_active":"2024-04-30T08:31:59.670020+0000","last_peered":"2024-04-30T08:31:59.670020+0000","last_clean":"2024-04-30T08:31:59.670020+0000","last_became_active":"2024-04-30T08:31:52.693532+0000","last_became_peered":"2024-04-30T08:31:52.693532+0000","last_unstale":"2024-04-30T08:31:59.670020+0000","last_undegraded":"2024-04-30T08:31:59.670020+0000","last_fullsized":"2024-04-30T08:31:59.670020+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:31:10.186667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548026+0000","last_change":"2024-04-30T08:31:52.688477+0000","last_active":"2024-04-30T08:32:30.548026+0000","last_peered":"2024-04-30T08:32:30.548026+0000","last_clean":"2024-04-30T08:32:30.548026+0000","last_became_active":"2024-04-30T08:31:52.688072+0000","last_became_peered":"2024-04-30T08:31:52.688072+0000","last_unstale":"2024-04-30T08:32:30.548026+0000","last_undegraded":"2024-04-30T08:32:30.548026+0000","last_fullsized":"2024-04-30T08:32:30.548026+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:35:42.659707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.553593+0000","last_change":"2024-04-30T08:31:52.716788+0000","last_active":"2024-04-30T08:32:30.553593+0000","last_peered":"2024-04-30T08:32:30.553593+0000","last_clean":"2024-04-30T08:32:30.553593+0000","last_became_active":"2024-04-30T08:31:52.716336+0000","last_became_peered":"2024-04-30T08:31:52.716336+0000","last_unstale":"2024-04-30T08:32:30.553593+0000","last_undegraded":"2024-04-30T08:32:30.553593+0000","last_fullsized":"2024-04-30T08:32:30.553593+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:23:23.477968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548294+0000","last_change":"2024-04-30T08:31:52.684571+0000","last_active":"2024-04-30T08:32:30.548294+0000","last_peered":"2024-04-30T08:32:30.548294+0000","last_clean":"2024-04-30T08:32:30.548294+0000","last_became_active":"2024-04-30T08:31:52.684192+0000","last_became_peered":"2024-04-30T08:31:52.684192+0000","last_unstale":"2024-04-30T08:32:30.548294+0000","last_undegraded":"2024-04-30T08:32:30.548294+0000","last_fullsized":"2024-04-30T08:32:30.548294+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:05:57.520133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284927,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10374,"num_objects":91,"num_object_clones":0,"num_object_copies":273,"num_objects_missing_on_primary":0,"num_objects_missing":8,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":198,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":61,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":72516,"data_compressed":1841,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},"log_size":174,"ondisk_log_size":174,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678038,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32060,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713092,"statfs":{"total":95995035648,"available":95962206208,"internally_reserved":0,"allocated":901120,"data_stored":450863,"data_compressed":17691,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874266,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":1031905,"data_compressed":50930,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037793,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":925696,"data_stored":450793,"data_compressed":17691,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234025,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32048,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713104,"statfs":{"total":95995035648,"available":95962218496,"internally_reserved":0,"allocated":892928,"data_stored":436731,"data_compressed":14704,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430254,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":987136,"data_stored":1057399,"data_compressed":56904,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626482,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32044,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713108,"statfs":{"total":95995035648,"available":95962222592,"internally_reserved":0,"allocated":888832,"data_stored":436715,"data_compressed":14704,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574929,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36248,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708904,"statfs":{"total":95995035648,"available":95957917696,"internally_reserved":0,"allocated":929792,"data_stored":470468,"data_compressed":17691,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738453,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32068,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713084,"statfs":{"total":95995035648,"available":95962198016,"internally_reserved":0,"allocated":913408,"data_stored":438171,"data_compressed":14704,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411404,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":917504,"data_stored":442198,"data_compressed":14704,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182469,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":897024,"data_stored":436401,"data_compressed":14704,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986240,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":921600,"data_stored":460102,"data_compressed":20678,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822714,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":995328,"data_stored":1052590,"data_compressed":56904,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34599,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34599,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34599,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":39559,"data_compressed":1841,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":12805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":5544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-30T08:32:34.265 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-30T08:32:34.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:33 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1036212298' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:32:34.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.587+0000 7fa703e00640 1 -- 172.21.15.32:0/4124248762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7041087c0 msgr2=0x7fa70410abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:34.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.587+0000 7fa703e00640 1 --2- 172.21.15.32:0/4124248762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7041087c0 0x7fa70410abe0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc009860 tx=0x7fa6fc02a340 comp rx=0 tx=0).stop 2024-04-30T08:32:34.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.587+0000 7fa703e00640 1 -- 172.21.15.32:0/4124248762 shutdown_connections 2024-04-30T08:32:34.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.587+0000 7fa703e00640 1 --2- 172.21.15.32:0/4124248762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa7041087c0 0x7fa70410abe0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:34.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.587+0000 7fa703e00640 1 --2- 172.21.15.32:0/4124248762 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa704103ac0 0x7fa704103ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:34.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.587+0000 7fa703e00640 1 --2- 172.21.15.32:0/4124248762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa704102880 0x7fa704103580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:34.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.587+0000 7fa703e00640 1 -- 172.21.15.32:0/4124248762 >> 172.21.15.32:0/4124248762 conn(0x7fa7040fda70 msgr2=0x7fa7040fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:34.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.587+0000 7fa703e00640 1 -- 172.21.15.32:0/4124248762 shutdown_connections 2024-04-30T08:32:34.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.587+0000 7fa703e00640 1 -- 172.21.15.32:0/4124248762 wait complete. 2024-04-30T08:32:34.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.588+0000 7fa703e00640 1 Processor -- start 2024-04-30T08:32:34.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.588+0000 7fa703e00640 1 -- start start 2024-04-30T08:32:34.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.588+0000 7fa703e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa704102880 0x7fa7040ff900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:34.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.588+0000 7fa703e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa704103ac0 0x7fa7040fdfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:34.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.588+0000 7fa703e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7041087c0 0x7fa7040fe4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:34.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.588+0000 7fa703e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa70410e390 con 0x7fa7041087c0 2024-04-30T08:32:34.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.588+0000 7fa703e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa70410e210 con 0x7fa704103ac0 2024-04-30T08:32:34.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.588+0000 7fa703e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa70410e510 con 0x7fa704102880 2024-04-30T08:32:34.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.589+0000 7fa703400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7041087c0 0x7fa7040fe4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:34.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.589+0000 7fa702000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa704103ac0 0x7fa7040fdfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:34.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.589+0000 7fa702000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa704103ac0 0x7fa7040fdfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:59702/0 (socket says 172.21.15.32:59702) 2024-04-30T08:32:34.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.589+0000 7fa702000640 1 -- 172.21.15.32:0/669624851 learned_addr learned my addr 172.21.15.32:0/669624851 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:34.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.589+0000 7fa702a00640 1 --2- 172.21.15.32:0/669624851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa704102880 0x7fa7040ff900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:34.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.589+0000 7fa702000640 1 -- 172.21.15.32:0/669624851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa704102880 msgr2=0x7fa7040ff900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:34.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.589+0000 7fa702000640 1 --2- 172.21.15.32:0/669624851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa704102880 0x7fa7040ff900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:34.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.589+0000 7fa702000640 1 -- 172.21.15.32:0/669624851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7041087c0 msgr2=0x7fa7040fe4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:34.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.589+0000 7fa702000640 1 --2- 172.21.15.32:0/669624851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7041087c0 0x7fa7040fe4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:34.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.589+0000 7fa702000640 1 -- 172.21.15.32:0/669624851 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7040fec10 con 0x7fa704103ac0 2024-04-30T08:32:34.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.590+0000 7fa702000640 1 --2- 172.21.15.32:0/669624851 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa704103ac0 0x7fa7040fdfa0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec00d950 tx=0x7fa6ec00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:34.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.590+0000 7fa6fb400640 1 -- 172.21.15.32:0/669624851 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec017030 con 0x7fa704103ac0 2024-04-30T08:32:34.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.590+0000 7fa703e00640 1 -- 172.21.15.32:0/669624851 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7041a41b0 con 0x7fa704103ac0 2024-04-30T08:32:34.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.590+0000 7fa6fb400640 1 -- 172.21.15.32:0/669624851 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6ec004020 con 0x7fa704103ac0 2024-04-30T08:32:34.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.590+0000 7fa6fb400640 1 -- 172.21.15.32:0/669624851 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6ec0047a0 con 0x7fa704103ac0 2024-04-30T08:32:34.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.590+0000 7fa703e00640 1 -- 172.21.15.32:0/669624851 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7041a4650 con 0x7fa704103ac0 2024-04-30T08:32:34.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.591+0000 7fa703e00640 1 -- 172.21.15.32:0/669624851 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7041a4910 con 0x7fa704103ac0 2024-04-30T08:32:34.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.594+0000 7fa6fb400640 1 -- 172.21.15.32:0/669624851 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fa6ec009a30 con 0x7fa704103ac0 2024-04-30T08:32:34.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.595+0000 7fa6fb400640 1 --2- 172.21.15.32:0/669624851 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa6e80afb70 0x7fa6e80b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:34.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.595+0000 7fa6fb400640 1 -- 172.21.15.32:0/669624851 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fa6ec0df120 con 0x7fa704103ac0 2024-04-30T08:32:34.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.595+0000 7fa702a00640 1 --2- 172.21.15.32:0/669624851 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa6e80afb70 0x7fa6e80b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:34.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.596+0000 7fa702a00640 1 --2- 172.21.15.32:0/669624851 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa6e80afb70 0x7fa6e80b2080 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa704103480 tx=0x7fa6f4010040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:34.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.597+0000 7fa6fb400640 1 -- 172.21.15.32:0/669624851 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa6ec0a77e0 con 0x7fa704103ac0 2024-04-30T08:32:34.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.803+0000 7fa703e00640 1 -- 172.21.15.32:0/669624851 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa70410a3d0 con 0x7fa6e80afb70 2024-04-30T08:32:34.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.831+0000 7fa6fb400640 1 -- 172.21.15.32:0/669624851 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340581 (secure 0 0 0) 0x7fa70410a3d0 con 0x7fa6e80afb70 2024-04-30T08:32:34.837 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:32:34.839 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-30T08:32:34.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.838+0000 7fa703e00640 1 -- 172.21.15.32:0/669624851 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa6e80afb70 msgr2=0x7fa6e80b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:34.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.838+0000 7fa703e00640 1 --2- 172.21.15.32:0/669624851 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa6e80afb70 0x7fa6e80b2080 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa704103480 tx=0x7fa6f4010040 comp rx=0 tx=0).stop 2024-04-30T08:32:34.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.838+0000 7fa703e00640 1 -- 172.21.15.32:0/669624851 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa704103ac0 msgr2=0x7fa7040fdfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:34.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.838+0000 7fa703e00640 1 --2- 172.21.15.32:0/669624851 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa704103ac0 0x7fa7040fdfa0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fa6ec00d950 tx=0x7fa6ec00de10 comp rx=0 tx=0).stop 2024-04-30T08:32:34.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.838+0000 7fa703e00640 1 -- 172.21.15.32:0/669624851 shutdown_connections 2024-04-30T08:32:34.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.838+0000 7fa703e00640 1 --2- 172.21.15.32:0/669624851 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa6e80afb70 0x7fa6e80b2080 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:34.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.838+0000 7fa703e00640 1 --2- 172.21.15.32:0/669624851 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa7041087c0 0x7fa7040fe4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:34.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.838+0000 7fa703e00640 1 --2- 172.21.15.32:0/669624851 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa704103ac0 0x7fa7040fdfa0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:34.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.838+0000 7fa703e00640 1 --2- 172.21.15.32:0/669624851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa704102880 0x7fa7040ff900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:34.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.838+0000 7fa703e00640 1 -- 172.21.15.32:0/669624851 >> 172.21.15.32:0/669624851 conn(0x7fa7040fda70 msgr2=0x7fa704109cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:34.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.838+0000 7fa703e00640 1 -- 172.21.15.32:0/669624851 shutdown_connections 2024-04-30T08:32:34.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:34.838+0000 7fa703e00640 1 -- 172.21.15.32:0/669624851 wait complete. 2024-04-30T08:32:34.872 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":345,"stamp":"2024-04-30T08:32:33.714194+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":600976,"num_objects":95,"num_object_clones":0,"num_object_copies":287,"num_objects_missing_on_primary":0,"num_objects_missing":8,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":95,"num_whiteouts":0,"num_read":105,"num_read_kb":83,"num_write":323,"num_write_kb":2220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":61,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":268,"ondisk_log_size":268,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":368384,"kb_used_data":10884,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124573440,"statfs":{"total":1151940427776,"available":1151563202560,"internally_reserved":0,"allocated":11145216,"data_stored":7164336,"data_compressed":312009,"data_compressed_allocated":417792,"data_compressed_original":2334720,"omap_allocated":19141,"internal_metadata":365933883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5968,"num_objects":71,"num_object_clones":0,"num_object_copies":215,"num_objects_missing_on_primary":0,"num_objects_missing":8,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":71,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":151,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":47,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004136"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548872+0000","last_change":"2024-04-30T08:31:52.711443+0000","last_active":"2024-04-30T08:32:30.548872+0000","last_peered":"2024-04-30T08:32:30.548872+0000","last_clean":"2024-04-30T08:32:30.548872+0000","last_became_active":"2024-04-30T08:31:52.711029+0000","last_became_peered":"2024-04-30T08:31:52.711029+0000","last_unstale":"2024-04-30T08:32:30.548872+0000","last_undegraded":"2024-04-30T08:32:30.548872+0000","last_fullsized":"2024-04-30T08:32:30.548872+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:51:15.208528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834854+0000","last_change":"2024-04-30T08:31:52.712324+0000","last_active":"2024-04-30T08:31:59.834854+0000","last_peered":"2024-04-30T08:31:59.834854+0000","last_clean":"2024-04-30T08:31:59.834854+0000","last_became_active":"2024-04-30T08:31:52.711923+0000","last_became_peered":"2024-04-30T08:31:52.711923+0000","last_unstale":"2024-04-30T08:31:59.834854+0000","last_undegraded":"2024-04-30T08:31:59.834854+0000","last_fullsized":"2024-04-30T08:31:59.834854+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:29:55.071459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834894+0000","last_change":"2024-04-30T08:31:52.712936+0000","last_active":"2024-04-30T08:31:59.834894+0000","last_peered":"2024-04-30T08:31:59.834894+0000","last_clean":"2024-04-30T08:31:59.834894+0000","last_became_active":"2024-04-30T08:31:52.712585+0000","last_became_peered":"2024-04-30T08:31:52.712585+0000","last_unstale":"2024-04-30T08:31:59.834894+0000","last_undegraded":"2024-04-30T08:31:59.834894+0000","last_fullsized":"2024-04-30T08:31:59.834894+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:13:43.745561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550312+0000","last_change":"2024-04-30T08:31:52.686880+0000","last_active":"2024-04-30T08:32:31.550312+0000","last_peered":"2024-04-30T08:32:31.550312+0000","last_clean":"2024-04-30T08:32:31.550312+0000","last_became_active":"2024-04-30T08:31:52.686506+0000","last_became_peered":"2024-04-30T08:31:52.686506+0000","last_unstale":"2024-04-30T08:32:31.550312+0000","last_undegraded":"2024-04-30T08:32:31.550312+0000","last_fullsized":"2024-04-30T08:32:31.550312+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:16:33.390951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.357817+0000","last_change":"2024-04-30T08:31:52.706014+0000","last_active":"2024-04-30T08:31:59.357817+0000","last_peered":"2024-04-30T08:31:59.357817+0000","last_clean":"2024-04-30T08:31:59.357817+0000","last_became_active":"2024-04-30T08:31:52.705582+0000","last_became_peered":"2024-04-30T08:31:52.705582+0000","last_unstale":"2024-04-30T08:31:59.357817+0000","last_undegraded":"2024-04-30T08:31:59.357817+0000","last_fullsized":"2024-04-30T08:31:59.357817+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:24:49.804592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:58.709591+0000","last_change":"2024-04-30T08:31:52.690462+0000","last_active":"2024-04-30T08:31:58.709591+0000","last_peered":"2024-04-30T08:31:58.709591+0000","last_clean":"2024-04-30T08:31:58.709591+0000","last_became_active":"2024-04-30T08:31:52.690092+0000","last_became_peered":"2024-04-30T08:31:52.690092+0000","last_unstale":"2024-04-30T08:31:58.709591+0000","last_undegraded":"2024-04-30T08:31:58.709591+0000","last_fullsized":"2024-04-30T08:31:58.709591+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:32:49.769383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.557763+0000","last_change":"2024-04-30T08:31:52.701786+0000","last_active":"2024-04-30T08:32:31.557763+0000","last_peered":"2024-04-30T08:32:31.557763+0000","last_clean":"2024-04-30T08:32:31.557763+0000","last_became_active":"2024-04-30T08:31:52.701402+0000","last_became_peered":"2024-04-30T08:31:52.701402+0000","last_unstale":"2024-04-30T08:32:31.557763+0000","last_undegraded":"2024-04-30T08:32:31.557763+0000","last_fullsized":"2024-04-30T08:32:31.557763+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:58:06.153479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.669311+0000","last_change":"2024-04-30T08:31:52.698037+0000","last_active":"2024-04-30T08:31:59.669311+0000","last_peered":"2024-04-30T08:31:59.669311+0000","last_clean":"2024-04-30T08:31:59.669311+0000","last_became_active":"2024-04-30T08:31:52.697646+0000","last_became_peered":"2024-04-30T08:31:52.697646+0000","last_unstale":"2024-04-30T08:31:59.669311+0000","last_undegraded":"2024-04-30T08:31:59.669311+0000","last_fullsized":"2024-04-30T08:31:59.669311+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:28:59.520091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548406+0000","last_change":"2024-04-30T08:31:52.680377+0000","last_active":"2024-04-30T08:32:30.548406+0000","last_peered":"2024-04-30T08:32:30.548406+0000","last_clean":"2024-04-30T08:32:30.548406+0000","last_became_active":"2024-04-30T08:31:52.679977+0000","last_became_peered":"2024-04-30T08:31:52.679977+0000","last_unstale":"2024-04-30T08:32:30.548406+0000","last_undegraded":"2024-04-30T08:32:30.548406+0000","last_fullsized":"2024-04-30T08:32:30.548406+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:35:22.488450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:58.710009+0000","last_change":"2024-04-30T08:31:52.686488+0000","last_active":"2024-04-30T08:31:58.710009+0000","last_peered":"2024-04-30T08:31:58.710009+0000","last_clean":"2024-04-30T08:31:58.710009+0000","last_became_active":"2024-04-30T08:31:52.686057+0000","last_became_peered":"2024-04-30T08:31:52.686057+0000","last_unstale":"2024-04-30T08:31:58.710009+0000","last_undegraded":"2024-04-30T08:31:58.710009+0000","last_fullsized":"2024-04-30T08:31:58.710009+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:51:47.170678+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.558068+0000","last_change":"2024-04-30T08:31:52.690344+0000","last_active":"2024-04-30T08:32:31.558068+0000","last_peered":"2024-04-30T08:32:31.558068+0000","last_clean":"2024-04-30T08:32:31.558068+0000","last_became_active":"2024-04-30T08:31:52.689857+0000","last_became_peered":"2024-04-30T08:31:52.689857+0000","last_unstale":"2024-04-30T08:32:31.558068+0000","last_undegraded":"2024-04-30T08:32:31.558068+0000","last_fullsized":"2024-04-30T08:32:31.558068+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:45:13.395589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550256+0000","last_change":"2024-04-30T08:31:52.685151+0000","last_active":"2024-04-30T08:32:31.550256+0000","last_peered":"2024-04-30T08:32:31.550256+0000","last_clean":"2024-04-30T08:32:31.550256+0000","last_became_active":"2024-04-30T08:31:52.684722+0000","last_became_peered":"2024-04-30T08:31:52.684722+0000","last_unstale":"2024-04-30T08:32:31.550256+0000","last_undegraded":"2024-04-30T08:32:31.550256+0000","last_fullsized":"2024-04-30T08:32:31.550256+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:14:00.660478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.553484+0000","last_change":"2024-04-30T08:31:52.714294+0000","last_active":"2024-04-30T08:32:30.553484+0000","last_peered":"2024-04-30T08:32:30.553484+0000","last_clean":"2024-04-30T08:32:30.553484+0000","last_became_active":"2024-04-30T08:31:52.710094+0000","last_became_peered":"2024-04-30T08:31:52.710094+0000","last_unstale":"2024-04-30T08:32:30.553484+0000","last_undegraded":"2024-04-30T08:32:30.553484+0000","last_fullsized":"2024-04-30T08:32:30.553484+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:34:44.203093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834299+0000","last_change":"2024-04-30T08:31:52.707992+0000","last_active":"2024-04-30T08:31:59.834299+0000","last_peered":"2024-04-30T08:31:59.834299+0000","last_clean":"2024-04-30T08:31:59.834299+0000","last_became_active":"2024-04-30T08:31:52.706456+0000","last_became_peered":"2024-04-30T08:31:52.706456+0000","last_unstale":"2024-04-30T08:31:59.834299+0000","last_undegraded":"2024-04-30T08:31:59.834299+0000","last_fullsized":"2024-04-30T08:31:59.834299+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:45:32.717842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834299+0000","last_change":"2024-04-30T08:31:52.695605+0000","last_active":"2024-04-30T08:31:59.834299+0000","last_peered":"2024-04-30T08:31:59.834299+0000","last_clean":"2024-04-30T08:31:59.834299+0000","last_became_active":"2024-04-30T08:31:52.692439+0000","last_became_peered":"2024-04-30T08:31:52.692439+0000","last_unstale":"2024-04-30T08:31:59.834299+0000","last_undegraded":"2024-04-30T08:31:59.834299+0000","last_fullsized":"2024-04-30T08:31:59.834299+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:39:38.746567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.547727+0000","last_change":"2024-04-30T08:31:52.707011+0000","last_active":"2024-04-30T08:32:30.547727+0000","last_peered":"2024-04-30T08:32:30.547727+0000","last_clean":"2024-04-30T08:32:30.547727+0000","last_became_active":"2024-04-30T08:31:52.706602+0000","last_became_peered":"2024-04-30T08:31:52.706602+0000","last_unstale":"2024-04-30T08:32:30.547727+0000","last_undegraded":"2024-04-30T08:32:30.547727+0000","last_fullsized":"2024-04-30T08:32:30.547727+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:43:13.922905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548723+0000","last_change":"2024-04-30T08:31:52.711439+0000","last_active":"2024-04-30T08:32:30.548723+0000","last_peered":"2024-04-30T08:32:30.548723+0000","last_clean":"2024-04-30T08:32:30.548723+0000","last_became_active":"2024-04-30T08:31:52.711023+0000","last_became_peered":"2024-04-30T08:31:52.711023+0000","last_unstale":"2024-04-30T08:32:30.548723+0000","last_undegraded":"2024-04-30T08:32:30.548723+0000","last_fullsized":"2024-04-30T08:32:30.548723+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:31:00.654348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.558525+0000","last_change":"2024-04-30T08:31:52.708476+0000","last_active":"2024-04-30T08:32:30.558525+0000","last_peered":"2024-04-30T08:32:30.558525+0000","last_clean":"2024-04-30T08:32:30.558525+0000","last_became_active":"2024-04-30T08:31:52.704735+0000","last_became_peered":"2024-04-30T08:31:52.704735+0000","last_unstale":"2024-04-30T08:32:30.558525+0000","last_undegraded":"2024-04-30T08:32:30.558525+0000","last_fullsized":"2024-04-30T08:32:30.558525+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:40:00.011783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834768+0000","last_change":"2024-04-30T08:31:52.704901+0000","last_active":"2024-04-30T08:31:59.834768+0000","last_peered":"2024-04-30T08:31:59.834768+0000","last_clean":"2024-04-30T08:31:59.834768+0000","last_became_active":"2024-04-30T08:31:52.704636+0000","last_became_peered":"2024-04-30T08:31:52.704636+0000","last_unstale":"2024-04-30T08:31:59.834768+0000","last_undegraded":"2024-04-30T08:31:59.834768+0000","last_fullsized":"2024-04-30T08:31:59.834768+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:08:53.580809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834299+0000","last_change":"2024-04-30T08:31:57.743342+0000","last_active":"2024-04-30T08:31:59.834299+0000","last_peered":"2024-04-30T08:31:59.834299+0000","last_clean":"2024-04-30T08:31:59.834299+0000","last_became_active":"2024-04-30T08:31:57.742850+0000","last_became_peered":"2024-04-30T08:31:57.742850+0000","last_unstale":"2024-04-30T08:31:59.834299+0000","last_undegraded":"2024-04-30T08:31:59.834299+0000","last_fullsized":"2024-04-30T08:31:59.834299+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:59:10.628822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"81'13","reported_seq":40,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.558153+0000","last_change":"2024-04-30T08:31:51.694984+0000","last_active":"2024-04-30T08:32:30.558153+0000","last_peered":"2024-04-30T08:32:30.558153+0000","last_clean":"2024-04-30T08:32:30.558153+0000","last_became_active":"2024-04-30T08:31:51.694514+0000","last_became_peered":"2024-04-30T08:31:51.694514+0000","last_unstale":"2024-04-30T08:32:30.558153+0000","last_undegraded":"2024-04-30T08:32:30.558153+0000","last_fullsized":"2024-04-30T08:32:30.558153+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:17:03.873090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.670928+0000","last_change":"2024-04-30T08:31:52.694276+0000","last_active":"2024-04-30T08:31:59.670928+0000","last_peered":"2024-04-30T08:31:59.670928+0000","last_clean":"2024-04-30T08:31:59.670928+0000","last_became_active":"2024-04-30T08:31:52.693839+0000","last_became_peered":"2024-04-30T08:31:52.693839+0000","last_unstale":"2024-04-30T08:31:59.670928+0000","last_undegraded":"2024-04-30T08:31:59.670928+0000","last_fullsized":"2024-04-30T08:31:59.670928+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:29:21.271040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.549942+0000","last_change":"2024-04-30T08:31:57.754828+0000","last_active":"2024-04-30T08:32:31.549942+0000","last_peered":"2024-04-30T08:32:31.549942+0000","last_clean":"2024-04-30T08:32:31.549942+0000","last_became_active":"2024-04-30T08:31:57.754322+0000","last_became_peered":"2024-04-30T08:31:57.754322+0000","last_unstale":"2024-04-30T08:32:31.549942+0000","last_undegraded":"2024-04-30T08:32:31.549942+0000","last_fullsized":"2024-04-30T08:32:31.549942+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:36:02.560134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"81'2","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.671373+0000","last_change":"2024-04-30T08:31:51.707637+0000","last_active":"2024-04-30T08:31:59.671373+0000","last_peered":"2024-04-30T08:31:59.671373+0000","last_clean":"2024-04-30T08:31:59.671373+0000","last_became_active":"2024-04-30T08:31:51.707233+0000","last_became_peered":"2024-04-30T08:31:51.707233+0000","last_unstale":"2024-04-30T08:31:59.671373+0000","last_undegraded":"2024-04-30T08:31:59.671373+0000","last_fullsized":"2024-04-30T08:31:59.671373+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:14:46.041169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.835049+0000","last_change":"2024-04-30T08:31:52.711442+0000","last_active":"2024-04-30T08:31:59.835049+0000","last_peered":"2024-04-30T08:31:59.835049+0000","last_clean":"2024-04-30T08:31:59.835049+0000","last_became_active":"2024-04-30T08:31:52.710942+0000","last_became_peered":"2024-04-30T08:31:52.710942+0000","last_unstale":"2024-04-30T08:31:59.835049+0000","last_undegraded":"2024-04-30T08:31:59.835049+0000","last_fullsized":"2024-04-30T08:31:59.835049+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:26:56.428770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550129+0000","last_change":"2024-04-30T08:31:57.760535+0000","last_active":"2024-04-30T08:32:31.550129+0000","last_peered":"2024-04-30T08:32:31.550129+0000","last_clean":"2024-04-30T08:32:31.550129+0000","last_became_active":"2024-04-30T08:31:57.760073+0000","last_became_peered":"2024-04-30T08:31:57.760073+0000","last_unstale":"2024-04-30T08:32:31.550129+0000","last_undegraded":"2024-04-30T08:32:31.550129+0000","last_fullsized":"2024-04-30T08:32:31.550129+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:03:51.359474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"81'7","reported_seq":31,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:32:28.528438+0000","last_change":"2024-04-30T08:31:51.678575+0000","last_active":"2024-04-30T08:32:28.528438+0000","last_peered":"2024-04-30T08:32:28.528438+0000","last_clean":"2024-04-30T08:32:28.528438+0000","last_became_active":"2024-04-30T08:31:51.678114+0000","last_became_peered":"2024-04-30T08:31:51.678114+0000","last_unstale":"2024-04-30T08:32:28.528438+0000","last_undegraded":"2024-04-30T08:32:28.528438+0000","last_fullsized":"2024-04-30T08:32:28.528438+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:10:35.517183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548286+0000","last_change":"2024-04-30T08:31:52.704137+0000","last_active":"2024-04-30T08:32:30.548286+0000","last_peered":"2024-04-30T08:32:30.548286+0000","last_clean":"2024-04-30T08:32:30.548286+0000","last_became_active":"2024-04-30T08:31:52.703695+0000","last_became_peered":"2024-04-30T08:31:52.703695+0000","last_unstale":"2024-04-30T08:32:30.548286+0000","last_undegraded":"2024-04-30T08:32:30.548286+0000","last_fullsized":"2024-04-30T08:32:30.548286+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:31:05.461106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.549998+0000","last_change":"2024-04-30T08:31:57.755237+0000","last_active":"2024-04-30T08:32:31.549998+0000","last_peered":"2024-04-30T08:32:31.549998+0000","last_clean":"2024-04-30T08:32:31.549998+0000","last_became_active":"2024-04-30T08:31:57.754438+0000","last_became_peered":"2024-04-30T08:31:57.754438+0000","last_unstale":"2024-04-30T08:32:31.549998+0000","last_undegraded":"2024-04-30T08:32:31.549998+0000","last_fullsized":"2024-04-30T08:32:31.549998+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:20:59.035110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"75'2","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.357899+0000","last_change":"2024-04-30T08:31:51.673793+0000","last_active":"2024-04-30T08:31:59.357899+0000","last_peered":"2024-04-30T08:31:59.357899+0000","last_clean":"2024-04-30T08:31:59.357899+0000","last_became_active":"2024-04-30T08:31:51.672703+0000","last_became_peered":"2024-04-30T08:31:51.672703+0000","last_unstale":"2024-04-30T08:31:59.357899+0000","last_undegraded":"2024-04-30T08:31:59.357899+0000","last_fullsized":"2024-04-30T08:31:59.357899+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:13:18.959542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.553233+0000","last_change":"2024-04-30T08:31:52.694320+0000","last_active":"2024-04-30T08:32:31.553233+0000","last_peered":"2024-04-30T08:32:31.553233+0000","last_clean":"2024-04-30T08:32:31.553233+0000","last_became_active":"2024-04-30T08:31:52.694032+0000","last_became_peered":"2024-04-30T08:31:52.694032+0000","last_unstale":"2024-04-30T08:32:31.553233+0000","last_undegraded":"2024-04-30T08:32:31.553233+0000","last_fullsized":"2024-04-30T08:32:31.553233+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:33:12.347369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834607+0000","last_change":"2024-04-30T08:31:57.739984+0000","last_active":"2024-04-30T08:31:59.834607+0000","last_peered":"2024-04-30T08:31:59.834607+0000","last_clean":"2024-04-30T08:31:59.834607+0000","last_became_active":"2024-04-30T08:31:57.739477+0000","last_became_peered":"2024-04-30T08:31:57.739477+0000","last_unstale":"2024-04-30T08:31:59.834607+0000","last_undegraded":"2024-04-30T08:31:59.834607+0000","last_fullsized":"2024-04-30T08:31:59.834607+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:10:11.325419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"81'6","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.564820+0000","last_change":"2024-04-30T08:31:51.678792+0000","last_active":"2024-04-30T08:32:31.564820+0000","last_peered":"2024-04-30T08:32:31.564820+0000","last_clean":"2024-04-30T08:32:31.564820+0000","last_became_active":"2024-04-30T08:31:51.678094+0000","last_became_peered":"2024-04-30T08:31:51.678094+0000","last_unstale":"2024-04-30T08:32:31.564820+0000","last_undegraded":"2024-04-30T08:32:31.564820+0000","last_fullsized":"2024-04-30T08:32:31.564820+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:03:17.526016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.357855+0000","last_change":"2024-04-30T08:31:52.701925+0000","last_active":"2024-04-30T08:31:59.357855+0000","last_peered":"2024-04-30T08:31:59.357855+0000","last_clean":"2024-04-30T08:31:59.357855+0000","last_became_active":"2024-04-30T08:31:52.701425+0000","last_became_peered":"2024-04-30T08:31:52.701425+0000","last_unstale":"2024-04-30T08:31:59.357855+0000","last_undegraded":"2024-04-30T08:31:59.357855+0000","last_fullsized":"2024-04-30T08:31:59.357855+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:20:35.767086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834905+0000","last_change":"2024-04-30T08:31:57.765047+0000","last_active":"2024-04-30T08:31:59.834905+0000","last_peered":"2024-04-30T08:31:59.834905+0000","last_clean":"2024-04-30T08:31:59.834905+0000","last_became_active":"2024-04-30T08:31:57.764519+0000","last_became_peered":"2024-04-30T08:31:57.764519+0000","last_unstale":"2024-04-30T08:31:59.834905+0000","last_undegraded":"2024-04-30T08:31:59.834905+0000","last_fullsized":"2024-04-30T08:31:59.834905+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:06:16.126396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"81'7","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:32:28.533731+0000","last_change":"2024-04-30T08:31:51.686403+0000","last_active":"2024-04-30T08:32:28.533731+0000","last_peered":"2024-04-30T08:32:28.533731+0000","last_clean":"2024-04-30T08:32:28.533731+0000","last_became_active":"2024-04-30T08:31:51.686096+0000","last_became_peered":"2024-04-30T08:31:51.686096+0000","last_unstale":"2024-04-30T08:32:28.533731+0000","last_undegraded":"2024-04-30T08:32:28.533731+0000","last_fullsized":"2024-04-30T08:32:28.533731+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:47:05.961930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.554940+0000","last_change":"2024-04-30T08:31:52.698512+0000","last_active":"2024-04-30T08:32:31.554940+0000","last_peered":"2024-04-30T08:32:31.554940+0000","last_clean":"2024-04-30T08:32:31.554940+0000","last_became_active":"2024-04-30T08:31:52.698168+0000","last_became_peered":"2024-04-30T08:31:52.698168+0000","last_unstale":"2024-04-30T08:32:31.554940+0000","last_undegraded":"2024-04-30T08:32:31.554940+0000","last_fullsized":"2024-04-30T08:32:31.554940+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:22:45.094365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548550+0000","last_change":"2024-04-30T08:31:57.761430+0000","last_active":"2024-04-30T08:32:30.548550+0000","last_peered":"2024-04-30T08:32:30.548550+0000","last_clean":"2024-04-30T08:32:30.548550+0000","last_became_active":"2024-04-30T08:31:57.761062+0000","last_became_peered":"2024-04-30T08:31:57.761062+0000","last_unstale":"2024-04-30T08:32:30.548550+0000","last_undegraded":"2024-04-30T08:32:30.548550+0000","last_fullsized":"2024-04-30T08:32:30.548550+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:32:05.403786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"81'3","reported_seq":29,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.552212+0000","last_change":"2024-04-30T08:31:51.686694+0000","last_active":"2024-04-30T08:32:30.552212+0000","last_peered":"2024-04-30T08:32:30.552212+0000","last_clean":"2024-04-30T08:32:30.552212+0000","last_became_active":"2024-04-30T08:31:51.685773+0000","last_became_peered":"2024-04-30T08:31:51.685773+0000","last_unstale":"2024-04-30T08:32:30.552212+0000","last_undegraded":"2024-04-30T08:32:30.552212+0000","last_fullsized":"2024-04-30T08:32:30.552212+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:44:25.327977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.669796+0000","last_change":"2024-04-30T08:31:52.709794+0000","last_active":"2024-04-30T08:31:59.669796+0000","last_peered":"2024-04-30T08:31:59.669796+0000","last_clean":"2024-04-30T08:31:59.669796+0000","last_became_active":"2024-04-30T08:31:52.709371+0000","last_became_peered":"2024-04-30T08:31:52.709371+0000","last_unstale":"2024-04-30T08:31:59.669796+0000","last_undegraded":"2024-04-30T08:31:59.669796+0000","last_fullsized":"2024-04-30T08:31:59.669796+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:36:59.892551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834943+0000","last_change":"2024-04-30T08:31:57.757224+0000","last_active":"2024-04-30T08:31:59.834943+0000","last_peered":"2024-04-30T08:31:59.834943+0000","last_clean":"2024-04-30T08:31:59.834943+0000","last_became_active":"2024-04-30T08:31:57.756658+0000","last_became_peered":"2024-04-30T08:31:57.756658+0000","last_unstale":"2024-04-30T08:31:59.834943+0000","last_undegraded":"2024-04-30T08:31:59.834943+0000","last_fullsized":"2024-04-30T08:31:59.834943+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:01:21.309322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"81'3","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.555826+0000","last_change":"2024-04-30T08:31:51.678944+0000","last_active":"2024-04-30T08:32:31.555826+0000","last_peered":"2024-04-30T08:32:31.555826+0000","last_clean":"2024-04-30T08:32:31.555826+0000","last_became_active":"2024-04-30T08:31:51.678493+0000","last_became_peered":"2024-04-30T08:31:51.678493+0000","last_unstale":"2024-04-30T08:32:31.555826+0000","last_undegraded":"2024-04-30T08:32:31.555826+0000","last_fullsized":"2024-04-30T08:32:31.555826+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:48:11.111466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.555177+0000","last_change":"2024-04-30T08:31:52.707627+0000","last_active":"2024-04-30T08:32:30.555177+0000","last_peered":"2024-04-30T08:32:30.555177+0000","last_clean":"2024-04-30T08:32:30.555177+0000","last_became_active":"2024-04-30T08:31:52.706441+0000","last_became_peered":"2024-04-30T08:31:52.706441+0000","last_unstale":"2024-04-30T08:32:30.555177+0000","last_undegraded":"2024-04-30T08:32:30.555177+0000","last_fullsized":"2024-04-30T08:32:30.555177+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:15:45.768927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.669809+0000","last_change":"2024-04-30T08:31:57.740324+0000","last_active":"2024-04-30T08:31:59.669809+0000","last_peered":"2024-04-30T08:31:59.669809+0000","last_clean":"2024-04-30T08:31:59.669809+0000","last_became_active":"2024-04-30T08:31:57.739464+0000","last_became_peered":"2024-04-30T08:31:57.739464+0000","last_unstale":"2024-04-30T08:31:59.669809+0000","last_undegraded":"2024-04-30T08:31:59.669809+0000","last_fullsized":"2024-04-30T08:31:59.669809+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:11:37.395545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"81'4","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:32:26.513947+0000","last_change":"2024-04-30T08:31:51.695283+0000","last_active":"2024-04-30T08:32:26.513947+0000","last_peered":"2024-04-30T08:32:26.513947+0000","last_clean":"2024-04-30T08:32:26.513947+0000","last_became_active":"2024-04-30T08:31:51.694848+0000","last_became_peered":"2024-04-30T08:31:51.694848+0000","last_unstale":"2024-04-30T08:32:26.513947+0000","last_undegraded":"2024-04-30T08:32:26.513947+0000","last_fullsized":"2024-04-30T08:32:26.513947+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:41:38.638034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548438+0000","last_change":"2024-04-30T08:31:52.693594+0000","last_active":"2024-04-30T08:32:30.548438+0000","last_peered":"2024-04-30T08:32:30.548438+0000","last_clean":"2024-04-30T08:32:30.548438+0000","last_became_active":"2024-04-30T08:31:52.693199+0000","last_became_peered":"2024-04-30T08:31:52.693199+0000","last_unstale":"2024-04-30T08:32:30.548438+0000","last_undegraded":"2024-04-30T08:32:30.548438+0000","last_fullsized":"2024-04-30T08:32:30.548438+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:57:49.908617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.553991+0000","last_change":"2024-04-30T08:31:57.749727+0000","last_active":"2024-04-30T08:32:30.553991+0000","last_peered":"2024-04-30T08:32:30.553991+0000","last_clean":"2024-04-30T08:32:30.553991+0000","last_became_active":"2024-04-30T08:31:57.749267+0000","last_became_peered":"2024-04-30T08:31:57.749267+0000","last_unstale":"2024-04-30T08:32:30.553991+0000","last_undegraded":"2024-04-30T08:32:30.553991+0000","last_fullsized":"2024-04-30T08:32:30.553991+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:26:12.683008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"81'4","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.557872+0000","last_change":"2024-04-30T08:31:51.677398+0000","last_active":"2024-04-30T08:32:31.557872+0000","last_peered":"2024-04-30T08:32:31.557872+0000","last_clean":"2024-04-30T08:32:31.557872+0000","last_became_active":"2024-04-30T08:31:51.677031+0000","last_became_peered":"2024-04-30T08:31:51.677031+0000","last_unstale":"2024-04-30T08:32:31.557872+0000","last_undegraded":"2024-04-30T08:32:31.557872+0000","last_fullsized":"2024-04-30T08:32:31.557872+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:24:25.154229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.835058+0000","last_change":"2024-04-30T08:31:52.717016+0000","last_active":"2024-04-30T08:31:59.835058+0000","last_peered":"2024-04-30T08:31:59.835058+0000","last_clean":"2024-04-30T08:31:59.835058+0000","last_became_active":"2024-04-30T08:31:52.714622+0000","last_became_peered":"2024-04-30T08:31:52.714622+0000","last_unstale":"2024-04-30T08:31:59.835058+0000","last_undegraded":"2024-04-30T08:31:59.835058+0000","last_fullsized":"2024-04-30T08:31:59.835058+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:56:44.550915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.586746+0000","last_change":"2024-04-30T08:32:31.586746+0000","last_active":"2024-04-30T08:32:31.586746+0000","last_peered":"2024-04-30T08:32:31.586746+0000","last_clean":"2024-04-30T08:32:31.586746+0000","last_became_active":"2024-04-30T08:32:31.586356+0000","last_became_peered":"2024-04-30T08:32:31.586356+0000","last_unstale":"2024-04-30T08:32:31.586746+0000","last_undegraded":"2024-04-30T08:32:31.586746+0000","last_fullsized":"2024-04-30T08:32:31.586746+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:22:39.371508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"81'5","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.553126+0000","last_change":"2024-04-30T08:31:51.694424+0000","last_active":"2024-04-30T08:32:31.553126+0000","last_peered":"2024-04-30T08:32:31.553126+0000","last_clean":"2024-04-30T08:32:31.553126+0000","last_became_active":"2024-04-30T08:31:51.694081+0000","last_became_peered":"2024-04-30T08:31:51.694081+0000","last_unstale":"2024-04-30T08:32:31.553126+0000","last_undegraded":"2024-04-30T08:32:31.553126+0000","last_fullsized":"2024-04-30T08:32:31.553126+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:27:33.328477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548544+0000","last_change":"2024-04-30T08:31:52.684739+0000","last_active":"2024-04-30T08:32:30.548544+0000","last_peered":"2024-04-30T08:32:30.548544+0000","last_clean":"2024-04-30T08:32:30.548544+0000","last_became_active":"2024-04-30T08:31:52.684316+0000","last_became_peered":"2024-04-30T08:31:52.684316+0000","last_unstale":"2024-04-30T08:32:30.548544+0000","last_undegraded":"2024-04-30T08:32:30.548544+0000","last_fullsized":"2024-04-30T08:32:30.548544+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:12:05.437493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.615550+0000","last_change":"2024-04-30T08:31:57.760643+0000","last_active":"2024-04-30T08:31:59.615550+0000","last_peered":"2024-04-30T08:31:59.615550+0000","last_clean":"2024-04-30T08:31:59.615550+0000","last_became_active":"2024-04-30T08:31:57.760053+0000","last_became_peered":"2024-04-30T08:31:57.760053+0000","last_unstale":"2024-04-30T08:31:59.615550+0000","last_undegraded":"2024-04-30T08:31:59.615550+0000","last_fullsized":"2024-04-30T08:31:59.615550+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:29:12.748550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"81'12","reported_seq":37,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:32:26.511027+0000","last_change":"2024-04-30T08:31:51.676223+0000","last_active":"2024-04-30T08:32:26.511027+0000","last_peered":"2024-04-30T08:32:26.511027+0000","last_clean":"2024-04-30T08:32:26.511027+0000","last_became_active":"2024-04-30T08:31:51.675830+0000","last_became_peered":"2024-04-30T08:31:51.675830+0000","last_unstale":"2024-04-30T08:32:26.511027+0000","last_undegraded":"2024-04-30T08:32:26.511027+0000","last_fullsized":"2024-04-30T08:32:26.511027+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:28:49.497719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.358155+0000","last_change":"2024-04-30T08:31:52.686892+0000","last_active":"2024-04-30T08:31:59.358155+0000","last_peered":"2024-04-30T08:31:59.358155+0000","last_clean":"2024-04-30T08:31:59.358155+0000","last_became_active":"2024-04-30T08:31:52.686533+0000","last_became_peered":"2024-04-30T08:31:52.686533+0000","last_unstale":"2024-04-30T08:31:59.358155+0000","last_undegraded":"2024-04-30T08:31:59.358155+0000","last_fullsized":"2024-04-30T08:31:59.358155+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:50:18.836161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.553606+0000","last_change":"2024-04-30T08:31:57.752865+0000","last_active":"2024-04-30T08:32:31.553606+0000","last_peered":"2024-04-30T08:32:31.553606+0000","last_clean":"2024-04-30T08:32:31.553606+0000","last_became_active":"2024-04-30T08:31:57.752385+0000","last_became_peered":"2024-04-30T08:31:57.752385+0000","last_unstale":"2024-04-30T08:32:31.553606+0000","last_undegraded":"2024-04-30T08:32:31.553606+0000","last_fullsized":"2024-04-30T08:32:31.553606+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:15:15.315580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"81'9","reported_seq":2,"reported_epoch":82,"state":"peering","last_fresh":"2024-04-30T08:32:29.552341+0000","last_change":"2024-04-30T08:32:30.944941+0000","last_active":"2024-04-30T08:32:30.944941+0000","last_peered":"2024-04-30T08:32:29.552341+0000","last_clean":"2024-04-30T08:32:29.552341+0000","last_became_active":"2024-04-30T08:31:51.702122+0000","last_became_peered":"2024-04-30T08:31:51.702122+0000","last_unstale":"2024-04-30T08:32:29.552341+0000","last_undegraded":"2024-04-30T08:32:29.552341+0000","last_fullsized":"2024-04-30T08:32:29.552341+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":6,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.553023+0000","last_change":"2024-04-30T08:31:52.706259+0000","last_active":"2024-04-30T08:32:30.553023+0000","last_peered":"2024-04-30T08:32:30.553023+0000","last_clean":"2024-04-30T08:32:30.553023+0000","last_became_active":"2024-04-30T08:31:52.705620+0000","last_became_peered":"2024-04-30T08:31:52.705620+0000","last_unstale":"2024-04-30T08:32:30.553023+0000","last_undegraded":"2024-04-30T08:32:30.553023+0000","last_fullsized":"2024-04-30T08:32:30.553023+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:22:29.816950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"peering","last_fresh":"2024-04-30T08:32:00.941417+0000","last_change":"2024-04-30T08:32:30.548960+0000","last_active":"2024-04-30T08:32:30.548960+0000","last_peered":"2024-04-30T08:32:00.941417+0000","last_clean":"2024-04-30T08:32:00.941417+0000","last_became_active":"2024-04-30T08:31:57.762531+0000","last_became_peered":"2024-04-30T08:31:57.762531+0000","last_unstale":"2024-04-30T08:32:00.941417+0000","last_undegraded":"2024-04-30T08:32:00.941417+0000","last_fullsized":"2024-04-30T08:32:00.941417+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7,8,9],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"81'5","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.554892+0000","last_change":"2024-04-30T08:31:51.701122+0000","last_active":"2024-04-30T08:32:31.554892+0000","last_peered":"2024-04-30T08:32:31.554892+0000","last_clean":"2024-04-30T08:32:31.554892+0000","last_became_active":"2024-04-30T08:31:51.700815+0000","last_became_peered":"2024-04-30T08:31:51.700815+0000","last_unstale":"2024-04-30T08:32:31.554892+0000","last_undegraded":"2024-04-30T08:32:31.554892+0000","last_fullsized":"2024-04-30T08:32:31.554892+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:33:12.714979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.558463+0000","last_change":"2024-04-30T08:31:52.716043+0000","last_active":"2024-04-30T08:32:31.558463+0000","last_peered":"2024-04-30T08:32:31.558463+0000","last_clean":"2024-04-30T08:32:31.558463+0000","last_became_active":"2024-04-30T08:31:52.715284+0000","last_became_peered":"2024-04-30T08:31:52.715284+0000","last_unstale":"2024-04-30T08:32:31.558463+0000","last_undegraded":"2024-04-30T08:32:31.558463+0000","last_fullsized":"2024-04-30T08:32:31.558463+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:00:01.863606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.557749+0000","last_change":"2024-04-30T08:31:57.765008+0000","last_active":"2024-04-30T08:32:30.557749+0000","last_peered":"2024-04-30T08:32:30.557749+0000","last_clean":"2024-04-30T08:32:30.557749+0000","last_became_active":"2024-04-30T08:31:57.762958+0000","last_became_peered":"2024-04-30T08:31:57.762958+0000","last_unstale":"2024-04-30T08:32:30.557749+0000","last_undegraded":"2024-04-30T08:32:30.557749+0000","last_fullsized":"2024-04-30T08:32:30.557749+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:48:47.116004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"81'6","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:32.572977+0000","last_change":"2024-04-30T08:31:51.676681+0000","last_active":"2024-04-30T08:32:32.572977+0000","last_peered":"2024-04-30T08:32:32.572977+0000","last_clean":"2024-04-30T08:32:32.572977+0000","last_became_active":"2024-04-30T08:31:51.676311+0000","last_became_peered":"2024-04-30T08:31:51.676311+0000","last_unstale":"2024-04-30T08:32:32.572977+0000","last_undegraded":"2024-04-30T08:32:32.572977+0000","last_fullsized":"2024-04-30T08:32:32.572977+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:24:49.757427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.558213+0000","last_change":"2024-04-30T08:31:52.689418+0000","last_active":"2024-04-30T08:32:31.558213+0000","last_peered":"2024-04-30T08:32:31.558213+0000","last_clean":"2024-04-30T08:32:31.558213+0000","last_became_active":"2024-04-30T08:31:52.688840+0000","last_became_peered":"2024-04-30T08:31:52.688840+0000","last_unstale":"2024-04-30T08:32:31.558213+0000","last_undegraded":"2024-04-30T08:32:31.558213+0000","last_fullsized":"2024-04-30T08:32:31.558213+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:40:05.781137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.554105+0000","last_change":"2024-04-30T08:31:57.765875+0000","last_active":"2024-04-30T08:32:30.554105+0000","last_peered":"2024-04-30T08:32:30.554105+0000","last_clean":"2024-04-30T08:32:30.554105+0000","last_became_active":"2024-04-30T08:31:57.765432+0000","last_became_peered":"2024-04-30T08:31:57.765432+0000","last_unstale":"2024-04-30T08:32:30.554105+0000","last_undegraded":"2024-04-30T08:32:30.554105+0000","last_fullsized":"2024-04-30T08:32:30.554105+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:24:44.329532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"81'16","reported_seq":40,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:32:27.524097+0000","last_change":"2024-04-30T08:31:51.695007+0000","last_active":"2024-04-30T08:32:27.524097+0000","last_peered":"2024-04-30T08:32:27.524097+0000","last_clean":"2024-04-30T08:32:27.524097+0000","last_became_active":"2024-04-30T08:31:51.694533+0000","last_became_peered":"2024-04-30T08:31:51.694533+0000","last_unstale":"2024-04-30T08:32:27.524097+0000","last_undegraded":"2024-04-30T08:32:27.524097+0000","last_fullsized":"2024-04-30T08:32:27.524097+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:47:15.104172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.553520+0000","last_change":"2024-04-30T08:31:52.692274+0000","last_active":"2024-04-30T08:32:31.553520+0000","last_peered":"2024-04-30T08:32:31.553520+0000","last_clean":"2024-04-30T08:32:31.553520+0000","last_became_active":"2024-04-30T08:31:52.691809+0000","last_became_peered":"2024-04-30T08:31:52.691809+0000","last_unstale":"2024-04-30T08:32:31.553520+0000","last_undegraded":"2024-04-30T08:32:31.553520+0000","last_fullsized":"2024-04-30T08:32:31.553520+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:50:00.201782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.555077+0000","last_change":"2024-04-30T08:31:57.732383+0000","last_active":"2024-04-30T08:32:31.555077+0000","last_peered":"2024-04-30T08:32:31.555077+0000","last_clean":"2024-04-30T08:32:31.555077+0000","last_became_active":"2024-04-30T08:31:57.731893+0000","last_became_peered":"2024-04-30T08:31:57.731893+0000","last_unstale":"2024-04-30T08:32:31.555077+0000","last_undegraded":"2024-04-30T08:32:31.555077+0000","last_fullsized":"2024-04-30T08:32:31.555077+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:08:55.243526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"81'2","reported_seq":26,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.358322+0000","last_change":"2024-04-30T08:31:51.671991+0000","last_active":"2024-04-30T08:31:59.358322+0000","last_peered":"2024-04-30T08:31:59.358322+0000","last_clean":"2024-04-30T08:31:59.358322+0000","last_became_active":"2024-04-30T08:31:51.671508+0000","last_became_peered":"2024-04-30T08:31:51.671508+0000","last_unstale":"2024-04-30T08:31:59.358322+0000","last_undegraded":"2024-04-30T08:31:59.358322+0000","last_fullsized":"2024-04-30T08:31:59.358322+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:54:30.106748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548288+0000","last_change":"2024-04-30T08:31:52.687555+0000","last_active":"2024-04-30T08:32:30.548288+0000","last_peered":"2024-04-30T08:32:30.548288+0000","last_clean":"2024-04-30T08:32:30.548288+0000","last_became_active":"2024-04-30T08:31:52.686000+0000","last_became_peered":"2024-04-30T08:31:52.686000+0000","last_unstale":"2024-04-30T08:32:30.548288+0000","last_undegraded":"2024-04-30T08:32:30.548288+0000","last_fullsized":"2024-04-30T08:32:30.548288+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:37:03.529774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.357960+0000","last_change":"2024-04-30T08:31:58.061077+0000","last_active":"2024-04-30T08:31:59.357960+0000","last_peered":"2024-04-30T08:31:59.357960+0000","last_clean":"2024-04-30T08:31:59.357960+0000","last_became_active":"2024-04-30T08:31:58.060486+0000","last_became_peered":"2024-04-30T08:31:58.060486+0000","last_unstale":"2024-04-30T08:31:59.357960+0000","last_undegraded":"2024-04-30T08:31:59.357960+0000","last_fullsized":"2024-04-30T08:31:59.357960+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:31:50.936560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"81'1","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.557573+0000","last_change":"2024-04-30T08:31:51.702869+0000","last_active":"2024-04-30T08:32:30.557573+0000","last_peered":"2024-04-30T08:32:30.557573+0000","last_clean":"2024-04-30T08:32:30.557573+0000","last_became_active":"2024-04-30T08:31:51.701150+0000","last_became_peered":"2024-04-30T08:31:51.701150+0000","last_unstale":"2024-04-30T08:32:30.557573+0000","last_undegraded":"2024-04-30T08:32:30.557573+0000","last_fullsized":"2024-04-30T08:32:30.557573+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:47:11.010310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:58.709852+0000","last_change":"2024-04-30T08:31:52.709159+0000","last_active":"2024-04-30T08:31:58.709852+0000","last_peered":"2024-04-30T08:31:58.709852+0000","last_clean":"2024-04-30T08:31:58.709852+0000","last_became_active":"2024-04-30T08:31:52.708799+0000","last_became_peered":"2024-04-30T08:31:52.708799+0000","last_unstale":"2024-04-30T08:31:58.709852+0000","last_undegraded":"2024-04-30T08:31:58.709852+0000","last_fullsized":"2024-04-30T08:31:58.709852+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:51:10.588599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.554141+0000","last_change":"2024-04-30T08:31:57.751652+0000","last_active":"2024-04-30T08:32:30.554141+0000","last_peered":"2024-04-30T08:32:30.554141+0000","last_clean":"2024-04-30T08:32:30.554141+0000","last_became_active":"2024-04-30T08:31:57.751176+0000","last_became_peered":"2024-04-30T08:31:57.751176+0000","last_unstale":"2024-04-30T08:32:30.554141+0000","last_undegraded":"2024-04-30T08:32:30.554141+0000","last_fullsized":"2024-04-30T08:32:30.554141+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:13:24.528842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"81'5","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.571485+0000","last_change":"2024-04-30T08:31:51.700816+0000","last_active":"2024-04-30T08:32:31.571485+0000","last_peered":"2024-04-30T08:32:31.571485+0000","last_clean":"2024-04-30T08:32:31.571485+0000","last_became_active":"2024-04-30T08:31:51.700470+0000","last_became_peered":"2024-04-30T08:31:51.700470+0000","last_unstale":"2024-04-30T08:32:31.571485+0000","last_undegraded":"2024-04-30T08:32:31.571485+0000","last_fullsized":"2024-04-30T08:32:31.571485+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:49:11.793412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:58.709513+0000","last_change":"2024-04-30T08:31:52.715236+0000","last_active":"2024-04-30T08:31:58.709513+0000","last_peered":"2024-04-30T08:31:58.709513+0000","last_clean":"2024-04-30T08:31:58.709513+0000","last_became_active":"2024-04-30T08:31:52.714801+0000","last_became_peered":"2024-04-30T08:31:52.714801+0000","last_unstale":"2024-04-30T08:31:58.709513+0000","last_undegraded":"2024-04-30T08:31:58.709513+0000","last_fullsized":"2024-04-30T08:31:58.709513+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:43:09.650002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.552783+0000","last_change":"2024-04-30T08:31:58.060927+0000","last_active":"2024-04-30T08:32:30.552783+0000","last_peered":"2024-04-30T08:32:30.552783+0000","last_clean":"2024-04-30T08:32:30.552783+0000","last_became_active":"2024-04-30T08:31:58.060182+0000","last_became_peered":"2024-04-30T08:31:58.060182+0000","last_unstale":"2024-04-30T08:32:30.552783+0000","last_undegraded":"2024-04-30T08:32:30.552783+0000","last_fullsized":"2024-04-30T08:32:30.552783+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:21:00.228064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.560485+0000","last_change":"2024-04-30T08:32:31.560485+0000","last_active":"2024-04-30T08:32:31.560485+0000","last_peered":"2024-04-30T08:32:31.560485+0000","last_clean":"2024-04-30T08:32:31.560485+0000","last_became_active":"2024-04-30T08:32:31.560066+0000","last_became_peered":"2024-04-30T08:32:31.560066+0000","last_unstale":"2024-04-30T08:32:31.560485+0000","last_undegraded":"2024-04-30T08:32:31.560485+0000","last_fullsized":"2024-04-30T08:32:31.560485+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:13:40.769761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"81'5","reported_seq":33,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.557878+0000","last_change":"2024-04-30T08:31:51.702808+0000","last_active":"2024-04-30T08:32:30.557878+0000","last_peered":"2024-04-30T08:32:30.557878+0000","last_clean":"2024-04-30T08:32:30.557878+0000","last_became_active":"2024-04-30T08:31:51.702326+0000","last_became_peered":"2024-04-30T08:31:51.702326+0000","last_unstale":"2024-04-30T08:32:30.557878+0000","last_undegraded":"2024-04-30T08:32:30.557878+0000","last_fullsized":"2024-04-30T08:32:30.557878+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:11:35.552670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"81'1","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.557866+0000","last_change":"2024-04-30T08:31:57.760359+0000","last_active":"2024-04-30T08:32:31.557866+0000","last_peered":"2024-04-30T08:32:31.557866+0000","last_clean":"2024-04-30T08:32:31.557866+0000","last_became_active":"2024-04-30T08:31:57.759821+0000","last_became_peered":"2024-04-30T08:31:57.759821+0000","last_unstale":"2024-04-30T08:32:31.557866+0000","last_undegraded":"2024-04-30T08:32:31.557866+0000","last_fullsized":"2024-04-30T08:32:31.557866+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:50:17.853791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.669652+0000","last_change":"2024-04-30T08:31:52.689281+0000","last_active":"2024-04-30T08:31:59.669652+0000","last_peered":"2024-04-30T08:31:59.669652+0000","last_clean":"2024-04-30T08:31:59.669652+0000","last_became_active":"2024-04-30T08:31:52.688834+0000","last_became_peered":"2024-04-30T08:31:52.688834+0000","last_unstale":"2024-04-30T08:31:59.669652+0000","last_undegraded":"2024-04-30T08:31:59.669652+0000","last_fullsized":"2024-04-30T08:31:59.669652+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:42:12.836649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'6","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550187+0000","last_change":"2024-04-30T08:31:51.683712+0000","last_active":"2024-04-30T08:32:31.550187+0000","last_peered":"2024-04-30T08:32:31.550187+0000","last_clean":"2024-04-30T08:32:31.550187+0000","last_became_active":"2024-04-30T08:31:51.683298+0000","last_became_peered":"2024-04-30T08:31:51.683298+0000","last_unstale":"2024-04-30T08:32:31.550187+0000","last_undegraded":"2024-04-30T08:32:31.550187+0000","last_fullsized":"2024-04-30T08:32:31.550187+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:43:08.832104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":103,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.557511+0000","last_change":"2024-04-30T08:30:26.503221+0000","last_active":"2024-04-30T08:32:31.557511+0000","last_peered":"2024-04-30T08:32:31.557511+0000","last_clean":"2024-04-30T08:32:31.557511+0000","last_became_active":"2024-04-30T08:30:26.498399+0000","last_became_peered":"2024-04-30T08:30:26.498399+0000","last_unstale":"2024-04-30T08:32:31.557511+0000","last_undegraded":"2024-04-30T08:32:31.557511+0000","last_fullsized":"2024-04-30T08:32:31.557511+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:26:20.170075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:26:20.170075+0000","last_clean_scrub_stamp":"2024-04-30T08:26:20.170075+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:34:44.067560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.552597+0000","last_change":"2024-04-30T08:31:58.061283+0000","last_active":"2024-04-30T08:32:31.552597+0000","last_peered":"2024-04-30T08:32:31.552597+0000","last_clean":"2024-04-30T08:32:31.552597+0000","last_became_active":"2024-04-30T08:31:58.060650+0000","last_became_peered":"2024-04-30T08:31:58.060650+0000","last_unstale":"2024-04-30T08:32:31.552597+0000","last_undegraded":"2024-04-30T08:32:31.552597+0000","last_fullsized":"2024-04-30T08:32:31.552597+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:13:01.811462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.551989+0000","last_change":"2024-04-30T08:31:52.699639+0000","last_active":"2024-04-30T08:32:30.551989+0000","last_peered":"2024-04-30T08:32:30.551989+0000","last_clean":"2024-04-30T08:32:30.551989+0000","last_became_active":"2024-04-30T08:31:52.699164+0000","last_became_peered":"2024-04-30T08:31:52.699164+0000","last_unstale":"2024-04-30T08:32:30.551989+0000","last_undegraded":"2024-04-30T08:32:30.551989+0000","last_fullsized":"2024-04-30T08:32:30.551989+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:22:18.873504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"81'4","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.558080+0000","last_change":"2024-04-30T08:31:51.679153+0000","last_active":"2024-04-30T08:32:31.558080+0000","last_peered":"2024-04-30T08:32:31.558080+0000","last_clean":"2024-04-30T08:32:31.558080+0000","last_became_active":"2024-04-30T08:31:51.678348+0000","last_became_peered":"2024-04-30T08:31:51.678348+0000","last_unstale":"2024-04-30T08:32:31.558080+0000","last_undegraded":"2024-04-30T08:32:31.558080+0000","last_fullsized":"2024-04-30T08:32:31.558080+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:02:56.240218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548074+0000","last_change":"2024-04-30T08:31:57.755423+0000","last_active":"2024-04-30T08:32:30.548074+0000","last_peered":"2024-04-30T08:32:30.548074+0000","last_clean":"2024-04-30T08:32:30.548074+0000","last_became_active":"2024-04-30T08:31:57.754835+0000","last_became_peered":"2024-04-30T08:31:57.754835+0000","last_unstale":"2024-04-30T08:32:30.548074+0000","last_undegraded":"2024-04-30T08:32:30.548074+0000","last_fullsized":"2024-04-30T08:32:30.548074+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:19:25.168920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"81'3","reported_seq":27,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:32:28.529739+0000","last_change":"2024-04-30T08:31:51.693839+0000","last_active":"2024-04-30T08:32:28.529739+0000","last_peered":"2024-04-30T08:32:28.529739+0000","last_clean":"2024-04-30T08:32:28.529739+0000","last_became_active":"2024-04-30T08:31:51.693349+0000","last_became_peered":"2024-04-30T08:31:51.693349+0000","last_unstale":"2024-04-30T08:32:28.529739+0000","last_undegraded":"2024-04-30T08:32:28.529739+0000","last_fullsized":"2024-04-30T08:32:28.529739+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:41:51.330901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.554333+0000","last_change":"2024-04-30T08:31:52.712360+0000","last_active":"2024-04-30T08:32:30.554333+0000","last_peered":"2024-04-30T08:32:30.554333+0000","last_clean":"2024-04-30T08:32:30.554333+0000","last_became_active":"2024-04-30T08:31:52.711652+0000","last_became_peered":"2024-04-30T08:31:52.711652+0000","last_unstale":"2024-04-30T08:32:30.554333+0000","last_undegraded":"2024-04-30T08:32:30.554333+0000","last_fullsized":"2024-04-30T08:32:30.554333+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:21:10.285386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.669947+0000","last_change":"2024-04-30T08:31:58.061654+0000","last_active":"2024-04-30T08:31:59.669947+0000","last_peered":"2024-04-30T08:31:59.669947+0000","last_clean":"2024-04-30T08:31:59.669947+0000","last_became_active":"2024-04-30T08:31:58.060181+0000","last_became_peered":"2024-04-30T08:31:58.060181+0000","last_unstale":"2024-04-30T08:31:59.669947+0000","last_undegraded":"2024-04-30T08:31:59.669947+0000","last_fullsized":"2024-04-30T08:31:59.669947+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:46:03.230626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"81'8","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.558636+0000","last_change":"2024-04-30T08:31:51.706940+0000","last_active":"2024-04-30T08:32:30.558636+0000","last_peered":"2024-04-30T08:32:30.558636+0000","last_clean":"2024-04-30T08:32:30.558636+0000","last_became_active":"2024-04-30T08:31:51.706481+0000","last_became_peered":"2024-04-30T08:31:51.706481+0000","last_unstale":"2024-04-30T08:32:30.558636+0000","last_undegraded":"2024-04-30T08:32:30.558636+0000","last_fullsized":"2024-04-30T08:32:30.558636+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:53:31.955595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.670853+0000","last_change":"2024-04-30T08:31:52.696625+0000","last_active":"2024-04-30T08:31:59.670853+0000","last_peered":"2024-04-30T08:31:59.670853+0000","last_clean":"2024-04-30T08:31:59.670853+0000","last_became_active":"2024-04-30T08:31:52.696230+0000","last_became_peered":"2024-04-30T08:31:52.696230+0000","last_unstale":"2024-04-30T08:31:59.670853+0000","last_undegraded":"2024-04-30T08:31:59.670853+0000","last_fullsized":"2024-04-30T08:31:59.670853+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:11:46.404251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834193+0000","last_change":"2024-04-30T08:31:57.738329+0000","last_active":"2024-04-30T08:31:59.834193+0000","last_peered":"2024-04-30T08:31:59.834193+0000","last_clean":"2024-04-30T08:31:59.834193+0000","last_became_active":"2024-04-30T08:31:57.737897+0000","last_became_peered":"2024-04-30T08:31:57.737897+0000","last_unstale":"2024-04-30T08:31:59.834193+0000","last_undegraded":"2024-04-30T08:31:59.834193+0000","last_fullsized":"2024-04-30T08:31:59.834193+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:02:43.074578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"81'7","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550515+0000","last_change":"2024-04-30T08:31:51.684066+0000","last_active":"2024-04-30T08:32:31.550515+0000","last_peered":"2024-04-30T08:32:31.550515+0000","last_clean":"2024-04-30T08:32:31.550515+0000","last_became_active":"2024-04-30T08:31:51.683699+0000","last_became_peered":"2024-04-30T08:31:51.683699+0000","last_unstale":"2024-04-30T08:32:31.550515+0000","last_undegraded":"2024-04-30T08:32:31.550515+0000","last_fullsized":"2024-04-30T08:32:31.550515+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:36:20.900733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.835032+0000","last_change":"2024-04-30T08:31:52.715219+0000","last_active":"2024-04-30T08:31:59.835032+0000","last_peered":"2024-04-30T08:31:59.835032+0000","last_clean":"2024-04-30T08:31:59.835032+0000","last_became_active":"2024-04-30T08:31:52.714774+0000","last_became_peered":"2024-04-30T08:31:52.714774+0000","last_unstale":"2024-04-30T08:31:59.835032+0000","last_undegraded":"2024-04-30T08:31:59.835032+0000","last_fullsized":"2024-04-30T08:31:59.835032+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:42:41.082610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550306+0000","last_change":"2024-04-30T08:31:58.060845+0000","last_active":"2024-04-30T08:32:31.550306+0000","last_peered":"2024-04-30T08:32:31.550306+0000","last_clean":"2024-04-30T08:32:31.550306+0000","last_became_active":"2024-04-30T08:31:58.060365+0000","last_became_peered":"2024-04-30T08:31:58.060365+0000","last_unstale":"2024-04-30T08:32:31.550306+0000","last_undegraded":"2024-04-30T08:32:31.550306+0000","last_fullsized":"2024-04-30T08:32:31.550306+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:21:39.583243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"81'2","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.547845+0000","last_change":"2024-04-30T08:31:51.694833+0000","last_active":"2024-04-30T08:32:30.547845+0000","last_peered":"2024-04-30T08:32:30.547845+0000","last_clean":"2024-04-30T08:32:30.547845+0000","last_became_active":"2024-04-30T08:31:51.694401+0000","last_became_peered":"2024-04-30T08:31:51.694401+0000","last_unstale":"2024-04-30T08:32:30.547845+0000","last_undegraded":"2024-04-30T08:32:30.547845+0000","last_fullsized":"2024-04-30T08:32:30.547845+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:38:52.605625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.357703+0000","last_change":"2024-04-30T08:31:52.686444+0000","last_active":"2024-04-30T08:31:59.357703+0000","last_peered":"2024-04-30T08:31:59.357703+0000","last_clean":"2024-04-30T08:31:59.357703+0000","last_became_active":"2024-04-30T08:31:52.686057+0000","last_became_peered":"2024-04-30T08:31:52.686057+0000","last_unstale":"2024-04-30T08:31:59.357703+0000","last_undegraded":"2024-04-30T08:31:59.357703+0000","last_fullsized":"2024-04-30T08:31:59.357703+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:54:29.824060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.557873+0000","last_change":"2024-04-30T08:31:57.766193+0000","last_active":"2024-04-30T08:32:30.557873+0000","last_peered":"2024-04-30T08:32:30.557873+0000","last_clean":"2024-04-30T08:32:30.557873+0000","last_became_active":"2024-04-30T08:31:57.765758+0000","last_became_peered":"2024-04-30T08:31:57.765758+0000","last_unstale":"2024-04-30T08:32:30.557873+0000","last_undegraded":"2024-04-30T08:32:30.557873+0000","last_fullsized":"2024-04-30T08:32:30.557873+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:59:57.282795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"81'7","reported_seq":32,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:32:29.554238+0000","last_change":"2024-04-30T08:31:51.704565+0000","last_active":"2024-04-30T08:32:29.554238+0000","last_peered":"2024-04-30T08:32:29.554238+0000","last_clean":"2024-04-30T08:32:29.554238+0000","last_became_active":"2024-04-30T08:31:51.703954+0000","last_became_peered":"2024-04-30T08:31:51.703954+0000","last_unstale":"2024-04-30T08:32:29.554238+0000","last_undegraded":"2024-04-30T08:32:29.554238+0000","last_fullsized":"2024-04-30T08:32:29.554238+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:03:00.517867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548041+0000","last_change":"2024-04-30T08:31:52.706043+0000","last_active":"2024-04-30T08:32:30.548041+0000","last_peered":"2024-04-30T08:32:30.548041+0000","last_clean":"2024-04-30T08:32:30.548041+0000","last_became_active":"2024-04-30T08:31:52.705678+0000","last_became_peered":"2024-04-30T08:31:52.705678+0000","last_unstale":"2024-04-30T08:32:30.548041+0000","last_undegraded":"2024-04-30T08:32:30.548041+0000","last_fullsized":"2024-04-30T08:32:30.548041+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:44:56.787835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"82'1","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550147+0000","last_change":"2024-04-30T08:31:57.764791+0000","last_active":"2024-04-30T08:32:31.550147+0000","last_peered":"2024-04-30T08:32:31.550147+0000","last_clean":"2024-04-30T08:32:31.550147+0000","last_became_active":"2024-04-30T08:31:57.764368+0000","last_became_peered":"2024-04-30T08:31:57.764368+0000","last_unstale":"2024-04-30T08:32:31.550147+0000","last_undegraded":"2024-04-30T08:32:31.550147+0000","last_fullsized":"2024-04-30T08:32:31.550147+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:19:28.545483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"81'3","reported_seq":2,"reported_epoch":82,"state":"peering","last_fresh":"2024-04-30T08:32:30.524198+0000","last_change":"2024-04-30T08:32:30.944913+0000","last_active":"2024-04-30T08:32:30.944913+0000","last_peered":"2024-04-30T08:32:30.524198+0000","last_clean":"2024-04-30T08:32:30.524198+0000","last_became_active":"2024-04-30T08:31:51.702141+0000","last_became_peered":"2024-04-30T08:31:51.702141+0000","last_unstale":"2024-04-30T08:32:30.524198+0000","last_undegraded":"2024-04-30T08:32:30.524198+0000","last_fullsized":"2024-04-30T08:32:30.524198+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.670593+0000","last_change":"2024-04-30T08:31:52.716707+0000","last_active":"2024-04-30T08:31:59.670593+0000","last_peered":"2024-04-30T08:31:59.670593+0000","last_clean":"2024-04-30T08:31:59.670593+0000","last_became_active":"2024-04-30T08:31:52.716293+0000","last_became_peered":"2024-04-30T08:31:52.716293+0000","last_unstale":"2024-04-30T08:31:59.670593+0000","last_undegraded":"2024-04-30T08:31:59.670593+0000","last_fullsized":"2024-04-30T08:31:59.670593+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:29:54.223472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550379+0000","last_change":"2024-04-30T08:31:58.060640+0000","last_active":"2024-04-30T08:32:31.550379+0000","last_peered":"2024-04-30T08:32:31.550379+0000","last_clean":"2024-04-30T08:32:31.550379+0000","last_became_active":"2024-04-30T08:31:58.060061+0000","last_became_peered":"2024-04-30T08:31:58.060061+0000","last_unstale":"2024-04-30T08:32:31.550379+0000","last_undegraded":"2024-04-30T08:32:31.550379+0000","last_fullsized":"2024-04-30T08:32:31.550379+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:46:00.844191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"81'4","reported_seq":28,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:32:26.512050+0000","last_change":"2024-04-30T08:31:51.679532+0000","last_active":"2024-04-30T08:32:26.512050+0000","last_peered":"2024-04-30T08:32:26.512050+0000","last_clean":"2024-04-30T08:32:26.512050+0000","last_became_active":"2024-04-30T08:31:51.679141+0000","last_became_peered":"2024-04-30T08:31:51.679141+0000","last_unstale":"2024-04-30T08:32:26.512050+0000","last_undegraded":"2024-04-30T08:32:26.512050+0000","last_fullsized":"2024-04-30T08:32:26.512050+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:19:50.526563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.557596+0000","last_change":"2024-04-30T08:31:52.703613+0000","last_active":"2024-04-30T08:32:31.557596+0000","last_peered":"2024-04-30T08:32:31.557596+0000","last_clean":"2024-04-30T08:32:31.557596+0000","last_became_active":"2024-04-30T08:31:52.703196+0000","last_became_peered":"2024-04-30T08:31:52.703196+0000","last_unstale":"2024-04-30T08:32:31.557596+0000","last_undegraded":"2024-04-30T08:32:31.557596+0000","last_fullsized":"2024-04-30T08:32:31.557596+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:33:11.136313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.547878+0000","last_change":"2024-04-30T08:31:57.755286+0000","last_active":"2024-04-30T08:32:30.547878+0000","last_peered":"2024-04-30T08:32:30.547878+0000","last_clean":"2024-04-30T08:32:30.547878+0000","last_became_active":"2024-04-30T08:31:57.754786+0000","last_became_peered":"2024-04-30T08:31:57.754786+0000","last_unstale":"2024-04-30T08:32:30.547878+0000","last_undegraded":"2024-04-30T08:32:30.547878+0000","last_fullsized":"2024-04-30T08:32:30.547878+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:02:50.931553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"81'5","reported_seq":29,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:32:28.529719+0000","last_change":"2024-04-30T08:31:51.693317+0000","last_active":"2024-04-30T08:32:28.529719+0000","last_peered":"2024-04-30T08:32:28.529719+0000","last_clean":"2024-04-30T08:32:28.529719+0000","last_became_active":"2024-04-30T08:31:51.692928+0000","last_became_peered":"2024-04-30T08:31:51.692928+0000","last_unstale":"2024-04-30T08:32:28.529719+0000","last_undegraded":"2024-04-30T08:32:28.529719+0000","last_fullsized":"2024-04-30T08:32:28.529719+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:35:09.851735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.558242+0000","last_change":"2024-04-30T08:31:52.694304+0000","last_active":"2024-04-30T08:32:31.558242+0000","last_peered":"2024-04-30T08:32:31.558242+0000","last_clean":"2024-04-30T08:32:31.558242+0000","last_became_active":"2024-04-30T08:31:52.693317+0000","last_became_peered":"2024-04-30T08:31:52.693317+0000","last_unstale":"2024-04-30T08:32:31.558242+0000","last_undegraded":"2024-04-30T08:32:31.558242+0000","last_fullsized":"2024-04-30T08:32:31.558242+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:31:39.294864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834714+0000","last_change":"2024-04-30T08:31:57.763598+0000","last_active":"2024-04-30T08:31:59.834714+0000","last_peered":"2024-04-30T08:31:59.834714+0000","last_clean":"2024-04-30T08:31:59.834714+0000","last_became_active":"2024-04-30T08:31:57.762307+0000","last_became_peered":"2024-04-30T08:31:57.762307+0000","last_unstale":"2024-04-30T08:31:59.834714+0000","last_undegraded":"2024-04-30T08:31:59.834714+0000","last_fullsized":"2024-04-30T08:31:59.834714+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:24:13.390595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"81'3","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.553337+0000","last_change":"2024-04-30T08:31:51.703476+0000","last_active":"2024-04-30T08:32:31.553337+0000","last_peered":"2024-04-30T08:32:31.553337+0000","last_clean":"2024-04-30T08:32:31.553337+0000","last_became_active":"2024-04-30T08:31:51.703072+0000","last_became_peered":"2024-04-30T08:31:51.703072+0000","last_unstale":"2024-04-30T08:32:31.553337+0000","last_undegraded":"2024-04-30T08:32:31.553337+0000","last_fullsized":"2024-04-30T08:32:31.553337+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:07:21.098223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.557803+0000","last_change":"2024-04-30T08:31:52.699223+0000","last_active":"2024-04-30T08:32:31.557803+0000","last_peered":"2024-04-30T08:32:31.557803+0000","last_clean":"2024-04-30T08:32:31.557803+0000","last_became_active":"2024-04-30T08:31:52.698837+0000","last_became_peered":"2024-04-30T08:31:52.698837+0000","last_unstale":"2024-04-30T08:32:31.557803+0000","last_undegraded":"2024-04-30T08:32:31.557803+0000","last_fullsized":"2024-04-30T08:32:31.557803+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:19:11.826470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.835310+0000","last_change":"2024-04-30T08:31:57.737438+0000","last_active":"2024-04-30T08:31:59.835310+0000","last_peered":"2024-04-30T08:31:59.835310+0000","last_clean":"2024-04-30T08:31:59.835310+0000","last_became_active":"2024-04-30T08:31:57.736949+0000","last_became_peered":"2024-04-30T08:31:57.736949+0000","last_unstale":"2024-04-30T08:31:59.835310+0000","last_undegraded":"2024-04-30T08:31:59.835310+0000","last_fullsized":"2024-04-30T08:31:59.835310+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:00:10.921846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"81'5","reported_seq":31,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.554486+0000","last_change":"2024-04-30T08:31:51.708943+0000","last_active":"2024-04-30T08:32:30.554486+0000","last_peered":"2024-04-30T08:32:30.554486+0000","last_clean":"2024-04-30T08:32:30.554486+0000","last_became_active":"2024-04-30T08:31:51.708376+0000","last_became_peered":"2024-04-30T08:31:51.708376+0000","last_unstale":"2024-04-30T08:32:30.554486+0000","last_undegraded":"2024-04-30T08:32:30.554486+0000","last_fullsized":"2024-04-30T08:32:30.554486+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:14:46.692192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.558028+0000","last_change":"2024-04-30T08:31:52.686634+0000","last_active":"2024-04-30T08:32:31.558028+0000","last_peered":"2024-04-30T08:32:31.558028+0000","last_clean":"2024-04-30T08:32:31.558028+0000","last_became_active":"2024-04-30T08:31:52.686231+0000","last_became_peered":"2024-04-30T08:31:52.686231+0000","last_unstale":"2024-04-30T08:32:31.558028+0000","last_undegraded":"2024-04-30T08:32:31.558028+0000","last_fullsized":"2024-04-30T08:32:31.558028+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:27:31.073041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548133+0000","last_change":"2024-04-30T08:31:52.688695+0000","last_active":"2024-04-30T08:32:30.548133+0000","last_peered":"2024-04-30T08:32:30.548133+0000","last_clean":"2024-04-30T08:32:30.548133+0000","last_became_active":"2024-04-30T08:31:52.686556+0000","last_became_peered":"2024-04-30T08:31:52.686556+0000","last_unstale":"2024-04-30T08:32:30.548133+0000","last_undegraded":"2024-04-30T08:32:30.548133+0000","last_fullsized":"2024-04-30T08:32:30.548133+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:22:48.976440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.835365+0000","last_change":"2024-04-30T08:31:52.688324+0000","last_active":"2024-04-30T08:31:59.835365+0000","last_peered":"2024-04-30T08:31:59.835365+0000","last_clean":"2024-04-30T08:31:59.835365+0000","last_became_active":"2024-04-30T08:31:52.687966+0000","last_became_peered":"2024-04-30T08:31:52.687966+0000","last_unstale":"2024-04-30T08:31:59.835365+0000","last_undegraded":"2024-04-30T08:31:59.835365+0000","last_fullsized":"2024-04-30T08:31:59.835365+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:35:38.237750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834334+0000","last_change":"2024-04-30T08:31:52.695638+0000","last_active":"2024-04-30T08:31:59.834334+0000","last_peered":"2024-04-30T08:31:59.834334+0000","last_clean":"2024-04-30T08:31:59.834334+0000","last_became_active":"2024-04-30T08:31:52.695283+0000","last_became_peered":"2024-04-30T08:31:52.695283+0000","last_unstale":"2024-04-30T08:31:59.834334+0000","last_undegraded":"2024-04-30T08:31:59.834334+0000","last_fullsized":"2024-04-30T08:31:59.834334+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:00:14.510381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.834598+0000","last_change":"2024-04-30T08:31:52.719112+0000","last_active":"2024-04-30T08:31:59.834598+0000","last_peered":"2024-04-30T08:31:59.834598+0000","last_clean":"2024-04-30T08:31:59.834598+0000","last_became_active":"2024-04-30T08:31:52.718207+0000","last_became_peered":"2024-04-30T08:31:52.718207+0000","last_unstale":"2024-04-30T08:31:59.834598+0000","last_undegraded":"2024-04-30T08:31:59.834598+0000","last_fullsized":"2024-04-30T08:31:59.834598+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:27:29.263962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.553965+0000","last_change":"2024-04-30T08:31:52.712008+0000","last_active":"2024-04-30T08:32:30.553965+0000","last_peered":"2024-04-30T08:32:30.553965+0000","last_clean":"2024-04-30T08:32:30.553965+0000","last_became_active":"2024-04-30T08:31:52.711064+0000","last_became_peered":"2024-04-30T08:31:52.711064+0000","last_unstale":"2024-04-30T08:32:30.553965+0000","last_undegraded":"2024-04-30T08:32:30.553965+0000","last_fullsized":"2024-04-30T08:32:30.553965+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:57:27.400602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548539+0000","last_change":"2024-04-30T08:31:52.691210+0000","last_active":"2024-04-30T08:32:30.548539+0000","last_peered":"2024-04-30T08:32:30.548539+0000","last_clean":"2024-04-30T08:32:30.548539+0000","last_became_active":"2024-04-30T08:31:52.688852+0000","last_became_peered":"2024-04-30T08:31:52.688852+0000","last_unstale":"2024-04-30T08:32:30.548539+0000","last_undegraded":"2024-04-30T08:32:30.548539+0000","last_fullsized":"2024-04-30T08:32:30.548539+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:04:46.502743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":22,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.835158+0000","last_change":"2024-04-30T08:31:52.684501+0000","last_active":"2024-04-30T08:31:59.835158+0000","last_peered":"2024-04-30T08:31:59.835158+0000","last_clean":"2024-04-30T08:31:59.835158+0000","last_became_active":"2024-04-30T08:31:52.684122+0000","last_became_peered":"2024-04-30T08:31:52.684122+0000","last_unstale":"2024-04-30T08:31:59.835158+0000","last_undegraded":"2024-04-30T08:31:59.835158+0000","last_fullsized":"2024-04-30T08:31:59.835158+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:55:58.093269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.671093+0000","last_change":"2024-04-30T08:31:52.698016+0000","last_active":"2024-04-30T08:31:59.671093+0000","last_peered":"2024-04-30T08:31:59.671093+0000","last_clean":"2024-04-30T08:31:59.671093+0000","last_became_active":"2024-04-30T08:31:52.695037+0000","last_became_peered":"2024-04-30T08:31:52.695037+0000","last_unstale":"2024-04-30T08:31:59.671093+0000","last_undegraded":"2024-04-30T08:31:59.671093+0000","last_fullsized":"2024-04-30T08:31:59.671093+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:06:01.360947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.555895+0000","last_change":"2024-04-30T08:31:52.698415+0000","last_active":"2024-04-30T08:32:31.555895+0000","last_peered":"2024-04-30T08:32:31.555895+0000","last_clean":"2024-04-30T08:32:31.555895+0000","last_became_active":"2024-04-30T08:31:52.697912+0000","last_became_peered":"2024-04-30T08:31:52.697912+0000","last_unstale":"2024-04-30T08:32:31.555895+0000","last_undegraded":"2024-04-30T08:32:31.555895+0000","last_fullsized":"2024-04-30T08:32:31.555895+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:05:44.420331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":23,"reported_epoch":81,"state":"active+clean","last_fresh":"2024-04-30T08:31:59.670020+0000","last_change":"2024-04-30T08:31:52.694776+0000","last_active":"2024-04-30T08:31:59.670020+0000","last_peered":"2024-04-30T08:31:59.670020+0000","last_clean":"2024-04-30T08:31:59.670020+0000","last_became_active":"2024-04-30T08:31:52.693532+0000","last_became_peered":"2024-04-30T08:31:52.693532+0000","last_unstale":"2024-04-30T08:31:59.670020+0000","last_undegraded":"2024-04-30T08:31:59.670020+0000","last_fullsized":"2024-04-30T08:31:59.670020+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:31:10.186667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548026+0000","last_change":"2024-04-30T08:31:52.688477+0000","last_active":"2024-04-30T08:32:30.548026+0000","last_peered":"2024-04-30T08:32:30.548026+0000","last_clean":"2024-04-30T08:32:30.548026+0000","last_became_active":"2024-04-30T08:31:52.688072+0000","last_became_peered":"2024-04-30T08:31:52.688072+0000","last_unstale":"2024-04-30T08:32:30.548026+0000","last_undegraded":"2024-04-30T08:32:30.548026+0000","last_fullsized":"2024-04-30T08:32:30.548026+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:35:42.659707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.553593+0000","last_change":"2024-04-30T08:31:52.716788+0000","last_active":"2024-04-30T08:32:30.553593+0000","last_peered":"2024-04-30T08:32:30.553593+0000","last_clean":"2024-04-30T08:32:30.553593+0000","last_became_active":"2024-04-30T08:31:52.716336+0000","last_became_peered":"2024-04-30T08:31:52.716336+0000","last_unstale":"2024-04-30T08:32:30.553593+0000","last_undegraded":"2024-04-30T08:32:30.553593+0000","last_fullsized":"2024-04-30T08:32:30.553593+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:23:23.477968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548294+0000","last_change":"2024-04-30T08:31:52.684571+0000","last_active":"2024-04-30T08:32:30.548294+0000","last_peered":"2024-04-30T08:32:30.548294+0000","last_clean":"2024-04-30T08:32:30.548294+0000","last_became_active":"2024-04-30T08:31:52.684192+0000","last_became_peered":"2024-04-30T08:31:52.684192+0000","last_unstale":"2024-04-30T08:32:30.548294+0000","last_undegraded":"2024-04-30T08:32:30.548294+0000","last_fullsized":"2024-04-30T08:32:30.548294+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:05:57.520133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":28672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284927,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":10374,"num_objects":91,"num_object_clones":0,"num_object_copies":273,"num_objects_missing_on_primary":0,"num_objects_missing":8,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":91,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":198,"num_write_kb":68,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":61,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":72516,"data_compressed":1841,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},"log_size":174,"ondisk_log_size":174,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678038,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32060,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713092,"statfs":{"total":95995035648,"available":95962206208,"internally_reserved":0,"allocated":901120,"data_stored":450863,"data_compressed":17691,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874266,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":1031905,"data_compressed":50930,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037793,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":925696,"data_stored":450793,"data_compressed":17691,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234025,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32048,"kb_used_data":872,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713104,"statfs":{"total":95995035648,"available":95962218496,"internally_reserved":0,"allocated":892928,"data_stored":436731,"data_compressed":14704,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430254,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":987136,"data_stored":1057399,"data_compressed":56904,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626482,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32044,"kb_used_data":868,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713108,"statfs":{"total":95995035648,"available":95962222592,"internally_reserved":0,"allocated":888832,"data_stored":436715,"data_compressed":14704,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574929,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36248,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708904,"statfs":{"total":95995035648,"available":95957917696,"internally_reserved":0,"allocated":929792,"data_stored":470468,"data_compressed":17691,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738453,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32068,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713084,"statfs":{"total":95995035648,"available":95962198016,"internally_reserved":0,"allocated":913408,"data_stored":438171,"data_compressed":14704,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411404,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32072,"kb_used_data":896,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713080,"statfs":{"total":95995035648,"available":95962193920,"internally_reserved":0,"allocated":917504,"data_stored":442198,"data_compressed":14704,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182469,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27892,"kb_used_data":876,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717260,"statfs":{"total":95995035648,"available":95966474240,"internally_reserved":0,"allocated":897024,"data_stored":436401,"data_compressed":14704,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986240,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":921600,"data_stored":460102,"data_compressed":20678,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822714,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":995328,"data_stored":1052590,"data_compressed":56904,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34599,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34599,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34599,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":39559,"data_compressed":1841,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":12805,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":5544,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-30T08:32:34.877 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.13 is not active+clean 2024-04-30T08:32:34.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.13', 'version': "81'9", 'reported_seq': 2, 'reported_epoch': 82, 'state': 'peering', 'last_fresh': '2024-04-30T08:32:29.552341+0000', 'last_change': '2024-04-30T08:32:30.944941+0000', 'last_active': '2024-04-30T08:32:30.944941+0000', 'last_peered': '2024-04-30T08:32:29.552341+0000', 'last_clean': '2024-04-30T08:32:29.552341+0000', 'last_became_active': '2024-04-30T08:31:51.702122+0000', 'last_became_peered': '2024-04-30T08:31:51.702122+0000', 'last_unstale': '2024-04-30T08:32:29.552341+0000', 'last_undegraded': '2024-04-30T08:32:29.552341+0000', 'last_fullsized': '2024-04-30T08:32:29.552341+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 73, 'last_epoch_clean': 74, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-30T08:31:50.645568+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-30T08:31:50.645568+0000', 'last_clean_scrub_stamp': '2024-04-30T08:31:50.645568+0000', 'objects_scrubbed': 0, 'log_size': 9, 'log_dups_size': 0, 'ondisk_log_size': 9, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 68, 'num_objects': 6, 'num_object_clones': 0, 'num_object_copies': 18, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 6, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 6, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 13, 'num_write_kb': 8, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 4, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [11, 10, 3], 'acting': [11, 10, 3], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 11, 'acting_primary': 11, 'purged_snaps': []} 2024-04-30T08:32:34.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 4.14 is not active+clean 2024-04-30T08:32:34.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '4.14', 'version': "0'0", 'reported_seq': 15, 'reported_epoch': 81, 'state': 'peering', 'last_fresh': '2024-04-30T08:32:00.941417+0000', 'last_change': '2024-04-30T08:32:30.548960+0000', 'last_active': '2024-04-30T08:32:30.548960+0000', 'last_peered': '2024-04-30T08:32:00.941417+0000', 'last_clean': '2024-04-30T08:32:00.941417+0000', 'last_became_active': '2024-04-30T08:31:57.762531+0000', 'last_became_peered': '2024-04-30T08:31:57.762531+0000', 'last_unstale': '2024-04-30T08:32:00.941417+0000', 'last_undegraded': '2024-04-30T08:32:00.941417+0000', 'last_fullsized': '2024-04-30T08:32:00.941417+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 79, 'last_epoch_clean': 80, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-30T08:31:56.685471+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-30T08:31:56.685471+0000', 'last_clean_scrub_stamp': '2024-04-30T08:31:56.685471+0000', 'objects_scrubbed': 0, 'log_size': 0, 'log_dups_size': 0, 'ondisk_log_size': 0, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 0, 'num_object_clones': 0, 'num_object_copies': 0, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 0, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 0, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 0, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 0, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [11, 8, 7, 6], 'acting': [11, 8, 7, 6], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [6, 7, 8, 9], 'up_primary': 11, 'acting_primary': 11, 'purged_snaps': []} 2024-04-30T08:32:34.878 INFO:tasks.cephfs.filesystem.ceph_manager:PG 2.a is not active+clean 2024-04-30T08:32:34.878 INFO:tasks.cephfs.filesystem.ceph_manager:{'pgid': '2.a', 'version': "81'3", 'reported_seq': 2, 'reported_epoch': 82, 'state': 'peering', 'last_fresh': '2024-04-30T08:32:30.524198+0000', 'last_change': '2024-04-30T08:32:30.944913+0000', 'last_active': '2024-04-30T08:32:30.944913+0000', 'last_peered': '2024-04-30T08:32:30.524198+0000', 'last_clean': '2024-04-30T08:32:30.524198+0000', 'last_became_active': '2024-04-30T08:31:51.702141+0000', 'last_became_peered': '2024-04-30T08:31:51.702141+0000', 'last_unstale': '2024-04-30T08:32:30.524198+0000', 'last_undegraded': '2024-04-30T08:32:30.524198+0000', 'last_fullsized': '2024-04-30T08:32:30.524198+0000', 'mapping_epoch': 82, 'log_start': "0'0", 'ondisk_log_start': "0'0", 'created': 73, 'last_epoch_clean': 74, 'parent': '0.0', 'parent_split_bits': 0, 'last_scrub': "0'0", 'last_scrub_stamp': '2024-04-30T08:31:50.645568+0000', 'last_deep_scrub': "0'0", 'last_deep_scrub_stamp': '2024-04-30T08:31:50.645568+0000', 'last_clean_scrub_stamp': '2024-04-30T08:31:50.645568+0000', 'objects_scrubbed': 0, 'log_size': 3, 'log_dups_size': 0, 'ondisk_log_size': 3, 'stats_invalid': False, 'dirty_stats_invalid': False, 'omap_stats_invalid': False, 'hitset_stats_invalid': False, 'hitset_bytes_stats_invalid': False, 'pin_stats_invalid': False, 'manifest_stats_invalid': False, 'snaptrimq_len': 0, 'last_scrub_duration': 0, 'scrub_schedule': 'no scrub is scheduled', 'scrub_duration': 0, 'objects_trimmed': 0, 'snaptrim_duration': 0, 'stat_sum': {'num_bytes': 0, 'num_objects': 2, 'num_object_clones': 0, 'num_object_copies': 6, 'num_objects_missing_on_primary': 0, 'num_objects_missing': 2, 'num_objects_degraded': 0, 'num_objects_misplaced': 0, 'num_objects_unfound': 0, 'num_objects_dirty': 2, 'num_whiteouts': 0, 'num_read': 0, 'num_read_kb': 0, 'num_write': 2, 'num_write_kb': 0, 'num_scrub_errors': 0, 'num_shallow_scrub_errors': 0, 'num_deep_scrub_errors': 0, 'num_objects_recovered': 0, 'num_bytes_recovered': 0, 'num_keys_recovered': 0, 'num_objects_omap': 2, 'num_objects_hit_set_archive': 0, 'num_bytes_hit_set_archive': 0, 'num_flush': 0, 'num_flush_kb': 0, 'num_evict': 0, 'num_evict_kb': 0, 'num_promote': 0, 'num_flush_mode_high': 0, 'num_flush_mode_low': 0, 'num_evict_mode_some': 0, 'num_evict_mode_full': 0, 'num_objects_pinned': 0, 'num_legacy_snapsets': 0, 'num_large_omap_objects': 0, 'num_objects_manifest': 0, 'num_omap_bytes': 0, 'num_omap_keys': 0, 'num_objects_repaired': 0}, 'up': [11, 3, 8], 'acting': [11, 3, 8], 'avail_no_missing': [], 'object_location_counts': [], 'blocked_by': [], 'up_primary': 11, 'acting_primary': 11, 'purged_snaps': []} 2024-04-30T08:32:34.879 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph status --format=json 2024-04-30T08:32:35.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:34 smithi032 ceph-mon[28331]: from='client.24878 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:32:35.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:34 smithi032 ceph-mon[28331]: pgmap v345: 129 pgs: 3 peering, 126 active+clean; 587 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6.4 KiB/s wr, 22 op/s 2024-04-30T08:32:35.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:34 smithi032 ceph-mon[28331]: from='client.24884 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:32:35.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.203+0000 7f8d3fe00640 1 -- 172.21.15.32:0/779355058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d40105bb0 msgr2=0x7f8d401068b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:35.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.203+0000 7f8d3fe00640 1 --2- 172.21.15.32:0/779355058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d40105bb0 0x7f8d401068b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8d30009900 tx=0x7f8d3002a400 comp rx=0 tx=0).stop 2024-04-30T08:32:35.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.204+0000 7f8d3fe00640 1 -- 172.21.15.32:0/779355058 shutdown_connections 2024-04-30T08:32:35.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.204+0000 7f8d3fe00640 1 --2- 172.21.15.32:0/779355058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d4010baf0 0x7f8d4010df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:35.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.204+0000 7f8d3fe00640 1 --2- 172.21.15.32:0/779355058 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d40106df0 0x7f8d40107200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:35.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.204+0000 7f8d3fe00640 1 --2- 172.21.15.32:0/779355058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d40105bb0 0x7f8d401068b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:35.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.204+0000 7f8d3fe00640 1 -- 172.21.15.32:0/779355058 >> 172.21.15.32:0/779355058 conn(0x7f8d40100da0 msgr2=0x7f8d40103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:35.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.204+0000 7f8d3fe00640 1 -- 172.21.15.32:0/779355058 shutdown_connections 2024-04-30T08:32:35.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.204+0000 7f8d3fe00640 1 -- 172.21.15.32:0/779355058 wait complete. 2024-04-30T08:32:35.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.205+0000 7f8d3fe00640 1 Processor -- start 2024-04-30T08:32:35.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.205+0000 7f8d3fe00640 1 -- start start 2024-04-30T08:32:35.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.206+0000 7f8d3fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d40105bb0 0x7f8d4019c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:35.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.206+0000 7f8d3fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d40106df0 0x7f8d4019c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:35.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.206+0000 7f8d3fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d4010baf0 0x7f8d401a37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:35.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.206+0000 7f8d3fe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8d401116e0 con 0x7f8d4010baf0 2024-04-30T08:32:35.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.206+0000 7f8d3fe00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8d40111560 con 0x7f8d40106df0 2024-04-30T08:32:35.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.206+0000 7f8d3fe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8d40111860 con 0x7f8d40105bb0 2024-04-30T08:32:35.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.206+0000 7f8d3ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d40105bb0 0x7f8d4019c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:35.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.206+0000 7f8d3ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d40105bb0 0x7f8d4019c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:40568/0 (socket says 172.21.15.32:40568) 2024-04-30T08:32:35.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.206+0000 7f8d3ea00640 1 -- 172.21.15.32:0/4185045098 learned_addr learned my addr 172.21.15.32:0/4185045098 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:35.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.206+0000 7f8d37e00640 1 --2- 172.21.15.32:0/4185045098 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d40106df0 0x7f8d4019c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:35.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.206+0000 7f8d3f400640 1 --2- 172.21.15.32:0/4185045098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d4010baf0 0x7f8d401a37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:35.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.207+0000 7f8d3ea00640 1 -- 172.21.15.32:0/4185045098 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d40106df0 msgr2=0x7f8d4019c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:35.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.207+0000 7f8d3ea00640 1 --2- 172.21.15.32:0/4185045098 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d40106df0 0x7f8d4019c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:35.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.207+0000 7f8d3ea00640 1 -- 172.21.15.32:0/4185045098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d4010baf0 msgr2=0x7f8d401a37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:35.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.207+0000 7f8d3ea00640 1 --2- 172.21.15.32:0/4185045098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d4010baf0 0x7f8d401a37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:35.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.207+0000 7f8d3ea00640 1 -- 172.21.15.32:0/4185045098 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d401a3e20 con 0x7f8d40105bb0 2024-04-30T08:32:35.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.207+0000 7f8d3f400640 1 --2- 172.21.15.32:0/4185045098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d4010baf0 0x7f8d401a37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:32:35.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.207+0000 7f8d3ea00640 1 --2- 172.21.15.32:0/4185045098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d40105bb0 0x7f8d4019c2e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8d30031440 tx=0x7f8d30003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:35.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.208+0000 7f8d37400640 1 -- 172.21.15.32:0/4185045098 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d3003f070 con 0x7f8d40105bb0 2024-04-30T08:32:35.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.208+0000 7f8d37400640 1 -- 172.21.15.32:0/4185045098 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d30031c50 con 0x7f8d40105bb0 2024-04-30T08:32:35.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.208+0000 7f8d37400640 1 -- 172.21.15.32:0/4185045098 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d30035030 con 0x7f8d40105bb0 2024-04-30T08:32:35.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.208+0000 7f8d3fe00640 1 -- 172.21.15.32:0/4185045098 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d401a40b0 con 0x7f8d40105bb0 2024-04-30T08:32:35.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.208+0000 7f8d3fe00640 1 -- 172.21.15.32:0/4185045098 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d401a45f0 con 0x7f8d40105bb0 2024-04-30T08:32:35.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.209+0000 7f8d3fe00640 1 -- 172.21.15.32:0/4185045098 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d4010a900 con 0x7f8d40105bb0 2024-04-30T08:32:35.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.211+0000 7f8d37400640 1 -- 172.21.15.32:0/4185045098 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f8d30042070 con 0x7f8d40105bb0 2024-04-30T08:32:35.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.213+0000 7f8d37400640 1 --2- 172.21.15.32:0/4185045098 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8d200afbf0 0x7f8d200b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:35.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.213+0000 7f8d37400640 1 -- 172.21.15.32:0/4185045098 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f8d300ffbc0 con 0x7f8d40105bb0 2024-04-30T08:32:35.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.213+0000 7f8d37e00640 1 --2- 172.21.15.32:0/4185045098 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8d200afbf0 0x7f8d200b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:35.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.214+0000 7f8d37e00640 1 --2- 172.21.15.32:0/4185045098 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8d200afbf0 0x7f8d200b2100 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8d4019d380 tx=0x7f8d28005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:35.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.215+0000 7f8d37400640 1 -- 172.21.15.32:0/4185045098 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8d300c7c30 con 0x7f8d40105bb0 2024-04-30T08:32:35.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:34 smithi071 ceph-mon[33385]: from='client.24878 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:32:35.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:34 smithi071 ceph-mon[33385]: pgmap v345: 129 pgs: 3 peering, 126 active+clean; 587 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6.4 KiB/s wr, 22 op/s 2024-04-30T08:32:35.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:34 smithi071 ceph-mon[33385]: from='client.24884 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:32:35.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:34 smithi189 ceph-mon[33038]: from='client.24878 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:32:35.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:34 smithi189 ceph-mon[33038]: pgmap v345: 129 pgs: 3 peering, 126 active+clean; 587 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6.4 KiB/s wr, 22 op/s 2024-04-30T08:32:35.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:34 smithi189 ceph-mon[33038]: from='client.24884 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:32:35.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.507+0000 7f8d3fe00640 1 -- 172.21.15.32:0/4185045098 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "status", "format": "json"} v 0) -- 0x7f8d4002dbd0 con 0x7f8d40105bb0 2024-04-30T08:32:35.510 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.509+0000 7f8d37400640 1 -- 172.21.15.32:0/4185045098 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json"}]=0 v0) ==== 72+0+2559 (secure 0 0 0) 0x7f8d300d0300 con 0x7f8d40105bb0 2024-04-30T08:32:35.510 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:32:35.511 INFO:teuthology.orchestra.run.smithi032.stdout:{"fsid":"98c0a670-06ca-11ef-bc93-c7b262605968","health":{"status":"HEALTH_OK","checks":{},"mutes":[]},"election_epoch":16,"quorum":[0,1,2],"quorum_names":["a","c","b"],"quorum_age":472,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":83,"num_osds":12,"num_up_osds":12,"osd_up_since":1714465850,"num_in_osds":12,"osd_in_since":1714465830,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":126},{"state_name":"peering","count":3}],"num_pgs":129,"num_pools":4,"num_objects":95,"data_bytes":600976,"bytes_used":377225216,"bytes_avail":1151563202560,"bytes_total":1151940427776,"inactive_pgs_ratio":0.023255813866853714,"read_bytes_sec":2430,"write_bytes_sec":6524,"read_op_per_sec":3,"write_op_per_sec":18},"fsmap":{"epoch":27,"id":1,"up":5,"in":5,"max":5,"by_rank":[{"filesystem_id":1,"rank":4,"name":"l","status":"up:standby-replay","gid":24442},{"filesystem_id":1,"rank":3,"name":"k","status":"up:standby-replay","gid":24458},{"filesystem_id":1,"rank":2,"name":"d","status":"up:standby-replay","gid":24484},{"filesystem_id":1,"rank":1,"name":"f","status":"up:standby-replay","gid":24505},{"filesystem_id":1,"rank":0,"name":"e","status":"up:standby-replay","gid":24506},{"filesystem_id":1,"rank":4,"name":"c","status":"up:active","gid":24517},{"filesystem_id":1,"rank":3,"name":"h","status":"up:active","gid":24533},{"filesystem_id":1,"rank":2,"name":"i","status":"up:active","gid":24541},{"filesystem_id":1,"rank":1,"name":"g","status":"up:active","gid":24560},{"filesystem_id":1,"rank":0,"name":"b","status":"up:active","gid":24590}],"up:standby":2},"mgrmap":{"available":true,"num_standbys":2,"modules":["cephadm","dashboard","iostat","nfs","restful"],"services":{"dashboard":"https://172.21.15.32:8443/"}},"servicemap":{"epoch":25,"modified":"2024-04-30T08:32:29.712804+0000","services":{"mds":{"daemons":{"summary":"","a":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"b":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"g":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"h":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}},"i":{"start_epoch":0,"start_stamp":"0.000000","gid":0,"addr":"(unrecognized address family 0)/0","metadata":{},"task_status":{}}}}}},"progress_events":{}} 2024-04-30T08:32:35.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.511+0000 7f8d3fe00640 1 -- 172.21.15.32:0/4185045098 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8d200afbf0 msgr2=0x7f8d200b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:35.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.511+0000 7f8d3fe00640 1 --2- 172.21.15.32:0/4185045098 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8d200afbf0 0x7f8d200b2100 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f8d4019d380 tx=0x7f8d28005f90 comp rx=0 tx=0).stop 2024-04-30T08:32:35.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.511+0000 7f8d3fe00640 1 -- 172.21.15.32:0/4185045098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d40105bb0 msgr2=0x7f8d4019c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:35.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.511+0000 7f8d3fe00640 1 --2- 172.21.15.32:0/4185045098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d40105bb0 0x7f8d4019c2e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8d30031440 tx=0x7f8d30003c10 comp rx=0 tx=0).stop 2024-04-30T08:32:35.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.512+0000 7f8d3fe00640 1 -- 172.21.15.32:0/4185045098 shutdown_connections 2024-04-30T08:32:35.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.512+0000 7f8d3fe00640 1 --2- 172.21.15.32:0/4185045098 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8d200afbf0 0x7f8d200b2100 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:35.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.512+0000 7f8d3fe00640 1 --2- 172.21.15.32:0/4185045098 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d4010baf0 0x7f8d401a37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:35.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.512+0000 7f8d3fe00640 1 --2- 172.21.15.32:0/4185045098 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d40106df0 0x7f8d4019c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:35.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.512+0000 7f8d3fe00640 1 --2- 172.21.15.32:0/4185045098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d40105bb0 0x7f8d4019c2e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:35.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.512+0000 7f8d3fe00640 1 -- 172.21.15.32:0/4185045098 >> 172.21.15.32:0/4185045098 conn(0x7f8d40100da0 msgr2=0x7f8d40107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:35.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.512+0000 7f8d3fe00640 1 -- 172.21.15.32:0/4185045098 shutdown_connections 2024-04-30T08:32:35.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.512+0000 7f8d3fe00640 1 -- 172.21.15.32:0/4185045098 wait complete. 2024-04-30T08:32:35.541 INFO:tasks.cephfs.filesystem.ceph_manager:no progress seen, keeping timeout for now 2024-04-30T08:32:35.541 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-30T08:32:35.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.817+0000 7f614a400640 1 -- 172.21.15.32:0/1114300732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6144109fc0 msgr2=0x7f614410acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:35.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.817+0000 7f614a400640 1 --2- 172.21.15.32:0/1114300732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6144109fc0 0x7f614410acc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f613c009ca0 tx=0x7f613c02a8c0 comp rx=0 tx=0).stop 2024-04-30T08:32:35.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.817+0000 7f614a400640 1 -- 172.21.15.32:0/1114300732 shutdown_connections 2024-04-30T08:32:35.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.817+0000 7f614a400640 1 --2- 172.21.15.32:0/1114300732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f614410ff00 0x7f6144112320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:35.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.817+0000 7f614a400640 1 --2- 172.21.15.32:0/1114300732 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f614410b200 0x7f614410b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:35.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.817+0000 7f614a400640 1 --2- 172.21.15.32:0/1114300732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6144109fc0 0x7f614410acc0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:35.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.817+0000 7f614a400640 1 -- 172.21.15.32:0/1114300732 >> 172.21.15.32:0/1114300732 conn(0x7f6144078bb0 msgr2=0x7f6144107a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:35.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.817+0000 7f614a400640 1 -- 172.21.15.32:0/1114300732 shutdown_connections 2024-04-30T08:32:35.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.817+0000 7f614a400640 1 -- 172.21.15.32:0/1114300732 wait complete. 2024-04-30T08:32:35.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.818+0000 7f614a400640 1 Processor -- start 2024-04-30T08:32:35.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.819+0000 7f614a400640 1 -- start start 2024-04-30T08:32:35.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.819+0000 7f614a400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6144109fc0 0x7f61441a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:35.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.819+0000 7f614a400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f614410b200 0x7f61441a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:35.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.819+0000 7f614a400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f614410ff00 0x7f61441a7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:35.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.819+0000 7f614a400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6144115ae0 con 0x7f614410ff00 2024-04-30T08:32:35.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.819+0000 7f614a400640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6144115960 con 0x7f614410b200 2024-04-30T08:32:35.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.819+0000 7f614a400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6144115c60 con 0x7f6144109fc0 2024-04-30T08:32:35.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.819+0000 7f6143e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f614410b200 0x7f61441a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:35.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.819+0000 7f6149a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f614410ff00 0x7f61441a7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:35.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.819+0000 7f6143e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f614410b200 0x7f61441a0c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:59750/0 (socket says 172.21.15.32:59750) 2024-04-30T08:32:35.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.819+0000 7f6149a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f614410ff00 0x7f61441a7b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45300/0 (socket says 172.21.15.32:45300) 2024-04-30T08:32:35.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.819+0000 7f6143e00640 1 -- 172.21.15.32:0/99958633 learned_addr learned my addr 172.21.15.32:0/99958633 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:35.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.819+0000 7f6149000640 1 --2- 172.21.15.32:0/99958633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6144109fc0 0x7f61441a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:35.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.820+0000 7f6143e00640 1 -- 172.21.15.32:0/99958633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6144109fc0 msgr2=0x7f61441a06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:35.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.820+0000 7f6143e00640 1 --2- 172.21.15.32:0/99958633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6144109fc0 0x7f61441a06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:35.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.820+0000 7f6143e00640 1 -- 172.21.15.32:0/99958633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f614410ff00 msgr2=0x7f61441a7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:35.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.820+0000 7f6143e00640 1 --2- 172.21.15.32:0/99958633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f614410ff00 0x7f61441a7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:35.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.820+0000 7f6143e00640 1 -- 172.21.15.32:0/99958633 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61441a8200 con 0x7f614410b200 2024-04-30T08:32:35.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.820+0000 7f6149a00640 1 --2- 172.21.15.32:0/99958633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f614410ff00 0x7f61441a7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:32:35.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.820+0000 7f6143e00640 1 --2- 172.21.15.32:0/99958633 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f614410b200 0x7f61441a0c00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f612c00e9f0 tx=0x7f612c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:35.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.821+0000 7f6141600640 1 -- 172.21.15.32:0/99958633 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f612c00cd80 con 0x7f614410b200 2024-04-30T08:32:35.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.821+0000 7f6141600640 1 -- 172.21.15.32:0/99958633 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f612c0108e0 con 0x7f614410b200 2024-04-30T08:32:35.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.821+0000 7f614a400640 1 -- 172.21.15.32:0/99958633 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61441a84f0 con 0x7f614410b200 2024-04-30T08:32:35.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.821+0000 7f6141600640 1 -- 172.21.15.32:0/99958633 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f612c018980 con 0x7f614410b200 2024-04-30T08:32:35.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.821+0000 7f614a400640 1 -- 172.21.15.32:0/99958633 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61441a8a40 con 0x7f614410b200 2024-04-30T08:32:35.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.822+0000 7f614a400640 1 -- 172.21.15.32:0/99958633 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f614410ed10 con 0x7f614410b200 2024-04-30T08:32:35.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.826+0000 7f6141600640 1 -- 172.21.15.32:0/99958633 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f612c010430 con 0x7f614410b200 2024-04-30T08:32:35.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.828+0000 7f6141600640 1 --2- 172.21.15.32:0/99958633 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f61240afb70 0x7f61240b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:35.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.828+0000 7f6141600640 1 -- 172.21.15.32:0/99958633 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f612c0df950 con 0x7f614410b200 2024-04-30T08:32:35.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.828+0000 7f6149000640 1 --2- 172.21.15.32:0/99958633 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f61240afb70 0x7f61240b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:35.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.828+0000 7f6141600640 1 -- 172.21.15.32:0/99958633 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f612c0a79c0 con 0x7f614410b200 2024-04-30T08:32:35.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:35.829+0000 7f6149000640 1 --2- 172.21.15.32:0/99958633 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f61240afb70 0x7f61240b2080 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f613c000f80 tx=0x7f613c004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:36.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:36.034+0000 7f614a400640 1 -- 172.21.15.32:0/99958633 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6144063060 con 0x7f61240afb70 2024-04-30T08:32:36.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:36.046+0000 7f6141600640 1 -- 172.21.15.32:0/99958633 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340643 (secure 0 0 0) 0x7f6144063060 con 0x7f61240afb70 2024-04-30T08:32:36.052 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:32:36.054 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-30T08:32:36.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:36.053+0000 7f614a400640 1 -- 172.21.15.32:0/99958633 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f61240afb70 msgr2=0x7f61240b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:36.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:36.053+0000 7f614a400640 1 --2- 172.21.15.32:0/99958633 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f61240afb70 0x7f61240b2080 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f613c000f80 tx=0x7f613c004a40 comp rx=0 tx=0).stop 2024-04-30T08:32:36.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:36.053+0000 7f614a400640 1 -- 172.21.15.32:0/99958633 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f614410b200 msgr2=0x7f61441a0c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:36.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:36.053+0000 7f614a400640 1 --2- 172.21.15.32:0/99958633 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f614410b200 0x7f61441a0c00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f612c00e9f0 tx=0x7f612c00eeb0 comp rx=0 tx=0).stop 2024-04-30T08:32:36.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:36.053+0000 7f614a400640 1 -- 172.21.15.32:0/99958633 shutdown_connections 2024-04-30T08:32:36.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:36.053+0000 7f614a400640 1 --2- 172.21.15.32:0/99958633 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f61240afb70 0x7f61240b2080 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:36.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:36.053+0000 7f614a400640 1 --2- 172.21.15.32:0/99958633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f614410ff00 0x7f61441a7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:36.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:36.053+0000 7f614a400640 1 --2- 172.21.15.32:0/99958633 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f614410b200 0x7f61441a0c00 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:36.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:36.053+0000 7f614a400640 1 --2- 172.21.15.32:0/99958633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6144109fc0 0x7f61441a06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:36.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:36.053+0000 7f614a400640 1 -- 172.21.15.32:0/99958633 >> 172.21.15.32:0/99958633 conn(0x7f6144078bb0 msgr2=0x7f614410bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:36.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:36.053+0000 7f614a400640 1 -- 172.21.15.32:0/99958633 shutdown_connections 2024-04-30T08:32:36.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:36.053+0000 7f614a400640 1 -- 172.21.15.32:0/99958633 wait complete. 2024-04-30T08:32:36.083 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2024-04-30T08:32:35.715334+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":639712,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":8,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":139,"num_read_kb":149,"num_write":350,"num_write_kb":2265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":292,"ondisk_log_size":292,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":421,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":368660,"kb_used_data":11116,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124573164,"statfs":{"total":1151940427776,"available":1151562919936,"internally_reserved":0,"allocated":11382784,"data_stored":7359233,"data_compressed":351535,"data_compressed_allocated":475136,"data_compressed_original":2449408,"omap_allocated":19141,"internal_metadata":365933883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":44704,"num_objects":78,"num_object_clones":0,"num_object_copies":236,"num_objects_missing_on_primary":0,"num_objects_missing":8,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":78,"num_whiteouts":0,"num_read":63,"num_read_kb":85,"num_write":178,"num_write_kb":96,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":52,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004086"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548872+0000","last_change":"2024-04-30T08:31:52.711443+0000","last_active":"2024-04-30T08:32:30.548872+0000","last_peered":"2024-04-30T08:32:30.548872+0000","last_clean":"2024-04-30T08:32:30.548872+0000","last_became_active":"2024-04-30T08:31:52.711029+0000","last_became_peered":"2024-04-30T08:31:52.711029+0000","last_unstale":"2024-04-30T08:32:30.548872+0000","last_undegraded":"2024-04-30T08:32:30.548872+0000","last_fullsized":"2024-04-30T08:32:30.548872+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:51:15.208528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.559664+0000","last_change":"2024-04-30T08:31:52.712324+0000","last_active":"2024-04-30T08:32:31.559664+0000","last_peered":"2024-04-30T08:32:31.559664+0000","last_clean":"2024-04-30T08:32:31.559664+0000","last_became_active":"2024-04-30T08:31:52.711923+0000","last_became_peered":"2024-04-30T08:31:52.711923+0000","last_unstale":"2024-04-30T08:32:31.559664+0000","last_undegraded":"2024-04-30T08:32:31.559664+0000","last_fullsized":"2024-04-30T08:32:31.559664+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:29:55.071459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.557006+0000","last_change":"2024-04-30T08:31:52.712936+0000","last_active":"2024-04-30T08:32:31.557006+0000","last_peered":"2024-04-30T08:32:31.557006+0000","last_clean":"2024-04-30T08:32:31.557006+0000","last_became_active":"2024-04-30T08:31:52.712585+0000","last_became_peered":"2024-04-30T08:31:52.712585+0000","last_unstale":"2024-04-30T08:32:31.557006+0000","last_undegraded":"2024-04-30T08:32:31.557006+0000","last_fullsized":"2024-04-30T08:32:31.557006+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:13:43.745561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550312+0000","last_change":"2024-04-30T08:31:52.686880+0000","last_active":"2024-04-30T08:32:31.550312+0000","last_peered":"2024-04-30T08:32:31.550312+0000","last_clean":"2024-04-30T08:32:31.550312+0000","last_became_active":"2024-04-30T08:31:52.686506+0000","last_became_peered":"2024-04-30T08:31:52.686506+0000","last_unstale":"2024-04-30T08:32:31.550312+0000","last_undegraded":"2024-04-30T08:32:31.550312+0000","last_fullsized":"2024-04-30T08:32:31.550312+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:16:33.390951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.555472+0000","last_change":"2024-04-30T08:31:52.706014+0000","last_active":"2024-04-30T08:32:31.555472+0000","last_peered":"2024-04-30T08:32:31.555472+0000","last_clean":"2024-04-30T08:32:31.555472+0000","last_became_active":"2024-04-30T08:31:52.705582+0000","last_became_peered":"2024-04-30T08:31:52.705582+0000","last_unstale":"2024-04-30T08:32:31.555472+0000","last_undegraded":"2024-04-30T08:32:31.555472+0000","last_fullsized":"2024-04-30T08:32:31.555472+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:24:49.804592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.552434+0000","last_change":"2024-04-30T08:31:52.690462+0000","last_active":"2024-04-30T08:32:31.552434+0000","last_peered":"2024-04-30T08:32:31.552434+0000","last_clean":"2024-04-30T08:32:31.552434+0000","last_became_active":"2024-04-30T08:31:52.690092+0000","last_became_peered":"2024-04-30T08:31:52.690092+0000","last_unstale":"2024-04-30T08:32:31.552434+0000","last_undegraded":"2024-04-30T08:32:31.552434+0000","last_fullsized":"2024-04-30T08:32:31.552434+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:32:49.769383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.557763+0000","last_change":"2024-04-30T08:31:52.701786+0000","last_active":"2024-04-30T08:32:31.557763+0000","last_peered":"2024-04-30T08:32:31.557763+0000","last_clean":"2024-04-30T08:32:31.557763+0000","last_became_active":"2024-04-30T08:31:52.701402+0000","last_became_peered":"2024-04-30T08:31:52.701402+0000","last_unstale":"2024-04-30T08:32:31.557763+0000","last_undegraded":"2024-04-30T08:32:31.557763+0000","last_fullsized":"2024-04-30T08:32:31.557763+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:58:06.153479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.567481+0000","last_change":"2024-04-30T08:31:52.698037+0000","last_active":"2024-04-30T08:32:31.567481+0000","last_peered":"2024-04-30T08:32:31.567481+0000","last_clean":"2024-04-30T08:32:31.567481+0000","last_became_active":"2024-04-30T08:31:52.697646+0000","last_became_peered":"2024-04-30T08:31:52.697646+0000","last_unstale":"2024-04-30T08:32:31.567481+0000","last_undegraded":"2024-04-30T08:32:31.567481+0000","last_fullsized":"2024-04-30T08:32:31.567481+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:28:59.520091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548406+0000","last_change":"2024-04-30T08:31:52.680377+0000","last_active":"2024-04-30T08:32:30.548406+0000","last_peered":"2024-04-30T08:32:30.548406+0000","last_clean":"2024-04-30T08:32:30.548406+0000","last_became_active":"2024-04-30T08:31:52.679977+0000","last_became_peered":"2024-04-30T08:31:52.679977+0000","last_unstale":"2024-04-30T08:32:30.548406+0000","last_undegraded":"2024-04-30T08:32:30.548406+0000","last_fullsized":"2024-04-30T08:32:30.548406+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:35:22.488450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.573546+0000","last_change":"2024-04-30T08:32:31.573546+0000","last_active":"2024-04-30T08:32:31.573546+0000","last_peered":"2024-04-30T08:32:31.573546+0000","last_clean":"2024-04-30T08:32:31.573546+0000","last_became_active":"2024-04-30T08:32:31.573289+0000","last_became_peered":"2024-04-30T08:32:31.573289+0000","last_unstale":"2024-04-30T08:32:31.573546+0000","last_undegraded":"2024-04-30T08:32:31.573546+0000","last_fullsized":"2024-04-30T08:32:31.573546+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:42:45.862326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.558068+0000","last_change":"2024-04-30T08:31:52.690344+0000","last_active":"2024-04-30T08:32:31.558068+0000","last_peered":"2024-04-30T08:32:31.558068+0000","last_clean":"2024-04-30T08:32:31.558068+0000","last_became_active":"2024-04-30T08:31:52.689857+0000","last_became_peered":"2024-04-30T08:31:52.689857+0000","last_unstale":"2024-04-30T08:32:31.558068+0000","last_undegraded":"2024-04-30T08:32:31.558068+0000","last_fullsized":"2024-04-30T08:32:31.558068+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:45:13.395589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550256+0000","last_change":"2024-04-30T08:31:52.685151+0000","last_active":"2024-04-30T08:32:31.550256+0000","last_peered":"2024-04-30T08:32:31.550256+0000","last_clean":"2024-04-30T08:32:31.550256+0000","last_became_active":"2024-04-30T08:31:52.684722+0000","last_became_peered":"2024-04-30T08:31:52.684722+0000","last_unstale":"2024-04-30T08:32:31.550256+0000","last_undegraded":"2024-04-30T08:32:31.550256+0000","last_fullsized":"2024-04-30T08:32:31.550256+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:14:00.660478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.553049+0000","last_change":"2024-04-30T08:31:52.714294+0000","last_active":"2024-04-30T08:32:31.553049+0000","last_peered":"2024-04-30T08:32:31.553049+0000","last_clean":"2024-04-30T08:32:31.553049+0000","last_became_active":"2024-04-30T08:31:52.710094+0000","last_became_peered":"2024-04-30T08:31:52.710094+0000","last_unstale":"2024-04-30T08:32:31.553049+0000","last_undegraded":"2024-04-30T08:32:31.553049+0000","last_fullsized":"2024-04-30T08:32:31.553049+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:34:44.203093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.556588+0000","last_change":"2024-04-30T08:31:52.707992+0000","last_active":"2024-04-30T08:32:31.556588+0000","last_peered":"2024-04-30T08:32:31.556588+0000","last_clean":"2024-04-30T08:32:31.556588+0000","last_became_active":"2024-04-30T08:31:52.706456+0000","last_became_peered":"2024-04-30T08:31:52.706456+0000","last_unstale":"2024-04-30T08:32:31.556588+0000","last_undegraded":"2024-04-30T08:32:31.556588+0000","last_fullsized":"2024-04-30T08:32:31.556588+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:45:32.717842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.556180+0000","last_change":"2024-04-30T08:31:52.695605+0000","last_active":"2024-04-30T08:32:31.556180+0000","last_peered":"2024-04-30T08:32:31.556180+0000","last_clean":"2024-04-30T08:32:31.556180+0000","last_became_active":"2024-04-30T08:31:52.692439+0000","last_became_peered":"2024-04-30T08:31:52.692439+0000","last_unstale":"2024-04-30T08:32:31.556180+0000","last_undegraded":"2024-04-30T08:32:31.556180+0000","last_fullsized":"2024-04-30T08:32:31.556180+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:39:38.746567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.547727+0000","last_change":"2024-04-30T08:31:52.707011+0000","last_active":"2024-04-30T08:32:30.547727+0000","last_peered":"2024-04-30T08:32:30.547727+0000","last_clean":"2024-04-30T08:32:30.547727+0000","last_became_active":"2024-04-30T08:31:52.706602+0000","last_became_peered":"2024-04-30T08:31:52.706602+0000","last_unstale":"2024-04-30T08:32:30.547727+0000","last_undegraded":"2024-04-30T08:32:30.547727+0000","last_fullsized":"2024-04-30T08:32:30.547727+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:43:13.922905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548723+0000","last_change":"2024-04-30T08:31:52.711439+0000","last_active":"2024-04-30T08:32:30.548723+0000","last_peered":"2024-04-30T08:32:30.548723+0000","last_clean":"2024-04-30T08:32:30.548723+0000","last_became_active":"2024-04-30T08:31:52.711023+0000","last_became_peered":"2024-04-30T08:31:52.711023+0000","last_unstale":"2024-04-30T08:32:30.548723+0000","last_undegraded":"2024-04-30T08:32:30.548723+0000","last_fullsized":"2024-04-30T08:32:30.548723+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:31:00.654348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.558525+0000","last_change":"2024-04-30T08:31:52.708476+0000","last_active":"2024-04-30T08:32:30.558525+0000","last_peered":"2024-04-30T08:32:30.558525+0000","last_clean":"2024-04-30T08:32:30.558525+0000","last_became_active":"2024-04-30T08:31:52.704735+0000","last_became_peered":"2024-04-30T08:31:52.704735+0000","last_unstale":"2024-04-30T08:32:30.558525+0000","last_undegraded":"2024-04-30T08:32:30.558525+0000","last_fullsized":"2024-04-30T08:32:30.558525+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:40:00.011783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550883+0000","last_change":"2024-04-30T08:31:52.704901+0000","last_active":"2024-04-30T08:32:31.550883+0000","last_peered":"2024-04-30T08:32:31.550883+0000","last_clean":"2024-04-30T08:32:31.550883+0000","last_became_active":"2024-04-30T08:31:52.704636+0000","last_became_peered":"2024-04-30T08:31:52.704636+0000","last_unstale":"2024-04-30T08:32:31.550883+0000","last_undegraded":"2024-04-30T08:32:31.550883+0000","last_fullsized":"2024-04-30T08:32:31.550883+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:08:53.580809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550486+0000","last_change":"2024-04-30T08:31:57.743342+0000","last_active":"2024-04-30T08:32:31.550486+0000","last_peered":"2024-04-30T08:32:31.550486+0000","last_clean":"2024-04-30T08:32:31.550486+0000","last_became_active":"2024-04-30T08:31:57.742850+0000","last_became_peered":"2024-04-30T08:31:57.742850+0000","last_unstale":"2024-04-30T08:32:31.550486+0000","last_undegraded":"2024-04-30T08:32:31.550486+0000","last_fullsized":"2024-04-30T08:32:31.550486+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:59:10.628822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"81'13","reported_seq":40,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.558153+0000","last_change":"2024-04-30T08:31:51.694984+0000","last_active":"2024-04-30T08:32:30.558153+0000","last_peered":"2024-04-30T08:32:30.558153+0000","last_clean":"2024-04-30T08:32:30.558153+0000","last_became_active":"2024-04-30T08:31:51.694514+0000","last_became_peered":"2024-04-30T08:31:51.694514+0000","last_unstale":"2024-04-30T08:32:30.558153+0000","last_undegraded":"2024-04-30T08:32:30.558153+0000","last_fullsized":"2024-04-30T08:32:30.558153+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:17:03.873090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.565234+0000","last_change":"2024-04-30T08:31:52.694276+0000","last_active":"2024-04-30T08:32:31.565234+0000","last_peered":"2024-04-30T08:32:31.565234+0000","last_clean":"2024-04-30T08:32:31.565234+0000","last_became_active":"2024-04-30T08:31:52.693839+0000","last_became_peered":"2024-04-30T08:31:52.693839+0000","last_unstale":"2024-04-30T08:32:31.565234+0000","last_undegraded":"2024-04-30T08:32:31.565234+0000","last_fullsized":"2024-04-30T08:32:31.565234+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:29:21.271040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.549942+0000","last_change":"2024-04-30T08:31:57.754828+0000","last_active":"2024-04-30T08:32:31.549942+0000","last_peered":"2024-04-30T08:32:31.549942+0000","last_clean":"2024-04-30T08:32:31.549942+0000","last_became_active":"2024-04-30T08:31:57.754322+0000","last_became_peered":"2024-04-30T08:31:57.754322+0000","last_unstale":"2024-04-30T08:32:31.549942+0000","last_undegraded":"2024-04-30T08:32:31.549942+0000","last_fullsized":"2024-04-30T08:32:31.549942+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:36:02.560134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"81'2","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.567668+0000","last_change":"2024-04-30T08:31:51.707637+0000","last_active":"2024-04-30T08:32:31.567668+0000","last_peered":"2024-04-30T08:32:31.567668+0000","last_clean":"2024-04-30T08:32:31.567668+0000","last_became_active":"2024-04-30T08:31:51.707233+0000","last_became_peered":"2024-04-30T08:31:51.707233+0000","last_unstale":"2024-04-30T08:32:31.567668+0000","last_undegraded":"2024-04-30T08:32:31.567668+0000","last_fullsized":"2024-04-30T08:32:31.567668+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:14:46.041169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.560581+0000","last_change":"2024-04-30T08:31:52.711442+0000","last_active":"2024-04-30T08:32:31.560581+0000","last_peered":"2024-04-30T08:32:31.560581+0000","last_clean":"2024-04-30T08:32:31.560581+0000","last_became_active":"2024-04-30T08:31:52.710942+0000","last_became_peered":"2024-04-30T08:31:52.710942+0000","last_unstale":"2024-04-30T08:32:31.560581+0000","last_undegraded":"2024-04-30T08:32:31.560581+0000","last_fullsized":"2024-04-30T08:32:31.560581+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:26:56.428770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550129+0000","last_change":"2024-04-30T08:31:57.760535+0000","last_active":"2024-04-30T08:32:31.550129+0000","last_peered":"2024-04-30T08:32:31.550129+0000","last_clean":"2024-04-30T08:32:31.550129+0000","last_became_active":"2024-04-30T08:31:57.760073+0000","last_became_peered":"2024-04-30T08:31:57.760073+0000","last_unstale":"2024-04-30T08:32:31.550129+0000","last_undegraded":"2024-04-30T08:32:31.550129+0000","last_fullsized":"2024-04-30T08:32:31.550129+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:03:51.359474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"81'12","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.557144+0000","last_change":"2024-04-30T08:31:51.678575+0000","last_active":"2024-04-30T08:32:31.557144+0000","last_peered":"2024-04-30T08:32:31.557144+0000","last_clean":"2024-04-30T08:32:31.557144+0000","last_became_active":"2024-04-30T08:31:51.678114+0000","last_became_peered":"2024-04-30T08:31:51.678114+0000","last_unstale":"2024-04-30T08:32:31.557144+0000","last_undegraded":"2024-04-30T08:32:31.557144+0000","last_fullsized":"2024-04-30T08:32:31.557144+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:10:35.517183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548286+0000","last_change":"2024-04-30T08:31:52.704137+0000","last_active":"2024-04-30T08:32:30.548286+0000","last_peered":"2024-04-30T08:32:30.548286+0000","last_clean":"2024-04-30T08:32:30.548286+0000","last_became_active":"2024-04-30T08:31:52.703695+0000","last_became_peered":"2024-04-30T08:31:52.703695+0000","last_unstale":"2024-04-30T08:32:30.548286+0000","last_undegraded":"2024-04-30T08:32:30.548286+0000","last_fullsized":"2024-04-30T08:32:30.548286+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:31:05.461106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.549998+0000","last_change":"2024-04-30T08:31:57.755237+0000","last_active":"2024-04-30T08:32:31.549998+0000","last_peered":"2024-04-30T08:32:31.549998+0000","last_clean":"2024-04-30T08:32:31.549998+0000","last_became_active":"2024-04-30T08:31:57.754438+0000","last_became_peered":"2024-04-30T08:31:57.754438+0000","last_unstale":"2024-04-30T08:32:31.549998+0000","last_undegraded":"2024-04-30T08:32:31.549998+0000","last_fullsized":"2024-04-30T08:32:31.549998+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:20:59.035110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"81'5","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.555740+0000","last_change":"2024-04-30T08:31:51.673793+0000","last_active":"2024-04-30T08:32:31.555740+0000","last_peered":"2024-04-30T08:32:31.555740+0000","last_clean":"2024-04-30T08:32:31.555740+0000","last_became_active":"2024-04-30T08:31:51.672703+0000","last_became_peered":"2024-04-30T08:31:51.672703+0000","last_unstale":"2024-04-30T08:32:31.555740+0000","last_undegraded":"2024-04-30T08:32:31.555740+0000","last_fullsized":"2024-04-30T08:32:31.555740+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:13:18.959542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.553233+0000","last_change":"2024-04-30T08:31:52.694320+0000","last_active":"2024-04-30T08:32:31.553233+0000","last_peered":"2024-04-30T08:32:31.553233+0000","last_clean":"2024-04-30T08:32:31.553233+0000","last_became_active":"2024-04-30T08:31:52.694032+0000","last_became_peered":"2024-04-30T08:31:52.694032+0000","last_unstale":"2024-04-30T08:32:31.553233+0000","last_undegraded":"2024-04-30T08:32:31.553233+0000","last_fullsized":"2024-04-30T08:32:31.553233+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:33:12.347369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550575+0000","last_change":"2024-04-30T08:31:57.739984+0000","last_active":"2024-04-30T08:32:31.550575+0000","last_peered":"2024-04-30T08:32:31.550575+0000","last_clean":"2024-04-30T08:32:31.550575+0000","last_became_active":"2024-04-30T08:31:57.739477+0000","last_became_peered":"2024-04-30T08:31:57.739477+0000","last_unstale":"2024-04-30T08:32:31.550575+0000","last_undegraded":"2024-04-30T08:32:31.550575+0000","last_fullsized":"2024-04-30T08:32:31.550575+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:10:11.325419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"81'6","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.564820+0000","last_change":"2024-04-30T08:31:51.678792+0000","last_active":"2024-04-30T08:32:31.564820+0000","last_peered":"2024-04-30T08:32:31.564820+0000","last_clean":"2024-04-30T08:32:31.564820+0000","last_became_active":"2024-04-30T08:31:51.678094+0000","last_became_peered":"2024-04-30T08:31:51.678094+0000","last_unstale":"2024-04-30T08:32:31.564820+0000","last_undegraded":"2024-04-30T08:32:31.564820+0000","last_fullsized":"2024-04-30T08:32:31.564820+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:03:17.526016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.572984+0000","last_change":"2024-04-30T08:32:31.572984+0000","last_active":"2024-04-30T08:32:31.572984+0000","last_peered":"2024-04-30T08:32:31.572984+0000","last_clean":"2024-04-30T08:32:31.572984+0000","last_became_active":"2024-04-30T08:32:31.572536+0000","last_became_peered":"2024-04-30T08:32:31.572536+0000","last_unstale":"2024-04-30T08:32:31.572984+0000","last_undegraded":"2024-04-30T08:32:31.572984+0000","last_fullsized":"2024-04-30T08:32:31.572984+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:46:15.884555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550816+0000","last_change":"2024-04-30T08:31:57.765047+0000","last_active":"2024-04-30T08:32:31.550816+0000","last_peered":"2024-04-30T08:32:31.550816+0000","last_clean":"2024-04-30T08:32:31.550816+0000","last_became_active":"2024-04-30T08:31:57.764519+0000","last_became_peered":"2024-04-30T08:31:57.764519+0000","last_unstale":"2024-04-30T08:32:31.550816+0000","last_undegraded":"2024-04-30T08:32:31.550816+0000","last_fullsized":"2024-04-30T08:32:31.550816+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:06:16.126396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"81'7","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.552381+0000","last_change":"2024-04-30T08:31:51.686403+0000","last_active":"2024-04-30T08:32:31.552381+0000","last_peered":"2024-04-30T08:32:31.552381+0000","last_clean":"2024-04-30T08:32:31.552381+0000","last_became_active":"2024-04-30T08:31:51.686096+0000","last_became_peered":"2024-04-30T08:31:51.686096+0000","last_unstale":"2024-04-30T08:32:31.552381+0000","last_undegraded":"2024-04-30T08:32:31.552381+0000","last_fullsized":"2024-04-30T08:32:31.552381+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:47:05.961930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.554940+0000","last_change":"2024-04-30T08:31:52.698512+0000","last_active":"2024-04-30T08:32:31.554940+0000","last_peered":"2024-04-30T08:32:31.554940+0000","last_clean":"2024-04-30T08:32:31.554940+0000","last_became_active":"2024-04-30T08:31:52.698168+0000","last_became_peered":"2024-04-30T08:31:52.698168+0000","last_unstale":"2024-04-30T08:32:31.554940+0000","last_undegraded":"2024-04-30T08:32:31.554940+0000","last_fullsized":"2024-04-30T08:32:31.554940+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:22:45.094365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548550+0000","last_change":"2024-04-30T08:31:57.761430+0000","last_active":"2024-04-30T08:32:30.548550+0000","last_peered":"2024-04-30T08:32:30.548550+0000","last_clean":"2024-04-30T08:32:30.548550+0000","last_became_active":"2024-04-30T08:31:57.761062+0000","last_became_peered":"2024-04-30T08:31:57.761062+0000","last_unstale":"2024-04-30T08:32:30.548550+0000","last_undegraded":"2024-04-30T08:32:30.548550+0000","last_fullsized":"2024-04-30T08:32:30.548550+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:32:05.403786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"81'3","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.552142+0000","last_change":"2024-04-30T08:31:51.686694+0000","last_active":"2024-04-30T08:32:31.552142+0000","last_peered":"2024-04-30T08:32:31.552142+0000","last_clean":"2024-04-30T08:32:31.552142+0000","last_became_active":"2024-04-30T08:31:51.685773+0000","last_became_peered":"2024-04-30T08:31:51.685773+0000","last_unstale":"2024-04-30T08:32:31.552142+0000","last_undegraded":"2024-04-30T08:32:31.552142+0000","last_fullsized":"2024-04-30T08:32:31.552142+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:44:25.327977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.567591+0000","last_change":"2024-04-30T08:31:52.709794+0000","last_active":"2024-04-30T08:32:31.567591+0000","last_peered":"2024-04-30T08:32:31.567591+0000","last_clean":"2024-04-30T08:32:31.567591+0000","last_became_active":"2024-04-30T08:31:52.709371+0000","last_became_peered":"2024-04-30T08:31:52.709371+0000","last_unstale":"2024-04-30T08:32:31.567591+0000","last_undegraded":"2024-04-30T08:32:31.567591+0000","last_fullsized":"2024-04-30T08:32:31.567591+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:36:59.892551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.559308+0000","last_change":"2024-04-30T08:31:57.757224+0000","last_active":"2024-04-30T08:32:31.559308+0000","last_peered":"2024-04-30T08:32:31.559308+0000","last_clean":"2024-04-30T08:32:31.559308+0000","last_became_active":"2024-04-30T08:31:57.756658+0000","last_became_peered":"2024-04-30T08:31:57.756658+0000","last_unstale":"2024-04-30T08:32:31.559308+0000","last_undegraded":"2024-04-30T08:32:31.559308+0000","last_fullsized":"2024-04-30T08:32:31.559308+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:01:21.309322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"81'3","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.555826+0000","last_change":"2024-04-30T08:31:51.678944+0000","last_active":"2024-04-30T08:32:31.555826+0000","last_peered":"2024-04-30T08:32:31.555826+0000","last_clean":"2024-04-30T08:32:31.555826+0000","last_became_active":"2024-04-30T08:31:51.678493+0000","last_became_peered":"2024-04-30T08:31:51.678493+0000","last_unstale":"2024-04-30T08:32:31.555826+0000","last_undegraded":"2024-04-30T08:32:31.555826+0000","last_fullsized":"2024-04-30T08:32:31.555826+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:48:11.111466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.555177+0000","last_change":"2024-04-30T08:31:52.707627+0000","last_active":"2024-04-30T08:32:30.555177+0000","last_peered":"2024-04-30T08:32:30.555177+0000","last_clean":"2024-04-30T08:32:30.555177+0000","last_became_active":"2024-04-30T08:31:52.706441+0000","last_became_peered":"2024-04-30T08:31:52.706441+0000","last_unstale":"2024-04-30T08:32:30.555177+0000","last_undegraded":"2024-04-30T08:32:30.555177+0000","last_fullsized":"2024-04-30T08:32:30.555177+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:15:45.768927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.565083+0000","last_change":"2024-04-30T08:31:57.740324+0000","last_active":"2024-04-30T08:32:31.565083+0000","last_peered":"2024-04-30T08:32:31.565083+0000","last_clean":"2024-04-30T08:32:31.565083+0000","last_became_active":"2024-04-30T08:31:57.739464+0000","last_became_peered":"2024-04-30T08:31:57.739464+0000","last_unstale":"2024-04-30T08:32:31.565083+0000","last_undegraded":"2024-04-30T08:32:31.565083+0000","last_fullsized":"2024-04-30T08:32:31.565083+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:11:37.395545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"81'4","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:33.567649+0000","last_change":"2024-04-30T08:31:51.695283+0000","last_active":"2024-04-30T08:32:33.567649+0000","last_peered":"2024-04-30T08:32:33.567649+0000","last_clean":"2024-04-30T08:32:33.567649+0000","last_became_active":"2024-04-30T08:31:51.694848+0000","last_became_peered":"2024-04-30T08:31:51.694848+0000","last_unstale":"2024-04-30T08:32:33.567649+0000","last_undegraded":"2024-04-30T08:32:33.567649+0000","last_fullsized":"2024-04-30T08:32:33.567649+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:41:38.638034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548438+0000","last_change":"2024-04-30T08:31:52.693594+0000","last_active":"2024-04-30T08:32:30.548438+0000","last_peered":"2024-04-30T08:32:30.548438+0000","last_clean":"2024-04-30T08:32:30.548438+0000","last_became_active":"2024-04-30T08:31:52.693199+0000","last_became_peered":"2024-04-30T08:31:52.693199+0000","last_unstale":"2024-04-30T08:32:30.548438+0000","last_undegraded":"2024-04-30T08:32:30.548438+0000","last_fullsized":"2024-04-30T08:32:30.548438+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:57:49.908617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.553991+0000","last_change":"2024-04-30T08:31:57.749727+0000","last_active":"2024-04-30T08:32:30.553991+0000","last_peered":"2024-04-30T08:32:30.553991+0000","last_clean":"2024-04-30T08:32:30.553991+0000","last_became_active":"2024-04-30T08:31:57.749267+0000","last_became_peered":"2024-04-30T08:31:57.749267+0000","last_unstale":"2024-04-30T08:32:30.553991+0000","last_undegraded":"2024-04-30T08:32:30.553991+0000","last_fullsized":"2024-04-30T08:32:30.553991+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:26:12.683008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"81'4","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.557872+0000","last_change":"2024-04-30T08:31:51.677398+0000","last_active":"2024-04-30T08:32:31.557872+0000","last_peered":"2024-04-30T08:32:31.557872+0000","last_clean":"2024-04-30T08:32:31.557872+0000","last_became_active":"2024-04-30T08:31:51.677031+0000","last_became_peered":"2024-04-30T08:31:51.677031+0000","last_unstale":"2024-04-30T08:32:31.557872+0000","last_undegraded":"2024-04-30T08:32:31.557872+0000","last_fullsized":"2024-04-30T08:32:31.557872+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:24:25.154229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.561554+0000","last_change":"2024-04-30T08:32:31.561554+0000","last_active":"2024-04-30T08:32:31.561554+0000","last_peered":"2024-04-30T08:32:31.561554+0000","last_clean":"2024-04-30T08:32:31.561554+0000","last_became_active":"2024-04-30T08:32:31.560253+0000","last_became_peered":"2024-04-30T08:32:31.560253+0000","last_unstale":"2024-04-30T08:32:31.561554+0000","last_undegraded":"2024-04-30T08:32:31.561554+0000","last_fullsized":"2024-04-30T08:32:31.561554+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:59:08.602282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.586746+0000","last_change":"2024-04-30T08:32:31.586746+0000","last_active":"2024-04-30T08:32:31.586746+0000","last_peered":"2024-04-30T08:32:31.586746+0000","last_clean":"2024-04-30T08:32:31.586746+0000","last_became_active":"2024-04-30T08:32:31.586356+0000","last_became_peered":"2024-04-30T08:32:31.586356+0000","last_unstale":"2024-04-30T08:32:31.586746+0000","last_undegraded":"2024-04-30T08:32:31.586746+0000","last_fullsized":"2024-04-30T08:32:31.586746+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:22:39.371508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"81'5","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.553126+0000","last_change":"2024-04-30T08:31:51.694424+0000","last_active":"2024-04-30T08:32:31.553126+0000","last_peered":"2024-04-30T08:32:31.553126+0000","last_clean":"2024-04-30T08:32:31.553126+0000","last_became_active":"2024-04-30T08:31:51.694081+0000","last_became_peered":"2024-04-30T08:31:51.694081+0000","last_unstale":"2024-04-30T08:32:31.553126+0000","last_undegraded":"2024-04-30T08:32:31.553126+0000","last_fullsized":"2024-04-30T08:32:31.553126+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:27:33.328477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548544+0000","last_change":"2024-04-30T08:31:52.684739+0000","last_active":"2024-04-30T08:32:30.548544+0000","last_peered":"2024-04-30T08:32:30.548544+0000","last_clean":"2024-04-30T08:32:30.548544+0000","last_became_active":"2024-04-30T08:31:52.684316+0000","last_became_peered":"2024-04-30T08:31:52.684316+0000","last_unstale":"2024-04-30T08:32:30.548544+0000","last_undegraded":"2024-04-30T08:32:30.548544+0000","last_fullsized":"2024-04-30T08:32:30.548544+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:12:05.437493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.558429+0000","last_change":"2024-04-30T08:32:31.557578+0000","last_active":"2024-04-30T08:32:31.558429+0000","last_peered":"2024-04-30T08:32:31.558429+0000","last_clean":"2024-04-30T08:32:31.558429+0000","last_became_active":"2024-04-30T08:32:31.557115+0000","last_became_peered":"2024-04-30T08:32:31.557115+0000","last_unstale":"2024-04-30T08:32:31.558429+0000","last_undegraded":"2024-04-30T08:32:31.558429+0000","last_fullsized":"2024-04-30T08:32:31.558429+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:41:33.225879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"82'26","reported_seq":64,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:33.585605+0000","last_change":"2024-04-30T08:31:51.676223+0000","last_active":"2024-04-30T08:32:33.585605+0000","last_peered":"2024-04-30T08:32:33.585605+0000","last_clean":"2024-04-30T08:32:33.585605+0000","last_became_active":"2024-04-30T08:31:51.675830+0000","last_became_peered":"2024-04-30T08:31:51.675830+0000","last_unstale":"2024-04-30T08:32:33.585605+0000","last_undegraded":"2024-04-30T08:32:33.585605+0000","last_fullsized":"2024-04-30T08:32:33.585605+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:28:49.497719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41906,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":45,"num_write":28,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.556767+0000","last_change":"2024-04-30T08:31:52.686892+0000","last_active":"2024-04-30T08:32:31.556767+0000","last_peered":"2024-04-30T08:32:31.556767+0000","last_clean":"2024-04-30T08:32:31.556767+0000","last_became_active":"2024-04-30T08:31:52.686533+0000","last_became_peered":"2024-04-30T08:31:52.686533+0000","last_unstale":"2024-04-30T08:32:31.556767+0000","last_undegraded":"2024-04-30T08:32:31.556767+0000","last_fullsized":"2024-04-30T08:32:31.556767+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:50:18.836161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.553606+0000","last_change":"2024-04-30T08:31:57.752865+0000","last_active":"2024-04-30T08:32:31.553606+0000","last_peered":"2024-04-30T08:32:31.553606+0000","last_clean":"2024-04-30T08:32:31.553606+0000","last_became_active":"2024-04-30T08:31:57.752385+0000","last_became_peered":"2024-04-30T08:31:57.752385+0000","last_unstale":"2024-04-30T08:32:31.553606+0000","last_undegraded":"2024-04-30T08:32:31.553606+0000","last_fullsized":"2024-04-30T08:32:31.553606+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:15:15.315580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"81'9","reported_seq":2,"reported_epoch":82,"state":"peering","last_fresh":"2024-04-30T08:32:29.552341+0000","last_change":"2024-04-30T08:32:30.944941+0000","last_active":"2024-04-30T08:32:30.944941+0000","last_peered":"2024-04-30T08:32:29.552341+0000","last_clean":"2024-04-30T08:32:29.552341+0000","last_became_active":"2024-04-30T08:31:51.702122+0000","last_became_peered":"2024-04-30T08:31:51.702122+0000","last_unstale":"2024-04-30T08:32:29.552341+0000","last_undegraded":"2024-04-30T08:32:29.552341+0000","last_fullsized":"2024-04-30T08:32:29.552341+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":6,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.552561+0000","last_change":"2024-04-30T08:31:52.706259+0000","last_active":"2024-04-30T08:32:31.552561+0000","last_peered":"2024-04-30T08:32:31.552561+0000","last_clean":"2024-04-30T08:32:31.552561+0000","last_became_active":"2024-04-30T08:31:52.705620+0000","last_became_peered":"2024-04-30T08:31:52.705620+0000","last_unstale":"2024-04-30T08:32:31.552561+0000","last_undegraded":"2024-04-30T08:32:31.552561+0000","last_fullsized":"2024-04-30T08:32:31.552561+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:22:29.816950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":15,"reported_epoch":81,"state":"peering","last_fresh":"2024-04-30T08:32:00.941417+0000","last_change":"2024-04-30T08:32:30.548960+0000","last_active":"2024-04-30T08:32:30.548960+0000","last_peered":"2024-04-30T08:32:00.941417+0000","last_clean":"2024-04-30T08:32:00.941417+0000","last_became_active":"2024-04-30T08:31:57.762531+0000","last_became_peered":"2024-04-30T08:31:57.762531+0000","last_unstale":"2024-04-30T08:32:00.941417+0000","last_undegraded":"2024-04-30T08:32:00.941417+0000","last_fullsized":"2024-04-30T08:32:00.941417+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[6,7,8,9],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"81'5","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.554892+0000","last_change":"2024-04-30T08:31:51.701122+0000","last_active":"2024-04-30T08:32:31.554892+0000","last_peered":"2024-04-30T08:32:31.554892+0000","last_clean":"2024-04-30T08:32:31.554892+0000","last_became_active":"2024-04-30T08:31:51.700815+0000","last_became_peered":"2024-04-30T08:31:51.700815+0000","last_unstale":"2024-04-30T08:32:31.554892+0000","last_undegraded":"2024-04-30T08:32:31.554892+0000","last_fullsized":"2024-04-30T08:32:31.554892+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:33:12.714979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.558463+0000","last_change":"2024-04-30T08:31:52.716043+0000","last_active":"2024-04-30T08:32:31.558463+0000","last_peered":"2024-04-30T08:32:31.558463+0000","last_clean":"2024-04-30T08:32:31.558463+0000","last_became_active":"2024-04-30T08:31:52.715284+0000","last_became_peered":"2024-04-30T08:31:52.715284+0000","last_unstale":"2024-04-30T08:32:31.558463+0000","last_undegraded":"2024-04-30T08:32:31.558463+0000","last_fullsized":"2024-04-30T08:32:31.558463+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:00:01.863606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.557749+0000","last_change":"2024-04-30T08:31:57.765008+0000","last_active":"2024-04-30T08:32:30.557749+0000","last_peered":"2024-04-30T08:32:30.557749+0000","last_clean":"2024-04-30T08:32:30.557749+0000","last_became_active":"2024-04-30T08:31:57.762958+0000","last_became_peered":"2024-04-30T08:31:57.762958+0000","last_unstale":"2024-04-30T08:32:30.557749+0000","last_undegraded":"2024-04-30T08:32:30.557749+0000","last_fullsized":"2024-04-30T08:32:30.557749+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:48:47.116004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"81'6","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:32.572977+0000","last_change":"2024-04-30T08:31:51.676681+0000","last_active":"2024-04-30T08:32:32.572977+0000","last_peered":"2024-04-30T08:32:32.572977+0000","last_clean":"2024-04-30T08:32:32.572977+0000","last_became_active":"2024-04-30T08:31:51.676311+0000","last_became_peered":"2024-04-30T08:31:51.676311+0000","last_unstale":"2024-04-30T08:32:32.572977+0000","last_undegraded":"2024-04-30T08:32:32.572977+0000","last_fullsized":"2024-04-30T08:32:32.572977+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:24:49.757427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.558213+0000","last_change":"2024-04-30T08:31:52.689418+0000","last_active":"2024-04-30T08:32:31.558213+0000","last_peered":"2024-04-30T08:32:31.558213+0000","last_clean":"2024-04-30T08:32:31.558213+0000","last_became_active":"2024-04-30T08:31:52.688840+0000","last_became_peered":"2024-04-30T08:31:52.688840+0000","last_unstale":"2024-04-30T08:32:31.558213+0000","last_undegraded":"2024-04-30T08:32:31.558213+0000","last_fullsized":"2024-04-30T08:32:31.558213+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:40:05.781137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.551981+0000","last_change":"2024-04-30T08:31:57.765875+0000","last_active":"2024-04-30T08:32:31.551981+0000","last_peered":"2024-04-30T08:32:31.551981+0000","last_clean":"2024-04-30T08:32:31.551981+0000","last_became_active":"2024-04-30T08:31:57.765432+0000","last_became_peered":"2024-04-30T08:31:57.765432+0000","last_unstale":"2024-04-30T08:32:31.551981+0000","last_undegraded":"2024-04-30T08:32:31.551981+0000","last_fullsized":"2024-04-30T08:32:31.551981+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:24:44.329532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"81'17","reported_seq":46,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.554050+0000","last_change":"2024-04-30T08:31:51.695007+0000","last_active":"2024-04-30T08:32:31.554050+0000","last_peered":"2024-04-30T08:32:31.554050+0000","last_clean":"2024-04-30T08:32:31.554050+0000","last_became_active":"2024-04-30T08:31:51.694533+0000","last_became_peered":"2024-04-30T08:31:51.694533+0000","last_unstale":"2024-04-30T08:32:31.554050+0000","last_undegraded":"2024-04-30T08:32:31.554050+0000","last_fullsized":"2024-04-30T08:32:31.554050+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:47:15.104172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.553520+0000","last_change":"2024-04-30T08:31:52.692274+0000","last_active":"2024-04-30T08:32:31.553520+0000","last_peered":"2024-04-30T08:32:31.553520+0000","last_clean":"2024-04-30T08:32:31.553520+0000","last_became_active":"2024-04-30T08:31:52.691809+0000","last_became_peered":"2024-04-30T08:31:52.691809+0000","last_unstale":"2024-04-30T08:32:31.553520+0000","last_undegraded":"2024-04-30T08:32:31.553520+0000","last_fullsized":"2024-04-30T08:32:31.553520+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:50:00.201782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.555077+0000","last_change":"2024-04-30T08:31:57.732383+0000","last_active":"2024-04-30T08:32:31.555077+0000","last_peered":"2024-04-30T08:32:31.555077+0000","last_clean":"2024-04-30T08:32:31.555077+0000","last_became_active":"2024-04-30T08:31:57.731893+0000","last_became_peered":"2024-04-30T08:31:57.731893+0000","last_unstale":"2024-04-30T08:32:31.555077+0000","last_undegraded":"2024-04-30T08:32:31.555077+0000","last_fullsized":"2024-04-30T08:32:31.555077+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:08:55.243526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"81'2","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.556534+0000","last_change":"2024-04-30T08:31:51.671991+0000","last_active":"2024-04-30T08:32:31.556534+0000","last_peered":"2024-04-30T08:32:31.556534+0000","last_clean":"2024-04-30T08:32:31.556534+0000","last_became_active":"2024-04-30T08:31:51.671508+0000","last_became_peered":"2024-04-30T08:31:51.671508+0000","last_unstale":"2024-04-30T08:32:31.556534+0000","last_undegraded":"2024-04-30T08:32:31.556534+0000","last_fullsized":"2024-04-30T08:32:31.556534+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:54:30.106748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548288+0000","last_change":"2024-04-30T08:31:52.687555+0000","last_active":"2024-04-30T08:32:30.548288+0000","last_peered":"2024-04-30T08:32:30.548288+0000","last_clean":"2024-04-30T08:32:30.548288+0000","last_became_active":"2024-04-30T08:31:52.686000+0000","last_became_peered":"2024-04-30T08:31:52.686000+0000","last_unstale":"2024-04-30T08:32:30.548288+0000","last_undegraded":"2024-04-30T08:32:30.548288+0000","last_fullsized":"2024-04-30T08:32:30.548288+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:37:03.529774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.555216+0000","last_change":"2024-04-30T08:31:58.061077+0000","last_active":"2024-04-30T08:32:31.555216+0000","last_peered":"2024-04-30T08:32:31.555216+0000","last_clean":"2024-04-30T08:32:31.555216+0000","last_became_active":"2024-04-30T08:31:58.060486+0000","last_became_peered":"2024-04-30T08:31:58.060486+0000","last_unstale":"2024-04-30T08:32:31.555216+0000","last_undegraded":"2024-04-30T08:32:31.555216+0000","last_fullsized":"2024-04-30T08:32:31.555216+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:31:50.936560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"81'1","reported_seq":27,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.557573+0000","last_change":"2024-04-30T08:31:51.702869+0000","last_active":"2024-04-30T08:32:30.557573+0000","last_peered":"2024-04-30T08:32:30.557573+0000","last_clean":"2024-04-30T08:32:30.557573+0000","last_became_active":"2024-04-30T08:31:51.701150+0000","last_became_peered":"2024-04-30T08:31:51.701150+0000","last_unstale":"2024-04-30T08:32:30.557573+0000","last_undegraded":"2024-04-30T08:32:30.557573+0000","last_fullsized":"2024-04-30T08:32:30.557573+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:47:11.010310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.552471+0000","last_change":"2024-04-30T08:31:52.709159+0000","last_active":"2024-04-30T08:32:31.552471+0000","last_peered":"2024-04-30T08:32:31.552471+0000","last_clean":"2024-04-30T08:32:31.552471+0000","last_became_active":"2024-04-30T08:31:52.708799+0000","last_became_peered":"2024-04-30T08:31:52.708799+0000","last_unstale":"2024-04-30T08:32:31.552471+0000","last_undegraded":"2024-04-30T08:32:31.552471+0000","last_fullsized":"2024-04-30T08:32:31.552471+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:51:10.588599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.553255+0000","last_change":"2024-04-30T08:31:57.751652+0000","last_active":"2024-04-30T08:32:31.553255+0000","last_peered":"2024-04-30T08:32:31.553255+0000","last_clean":"2024-04-30T08:32:31.553255+0000","last_became_active":"2024-04-30T08:31:57.751176+0000","last_became_peered":"2024-04-30T08:31:57.751176+0000","last_unstale":"2024-04-30T08:32:31.553255+0000","last_undegraded":"2024-04-30T08:32:31.553255+0000","last_fullsized":"2024-04-30T08:32:31.553255+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:13:24.528842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"81'5","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.571485+0000","last_change":"2024-04-30T08:31:51.700816+0000","last_active":"2024-04-30T08:32:31.571485+0000","last_peered":"2024-04-30T08:32:31.571485+0000","last_clean":"2024-04-30T08:32:31.571485+0000","last_became_active":"2024-04-30T08:31:51.700470+0000","last_became_peered":"2024-04-30T08:31:51.700470+0000","last_unstale":"2024-04-30T08:32:31.571485+0000","last_undegraded":"2024-04-30T08:32:31.571485+0000","last_fullsized":"2024-04-30T08:32:31.571485+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:49:11.793412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.552011+0000","last_change":"2024-04-30T08:31:52.715236+0000","last_active":"2024-04-30T08:32:31.552011+0000","last_peered":"2024-04-30T08:32:31.552011+0000","last_clean":"2024-04-30T08:32:31.552011+0000","last_became_active":"2024-04-30T08:31:52.714801+0000","last_became_peered":"2024-04-30T08:31:52.714801+0000","last_unstale":"2024-04-30T08:32:31.552011+0000","last_undegraded":"2024-04-30T08:32:31.552011+0000","last_fullsized":"2024-04-30T08:32:31.552011+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:43:09.650002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.553679+0000","last_change":"2024-04-30T08:31:58.060927+0000","last_active":"2024-04-30T08:32:31.553679+0000","last_peered":"2024-04-30T08:32:31.553679+0000","last_clean":"2024-04-30T08:32:31.553679+0000","last_became_active":"2024-04-30T08:31:58.060182+0000","last_became_peered":"2024-04-30T08:31:58.060182+0000","last_unstale":"2024-04-30T08:32:31.553679+0000","last_undegraded":"2024-04-30T08:32:31.553679+0000","last_fullsized":"2024-04-30T08:32:31.553679+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:21:00.228064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.560485+0000","last_change":"2024-04-30T08:32:31.560485+0000","last_active":"2024-04-30T08:32:31.560485+0000","last_peered":"2024-04-30T08:32:31.560485+0000","last_clean":"2024-04-30T08:32:31.560485+0000","last_became_active":"2024-04-30T08:32:31.560066+0000","last_became_peered":"2024-04-30T08:32:31.560066+0000","last_unstale":"2024-04-30T08:32:31.560485+0000","last_undegraded":"2024-04-30T08:32:31.560485+0000","last_fullsized":"2024-04-30T08:32:31.560485+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:13:40.769761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"81'5","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:33.574285+0000","last_change":"2024-04-30T08:31:51.702808+0000","last_active":"2024-04-30T08:32:33.574285+0000","last_peered":"2024-04-30T08:32:33.574285+0000","last_clean":"2024-04-30T08:32:33.574285+0000","last_became_active":"2024-04-30T08:31:51.702326+0000","last_became_peered":"2024-04-30T08:31:51.702326+0000","last_unstale":"2024-04-30T08:32:33.574285+0000","last_undegraded":"2024-04-30T08:32:33.574285+0000","last_fullsized":"2024-04-30T08:32:33.574285+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:11:35.552670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"81'1","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.557866+0000","last_change":"2024-04-30T08:31:57.760359+0000","last_active":"2024-04-30T08:32:31.557866+0000","last_peered":"2024-04-30T08:32:31.557866+0000","last_clean":"2024-04-30T08:32:31.557866+0000","last_became_active":"2024-04-30T08:31:57.759821+0000","last_became_peered":"2024-04-30T08:31:57.759821+0000","last_unstale":"2024-04-30T08:32:31.557866+0000","last_undegraded":"2024-04-30T08:32:31.557866+0000","last_fullsized":"2024-04-30T08:32:31.557866+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:50:17.853791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.566198+0000","last_change":"2024-04-30T08:31:52.689281+0000","last_active":"2024-04-30T08:32:31.566198+0000","last_peered":"2024-04-30T08:32:31.566198+0000","last_clean":"2024-04-30T08:32:31.566198+0000","last_became_active":"2024-04-30T08:31:52.688834+0000","last_became_peered":"2024-04-30T08:31:52.688834+0000","last_unstale":"2024-04-30T08:32:31.566198+0000","last_undegraded":"2024-04-30T08:32:31.566198+0000","last_fullsized":"2024-04-30T08:32:31.566198+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:42:12.836649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'6","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550187+0000","last_change":"2024-04-30T08:31:51.683712+0000","last_active":"2024-04-30T08:32:31.550187+0000","last_peered":"2024-04-30T08:32:31.550187+0000","last_clean":"2024-04-30T08:32:31.550187+0000","last_became_active":"2024-04-30T08:31:51.683298+0000","last_became_peered":"2024-04-30T08:31:51.683298+0000","last_unstale":"2024-04-30T08:32:31.550187+0000","last_undegraded":"2024-04-30T08:32:31.550187+0000","last_fullsized":"2024-04-30T08:32:31.550187+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:43:08.832104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":103,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.557511+0000","last_change":"2024-04-30T08:30:26.503221+0000","last_active":"2024-04-30T08:32:31.557511+0000","last_peered":"2024-04-30T08:32:31.557511+0000","last_clean":"2024-04-30T08:32:31.557511+0000","last_became_active":"2024-04-30T08:30:26.498399+0000","last_became_peered":"2024-04-30T08:30:26.498399+0000","last_unstale":"2024-04-30T08:32:31.557511+0000","last_undegraded":"2024-04-30T08:32:31.557511+0000","last_fullsized":"2024-04-30T08:32:31.557511+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:26:20.170075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:26:20.170075+0000","last_clean_scrub_stamp":"2024-04-30T08:26:20.170075+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:34:44.067560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.552597+0000","last_change":"2024-04-30T08:31:58.061283+0000","last_active":"2024-04-30T08:32:31.552597+0000","last_peered":"2024-04-30T08:32:31.552597+0000","last_clean":"2024-04-30T08:32:31.552597+0000","last_became_active":"2024-04-30T08:31:58.060650+0000","last_became_peered":"2024-04-30T08:31:58.060650+0000","last_unstale":"2024-04-30T08:32:31.552597+0000","last_undegraded":"2024-04-30T08:32:31.552597+0000","last_fullsized":"2024-04-30T08:32:31.552597+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:13:01.811462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.551872+0000","last_change":"2024-04-30T08:31:52.699639+0000","last_active":"2024-04-30T08:32:31.551872+0000","last_peered":"2024-04-30T08:32:31.551872+0000","last_clean":"2024-04-30T08:32:31.551872+0000","last_became_active":"2024-04-30T08:31:52.699164+0000","last_became_peered":"2024-04-30T08:31:52.699164+0000","last_unstale":"2024-04-30T08:32:31.551872+0000","last_undegraded":"2024-04-30T08:32:31.551872+0000","last_fullsized":"2024-04-30T08:32:31.551872+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:22:18.873504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"81'4","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.558080+0000","last_change":"2024-04-30T08:31:51.679153+0000","last_active":"2024-04-30T08:32:31.558080+0000","last_peered":"2024-04-30T08:32:31.558080+0000","last_clean":"2024-04-30T08:32:31.558080+0000","last_became_active":"2024-04-30T08:31:51.678348+0000","last_became_peered":"2024-04-30T08:31:51.678348+0000","last_unstale":"2024-04-30T08:32:31.558080+0000","last_undegraded":"2024-04-30T08:32:31.558080+0000","last_fullsized":"2024-04-30T08:32:31.558080+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:02:56.240218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548074+0000","last_change":"2024-04-30T08:31:57.755423+0000","last_active":"2024-04-30T08:32:30.548074+0000","last_peered":"2024-04-30T08:32:30.548074+0000","last_clean":"2024-04-30T08:32:30.548074+0000","last_became_active":"2024-04-30T08:31:57.754835+0000","last_became_peered":"2024-04-30T08:31:57.754835+0000","last_unstale":"2024-04-30T08:32:30.548074+0000","last_undegraded":"2024-04-30T08:32:30.548074+0000","last_fullsized":"2024-04-30T08:32:30.548074+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:19:25.168920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"81'4","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.556916+0000","last_change":"2024-04-30T08:31:51.693839+0000","last_active":"2024-04-30T08:32:31.556916+0000","last_peered":"2024-04-30T08:32:31.556916+0000","last_clean":"2024-04-30T08:32:31.556916+0000","last_became_active":"2024-04-30T08:31:51.693349+0000","last_became_peered":"2024-04-30T08:31:51.693349+0000","last_unstale":"2024-04-30T08:32:31.556916+0000","last_undegraded":"2024-04-30T08:32:31.556916+0000","last_fullsized":"2024-04-30T08:32:31.556916+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:41:51.330901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.553463+0000","last_change":"2024-04-30T08:31:52.712360+0000","last_active":"2024-04-30T08:32:31.553463+0000","last_peered":"2024-04-30T08:32:31.553463+0000","last_clean":"2024-04-30T08:32:31.553463+0000","last_became_active":"2024-04-30T08:31:52.711652+0000","last_became_peered":"2024-04-30T08:31:52.711652+0000","last_unstale":"2024-04-30T08:32:31.553463+0000","last_undegraded":"2024-04-30T08:32:31.553463+0000","last_fullsized":"2024-04-30T08:32:31.553463+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:21:10.285386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.567607+0000","last_change":"2024-04-30T08:31:58.061654+0000","last_active":"2024-04-30T08:32:31.567607+0000","last_peered":"2024-04-30T08:32:31.567607+0000","last_clean":"2024-04-30T08:32:31.567607+0000","last_became_active":"2024-04-30T08:31:58.060181+0000","last_became_peered":"2024-04-30T08:31:58.060181+0000","last_unstale":"2024-04-30T08:32:31.567607+0000","last_undegraded":"2024-04-30T08:32:31.567607+0000","last_fullsized":"2024-04-30T08:32:31.567607+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:46:03.230626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"81'8","reported_seq":34,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.558636+0000","last_change":"2024-04-30T08:31:51.706940+0000","last_active":"2024-04-30T08:32:30.558636+0000","last_peered":"2024-04-30T08:32:30.558636+0000","last_clean":"2024-04-30T08:32:30.558636+0000","last_became_active":"2024-04-30T08:31:51.706481+0000","last_became_peered":"2024-04-30T08:31:51.706481+0000","last_unstale":"2024-04-30T08:32:30.558636+0000","last_undegraded":"2024-04-30T08:32:30.558636+0000","last_fullsized":"2024-04-30T08:32:30.558636+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:53:31.955595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.565718+0000","last_change":"2024-04-30T08:31:52.696625+0000","last_active":"2024-04-30T08:32:31.565718+0000","last_peered":"2024-04-30T08:32:31.565718+0000","last_clean":"2024-04-30T08:32:31.565718+0000","last_became_active":"2024-04-30T08:31:52.696230+0000","last_became_peered":"2024-04-30T08:31:52.696230+0000","last_unstale":"2024-04-30T08:32:31.565718+0000","last_undegraded":"2024-04-30T08:32:31.565718+0000","last_fullsized":"2024-04-30T08:32:31.565718+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:11:46.404251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550372+0000","last_change":"2024-04-30T08:31:57.738329+0000","last_active":"2024-04-30T08:32:31.550372+0000","last_peered":"2024-04-30T08:32:31.550372+0000","last_clean":"2024-04-30T08:32:31.550372+0000","last_became_active":"2024-04-30T08:31:57.737897+0000","last_became_peered":"2024-04-30T08:31:57.737897+0000","last_unstale":"2024-04-30T08:32:31.550372+0000","last_undegraded":"2024-04-30T08:32:31.550372+0000","last_fullsized":"2024-04-30T08:32:31.550372+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:02:43.074578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"81'7","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550515+0000","last_change":"2024-04-30T08:31:51.684066+0000","last_active":"2024-04-30T08:32:31.550515+0000","last_peered":"2024-04-30T08:32:31.550515+0000","last_clean":"2024-04-30T08:32:31.550515+0000","last_became_active":"2024-04-30T08:31:51.683699+0000","last_became_peered":"2024-04-30T08:31:51.683699+0000","last_unstale":"2024-04-30T08:32:31.550515+0000","last_undegraded":"2024-04-30T08:32:31.550515+0000","last_fullsized":"2024-04-30T08:32:31.550515+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:36:20.900733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.559969+0000","last_change":"2024-04-30T08:31:52.715219+0000","last_active":"2024-04-30T08:32:31.559969+0000","last_peered":"2024-04-30T08:32:31.559969+0000","last_clean":"2024-04-30T08:32:31.559969+0000","last_became_active":"2024-04-30T08:31:52.714774+0000","last_became_peered":"2024-04-30T08:31:52.714774+0000","last_unstale":"2024-04-30T08:32:31.559969+0000","last_undegraded":"2024-04-30T08:32:31.559969+0000","last_fullsized":"2024-04-30T08:32:31.559969+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:42:41.082610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550306+0000","last_change":"2024-04-30T08:31:58.060845+0000","last_active":"2024-04-30T08:32:31.550306+0000","last_peered":"2024-04-30T08:32:31.550306+0000","last_clean":"2024-04-30T08:32:31.550306+0000","last_became_active":"2024-04-30T08:31:58.060365+0000","last_became_peered":"2024-04-30T08:31:58.060365+0000","last_unstale":"2024-04-30T08:32:31.550306+0000","last_undegraded":"2024-04-30T08:32:31.550306+0000","last_fullsized":"2024-04-30T08:32:31.550306+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:21:39.583243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"81'2","reported_seq":28,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.547845+0000","last_change":"2024-04-30T08:31:51.694833+0000","last_active":"2024-04-30T08:32:30.547845+0000","last_peered":"2024-04-30T08:32:30.547845+0000","last_clean":"2024-04-30T08:32:30.547845+0000","last_became_active":"2024-04-30T08:31:51.694401+0000","last_became_peered":"2024-04-30T08:31:51.694401+0000","last_unstale":"2024-04-30T08:32:30.547845+0000","last_undegraded":"2024-04-30T08:32:30.547845+0000","last_fullsized":"2024-04-30T08:32:30.547845+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:38:52.605625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.553210+0000","last_change":"2024-04-30T08:31:52.686444+0000","last_active":"2024-04-30T08:32:31.553210+0000","last_peered":"2024-04-30T08:32:31.553210+0000","last_clean":"2024-04-30T08:32:31.553210+0000","last_became_active":"2024-04-30T08:31:52.686057+0000","last_became_peered":"2024-04-30T08:31:52.686057+0000","last_unstale":"2024-04-30T08:32:31.553210+0000","last_undegraded":"2024-04-30T08:32:31.553210+0000","last_fullsized":"2024-04-30T08:32:31.553210+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:54:29.824060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.557873+0000","last_change":"2024-04-30T08:31:57.766193+0000","last_active":"2024-04-30T08:32:30.557873+0000","last_peered":"2024-04-30T08:32:30.557873+0000","last_clean":"2024-04-30T08:32:30.557873+0000","last_became_active":"2024-04-30T08:31:57.765758+0000","last_became_peered":"2024-04-30T08:31:57.765758+0000","last_unstale":"2024-04-30T08:32:30.557873+0000","last_undegraded":"2024-04-30T08:32:30.557873+0000","last_fullsized":"2024-04-30T08:32:30.557873+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:59:57.282795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"81'7","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.565708+0000","last_change":"2024-04-30T08:31:51.704565+0000","last_active":"2024-04-30T08:32:31.565708+0000","last_peered":"2024-04-30T08:32:31.565708+0000","last_clean":"2024-04-30T08:32:31.565708+0000","last_became_active":"2024-04-30T08:31:51.703954+0000","last_became_peered":"2024-04-30T08:31:51.703954+0000","last_unstale":"2024-04-30T08:32:31.565708+0000","last_undegraded":"2024-04-30T08:32:31.565708+0000","last_fullsized":"2024-04-30T08:32:31.565708+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:03:00.517867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548041+0000","last_change":"2024-04-30T08:31:52.706043+0000","last_active":"2024-04-30T08:32:30.548041+0000","last_peered":"2024-04-30T08:32:30.548041+0000","last_clean":"2024-04-30T08:32:30.548041+0000","last_became_active":"2024-04-30T08:31:52.705678+0000","last_became_peered":"2024-04-30T08:31:52.705678+0000","last_unstale":"2024-04-30T08:32:30.548041+0000","last_undegraded":"2024-04-30T08:32:30.548041+0000","last_fullsized":"2024-04-30T08:32:30.548041+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:44:56.787835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"82'1","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550147+0000","last_change":"2024-04-30T08:31:57.764791+0000","last_active":"2024-04-30T08:32:31.550147+0000","last_peered":"2024-04-30T08:32:31.550147+0000","last_clean":"2024-04-30T08:32:31.550147+0000","last_became_active":"2024-04-30T08:31:57.764368+0000","last_became_peered":"2024-04-30T08:31:57.764368+0000","last_unstale":"2024-04-30T08:32:31.550147+0000","last_undegraded":"2024-04-30T08:32:31.550147+0000","last_fullsized":"2024-04-30T08:32:31.550147+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:19:28.545483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"81'3","reported_seq":2,"reported_epoch":82,"state":"peering","last_fresh":"2024-04-30T08:32:30.524198+0000","last_change":"2024-04-30T08:32:30.944913+0000","last_active":"2024-04-30T08:32:30.944913+0000","last_peered":"2024-04-30T08:32:30.524198+0000","last_clean":"2024-04-30T08:32:30.524198+0000","last_became_active":"2024-04-30T08:31:51.702141+0000","last_became_peered":"2024-04-30T08:31:51.702141+0000","last_unstale":"2024-04-30T08:32:30.524198+0000","last_undegraded":"2024-04-30T08:32:30.524198+0000","last_fullsized":"2024-04-30T08:32:30.524198+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.566691+0000","last_change":"2024-04-30T08:31:52.716707+0000","last_active":"2024-04-30T08:32:31.566691+0000","last_peered":"2024-04-30T08:32:31.566691+0000","last_clean":"2024-04-30T08:32:31.566691+0000","last_became_active":"2024-04-30T08:31:52.716293+0000","last_became_peered":"2024-04-30T08:31:52.716293+0000","last_unstale":"2024-04-30T08:32:31.566691+0000","last_undegraded":"2024-04-30T08:32:31.566691+0000","last_fullsized":"2024-04-30T08:32:31.566691+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:29:54.223472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550379+0000","last_change":"2024-04-30T08:31:58.060640+0000","last_active":"2024-04-30T08:32:31.550379+0000","last_peered":"2024-04-30T08:32:31.550379+0000","last_clean":"2024-04-30T08:32:31.550379+0000","last_became_active":"2024-04-30T08:31:58.060061+0000","last_became_peered":"2024-04-30T08:31:58.060061+0000","last_unstale":"2024-04-30T08:32:31.550379+0000","last_undegraded":"2024-04-30T08:32:31.550379+0000","last_fullsized":"2024-04-30T08:32:31.550379+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:46:00.844191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"81'4","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.555331+0000","last_change":"2024-04-30T08:31:51.679532+0000","last_active":"2024-04-30T08:32:31.555331+0000","last_peered":"2024-04-30T08:32:31.555331+0000","last_clean":"2024-04-30T08:32:31.555331+0000","last_became_active":"2024-04-30T08:31:51.679141+0000","last_became_peered":"2024-04-30T08:31:51.679141+0000","last_unstale":"2024-04-30T08:32:31.555331+0000","last_undegraded":"2024-04-30T08:32:31.555331+0000","last_fullsized":"2024-04-30T08:32:31.555331+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:19:50.526563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.557596+0000","last_change":"2024-04-30T08:31:52.703613+0000","last_active":"2024-04-30T08:32:31.557596+0000","last_peered":"2024-04-30T08:32:31.557596+0000","last_clean":"2024-04-30T08:32:31.557596+0000","last_became_active":"2024-04-30T08:31:52.703196+0000","last_became_peered":"2024-04-30T08:31:52.703196+0000","last_unstale":"2024-04-30T08:32:31.557596+0000","last_undegraded":"2024-04-30T08:32:31.557596+0000","last_fullsized":"2024-04-30T08:32:31.557596+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:33:11.136313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":16,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.547878+0000","last_change":"2024-04-30T08:31:57.755286+0000","last_active":"2024-04-30T08:32:30.547878+0000","last_peered":"2024-04-30T08:32:30.547878+0000","last_clean":"2024-04-30T08:32:30.547878+0000","last_became_active":"2024-04-30T08:31:57.754786+0000","last_became_peered":"2024-04-30T08:31:57.754786+0000","last_unstale":"2024-04-30T08:32:30.547878+0000","last_undegraded":"2024-04-30T08:32:30.547878+0000","last_fullsized":"2024-04-30T08:32:30.547878+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:02:50.931553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"81'5","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:33.580030+0000","last_change":"2024-04-30T08:31:51.693317+0000","last_active":"2024-04-30T08:32:33.580030+0000","last_peered":"2024-04-30T08:32:33.580030+0000","last_clean":"2024-04-30T08:32:33.580030+0000","last_became_active":"2024-04-30T08:31:51.692928+0000","last_became_peered":"2024-04-30T08:31:51.692928+0000","last_unstale":"2024-04-30T08:32:33.580030+0000","last_undegraded":"2024-04-30T08:32:33.580030+0000","last_fullsized":"2024-04-30T08:32:33.580030+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:35:09.851735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.558242+0000","last_change":"2024-04-30T08:31:52.694304+0000","last_active":"2024-04-30T08:32:31.558242+0000","last_peered":"2024-04-30T08:32:31.558242+0000","last_clean":"2024-04-30T08:32:31.558242+0000","last_became_active":"2024-04-30T08:31:52.693317+0000","last_became_peered":"2024-04-30T08:31:52.693317+0000","last_unstale":"2024-04-30T08:32:31.558242+0000","last_undegraded":"2024-04-30T08:32:31.558242+0000","last_fullsized":"2024-04-30T08:32:31.558242+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:31:39.294864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.556289+0000","last_change":"2024-04-30T08:31:57.763598+0000","last_active":"2024-04-30T08:32:31.556289+0000","last_peered":"2024-04-30T08:32:31.556289+0000","last_clean":"2024-04-30T08:32:31.556289+0000","last_became_active":"2024-04-30T08:31:57.762307+0000","last_became_peered":"2024-04-30T08:31:57.762307+0000","last_unstale":"2024-04-30T08:32:31.556289+0000","last_undegraded":"2024-04-30T08:32:31.556289+0000","last_fullsized":"2024-04-30T08:32:31.556289+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:24:13.390595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"81'3","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.553337+0000","last_change":"2024-04-30T08:31:51.703476+0000","last_active":"2024-04-30T08:32:31.553337+0000","last_peered":"2024-04-30T08:32:31.553337+0000","last_clean":"2024-04-30T08:32:31.553337+0000","last_became_active":"2024-04-30T08:31:51.703072+0000","last_became_peered":"2024-04-30T08:31:51.703072+0000","last_unstale":"2024-04-30T08:32:31.553337+0000","last_undegraded":"2024-04-30T08:32:31.553337+0000","last_fullsized":"2024-04-30T08:32:31.553337+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:07:21.098223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.557803+0000","last_change":"2024-04-30T08:31:52.699223+0000","last_active":"2024-04-30T08:32:31.557803+0000","last_peered":"2024-04-30T08:32:31.557803+0000","last_clean":"2024-04-30T08:32:31.557803+0000","last_became_active":"2024-04-30T08:31:52.698837+0000","last_became_peered":"2024-04-30T08:31:52.698837+0000","last_unstale":"2024-04-30T08:32:31.557803+0000","last_undegraded":"2024-04-30T08:32:31.557803+0000","last_fullsized":"2024-04-30T08:32:31.557803+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:19:11.826470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.587430+0000","last_change":"2024-04-30T08:32:31.586667+0000","last_active":"2024-04-30T08:32:31.587430+0000","last_peered":"2024-04-30T08:32:31.587430+0000","last_clean":"2024-04-30T08:32:31.587430+0000","last_became_active":"2024-04-30T08:32:31.586144+0000","last_became_peered":"2024-04-30T08:32:31.586144+0000","last_unstale":"2024-04-30T08:32:31.587430+0000","last_undegraded":"2024-04-30T08:32:31.587430+0000","last_fullsized":"2024-04-30T08:32:31.587430+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:50:39.752485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7,1],"acting":[8,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"81'5","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.552340+0000","last_change":"2024-04-30T08:31:51.708943+0000","last_active":"2024-04-30T08:32:31.552340+0000","last_peered":"2024-04-30T08:32:31.552340+0000","last_clean":"2024-04-30T08:32:31.552340+0000","last_became_active":"2024-04-30T08:31:51.708376+0000","last_became_peered":"2024-04-30T08:31:51.708376+0000","last_unstale":"2024-04-30T08:32:31.552340+0000","last_undegraded":"2024-04-30T08:32:31.552340+0000","last_fullsized":"2024-04-30T08:32:31.552340+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:14:46.692192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.558028+0000","last_change":"2024-04-30T08:31:52.686634+0000","last_active":"2024-04-30T08:32:31.558028+0000","last_peered":"2024-04-30T08:32:31.558028+0000","last_clean":"2024-04-30T08:32:31.558028+0000","last_became_active":"2024-04-30T08:31:52.686231+0000","last_became_peered":"2024-04-30T08:31:52.686231+0000","last_unstale":"2024-04-30T08:32:31.558028+0000","last_undegraded":"2024-04-30T08:32:31.558028+0000","last_fullsized":"2024-04-30T08:32:31.558028+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:27:31.073041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548133+0000","last_change":"2024-04-30T08:31:52.688695+0000","last_active":"2024-04-30T08:32:30.548133+0000","last_peered":"2024-04-30T08:32:30.548133+0000","last_clean":"2024-04-30T08:32:30.548133+0000","last_became_active":"2024-04-30T08:31:52.686556+0000","last_became_peered":"2024-04-30T08:31:52.686556+0000","last_unstale":"2024-04-30T08:32:30.548133+0000","last_undegraded":"2024-04-30T08:32:30.548133+0000","last_fullsized":"2024-04-30T08:32:30.548133+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:22:48.976440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550665+0000","last_change":"2024-04-30T08:31:52.688324+0000","last_active":"2024-04-30T08:32:31.550665+0000","last_peered":"2024-04-30T08:32:31.550665+0000","last_clean":"2024-04-30T08:32:31.550665+0000","last_became_active":"2024-04-30T08:31:52.687966+0000","last_became_peered":"2024-04-30T08:31:52.687966+0000","last_unstale":"2024-04-30T08:32:31.550665+0000","last_undegraded":"2024-04-30T08:32:31.550665+0000","last_fullsized":"2024-04-30T08:32:31.550665+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:35:38.237750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.551643+0000","last_change":"2024-04-30T08:31:52.695638+0000","last_active":"2024-04-30T08:32:31.551643+0000","last_peered":"2024-04-30T08:32:31.551643+0000","last_clean":"2024-04-30T08:32:31.551643+0000","last_became_active":"2024-04-30T08:31:52.695283+0000","last_became_peered":"2024-04-30T08:31:52.695283+0000","last_unstale":"2024-04-30T08:32:31.551643+0000","last_undegraded":"2024-04-30T08:32:31.551643+0000","last_fullsized":"2024-04-30T08:32:31.551643+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:00:14.510381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.556661+0000","last_change":"2024-04-30T08:31:52.719112+0000","last_active":"2024-04-30T08:32:31.556661+0000","last_peered":"2024-04-30T08:32:31.556661+0000","last_clean":"2024-04-30T08:32:31.556661+0000","last_became_active":"2024-04-30T08:31:52.718207+0000","last_became_peered":"2024-04-30T08:31:52.718207+0000","last_unstale":"2024-04-30T08:32:31.556661+0000","last_undegraded":"2024-04-30T08:32:31.556661+0000","last_fullsized":"2024-04-30T08:32:31.556661+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:27:29.263962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.553131+0000","last_change":"2024-04-30T08:31:52.712008+0000","last_active":"2024-04-30T08:32:31.553131+0000","last_peered":"2024-04-30T08:32:31.553131+0000","last_clean":"2024-04-30T08:32:31.553131+0000","last_became_active":"2024-04-30T08:31:52.711064+0000","last_became_peered":"2024-04-30T08:31:52.711064+0000","last_unstale":"2024-04-30T08:32:31.553131+0000","last_undegraded":"2024-04-30T08:32:31.553131+0000","last_fullsized":"2024-04-30T08:32:31.553131+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:57:27.400602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548539+0000","last_change":"2024-04-30T08:31:52.691210+0000","last_active":"2024-04-30T08:32:30.548539+0000","last_peered":"2024-04-30T08:32:30.548539+0000","last_clean":"2024-04-30T08:32:30.548539+0000","last_became_active":"2024-04-30T08:31:52.688852+0000","last_became_peered":"2024-04-30T08:31:52.688852+0000","last_unstale":"2024-04-30T08:32:30.548539+0000","last_undegraded":"2024-04-30T08:32:30.548539+0000","last_fullsized":"2024-04-30T08:32:30.548539+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:04:46.502743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.551202+0000","last_change":"2024-04-30T08:31:52.684501+0000","last_active":"2024-04-30T08:32:31.551202+0000","last_peered":"2024-04-30T08:32:31.551202+0000","last_clean":"2024-04-30T08:32:31.551202+0000","last_became_active":"2024-04-30T08:31:52.684122+0000","last_became_peered":"2024-04-30T08:31:52.684122+0000","last_unstale":"2024-04-30T08:32:31.551202+0000","last_undegraded":"2024-04-30T08:32:31.551202+0000","last_fullsized":"2024-04-30T08:32:31.551202+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:55:58.093269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.565964+0000","last_change":"2024-04-30T08:31:52.698016+0000","last_active":"2024-04-30T08:32:31.565964+0000","last_peered":"2024-04-30T08:32:31.565964+0000","last_clean":"2024-04-30T08:32:31.565964+0000","last_became_active":"2024-04-30T08:31:52.695037+0000","last_became_peered":"2024-04-30T08:31:52.695037+0000","last_unstale":"2024-04-30T08:32:31.565964+0000","last_undegraded":"2024-04-30T08:32:31.565964+0000","last_fullsized":"2024-04-30T08:32:31.565964+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:06:01.360947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.555895+0000","last_change":"2024-04-30T08:31:52.698415+0000","last_active":"2024-04-30T08:32:31.555895+0000","last_peered":"2024-04-30T08:32:31.555895+0000","last_clean":"2024-04-30T08:32:31.555895+0000","last_became_active":"2024-04-30T08:31:52.697912+0000","last_became_peered":"2024-04-30T08:31:52.697912+0000","last_unstale":"2024-04-30T08:32:31.555895+0000","last_undegraded":"2024-04-30T08:32:31.555895+0000","last_fullsized":"2024-04-30T08:32:31.555895+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:05:44.420331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.566022+0000","last_change":"2024-04-30T08:31:52.694776+0000","last_active":"2024-04-30T08:32:31.566022+0000","last_peered":"2024-04-30T08:32:31.566022+0000","last_clean":"2024-04-30T08:32:31.566022+0000","last_became_active":"2024-04-30T08:31:52.693532+0000","last_became_peered":"2024-04-30T08:31:52.693532+0000","last_unstale":"2024-04-30T08:32:31.566022+0000","last_undegraded":"2024-04-30T08:32:31.566022+0000","last_fullsized":"2024-04-30T08:32:31.566022+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:31:10.186667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":24,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548026+0000","last_change":"2024-04-30T08:31:52.688477+0000","last_active":"2024-04-30T08:32:30.548026+0000","last_peered":"2024-04-30T08:32:30.548026+0000","last_clean":"2024-04-30T08:32:30.548026+0000","last_became_active":"2024-04-30T08:31:52.688072+0000","last_became_peered":"2024-04-30T08:31:52.688072+0000","last_unstale":"2024-04-30T08:32:30.548026+0000","last_undegraded":"2024-04-30T08:32:30.548026+0000","last_fullsized":"2024-04-30T08:32:30.548026+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:35:42.659707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.553210+0000","last_change":"2024-04-30T08:31:52.716788+0000","last_active":"2024-04-30T08:32:31.553210+0000","last_peered":"2024-04-30T08:32:31.553210+0000","last_clean":"2024-04-30T08:32:31.553210+0000","last_became_active":"2024-04-30T08:31:52.716336+0000","last_became_peered":"2024-04-30T08:31:52.716336+0000","last_unstale":"2024-04-30T08:32:31.553210+0000","last_undegraded":"2024-04-30T08:32:31.553210+0000","last_fullsized":"2024-04-30T08:32:31.553210+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:23:23.477968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":25,"reported_epoch":82,"state":"active+clean","last_fresh":"2024-04-30T08:32:30.548294+0000","last_change":"2024-04-30T08:31:52.684571+0000","last_active":"2024-04-30T08:32:30.548294+0000","last_peered":"2024-04-30T08:32:30.548294+0000","last_clean":"2024-04-30T08:32:30.548294+0000","last_became_active":"2024-04-30T08:31:52.684192+0000","last_became_peered":"2024-04-30T08:31:52.684192+0000","last_unstale":"2024-04-30T08:32:30.548294+0000","last_undegraded":"2024-04-30T08:32:30.548294+0000","last_fullsized":"2024-04-30T08:32:30.548294+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:05:57.520133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284927,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":49110,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":8,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":63,"num_read_kb":85,"num_write":225,"num_write_kb":113,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":491520,"data_stored":143163,"data_compressed":5523,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":198,"ondisk_log_size":198,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678038,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32060,"kb_used_data":880,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713092,"statfs":{"total":95995035648,"available":95962206208,"internally_reserved":0,"allocated":901120,"data_stored":450863,"data_compressed":17691,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874267,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":1089316,"data_compressed":58745,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037793,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":925696,"data_stored":450793,"data_compressed":17691,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234026,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":933888,"data_stored":460511,"data_compressed":20678,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430254,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":987136,"data_stored":1057399,"data_compressed":56904,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626483,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":913408,"data_stored":455637,"data_compressed":20678,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574929,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36248,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708904,"statfs":{"total":95995035648,"available":95957917696,"internally_reserved":0,"allocated":929792,"data_stored":470468,"data_compressed":17691,"data_compressed_allocated":24576,"data_compressed_original":49152,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738454,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":937984,"data_stored":457093,"data_compressed":20678,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411405,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":970752,"data_stored":499116,"data_compressed":22519,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182470,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":925696,"data_stored":455345,"data_compressed":20678,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986240,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":921600,"data_stored":460102,"data_compressed":20678,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822714,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":995328,"data_stored":1052590,"data_compressed":56904,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34599,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34599,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34599,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":39559,"data_compressed":1841,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":44201,"data_compressed":1841,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":44033,"data_compressed":1841,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-30T08:32:36.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:35 smithi032 ceph-mon[28331]: from='client.24890 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:32:36.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:35 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/4185045098' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-30T08:32:36.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:35 smithi071 ceph-mon[33385]: from='client.24890 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:32:36.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:35 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/4185045098' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-30T08:32:36.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:35 smithi189 ceph-mon[33038]: from='client.24890 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:32:36.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:35 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/4185045098' entity='client.admin' cmd=[{"prefix": "status", "format": "json"}]: dispatch 2024-04-30T08:32:37.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:36 smithi032 ceph-mon[28331]: pgmap v346: 129 pgs: 3 peering, 126 active+clean; 625 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 KiB/s wr, 30 op/s 2024-04-30T08:32:37.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:36 smithi032 ceph-mon[28331]: from='client.24902 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:32:37.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:36 smithi071 ceph-mon[33385]: pgmap v346: 129 pgs: 3 peering, 126 active+clean; 625 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 KiB/s wr, 30 op/s 2024-04-30T08:32:37.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:36 smithi071 ceph-mon[33385]: from='client.24902 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:32:37.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:36 smithi189 ceph-mon[33038]: pgmap v346: 129 pgs: 3 peering, 126 active+clean; 625 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 12 KiB/s wr, 30 op/s 2024-04-30T08:32:37.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:36 smithi189 ceph-mon[33038]: from='client.24902 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:32:39.089 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-30T08:32:39.172 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:38 smithi032 ceph-mon[28331]: pgmap v347: 129 pgs: 129 active+clean; 625 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.4 KiB/s wr, 23 op/s; 8 B/s, 1 keys/s, 0 objects/s recovering 2024-04-30T08:32:39.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:38 smithi071 ceph-mon[33385]: pgmap v347: 129 pgs: 129 active+clean; 625 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.4 KiB/s wr, 23 op/s; 8 B/s, 1 keys/s, 0 objects/s recovering 2024-04-30T08:32:39.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:38 smithi189 ceph-mon[33038]: pgmap v347: 129 pgs: 129 active+clean; 625 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 9.4 KiB/s wr, 23 op/s; 8 B/s, 1 keys/s, 0 objects/s recovering 2024-04-30T08:32:39.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.386+0000 7f3f5d800640 1 -- 172.21.15.32:0/2701585005 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3f5810b200 msgr2=0x7f3f5810b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:39.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.386+0000 7f3f5d800640 1 --2- 172.21.15.32:0/2701585005 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3f5810b200 0x7f3f5810b610 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f3f44009860 tx=0x7f3f4402a340 comp rx=0 tx=0).stop 2024-04-30T08:32:39.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.387+0000 7f3f5d800640 1 -- 172.21.15.32:0/2701585005 shutdown_connections 2024-04-30T08:32:39.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.387+0000 7f3f5d800640 1 --2- 172.21.15.32:0/2701585005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f5810ff00 0x7f3f58112320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:39.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.387+0000 7f3f5d800640 1 --2- 172.21.15.32:0/2701585005 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3f5810b200 0x7f3f5810b610 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:39.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.387+0000 7f3f5d800640 1 --2- 172.21.15.32:0/2701585005 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f58109fc0 0x7f3f5810acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:39.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.387+0000 7f3f5d800640 1 -- 172.21.15.32:0/2701585005 >> 172.21.15.32:0/2701585005 conn(0x7f3f58078bb0 msgr2=0x7f3f58107a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:39.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.387+0000 7f3f5d800640 1 -- 172.21.15.32:0/2701585005 shutdown_connections 2024-04-30T08:32:39.389 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.387+0000 7f3f5d800640 1 -- 172.21.15.32:0/2701585005 wait complete. 2024-04-30T08:32:39.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.388+0000 7f3f5d800640 1 Processor -- start 2024-04-30T08:32:39.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.388+0000 7f3f5d800640 1 -- start start 2024-04-30T08:32:39.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.389+0000 7f3f5d800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f58109fc0 0x7f3f581a0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:39.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.389+0000 7f3f5d800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f5810b200 0x7f3f581a0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:39.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.389+0000 7f3f5d800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3f5810ff00 0x7f3f581a7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:39.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.389+0000 7f3f5d800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3f58115ae0 con 0x7f3f5810b200 2024-04-30T08:32:39.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.389+0000 7f3f5d800640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f3f58115960 con 0x7f3f5810ff00 2024-04-30T08:32:39.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.389+0000 7f3f5d800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3f58115c60 con 0x7f3f58109fc0 2024-04-30T08:32:39.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.389+0000 7f3f57e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f58109fc0 0x7f3f581a0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:39.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.389+0000 7f3f57400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f5810b200 0x7f3f581a0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:39.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.389+0000 7f3f57e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f58109fc0 0x7f3f581a0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:53048/0 (socket says 172.21.15.32:53048) 2024-04-30T08:32:39.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.389+0000 7f3f57e00640 1 -- 172.21.15.32:0/1302800001 learned_addr learned my addr 172.21.15.32:0/1302800001 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:39.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.390+0000 7f3f5ce00640 1 --2- 172.21.15.32:0/1302800001 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3f5810ff00 0x7f3f581a7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:39.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.390+0000 7f3f57e00640 1 -- 172.21.15.32:0/1302800001 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3f5810ff00 msgr2=0x7f3f581a7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:39.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.390+0000 7f3f57e00640 1 --2- 172.21.15.32:0/1302800001 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3f5810ff00 0x7f3f581a7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:39.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.390+0000 7f3f57e00640 1 -- 172.21.15.32:0/1302800001 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f5810b200 msgr2=0x7f3f581a0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:39.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.390+0000 7f3f57e00640 1 --2- 172.21.15.32:0/1302800001 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f5810b200 0x7f3f581a0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:39.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.390+0000 7f3f57e00640 1 -- 172.21.15.32:0/1302800001 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f581a8260 con 0x7f3f58109fc0 2024-04-30T08:32:39.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.390+0000 7f3f5ce00640 1 --2- 172.21.15.32:0/1302800001 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3f5810ff00 0x7f3f581a7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:32:39.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.390+0000 7f3f57e00640 1 --2- 172.21.15.32:0/1302800001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f58109fc0 0x7f3f581a0720 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3f4800e9f0 tx=0x7f3f4800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:39.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.391+0000 7f3f54c00640 1 -- 172.21.15.32:0/1302800001 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4800ccc0 con 0x7f3f58109fc0 2024-04-30T08:32:39.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.391+0000 7f3f54c00640 1 -- 172.21.15.32:0/1302800001 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3f480035f0 con 0x7f3f58109fc0 2024-04-30T08:32:39.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.391+0000 7f3f5d800640 1 -- 172.21.15.32:0/1302800001 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f581a8550 con 0x7f3f58109fc0 2024-04-30T08:32:39.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.391+0000 7f3f54c00640 1 -- 172.21.15.32:0/1302800001 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4800ce60 con 0x7f3f58109fc0 2024-04-30T08:32:39.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.391+0000 7f3f5d800640 1 -- 172.21.15.32:0/1302800001 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f581a8a70 con 0x7f3f58109fc0 2024-04-30T08:32:39.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.392+0000 7f3f5d800640 1 -- 172.21.15.32:0/1302800001 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f5810ed10 con 0x7f3f58109fc0 2024-04-30T08:32:39.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.394+0000 7f3f54c00640 1 -- 172.21.15.32:0/1302800001 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f3f48003aa0 con 0x7f3f58109fc0 2024-04-30T08:32:39.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.396+0000 7f3f54c00640 1 --2- 172.21.15.32:0/1302800001 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3f3c0afb70 0x7f3f3c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:39.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.396+0000 7f3f54c00640 1 -- 172.21.15.32:0/1302800001 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f3f480df690 con 0x7f3f58109fc0 2024-04-30T08:32:39.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.396+0000 7f3f57400640 1 --2- 172.21.15.32:0/1302800001 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3f3c0afb70 0x7f3f3c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:39.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.397+0000 7f3f57400640 1 --2- 172.21.15.32:0/1302800001 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3f3c0afb70 0x7f3f3c0b2080 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3f440304b0 tx=0x7f3f44000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:39.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.398+0000 7f3f54c00640 1 -- 172.21.15.32:0/1302800001 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3f480a7d50 con 0x7f3f58109fc0 2024-04-30T08:32:39.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.605+0000 7f3f5d800640 1 -- 172.21.15.32:0/1302800001 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3f58063060 con 0x7f3f3c0afb70 2024-04-30T08:32:39.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.632+0000 7f3f54c00640 1 -- 172.21.15.32:0/1302800001 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340776 (secure 0 0 0) 0x7f3f58063060 con 0x7f3f3c0afb70 2024-04-30T08:32:39.638 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:32:39.641 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-30T08:32:39.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.641+0000 7f3f5d800640 1 -- 172.21.15.32:0/1302800001 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3f3c0afb70 msgr2=0x7f3f3c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:39.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.641+0000 7f3f5d800640 1 --2- 172.21.15.32:0/1302800001 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3f3c0afb70 0x7f3f3c0b2080 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f3f440304b0 tx=0x7f3f44000bc0 comp rx=0 tx=0).stop 2024-04-30T08:32:39.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.641+0000 7f3f5d800640 1 -- 172.21.15.32:0/1302800001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f58109fc0 msgr2=0x7f3f581a0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:39.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.641+0000 7f3f5d800640 1 --2- 172.21.15.32:0/1302800001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f58109fc0 0x7f3f581a0720 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3f4800e9f0 tx=0x7f3f4800eeb0 comp rx=0 tx=0).stop 2024-04-30T08:32:39.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.641+0000 7f3f5d800640 1 -- 172.21.15.32:0/1302800001 shutdown_connections 2024-04-30T08:32:39.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.641+0000 7f3f5d800640 1 --2- 172.21.15.32:0/1302800001 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3f3c0afb70 0x7f3f3c0b2080 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:39.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.641+0000 7f3f5d800640 1 --2- 172.21.15.32:0/1302800001 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3f5810ff00 0x7f3f581a7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:39.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.641+0000 7f3f5d800640 1 --2- 172.21.15.32:0/1302800001 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3f5810b200 0x7f3f581a0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:39.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.641+0000 7f3f5d800640 1 --2- 172.21.15.32:0/1302800001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f58109fc0 0x7f3f581a0720 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:39.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.641+0000 7f3f5d800640 1 -- 172.21.15.32:0/1302800001 >> 172.21.15.32:0/1302800001 conn(0x7f3f58078bb0 msgr2=0x7f3f5810bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:39.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.641+0000 7f3f5d800640 1 -- 172.21.15.32:0/1302800001 shutdown_connections 2024-04-30T08:32:39.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:39.641+0000 7f3f5d800640 1 -- 172.21.15.32:0/1302800001 wait complete. 2024-04-30T08:32:39.673 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":347,"stamp":"2024-04-30T08:32:37.716566+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":639712,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":147,"num_read_kb":157,"num_write":356,"num_write_kb":2267,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":2951908,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":295,"ondisk_log_size":295,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":421,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":368732,"kb_used_data":11184,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124573092,"statfs":{"total":1151940427776,"available":1151562846208,"internally_reserved":0,"allocated":11452416,"data_stored":7415935,"data_compressed":362337,"data_compressed_allocated":491520,"data_compressed_original":2482176,"omap_allocated":19141,"internal_metadata":365933883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":42542,"num_objects":45,"num_object_clones":0,"num_object_copies":137,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":45,"num_whiteouts":0,"num_read":71,"num_read_kb":93,"num_write":117,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.004320"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.554514+0000","last_change":"2024-04-30T08:31:52.711443+0000","last_active":"2024-04-30T08:32:31.554514+0000","last_peered":"2024-04-30T08:32:31.554514+0000","last_clean":"2024-04-30T08:32:31.554514+0000","last_became_active":"2024-04-30T08:31:52.711029+0000","last_became_peered":"2024-04-30T08:31:52.711029+0000","last_unstale":"2024-04-30T08:32:31.554514+0000","last_undegraded":"2024-04-30T08:32:31.554514+0000","last_fullsized":"2024-04-30T08:32:31.554514+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:51:15.208528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.559664+0000","last_change":"2024-04-30T08:31:52.712324+0000","last_active":"2024-04-30T08:32:31.559664+0000","last_peered":"2024-04-30T08:32:31.559664+0000","last_clean":"2024-04-30T08:32:31.559664+0000","last_became_active":"2024-04-30T08:31:52.711923+0000","last_became_peered":"2024-04-30T08:31:52.711923+0000","last_unstale":"2024-04-30T08:32:31.559664+0000","last_undegraded":"2024-04-30T08:32:31.559664+0000","last_fullsized":"2024-04-30T08:32:31.559664+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:29:55.071459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.557006+0000","last_change":"2024-04-30T08:31:52.712936+0000","last_active":"2024-04-30T08:32:31.557006+0000","last_peered":"2024-04-30T08:32:31.557006+0000","last_clean":"2024-04-30T08:32:31.557006+0000","last_became_active":"2024-04-30T08:31:52.712585+0000","last_became_peered":"2024-04-30T08:31:52.712585+0000","last_unstale":"2024-04-30T08:32:31.557006+0000","last_undegraded":"2024-04-30T08:32:31.557006+0000","last_fullsized":"2024-04-30T08:32:31.557006+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:13:43.745561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550312+0000","last_change":"2024-04-30T08:31:52.686880+0000","last_active":"2024-04-30T08:32:31.550312+0000","last_peered":"2024-04-30T08:32:31.550312+0000","last_clean":"2024-04-30T08:32:31.550312+0000","last_became_active":"2024-04-30T08:31:52.686506+0000","last_became_peered":"2024-04-30T08:31:52.686506+0000","last_unstale":"2024-04-30T08:32:31.550312+0000","last_undegraded":"2024-04-30T08:32:31.550312+0000","last_fullsized":"2024-04-30T08:32:31.550312+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:16:33.390951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.555472+0000","last_change":"2024-04-30T08:31:52.706014+0000","last_active":"2024-04-30T08:32:31.555472+0000","last_peered":"2024-04-30T08:32:31.555472+0000","last_clean":"2024-04-30T08:32:31.555472+0000","last_became_active":"2024-04-30T08:31:52.705582+0000","last_became_peered":"2024-04-30T08:31:52.705582+0000","last_unstale":"2024-04-30T08:32:31.555472+0000","last_undegraded":"2024-04-30T08:32:31.555472+0000","last_fullsized":"2024-04-30T08:32:31.555472+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:24:49.804592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.552434+0000","last_change":"2024-04-30T08:31:52.690462+0000","last_active":"2024-04-30T08:32:31.552434+0000","last_peered":"2024-04-30T08:32:31.552434+0000","last_clean":"2024-04-30T08:32:31.552434+0000","last_became_active":"2024-04-30T08:31:52.690092+0000","last_became_peered":"2024-04-30T08:31:52.690092+0000","last_unstale":"2024-04-30T08:32:31.552434+0000","last_undegraded":"2024-04-30T08:32:31.552434+0000","last_fullsized":"2024-04-30T08:32:31.552434+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:32:49.769383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.557763+0000","last_change":"2024-04-30T08:31:52.701786+0000","last_active":"2024-04-30T08:32:31.557763+0000","last_peered":"2024-04-30T08:32:31.557763+0000","last_clean":"2024-04-30T08:32:31.557763+0000","last_became_active":"2024-04-30T08:31:52.701402+0000","last_became_peered":"2024-04-30T08:31:52.701402+0000","last_unstale":"2024-04-30T08:32:31.557763+0000","last_undegraded":"2024-04-30T08:32:31.557763+0000","last_fullsized":"2024-04-30T08:32:31.557763+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:58:06.153479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.567481+0000","last_change":"2024-04-30T08:31:52.698037+0000","last_active":"2024-04-30T08:32:31.567481+0000","last_peered":"2024-04-30T08:32:31.567481+0000","last_clean":"2024-04-30T08:32:31.567481+0000","last_became_active":"2024-04-30T08:31:52.697646+0000","last_became_peered":"2024-04-30T08:31:52.697646+0000","last_unstale":"2024-04-30T08:32:31.567481+0000","last_undegraded":"2024-04-30T08:32:31.567481+0000","last_fullsized":"2024-04-30T08:32:31.567481+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:28:59.520091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.585816+0000","last_change":"2024-04-30T08:31:52.680377+0000","last_active":"2024-04-30T08:32:31.585816+0000","last_peered":"2024-04-30T08:32:31.585816+0000","last_clean":"2024-04-30T08:32:31.585816+0000","last_became_active":"2024-04-30T08:31:52.679977+0000","last_became_peered":"2024-04-30T08:31:52.679977+0000","last_unstale":"2024-04-30T08:32:31.585816+0000","last_undegraded":"2024-04-30T08:32:31.585816+0000","last_fullsized":"2024-04-30T08:32:31.585816+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:35:22.488450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.573546+0000","last_change":"2024-04-30T08:32:31.573546+0000","last_active":"2024-04-30T08:32:31.573546+0000","last_peered":"2024-04-30T08:32:31.573546+0000","last_clean":"2024-04-30T08:32:31.573546+0000","last_became_active":"2024-04-30T08:32:31.573289+0000","last_became_peered":"2024-04-30T08:32:31.573289+0000","last_unstale":"2024-04-30T08:32:31.573546+0000","last_undegraded":"2024-04-30T08:32:31.573546+0000","last_fullsized":"2024-04-30T08:32:31.573546+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:42:45.862326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.558068+0000","last_change":"2024-04-30T08:31:52.690344+0000","last_active":"2024-04-30T08:32:31.558068+0000","last_peered":"2024-04-30T08:32:31.558068+0000","last_clean":"2024-04-30T08:32:31.558068+0000","last_became_active":"2024-04-30T08:31:52.689857+0000","last_became_peered":"2024-04-30T08:31:52.689857+0000","last_unstale":"2024-04-30T08:32:31.558068+0000","last_undegraded":"2024-04-30T08:32:31.558068+0000","last_fullsized":"2024-04-30T08:32:31.558068+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:45:13.395589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550256+0000","last_change":"2024-04-30T08:31:52.685151+0000","last_active":"2024-04-30T08:32:31.550256+0000","last_peered":"2024-04-30T08:32:31.550256+0000","last_clean":"2024-04-30T08:32:31.550256+0000","last_became_active":"2024-04-30T08:31:52.684722+0000","last_became_peered":"2024-04-30T08:31:52.684722+0000","last_unstale":"2024-04-30T08:32:31.550256+0000","last_undegraded":"2024-04-30T08:32:31.550256+0000","last_fullsized":"2024-04-30T08:32:31.550256+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:14:00.660478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.553049+0000","last_change":"2024-04-30T08:31:52.714294+0000","last_active":"2024-04-30T08:32:31.553049+0000","last_peered":"2024-04-30T08:32:31.553049+0000","last_clean":"2024-04-30T08:32:31.553049+0000","last_became_active":"2024-04-30T08:31:52.710094+0000","last_became_peered":"2024-04-30T08:31:52.710094+0000","last_unstale":"2024-04-30T08:32:31.553049+0000","last_undegraded":"2024-04-30T08:32:31.553049+0000","last_fullsized":"2024-04-30T08:32:31.553049+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:34:44.203093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.556588+0000","last_change":"2024-04-30T08:31:52.707992+0000","last_active":"2024-04-30T08:32:31.556588+0000","last_peered":"2024-04-30T08:32:31.556588+0000","last_clean":"2024-04-30T08:32:31.556588+0000","last_became_active":"2024-04-30T08:31:52.706456+0000","last_became_peered":"2024-04-30T08:31:52.706456+0000","last_unstale":"2024-04-30T08:32:31.556588+0000","last_undegraded":"2024-04-30T08:32:31.556588+0000","last_fullsized":"2024-04-30T08:32:31.556588+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:45:32.717842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.556180+0000","last_change":"2024-04-30T08:31:52.695605+0000","last_active":"2024-04-30T08:32:31.556180+0000","last_peered":"2024-04-30T08:32:31.556180+0000","last_clean":"2024-04-30T08:32:31.556180+0000","last_became_active":"2024-04-30T08:31:52.692439+0000","last_became_peered":"2024-04-30T08:31:52.692439+0000","last_unstale":"2024-04-30T08:32:31.556180+0000","last_undegraded":"2024-04-30T08:32:31.556180+0000","last_fullsized":"2024-04-30T08:32:31.556180+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:39:38.746567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.585229+0000","last_change":"2024-04-30T08:31:52.707011+0000","last_active":"2024-04-30T08:32:31.585229+0000","last_peered":"2024-04-30T08:32:31.585229+0000","last_clean":"2024-04-30T08:32:31.585229+0000","last_became_active":"2024-04-30T08:31:52.706602+0000","last_became_peered":"2024-04-30T08:31:52.706602+0000","last_unstale":"2024-04-30T08:32:31.585229+0000","last_undegraded":"2024-04-30T08:32:31.585229+0000","last_fullsized":"2024-04-30T08:32:31.585229+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:43:13.922905+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.554345+0000","last_change":"2024-04-30T08:31:52.711439+0000","last_active":"2024-04-30T08:32:31.554345+0000","last_peered":"2024-04-30T08:32:31.554345+0000","last_clean":"2024-04-30T08:32:31.554345+0000","last_became_active":"2024-04-30T08:31:52.711023+0000","last_became_peered":"2024-04-30T08:31:52.711023+0000","last_unstale":"2024-04-30T08:32:31.554345+0000","last_undegraded":"2024-04-30T08:32:31.554345+0000","last_fullsized":"2024-04-30T08:32:31.554345+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:31:00.654348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.570096+0000","last_change":"2024-04-30T08:31:52.708476+0000","last_active":"2024-04-30T08:32:31.570096+0000","last_peered":"2024-04-30T08:32:31.570096+0000","last_clean":"2024-04-30T08:32:31.570096+0000","last_became_active":"2024-04-30T08:31:52.704735+0000","last_became_peered":"2024-04-30T08:31:52.704735+0000","last_unstale":"2024-04-30T08:32:31.570096+0000","last_undegraded":"2024-04-30T08:32:31.570096+0000","last_fullsized":"2024-04-30T08:32:31.570096+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:40:00.011783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550883+0000","last_change":"2024-04-30T08:31:52.704901+0000","last_active":"2024-04-30T08:32:31.550883+0000","last_peered":"2024-04-30T08:32:31.550883+0000","last_clean":"2024-04-30T08:32:31.550883+0000","last_became_active":"2024-04-30T08:31:52.704636+0000","last_became_peered":"2024-04-30T08:31:52.704636+0000","last_unstale":"2024-04-30T08:32:31.550883+0000","last_undegraded":"2024-04-30T08:32:31.550883+0000","last_fullsized":"2024-04-30T08:32:31.550883+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:08:53.580809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550486+0000","last_change":"2024-04-30T08:31:57.743342+0000","last_active":"2024-04-30T08:32:31.550486+0000","last_peered":"2024-04-30T08:32:31.550486+0000","last_clean":"2024-04-30T08:32:31.550486+0000","last_became_active":"2024-04-30T08:31:57.742850+0000","last_became_peered":"2024-04-30T08:31:57.742850+0000","last_unstale":"2024-04-30T08:32:31.550486+0000","last_undegraded":"2024-04-30T08:32:31.550486+0000","last_fullsized":"2024-04-30T08:32:31.550486+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:59:10.628822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"81'13","reported_seq":48,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:34.576316+0000","last_change":"2024-04-30T08:31:51.694984+0000","last_active":"2024-04-30T08:32:34.576316+0000","last_peered":"2024-04-30T08:32:34.576316+0000","last_clean":"2024-04-30T08:32:34.576316+0000","last_became_active":"2024-04-30T08:31:51.694514+0000","last_became_peered":"2024-04-30T08:31:51.694514+0000","last_unstale":"2024-04-30T08:32:34.576316+0000","last_undegraded":"2024-04-30T08:32:34.576316+0000","last_fullsized":"2024-04-30T08:32:34.576316+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:17:03.873090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.565234+0000","last_change":"2024-04-30T08:31:52.694276+0000","last_active":"2024-04-30T08:32:31.565234+0000","last_peered":"2024-04-30T08:32:31.565234+0000","last_clean":"2024-04-30T08:32:31.565234+0000","last_became_active":"2024-04-30T08:31:52.693839+0000","last_became_peered":"2024-04-30T08:31:52.693839+0000","last_unstale":"2024-04-30T08:32:31.565234+0000","last_undegraded":"2024-04-30T08:32:31.565234+0000","last_fullsized":"2024-04-30T08:32:31.565234+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:29:21.271040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.549942+0000","last_change":"2024-04-30T08:31:57.754828+0000","last_active":"2024-04-30T08:32:31.549942+0000","last_peered":"2024-04-30T08:32:31.549942+0000","last_clean":"2024-04-30T08:32:31.549942+0000","last_became_active":"2024-04-30T08:31:57.754322+0000","last_became_peered":"2024-04-30T08:31:57.754322+0000","last_unstale":"2024-04-30T08:32:31.549942+0000","last_undegraded":"2024-04-30T08:32:31.549942+0000","last_fullsized":"2024-04-30T08:32:31.549942+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:36:02.560134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"81'2","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.567668+0000","last_change":"2024-04-30T08:31:51.707637+0000","last_active":"2024-04-30T08:32:31.567668+0000","last_peered":"2024-04-30T08:32:31.567668+0000","last_clean":"2024-04-30T08:32:31.567668+0000","last_became_active":"2024-04-30T08:31:51.707233+0000","last_became_peered":"2024-04-30T08:31:51.707233+0000","last_unstale":"2024-04-30T08:32:31.567668+0000","last_undegraded":"2024-04-30T08:32:31.567668+0000","last_fullsized":"2024-04-30T08:32:31.567668+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:14:46.041169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.560581+0000","last_change":"2024-04-30T08:31:52.711442+0000","last_active":"2024-04-30T08:32:31.560581+0000","last_peered":"2024-04-30T08:32:31.560581+0000","last_clean":"2024-04-30T08:32:31.560581+0000","last_became_active":"2024-04-30T08:31:52.710942+0000","last_became_peered":"2024-04-30T08:31:52.710942+0000","last_unstale":"2024-04-30T08:32:31.560581+0000","last_undegraded":"2024-04-30T08:32:31.560581+0000","last_fullsized":"2024-04-30T08:32:31.560581+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:26:56.428770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550129+0000","last_change":"2024-04-30T08:31:57.760535+0000","last_active":"2024-04-30T08:32:31.550129+0000","last_peered":"2024-04-30T08:32:31.550129+0000","last_clean":"2024-04-30T08:32:31.550129+0000","last_became_active":"2024-04-30T08:31:57.760073+0000","last_became_peered":"2024-04-30T08:31:57.760073+0000","last_unstale":"2024-04-30T08:32:31.550129+0000","last_undegraded":"2024-04-30T08:32:31.550129+0000","last_fullsized":"2024-04-30T08:32:31.550129+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:03:51.359474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"81'12","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.557144+0000","last_change":"2024-04-30T08:31:51.678575+0000","last_active":"2024-04-30T08:32:31.557144+0000","last_peered":"2024-04-30T08:32:31.557144+0000","last_clean":"2024-04-30T08:32:31.557144+0000","last_became_active":"2024-04-30T08:31:51.678114+0000","last_became_peered":"2024-04-30T08:31:51.678114+0000","last_unstale":"2024-04-30T08:32:31.557144+0000","last_undegraded":"2024-04-30T08:32:31.557144+0000","last_fullsized":"2024-04-30T08:32:31.557144+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:10:35.517183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.585186+0000","last_change":"2024-04-30T08:31:52.704137+0000","last_active":"2024-04-30T08:32:31.585186+0000","last_peered":"2024-04-30T08:32:31.585186+0000","last_clean":"2024-04-30T08:32:31.585186+0000","last_became_active":"2024-04-30T08:31:52.703695+0000","last_became_peered":"2024-04-30T08:31:52.703695+0000","last_unstale":"2024-04-30T08:32:31.585186+0000","last_undegraded":"2024-04-30T08:32:31.585186+0000","last_fullsized":"2024-04-30T08:32:31.585186+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:31:05.461106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.549998+0000","last_change":"2024-04-30T08:31:57.755237+0000","last_active":"2024-04-30T08:32:31.549998+0000","last_peered":"2024-04-30T08:32:31.549998+0000","last_clean":"2024-04-30T08:32:31.549998+0000","last_became_active":"2024-04-30T08:31:57.754438+0000","last_became_peered":"2024-04-30T08:31:57.754438+0000","last_unstale":"2024-04-30T08:32:31.549998+0000","last_undegraded":"2024-04-30T08:32:31.549998+0000","last_fullsized":"2024-04-30T08:32:31.549998+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:20:59.035110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"81'5","reported_seq":34,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.555740+0000","last_change":"2024-04-30T08:31:51.673793+0000","last_active":"2024-04-30T08:32:31.555740+0000","last_peered":"2024-04-30T08:32:31.555740+0000","last_clean":"2024-04-30T08:32:31.555740+0000","last_became_active":"2024-04-30T08:31:51.672703+0000","last_became_peered":"2024-04-30T08:31:51.672703+0000","last_unstale":"2024-04-30T08:32:31.555740+0000","last_undegraded":"2024-04-30T08:32:31.555740+0000","last_fullsized":"2024-04-30T08:32:31.555740+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:13:18.959542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.553233+0000","last_change":"2024-04-30T08:31:52.694320+0000","last_active":"2024-04-30T08:32:31.553233+0000","last_peered":"2024-04-30T08:32:31.553233+0000","last_clean":"2024-04-30T08:32:31.553233+0000","last_became_active":"2024-04-30T08:31:52.694032+0000","last_became_peered":"2024-04-30T08:31:52.694032+0000","last_unstale":"2024-04-30T08:32:31.553233+0000","last_undegraded":"2024-04-30T08:32:31.553233+0000","last_fullsized":"2024-04-30T08:32:31.553233+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:33:12.347369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550575+0000","last_change":"2024-04-30T08:31:57.739984+0000","last_active":"2024-04-30T08:32:31.550575+0000","last_peered":"2024-04-30T08:32:31.550575+0000","last_clean":"2024-04-30T08:32:31.550575+0000","last_became_active":"2024-04-30T08:31:57.739477+0000","last_became_peered":"2024-04-30T08:31:57.739477+0000","last_unstale":"2024-04-30T08:32:31.550575+0000","last_undegraded":"2024-04-30T08:32:31.550575+0000","last_fullsized":"2024-04-30T08:32:31.550575+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:10:11.325419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"81'6","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.564820+0000","last_change":"2024-04-30T08:31:51.678792+0000","last_active":"2024-04-30T08:32:31.564820+0000","last_peered":"2024-04-30T08:32:31.564820+0000","last_clean":"2024-04-30T08:32:31.564820+0000","last_became_active":"2024-04-30T08:31:51.678094+0000","last_became_peered":"2024-04-30T08:31:51.678094+0000","last_unstale":"2024-04-30T08:32:31.564820+0000","last_undegraded":"2024-04-30T08:32:31.564820+0000","last_fullsized":"2024-04-30T08:32:31.564820+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:03:17.526016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.572984+0000","last_change":"2024-04-30T08:32:31.572984+0000","last_active":"2024-04-30T08:32:31.572984+0000","last_peered":"2024-04-30T08:32:31.572984+0000","last_clean":"2024-04-30T08:32:31.572984+0000","last_became_active":"2024-04-30T08:32:31.572536+0000","last_became_peered":"2024-04-30T08:32:31.572536+0000","last_unstale":"2024-04-30T08:32:31.572984+0000","last_undegraded":"2024-04-30T08:32:31.572984+0000","last_fullsized":"2024-04-30T08:32:31.572984+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:46:15.884555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550816+0000","last_change":"2024-04-30T08:31:57.765047+0000","last_active":"2024-04-30T08:32:31.550816+0000","last_peered":"2024-04-30T08:32:31.550816+0000","last_clean":"2024-04-30T08:32:31.550816+0000","last_became_active":"2024-04-30T08:31:57.764519+0000","last_became_peered":"2024-04-30T08:31:57.764519+0000","last_unstale":"2024-04-30T08:32:31.550816+0000","last_undegraded":"2024-04-30T08:32:31.550816+0000","last_fullsized":"2024-04-30T08:32:31.550816+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:06:16.126396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"81'7","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.552381+0000","last_change":"2024-04-30T08:31:51.686403+0000","last_active":"2024-04-30T08:32:31.552381+0000","last_peered":"2024-04-30T08:32:31.552381+0000","last_clean":"2024-04-30T08:32:31.552381+0000","last_became_active":"2024-04-30T08:31:51.686096+0000","last_became_peered":"2024-04-30T08:31:51.686096+0000","last_unstale":"2024-04-30T08:32:31.552381+0000","last_undegraded":"2024-04-30T08:32:31.552381+0000","last_fullsized":"2024-04-30T08:32:31.552381+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:47:05.961930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.554940+0000","last_change":"2024-04-30T08:31:52.698512+0000","last_active":"2024-04-30T08:32:31.554940+0000","last_peered":"2024-04-30T08:32:31.554940+0000","last_clean":"2024-04-30T08:32:31.554940+0000","last_became_active":"2024-04-30T08:31:52.698168+0000","last_became_peered":"2024-04-30T08:31:52.698168+0000","last_unstale":"2024-04-30T08:32:31.554940+0000","last_undegraded":"2024-04-30T08:32:31.554940+0000","last_fullsized":"2024-04-30T08:32:31.554940+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:22:45.094365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.554130+0000","last_change":"2024-04-30T08:31:57.761430+0000","last_active":"2024-04-30T08:32:31.554130+0000","last_peered":"2024-04-30T08:32:31.554130+0000","last_clean":"2024-04-30T08:32:31.554130+0000","last_became_active":"2024-04-30T08:31:57.761062+0000","last_became_peered":"2024-04-30T08:31:57.761062+0000","last_unstale":"2024-04-30T08:32:31.554130+0000","last_undegraded":"2024-04-30T08:32:31.554130+0000","last_fullsized":"2024-04-30T08:32:31.554130+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:32:05.403786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"81'3","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.552142+0000","last_change":"2024-04-30T08:31:51.686694+0000","last_active":"2024-04-30T08:32:31.552142+0000","last_peered":"2024-04-30T08:32:31.552142+0000","last_clean":"2024-04-30T08:32:31.552142+0000","last_became_active":"2024-04-30T08:31:51.685773+0000","last_became_peered":"2024-04-30T08:31:51.685773+0000","last_unstale":"2024-04-30T08:32:31.552142+0000","last_undegraded":"2024-04-30T08:32:31.552142+0000","last_fullsized":"2024-04-30T08:32:31.552142+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:44:25.327977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.567591+0000","last_change":"2024-04-30T08:31:52.709794+0000","last_active":"2024-04-30T08:32:31.567591+0000","last_peered":"2024-04-30T08:32:31.567591+0000","last_clean":"2024-04-30T08:32:31.567591+0000","last_became_active":"2024-04-30T08:31:52.709371+0000","last_became_peered":"2024-04-30T08:31:52.709371+0000","last_unstale":"2024-04-30T08:32:31.567591+0000","last_undegraded":"2024-04-30T08:32:31.567591+0000","last_fullsized":"2024-04-30T08:32:31.567591+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:36:59.892551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.559308+0000","last_change":"2024-04-30T08:31:57.757224+0000","last_active":"2024-04-30T08:32:31.559308+0000","last_peered":"2024-04-30T08:32:31.559308+0000","last_clean":"2024-04-30T08:32:31.559308+0000","last_became_active":"2024-04-30T08:31:57.756658+0000","last_became_peered":"2024-04-30T08:31:57.756658+0000","last_unstale":"2024-04-30T08:32:31.559308+0000","last_undegraded":"2024-04-30T08:32:31.559308+0000","last_fullsized":"2024-04-30T08:32:31.559308+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:01:21.309322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"81'3","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.555826+0000","last_change":"2024-04-30T08:31:51.678944+0000","last_active":"2024-04-30T08:32:31.555826+0000","last_peered":"2024-04-30T08:32:31.555826+0000","last_clean":"2024-04-30T08:32:31.555826+0000","last_became_active":"2024-04-30T08:31:51.678493+0000","last_became_peered":"2024-04-30T08:31:51.678493+0000","last_unstale":"2024-04-30T08:32:31.555826+0000","last_undegraded":"2024-04-30T08:32:31.555826+0000","last_fullsized":"2024-04-30T08:32:31.555826+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:48:11.111466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.568701+0000","last_change":"2024-04-30T08:31:52.707627+0000","last_active":"2024-04-30T08:32:31.568701+0000","last_peered":"2024-04-30T08:32:31.568701+0000","last_clean":"2024-04-30T08:32:31.568701+0000","last_became_active":"2024-04-30T08:31:52.706441+0000","last_became_peered":"2024-04-30T08:31:52.706441+0000","last_unstale":"2024-04-30T08:32:31.568701+0000","last_undegraded":"2024-04-30T08:32:31.568701+0000","last_fullsized":"2024-04-30T08:32:31.568701+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:15:45.768927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.565083+0000","last_change":"2024-04-30T08:31:57.740324+0000","last_active":"2024-04-30T08:32:31.565083+0000","last_peered":"2024-04-30T08:32:31.565083+0000","last_clean":"2024-04-30T08:32:31.565083+0000","last_became_active":"2024-04-30T08:31:57.739464+0000","last_became_peered":"2024-04-30T08:31:57.739464+0000","last_unstale":"2024-04-30T08:32:31.565083+0000","last_undegraded":"2024-04-30T08:32:31.565083+0000","last_fullsized":"2024-04-30T08:32:31.565083+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:11:37.395545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"81'4","reported_seq":40,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:33.567649+0000","last_change":"2024-04-30T08:31:51.695283+0000","last_active":"2024-04-30T08:32:33.567649+0000","last_peered":"2024-04-30T08:32:33.567649+0000","last_clean":"2024-04-30T08:32:33.567649+0000","last_became_active":"2024-04-30T08:31:51.694848+0000","last_became_peered":"2024-04-30T08:31:51.694848+0000","last_unstale":"2024-04-30T08:32:33.567649+0000","last_undegraded":"2024-04-30T08:32:33.567649+0000","last_fullsized":"2024-04-30T08:32:33.567649+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:41:38.638034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.585668+0000","last_change":"2024-04-30T08:31:52.693594+0000","last_active":"2024-04-30T08:32:31.585668+0000","last_peered":"2024-04-30T08:32:31.585668+0000","last_clean":"2024-04-30T08:32:31.585668+0000","last_became_active":"2024-04-30T08:31:52.693199+0000","last_became_peered":"2024-04-30T08:31:52.693199+0000","last_unstale":"2024-04-30T08:32:31.585668+0000","last_undegraded":"2024-04-30T08:32:31.585668+0000","last_fullsized":"2024-04-30T08:32:31.585668+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:57:49.908617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.568191+0000","last_change":"2024-04-30T08:31:57.749727+0000","last_active":"2024-04-30T08:32:31.568191+0000","last_peered":"2024-04-30T08:32:31.568191+0000","last_clean":"2024-04-30T08:32:31.568191+0000","last_became_active":"2024-04-30T08:31:57.749267+0000","last_became_peered":"2024-04-30T08:31:57.749267+0000","last_unstale":"2024-04-30T08:32:31.568191+0000","last_undegraded":"2024-04-30T08:32:31.568191+0000","last_fullsized":"2024-04-30T08:32:31.568191+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:26:12.683008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"81'4","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.557872+0000","last_change":"2024-04-30T08:31:51.677398+0000","last_active":"2024-04-30T08:32:31.557872+0000","last_peered":"2024-04-30T08:32:31.557872+0000","last_clean":"2024-04-30T08:32:31.557872+0000","last_became_active":"2024-04-30T08:31:51.677031+0000","last_became_peered":"2024-04-30T08:31:51.677031+0000","last_unstale":"2024-04-30T08:32:31.557872+0000","last_undegraded":"2024-04-30T08:32:31.557872+0000","last_fullsized":"2024-04-30T08:32:31.557872+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:24:25.154229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":32,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.561554+0000","last_change":"2024-04-30T08:32:31.561554+0000","last_active":"2024-04-30T08:32:31.561554+0000","last_peered":"2024-04-30T08:32:31.561554+0000","last_clean":"2024-04-30T08:32:31.561554+0000","last_became_active":"2024-04-30T08:32:31.560253+0000","last_became_peered":"2024-04-30T08:32:31.560253+0000","last_unstale":"2024-04-30T08:32:31.561554+0000","last_undegraded":"2024-04-30T08:32:31.561554+0000","last_fullsized":"2024-04-30T08:32:31.561554+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:59:08.602282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.586746+0000","last_change":"2024-04-30T08:32:31.586746+0000","last_active":"2024-04-30T08:32:31.586746+0000","last_peered":"2024-04-30T08:32:31.586746+0000","last_clean":"2024-04-30T08:32:31.586746+0000","last_became_active":"2024-04-30T08:32:31.586356+0000","last_became_peered":"2024-04-30T08:32:31.586356+0000","last_unstale":"2024-04-30T08:32:31.586746+0000","last_undegraded":"2024-04-30T08:32:31.586746+0000","last_fullsized":"2024-04-30T08:32:31.586746+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:22:39.371508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"81'5","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.553126+0000","last_change":"2024-04-30T08:31:51.694424+0000","last_active":"2024-04-30T08:32:31.553126+0000","last_peered":"2024-04-30T08:32:31.553126+0000","last_clean":"2024-04-30T08:32:31.553126+0000","last_became_active":"2024-04-30T08:31:51.694081+0000","last_became_peered":"2024-04-30T08:31:51.694081+0000","last_unstale":"2024-04-30T08:32:31.553126+0000","last_undegraded":"2024-04-30T08:32:31.553126+0000","last_fullsized":"2024-04-30T08:32:31.553126+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:27:33.328477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.585313+0000","last_change":"2024-04-30T08:31:52.684739+0000","last_active":"2024-04-30T08:32:31.585313+0000","last_peered":"2024-04-30T08:32:31.585313+0000","last_clean":"2024-04-30T08:32:31.585313+0000","last_became_active":"2024-04-30T08:31:52.684316+0000","last_became_peered":"2024-04-30T08:31:52.684316+0000","last_unstale":"2024-04-30T08:32:31.585313+0000","last_undegraded":"2024-04-30T08:32:31.585313+0000","last_fullsized":"2024-04-30T08:32:31.585313+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:12:05.437493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.558429+0000","last_change":"2024-04-30T08:32:31.557578+0000","last_active":"2024-04-30T08:32:31.558429+0000","last_peered":"2024-04-30T08:32:31.558429+0000","last_clean":"2024-04-30T08:32:31.558429+0000","last_became_active":"2024-04-30T08:32:31.557115+0000","last_became_peered":"2024-04-30T08:32:31.557115+0000","last_unstale":"2024-04-30T08:32:31.558429+0000","last_undegraded":"2024-04-30T08:32:31.558429+0000","last_fullsized":"2024-04-30T08:32:31.558429+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:41:33.225879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"82'26","reported_seq":64,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:33.585605+0000","last_change":"2024-04-30T08:31:51.676223+0000","last_active":"2024-04-30T08:32:33.585605+0000","last_peered":"2024-04-30T08:32:33.585605+0000","last_clean":"2024-04-30T08:32:33.585605+0000","last_became_active":"2024-04-30T08:31:51.675830+0000","last_became_peered":"2024-04-30T08:31:51.675830+0000","last_unstale":"2024-04-30T08:32:33.585605+0000","last_undegraded":"2024-04-30T08:32:33.585605+0000","last_fullsized":"2024-04-30T08:32:33.585605+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:28:49.497719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":41906,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":9,"num_read_kb":45,"num_write":28,"num_write_kb":51,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.556767+0000","last_change":"2024-04-30T08:31:52.686892+0000","last_active":"2024-04-30T08:32:31.556767+0000","last_peered":"2024-04-30T08:32:31.556767+0000","last_clean":"2024-04-30T08:32:31.556767+0000","last_became_active":"2024-04-30T08:31:52.686533+0000","last_became_peered":"2024-04-30T08:31:52.686533+0000","last_unstale":"2024-04-30T08:32:31.556767+0000","last_undegraded":"2024-04-30T08:32:31.556767+0000","last_fullsized":"2024-04-30T08:32:31.556767+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:50:18.836161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.553606+0000","last_change":"2024-04-30T08:31:57.752865+0000","last_active":"2024-04-30T08:32:31.553606+0000","last_peered":"2024-04-30T08:32:31.553606+0000","last_clean":"2024-04-30T08:32:31.553606+0000","last_became_active":"2024-04-30T08:31:57.752385+0000","last_became_peered":"2024-04-30T08:31:57.752385+0000","last_unstale":"2024-04-30T08:32:31.553606+0000","last_undegraded":"2024-04-30T08:32:31.553606+0000","last_fullsized":"2024-04-30T08:32:31.553606+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:15:15.315580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"81'9","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.585282+0000","last_change":"2024-04-30T08:32:31.585282+0000","last_active":"2024-04-30T08:32:31.585282+0000","last_peered":"2024-04-30T08:32:31.585282+0000","last_clean":"2024-04-30T08:32:31.585282+0000","last_became_active":"2024-04-30T08:32:31.571676+0000","last_became_peered":"2024-04-30T08:32:31.571676+0000","last_unstale":"2024-04-30T08:32:31.585282+0000","last_undegraded":"2024-04-30T08:32:31.585282+0000","last_fullsized":"2024-04-30T08:32:31.585282+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:18:29.910201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.552561+0000","last_change":"2024-04-30T08:31:52.706259+0000","last_active":"2024-04-30T08:32:31.552561+0000","last_peered":"2024-04-30T08:32:31.552561+0000","last_clean":"2024-04-30T08:32:31.552561+0000","last_became_active":"2024-04-30T08:31:52.705620+0000","last_became_peered":"2024-04-30T08:31:52.705620+0000","last_unstale":"2024-04-30T08:32:31.552561+0000","last_undegraded":"2024-04-30T08:32:31.552561+0000","last_fullsized":"2024-04-30T08:32:31.552561+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:22:29.816950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":25,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.562571+0000","last_change":"2024-04-30T08:32:31.562571+0000","last_active":"2024-04-30T08:32:31.562571+0000","last_peered":"2024-04-30T08:32:31.562571+0000","last_clean":"2024-04-30T08:32:31.562571+0000","last_became_active":"2024-04-30T08:32:31.562012+0000","last_became_peered":"2024-04-30T08:32:31.562012+0000","last_unstale":"2024-04-30T08:32:31.562571+0000","last_undegraded":"2024-04-30T08:32:31.562571+0000","last_fullsized":"2024-04-30T08:32:31.562571+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:12:22.928286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"81'5","reported_seq":35,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.554892+0000","last_change":"2024-04-30T08:31:51.701122+0000","last_active":"2024-04-30T08:32:31.554892+0000","last_peered":"2024-04-30T08:32:31.554892+0000","last_clean":"2024-04-30T08:32:31.554892+0000","last_became_active":"2024-04-30T08:31:51.700815+0000","last_became_peered":"2024-04-30T08:31:51.700815+0000","last_unstale":"2024-04-30T08:32:31.554892+0000","last_undegraded":"2024-04-30T08:32:31.554892+0000","last_fullsized":"2024-04-30T08:32:31.554892+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:33:12.714979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.558463+0000","last_change":"2024-04-30T08:31:52.716043+0000","last_active":"2024-04-30T08:32:31.558463+0000","last_peered":"2024-04-30T08:32:31.558463+0000","last_clean":"2024-04-30T08:32:31.558463+0000","last_became_active":"2024-04-30T08:31:52.715284+0000","last_became_peered":"2024-04-30T08:31:52.715284+0000","last_unstale":"2024-04-30T08:32:31.558463+0000","last_undegraded":"2024-04-30T08:32:31.558463+0000","last_fullsized":"2024-04-30T08:32:31.558463+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:00:01.863606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.569689+0000","last_change":"2024-04-30T08:31:57.765008+0000","last_active":"2024-04-30T08:32:31.569689+0000","last_peered":"2024-04-30T08:32:31.569689+0000","last_clean":"2024-04-30T08:32:31.569689+0000","last_became_active":"2024-04-30T08:31:57.762958+0000","last_became_peered":"2024-04-30T08:31:57.762958+0000","last_unstale":"2024-04-30T08:32:31.569689+0000","last_undegraded":"2024-04-30T08:32:31.569689+0000","last_fullsized":"2024-04-30T08:32:31.569689+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:48:47.116004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"81'6","reported_seq":41,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:32.572977+0000","last_change":"2024-04-30T08:31:51.676681+0000","last_active":"2024-04-30T08:32:32.572977+0000","last_peered":"2024-04-30T08:32:32.572977+0000","last_clean":"2024-04-30T08:32:32.572977+0000","last_became_active":"2024-04-30T08:31:51.676311+0000","last_became_peered":"2024-04-30T08:31:51.676311+0000","last_unstale":"2024-04-30T08:32:32.572977+0000","last_undegraded":"2024-04-30T08:32:32.572977+0000","last_fullsized":"2024-04-30T08:32:32.572977+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:24:49.757427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":7,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.558213+0000","last_change":"2024-04-30T08:31:52.689418+0000","last_active":"2024-04-30T08:32:31.558213+0000","last_peered":"2024-04-30T08:32:31.558213+0000","last_clean":"2024-04-30T08:32:31.558213+0000","last_became_active":"2024-04-30T08:31:52.688840+0000","last_became_peered":"2024-04-30T08:31:52.688840+0000","last_unstale":"2024-04-30T08:32:31.558213+0000","last_undegraded":"2024-04-30T08:32:31.558213+0000","last_fullsized":"2024-04-30T08:32:31.558213+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:40:05.781137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.551981+0000","last_change":"2024-04-30T08:31:57.765875+0000","last_active":"2024-04-30T08:32:31.551981+0000","last_peered":"2024-04-30T08:32:31.551981+0000","last_clean":"2024-04-30T08:32:31.551981+0000","last_became_active":"2024-04-30T08:31:57.765432+0000","last_became_peered":"2024-04-30T08:31:57.765432+0000","last_unstale":"2024-04-30T08:32:31.551981+0000","last_undegraded":"2024-04-30T08:32:31.551981+0000","last_fullsized":"2024-04-30T08:32:31.551981+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:24:44.329532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"81'17","reported_seq":46,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.554050+0000","last_change":"2024-04-30T08:31:51.695007+0000","last_active":"2024-04-30T08:32:31.554050+0000","last_peered":"2024-04-30T08:32:31.554050+0000","last_clean":"2024-04-30T08:32:31.554050+0000","last_became_active":"2024-04-30T08:31:51.694533+0000","last_became_peered":"2024-04-30T08:31:51.694533+0000","last_unstale":"2024-04-30T08:32:31.554050+0000","last_undegraded":"2024-04-30T08:32:31.554050+0000","last_fullsized":"2024-04-30T08:32:31.554050+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:47:15.104172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.553520+0000","last_change":"2024-04-30T08:31:52.692274+0000","last_active":"2024-04-30T08:32:31.553520+0000","last_peered":"2024-04-30T08:32:31.553520+0000","last_clean":"2024-04-30T08:32:31.553520+0000","last_became_active":"2024-04-30T08:31:52.691809+0000","last_became_peered":"2024-04-30T08:31:52.691809+0000","last_unstale":"2024-04-30T08:32:31.553520+0000","last_undegraded":"2024-04-30T08:32:31.553520+0000","last_fullsized":"2024-04-30T08:32:31.553520+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:50:00.201782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.555077+0000","last_change":"2024-04-30T08:31:57.732383+0000","last_active":"2024-04-30T08:32:31.555077+0000","last_peered":"2024-04-30T08:32:31.555077+0000","last_clean":"2024-04-30T08:32:31.555077+0000","last_became_active":"2024-04-30T08:31:57.731893+0000","last_became_peered":"2024-04-30T08:31:57.731893+0000","last_unstale":"2024-04-30T08:32:31.555077+0000","last_undegraded":"2024-04-30T08:32:31.555077+0000","last_fullsized":"2024-04-30T08:32:31.555077+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:08:55.243526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"81'2","reported_seq":31,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.556534+0000","last_change":"2024-04-30T08:31:51.671991+0000","last_active":"2024-04-30T08:32:31.556534+0000","last_peered":"2024-04-30T08:32:31.556534+0000","last_clean":"2024-04-30T08:32:31.556534+0000","last_became_active":"2024-04-30T08:31:51.671508+0000","last_became_peered":"2024-04-30T08:31:51.671508+0000","last_unstale":"2024-04-30T08:32:31.556534+0000","last_undegraded":"2024-04-30T08:32:31.556534+0000","last_fullsized":"2024-04-30T08:32:31.556534+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:54:30.106748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.554259+0000","last_change":"2024-04-30T08:31:52.687555+0000","last_active":"2024-04-30T08:32:31.554259+0000","last_peered":"2024-04-30T08:32:31.554259+0000","last_clean":"2024-04-30T08:32:31.554259+0000","last_became_active":"2024-04-30T08:31:52.686000+0000","last_became_peered":"2024-04-30T08:31:52.686000+0000","last_unstale":"2024-04-30T08:32:31.554259+0000","last_undegraded":"2024-04-30T08:32:31.554259+0000","last_fullsized":"2024-04-30T08:32:31.554259+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:37:03.529774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.555216+0000","last_change":"2024-04-30T08:31:58.061077+0000","last_active":"2024-04-30T08:32:31.555216+0000","last_peered":"2024-04-30T08:32:31.555216+0000","last_clean":"2024-04-30T08:32:31.555216+0000","last_became_active":"2024-04-30T08:31:58.060486+0000","last_became_peered":"2024-04-30T08:31:58.060486+0000","last_unstale":"2024-04-30T08:32:31.555216+0000","last_undegraded":"2024-04-30T08:32:31.555216+0000","last_fullsized":"2024-04-30T08:32:31.555216+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:31:50.936560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"81'1","reported_seq":29,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.570855+0000","last_change":"2024-04-30T08:31:51.702869+0000","last_active":"2024-04-30T08:32:31.570855+0000","last_peered":"2024-04-30T08:32:31.570855+0000","last_clean":"2024-04-30T08:32:31.570855+0000","last_became_active":"2024-04-30T08:31:51.701150+0000","last_became_peered":"2024-04-30T08:31:51.701150+0000","last_unstale":"2024-04-30T08:32:31.570855+0000","last_undegraded":"2024-04-30T08:32:31.570855+0000","last_fullsized":"2024-04-30T08:32:31.570855+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:47:11.010310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.552471+0000","last_change":"2024-04-30T08:31:52.709159+0000","last_active":"2024-04-30T08:32:31.552471+0000","last_peered":"2024-04-30T08:32:31.552471+0000","last_clean":"2024-04-30T08:32:31.552471+0000","last_became_active":"2024-04-30T08:31:52.708799+0000","last_became_peered":"2024-04-30T08:31:52.708799+0000","last_unstale":"2024-04-30T08:32:31.552471+0000","last_undegraded":"2024-04-30T08:32:31.552471+0000","last_fullsized":"2024-04-30T08:32:31.552471+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:51:10.588599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.553255+0000","last_change":"2024-04-30T08:31:57.751652+0000","last_active":"2024-04-30T08:32:31.553255+0000","last_peered":"2024-04-30T08:32:31.553255+0000","last_clean":"2024-04-30T08:32:31.553255+0000","last_became_active":"2024-04-30T08:31:57.751176+0000","last_became_peered":"2024-04-30T08:31:57.751176+0000","last_unstale":"2024-04-30T08:32:31.553255+0000","last_undegraded":"2024-04-30T08:32:31.553255+0000","last_fullsized":"2024-04-30T08:32:31.553255+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:13:24.528842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"81'5","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.571485+0000","last_change":"2024-04-30T08:31:51.700816+0000","last_active":"2024-04-30T08:32:31.571485+0000","last_peered":"2024-04-30T08:32:31.571485+0000","last_clean":"2024-04-30T08:32:31.571485+0000","last_became_active":"2024-04-30T08:31:51.700470+0000","last_became_peered":"2024-04-30T08:31:51.700470+0000","last_unstale":"2024-04-30T08:32:31.571485+0000","last_undegraded":"2024-04-30T08:32:31.571485+0000","last_fullsized":"2024-04-30T08:32:31.571485+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:49:11.793412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.552011+0000","last_change":"2024-04-30T08:31:52.715236+0000","last_active":"2024-04-30T08:32:31.552011+0000","last_peered":"2024-04-30T08:32:31.552011+0000","last_clean":"2024-04-30T08:32:31.552011+0000","last_became_active":"2024-04-30T08:31:52.714801+0000","last_became_peered":"2024-04-30T08:31:52.714801+0000","last_unstale":"2024-04-30T08:32:31.552011+0000","last_undegraded":"2024-04-30T08:32:31.552011+0000","last_fullsized":"2024-04-30T08:32:31.552011+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:43:09.650002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.553679+0000","last_change":"2024-04-30T08:31:58.060927+0000","last_active":"2024-04-30T08:32:31.553679+0000","last_peered":"2024-04-30T08:32:31.553679+0000","last_clean":"2024-04-30T08:32:31.553679+0000","last_became_active":"2024-04-30T08:31:58.060182+0000","last_became_peered":"2024-04-30T08:31:58.060182+0000","last_unstale":"2024-04-30T08:32:31.553679+0000","last_undegraded":"2024-04-30T08:32:31.553679+0000","last_fullsized":"2024-04-30T08:32:31.553679+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:21:00.228064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.560485+0000","last_change":"2024-04-30T08:32:31.560485+0000","last_active":"2024-04-30T08:32:31.560485+0000","last_peered":"2024-04-30T08:32:31.560485+0000","last_clean":"2024-04-30T08:32:31.560485+0000","last_became_active":"2024-04-30T08:32:31.560066+0000","last_became_peered":"2024-04-30T08:32:31.560066+0000","last_unstale":"2024-04-30T08:32:31.560485+0000","last_undegraded":"2024-04-30T08:32:31.560485+0000","last_fullsized":"2024-04-30T08:32:31.560485+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:13:40.769761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"81'5","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:33.574285+0000","last_change":"2024-04-30T08:31:51.702808+0000","last_active":"2024-04-30T08:32:33.574285+0000","last_peered":"2024-04-30T08:32:33.574285+0000","last_clean":"2024-04-30T08:32:33.574285+0000","last_became_active":"2024-04-30T08:31:51.702326+0000","last_became_peered":"2024-04-30T08:31:51.702326+0000","last_unstale":"2024-04-30T08:32:33.574285+0000","last_undegraded":"2024-04-30T08:32:33.574285+0000","last_fullsized":"2024-04-30T08:32:33.574285+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:11:35.552670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":6,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"81'1","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.557866+0000","last_change":"2024-04-30T08:31:57.760359+0000","last_active":"2024-04-30T08:32:31.557866+0000","last_peered":"2024-04-30T08:32:31.557866+0000","last_clean":"2024-04-30T08:32:31.557866+0000","last_became_active":"2024-04-30T08:31:57.759821+0000","last_became_peered":"2024-04-30T08:31:57.759821+0000","last_unstale":"2024-04-30T08:32:31.557866+0000","last_undegraded":"2024-04-30T08:32:31.557866+0000","last_fullsized":"2024-04-30T08:32:31.557866+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:50:17.853791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.566198+0000","last_change":"2024-04-30T08:31:52.689281+0000","last_active":"2024-04-30T08:32:31.566198+0000","last_peered":"2024-04-30T08:32:31.566198+0000","last_clean":"2024-04-30T08:32:31.566198+0000","last_became_active":"2024-04-30T08:31:52.688834+0000","last_became_peered":"2024-04-30T08:31:52.688834+0000","last_unstale":"2024-04-30T08:32:31.566198+0000","last_undegraded":"2024-04-30T08:32:31.566198+0000","last_fullsized":"2024-04-30T08:32:31.566198+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:42:12.836649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'6","reported_seq":38,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550187+0000","last_change":"2024-04-30T08:31:51.683712+0000","last_active":"2024-04-30T08:32:31.550187+0000","last_peered":"2024-04-30T08:32:31.550187+0000","last_clean":"2024-04-30T08:32:31.550187+0000","last_became_active":"2024-04-30T08:31:51.683298+0000","last_became_peered":"2024-04-30T08:31:51.683298+0000","last_unstale":"2024-04-30T08:32:31.550187+0000","last_undegraded":"2024-04-30T08:32:31.550187+0000","last_fullsized":"2024-04-30T08:32:31.550187+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:43:08.832104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":103,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.557511+0000","last_change":"2024-04-30T08:30:26.503221+0000","last_active":"2024-04-30T08:32:31.557511+0000","last_peered":"2024-04-30T08:32:31.557511+0000","last_clean":"2024-04-30T08:32:31.557511+0000","last_became_active":"2024-04-30T08:30:26.498399+0000","last_became_peered":"2024-04-30T08:30:26.498399+0000","last_unstale":"2024-04-30T08:32:31.557511+0000","last_undegraded":"2024-04-30T08:32:31.557511+0000","last_fullsized":"2024-04-30T08:32:31.557511+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:26:20.170075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:26:20.170075+0000","last_clean_scrub_stamp":"2024-04-30T08:26:20.170075+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:34:44.067560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.552597+0000","last_change":"2024-04-30T08:31:58.061283+0000","last_active":"2024-04-30T08:32:31.552597+0000","last_peered":"2024-04-30T08:32:31.552597+0000","last_clean":"2024-04-30T08:32:31.552597+0000","last_became_active":"2024-04-30T08:31:58.060650+0000","last_became_peered":"2024-04-30T08:31:58.060650+0000","last_unstale":"2024-04-30T08:32:31.552597+0000","last_undegraded":"2024-04-30T08:32:31.552597+0000","last_fullsized":"2024-04-30T08:32:31.552597+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:13:01.811462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.551872+0000","last_change":"2024-04-30T08:31:52.699639+0000","last_active":"2024-04-30T08:32:31.551872+0000","last_peered":"2024-04-30T08:32:31.551872+0000","last_clean":"2024-04-30T08:32:31.551872+0000","last_became_active":"2024-04-30T08:31:52.699164+0000","last_became_peered":"2024-04-30T08:31:52.699164+0000","last_unstale":"2024-04-30T08:32:31.551872+0000","last_undegraded":"2024-04-30T08:32:31.551872+0000","last_fullsized":"2024-04-30T08:32:31.551872+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:22:18.873504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"81'4","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.558080+0000","last_change":"2024-04-30T08:31:51.679153+0000","last_active":"2024-04-30T08:32:31.558080+0000","last_peered":"2024-04-30T08:32:31.558080+0000","last_clean":"2024-04-30T08:32:31.558080+0000","last_became_active":"2024-04-30T08:31:51.678348+0000","last_became_peered":"2024-04-30T08:31:51.678348+0000","last_unstale":"2024-04-30T08:32:31.558080+0000","last_undegraded":"2024-04-30T08:32:31.558080+0000","last_fullsized":"2024-04-30T08:32:31.558080+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:02:56.240218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.555987+0000","last_change":"2024-04-30T08:31:57.755423+0000","last_active":"2024-04-30T08:32:31.555987+0000","last_peered":"2024-04-30T08:32:31.555987+0000","last_clean":"2024-04-30T08:32:31.555987+0000","last_became_active":"2024-04-30T08:31:57.754835+0000","last_became_peered":"2024-04-30T08:31:57.754835+0000","last_unstale":"2024-04-30T08:32:31.555987+0000","last_undegraded":"2024-04-30T08:32:31.555987+0000","last_fullsized":"2024-04-30T08:32:31.555987+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:19:25.168920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"81'4","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.556916+0000","last_change":"2024-04-30T08:31:51.693839+0000","last_active":"2024-04-30T08:32:31.556916+0000","last_peered":"2024-04-30T08:32:31.556916+0000","last_clean":"2024-04-30T08:32:31.556916+0000","last_became_active":"2024-04-30T08:31:51.693349+0000","last_became_peered":"2024-04-30T08:31:51.693349+0000","last_unstale":"2024-04-30T08:32:31.556916+0000","last_undegraded":"2024-04-30T08:32:31.556916+0000","last_fullsized":"2024-04-30T08:32:31.556916+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:41:51.330901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.553463+0000","last_change":"2024-04-30T08:31:52.712360+0000","last_active":"2024-04-30T08:32:31.553463+0000","last_peered":"2024-04-30T08:32:31.553463+0000","last_clean":"2024-04-30T08:32:31.553463+0000","last_became_active":"2024-04-30T08:31:52.711652+0000","last_became_peered":"2024-04-30T08:31:52.711652+0000","last_unstale":"2024-04-30T08:32:31.553463+0000","last_undegraded":"2024-04-30T08:32:31.553463+0000","last_fullsized":"2024-04-30T08:32:31.553463+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:21:10.285386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.567607+0000","last_change":"2024-04-30T08:31:58.061654+0000","last_active":"2024-04-30T08:32:31.567607+0000","last_peered":"2024-04-30T08:32:31.567607+0000","last_clean":"2024-04-30T08:32:31.567607+0000","last_became_active":"2024-04-30T08:31:58.060181+0000","last_became_peered":"2024-04-30T08:31:58.060181+0000","last_unstale":"2024-04-30T08:32:31.567607+0000","last_undegraded":"2024-04-30T08:32:31.567607+0000","last_fullsized":"2024-04-30T08:32:31.567607+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:46:03.230626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"83'11","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:35.245365+0000","last_change":"2024-04-30T08:31:51.706940+0000","last_active":"2024-04-30T08:32:35.245365+0000","last_peered":"2024-04-30T08:32:35.245365+0000","last_clean":"2024-04-30T08:32:35.245365+0000","last_became_active":"2024-04-30T08:31:51.706481+0000","last_became_peered":"2024-04-30T08:31:51.706481+0000","last_unstale":"2024-04-30T08:32:35.245365+0000","last_undegraded":"2024-04-30T08:32:35.245365+0000","last_fullsized":"2024-04-30T08:32:35.245365+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:53:31.955595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.565718+0000","last_change":"2024-04-30T08:31:52.696625+0000","last_active":"2024-04-30T08:32:31.565718+0000","last_peered":"2024-04-30T08:32:31.565718+0000","last_clean":"2024-04-30T08:32:31.565718+0000","last_became_active":"2024-04-30T08:31:52.696230+0000","last_became_peered":"2024-04-30T08:31:52.696230+0000","last_unstale":"2024-04-30T08:32:31.565718+0000","last_undegraded":"2024-04-30T08:32:31.565718+0000","last_fullsized":"2024-04-30T08:32:31.565718+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:11:46.404251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550372+0000","last_change":"2024-04-30T08:31:57.738329+0000","last_active":"2024-04-30T08:32:31.550372+0000","last_peered":"2024-04-30T08:32:31.550372+0000","last_clean":"2024-04-30T08:32:31.550372+0000","last_became_active":"2024-04-30T08:31:57.737897+0000","last_became_peered":"2024-04-30T08:31:57.737897+0000","last_unstale":"2024-04-30T08:32:31.550372+0000","last_undegraded":"2024-04-30T08:32:31.550372+0000","last_fullsized":"2024-04-30T08:32:31.550372+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:02:43.074578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"81'7","reported_seq":37,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550515+0000","last_change":"2024-04-30T08:31:51.684066+0000","last_active":"2024-04-30T08:32:31.550515+0000","last_peered":"2024-04-30T08:32:31.550515+0000","last_clean":"2024-04-30T08:32:31.550515+0000","last_became_active":"2024-04-30T08:31:51.683699+0000","last_became_peered":"2024-04-30T08:31:51.683699+0000","last_unstale":"2024-04-30T08:32:31.550515+0000","last_undegraded":"2024-04-30T08:32:31.550515+0000","last_fullsized":"2024-04-30T08:32:31.550515+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:36:20.900733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.559969+0000","last_change":"2024-04-30T08:31:52.715219+0000","last_active":"2024-04-30T08:32:31.559969+0000","last_peered":"2024-04-30T08:32:31.559969+0000","last_clean":"2024-04-30T08:32:31.559969+0000","last_became_active":"2024-04-30T08:31:52.714774+0000","last_became_peered":"2024-04-30T08:31:52.714774+0000","last_unstale":"2024-04-30T08:32:31.559969+0000","last_undegraded":"2024-04-30T08:32:31.559969+0000","last_fullsized":"2024-04-30T08:32:31.559969+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:42:41.082610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550306+0000","last_change":"2024-04-30T08:31:58.060845+0000","last_active":"2024-04-30T08:32:31.550306+0000","last_peered":"2024-04-30T08:32:31.550306+0000","last_clean":"2024-04-30T08:32:31.550306+0000","last_became_active":"2024-04-30T08:31:58.060365+0000","last_became_peered":"2024-04-30T08:31:58.060365+0000","last_unstale":"2024-04-30T08:32:31.550306+0000","last_undegraded":"2024-04-30T08:32:31.550306+0000","last_fullsized":"2024-04-30T08:32:31.550306+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:21:39.583243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"81'2","reported_seq":30,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.585367+0000","last_change":"2024-04-30T08:31:51.694833+0000","last_active":"2024-04-30T08:32:31.585367+0000","last_peered":"2024-04-30T08:32:31.585367+0000","last_clean":"2024-04-30T08:32:31.585367+0000","last_became_active":"2024-04-30T08:31:51.694401+0000","last_became_peered":"2024-04-30T08:31:51.694401+0000","last_unstale":"2024-04-30T08:32:31.585367+0000","last_undegraded":"2024-04-30T08:32:31.585367+0000","last_fullsized":"2024-04-30T08:32:31.585367+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:38:52.605625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.553210+0000","last_change":"2024-04-30T08:31:52.686444+0000","last_active":"2024-04-30T08:32:31.553210+0000","last_peered":"2024-04-30T08:32:31.553210+0000","last_clean":"2024-04-30T08:32:31.553210+0000","last_became_active":"2024-04-30T08:31:52.686057+0000","last_became_peered":"2024-04-30T08:31:52.686057+0000","last_unstale":"2024-04-30T08:32:31.553210+0000","last_undegraded":"2024-04-30T08:32:31.553210+0000","last_fullsized":"2024-04-30T08:32:31.553210+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:54:29.824060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.569963+0000","last_change":"2024-04-30T08:31:57.766193+0000","last_active":"2024-04-30T08:32:31.569963+0000","last_peered":"2024-04-30T08:32:31.569963+0000","last_clean":"2024-04-30T08:32:31.569963+0000","last_became_active":"2024-04-30T08:31:57.765758+0000","last_became_peered":"2024-04-30T08:31:57.765758+0000","last_unstale":"2024-04-30T08:32:31.569963+0000","last_undegraded":"2024-04-30T08:32:31.569963+0000","last_fullsized":"2024-04-30T08:32:31.569963+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:59:57.282795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"81'7","reported_seq":36,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.565708+0000","last_change":"2024-04-30T08:31:51.704565+0000","last_active":"2024-04-30T08:32:31.565708+0000","last_peered":"2024-04-30T08:32:31.565708+0000","last_clean":"2024-04-30T08:32:31.565708+0000","last_became_active":"2024-04-30T08:31:51.703954+0000","last_became_peered":"2024-04-30T08:31:51.703954+0000","last_unstale":"2024-04-30T08:32:31.565708+0000","last_undegraded":"2024-04-30T08:32:31.565708+0000","last_fullsized":"2024-04-30T08:32:31.565708+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:03:00.517867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.553953+0000","last_change":"2024-04-30T08:31:52.706043+0000","last_active":"2024-04-30T08:32:31.553953+0000","last_peered":"2024-04-30T08:32:31.553953+0000","last_clean":"2024-04-30T08:32:31.553953+0000","last_became_active":"2024-04-30T08:31:52.705678+0000","last_became_peered":"2024-04-30T08:31:52.705678+0000","last_unstale":"2024-04-30T08:32:31.553953+0000","last_undegraded":"2024-04-30T08:32:31.553953+0000","last_fullsized":"2024-04-30T08:32:31.553953+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:44:56.787835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"82'1","reported_seq":19,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550147+0000","last_change":"2024-04-30T08:31:57.764791+0000","last_active":"2024-04-30T08:32:31.550147+0000","last_peered":"2024-04-30T08:32:31.550147+0000","last_clean":"2024-04-30T08:32:31.550147+0000","last_became_active":"2024-04-30T08:31:57.764368+0000","last_became_peered":"2024-04-30T08:31:57.764368+0000","last_unstale":"2024-04-30T08:32:31.550147+0000","last_undegraded":"2024-04-30T08:32:31.550147+0000","last_fullsized":"2024-04-30T08:32:31.550147+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:19:28.545483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"81'3","reported_seq":17,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.577309+0000","last_change":"2024-04-30T08:32:31.577309+0000","last_active":"2024-04-30T08:32:31.577309+0000","last_peered":"2024-04-30T08:32:31.577309+0000","last_clean":"2024-04-30T08:32:31.577309+0000","last_became_active":"2024-04-30T08:32:31.571515+0000","last_became_peered":"2024-04-30T08:32:31.571515+0000","last_unstale":"2024-04-30T08:32:31.577309+0000","last_undegraded":"2024-04-30T08:32:31.577309+0000","last_fullsized":"2024-04-30T08:32:31.577309+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:28:48.408414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.566691+0000","last_change":"2024-04-30T08:31:52.716707+0000","last_active":"2024-04-30T08:32:31.566691+0000","last_peered":"2024-04-30T08:32:31.566691+0000","last_clean":"2024-04-30T08:32:31.566691+0000","last_became_active":"2024-04-30T08:31:52.716293+0000","last_became_peered":"2024-04-30T08:31:52.716293+0000","last_unstale":"2024-04-30T08:32:31.566691+0000","last_undegraded":"2024-04-30T08:32:31.566691+0000","last_fullsized":"2024-04-30T08:32:31.566691+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:29:54.223472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550379+0000","last_change":"2024-04-30T08:31:58.060640+0000","last_active":"2024-04-30T08:32:31.550379+0000","last_peered":"2024-04-30T08:32:31.550379+0000","last_clean":"2024-04-30T08:32:31.550379+0000","last_became_active":"2024-04-30T08:31:58.060061+0000","last_became_peered":"2024-04-30T08:31:58.060061+0000","last_unstale":"2024-04-30T08:32:31.550379+0000","last_undegraded":"2024-04-30T08:32:31.550379+0000","last_fullsized":"2024-04-30T08:32:31.550379+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:46:00.844191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"81'4","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.555331+0000","last_change":"2024-04-30T08:31:51.679532+0000","last_active":"2024-04-30T08:32:31.555331+0000","last_peered":"2024-04-30T08:32:31.555331+0000","last_clean":"2024-04-30T08:32:31.555331+0000","last_became_active":"2024-04-30T08:31:51.679141+0000","last_became_peered":"2024-04-30T08:31:51.679141+0000","last_unstale":"2024-04-30T08:32:31.555331+0000","last_undegraded":"2024-04-30T08:32:31.555331+0000","last_fullsized":"2024-04-30T08:32:31.555331+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:19:50.526563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.557596+0000","last_change":"2024-04-30T08:31:52.703613+0000","last_active":"2024-04-30T08:32:31.557596+0000","last_peered":"2024-04-30T08:32:31.557596+0000","last_clean":"2024-04-30T08:32:31.557596+0000","last_became_active":"2024-04-30T08:31:52.703196+0000","last_became_peered":"2024-04-30T08:31:52.703196+0000","last_unstale":"2024-04-30T08:32:31.557596+0000","last_undegraded":"2024-04-30T08:32:31.557596+0000","last_fullsized":"2024-04-30T08:32:31.557596+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:33:11.136313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.584954+0000","last_change":"2024-04-30T08:31:57.755286+0000","last_active":"2024-04-30T08:32:31.584954+0000","last_peered":"2024-04-30T08:32:31.584954+0000","last_clean":"2024-04-30T08:32:31.584954+0000","last_became_active":"2024-04-30T08:31:57.754786+0000","last_became_peered":"2024-04-30T08:31:57.754786+0000","last_unstale":"2024-04-30T08:32:31.584954+0000","last_undegraded":"2024-04-30T08:32:31.584954+0000","last_fullsized":"2024-04-30T08:32:31.584954+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:02:50.931553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"81'5","reported_seq":39,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:33.580030+0000","last_change":"2024-04-30T08:31:51.693317+0000","last_active":"2024-04-30T08:32:33.580030+0000","last_peered":"2024-04-30T08:32:33.580030+0000","last_clean":"2024-04-30T08:32:33.580030+0000","last_became_active":"2024-04-30T08:31:51.692928+0000","last_became_peered":"2024-04-30T08:31:51.692928+0000","last_unstale":"2024-04-30T08:32:33.580030+0000","last_undegraded":"2024-04-30T08:32:33.580030+0000","last_fullsized":"2024-04-30T08:32:33.580030+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:35:09.851735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":6,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.558242+0000","last_change":"2024-04-30T08:31:52.694304+0000","last_active":"2024-04-30T08:32:31.558242+0000","last_peered":"2024-04-30T08:32:31.558242+0000","last_clean":"2024-04-30T08:32:31.558242+0000","last_became_active":"2024-04-30T08:31:52.693317+0000","last_became_peered":"2024-04-30T08:31:52.693317+0000","last_unstale":"2024-04-30T08:32:31.558242+0000","last_undegraded":"2024-04-30T08:32:31.558242+0000","last_fullsized":"2024-04-30T08:32:31.558242+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:31:39.294864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.556289+0000","last_change":"2024-04-30T08:31:57.763598+0000","last_active":"2024-04-30T08:32:31.556289+0000","last_peered":"2024-04-30T08:32:31.556289+0000","last_clean":"2024-04-30T08:32:31.556289+0000","last_became_active":"2024-04-30T08:31:57.762307+0000","last_became_peered":"2024-04-30T08:31:57.762307+0000","last_unstale":"2024-04-30T08:32:31.556289+0000","last_undegraded":"2024-04-30T08:32:31.556289+0000","last_fullsized":"2024-04-30T08:32:31.556289+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:24:13.390595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"81'3","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.553337+0000","last_change":"2024-04-30T08:31:51.703476+0000","last_active":"2024-04-30T08:32:31.553337+0000","last_peered":"2024-04-30T08:32:31.553337+0000","last_clean":"2024-04-30T08:32:31.553337+0000","last_became_active":"2024-04-30T08:31:51.703072+0000","last_became_peered":"2024-04-30T08:31:51.703072+0000","last_unstale":"2024-04-30T08:32:31.553337+0000","last_undegraded":"2024-04-30T08:32:31.553337+0000","last_fullsized":"2024-04-30T08:32:31.553337+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:07:21.098223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.557803+0000","last_change":"2024-04-30T08:31:52.699223+0000","last_active":"2024-04-30T08:32:31.557803+0000","last_peered":"2024-04-30T08:32:31.557803+0000","last_clean":"2024-04-30T08:32:31.557803+0000","last_became_active":"2024-04-30T08:31:52.698837+0000","last_became_peered":"2024-04-30T08:31:52.698837+0000","last_unstale":"2024-04-30T08:32:31.557803+0000","last_undegraded":"2024-04-30T08:32:31.557803+0000","last_fullsized":"2024-04-30T08:32:31.557803+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:19:11.826470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":12,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.587430+0000","last_change":"2024-04-30T08:32:31.586667+0000","last_active":"2024-04-30T08:32:31.587430+0000","last_peered":"2024-04-30T08:32:31.587430+0000","last_clean":"2024-04-30T08:32:31.587430+0000","last_became_active":"2024-04-30T08:32:31.586144+0000","last_became_peered":"2024-04-30T08:32:31.586144+0000","last_unstale":"2024-04-30T08:32:31.587430+0000","last_undegraded":"2024-04-30T08:32:31.587430+0000","last_fullsized":"2024-04-30T08:32:31.587430+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:50:39.752485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7,1],"acting":[8,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"81'5","reported_seq":33,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.552340+0000","last_change":"2024-04-30T08:31:51.708943+0000","last_active":"2024-04-30T08:32:31.552340+0000","last_peered":"2024-04-30T08:32:31.552340+0000","last_clean":"2024-04-30T08:32:31.552340+0000","last_became_active":"2024-04-30T08:31:51.708376+0000","last_became_peered":"2024-04-30T08:31:51.708376+0000","last_unstale":"2024-04-30T08:32:31.552340+0000","last_undegraded":"2024-04-30T08:32:31.552340+0000","last_fullsized":"2024-04-30T08:32:31.552340+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:14:46.692192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.558028+0000","last_change":"2024-04-30T08:31:52.686634+0000","last_active":"2024-04-30T08:32:31.558028+0000","last_peered":"2024-04-30T08:32:31.558028+0000","last_clean":"2024-04-30T08:32:31.558028+0000","last_became_active":"2024-04-30T08:31:52.686231+0000","last_became_peered":"2024-04-30T08:31:52.686231+0000","last_unstale":"2024-04-30T08:32:31.558028+0000","last_undegraded":"2024-04-30T08:32:31.558028+0000","last_fullsized":"2024-04-30T08:32:31.558028+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:27:31.073041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.554015+0000","last_change":"2024-04-30T08:31:52.688695+0000","last_active":"2024-04-30T08:32:31.554015+0000","last_peered":"2024-04-30T08:32:31.554015+0000","last_clean":"2024-04-30T08:32:31.554015+0000","last_became_active":"2024-04-30T08:31:52.686556+0000","last_became_peered":"2024-04-30T08:31:52.686556+0000","last_unstale":"2024-04-30T08:32:31.554015+0000","last_undegraded":"2024-04-30T08:32:31.554015+0000","last_fullsized":"2024-04-30T08:32:31.554015+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:22:48.976440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.550665+0000","last_change":"2024-04-30T08:31:52.688324+0000","last_active":"2024-04-30T08:32:31.550665+0000","last_peered":"2024-04-30T08:32:31.550665+0000","last_clean":"2024-04-30T08:32:31.550665+0000","last_became_active":"2024-04-30T08:31:52.687966+0000","last_became_peered":"2024-04-30T08:31:52.687966+0000","last_unstale":"2024-04-30T08:32:31.550665+0000","last_undegraded":"2024-04-30T08:32:31.550665+0000","last_fullsized":"2024-04-30T08:32:31.550665+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:35:38.237750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.551643+0000","last_change":"2024-04-30T08:31:52.695638+0000","last_active":"2024-04-30T08:32:31.551643+0000","last_peered":"2024-04-30T08:32:31.551643+0000","last_clean":"2024-04-30T08:32:31.551643+0000","last_became_active":"2024-04-30T08:31:52.695283+0000","last_became_peered":"2024-04-30T08:31:52.695283+0000","last_unstale":"2024-04-30T08:32:31.551643+0000","last_undegraded":"2024-04-30T08:32:31.551643+0000","last_fullsized":"2024-04-30T08:32:31.551643+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:00:14.510381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.556661+0000","last_change":"2024-04-30T08:31:52.719112+0000","last_active":"2024-04-30T08:32:31.556661+0000","last_peered":"2024-04-30T08:32:31.556661+0000","last_clean":"2024-04-30T08:32:31.556661+0000","last_became_active":"2024-04-30T08:31:52.718207+0000","last_became_peered":"2024-04-30T08:31:52.718207+0000","last_unstale":"2024-04-30T08:32:31.556661+0000","last_undegraded":"2024-04-30T08:32:31.556661+0000","last_fullsized":"2024-04-30T08:32:31.556661+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:27:29.263962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.553131+0000","last_change":"2024-04-30T08:31:52.712008+0000","last_active":"2024-04-30T08:32:31.553131+0000","last_peered":"2024-04-30T08:32:31.553131+0000","last_clean":"2024-04-30T08:32:31.553131+0000","last_became_active":"2024-04-30T08:31:52.711064+0000","last_became_peered":"2024-04-30T08:31:52.711064+0000","last_unstale":"2024-04-30T08:32:31.553131+0000","last_undegraded":"2024-04-30T08:32:31.553131+0000","last_fullsized":"2024-04-30T08:32:31.553131+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:57:27.400602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.558375+0000","last_change":"2024-04-30T08:31:52.691210+0000","last_active":"2024-04-30T08:32:31.558375+0000","last_peered":"2024-04-30T08:32:31.558375+0000","last_clean":"2024-04-30T08:32:31.558375+0000","last_became_active":"2024-04-30T08:31:52.688852+0000","last_became_peered":"2024-04-30T08:31:52.688852+0000","last_unstale":"2024-04-30T08:32:31.558375+0000","last_undegraded":"2024-04-30T08:32:31.558375+0000","last_fullsized":"2024-04-30T08:32:31.558375+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:04:46.502743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.551202+0000","last_change":"2024-04-30T08:31:52.684501+0000","last_active":"2024-04-30T08:32:31.551202+0000","last_peered":"2024-04-30T08:32:31.551202+0000","last_clean":"2024-04-30T08:32:31.551202+0000","last_became_active":"2024-04-30T08:31:52.684122+0000","last_became_peered":"2024-04-30T08:31:52.684122+0000","last_unstale":"2024-04-30T08:32:31.551202+0000","last_undegraded":"2024-04-30T08:32:31.551202+0000","last_fullsized":"2024-04-30T08:32:31.551202+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:55:58.093269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.565964+0000","last_change":"2024-04-30T08:31:52.698016+0000","last_active":"2024-04-30T08:32:31.565964+0000","last_peered":"2024-04-30T08:32:31.565964+0000","last_clean":"2024-04-30T08:32:31.565964+0000","last_became_active":"2024-04-30T08:31:52.695037+0000","last_became_peered":"2024-04-30T08:31:52.695037+0000","last_unstale":"2024-04-30T08:32:31.565964+0000","last_undegraded":"2024-04-30T08:32:31.565964+0000","last_fullsized":"2024-04-30T08:32:31.565964+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:06:01.360947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.555895+0000","last_change":"2024-04-30T08:31:52.698415+0000","last_active":"2024-04-30T08:32:31.555895+0000","last_peered":"2024-04-30T08:32:31.555895+0000","last_clean":"2024-04-30T08:32:31.555895+0000","last_became_active":"2024-04-30T08:31:52.697912+0000","last_became_peered":"2024-04-30T08:31:52.697912+0000","last_unstale":"2024-04-30T08:32:31.555895+0000","last_undegraded":"2024-04-30T08:32:31.555895+0000","last_fullsized":"2024-04-30T08:32:31.555895+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:05:44.420331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.566022+0000","last_change":"2024-04-30T08:31:52.694776+0000","last_active":"2024-04-30T08:32:31.566022+0000","last_peered":"2024-04-30T08:32:31.566022+0000","last_clean":"2024-04-30T08:32:31.566022+0000","last_became_active":"2024-04-30T08:31:52.693532+0000","last_became_peered":"2024-04-30T08:31:52.693532+0000","last_unstale":"2024-04-30T08:32:31.566022+0000","last_undegraded":"2024-04-30T08:32:31.566022+0000","last_fullsized":"2024-04-30T08:32:31.566022+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:31:10.186667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.585119+0000","last_change":"2024-04-30T08:31:52.688477+0000","last_active":"2024-04-30T08:32:31.585119+0000","last_peered":"2024-04-30T08:32:31.585119+0000","last_clean":"2024-04-30T08:32:31.585119+0000","last_became_active":"2024-04-30T08:31:52.688072+0000","last_became_peered":"2024-04-30T08:31:52.688072+0000","last_unstale":"2024-04-30T08:32:31.585119+0000","last_undegraded":"2024-04-30T08:32:31.585119+0000","last_fullsized":"2024-04-30T08:32:31.585119+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:35:42.659707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":26,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.553210+0000","last_change":"2024-04-30T08:31:52.716788+0000","last_active":"2024-04-30T08:32:31.553210+0000","last_peered":"2024-04-30T08:32:31.553210+0000","last_clean":"2024-04-30T08:32:31.553210+0000","last_became_active":"2024-04-30T08:31:52.716336+0000","last_became_peered":"2024-04-30T08:31:52.716336+0000","last_unstale":"2024-04-30T08:32:31.553210+0000","last_undegraded":"2024-04-30T08:32:31.553210+0000","last_fullsized":"2024-04-30T08:32:31.553210+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:23:23.477968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":27,"reported_epoch":83,"state":"active+clean","last_fresh":"2024-04-30T08:32:31.556443+0000","last_change":"2024-04-30T08:31:52.684571+0000","last_active":"2024-04-30T08:32:31.556443+0000","last_peered":"2024-04-30T08:32:31.556443+0000","last_clean":"2024-04-30T08:32:31.556443+0000","last_became_active":"2024-04-30T08:31:52.684192+0000","last_became_peered":"2024-04-30T08:31:52.684192+0000","last_unstale":"2024-04-30T08:32:31.556443+0000","last_undegraded":"2024-04-30T08:32:31.556443+0000","last_fullsized":"2024-04-30T08:32:31.556443+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:05:57.520133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284927,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":49110,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":71,"num_read_kb":93,"num_write":231,"num_write_kb":115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":507904,"data_stored":153083,"data_compressed":5523,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":201,"ondisk_log_size":201,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678039,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32084,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713068,"statfs":{"total":95995035648,"available":95962181632,"internally_reserved":0,"allocated":921600,"data_stored":460271,"data_compressed":20678,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874267,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":1089316,"data_compressed":58745,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037794,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":942080,"data_stored":464229,"data_compressed":20678,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234026,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":933888,"data_stored":460511,"data_compressed":20678,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430254,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":987136,"data_stored":1057399,"data_compressed":56904,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626483,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32076,"kb_used_data":892,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713076,"statfs":{"total":95995035648,"available":95962189824,"internally_reserved":0,"allocated":913408,"data_stored":455637,"data_compressed":20678,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574930,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36276,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708876,"statfs":{"total":95995035648,"available":95957889024,"internally_reserved":0,"allocated":962560,"data_stored":504326,"data_compressed":22519,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738454,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":937984,"data_stored":457093,"data_compressed":20678,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411405,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":970752,"data_stored":499116,"data_compressed":22519,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182470,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":925696,"data_stored":455345,"data_compressed":20678,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986240,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":900,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":921600,"data_stored":460102,"data_compressed":20678,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822714,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":995328,"data_stored":1052590,"data_compressed":56904,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34599,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34599,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34599,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":49411,"data_compressed":1841,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":44201,"data_compressed":1841,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":44033,"data_compressed":1841,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-30T08:32:39.679 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-30T08:32:39.680 INFO:tasks.cephadm:Setting up client nodes... 2024-04-30T08:32:39.681 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-30T08:32:40.186 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:32:41.102 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:40 smithi189 ceph-mon[33038]: from='client.24904 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:32:41.102 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:40 smithi189 ceph-mon[33038]: pgmap v348: 129 pgs: 129 active+clean; 630 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.5 KiB/s wr, 21 op/s; 7 B/s, 1 keys/s, 0 objects/s recovering 2024-04-30T08:32:41.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:40 smithi032 ceph-mon[28331]: from='client.24904 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:32:41.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:40 smithi032 ceph-mon[28331]: pgmap v348: 129 pgs: 129 active+clean; 630 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.5 KiB/s wr, 21 op/s; 7 B/s, 1 keys/s, 0 objects/s recovering 2024-04-30T08:32:41.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:40 smithi071 ceph-mon[33385]: from='client.24904 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:32:41.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:40 smithi071 ceph-mon[33385]: pgmap v348: 129 pgs: 129 active+clean; 630 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 8.5 KiB/s wr, 21 op/s; 7 B/s, 1 keys/s, 0 objects/s recovering 2024-04-30T08:32:42.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.579+0000 7f3abda00700 1 -- 172.21.15.32:0/2633888981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab81040c0 msgr2=0x7f3ab81044d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:42.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.579+0000 7f3abda00700 1 --2- 172.21.15.32:0/2633888981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab81040c0 0x7f3ab81044d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3aa8009990 tx=0x7f3aa8009ca0 comp rx=0 tx=0).stop 2024-04-30T08:32:42.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.579+0000 7f3abda00700 1 -- 172.21.15.32:0/2633888981 shutdown_connections 2024-04-30T08:32:42.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.579+0000 7f3abda00700 1 --2- 172.21.15.32:0/2633888981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ab8105b90 0x7f3ab810c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:42.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.579+0000 7f3abda00700 1 --2- 172.21.15.32:0/2633888981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab81040c0 0x7f3ab81044d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:42.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.579+0000 7f3abda00700 1 --2- 172.21.15.32:0/2633888981 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ab81051f0 0x7f3ab8105600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:42.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.579+0000 7f3abda00700 1 -- 172.21.15.32:0/2633888981 >> 172.21.15.32:0/2633888981 conn(0x7f3ab80ff700 msgr2=0x7f3ab8101bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:42.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.579+0000 7f3abda00700 1 -- 172.21.15.32:0/2633888981 shutdown_connections 2024-04-30T08:32:42.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.579+0000 7f3abda00700 1 -- 172.21.15.32:0/2633888981 wait complete. 2024-04-30T08:32:42.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.581+0000 7f3abda00700 1 Processor -- start 2024-04-30T08:32:42.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.581+0000 7f3abda00700 1 -- start start 2024-04-30T08:32:42.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.582+0000 7f3abda00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab81040c0 0x7f3ab8077340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:42.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.582+0000 7f3abda00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ab81051f0 0x7f3ab8075a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:42.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.582+0000 7f3abda00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ab8105b90 0x7f3ab8075fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:42.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.582+0000 7f3abda00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab810fb50 con 0x7f3ab81040c0 2024-04-30T08:32:42.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.582+0000 7f3abda00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab810f9d0 con 0x7f3ab8105b90 2024-04-30T08:32:42.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.582+0000 7f3abda00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3ab810fcd0 con 0x7f3ab81051f0 2024-04-30T08:32:42.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.582+0000 7f3ab7e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab81040c0 0x7f3ab8077340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:42.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.582+0000 7f3ab7400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ab81051f0 0x7f3ab8075a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:42.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.582+0000 7f3ab7e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab81040c0 0x7f3ab8077340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57444/0 (socket says 172.21.15.32:57444) 2024-04-30T08:32:42.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.582+0000 7f3ab7400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ab81051f0 0x7f3ab8075a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:53060/0 (socket says 172.21.15.32:53060) 2024-04-30T08:32:42.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.582+0000 7f3ab7e00700 1 -- 172.21.15.32:0/1295452567 learned_addr learned my addr 172.21.15.32:0/1295452567 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:42.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.582+0000 7f3abd000700 1 --2- 172.21.15.32:0/1295452567 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ab8105b90 0x7f3ab8075fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:42.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.583+0000 7f3ab7400700 1 -- 172.21.15.32:0/1295452567 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ab8105b90 msgr2=0x7f3ab8075fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:42.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.583+0000 7f3ab7400700 1 --2- 172.21.15.32:0/1295452567 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ab8105b90 0x7f3ab8075fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:42.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.583+0000 7f3ab7400700 1 -- 172.21.15.32:0/1295452567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab81040c0 msgr2=0x7f3ab8077340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:42.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.583+0000 7f3ab7400700 1 --2- 172.21.15.32:0/1295452567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab81040c0 0x7f3ab8077340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:42.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.583+0000 7f3ab7400700 1 -- 172.21.15.32:0/1295452567 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ab80766a0 con 0x7f3ab81051f0 2024-04-30T08:32:42.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.583+0000 7f3ab7e00700 1 --2- 172.21.15.32:0/1295452567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab81040c0 0x7f3ab8077340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:32:42.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.583+0000 7f3ab7400700 1 --2- 172.21.15.32:0/1295452567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ab81051f0 0x7f3ab8075a60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3aa400e9f0 tx=0x7f3aa400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:42.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.583+0000 7f3ab4c00700 1 -- 172.21.15.32:0/1295452567 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa400c9c0 con 0x7f3ab81051f0 2024-04-30T08:32:42.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.583+0000 7f3ab4c00700 1 -- 172.21.15.32:0/1295452567 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3aa400cb60 con 0x7f3ab81051f0 2024-04-30T08:32:42.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.583+0000 7f3abda00700 1 -- 172.21.15.32:0/1295452567 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ab81a4850 con 0x7f3ab81051f0 2024-04-30T08:32:42.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.583+0000 7f3ab4c00700 1 -- 172.21.15.32:0/1295452567 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3aa4018680 con 0x7f3ab81051f0 2024-04-30T08:32:42.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.584+0000 7f3abda00700 1 -- 172.21.15.32:0/1295452567 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ab81a4cb0 con 0x7f3ab81051f0 2024-04-30T08:32:42.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.585+0000 7f3abda00700 1 -- 172.21.15.32:0/1295452567 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ab8051210 con 0x7f3ab81051f0 2024-04-30T08:32:42.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.587+0000 7f3ab4c00700 1 -- 172.21.15.32:0/1295452567 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f3aa4010430 con 0x7f3ab81051f0 2024-04-30T08:32:42.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.589+0000 7f3ab4c00700 1 --2- 172.21.15.32:0/1295452567 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3a9c0afc10 0x7f3a9c0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:42.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.589+0000 7f3ab4c00700 1 -- 172.21.15.32:0/1295452567 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f3aa40df300 con 0x7f3ab81051f0 2024-04-30T08:32:42.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.589+0000 7f3ab7e00700 1 --2- 172.21.15.32:0/1295452567 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3a9c0afc10 0x7f3a9c0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:42.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.590+0000 7f3ab7e00700 1 --2- 172.21.15.32:0/1295452567 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3a9c0afc10 0x7f3a9c0b2120 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3aa800b2e0 tx=0x7f3aa8004710 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:42.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.593+0000 7f3ab4c00700 1 -- 172.21.15.32:0/1295452567 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3aa40a79c0 con 0x7f3ab81051f0 2024-04-30T08:32:42.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.924+0000 7f3abda00700 1 -- 172.21.15.32:0/1295452567 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f3ab802e8e0 con 0x7f3ab81051f0 2024-04-30T08:32:42.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.936+0000 7f3ab4c00700 1 -- 172.21.15.32:0/1295452567 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f3aa40a72c0 con 0x7f3ab81051f0 2024-04-30T08:32:42.938 INFO:teuthology.orchestra.run.smithi032.stdout:[client.0] 2024-04-30T08:32:42.939 INFO:teuthology.orchestra.run.smithi032.stdout: key = AQCqrDBm4QFNNxAAh8YeFF+tg8OkheEQS9uTow== 2024-04-30T08:32:42.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.939+0000 7f3abda00700 1 -- 172.21.15.32:0/1295452567 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3a9c0afc10 msgr2=0x7f3a9c0b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:42.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.939+0000 7f3abda00700 1 --2- 172.21.15.32:0/1295452567 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3a9c0afc10 0x7f3a9c0b2120 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f3aa800b2e0 tx=0x7f3aa8004710 comp rx=0 tx=0).stop 2024-04-30T08:32:42.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.939+0000 7f3abda00700 1 -- 172.21.15.32:0/1295452567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ab81051f0 msgr2=0x7f3ab8075a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:42.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.939+0000 7f3abda00700 1 --2- 172.21.15.32:0/1295452567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ab81051f0 0x7f3ab8075a60 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f3aa400e9f0 tx=0x7f3aa400ed00 comp rx=0 tx=0).stop 2024-04-30T08:32:42.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.940+0000 7f3abda00700 1 -- 172.21.15.32:0/1295452567 shutdown_connections 2024-04-30T08:32:42.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.940+0000 7f3abda00700 1 --2- 172.21.15.32:0/1295452567 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3a9c0afc10 0x7f3a9c0b2120 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:42.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.940+0000 7f3abda00700 1 --2- 172.21.15.32:0/1295452567 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ab8105b90 0x7f3ab8075fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:42.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.940+0000 7f3abda00700 1 --2- 172.21.15.32:0/1295452567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ab81040c0 0x7f3ab8077340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:42.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.940+0000 7f3abda00700 1 --2- 172.21.15.32:0/1295452567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ab81051f0 0x7f3ab8075a60 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:42.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.940+0000 7f3abda00700 1 -- 172.21.15.32:0/1295452567 >> 172.21.15.32:0/1295452567 conn(0x7f3ab80ff700 msgr2=0x7f3ab8108420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:42.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.940+0000 7f3abda00700 1 -- 172.21.15.32:0/1295452567 shutdown_connections 2024-04-30T08:32:42.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:42.940+0000 7f3abda00700 1 -- 172.21.15.32:0/1295452567 wait complete. 2024-04-30T08:32:43.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:42 smithi032 ceph-mon[28331]: pgmap v349: 129 pgs: 129 active+clean; 630 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.7 KiB/s wr, 17 op/s; 6 B/s, 0 keys/s, 0 objects/s recovering 2024-04-30T08:32:43.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:42 smithi071 ceph-mon[33385]: pgmap v349: 129 pgs: 129 active+clean; 630 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.7 KiB/s wr, 17 op/s; 6 B/s, 0 keys/s, 0 objects/s recovering 2024-04-30T08:32:43.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:42 smithi189 ceph-mon[33038]: pgmap v349: 129 pgs: 129 active+clean; 630 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 6.7 KiB/s wr, 17 op/s; 6 B/s, 0 keys/s, 0 objects/s recovering 2024-04-30T08:32:43.789 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-30T08:32:43.789 DEBUG:teuthology.orchestra.run.smithi032:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-30T08:32:43.789 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-30T08:32:43.906 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-30T08:32:43.906 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-30T08:32:43.906 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-30T08:32:44.019 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:43 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1295452567' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-30T08:32:44.019 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:43 smithi032 ceph-mon[28331]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-30T08:32:44.019 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:43 smithi032 ceph-mon[28331]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-30T08:32:44.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:43 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1295452567' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-30T08:32:44.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:43 smithi071 ceph-mon[33385]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-30T08:32:44.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:43 smithi071 ceph-mon[33385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-30T08:32:44.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:43 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1295452567' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-30T08:32:44.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:43 smithi189 ceph-mon[33038]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-30T08:32:44.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:43 smithi189 ceph-mon[33038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-30T08:32:44.411 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:32:45.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:44 smithi032 ceph-mon[28331]: pgmap v350: 129 pgs: 129 active+clean; 630 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.7 KiB/s wr, 15 op/s; 5 B/s, 0 keys/s, 0 objects/s recovering 2024-04-30T08:32:45.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:44 smithi071 ceph-mon[33385]: pgmap v350: 129 pgs: 129 active+clean; 630 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.7 KiB/s wr, 15 op/s; 5 B/s, 0 keys/s, 0 objects/s recovering 2024-04-30T08:32:45.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:44 smithi189 ceph-mon[33038]: pgmap v350: 129 pgs: 129 active+clean; 630 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 5.7 KiB/s wr, 15 op/s; 5 B/s, 0 keys/s, 0 objects/s recovering 2024-04-30T08:32:46.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.773+0000 7fb393400700 1 -- 172.21.15.32:0/2679309316 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb38c1064d0 msgr2=0x7fb38c1068e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:46.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.773+0000 7fb393400700 1 --2- 172.21.15.32:0/2679309316 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb38c1064d0 0x7fb38c1068e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fb37c009990 tx=0x7fb37c009ca0 comp rx=0 tx=0).stop 2024-04-30T08:32:46.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.773+0000 7fb393400700 1 -- 172.21.15.32:0/2679309316 shutdown_connections 2024-04-30T08:32:46.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.773+0000 7fb393400700 1 --2- 172.21.15.32:0/2679309316 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb38c106e20 0x7fb38c1132b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:46.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.773+0000 7fb393400700 1 --2- 172.21.15.32:0/2679309316 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb38c10c510 0x7fb38c10c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:46.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.773+0000 7fb393400700 1 --2- 172.21.15.32:0/2679309316 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb38c1064d0 0x7fb38c1068e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:46.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.773+0000 7fb393400700 1 -- 172.21.15.32:0/2679309316 >> 172.21.15.32:0/2679309316 conn(0x7fb38c101fd0 msgr2=0x7fb38c104420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:46.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.773+0000 7fb393400700 1 -- 172.21.15.32:0/2679309316 shutdown_connections 2024-04-30T08:32:46.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.773+0000 7fb393400700 1 -- 172.21.15.32:0/2679309316 wait complete. 2024-04-30T08:32:46.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.775+0000 7fb393400700 1 Processor -- start 2024-04-30T08:32:46.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.775+0000 7fb393400700 1 -- start start 2024-04-30T08:32:46.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.775+0000 7fb393400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb38c1064d0 0x7fb38c19cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:46.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.776+0000 7fb393400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb38c106e20 0x7fb38c19d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:46.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.776+0000 7fb393400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb38c10c510 0x7fb38c1a17e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:46.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.776+0000 7fb393400700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb38c116640 con 0x7fb38c1064d0 2024-04-30T08:32:46.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.776+0000 7fb393400700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb38c1164c0 con 0x7fb38c106e20 2024-04-30T08:32:46.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.776+0000 7fb393400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb38c1167c0 con 0x7fb38c10c510 2024-04-30T08:32:46.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.776+0000 7fb390a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb38c1064d0 0x7fb38c19cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:46.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.776+0000 7fb38be00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb38c106e20 0x7fb38c19d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:46.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.776+0000 7fb390a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb38c1064d0 0x7fb38c19cfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57468/0 (socket says 172.21.15.32:57468) 2024-04-30T08:32:46.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.776+0000 7fb390a00700 1 -- 172.21.15.32:0/3149285134 learned_addr learned my addr 172.21.15.32:0/3149285134 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:46.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.776+0000 7fb391400700 1 --2- 172.21.15.32:0/3149285134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb38c10c510 0x7fb38c1a17e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:46.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.776+0000 7fb38be00700 1 -- 172.21.15.32:0/3149285134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb38c10c510 msgr2=0x7fb38c1a17e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:46.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.776+0000 7fb38be00700 1 --2- 172.21.15.32:0/3149285134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb38c10c510 0x7fb38c1a17e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:46.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.776+0000 7fb38be00700 1 -- 172.21.15.32:0/3149285134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb38c1064d0 msgr2=0x7fb38c19cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:46.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.776+0000 7fb38be00700 1 --2- 172.21.15.32:0/3149285134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb38c1064d0 0x7fb38c19cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:46.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.776+0000 7fb38be00700 1 -- 172.21.15.32:0/3149285134 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb38c1a1e80 con 0x7fb38c106e20 2024-04-30T08:32:46.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.776+0000 7fb391400700 1 --2- 172.21.15.32:0/3149285134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb38c10c510 0x7fb38c1a17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:32:46.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.777+0000 7fb38be00700 1 --2- 172.21.15.32:0/3149285134 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb38c106e20 0x7fb38c19d550 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb37c0098e0 tx=0x7fb37c0047b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:46.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.777+0000 7fb389600700 1 -- 172.21.15.32:0/3149285134 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb37c026070 con 0x7fb38c106e20 2024-04-30T08:32:46.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.777+0000 7fb393400700 1 -- 172.21.15.32:0/3149285134 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb38c1a2110 con 0x7fb38c106e20 2024-04-30T08:32:46.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.777+0000 7fb389600700 1 -- 172.21.15.32:0/3149285134 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb37c021410 con 0x7fb38c106e20 2024-04-30T08:32:46.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.777+0000 7fb389600700 1 -- 172.21.15.32:0/3149285134 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb37c017650 con 0x7fb38c106e20 2024-04-30T08:32:46.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.777+0000 7fb393400700 1 -- 172.21.15.32:0/3149285134 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb38c1a2610 con 0x7fb38c106e20 2024-04-30T08:32:46.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.778+0000 7fb393400700 1 -- 172.21.15.32:0/3149285134 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb38c051210 con 0x7fb38c106e20 2024-04-30T08:32:46.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.781+0000 7fb389600700 1 -- 172.21.15.32:0/3149285134 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fb37c029070 con 0x7fb38c106e20 2024-04-30T08:32:46.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.783+0000 7fb389600700 1 --2- 172.21.15.32:0/3149285134 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb3740afc10 0x7fb3740b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:46.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.783+0000 7fb389600700 1 -- 172.21.15.32:0/3149285134 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fb37c0e05c0 con 0x7fb38c106e20 2024-04-30T08:32:46.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.783+0000 7fb390a00700 1 --2- 172.21.15.32:0/3149285134 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb3740afc10 0x7fb3740b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:46.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.784+0000 7fb390a00700 1 --2- 172.21.15.32:0/3149285134 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb3740afc10 0x7fb3740b2120 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb380005950 tx=0x7fb380009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:46.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:46.785+0000 7fb389600700 1 -- 172.21.15.32:0/3149285134 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb37c030020 con 0x7fb38c106e20 2024-04-30T08:32:47.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:47.082+0000 7fb393400700 1 -- 172.21.15.32:0/3149285134 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fb38c02e590 con 0x7fb38c106e20 2024-04-30T08:32:47.084 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:46 smithi032 ceph-mon[28331]: pgmap v351: 129 pgs: 129 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.6 KiB/s wr, 14 op/s; 5 B/s, 0 keys/s, 0 objects/s recovering 2024-04-30T08:32:47.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:47.089+0000 7fb389600700 1 -- 172.21.15.32:0/3149285134 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+284480 (secure 0 0 0) 0x7fb38c1a2610 con 0x7fb38c106e20 2024-04-30T08:32:47.094 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:32:47.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:47.097+0000 7fb393400700 1 -- 172.21.15.32:0/3149285134 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb3740afc10 msgr2=0x7fb3740b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:47.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:47.097+0000 7fb393400700 1 --2- 172.21.15.32:0/3149285134 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb3740afc10 0x7fb3740b2120 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fb380005950 tx=0x7fb380009260 comp rx=0 tx=0).stop 2024-04-30T08:32:47.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:47.097+0000 7fb393400700 1 -- 172.21.15.32:0/3149285134 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb38c106e20 msgr2=0x7fb38c19d550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:47.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:47.097+0000 7fb393400700 1 --2- 172.21.15.32:0/3149285134 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb38c106e20 0x7fb38c19d550 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fb37c0098e0 tx=0x7fb37c0047b0 comp rx=0 tx=0).stop 2024-04-30T08:32:47.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:47.097+0000 7fb393400700 1 -- 172.21.15.32:0/3149285134 shutdown_connections 2024-04-30T08:32:47.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:47.097+0000 7fb393400700 1 --2- 172.21.15.32:0/3149285134 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb3740afc10 0x7fb3740b2120 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:47.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:47.097+0000 7fb393400700 1 --2- 172.21.15.32:0/3149285134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb38c10c510 0x7fb38c1a17e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:47.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:47.097+0000 7fb393400700 1 --2- 172.21.15.32:0/3149285134 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb38c1064d0 0x7fb38c19cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:47.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:47.097+0000 7fb393400700 1 --2- 172.21.15.32:0/3149285134 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb38c106e20 0x7fb38c19d550 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:47.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:47.097+0000 7fb393400700 1 -- 172.21.15.32:0/3149285134 >> 172.21.15.32:0/3149285134 conn(0x7fb38c101fd0 msgr2=0x7fb38c1047e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:47.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:47.097+0000 7fb393400700 1 -- 172.21.15.32:0/3149285134 shutdown_connections 2024-04-30T08:32:47.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:47.097+0000 7fb393400700 1 -- 172.21.15.32:0/3149285134 wait complete. 2024-04-30T08:32:47.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:46 smithi071 ceph-mon[33385]: pgmap v351: 129 pgs: 129 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.6 KiB/s wr, 14 op/s; 5 B/s, 0 keys/s, 0 objects/s recovering 2024-04-30T08:32:47.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:46 smithi189 ceph-mon[33038]: pgmap v351: 129 pgs: 129 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 5.6 KiB/s wr, 14 op/s; 5 B/s, 0 keys/s, 0 objects/s recovering 2024-04-30T08:32:48.033 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6800","nonce":2930853529},{"type":"v1","addr":"172.21.15.32:6801","nonce":2930853529}]},"active_addr":"172.21.15.32:6801/2930853529","active_change":"2024-04-30T08:23:29.526909+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.32:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.32:0","nonce":1718012592}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.32:0","nonce":2970162436}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.32:0","nonce":3418667628}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.32:0","nonce":4288389191}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.32:0","nonce":2628428881}]}]} 2024-04-30T08:32:48.039 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-30T08:32:48.039 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-30T08:32:48.039 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-30T08:32:48.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:47 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3149285134' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-30T08:32:48.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:47 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3149285134' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-30T08:32:48.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:47 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3149285134' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-30T08:32:48.502 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:32:49.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:48 smithi071 ceph-mon[33385]: pgmap v352: 129 pgs: 129 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.8 KiB/s wr, 10 op/s; 5 B/s, 0 keys/s, 0 objects/s recovering 2024-04-30T08:32:49.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:48 smithi189 ceph-mon[33038]: pgmap v352: 129 pgs: 129 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.8 KiB/s wr, 10 op/s; 5 B/s, 0 keys/s, 0 objects/s recovering 2024-04-30T08:32:49.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:48 smithi032 ceph-mon[28331]: pgmap v352: 129 pgs: 129 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.8 KiB/s wr, 10 op/s; 5 B/s, 0 keys/s, 0 objects/s recovering 2024-04-30T08:32:50.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.954+0000 7f2859600700 1 -- 172.21.15.32:0/2642045898 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f285410a710 msgr2=0x7f285410ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:50.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.954+0000 7f2859600700 1 --2- 172.21.15.32:0/2642045898 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f285410a710 0x7f285410ab20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2840009990 tx=0x7f2840009ca0 comp rx=0 tx=0).stop 2024-04-30T08:32:50.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.954+0000 7f2859600700 1 -- 172.21.15.32:0/2642045898 shutdown_connections 2024-04-30T08:32:50.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.954+0000 7f2859600700 1 --2- 172.21.15.32:0/2642045898 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f285410b060 0x7f2854117570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:50.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.954+0000 7f2859600700 1 --2- 172.21.15.32:0/2642045898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f285410f610 0x7f285410fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:50.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.954+0000 7f2859600700 1 --2- 172.21.15.32:0/2642045898 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f285410a710 0x7f285410ab20 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:50.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.954+0000 7f2859600700 1 -- 172.21.15.32:0/2642045898 >> 172.21.15.32:0/2642045898 conn(0x7f2854076a50 msgr2=0x7f2854076e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:50.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.954+0000 7f2859600700 1 -- 172.21.15.32:0/2642045898 shutdown_connections 2024-04-30T08:32:50.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.954+0000 7f2859600700 1 -- 172.21.15.32:0/2642045898 wait complete. 2024-04-30T08:32:50.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.956+0000 7f2859600700 1 Processor -- start 2024-04-30T08:32:50.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.957+0000 7f2859600700 1 -- start start 2024-04-30T08:32:50.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.957+0000 7f2859600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f285410a710 0x7f28541a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:50.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.957+0000 7f2859600700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f285410b060 0x7f28541a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:50.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.957+0000 7f2859600700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f285410f610 0x7f28541a59f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:50.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.958+0000 7f2859600700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f285411a7f0 con 0x7f285410b060 2024-04-30T08:32:50.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.958+0000 7f2859600700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f285411a670 con 0x7f285410f610 2024-04-30T08:32:50.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.958+0000 7f2859600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f285411a970 con 0x7f285410a710 2024-04-30T08:32:50.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.958+0000 7f2853400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f285410b060 0x7f28541a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:50.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.958+0000 7f2853e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f285410a710 0x7f28541a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:50.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.958+0000 7f2853400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f285410b060 0x7f28541a1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60266/0 (socket says 172.21.15.32:60266) 2024-04-30T08:32:50.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.958+0000 7f2853400700 1 -- 172.21.15.32:0/3127386184 learned_addr learned my addr 172.21.15.32:0/3127386184 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:50.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.958+0000 7f2858c00700 1 --2- 172.21.15.32:0/3127386184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f285410f610 0x7f28541a59f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:50.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.958+0000 7f2853e00700 1 -- 172.21.15.32:0/3127386184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f285410f610 msgr2=0x7f28541a59f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:50.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.958+0000 7f2853e00700 1 --2- 172.21.15.32:0/3127386184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f285410f610 0x7f28541a59f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:50.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.958+0000 7f2853e00700 1 -- 172.21.15.32:0/3127386184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f285410b060 msgr2=0x7f28541a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:50.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.958+0000 7f2853e00700 1 --2- 172.21.15.32:0/3127386184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f285410b060 0x7f28541a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:50.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.958+0000 7f2853e00700 1 -- 172.21.15.32:0/3127386184 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28541a6090 con 0x7f285410a710 2024-04-30T08:32:50.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.959+0000 7f2853e00700 1 --2- 172.21.15.32:0/3127386184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f285410a710 0x7f28541a11c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f284400e9f0 tx=0x7f284400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:50.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.959+0000 7f2850c00700 1 -- 172.21.15.32:0/3127386184 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f284400c9c0 con 0x7f285410a710 2024-04-30T08:32:50.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.959+0000 7f2850c00700 1 -- 172.21.15.32:0/3127386184 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f284400cb60 con 0x7f285410a710 2024-04-30T08:32:50.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.959+0000 7f2850c00700 1 -- 172.21.15.32:0/3127386184 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2844018680 con 0x7f285410a710 2024-04-30T08:32:50.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.960+0000 7f2859600700 1 -- 172.21.15.32:0/3127386184 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28541a6380 con 0x7f285410a710 2024-04-30T08:32:50.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.960+0000 7f2859600700 1 -- 172.21.15.32:0/3127386184 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28541adca0 con 0x7f285410a710 2024-04-30T08:32:50.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.961+0000 7f2859600700 1 -- 172.21.15.32:0/3127386184 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2854051210 con 0x7f285410a710 2024-04-30T08:32:50.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.968+0000 7f2850c00700 1 -- 172.21.15.32:0/3127386184 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f2844010430 con 0x7f285410a710 2024-04-30T08:32:50.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.969+0000 7f2850c00700 1 --2- 172.21.15.32:0/3127386184 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f28380afc10 0x7f28380b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:50.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.970+0000 7f2850c00700 1 -- 172.21.15.32:0/3127386184 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f28440df5d0 con 0x7f285410a710 2024-04-30T08:32:50.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.970+0000 7f2853400700 1 --2- 172.21.15.32:0/3127386184 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f28380afc10 0x7f28380b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:50.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.970+0000 7f2853400700 1 --2- 172.21.15.32:0/3127386184 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f28380afc10 0x7f28380b2120 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2854077e80 tx=0x7f2840009550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:50.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:50.971+0000 7f2850c00700 1 -- 172.21.15.32:0/3127386184 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f28440a7c90 con 0x7f285410a710 2024-04-30T08:32:51.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:51.209+0000 7f2859600700 1 -- 172.21.15.32:0/3127386184 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f285402e590 con 0x7f285410a710 2024-04-30T08:32:51.210 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:50 smithi032 ceph-mon[28331]: pgmap v353: 129 pgs: 129 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-30T08:32:51.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:51.211+0000 7f2850c00700 1 -- 172.21.15.32:0/3127386184 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+24525 (secure 0 0 0) 0x7f28440a7590 con 0x7f285410a710 2024-04-30T08:32:51.212 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:32:51.212 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":83,"fsid":"98c0a670-06ca-11ef-bc93-c7b262605968","created":"2024-04-30T08:22:11.122875+0000","modified":"2024-04-30T08:32:31.544477+0000","last_up_change":"2024-04-30T08:30:50.350439+0000","last_in_change":"2024-04-30T08:30:30.210620+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-30T08:26:19.728331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-30T08:31:49.800542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-30T08:31:51.211320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-30T08:31:55.716625+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a7e060c8-1cbb-418f-9a03-06693f6b3f52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6803","nonce":2624700402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6805","nonce":2624700402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6809","nonce":2624700402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6807","nonce":2624700402}]},"public_addr":"172.21.15.32:6803/2624700402","cluster_addr":"172.21.15.32:6805/2624700402","heartbeat_back_addr":"172.21.15.32:6809/2624700402","heartbeat_front_addr":"172.21.15.32:6807/2624700402","state":["exists","up"]},{"osd":1,"uuid":"72ef2fb1-e5ca-48ef-aa3a-4796a64b922f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6801","nonce":444649711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6803","nonce":444649711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6807","nonce":444649711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6805","nonce":444649711}]},"public_addr":"172.21.15.71:6801/444649711","cluster_addr":"172.21.15.71:6803/444649711","heartbeat_back_addr":"172.21.15.71:6807/444649711","heartbeat_front_addr":"172.21.15.71:6805/444649711","state":["exists","up"]},{"osd":2,"uuid":"704e5e8a-935e-47c7-ae28-27119876ac09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6801","nonce":3255528671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6803","nonce":3255528671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6807","nonce":3255528671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6805","nonce":3255528671}]},"public_addr":"172.21.15.189:6801/3255528671","cluster_addr":"172.21.15.189:6803/3255528671","heartbeat_back_addr":"172.21.15.189:6807/3255528671","heartbeat_front_addr":"172.21.15.189:6805/3255528671","state":["exists","up"]},{"osd":3,"uuid":"575004f7-6559-4a62-863e-e33078b0eebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6811","nonce":1384702112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6813","nonce":1384702112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6817","nonce":1384702112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6815","nonce":1384702112}]},"public_addr":"172.21.15.32:6811/1384702112","cluster_addr":"172.21.15.32:6813/1384702112","heartbeat_back_addr":"172.21.15.32:6817/1384702112","heartbeat_front_addr":"172.21.15.32:6815/1384702112","state":["exists","up"]},{"osd":4,"uuid":"e1c1e894-074d-4c78-89ef-9e5a114268bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6809","nonce":4250534574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6811","nonce":4250534574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6815","nonce":4250534574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6813","nonce":4250534574}]},"public_addr":"172.21.15.71:6809/4250534574","cluster_addr":"172.21.15.71:6811/4250534574","heartbeat_back_addr":"172.21.15.71:6815/4250534574","heartbeat_front_addr":"172.21.15.71:6813/4250534574","state":["exists","up"]},{"osd":5,"uuid":"12c022d4-fac6-40c4-ae7a-691571bee5d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6809","nonce":4245346198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6811","nonce":4245346198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6815","nonce":4245346198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6813","nonce":4245346198}]},"public_addr":"172.21.15.189:6809/4245346198","cluster_addr":"172.21.15.189:6811/4245346198","heartbeat_back_addr":"172.21.15.189:6815/4245346198","heartbeat_front_addr":"172.21.15.189:6813/4245346198","state":["exists","up"]},{"osd":6,"uuid":"b413f8ee-3d4a-415a-a2ba-d68437b7330d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6819","nonce":1910389190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6821","nonce":1910389190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6825","nonce":1910389190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6823","nonce":1910389190}]},"public_addr":"172.21.15.32:6819/1910389190","cluster_addr":"172.21.15.32:6821/1910389190","heartbeat_back_addr":"172.21.15.32:6825/1910389190","heartbeat_front_addr":"172.21.15.32:6823/1910389190","state":["exists","up"]},{"osd":7,"uuid":"aac64fb3-fc13-4126-b842-7b06ad5ce238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6817","nonce":692607380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6819","nonce":692607380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6823","nonce":692607380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6821","nonce":692607380}]},"public_addr":"172.21.15.71:6817/692607380","cluster_addr":"172.21.15.71:6819/692607380","heartbeat_back_addr":"172.21.15.71:6823/692607380","heartbeat_front_addr":"172.21.15.71:6821/692607380","state":["exists","up"]},{"osd":8,"uuid":"cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6817","nonce":3973697069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6819","nonce":3973697069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6823","nonce":3973697069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6821","nonce":3973697069}]},"public_addr":"172.21.15.189:6817/3973697069","cluster_addr":"172.21.15.189:6819/3973697069","heartbeat_back_addr":"172.21.15.189:6823/3973697069","heartbeat_front_addr":"172.21.15.189:6821/3973697069","state":["exists","up"]},{"osd":9,"uuid":"ef4a1645-ae6e-45c3-82ad-161510e62aa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6827","nonce":46066685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6829","nonce":46066685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6833","nonce":46066685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6831","nonce":46066685}]},"public_addr":"172.21.15.32:6827/46066685","cluster_addr":"172.21.15.32:6829/46066685","heartbeat_back_addr":"172.21.15.32:6833/46066685","heartbeat_front_addr":"172.21.15.32:6831/46066685","state":["exists","up"]},{"osd":10,"uuid":"25b26a38-a644-409f-b941-59efdf04a3ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6825","nonce":1886290293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6827","nonce":1886290293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6831","nonce":1886290293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6829","nonce":1886290293}]},"public_addr":"172.21.15.71:6825/1886290293","cluster_addr":"172.21.15.71:6827/1886290293","heartbeat_back_addr":"172.21.15.71:6831/1886290293","heartbeat_front_addr":"172.21.15.71:6829/1886290293","state":["exists","up"]},{"osd":11,"uuid":"ab96b3ff-3688-455b-af64-60226c5bfbff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6825","nonce":850185001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6827","nonce":850185001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6831","nonce":850185001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6829","nonce":850185001}]},"public_addr":"172.21.15.189:6825/850185001","cluster_addr":"172.21.15.189:6827/850185001","heartbeat_back_addr":"172.21.15.189:6831/850185001","heartbeat_front_addr":"172.21.15.189:6829/850185001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:27.220308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:53.230783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:17.190642+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:52.603799+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:20.568671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:46.158363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:22.706933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:50.665173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:16.602410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:53.814698+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:22.435189+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:48.651949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1266897222":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/2915689533":"2024-05-01T08:23:29.526664+0000","172.21.15.32:6801/4019351306":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/425836193":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/1134075722":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6801/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/21059253":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/3245550894":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/4019351306":"2024-05-01T08:23:29.526664+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-30T08:32:51.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:51.215+0000 7f2859600700 1 -- 172.21.15.32:0/3127386184 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f28380afc10 msgr2=0x7f28380b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:51.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:51.215+0000 7f2859600700 1 --2- 172.21.15.32:0/3127386184 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f28380afc10 0x7f28380b2120 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f2854077e80 tx=0x7f2840009550 comp rx=0 tx=0).stop 2024-04-30T08:32:51.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:51.215+0000 7f2859600700 1 -- 172.21.15.32:0/3127386184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f285410a710 msgr2=0x7f28541a11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:51.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:51.215+0000 7f2859600700 1 --2- 172.21.15.32:0/3127386184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f285410a710 0x7f28541a11c0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f284400e9f0 tx=0x7f284400ed00 comp rx=0 tx=0).stop 2024-04-30T08:32:51.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:51.215+0000 7f2859600700 1 -- 172.21.15.32:0/3127386184 shutdown_connections 2024-04-30T08:32:51.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:51.215+0000 7f2859600700 1 --2- 172.21.15.32:0/3127386184 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f28380afc10 0x7f28380b2120 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:51.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:51.215+0000 7f2859600700 1 --2- 172.21.15.32:0/3127386184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f285410f610 0x7f28541a59f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:51.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:51.215+0000 7f2859600700 1 --2- 172.21.15.32:0/3127386184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f285410a710 0x7f28541a11c0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:51.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:51.215+0000 7f2859600700 1 --2- 172.21.15.32:0/3127386184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f285410b060 0x7f28541a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:51.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:51.215+0000 7f2859600700 1 -- 172.21.15.32:0/3127386184 >> 172.21.15.32:0/3127386184 conn(0x7f2854076a50 msgr2=0x7f28541087b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:51.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:51.216+0000 7f2859600700 1 -- 172.21.15.32:0/3127386184 shutdown_connections 2024-04-30T08:32:51.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:51.216+0000 7f2859600700 1 -- 172.21.15.32:0/3127386184 wait complete. 2024-04-30T08:32:51.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:50 smithi071 ceph-mon[33385]: pgmap v353: 129 pgs: 129 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-30T08:32:51.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:50 smithi189 ceph-mon[33038]: pgmap v353: 129 pgs: 129 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-04-30T08:32:52.204 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-30T08:32:52.205 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-30T08:32:52.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:51 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3127386184' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:32:52.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:51 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:32:52.277 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:51 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3127386184' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:32:52.277 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:51 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:32:52.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:51 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3127386184' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:32:52.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:51 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:32:52.674 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:32:53.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:52 smithi189 ceph-mon[33038]: pgmap v354: 129 pgs: 129 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-04-30T08:32:53.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:52 smithi032 ceph-mon[28331]: pgmap v354: 129 pgs: 129 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-04-30T08:32:53.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:52 smithi071 ceph-mon[33385]: pgmap v354: 129 pgs: 129 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 1.1 KiB/s wr, 10 op/s 2024-04-30T08:32:54.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:32:54.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:32:54.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:32:54.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:32:54.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:54 smithi032 ceph-mon[28331]: pgmap v355: 129 pgs: 129 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-04-30T08:32:54.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:32:54.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:32:54.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:32:54.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:32:54.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:54 smithi071 ceph-mon[33385]: pgmap v355: 129 pgs: 129 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-04-30T08:32:54.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:32:54.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:32:54.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:32:54.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:32:54.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:54 smithi189 ceph-mon[33038]: pgmap v355: 129 pgs: 129 active+clean; 642 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-04-30T08:32:55.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.703+0000 7f5b9ce00700 1 -- 172.21.15.32:0/3992119128 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b980763a0 msgr2=0x7f5b98110610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:55.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.703+0000 7f5b9ce00700 1 --2- 172.21.15.32:0/3992119128 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b980763a0 0x7f5b98110610 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5b84009990 tx=0x7f5b84009ca0 comp rx=0 tx=0).stop 2024-04-30T08:32:55.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.703+0000 7f5b9ce00700 1 -- 172.21.15.32:0/3992119128 shutdown_connections 2024-04-30T08:32:55.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.703+0000 7f5b9ce00700 1 --2- 172.21.15.32:0/3992119128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b98110b50 0x7f5b98112f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:55.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.703+0000 7f5b9ce00700 1 --2- 172.21.15.32:0/3992119128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b98075a50 0x7f5b98075e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:55.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.703+0000 7f5b9ce00700 1 --2- 172.21.15.32:0/3992119128 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b980763a0 0x7f5b98110610 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:55.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.703+0000 7f5b9ce00700 1 -- 172.21.15.32:0/3992119128 >> 172.21.15.32:0/3992119128 conn(0x7f5b980fff90 msgr2=0x7f5b981023b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:55.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.704+0000 7f5b9ce00700 1 -- 172.21.15.32:0/3992119128 shutdown_connections 2024-04-30T08:32:55.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.704+0000 7f5b9ce00700 1 -- 172.21.15.32:0/3992119128 wait complete. 2024-04-30T08:32:55.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.705+0000 7f5b9ce00700 1 Processor -- start 2024-04-30T08:32:55.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.705+0000 7f5b9ce00700 1 -- start start 2024-04-30T08:32:55.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.705+0000 7f5b9ce00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b98075a50 0x7f5b9819cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:55.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.705+0000 7f5b9ce00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b980763a0 0x7f5b9819d560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:55.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.705+0000 7f5b9ce00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b98110b50 0x7f5b981a17f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:55.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.706+0000 7f5b9ce00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5b98116630 con 0x7f5b98110b50 2024-04-30T08:32:55.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.706+0000 7f5b9ce00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5b981164b0 con 0x7f5b980763a0 2024-04-30T08:32:55.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.706+0000 7f5b9ce00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5b981167b0 con 0x7f5b98075a50 2024-04-30T08:32:55.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.706+0000 7f5b95e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b98075a50 0x7f5b9819cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:55.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.706+0000 7f5b96800700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b98110b50 0x7f5b981a17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:55.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.706+0000 7f5b95e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b98075a50 0x7f5b9819cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:53096/0 (socket says 172.21.15.32:53096) 2024-04-30T08:32:55.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.706+0000 7f5b95e00700 1 -- 172.21.15.32:0/4002286629 learned_addr learned my addr 172.21.15.32:0/4002286629 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:32:55.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.706+0000 7f5b95400700 1 --2- 172.21.15.32:0/4002286629 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b980763a0 0x7f5b9819d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:55.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.706+0000 7f5b95400700 1 -- 172.21.15.32:0/4002286629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b98075a50 msgr2=0x7f5b9819cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:55.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.706+0000 7f5b95400700 1 --2- 172.21.15.32:0/4002286629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b98075a50 0x7f5b9819cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:55.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.706+0000 7f5b95400700 1 -- 172.21.15.32:0/4002286629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b98110b50 msgr2=0x7f5b981a17f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:55.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.706+0000 7f5b95400700 1 --2- 172.21.15.32:0/4002286629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b98110b50 0x7f5b981a17f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:55.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.706+0000 7f5b95400700 1 -- 172.21.15.32:0/4002286629 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b981a1e90 con 0x7f5b980763a0 2024-04-30T08:32:55.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.707+0000 7f5b95400700 1 --2- 172.21.15.32:0/4002286629 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b980763a0 0x7f5b9819d560 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5b840059d0 tx=0x7f5b840057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:55.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.707+0000 7f5b8ea00700 1 -- 172.21.15.32:0/4002286629 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b84027070 con 0x7f5b980763a0 2024-04-30T08:32:55.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.707+0000 7f5b9ce00700 1 -- 172.21.15.32:0/4002286629 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b981a2120 con 0x7f5b980763a0 2024-04-30T08:32:55.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.707+0000 7f5b8ea00700 1 -- 172.21.15.32:0/4002286629 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5b84022470 con 0x7f5b980763a0 2024-04-30T08:32:55.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.707+0000 7f5b8ea00700 1 -- 172.21.15.32:0/4002286629 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b84017650 con 0x7f5b980763a0 2024-04-30T08:32:55.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.707+0000 7f5b9ce00700 1 -- 172.21.15.32:0/4002286629 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b981a2620 con 0x7f5b980763a0 2024-04-30T08:32:55.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.709+0000 7f5b9ce00700 1 -- 172.21.15.32:0/4002286629 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5b98051210 con 0x7f5b980763a0 2024-04-30T08:32:55.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.711+0000 7f5b8ea00700 1 -- 172.21.15.32:0/4002286629 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f5b8402a070 con 0x7f5b980763a0 2024-04-30T08:32:55.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.713+0000 7f5b8ea00700 1 --2- 172.21.15.32:0/4002286629 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f5b7c0afc10 0x7f5b7c0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:32:55.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.713+0000 7f5b8ea00700 1 -- 172.21.15.32:0/4002286629 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f5b840e0180 con 0x7f5b980763a0 2024-04-30T08:32:55.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.713+0000 7f5b95e00700 1 --2- 172.21.15.32:0/4002286629 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f5b7c0afc10 0x7f5b7c0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:32:55.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.714+0000 7f5b95e00700 1 --2- 172.21.15.32:0/4002286629 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f5b7c0afc10 0x7f5b7c0b2120 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5b88009d60 tx=0x7f5b88009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:32:55.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.715+0000 7f5b8ea00700 1 -- 172.21.15.32:0/4002286629 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5b8405eff0 con 0x7f5b980763a0 2024-04-30T08:32:55.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.957+0000 7f5b9ce00700 1 -- 172.21.15.32:0/4002286629 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5b9802e590 con 0x7f5b980763a0 2024-04-30T08:32:55.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.960+0000 7f5b8ea00700 1 -- 172.21.15.32:0/4002286629 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v83) ==== 74+0+24525 (secure 0 0 0) 0x7f5b981a2620 con 0x7f5b980763a0 2024-04-30T08:32:55.969 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:32:55.972 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":83,"fsid":"98c0a670-06ca-11ef-bc93-c7b262605968","created":"2024-04-30T08:22:11.122875+0000","modified":"2024-04-30T08:32:31.544477+0000","last_up_change":"2024-04-30T08:30:50.350439+0000","last_in_change":"2024-04-30T08:30:30.210620+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-30T08:26:19.728331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-30T08:31:49.800542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-30T08:31:51.211320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-30T08:31:55.716625+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a7e060c8-1cbb-418f-9a03-06693f6b3f52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6803","nonce":2624700402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6805","nonce":2624700402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6809","nonce":2624700402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6807","nonce":2624700402}]},"public_addr":"172.21.15.32:6803/2624700402","cluster_addr":"172.21.15.32:6805/2624700402","heartbeat_back_addr":"172.21.15.32:6809/2624700402","heartbeat_front_addr":"172.21.15.32:6807/2624700402","state":["exists","up"]},{"osd":1,"uuid":"72ef2fb1-e5ca-48ef-aa3a-4796a64b922f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6801","nonce":444649711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6803","nonce":444649711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6807","nonce":444649711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6805","nonce":444649711}]},"public_addr":"172.21.15.71:6801/444649711","cluster_addr":"172.21.15.71:6803/444649711","heartbeat_back_addr":"172.21.15.71:6807/444649711","heartbeat_front_addr":"172.21.15.71:6805/444649711","state":["exists","up"]},{"osd":2,"uuid":"704e5e8a-935e-47c7-ae28-27119876ac09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6801","nonce":3255528671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6803","nonce":3255528671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6807","nonce":3255528671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6805","nonce":3255528671}]},"public_addr":"172.21.15.189:6801/3255528671","cluster_addr":"172.21.15.189:6803/3255528671","heartbeat_back_addr":"172.21.15.189:6807/3255528671","heartbeat_front_addr":"172.21.15.189:6805/3255528671","state":["exists","up"]},{"osd":3,"uuid":"575004f7-6559-4a62-863e-e33078b0eebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6811","nonce":1384702112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6813","nonce":1384702112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6817","nonce":1384702112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6815","nonce":1384702112}]},"public_addr":"172.21.15.32:6811/1384702112","cluster_addr":"172.21.15.32:6813/1384702112","heartbeat_back_addr":"172.21.15.32:6817/1384702112","heartbeat_front_addr":"172.21.15.32:6815/1384702112","state":["exists","up"]},{"osd":4,"uuid":"e1c1e894-074d-4c78-89ef-9e5a114268bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6809","nonce":4250534574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6811","nonce":4250534574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6815","nonce":4250534574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6813","nonce":4250534574}]},"public_addr":"172.21.15.71:6809/4250534574","cluster_addr":"172.21.15.71:6811/4250534574","heartbeat_back_addr":"172.21.15.71:6815/4250534574","heartbeat_front_addr":"172.21.15.71:6813/4250534574","state":["exists","up"]},{"osd":5,"uuid":"12c022d4-fac6-40c4-ae7a-691571bee5d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6809","nonce":4245346198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6811","nonce":4245346198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6815","nonce":4245346198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6813","nonce":4245346198}]},"public_addr":"172.21.15.189:6809/4245346198","cluster_addr":"172.21.15.189:6811/4245346198","heartbeat_back_addr":"172.21.15.189:6815/4245346198","heartbeat_front_addr":"172.21.15.189:6813/4245346198","state":["exists","up"]},{"osd":6,"uuid":"b413f8ee-3d4a-415a-a2ba-d68437b7330d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6819","nonce":1910389190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6821","nonce":1910389190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6825","nonce":1910389190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6823","nonce":1910389190}]},"public_addr":"172.21.15.32:6819/1910389190","cluster_addr":"172.21.15.32:6821/1910389190","heartbeat_back_addr":"172.21.15.32:6825/1910389190","heartbeat_front_addr":"172.21.15.32:6823/1910389190","state":["exists","up"]},{"osd":7,"uuid":"aac64fb3-fc13-4126-b842-7b06ad5ce238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6817","nonce":692607380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6819","nonce":692607380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6823","nonce":692607380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6821","nonce":692607380}]},"public_addr":"172.21.15.71:6817/692607380","cluster_addr":"172.21.15.71:6819/692607380","heartbeat_back_addr":"172.21.15.71:6823/692607380","heartbeat_front_addr":"172.21.15.71:6821/692607380","state":["exists","up"]},{"osd":8,"uuid":"cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6817","nonce":3973697069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6819","nonce":3973697069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6823","nonce":3973697069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6821","nonce":3973697069}]},"public_addr":"172.21.15.189:6817/3973697069","cluster_addr":"172.21.15.189:6819/3973697069","heartbeat_back_addr":"172.21.15.189:6823/3973697069","heartbeat_front_addr":"172.21.15.189:6821/3973697069","state":["exists","up"]},{"osd":9,"uuid":"ef4a1645-ae6e-45c3-82ad-161510e62aa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6827","nonce":46066685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6829","nonce":46066685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6833","nonce":46066685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6831","nonce":46066685}]},"public_addr":"172.21.15.32:6827/46066685","cluster_addr":"172.21.15.32:6829/46066685","heartbeat_back_addr":"172.21.15.32:6833/46066685","heartbeat_front_addr":"172.21.15.32:6831/46066685","state":["exists","up"]},{"osd":10,"uuid":"25b26a38-a644-409f-b941-59efdf04a3ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6825","nonce":1886290293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6827","nonce":1886290293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6831","nonce":1886290293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6829","nonce":1886290293}]},"public_addr":"172.21.15.71:6825/1886290293","cluster_addr":"172.21.15.71:6827/1886290293","heartbeat_back_addr":"172.21.15.71:6831/1886290293","heartbeat_front_addr":"172.21.15.71:6829/1886290293","state":["exists","up"]},{"osd":11,"uuid":"ab96b3ff-3688-455b-af64-60226c5bfbff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6825","nonce":850185001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6827","nonce":850185001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6831","nonce":850185001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6829","nonce":850185001}]},"public_addr":"172.21.15.189:6825/850185001","cluster_addr":"172.21.15.189:6827/850185001","heartbeat_back_addr":"172.21.15.189:6831/850185001","heartbeat_front_addr":"172.21.15.189:6829/850185001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:27.220308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:53.230783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:17.190642+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:52.603799+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:20.568671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:46.158363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:22.706933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:50.665173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:16.602410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:53.814698+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:22.435189+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:48.651949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1266897222":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/2915689533":"2024-05-01T08:23:29.526664+0000","172.21.15.32:6801/4019351306":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/425836193":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/1134075722":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6801/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/21059253":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/3245550894":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/4019351306":"2024-05-01T08:23:29.526664+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-30T08:32:55.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.963+0000 7f5b9ce00700 1 -- 172.21.15.32:0/4002286629 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f5b7c0afc10 msgr2=0x7f5b7c0b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:55.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.963+0000 7f5b9ce00700 1 --2- 172.21.15.32:0/4002286629 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f5b7c0afc10 0x7f5b7c0b2120 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5b88009d60 tx=0x7f5b88009260 comp rx=0 tx=0).stop 2024-04-30T08:32:55.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.963+0000 7f5b9ce00700 1 -- 172.21.15.32:0/4002286629 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b980763a0 msgr2=0x7f5b9819d560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:32:55.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.963+0000 7f5b9ce00700 1 --2- 172.21.15.32:0/4002286629 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b980763a0 0x7f5b9819d560 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5b840059d0 tx=0x7f5b840057c0 comp rx=0 tx=0).stop 2024-04-30T08:32:55.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.963+0000 7f5b9ce00700 1 -- 172.21.15.32:0/4002286629 shutdown_connections 2024-04-30T08:32:55.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.963+0000 7f5b9ce00700 1 --2- 172.21.15.32:0/4002286629 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f5b7c0afc10 0x7f5b7c0b2120 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:55.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.963+0000 7f5b9ce00700 1 --2- 172.21.15.32:0/4002286629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5b98110b50 0x7f5b981a17f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:55.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.963+0000 7f5b9ce00700 1 --2- 172.21.15.32:0/4002286629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b98075a50 0x7f5b9819cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:55.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.963+0000 7f5b9ce00700 1 --2- 172.21.15.32:0/4002286629 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5b980763a0 0x7f5b9819d560 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:32:55.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.963+0000 7f5b9ce00700 1 -- 172.21.15.32:0/4002286629 >> 172.21.15.32:0/4002286629 conn(0x7f5b980fff90 msgr2=0x7f5b98100620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:32:55.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.963+0000 7f5b9ce00700 1 -- 172.21.15.32:0/4002286629 shutdown_connections 2024-04-30T08:32:55.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:32:55.963+0000 7f5b9ce00700 1 -- 172.21.15.32:0/4002286629 wait complete. 2024-04-30T08:32:56.806 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-30T08:32:56.807 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-30T08:32:56.807 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-30T08:32:56.808 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-30T08:32:56.808 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-30T08:32:56.808 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-30T08:32:56.809 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-30T08:32:56.809 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-30T08:32:56.810 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-30T08:32:56.810 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-30T08:32:56.810 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-30T08:32:56.811 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-30T08:32:57.012 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:56 smithi032 ceph-mon[28331]: pgmap v356: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-04-30T08:32:57.013 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:56 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/4002286629' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:32:57.013 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:56 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:32:57.013 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:56 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:32:57.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:56 smithi189 ceph-mon[33038]: pgmap v356: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-04-30T08:32:57.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:56 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/4002286629' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:32:57.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:56 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:32:57.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:56 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:32:57.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:56 smithi071 ceph-mon[33385]: pgmap v356: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-04-30T08:32:57.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:56 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/4002286629' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:32:57.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:56 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:32:57.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:56 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:32:57.864 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:32:57.958 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:32:57.984 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:32:58.032 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:32:58.404 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:32:58.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:58 smithi032 ceph-mon[28331]: pgmap v357: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 8 op/s 2024-04-30T08:32:58.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:58 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:32:58.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:58 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:32:58.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:32:58 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:32:59.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:58 smithi189 ceph-mon[33038]: pgmap v357: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 8 op/s 2024-04-30T08:32:59.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:58 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:32:59.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:58 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:32:59.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:32:58 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:32:59.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:58 smithi071 ceph-mon[33385]: pgmap v357: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 8 op/s 2024-04-30T08:32:59.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:58 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:32:59.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:58 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:32:59.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:32:58 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:33:01.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:00 smithi189 ceph-mon[33038]: pgmap v358: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-30T08:33:01.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:00 smithi032 ceph-mon[28331]: pgmap v358: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-30T08:33:01.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:00 smithi071 ceph-mon[33385]: pgmap v358: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-30T08:33:01.874 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:33:02.474 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:33:02.613 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:33:02.613 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:33:02.614 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:33:02.614 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:33:02.617 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:33:02.919 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:02 smithi032 ceph-mon[28331]: pgmap v359: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 341 B/s wr, 9 op/s 2024-04-30T08:33:03.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:02 smithi071 ceph-mon[33385]: pgmap v359: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 341 B/s wr, 9 op/s 2024-04-30T08:33:03.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:02 smithi189 ceph-mon[33038]: pgmap v359: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 341 B/s wr, 9 op/s 2024-04-30T08:33:05.105 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:04 smithi032 ceph-mon[28331]: pgmap v360: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 9 op/s 2024-04-30T08:33:05.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:04 smithi071 ceph-mon[33385]: pgmap v360: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 9 op/s 2024-04-30T08:33:05.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:04 smithi189 ceph-mon[33038]: pgmap v360: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 9 op/s 2024-04-30T08:33:06.992 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:06 smithi032 ceph-mon[28331]: pgmap v361: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-30T08:33:07.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:06 smithi071 ceph-mon[33385]: pgmap v361: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-30T08:33:07.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:06 smithi189 ceph-mon[33038]: pgmap v361: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-30T08:33:09.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.944+0000 7fda7d200700 1 -- 172.21.15.32:0/536532327 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda781109b0 msgr2=0x7fda78112dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:09.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.944+0000 7fda7d200700 1 --2- 172.21.15.32:0/536532327 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda781109b0 0x7fda78112dd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fda7000b0a0 tx=0x7fda7000b3b0 comp rx=0 tx=0).stop 2024-04-30T08:33:09.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.945+0000 7fda7d200700 1 -- 172.21.15.32:0/536532327 shutdown_connections 2024-04-30T08:33:09.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.945+0000 7fda7d200700 1 --2- 172.21.15.32:0/536532327 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda781109b0 0x7fda78112dd0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:09.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.945+0000 7fda7d200700 1 --2- 172.21.15.32:0/536532327 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda78102e00 0x7fda78103210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:09.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.945+0000 7fda7d200700 1 --2- 172.21.15.32:0/536532327 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda781037e0 0x7fda78110470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:09.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.945+0000 7fda7d200700 1 -- 172.21.15.32:0/536532327 >> 172.21.15.32:0/536532327 conn(0x7fda78078a50 msgr2=0x7fda78078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:09.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.945+0000 7fda7d200700 1 -- 172.21.15.32:0/536532327 shutdown_connections 2024-04-30T08:33:09.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.945+0000 7fda7d200700 1 -- 172.21.15.32:0/536532327 wait complete. 2024-04-30T08:33:09.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.946+0000 7fda7d200700 1 Processor -- start 2024-04-30T08:33:09.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.946+0000 7fda7d200700 1 -- start start 2024-04-30T08:33:09.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.947+0000 7fda7d200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda78102e00 0x7fda7819ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:09.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.947+0000 7fda7d200700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda781037e0 0x7fda7819d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:09.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.947+0000 7fda7d200700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda781109b0 0x7fda781a1650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:09.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.947+0000 7fda7d200700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fda78116470 con 0x7fda781037e0 2024-04-30T08:33:09.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.947+0000 7fda7d200700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fda781162f0 con 0x7fda781109b0 2024-04-30T08:33:09.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.947+0000 7fda7d200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fda781165f0 con 0x7fda78102e00 2024-04-30T08:33:09.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.947+0000 7fda6fe00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda781037e0 0x7fda7819d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:09.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.947+0000 7fda6fe00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda781037e0 0x7fda7819d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51276/0 (socket says 172.21.15.32:51276) 2024-04-30T08:33:09.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.947+0000 7fda6fe00700 1 -- 172.21.15.32:0/226227478 learned_addr learned my addr 172.21.15.32:0/226227478 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:33:09.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.947+0000 7fda77400700 1 --2- 172.21.15.32:0/226227478 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda78102e00 0x7fda7819ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:09.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.948+0000 7fda77e00700 1 --2- 172.21.15.32:0/226227478 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda781109b0 0x7fda781a1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:09.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.948+0000 7fda6fe00700 1 -- 172.21.15.32:0/226227478 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda78102e00 msgr2=0x7fda7819ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:09.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.948+0000 7fda6fe00700 1 --2- 172.21.15.32:0/226227478 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda78102e00 0x7fda7819ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:09.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.948+0000 7fda6fe00700 1 -- 172.21.15.32:0/226227478 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda781109b0 msgr2=0x7fda781a1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:09.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.948+0000 7fda6fe00700 1 --2- 172.21.15.32:0/226227478 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fda781109b0 0x7fda781a1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:09.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.948+0000 7fda6fe00700 1 -- 172.21.15.32:0/226227478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fda781a1cf0 con 0x7fda781037e0 2024-04-30T08:33:09.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.948+0000 7fda77400700 1 --2- 172.21.15.32:0/226227478 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda78102e00 0x7fda7819ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:33:09.115 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:08 smithi032 ceph-mon[28331]: pgmap v362: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-30T08:33:09.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.948+0000 7fda6fe00700 1 --2- 172.21.15.32:0/226227478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda781037e0 0x7fda7819d3c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fda6000d950 tx=0x7fda6000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:09.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.948+0000 7fda74c00700 1 -- 172.21.15.32:0/226227478 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda60016070 con 0x7fda781037e0 2024-04-30T08:33:09.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.949+0000 7fda7d200700 1 -- 172.21.15.32:0/226227478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fda781a1fe0 con 0x7fda781037e0 2024-04-30T08:33:09.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.949+0000 7fda74c00700 1 -- 172.21.15.32:0/226227478 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fda60005020 con 0x7fda781037e0 2024-04-30T08:33:09.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.949+0000 7fda74c00700 1 -- 172.21.15.32:0/226227478 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fda6000f9d0 con 0x7fda781037e0 2024-04-30T08:33:09.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.949+0000 7fda7d200700 1 -- 172.21.15.32:0/226227478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fda781a9900 con 0x7fda781037e0 2024-04-30T08:33:09.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.950+0000 7fda7d200700 1 -- 172.21.15.32:0/226227478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fda78063e20 con 0x7fda781037e0 2024-04-30T08:33:09.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.953+0000 7fda74c00700 1 -- 172.21.15.32:0/226227478 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fda600051d0 con 0x7fda781037e0 2024-04-30T08:33:09.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.955+0000 7fda74c00700 1 --2- 172.21.15.32:0/226227478 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fda5c0afc10 0x7fda5c0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:09.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.955+0000 7fda74c00700 1 -- 172.21.15.32:0/226227478 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fda600deec0 con 0x7fda781037e0 2024-04-30T08:33:09.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.955+0000 7fda77400700 1 --2- 172.21.15.32:0/226227478 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fda5c0afc10 0x7fda5c0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:09.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.956+0000 7fda74c00700 1 --2- 172.21.15.32:0/226227478 >> [v2:172.21.15.189:6816/3973697069,v1:172.21.15.189:6817/3973697069] conn(0x7fda5c0badc0 0x7fda5c0bd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:09.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.956+0000 7fda74c00700 1 -- 172.21.15.32:0/226227478 --> [v2:172.21.15.189:6816/3973697069,v1:172.21.15.189:6817/3973697069] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fda60024ea0 con 0x7fda5c0badc0 2024-04-30T08:33:09.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.956+0000 7fda74c00700 1 -- 172.21.15.32:0/226227478 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7fda600df270 con 0x7fda781037e0 2024-04-30T08:33:09.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.956+0000 7fda77e00700 1 --2- 172.21.15.32:0/226227478 >> [v2:172.21.15.189:6816/3973697069,v1:172.21.15.189:6817/3973697069] conn(0x7fda5c0badc0 0x7fda5c0bd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:09.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.956+0000 7fda77400700 1 --2- 172.21.15.32:0/226227478 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fda5c0afc10 0x7fda5c0b2120 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fda68011490 tx=0x7fda68009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:09.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.957+0000 7fda77e00700 1 --2- 172.21.15.32:0/226227478 >> [v2:172.21.15.189:6816/3973697069,v1:172.21.15.189:6817/3973697069] conn(0x7fda5c0badc0 0x7fda5c0bd2b0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:09.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.958+0000 7fda74c00700 1 -- 172.21.15.32:0/226227478 <== osd.8 v2:172.21.15.189:6816/3973697069 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fda60024ea0 con 0x7fda5c0badc0 2024-04-30T08:33:09.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.977+0000 7fda7d200700 1 -- 172.21.15.32:0/226227478 --> [v2:172.21.15.189:6816/3973697069,v1:172.21.15.189:6817/3973697069] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fda78064220 con 0x7fda5c0badc0 2024-04-30T08:33:09.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.978+0000 7fda74c00700 1 -- 172.21.15.32:0/226227478 <== osd.8 v2:172.21.15.189:6816/3973697069 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fda78064220 con 0x7fda5c0badc0 2024-04-30T08:33:09.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.978+0000 7fda7d200700 1 -- 172.21.15.32:0/226227478 >> [v2:172.21.15.189:6816/3973697069,v1:172.21.15.189:6817/3973697069] conn(0x7fda5c0badc0 msgr2=0x7fda5c0bd2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:09.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.978+0000 7fda7d200700 1 --2- 172.21.15.32:0/226227478 >> [v2:172.21.15.189:6816/3973697069,v1:172.21.15.189:6817/3973697069] conn(0x7fda5c0badc0 0x7fda5c0bd2b0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:09.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.978+0000 7fda7d200700 1 -- 172.21.15.32:0/226227478 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fda5c0afc10 msgr2=0x7fda5c0b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:09.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.978+0000 7fda7d200700 1 --2- 172.21.15.32:0/226227478 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fda5c0afc10 0x7fda5c0b2120 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fda68011490 tx=0x7fda68009260 comp rx=0 tx=0).stop 2024-04-30T08:33:09.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.978+0000 7fda7d200700 1 -- 172.21.15.32:0/226227478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda781037e0 msgr2=0x7fda7819d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:09.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.978+0000 7fda7d200700 1 --2- 172.21.15.32:0/226227478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fda781037e0 0x7fda7819d3c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fda6000d950 tx=0x7fda6000dc60 comp rx=0 tx=0).stop 2024-04-30T08:33:09.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.978+0000 7fda77e00700 1 -- 172.21.15.32:0/226227478 reap_dead start 2024-04-30T08:33:09.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.979+0000 7fda7d200700 1 -- 172.21.15.32:0/226227478 shutdown_connections 2024-04-30T08:33:09.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.979+0000 7fda7d200700 1 -- 172.21.15.32:0/226227478 >> 172.21.15.32:0/226227478 conn(0x7fda78078a50 msgr2=0x7fda78111c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:09.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.979+0000 7fda7d200700 1 -- 172.21.15.32:0/226227478 shutdown_connections 2024-04-30T08:33:09.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:08.979+0000 7fda7d200700 1 -- 172.21.15.32:0/226227478 wait complete. 2024-04-30T08:33:09.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:08 smithi071 ceph-mon[33385]: pgmap v362: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-30T08:33:09.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:08 smithi189 ceph-mon[33038]: pgmap v362: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-30T08:33:09.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.664+0000 7fa40a400700 1 -- 172.21.15.32:0/2219317476 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa404102fd0 msgr2=0x7fa4041033e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:09.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.664+0000 7fa40a400700 1 --2- 172.21.15.32:0/2219317476 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa404102fd0 0x7fa4041033e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa3f40098f0 tx=0x7fa3f4009c00 comp rx=0 tx=0).stop 2024-04-30T08:33:09.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.664+0000 7fa40a400700 1 -- 172.21.15.32:0/2219317476 shutdown_connections 2024-04-30T08:33:09.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.664+0000 7fa40a400700 1 --2- 172.21.15.32:0/2219317476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa404108420 0x7fa4041088f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:09.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.664+0000 7fa40a400700 1 --2- 172.21.15.32:0/2219317476 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa404102fd0 0x7fa4041033e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:09.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.664+0000 7fa40a400700 1 --2- 172.21.15.32:0/2219317476 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa404103920 0x7fa404107e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:09.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.664+0000 7fa40a400700 1 -- 172.21.15.32:0/2219317476 >> 172.21.15.32:0/2219317476 conn(0x7fa404078a10 msgr2=0x7fa404078e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:09.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.664+0000 7fa40a400700 1 -- 172.21.15.32:0/2219317476 shutdown_connections 2024-04-30T08:33:09.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.665+0000 7fa40a400700 1 -- 172.21.15.32:0/2219317476 wait complete. 2024-04-30T08:33:09.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.666+0000 7fa40a400700 1 Processor -- start 2024-04-30T08:33:09.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.666+0000 7fa40a400700 1 -- start start 2024-04-30T08:33:09.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.667+0000 7fa40a400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa404102fd0 0x7fa4041a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:09.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.667+0000 7fa40a400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa404103920 0x7fa4041a1790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:09.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.667+0000 7fa40a400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa404108420 0x7fa4041a5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:09.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.667+0000 7fa40a400700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa40411a800 con 0x7fa404108420 2024-04-30T08:33:09.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.667+0000 7fa40a400700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa40411a680 con 0x7fa404103920 2024-04-30T08:33:09.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.667+0000 7fa40a400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa40411a980 con 0x7fa404102fd0 2024-04-30T08:33:09.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.667+0000 7fa409a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa404108420 0x7fa4041a5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:09.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.667+0000 7fa409000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa404102fd0 0x7fa4041a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:09.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.667+0000 7fa3fbe00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa404103920 0x7fa4041a1790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:09.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.667+0000 7fa409000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa404102fd0 0x7fa4041a11f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:58974/0 (socket says 172.21.15.32:58974) 2024-04-30T08:33:09.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.667+0000 7fa3fbe00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa404103920 0x7fa4041a1790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:57820/0 (socket says 172.21.15.32:57820) 2024-04-30T08:33:09.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.667+0000 7fa409000700 1 -- 172.21.15.32:0/2040098096 learned_addr learned my addr 172.21.15.32:0/2040098096 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:33:09.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.668+0000 7fa409000700 1 -- 172.21.15.32:0/2040098096 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa404103920 msgr2=0x7fa4041a1790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:09.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.668+0000 7fa409000700 1 --2- 172.21.15.32:0/2040098096 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa404103920 0x7fa4041a1790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:09.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.668+0000 7fa409000700 1 -- 172.21.15.32:0/2040098096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa404108420 msgr2=0x7fa4041a5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:09.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.668+0000 7fa409000700 1 --2- 172.21.15.32:0/2040098096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa404108420 0x7fa4041a5a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:09.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.668+0000 7fa409000700 1 -- 172.21.15.32:0/2040098096 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4041a60c0 con 0x7fa404102fd0 2024-04-30T08:33:09.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.668+0000 7fa409a00700 1 --2- 172.21.15.32:0/2040098096 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa404108420 0x7fa4041a5a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:33:09.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.668+0000 7fa409000700 1 --2- 172.21.15.32:0/2040098096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa404102fd0 0x7fa4041a11f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4009230 tx=0x7fa3f400fa70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:09.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.669+0000 7fa402000700 1 -- 172.21.15.32:0/2040098096 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4005140 con 0x7fa404102fd0 2024-04-30T08:33:09.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.669+0000 7fa40a400700 1 -- 172.21.15.32:0/2040098096 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4041a6350 con 0x7fa404102fd0 2024-04-30T08:33:09.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.669+0000 7fa402000700 1 -- 172.21.15.32:0/2040098096 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa3f400b560 con 0x7fa404102fd0 2024-04-30T08:33:09.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.669+0000 7fa402000700 1 -- 172.21.15.32:0/2040098096 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3f4003970 con 0x7fa404102fd0 2024-04-30T08:33:09.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.669+0000 7fa40a400700 1 -- 172.21.15.32:0/2040098096 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4041a6830 con 0x7fa404102fd0 2024-04-30T08:33:09.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.670+0000 7fa40a400700 1 -- 172.21.15.32:0/2040098096 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa404107890 con 0x7fa404102fd0 2024-04-30T08:33:09.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.673+0000 7fa402000700 1 -- 172.21.15.32:0/2040098096 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fa3f400ba50 con 0x7fa404102fd0 2024-04-30T08:33:09.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.674+0000 7fa402000700 1 --2- 172.21.15.32:0/2040098096 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa3e80afc10 0x7fa3e80b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:09.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.674+0000 7fa402000700 1 -- 172.21.15.32:0/2040098096 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fa3f40e0570 con 0x7fa404102fd0 2024-04-30T08:33:09.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.675+0000 7fa3fbe00700 1 --2- 172.21.15.32:0/2040098096 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa3e80afc10 0x7fa3e80b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:09.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.675+0000 7fa402000700 1 --2- 172.21.15.32:0/2040098096 >> [v2:172.21.15.71:6800/444649711,v1:172.21.15.71:6801/444649711] conn(0x7fa3e80badc0 0x7fa3e80bd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:09.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.675+0000 7fa402000700 1 -- 172.21.15.32:0/2040098096 --> [v2:172.21.15.71:6800/444649711,v1:172.21.15.71:6801/444649711] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa3f400bdb0 con 0x7fa3e80badc0 2024-04-30T08:33:09.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.675+0000 7fa402000700 1 -- 172.21.15.32:0/2040098096 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7fa3f40e0920 con 0x7fa404102fd0 2024-04-30T08:33:09.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.675+0000 7fa409a00700 1 --2- 172.21.15.32:0/2040098096 >> [v2:172.21.15.71:6800/444649711,v1:172.21.15.71:6801/444649711] conn(0x7fa3e80badc0 0x7fa3e80bd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:09.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.675+0000 7fa3fbe00700 1 --2- 172.21.15.32:0/2040098096 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa3e80afc10 0x7fa3e80b2120 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa404101900 tx=0x7fa3ec009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:09.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.676+0000 7fa409a00700 1 --2- 172.21.15.32:0/2040098096 >> [v2:172.21.15.71:6800/444649711,v1:172.21.15.71:6801/444649711] conn(0x7fa3e80badc0 0x7fa3e80bd2b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:09.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.677+0000 7fa402000700 1 -- 172.21.15.32:0/2040098096 <== osd.1 v2:172.21.15.71:6800/444649711 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa3f400bdb0 con 0x7fa3e80badc0 2024-04-30T08:33:09.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.700+0000 7fa40a400700 1 -- 172.21.15.32:0/2040098096 --> [v2:172.21.15.71:6800/444649711,v1:172.21.15.71:6801/444649711] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa40406be30 con 0x7fa3e80badc0 2024-04-30T08:33:09.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.700+0000 7fa402000700 1 -- 172.21.15.32:0/2040098096 <== osd.1 v2:172.21.15.71:6800/444649711 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fa40406be30 con 0x7fa3e80badc0 2024-04-30T08:33:09.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.701+0000 7fa40a400700 1 -- 172.21.15.32:0/2040098096 >> [v2:172.21.15.71:6800/444649711,v1:172.21.15.71:6801/444649711] conn(0x7fa3e80badc0 msgr2=0x7fa3e80bd2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:09.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.701+0000 7fa40a400700 1 --2- 172.21.15.32:0/2040098096 >> [v2:172.21.15.71:6800/444649711,v1:172.21.15.71:6801/444649711] conn(0x7fa3e80badc0 0x7fa3e80bd2b0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:09.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.701+0000 7fa40a400700 1 -- 172.21.15.32:0/2040098096 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa3e80afc10 msgr2=0x7fa3e80b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:09.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.701+0000 7fa40a400700 1 --2- 172.21.15.32:0/2040098096 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa3e80afc10 0x7fa3e80b2120 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fa404101900 tx=0x7fa3ec009310 comp rx=0 tx=0).stop 2024-04-30T08:33:09.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.701+0000 7fa40a400700 1 -- 172.21.15.32:0/2040098096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa404102fd0 msgr2=0x7fa4041a11f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:09.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.701+0000 7fa40a400700 1 --2- 172.21.15.32:0/2040098096 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa404102fd0 0x7fa4041a11f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fa3f4009230 tx=0x7fa3f400fa70 comp rx=0 tx=0).stop 2024-04-30T08:33:09.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.701+0000 7fa409a00700 1 -- 172.21.15.32:0/2040098096 reap_dead start 2024-04-30T08:33:09.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.701+0000 7fa40a400700 1 -- 172.21.15.32:0/2040098096 shutdown_connections 2024-04-30T08:33:09.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.701+0000 7fa40a400700 1 -- 172.21.15.32:0/2040098096 >> 172.21.15.32:0/2040098096 conn(0x7fa404078a10 msgr2=0x7fa404105e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:09.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.701+0000 7fa40a400700 1 -- 172.21.15.32:0/2040098096 shutdown_connections 2024-04-30T08:33:09.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:09.701+0000 7fa40a400700 1 -- 172.21.15.32:0/2040098096 wait complete. 2024-04-30T08:33:11.102 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:10 smithi189 ceph-mon[33038]: pgmap v363: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-30T08:33:11.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:10 smithi032 ceph-mon[28331]: pgmap v363: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-30T08:33:11.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:10 smithi071 ceph-mon[33385]: pgmap v363: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-30T08:33:11.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.590+0000 7f8a98600700 1 -- 172.21.15.32:0/3553820302 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8a90107d40 msgr2=0x7f8a90108150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:11.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.590+0000 7f8a98600700 1 --2- 172.21.15.32:0/3553820302 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8a90107d40 0x7f8a90108150 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c009990 tx=0x7f8a7c009ca0 comp rx=0 tx=0).stop 2024-04-30T08:33:11.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.590+0000 7f8a98600700 1 -- 172.21.15.32:0/3553820302 shutdown_connections 2024-04-30T08:33:11.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.590+0000 7f8a98600700 1 --2- 172.21.15.32:0/3553820302 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a90108690 0x7f8a9010fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:11.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.590+0000 7f8a98600700 1 --2- 172.21.15.32:0/3553820302 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a90103410 0x7f8a90103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:11.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.590+0000 7f8a98600700 1 --2- 172.21.15.32:0/3553820302 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8a90107d40 0x7f8a90108150 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:11.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.590+0000 7f8a98600700 1 -- 172.21.15.32:0/3553820302 >> 172.21.15.32:0/3553820302 conn(0x7f8a900fed10 msgr2=0x7f8a90101130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:11.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.590+0000 7f8a98600700 1 -- 172.21.15.32:0/3553820302 shutdown_connections 2024-04-30T08:33:11.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.591+0000 7f8a98600700 1 -- 172.21.15.32:0/3553820302 wait complete. 2024-04-30T08:33:11.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.592+0000 7f8a98600700 1 Processor -- start 2024-04-30T08:33:11.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.592+0000 7f8a98600700 1 -- start start 2024-04-30T08:33:11.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.593+0000 7f8a98600700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a90103410 0x7f8a9019f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:11.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.593+0000 7f8a98600700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8a90107d40 0x7f8a9019fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:11.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.593+0000 7f8a98600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a90108690 0x7f8a90199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:11.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.593+0000 7f8a98600700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8a901132f0 con 0x7f8a90103410 2024-04-30T08:33:11.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.593+0000 7f8a98600700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8a90113170 con 0x7f8a90107d40 2024-04-30T08:33:11.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.593+0000 7f8a98600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8a90113470 con 0x7f8a90108690 2024-04-30T08:33:11.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.593+0000 7f8a95c00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a90103410 0x7f8a9019f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:11.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.593+0000 7f8a95200700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8a90107d40 0x7f8a9019fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:11.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.593+0000 7f8a95c00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a90103410 0x7f8a9019f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51330/0 (socket says 172.21.15.32:51330) 2024-04-30T08:33:11.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.593+0000 7f8a95c00700 1 -- 172.21.15.32:0/1211851565 learned_addr learned my addr 172.21.15.32:0/1211851565 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:33:11.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.593+0000 7f8a96600700 1 --2- 172.21.15.32:0/1211851565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a90108690 0x7f8a90199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:11.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.594+0000 7f8a95200700 1 -- 172.21.15.32:0/1211851565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a90108690 msgr2=0x7f8a90199a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:11.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.594+0000 7f8a95200700 1 --2- 172.21.15.32:0/1211851565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8a90108690 0x7f8a90199a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:11.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.594+0000 7f8a95200700 1 -- 172.21.15.32:0/1211851565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a90103410 msgr2=0x7f8a9019f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:11.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.594+0000 7f8a95200700 1 --2- 172.21.15.32:0/1211851565 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8a90103410 0x7f8a9019f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:11.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.594+0000 7f8a95200700 1 -- 172.21.15.32:0/1211851565 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a9019a140 con 0x7f8a90107d40 2024-04-30T08:33:11.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.594+0000 7f8a95200700 1 --2- 172.21.15.32:0/1211851565 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8a90107d40 0x7f8a9019fdb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c000c00 tx=0x7f8a7c009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:11.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.594+0000 7f8a8e000700 1 -- 172.21.15.32:0/1211851565 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a7c026070 con 0x7f8a90107d40 2024-04-30T08:33:11.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.595+0000 7f8a98600700 1 -- 172.21.15.32:0/1211851565 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a9019a3d0 con 0x7f8a90107d40 2024-04-30T08:33:11.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.595+0000 7f8a8e000700 1 -- 172.21.15.32:0/1211851565 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8a7c00b610 con 0x7f8a90107d40 2024-04-30T08:33:11.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.595+0000 7f8a8e000700 1 -- 172.21.15.32:0/1211851565 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a7c021e00 con 0x7f8a90107d40 2024-04-30T08:33:11.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.595+0000 7f8a98600700 1 -- 172.21.15.32:0/1211851565 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a901a6700 con 0x7f8a90107d40 2024-04-30T08:33:11.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.596+0000 7f8a98600700 1 -- 172.21.15.32:0/1211851565 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8a90063e60 con 0x7f8a90107d40 2024-04-30T08:33:11.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.599+0000 7f8a8e000700 1 -- 172.21.15.32:0/1211851565 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f8a7c029070 con 0x7f8a90107d40 2024-04-30T08:33:11.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.600+0000 7f8a8e000700 1 --2- 172.21.15.32:0/1211851565 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8a780afc90 0x7f8a780b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:11.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.600+0000 7f8a8e000700 1 -- 172.21.15.32:0/1211851565 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f8a7c0e0370 con 0x7f8a90107d40 2024-04-30T08:33:11.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.600+0000 7f8a95c00700 1 --2- 172.21.15.32:0/1211851565 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8a780afc90 0x7f8a780b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:11.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.601+0000 7f8a8e000700 1 --2- 172.21.15.32:0/1211851565 >> [v2:172.21.15.189:6800/3255528671,v1:172.21.15.189:6801/3255528671] conn(0x7f8a780bae40 0x7f8a780bd330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:11.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.601+0000 7f8a8e000700 1 -- 172.21.15.32:0/1211851565 --> [v2:172.21.15.189:6800/3255528671,v1:172.21.15.189:6801/3255528671] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8a7c010ea0 con 0x7f8a780bae40 2024-04-30T08:33:11.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.601+0000 7f8a8e000700 1 -- 172.21.15.32:0/1211851565 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7f8a7c0e0720 con 0x7f8a90107d40 2024-04-30T08:33:11.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.601+0000 7f8a96600700 1 --2- 172.21.15.32:0/1211851565 >> [v2:172.21.15.189:6800/3255528671,v1:172.21.15.189:6801/3255528671] conn(0x7f8a780bae40 0x7f8a780bd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:11.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.601+0000 7f8a95c00700 1 --2- 172.21.15.32:0/1211851565 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8a780afc90 0x7f8a780b21a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8a84009da0 tx=0x7f8a8400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:11.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.602+0000 7f8a96600700 1 --2- 172.21.15.32:0/1211851565 >> [v2:172.21.15.189:6800/3255528671,v1:172.21.15.189:6801/3255528671] conn(0x7f8a780bae40 0x7f8a780bd330 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:11.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.603+0000 7f8a8e000700 1 -- 172.21.15.32:0/1211851565 <== osd.2 v2:172.21.15.189:6800/3255528671 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8a7c010ea0 con 0x7f8a780bae40 2024-04-30T08:33:11.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.621+0000 7f8a98600700 1 -- 172.21.15.32:0/1211851565 --> [v2:172.21.15.189:6800/3255528671,v1:172.21.15.189:6801/3255528671] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8a90064260 con 0x7f8a780bae40 2024-04-30T08:33:11.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.622+0000 7f8a8e000700 1 -- 172.21.15.32:0/1211851565 <== osd.2 v2:172.21.15.189:6800/3255528671 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8a90064260 con 0x7f8a780bae40 2024-04-30T08:33:11.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.622+0000 7f8a98600700 1 -- 172.21.15.32:0/1211851565 >> [v2:172.21.15.189:6800/3255528671,v1:172.21.15.189:6801/3255528671] conn(0x7f8a780bae40 msgr2=0x7f8a780bd330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:11.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.622+0000 7f8a98600700 1 --2- 172.21.15.32:0/1211851565 >> [v2:172.21.15.189:6800/3255528671,v1:172.21.15.189:6801/3255528671] conn(0x7f8a780bae40 0x7f8a780bd330 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:11.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.623+0000 7f8a98600700 1 -- 172.21.15.32:0/1211851565 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8a780afc90 msgr2=0x7f8a780b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:11.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.623+0000 7f8a98600700 1 --2- 172.21.15.32:0/1211851565 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8a780afc90 0x7f8a780b21a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8a84009da0 tx=0x7f8a8400b040 comp rx=0 tx=0).stop 2024-04-30T08:33:11.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.623+0000 7f8a98600700 1 -- 172.21.15.32:0/1211851565 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8a90107d40 msgr2=0x7f8a9019fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:11.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.623+0000 7f8a98600700 1 --2- 172.21.15.32:0/1211851565 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8a90107d40 0x7f8a9019fdb0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8a7c000c00 tx=0x7f8a7c009580 comp rx=0 tx=0).stop 2024-04-30T08:33:11.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.623+0000 7f8a96600700 1 -- 172.21.15.32:0/1211851565 reap_dead start 2024-04-30T08:33:11.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.623+0000 7f8a98600700 1 -- 172.21.15.32:0/1211851565 shutdown_connections 2024-04-30T08:33:11.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.623+0000 7f8a98600700 1 -- 172.21.15.32:0/1211851565 >> 172.21.15.32:0/1211851565 conn(0x7f8a900fed10 msgr2=0x7f8a901095f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:11.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.623+0000 7f8a98600700 1 -- 172.21.15.32:0/1211851565 shutdown_connections 2024-04-30T08:33:11.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:11.623+0000 7f8a98600700 1 -- 172.21.15.32:0/1211851565 wait complete. 2024-04-30T08:33:12.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.280+0000 7fbbc1600700 1 -- 172.21.15.32:0/2616691550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbbc10b060 msgr2=0x7fbbbc117570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:12.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.280+0000 7fbbc1600700 1 --2- 172.21.15.32:0/2616691550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbbc10b060 0x7fbbbc117570 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbbb4009f90 tx=0x7fbbb400b330 comp rx=0 tx=0).stop 2024-04-30T08:33:12.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.280+0000 7fbbc1600700 1 -- 172.21.15.32:0/2616691550 shutdown_connections 2024-04-30T08:33:12.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.280+0000 7fbbc1600700 1 --2- 172.21.15.32:0/2616691550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbbc10b060 0x7fbbbc117570 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:12.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.280+0000 7fbbc1600700 1 --2- 172.21.15.32:0/2616691550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbbbc10f610 0x7fbbbc10fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:12.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.280+0000 7fbbc1600700 1 --2- 172.21.15.32:0/2616691550 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbbbc10a710 0x7fbbbc10ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:12.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.280+0000 7fbbc1600700 1 -- 172.21.15.32:0/2616691550 >> 172.21.15.32:0/2616691550 conn(0x7fbbbc076a50 msgr2=0x7fbbbc076e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:12.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.280+0000 7fbbc1600700 1 -- 172.21.15.32:0/2616691550 shutdown_connections 2024-04-30T08:33:12.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.280+0000 7fbbc1600700 1 -- 172.21.15.32:0/2616691550 wait complete. 2024-04-30T08:33:12.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.282+0000 7fbbc1600700 1 Processor -- start 2024-04-30T08:33:12.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.282+0000 7fbbc1600700 1 -- start start 2024-04-30T08:33:12.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.283+0000 7fbbc1600700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbbbc10a710 0x7fbbbc1a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:12.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.283+0000 7fbbc1600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbbc10b060 0x7fbbbc1a16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:12.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.283+0000 7fbbc1600700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbbbc10f610 0x7fbbbc1a5960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:12.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.283+0000 7fbbc1600700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbbbc11a840 con 0x7fbbbc10a710 2024-04-30T08:33:12.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.283+0000 7fbbc1600700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fbbbc11a6c0 con 0x7fbbbc10f610 2024-04-30T08:33:12.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.283+0000 7fbbc1600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbbbc11a9c0 con 0x7fbbbc10b060 2024-04-30T08:33:12.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.283+0000 7fbbbbe00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbbbc10a710 0x7fbbbc1a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:12.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.283+0000 7fbbbbe00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbbbc10a710 0x7fbbbc1a1130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51256/0 (socket says 172.21.15.32:51256) 2024-04-30T08:33:12.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.283+0000 7fbbbbe00700 1 -- 172.21.15.32:0/3078490058 learned_addr learned my addr 172.21.15.32:0/3078490058 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:33:12.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.283+0000 7fbbc0c00700 1 --2- 172.21.15.32:0/3078490058 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbbbc10f610 0x7fbbbc1a5960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:12.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.283+0000 7fbbbb400700 1 --2- 172.21.15.32:0/3078490058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbbc10b060 0x7fbbbc1a16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:12.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.284+0000 7fbbbbe00700 1 -- 172.21.15.32:0/3078490058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbbc10b060 msgr2=0x7fbbbc1a16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:12.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.284+0000 7fbbbbe00700 1 --2- 172.21.15.32:0/3078490058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbbbc10b060 0x7fbbbc1a16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:12.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.284+0000 7fbbbbe00700 1 -- 172.21.15.32:0/3078490058 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbbbc10f610 msgr2=0x7fbbbc1a5960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:12.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.284+0000 7fbbbbe00700 1 --2- 172.21.15.32:0/3078490058 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbbbc10f610 0x7fbbbc1a5960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:12.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.284+0000 7fbbbbe00700 1 -- 172.21.15.32:0/3078490058 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbbc1a6000 con 0x7fbbbc10a710 2024-04-30T08:33:12.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.284+0000 7fbbc0c00700 1 --2- 172.21.15.32:0/3078490058 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbbbc10f610 0x7fbbbc1a5960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:33:12.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.284+0000 7fbbbbe00700 1 --2- 172.21.15.32:0/3078490058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbbbc10a710 0x7fbbbc1a1130 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbbac00e9c0 tx=0x7fbbac00ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:12.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.284+0000 7fbbb8c00700 1 -- 172.21.15.32:0/3078490058 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbac004a30 con 0x7fbbbc10a710 2024-04-30T08:33:12.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.284+0000 7fbbb8c00700 1 -- 172.21.15.32:0/3078490058 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbbac00cc70 con 0x7fbbbc10a710 2024-04-30T08:33:12.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.284+0000 7fbbb8c00700 1 -- 172.21.15.32:0/3078490058 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbac005490 con 0x7fbbbc10a710 2024-04-30T08:33:12.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.284+0000 7fbbc1600700 1 -- 172.21.15.32:0/3078490058 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbbc1a62f0 con 0x7fbbbc10a710 2024-04-30T08:33:12.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.285+0000 7fbbc1600700 1 -- 172.21.15.32:0/3078490058 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbbc1adc10 con 0x7fbbbc10a710 2024-04-30T08:33:12.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.285+0000 7fbbc1600700 1 -- 172.21.15.32:0/3078490058 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbbbc063ea0 con 0x7fbbbc10a710 2024-04-30T08:33:12.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.288+0000 7fbbb8c00700 1 -- 172.21.15.32:0/3078490058 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fbbac004bd0 con 0x7fbbbc10a710 2024-04-30T08:33:12.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.290+0000 7fbbb8c00700 1 --2- 172.21.15.32:0/3078490058 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbba00afc10 0x7fbba00b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:12.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.290+0000 7fbbb8c00700 1 -- 172.21.15.32:0/3078490058 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7fbbac0df5f0 con 0x7fbbbc10a710 2024-04-30T08:33:12.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.290+0000 7fbbbb400700 1 --2- 172.21.15.32:0/3078490058 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbba00afc10 0x7fbba00b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:12.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.290+0000 7fbbb8c00700 1 --2- 172.21.15.32:0/3078490058 >> [v2:172.21.15.71:6808/4250534574,v1:172.21.15.71:6809/4250534574] conn(0x7fbba00badc0 0x7fbba00bd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:12.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.290+0000 7fbbb8c00700 1 -- 172.21.15.32:0/3078490058 --> [v2:172.21.15.71:6808/4250534574,v1:172.21.15.71:6809/4250534574] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbba00bd980 con 0x7fbba00badc0 2024-04-30T08:33:12.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.290+0000 7fbbb8c00700 1 -- 172.21.15.32:0/3078490058 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7fbbac0df9a0 con 0x7fbbbc10a710 2024-04-30T08:33:12.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.290+0000 7fbbc0c00700 1 --2- 172.21.15.32:0/3078490058 >> [v2:172.21.15.71:6808/4250534574,v1:172.21.15.71:6809/4250534574] conn(0x7fbba00badc0 0x7fbba00bd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:12.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.291+0000 7fbbbb400700 1 --2- 172.21.15.32:0/3078490058 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbba00afc10 0x7fbba00b2120 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc077e10 tx=0x7fbba8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:12.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.291+0000 7fbbc0c00700 1 --2- 172.21.15.32:0/3078490058 >> [v2:172.21.15.71:6808/4250534574,v1:172.21.15.71:6809/4250534574] conn(0x7fbba00badc0 0x7fbba00bd2b0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:12.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.292+0000 7fbbb8c00700 1 -- 172.21.15.32:0/3078490058 <== osd.4 v2:172.21.15.71:6808/4250534574 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbba00bd980 con 0x7fbba00badc0 2024-04-30T08:33:12.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.311+0000 7fbbc1600700 1 -- 172.21.15.32:0/3078490058 --> [v2:172.21.15.71:6808/4250534574,v1:172.21.15.71:6809/4250534574] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbbbc064320 con 0x7fbba00badc0 2024-04-30T08:33:12.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.311+0000 7fbbb8c00700 1 -- 172.21.15.32:0/3078490058 <== osd.4 v2:172.21.15.71:6808/4250534574 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fbbbc064320 con 0x7fbba00badc0 2024-04-30T08:33:12.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.312+0000 7fbbc1600700 1 -- 172.21.15.32:0/3078490058 >> [v2:172.21.15.71:6808/4250534574,v1:172.21.15.71:6809/4250534574] conn(0x7fbba00badc0 msgr2=0x7fbba00bd2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:12.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.312+0000 7fbbc1600700 1 --2- 172.21.15.32:0/3078490058 >> [v2:172.21.15.71:6808/4250534574,v1:172.21.15.71:6809/4250534574] conn(0x7fbba00badc0 0x7fbba00bd2b0 crc :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:12.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.312+0000 7fbbc1600700 1 -- 172.21.15.32:0/3078490058 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbba00afc10 msgr2=0x7fbba00b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:12.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.312+0000 7fbbc1600700 1 --2- 172.21.15.32:0/3078490058 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbba00afc10 0x7fbba00b2120 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc077e10 tx=0x7fbba8009260 comp rx=0 tx=0).stop 2024-04-30T08:33:12.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.312+0000 7fbbc1600700 1 -- 172.21.15.32:0/3078490058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbbbc10a710 msgr2=0x7fbbbc1a1130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:12.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.312+0000 7fbbc1600700 1 --2- 172.21.15.32:0/3078490058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbbbc10a710 0x7fbbbc1a1130 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbbac00e9c0 tx=0x7fbbac00ecd0 comp rx=0 tx=0).stop 2024-04-30T08:33:12.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.312+0000 7fbbc0c00700 1 -- 172.21.15.32:0/3078490058 reap_dead start 2024-04-30T08:33:12.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.312+0000 7fbbc1600700 1 -- 172.21.15.32:0/3078490058 shutdown_connections 2024-04-30T08:33:12.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.312+0000 7fbbc1600700 1 -- 172.21.15.32:0/3078490058 >> 172.21.15.32:0/3078490058 conn(0x7fbbbc076a50 msgr2=0x7fbbbc108770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:12.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.313+0000 7fbbc1600700 1 -- 172.21.15.32:0/3078490058 shutdown_connections 2024-04-30T08:33:12.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:07.313+0000 7fbbc1600700 1 -- 172.21.15.32:0/3078490058 wait complete. 2024-04-30T08:33:13.157 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:12 smithi032 ceph-mon[28331]: pgmap v364: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-30T08:33:13.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:12 smithi071 ceph-mon[33385]: pgmap v364: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-30T08:33:13.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:12 smithi189 ceph-mon[33038]: pgmap v364: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-30T08:33:14.072 INFO:teuthology.orchestra.run.smithi032.stdout:231928234034 2024-04-30T08:33:14.072 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-30T08:33:14.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:14 smithi032 ceph-mon[28331]: pgmap v365: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:33:15.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:14 smithi071 ceph-mon[33385]: pgmap v365: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:33:15.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:14 smithi189 ceph-mon[33038]: pgmap v365: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:33:15.388 INFO:teuthology.orchestra.run.smithi032.stdout:133143986247 2024-04-30T08:33:15.388 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-30T08:33:17.036 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:16 smithi032 ceph-mon[28331]: pgmap v366: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-30T08:33:17.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:16 smithi071 ceph-mon[33385]: pgmap v366: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-30T08:33:17.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.286+0000 7efe33e00700 1 -- 172.21.15.32:0/3121792420 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efe34104350 msgr2=0x7efe34104760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:17.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.286+0000 7efe33e00700 1 --2- 172.21.15.32:0/3121792420 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efe34104350 0x7efe34104760 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7efe24009f90 tx=0x7efe2400b330 comp rx=0 tx=0).stop 2024-04-30T08:33:17.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.286+0000 7efe33e00700 1 -- 172.21.15.32:0/3121792420 shutdown_connections 2024-04-30T08:33:17.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.286+0000 7efe33e00700 1 --2- 172.21.15.32:0/3121792420 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe341095d0 0x7efe34110f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:17.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.286+0000 7efe33e00700 1 --2- 172.21.15.32:0/3121792420 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efe34104350 0x7efe34104760 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:17.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.286+0000 7efe33e00700 1 --2- 172.21.15.32:0/3121792420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe34108c80 0x7efe34109090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:17.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.286+0000 7efe33e00700 1 -- 172.21.15.32:0/3121792420 >> 172.21.15.32:0/3121792420 conn(0x7efe34079c60 msgr2=0x7efe3407a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:17.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.287+0000 7efe33e00700 1 -- 172.21.15.32:0/3121792420 shutdown_connections 2024-04-30T08:33:17.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.287+0000 7efe33e00700 1 -- 172.21.15.32:0/3121792420 wait complete. 2024-04-30T08:33:17.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.288+0000 7efe33e00700 1 Processor -- start 2024-04-30T08:33:17.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.288+0000 7efe33e00700 1 -- start start 2024-04-30T08:33:17.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.289+0000 7efe33e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe34104350 0x7efe3419ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:17.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.289+0000 7efe33e00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efe34108c80 0x7efe3419d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:17.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.289+0000 7efe33e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe341095d0 0x7efe341a1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:17.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.289+0000 7efe33e00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7efe34114250 con 0x7efe34104350 2024-04-30T08:33:17.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.289+0000 7efe33e00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7efe341140d0 con 0x7efe34108c80 2024-04-30T08:33:17.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.289+0000 7efe33e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efe341143d0 con 0x7efe341095d0 2024-04-30T08:33:17.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.290+0000 7efe32a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe34104350 0x7efe3419ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:17.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.290+0000 7efe32000700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efe34108c80 0x7efe3419d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:17.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.290+0000 7efe32000700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efe34108c80 0x7efe3419d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:60056/0 (socket says 172.21.15.32:60056) 2024-04-30T08:33:17.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.290+0000 7efe32000700 1 -- 172.21.15.32:0/1731903815 learned_addr learned my addr 172.21.15.32:0/1731903815 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:33:17.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.290+0000 7efe33400700 1 --2- 172.21.15.32:0/1731903815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe341095d0 0x7efe341a1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:17.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.290+0000 7efe32000700 1 -- 172.21.15.32:0/1731903815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe341095d0 msgr2=0x7efe341a1690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:17.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.290+0000 7efe32000700 1 --2- 172.21.15.32:0/1731903815 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efe341095d0 0x7efe341a1690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:17.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.290+0000 7efe32000700 1 -- 172.21.15.32:0/1731903815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe34104350 msgr2=0x7efe3419ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:17.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.290+0000 7efe32000700 1 --2- 172.21.15.32:0/1731903815 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efe34104350 0x7efe3419ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:17.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.290+0000 7efe32000700 1 -- 172.21.15.32:0/1731903815 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe341a1d30 con 0x7efe34108c80 2024-04-30T08:33:17.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.290+0000 7efe32000700 1 --2- 172.21.15.32:0/1731903815 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efe34108c80 0x7efe3419d400 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7efe2000e970 tx=0x7efe2000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:17.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.291+0000 7efe2b400700 1 -- 172.21.15.32:0/1731903815 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe20004a30 con 0x7efe34108c80 2024-04-30T08:33:17.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.291+0000 7efe2b400700 1 -- 172.21.15.32:0/1731903815 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efe2000cc70 con 0x7efe34108c80 2024-04-30T08:33:17.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.291+0000 7efe33e00700 1 -- 172.21.15.32:0/1731903815 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe341a2020 con 0x7efe34108c80 2024-04-30T08:33:17.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.291+0000 7efe2b400700 1 -- 172.21.15.32:0/1731903815 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe20005580 con 0x7efe34108c80 2024-04-30T08:33:17.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.291+0000 7efe33e00700 1 -- 172.21.15.32:0/1731903815 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe341a9940 con 0x7efe34108c80 2024-04-30T08:33:17.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.292+0000 7efe33e00700 1 -- 172.21.15.32:0/1731903815 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7efe341024b0 con 0x7efe34108c80 2024-04-30T08:33:17.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.295+0000 7efe2b400700 1 -- 172.21.15.32:0/1731903815 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7efe2000c7c0 con 0x7efe34108c80 2024-04-30T08:33:17.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.296+0000 7efe2b400700 1 --2- 172.21.15.32:0/1731903815 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7efe180afc90 0x7efe180b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:17.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.296+0000 7efe2b400700 1 -- 172.21.15.32:0/1731903815 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7efe200df690 con 0x7efe34108c80 2024-04-30T08:33:17.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.296+0000 7efe32a00700 1 --2- 172.21.15.32:0/1731903815 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7efe180afc90 0x7efe180b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:17.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.297+0000 7efe2b400700 1 --2- 172.21.15.32:0/1731903815 >> [v2:172.21.15.32:6802/2624700402,v1:172.21.15.32:6803/2624700402] conn(0x7efe180bae40 0x7efe180bd330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:17.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.297+0000 7efe2b400700 1 -- 172.21.15.32:0/1731903815 --> [v2:172.21.15.32:6802/2624700402,v1:172.21.15.32:6803/2624700402] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efe180bda00 con 0x7efe180bae40 2024-04-30T08:33:17.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.297+0000 7efe2b400700 1 -- 172.21.15.32:0/1731903815 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7efe200dfa40 con 0x7efe34108c80 2024-04-30T08:33:17.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.297+0000 7efe33400700 1 --2- 172.21.15.32:0/1731903815 >> [v2:172.21.15.32:6802/2624700402,v1:172.21.15.32:6803/2624700402] conn(0x7efe180bae40 0x7efe180bd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:17.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.297+0000 7efe32a00700 1 --2- 172.21.15.32:0/1731903815 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7efe180afc90 0x7efe180b21a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7efe24009da0 tx=0x7efe240096c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:17.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.298+0000 7efe33400700 1 --2- 172.21.15.32:0/1731903815 >> [v2:172.21.15.32:6802/2624700402,v1:172.21.15.32:6803/2624700402] conn(0x7efe180bae40 0x7efe180bd330 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:17.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:16 smithi189 ceph-mon[33038]: pgmap v366: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-30T08:33:17.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.299+0000 7efe2b400700 1 -- 172.21.15.32:0/1731903815 <== osd.0 v2:172.21.15.32:6802/2624700402 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7efe180bda00 con 0x7efe180bae40 2024-04-30T08:33:17.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.317+0000 7efe33e00700 1 -- 172.21.15.32:0/1731903815 --> [v2:172.21.15.32:6802/2624700402,v1:172.21.15.32:6803/2624700402] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7efe34105800 con 0x7efe180bae40 2024-04-30T08:33:17.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.318+0000 7efe2b400700 1 -- 172.21.15.32:0/1731903815 <== osd.0 v2:172.21.15.32:6802/2624700402 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7efe34105800 con 0x7efe180bae40 2024-04-30T08:33:17.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.318+0000 7efe33e00700 1 -- 172.21.15.32:0/1731903815 >> [v2:172.21.15.32:6802/2624700402,v1:172.21.15.32:6803/2624700402] conn(0x7efe180bae40 msgr2=0x7efe180bd330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:17.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.318+0000 7efe33e00700 1 --2- 172.21.15.32:0/1731903815 >> [v2:172.21.15.32:6802/2624700402,v1:172.21.15.32:6803/2624700402] conn(0x7efe180bae40 0x7efe180bd330 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:17.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.318+0000 7efe33e00700 1 -- 172.21.15.32:0/1731903815 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7efe180afc90 msgr2=0x7efe180b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:17.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.318+0000 7efe33e00700 1 --2- 172.21.15.32:0/1731903815 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7efe180afc90 0x7efe180b21a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7efe24009da0 tx=0x7efe240096c0 comp rx=0 tx=0).stop 2024-04-30T08:33:17.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.318+0000 7efe33e00700 1 -- 172.21.15.32:0/1731903815 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efe34108c80 msgr2=0x7efe3419d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:17.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.318+0000 7efe33e00700 1 --2- 172.21.15.32:0/1731903815 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efe34108c80 0x7efe3419d400 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7efe2000e970 tx=0x7efe2000ec80 comp rx=0 tx=0).stop 2024-04-30T08:33:17.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.318+0000 7efe33400700 1 -- 172.21.15.32:0/1731903815 reap_dead start 2024-04-30T08:33:17.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.319+0000 7efe33e00700 1 -- 172.21.15.32:0/1731903815 shutdown_connections 2024-04-30T08:33:17.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.319+0000 7efe33e00700 1 -- 172.21.15.32:0/1731903815 >> 172.21.15.32:0/1731903815 conn(0x7efe34079c60 msgr2=0x7efe34101ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:17.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.319+0000 7efe33e00700 1 -- 172.21.15.32:0/1731903815 shutdown_connections 2024-04-30T08:33:17.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.319+0000 7efe33e00700 1 -- 172.21.15.32:0/1731903815 wait complete. 2024-04-30T08:33:17.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.613+0000 7f51b6a00700 1 -- 172.21.15.32:0/4054469228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51b010a8e0 msgr2=0x7f51b010acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:17.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.613+0000 7f51b6a00700 1 --2- 172.21.15.32:0/4054469228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51b010a8e0 0x7f51b010acf0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f519c009990 tx=0x7f519c009ca0 comp rx=0 tx=0).stop 2024-04-30T08:33:17.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.613+0000 7f51b6a00700 1 -- 172.21.15.32:0/4054469228 shutdown_connections 2024-04-30T08:33:17.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.613+0000 7f51b6a00700 1 --2- 172.21.15.32:0/4054469228 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f51b010b230 0x7f51b0117760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:17.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.613+0000 7f51b6a00700 1 --2- 172.21.15.32:0/4054469228 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51b01108c0 0x7f51b0110cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:17.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.613+0000 7f51b6a00700 1 --2- 172.21.15.32:0/4054469228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51b010a8e0 0x7f51b010acf0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:17.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.613+0000 7f51b6a00700 1 -- 172.21.15.32:0/4054469228 >> 172.21.15.32:0/4054469228 conn(0x7f51b0077bd0 msgr2=0x7f51b0078030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:17.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.613+0000 7f51b6a00700 1 -- 172.21.15.32:0/4054469228 shutdown_connections 2024-04-30T08:33:17.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.614+0000 7f51b6a00700 1 -- 172.21.15.32:0/4054469228 wait complete. 2024-04-30T08:33:17.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.615+0000 7f51b6a00700 1 Processor -- start 2024-04-30T08:33:17.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.615+0000 7f51b6a00700 1 -- start start 2024-04-30T08:33:17.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.616+0000 7f51b6a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51b010a8e0 0x7f51b01a1420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:17.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.616+0000 7f51b6a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51b010b230 0x7f51b01a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:17.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.616+0000 7f51b6a00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f51b01108c0 0x7f51b01a5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:17.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.616+0000 7f51b6a00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f51b011aa30 con 0x7f51b010b230 2024-04-30T08:33:17.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.616+0000 7f51b6a00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f51b011a8b0 con 0x7f51b01108c0 2024-04-30T08:33:17.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.616+0000 7f51b6a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f51b011abb0 con 0x7f51b010a8e0 2024-04-30T08:33:17.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.616+0000 7f51af400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51b010b230 0x7f51b01a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:17.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.616+0000 7f51afe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51b010a8e0 0x7f51b01a1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:17.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.616+0000 7f51afe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51b010a8e0 0x7f51b01a1420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:46808/0 (socket says 172.21.15.32:46808) 2024-04-30T08:33:17.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.616+0000 7f51afe00700 1 -- 172.21.15.32:0/2283957061 learned_addr learned my addr 172.21.15.32:0/2283957061 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:33:17.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.616+0000 7f51b4a00700 1 --2- 172.21.15.32:0/2283957061 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f51b01108c0 0x7f51b01a5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:17.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.617+0000 7f51afe00700 1 -- 172.21.15.32:0/2283957061 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f51b01108c0 msgr2=0x7f51b01a5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:17.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.617+0000 7f51afe00700 1 --2- 172.21.15.32:0/2283957061 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f51b01108c0 0x7f51b01a5c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:17.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.617+0000 7f51afe00700 1 -- 172.21.15.32:0/2283957061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51b010b230 msgr2=0x7f51b01a19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:17.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.617+0000 7f51afe00700 1 --2- 172.21.15.32:0/2283957061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51b010b230 0x7f51b01a19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:17.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.617+0000 7f51afe00700 1 -- 172.21.15.32:0/2283957061 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51b01a62f0 con 0x7f51b010a8e0 2024-04-30T08:33:17.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.617+0000 7f51af400700 1 --2- 172.21.15.32:0/2283957061 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51b010b230 0x7f51b01a19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:33:17.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.617+0000 7f51afe00700 1 --2- 172.21.15.32:0/2283957061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51b010a8e0 0x7f51b01a1420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f51a000d950 tx=0x7f51a000dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:17.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.618+0000 7f51acc00700 1 -- 172.21.15.32:0/2283957061 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51a0016070 con 0x7f51b010a8e0 2024-04-30T08:33:17.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.618+0000 7f51b6a00700 1 -- 172.21.15.32:0/2283957061 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51b01a65e0 con 0x7f51b010a8e0 2024-04-30T08:33:17.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.618+0000 7f51acc00700 1 -- 172.21.15.32:0/2283957061 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f51a0004830 con 0x7f51b010a8e0 2024-04-30T08:33:17.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.618+0000 7f51acc00700 1 -- 172.21.15.32:0/2283957061 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51a0005520 con 0x7f51b010a8e0 2024-04-30T08:33:17.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.618+0000 7f51b6a00700 1 -- 172.21.15.32:0/2283957061 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51b01adf00 con 0x7f51b010a8e0 2024-04-30T08:33:17.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.619+0000 7f51b6a00700 1 -- 172.21.15.32:0/2283957061 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f51b0063e20 con 0x7f51b010a8e0 2024-04-30T08:33:17.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.622+0000 7f51acc00700 1 -- 172.21.15.32:0/2283957061 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f51a0004220 con 0x7f51b010a8e0 2024-04-30T08:33:17.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.623+0000 7f51acc00700 1 --2- 172.21.15.32:0/2283957061 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f51940afc10 0x7f51940b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:17.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.624+0000 7f51acc00700 1 -- 172.21.15.32:0/2283957061 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f51a00df4c0 con 0x7f51b010a8e0 2024-04-30T08:33:17.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.624+0000 7f51af400700 1 --2- 172.21.15.32:0/2283957061 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f51940afc10 0x7f51940b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:17.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.624+0000 7f51acc00700 1 --2- 172.21.15.32:0/2283957061 >> [v2:172.21.15.32:6826/46066685,v1:172.21.15.32:6827/46066685] conn(0x7f51940badc0 0x7f51940bd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:17.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.624+0000 7f51acc00700 1 -- 172.21.15.32:0/2283957061 --> [v2:172.21.15.32:6826/46066685,v1:172.21.15.32:6827/46066685] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f51a001ce50 con 0x7f51940badc0 2024-04-30T08:33:17.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.624+0000 7f51acc00700 1 -- 172.21.15.32:0/2283957061 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7f51a00df870 con 0x7f51b010a8e0 2024-04-30T08:33:17.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.624+0000 7f51b4a00700 1 --2- 172.21.15.32:0/2283957061 >> [v2:172.21.15.32:6826/46066685,v1:172.21.15.32:6827/46066685] conn(0x7f51940badc0 0x7f51940bd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:17.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.624+0000 7f51af400700 1 --2- 172.21.15.32:0/2283957061 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f51940afc10 0x7f51940b2120 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f51b01099f0 tx=0x7f519c00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:17.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.625+0000 7f51b4a00700 1 --2- 172.21.15.32:0/2283957061 >> [v2:172.21.15.32:6826/46066685,v1:172.21.15.32:6827/46066685] conn(0x7f51940badc0 0x7f51940bd2b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:17.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.626+0000 7f51acc00700 1 -- 172.21.15.32:0/2283957061 <== osd.9 v2:172.21.15.32:6826/46066685 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f51a001ce50 con 0x7f51940badc0 2024-04-30T08:33:17.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.644+0000 7f51b6a00700 1 -- 172.21.15.32:0/2283957061 --> [v2:172.21.15.32:6826/46066685,v1:172.21.15.32:6827/46066685] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f51b0064220 con 0x7f51940badc0 2024-04-30T08:33:17.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.645+0000 7f51acc00700 1 -- 172.21.15.32:0/2283957061 <== osd.9 v2:172.21.15.32:6826/46066685 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f51b0064220 con 0x7f51940badc0 2024-04-30T08:33:17.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.645+0000 7f51b6a00700 1 -- 172.21.15.32:0/2283957061 >> [v2:172.21.15.32:6826/46066685,v1:172.21.15.32:6827/46066685] conn(0x7f51940badc0 msgr2=0x7f51940bd2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:17.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.645+0000 7f51b6a00700 1 --2- 172.21.15.32:0/2283957061 >> [v2:172.21.15.32:6826/46066685,v1:172.21.15.32:6827/46066685] conn(0x7f51940badc0 0x7f51940bd2b0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:17.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.646+0000 7f51b6a00700 1 -- 172.21.15.32:0/2283957061 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f51940afc10 msgr2=0x7f51940b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:17.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.646+0000 7f51b6a00700 1 --2- 172.21.15.32:0/2283957061 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f51940afc10 0x7f51940b2120 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f51b01099f0 tx=0x7f519c00b540 comp rx=0 tx=0).stop 2024-04-30T08:33:17.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.646+0000 7f51b6a00700 1 -- 172.21.15.32:0/2283957061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51b010a8e0 msgr2=0x7f51b01a1420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:17.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.646+0000 7f51b6a00700 1 --2- 172.21.15.32:0/2283957061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f51b010a8e0 0x7f51b01a1420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f51a000d950 tx=0x7f51a000dc60 comp rx=0 tx=0).stop 2024-04-30T08:33:17.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.646+0000 7f51b4a00700 1 -- 172.21.15.32:0/2283957061 reap_dead start 2024-04-30T08:33:17.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.646+0000 7f51b6a00700 1 -- 172.21.15.32:0/2283957061 shutdown_connections 2024-04-30T08:33:17.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.646+0000 7f51b6a00700 1 -- 172.21.15.32:0/2283957061 >> 172.21.15.32:0/2283957061 conn(0x7f51b0077bd0 msgr2=0x7f51b007b240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:17.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.646+0000 7f51b6a00700 1 -- 172.21.15.32:0/2283957061 shutdown_connections 2024-04-30T08:33:17.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:17.646+0000 7f51b6a00700 1 -- 172.21.15.32:0/2283957061 wait complete. 2024-04-30T08:33:18.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.553+0000 7f5c88600700 1 -- 172.21.15.32:0/3211499650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c801053d0 msgr2=0x7f5c801057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:18.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.553+0000 7f5c88600700 1 --2- 172.21.15.32:0/3211499650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c801053d0 0x7f5c801057e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5c74009990 tx=0x7f5c74009ca0 comp rx=0 tx=0).stop 2024-04-30T08:33:18.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.553+0000 7f5c88600700 1 -- 172.21.15.32:0/3211499650 shutdown_connections 2024-04-30T08:33:18.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.553+0000 7f5c88600700 1 --2- 172.21.15.32:0/3211499650 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c8010e840 0x7f5c80110d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:18.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.553+0000 7f5c88600700 1 --2- 172.21.15.32:0/3211499650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c801053d0 0x7f5c801057e0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:18.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.553+0000 7f5c88600700 1 --2- 172.21.15.32:0/3211499650 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c80105d20 0x7f5c8010e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:18.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.553+0000 7f5c88600700 1 -- 172.21.15.32:0/3211499650 >> 172.21.15.32:0/3211499650 conn(0x7f5c800fee80 msgr2=0x7f5c801012a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:18.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.554+0000 7f5c88600700 1 -- 172.21.15.32:0/3211499650 shutdown_connections 2024-04-30T08:33:18.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.554+0000 7f5c88600700 1 -- 172.21.15.32:0/3211499650 wait complete. 2024-04-30T08:33:18.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.555+0000 7f5c88600700 1 Processor -- start 2024-04-30T08:33:18.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.555+0000 7f5c88600700 1 -- start start 2024-04-30T08:33:18.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.556+0000 7f5c88600700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c801053d0 0x7f5c8019cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:18.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.556+0000 7f5c88600700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c80105d20 0x7f5c8019d530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:18.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.556+0000 7f5c88600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c8010e840 0x7f5c801a17c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:18.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.556+0000 7f5c88600700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5c801143e0 con 0x7f5c80105d20 2024-04-30T08:33:18.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.556+0000 7f5c88600700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5c80114260 con 0x7f5c801053d0 2024-04-30T08:33:18.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.556+0000 7f5c88600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5c80114560 con 0x7f5c8010e840 2024-04-30T08:33:18.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.556+0000 7f5c85c00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c801053d0 0x7f5c8019cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:18.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.556+0000 7f5c85c00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c801053d0 0x7f5c8019cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:60106/0 (socket says 172.21.15.32:60106) 2024-04-30T08:33:18.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.556+0000 7f5c85c00700 1 -- 172.21.15.32:0/1620260658 learned_addr learned my addr 172.21.15.32:0/1620260658 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:33:18.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.556+0000 7f5c86600700 1 --2- 172.21.15.32:0/1620260658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c8010e840 0x7f5c801a17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:18.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.556+0000 7f5c85200700 1 --2- 172.21.15.32:0/1620260658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c80105d20 0x7f5c8019d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:18.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.557+0000 7f5c85c00700 1 -- 172.21.15.32:0/1620260658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c8010e840 msgr2=0x7f5c801a17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:18.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.557+0000 7f5c85c00700 1 --2- 172.21.15.32:0/1620260658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c8010e840 0x7f5c801a17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:18.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.557+0000 7f5c85c00700 1 -- 172.21.15.32:0/1620260658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c80105d20 msgr2=0x7f5c8019d530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:18.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.557+0000 7f5c85c00700 1 --2- 172.21.15.32:0/1620260658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5c80105d20 0x7f5c8019d530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:18.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.557+0000 7f5c85c00700 1 -- 172.21.15.32:0/1620260658 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c801a1e60 con 0x7f5c801053d0 2024-04-30T08:33:18.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.557+0000 7f5c85c00700 1 --2- 172.21.15.32:0/1620260658 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c801053d0 0x7f5c8019cf90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5c74000c00 tx=0x7f5c740057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:18.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.557+0000 7f5c7e000700 1 -- 172.21.15.32:0/1620260658 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c74027070 con 0x7f5c801053d0 2024-04-30T08:33:18.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.558+0000 7f5c7e000700 1 -- 172.21.15.32:0/1620260658 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5c74022470 con 0x7f5c801053d0 2024-04-30T08:33:18.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.558+0000 7f5c7e000700 1 -- 172.21.15.32:0/1620260658 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c74017650 con 0x7f5c801053d0 2024-04-30T08:33:18.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.558+0000 7f5c88600700 1 -- 172.21.15.32:0/1620260658 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c801a20f0 con 0x7f5c801053d0 2024-04-30T08:33:18.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.558+0000 7f5c88600700 1 -- 172.21.15.32:0/1620260658 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c801a9a70 con 0x7f5c801053d0 2024-04-30T08:33:18.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.559+0000 7f5c88600700 1 -- 172.21.15.32:0/1620260658 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5c80103a50 con 0x7f5c801053d0 2024-04-30T08:33:18.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.562+0000 7f5c7e000700 1 -- 172.21.15.32:0/1620260658 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f5c7402a070 con 0x7f5c801053d0 2024-04-30T08:33:18.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.563+0000 7f5c7e000700 1 --2- 172.21.15.32:0/1620260658 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f5c680afc90 0x7f5c680b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:18.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.563+0000 7f5c7e000700 1 -- 172.21.15.32:0/1620260658 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f5c740e0100 con 0x7f5c801053d0 2024-04-30T08:33:18.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.564+0000 7f5c85200700 1 --2- 172.21.15.32:0/1620260658 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f5c680afc90 0x7f5c680b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:18.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.564+0000 7f5c7e000700 1 --2- 172.21.15.32:0/1620260658 >> [v2:172.21.15.189:6808/4245346198,v1:172.21.15.189:6809/4245346198] conn(0x7f5c680bae40 0x7f5c680bd330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:18.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.564+0000 7f5c7e000700 1 -- 172.21.15.32:0/1620260658 --> [v2:172.21.15.189:6808/4245346198,v1:172.21.15.189:6809/4245346198] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5c7402ee50 con 0x7f5c680bae40 2024-04-30T08:33:18.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.564+0000 7f5c7e000700 1 -- 172.21.15.32:0/1620260658 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7f5c740e04b0 con 0x7f5c801053d0 2024-04-30T08:33:18.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.564+0000 7f5c86600700 1 --2- 172.21.15.32:0/1620260658 >> [v2:172.21.15.189:6808/4245346198,v1:172.21.15.189:6809/4245346198] conn(0x7f5c680bae40 0x7f5c680bd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:18.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.565+0000 7f5c85200700 1 --2- 172.21.15.32:0/1620260658 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f5c680afc90 0x7f5c680b21a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5c80110750 tx=0x7f5c6c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:18.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.565+0000 7f5c86600700 1 --2- 172.21.15.32:0/1620260658 >> [v2:172.21.15.189:6808/4245346198,v1:172.21.15.189:6809/4245346198] conn(0x7f5c680bae40 0x7f5c680bd330 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:18.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.566+0000 7f5c7e000700 1 -- 172.21.15.32:0/1620260658 <== osd.5 v2:172.21.15.189:6808/4245346198 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5c7402ee50 con 0x7f5c680bae40 2024-04-30T08:33:18.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.585+0000 7f5c88600700 1 -- 172.21.15.32:0/1620260658 --> [v2:172.21.15.189:6808/4245346198,v1:172.21.15.189:6809/4245346198] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5c8006c870 con 0x7f5c680bae40 2024-04-30T08:33:18.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.586+0000 7f5c7e000700 1 -- 172.21.15.32:0/1620260658 <== osd.5 v2:172.21.15.189:6808/4245346198 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f5c8006c870 con 0x7f5c680bae40 2024-04-30T08:33:18.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.586+0000 7f5c88600700 1 -- 172.21.15.32:0/1620260658 >> [v2:172.21.15.189:6808/4245346198,v1:172.21.15.189:6809/4245346198] conn(0x7f5c680bae40 msgr2=0x7f5c680bd330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:18.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.586+0000 7f5c88600700 1 --2- 172.21.15.32:0/1620260658 >> [v2:172.21.15.189:6808/4245346198,v1:172.21.15.189:6809/4245346198] conn(0x7f5c680bae40 0x7f5c680bd330 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:18.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.587+0000 7f5c88600700 1 -- 172.21.15.32:0/1620260658 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f5c680afc90 msgr2=0x7f5c680b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:18.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.587+0000 7f5c88600700 1 --2- 172.21.15.32:0/1620260658 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f5c680afc90 0x7f5c680b21a0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5c80110750 tx=0x7f5c6c009260 comp rx=0 tx=0).stop 2024-04-30T08:33:18.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.587+0000 7f5c88600700 1 -- 172.21.15.32:0/1620260658 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c801053d0 msgr2=0x7f5c8019cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:18.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.587+0000 7f5c88600700 1 --2- 172.21.15.32:0/1620260658 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5c801053d0 0x7f5c8019cf90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5c74000c00 tx=0x7f5c740057c0 comp rx=0 tx=0).stop 2024-04-30T08:33:18.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.587+0000 7f5c86600700 1 -- 172.21.15.32:0/1620260658 reap_dead start 2024-04-30T08:33:18.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.587+0000 7f5c88600700 1 -- 172.21.15.32:0/1620260658 shutdown_connections 2024-04-30T08:33:18.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.587+0000 7f5c88600700 1 -- 172.21.15.32:0/1620260658 >> 172.21.15.32:0/1620260658 conn(0x7f5c800fee80 msgr2=0x7f5c80102ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:18.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.587+0000 7f5c88600700 1 -- 172.21.15.32:0/1620260658 shutdown_connections 2024-04-30T08:33:18.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:18.587+0000 7f5c88600700 1 -- 172.21.15.32:0/1620260658 wait complete. 2024-04-30T08:33:19.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.780+0000 7f3459400700 1 -- 172.21.15.32:0/3893605159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34541064d0 msgr2=0x7f34541068e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:19.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.780+0000 7f3459400700 1 --2- 172.21.15.32:0/3893605159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34541064d0 0x7f34541068e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f34400098f0 tx=0x7f3440009c00 comp rx=0 tx=0).stop 2024-04-30T08:33:19.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.780+0000 7f3459400700 1 -- 172.21.15.32:0/3893605159 shutdown_connections 2024-04-30T08:33:19.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.780+0000 7f3459400700 1 --2- 172.21.15.32:0/3893605159 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3454106e20 0x7f34541132b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:19.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.780+0000 7f3459400700 1 --2- 172.21.15.32:0/3893605159 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f345410c510 0x7f345410c920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:19.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.780+0000 7f3459400700 1 --2- 172.21.15.32:0/3893605159 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34541064d0 0x7f34541068e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:19.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.780+0000 7f3459400700 1 -- 172.21.15.32:0/3893605159 >> 172.21.15.32:0/3893605159 conn(0x7f3454101fd0 msgr2=0x7f3454104420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:19.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.780+0000 7f3459400700 1 -- 172.21.15.32:0/3893605159 shutdown_connections 2024-04-30T08:33:19.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.780+0000 7f3459400700 1 -- 172.21.15.32:0/3893605159 wait complete. 2024-04-30T08:33:19.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.781+0000 7f3459400700 1 Processor -- start 2024-04-30T08:33:19.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.781+0000 7f3459400700 1 -- start start 2024-04-30T08:33:19.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.782+0000 7f3459400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34541064d0 0x7f345419d010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:19.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.782+0000 7f3459400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3454106e20 0x7f345419d5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:19.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.782+0000 7f3459400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f345410c510 0x7f34541a1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:19.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.782+0000 7f3459400700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3454116640 con 0x7f34541064d0 2024-04-30T08:33:19.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.782+0000 7f3459400700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f34541164c0 con 0x7f345410c510 2024-04-30T08:33:19.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.782+0000 7f3459400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f34541167c0 con 0x7f3454106e20 2024-04-30T08:33:19.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.782+0000 7f3452800700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34541064d0 0x7f345419d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:19.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.782+0000 7f3451e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3454106e20 0x7f345419d5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:19.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.782+0000 7f3453200700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f345410c510 0x7f34541a1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:19.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.782+0000 7f3452800700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34541064d0 0x7f345419d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51362/0 (socket says 172.21.15.32:51362) 2024-04-30T08:33:19.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.782+0000 7f3452800700 1 -- 172.21.15.32:0/1482255478 learned_addr learned my addr 172.21.15.32:0/1482255478 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:33:19.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.783+0000 7f3452800700 1 -- 172.21.15.32:0/1482255478 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3454106e20 msgr2=0x7f345419d5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:19.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.783+0000 7f3452800700 1 --2- 172.21.15.32:0/1482255478 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3454106e20 0x7f345419d5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:19.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.783+0000 7f3452800700 1 -- 172.21.15.32:0/1482255478 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f345410c510 msgr2=0x7f34541a1840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:19.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.783+0000 7f3452800700 1 --2- 172.21.15.32:0/1482255478 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f345410c510 0x7f34541a1840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:19.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.783+0000 7f3452800700 1 -- 172.21.15.32:0/1482255478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34541a1ee0 con 0x7f34541064d0 2024-04-30T08:33:19.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.783+0000 7f3453200700 1 --2- 172.21.15.32:0/1482255478 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f345410c510 0x7f34541a1840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:33:19.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.783+0000 7f3452800700 1 --2- 172.21.15.32:0/1482255478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34541064d0 0x7f345419d010 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f344400ec70 tx=0x7f344400ef80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:19.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.783+0000 7f344b400700 1 -- 172.21.15.32:0/1482255478 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f344400cc80 con 0x7f34541064d0 2024-04-30T08:33:19.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.783+0000 7f344b400700 1 -- 172.21.15.32:0/1482255478 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f344400ce20 con 0x7f34541064d0 2024-04-30T08:33:19.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.783+0000 7f3459400700 1 -- 172.21.15.32:0/1482255478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34541a21d0 con 0x7f34541064d0 2024-04-30T08:33:19.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.783+0000 7f344b400700 1 -- 172.21.15.32:0/1482255478 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3444010430 con 0x7f34541064d0 2024-04-30T08:33:19.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.784+0000 7f3459400700 1 -- 172.21.15.32:0/1482255478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34541a9af0 con 0x7f34541064d0 2024-04-30T08:33:19.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.785+0000 7f3459400700 1 -- 172.21.15.32:0/1482255478 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3454063ea0 con 0x7f34541064d0 2024-04-30T08:33:19.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.787+0000 7f344b400700 1 -- 172.21.15.32:0/1482255478 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f3444003680 con 0x7f34541064d0 2024-04-30T08:33:19.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.789+0000 7f344b400700 1 --2- 172.21.15.32:0/1482255478 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f34380afc90 0x7f34380b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:19.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.789+0000 7f344b400700 1 -- 172.21.15.32:0/1482255478 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f34440df5f0 con 0x7f34541064d0 2024-04-30T08:33:19.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.789+0000 7f3451e00700 1 --2- 172.21.15.32:0/1482255478 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f34380afc90 0x7f34380b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:19.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.789+0000 7f344b400700 1 --2- 172.21.15.32:0/1482255478 >> [v2:172.21.15.32:6818/1910389190,v1:172.21.15.32:6819/1910389190] conn(0x7f34380bae40 0x7f34380bd330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:19.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.789+0000 7f344b400700 1 -- 172.21.15.32:0/1482255478 --> [v2:172.21.15.32:6818/1910389190,v1:172.21.15.32:6819/1910389190] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f344400be50 con 0x7f34380bae40 2024-04-30T08:33:19.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.789+0000 7f344b400700 1 -- 172.21.15.32:0/1482255478 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7f34440df9a0 con 0x7f34541064d0 2024-04-30T08:33:19.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.790+0000 7f3453200700 1 --2- 172.21.15.32:0/1482255478 >> [v2:172.21.15.32:6818/1910389190,v1:172.21.15.32:6819/1910389190] conn(0x7f34380bae40 0x7f34380bd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:19.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.790+0000 7f3451e00700 1 --2- 172.21.15.32:0/1482255478 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f34380afc90 0x7f34380b21a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f344000c010 tx=0x7f3440019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:19.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.790+0000 7f3453200700 1 --2- 172.21.15.32:0/1482255478 >> [v2:172.21.15.32:6818/1910389190,v1:172.21.15.32:6819/1910389190] conn(0x7f34380bae40 0x7f34380bd330 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:19.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.791+0000 7f344b400700 1 -- 172.21.15.32:0/1482255478 <== osd.6 v2:172.21.15.32:6818/1910389190 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f344400be50 con 0x7f34380bae40 2024-04-30T08:33:19.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.810+0000 7f3459400700 1 -- 172.21.15.32:0/1482255478 --> [v2:172.21.15.32:6818/1910389190,v1:172.21.15.32:6819/1910389190] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3454064320 con 0x7f34380bae40 2024-04-30T08:33:19.104 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:18 smithi032 ceph-mon[28331]: pgmap v367: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-30T08:33:19.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.810+0000 7f344b400700 1 -- 172.21.15.32:0/1482255478 <== osd.6 v2:172.21.15.32:6818/1910389190 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f3454064320 con 0x7f34380bae40 2024-04-30T08:33:19.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.811+0000 7f3459400700 1 -- 172.21.15.32:0/1482255478 >> [v2:172.21.15.32:6818/1910389190,v1:172.21.15.32:6819/1910389190] conn(0x7f34380bae40 msgr2=0x7f34380bd330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:19.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.811+0000 7f3459400700 1 --2- 172.21.15.32:0/1482255478 >> [v2:172.21.15.32:6818/1910389190,v1:172.21.15.32:6819/1910389190] conn(0x7f34380bae40 0x7f34380bd330 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:19.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.811+0000 7f3459400700 1 -- 172.21.15.32:0/1482255478 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f34380afc90 msgr2=0x7f34380b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:19.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.811+0000 7f3459400700 1 --2- 172.21.15.32:0/1482255478 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f34380afc90 0x7f34380b21a0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f344000c010 tx=0x7f3440019040 comp rx=0 tx=0).stop 2024-04-30T08:33:19.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.811+0000 7f3459400700 1 -- 172.21.15.32:0/1482255478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34541064d0 msgr2=0x7f345419d010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:19.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.811+0000 7f3459400700 1 --2- 172.21.15.32:0/1482255478 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f34541064d0 0x7f345419d010 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f344400ec70 tx=0x7f344400ef80 comp rx=0 tx=0).stop 2024-04-30T08:33:19.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.811+0000 7f3453200700 1 -- 172.21.15.32:0/1482255478 reap_dead start 2024-04-30T08:33:19.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.811+0000 7f3459400700 1 -- 172.21.15.32:0/1482255478 shutdown_connections 2024-04-30T08:33:19.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.811+0000 7f3459400700 1 -- 172.21.15.32:0/1482255478 >> 172.21.15.32:0/1482255478 conn(0x7f3454101fd0 msgr2=0x7f34541047e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:19.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.811+0000 7f3459400700 1 -- 172.21.15.32:0/1482255478 shutdown_connections 2024-04-30T08:33:19.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:16.811+0000 7f3459400700 1 -- 172.21.15.32:0/1482255478 wait complete. 2024-04-30T08:33:19.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:18 smithi071 ceph-mon[33385]: pgmap v367: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-30T08:33:19.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:18 smithi189 ceph-mon[33038]: pgmap v367: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 8 op/s 2024-04-30T08:33:20.135 INFO:teuthology.orchestra.run.smithi032.stdout:55834574937 2024-04-30T08:33:20.136 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-30T08:33:20.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.904+0000 7f8f38a00700 1 -- 172.21.15.32:0/996186859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f340765d0 msgr2=0x7f8f341103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:20.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.904+0000 7f8f38a00700 1 --2- 172.21.15.32:0/996186859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f340765d0 0x7f8f341103c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8f20009990 tx=0x7f8f20009ca0 comp rx=0 tx=0).stop 2024-04-30T08:33:20.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.904+0000 7f8f38a00700 1 -- 172.21.15.32:0/996186859 shutdown_connections 2024-04-30T08:33:20.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.904+0000 7f8f38a00700 1 --2- 172.21.15.32:0/996186859 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f34110900 0x7f8f34112d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:20.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.905+0000 7f8f38a00700 1 --2- 172.21.15.32:0/996186859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f34075c80 0x7f8f34076090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:20.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.905+0000 7f8f38a00700 1 --2- 172.21.15.32:0/996186859 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f340765d0 0x7f8f341103c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:20.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.905+0000 7f8f38a00700 1 -- 172.21.15.32:0/996186859 >> 172.21.15.32:0/996186859 conn(0x7f8f34079c60 msgr2=0x7f8f3407a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:20.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.905+0000 7f8f38a00700 1 -- 172.21.15.32:0/996186859 shutdown_connections 2024-04-30T08:33:20.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.905+0000 7f8f38a00700 1 -- 172.21.15.32:0/996186859 wait complete. 2024-04-30T08:33:20.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.906+0000 7f8f38a00700 1 Processor -- start 2024-04-30T08:33:20.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.906+0000 7f8f38a00700 1 -- start start 2024-04-30T08:33:20.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.907+0000 7f8f38a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f34075c80 0x7f8f3419cdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:20.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.907+0000 7f8f38a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f340765d0 0x7f8f3419d390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:20.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.907+0000 7f8f38a00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f34110900 0x7f8f341a1620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:20.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.907+0000 7f8f38a00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8f34116400 con 0x7f8f340765d0 2024-04-30T08:33:20.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.907+0000 7f8f38a00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8f34116280 con 0x7f8f34110900 2024-04-30T08:33:20.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.907+0000 7f8f38a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8f34116580 con 0x7f8f34075c80 2024-04-30T08:33:20.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.907+0000 7f8f33400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f34075c80 0x7f8f3419cdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:20.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.907+0000 7f8f32a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f340765d0 0x7f8f3419d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:20.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.907+0000 7f8f33400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f34075c80 0x7f8f3419cdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:46834/0 (socket says 172.21.15.32:46834) 2024-04-30T08:33:20.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.907+0000 7f8f32a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f340765d0 0x7f8f3419d390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52016/0 (socket says 172.21.15.32:52016) 2024-04-30T08:33:20.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.907+0000 7f8f32a00700 1 -- 172.21.15.32:0/1681751409 learned_addr learned my addr 172.21.15.32:0/1681751409 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:33:20.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.907+0000 7f8f33e00700 1 --2- 172.21.15.32:0/1681751409 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f34110900 0x7f8f341a1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:20.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.907+0000 7f8f33400700 1 -- 172.21.15.32:0/1681751409 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f34110900 msgr2=0x7f8f341a1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:20.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.908+0000 7f8f33400700 1 --2- 172.21.15.32:0/1681751409 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f34110900 0x7f8f341a1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:20.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.908+0000 7f8f33400700 1 -- 172.21.15.32:0/1681751409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f340765d0 msgr2=0x7f8f3419d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:20.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.908+0000 7f8f33400700 1 --2- 172.21.15.32:0/1681751409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f340765d0 0x7f8f3419d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:20.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.908+0000 7f8f33400700 1 -- 172.21.15.32:0/1681751409 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f341a1cc0 con 0x7f8f34075c80 2024-04-30T08:33:20.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.908+0000 7f8f32a00700 1 --2- 172.21.15.32:0/1681751409 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f340765d0 0x7f8f3419d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:33:20.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.908+0000 7f8f33e00700 1 --2- 172.21.15.32:0/1681751409 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f34110900 0x7f8f341a1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:33:20.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.908+0000 7f8f33400700 1 --2- 172.21.15.32:0/1681751409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f34075c80 0x7f8f3419cdf0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8f2400e9f0 tx=0x7f8f2400ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:20.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.908+0000 7f8f2be00700 1 -- 172.21.15.32:0/1681751409 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f2400c9c0 con 0x7f8f34075c80 2024-04-30T08:33:20.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.908+0000 7f8f2be00700 1 -- 172.21.15.32:0/1681751409 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f2400cb60 con 0x7f8f34075c80 2024-04-30T08:33:20.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.908+0000 7f8f38a00700 1 -- 172.21.15.32:0/1681751409 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f341a1fb0 con 0x7f8f34075c80 2024-04-30T08:33:20.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.908+0000 7f8f2be00700 1 -- 172.21.15.32:0/1681751409 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f24018680 con 0x7f8f34075c80 2024-04-30T08:33:20.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.909+0000 7f8f38a00700 1 -- 172.21.15.32:0/1681751409 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f341a98d0 con 0x7f8f34075c80 2024-04-30T08:33:20.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.909+0000 7f8f38a00700 1 -- 172.21.15.32:0/1681751409 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8f34063ea0 con 0x7f8f34075c80 2024-04-30T08:33:20.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.912+0000 7f8f2be00700 1 -- 172.21.15.32:0/1681751409 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f8f24010430 con 0x7f8f34075c80 2024-04-30T08:33:20.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.914+0000 7f8f2be00700 1 --2- 172.21.15.32:0/1681751409 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8f180afc10 0x7f8f180b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:20.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.914+0000 7f8f2be00700 1 -- 172.21.15.32:0/1681751409 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f8f240df8d0 con 0x7f8f34075c80 2024-04-30T08:33:20.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.914+0000 7f8f32a00700 1 --2- 172.21.15.32:0/1681751409 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8f180afc10 0x7f8f180b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:20.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.914+0000 7f8f2be00700 1 --2- 172.21.15.32:0/1681751409 >> [v2:172.21.15.32:6810/1384702112,v1:172.21.15.32:6811/1384702112] conn(0x7f8f180badc0 0x7f8f180bd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:20.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.914+0000 7f8f2be00700 1 -- 172.21.15.32:0/1681751409 --> [v2:172.21.15.32:6810/1384702112,v1:172.21.15.32:6811/1384702112] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8f180bd980 con 0x7f8f180badc0 2024-04-30T08:33:20.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.914+0000 7f8f2be00700 1 -- 172.21.15.32:0/1681751409 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7f8f240dfc80 con 0x7f8f34075c80 2024-04-30T08:33:20.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.915+0000 7f8f33e00700 1 --2- 172.21.15.32:0/1681751409 >> [v2:172.21.15.32:6810/1384702112,v1:172.21.15.32:6811/1384702112] conn(0x7f8f180badc0 0x7f8f180bd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:20.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.915+0000 7f8f32a00700 1 --2- 172.21.15.32:0/1681751409 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8f180afc10 0x7f8f180b2120 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8f3407a6c0 tx=0x7f8f20009590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:20.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.915+0000 7f8f33e00700 1 --2- 172.21.15.32:0/1681751409 >> [v2:172.21.15.32:6810/1384702112,v1:172.21.15.32:6811/1384702112] conn(0x7f8f180badc0 0x7f8f180bd2b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:20.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.917+0000 7f8f2be00700 1 -- 172.21.15.32:0/1681751409 <== osd.3 v2:172.21.15.32:6810/1384702112 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8f180bd980 con 0x7f8f180badc0 2024-04-30T08:33:20.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.935+0000 7f8f38a00700 1 -- 172.21.15.32:0/1681751409 --> [v2:172.21.15.32:6810/1384702112,v1:172.21.15.32:6811/1384702112] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8f34064320 con 0x7f8f180badc0 2024-04-30T08:33:20.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.936+0000 7f8f2be00700 1 -- 172.21.15.32:0/1681751409 <== osd.3 v2:172.21.15.32:6810/1384702112 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f8f34064320 con 0x7f8f180badc0 2024-04-30T08:33:20.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.937+0000 7f8f38a00700 1 -- 172.21.15.32:0/1681751409 >> [v2:172.21.15.32:6810/1384702112,v1:172.21.15.32:6811/1384702112] conn(0x7f8f180badc0 msgr2=0x7f8f180bd2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:20.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.937+0000 7f8f38a00700 1 --2- 172.21.15.32:0/1681751409 >> [v2:172.21.15.32:6810/1384702112,v1:172.21.15.32:6811/1384702112] conn(0x7f8f180badc0 0x7f8f180bd2b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:20.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.937+0000 7f8f38a00700 1 -- 172.21.15.32:0/1681751409 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8f180afc10 msgr2=0x7f8f180b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:20.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.937+0000 7f8f38a00700 1 --2- 172.21.15.32:0/1681751409 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8f180afc10 0x7f8f180b2120 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8f3407a6c0 tx=0x7f8f20009590 comp rx=0 tx=0).stop 2024-04-30T08:33:20.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.937+0000 7f8f38a00700 1 -- 172.21.15.32:0/1681751409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f34075c80 msgr2=0x7f8f3419cdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:20.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.937+0000 7f8f38a00700 1 --2- 172.21.15.32:0/1681751409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f34075c80 0x7f8f3419cdf0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f8f2400e9f0 tx=0x7f8f2400ed00 comp rx=0 tx=0).stop 2024-04-30T08:33:20.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.937+0000 7f8f33e00700 1 -- 172.21.15.32:0/1681751409 reap_dead start 2024-04-30T08:33:20.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.937+0000 7f8f38a00700 1 -- 172.21.15.32:0/1681751409 shutdown_connections 2024-04-30T08:33:20.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.937+0000 7f8f38a00700 1 -- 172.21.15.32:0/1681751409 >> 172.21.15.32:0/1681751409 conn(0x7f8f34079c60 msgr2=0x7f8f34100ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:20.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.937+0000 7f8f38a00700 1 -- 172.21.15.32:0/1681751409 shutdown_connections 2024-04-30T08:33:20.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:19.937+0000 7f8f38a00700 1 -- 172.21.15.32:0/1681751409 wait complete. 2024-04-30T08:33:21.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:20 smithi032 ceph-mon[28331]: pgmap v368: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-30T08:33:21.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:20 smithi071 ceph-mon[33385]: pgmap v368: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-30T08:33:21.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:20 smithi189 ceph-mon[33038]: pgmap v368: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s 2024-04-30T08:33:21.488 INFO:teuthology.orchestra.run.smithi032.stdout:257698037803 2024-04-30T08:33:21.489 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-30T08:33:21.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.806+0000 7f6ec1200700 1 -- 172.21.15.32:0/2455451838 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6ebc108e70 msgr2=0x7f6ebc109280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:21.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.806+0000 7f6ec1200700 1 --2- 172.21.15.32:0/2455451838 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6ebc108e70 0x7f6ebc109280 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f6ea80098f0 tx=0x7f6ea8009c00 comp rx=0 tx=0).stop 2024-04-30T08:33:21.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.807+0000 7f6ec1200700 1 -- 172.21.15.32:0/2455451838 shutdown_connections 2024-04-30T08:33:21.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.807+0000 7f6ec1200700 1 --2- 172.21.15.32:0/2455451838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ebc1097c0 0x7f6ebc111120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:21.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.807+0000 7f6ec1200700 1 --2- 172.21.15.32:0/2455451838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ebc104540 0x7f6ebc104950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:21.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.807+0000 7f6ec1200700 1 --2- 172.21.15.32:0/2455451838 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6ebc108e70 0x7f6ebc109280 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:21.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.807+0000 7f6ec1200700 1 -- 172.21.15.32:0/2455451838 >> 172.21.15.32:0/2455451838 conn(0x7f6ebc0fff90 msgr2=0x7f6ebc1023b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:21.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.807+0000 7f6ec1200700 1 -- 172.21.15.32:0/2455451838 shutdown_connections 2024-04-30T08:33:21.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.807+0000 7f6ec1200700 1 -- 172.21.15.32:0/2455451838 wait complete. 2024-04-30T08:33:21.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.808+0000 7f6ec1200700 1 Processor -- start 2024-04-30T08:33:21.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.809+0000 7f6ec1200700 1 -- start start 2024-04-30T08:33:21.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.809+0000 7f6ec1200700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6ebc104540 0x7f6ebc19d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:21.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.809+0000 7f6ec1200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ebc108e70 0x7f6ebc19d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:21.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.809+0000 7f6ec1200700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ebc1097c0 0x7f6ebc1a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:21.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.809+0000 7f6ec1200700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6ebc114440 con 0x7f6ebc1097c0 2024-04-30T08:33:21.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.809+0000 7f6ec1200700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6ebc1142c0 con 0x7f6ebc104540 2024-04-30T08:33:21.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.809+0000 7f6ec1200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6ebc1145c0 con 0x7f6ebc108e70 2024-04-30T08:33:21.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.810+0000 7f6eb9e00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6ebc104540 0x7f6ebc19d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:21.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.810+0000 7f6eb9400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ebc108e70 0x7f6ebc19d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:21.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.810+0000 7f6eb9e00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6ebc104540 0x7f6ebc19d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:60140/0 (socket says 172.21.15.32:60140) 2024-04-30T08:33:21.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.810+0000 7f6eb9e00700 1 -- 172.21.15.32:0/1870784586 learned_addr learned my addr 172.21.15.32:0/1870784586 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:33:21.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.810+0000 7f6eba800700 1 --2- 172.21.15.32:0/1870784586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ebc1097c0 0x7f6ebc1a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:21.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.810+0000 7f6eb9400700 1 -- 172.21.15.32:0/1870784586 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6ebc104540 msgr2=0x7f6ebc19d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:21.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.810+0000 7f6eb9400700 1 --2- 172.21.15.32:0/1870784586 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6ebc104540 0x7f6ebc19d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:21.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.810+0000 7f6eb9400700 1 -- 172.21.15.32:0/1870784586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ebc1097c0 msgr2=0x7f6ebc1a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:21.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.810+0000 7f6eb9400700 1 --2- 172.21.15.32:0/1870784586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ebc1097c0 0x7f6ebc1a1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:21.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.810+0000 7f6eb9400700 1 -- 172.21.15.32:0/1870784586 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ebc1a1fc0 con 0x7f6ebc108e70 2024-04-30T08:33:21.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.810+0000 7f6eba800700 1 --2- 172.21.15.32:0/1870784586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6ebc1097c0 0x7f6ebc1a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:33:21.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.811+0000 7f6eb9400700 1 --2- 172.21.15.32:0/1870784586 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ebc108e70 0x7f6ebc19d690 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6ea8000c00 tx=0x7f6ea800fa20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:21.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.811+0000 7f6eb2a00700 1 -- 172.21.15.32:0/1870784586 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea8005140 con 0x7f6ebc108e70 2024-04-30T08:33:21.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.811+0000 7f6eb2a00700 1 -- 172.21.15.32:0/1870784586 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6ea800b560 con 0x7f6ebc108e70 2024-04-30T08:33:21.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.811+0000 7f6eb2a00700 1 -- 172.21.15.32:0/1870784586 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ea8003a40 con 0x7f6ebc108e70 2024-04-30T08:33:21.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.811+0000 7f6ec1200700 1 -- 172.21.15.32:0/1870784586 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ebc1a2250 con 0x7f6ebc108e70 2024-04-30T08:33:21.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.811+0000 7f6ec1200700 1 -- 172.21.15.32:0/1870784586 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ebc1a26b0 con 0x7f6ebc108e70 2024-04-30T08:33:21.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.812+0000 7f6ea7e00700 1 -- 172.21.15.32:0/1870784586 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6e94001030 con 0x7f6ebc108e70 2024-04-30T08:33:21.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.816+0000 7f6eb2a00700 1 -- 172.21.15.32:0/1870784586 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f6ea800b9e0 con 0x7f6ebc108e70 2024-04-30T08:33:21.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.817+0000 7f6eb2a00700 1 --2- 172.21.15.32:0/1870784586 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f6ea00afc10 0x7f6ea00b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:21.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.817+0000 7f6eb2a00700 1 -- 172.21.15.32:0/1870784586 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f6ea80e05d0 con 0x7f6ebc108e70 2024-04-30T08:33:21.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.817+0000 7f6eb2a00700 1 --2- 172.21.15.32:0/1870784586 >> [v2:172.21.15.71:6816/692607380,v1:172.21.15.71:6817/692607380] conn(0x7f6ea00badb0 0x7f6ea00bd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:21.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.817+0000 7f6eb9e00700 1 --2- 172.21.15.32:0/1870784586 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f6ea00afc10 0x7f6ea00b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:21.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.817+0000 7f6eb2a00700 1 -- 172.21.15.32:0/1870784586 --> [v2:172.21.15.71:6816/692607380,v1:172.21.15.71:6817/692607380] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6ea8004e50 con 0x7f6ea00badb0 2024-04-30T08:33:21.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.817+0000 7f6eb2a00700 1 -- 172.21.15.32:0/1870784586 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7f6ea80e0980 con 0x7f6ebc108e70 2024-04-30T08:33:21.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.818+0000 7f6eba800700 1 --2- 172.21.15.32:0/1870784586 >> [v2:172.21.15.71:6816/692607380,v1:172.21.15.71:6817/692607380] conn(0x7f6ea00badb0 0x7f6ea00bd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:21.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.818+0000 7f6eb9e00700 1 --2- 172.21.15.32:0/1870784586 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f6ea00afc10 0x7f6ea00b2120 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6eac009d30 tx=0x7f6eac009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:21.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.819+0000 7f6eba800700 1 --2- 172.21.15.32:0/1870784586 >> [v2:172.21.15.71:6816/692607380,v1:172.21.15.71:6817/692607380] conn(0x7f6ea00badb0 0x7f6ea00bd2a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:21.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.820+0000 7f6eb2a00700 1 -- 172.21.15.32:0/1870784586 <== osd.7 v2:172.21.15.71:6816/692607380 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6ea8004e50 con 0x7f6ea00badb0 2024-04-30T08:33:21.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.839+0000 7f6ea7e00700 1 -- 172.21.15.32:0/1870784586 --> [v2:172.21.15.71:6816/692607380,v1:172.21.15.71:6817/692607380] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6e94002e60 con 0x7f6ea00badb0 2024-04-30T08:33:21.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.839+0000 7f6eb2a00700 1 -- 172.21.15.32:0/1870784586 <== osd.7 v2:172.21.15.71:6816/692607380 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f6e94002e60 con 0x7f6ea00badb0 2024-04-30T08:33:21.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.840+0000 7f6ea7e00700 1 -- 172.21.15.32:0/1870784586 >> [v2:172.21.15.71:6816/692607380,v1:172.21.15.71:6817/692607380] conn(0x7f6ea00badb0 msgr2=0x7f6ea00bd2a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:21.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.840+0000 7f6ea7e00700 1 --2- 172.21.15.32:0/1870784586 >> [v2:172.21.15.71:6816/692607380,v1:172.21.15.71:6817/692607380] conn(0x7f6ea00badb0 0x7f6ea00bd2a0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:21.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.840+0000 7f6ea7e00700 1 -- 172.21.15.32:0/1870784586 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f6ea00afc10 msgr2=0x7f6ea00b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:21.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.840+0000 7f6ea7e00700 1 --2- 172.21.15.32:0/1870784586 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f6ea00afc10 0x7f6ea00b2120 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6eac009d30 tx=0x7f6eac009310 comp rx=0 tx=0).stop 2024-04-30T08:33:21.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.840+0000 7f6ea7e00700 1 -- 172.21.15.32:0/1870784586 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ebc108e70 msgr2=0x7f6ebc19d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:21.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.840+0000 7f6ea7e00700 1 --2- 172.21.15.32:0/1870784586 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6ebc108e70 0x7f6ebc19d690 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f6ea8000c00 tx=0x7f6ea800fa20 comp rx=0 tx=0).stop 2024-04-30T08:33:21.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.840+0000 7f6eba800700 1 -- 172.21.15.32:0/1870784586 reap_dead start 2024-04-30T08:33:21.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.840+0000 7f6ea7e00700 1 -- 172.21.15.32:0/1870784586 shutdown_connections 2024-04-30T08:33:21.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.840+0000 7f6ea7e00700 1 -- 172.21.15.32:0/1870784586 >> 172.21.15.32:0/1870784586 conn(0x7f6ebc0fff90 msgr2=0x7f6ebc10a720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:21.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.841+0000 7f6ea7e00700 1 -- 172.21.15.32:0/1870784586 shutdown_connections 2024-04-30T08:33:21.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:21.841+0000 7f6ea7e00700 1 -- 172.21.15.32:0/1870784586 wait complete. 2024-04-30T08:33:22.231 INFO:teuthology.orchestra.run.smithi032.stdout:77309411413 2024-04-30T08:33:22.231 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-30T08:33:22.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:22 smithi032 ceph-mon[28331]: pgmap v369: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-30T08:33:23.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:22 smithi071 ceph-mon[33385]: pgmap v369: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-30T08:33:23.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:22 smithi189 ceph-mon[33038]: pgmap v369: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-30T08:33:23.941 INFO:teuthology.orchestra.run.smithi032.stdout:180388626492 2024-04-30T08:33:23.942 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-30T08:33:24.519 INFO:teuthology.orchestra.run.smithi032.stdout:107374182480 2024-04-30T08:33:24.519 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-30T08:33:24.614 INFO:teuthology.orchestra.run.smithi032.stdout:206158430264 2024-04-30T08:33:24.614 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-30T08:33:24.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.385+0000 7f10a2600700 1 -- 172.21.15.32:0/2529640221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f109c078d50 msgr2=0x7f109c079160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:24.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.385+0000 7f10a2600700 1 --2- 172.21.15.32:0/2529640221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f109c078d50 0x7f109c079160 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f108c009990 tx=0x7f108c009ca0 comp rx=0 tx=0).stop 2024-04-30T08:33:24.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.385+0000 7f10a2600700 1 -- 172.21.15.32:0/2529640221 shutdown_connections 2024-04-30T08:33:24.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.385+0000 7f10a2600700 1 --2- 172.21.15.32:0/2529640221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f109c115150 0x7f109c117570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:24.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.385+0000 7f10a2600700 1 --2- 172.21.15.32:0/2529640221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f109c078d50 0x7f109c079160 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:24.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.385+0000 7f10a2600700 1 --2- 172.21.15.32:0/2529640221 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f109c0796a0 0x7f109c114c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:24.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.385+0000 7f10a2600700 1 -- 172.21.15.32:0/2529640221 >> 172.21.15.32:0/2529640221 conn(0x7f109c101fd0 msgr2=0x7f109c104420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:24.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.385+0000 7f10a2600700 1 -- 172.21.15.32:0/2529640221 shutdown_connections 2024-04-30T08:33:24.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.385+0000 7f10a2600700 1 -- 172.21.15.32:0/2529640221 wait complete. 2024-04-30T08:33:24.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.386+0000 7f10a2600700 1 Processor -- start 2024-04-30T08:33:24.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.387+0000 7f10a2600700 1 -- start start 2024-04-30T08:33:24.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.387+0000 7f10a2600700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f109c078d50 0x7f109c1a13b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:24.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.387+0000 7f10a2600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f109c0796a0 0x7f109c1a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:24.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.387+0000 7f10a2600700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f109c115150 0x7f109c1a5be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:24.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.387+0000 7f10a2600700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f109c11aa40 con 0x7f109c115150 2024-04-30T08:33:24.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.387+0000 7f10a2600700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f109c11a8c0 con 0x7f109c078d50 2024-04-30T08:33:24.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.387+0000 7f10a2600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f109c11abc0 con 0x7f109c0796a0 2024-04-30T08:33:24.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.388+0000 7f109be00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f109c115150 0x7f109c1a5be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:24.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.388+0000 7f109b400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f109c078d50 0x7f109c1a13b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:24.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.388+0000 7f109be00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f109c115150 0x7f109c1a5be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:52046/0 (socket says 172.21.15.32:52046) 2024-04-30T08:33:24.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.388+0000 7f109aa00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f109c0796a0 0x7f109c1a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:24.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.388+0000 7f109be00700 1 -- 172.21.15.32:0/1391022981 learned_addr learned my addr 172.21.15.32:0/1391022981 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:33:24.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.388+0000 7f109be00700 1 -- 172.21.15.32:0/1391022981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f109c0796a0 msgr2=0x7f109c1a1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:24.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.388+0000 7f109be00700 1 --2- 172.21.15.32:0/1391022981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f109c0796a0 0x7f109c1a1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:24.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.388+0000 7f109be00700 1 -- 172.21.15.32:0/1391022981 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f109c078d50 msgr2=0x7f109c1a13b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:24.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.388+0000 7f109be00700 1 --2- 172.21.15.32:0/1391022981 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f109c078d50 0x7f109c1a13b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:24.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.388+0000 7f109be00700 1 -- 172.21.15.32:0/1391022981 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f109c1a6280 con 0x7f109c115150 2024-04-30T08:33:24.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.388+0000 7f109aa00700 1 --2- 172.21.15.32:0/1391022981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f109c0796a0 0x7f109c1a1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:33:24.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.388+0000 7f109be00700 1 --2- 172.21.15.32:0/1391022981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f109c115150 0x7f109c1a5be0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f109400c370 tx=0x7f109400c680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:24.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.389+0000 7f1093e00700 1 -- 172.21.15.32:0/1391022981 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10940092e0 con 0x7f109c115150 2024-04-30T08:33:24.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.389+0000 7f1093e00700 1 -- 172.21.15.32:0/1391022981 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1094003a00 con 0x7f109c115150 2024-04-30T08:33:24.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.389+0000 7f1093e00700 1 -- 172.21.15.32:0/1391022981 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10940046f0 con 0x7f109c115150 2024-04-30T08:33:24.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.389+0000 7f10a2600700 1 -- 172.21.15.32:0/1391022981 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f109c1a6510 con 0x7f109c115150 2024-04-30T08:33:24.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.389+0000 7f10a2600700 1 -- 172.21.15.32:0/1391022981 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f109c1a6a00 con 0x7f109c115150 2024-04-30T08:33:24.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.390+0000 7f10a2600700 1 -- 172.21.15.32:0/1391022981 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f109c063ea0 con 0x7f109c115150 2024-04-30T08:33:24.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.393+0000 7f1093e00700 1 -- 172.21.15.32:0/1391022981 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f109401a020 con 0x7f109c115150 2024-04-30T08:33:24.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.394+0000 7f1093e00700 1 --2- 172.21.15.32:0/1391022981 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f10800afc90 0x7f10800b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:24.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.394+0000 7f1093e00700 1 -- 172.21.15.32:0/1391022981 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f10940df400 con 0x7f109c115150 2024-04-30T08:33:24.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.395+0000 7f109b400700 1 --2- 172.21.15.32:0/1391022981 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f10800afc90 0x7f10800b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:24.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.395+0000 7f1093e00700 1 --2- 172.21.15.32:0/1391022981 >> [v2:172.21.15.189:6824/850185001,v1:172.21.15.189:6825/850185001] conn(0x7f10800bae40 0x7f10800bd330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:24.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.395+0000 7f1093e00700 1 -- 172.21.15.32:0/1391022981 --> [v2:172.21.15.189:6824/850185001,v1:172.21.15.189:6825/850185001] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1094009c20 con 0x7f10800bae40 2024-04-30T08:33:24.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.395+0000 7f1093e00700 1 -- 172.21.15.32:0/1391022981 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7f10940df7b0 con 0x7f109c115150 2024-04-30T08:33:24.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.395+0000 7f109aa00700 1 --2- 172.21.15.32:0/1391022981 >> [v2:172.21.15.189:6824/850185001,v1:172.21.15.189:6825/850185001] conn(0x7f10800bae40 0x7f10800bd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:24.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.396+0000 7f109b400700 1 --2- 172.21.15.32:0/1391022981 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f10800afc90 0x7f10800b21a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f108c006010 tx=0x7f108c00b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:24.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.396+0000 7f109aa00700 1 --2- 172.21.15.32:0/1391022981 >> [v2:172.21.15.189:6824/850185001,v1:172.21.15.189:6825/850185001] conn(0x7f10800bae40 0x7f10800bd330 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:24.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.397+0000 7f1093e00700 1 -- 172.21.15.32:0/1391022981 <== osd.11 v2:172.21.15.189:6824/850185001 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1094009c20 con 0x7f10800bae40 2024-04-30T08:33:24.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.415+0000 7f10a2600700 1 -- 172.21.15.32:0/1391022981 --> [v2:172.21.15.189:6824/850185001,v1:172.21.15.189:6825/850185001] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f109c064320 con 0x7f10800bae40 2024-04-30T08:33:24.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.416+0000 7f1093e00700 1 -- 172.21.15.32:0/1391022981 <== osd.11 v2:172.21.15.189:6824/850185001 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f109c064320 con 0x7f10800bae40 2024-04-30T08:33:24.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.416+0000 7f10a2600700 1 -- 172.21.15.32:0/1391022981 >> [v2:172.21.15.189:6824/850185001,v1:172.21.15.189:6825/850185001] conn(0x7f10800bae40 msgr2=0x7f10800bd330 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:24.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.416+0000 7f10a2600700 1 --2- 172.21.15.32:0/1391022981 >> [v2:172.21.15.189:6824/850185001,v1:172.21.15.189:6825/850185001] conn(0x7f10800bae40 0x7f10800bd330 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:24.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.416+0000 7f10a2600700 1 -- 172.21.15.32:0/1391022981 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f10800afc90 msgr2=0x7f10800b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:24.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.416+0000 7f10a2600700 1 --2- 172.21.15.32:0/1391022981 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f10800afc90 0x7f10800b21a0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f108c006010 tx=0x7f108c00b260 comp rx=0 tx=0).stop 2024-04-30T08:33:24.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.416+0000 7f10a2600700 1 -- 172.21.15.32:0/1391022981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f109c115150 msgr2=0x7f109c1a5be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:24.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.416+0000 7f10a2600700 1 --2- 172.21.15.32:0/1391022981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f109c115150 0x7f109c1a5be0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f109400c370 tx=0x7f109400c680 comp rx=0 tx=0).stop 2024-04-30T08:33:24.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.416+0000 7f109be00700 1 -- 172.21.15.32:0/1391022981 reap_dead start 2024-04-30T08:33:24.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.417+0000 7f10a2600700 1 -- 172.21.15.32:0/1391022981 shutdown_connections 2024-04-30T08:33:24.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.417+0000 7f10a2600700 1 -- 172.21.15.32:0/1391022981 >> 172.21.15.32:0/1391022981 conn(0x7f109c101fd0 msgr2=0x7f109c07bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:24.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.417+0000 7f10a2600700 1 -- 172.21.15.32:0/1391022981 shutdown_connections 2024-04-30T08:33:24.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:23.417+0000 7f10a2600700 1 -- 172.21.15.32:0/1391022981 wait complete. 2024-04-30T08:33:25.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:24 smithi032 ceph-mon[28331]: pgmap v370: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:33:25.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:24 smithi071 ceph-mon[33385]: pgmap v370: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:33:25.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:24 smithi189 ceph-mon[33038]: pgmap v370: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:33:25.479 INFO:teuthology.orchestra.run.smithi032.stdout:304942678049 2024-04-30T08:33:25.479 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-30T08:33:25.914 INFO:teuthology.orchestra.run.smithi032.stdout:154618822724 2024-04-30T08:33:25.915 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-30T08:33:26.174 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:33:26.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.256+0000 7f14bde00700 1 -- 172.21.15.32:0/1354170366 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14b81053d0 msgr2=0x7f14b81057e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:26.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.256+0000 7f14bde00700 1 --2- 172.21.15.32:0/1354170366 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14b81053d0 0x7f14b81057e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f14a8009990 tx=0x7f14a8009ca0 comp rx=0 tx=0).stop 2024-04-30T08:33:26.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.256+0000 7f14bde00700 1 -- 172.21.15.32:0/1354170366 shutdown_connections 2024-04-30T08:33:26.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.256+0000 7f14bde00700 1 --2- 172.21.15.32:0/1354170366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14b810e840 0x7f14b8110d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:26.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.256+0000 7f14bde00700 1 --2- 172.21.15.32:0/1354170366 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14b81053d0 0x7f14b81057e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:26.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.256+0000 7f14bde00700 1 --2- 172.21.15.32:0/1354170366 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14b8105d20 0x7f14b810e270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:26.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.256+0000 7f14bde00700 1 -- 172.21.15.32:0/1354170366 >> 172.21.15.32:0/1354170366 conn(0x7f14b80fee80 msgr2=0x7f14b81012a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:26.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.256+0000 7f14bde00700 1 -- 172.21.15.32:0/1354170366 shutdown_connections 2024-04-30T08:33:26.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.256+0000 7f14bde00700 1 -- 172.21.15.32:0/1354170366 wait complete. 2024-04-30T08:33:26.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.257+0000 7f14bde00700 1 Processor -- start 2024-04-30T08:33:26.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.258+0000 7f14bde00700 1 -- start start 2024-04-30T08:33:26.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.258+0000 7f14bde00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14b81053d0 0x7f14b819ad50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:26.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.258+0000 7f14bde00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14b8105d20 0x7f14b819b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:26.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.258+0000 7f14bde00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14b810e840 0x7f14b819f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:26.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.258+0000 7f14bde00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f14b81143e0 con 0x7f14b810e840 2024-04-30T08:33:26.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.258+0000 7f14bde00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f14b8114260 con 0x7f14b8105d20 2024-04-30T08:33:26.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.258+0000 7f14bde00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f14b8114560 con 0x7f14b81053d0 2024-04-30T08:33:26.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.258+0000 7f14b6a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14b81053d0 0x7f14b819ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:26.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.258+0000 7f14b6000700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14b8105d20 0x7f14b819b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:26.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.258+0000 7f14b6a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14b81053d0 0x7f14b819ad50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:46866/0 (socket says 172.21.15.32:46866) 2024-04-30T08:33:26.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.259+0000 7f14b6a00700 1 -- 172.21.15.32:0/4193586206 learned_addr learned my addr 172.21.15.32:0/4193586206 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:33:26.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.259+0000 7f14b7400700 1 --2- 172.21.15.32:0/4193586206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14b810e840 0x7f14b819f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:26.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.259+0000 7f14b6a00700 1 -- 172.21.15.32:0/4193586206 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14b8105d20 msgr2=0x7f14b819b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:26.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.259+0000 7f14b6a00700 1 --2- 172.21.15.32:0/4193586206 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14b8105d20 0x7f14b819b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:26.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.259+0000 7f14b6a00700 1 -- 172.21.15.32:0/4193586206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14b810e840 msgr2=0x7f14b819f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:26.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.259+0000 7f14b6a00700 1 --2- 172.21.15.32:0/4193586206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14b810e840 0x7f14b819f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:26.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.259+0000 7f14b6a00700 1 -- 172.21.15.32:0/4193586206 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14b819fc20 con 0x7f14b81053d0 2024-04-30T08:33:26.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.259+0000 7f14b6a00700 1 --2- 172.21.15.32:0/4193586206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14b81053d0 0x7f14b819ad50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f14a8005910 tx=0x7f14a800b670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:26.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.260+0000 7f14bca00700 1 -- 172.21.15.32:0/4193586206 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a8027070 con 0x7f14b81053d0 2024-04-30T08:33:26.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.260+0000 7f14bca00700 1 -- 172.21.15.32:0/4193586206 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14a8022470 con 0x7f14b81053d0 2024-04-30T08:33:26.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.260+0000 7f14bde00700 1 -- 172.21.15.32:0/4193586206 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14b819feb0 con 0x7f14b81053d0 2024-04-30T08:33:26.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.260+0000 7f14bca00700 1 -- 172.21.15.32:0/4193586206 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14a8017650 con 0x7f14b81053d0 2024-04-30T08:33:26.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.260+0000 7f14bde00700 1 -- 172.21.15.32:0/4193586206 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14b81a0390 con 0x7f14b81053d0 2024-04-30T08:33:26.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.261+0000 7f14bde00700 1 -- 172.21.15.32:0/4193586206 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f14b8063ea0 con 0x7f14b81053d0 2024-04-30T08:33:26.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.264+0000 7f14bca00700 1 -- 172.21.15.32:0/4193586206 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f14a802a070 con 0x7f14b81053d0 2024-04-30T08:33:26.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.266+0000 7f14bca00700 1 --2- 172.21.15.32:0/4193586206 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f149c0afc10 0x7f149c0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:26.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.267+0000 7f14bca00700 1 -- 172.21.15.32:0/4193586206 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 9092+0+0 (secure 0 0 0) 0x7f14a80e0370 con 0x7f14b81053d0 2024-04-30T08:33:26.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.267+0000 7f14bca00700 1 --2- 172.21.15.32:0/4193586206 >> [v2:172.21.15.71:6824/1886290293,v1:172.21.15.71:6825/1886290293] conn(0x7f149c0badc0 0x7f149c0bd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:26.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.267+0000 7f14bca00700 1 -- 172.21.15.32:0/4193586206 --> [v2:172.21.15.71:6824/1886290293,v1:172.21.15.71:6825/1886290293] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f149c0bd980 con 0x7f149c0badc0 2024-04-30T08:33:26.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.267+0000 7f14bca00700 1 -- 172.21.15.32:0/4193586206 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=83) ==== 24+0+0 (secure 0 0 0) 0x7f14a80e07c0 con 0x7f14b81053d0 2024-04-30T08:33:26.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.268+0000 7f14b7400700 1 --2- 172.21.15.32:0/4193586206 >> [v2:172.21.15.71:6824/1886290293,v1:172.21.15.71:6825/1886290293] conn(0x7f149c0badc0 0x7f149c0bd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:26.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.268+0000 7f14b6000700 1 --2- 172.21.15.32:0/4193586206 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f149c0afc10 0x7f149c0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:26.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.268+0000 7f14b7400700 1 --2- 172.21.15.32:0/4193586206 >> [v2:172.21.15.71:6824/1886290293,v1:172.21.15.71:6825/1886290293] conn(0x7f149c0badc0 0x7f149c0bd2b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:26.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.269+0000 7f14b6000700 1 --2- 172.21.15.32:0/4193586206 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f149c0afc10 0x7f149c0b2120 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f14b8102270 tx=0x7f14a4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:26.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.269+0000 7f14bca00700 1 -- 172.21.15.32:0/4193586206 <== osd.10 v2:172.21.15.71:6824/1886290293 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f149c0bd980 con 0x7f149c0badc0 2024-04-30T08:33:26.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.290+0000 7f14bde00700 1 -- 172.21.15.32:0/4193586206 --> [v2:172.21.15.71:6824/1886290293,v1:172.21.15.71:6825/1886290293] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f14b8064320 con 0x7f149c0badc0 2024-04-30T08:33:26.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.291+0000 7f14bca00700 1 -- 172.21.15.32:0/4193586206 <== osd.10 v2:172.21.15.71:6824/1886290293 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f14b8064320 con 0x7f149c0badc0 2024-04-30T08:33:26.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.291+0000 7f14bde00700 1 -- 172.21.15.32:0/4193586206 >> [v2:172.21.15.71:6824/1886290293,v1:172.21.15.71:6825/1886290293] conn(0x7f149c0badc0 msgr2=0x7f149c0bd2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:26.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.291+0000 7f14bde00700 1 --2- 172.21.15.32:0/4193586206 >> [v2:172.21.15.71:6824/1886290293,v1:172.21.15.71:6825/1886290293] conn(0x7f149c0badc0 0x7f149c0bd2b0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:26.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.291+0000 7f14bde00700 1 -- 172.21.15.32:0/4193586206 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f149c0afc10 msgr2=0x7f149c0b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:26.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.291+0000 7f14bde00700 1 --2- 172.21.15.32:0/4193586206 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f149c0afc10 0x7f149c0b2120 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f14b8102270 tx=0x7f14a4009260 comp rx=0 tx=0).stop 2024-04-30T08:33:26.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.291+0000 7f14bde00700 1 -- 172.21.15.32:0/4193586206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14b81053d0 msgr2=0x7f14b819ad50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:26.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.291+0000 7f14bde00700 1 --2- 172.21.15.32:0/4193586206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14b81053d0 0x7f14b819ad50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f14a8005910 tx=0x7f14a800b670 comp rx=0 tx=0).stop 2024-04-30T08:33:26.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.291+0000 7f14b7400700 1 -- 172.21.15.32:0/4193586206 reap_dead start 2024-04-30T08:33:26.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.292+0000 7f14bde00700 1 -- 172.21.15.32:0/4193586206 shutdown_connections 2024-04-30T08:33:26.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.292+0000 7f14bde00700 1 -- 172.21.15.32:0/4193586206 >> 172.21.15.32:0/4193586206 conn(0x7f14b80fee80 msgr2=0x7f14b80ff4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:26.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.292+0000 7f14bde00700 1 -- 172.21.15.32:0/4193586206 shutdown_connections 2024-04-30T08:33:26.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:26.292+0000 7f14bde00700 1 -- 172.21.15.32:0/4193586206 wait complete. 2024-04-30T08:33:26.741 INFO:teuthology.orchestra.run.smithi032.stdout:34359738463 2024-04-30T08:33:26.741 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-30T08:33:26.944 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:26 smithi032 ceph-mon[28331]: pgmap v371: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-30T08:33:27.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:26 smithi071 ceph-mon[33385]: pgmap v371: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-30T08:33:27.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:26 smithi189 ceph-mon[33038]: pgmap v371: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-30T08:33:27.474 INFO:teuthology.orchestra.run.smithi032.stdout:279172874278 2024-04-30T08:33:27.475 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-30T08:33:27.811 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:33:27.816 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:33:28.921 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:28 smithi032 ceph-mon[28331]: pgmap v372: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-04-30T08:33:29.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:28 smithi071 ceph-mon[33385]: pgmap v372: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-04-30T08:33:29.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:28 smithi189 ceph-mon[33038]: pgmap v372: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 9 op/s 2024-04-30T08:33:30.411 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:33:30.414 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:33:30.415 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:33:30.424 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:33:30.424 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:33:30.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:30 smithi032 ceph-mon[28331]: pgmap v373: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-04-30T08:33:30.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:30 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 7]}]: dispatch 2024-04-30T08:33:30.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:30 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]: dispatch 2024-04-30T08:33:30.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:30 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [3, 6, 5, 8]}]: dispatch 2024-04-30T08:33:30.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:30 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.30", "id": [1, 9]}]: dispatch 2024-04-30T08:33:31.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:30 smithi071 ceph-mon[33385]: pgmap v373: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-04-30T08:33:31.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:30 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 7]}]: dispatch 2024-04-30T08:33:31.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:30 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]: dispatch 2024-04-30T08:33:31.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:30 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [3, 6, 5, 8]}]: dispatch 2024-04-30T08:33:31.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:30 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.30", "id": [1, 9]}]: dispatch 2024-04-30T08:33:31.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.259+0000 7f3137200700 1 -- 172.21.15.32:0/3875069498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31301097a0 msgr2=0x7f3130111100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:31.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.259+0000 7f3137200700 1 --2- 172.21.15.32:0/3875069498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31301097a0 0x7f3130111100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f3128009fc0 tx=0x7f312800b330 comp rx=0 tx=0).stop 2024-04-30T08:33:31.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.259+0000 7f3137200700 1 -- 172.21.15.32:0/3875069498 shutdown_connections 2024-04-30T08:33:31.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.259+0000 7f3137200700 1 --2- 172.21.15.32:0/3875069498 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31301097a0 0x7f3130111100 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:31.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.259+0000 7f3137200700 1 --2- 172.21.15.32:0/3875069498 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3130104520 0x7f3130104930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:31.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.259+0000 7f3137200700 1 --2- 172.21.15.32:0/3875069498 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3130108e50 0x7f3130109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:31.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.259+0000 7f3137200700 1 -- 172.21.15.32:0/3875069498 >> 172.21.15.32:0/3875069498 conn(0x7f31300ffdf0 msgr2=0x7f3130102240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:31.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.259+0000 7f3137200700 1 -- 172.21.15.32:0/3875069498 shutdown_connections 2024-04-30T08:33:31.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.259+0000 7f3137200700 1 -- 172.21.15.32:0/3875069498 wait complete. 2024-04-30T08:33:31.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.260+0000 7f3137200700 1 Processor -- start 2024-04-30T08:33:31.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.261+0000 7f3137200700 1 -- start start 2024-04-30T08:33:31.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.261+0000 7f3137200700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3130104520 0x7f313019cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:31.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.261+0000 7f3137200700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3130108e50 0x7f313019d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:31.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.261+0000 7f3137200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31301097a0 0x7f31301a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:31.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.261+0000 7f3137200700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3130114420 con 0x7f3130108e50 2024-04-30T08:33:31.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.261+0000 7f3137200700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f31301142a0 con 0x7f3130104520 2024-04-30T08:33:31.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.261+0000 7f3137200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f31301145a0 con 0x7f31301097a0 2024-04-30T08:33:31.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.261+0000 7f312fe00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3130104520 0x7f313019cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:31.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.261+0000 7f3127e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3130108e50 0x7f313019d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:31.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.261+0000 7f312fe00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3130104520 0x7f313019cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:59872/0 (socket says 172.21.15.32:59872) 2024-04-30T08:33:31.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.261+0000 7f312fe00700 1 -- 172.21.15.32:0/2738467747 learned_addr learned my addr 172.21.15.32:0/2738467747 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:33:31.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.261+0000 7f3127e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3130108e50 0x7f313019d4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55284/0 (socket says 172.21.15.32:55284) 2024-04-30T08:33:31.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.261+0000 7f3135200700 1 --2- 172.21.15.32:0/2738467747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31301097a0 0x7f31301a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:31.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.262+0000 7f3135200700 1 -- 172.21.15.32:0/2738467747 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3130104520 msgr2=0x7f313019cf30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:31.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.262+0000 7f3135200700 1 --2- 172.21.15.32:0/2738467747 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3130104520 0x7f313019cf30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:31.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.262+0000 7f3135200700 1 -- 172.21.15.32:0/2738467747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3130108e50 msgr2=0x7f313019d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:31.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.262+0000 7f3135200700 1 --2- 172.21.15.32:0/2738467747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3130108e50 0x7f313019d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:31.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.262+0000 7f3135200700 1 -- 172.21.15.32:0/2738467747 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31301a1e00 con 0x7f31301097a0 2024-04-30T08:33:31.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.262+0000 7f312fe00700 1 --2- 172.21.15.32:0/2738467747 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3130104520 0x7f313019cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:33:31.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.262+0000 7f3135200700 1 --2- 172.21.15.32:0/2738467747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31301097a0 0x7f31301a1760 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f312800b6c0 tx=0x7f312800cc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:31.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.263+0000 7f312d600700 1 -- 172.21.15.32:0/2738467747 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3128021070 con 0x7f31301097a0 2024-04-30T08:33:31.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.263+0000 7f3137200700 1 -- 172.21.15.32:0/2738467747 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31301a2090 con 0x7f31301097a0 2024-04-30T08:33:31.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.263+0000 7f312d600700 1 -- 172.21.15.32:0/2738467747 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3128007ac0 con 0x7f31301097a0 2024-04-30T08:33:31.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.263+0000 7f312d600700 1 -- 172.21.15.32:0/2738467747 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3128005040 con 0x7f31301097a0 2024-04-30T08:33:31.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.263+0000 7f3137200700 1 -- 172.21.15.32:0/2738467747 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31301a9a10 con 0x7f31301097a0 2024-04-30T08:33:31.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.264+0000 7f3137200700 1 -- 172.21.15.32:0/2738467747 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3130051210 con 0x7f31301097a0 2024-04-30T08:33:31.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.267+0000 7f312d600700 1 -- 172.21.15.32:0/2738467747 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f3128024070 con 0x7f31301097a0 2024-04-30T08:33:31.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.268+0000 7f312d600700 1 --2- 172.21.15.32:0/2738467747 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f31140afc10 0x7f31140b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:31.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.269+0000 7f312d600700 1 -- 172.21.15.32:0/2738467747 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 9216+0+0 (secure 0 0 0) 0x7f312800f020 con 0x7f31301097a0 2024-04-30T08:33:31.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.269+0000 7f312fe00700 1 --2- 172.21.15.32:0/2738467747 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f31140afc10 0x7f31140b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:31.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.270+0000 7f312fe00700 1 --2- 172.21.15.32:0/2738467747 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f31140afc10 0x7f31140b2120 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3120011630 tx=0x7f3120009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:31.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.270+0000 7f312d600700 1 -- 172.21.15.32:0/2738467747 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f312802b020 con 0x7f31301097a0 2024-04-30T08:33:31.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:30 smithi189 ceph-mon[33038]: pgmap v373: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 10 op/s 2024-04-30T08:33:31.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:30 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 7]}]: dispatch 2024-04-30T08:33:31.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:30 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]: dispatch 2024-04-30T08:33:31.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:30 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [3, 6, 5, 8]}]: dispatch 2024-04-30T08:33:31.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:30 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.30", "id": [1, 9]}]: dispatch 2024-04-30T08:33:31.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.507+0000 7f3137200700 1 -- 172.21.15.32:0/2738467747 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f31301a9be0 con 0x7f31301097a0 2024-04-30T08:33:31.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.508+0000 7f312d600700 1 -- 172.21.15.32:0/2738467747 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f31280a9100 con 0x7f31301097a0 2024-04-30T08:33:31.510 INFO:teuthology.orchestra.run.smithi032.stdout:55834574941 2024-04-30T08:33:31.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.511+0000 7f3137200700 1 -- 172.21.15.32:0/2738467747 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f31140afc10 msgr2=0x7f31140b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:31.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.511+0000 7f3137200700 1 --2- 172.21.15.32:0/2738467747 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f31140afc10 0x7f31140b2120 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f3120011630 tx=0x7f3120009260 comp rx=0 tx=0).stop 2024-04-30T08:33:31.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.511+0000 7f3137200700 1 -- 172.21.15.32:0/2738467747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31301097a0 msgr2=0x7f31301a1760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:31.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.511+0000 7f3137200700 1 --2- 172.21.15.32:0/2738467747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31301097a0 0x7f31301a1760 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f312800b6c0 tx=0x7f312800cc30 comp rx=0 tx=0).stop 2024-04-30T08:33:31.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.512+0000 7f3137200700 1 -- 172.21.15.32:0/2738467747 shutdown_connections 2024-04-30T08:33:31.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.512+0000 7f3137200700 1 --2- 172.21.15.32:0/2738467747 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f31140afc10 0x7f31140b2120 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:31.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.512+0000 7f3137200700 1 --2- 172.21.15.32:0/2738467747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31301097a0 0x7f31301a1760 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:31.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.512+0000 7f3137200700 1 --2- 172.21.15.32:0/2738467747 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3130104520 0x7f313019cf30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:31.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.512+0000 7f3137200700 1 --2- 172.21.15.32:0/2738467747 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3130108e50 0x7f313019d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:31.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.512+0000 7f3137200700 1 -- 172.21.15.32:0/2738467747 >> 172.21.15.32:0/2738467747 conn(0x7f31300ffdf0 msgr2=0x7f3130110dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:31.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.512+0000 7f3137200700 1 -- 172.21.15.32:0/2738467747 shutdown_connections 2024-04-30T08:33:31.514 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:31.512+0000 7f3137200700 1 -- 172.21.15.32:0/2738467747 wait complete. 2024-04-30T08:33:32.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:31 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 7]}]': finished 2024-04-30T08:33:32.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:31 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]': finished 2024-04-30T08:33:32.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:31 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [3, 6, 5, 8]}]': finished 2024-04-30T08:33:32.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:31 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.30", "id": [1, 9]}]': finished 2024-04-30T08:33:32.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:31 smithi032 ceph-mon[28331]: osdmap e84: 12 total, 12 up, 12 in 2024-04-30T08:33:32.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:31 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2738467747' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-30T08:33:32.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:31 smithi032 ceph-mon[28331]: osdmap e85: 12 total, 12 up, 12 in 2024-04-30T08:33:32.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:31 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 7]}]': finished 2024-04-30T08:33:32.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:31 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]': finished 2024-04-30T08:33:32.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:31 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [3, 6, 5, 8]}]': finished 2024-04-30T08:33:32.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:31 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.30", "id": [1, 9]}]': finished 2024-04-30T08:33:32.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:31 smithi071 ceph-mon[33385]: osdmap e84: 12 total, 12 up, 12 in 2024-04-30T08:33:32.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:31 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2738467747' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-30T08:33:32.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:31 smithi071 ceph-mon[33385]: osdmap e85: 12 total, 12 up, 12 in 2024-04-30T08:33:32.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:31 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 7]}]': finished 2024-04-30T08:33:32.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:31 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.18", "id": [3, 9]}]': finished 2024-04-30T08:33:32.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:31 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [3, 6, 5, 8]}]': finished 2024-04-30T08:33:32.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:31 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.30", "id": [1, 9]}]': finished 2024-04-30T08:33:32.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:31 smithi189 ceph-mon[33038]: osdmap e84: 12 total, 12 up, 12 in 2024-04-30T08:33:32.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:31 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2738467747' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-30T08:33:32.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:31 smithi189 ceph-mon[33038]: osdmap e85: 12 total, 12 up, 12 in 2024-04-30T08:33:33.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:32 smithi032 ceph-mon[28331]: pgmap v375: 129 pgs: 3 peering, 126 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-30T08:33:33.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:32 smithi032 ceph-mon[28331]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-04-30T08:33:33.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:32 smithi071 ceph-mon[33385]: pgmap v375: 129 pgs: 3 peering, 126 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-30T08:33:33.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:32 smithi071 ceph-mon[33385]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-04-30T08:33:33.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:32 smithi189 ceph-mon[33038]: pgmap v375: 129 pgs: 3 peering, 126 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-30T08:33:33.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:32 smithi189 ceph-mon[33038]: Health check failed: Reduced data availability: 1 pg inactive, 3 pgs peering (PG_AVAILABILITY) 2024-04-30T08:33:35.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:34 smithi032 ceph-mon[28331]: pgmap v377: 129 pgs: 3 peering, 126 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-04-30T08:33:35.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:34 smithi071 ceph-mon[33385]: pgmap v377: 129 pgs: 3 peering, 126 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-04-30T08:33:35.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:34 smithi189 ceph-mon[33038]: pgmap v377: 129 pgs: 3 peering, 126 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-04-30T08:33:37.079 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:36 smithi032 ceph-mon[28331]: pgmap v378: 129 pgs: 3 peering, 126 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s 2024-04-30T08:33:37.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:36 smithi071 ceph-mon[33385]: pgmap v378: 129 pgs: 3 peering, 126 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s 2024-04-30T08:33:37.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:36 smithi189 ceph-mon[33038]: pgmap v378: 129 pgs: 3 peering, 126 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 10 op/s 2024-04-30T08:33:38.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.028+0000 7f8d9d000700 1 -- 172.21.15.32:0/3215599813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d98105240 msgr2=0x7f8d98105650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:38.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.028+0000 7f8d9d000700 1 --2- 172.21.15.32:0/3215599813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d98105240 0x7f8d98105650 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f8d88009990 tx=0x7f8d88009ca0 comp rx=0 tx=0).stop 2024-04-30T08:33:38.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.028+0000 7f8d9d000700 1 -- 172.21.15.32:0/3215599813 shutdown_connections 2024-04-30T08:33:38.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.028+0000 7f8d9d000700 1 --2- 172.21.15.32:0/3215599813 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d9810e670 0x7f8d98110b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:38.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.028+0000 7f8d9d000700 1 --2- 172.21.15.32:0/3215599813 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d98105240 0x7f8d98105650 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:38.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.028+0000 7f8d9d000700 1 --2- 172.21.15.32:0/3215599813 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d98105b90 0x7f8d9810e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:38.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.028+0000 7f8d9d000700 1 -- 172.21.15.32:0/3215599813 >> 172.21.15.32:0/3215599813 conn(0x7f8d98078a50 msgr2=0x7f8d98078eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:38.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.028+0000 7f8d9d000700 1 -- 172.21.15.32:0/3215599813 shutdown_connections 2024-04-30T08:33:38.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.028+0000 7f8d9d000700 1 -- 172.21.15.32:0/3215599813 wait complete. 2024-04-30T08:33:38.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.030+0000 7f8d9d000700 1 Processor -- start 2024-04-30T08:33:38.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.030+0000 7f8d9d000700 1 -- start start 2024-04-30T08:33:38.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.030+0000 7f8d9d000700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d98105240 0x7f8d9819ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:38.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.030+0000 7f8d9d000700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d98105b90 0x7f8d9819d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:38.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.030+0000 7f8d9d000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d9810e670 0x7f8d981a1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:38.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.030+0000 7f8d9d000700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8d98114230 con 0x7f8d98105240 2024-04-30T08:33:38.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.030+0000 7f8d9d000700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8d981140b0 con 0x7f8d98105b90 2024-04-30T08:33:38.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.030+0000 7f8d9d000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8d981143b0 con 0x7f8d9810e670 2024-04-30T08:33:38.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.031+0000 7f8d97400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d98105240 0x7f8d9819ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:38.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.031+0000 7f8d97e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d9810e670 0x7f8d981a1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:38.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.031+0000 7f8d96a00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d98105b90 0x7f8d9819d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:38.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.031+0000 7f8d97400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d98105240 0x7f8d9819ce40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47688/0 (socket says 172.21.15.32:47688) 2024-04-30T08:33:38.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.031+0000 7f8d97400700 1 -- 172.21.15.32:0/3618273147 learned_addr learned my addr 172.21.15.32:0/3618273147 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:33:38.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.031+0000 7f8d97e00700 1 -- 172.21.15.32:0/3618273147 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d98105b90 msgr2=0x7f8d9819d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:38.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.031+0000 7f8d97e00700 1 --2- 172.21.15.32:0/3618273147 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d98105b90 0x7f8d9819d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:38.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.031+0000 7f8d97e00700 1 -- 172.21.15.32:0/3618273147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d98105240 msgr2=0x7f8d9819ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:38.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.031+0000 7f8d97e00700 1 --2- 172.21.15.32:0/3618273147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d98105240 0x7f8d9819ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:38.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.031+0000 7f8d97e00700 1 -- 172.21.15.32:0/3618273147 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d981a1d10 con 0x7f8d9810e670 2024-04-30T08:33:38.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.031+0000 7f8d96a00700 1 --2- 172.21.15.32:0/3618273147 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d98105b90 0x7f8d9819d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:33:38.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.032+0000 7f8d97e00700 1 --2- 172.21.15.32:0/3618273147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d9810e670 0x7f8d981a1670 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8d9000efc0 tx=0x7f8d9000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:38.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.032+0000 7f8d8fe00700 1 -- 172.21.15.32:0/3618273147 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d900092e0 con 0x7f8d9810e670 2024-04-30T08:33:38.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.032+0000 7f8d9d000700 1 -- 172.21.15.32:0/3618273147 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d981a1fa0 con 0x7f8d9810e670 2024-04-30T08:33:38.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.032+0000 7f8d8fe00700 1 -- 172.21.15.32:0/3618273147 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d90004bb0 con 0x7f8d9810e670 2024-04-30T08:33:38.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.032+0000 7f8d8fe00700 1 -- 172.21.15.32:0/3618273147 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d90004020 con 0x7f8d9810e670 2024-04-30T08:33:38.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.032+0000 7f8d9d000700 1 -- 172.21.15.32:0/3618273147 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d981a9920 con 0x7f8d9810e670 2024-04-30T08:33:38.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.034+0000 7f8d9d000700 1 -- 172.21.15.32:0/3618273147 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d9806c810 con 0x7f8d9810e670 2024-04-30T08:33:38.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.036+0000 7f8d8fe00700 1 -- 172.21.15.32:0/3618273147 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f8d9001b030 con 0x7f8d9810e670 2024-04-30T08:33:38.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.037+0000 7f8d8fe00700 1 --2- 172.21.15.32:0/3618273147 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8d7c0afc10 0x7f8d7c0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:38.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.037+0000 7f8d8fe00700 1 -- 172.21.15.32:0/3618273147 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f8d900dec10 con 0x7f8d9810e670 2024-04-30T08:33:38.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.038+0000 7f8d97400700 1 --2- 172.21.15.32:0/3618273147 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8d7c0afc10 0x7f8d7c0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:38.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.038+0000 7f8d97400700 1 --2- 172.21.15.32:0/3618273147 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8d7c0afc10 0x7f8d7c0b2120 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8d88006010 tx=0x7f8d8800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:38.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.042+0000 7f8d8fe00700 1 -- 172.21.15.32:0/3618273147 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8d90022020 con 0x7f8d9810e670 2024-04-30T08:33:38.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:37 smithi071 ceph-mon[33385]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-04-30T08:33:38.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:37 smithi071 ceph-mon[33385]: Cluster is now healthy 2024-04-30T08:33:38.282 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:37 smithi032 ceph-mon[28331]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-04-30T08:33:38.282 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:37 smithi032 ceph-mon[28331]: Cluster is now healthy 2024-04-30T08:33:38.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.281+0000 7f8d9d000700 1 -- 172.21.15.32:0/3618273147 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f8d9819e1a0 con 0x7f8d9810e670 2024-04-30T08:33:38.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.282+0000 7f8d8fe00700 1 -- 172.21.15.32:0/3618273147 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f8d90010040 con 0x7f8d9810e670 2024-04-30T08:33:38.283 INFO:teuthology.orchestra.run.smithi032.stdout:231928234039 2024-04-30T08:33:38.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.285+0000 7f8d9d000700 1 -- 172.21.15.32:0/3618273147 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8d7c0afc10 msgr2=0x7f8d7c0b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:38.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.285+0000 7f8d9d000700 1 --2- 172.21.15.32:0/3618273147 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8d7c0afc10 0x7f8d7c0b2120 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8d88006010 tx=0x7f8d8800b260 comp rx=0 tx=0).stop 2024-04-30T08:33:38.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.285+0000 7f8d9d000700 1 -- 172.21.15.32:0/3618273147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d9810e670 msgr2=0x7f8d981a1670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:38.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.285+0000 7f8d9d000700 1 --2- 172.21.15.32:0/3618273147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d9810e670 0x7f8d981a1670 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f8d9000efc0 tx=0x7f8d9000c3a0 comp rx=0 tx=0).stop 2024-04-30T08:33:38.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.285+0000 7f8d9d000700 1 -- 172.21.15.32:0/3618273147 shutdown_connections 2024-04-30T08:33:38.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.285+0000 7f8d9d000700 1 --2- 172.21.15.32:0/3618273147 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8d7c0afc10 0x7f8d7c0b2120 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:38.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.285+0000 7f8d9d000700 1 --2- 172.21.15.32:0/3618273147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d9810e670 0x7f8d981a1670 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:38.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.285+0000 7f8d9d000700 1 --2- 172.21.15.32:0/3618273147 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d98105240 0x7f8d9819ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:38.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.285+0000 7f8d9d000700 1 --2- 172.21.15.32:0/3618273147 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d98105b90 0x7f8d9819d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:38.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.285+0000 7f8d9d000700 1 -- 172.21.15.32:0/3618273147 >> 172.21.15.32:0/3618273147 conn(0x7f8d98078a50 msgr2=0x7f8d98103fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:38.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.285+0000 7f8d9d000700 1 -- 172.21.15.32:0/3618273147 shutdown_connections 2024-04-30T08:33:38.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:38.286+0000 7f8d9d000700 1 -- 172.21.15.32:0/3618273147 wait complete. 2024-04-30T08:33:38.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:37 smithi189 ceph-mon[33038]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 3 pgs peering) 2024-04-30T08:33:38.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:37 smithi189 ceph-mon[33038]: Cluster is now healthy 2024-04-30T08:33:38.798 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:33:38.837 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574937 got 55834574941 for osd.1 2024-04-30T08:33:38.838 DEBUG:teuthology.parallel:result is None 2024-04-30T08:33:39.226 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:38 smithi032 ceph-mon[28331]: pgmap v379: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-30T08:33:39.226 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:38 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3618273147' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-30T08:33:39.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:38 smithi071 ceph-mon[33385]: pgmap v379: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-30T08:33:39.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:38 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3618273147' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-30T08:33:39.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:38 smithi189 ceph-mon[33038]: pgmap v379: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 8 op/s 2024-04-30T08:33:39.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:38 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3618273147' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-30T08:33:40.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.394+0000 7f3ec7200700 1 -- 172.21.15.32:0/2739702296 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ec80765d0 msgr2=0x7f3ec81103c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:40.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.394+0000 7f3ec7200700 1 --2- 172.21.15.32:0/2739702296 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ec80765d0 0x7f3ec81103c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3eb4009990 tx=0x7f3eb4009ca0 comp rx=0 tx=0).stop 2024-04-30T08:33:40.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.394+0000 7f3ec7200700 1 -- 172.21.15.32:0/2739702296 shutdown_connections 2024-04-30T08:33:40.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.394+0000 7f3ec7200700 1 --2- 172.21.15.32:0/2739702296 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ec8110900 0x7f3ec8112d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:40.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.394+0000 7f3ec7200700 1 --2- 172.21.15.32:0/2739702296 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ec8075c80 0x7f3ec8076090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:40.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.394+0000 7f3ec7200700 1 --2- 172.21.15.32:0/2739702296 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ec80765d0 0x7f3ec81103c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:40.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.394+0000 7f3ec7200700 1 -- 172.21.15.32:0/2739702296 >> 172.21.15.32:0/2739702296 conn(0x7f3ec8079c60 msgr2=0x7f3ec807a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:40.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.394+0000 7f3ec7200700 1 -- 172.21.15.32:0/2739702296 shutdown_connections 2024-04-30T08:33:40.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.394+0000 7f3ec7200700 1 -- 172.21.15.32:0/2739702296 wait complete. 2024-04-30T08:33:40.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.396+0000 7f3ec7200700 1 Processor -- start 2024-04-30T08:33:40.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.396+0000 7f3ec7200700 1 -- start start 2024-04-30T08:33:40.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.397+0000 7f3ec7200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ec8075c80 0x7f3ec819cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:40.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.397+0000 7f3ec7200700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ec80765d0 0x7f3ec819d350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:40.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.397+0000 7f3ec7200700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ec8110900 0x7f3ec81a15e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:40.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.397+0000 7f3ec7200700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec8116420 con 0x7f3ec80765d0 2024-04-30T08:33:40.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.397+0000 7f3ec7200700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec81162a0 con 0x7f3ec8110900 2024-04-30T08:33:40.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.397+0000 7f3ec7200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3ec81165a0 con 0x7f3ec8075c80 2024-04-30T08:33:40.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.397+0000 7f3ec5e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ec8075c80 0x7f3ec819cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:40.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.397+0000 7f3ec5e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ec8075c80 0x7f3ec819cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:38118/0 (socket says 172.21.15.32:38118) 2024-04-30T08:33:40.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.397+0000 7f3ec5e00700 1 -- 172.21.15.32:0/1762517852 learned_addr learned my addr 172.21.15.32:0/1762517852 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:33:40.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.397+0000 7f3ec5400700 1 --2- 172.21.15.32:0/1762517852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ec80765d0 0x7f3ec819d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:40.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.397+0000 7f3ec6800700 1 --2- 172.21.15.32:0/1762517852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ec8110900 0x7f3ec81a15e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:40.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.398+0000 7f3ec5e00700 1 -- 172.21.15.32:0/1762517852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ec8110900 msgr2=0x7f3ec81a15e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:40.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.398+0000 7f3ec5e00700 1 --2- 172.21.15.32:0/1762517852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ec8110900 0x7f3ec81a15e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:40.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.398+0000 7f3ec5e00700 1 -- 172.21.15.32:0/1762517852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ec80765d0 msgr2=0x7f3ec819d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:40.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.398+0000 7f3ec5e00700 1 --2- 172.21.15.32:0/1762517852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ec80765d0 0x7f3ec819d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:40.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.398+0000 7f3ec5e00700 1 -- 172.21.15.32:0/1762517852 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ec81a1c80 con 0x7f3ec8075c80 2024-04-30T08:33:40.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.398+0000 7f3ec5400700 1 --2- 172.21.15.32:0/1762517852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ec80765d0 0x7f3ec819d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:33:40.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.398+0000 7f3ec5e00700 1 --2- 172.21.15.32:0/1762517852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ec8075c80 0x7f3ec819cdb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3eb800e9a0 tx=0x7f3eb800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:40.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.399+0000 7f3ebea00700 1 -- 172.21.15.32:0/1762517852 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb800ca10 con 0x7f3ec8075c80 2024-04-30T08:33:40.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.399+0000 7f3ebea00700 1 -- 172.21.15.32:0/1762517852 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3eb800cbb0 con 0x7f3ec8075c80 2024-04-30T08:33:40.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.399+0000 7f3ec7200700 1 -- 172.21.15.32:0/1762517852 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ec81a1f70 con 0x7f3ec8075c80 2024-04-30T08:33:40.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.399+0000 7f3ebea00700 1 -- 172.21.15.32:0/1762517852 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3eb8018680 con 0x7f3ec8075c80 2024-04-30T08:33:40.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.399+0000 7f3ec7200700 1 -- 172.21.15.32:0/1762517852 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ec81a9890 con 0x7f3ec8075c80 2024-04-30T08:33:40.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.400+0000 7f3ec7200700 1 -- 172.21.15.32:0/1762517852 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3ec80771f0 con 0x7f3ec8075c80 2024-04-30T08:33:40.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.402+0000 7f3ebea00700 1 -- 172.21.15.32:0/1762517852 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f3eb8010430 con 0x7f3ec8075c80 2024-04-30T08:33:40.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.404+0000 7f3ebea00700 1 --2- 172.21.15.32:0/1762517852 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3eac0afc90 0x7f3eac0b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:40.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.404+0000 7f3ebea00700 1 -- 172.21.15.32:0/1762517852 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f3eb80df6a0 con 0x7f3ec8075c80 2024-04-30T08:33:40.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.404+0000 7f3ec5400700 1 --2- 172.21.15.32:0/1762517852 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3eac0afc90 0x7f3eac0b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:40.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.405+0000 7f3ec5400700 1 --2- 172.21.15.32:0/1762517852 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3eac0afc90 0x7f3eac0b21a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3eb400b2e0 tx=0x7f3eb4005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:40.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.406+0000 7f3ebea00700 1 -- 172.21.15.32:0/1762517852 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3eb80a7d10 con 0x7f3ec8075c80 2024-04-30T08:33:40.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.649+0000 7f3ec7200700 1 -- 172.21.15.32:0/1762517852 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f3ec802e590 con 0x7f3ec8075c80 2024-04-30T08:33:40.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.650+0000 7f3ebea00700 1 -- 172.21.15.32:0/1762517852 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f3eb80a7610 con 0x7f3ec8075c80 2024-04-30T08:33:40.650 INFO:teuthology.orchestra.run.smithi032.stdout:257698037807 2024-04-30T08:33:40.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.652+0000 7f3ec7200700 1 -- 172.21.15.32:0/1762517852 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3eac0afc90 msgr2=0x7f3eac0b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:40.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.652+0000 7f3ec7200700 1 --2- 172.21.15.32:0/1762517852 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3eac0afc90 0x7f3eac0b21a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3eb400b2e0 tx=0x7f3eb4005960 comp rx=0 tx=0).stop 2024-04-30T08:33:40.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.652+0000 7f3ec7200700 1 -- 172.21.15.32:0/1762517852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ec8075c80 msgr2=0x7f3ec819cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:40.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.652+0000 7f3ec7200700 1 --2- 172.21.15.32:0/1762517852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ec8075c80 0x7f3ec819cdb0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f3eb800e9a0 tx=0x7f3eb800ecb0 comp rx=0 tx=0).stop 2024-04-30T08:33:40.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.652+0000 7f3ec7200700 1 -- 172.21.15.32:0/1762517852 shutdown_connections 2024-04-30T08:33:40.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.652+0000 7f3ec7200700 1 --2- 172.21.15.32:0/1762517852 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3eac0afc90 0x7f3eac0b21a0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:40.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.652+0000 7f3ec7200700 1 --2- 172.21.15.32:0/1762517852 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3ec8110900 0x7f3ec81a15e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:40.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.652+0000 7f3ec7200700 1 --2- 172.21.15.32:0/1762517852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3ec8075c80 0x7f3ec819cdb0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:40.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.652+0000 7f3ec7200700 1 --2- 172.21.15.32:0/1762517852 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3ec80765d0 0x7f3ec819d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:40.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.652+0000 7f3ec7200700 1 -- 172.21.15.32:0/1762517852 >> 172.21.15.32:0/1762517852 conn(0x7f3ec8079c60 msgr2=0x7f3ec8100ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:40.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.653+0000 7f3ec7200700 1 -- 172.21.15.32:0/1762517852 shutdown_connections 2024-04-30T08:33:40.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:40.653+0000 7f3ec7200700 1 -- 172.21.15.32:0/1762517852 wait complete. 2024-04-30T08:33:41.013 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:33:41.024 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:33:41.211 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:33:41.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:40 smithi071 ceph-mon[33385]: pgmap v380: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-04-30T08:33:41.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:40 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1762517852' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-30T08:33:41.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:40 smithi189 ceph-mon[33038]: pgmap v380: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-04-30T08:33:41.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:40 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1762517852' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-30T08:33:41.304 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:40 smithi032 ceph-mon[28331]: pgmap v380: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 10 op/s 2024-04-30T08:33:41.304 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:40 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1762517852' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-30T08:33:42.054 INFO:tasks.cephadm.ceph_manager.ceph:need seq 231928234034 got 231928234039 for osd.8 2024-04-30T08:33:42.054 DEBUG:teuthology.parallel:result is None 2024-04-30T08:33:43.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:42 smithi071 ceph-mon[33385]: pgmap v381: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-30T08:33:43.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:42 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3987630753' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-30T08:33:43.267 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:42 smithi032 ceph-mon[28331]: pgmap v381: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-30T08:33:43.267 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:42 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3987630753' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-30T08:33:43.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:42 smithi189 ceph-mon[33038]: pgmap v381: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-30T08:33:43.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:42 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3987630753' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-30T08:33:43.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.740+0000 7fa8e7e00700 1 -- 172.21.15.32:0/295229263 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8e8105b30 msgr2=0x7fa8e810e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:43.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.740+0000 7fa8e7e00700 1 --2- 172.21.15.32:0/295229263 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8e8105b30 0x7fa8e810e080 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa8d40098f0 tx=0x7fa8d4009c00 comp rx=0 tx=0).stop 2024-04-30T08:33:43.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.740+0000 7fa8e7e00700 1 -- 172.21.15.32:0/295229263 shutdown_connections 2024-04-30T08:33:43.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.740+0000 7fa8e7e00700 1 --2- 172.21.15.32:0/295229263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8e810e650 0x7fa8e8110b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:43.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.740+0000 7fa8e7e00700 1 --2- 172.21.15.32:0/295229263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8e81051e0 0x7fa8e81055f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:43.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.740+0000 7fa8e7e00700 1 --2- 172.21.15.32:0/295229263 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8e8105b30 0x7fa8e810e080 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:43.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.740+0000 7fa8e7e00700 1 -- 172.21.15.32:0/295229263 >> 172.21.15.32:0/295229263 conn(0x7fa8e8078aa0 msgr2=0x7fa8e8078ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:43.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.740+0000 7fa8e7e00700 1 -- 172.21.15.32:0/295229263 shutdown_connections 2024-04-30T08:33:43.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.740+0000 7fa8e7e00700 1 -- 172.21.15.32:0/295229263 wait complete. 2024-04-30T08:33:43.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.741+0000 7fa8e7e00700 1 Processor -- start 2024-04-30T08:33:43.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.742+0000 7fa8e7e00700 1 -- start start 2024-04-30T08:33:43.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.742+0000 7fa8e7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8e81051e0 0x7fa8e81060f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:43.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.742+0000 7fa8e7e00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8e8105b30 0x7fa8e8106630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:43.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.742+0000 7fa8e7e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8e810e650 0x7fa8e8106b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:43.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.742+0000 7fa8e7e00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e8114200 con 0x7fa8e810e650 2024-04-30T08:33:43.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.742+0000 7fa8e7e00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e8114080 con 0x7fa8e8105b30 2024-04-30T08:33:43.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.742+0000 7fa8e7e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa8e8114380 con 0x7fa8e81051e0 2024-04-30T08:33:43.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.742+0000 7fa8e6a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8e81051e0 0x7fa8e81060f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:43.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.742+0000 7fa8e7400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8e810e650 0x7fa8e8106b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:43.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.742+0000 7fa8e6a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8e81051e0 0x7fa8e81060f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:38130/0 (socket says 172.21.15.32:38130) 2024-04-30T08:33:43.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.742+0000 7fa8e6a00700 1 -- 172.21.15.32:0/3987630753 learned_addr learned my addr 172.21.15.32:0/3987630753 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:33:43.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.742+0000 7fa8e6000700 1 --2- 172.21.15.32:0/3987630753 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8e8105b30 0x7fa8e8106630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:43.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.743+0000 7fa8e6a00700 1 -- 172.21.15.32:0/3987630753 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8e8105b30 msgr2=0x7fa8e8106630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:43.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.743+0000 7fa8e6a00700 1 --2- 172.21.15.32:0/3987630753 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8e8105b30 0x7fa8e8106630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:43.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.743+0000 7fa8e6a00700 1 -- 172.21.15.32:0/3987630753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8e810e650 msgr2=0x7fa8e8106b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:43.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.743+0000 7fa8e6a00700 1 --2- 172.21.15.32:0/3987630753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8e810e650 0x7fa8e8106b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:43.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.743+0000 7fa8e6a00700 1 -- 172.21.15.32:0/3987630753 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8e81ad4f0 con 0x7fa8e81051e0 2024-04-30T08:33:43.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.743+0000 7fa8e6000700 1 --2- 172.21.15.32:0/3987630753 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8e8105b30 0x7fa8e8106630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:33:43.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.743+0000 7fa8e6a00700 1 --2- 172.21.15.32:0/3987630753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8e81051e0 0x7fa8e81060f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa8d800ec10 tx=0x7fa8d800ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:43.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.743+0000 7fa8eca00700 1 -- 172.21.15.32:0/3987630753 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d800cc90 con 0x7fa8e81051e0 2024-04-30T08:33:43.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.743+0000 7fa8e7e00700 1 -- 172.21.15.32:0/3987630753 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8e81ad780 con 0x7fa8e81051e0 2024-04-30T08:33:43.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.743+0000 7fa8eca00700 1 -- 172.21.15.32:0/3987630753 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa8d800ce30 con 0x7fa8e81051e0 2024-04-30T08:33:43.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.744+0000 7fa8eca00700 1 -- 172.21.15.32:0/3987630753 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8d8010430 con 0x7fa8e81051e0 2024-04-30T08:33:43.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.744+0000 7fa8e7e00700 1 -- 172.21.15.32:0/3987630753 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8e81adc90 con 0x7fa8e81051e0 2024-04-30T08:33:43.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.745+0000 7fa8e7e00700 1 -- 172.21.15.32:0/3987630753 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa8e810bbb0 con 0x7fa8e81051e0 2024-04-30T08:33:43.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.747+0000 7fa8eca00700 1 -- 172.21.15.32:0/3987630753 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fa8d8004000 con 0x7fa8e81051e0 2024-04-30T08:33:43.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.749+0000 7fa8eca00700 1 --2- 172.21.15.32:0/3987630753 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa8cc0afc10 0x7fa8cc0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:43.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.749+0000 7fa8eca00700 1 -- 172.21.15.32:0/3987630753 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fa8d80df5a0 con 0x7fa8e81051e0 2024-04-30T08:33:43.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.749+0000 7fa8e6000700 1 --2- 172.21.15.32:0/3987630753 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa8cc0afc10 0x7fa8cc0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:43.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.750+0000 7fa8e6000700 1 --2- 172.21.15.32:0/3987630753 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa8cc0afc10 0x7fa8cc0b2120 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa8d40098c0 tx=0x7fa8d4019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:43.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.751+0000 7fa8eca00700 1 -- 172.21.15.32:0/3987630753 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa8d80a7c10 con 0x7fa8e81051e0 2024-04-30T08:33:43.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.988+0000 7fa8e7e00700 1 -- 172.21.15.32:0/3987630753 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fa8e8107620 con 0x7fa8e81051e0 2024-04-30T08:33:43.333 INFO:teuthology.orchestra.run.smithi032.stdout:304942678053 2024-04-30T08:33:43.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.989+0000 7fa8eca00700 1 -- 172.21.15.32:0/3987630753 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fa8d80a7510 con 0x7fa8e81051e0 2024-04-30T08:33:43.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.992+0000 7fa8e7e00700 1 -- 172.21.15.32:0/3987630753 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa8cc0afc10 msgr2=0x7fa8cc0b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:43.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.992+0000 7fa8e7e00700 1 --2- 172.21.15.32:0/3987630753 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa8cc0afc10 0x7fa8cc0b2120 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa8d40098c0 tx=0x7fa8d4019040 comp rx=0 tx=0).stop 2024-04-30T08:33:43.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.992+0000 7fa8e7e00700 1 -- 172.21.15.32:0/3987630753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8e81051e0 msgr2=0x7fa8e81060f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:43.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.992+0000 7fa8e7e00700 1 --2- 172.21.15.32:0/3987630753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8e81051e0 0x7fa8e81060f0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fa8d800ec10 tx=0x7fa8d800ef20 comp rx=0 tx=0).stop 2024-04-30T08:33:43.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.992+0000 7fa8e7e00700 1 -- 172.21.15.32:0/3987630753 shutdown_connections 2024-04-30T08:33:43.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.992+0000 7fa8e7e00700 1 --2- 172.21.15.32:0/3987630753 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa8cc0afc10 0x7fa8cc0b2120 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:43.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.992+0000 7fa8e7e00700 1 --2- 172.21.15.32:0/3987630753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8e810e650 0x7fa8e8106b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:43.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.992+0000 7fa8e7e00700 1 --2- 172.21.15.32:0/3987630753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8e81051e0 0x7fa8e81060f0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:43.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.992+0000 7fa8e7e00700 1 --2- 172.21.15.32:0/3987630753 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8e8105b30 0x7fa8e8106630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:43.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.992+0000 7fa8e7e00700 1 -- 172.21.15.32:0/3987630753 >> 172.21.15.32:0/3987630753 conn(0x7fa8e8078aa0 msgr2=0x7fa8e8102fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:43.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.992+0000 7fa8e7e00700 1 -- 172.21.15.32:0/3987630753 shutdown_connections 2024-04-30T08:33:43.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:41.993+0000 7fa8e7e00700 1 -- 172.21.15.32:0/3987630753 wait complete. 2024-04-30T08:33:44.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.663+0000 7f22f9e00700 1 -- 172.21.15.32:0/2961317384 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22f41095d0 msgr2=0x7f22f4110f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:44.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.663+0000 7f22f9e00700 1 --2- 172.21.15.32:0/2961317384 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22f41095d0 0x7f22f4110f30 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f22ec00b0d0 tx=0x7f22ec00b3e0 comp rx=0 tx=0).stop 2024-04-30T08:33:44.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.664+0000 7f22f9e00700 1 -- 172.21.15.32:0/2961317384 shutdown_connections 2024-04-30T08:33:44.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.664+0000 7f22f9e00700 1 --2- 172.21.15.32:0/2961317384 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22f41095d0 0x7f22f4110f30 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:44.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.664+0000 7f22f9e00700 1 --2- 172.21.15.32:0/2961317384 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f22f4104350 0x7f22f4104760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:44.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.664+0000 7f22f9e00700 1 --2- 172.21.15.32:0/2961317384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22f4108c80 0x7f22f4109090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:44.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.664+0000 7f22f9e00700 1 -- 172.21.15.32:0/2961317384 >> 172.21.15.32:0/2961317384 conn(0x7f22f4079c60 msgr2=0x7f22f407a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:44.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.664+0000 7f22f9e00700 1 -- 172.21.15.32:0/2961317384 shutdown_connections 2024-04-30T08:33:44.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.664+0000 7f22f9e00700 1 -- 172.21.15.32:0/2961317384 wait complete. 2024-04-30T08:33:44.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.665+0000 7f22f9e00700 1 Processor -- start 2024-04-30T08:33:44.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.665+0000 7f22f9e00700 1 -- start start 2024-04-30T08:33:44.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.666+0000 7f22f9e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22f4104350 0x7f22f419cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:44.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.666+0000 7f22f9e00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f22f4108c80 0x7f22f419d340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:44.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.666+0000 7f22f9e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22f41095d0 0x7f22f41a15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:44.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.666+0000 7f22f9e00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f22f4114250 con 0x7f22f4104350 2024-04-30T08:33:44.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.666+0000 7f22f9e00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f22f41140d0 con 0x7f22f4108c80 2024-04-30T08:33:44.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.666+0000 7f22f9e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f22f41143d0 con 0x7f22f41095d0 2024-04-30T08:33:44.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.666+0000 7f22f8a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22f4104350 0x7f22f419cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:44.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.666+0000 7f22f8a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22f4104350 0x7f22f419cda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47736/0 (socket says 172.21.15.32:47736) 2024-04-30T08:33:44.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.666+0000 7f22f8a00700 1 -- 172.21.15.32:0/3800693185 learned_addr learned my addr 172.21.15.32:0/3800693185 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:33:44.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.666+0000 7f22f3e00700 1 --2- 172.21.15.32:0/3800693185 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f22f4108c80 0x7f22f419d340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.667+0000 7f22f8a00700 1 -- 172.21.15.32:0/3800693185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22f41095d0 msgr2=0x7f22f41a15d0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-30T08:33:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.667+0000 7f22f8a00700 1 --2- 172.21.15.32:0/3800693185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22f41095d0 0x7f22f41a15d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.667+0000 7f22f8a00700 1 -- 172.21.15.32:0/3800693185 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f22f4108c80 msgr2=0x7f22f419d340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.667+0000 7f22f8a00700 1 --2- 172.21.15.32:0/3800693185 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f22f4108c80 0x7f22f419d340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.667+0000 7f22f8a00700 1 -- 172.21.15.32:0/3800693185 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22f41a1c70 con 0x7f22f4104350 2024-04-30T08:33:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.667+0000 7f22f8a00700 1 --2- 172.21.15.32:0/3800693185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22f4104350 0x7f22f419cda0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f22e800e9a0 tx=0x7f22e800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.667+0000 7f22f1600700 1 -- 172.21.15.32:0/3800693185 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e800cbd0 con 0x7f22f4104350 2024-04-30T08:33:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.668+0000 7f22f9e00700 1 -- 172.21.15.32:0/3800693185 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22f41a1f60 con 0x7f22f4104350 2024-04-30T08:33:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.668+0000 7f22f1600700 1 -- 172.21.15.32:0/3800693185 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f22e800cd70 con 0x7f22f4104350 2024-04-30T08:33:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.668+0000 7f22f1600700 1 -- 172.21.15.32:0/3800693185 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22e8018680 con 0x7f22f4104350 2024-04-30T08:33:44.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.668+0000 7f22f9e00700 1 -- 172.21.15.32:0/3800693185 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22f41a9880 con 0x7f22f4104350 2024-04-30T08:33:44.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.669+0000 7f22f9e00700 1 -- 172.21.15.32:0/3800693185 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22f4105800 con 0x7f22f4104350 2024-04-30T08:33:44.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.672+0000 7f22f1600700 1 -- 172.21.15.32:0/3800693185 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f22e8010430 con 0x7f22f4104350 2024-04-30T08:33:44.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.675+0000 7f22f1600700 1 --2- 172.21.15.32:0/3800693185 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f22dc0afc90 0x7f22dc0b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:44.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.675+0000 7f22f1600700 1 -- 172.21.15.32:0/3800693185 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f22e80df5c0 con 0x7f22f4104350 2024-04-30T08:33:44.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.675+0000 7f22f3e00700 1 --2- 172.21.15.32:0/3800693185 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f22dc0afc90 0x7f22dc0b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:44.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.677+0000 7f22f3e00700 1 --2- 172.21.15.32:0/3800693185 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f22dc0afc90 0x7f22dc0b21a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f22f4111000 tx=0x7f22e0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:44.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.678+0000 7f22f1600700 1 -- 172.21.15.32:0/3800693185 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f22e80a7c70 con 0x7f22f4104350 2024-04-30T08:33:44.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.930+0000 7f22f9e00700 1 -- 172.21.15.32:0/3800693185 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f22f41a9ab0 con 0x7f22f4104350 2024-04-30T08:33:44.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.931+0000 7f22f1600700 1 -- 172.21.15.32:0/3800693185 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f22e80a7570 con 0x7f22f4104350 2024-04-30T08:33:44.933 INFO:teuthology.orchestra.run.smithi032.stdout:107374182484 2024-04-30T08:33:44.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.935+0000 7f22f9e00700 1 -- 172.21.15.32:0/3800693185 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f22dc0afc90 msgr2=0x7f22dc0b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:44.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.935+0000 7f22f9e00700 1 --2- 172.21.15.32:0/3800693185 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f22dc0afc90 0x7f22dc0b21a0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f22f4111000 tx=0x7f22e0009260 comp rx=0 tx=0).stop 2024-04-30T08:33:44.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.935+0000 7f22f9e00700 1 -- 172.21.15.32:0/3800693185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22f4104350 msgr2=0x7f22f419cda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:44.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.935+0000 7f22f9e00700 1 --2- 172.21.15.32:0/3800693185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22f4104350 0x7f22f419cda0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f22e800e9a0 tx=0x7f22e800ecb0 comp rx=0 tx=0).stop 2024-04-30T08:33:44.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.935+0000 7f22f9e00700 1 -- 172.21.15.32:0/3800693185 shutdown_connections 2024-04-30T08:33:44.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.935+0000 7f22f9e00700 1 --2- 172.21.15.32:0/3800693185 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f22dc0afc90 0x7f22dc0b21a0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:44.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.935+0000 7f22f9e00700 1 --2- 172.21.15.32:0/3800693185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22f41095d0 0x7f22f41a15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:44.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.935+0000 7f22f9e00700 1 --2- 172.21.15.32:0/3800693185 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f22f4104350 0x7f22f419cda0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:44.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.935+0000 7f22f9e00700 1 --2- 172.21.15.32:0/3800693185 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f22f4108c80 0x7f22f419d340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:44.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.935+0000 7f22f9e00700 1 -- 172.21.15.32:0/3800693185 >> 172.21.15.32:0/3800693185 conn(0x7f22f4079c60 msgr2=0x7f22f410f2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:44.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.935+0000 7f22f9e00700 1 -- 172.21.15.32:0/3800693185 shutdown_connections 2024-04-30T08:33:44.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:44.935+0000 7f22f9e00700 1 -- 172.21.15.32:0/3800693185 wait complete. 2024-04-30T08:33:45.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:44 smithi032 ceph-mon[28331]: pgmap v382: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:33:45.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:44 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3800693185' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-30T08:33:45.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:44 smithi071 ceph-mon[33385]: pgmap v382: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:33:45.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:44 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3800693185' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-30T08:33:45.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:44 smithi189 ceph-mon[33038]: pgmap v382: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:33:45.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:44 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3800693185' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-30T08:33:45.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.837+0000 7fc5c5600700 1 -- 172.21.15.32:0/1954614753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5c01051e0 msgr2=0x7fc5c01055f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:45.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.837+0000 7fc5c5600700 1 --2- 172.21.15.32:0/1954614753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5c01051e0 0x7fc5c01055f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0009990 tx=0x7fc5b0009ca0 comp rx=0 tx=0).stop 2024-04-30T08:33:45.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.838+0000 7fc5c5600700 1 -- 172.21.15.32:0/1954614753 shutdown_connections 2024-04-30T08:33:45.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.838+0000 7fc5c5600700 1 --2- 172.21.15.32:0/1954614753 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5c010e650 0x7fc5c0110b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:45.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.838+0000 7fc5c5600700 1 --2- 172.21.15.32:0/1954614753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5c01051e0 0x7fc5c01055f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:45.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.838+0000 7fc5c5600700 1 --2- 172.21.15.32:0/1954614753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5c0105b30 0x7fc5c010e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:45.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.838+0000 7fc5c5600700 1 -- 172.21.15.32:0/1954614753 >> 172.21.15.32:0/1954614753 conn(0x7fc5c0078a10 msgr2=0x7fc5c0078e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:45.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.838+0000 7fc5c5600700 1 -- 172.21.15.32:0/1954614753 shutdown_connections 2024-04-30T08:33:45.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.838+0000 7fc5c5600700 1 -- 172.21.15.32:0/1954614753 wait complete. 2024-04-30T08:33:45.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.839+0000 7fc5c5600700 1 Processor -- start 2024-04-30T08:33:45.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.839+0000 7fc5c5600700 1 -- start start 2024-04-30T08:33:45.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.840+0000 7fc5c5600700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5c01051e0 0x7fc5c019ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:45.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.840+0000 7fc5c5600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5c0105b30 0x7fc5c019d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:45.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.840+0000 7fc5c5600700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5c010e650 0x7fc5c01a1690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:45.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.840+0000 7fc5c5600700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c01141f0 con 0x7fc5c01051e0 2024-04-30T08:33:45.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.840+0000 7fc5c5600700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c0114070 con 0x7fc5c010e650 2024-04-30T08:33:45.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.840+0000 7fc5c5600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c0114370 con 0x7fc5c0105b30 2024-04-30T08:33:45.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.840+0000 7fc5bf400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5c0105b30 0x7fc5c019d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:45.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.840+0000 7fc5c4c00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5c010e650 0x7fc5c01a1690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:45.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.840+0000 7fc5bf400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5c0105b30 0x7fc5c019d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:38164/0 (socket says 172.21.15.32:38164) 2024-04-30T08:33:45.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.840+0000 7fc5bf400700 1 -- 172.21.15.32:0/2670251967 learned_addr learned my addr 172.21.15.32:0/2670251967 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:33:45.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.841+0000 7fc5bfe00700 1 --2- 172.21.15.32:0/2670251967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5c01051e0 0x7fc5c019ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:45.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.841+0000 7fc5c4c00700 1 -- 172.21.15.32:0/2670251967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5c0105b30 msgr2=0x7fc5c019d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:45.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.841+0000 7fc5c4c00700 1 --2- 172.21.15.32:0/2670251967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5c0105b30 0x7fc5c019d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:45.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.841+0000 7fc5c4c00700 1 -- 172.21.15.32:0/2670251967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5c01051e0 msgr2=0x7fc5c019ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:45.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.841+0000 7fc5c4c00700 1 --2- 172.21.15.32:0/2670251967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5c01051e0 0x7fc5c019ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:45.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.841+0000 7fc5c4c00700 1 -- 172.21.15.32:0/2670251967 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5c01a1d30 con 0x7fc5c010e650 2024-04-30T08:33:45.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.841+0000 7fc5c4c00700 1 --2- 172.21.15.32:0/2670251967 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5c010e650 0x7fc5c01a1690 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800efb0 tx=0x7fc5b800c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:45.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.842+0000 7fc5bcc00700 1 -- 172.21.15.32:0/2670251967 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b8003a00 con 0x7fc5c010e650 2024-04-30T08:33:45.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.842+0000 7fc5c5600700 1 -- 172.21.15.32:0/2670251967 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5c01a2020 con 0x7fc5c010e650 2024-04-30T08:33:45.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.842+0000 7fc5bcc00700 1 -- 172.21.15.32:0/2670251967 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc5b800ce50 con 0x7fc5c010e650 2024-04-30T08:33:45.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.842+0000 7fc5bcc00700 1 -- 172.21.15.32:0/2670251967 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b80047c0 con 0x7fc5c010e650 2024-04-30T08:33:45.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.842+0000 7fc5c5600700 1 -- 172.21.15.32:0/2670251967 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5c01a9940 con 0x7fc5c010e650 2024-04-30T08:33:45.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.843+0000 7fc5c5600700 1 -- 172.21.15.32:0/2670251967 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5c0051210 con 0x7fc5c010e650 2024-04-30T08:33:45.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.846+0000 7fc5bcc00700 1 -- 172.21.15.32:0/2670251967 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fc5b8012050 con 0x7fc5c010e650 2024-04-30T08:33:45.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.847+0000 7fc5bcc00700 1 --2- 172.21.15.32:0/2670251967 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc5a40afc10 0x7fc5a40b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:45.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.847+0000 7fc5bcc00700 1 -- 172.21.15.32:0/2670251967 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fc5b80dec90 con 0x7fc5c010e650 2024-04-30T08:33:45.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.847+0000 7fc5bfe00700 1 --2- 172.21.15.32:0/2670251967 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc5a40afc10 0x7fc5a40b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:45.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.848+0000 7fc5bfe00700 1 --2- 172.21.15.32:0/2670251967 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc5a40afc10 0x7fc5a40b2120 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0006010 tx=0x7fc5b000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:45.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:45.849+0000 7fc5bcc00700 1 -- 172.21.15.32:0/2670251967 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc5b8022060 con 0x7fc5c010e650 2024-04-30T08:33:46.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.095+0000 7fc5c5600700 1 -- 172.21.15.32:0/2670251967 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fc5c01a9b10 con 0x7fc5c010e650 2024-04-30T08:33:46.097 INFO:teuthology.orchestra.run.smithi032.stdout:180388626498 2024-04-30T08:33:46.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.096+0000 7fc5bcc00700 1 -- 172.21.15.32:0/2670251967 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fc5b8010040 con 0x7fc5c010e650 2024-04-30T08:33:46.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.100+0000 7fc5c5600700 1 -- 172.21.15.32:0/2670251967 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc5a40afc10 msgr2=0x7fc5a40b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:46.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.101+0000 7fc5c5600700 1 --2- 172.21.15.32:0/2670251967 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc5a40afc10 0x7fc5a40b2120 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0006010 tx=0x7fc5b000b260 comp rx=0 tx=0).stop 2024-04-30T08:33:46.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.101+0000 7fc5c5600700 1 -- 172.21.15.32:0/2670251967 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5c010e650 msgr2=0x7fc5c01a1690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:46.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.101+0000 7fc5c5600700 1 --2- 172.21.15.32:0/2670251967 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5c010e650 0x7fc5c01a1690 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800efb0 tx=0x7fc5b800c3a0 comp rx=0 tx=0).stop 2024-04-30T08:33:46.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.101+0000 7fc5c5600700 1 -- 172.21.15.32:0/2670251967 shutdown_connections 2024-04-30T08:33:46.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.101+0000 7fc5c5600700 1 --2- 172.21.15.32:0/2670251967 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc5a40afc10 0x7fc5a40b2120 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:46.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.101+0000 7fc5c5600700 1 --2- 172.21.15.32:0/2670251967 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc5c010e650 0x7fc5c01a1690 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:46.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.101+0000 7fc5c5600700 1 --2- 172.21.15.32:0/2670251967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc5c01051e0 0x7fc5c019ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:46.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.101+0000 7fc5c5600700 1 --2- 172.21.15.32:0/2670251967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5c0105b30 0x7fc5c019d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:46.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.101+0000 7fc5c5600700 1 -- 172.21.15.32:0/2670251967 >> 172.21.15.32:0/2670251967 conn(0x7fc5c0078a10 msgr2=0x7fc5c0102ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:46.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.101+0000 7fc5c5600700 1 -- 172.21.15.32:0/2670251967 shutdown_connections 2024-04-30T08:33:46.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.101+0000 7fc5c5600700 1 -- 172.21.15.32:0/2670251967 wait complete. 2024-04-30T08:33:46.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.648+0000 7fa9edc00700 1 -- 172.21.15.32:0/2234379715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9e81037d0 msgr2=0x7fa9e81104f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:46.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.648+0000 7fa9edc00700 1 --2- 172.21.15.32:0/2234379715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9e81037d0 0x7fa9e81104f0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4009990 tx=0x7fa9d4009ca0 comp rx=0 tx=0).stop 2024-04-30T08:33:46.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.648+0000 7fa9edc00700 1 -- 172.21.15.32:0/2234379715 shutdown_connections 2024-04-30T08:33:46.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.648+0000 7fa9edc00700 1 --2- 172.21.15.32:0/2234379715 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9e8110a30 0x7fa9e8112e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:46.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.648+0000 7fa9edc00700 1 --2- 172.21.15.32:0/2234379715 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9e8102e80 0x7fa9e8103290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:46.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.648+0000 7fa9edc00700 1 --2- 172.21.15.32:0/2234379715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9e81037d0 0x7fa9e81104f0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:46.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.648+0000 7fa9edc00700 1 -- 172.21.15.32:0/2234379715 >> 172.21.15.32:0/2234379715 conn(0x7fa9e8078a10 msgr2=0x7fa9e8078e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:46.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.648+0000 7fa9edc00700 1 -- 172.21.15.32:0/2234379715 shutdown_connections 2024-04-30T08:33:46.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.648+0000 7fa9edc00700 1 -- 172.21.15.32:0/2234379715 wait complete. 2024-04-30T08:33:46.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.649+0000 7fa9edc00700 1 Processor -- start 2024-04-30T08:33:46.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.650+0000 7fa9edc00700 1 -- start start 2024-04-30T08:33:46.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.650+0000 7fa9edc00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9e8102e80 0x7fa9e819ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:46.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.650+0000 7fa9edc00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9e81037d0 0x7fa9e819d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:46.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.650+0000 7fa9edc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9e8110a30 0x7fa9e81a1670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:46.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.650+0000 7fa9edc00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e8116470 con 0x7fa9e81037d0 2024-04-30T08:33:46.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.650+0000 7fa9edc00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e81162f0 con 0x7fa9e8102e80 2024-04-30T08:33:46.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.650+0000 7fa9edc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa9e81165f0 con 0x7fa9e8110a30 2024-04-30T08:33:46.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.650+0000 7fa9e7400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9e81037d0 0x7fa9e819d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:46.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.650+0000 7fa9e7400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9e81037d0 0x7fa9e819d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:47764/0 (socket says 172.21.15.32:47764) 2024-04-30T08:33:46.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.650+0000 7fa9e7400700 1 -- 172.21.15.32:0/2511233589 learned_addr learned my addr 172.21.15.32:0/2511233589 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:33:46.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.651+0000 7fa9ed200700 1 --2- 172.21.15.32:0/2511233589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9e8110a30 0x7fa9e81a1670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:46.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.651+0000 7fa9e7400700 1 -- 172.21.15.32:0/2511233589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9e8110a30 msgr2=0x7fa9e81a1670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:46.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.651+0000 7fa9e7400700 1 --2- 172.21.15.32:0/2511233589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9e8110a30 0x7fa9e81a1670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:46.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.651+0000 7fa9e7400700 1 -- 172.21.15.32:0/2511233589 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9e8102e80 msgr2=0x7fa9e819ce40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-30T08:33:46.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.651+0000 7fa9e7400700 1 --2- 172.21.15.32:0/2511233589 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9e8102e80 0x7fa9e819ce40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:46.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.651+0000 7fa9e7400700 1 -- 172.21.15.32:0/2511233589 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9e81a1d10 con 0x7fa9e81037d0 2024-04-30T08:33:46.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.651+0000 7fa9ed200700 1 --2- 172.21.15.32:0/2511233589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9e8110a30 0x7fa9e81a1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:33:46.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.651+0000 7fa9e7400700 1 --2- 172.21.15.32:0/2511233589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9e81037d0 0x7fa9e819d3e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4000c00 tx=0x7fa9d4005ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:46.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.651+0000 7fa9e4c00700 1 -- 172.21.15.32:0/2511233589 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4027070 con 0x7fa9e81037d0 2024-04-30T08:33:46.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.652+0000 7fa9edc00700 1 -- 172.21.15.32:0/2511233589 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9e81a1fa0 con 0x7fa9e81037d0 2024-04-30T08:33:46.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.652+0000 7fa9e4c00700 1 -- 172.21.15.32:0/2511233589 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa9d4022470 con 0x7fa9e81037d0 2024-04-30T08:33:46.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.652+0000 7fa9e4c00700 1 -- 172.21.15.32:0/2511233589 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4017630 con 0x7fa9e81037d0 2024-04-30T08:33:46.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.652+0000 7fa9edc00700 1 -- 172.21.15.32:0/2511233589 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9e81a9920 con 0x7fa9e81037d0 2024-04-30T08:33:46.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.653+0000 7fa9edc00700 1 -- 172.21.15.32:0/2511233589 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa9e8051210 con 0x7fa9e81037d0 2024-04-30T08:33:46.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.656+0000 7fa9e4c00700 1 -- 172.21.15.32:0/2511233589 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fa9d402a070 con 0x7fa9e81037d0 2024-04-30T08:33:46.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.658+0000 7fa9e4c00700 1 --2- 172.21.15.32:0/2511233589 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa9cc0afc10 0x7fa9cc0b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:46.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.658+0000 7fa9e4c00700 1 -- 172.21.15.32:0/2511233589 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fa9d40e0630 con 0x7fa9e81037d0 2024-04-30T08:33:46.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.658+0000 7fa9e7e00700 1 --2- 172.21.15.32:0/2511233589 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa9cc0afc10 0x7fa9cc0b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:46.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.659+0000 7fa9e7e00700 1 --2- 172.21.15.32:0/2511233589 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa9cc0afc10 0x7fa9cc0b2120 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8009710 tx=0x7fa9d8006c60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:46.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.661+0000 7fa9e4c00700 1 -- 172.21.15.32:0/2511233589 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa9d4013070 con 0x7fa9e81037d0 2024-04-30T08:33:46.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.889+0000 7fa9edc00700 1 -- 172.21.15.32:0/2511233589 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fa9e802e590 con 0x7fa9e81037d0 2024-04-30T08:33:46.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.890+0000 7fa9e4c00700 1 -- 172.21.15.32:0/2511233589 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fa9d40a8d30 con 0x7fa9e81037d0 2024-04-30T08:33:46.890 INFO:teuthology.orchestra.run.smithi032.stdout:77309411420 2024-04-30T08:33:46.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.892+0000 7fa9edc00700 1 -- 172.21.15.32:0/2511233589 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa9cc0afc10 msgr2=0x7fa9cc0b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:46.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.892+0000 7fa9edc00700 1 --2- 172.21.15.32:0/2511233589 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa9cc0afc10 0x7fa9cc0b2120 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fa9d8009710 tx=0x7fa9d8006c60 comp rx=0 tx=0).stop 2024-04-30T08:33:46.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.892+0000 7fa9edc00700 1 -- 172.21.15.32:0/2511233589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9e81037d0 msgr2=0x7fa9e819d3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:46.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.892+0000 7fa9edc00700 1 --2- 172.21.15.32:0/2511233589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9e81037d0 0x7fa9e819d3e0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4000c00 tx=0x7fa9d4005ef0 comp rx=0 tx=0).stop 2024-04-30T08:33:46.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.892+0000 7fa9edc00700 1 -- 172.21.15.32:0/2511233589 shutdown_connections 2024-04-30T08:33:46.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.892+0000 7fa9edc00700 1 --2- 172.21.15.32:0/2511233589 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa9cc0afc10 0x7fa9cc0b2120 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:46.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.892+0000 7fa9edc00700 1 --2- 172.21.15.32:0/2511233589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa9e8110a30 0x7fa9e81a1670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:46.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.892+0000 7fa9edc00700 1 --2- 172.21.15.32:0/2511233589 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa9e8102e80 0x7fa9e819ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:46.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.892+0000 7fa9edc00700 1 --2- 172.21.15.32:0/2511233589 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa9e81037d0 0x7fa9e819d3e0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:46.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.892+0000 7fa9edc00700 1 -- 172.21.15.32:0/2511233589 >> 172.21.15.32:0/2511233589 conn(0x7fa9e8078a10 msgr2=0x7fa9e8111d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:46.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.893+0000 7fa9edc00700 1 -- 172.21.15.32:0/2511233589 shutdown_connections 2024-04-30T08:33:46.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:46.893+0000 7fa9edc00700 1 -- 172.21.15.32:0/2511233589 wait complete. 2024-04-30T08:33:47.025 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037803 got 257698037807 for osd.9 2024-04-30T08:33:47.026 DEBUG:teuthology.parallel:result is None 2024-04-30T08:33:47.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:46 smithi032 ceph-mon[28331]: pgmap v383: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-30T08:33:47.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:46 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2670251967' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-30T08:33:47.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:46 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2511233589' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-30T08:33:47.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:46 smithi071 ceph-mon[33385]: pgmap v383: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-30T08:33:47.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:46 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2670251967' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-30T08:33:47.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:46 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2511233589' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-30T08:33:47.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:46 smithi189 ceph-mon[33038]: pgmap v383: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-30T08:33:47.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:46 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2670251967' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-30T08:33:47.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:46 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2511233589' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-30T08:33:47.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.700+0000 7f950fe00700 1 -- 172.21.15.32:0/1821010282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9508106e00 msgr2=0x7f9508113290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:47.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.700+0000 7f950fe00700 1 --2- 172.21.15.32:0/1821010282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9508106e00 0x7f9508113290 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9500009f90 tx=0x7f950000b330 comp rx=0 tx=0).stop 2024-04-30T08:33:47.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.700+0000 7f950fe00700 1 -- 172.21.15.32:0/1821010282 shutdown_connections 2024-04-30T08:33:47.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.700+0000 7f950fe00700 1 --2- 172.21.15.32:0/1821010282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9508106e00 0x7f9508113290 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:47.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.700+0000 7f950fe00700 1 --2- 172.21.15.32:0/1821010282 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f950810c4f0 0x7f950810c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:47.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.700+0000 7f950fe00700 1 --2- 172.21.15.32:0/1821010282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95081064b0 0x7f95081068c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:47.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.700+0000 7f950fe00700 1 -- 172.21.15.32:0/1821010282 >> 172.21.15.32:0/1821010282 conn(0x7f9508101fb0 msgr2=0x7f9508104400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:47.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.700+0000 7f950fe00700 1 -- 172.21.15.32:0/1821010282 shutdown_connections 2024-04-30T08:33:47.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.700+0000 7f950fe00700 1 -- 172.21.15.32:0/1821010282 wait complete. 2024-04-30T08:33:47.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.702+0000 7f950fe00700 1 Processor -- start 2024-04-30T08:33:47.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.702+0000 7f950fe00700 1 -- start start 2024-04-30T08:33:47.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.703+0000 7f950fe00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95081064b0 0x7f950819cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:47.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.703+0000 7f950fe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9508106e00 0x7f950819d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:47.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.703+0000 7f950fe00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f950810c4f0 0x7f95081a1770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:47.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.703+0000 7f950fe00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9508116670 con 0x7f95081064b0 2024-04-30T08:33:47.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.703+0000 7f950fe00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f95081164f0 con 0x7f950810c4f0 2024-04-30T08:33:47.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.703+0000 7f950fe00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f95081167f0 con 0x7f9508106e00 2024-04-30T08:33:47.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.703+0000 7f950ca00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9508106e00 0x7f950819d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:47.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.703+0000 7f950d400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95081064b0 0x7f950819cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:47.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.703+0000 7f950ca00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9508106e00 0x7f950819d4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:42772/0 (socket says 172.21.15.32:42772) 2024-04-30T08:33:47.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.703+0000 7f950ca00700 1 -- 172.21.15.32:0/1206544501 learned_addr learned my addr 172.21.15.32:0/1206544501 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:33:47.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.703+0000 7f950de00700 1 --2- 172.21.15.32:0/1206544501 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f950810c4f0 0x7f95081a1770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:47.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.704+0000 7f950ca00700 1 -- 172.21.15.32:0/1206544501 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f950810c4f0 msgr2=0x7f95081a1770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:47.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.704+0000 7f950ca00700 1 --2- 172.21.15.32:0/1206544501 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f950810c4f0 0x7f95081a1770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:47.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.704+0000 7f950ca00700 1 -- 172.21.15.32:0/1206544501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95081064b0 msgr2=0x7f950819cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:47.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.704+0000 7f950ca00700 1 --2- 172.21.15.32:0/1206544501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95081064b0 0x7f950819cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:47.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.704+0000 7f950ca00700 1 -- 172.21.15.32:0/1206544501 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95081a1e10 con 0x7f9508106e00 2024-04-30T08:33:47.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.704+0000 7f950d400700 1 --2- 172.21.15.32:0/1206544501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95081064b0 0x7f950819cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:33:47.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.704+0000 7f950ca00700 1 --2- 172.21.15.32:0/1206544501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9508106e00 0x7f950819d4e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f94f400e940 tx=0x7f94f400ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:47.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.704+0000 7f9506000700 1 -- 172.21.15.32:0/1206544501 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f400c9e0 con 0x7f9508106e00 2024-04-30T08:33:47.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.704+0000 7f9506000700 1 -- 172.21.15.32:0/1206544501 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94f400cb80 con 0x7f9508106e00 2024-04-30T08:33:47.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.704+0000 7f9506000700 1 -- 172.21.15.32:0/1206544501 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f4010e60 con 0x7f9508106e00 2024-04-30T08:33:47.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.705+0000 7f950fe00700 1 -- 172.21.15.32:0/1206544501 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95081a2100 con 0x7f9508106e00 2024-04-30T08:33:47.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.705+0000 7f950fe00700 1 -- 172.21.15.32:0/1206544501 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95081a9a20 con 0x7f9508106e00 2024-04-30T08:33:47.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.706+0000 7f950fe00700 1 -- 172.21.15.32:0/1206544501 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9508051210 con 0x7f9508106e00 2024-04-30T08:33:47.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.708+0000 7f9506000700 1 -- 172.21.15.32:0/1206544501 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f94f4010430 con 0x7f9508106e00 2024-04-30T08:33:47.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.710+0000 7f9506000700 1 --2- 172.21.15.32:0/1206544501 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f94f00afc10 0x7f94f00b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:47.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.710+0000 7f9506000700 1 -- 172.21.15.32:0/1206544501 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f94f40df810 con 0x7f9508106e00 2024-04-30T08:33:47.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.710+0000 7f950d400700 1 --2- 172.21.15.32:0/1206544501 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f94f00afc10 0x7f94f00b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:47.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.711+0000 7f950d400700 1 --2- 172.21.15.32:0/1206544501 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f94f00afc10 0x7f94f00b2120 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f94fc009750 tx=0x7f94fc009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:47.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.712+0000 7f9506000700 1 -- 172.21.15.32:0/1206544501 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f94f40a7e80 con 0x7f9508106e00 2024-04-30T08:33:47.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.947+0000 7f950fe00700 1 -- 172.21.15.32:0/1206544501 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f950802e590 con 0x7f9508106e00 2024-04-30T08:33:47.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.948+0000 7f9506000700 1 -- 172.21.15.32:0/1206544501 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f94f40a7780 con 0x7f9508106e00 2024-04-30T08:33:47.949 INFO:teuthology.orchestra.run.smithi032.stdout:133143986255 2024-04-30T08:33:47.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.951+0000 7f950fe00700 1 -- 172.21.15.32:0/1206544501 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f94f00afc10 msgr2=0x7f94f00b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:47.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.951+0000 7f950fe00700 1 --2- 172.21.15.32:0/1206544501 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f94f00afc10 0x7f94f00b2120 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f94fc009750 tx=0x7f94fc009310 comp rx=0 tx=0).stop 2024-04-30T08:33:47.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.951+0000 7f950fe00700 1 -- 172.21.15.32:0/1206544501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9508106e00 msgr2=0x7f950819d4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:47.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.951+0000 7f950fe00700 1 --2- 172.21.15.32:0/1206544501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9508106e00 0x7f950819d4e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f94f400e940 tx=0x7f94f400ec50 comp rx=0 tx=0).stop 2024-04-30T08:33:47.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.951+0000 7f950fe00700 1 -- 172.21.15.32:0/1206544501 shutdown_connections 2024-04-30T08:33:47.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.951+0000 7f950fe00700 1 --2- 172.21.15.32:0/1206544501 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f94f00afc10 0x7f94f00b2120 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:47.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.951+0000 7f950fe00700 1 --2- 172.21.15.32:0/1206544501 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f950810c4f0 0x7f95081a1770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:47.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.951+0000 7f950fe00700 1 --2- 172.21.15.32:0/1206544501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f95081064b0 0x7f950819cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:47.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.951+0000 7f950fe00700 1 --2- 172.21.15.32:0/1206544501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9508106e00 0x7f950819d4e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:47.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.951+0000 7f950fe00700 1 -- 172.21.15.32:0/1206544501 >> 172.21.15.32:0/1206544501 conn(0x7f9508101fb0 msgr2=0x7f9508104530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:47.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.952+0000 7f950fe00700 1 -- 172.21.15.32:0/1206544501 shutdown_connections 2024-04-30T08:33:47.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:47.952+0000 7f950fe00700 1 -- 172.21.15.32:0/1206544501 wait complete. 2024-04-30T08:33:48.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:47 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1206544501' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-30T08:33:48.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:47 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1206544501' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-30T08:33:48.341 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:47 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1206544501' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-30T08:33:48.873 INFO:tasks.cephadm.ceph_manager.ceph:need seq 304942678049 got 304942678053 for osd.11 2024-04-30T08:33:48.873 DEBUG:teuthology.parallel:result is None 2024-04-30T08:33:49.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:48 smithi032 ceph-mon[28331]: pgmap v384: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-30T08:33:49.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:48 smithi071 ceph-mon[33385]: pgmap v384: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-30T08:33:49.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:48 smithi189 ceph-mon[33038]: pgmap v384: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-30T08:33:49.327 INFO:tasks.cephadm.ceph_manager.ceph:need seq 180388626492 got 180388626498 for osd.6 2024-04-30T08:33:49.327 DEBUG:teuthology.parallel:result is None 2024-04-30T08:33:49.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.871+0000 7f10e8e00700 1 -- 172.21.15.32:0/792249671 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10e410b230 msgr2=0x7f10e4117760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:49.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.871+0000 7f10e8e00700 1 --2- 172.21.15.32:0/792249671 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10e410b230 0x7f10e4117760 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f10dc009f90 tx=0x7f10dc00b330 comp rx=0 tx=0).stop 2024-04-30T08:33:49.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.871+0000 7f10e8e00700 1 -- 172.21.15.32:0/792249671 shutdown_connections 2024-04-30T08:33:49.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.871+0000 7f10e8e00700 1 --2- 172.21.15.32:0/792249671 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10e410b230 0x7f10e4117760 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:49.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.871+0000 7f10e8e00700 1 --2- 172.21.15.32:0/792249671 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10e41108c0 0x7f10e4110cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:49.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.871+0000 7f10e8e00700 1 --2- 172.21.15.32:0/792249671 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10e410a8e0 0x7f10e410acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:49.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.871+0000 7f10e8e00700 1 -- 172.21.15.32:0/792249671 >> 172.21.15.32:0/792249671 conn(0x7f10e4077bd0 msgr2=0x7f10e4078030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:49.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.871+0000 7f10e8e00700 1 -- 172.21.15.32:0/792249671 shutdown_connections 2024-04-30T08:33:49.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.871+0000 7f10e8e00700 1 -- 172.21.15.32:0/792249671 wait complete. 2024-04-30T08:33:49.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.872+0000 7f10e8e00700 1 Processor -- start 2024-04-30T08:33:49.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.872+0000 7f10e8e00700 1 -- start start 2024-04-30T08:33:49.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.873+0000 7f10e8e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10e410a8e0 0x7f10e41a1360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:49.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.873+0000 7f10e8e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10e410b230 0x7f10e41a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:49.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.873+0000 7f10e8e00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10e41108c0 0x7f10e41a5b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:49.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.873+0000 7f10e8e00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f10e411aa30 con 0x7f10e410b230 2024-04-30T08:33:49.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.873+0000 7f10e8e00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f10e411a8b0 con 0x7f10e41108c0 2024-04-30T08:33:49.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.873+0000 7f10e8e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f10e411abb0 con 0x7f10e410a8e0 2024-04-30T08:33:49.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.873+0000 7f10dbe00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10e410b230 0x7f10e41a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:49.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.873+0000 7f10e1e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10e410a8e0 0x7f10e41a1360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:49.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.873+0000 7f10e2800700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10e41108c0 0x7f10e41a5b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:49.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.873+0000 7f10e1e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10e410a8e0 0x7f10e41a1360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:42796/0 (socket says 172.21.15.32:42796) 2024-04-30T08:33:49.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.873+0000 7f10e1e00700 1 -- 172.21.15.32:0/1391672720 learned_addr learned my addr 172.21.15.32:0/1391672720 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:33:49.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.873+0000 7f10dbe00700 1 --2- 172.21.15.32:0/1391672720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10e410b230 0x7f10e41a1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55092/0 (socket says 172.21.15.32:55092) 2024-04-30T08:33:49.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.874+0000 7f10e2800700 1 -- 172.21.15.32:0/1391672720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10e410a8e0 msgr2=0x7f10e41a1360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:49.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.874+0000 7f10e2800700 1 --2- 172.21.15.32:0/1391672720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10e410a8e0 0x7f10e41a1360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:49.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.874+0000 7f10e2800700 1 -- 172.21.15.32:0/1391672720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10e410b230 msgr2=0x7f10e41a1900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:49.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.874+0000 7f10e2800700 1 --2- 172.21.15.32:0/1391672720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10e410b230 0x7f10e41a1900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:49.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.874+0000 7f10e2800700 1 -- 172.21.15.32:0/1391672720 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10e41a6230 con 0x7f10e41108c0 2024-04-30T08:33:49.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.874+0000 7f10e2800700 1 --2- 172.21.15.32:0/1391672720 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10e41108c0 0x7f10e41a5b90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f10dc00b6c0 tx=0x7f10dc00cc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:49.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.874+0000 7f10daa00700 1 -- 172.21.15.32:0/1391672720 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10dc021070 con 0x7f10e41108c0 2024-04-30T08:33:49.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.874+0000 7f10e8e00700 1 -- 172.21.15.32:0/1391672720 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10e41a64c0 con 0x7f10e41108c0 2024-04-30T08:33:49.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.874+0000 7f10daa00700 1 -- 172.21.15.32:0/1391672720 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f10dc007d60 con 0x7f10e41108c0 2024-04-30T08:33:49.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.875+0000 7f10daa00700 1 -- 172.21.15.32:0/1391672720 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10dc005040 con 0x7f10e41108c0 2024-04-30T08:33:49.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.875+0000 7f10e8e00700 1 -- 172.21.15.32:0/1391672720 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10e41ade40 con 0x7f10e41108c0 2024-04-30T08:33:49.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.876+0000 7f10e8e00700 1 -- 172.21.15.32:0/1391672720 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10e410bfd0 con 0x7f10e41108c0 2024-04-30T08:33:49.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.878+0000 7f10daa00700 1 -- 172.21.15.32:0/1391672720 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f10dc024070 con 0x7f10e41108c0 2024-04-30T08:33:49.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.880+0000 7f10daa00700 1 --2- 172.21.15.32:0/1391672720 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f10c80afc10 0x7f10c80b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:49.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.880+0000 7f10daa00700 1 -- 172.21.15.32:0/1391672720 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f10dc00f020 con 0x7f10e41108c0 2024-04-30T08:33:49.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.880+0000 7f10e1e00700 1 --2- 172.21.15.32:0/1391672720 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f10c80afc10 0x7f10c80b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:49.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.881+0000 7f10e1e00700 1 --2- 172.21.15.32:0/1391672720 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f10c80afc10 0x7f10c80b2120 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f10d4009750 tx=0x7f10d4009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:49.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:49.882+0000 7f10daa00700 1 -- 172.21.15.32:0/1391672720 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f10dc02b020 con 0x7f10e41108c0 2024-04-30T08:33:50.121 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.120+0000 7f10e8e00700 1 -- 172.21.15.32:0/1391672720 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f10e41a26c0 con 0x7f10e41108c0 2024-04-30T08:33:50.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.121+0000 7f10daa00700 1 -- 172.21.15.32:0/1391672720 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f10dc0a9200 con 0x7f10e41108c0 2024-04-30T08:33:50.122 INFO:teuthology.orchestra.run.smithi032.stdout:279172874283 2024-04-30T08:33:50.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.125+0000 7f10e8e00700 1 -- 172.21.15.32:0/1391672720 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f10c80afc10 msgr2=0x7f10c80b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:50.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.125+0000 7f10e8e00700 1 --2- 172.21.15.32:0/1391672720 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f10c80afc10 0x7f10c80b2120 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f10d4009750 tx=0x7f10d4009310 comp rx=0 tx=0).stop 2024-04-30T08:33:50.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.125+0000 7f10e8e00700 1 -- 172.21.15.32:0/1391672720 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10e41108c0 msgr2=0x7f10e41a5b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:50.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.125+0000 7f10e8e00700 1 --2- 172.21.15.32:0/1391672720 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10e41108c0 0x7f10e41a5b90 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f10dc00b6c0 tx=0x7f10dc00cc30 comp rx=0 tx=0).stop 2024-04-30T08:33:50.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.125+0000 7f10e8e00700 1 -- 172.21.15.32:0/1391672720 shutdown_connections 2024-04-30T08:33:50.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.125+0000 7f10e8e00700 1 --2- 172.21.15.32:0/1391672720 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f10c80afc10 0x7f10c80b2120 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:50.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.125+0000 7f10e8e00700 1 --2- 172.21.15.32:0/1391672720 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f10e41108c0 0x7f10e41a5b90 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:50.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.125+0000 7f10e8e00700 1 --2- 172.21.15.32:0/1391672720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10e410a8e0 0x7f10e41a1360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:50.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.125+0000 7f10e8e00700 1 --2- 172.21.15.32:0/1391672720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f10e410b230 0x7f10e41a1900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:50.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.125+0000 7f10e8e00700 1 -- 172.21.15.32:0/1391672720 >> 172.21.15.32:0/1391672720 conn(0x7f10e4077bd0 msgr2=0x7f10e407b1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:50.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.125+0000 7f10e8e00700 1 -- 172.21.15.32:0/1391672720 shutdown_connections 2024-04-30T08:33:50.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.125+0000 7f10e8e00700 1 -- 172.21.15.32:0/1391672720 wait complete. 2024-04-30T08:33:50.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.851+0000 7fbadd000700 1 -- 172.21.15.32:0/227541404 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbad8105d00 msgr2=0x7fbad810e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:50.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.851+0000 7fbadd000700 1 --2- 172.21.15.32:0/227541404 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbad8105d00 0x7fbad810e250 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fbac4009990 tx=0x7fbac4009ca0 comp rx=0 tx=0).stop 2024-04-30T08:33:50.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.851+0000 7fbadd000700 1 -- 172.21.15.32:0/227541404 shutdown_connections 2024-04-30T08:33:50.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.851+0000 7fbadd000700 1 --2- 172.21.15.32:0/227541404 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbad810e820 0x7fbad8110d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:50.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.851+0000 7fbadd000700 1 --2- 172.21.15.32:0/227541404 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbad81053b0 0x7fbad81057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:50.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.851+0000 7fbadd000700 1 --2- 172.21.15.32:0/227541404 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbad8105d00 0x7fbad810e250 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:50.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.851+0000 7fbadd000700 1 -- 172.21.15.32:0/227541404 >> 172.21.15.32:0/227541404 conn(0x7fbad80fed10 msgr2=0x7fbad8101130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:50.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.852+0000 7fbadd000700 1 -- 172.21.15.32:0/227541404 shutdown_connections 2024-04-30T08:33:50.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.852+0000 7fbadd000700 1 -- 172.21.15.32:0/227541404 wait complete. 2024-04-30T08:33:50.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.853+0000 7fbadd000700 1 Processor -- start 2024-04-30T08:33:50.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.853+0000 7fbadd000700 1 -- start start 2024-04-30T08:33:50.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.853+0000 7fbadd000700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbad81053b0 0x7fbad819cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:50.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.853+0000 7fbadd000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbad8105d00 0x7fbad819d4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:50.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.854+0000 7fbadd000700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbad810e820 0x7fbad81a1760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:50.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.854+0000 7fbadd000700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbad8114380 con 0x7fbad810e820 2024-04-30T08:33:50.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.854+0000 7fbadd000700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fbad8114200 con 0x7fbad81053b0 2024-04-30T08:33:50.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.854+0000 7fbadd000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbad8114500 con 0x7fbad8105d00 2024-04-30T08:33:50.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.854+0000 7fbad6800700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbad810e820 0x7fbad81a1760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:50.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.854+0000 7fbad5e00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbad81053b0 0x7fbad819cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:50.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.854+0000 7fbad6800700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbad810e820 0x7fbad81a1760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55096/0 (socket says 172.21.15.32:55096) 2024-04-30T08:33:50.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.854+0000 7fbad6800700 1 -- 172.21.15.32:0/3170965012 learned_addr learned my addr 172.21.15.32:0/3170965012 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:33:50.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.854+0000 7fbad5400700 1 --2- 172.21.15.32:0/3170965012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbad8105d00 0x7fbad819d4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:50.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.854+0000 7fbad5e00700 1 -- 172.21.15.32:0/3170965012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbad8105d00 msgr2=0x7fbad819d4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:50.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.854+0000 7fbad5e00700 1 --2- 172.21.15.32:0/3170965012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbad8105d00 0x7fbad819d4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:50.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.854+0000 7fbad5e00700 1 -- 172.21.15.32:0/3170965012 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbad810e820 msgr2=0x7fbad81a1760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:50.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.854+0000 7fbad5e00700 1 --2- 172.21.15.32:0/3170965012 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbad810e820 0x7fbad81a1760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:50.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.854+0000 7fbad5e00700 1 -- 172.21.15.32:0/3170965012 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbad81a1e00 con 0x7fbad81053b0 2024-04-30T08:33:50.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.854+0000 7fbad5400700 1 --2- 172.21.15.32:0/3170965012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbad8105d00 0x7fbad819d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:33:50.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.855+0000 7fbad5e00700 1 --2- 172.21.15.32:0/3170965012 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbad81053b0 0x7fbad819cf30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fbac800e9f0 tx=0x7fbac800ed00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:50.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.855+0000 7fbacea00700 1 -- 172.21.15.32:0/3170965012 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac800c9c0 con 0x7fbad81053b0 2024-04-30T08:33:50.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.855+0000 7fbacea00700 1 -- 172.21.15.32:0/3170965012 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbac800cb60 con 0x7fbad81053b0 2024-04-30T08:33:50.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.855+0000 7fbacea00700 1 -- 172.21.15.32:0/3170965012 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbac8018680 con 0x7fbad81053b0 2024-04-30T08:33:50.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.855+0000 7fbadd000700 1 -- 172.21.15.32:0/3170965012 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbad81a20f0 con 0x7fbad81053b0 2024-04-30T08:33:50.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.855+0000 7fbadd000700 1 -- 172.21.15.32:0/3170965012 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbad81a9a10 con 0x7fbad81053b0 2024-04-30T08:33:50.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.857+0000 7fbadd000700 1 -- 172.21.15.32:0/3170965012 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbad806be40 con 0x7fbad81053b0 2024-04-30T08:33:50.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.859+0000 7fbacea00700 1 -- 172.21.15.32:0/3170965012 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fbac8010430 con 0x7fbad81053b0 2024-04-30T08:33:50.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.861+0000 7fbacea00700 1 --2- 172.21.15.32:0/3170965012 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbabc0afc90 0x7fbabc0b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:50.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.861+0000 7fbacea00700 1 -- 172.21.15.32:0/3170965012 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fbac80df600 con 0x7fbad81053b0 2024-04-30T08:33:50.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.861+0000 7fbad5400700 1 --2- 172.21.15.32:0/3170965012 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbabc0afc90 0x7fbabc0b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:50.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.862+0000 7fbad5400700 1 --2- 172.21.15.32:0/3170965012 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbabc0afc90 0x7fbabc0b21a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbac40098e0 tx=0x7fbac4005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:50.937 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:50.862+0000 7fbacea00700 1 -- 172.21.15.32:0/3170965012 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbac80a7c70 con 0x7fbad81053b0 2024-04-30T08:33:51.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:51.098+0000 7fbadd000700 1 -- 172.21.15.32:0/3170965012 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fbad802e590 con 0x7fbad81053b0 2024-04-30T08:33:51.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:51.099+0000 7fbacea00700 1 -- 172.21.15.32:0/3170965012 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fbac80a7570 con 0x7fbad81053b0 2024-04-30T08:33:51.100 INFO:teuthology.orchestra.run.smithi032.stdout:206158430270 2024-04-30T08:33:51.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:51.102+0000 7fbadd000700 1 -- 172.21.15.32:0/3170965012 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbabc0afc90 msgr2=0x7fbabc0b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:51.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:51.102+0000 7fbadd000700 1 --2- 172.21.15.32:0/3170965012 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbabc0afc90 0x7fbabc0b21a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fbac40098e0 tx=0x7fbac4005960 comp rx=0 tx=0).stop 2024-04-30T08:33:51.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:51.102+0000 7fbadd000700 1 -- 172.21.15.32:0/3170965012 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbad81053b0 msgr2=0x7fbad819cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:51.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:51.102+0000 7fbadd000700 1 --2- 172.21.15.32:0/3170965012 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbad81053b0 0x7fbad819cf30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fbac800e9f0 tx=0x7fbac800ed00 comp rx=0 tx=0).stop 2024-04-30T08:33:51.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:51.102+0000 7fbadd000700 1 -- 172.21.15.32:0/3170965012 shutdown_connections 2024-04-30T08:33:51.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:51.102+0000 7fbadd000700 1 --2- 172.21.15.32:0/3170965012 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbabc0afc90 0x7fbabc0b21a0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:51.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:51.102+0000 7fbadd000700 1 --2- 172.21.15.32:0/3170965012 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbad810e820 0x7fbad81a1760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:51.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:51.102+0000 7fbadd000700 1 --2- 172.21.15.32:0/3170965012 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbad81053b0 0x7fbad819cf30 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:51.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:51.102+0000 7fbadd000700 1 --2- 172.21.15.32:0/3170965012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbad8105d00 0x7fbad819d4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:51.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:51.102+0000 7fbadd000700 1 -- 172.21.15.32:0/3170965012 >> 172.21.15.32:0/3170965012 conn(0x7fbad80fed10 msgr2=0x7fbad81030f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:51.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:51.102+0000 7fbadd000700 1 -- 172.21.15.32:0/3170965012 shutdown_connections 2024-04-30T08:33:51.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:51.102+0000 7fbadd000700 1 -- 172.21.15.32:0/3170965012 wait complete. 2024-04-30T08:33:51.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:51 smithi071 ceph-mon[33385]: pgmap v385: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-30T08:33:51.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:51 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1391672720' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-30T08:33:51.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:51 smithi189 ceph-mon[33038]: pgmap v385: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-30T08:33:51.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:51 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1391672720' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-30T08:33:51.346 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:51 smithi032 ceph-mon[28331]: pgmap v385: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-30T08:33:51.346 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:51 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1391672720' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-30T08:33:51.683 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182480 got 107374182484 for osd.3 2024-04-30T08:33:51.683 DEBUG:teuthology.parallel:result is None 2024-04-30T08:33:52.281 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:52 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3170965012' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-30T08:33:52.282 INFO:tasks.cephadm.ceph_manager.ceph:need seq 279172874278 got 279172874283 for osd.10 2024-04-30T08:33:52.283 DEBUG:teuthology.parallel:result is None 2024-04-30T08:33:52.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:52 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3170965012' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-30T08:33:52.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:52 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3170965012' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-30T08:33:52.646 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986247 got 133143986255 for osd.4 2024-04-30T08:33:52.646 DEBUG:teuthology.parallel:result is None 2024-04-30T08:33:53.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:53 smithi032 ceph-mon[28331]: pgmap v386: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-30T08:33:53.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:53 smithi189 ceph-mon[33038]: pgmap v386: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-30T08:33:53.422 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411413 got 77309411420 for osd.2 2024-04-30T08:33:53.423 DEBUG:teuthology.parallel:result is None 2024-04-30T08:33:53.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:53 smithi071 ceph-mon[33385]: pgmap v386: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-30T08:33:53.808 INFO:tasks.cephadm.ceph_manager.ceph:need seq 206158430264 got 206158430270 for osd.7 2024-04-30T08:33:53.808 DEBUG:teuthology.parallel:result is None 2024-04-30T08:33:55.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.073+0000 7fd423200700 1 -- 172.21.15.32:0/2362589601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd41c105d00 msgr2=0x7fd41c10e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:55.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.073+0000 7fd423200700 1 --2- 172.21.15.32:0/2362589601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd41c105d00 0x7fd41c10e250 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fd408009990 tx=0x7fd408009ca0 comp rx=0 tx=0).stop 2024-04-30T08:33:55.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.073+0000 7fd423200700 1 -- 172.21.15.32:0/2362589601 shutdown_connections 2024-04-30T08:33:55.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.073+0000 7fd423200700 1 --2- 172.21.15.32:0/2362589601 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd41c10e820 0x7fd41c110d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:55.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.073+0000 7fd423200700 1 --2- 172.21.15.32:0/2362589601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd41c1053b0 0x7fd41c1057c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:55.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.073+0000 7fd423200700 1 --2- 172.21.15.32:0/2362589601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd41c105d00 0x7fd41c10e250 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:55.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.073+0000 7fd423200700 1 -- 172.21.15.32:0/2362589601 >> 172.21.15.32:0/2362589601 conn(0x7fd41c0fed10 msgr2=0x7fd41c101130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:55.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.074+0000 7fd423200700 1 -- 172.21.15.32:0/2362589601 shutdown_connections 2024-04-30T08:33:55.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.074+0000 7fd423200700 1 -- 172.21.15.32:0/2362589601 wait complete. 2024-04-30T08:33:55.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.075+0000 7fd423200700 1 Processor -- start 2024-04-30T08:33:55.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.075+0000 7fd423200700 1 -- start start 2024-04-30T08:33:55.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.075+0000 7fd423200700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd41c1053b0 0x7fd41c19ad10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:55.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.076+0000 7fd423200700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd41c105d00 0x7fd41c19b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:55.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.076+0000 7fd423200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd41c10e820 0x7fd41c19f540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:55.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.076+0000 7fd423200700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd41c114380 con 0x7fd41c1053b0 2024-04-30T08:33:55.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.076+0000 7fd423200700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd41c114200 con 0x7fd41c105d00 2024-04-30T08:33:55.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.076+0000 7fd423200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd41c114500 con 0x7fd41c10e820 2024-04-30T08:33:55.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.076+0000 7fd41be00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd41c1053b0 0x7fd41c19ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:55.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.076+0000 7fd41be00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd41c1053b0 0x7fd41c19ad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55112/0 (socket says 172.21.15.32:55112) 2024-04-30T08:33:55.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.076+0000 7fd41be00700 1 -- 172.21.15.32:0/2886561981 learned_addr learned my addr 172.21.15.32:0/2886561981 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:33:55.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.076+0000 7fd41b400700 1 --2- 172.21.15.32:0/2886561981 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd41c105d00 0x7fd41c19b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:55.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.076+0000 7fd421200700 1 --2- 172.21.15.32:0/2886561981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd41c10e820 0x7fd41c19f540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:55.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.076+0000 7fd41be00700 1 -- 172.21.15.32:0/2886561981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd41c10e820 msgr2=0x7fd41c19f540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:55.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.076+0000 7fd41be00700 1 --2- 172.21.15.32:0/2886561981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd41c10e820 0x7fd41c19f540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:55.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.076+0000 7fd41be00700 1 -- 172.21.15.32:0/2886561981 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd41c105d00 msgr2=0x7fd41c19b2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:55.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.076+0000 7fd41be00700 1 --2- 172.21.15.32:0/2886561981 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd41c105d00 0x7fd41c19b2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:55.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.077+0000 7fd41be00700 1 -- 172.21.15.32:0/2886561981 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd41c19fbe0 con 0x7fd41c1053b0 2024-04-30T08:33:55.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.077+0000 7fd41be00700 1 --2- 172.21.15.32:0/2886561981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd41c1053b0 0x7fd41c19ad10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd40c00e9a0 tx=0x7fd40c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:55.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.077+0000 7fd418c00700 1 -- 172.21.15.32:0/2886561981 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd40c00ca10 con 0x7fd41c1053b0 2024-04-30T08:33:55.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.077+0000 7fd423200700 1 -- 172.21.15.32:0/2886561981 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd41c19fed0 con 0x7fd41c1053b0 2024-04-30T08:33:55.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.077+0000 7fd418c00700 1 -- 172.21.15.32:0/2886561981 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd40c00cbb0 con 0x7fd41c1053b0 2024-04-30T08:33:55.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.077+0000 7fd423200700 1 -- 172.21.15.32:0/2886561981 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd41c1a77f0 con 0x7fd41c1053b0 2024-04-30T08:33:55.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.077+0000 7fd418c00700 1 -- 172.21.15.32:0/2886561981 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd40c018620 con 0x7fd41c1053b0 2024-04-30T08:33:55.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.079+0000 7fd423200700 1 -- 172.21.15.32:0/2886561981 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd41c051210 con 0x7fd41c1053b0 2024-04-30T08:33:55.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.083+0000 7fd418c00700 1 -- 172.21.15.32:0/2886561981 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fd40c010430 con 0x7fd41c1053b0 2024-04-30T08:33:55.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.085+0000 7fd418c00700 1 --2- 172.21.15.32:0/2886561981 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd4000afc10 0x7fd4000b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:55.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.085+0000 7fd418c00700 1 -- 172.21.15.32:0/2886561981 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fd40c0df630 con 0x7fd41c1053b0 2024-04-30T08:33:55.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.085+0000 7fd41b400700 1 --2- 172.21.15.32:0/2886561981 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd4000afc10 0x7fd4000b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:55.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.086+0000 7fd41b400700 1 --2- 172.21.15.32:0/2886561981 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd4000afc10 0x7fd4000b2120 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd40800b2e0 tx=0x7fd408005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:55.163 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.088+0000 7fd418c00700 1 -- 172.21.15.32:0/2886561981 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd40c0a7ce0 con 0x7fd41c1053b0 2024-04-30T08:33:55.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.173+0000 7f9735a00700 1 -- 172.21.15.32:0/2541564102 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9730104520 msgr2=0x7f9730104930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:55.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.173+0000 7f9735a00700 1 --2- 172.21.15.32:0/2541564102 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9730104520 0x7f9730104930 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f9720009990 tx=0x7f9720009ca0 comp rx=0 tx=0).stop 2024-04-30T08:33:55.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.173+0000 7f9735a00700 1 -- 172.21.15.32:0/2541564102 shutdown_connections 2024-04-30T08:33:55.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.173+0000 7f9735a00700 1 --2- 172.21.15.32:0/2541564102 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97301097a0 0x7f9730111100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:55.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.173+0000 7f9735a00700 1 --2- 172.21.15.32:0/2541564102 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9730104520 0x7f9730104930 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:55.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.173+0000 7f9735a00700 1 --2- 172.21.15.32:0/2541564102 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9730108e50 0x7f9730109260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:55.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.173+0000 7f9735a00700 1 -- 172.21.15.32:0/2541564102 >> 172.21.15.32:0/2541564102 conn(0x7f97300ffdf0 msgr2=0x7f9730102240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:55.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.173+0000 7f9735a00700 1 -- 172.21.15.32:0/2541564102 shutdown_connections 2024-04-30T08:33:55.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.173+0000 7f9735a00700 1 -- 172.21.15.32:0/2541564102 wait complete. 2024-04-30T08:33:55.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.175+0000 7f9735a00700 1 Processor -- start 2024-04-30T08:33:55.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.175+0000 7f9735a00700 1 -- start start 2024-04-30T08:33:55.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.175+0000 7f9735a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9730104520 0x7f973019adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:55.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.175+0000 7f9735a00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9730108e50 0x7f973019b350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:55.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.175+0000 7f9735a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97301097a0 0x7f973019f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:55.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.175+0000 7f9735a00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9730114420 con 0x7f9730104520 2024-04-30T08:33:55.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.175+0000 7f9735a00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f97301142a0 con 0x7f9730108e50 2024-04-30T08:33:55.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.175+0000 7f9735a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f97301145a0 con 0x7f97301097a0 2024-04-30T08:33:55.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.176+0000 7f972ea00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9730104520 0x7f973019adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:55.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.176+0000 7f972ea00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9730104520 0x7f973019adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:55136/0 (socket says 172.21.15.32:55136) 2024-04-30T08:33:55.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.176+0000 7f972ea00700 1 -- 172.21.15.32:0/3420008988 learned_addr learned my addr 172.21.15.32:0/3420008988 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:33:55.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.176+0000 7f972f400700 1 --2- 172.21.15.32:0/3420008988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97301097a0 0x7f973019f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:55.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.176+0000 7f972e000700 1 --2- 172.21.15.32:0/3420008988 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9730108e50 0x7f973019b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:55.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.176+0000 7f972ea00700 1 -- 172.21.15.32:0/3420008988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97301097a0 msgr2=0x7f973019f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:55.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.176+0000 7f972ea00700 1 --2- 172.21.15.32:0/3420008988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97301097a0 0x7f973019f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:55.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.176+0000 7f972ea00700 1 -- 172.21.15.32:0/3420008988 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9730108e50 msgr2=0x7f973019b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:55.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.176+0000 7f972ea00700 1 --2- 172.21.15.32:0/3420008988 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9730108e50 0x7f973019b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:55.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.176+0000 7f972ea00700 1 -- 172.21.15.32:0/3420008988 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f973019fc80 con 0x7f9730104520 2024-04-30T08:33:55.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.177+0000 7f972ea00700 1 --2- 172.21.15.32:0/3420008988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9730104520 0x7f973019adb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9720000c00 tx=0x7f97200057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:55.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.177+0000 7f9727400700 1 -- 172.21.15.32:0/3420008988 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9720027070 con 0x7f9730104520 2024-04-30T08:33:55.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.177+0000 7f9735a00700 1 -- 172.21.15.32:0/3420008988 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f973019ff10 con 0x7f9730104520 2024-04-30T08:33:55.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.177+0000 7f9727400700 1 -- 172.21.15.32:0/3420008988 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f972000b610 con 0x7f9730104520 2024-04-30T08:33:55.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.177+0000 7f9735a00700 1 -- 172.21.15.32:0/3420008988 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97301a03f0 con 0x7f9730104520 2024-04-30T08:33:55.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.178+0000 7f9727400700 1 -- 172.21.15.32:0/3420008988 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97200173f0 con 0x7f9730104520 2024-04-30T08:33:55.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.179+0000 7f9735a00700 1 -- 172.21.15.32:0/3420008988 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9730051210 con 0x7f9730104520 2024-04-30T08:33:55.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.182+0000 7f9727400700 1 -- 172.21.15.32:0/3420008988 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7f972002a070 con 0x7f9730104520 2024-04-30T08:33:55.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.184+0000 7f9727400700 1 --2- 172.21.15.32:0/3420008988 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f97140afc10 0x7f97140b2120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:55.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.184+0000 7f9727400700 1 -- 172.21.15.32:0/3420008988 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f97200e0120 con 0x7f9730104520 2024-04-30T08:33:55.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.184+0000 7f972e000700 1 --2- 172.21.15.32:0/3420008988 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f97140afc10 0x7f97140b2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:55.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.185+0000 7f972e000700 1 --2- 172.21.15.32:0/3420008988 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f97140afc10 0x7f97140b2120 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9730103390 tx=0x7f971c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:55.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.186+0000 7f9727400700 1 -- 172.21.15.32:0/3420008988 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9720013070 con 0x7f9730104520 2024-04-30T08:33:55.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:55 smithi189 ceph-mon[33038]: pgmap v387: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:33:55.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.325+0000 7fd423200700 1 -- 172.21.15.32:0/2886561981 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fd41c02e590 con 0x7fd41c1053b0 2024-04-30T08:33:55.326 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:55 smithi032 ceph-mon[28331]: pgmap v387: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:33:55.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.326+0000 7fd418c00700 1 -- 172.21.15.32:0/2886561981 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd40c0a75e0 con 0x7fd41c1053b0 2024-04-30T08:33:55.327 INFO:teuthology.orchestra.run.smithi032.stdout:34359738470 2024-04-30T08:33:55.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.330+0000 7fd423200700 1 -- 172.21.15.32:0/2886561981 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd4000afc10 msgr2=0x7fd4000b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:55.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.330+0000 7fd423200700 1 --2- 172.21.15.32:0/2886561981 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd4000afc10 0x7fd4000b2120 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd40800b2e0 tx=0x7fd408005960 comp rx=0 tx=0).stop 2024-04-30T08:33:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.330+0000 7fd423200700 1 -- 172.21.15.32:0/2886561981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd41c1053b0 msgr2=0x7fd41c19ad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.330+0000 7fd423200700 1 --2- 172.21.15.32:0/2886561981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd41c1053b0 0x7fd41c19ad10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fd40c00e9a0 tx=0x7fd40c00ecb0 comp rx=0 tx=0).stop 2024-04-30T08:33:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.331+0000 7fd423200700 1 -- 172.21.15.32:0/2886561981 shutdown_connections 2024-04-30T08:33:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.331+0000 7fd423200700 1 --2- 172.21.15.32:0/2886561981 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd4000afc10 0x7fd4000b2120 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.331+0000 7fd423200700 1 --2- 172.21.15.32:0/2886561981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd41c10e820 0x7fd41c19f540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.331+0000 7fd423200700 1 --2- 172.21.15.32:0/2886561981 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd41c1053b0 0x7fd41c19ad10 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.331+0000 7fd423200700 1 --2- 172.21.15.32:0/2886561981 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd41c105d00 0x7fd41c19b2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.331+0000 7fd423200700 1 -- 172.21.15.32:0/2886561981 >> 172.21.15.32:0/2886561981 conn(0x7fd41c0fed10 msgr2=0x7fd41c0ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.331+0000 7fd423200700 1 -- 172.21.15.32:0/2886561981 shutdown_connections 2024-04-30T08:33:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.331+0000 7fd423200700 1 -- 172.21.15.32:0/2886561981 wait complete. 2024-04-30T08:33:55.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.420+0000 7f9735a00700 1 -- 172.21.15.32:0/3420008988 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f973002e590 con 0x7f9730104520 2024-04-30T08:33:55.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.421+0000 7f9727400700 1 -- 172.21.15.32:0/3420008988 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f97200a8860 con 0x7f9730104520 2024-04-30T08:33:55.422 INFO:teuthology.orchestra.run.smithi032.stdout:154618822731 2024-04-30T08:33:55.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.424+0000 7f9735a00700 1 -- 172.21.15.32:0/3420008988 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f97140afc10 msgr2=0x7f97140b2120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:55.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.424+0000 7f9735a00700 1 --2- 172.21.15.32:0/3420008988 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f97140afc10 0x7f97140b2120 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f9730103390 tx=0x7f971c009260 comp rx=0 tx=0).stop 2024-04-30T08:33:55.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.424+0000 7f9735a00700 1 -- 172.21.15.32:0/3420008988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9730104520 msgr2=0x7f973019adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:55.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.424+0000 7f9735a00700 1 --2- 172.21.15.32:0/3420008988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9730104520 0x7f973019adb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f9720000c00 tx=0x7f97200057c0 comp rx=0 tx=0).stop 2024-04-30T08:33:55.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.424+0000 7f9735a00700 1 -- 172.21.15.32:0/3420008988 shutdown_connections 2024-04-30T08:33:55.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.424+0000 7f9735a00700 1 --2- 172.21.15.32:0/3420008988 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f97140afc10 0x7f97140b2120 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:55.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.424+0000 7f9735a00700 1 --2- 172.21.15.32:0/3420008988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97301097a0 0x7f973019f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:55.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.424+0000 7f9735a00700 1 --2- 172.21.15.32:0/3420008988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9730104520 0x7f973019adb0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:55.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.424+0000 7f9735a00700 1 --2- 172.21.15.32:0/3420008988 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9730108e50 0x7f973019b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:55.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.424+0000 7f9735a00700 1 -- 172.21.15.32:0/3420008988 >> 172.21.15.32:0/3420008988 conn(0x7f97300ffdf0 msgr2=0x7f973010a700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:55.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.425+0000 7f9735a00700 1 -- 172.21.15.32:0/3420008988 shutdown_connections 2024-04-30T08:33:55.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:55.425+0000 7f9735a00700 1 -- 172.21.15.32:0/3420008988 wait complete. 2024-04-30T08:33:55.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:55 smithi071 ceph-mon[33385]: pgmap v387: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:33:56.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:56 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2886561981' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-30T08:33:56.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:56 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3420008988' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-30T08:33:56.318 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:56 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2886561981' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-30T08:33:56.318 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:56 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3420008988' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-30T08:33:56.373 INFO:tasks.cephadm.ceph_manager.ceph:need seq 154618822724 got 154618822731 for osd.5 2024-04-30T08:33:56.373 DEBUG:teuthology.parallel:result is None 2024-04-30T08:33:56.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:56 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2886561981' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-30T08:33:56.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:56 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3420008988' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-30T08:33:57.012 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738463 got 34359738470 for osd.0 2024-04-30T08:33:57.012 DEBUG:teuthology.parallel:result is None 2024-04-30T08:33:57.013 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-30T08:33:57.013 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-30T08:33:57.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:57 smithi189 ceph-mon[33038]: pgmap v388: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-30T08:33:57.405 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:57 smithi071 ceph-mon[33385]: pgmap v388: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-30T08:33:57.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:57 smithi032 ceph-mon[28331]: pgmap v388: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-30T08:33:57.487 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:33:59.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:59 smithi032 ceph-mon[28331]: pgmap v389: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-30T08:33:59.256 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:33:59 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:33:59.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:59 smithi189 ceph-mon[33038]: pgmap v389: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-30T08:33:59.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:33:59 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:33:59.404 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:59 smithi071 ceph-mon[33385]: pgmap v389: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-30T08:33:59.405 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:33:59 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:33:59.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.943+0000 7fd1ae200700 1 -- 172.21.15.32:0/3760337341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1a8075c80 msgr2=0x7fd1a8076090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:59.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.943+0000 7fd1ae200700 1 --2- 172.21.15.32:0/3760337341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1a8075c80 0x7fd1a8076090 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fd19c009990 tx=0x7fd19c009ca0 comp rx=0 tx=0).stop 2024-04-30T08:33:59.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.944+0000 7fd1ae200700 1 -- 172.21.15.32:0/3760337341 shutdown_connections 2024-04-30T08:33:59.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.944+0000 7fd1ae200700 1 --2- 172.21.15.32:0/3760337341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1a8110900 0x7fd1a8112d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:59.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.944+0000 7fd1ae200700 1 --2- 172.21.15.32:0/3760337341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1a8075c80 0x7fd1a8076090 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:59.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.944+0000 7fd1ae200700 1 --2- 172.21.15.32:0/3760337341 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd1a80765d0 0x7fd1a81103c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:59.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.944+0000 7fd1ae200700 1 -- 172.21.15.32:0/3760337341 >> 172.21.15.32:0/3760337341 conn(0x7fd1a8079c60 msgr2=0x7fd1a807a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:33:59.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.944+0000 7fd1ae200700 1 -- 172.21.15.32:0/3760337341 shutdown_connections 2024-04-30T08:33:59.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.944+0000 7fd1ae200700 1 -- 172.21.15.32:0/3760337341 wait complete. 2024-04-30T08:33:59.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.945+0000 7fd1ae200700 1 Processor -- start 2024-04-30T08:33:59.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.945+0000 7fd1ae200700 1 -- start start 2024-04-30T08:33:59.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.946+0000 7fd1ae200700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd1a8075c80 0x7fd1a819ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:59.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.946+0000 7fd1ae200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1a80765d0 0x7fd1a819d3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:59.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.946+0000 7fd1ae200700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1a8110900 0x7fd1a81a1640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:59.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.946+0000 7fd1ae200700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a8116440 con 0x7fd1a8110900 2024-04-30T08:33:59.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.946+0000 7fd1ae200700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a81162c0 con 0x7fd1a8075c80 2024-04-30T08:33:59.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.946+0000 7fd1ae200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd1a81165c0 con 0x7fd1a80765d0 2024-04-30T08:33:59.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.946+0000 7fd1ace00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd1a8075c80 0x7fd1a819ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:59.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.946+0000 7fd1ace00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd1a8075c80 0x7fd1a819ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:55444/0 (socket says 172.21.15.32:55444) 2024-04-30T08:33:59.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.946+0000 7fd1a7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1a80765d0 0x7fd1a819d3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:59.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.946+0000 7fd1ace00700 1 -- 172.21.15.32:0/2187500663 learned_addr learned my addr 172.21.15.32:0/2187500663 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:33:59.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.946+0000 7fd1ad800700 1 --2- 172.21.15.32:0/2187500663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1a8110900 0x7fd1a81a1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:59.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.947+0000 7fd1ace00700 1 -- 172.21.15.32:0/2187500663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1a80765d0 msgr2=0x7fd1a819d3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:59.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.947+0000 7fd1ace00700 1 --2- 172.21.15.32:0/2187500663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1a80765d0 0x7fd1a819d3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:59.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.947+0000 7fd1ace00700 1 -- 172.21.15.32:0/2187500663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1a8110900 msgr2=0x7fd1a81a1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:33:59.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.947+0000 7fd1ace00700 1 --2- 172.21.15.32:0/2187500663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1a8110900 0x7fd1a81a1640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:33:59.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.947+0000 7fd1ace00700 1 -- 172.21.15.32:0/2187500663 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1a81a1ce0 con 0x7fd1a8075c80 2024-04-30T08:33:59.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.947+0000 7fd1ace00700 1 --2- 172.21.15.32:0/2187500663 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd1a8075c80 0x7fd1a819ce10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd19c000c00 tx=0x7fd19c0057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:59.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.947+0000 7fd1a5600700 1 -- 172.21.15.32:0/2187500663 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd19c027070 con 0x7fd1a8075c80 2024-04-30T08:33:59.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.947+0000 7fd1a5600700 1 -- 172.21.15.32:0/2187500663 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd19c022470 con 0x7fd1a8075c80 2024-04-30T08:33:59.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.947+0000 7fd1a5600700 1 -- 172.21.15.32:0/2187500663 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd19c017650 con 0x7fd1a8075c80 2024-04-30T08:33:59.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.947+0000 7fd1ae200700 1 -- 172.21.15.32:0/2187500663 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1a81a1f70 con 0x7fd1a8075c80 2024-04-30T08:33:59.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.948+0000 7fd1ae200700 1 -- 172.21.15.32:0/2187500663 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1a81a2400 con 0x7fd1a8075c80 2024-04-30T08:33:59.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.949+0000 7fd1ae200700 1 -- 172.21.15.32:0/2187500663 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1a80771f0 con 0x7fd1a8075c80 2024-04-30T08:33:59.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.951+0000 7fd1a5600700 1 -- 172.21.15.32:0/2187500663 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 17) ==== 147571+0+0 (secure 0 0 0) 0x7fd19c02a070 con 0x7fd1a8075c80 2024-04-30T08:33:59.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.953+0000 7fd1a5600700 1 --2- 172.21.15.32:0/2187500663 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd1900afc90 0x7fd1900b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:33:59.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.953+0000 7fd1a5600700 1 -- 172.21.15.32:0/2187500663 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fd19c0e00d0 con 0x7fd1a8075c80 2024-04-30T08:33:59.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.953+0000 7fd1a7e00700 1 --2- 172.21.15.32:0/2187500663 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd1900afc90 0x7fd1900b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:33:59.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.954+0000 7fd1a7e00700 1 --2- 172.21.15.32:0/2187500663 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd1900afc90 0x7fd1900b21a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd1a807a690 tx=0x7fd194009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:33:59.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:33:59.955+0000 7fd1a5600700 1 -- 172.21.15.32:0/2187500663 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd19c013070 con 0x7fd1a8075c80 2024-04-30T08:34:00.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:00.171+0000 7fd1ae200700 1 -- 172.21.15.32:0/2187500663 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd1a8063b50 con 0x7fd1900afc90 2024-04-30T08:34:00.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:00.194+0000 7fd1a5600700 1 -- 172.21.15.32:0/2187500663 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340711 (secure 0 0 0) 0x7fd1a8063b50 con 0x7fd1900afc90 2024-04-30T08:34:00.203 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:34:00.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:00.201+0000 7fd1ae200700 1 -- 172.21.15.32:0/2187500663 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd1900afc90 msgr2=0x7fd1900b21a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:00.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:00.201+0000 7fd1ae200700 1 --2- 172.21.15.32:0/2187500663 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd1900afc90 0x7fd1900b21a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fd1a807a690 tx=0x7fd194009260 comp rx=0 tx=0).stop 2024-04-30T08:34:00.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:00.201+0000 7fd1ae200700 1 -- 172.21.15.32:0/2187500663 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd1a8075c80 msgr2=0x7fd1a819ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:00.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:00.201+0000 7fd1ae200700 1 --2- 172.21.15.32:0/2187500663 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd1a8075c80 0x7fd1a819ce10 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd19c000c00 tx=0x7fd19c0057c0 comp rx=0 tx=0).stop 2024-04-30T08:34:00.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:00.201+0000 7fd1ae200700 1 -- 172.21.15.32:0/2187500663 shutdown_connections 2024-04-30T08:34:00.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:00.201+0000 7fd1ae200700 1 --2- 172.21.15.32:0/2187500663 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd1900afc90 0x7fd1900b21a0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:00.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:00.202+0000 7fd1ae200700 1 --2- 172.21.15.32:0/2187500663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd1a8110900 0x7fd1a81a1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:00.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:00.202+0000 7fd1ae200700 1 --2- 172.21.15.32:0/2187500663 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd1a8075c80 0x7fd1a819ce10 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:00.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:00.202+0000 7fd1ae200700 1 --2- 172.21.15.32:0/2187500663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd1a80765d0 0x7fd1a819d3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:00.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:00.202+0000 7fd1ae200700 1 -- 172.21.15.32:0/2187500663 >> 172.21.15.32:0/2187500663 conn(0x7fd1a8079c60 msgr2=0x7fd1a8100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:00.206 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:00.202+0000 7fd1ae200700 1 -- 172.21.15.32:0/2187500663 shutdown_connections 2024-04-30T08:34:00.207 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:00.202+0000 7fd1ae200700 1 -- 172.21.15.32:0/2187500663 wait complete. 2024-04-30T08:34:00.207 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-30T08:34:00.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:00 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:34:00.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:00 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:34:00.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:00 smithi032 ceph-mon[28331]: pgmap v390: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-30T08:34:00.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:00 smithi032 ceph-mon[28331]: from='client.25076 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:34:00.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:00 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:34:00.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:00 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:34:00.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:00 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:34:00.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:00 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:34:01.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:00 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:34:01.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:00 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:34:01.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:00 smithi071 ceph-mon[33385]: pgmap v390: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-30T08:34:01.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:00 smithi071 ceph-mon[33385]: from='client.25076 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:34:01.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:00 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:34:01.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:00 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:34:01.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:00 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:34:01.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:00 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:34:01.008 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":390,"stamp":"2024-04-30T08:33:59.758279+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659878,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":984,"num_read_kb":593,"num_write":365,"num_write_kb":2291,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":2951908,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":303,"ondisk_log_size":303,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369048,"kb_used_data":11516,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572776,"statfs":{"total":1151940427776,"available":1151562522624,"internally_reserved":0,"allocated":11792384,"data_stored":7698440,"data_compressed":434025,"data_compressed_allocated":589824,"data_compressed_original":2678784,"omap_allocated":19141,"internal_metadata":365933883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":135,"num_read_kb":65,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005926"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.887373+0000","last_change":"2024-04-30T08:31:52.711443+0000","last_active":"2024-04-30T08:33:31.887373+0000","last_peered":"2024-04-30T08:33:31.887373+0000","last_clean":"2024-04-30T08:33:31.887373+0000","last_became_active":"2024-04-30T08:31:52.711029+0000","last_became_peered":"2024-04-30T08:31:52.711029+0000","last_unstale":"2024-04-30T08:33:31.887373+0000","last_undegraded":"2024-04-30T08:33:31.887373+0000","last_fullsized":"2024-04-30T08:33:31.887373+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:51:15.208528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.799749+0000","last_change":"2024-04-30T08:31:52.712324+0000","last_active":"2024-04-30T08:33:32.799749+0000","last_peered":"2024-04-30T08:33:32.799749+0000","last_clean":"2024-04-30T08:33:32.799749+0000","last_became_active":"2024-04-30T08:31:52.711923+0000","last_became_peered":"2024-04-30T08:31:52.711923+0000","last_unstale":"2024-04-30T08:33:32.799749+0000","last_undegraded":"2024-04-30T08:33:32.799749+0000","last_fullsized":"2024-04-30T08:33:32.799749+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:29:55.071459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.800085+0000","last_change":"2024-04-30T08:31:52.712936+0000","last_active":"2024-04-30T08:33:32.800085+0000","last_peered":"2024-04-30T08:33:32.800085+0000","last_clean":"2024-04-30T08:33:32.800085+0000","last_became_active":"2024-04-30T08:31:52.712585+0000","last_became_peered":"2024-04-30T08:31:52.712585+0000","last_unstale":"2024-04-30T08:33:32.800085+0000","last_undegraded":"2024-04-30T08:33:32.800085+0000","last_fullsized":"2024-04-30T08:33:32.800085+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:13:43.745561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.170358+0000","last_change":"2024-04-30T08:31:52.686880+0000","last_active":"2024-04-30T08:33:32.170358+0000","last_peered":"2024-04-30T08:33:32.170358+0000","last_clean":"2024-04-30T08:33:32.170358+0000","last_became_active":"2024-04-30T08:31:52.686506+0000","last_became_peered":"2024-04-30T08:31:52.686506+0000","last_unstale":"2024-04-30T08:33:32.170358+0000","last_undegraded":"2024-04-30T08:33:32.170358+0000","last_fullsized":"2024-04-30T08:33:32.170358+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:16:33.390951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886150+0000","last_change":"2024-04-30T08:31:52.706014+0000","last_active":"2024-04-30T08:33:31.886150+0000","last_peered":"2024-04-30T08:33:31.886150+0000","last_clean":"2024-04-30T08:33:31.886150+0000","last_became_active":"2024-04-30T08:31:52.705582+0000","last_became_peered":"2024-04-30T08:31:52.705582+0000","last_unstale":"2024-04-30T08:33:31.886150+0000","last_undegraded":"2024-04-30T08:33:31.886150+0000","last_fullsized":"2024-04-30T08:33:31.886150+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:24:49.804592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881602+0000","last_change":"2024-04-30T08:31:52.690462+0000","last_active":"2024-04-30T08:33:31.881602+0000","last_peered":"2024-04-30T08:33:31.881602+0000","last_clean":"2024-04-30T08:33:31.881602+0000","last_became_active":"2024-04-30T08:31:52.690092+0000","last_became_peered":"2024-04-30T08:31:52.690092+0000","last_unstale":"2024-04-30T08:33:31.881602+0000","last_undegraded":"2024-04-30T08:33:31.881602+0000","last_fullsized":"2024-04-30T08:33:31.881602+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:32:49.769383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.218008+0000","last_change":"2024-04-30T08:31:52.701786+0000","last_active":"2024-04-30T08:33:32.218008+0000","last_peered":"2024-04-30T08:33:32.218008+0000","last_clean":"2024-04-30T08:33:32.218008+0000","last_became_active":"2024-04-30T08:31:52.701402+0000","last_became_peered":"2024-04-30T08:31:52.701402+0000","last_unstale":"2024-04-30T08:33:32.218008+0000","last_undegraded":"2024-04-30T08:33:32.218008+0000","last_fullsized":"2024-04-30T08:33:32.218008+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:58:06.153479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.276349+0000","last_change":"2024-04-30T08:31:52.698037+0000","last_active":"2024-04-30T08:33:32.276349+0000","last_peered":"2024-04-30T08:33:32.276349+0000","last_clean":"2024-04-30T08:33:32.276349+0000","last_became_active":"2024-04-30T08:31:52.697646+0000","last_became_peered":"2024-04-30T08:31:52.697646+0000","last_unstale":"2024-04-30T08:33:32.276349+0000","last_undegraded":"2024-04-30T08:33:32.276349+0000","last_fullsized":"2024-04-30T08:33:32.276349+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:28:59.520091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273164+0000","last_change":"2024-04-30T08:31:52.680377+0000","last_active":"2024-04-30T08:33:32.273164+0000","last_peered":"2024-04-30T08:33:32.273164+0000","last_clean":"2024-04-30T08:33:32.273164+0000","last_became_active":"2024-04-30T08:31:52.679977+0000","last_became_peered":"2024-04-30T08:31:52.679977+0000","last_unstale":"2024-04-30T08:33:32.273164+0000","last_undegraded":"2024-04-30T08:33:32.273164+0000","last_fullsized":"2024-04-30T08:33:32.273164+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:35:22.488450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881954+0000","last_change":"2024-04-30T08:32:31.573546+0000","last_active":"2024-04-30T08:33:31.881954+0000","last_peered":"2024-04-30T08:33:31.881954+0000","last_clean":"2024-04-30T08:33:31.881954+0000","last_became_active":"2024-04-30T08:32:31.573289+0000","last_became_peered":"2024-04-30T08:32:31.573289+0000","last_unstale":"2024-04-30T08:33:31.881954+0000","last_undegraded":"2024-04-30T08:33:31.881954+0000","last_fullsized":"2024-04-30T08:33:31.881954+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:42:45.862326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.218613+0000","last_change":"2024-04-30T08:31:52.690344+0000","last_active":"2024-04-30T08:33:32.218613+0000","last_peered":"2024-04-30T08:33:32.218613+0000","last_clean":"2024-04-30T08:33:32.218613+0000","last_became_active":"2024-04-30T08:31:52.689857+0000","last_became_peered":"2024-04-30T08:31:52.689857+0000","last_unstale":"2024-04-30T08:33:32.218613+0000","last_undegraded":"2024-04-30T08:33:32.218613+0000","last_fullsized":"2024-04-30T08:33:32.218613+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:45:13.395589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.170332+0000","last_change":"2024-04-30T08:31:52.685151+0000","last_active":"2024-04-30T08:33:32.170332+0000","last_peered":"2024-04-30T08:33:32.170332+0000","last_clean":"2024-04-30T08:33:32.170332+0000","last_became_active":"2024-04-30T08:31:52.684722+0000","last_became_peered":"2024-04-30T08:31:52.684722+0000","last_unstale":"2024-04-30T08:33:32.170332+0000","last_undegraded":"2024-04-30T08:33:32.170332+0000","last_fullsized":"2024-04-30T08:33:32.170332+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:14:00.660478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.880770+0000","last_change":"2024-04-30T08:31:52.714294+0000","last_active":"2024-04-30T08:33:31.880770+0000","last_peered":"2024-04-30T08:33:31.880770+0000","last_clean":"2024-04-30T08:33:31.880770+0000","last_became_active":"2024-04-30T08:31:52.710094+0000","last_became_peered":"2024-04-30T08:31:52.710094+0000","last_unstale":"2024-04-30T08:33:31.880770+0000","last_undegraded":"2024-04-30T08:33:31.880770+0000","last_fullsized":"2024-04-30T08:33:31.880770+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:34:44.203093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.798873+0000","last_change":"2024-04-30T08:31:52.707992+0000","last_active":"2024-04-30T08:33:32.798873+0000","last_peered":"2024-04-30T08:33:32.798873+0000","last_clean":"2024-04-30T08:33:32.798873+0000","last_became_active":"2024-04-30T08:31:52.706456+0000","last_became_peered":"2024-04-30T08:31:52.706456+0000","last_unstale":"2024-04-30T08:33:32.798873+0000","last_undegraded":"2024-04-30T08:33:32.798873+0000","last_fullsized":"2024-04-30T08:33:32.798873+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:45:32.717842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.798865+0000","last_change":"2024-04-30T08:31:52.695605+0000","last_active":"2024-04-30T08:33:32.798865+0000","last_peered":"2024-04-30T08:33:32.798865+0000","last_clean":"2024-04-30T08:33:32.798865+0000","last_became_active":"2024-04-30T08:31:52.692439+0000","last_became_peered":"2024-04-30T08:31:52.692439+0000","last_unstale":"2024-04-30T08:33:32.798865+0000","last_undegraded":"2024-04-30T08:33:32.798865+0000","last_fullsized":"2024-04-30T08:33:32.798865+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:39:38.746567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.802311+0000","last_change":"2024-04-30T08:33:32.802311+0000","last_active":"2024-04-30T08:33:32.802311+0000","last_peered":"2024-04-30T08:33:32.802311+0000","last_clean":"2024-04-30T08:33:32.802311+0000","last_became_active":"2024-04-30T08:33:32.801601+0000","last_became_peered":"2024-04-30T08:33:32.801601+0000","last_unstale":"2024-04-30T08:33:32.802311+0000","last_undegraded":"2024-04-30T08:33:32.802311+0000","last_fullsized":"2024-04-30T08:33:32.802311+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:59:34.877824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0],"acting":[9,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886596+0000","last_change":"2024-04-30T08:31:52.711439+0000","last_active":"2024-04-30T08:33:31.886596+0000","last_peered":"2024-04-30T08:33:31.886596+0000","last_clean":"2024-04-30T08:33:31.886596+0000","last_became_active":"2024-04-30T08:31:52.711023+0000","last_became_peered":"2024-04-30T08:31:52.711023+0000","last_unstale":"2024-04-30T08:33:31.886596+0000","last_undegraded":"2024-04-30T08:33:31.886596+0000","last_fullsized":"2024-04-30T08:33:31.886596+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:31:00.654348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886786+0000","last_change":"2024-04-30T08:31:52.708476+0000","last_active":"2024-04-30T08:33:31.886786+0000","last_peered":"2024-04-30T08:33:31.886786+0000","last_clean":"2024-04-30T08:33:31.886786+0000","last_became_active":"2024-04-30T08:31:52.704735+0000","last_became_peered":"2024-04-30T08:31:52.704735+0000","last_unstale":"2024-04-30T08:33:31.886786+0000","last_undegraded":"2024-04-30T08:33:31.886786+0000","last_fullsized":"2024-04-30T08:33:31.886786+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:40:00.011783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.880121+0000","last_change":"2024-04-30T08:31:52.704901+0000","last_active":"2024-04-30T08:33:31.880121+0000","last_peered":"2024-04-30T08:33:31.880121+0000","last_clean":"2024-04-30T08:33:31.880121+0000","last_became_active":"2024-04-30T08:31:52.704636+0000","last_became_peered":"2024-04-30T08:31:52.704636+0000","last_unstale":"2024-04-30T08:33:31.880121+0000","last_undegraded":"2024-04-30T08:33:31.880121+0000","last_fullsized":"2024-04-30T08:33:31.880121+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:08:53.580809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.879520+0000","last_change":"2024-04-30T08:31:57.743342+0000","last_active":"2024-04-30T08:33:31.879520+0000","last_peered":"2024-04-30T08:33:31.879520+0000","last_clean":"2024-04-30T08:33:31.879520+0000","last_became_active":"2024-04-30T08:31:57.742850+0000","last_became_peered":"2024-04-30T08:31:57.742850+0000","last_unstale":"2024-04-30T08:33:31.879520+0000","last_undegraded":"2024-04-30T08:33:31.879520+0000","last_fullsized":"2024-04-30T08:33:31.879520+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:59:10.628822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"81'13","reported_seq":133,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:55.761942+0000","last_change":"2024-04-30T08:31:51.694984+0000","last_active":"2024-04-30T08:33:55.761942+0000","last_peered":"2024-04-30T08:33:55.761942+0000","last_clean":"2024-04-30T08:33:55.761942+0000","last_became_active":"2024-04-30T08:31:51.694514+0000","last_became_peered":"2024-04-30T08:31:51.694514+0000","last_unstale":"2024-04-30T08:33:55.761942+0000","last_undegraded":"2024-04-30T08:33:55.761942+0000","last_fullsized":"2024-04-30T08:33:55.761942+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:17:03.873090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":87,"num_read_kb":87,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.277033+0000","last_change":"2024-04-30T08:31:52.694276+0000","last_active":"2024-04-30T08:33:32.277033+0000","last_peered":"2024-04-30T08:33:32.277033+0000","last_clean":"2024-04-30T08:33:32.277033+0000","last_became_active":"2024-04-30T08:31:52.693839+0000","last_became_peered":"2024-04-30T08:31:52.693839+0000","last_unstale":"2024-04-30T08:33:32.277033+0000","last_undegraded":"2024-04-30T08:33:32.277033+0000","last_fullsized":"2024-04-30T08:33:32.277033+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:29:21.271040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.169659+0000","last_change":"2024-04-30T08:31:57.754828+0000","last_active":"2024-04-30T08:33:32.169659+0000","last_peered":"2024-04-30T08:33:32.169659+0000","last_clean":"2024-04-30T08:33:32.169659+0000","last_became_active":"2024-04-30T08:31:57.754322+0000","last_became_peered":"2024-04-30T08:31:57.754322+0000","last_unstale":"2024-04-30T08:33:32.169659+0000","last_undegraded":"2024-04-30T08:33:32.169659+0000","last_fullsized":"2024-04-30T08:33:32.169659+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:36:02.560134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"81'2","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.277887+0000","last_change":"2024-04-30T08:31:51.707637+0000","last_active":"2024-04-30T08:33:32.277887+0000","last_peered":"2024-04-30T08:33:32.277887+0000","last_clean":"2024-04-30T08:33:32.277887+0000","last_became_active":"2024-04-30T08:31:51.707233+0000","last_became_peered":"2024-04-30T08:31:51.707233+0000","last_unstale":"2024-04-30T08:33:32.277887+0000","last_undegraded":"2024-04-30T08:33:32.277887+0000","last_fullsized":"2024-04-30T08:33:32.277887+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:14:46.041169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.799831+0000","last_change":"2024-04-30T08:31:52.711442+0000","last_active":"2024-04-30T08:33:32.799831+0000","last_peered":"2024-04-30T08:33:32.799831+0000","last_clean":"2024-04-30T08:33:32.799831+0000","last_became_active":"2024-04-30T08:31:52.710942+0000","last_became_peered":"2024-04-30T08:31:52.710942+0000","last_unstale":"2024-04-30T08:33:32.799831+0000","last_undegraded":"2024-04-30T08:33:32.799831+0000","last_fullsized":"2024-04-30T08:33:32.799831+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:26:56.428770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.169844+0000","last_change":"2024-04-30T08:31:57.760535+0000","last_active":"2024-04-30T08:33:32.169844+0000","last_peered":"2024-04-30T08:33:32.169844+0000","last_clean":"2024-04-30T08:33:32.169844+0000","last_became_active":"2024-04-30T08:31:57.760073+0000","last_became_peered":"2024-04-30T08:31:57.760073+0000","last_unstale":"2024-04-30T08:33:32.169844+0000","last_undegraded":"2024-04-30T08:33:32.169844+0000","last_fullsized":"2024-04-30T08:33:32.169844+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:03:51.359474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"81'12","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.887039+0000","last_change":"2024-04-30T08:31:51.678575+0000","last_active":"2024-04-30T08:33:31.887039+0000","last_peered":"2024-04-30T08:33:31.887039+0000","last_clean":"2024-04-30T08:33:31.887039+0000","last_became_active":"2024-04-30T08:31:51.678114+0000","last_became_peered":"2024-04-30T08:31:51.678114+0000","last_unstale":"2024-04-30T08:33:31.887039+0000","last_undegraded":"2024-04-30T08:33:31.887039+0000","last_fullsized":"2024-04-30T08:33:31.887039+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:10:35.517183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.272847+0000","last_change":"2024-04-30T08:31:52.704137+0000","last_active":"2024-04-30T08:33:32.272847+0000","last_peered":"2024-04-30T08:33:32.272847+0000","last_clean":"2024-04-30T08:33:32.272847+0000","last_became_active":"2024-04-30T08:31:52.703695+0000","last_became_peered":"2024-04-30T08:31:52.703695+0000","last_unstale":"2024-04-30T08:33:32.272847+0000","last_undegraded":"2024-04-30T08:33:32.272847+0000","last_fullsized":"2024-04-30T08:33:32.272847+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:31:05.461106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.169733+0000","last_change":"2024-04-30T08:31:57.755237+0000","last_active":"2024-04-30T08:33:32.169733+0000","last_peered":"2024-04-30T08:33:32.169733+0000","last_clean":"2024-04-30T08:33:32.169733+0000","last_became_active":"2024-04-30T08:31:57.754438+0000","last_became_peered":"2024-04-30T08:31:57.754438+0000","last_unstale":"2024-04-30T08:33:32.169733+0000","last_undegraded":"2024-04-30T08:33:32.169733+0000","last_fullsized":"2024-04-30T08:33:32.169733+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:20:59.035110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"81'5","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886326+0000","last_change":"2024-04-30T08:31:51.673793+0000","last_active":"2024-04-30T08:33:31.886326+0000","last_peered":"2024-04-30T08:33:31.886326+0000","last_clean":"2024-04-30T08:33:31.886326+0000","last_became_active":"2024-04-30T08:31:51.672703+0000","last_became_peered":"2024-04-30T08:31:51.672703+0000","last_unstale":"2024-04-30T08:33:31.886326+0000","last_undegraded":"2024-04-30T08:33:31.886326+0000","last_fullsized":"2024-04-30T08:33:31.886326+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:13:18.959542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273553+0000","last_change":"2024-04-30T08:31:52.694320+0000","last_active":"2024-04-30T08:33:32.273553+0000","last_peered":"2024-04-30T08:33:32.273553+0000","last_clean":"2024-04-30T08:33:32.273553+0000","last_became_active":"2024-04-30T08:31:52.694032+0000","last_became_peered":"2024-04-30T08:31:52.694032+0000","last_unstale":"2024-04-30T08:33:32.273553+0000","last_undegraded":"2024-04-30T08:33:32.273553+0000","last_fullsized":"2024-04-30T08:33:32.273553+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:33:12.347369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.879955+0000","last_change":"2024-04-30T08:31:57.739984+0000","last_active":"2024-04-30T08:33:31.879955+0000","last_peered":"2024-04-30T08:33:31.879955+0000","last_clean":"2024-04-30T08:33:31.879955+0000","last_became_active":"2024-04-30T08:31:57.739477+0000","last_became_peered":"2024-04-30T08:31:57.739477+0000","last_unstale":"2024-04-30T08:33:31.879955+0000","last_undegraded":"2024-04-30T08:33:31.879955+0000","last_fullsized":"2024-04-30T08:33:31.879955+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:10:11.325419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"81'6","reported_seq":129,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:56.702491+0000","last_change":"2024-04-30T08:31:51.678792+0000","last_active":"2024-04-30T08:33:56.702491+0000","last_peered":"2024-04-30T08:33:56.702491+0000","last_clean":"2024-04-30T08:33:56.702491+0000","last_became_active":"2024-04-30T08:31:51.678094+0000","last_became_peered":"2024-04-30T08:31:51.678094+0000","last_unstale":"2024-04-30T08:33:56.702491+0000","last_undegraded":"2024-04-30T08:33:56.702491+0000","last_fullsized":"2024-04-30T08:33:56.702491+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:03:17.526016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":90,"num_read_kb":2,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886172+0000","last_change":"2024-04-30T08:32:31.572984+0000","last_active":"2024-04-30T08:33:31.886172+0000","last_peered":"2024-04-30T08:33:31.886172+0000","last_clean":"2024-04-30T08:33:31.886172+0000","last_became_active":"2024-04-30T08:32:31.572536+0000","last_became_peered":"2024-04-30T08:32:31.572536+0000","last_unstale":"2024-04-30T08:33:31.886172+0000","last_undegraded":"2024-04-30T08:33:31.886172+0000","last_fullsized":"2024-04-30T08:33:31.886172+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:46:15.884555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.879808+0000","last_change":"2024-04-30T08:31:57.765047+0000","last_active":"2024-04-30T08:33:31.879808+0000","last_peered":"2024-04-30T08:33:31.879808+0000","last_clean":"2024-04-30T08:33:31.879808+0000","last_became_active":"2024-04-30T08:31:57.764519+0000","last_became_peered":"2024-04-30T08:31:57.764519+0000","last_unstale":"2024-04-30T08:33:31.879808+0000","last_undegraded":"2024-04-30T08:33:31.879808+0000","last_fullsized":"2024-04-30T08:33:31.879808+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:06:16.126396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"81'7","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881554+0000","last_change":"2024-04-30T08:31:51.686403+0000","last_active":"2024-04-30T08:33:31.881554+0000","last_peered":"2024-04-30T08:33:31.881554+0000","last_clean":"2024-04-30T08:33:31.881554+0000","last_became_active":"2024-04-30T08:31:51.686096+0000","last_became_peered":"2024-04-30T08:31:51.686096+0000","last_unstale":"2024-04-30T08:33:31.881554+0000","last_undegraded":"2024-04-30T08:33:31.881554+0000","last_fullsized":"2024-04-30T08:33:31.881554+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:47:05.961930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273383+0000","last_change":"2024-04-30T08:31:52.698512+0000","last_active":"2024-04-30T08:33:32.273383+0000","last_peered":"2024-04-30T08:33:32.273383+0000","last_clean":"2024-04-30T08:33:32.273383+0000","last_became_active":"2024-04-30T08:31:52.698168+0000","last_became_peered":"2024-04-30T08:31:52.698168+0000","last_unstale":"2024-04-30T08:33:32.273383+0000","last_undegraded":"2024-04-30T08:33:32.273383+0000","last_fullsized":"2024-04-30T08:33:32.273383+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:22:45.094365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886339+0000","last_change":"2024-04-30T08:31:57.761430+0000","last_active":"2024-04-30T08:33:31.886339+0000","last_peered":"2024-04-30T08:33:31.886339+0000","last_clean":"2024-04-30T08:33:31.886339+0000","last_became_active":"2024-04-30T08:31:57.761062+0000","last_became_peered":"2024-04-30T08:31:57.761062+0000","last_unstale":"2024-04-30T08:33:31.886339+0000","last_undegraded":"2024-04-30T08:33:31.886339+0000","last_fullsized":"2024-04-30T08:33:31.886339+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:32:05.403786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"81'3","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.879731+0000","last_change":"2024-04-30T08:31:51.686694+0000","last_active":"2024-04-30T08:33:31.879731+0000","last_peered":"2024-04-30T08:33:31.879731+0000","last_clean":"2024-04-30T08:33:31.879731+0000","last_became_active":"2024-04-30T08:31:51.685773+0000","last_became_peered":"2024-04-30T08:31:51.685773+0000","last_unstale":"2024-04-30T08:33:31.879731+0000","last_undegraded":"2024-04-30T08:33:31.879731+0000","last_fullsized":"2024-04-30T08:33:31.879731+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:44:25.327977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.888698+0000","last_change":"2024-04-30T08:33:31.888698+0000","last_active":"2024-04-30T08:33:31.888698+0000","last_peered":"2024-04-30T08:33:31.888698+0000","last_clean":"2024-04-30T08:33:31.888698+0000","last_became_active":"2024-04-30T08:33:31.887935+0000","last_became_peered":"2024-04-30T08:33:31.887935+0000","last_unstale":"2024-04-30T08:33:31.888698+0000","last_undegraded":"2024-04-30T08:33:31.888698+0000","last_fullsized":"2024-04-30T08:33:31.888698+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:15:40.989004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.799695+0000","last_change":"2024-04-30T08:31:57.757224+0000","last_active":"2024-04-30T08:33:32.799695+0000","last_peered":"2024-04-30T08:33:32.799695+0000","last_clean":"2024-04-30T08:33:32.799695+0000","last_became_active":"2024-04-30T08:31:57.756658+0000","last_became_peered":"2024-04-30T08:31:57.756658+0000","last_unstale":"2024-04-30T08:33:32.799695+0000","last_undegraded":"2024-04-30T08:33:32.799695+0000","last_fullsized":"2024-04-30T08:33:32.799695+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:01:21.309322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"81'3","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.272944+0000","last_change":"2024-04-30T08:31:51.678944+0000","last_active":"2024-04-30T08:33:32.272944+0000","last_peered":"2024-04-30T08:33:32.272944+0000","last_clean":"2024-04-30T08:33:32.272944+0000","last_became_active":"2024-04-30T08:31:51.678493+0000","last_became_peered":"2024-04-30T08:31:51.678493+0000","last_unstale":"2024-04-30T08:33:32.272944+0000","last_undegraded":"2024-04-30T08:33:32.272944+0000","last_fullsized":"2024-04-30T08:33:32.272944+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:48:11.111466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.882843+0000","last_change":"2024-04-30T08:31:52.707627+0000","last_active":"2024-04-30T08:33:31.882843+0000","last_peered":"2024-04-30T08:33:31.882843+0000","last_clean":"2024-04-30T08:33:31.882843+0000","last_became_active":"2024-04-30T08:31:52.706441+0000","last_became_peered":"2024-04-30T08:31:52.706441+0000","last_unstale":"2024-04-30T08:33:31.882843+0000","last_undegraded":"2024-04-30T08:33:31.882843+0000","last_fullsized":"2024-04-30T08:33:31.882843+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:15:45.768927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.275059+0000","last_change":"2024-04-30T08:31:57.740324+0000","last_active":"2024-04-30T08:33:32.275059+0000","last_peered":"2024-04-30T08:33:32.275059+0000","last_clean":"2024-04-30T08:33:32.275059+0000","last_became_active":"2024-04-30T08:31:57.739464+0000","last_became_peered":"2024-04-30T08:31:57.739464+0000","last_unstale":"2024-04-30T08:33:32.275059+0000","last_undegraded":"2024-04-30T08:33:32.275059+0000","last_fullsized":"2024-04-30T08:33:32.275059+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:11:37.395545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"81'4","reported_seq":214,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:58.704283+0000","last_change":"2024-04-30T08:31:51.695283+0000","last_active":"2024-04-30T08:33:58.704283+0000","last_peered":"2024-04-30T08:33:58.704283+0000","last_clean":"2024-04-30T08:33:58.704283+0000","last_became_active":"2024-04-30T08:31:51.694848+0000","last_became_peered":"2024-04-30T08:31:51.694848+0000","last_unstale":"2024-04-30T08:33:58.704283+0000","last_undegraded":"2024-04-30T08:33:58.704283+0000","last_fullsized":"2024-04-30T08:33:58.704283+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:41:38.638034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":177,"num_read_kb":177,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273167+0000","last_change":"2024-04-30T08:31:52.693594+0000","last_active":"2024-04-30T08:33:32.273167+0000","last_peered":"2024-04-30T08:33:32.273167+0000","last_clean":"2024-04-30T08:33:32.273167+0000","last_became_active":"2024-04-30T08:31:52.693199+0000","last_became_peered":"2024-04-30T08:31:52.693199+0000","last_unstale":"2024-04-30T08:33:32.273167+0000","last_undegraded":"2024-04-30T08:33:32.273167+0000","last_fullsized":"2024-04-30T08:33:32.273167+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:57:49.908617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.883592+0000","last_change":"2024-04-30T08:31:57.749727+0000","last_active":"2024-04-30T08:33:31.883592+0000","last_peered":"2024-04-30T08:33:31.883592+0000","last_clean":"2024-04-30T08:33:31.883592+0000","last_became_active":"2024-04-30T08:31:57.749267+0000","last_became_peered":"2024-04-30T08:31:57.749267+0000","last_unstale":"2024-04-30T08:33:31.883592+0000","last_undegraded":"2024-04-30T08:33:31.883592+0000","last_fullsized":"2024-04-30T08:33:31.883592+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:26:12.683008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"81'4","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.218604+0000","last_change":"2024-04-30T08:31:51.677398+0000","last_active":"2024-04-30T08:33:32.218604+0000","last_peered":"2024-04-30T08:33:32.218604+0000","last_clean":"2024-04-30T08:33:32.218604+0000","last_became_active":"2024-04-30T08:31:51.677031+0000","last_became_peered":"2024-04-30T08:31:51.677031+0000","last_unstale":"2024-04-30T08:33:32.218604+0000","last_undegraded":"2024-04-30T08:33:32.218604+0000","last_fullsized":"2024-04-30T08:33:32.218604+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:24:25.154229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.800006+0000","last_change":"2024-04-30T08:32:31.561554+0000","last_active":"2024-04-30T08:33:32.800006+0000","last_peered":"2024-04-30T08:33:32.800006+0000","last_clean":"2024-04-30T08:33:32.800006+0000","last_became_active":"2024-04-30T08:32:31.560253+0000","last_became_peered":"2024-04-30T08:32:31.560253+0000","last_unstale":"2024-04-30T08:33:32.800006+0000","last_undegraded":"2024-04-30T08:33:32.800006+0000","last_fullsized":"2024-04-30T08:33:32.800006+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:59:08.602282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":29,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.169998+0000","last_change":"2024-04-30T08:32:31.586746+0000","last_active":"2024-04-30T08:33:32.169998+0000","last_peered":"2024-04-30T08:33:32.169998+0000","last_clean":"2024-04-30T08:33:32.169998+0000","last_became_active":"2024-04-30T08:32:31.586356+0000","last_became_peered":"2024-04-30T08:32:31.586356+0000","last_unstale":"2024-04-30T08:33:32.169998+0000","last_undegraded":"2024-04-30T08:33:32.169998+0000","last_fullsized":"2024-04-30T08:33:32.169998+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:22:39.371508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"81'5","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273422+0000","last_change":"2024-04-30T08:31:51.694424+0000","last_active":"2024-04-30T08:33:32.273422+0000","last_peered":"2024-04-30T08:33:32.273422+0000","last_clean":"2024-04-30T08:33:32.273422+0000","last_became_active":"2024-04-30T08:31:51.694081+0000","last_became_peered":"2024-04-30T08:31:51.694081+0000","last_unstale":"2024-04-30T08:33:32.273422+0000","last_undegraded":"2024-04-30T08:33:32.273422+0000","last_fullsized":"2024-04-30T08:33:32.273422+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:27:33.328477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273039+0000","last_change":"2024-04-30T08:31:52.684739+0000","last_active":"2024-04-30T08:33:32.273039+0000","last_peered":"2024-04-30T08:33:32.273039+0000","last_clean":"2024-04-30T08:33:32.273039+0000","last_became_active":"2024-04-30T08:31:52.684316+0000","last_became_peered":"2024-04-30T08:31:52.684316+0000","last_unstale":"2024-04-30T08:33:32.273039+0000","last_undegraded":"2024-04-30T08:33:32.273039+0000","last_fullsized":"2024-04-30T08:33:32.273039+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:12:05.437493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881339+0000","last_change":"2024-04-30T08:32:31.557578+0000","last_active":"2024-04-30T08:33:31.881339+0000","last_peered":"2024-04-30T08:33:31.881339+0000","last_clean":"2024-04-30T08:33:31.881339+0000","last_became_active":"2024-04-30T08:32:31.557115+0000","last_became_peered":"2024-04-30T08:32:31.557115+0000","last_unstale":"2024-04-30T08:33:31.881339+0000","last_undegraded":"2024-04-30T08:33:31.881339+0000","last_fullsized":"2024-04-30T08:33:31.881339+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:41:33.225879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"83'33","reported_seq":248,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:58.704314+0000","last_change":"2024-04-30T08:31:51.676223+0000","last_active":"2024-04-30T08:33:58.704314+0000","last_peered":"2024-04-30T08:33:58.704314+0000","last_clean":"2024-04-30T08:33:58.704314+0000","last_became_active":"2024-04-30T08:31:51.675830+0000","last_became_peered":"2024-04-30T08:31:51.675830+0000","last_unstale":"2024-04-30T08:33:58.704314+0000","last_undegraded":"2024-04-30T08:33:58.704314+0000","last_fullsized":"2024-04-30T08:33:58.704314+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:28:49.497719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62072,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":182,"num_read_kb":151,"num_write":35,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886419+0000","last_change":"2024-04-30T08:31:52.686892+0000","last_active":"2024-04-30T08:33:31.886419+0000","last_peered":"2024-04-30T08:33:31.886419+0000","last_clean":"2024-04-30T08:33:31.886419+0000","last_became_active":"2024-04-30T08:31:52.686533+0000","last_became_peered":"2024-04-30T08:31:52.686533+0000","last_unstale":"2024-04-30T08:33:31.886419+0000","last_undegraded":"2024-04-30T08:33:31.886419+0000","last_fullsized":"2024-04-30T08:33:31.886419+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:50:18.836161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273957+0000","last_change":"2024-04-30T08:31:57.752865+0000","last_active":"2024-04-30T08:33:32.273957+0000","last_peered":"2024-04-30T08:33:32.273957+0000","last_clean":"2024-04-30T08:33:32.273957+0000","last_became_active":"2024-04-30T08:31:57.752385+0000","last_became_peered":"2024-04-30T08:31:57.752385+0000","last_unstale":"2024-04-30T08:33:32.273957+0000","last_undegraded":"2024-04-30T08:33:32.273957+0000","last_fullsized":"2024-04-30T08:33:32.273957+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:15:15.315580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"81'9","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886198+0000","last_change":"2024-04-30T08:32:31.585282+0000","last_active":"2024-04-30T08:33:31.886198+0000","last_peered":"2024-04-30T08:33:31.886198+0000","last_clean":"2024-04-30T08:33:31.886198+0000","last_became_active":"2024-04-30T08:32:31.571676+0000","last_became_peered":"2024-04-30T08:32:31.571676+0000","last_unstale":"2024-04-30T08:33:31.886198+0000","last_undegraded":"2024-04-30T08:33:31.886198+0000","last_fullsized":"2024-04-30T08:33:31.886198+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:18:29.910201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.879936+0000","last_change":"2024-04-30T08:31:52.706259+0000","last_active":"2024-04-30T08:33:31.879936+0000","last_peered":"2024-04-30T08:33:31.879936+0000","last_clean":"2024-04-30T08:33:31.879936+0000","last_became_active":"2024-04-30T08:31:52.705620+0000","last_became_peered":"2024-04-30T08:31:52.705620+0000","last_unstale":"2024-04-30T08:33:31.879936+0000","last_undegraded":"2024-04-30T08:33:31.879936+0000","last_fullsized":"2024-04-30T08:33:31.879936+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:22:29.816950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":29,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886484+0000","last_change":"2024-04-30T08:32:31.562571+0000","last_active":"2024-04-30T08:33:31.886484+0000","last_peered":"2024-04-30T08:33:31.886484+0000","last_clean":"2024-04-30T08:33:31.886484+0000","last_became_active":"2024-04-30T08:32:31.562012+0000","last_became_peered":"2024-04-30T08:32:31.562012+0000","last_unstale":"2024-04-30T08:33:31.886484+0000","last_undegraded":"2024-04-30T08:33:31.886484+0000","last_fullsized":"2024-04-30T08:33:31.886484+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:12:22.928286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"81'5","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273089+0000","last_change":"2024-04-30T08:31:51.701122+0000","last_active":"2024-04-30T08:33:32.273089+0000","last_peered":"2024-04-30T08:33:32.273089+0000","last_clean":"2024-04-30T08:33:32.273089+0000","last_became_active":"2024-04-30T08:31:51.700815+0000","last_became_peered":"2024-04-30T08:31:51.700815+0000","last_unstale":"2024-04-30T08:33:32.273089+0000","last_undegraded":"2024-04-30T08:33:32.273089+0000","last_fullsized":"2024-04-30T08:33:32.273089+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:33:12.714979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.218401+0000","last_change":"2024-04-30T08:31:52.716043+0000","last_active":"2024-04-30T08:33:32.218401+0000","last_peered":"2024-04-30T08:33:32.218401+0000","last_clean":"2024-04-30T08:33:32.218401+0000","last_became_active":"2024-04-30T08:31:52.715284+0000","last_became_peered":"2024-04-30T08:31:52.715284+0000","last_unstale":"2024-04-30T08:33:32.218401+0000","last_undegraded":"2024-04-30T08:33:32.218401+0000","last_fullsized":"2024-04-30T08:33:32.218401+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:00:01.863606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.885093+0000","last_change":"2024-04-30T08:31:57.765008+0000","last_active":"2024-04-30T08:33:31.885093+0000","last_peered":"2024-04-30T08:33:31.885093+0000","last_clean":"2024-04-30T08:33:31.885093+0000","last_became_active":"2024-04-30T08:31:57.762958+0000","last_became_peered":"2024-04-30T08:31:57.762958+0000","last_unstale":"2024-04-30T08:33:31.885093+0000","last_undegraded":"2024-04-30T08:33:31.885093+0000","last_fullsized":"2024-04-30T08:33:31.885093+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:48:47.116004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"81'6","reported_seq":129,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:56.767821+0000","last_change":"2024-04-30T08:31:51.676681+0000","last_active":"2024-04-30T08:33:56.767821+0000","last_peered":"2024-04-30T08:33:56.767821+0000","last_clean":"2024-04-30T08:33:56.767821+0000","last_became_active":"2024-04-30T08:31:51.676311+0000","last_became_peered":"2024-04-30T08:31:51.676311+0000","last_unstale":"2024-04-30T08:33:56.767821+0000","last_undegraded":"2024-04-30T08:33:56.767821+0000","last_fullsized":"2024-04-30T08:33:56.767821+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:24:49.757427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":91,"num_read_kb":3,"num_write":6,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.217888+0000","last_change":"2024-04-30T08:31:52.689418+0000","last_active":"2024-04-30T08:33:32.217888+0000","last_peered":"2024-04-30T08:33:32.217888+0000","last_clean":"2024-04-30T08:33:32.217888+0000","last_became_active":"2024-04-30T08:31:52.688840+0000","last_became_peered":"2024-04-30T08:31:52.688840+0000","last_unstale":"2024-04-30T08:33:32.217888+0000","last_undegraded":"2024-04-30T08:33:32.217888+0000","last_fullsized":"2024-04-30T08:33:32.217888+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:40:05.781137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881164+0000","last_change":"2024-04-30T08:31:57.765875+0000","last_active":"2024-04-30T08:33:31.881164+0000","last_peered":"2024-04-30T08:33:31.881164+0000","last_clean":"2024-04-30T08:33:31.881164+0000","last_became_active":"2024-04-30T08:31:57.765432+0000","last_became_peered":"2024-04-30T08:31:57.765432+0000","last_unstale":"2024-04-30T08:33:31.881164+0000","last_undegraded":"2024-04-30T08:33:31.881164+0000","last_fullsized":"2024-04-30T08:33:31.881164+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:24:44.329532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"81'17","reported_seq":50,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.887011+0000","last_change":"2024-04-30T08:31:51.695007+0000","last_active":"2024-04-30T08:33:31.887011+0000","last_peered":"2024-04-30T08:33:31.887011+0000","last_clean":"2024-04-30T08:33:31.887011+0000","last_became_active":"2024-04-30T08:31:51.694533+0000","last_became_peered":"2024-04-30T08:31:51.694533+0000","last_unstale":"2024-04-30T08:33:31.887011+0000","last_undegraded":"2024-04-30T08:33:31.887011+0000","last_fullsized":"2024-04-30T08:33:31.887011+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:47:15.104172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273155+0000","last_change":"2024-04-30T08:31:52.692274+0000","last_active":"2024-04-30T08:33:32.273155+0000","last_peered":"2024-04-30T08:33:32.273155+0000","last_clean":"2024-04-30T08:33:32.273155+0000","last_became_active":"2024-04-30T08:31:52.691809+0000","last_became_peered":"2024-04-30T08:31:52.691809+0000","last_unstale":"2024-04-30T08:33:32.273155+0000","last_undegraded":"2024-04-30T08:33:32.273155+0000","last_fullsized":"2024-04-30T08:33:32.273155+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:50:00.201782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273070+0000","last_change":"2024-04-30T08:31:57.732383+0000","last_active":"2024-04-30T08:33:32.273070+0000","last_peered":"2024-04-30T08:33:32.273070+0000","last_clean":"2024-04-30T08:33:32.273070+0000","last_became_active":"2024-04-30T08:31:57.731893+0000","last_became_peered":"2024-04-30T08:31:57.731893+0000","last_unstale":"2024-04-30T08:33:32.273070+0000","last_undegraded":"2024-04-30T08:33:32.273070+0000","last_fullsized":"2024-04-30T08:33:32.273070+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:08:55.243526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"81'2","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886769+0000","last_change":"2024-04-30T08:31:51.671991+0000","last_active":"2024-04-30T08:33:31.886769+0000","last_peered":"2024-04-30T08:33:31.886769+0000","last_clean":"2024-04-30T08:33:31.886769+0000","last_became_active":"2024-04-30T08:31:51.671508+0000","last_became_peered":"2024-04-30T08:31:51.671508+0000","last_unstale":"2024-04-30T08:33:31.886769+0000","last_undegraded":"2024-04-30T08:33:31.886769+0000","last_fullsized":"2024-04-30T08:33:31.886769+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:54:30.106748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.274901+0000","last_change":"2024-04-30T08:33:32.274901+0000","last_active":"2024-04-30T08:33:32.274901+0000","last_peered":"2024-04-30T08:33:32.274901+0000","last_clean":"2024-04-30T08:33:32.274901+0000","last_became_active":"2024-04-30T08:33:32.274401+0000","last_became_peered":"2024-04-30T08:33:32.274401+0000","last_unstale":"2024-04-30T08:33:32.274901+0000","last_undegraded":"2024-04-30T08:33:32.274901+0000","last_fullsized":"2024-04-30T08:33:32.274901+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:23:08.703092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886029+0000","last_change":"2024-04-30T08:31:58.061077+0000","last_active":"2024-04-30T08:33:31.886029+0000","last_peered":"2024-04-30T08:33:31.886029+0000","last_clean":"2024-04-30T08:33:31.886029+0000","last_became_active":"2024-04-30T08:31:58.060486+0000","last_became_peered":"2024-04-30T08:31:58.060486+0000","last_unstale":"2024-04-30T08:33:31.886029+0000","last_undegraded":"2024-04-30T08:33:31.886029+0000","last_fullsized":"2024-04-30T08:33:31.886029+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:31:50.936560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"81'1","reported_seq":34,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.885199+0000","last_change":"2024-04-30T08:31:51.702869+0000","last_active":"2024-04-30T08:33:31.885199+0000","last_peered":"2024-04-30T08:33:31.885199+0000","last_clean":"2024-04-30T08:33:31.885199+0000","last_became_active":"2024-04-30T08:31:51.701150+0000","last_became_peered":"2024-04-30T08:31:51.701150+0000","last_unstale":"2024-04-30T08:33:31.885199+0000","last_undegraded":"2024-04-30T08:33:31.885199+0000","last_fullsized":"2024-04-30T08:33:31.885199+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:47:11.010310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881903+0000","last_change":"2024-04-30T08:31:52.709159+0000","last_active":"2024-04-30T08:33:31.881903+0000","last_peered":"2024-04-30T08:33:31.881903+0000","last_clean":"2024-04-30T08:33:31.881903+0000","last_became_active":"2024-04-30T08:31:52.708799+0000","last_became_peered":"2024-04-30T08:31:52.708799+0000","last_unstale":"2024-04-30T08:33:31.881903+0000","last_undegraded":"2024-04-30T08:33:31.881903+0000","last_fullsized":"2024-04-30T08:33:31.881903+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:51:10.588599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881128+0000","last_change":"2024-04-30T08:31:57.751652+0000","last_active":"2024-04-30T08:33:31.881128+0000","last_peered":"2024-04-30T08:33:31.881128+0000","last_clean":"2024-04-30T08:33:31.881128+0000","last_became_active":"2024-04-30T08:31:57.751176+0000","last_became_peered":"2024-04-30T08:31:57.751176+0000","last_unstale":"2024-04-30T08:33:31.881128+0000","last_undegraded":"2024-04-30T08:33:31.881128+0000","last_fullsized":"2024-04-30T08:33:31.881128+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:13:24.528842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"81'5","reported_seq":128,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:57.767303+0000","last_change":"2024-04-30T08:31:51.700816+0000","last_active":"2024-04-30T08:33:57.767303+0000","last_peered":"2024-04-30T08:33:57.767303+0000","last_clean":"2024-04-30T08:33:57.767303+0000","last_became_active":"2024-04-30T08:31:51.700470+0000","last_became_peered":"2024-04-30T08:31:51.700470+0000","last_unstale":"2024-04-30T08:33:57.767303+0000","last_undegraded":"2024-04-30T08:33:57.767303+0000","last_fullsized":"2024-04-30T08:33:57.767303+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:49:11.793412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":89,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881562+0000","last_change":"2024-04-30T08:31:52.715236+0000","last_active":"2024-04-30T08:33:31.881562+0000","last_peered":"2024-04-30T08:33:31.881562+0000","last_clean":"2024-04-30T08:33:31.881562+0000","last_became_active":"2024-04-30T08:31:52.714801+0000","last_became_peered":"2024-04-30T08:31:52.714801+0000","last_unstale":"2024-04-30T08:33:31.881562+0000","last_undegraded":"2024-04-30T08:33:31.881562+0000","last_fullsized":"2024-04-30T08:33:31.881562+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:43:09.650002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881625+0000","last_change":"2024-04-30T08:31:58.060927+0000","last_active":"2024-04-30T08:33:31.881625+0000","last_peered":"2024-04-30T08:33:31.881625+0000","last_clean":"2024-04-30T08:33:31.881625+0000","last_became_active":"2024-04-30T08:31:58.060182+0000","last_became_peered":"2024-04-30T08:31:58.060182+0000","last_unstale":"2024-04-30T08:33:31.881625+0000","last_undegraded":"2024-04-30T08:33:31.881625+0000","last_fullsized":"2024-04-30T08:33:31.881625+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:21:00.228064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273509+0000","last_change":"2024-04-30T08:32:31.560485+0000","last_active":"2024-04-30T08:33:32.273509+0000","last_peered":"2024-04-30T08:33:32.273509+0000","last_clean":"2024-04-30T08:33:32.273509+0000","last_became_active":"2024-04-30T08:32:31.560066+0000","last_became_peered":"2024-04-30T08:32:31.560066+0000","last_unstale":"2024-04-30T08:33:32.273509+0000","last_undegraded":"2024-04-30T08:33:32.273509+0000","last_fullsized":"2024-04-30T08:33:32.273509+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:13:40.769761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"81'5","reported_seq":123,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:53.759720+0000","last_change":"2024-04-30T08:31:51.702808+0000","last_active":"2024-04-30T08:33:53.759720+0000","last_peered":"2024-04-30T08:33:53.759720+0000","last_clean":"2024-04-30T08:33:53.759720+0000","last_became_active":"2024-04-30T08:31:51.702326+0000","last_became_peered":"2024-04-30T08:31:51.702326+0000","last_unstale":"2024-04-30T08:33:53.759720+0000","last_undegraded":"2024-04-30T08:33:53.759720+0000","last_fullsized":"2024-04-30T08:33:53.759720+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:11:35.552670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":86,"num_read_kb":86,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"81'1","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.218441+0000","last_change":"2024-04-30T08:31:57.760359+0000","last_active":"2024-04-30T08:33:32.218441+0000","last_peered":"2024-04-30T08:33:32.218441+0000","last_clean":"2024-04-30T08:33:32.218441+0000","last_became_active":"2024-04-30T08:31:57.759821+0000","last_became_peered":"2024-04-30T08:31:57.759821+0000","last_unstale":"2024-04-30T08:33:32.218441+0000","last_undegraded":"2024-04-30T08:33:32.218441+0000","last_fullsized":"2024-04-30T08:33:32.218441+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:50:17.853791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.275599+0000","last_change":"2024-04-30T08:31:52.689281+0000","last_active":"2024-04-30T08:33:32.275599+0000","last_peered":"2024-04-30T08:33:32.275599+0000","last_clean":"2024-04-30T08:33:32.275599+0000","last_became_active":"2024-04-30T08:31:52.688834+0000","last_became_peered":"2024-04-30T08:31:52.688834+0000","last_unstale":"2024-04-30T08:33:32.275599+0000","last_undegraded":"2024-04-30T08:33:32.275599+0000","last_fullsized":"2024-04-30T08:33:32.275599+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:42:12.836649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'6","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.169972+0000","last_change":"2024-04-30T08:31:51.683712+0000","last_active":"2024-04-30T08:33:32.169972+0000","last_peered":"2024-04-30T08:33:32.169972+0000","last_clean":"2024-04-30T08:33:32.169972+0000","last_became_active":"2024-04-30T08:31:51.683298+0000","last_became_peered":"2024-04-30T08:31:51.683298+0000","last_unstale":"2024-04-30T08:33:32.169972+0000","last_undegraded":"2024-04-30T08:33:32.169972+0000","last_fullsized":"2024-04-30T08:33:32.169972+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:43:08.832104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":108,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.217728+0000","last_change":"2024-04-30T08:30:26.503221+0000","last_active":"2024-04-30T08:33:32.217728+0000","last_peered":"2024-04-30T08:33:32.217728+0000","last_clean":"2024-04-30T08:33:32.217728+0000","last_became_active":"2024-04-30T08:30:26.498399+0000","last_became_peered":"2024-04-30T08:30:26.498399+0000","last_unstale":"2024-04-30T08:33:32.217728+0000","last_undegraded":"2024-04-30T08:33:32.217728+0000","last_fullsized":"2024-04-30T08:33:32.217728+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:26:20.170075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:26:20.170075+0000","last_clean_scrub_stamp":"2024-04-30T08:26:20.170075+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:34:44.067560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273473+0000","last_change":"2024-04-30T08:31:58.061283+0000","last_active":"2024-04-30T08:33:32.273473+0000","last_peered":"2024-04-30T08:33:32.273473+0000","last_clean":"2024-04-30T08:33:32.273473+0000","last_became_active":"2024-04-30T08:31:58.060650+0000","last_became_peered":"2024-04-30T08:31:58.060650+0000","last_unstale":"2024-04-30T08:33:32.273473+0000","last_undegraded":"2024-04-30T08:33:32.273473+0000","last_fullsized":"2024-04-30T08:33:32.273473+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:13:01.811462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.879517+0000","last_change":"2024-04-30T08:31:52.699639+0000","last_active":"2024-04-30T08:33:31.879517+0000","last_peered":"2024-04-30T08:33:31.879517+0000","last_clean":"2024-04-30T08:33:31.879517+0000","last_became_active":"2024-04-30T08:31:52.699164+0000","last_became_peered":"2024-04-30T08:31:52.699164+0000","last_unstale":"2024-04-30T08:33:31.879517+0000","last_undegraded":"2024-04-30T08:33:31.879517+0000","last_fullsized":"2024-04-30T08:33:31.879517+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:22:18.873504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"81'4","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.217821+0000","last_change":"2024-04-30T08:31:51.679153+0000","last_active":"2024-04-30T08:33:32.217821+0000","last_peered":"2024-04-30T08:33:32.217821+0000","last_clean":"2024-04-30T08:33:32.217821+0000","last_became_active":"2024-04-30T08:31:51.678348+0000","last_became_peered":"2024-04-30T08:31:51.678348+0000","last_unstale":"2024-04-30T08:33:32.217821+0000","last_undegraded":"2024-04-30T08:33:32.217821+0000","last_fullsized":"2024-04-30T08:33:32.217821+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:02:56.240218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886545+0000","last_change":"2024-04-30T08:31:57.755423+0000","last_active":"2024-04-30T08:33:31.886545+0000","last_peered":"2024-04-30T08:33:31.886545+0000","last_clean":"2024-04-30T08:33:31.886545+0000","last_became_active":"2024-04-30T08:31:57.754835+0000","last_became_peered":"2024-04-30T08:31:57.754835+0000","last_unstale":"2024-04-30T08:33:31.886545+0000","last_undegraded":"2024-04-30T08:33:31.886545+0000","last_fullsized":"2024-04-30T08:33:31.886545+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:19:25.168920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"81'4","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886919+0000","last_change":"2024-04-30T08:31:51.693839+0000","last_active":"2024-04-30T08:33:31.886919+0000","last_peered":"2024-04-30T08:33:31.886919+0000","last_clean":"2024-04-30T08:33:31.886919+0000","last_became_active":"2024-04-30T08:31:51.693349+0000","last_became_peered":"2024-04-30T08:31:51.693349+0000","last_unstale":"2024-04-30T08:33:31.886919+0000","last_undegraded":"2024-04-30T08:33:31.886919+0000","last_fullsized":"2024-04-30T08:33:31.886919+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:41:51.330901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881271+0000","last_change":"2024-04-30T08:31:52.712360+0000","last_active":"2024-04-30T08:33:31.881271+0000","last_peered":"2024-04-30T08:33:31.881271+0000","last_clean":"2024-04-30T08:33:31.881271+0000","last_became_active":"2024-04-30T08:31:52.711652+0000","last_became_peered":"2024-04-30T08:31:52.711652+0000","last_unstale":"2024-04-30T08:33:31.881271+0000","last_undegraded":"2024-04-30T08:33:31.881271+0000","last_fullsized":"2024-04-30T08:33:31.881271+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:21:10.285386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.276577+0000","last_change":"2024-04-30T08:31:58.061654+0000","last_active":"2024-04-30T08:33:32.276577+0000","last_peered":"2024-04-30T08:33:32.276577+0000","last_clean":"2024-04-30T08:33:32.276577+0000","last_became_active":"2024-04-30T08:31:58.060181+0000","last_became_peered":"2024-04-30T08:31:58.060181+0000","last_unstale":"2024-04-30T08:33:32.276577+0000","last_undegraded":"2024-04-30T08:33:32.276577+0000","last_fullsized":"2024-04-30T08:33:32.276577+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:46:03.230626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"83'12","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886549+0000","last_change":"2024-04-30T08:31:51.706940+0000","last_active":"2024-04-30T08:33:31.886549+0000","last_peered":"2024-04-30T08:33:31.886549+0000","last_clean":"2024-04-30T08:33:31.886549+0000","last_became_active":"2024-04-30T08:31:51.706481+0000","last_became_peered":"2024-04-30T08:31:51.706481+0000","last_unstale":"2024-04-30T08:33:31.886549+0000","last_undegraded":"2024-04-30T08:33:31.886549+0000","last_fullsized":"2024-04-30T08:33:31.886549+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:53:31.955595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.277222+0000","last_change":"2024-04-30T08:31:52.696625+0000","last_active":"2024-04-30T08:33:32.277222+0000","last_peered":"2024-04-30T08:33:32.277222+0000","last_clean":"2024-04-30T08:33:32.277222+0000","last_became_active":"2024-04-30T08:31:52.696230+0000","last_became_peered":"2024-04-30T08:31:52.696230+0000","last_unstale":"2024-04-30T08:33:32.277222+0000","last_undegraded":"2024-04-30T08:33:32.277222+0000","last_fullsized":"2024-04-30T08:33:32.277222+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:11:46.404251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.879348+0000","last_change":"2024-04-30T08:31:57.738329+0000","last_active":"2024-04-30T08:33:31.879348+0000","last_peered":"2024-04-30T08:33:31.879348+0000","last_clean":"2024-04-30T08:33:31.879348+0000","last_became_active":"2024-04-30T08:31:57.737897+0000","last_became_peered":"2024-04-30T08:31:57.737897+0000","last_unstale":"2024-04-30T08:33:31.879348+0000","last_undegraded":"2024-04-30T08:33:31.879348+0000","last_fullsized":"2024-04-30T08:33:31.879348+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:02:43.074578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"81'7","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.170306+0000","last_change":"2024-04-30T08:31:51.684066+0000","last_active":"2024-04-30T08:33:32.170306+0000","last_peered":"2024-04-30T08:33:32.170306+0000","last_clean":"2024-04-30T08:33:32.170306+0000","last_became_active":"2024-04-30T08:31:51.683699+0000","last_became_peered":"2024-04-30T08:31:51.683699+0000","last_unstale":"2024-04-30T08:33:32.170306+0000","last_undegraded":"2024-04-30T08:33:32.170306+0000","last_fullsized":"2024-04-30T08:33:32.170306+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:36:20.900733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.799887+0000","last_change":"2024-04-30T08:31:52.715219+0000","last_active":"2024-04-30T08:33:32.799887+0000","last_peered":"2024-04-30T08:33:32.799887+0000","last_clean":"2024-04-30T08:33:32.799887+0000","last_became_active":"2024-04-30T08:31:52.714774+0000","last_became_peered":"2024-04-30T08:31:52.714774+0000","last_unstale":"2024-04-30T08:33:32.799887+0000","last_undegraded":"2024-04-30T08:33:32.799887+0000","last_fullsized":"2024-04-30T08:33:32.799887+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:42:41.082610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.170102+0000","last_change":"2024-04-30T08:31:58.060845+0000","last_active":"2024-04-30T08:33:32.170102+0000","last_peered":"2024-04-30T08:33:32.170102+0000","last_clean":"2024-04-30T08:33:32.170102+0000","last_became_active":"2024-04-30T08:31:58.060365+0000","last_became_peered":"2024-04-30T08:31:58.060365+0000","last_unstale":"2024-04-30T08:33:32.170102+0000","last_undegraded":"2024-04-30T08:33:32.170102+0000","last_fullsized":"2024-04-30T08:33:32.170102+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:21:39.583243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"81'2","reported_seq":34,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.272525+0000","last_change":"2024-04-30T08:31:51.694833+0000","last_active":"2024-04-30T08:33:32.272525+0000","last_peered":"2024-04-30T08:33:32.272525+0000","last_clean":"2024-04-30T08:33:32.272525+0000","last_became_active":"2024-04-30T08:31:51.694401+0000","last_became_peered":"2024-04-30T08:31:51.694401+0000","last_unstale":"2024-04-30T08:33:32.272525+0000","last_undegraded":"2024-04-30T08:33:32.272525+0000","last_fullsized":"2024-04-30T08:33:32.272525+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:38:52.605625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.885889+0000","last_change":"2024-04-30T08:31:52.686444+0000","last_active":"2024-04-30T08:33:31.885889+0000","last_peered":"2024-04-30T08:33:31.885889+0000","last_clean":"2024-04-30T08:33:31.885889+0000","last_became_active":"2024-04-30T08:31:52.686057+0000","last_became_peered":"2024-04-30T08:31:52.686057+0000","last_unstale":"2024-04-30T08:33:31.885889+0000","last_undegraded":"2024-04-30T08:33:31.885889+0000","last_fullsized":"2024-04-30T08:33:31.885889+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:54:29.824060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.885328+0000","last_change":"2024-04-30T08:31:57.766193+0000","last_active":"2024-04-30T08:33:31.885328+0000","last_peered":"2024-04-30T08:33:31.885328+0000","last_clean":"2024-04-30T08:33:31.885328+0000","last_became_active":"2024-04-30T08:31:57.765758+0000","last_became_peered":"2024-04-30T08:31:57.765758+0000","last_unstale":"2024-04-30T08:33:31.885328+0000","last_undegraded":"2024-04-30T08:33:31.885328+0000","last_fullsized":"2024-04-30T08:33:31.885328+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:59:57.282795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"81'7","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.275484+0000","last_change":"2024-04-30T08:31:51.704565+0000","last_active":"2024-04-30T08:33:32.275484+0000","last_peered":"2024-04-30T08:33:32.275484+0000","last_clean":"2024-04-30T08:33:32.275484+0000","last_became_active":"2024-04-30T08:31:51.703954+0000","last_became_peered":"2024-04-30T08:31:51.703954+0000","last_unstale":"2024-04-30T08:33:32.275484+0000","last_undegraded":"2024-04-30T08:33:32.275484+0000","last_fullsized":"2024-04-30T08:33:32.275484+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:03:00.517867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886804+0000","last_change":"2024-04-30T08:31:52.706043+0000","last_active":"2024-04-30T08:33:31.886804+0000","last_peered":"2024-04-30T08:33:31.886804+0000","last_clean":"2024-04-30T08:33:31.886804+0000","last_became_active":"2024-04-30T08:31:52.705678+0000","last_became_peered":"2024-04-30T08:31:52.705678+0000","last_unstale":"2024-04-30T08:33:31.886804+0000","last_undegraded":"2024-04-30T08:33:31.886804+0000","last_fullsized":"2024-04-30T08:33:31.886804+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:44:56.787835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"82'1","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.170030+0000","last_change":"2024-04-30T08:31:57.764791+0000","last_active":"2024-04-30T08:33:32.170030+0000","last_peered":"2024-04-30T08:33:32.170030+0000","last_clean":"2024-04-30T08:33:32.170030+0000","last_became_active":"2024-04-30T08:31:57.764368+0000","last_became_peered":"2024-04-30T08:31:57.764368+0000","last_unstale":"2024-04-30T08:33:32.170030+0000","last_undegraded":"2024-04-30T08:33:32.170030+0000","last_fullsized":"2024-04-30T08:33:32.170030+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:19:28.545483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"81'3","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886360+0000","last_change":"2024-04-30T08:32:31.577309+0000","last_active":"2024-04-30T08:33:31.886360+0000","last_peered":"2024-04-30T08:33:31.886360+0000","last_clean":"2024-04-30T08:33:31.886360+0000","last_became_active":"2024-04-30T08:32:31.571515+0000","last_became_peered":"2024-04-30T08:32:31.571515+0000","last_unstale":"2024-04-30T08:33:31.886360+0000","last_undegraded":"2024-04-30T08:33:31.886360+0000","last_fullsized":"2024-04-30T08:33:31.886360+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:28:48.408414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.275847+0000","last_change":"2024-04-30T08:31:52.716707+0000","last_active":"2024-04-30T08:33:32.275847+0000","last_peered":"2024-04-30T08:33:32.275847+0000","last_clean":"2024-04-30T08:33:32.275847+0000","last_became_active":"2024-04-30T08:31:52.716293+0000","last_became_peered":"2024-04-30T08:31:52.716293+0000","last_unstale":"2024-04-30T08:33:32.275847+0000","last_undegraded":"2024-04-30T08:33:32.275847+0000","last_fullsized":"2024-04-30T08:33:32.275847+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:29:54.223472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.170072+0000","last_change":"2024-04-30T08:31:58.060640+0000","last_active":"2024-04-30T08:33:32.170072+0000","last_peered":"2024-04-30T08:33:32.170072+0000","last_clean":"2024-04-30T08:33:32.170072+0000","last_became_active":"2024-04-30T08:31:58.060061+0000","last_became_peered":"2024-04-30T08:31:58.060061+0000","last_unstale":"2024-04-30T08:33:32.170072+0000","last_undegraded":"2024-04-30T08:33:32.170072+0000","last_fullsized":"2024-04-30T08:33:32.170072+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:46:00.844191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"81'4","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886101+0000","last_change":"2024-04-30T08:31:51.679532+0000","last_active":"2024-04-30T08:33:31.886101+0000","last_peered":"2024-04-30T08:33:31.886101+0000","last_clean":"2024-04-30T08:33:31.886101+0000","last_became_active":"2024-04-30T08:31:51.679141+0000","last_became_peered":"2024-04-30T08:31:51.679141+0000","last_unstale":"2024-04-30T08:33:31.886101+0000","last_undegraded":"2024-04-30T08:33:31.886101+0000","last_fullsized":"2024-04-30T08:33:31.886101+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:19:50.526563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.218018+0000","last_change":"2024-04-30T08:31:52.703613+0000","last_active":"2024-04-30T08:33:32.218018+0000","last_peered":"2024-04-30T08:33:32.218018+0000","last_clean":"2024-04-30T08:33:32.218018+0000","last_became_active":"2024-04-30T08:31:52.703196+0000","last_became_peered":"2024-04-30T08:31:52.703196+0000","last_unstale":"2024-04-30T08:33:32.218018+0000","last_undegraded":"2024-04-30T08:33:32.218018+0000","last_fullsized":"2024-04-30T08:33:32.218018+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:33:11.136313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.272565+0000","last_change":"2024-04-30T08:31:57.755286+0000","last_active":"2024-04-30T08:33:32.272565+0000","last_peered":"2024-04-30T08:33:32.272565+0000","last_clean":"2024-04-30T08:33:32.272565+0000","last_became_active":"2024-04-30T08:31:57.754786+0000","last_became_peered":"2024-04-30T08:31:57.754786+0000","last_unstale":"2024-04-30T08:33:32.272565+0000","last_undegraded":"2024-04-30T08:33:32.272565+0000","last_fullsized":"2024-04-30T08:33:32.272565+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:02:50.931553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"81'5","reported_seq":123,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:53.698061+0000","last_change":"2024-04-30T08:31:51.693317+0000","last_active":"2024-04-30T08:33:53.698061+0000","last_peered":"2024-04-30T08:33:53.698061+0000","last_clean":"2024-04-30T08:33:53.698061+0000","last_became_active":"2024-04-30T08:31:51.692928+0000","last_became_peered":"2024-04-30T08:31:51.692928+0000","last_unstale":"2024-04-30T08:33:53.698061+0000","last_undegraded":"2024-04-30T08:33:53.698061+0000","last_fullsized":"2024-04-30T08:33:53.698061+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:35:09.851735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":86,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.218738+0000","last_change":"2024-04-30T08:31:52.694304+0000","last_active":"2024-04-30T08:33:32.218738+0000","last_peered":"2024-04-30T08:33:32.218738+0000","last_clean":"2024-04-30T08:33:32.218738+0000","last_became_active":"2024-04-30T08:31:52.693317+0000","last_became_peered":"2024-04-30T08:31:52.693317+0000","last_unstale":"2024-04-30T08:33:32.218738+0000","last_undegraded":"2024-04-30T08:33:32.218738+0000","last_fullsized":"2024-04-30T08:33:32.218738+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:31:39.294864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.798945+0000","last_change":"2024-04-30T08:31:57.763598+0000","last_active":"2024-04-30T08:33:32.798945+0000","last_peered":"2024-04-30T08:33:32.798945+0000","last_clean":"2024-04-30T08:33:32.798945+0000","last_became_active":"2024-04-30T08:31:57.762307+0000","last_became_peered":"2024-04-30T08:31:57.762307+0000","last_unstale":"2024-04-30T08:33:32.798945+0000","last_undegraded":"2024-04-30T08:33:32.798945+0000","last_fullsized":"2024-04-30T08:33:32.798945+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:24:13.390595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"81'3","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273690+0000","last_change":"2024-04-30T08:31:51.703476+0000","last_active":"2024-04-30T08:33:32.273690+0000","last_peered":"2024-04-30T08:33:32.273690+0000","last_clean":"2024-04-30T08:33:32.273690+0000","last_became_active":"2024-04-30T08:31:51.703072+0000","last_became_peered":"2024-04-30T08:31:51.703072+0000","last_unstale":"2024-04-30T08:33:32.273690+0000","last_undegraded":"2024-04-30T08:33:32.273690+0000","last_fullsized":"2024-04-30T08:33:32.273690+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:07:21.098223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.218411+0000","last_change":"2024-04-30T08:31:52.699223+0000","last_active":"2024-04-30T08:33:32.218411+0000","last_peered":"2024-04-30T08:33:32.218411+0000","last_clean":"2024-04-30T08:33:32.218411+0000","last_became_active":"2024-04-30T08:31:52.698837+0000","last_became_peered":"2024-04-30T08:31:52.698837+0000","last_unstale":"2024-04-30T08:33:32.218411+0000","last_undegraded":"2024-04-30T08:33:32.218411+0000","last_fullsized":"2024-04-30T08:33:32.218411+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:19:11.826470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886711+0000","last_change":"2024-04-30T08:32:31.586667+0000","last_active":"2024-04-30T08:33:31.886711+0000","last_peered":"2024-04-30T08:33:31.886711+0000","last_clean":"2024-04-30T08:33:31.886711+0000","last_became_active":"2024-04-30T08:32:31.586144+0000","last_became_peered":"2024-04-30T08:32:31.586144+0000","last_unstale":"2024-04-30T08:33:31.886711+0000","last_undegraded":"2024-04-30T08:33:31.886711+0000","last_fullsized":"2024-04-30T08:33:31.886711+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:50:39.752485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7,1],"acting":[8,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"81'5","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881571+0000","last_change":"2024-04-30T08:31:51.708943+0000","last_active":"2024-04-30T08:33:31.881571+0000","last_peered":"2024-04-30T08:33:31.881571+0000","last_clean":"2024-04-30T08:33:31.881571+0000","last_became_active":"2024-04-30T08:31:51.708376+0000","last_became_peered":"2024-04-30T08:31:51.708376+0000","last_unstale":"2024-04-30T08:33:31.881571+0000","last_undegraded":"2024-04-30T08:33:31.881571+0000","last_fullsized":"2024-04-30T08:33:31.881571+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:14:46.692192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.218781+0000","last_change":"2024-04-30T08:31:52.686634+0000","last_active":"2024-04-30T08:33:32.218781+0000","last_peered":"2024-04-30T08:33:32.218781+0000","last_clean":"2024-04-30T08:33:32.218781+0000","last_became_active":"2024-04-30T08:31:52.686231+0000","last_became_peered":"2024-04-30T08:31:52.686231+0000","last_unstale":"2024-04-30T08:33:32.218781+0000","last_undegraded":"2024-04-30T08:33:32.218781+0000","last_fullsized":"2024-04-30T08:33:32.218781+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:27:31.073041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886909+0000","last_change":"2024-04-30T08:31:52.688695+0000","last_active":"2024-04-30T08:33:31.886909+0000","last_peered":"2024-04-30T08:33:31.886909+0000","last_clean":"2024-04-30T08:33:31.886909+0000","last_became_active":"2024-04-30T08:31:52.686556+0000","last_became_peered":"2024-04-30T08:31:52.686556+0000","last_unstale":"2024-04-30T08:33:31.886909+0000","last_undegraded":"2024-04-30T08:33:31.886909+0000","last_fullsized":"2024-04-30T08:33:31.886909+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:22:48.976440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.879685+0000","last_change":"2024-04-30T08:31:52.688324+0000","last_active":"2024-04-30T08:33:31.879685+0000","last_peered":"2024-04-30T08:33:31.879685+0000","last_clean":"2024-04-30T08:33:31.879685+0000","last_became_active":"2024-04-30T08:31:52.687966+0000","last_became_peered":"2024-04-30T08:31:52.687966+0000","last_unstale":"2024-04-30T08:33:31.879685+0000","last_undegraded":"2024-04-30T08:33:31.879685+0000","last_fullsized":"2024-04-30T08:33:31.879685+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:35:38.237750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.879470+0000","last_change":"2024-04-30T08:31:52.695638+0000","last_active":"2024-04-30T08:33:31.879470+0000","last_peered":"2024-04-30T08:33:31.879470+0000","last_clean":"2024-04-30T08:33:31.879470+0000","last_became_active":"2024-04-30T08:31:52.695283+0000","last_became_peered":"2024-04-30T08:31:52.695283+0000","last_unstale":"2024-04-30T08:33:31.879470+0000","last_undegraded":"2024-04-30T08:33:31.879470+0000","last_fullsized":"2024-04-30T08:33:31.879470+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:00:14.510381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.799495+0000","last_change":"2024-04-30T08:31:52.719112+0000","last_active":"2024-04-30T08:33:32.799495+0000","last_peered":"2024-04-30T08:33:32.799495+0000","last_clean":"2024-04-30T08:33:32.799495+0000","last_became_active":"2024-04-30T08:31:52.718207+0000","last_became_peered":"2024-04-30T08:31:52.718207+0000","last_unstale":"2024-04-30T08:33:32.799495+0000","last_undegraded":"2024-04-30T08:33:32.799495+0000","last_fullsized":"2024-04-30T08:33:32.799495+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:27:29.263962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881008+0000","last_change":"2024-04-30T08:31:52.712008+0000","last_active":"2024-04-30T08:33:31.881008+0000","last_peered":"2024-04-30T08:33:31.881008+0000","last_clean":"2024-04-30T08:33:31.881008+0000","last_became_active":"2024-04-30T08:31:52.711064+0000","last_became_peered":"2024-04-30T08:31:52.711064+0000","last_unstale":"2024-04-30T08:33:31.881008+0000","last_undegraded":"2024-04-30T08:33:31.881008+0000","last_fullsized":"2024-04-30T08:33:31.881008+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:57:27.400602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.887037+0000","last_change":"2024-04-30T08:31:52.691210+0000","last_active":"2024-04-30T08:33:31.887037+0000","last_peered":"2024-04-30T08:33:31.887037+0000","last_clean":"2024-04-30T08:33:31.887037+0000","last_became_active":"2024-04-30T08:31:52.688852+0000","last_became_peered":"2024-04-30T08:31:52.688852+0000","last_unstale":"2024-04-30T08:33:31.887037+0000","last_undegraded":"2024-04-30T08:33:31.887037+0000","last_fullsized":"2024-04-30T08:33:31.887037+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:04:46.502743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.880209+0000","last_change":"2024-04-30T08:31:52.684501+0000","last_active":"2024-04-30T08:33:31.880209+0000","last_peered":"2024-04-30T08:33:31.880209+0000","last_clean":"2024-04-30T08:33:31.880209+0000","last_became_active":"2024-04-30T08:31:52.684122+0000","last_became_peered":"2024-04-30T08:31:52.684122+0000","last_unstale":"2024-04-30T08:33:31.880209+0000","last_undegraded":"2024-04-30T08:33:31.880209+0000","last_fullsized":"2024-04-30T08:33:31.880209+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:55:58.093269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.277504+0000","last_change":"2024-04-30T08:31:52.698016+0000","last_active":"2024-04-30T08:33:32.277504+0000","last_peered":"2024-04-30T08:33:32.277504+0000","last_clean":"2024-04-30T08:33:32.277504+0000","last_became_active":"2024-04-30T08:31:52.695037+0000","last_became_peered":"2024-04-30T08:31:52.695037+0000","last_unstale":"2024-04-30T08:33:32.277504+0000","last_undegraded":"2024-04-30T08:33:32.277504+0000","last_fullsized":"2024-04-30T08:33:32.277504+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:06:01.360947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.889900+0000","last_change":"2024-04-30T08:33:31.889445+0000","last_active":"2024-04-30T08:33:31.889900+0000","last_peered":"2024-04-30T08:33:31.889900+0000","last_clean":"2024-04-30T08:33:31.889900+0000","last_became_active":"2024-04-30T08:33:31.889110+0000","last_became_peered":"2024-04-30T08:33:31.889110+0000","last_unstale":"2024-04-30T08:33:31.889900+0000","last_undegraded":"2024-04-30T08:33:31.889900+0000","last_fullsized":"2024-04-30T08:33:31.889900+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:34:26.741854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.275608+0000","last_change":"2024-04-30T08:31:52.694776+0000","last_active":"2024-04-30T08:33:32.275608+0000","last_peered":"2024-04-30T08:33:32.275608+0000","last_clean":"2024-04-30T08:33:32.275608+0000","last_became_active":"2024-04-30T08:31:52.693532+0000","last_became_peered":"2024-04-30T08:31:52.693532+0000","last_unstale":"2024-04-30T08:33:32.275608+0000","last_undegraded":"2024-04-30T08:33:32.275608+0000","last_fullsized":"2024-04-30T08:33:32.275608+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:31:10.186667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.272820+0000","last_change":"2024-04-30T08:31:52.688477+0000","last_active":"2024-04-30T08:33:32.272820+0000","last_peered":"2024-04-30T08:33:32.272820+0000","last_clean":"2024-04-30T08:33:32.272820+0000","last_became_active":"2024-04-30T08:31:52.688072+0000","last_became_peered":"2024-04-30T08:31:52.688072+0000","last_unstale":"2024-04-30T08:33:32.272820+0000","last_undegraded":"2024-04-30T08:33:32.272820+0000","last_fullsized":"2024-04-30T08:33:32.272820+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:35:42.659707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.880875+0000","last_change":"2024-04-30T08:31:52.716788+0000","last_active":"2024-04-30T08:33:31.880875+0000","last_peered":"2024-04-30T08:33:31.880875+0000","last_clean":"2024-04-30T08:33:31.880875+0000","last_became_active":"2024-04-30T08:31:52.716336+0000","last_became_peered":"2024-04-30T08:31:52.716336+0000","last_unstale":"2024-04-30T08:33:31.880875+0000","last_undegraded":"2024-04-30T08:33:31.880875+0000","last_fullsized":"2024-04-30T08:33:31.880875+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:23:23.477968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886684+0000","last_change":"2024-04-30T08:31:52.684571+0000","last_active":"2024-04-30T08:33:31.886684+0000","last_peered":"2024-04-30T08:33:31.886684+0000","last_clean":"2024-04-30T08:33:31.886684+0000","last_became_active":"2024-04-30T08:31:52.684192+0000","last_became_peered":"2024-04-30T08:31:52.684192+0000","last_unstale":"2024-04-30T08:33:31.886684+0000","last_undegraded":"2024-04-30T08:33:31.886684+0000","last_fullsized":"2024-04-30T08:33:31.886684+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:05:57.520133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284927,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":69276,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":908,"num_read_kb":529,"num_write":240,"num_write_kb":139,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":552960,"data_stored":207828,"data_compressed":5523,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":209,"ondisk_log_size":209,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678056,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":946176,"data_stored":479251,"data_compressed":26652,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874285,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1081344,"data_stored":1128462,"data_compressed":64719,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037811,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":483209,"data_compressed":26652,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234044,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":958464,"data_stored":479491,"data_compressed":26652,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430272,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32168,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712984,"statfs":{"total":95995035648,"available":95962095616,"internally_reserved":0,"allocated":1011712,"data_stored":1076379,"data_compressed":62878,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626500,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":937984,"data_stored":474617,"data_compressed":26652,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574947,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":999424,"data_stored":537719,"data_compressed":28493,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738471,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":962560,"data_stored":476073,"data_compressed":26652,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411422,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1011712,"data_stored":538262,"data_compressed":28493,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182487,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":950272,"data_stored":474325,"data_compressed":26652,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986258,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":946176,"data_stored":479082,"data_compressed":26652,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822732,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1019904,"data_stored":1071570,"data_compressed":62878,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34599,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34599,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34599,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":63824,"data_compressed":1841,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":64367,"data_compressed":1841,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3211,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3924,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":64199,"data_compressed":1841,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-30T08:34:01.013 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-30T08:34:01.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:00 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:34:01.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:00 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:34:01.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:00 smithi189 ceph-mon[33038]: pgmap v390: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-30T08:34:01.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:00 smithi189 ceph-mon[33038]: from='client.25076 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:34:01.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:00 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:34:01.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:00 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:34:01.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:00 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:34:01.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:00 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:34:01.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:01 smithi032 ceph-mon[28331]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-04-30T08:34:02.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:01 smithi071 ceph-mon[33385]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-04-30T08:34:02.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:01 smithi189 ceph-mon[33038]: mgrmap e18: x(active, since 10m), standbys: z, y 2024-04-30T08:34:02.100 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:34:02.760 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:02 smithi032 ceph-mon[28331]: pgmap v391: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-30T08:34:03.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:02 smithi071 ceph-mon[33385]: pgmap v391: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-30T08:34:03.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:02 smithi189 ceph-mon[33038]: pgmap v391: 129 pgs: 129 active+clean; 644 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 85 B/s wr, 9 op/s 2024-04-30T08:34:04.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.572+0000 7fb460a00700 1 -- 172.21.15.32:0/3996872670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb45c108e70 msgr2=0x7fb45c109280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:04.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.572+0000 7fb460a00700 1 --2- 172.21.15.32:0/3996872670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb45c108e70 0x7fb45c109280 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fb4480099c0 tx=0x7fb448009cd0 comp rx=0 tx=0).stop 2024-04-30T08:34:04.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.572+0000 7fb460a00700 1 -- 172.21.15.32:0/3996872670 shutdown_connections 2024-04-30T08:34:04.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.572+0000 7fb460a00700 1 --2- 172.21.15.32:0/3996872670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb45c1097c0 0x7fb45c111120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:04.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.572+0000 7fb460a00700 1 --2- 172.21.15.32:0/3996872670 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb45c104540 0x7fb45c104950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:04.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.572+0000 7fb460a00700 1 --2- 172.21.15.32:0/3996872670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb45c108e70 0x7fb45c109280 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:04.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.572+0000 7fb460a00700 1 -- 172.21.15.32:0/3996872670 >> 172.21.15.32:0/3996872670 conn(0x7fb45c0fff90 msgr2=0x7fb45c1023b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:04.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.572+0000 7fb460a00700 1 -- 172.21.15.32:0/3996872670 shutdown_connections 2024-04-30T08:34:04.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.572+0000 7fb460a00700 1 -- 172.21.15.32:0/3996872670 wait complete. 2024-04-30T08:34:04.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.573+0000 7fb460a00700 1 Processor -- start 2024-04-30T08:34:04.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.574+0000 7fb460a00700 1 -- start start 2024-04-30T08:34:04.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.574+0000 7fb460a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb45c104540 0x7fb45c19adf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:04.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.574+0000 7fb460a00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb45c108e70 0x7fb45c19b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:04.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.574+0000 7fb460a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb45c1097c0 0x7fb45c19f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:04.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.574+0000 7fb460a00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb45c114420 con 0x7fb45c104540 2024-04-30T08:34:04.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.574+0000 7fb460a00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb45c1142a0 con 0x7fb45c108e70 2024-04-30T08:34:04.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.574+0000 7fb460a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb45c1145a0 con 0x7fb45c1097c0 2024-04-30T08:34:04.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.575+0000 7fb459e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb45c104540 0x7fb45c19adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:04.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.575+0000 7fb459e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb45c104540 0x7fb45c19adf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43826/0 (socket says 172.21.15.32:43826) 2024-04-30T08:34:04.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.575+0000 7fb459e00700 1 -- 172.21.15.32:0/3265482647 learned_addr learned my addr 172.21.15.32:0/3265482647 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:04.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.575+0000 7fb459400700 1 --2- 172.21.15.32:0/3265482647 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb45c108e70 0x7fb45c19b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:04.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.575+0000 7fb45a800700 1 --2- 172.21.15.32:0/3265482647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb45c1097c0 0x7fb45c19f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:04.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.575+0000 7fb459e00700 1 -- 172.21.15.32:0/3265482647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb45c1097c0 msgr2=0x7fb45c19f620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:04.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.575+0000 7fb459e00700 1 --2- 172.21.15.32:0/3265482647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb45c1097c0 0x7fb45c19f620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:04.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.575+0000 7fb459e00700 1 -- 172.21.15.32:0/3265482647 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb45c108e70 msgr2=0x7fb45c19b390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:04.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.575+0000 7fb459e00700 1 --2- 172.21.15.32:0/3265482647 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb45c108e70 0x7fb45c19b390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:04.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.575+0000 7fb459e00700 1 -- 172.21.15.32:0/3265482647 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb45c19fcc0 con 0x7fb45c104540 2024-04-30T08:34:04.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.575+0000 7fb459400700 1 --2- 172.21.15.32:0/3265482647 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb45c108e70 0x7fb45c19b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:34:04.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.576+0000 7fb459e00700 1 --2- 172.21.15.32:0/3265482647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb45c104540 0x7fb45c19adf0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb44c00e9a0 tx=0x7fb44c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:04.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.576+0000 7fb452a00700 1 -- 172.21.15.32:0/3265482647 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb44c00ca10 con 0x7fb45c104540 2024-04-30T08:34:04.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.576+0000 7fb452a00700 1 -- 172.21.15.32:0/3265482647 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb44c00cbb0 con 0x7fb45c104540 2024-04-30T08:34:04.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.576+0000 7fb452a00700 1 -- 172.21.15.32:0/3265482647 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb44c018680 con 0x7fb45c104540 2024-04-30T08:34:04.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.576+0000 7fb460a00700 1 -- 172.21.15.32:0/3265482647 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb45c19ffb0 con 0x7fb45c104540 2024-04-30T08:34:04.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.576+0000 7fb460a00700 1 -- 172.21.15.32:0/3265482647 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb45c1a78d0 con 0x7fb45c104540 2024-04-30T08:34:04.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.578+0000 7fb460a00700 1 -- 172.21.15.32:0/3265482647 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb45c1059f0 con 0x7fb45c104540 2024-04-30T08:34:04.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.580+0000 7fb452a00700 1 -- 172.21.15.32:0/3265482647 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fb44c010430 con 0x7fb45c104540 2024-04-30T08:34:04.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.583+0000 7fb452a00700 1 --2- 172.21.15.32:0/3265482647 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb4400afb80 0x7fb4400b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:04.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.583+0000 7fb452a00700 1 -- 172.21.15.32:0/3265482647 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fb44c0df1d0 con 0x7fb45c104540 2024-04-30T08:34:04.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.583+0000 7fb459400700 1 --2- 172.21.15.32:0/3265482647 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb4400afb80 0x7fb4400b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:04.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.584+0000 7fb459400700 1 --2- 172.21.15.32:0/3265482647 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb4400afb80 0x7fb4400b2090 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb448000c00 tx=0x7fb44801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:04.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.587+0000 7fb452a00700 1 -- 172.21.15.32:0/3265482647 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb44c0a79a0 con 0x7fb45c104540 2024-04-30T08:34:04.844 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:04 smithi032 ceph-mon[28331]: pgmap v392: 129 pgs: 129 active+clean; 645 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-30T08:34:04.844 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:04 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:34:04.844 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:04 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:34:04.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.842+0000 7fb460a00700 1 -- 172.21.15.32:0/3265482647 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb45c063b50 con 0x7fb4400afb80 2024-04-30T08:34:04.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.854+0000 7fb452a00700 1 -- 172.21.15.32:0/3265482647 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340951 (secure 0 0 0) 0x7fb44807b3f0 con 0x7fb4400afb80 2024-04-30T08:34:04.858 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:34:04.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.859+0000 7fb460a00700 1 -- 172.21.15.32:0/3265482647 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb4400afb80 msgr2=0x7fb4400b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:04.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.859+0000 7fb460a00700 1 --2- 172.21.15.32:0/3265482647 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb4400afb80 0x7fb4400b2090 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb448000c00 tx=0x7fb44801a040 comp rx=0 tx=0).stop 2024-04-30T08:34:04.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.859+0000 7fb460a00700 1 -- 172.21.15.32:0/3265482647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb45c104540 msgr2=0x7fb45c19adf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:04.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.859+0000 7fb460a00700 1 --2- 172.21.15.32:0/3265482647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb45c104540 0x7fb45c19adf0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb44c00e9a0 tx=0x7fb44c00ecb0 comp rx=0 tx=0).stop 2024-04-30T08:34:04.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.859+0000 7fb460a00700 1 -- 172.21.15.32:0/3265482647 shutdown_connections 2024-04-30T08:34:04.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.859+0000 7fb460a00700 1 --2- 172.21.15.32:0/3265482647 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb4400afb80 0x7fb4400b2090 secure :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fb448000c00 tx=0x7fb44801a040 comp rx=0 tx=0).stop 2024-04-30T08:34:04.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.859+0000 7fb460a00700 1 --2- 172.21.15.32:0/3265482647 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb45c1097c0 0x7fb45c19f620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:04.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.859+0000 7fb460a00700 1 --2- 172.21.15.32:0/3265482647 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb45c104540 0x7fb45c19adf0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:04.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.859+0000 7fb460a00700 1 --2- 172.21.15.32:0/3265482647 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb45c108e70 0x7fb45c19b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:04.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.859+0000 7fb460a00700 1 -- 172.21.15.32:0/3265482647 >> 172.21.15.32:0/3265482647 conn(0x7fb45c0fff90 msgr2=0x7fb45c1085e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:04.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.859+0000 7fb460a00700 1 -- 172.21.15.32:0/3265482647 shutdown_connections 2024-04-30T08:34:04.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:04.859+0000 7fb460a00700 1 -- 172.21.15.32:0/3265482647 wait complete. 2024-04-30T08:34:04.867 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-30T08:34:05.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:04 smithi071 ceph-mon[33385]: pgmap v392: 129 pgs: 129 active+clean; 645 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-30T08:34:05.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:04 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:34:05.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:04 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:34:05.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:04 smithi189 ceph-mon[33038]: pgmap v392: 129 pgs: 129 active+clean; 645 KiB data, 360 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-30T08:34:05.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:04 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:34:05.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:04 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:34:05.659 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":392,"stamp":"2024-04-30T08:34:03.759462+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":660160,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1016,"num_read_kb":605,"num_write":370,"num_write_kb":2294,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":2951908,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":749,"ondisk_log_size":749,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":369020,"kb_used_data":11520,"kb_used_omap":18,"kb_used_meta":357357,"kb_avail":1124572804,"statfs":{"total":1151940427776,"available":1151562551296,"internally_reserved":0,"allocated":11796480,"data_stored":7699253,"data_compressed":434025,"data_compressed_allocated":589824,"data_compressed_original":2678784,"omap_allocated":19141,"internal_metadata":365933883},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":47,"apply_latency_ms":47,"commit_latency_ns":47000000,"apply_latency_ns":47000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":282,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":117,"num_read_kb":52,"num_write":5,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005083"},"pg_stats":[{"pgid":"3.3f","version":"85'8","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.737715+0000","last_change":"2024-04-30T08:31:52.711443+0000","last_active":"2024-04-30T08:33:59.737715+0000","last_peered":"2024-04-30T08:33:59.737715+0000","last_clean":"2024-04-30T08:33:59.737715+0000","last_became_active":"2024-04-30T08:31:52.711029+0000","last_became_peered":"2024-04-30T08:31:52.711029+0000","last_unstale":"2024-04-30T08:33:59.737715+0000","last_undegraded":"2024-04-30T08:33:59.737715+0000","last_fullsized":"2024-04-30T08:33:59.737715+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:51:15.208528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"85'4","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.736806+0000","last_change":"2024-04-30T08:31:52.712324+0000","last_active":"2024-04-30T08:33:59.736806+0000","last_peered":"2024-04-30T08:33:59.736806+0000","last_clean":"2024-04-30T08:33:59.736806+0000","last_became_active":"2024-04-30T08:31:52.711923+0000","last_became_peered":"2024-04-30T08:31:52.711923+0000","last_unstale":"2024-04-30T08:33:59.736806+0000","last_undegraded":"2024-04-30T08:33:59.736806+0000","last_fullsized":"2024-04-30T08:33:59.736806+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:29:55.071459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"85'7","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.735218+0000","last_change":"2024-04-30T08:31:52.712936+0000","last_active":"2024-04-30T08:33:59.735218+0000","last_peered":"2024-04-30T08:33:59.735218+0000","last_clean":"2024-04-30T08:33:59.735218+0000","last_became_active":"2024-04-30T08:31:52.712585+0000","last_became_peered":"2024-04-30T08:31:52.712585+0000","last_unstale":"2024-04-30T08:33:59.735218+0000","last_undegraded":"2024-04-30T08:33:59.735218+0000","last_fullsized":"2024-04-30T08:33:59.735218+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:13:43.745561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"85'8","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.688422+0000","last_change":"2024-04-30T08:31:52.686880+0000","last_active":"2024-04-30T08:33:59.688422+0000","last_peered":"2024-04-30T08:33:59.688422+0000","last_clean":"2024-04-30T08:33:59.688422+0000","last_became_active":"2024-04-30T08:31:52.686506+0000","last_became_peered":"2024-04-30T08:31:52.686506+0000","last_unstale":"2024-04-30T08:33:59.688422+0000","last_undegraded":"2024-04-30T08:33:59.688422+0000","last_fullsized":"2024-04-30T08:33:59.688422+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:16:33.390951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886150+0000","last_change":"2024-04-30T08:31:52.706014+0000","last_active":"2024-04-30T08:33:31.886150+0000","last_peered":"2024-04-30T08:33:31.886150+0000","last_clean":"2024-04-30T08:33:31.886150+0000","last_became_active":"2024-04-30T08:31:52.705582+0000","last_became_peered":"2024-04-30T08:31:52.705582+0000","last_unstale":"2024-04-30T08:33:31.886150+0000","last_undegraded":"2024-04-30T08:33:31.886150+0000","last_fullsized":"2024-04-30T08:33:31.886150+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:24:49.804592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881602+0000","last_change":"2024-04-30T08:31:52.690462+0000","last_active":"2024-04-30T08:33:31.881602+0000","last_peered":"2024-04-30T08:33:31.881602+0000","last_clean":"2024-04-30T08:33:31.881602+0000","last_became_active":"2024-04-30T08:31:52.690092+0000","last_became_peered":"2024-04-30T08:31:52.690092+0000","last_unstale":"2024-04-30T08:33:31.881602+0000","last_undegraded":"2024-04-30T08:33:31.881602+0000","last_fullsized":"2024-04-30T08:33:31.881602+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:32:49.769383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"85'5","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.704484+0000","last_change":"2024-04-30T08:31:52.701786+0000","last_active":"2024-04-30T08:33:59.704484+0000","last_peered":"2024-04-30T08:33:59.704484+0000","last_clean":"2024-04-30T08:33:59.704484+0000","last_became_active":"2024-04-30T08:31:52.701402+0000","last_became_peered":"2024-04-30T08:31:52.701402+0000","last_unstale":"2024-04-30T08:33:59.704484+0000","last_undegraded":"2024-04-30T08:33:59.704484+0000","last_fullsized":"2024-04-30T08:33:59.704484+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:58:06.153479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"85'9","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.718793+0000","last_change":"2024-04-30T08:31:52.698037+0000","last_active":"2024-04-30T08:33:59.718793+0000","last_peered":"2024-04-30T08:33:59.718793+0000","last_clean":"2024-04-30T08:33:59.718793+0000","last_became_active":"2024-04-30T08:31:52.697646+0000","last_became_peered":"2024-04-30T08:31:52.697646+0000","last_unstale":"2024-04-30T08:33:59.718793+0000","last_undegraded":"2024-04-30T08:33:59.718793+0000","last_fullsized":"2024-04-30T08:33:59.718793+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:28:59.520091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"85'11","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.691962+0000","last_change":"2024-04-30T08:31:52.680377+0000","last_active":"2024-04-30T08:33:59.691962+0000","last_peered":"2024-04-30T08:33:59.691962+0000","last_clean":"2024-04-30T08:33:59.691962+0000","last_became_active":"2024-04-30T08:31:52.679977+0000","last_became_peered":"2024-04-30T08:31:52.679977+0000","last_unstale":"2024-04-30T08:33:59.691962+0000","last_undegraded":"2024-04-30T08:33:59.691962+0000","last_fullsized":"2024-04-30T08:33:59.691962+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:35:22.488450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881954+0000","last_change":"2024-04-30T08:32:31.573546+0000","last_active":"2024-04-30T08:33:31.881954+0000","last_peered":"2024-04-30T08:33:31.881954+0000","last_clean":"2024-04-30T08:33:31.881954+0000","last_became_active":"2024-04-30T08:32:31.573289+0000","last_became_peered":"2024-04-30T08:32:31.573289+0000","last_unstale":"2024-04-30T08:33:31.881954+0000","last_undegraded":"2024-04-30T08:33:31.881954+0000","last_fullsized":"2024-04-30T08:33:31.881954+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:42:45.862326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"85'6","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.691865+0000","last_change":"2024-04-30T08:31:52.690344+0000","last_active":"2024-04-30T08:33:59.691865+0000","last_peered":"2024-04-30T08:33:59.691865+0000","last_clean":"2024-04-30T08:33:59.691865+0000","last_became_active":"2024-04-30T08:31:52.689857+0000","last_became_peered":"2024-04-30T08:31:52.689857+0000","last_unstale":"2024-04-30T08:33:59.691865+0000","last_undegraded":"2024-04-30T08:33:59.691865+0000","last_fullsized":"2024-04-30T08:33:59.691865+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:45:13.395589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"85'7","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.688301+0000","last_change":"2024-04-30T08:31:52.685151+0000","last_active":"2024-04-30T08:33:59.688301+0000","last_peered":"2024-04-30T08:33:59.688301+0000","last_clean":"2024-04-30T08:33:59.688301+0000","last_became_active":"2024-04-30T08:31:52.684722+0000","last_became_peered":"2024-04-30T08:31:52.684722+0000","last_unstale":"2024-04-30T08:33:59.688301+0000","last_undegraded":"2024-04-30T08:33:59.688301+0000","last_fullsized":"2024-04-30T08:33:59.688301+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:14:00.660478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"85'10","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.723157+0000","last_change":"2024-04-30T08:31:52.714294+0000","last_active":"2024-04-30T08:33:59.723157+0000","last_peered":"2024-04-30T08:33:59.723157+0000","last_clean":"2024-04-30T08:33:59.723157+0000","last_became_active":"2024-04-30T08:31:52.710094+0000","last_became_peered":"2024-04-30T08:31:52.710094+0000","last_unstale":"2024-04-30T08:33:59.723157+0000","last_undegraded":"2024-04-30T08:33:59.723157+0000","last_fullsized":"2024-04-30T08:33:59.723157+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:34:44.203093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"85'7","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.718057+0000","last_change":"2024-04-30T08:31:52.707992+0000","last_active":"2024-04-30T08:33:59.718057+0000","last_peered":"2024-04-30T08:33:59.718057+0000","last_clean":"2024-04-30T08:33:59.718057+0000","last_became_active":"2024-04-30T08:31:52.706456+0000","last_became_peered":"2024-04-30T08:31:52.706456+0000","last_unstale":"2024-04-30T08:33:59.718057+0000","last_undegraded":"2024-04-30T08:33:59.718057+0000","last_fullsized":"2024-04-30T08:33:59.718057+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:45:32.717842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"85'7","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.734748+0000","last_change":"2024-04-30T08:31:52.695605+0000","last_active":"2024-04-30T08:33:59.734748+0000","last_peered":"2024-04-30T08:33:59.734748+0000","last_clean":"2024-04-30T08:33:59.734748+0000","last_became_active":"2024-04-30T08:31:52.692439+0000","last_became_peered":"2024-04-30T08:31:52.692439+0000","last_unstale":"2024-04-30T08:33:59.734748+0000","last_undegraded":"2024-04-30T08:33:59.734748+0000","last_fullsized":"2024-04-30T08:33:59.734748+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:39:38.746567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"85'9","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.740363+0000","last_change":"2024-04-30T08:33:32.802311+0000","last_active":"2024-04-30T08:33:59.740363+0000","last_peered":"2024-04-30T08:33:59.740363+0000","last_clean":"2024-04-30T08:33:59.740363+0000","last_became_active":"2024-04-30T08:33:32.801601+0000","last_became_peered":"2024-04-30T08:33:32.801601+0000","last_unstale":"2024-04-30T08:33:59.740363+0000","last_undegraded":"2024-04-30T08:33:59.740363+0000","last_fullsized":"2024-04-30T08:33:59.740363+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:59:34.877824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0],"acting":[9,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"85'8","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.737740+0000","last_change":"2024-04-30T08:31:52.711439+0000","last_active":"2024-04-30T08:33:59.737740+0000","last_peered":"2024-04-30T08:33:59.737740+0000","last_clean":"2024-04-30T08:33:59.737740+0000","last_became_active":"2024-04-30T08:31:52.711023+0000","last_became_peered":"2024-04-30T08:31:52.711023+0000","last_unstale":"2024-04-30T08:33:59.737740+0000","last_undegraded":"2024-04-30T08:33:59.737740+0000","last_fullsized":"2024-04-30T08:33:59.737740+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:31:00.654348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"85'10","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.721167+0000","last_change":"2024-04-30T08:31:52.708476+0000","last_active":"2024-04-30T08:33:59.721167+0000","last_peered":"2024-04-30T08:33:59.721167+0000","last_clean":"2024-04-30T08:33:59.721167+0000","last_became_active":"2024-04-30T08:31:52.704735+0000","last_became_peered":"2024-04-30T08:31:52.704735+0000","last_unstale":"2024-04-30T08:33:59.721167+0000","last_undegraded":"2024-04-30T08:33:59.721167+0000","last_fullsized":"2024-04-30T08:33:59.721167+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:40:00.011783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"85'7","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.719953+0000","last_change":"2024-04-30T08:31:52.704901+0000","last_active":"2024-04-30T08:33:59.719953+0000","last_peered":"2024-04-30T08:33:59.719953+0000","last_clean":"2024-04-30T08:33:59.719953+0000","last_became_active":"2024-04-30T08:31:52.704636+0000","last_became_peered":"2024-04-30T08:31:52.704636+0000","last_unstale":"2024-04-30T08:33:59.719953+0000","last_undegraded":"2024-04-30T08:33:59.719953+0000","last_fullsized":"2024-04-30T08:33:59.719953+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:08:53.580809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.879520+0000","last_change":"2024-04-30T08:31:57.743342+0000","last_active":"2024-04-30T08:33:31.879520+0000","last_peered":"2024-04-30T08:33:31.879520+0000","last_clean":"2024-04-30T08:33:31.879520+0000","last_became_active":"2024-04-30T08:31:57.742850+0000","last_became_peered":"2024-04-30T08:31:57.742850+0000","last_unstale":"2024-04-30T08:33:31.879520+0000","last_undegraded":"2024-04-30T08:33:31.879520+0000","last_fullsized":"2024-04-30T08:33:31.879520+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:59:10.628822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"81'13","reported_seq":138,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.770609+0000","last_change":"2024-04-30T08:31:51.694984+0000","last_active":"2024-04-30T08:33:59.770609+0000","last_peered":"2024-04-30T08:33:59.770609+0000","last_clean":"2024-04-30T08:33:59.770609+0000","last_became_active":"2024-04-30T08:31:51.694514+0000","last_became_peered":"2024-04-30T08:31:51.694514+0000","last_unstale":"2024-04-30T08:33:59.770609+0000","last_undegraded":"2024-04-30T08:33:59.770609+0000","last_fullsized":"2024-04-30T08:33:59.770609+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:17:03.873090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":92,"num_read_kb":92,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"85'5","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.717378+0000","last_change":"2024-04-30T08:31:52.694276+0000","last_active":"2024-04-30T08:33:59.717378+0000","last_peered":"2024-04-30T08:33:59.717378+0000","last_clean":"2024-04-30T08:33:59.717378+0000","last_became_active":"2024-04-30T08:31:52.693839+0000","last_became_peered":"2024-04-30T08:31:52.693839+0000","last_unstale":"2024-04-30T08:33:59.717378+0000","last_undegraded":"2024-04-30T08:33:59.717378+0000","last_fullsized":"2024-04-30T08:33:59.717378+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:29:21.271040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.169659+0000","last_change":"2024-04-30T08:31:57.754828+0000","last_active":"2024-04-30T08:33:32.169659+0000","last_peered":"2024-04-30T08:33:32.169659+0000","last_clean":"2024-04-30T08:33:32.169659+0000","last_became_active":"2024-04-30T08:31:57.754322+0000","last_became_peered":"2024-04-30T08:31:57.754322+0000","last_unstale":"2024-04-30T08:33:32.169659+0000","last_undegraded":"2024-04-30T08:33:32.169659+0000","last_fullsized":"2024-04-30T08:33:32.169659+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:36:02.560134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"81'2","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.277887+0000","last_change":"2024-04-30T08:31:51.707637+0000","last_active":"2024-04-30T08:33:32.277887+0000","last_peered":"2024-04-30T08:33:32.277887+0000","last_clean":"2024-04-30T08:33:32.277887+0000","last_became_active":"2024-04-30T08:31:51.707233+0000","last_became_peered":"2024-04-30T08:31:51.707233+0000","last_unstale":"2024-04-30T08:33:32.277887+0000","last_undegraded":"2024-04-30T08:33:32.277887+0000","last_fullsized":"2024-04-30T08:33:32.277887+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:14:46.041169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"85'6","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.741653+0000","last_change":"2024-04-30T08:31:52.711442+0000","last_active":"2024-04-30T08:33:59.741653+0000","last_peered":"2024-04-30T08:33:59.741653+0000","last_clean":"2024-04-30T08:33:59.741653+0000","last_became_active":"2024-04-30T08:31:52.710942+0000","last_became_peered":"2024-04-30T08:31:52.710942+0000","last_unstale":"2024-04-30T08:33:59.741653+0000","last_undegraded":"2024-04-30T08:33:59.741653+0000","last_fullsized":"2024-04-30T08:33:59.741653+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:26:56.428770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.169844+0000","last_change":"2024-04-30T08:31:57.760535+0000","last_active":"2024-04-30T08:33:32.169844+0000","last_peered":"2024-04-30T08:33:32.169844+0000","last_clean":"2024-04-30T08:33:32.169844+0000","last_became_active":"2024-04-30T08:31:57.760073+0000","last_became_peered":"2024-04-30T08:31:57.760073+0000","last_unstale":"2024-04-30T08:33:32.169844+0000","last_undegraded":"2024-04-30T08:33:32.169844+0000","last_fullsized":"2024-04-30T08:33:32.169844+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:03:51.359474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"81'12","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.887039+0000","last_change":"2024-04-30T08:31:51.678575+0000","last_active":"2024-04-30T08:33:31.887039+0000","last_peered":"2024-04-30T08:33:31.887039+0000","last_clean":"2024-04-30T08:33:31.887039+0000","last_became_active":"2024-04-30T08:31:51.678114+0000","last_became_peered":"2024-04-30T08:31:51.678114+0000","last_unstale":"2024-04-30T08:33:31.887039+0000","last_undegraded":"2024-04-30T08:33:31.887039+0000","last_fullsized":"2024-04-30T08:33:31.887039+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:10:35.517183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"85'9","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.721239+0000","last_change":"2024-04-30T08:31:52.704137+0000","last_active":"2024-04-30T08:33:59.721239+0000","last_peered":"2024-04-30T08:33:59.721239+0000","last_clean":"2024-04-30T08:33:59.721239+0000","last_became_active":"2024-04-30T08:31:52.703695+0000","last_became_peered":"2024-04-30T08:31:52.703695+0000","last_unstale":"2024-04-30T08:33:59.721239+0000","last_undegraded":"2024-04-30T08:33:59.721239+0000","last_fullsized":"2024-04-30T08:33:59.721239+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:31:05.461106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.169733+0000","last_change":"2024-04-30T08:31:57.755237+0000","last_active":"2024-04-30T08:33:32.169733+0000","last_peered":"2024-04-30T08:33:32.169733+0000","last_clean":"2024-04-30T08:33:32.169733+0000","last_became_active":"2024-04-30T08:31:57.754438+0000","last_became_peered":"2024-04-30T08:31:57.754438+0000","last_unstale":"2024-04-30T08:33:32.169733+0000","last_undegraded":"2024-04-30T08:33:32.169733+0000","last_fullsized":"2024-04-30T08:33:32.169733+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:20:59.035110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"81'5","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886326+0000","last_change":"2024-04-30T08:31:51.673793+0000","last_active":"2024-04-30T08:33:31.886326+0000","last_peered":"2024-04-30T08:33:31.886326+0000","last_clean":"2024-04-30T08:33:31.886326+0000","last_became_active":"2024-04-30T08:31:51.672703+0000","last_became_peered":"2024-04-30T08:31:51.672703+0000","last_unstale":"2024-04-30T08:33:31.886326+0000","last_undegraded":"2024-04-30T08:33:31.886326+0000","last_fullsized":"2024-04-30T08:33:31.886326+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:13:18.959542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"85'15","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.686311+0000","last_change":"2024-04-30T08:31:52.694320+0000","last_active":"2024-04-30T08:33:59.686311+0000","last_peered":"2024-04-30T08:33:59.686311+0000","last_clean":"2024-04-30T08:33:59.686311+0000","last_became_active":"2024-04-30T08:31:52.694032+0000","last_became_peered":"2024-04-30T08:31:52.694032+0000","last_unstale":"2024-04-30T08:33:59.686311+0000","last_undegraded":"2024-04-30T08:33:59.686311+0000","last_fullsized":"2024-04-30T08:33:59.686311+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:33:12.347369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.879955+0000","last_change":"2024-04-30T08:31:57.739984+0000","last_active":"2024-04-30T08:33:31.879955+0000","last_peered":"2024-04-30T08:33:31.879955+0000","last_clean":"2024-04-30T08:33:31.879955+0000","last_became_active":"2024-04-30T08:31:57.739477+0000","last_became_peered":"2024-04-30T08:31:57.739477+0000","last_unstale":"2024-04-30T08:33:31.879955+0000","last_undegraded":"2024-04-30T08:33:31.879955+0000","last_fullsized":"2024-04-30T08:33:31.879955+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:10:11.325419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'7","reported_seq":136,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:34:01.709839+0000","last_change":"2024-04-30T08:31:51.678792+0000","last_active":"2024-04-30T08:34:01.709839+0000","last_peered":"2024-04-30T08:34:01.709839+0000","last_clean":"2024-04-30T08:34:01.709839+0000","last_became_active":"2024-04-30T08:31:51.678094+0000","last_became_peered":"2024-04-30T08:31:51.678094+0000","last_unstale":"2024-04-30T08:34:01.709839+0000","last_undegraded":"2024-04-30T08:34:01.709839+0000","last_fullsized":"2024-04-30T08:34:01.709839+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:03:17.526016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886172+0000","last_change":"2024-04-30T08:32:31.572984+0000","last_active":"2024-04-30T08:33:31.886172+0000","last_peered":"2024-04-30T08:33:31.886172+0000","last_clean":"2024-04-30T08:33:31.886172+0000","last_became_active":"2024-04-30T08:32:31.572536+0000","last_became_peered":"2024-04-30T08:32:31.572536+0000","last_unstale":"2024-04-30T08:33:31.886172+0000","last_undegraded":"2024-04-30T08:33:31.886172+0000","last_fullsized":"2024-04-30T08:33:31.886172+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:46:15.884555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.879808+0000","last_change":"2024-04-30T08:31:57.765047+0000","last_active":"2024-04-30T08:33:31.879808+0000","last_peered":"2024-04-30T08:33:31.879808+0000","last_clean":"2024-04-30T08:33:31.879808+0000","last_became_active":"2024-04-30T08:31:57.764519+0000","last_became_peered":"2024-04-30T08:31:57.764519+0000","last_unstale":"2024-04-30T08:33:31.879808+0000","last_undegraded":"2024-04-30T08:33:31.879808+0000","last_fullsized":"2024-04-30T08:33:31.879808+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:06:16.126396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"81'7","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881554+0000","last_change":"2024-04-30T08:31:51.686403+0000","last_active":"2024-04-30T08:33:31.881554+0000","last_peered":"2024-04-30T08:33:31.881554+0000","last_clean":"2024-04-30T08:33:31.881554+0000","last_became_active":"2024-04-30T08:31:51.686096+0000","last_became_peered":"2024-04-30T08:31:51.686096+0000","last_unstale":"2024-04-30T08:33:31.881554+0000","last_undegraded":"2024-04-30T08:33:31.881554+0000","last_fullsized":"2024-04-30T08:33:31.881554+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:47:05.961930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"85'8","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.698886+0000","last_change":"2024-04-30T08:31:52.698512+0000","last_active":"2024-04-30T08:33:59.698886+0000","last_peered":"2024-04-30T08:33:59.698886+0000","last_clean":"2024-04-30T08:33:59.698886+0000","last_became_active":"2024-04-30T08:31:52.698168+0000","last_became_peered":"2024-04-30T08:31:52.698168+0000","last_unstale":"2024-04-30T08:33:59.698886+0000","last_undegraded":"2024-04-30T08:33:59.698886+0000","last_fullsized":"2024-04-30T08:33:59.698886+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:22:45.094365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886339+0000","last_change":"2024-04-30T08:31:57.761430+0000","last_active":"2024-04-30T08:33:31.886339+0000","last_peered":"2024-04-30T08:33:31.886339+0000","last_clean":"2024-04-30T08:33:31.886339+0000","last_became_active":"2024-04-30T08:31:57.761062+0000","last_became_peered":"2024-04-30T08:31:57.761062+0000","last_unstale":"2024-04-30T08:33:31.886339+0000","last_undegraded":"2024-04-30T08:33:31.886339+0000","last_fullsized":"2024-04-30T08:33:31.886339+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:32:05.403786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"81'3","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.879731+0000","last_change":"2024-04-30T08:31:51.686694+0000","last_active":"2024-04-30T08:33:31.879731+0000","last_peered":"2024-04-30T08:33:31.879731+0000","last_clean":"2024-04-30T08:33:31.879731+0000","last_became_active":"2024-04-30T08:31:51.685773+0000","last_became_peered":"2024-04-30T08:31:51.685773+0000","last_unstale":"2024-04-30T08:33:31.879731+0000","last_undegraded":"2024-04-30T08:33:31.879731+0000","last_fullsized":"2024-04-30T08:33:31.879731+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:44:25.327977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"85'8","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.734178+0000","last_change":"2024-04-30T08:33:31.888698+0000","last_active":"2024-04-30T08:33:59.734178+0000","last_peered":"2024-04-30T08:33:59.734178+0000","last_clean":"2024-04-30T08:33:59.734178+0000","last_became_active":"2024-04-30T08:33:31.887935+0000","last_became_peered":"2024-04-30T08:33:31.887935+0000","last_unstale":"2024-04-30T08:33:59.734178+0000","last_undegraded":"2024-04-30T08:33:59.734178+0000","last_fullsized":"2024-04-30T08:33:59.734178+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:15:40.989004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.799780+0000","last_change":"2024-04-30T08:31:57.757224+0000","last_active":"2024-04-30T08:33:32.799780+0000","last_peered":"2024-04-30T08:33:32.799780+0000","last_clean":"2024-04-30T08:33:32.799780+0000","last_became_active":"2024-04-30T08:31:57.756658+0000","last_became_peered":"2024-04-30T08:31:57.756658+0000","last_unstale":"2024-04-30T08:33:32.799780+0000","last_undegraded":"2024-04-30T08:33:32.799780+0000","last_fullsized":"2024-04-30T08:33:32.799780+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:01:21.309322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"81'3","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.272944+0000","last_change":"2024-04-30T08:31:51.678944+0000","last_active":"2024-04-30T08:33:32.272944+0000","last_peered":"2024-04-30T08:33:32.272944+0000","last_clean":"2024-04-30T08:33:32.272944+0000","last_became_active":"2024-04-30T08:31:51.678493+0000","last_became_peered":"2024-04-30T08:31:51.678493+0000","last_unstale":"2024-04-30T08:33:32.272944+0000","last_undegraded":"2024-04-30T08:33:32.272944+0000","last_fullsized":"2024-04-30T08:33:32.272944+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:48:11.111466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"85'11","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.722548+0000","last_change":"2024-04-30T08:31:52.707627+0000","last_active":"2024-04-30T08:33:59.722548+0000","last_peered":"2024-04-30T08:33:59.722548+0000","last_clean":"2024-04-30T08:33:59.722548+0000","last_became_active":"2024-04-30T08:31:52.706441+0000","last_became_peered":"2024-04-30T08:31:52.706441+0000","last_unstale":"2024-04-30T08:33:59.722548+0000","last_undegraded":"2024-04-30T08:33:59.722548+0000","last_fullsized":"2024-04-30T08:33:59.722548+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:15:45.768927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.275059+0000","last_change":"2024-04-30T08:31:57.740324+0000","last_active":"2024-04-30T08:33:32.275059+0000","last_peered":"2024-04-30T08:33:32.275059+0000","last_clean":"2024-04-30T08:33:32.275059+0000","last_became_active":"2024-04-30T08:31:57.739464+0000","last_became_peered":"2024-04-30T08:31:57.739464+0000","last_unstale":"2024-04-30T08:33:32.275059+0000","last_undegraded":"2024-04-30T08:33:32.275059+0000","last_fullsized":"2024-04-30T08:33:32.275059+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:11:37.395545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"81'4","reported_seq":214,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:58.704283+0000","last_change":"2024-04-30T08:31:51.695283+0000","last_active":"2024-04-30T08:33:58.704283+0000","last_peered":"2024-04-30T08:33:58.704283+0000","last_clean":"2024-04-30T08:33:58.704283+0000","last_became_active":"2024-04-30T08:31:51.694848+0000","last_became_peered":"2024-04-30T08:31:51.694848+0000","last_unstale":"2024-04-30T08:33:58.704283+0000","last_undegraded":"2024-04-30T08:33:58.704283+0000","last_fullsized":"2024-04-30T08:33:58.704283+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:41:38.638034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":177,"num_read_kb":177,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"85'10","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.729450+0000","last_change":"2024-04-30T08:31:52.693594+0000","last_active":"2024-04-30T08:33:59.729450+0000","last_peered":"2024-04-30T08:33:59.729450+0000","last_clean":"2024-04-30T08:33:59.729450+0000","last_became_active":"2024-04-30T08:31:52.693199+0000","last_became_peered":"2024-04-30T08:31:52.693199+0000","last_unstale":"2024-04-30T08:33:59.729450+0000","last_undegraded":"2024-04-30T08:33:59.729450+0000","last_fullsized":"2024-04-30T08:33:59.729450+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:57:49.908617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.883592+0000","last_change":"2024-04-30T08:31:57.749727+0000","last_active":"2024-04-30T08:33:31.883592+0000","last_peered":"2024-04-30T08:33:31.883592+0000","last_clean":"2024-04-30T08:33:31.883592+0000","last_became_active":"2024-04-30T08:31:57.749267+0000","last_became_peered":"2024-04-30T08:31:57.749267+0000","last_unstale":"2024-04-30T08:33:31.883592+0000","last_undegraded":"2024-04-30T08:33:31.883592+0000","last_fullsized":"2024-04-30T08:33:31.883592+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:26:12.683008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"81'4","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.218604+0000","last_change":"2024-04-30T08:31:51.677398+0000","last_active":"2024-04-30T08:33:32.218604+0000","last_peered":"2024-04-30T08:33:32.218604+0000","last_clean":"2024-04-30T08:33:32.218604+0000","last_became_active":"2024-04-30T08:31:51.677031+0000","last_became_peered":"2024-04-30T08:31:51.677031+0000","last_unstale":"2024-04-30T08:33:32.218604+0000","last_undegraded":"2024-04-30T08:33:32.218604+0000","last_fullsized":"2024-04-30T08:33:32.218604+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:24:25.154229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"85'9","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.741323+0000","last_change":"2024-04-30T08:32:31.561554+0000","last_active":"2024-04-30T08:33:59.741323+0000","last_peered":"2024-04-30T08:33:59.741323+0000","last_clean":"2024-04-30T08:33:59.741323+0000","last_became_active":"2024-04-30T08:32:31.560253+0000","last_became_peered":"2024-04-30T08:32:31.560253+0000","last_unstale":"2024-04-30T08:33:59.741323+0000","last_undegraded":"2024-04-30T08:33:59.741323+0000","last_fullsized":"2024-04-30T08:33:59.741323+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:59:08.602282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":29,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.169998+0000","last_change":"2024-04-30T08:32:31.586746+0000","last_active":"2024-04-30T08:33:32.169998+0000","last_peered":"2024-04-30T08:33:32.169998+0000","last_clean":"2024-04-30T08:33:32.169998+0000","last_became_active":"2024-04-30T08:32:31.586356+0000","last_became_peered":"2024-04-30T08:32:31.586356+0000","last_unstale":"2024-04-30T08:33:32.169998+0000","last_undegraded":"2024-04-30T08:33:32.169998+0000","last_fullsized":"2024-04-30T08:33:32.169998+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:22:39.371508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"81'5","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273422+0000","last_change":"2024-04-30T08:31:51.694424+0000","last_active":"2024-04-30T08:33:32.273422+0000","last_peered":"2024-04-30T08:33:32.273422+0000","last_clean":"2024-04-30T08:33:32.273422+0000","last_became_active":"2024-04-30T08:31:51.694081+0000","last_became_peered":"2024-04-30T08:31:51.694081+0000","last_unstale":"2024-04-30T08:33:32.273422+0000","last_undegraded":"2024-04-30T08:33:32.273422+0000","last_fullsized":"2024-04-30T08:33:32.273422+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:27:33.328477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"85'11","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.691897+0000","last_change":"2024-04-30T08:31:52.684739+0000","last_active":"2024-04-30T08:33:59.691897+0000","last_peered":"2024-04-30T08:33:59.691897+0000","last_clean":"2024-04-30T08:33:59.691897+0000","last_became_active":"2024-04-30T08:31:52.684316+0000","last_became_peered":"2024-04-30T08:31:52.684316+0000","last_unstale":"2024-04-30T08:33:59.691897+0000","last_undegraded":"2024-04-30T08:33:59.691897+0000","last_fullsized":"2024-04-30T08:33:59.691897+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:12:05.437493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881339+0000","last_change":"2024-04-30T08:32:31.557578+0000","last_active":"2024-04-30T08:33:31.881339+0000","last_peered":"2024-04-30T08:33:31.881339+0000","last_clean":"2024-04-30T08:33:31.881339+0000","last_became_active":"2024-04-30T08:32:31.557115+0000","last_became_peered":"2024-04-30T08:32:31.557115+0000","last_unstale":"2024-04-30T08:33:31.881339+0000","last_undegraded":"2024-04-30T08:33:31.881339+0000","last_fullsized":"2024-04-30T08:33:31.881339+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:41:33.225879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"83'33","reported_seq":248,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:58.704314+0000","last_change":"2024-04-30T08:31:51.676223+0000","last_active":"2024-04-30T08:33:58.704314+0000","last_peered":"2024-04-30T08:33:58.704314+0000","last_clean":"2024-04-30T08:33:58.704314+0000","last_became_active":"2024-04-30T08:31:51.675830+0000","last_became_peered":"2024-04-30T08:31:51.675830+0000","last_unstale":"2024-04-30T08:33:58.704314+0000","last_undegraded":"2024-04-30T08:33:58.704314+0000","last_fullsized":"2024-04-30T08:33:58.704314+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:28:49.497719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62072,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":182,"num_read_kb":151,"num_write":35,"num_write_kb":75,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886419+0000","last_change":"2024-04-30T08:31:52.686892+0000","last_active":"2024-04-30T08:33:31.886419+0000","last_peered":"2024-04-30T08:33:31.886419+0000","last_clean":"2024-04-30T08:33:31.886419+0000","last_became_active":"2024-04-30T08:31:52.686533+0000","last_became_peered":"2024-04-30T08:31:52.686533+0000","last_unstale":"2024-04-30T08:33:31.886419+0000","last_undegraded":"2024-04-30T08:33:31.886419+0000","last_fullsized":"2024-04-30T08:33:31.886419+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:50:18.836161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273957+0000","last_change":"2024-04-30T08:31:57.752865+0000","last_active":"2024-04-30T08:33:32.273957+0000","last_peered":"2024-04-30T08:33:32.273957+0000","last_clean":"2024-04-30T08:33:32.273957+0000","last_became_active":"2024-04-30T08:31:57.752385+0000","last_became_peered":"2024-04-30T08:31:57.752385+0000","last_unstale":"2024-04-30T08:33:32.273957+0000","last_undegraded":"2024-04-30T08:33:32.273957+0000","last_fullsized":"2024-04-30T08:33:32.273957+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:15:15.315580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"81'9","reported_seq":31,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886198+0000","last_change":"2024-04-30T08:32:31.585282+0000","last_active":"2024-04-30T08:33:31.886198+0000","last_peered":"2024-04-30T08:33:31.886198+0000","last_clean":"2024-04-30T08:33:31.886198+0000","last_became_active":"2024-04-30T08:32:31.571676+0000","last_became_peered":"2024-04-30T08:32:31.571676+0000","last_unstale":"2024-04-30T08:33:31.886198+0000","last_undegraded":"2024-04-30T08:33:31.886198+0000","last_fullsized":"2024-04-30T08:33:31.886198+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:18:29.910201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"85'9","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.723477+0000","last_change":"2024-04-30T08:31:52.706259+0000","last_active":"2024-04-30T08:33:59.723477+0000","last_peered":"2024-04-30T08:33:59.723477+0000","last_clean":"2024-04-30T08:33:59.723477+0000","last_became_active":"2024-04-30T08:31:52.705620+0000","last_became_peered":"2024-04-30T08:31:52.705620+0000","last_unstale":"2024-04-30T08:33:59.723477+0000","last_undegraded":"2024-04-30T08:33:59.723477+0000","last_fullsized":"2024-04-30T08:33:59.723477+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:22:29.816950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":29,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886484+0000","last_change":"2024-04-30T08:32:31.562571+0000","last_active":"2024-04-30T08:33:31.886484+0000","last_peered":"2024-04-30T08:33:31.886484+0000","last_clean":"2024-04-30T08:33:31.886484+0000","last_became_active":"2024-04-30T08:32:31.562012+0000","last_became_peered":"2024-04-30T08:32:31.562012+0000","last_unstale":"2024-04-30T08:33:31.886484+0000","last_undegraded":"2024-04-30T08:33:31.886484+0000","last_fullsized":"2024-04-30T08:33:31.886484+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:12:22.928286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"81'5","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273089+0000","last_change":"2024-04-30T08:31:51.701122+0000","last_active":"2024-04-30T08:33:32.273089+0000","last_peered":"2024-04-30T08:33:32.273089+0000","last_clean":"2024-04-30T08:33:32.273089+0000","last_became_active":"2024-04-30T08:31:51.700815+0000","last_became_peered":"2024-04-30T08:31:51.700815+0000","last_unstale":"2024-04-30T08:33:32.273089+0000","last_undegraded":"2024-04-30T08:33:32.273089+0000","last_fullsized":"2024-04-30T08:33:32.273089+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:33:12.714979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"85'15","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.721444+0000","last_change":"2024-04-30T08:31:52.716043+0000","last_active":"2024-04-30T08:33:59.721444+0000","last_peered":"2024-04-30T08:33:59.721444+0000","last_clean":"2024-04-30T08:33:59.721444+0000","last_became_active":"2024-04-30T08:31:52.715284+0000","last_became_peered":"2024-04-30T08:31:52.715284+0000","last_unstale":"2024-04-30T08:33:59.721444+0000","last_undegraded":"2024-04-30T08:33:59.721444+0000","last_fullsized":"2024-04-30T08:33:59.721444+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:00:01.863606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.885093+0000","last_change":"2024-04-30T08:31:57.765008+0000","last_active":"2024-04-30T08:33:31.885093+0000","last_peered":"2024-04-30T08:33:31.885093+0000","last_clean":"2024-04-30T08:33:31.885093+0000","last_became_active":"2024-04-30T08:31:57.762958+0000","last_became_peered":"2024-04-30T08:31:57.762958+0000","last_unstale":"2024-04-30T08:33:31.885093+0000","last_undegraded":"2024-04-30T08:33:31.885093+0000","last_fullsized":"2024-04-30T08:33:31.885093+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:48:47.116004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"85'7","reported_seq":136,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:34:01.779292+0000","last_change":"2024-04-30T08:31:51.676681+0000","last_active":"2024-04-30T08:34:01.779292+0000","last_peered":"2024-04-30T08:34:01.779292+0000","last_clean":"2024-04-30T08:34:01.779292+0000","last_became_active":"2024-04-30T08:31:51.676311+0000","last_became_peered":"2024-04-30T08:31:51.676311+0000","last_unstale":"2024-04-30T08:34:01.779292+0000","last_undegraded":"2024-04-30T08:34:01.779292+0000","last_fullsized":"2024-04-30T08:34:01.779292+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:24:49.757427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":97,"num_read_kb":4,"num_write":7,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"85'6","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.687241+0000","last_change":"2024-04-30T08:31:52.689418+0000","last_active":"2024-04-30T08:33:59.687241+0000","last_peered":"2024-04-30T08:33:59.687241+0000","last_clean":"2024-04-30T08:33:59.687241+0000","last_became_active":"2024-04-30T08:31:52.688840+0000","last_became_peered":"2024-04-30T08:31:52.688840+0000","last_unstale":"2024-04-30T08:33:59.687241+0000","last_undegraded":"2024-04-30T08:33:59.687241+0000","last_fullsized":"2024-04-30T08:33:59.687241+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:40:05.781137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881164+0000","last_change":"2024-04-30T08:31:57.765875+0000","last_active":"2024-04-30T08:33:31.881164+0000","last_peered":"2024-04-30T08:33:31.881164+0000","last_clean":"2024-04-30T08:33:31.881164+0000","last_became_active":"2024-04-30T08:31:57.765432+0000","last_became_peered":"2024-04-30T08:31:57.765432+0000","last_unstale":"2024-04-30T08:33:31.881164+0000","last_undegraded":"2024-04-30T08:33:31.881164+0000","last_fullsized":"2024-04-30T08:33:31.881164+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:24:44.329532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"81'17","reported_seq":50,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.887011+0000","last_change":"2024-04-30T08:31:51.695007+0000","last_active":"2024-04-30T08:33:31.887011+0000","last_peered":"2024-04-30T08:33:31.887011+0000","last_clean":"2024-04-30T08:33:31.887011+0000","last_became_active":"2024-04-30T08:31:51.694533+0000","last_became_peered":"2024-04-30T08:31:51.694533+0000","last_unstale":"2024-04-30T08:33:31.887011+0000","last_undegraded":"2024-04-30T08:33:31.887011+0000","last_fullsized":"2024-04-30T08:33:31.887011+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:47:15.104172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"85'6","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.685513+0000","last_change":"2024-04-30T08:31:52.692274+0000","last_active":"2024-04-30T08:33:59.685513+0000","last_peered":"2024-04-30T08:33:59.685513+0000","last_clean":"2024-04-30T08:33:59.685513+0000","last_became_active":"2024-04-30T08:31:52.691809+0000","last_became_peered":"2024-04-30T08:31:52.691809+0000","last_unstale":"2024-04-30T08:33:59.685513+0000","last_undegraded":"2024-04-30T08:33:59.685513+0000","last_fullsized":"2024-04-30T08:33:59.685513+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:50:00.201782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273070+0000","last_change":"2024-04-30T08:31:57.732383+0000","last_active":"2024-04-30T08:33:32.273070+0000","last_peered":"2024-04-30T08:33:32.273070+0000","last_clean":"2024-04-30T08:33:32.273070+0000","last_became_active":"2024-04-30T08:31:57.731893+0000","last_became_peered":"2024-04-30T08:31:57.731893+0000","last_unstale":"2024-04-30T08:33:32.273070+0000","last_undegraded":"2024-04-30T08:33:32.273070+0000","last_fullsized":"2024-04-30T08:33:32.273070+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:08:55.243526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"81'2","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886769+0000","last_change":"2024-04-30T08:31:51.671991+0000","last_active":"2024-04-30T08:33:31.886769+0000","last_peered":"2024-04-30T08:33:31.886769+0000","last_clean":"2024-04-30T08:33:31.886769+0000","last_became_active":"2024-04-30T08:31:51.671508+0000","last_became_peered":"2024-04-30T08:31:51.671508+0000","last_unstale":"2024-04-30T08:33:31.886769+0000","last_undegraded":"2024-04-30T08:33:31.886769+0000","last_fullsized":"2024-04-30T08:33:31.886769+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:54:30.106748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"85'5","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.689495+0000","last_change":"2024-04-30T08:33:32.274901+0000","last_active":"2024-04-30T08:33:59.689495+0000","last_peered":"2024-04-30T08:33:59.689495+0000","last_clean":"2024-04-30T08:33:59.689495+0000","last_became_active":"2024-04-30T08:33:32.274401+0000","last_became_peered":"2024-04-30T08:33:32.274401+0000","last_unstale":"2024-04-30T08:33:59.689495+0000","last_undegraded":"2024-04-30T08:33:59.689495+0000","last_fullsized":"2024-04-30T08:33:59.689495+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:23:08.703092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886029+0000","last_change":"2024-04-30T08:31:58.061077+0000","last_active":"2024-04-30T08:33:31.886029+0000","last_peered":"2024-04-30T08:33:31.886029+0000","last_clean":"2024-04-30T08:33:31.886029+0000","last_became_active":"2024-04-30T08:31:58.060486+0000","last_became_peered":"2024-04-30T08:31:58.060486+0000","last_unstale":"2024-04-30T08:33:31.886029+0000","last_undegraded":"2024-04-30T08:33:31.886029+0000","last_fullsized":"2024-04-30T08:33:31.886029+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:31:50.936560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"81'1","reported_seq":34,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.885199+0000","last_change":"2024-04-30T08:31:51.702869+0000","last_active":"2024-04-30T08:33:31.885199+0000","last_peered":"2024-04-30T08:33:31.885199+0000","last_clean":"2024-04-30T08:33:31.885199+0000","last_became_active":"2024-04-30T08:31:51.701150+0000","last_became_peered":"2024-04-30T08:31:51.701150+0000","last_unstale":"2024-04-30T08:33:31.885199+0000","last_undegraded":"2024-04-30T08:33:31.885199+0000","last_fullsized":"2024-04-30T08:33:31.885199+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:47:11.010310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881903+0000","last_change":"2024-04-30T08:31:52.709159+0000","last_active":"2024-04-30T08:33:31.881903+0000","last_peered":"2024-04-30T08:33:31.881903+0000","last_clean":"2024-04-30T08:33:31.881903+0000","last_became_active":"2024-04-30T08:31:52.708799+0000","last_became_peered":"2024-04-30T08:31:52.708799+0000","last_unstale":"2024-04-30T08:33:31.881903+0000","last_undegraded":"2024-04-30T08:33:31.881903+0000","last_fullsized":"2024-04-30T08:33:31.881903+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:51:10.588599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881128+0000","last_change":"2024-04-30T08:31:57.751652+0000","last_active":"2024-04-30T08:33:31.881128+0000","last_peered":"2024-04-30T08:33:31.881128+0000","last_clean":"2024-04-30T08:33:31.881128+0000","last_became_active":"2024-04-30T08:31:57.751176+0000","last_became_peered":"2024-04-30T08:31:57.751176+0000","last_unstale":"2024-04-30T08:33:31.881128+0000","last_undegraded":"2024-04-30T08:33:31.881128+0000","last_fullsized":"2024-04-30T08:33:31.881128+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:13:24.528842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"81'5","reported_seq":133,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:34:01.775178+0000","last_change":"2024-04-30T08:31:51.700816+0000","last_active":"2024-04-30T08:34:01.775178+0000","last_peered":"2024-04-30T08:34:01.775178+0000","last_clean":"2024-04-30T08:34:01.775178+0000","last_became_active":"2024-04-30T08:31:51.700470+0000","last_became_peered":"2024-04-30T08:31:51.700470+0000","last_unstale":"2024-04-30T08:34:01.775178+0000","last_undegraded":"2024-04-30T08:34:01.775178+0000","last_fullsized":"2024-04-30T08:34:01.775178+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:49:11.793412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":94,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881562+0000","last_change":"2024-04-30T08:31:52.715236+0000","last_active":"2024-04-30T08:33:31.881562+0000","last_peered":"2024-04-30T08:33:31.881562+0000","last_clean":"2024-04-30T08:33:31.881562+0000","last_became_active":"2024-04-30T08:31:52.714801+0000","last_became_peered":"2024-04-30T08:31:52.714801+0000","last_unstale":"2024-04-30T08:33:31.881562+0000","last_undegraded":"2024-04-30T08:33:31.881562+0000","last_fullsized":"2024-04-30T08:33:31.881562+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:43:09.650002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881625+0000","last_change":"2024-04-30T08:31:58.060927+0000","last_active":"2024-04-30T08:33:31.881625+0000","last_peered":"2024-04-30T08:33:31.881625+0000","last_clean":"2024-04-30T08:33:31.881625+0000","last_became_active":"2024-04-30T08:31:58.060182+0000","last_became_peered":"2024-04-30T08:31:58.060182+0000","last_unstale":"2024-04-30T08:33:31.881625+0000","last_undegraded":"2024-04-30T08:33:31.881625+0000","last_fullsized":"2024-04-30T08:33:31.881625+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:21:00.228064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"85'8","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.679643+0000","last_change":"2024-04-30T08:32:31.560485+0000","last_active":"2024-04-30T08:33:59.679643+0000","last_peered":"2024-04-30T08:33:59.679643+0000","last_clean":"2024-04-30T08:33:59.679643+0000","last_became_active":"2024-04-30T08:32:31.560066+0000","last_became_peered":"2024-04-30T08:32:31.560066+0000","last_unstale":"2024-04-30T08:33:59.679643+0000","last_undegraded":"2024-04-30T08:33:59.679643+0000","last_fullsized":"2024-04-30T08:33:59.679643+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:13:40.769761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'6","reported_seq":130,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.772538+0000","last_change":"2024-04-30T08:31:51.702808+0000","last_active":"2024-04-30T08:33:59.772538+0000","last_peered":"2024-04-30T08:33:59.772538+0000","last_clean":"2024-04-30T08:33:59.772538+0000","last_became_active":"2024-04-30T08:31:51.702326+0000","last_became_peered":"2024-04-30T08:31:51.702326+0000","last_unstale":"2024-04-30T08:33:59.772538+0000","last_undegraded":"2024-04-30T08:33:59.772538+0000","last_fullsized":"2024-04-30T08:33:59.772538+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:11:35.552670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":91,"num_read_kb":91,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"81'1","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.218503+0000","last_change":"2024-04-30T08:31:57.760359+0000","last_active":"2024-04-30T08:33:32.218503+0000","last_peered":"2024-04-30T08:33:32.218503+0000","last_clean":"2024-04-30T08:33:32.218503+0000","last_became_active":"2024-04-30T08:31:57.759821+0000","last_became_peered":"2024-04-30T08:31:57.759821+0000","last_unstale":"2024-04-30T08:33:32.218503+0000","last_undegraded":"2024-04-30T08:33:32.218503+0000","last_fullsized":"2024-04-30T08:33:32.218503+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:50:17.853791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"85'8","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.719174+0000","last_change":"2024-04-30T08:31:52.689281+0000","last_active":"2024-04-30T08:33:59.719174+0000","last_peered":"2024-04-30T08:33:59.719174+0000","last_clean":"2024-04-30T08:33:59.719174+0000","last_became_active":"2024-04-30T08:31:52.688834+0000","last_became_peered":"2024-04-30T08:31:52.688834+0000","last_unstale":"2024-04-30T08:33:59.719174+0000","last_undegraded":"2024-04-30T08:33:59.719174+0000","last_fullsized":"2024-04-30T08:33:59.719174+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:42:12.836649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'6","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.169972+0000","last_change":"2024-04-30T08:31:51.683712+0000","last_active":"2024-04-30T08:33:32.169972+0000","last_peered":"2024-04-30T08:33:32.169972+0000","last_clean":"2024-04-30T08:33:32.169972+0000","last_became_active":"2024-04-30T08:31:51.683298+0000","last_became_peered":"2024-04-30T08:31:51.683298+0000","last_unstale":"2024-04-30T08:33:32.169972+0000","last_undegraded":"2024-04-30T08:33:32.169972+0000","last_fullsized":"2024-04-30T08:33:32.169972+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:43:08.832104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":108,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.217728+0000","last_change":"2024-04-30T08:30:26.503221+0000","last_active":"2024-04-30T08:33:32.217728+0000","last_peered":"2024-04-30T08:33:32.217728+0000","last_clean":"2024-04-30T08:33:32.217728+0000","last_became_active":"2024-04-30T08:30:26.498399+0000","last_became_peered":"2024-04-30T08:30:26.498399+0000","last_unstale":"2024-04-30T08:33:32.217728+0000","last_undegraded":"2024-04-30T08:33:32.217728+0000","last_fullsized":"2024-04-30T08:33:32.217728+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:26:20.170075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:26:20.170075+0000","last_clean_scrub_stamp":"2024-04-30T08:26:20.170075+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:34:44.067560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273473+0000","last_change":"2024-04-30T08:31:58.061283+0000","last_active":"2024-04-30T08:33:32.273473+0000","last_peered":"2024-04-30T08:33:32.273473+0000","last_clean":"2024-04-30T08:33:32.273473+0000","last_became_active":"2024-04-30T08:31:58.060650+0000","last_became_peered":"2024-04-30T08:31:58.060650+0000","last_unstale":"2024-04-30T08:33:32.273473+0000","last_undegraded":"2024-04-30T08:33:32.273473+0000","last_fullsized":"2024-04-30T08:33:32.273473+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:13:01.811462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"85'5","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.728420+0000","last_change":"2024-04-30T08:31:52.699639+0000","last_active":"2024-04-30T08:33:59.728420+0000","last_peered":"2024-04-30T08:33:59.728420+0000","last_clean":"2024-04-30T08:33:59.728420+0000","last_became_active":"2024-04-30T08:31:52.699164+0000","last_became_peered":"2024-04-30T08:31:52.699164+0000","last_unstale":"2024-04-30T08:33:59.728420+0000","last_undegraded":"2024-04-30T08:33:59.728420+0000","last_fullsized":"2024-04-30T08:33:59.728420+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:22:18.873504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"81'4","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.217821+0000","last_change":"2024-04-30T08:31:51.679153+0000","last_active":"2024-04-30T08:33:32.217821+0000","last_peered":"2024-04-30T08:33:32.217821+0000","last_clean":"2024-04-30T08:33:32.217821+0000","last_became_active":"2024-04-30T08:31:51.678348+0000","last_became_peered":"2024-04-30T08:31:51.678348+0000","last_unstale":"2024-04-30T08:33:32.217821+0000","last_undegraded":"2024-04-30T08:33:32.217821+0000","last_fullsized":"2024-04-30T08:33:32.217821+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:02:56.240218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886545+0000","last_change":"2024-04-30T08:31:57.755423+0000","last_active":"2024-04-30T08:33:31.886545+0000","last_peered":"2024-04-30T08:33:31.886545+0000","last_clean":"2024-04-30T08:33:31.886545+0000","last_became_active":"2024-04-30T08:31:57.754835+0000","last_became_peered":"2024-04-30T08:31:57.754835+0000","last_unstale":"2024-04-30T08:33:31.886545+0000","last_undegraded":"2024-04-30T08:33:31.886545+0000","last_fullsized":"2024-04-30T08:33:31.886545+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:19:25.168920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"81'4","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886919+0000","last_change":"2024-04-30T08:31:51.693839+0000","last_active":"2024-04-30T08:33:31.886919+0000","last_peered":"2024-04-30T08:33:31.886919+0000","last_clean":"2024-04-30T08:33:31.886919+0000","last_became_active":"2024-04-30T08:31:51.693349+0000","last_became_peered":"2024-04-30T08:31:51.693349+0000","last_unstale":"2024-04-30T08:33:31.886919+0000","last_undegraded":"2024-04-30T08:33:31.886919+0000","last_fullsized":"2024-04-30T08:33:31.886919+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:41:51.330901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"85'9","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.733847+0000","last_change":"2024-04-30T08:31:52.712360+0000","last_active":"2024-04-30T08:33:59.733847+0000","last_peered":"2024-04-30T08:33:59.733847+0000","last_clean":"2024-04-30T08:33:59.733847+0000","last_became_active":"2024-04-30T08:31:52.711652+0000","last_became_peered":"2024-04-30T08:31:52.711652+0000","last_unstale":"2024-04-30T08:33:59.733847+0000","last_undegraded":"2024-04-30T08:33:59.733847+0000","last_fullsized":"2024-04-30T08:33:59.733847+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:21:10.285386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.276738+0000","last_change":"2024-04-30T08:31:58.061654+0000","last_active":"2024-04-30T08:33:32.276738+0000","last_peered":"2024-04-30T08:33:32.276738+0000","last_clean":"2024-04-30T08:33:32.276738+0000","last_became_active":"2024-04-30T08:31:58.060181+0000","last_became_peered":"2024-04-30T08:31:58.060181+0000","last_unstale":"2024-04-30T08:33:32.276738+0000","last_undegraded":"2024-04-30T08:33:32.276738+0000","last_fullsized":"2024-04-30T08:33:32.276738+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:46:03.230626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'13","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:34:00.257425+0000","last_change":"2024-04-30T08:31:51.706940+0000","last_active":"2024-04-30T08:34:00.257425+0000","last_peered":"2024-04-30T08:34:00.257425+0000","last_clean":"2024-04-30T08:34:00.257425+0000","last_became_active":"2024-04-30T08:31:51.706481+0000","last_became_peered":"2024-04-30T08:31:51.706481+0000","last_unstale":"2024-04-30T08:34:00.257425+0000","last_undegraded":"2024-04-30T08:34:00.257425+0000","last_fullsized":"2024-04-30T08:34:00.257425+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:53:31.955595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"85'5","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.714214+0000","last_change":"2024-04-30T08:31:52.696625+0000","last_active":"2024-04-30T08:33:59.714214+0000","last_peered":"2024-04-30T08:33:59.714214+0000","last_clean":"2024-04-30T08:33:59.714214+0000","last_became_active":"2024-04-30T08:31:52.696230+0000","last_became_peered":"2024-04-30T08:31:52.696230+0000","last_unstale":"2024-04-30T08:33:59.714214+0000","last_undegraded":"2024-04-30T08:33:59.714214+0000","last_fullsized":"2024-04-30T08:33:59.714214+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:11:46.404251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.879348+0000","last_change":"2024-04-30T08:31:57.738329+0000","last_active":"2024-04-30T08:33:31.879348+0000","last_peered":"2024-04-30T08:33:31.879348+0000","last_clean":"2024-04-30T08:33:31.879348+0000","last_became_active":"2024-04-30T08:31:57.737897+0000","last_became_peered":"2024-04-30T08:31:57.737897+0000","last_unstale":"2024-04-30T08:33:31.879348+0000","last_undegraded":"2024-04-30T08:33:31.879348+0000","last_fullsized":"2024-04-30T08:33:31.879348+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:02:43.074578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"81'7","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.170374+0000","last_change":"2024-04-30T08:31:51.684066+0000","last_active":"2024-04-30T08:33:32.170374+0000","last_peered":"2024-04-30T08:33:32.170374+0000","last_clean":"2024-04-30T08:33:32.170374+0000","last_became_active":"2024-04-30T08:31:51.683699+0000","last_became_peered":"2024-04-30T08:31:51.683699+0000","last_unstale":"2024-04-30T08:33:32.170374+0000","last_undegraded":"2024-04-30T08:33:32.170374+0000","last_fullsized":"2024-04-30T08:33:32.170374+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:36:20.900733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"85'7","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.739355+0000","last_change":"2024-04-30T08:31:52.715219+0000","last_active":"2024-04-30T08:33:59.739355+0000","last_peered":"2024-04-30T08:33:59.739355+0000","last_clean":"2024-04-30T08:33:59.739355+0000","last_became_active":"2024-04-30T08:31:52.714774+0000","last_became_peered":"2024-04-30T08:31:52.714774+0000","last_unstale":"2024-04-30T08:33:59.739355+0000","last_undegraded":"2024-04-30T08:33:59.739355+0000","last_fullsized":"2024-04-30T08:33:59.739355+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:42:41.082610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.170102+0000","last_change":"2024-04-30T08:31:58.060845+0000","last_active":"2024-04-30T08:33:32.170102+0000","last_peered":"2024-04-30T08:33:32.170102+0000","last_clean":"2024-04-30T08:33:32.170102+0000","last_became_active":"2024-04-30T08:31:58.060365+0000","last_became_peered":"2024-04-30T08:31:58.060365+0000","last_unstale":"2024-04-30T08:33:32.170102+0000","last_undegraded":"2024-04-30T08:33:32.170102+0000","last_fullsized":"2024-04-30T08:33:32.170102+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:21:39.583243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"81'2","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.272616+0000","last_change":"2024-04-30T08:31:51.694833+0000","last_active":"2024-04-30T08:33:32.272616+0000","last_peered":"2024-04-30T08:33:32.272616+0000","last_clean":"2024-04-30T08:33:32.272616+0000","last_became_active":"2024-04-30T08:31:51.694401+0000","last_became_peered":"2024-04-30T08:31:51.694401+0000","last_unstale":"2024-04-30T08:33:32.272616+0000","last_undegraded":"2024-04-30T08:33:32.272616+0000","last_fullsized":"2024-04-30T08:33:32.272616+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:38:52.605625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.885889+0000","last_change":"2024-04-30T08:31:52.686444+0000","last_active":"2024-04-30T08:33:31.885889+0000","last_peered":"2024-04-30T08:33:31.885889+0000","last_clean":"2024-04-30T08:33:31.885889+0000","last_became_active":"2024-04-30T08:31:52.686057+0000","last_became_peered":"2024-04-30T08:31:52.686057+0000","last_unstale":"2024-04-30T08:33:31.885889+0000","last_undegraded":"2024-04-30T08:33:31.885889+0000","last_fullsized":"2024-04-30T08:33:31.885889+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:54:29.824060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.885328+0000","last_change":"2024-04-30T08:31:57.766193+0000","last_active":"2024-04-30T08:33:31.885328+0000","last_peered":"2024-04-30T08:33:31.885328+0000","last_clean":"2024-04-30T08:33:31.885328+0000","last_became_active":"2024-04-30T08:31:57.765758+0000","last_became_peered":"2024-04-30T08:31:57.765758+0000","last_unstale":"2024-04-30T08:33:31.885328+0000","last_undegraded":"2024-04-30T08:33:31.885328+0000","last_fullsized":"2024-04-30T08:33:31.885328+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:59:57.282795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"81'7","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.275484+0000","last_change":"2024-04-30T08:31:51.704565+0000","last_active":"2024-04-30T08:33:32.275484+0000","last_peered":"2024-04-30T08:33:32.275484+0000","last_clean":"2024-04-30T08:33:32.275484+0000","last_became_active":"2024-04-30T08:31:51.703954+0000","last_became_peered":"2024-04-30T08:31:51.703954+0000","last_unstale":"2024-04-30T08:33:32.275484+0000","last_undegraded":"2024-04-30T08:33:32.275484+0000","last_fullsized":"2024-04-30T08:33:32.275484+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:03:00.517867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"85'9","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.717277+0000","last_change":"2024-04-30T08:31:52.706043+0000","last_active":"2024-04-30T08:33:59.717277+0000","last_peered":"2024-04-30T08:33:59.717277+0000","last_clean":"2024-04-30T08:33:59.717277+0000","last_became_active":"2024-04-30T08:31:52.705678+0000","last_became_peered":"2024-04-30T08:31:52.705678+0000","last_unstale":"2024-04-30T08:33:59.717277+0000","last_undegraded":"2024-04-30T08:33:59.717277+0000","last_fullsized":"2024-04-30T08:33:59.717277+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:44:56.787835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"82'1","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.170030+0000","last_change":"2024-04-30T08:31:57.764791+0000","last_active":"2024-04-30T08:33:32.170030+0000","last_peered":"2024-04-30T08:33:32.170030+0000","last_clean":"2024-04-30T08:33:32.170030+0000","last_became_active":"2024-04-30T08:31:57.764368+0000","last_became_peered":"2024-04-30T08:31:57.764368+0000","last_unstale":"2024-04-30T08:33:32.170030+0000","last_undegraded":"2024-04-30T08:33:32.170030+0000","last_fullsized":"2024-04-30T08:33:32.170030+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:19:28.545483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"81'3","reported_seq":21,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886360+0000","last_change":"2024-04-30T08:32:31.577309+0000","last_active":"2024-04-30T08:33:31.886360+0000","last_peered":"2024-04-30T08:33:31.886360+0000","last_clean":"2024-04-30T08:33:31.886360+0000","last_became_active":"2024-04-30T08:32:31.571515+0000","last_became_peered":"2024-04-30T08:32:31.571515+0000","last_unstale":"2024-04-30T08:33:31.886360+0000","last_undegraded":"2024-04-30T08:33:31.886360+0000","last_fullsized":"2024-04-30T08:33:31.886360+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:28:48.408414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"85'8","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.730067+0000","last_change":"2024-04-30T08:31:52.716707+0000","last_active":"2024-04-30T08:33:59.730067+0000","last_peered":"2024-04-30T08:33:59.730067+0000","last_clean":"2024-04-30T08:33:59.730067+0000","last_became_active":"2024-04-30T08:31:52.716293+0000","last_became_peered":"2024-04-30T08:31:52.716293+0000","last_unstale":"2024-04-30T08:33:59.730067+0000","last_undegraded":"2024-04-30T08:33:59.730067+0000","last_fullsized":"2024-04-30T08:33:59.730067+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:29:54.223472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.170072+0000","last_change":"2024-04-30T08:31:58.060640+0000","last_active":"2024-04-30T08:33:32.170072+0000","last_peered":"2024-04-30T08:33:32.170072+0000","last_clean":"2024-04-30T08:33:32.170072+0000","last_became_active":"2024-04-30T08:31:58.060061+0000","last_became_peered":"2024-04-30T08:31:58.060061+0000","last_unstale":"2024-04-30T08:33:32.170072+0000","last_undegraded":"2024-04-30T08:33:32.170072+0000","last_fullsized":"2024-04-30T08:33:32.170072+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:46:00.844191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"81'4","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886101+0000","last_change":"2024-04-30T08:31:51.679532+0000","last_active":"2024-04-30T08:33:31.886101+0000","last_peered":"2024-04-30T08:33:31.886101+0000","last_clean":"2024-04-30T08:33:31.886101+0000","last_became_active":"2024-04-30T08:31:51.679141+0000","last_became_peered":"2024-04-30T08:31:51.679141+0000","last_unstale":"2024-04-30T08:33:31.886101+0000","last_undegraded":"2024-04-30T08:33:31.886101+0000","last_fullsized":"2024-04-30T08:33:31.886101+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:19:50.526563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"85'5","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.720554+0000","last_change":"2024-04-30T08:31:52.703613+0000","last_active":"2024-04-30T08:33:59.720554+0000","last_peered":"2024-04-30T08:33:59.720554+0000","last_clean":"2024-04-30T08:33:59.720554+0000","last_became_active":"2024-04-30T08:31:52.703196+0000","last_became_peered":"2024-04-30T08:31:52.703196+0000","last_unstale":"2024-04-30T08:33:59.720554+0000","last_undegraded":"2024-04-30T08:33:59.720554+0000","last_fullsized":"2024-04-30T08:33:59.720554+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:33:11.136313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.272565+0000","last_change":"2024-04-30T08:31:57.755286+0000","last_active":"2024-04-30T08:33:32.272565+0000","last_peered":"2024-04-30T08:33:32.272565+0000","last_clean":"2024-04-30T08:33:32.272565+0000","last_became_active":"2024-04-30T08:31:57.754786+0000","last_became_peered":"2024-04-30T08:31:57.754786+0000","last_unstale":"2024-04-30T08:33:32.272565+0000","last_undegraded":"2024-04-30T08:33:32.272565+0000","last_fullsized":"2024-04-30T08:33:32.272565+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:02:50.931553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"81'5","reported_seq":128,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:58.704918+0000","last_change":"2024-04-30T08:31:51.693317+0000","last_active":"2024-04-30T08:33:58.704918+0000","last_peered":"2024-04-30T08:33:58.704918+0000","last_clean":"2024-04-30T08:33:58.704918+0000","last_became_active":"2024-04-30T08:31:51.692928+0000","last_became_peered":"2024-04-30T08:31:51.692928+0000","last_unstale":"2024-04-30T08:33:58.704918+0000","last_undegraded":"2024-04-30T08:33:58.704918+0000","last_fullsized":"2024-04-30T08:33:58.704918+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:35:09.851735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":91,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"85'7","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.691997+0000","last_change":"2024-04-30T08:31:52.694304+0000","last_active":"2024-04-30T08:33:59.691997+0000","last_peered":"2024-04-30T08:33:59.691997+0000","last_clean":"2024-04-30T08:33:59.691997+0000","last_became_active":"2024-04-30T08:31:52.693317+0000","last_became_peered":"2024-04-30T08:31:52.693317+0000","last_unstale":"2024-04-30T08:33:59.691997+0000","last_undegraded":"2024-04-30T08:33:59.691997+0000","last_fullsized":"2024-04-30T08:33:59.691997+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:31:39.294864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.798945+0000","last_change":"2024-04-30T08:31:57.763598+0000","last_active":"2024-04-30T08:33:32.798945+0000","last_peered":"2024-04-30T08:33:32.798945+0000","last_clean":"2024-04-30T08:33:32.798945+0000","last_became_active":"2024-04-30T08:31:57.762307+0000","last_became_peered":"2024-04-30T08:31:57.762307+0000","last_unstale":"2024-04-30T08:33:32.798945+0000","last_undegraded":"2024-04-30T08:33:32.798945+0000","last_fullsized":"2024-04-30T08:33:32.798945+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:24:13.390595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"81'3","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273690+0000","last_change":"2024-04-30T08:31:51.703476+0000","last_active":"2024-04-30T08:33:32.273690+0000","last_peered":"2024-04-30T08:33:32.273690+0000","last_clean":"2024-04-30T08:33:32.273690+0000","last_became_active":"2024-04-30T08:31:51.703072+0000","last_became_peered":"2024-04-30T08:31:51.703072+0000","last_unstale":"2024-04-30T08:33:32.273690+0000","last_undegraded":"2024-04-30T08:33:32.273690+0000","last_fullsized":"2024-04-30T08:33:32.273690+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:07:21.098223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"85'6","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.698573+0000","last_change":"2024-04-30T08:31:52.699223+0000","last_active":"2024-04-30T08:33:59.698573+0000","last_peered":"2024-04-30T08:33:59.698573+0000","last_clean":"2024-04-30T08:33:59.698573+0000","last_became_active":"2024-04-30T08:31:52.698837+0000","last_became_peered":"2024-04-30T08:31:52.698837+0000","last_unstale":"2024-04-30T08:33:59.698573+0000","last_undegraded":"2024-04-30T08:33:59.698573+0000","last_fullsized":"2024-04-30T08:33:59.698573+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:19:11.826470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":16,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886711+0000","last_change":"2024-04-30T08:32:31.586667+0000","last_active":"2024-04-30T08:33:31.886711+0000","last_peered":"2024-04-30T08:33:31.886711+0000","last_clean":"2024-04-30T08:33:31.886711+0000","last_became_active":"2024-04-30T08:32:31.586144+0000","last_became_peered":"2024-04-30T08:32:31.586144+0000","last_unstale":"2024-04-30T08:33:31.886711+0000","last_undegraded":"2024-04-30T08:33:31.886711+0000","last_fullsized":"2024-04-30T08:33:31.886711+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:50:39.752485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7,1],"acting":[8,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"81'5","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881571+0000","last_change":"2024-04-30T08:31:51.708943+0000","last_active":"2024-04-30T08:33:31.881571+0000","last_peered":"2024-04-30T08:33:31.881571+0000","last_clean":"2024-04-30T08:33:31.881571+0000","last_became_active":"2024-04-30T08:31:51.708376+0000","last_became_peered":"2024-04-30T08:31:51.708376+0000","last_unstale":"2024-04-30T08:33:31.881571+0000","last_undegraded":"2024-04-30T08:33:31.881571+0000","last_fullsized":"2024-04-30T08:33:31.881571+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:14:46.692192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"85'9","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.690147+0000","last_change":"2024-04-30T08:31:52.686634+0000","last_active":"2024-04-30T08:33:59.690147+0000","last_peered":"2024-04-30T08:33:59.690147+0000","last_clean":"2024-04-30T08:33:59.690147+0000","last_became_active":"2024-04-30T08:31:52.686231+0000","last_became_peered":"2024-04-30T08:31:52.686231+0000","last_unstale":"2024-04-30T08:33:59.690147+0000","last_undegraded":"2024-04-30T08:33:59.690147+0000","last_fullsized":"2024-04-30T08:33:59.690147+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:27:31.073041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"85'9","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.689734+0000","last_change":"2024-04-30T08:31:52.688695+0000","last_active":"2024-04-30T08:33:59.689734+0000","last_peered":"2024-04-30T08:33:59.689734+0000","last_clean":"2024-04-30T08:33:59.689734+0000","last_became_active":"2024-04-30T08:31:52.686556+0000","last_became_peered":"2024-04-30T08:31:52.686556+0000","last_unstale":"2024-04-30T08:33:59.689734+0000","last_undegraded":"2024-04-30T08:33:59.689734+0000","last_fullsized":"2024-04-30T08:33:59.689734+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:22:48.976440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"85'12","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.683087+0000","last_change":"2024-04-30T08:31:52.688324+0000","last_active":"2024-04-30T08:33:59.683087+0000","last_peered":"2024-04-30T08:33:59.683087+0000","last_clean":"2024-04-30T08:33:59.683087+0000","last_became_active":"2024-04-30T08:31:52.687966+0000","last_became_peered":"2024-04-30T08:31:52.687966+0000","last_unstale":"2024-04-30T08:33:59.683087+0000","last_undegraded":"2024-04-30T08:33:59.683087+0000","last_fullsized":"2024-04-30T08:33:59.683087+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:35:38.237750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"85'8","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.706258+0000","last_change":"2024-04-30T08:31:52.695638+0000","last_active":"2024-04-30T08:33:59.706258+0000","last_peered":"2024-04-30T08:33:59.706258+0000","last_clean":"2024-04-30T08:33:59.706258+0000","last_became_active":"2024-04-30T08:31:52.695283+0000","last_became_peered":"2024-04-30T08:31:52.695283+0000","last_unstale":"2024-04-30T08:33:59.706258+0000","last_undegraded":"2024-04-30T08:33:59.706258+0000","last_fullsized":"2024-04-30T08:33:59.706258+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:00:14.510381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"85'7","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.739539+0000","last_change":"2024-04-30T08:31:52.719112+0000","last_active":"2024-04-30T08:33:59.739539+0000","last_peered":"2024-04-30T08:33:59.739539+0000","last_clean":"2024-04-30T08:33:59.739539+0000","last_became_active":"2024-04-30T08:31:52.718207+0000","last_became_peered":"2024-04-30T08:31:52.718207+0000","last_unstale":"2024-04-30T08:33:59.739539+0000","last_undegraded":"2024-04-30T08:33:59.739539+0000","last_fullsized":"2024-04-30T08:33:59.739539+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:27:29.263962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"85'4","reported_seq":34,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.733486+0000","last_change":"2024-04-30T08:31:52.712008+0000","last_active":"2024-04-30T08:33:59.733486+0000","last_peered":"2024-04-30T08:33:59.733486+0000","last_clean":"2024-04-30T08:33:59.733486+0000","last_became_active":"2024-04-30T08:31:52.711064+0000","last_became_peered":"2024-04-30T08:31:52.711064+0000","last_unstale":"2024-04-30T08:33:59.733486+0000","last_undegraded":"2024-04-30T08:33:59.733486+0000","last_fullsized":"2024-04-30T08:33:59.733486+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:57:27.400602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"85'6","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.688526+0000","last_change":"2024-04-30T08:31:52.691210+0000","last_active":"2024-04-30T08:33:59.688526+0000","last_peered":"2024-04-30T08:33:59.688526+0000","last_clean":"2024-04-30T08:33:59.688526+0000","last_became_active":"2024-04-30T08:31:52.688852+0000","last_became_peered":"2024-04-30T08:31:52.688852+0000","last_unstale":"2024-04-30T08:33:59.688526+0000","last_undegraded":"2024-04-30T08:33:59.688526+0000","last_fullsized":"2024-04-30T08:33:59.688526+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:04:46.502743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"85'15","reported_seq":45,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.684276+0000","last_change":"2024-04-30T08:31:52.684501+0000","last_active":"2024-04-30T08:33:59.684276+0000","last_peered":"2024-04-30T08:33:59.684276+0000","last_clean":"2024-04-30T08:33:59.684276+0000","last_became_active":"2024-04-30T08:31:52.684122+0000","last_became_peered":"2024-04-30T08:31:52.684122+0000","last_unstale":"2024-04-30T08:33:59.684276+0000","last_undegraded":"2024-04-30T08:33:59.684276+0000","last_fullsized":"2024-04-30T08:33:59.684276+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:55:58.093269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"85'2","reported_seq":33,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.705928+0000","last_change":"2024-04-30T08:31:52.698016+0000","last_active":"2024-04-30T08:33:59.705928+0000","last_peered":"2024-04-30T08:33:59.705928+0000","last_clean":"2024-04-30T08:33:59.705928+0000","last_became_active":"2024-04-30T08:31:52.695037+0000","last_became_peered":"2024-04-30T08:31:52.695037+0000","last_unstale":"2024-04-30T08:33:59.705928+0000","last_undegraded":"2024-04-30T08:33:59.705928+0000","last_fullsized":"2024-04-30T08:33:59.705928+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:06:01.360947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":11,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.889900+0000","last_change":"2024-04-30T08:33:31.889445+0000","last_active":"2024-04-30T08:33:31.889900+0000","last_peered":"2024-04-30T08:33:31.889900+0000","last_clean":"2024-04-30T08:33:31.889900+0000","last_became_active":"2024-04-30T08:33:31.889110+0000","last_became_peered":"2024-04-30T08:33:31.889110+0000","last_unstale":"2024-04-30T08:33:31.889900+0000","last_undegraded":"2024-04-30T08:33:31.889900+0000","last_fullsized":"2024-04-30T08:33:31.889900+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:34:26.741854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.29","version":"85'8","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.699434+0000","last_change":"2024-04-30T08:31:52.694776+0000","last_active":"2024-04-30T08:33:59.699434+0000","last_peered":"2024-04-30T08:33:59.699434+0000","last_clean":"2024-04-30T08:33:59.699434+0000","last_became_active":"2024-04-30T08:31:52.693532+0000","last_became_peered":"2024-04-30T08:31:52.693532+0000","last_unstale":"2024-04-30T08:33:59.699434+0000","last_undegraded":"2024-04-30T08:33:59.699434+0000","last_fullsized":"2024-04-30T08:33:59.699434+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:31:10.186667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"85'10","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.706117+0000","last_change":"2024-04-30T08:31:52.688477+0000","last_active":"2024-04-30T08:33:59.706117+0000","last_peered":"2024-04-30T08:33:59.706117+0000","last_clean":"2024-04-30T08:33:59.706117+0000","last_became_active":"2024-04-30T08:31:52.688072+0000","last_became_peered":"2024-04-30T08:31:52.688072+0000","last_unstale":"2024-04-30T08:33:59.706117+0000","last_undegraded":"2024-04-30T08:33:59.706117+0000","last_fullsized":"2024-04-30T08:33:59.706117+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:35:42.659707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"85'11","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.738132+0000","last_change":"2024-04-30T08:31:52.716788+0000","last_active":"2024-04-30T08:33:59.738132+0000","last_peered":"2024-04-30T08:33:59.738132+0000","last_clean":"2024-04-30T08:33:59.738132+0000","last_became_active":"2024-04-30T08:31:52.716336+0000","last_became_peered":"2024-04-30T08:31:52.716336+0000","last_unstale":"2024-04-30T08:33:59.738132+0000","last_undegraded":"2024-04-30T08:33:59.738132+0000","last_fullsized":"2024-04-30T08:33:59.738132+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:23:23.477968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"85'8","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.681080+0000","last_change":"2024-04-30T08:31:52.684571+0000","last_active":"2024-04-30T08:33:59.681080+0000","last_peered":"2024-04-30T08:33:59.681080+0000","last_clean":"2024-04-30T08:33:59.681080+0000","last_became_active":"2024-04-30T08:31:52.684192+0000","last_became_peered":"2024-04-30T08:31:52.684192+0000","last_unstale":"2024-04-30T08:33:59.681080+0000","last_undegraded":"2024-04-30T08:33:59.681080+0000","last_fullsized":"2024-04-30T08:33:59.681080+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:05:57.520133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":442,"ondisk_log_size":442,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284927,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":69558,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":940,"num_read_kb":541,"num_write":245,"num_write_kb":142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":561152,"data_stored":209031,"data_compressed":5523,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":213,"ondisk_log_size":213,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678057,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":946176,"data_stored":479251,"data_compressed":26652,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874285,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1081344,"data_stored":1128462,"data_compressed":64719,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037812,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":483209,"data_compressed":26652,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":15,"apply_latency_ms":15,"commit_latency_ns":15000000,"apply_latency_ns":15000000},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234044,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":958464,"data_stored":479491,"data_compressed":26652,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430273,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":1076661,"data_compressed":62878,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626501,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":937984,"data_stored":474617,"data_compressed":26652,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574948,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36316,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708836,"statfs":{"total":95995035648,"available":95957848064,"internally_reserved":0,"allocated":1003520,"data_stored":537968,"data_compressed":28493,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738472,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":476073,"data_compressed":26652,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":9,"apply_latency_ms":9,"commit_latency_ns":9000000,"apply_latency_ns":9000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411423,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1011712,"data_stored":538262,"data_compressed":28493,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182488,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27948,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717204,"statfs":{"total":95995035648,"available":95966416896,"internally_reserved":0,"allocated":950272,"data_stored":474325,"data_compressed":26652,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986259,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":479223,"data_compressed":26652,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822733,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":1071711,"data_compressed":62878,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34599,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34599,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34599,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":64073,"data_compressed":1841,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":64757,"data_compressed":1841,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1232,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4206,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":64199,"data_compressed":1841,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-30T08:34:05.664 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-30T08:34:05.664 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-30T08:34:05.664 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-30T08:34:05.664 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-30T08:34:05.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:05 smithi032 ceph-mon[28331]: from='client.14964 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:34:06.109 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:34:06.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:05 smithi071 ceph-mon[33385]: from='client.14964 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:34:06.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:05 smithi189 ceph-mon[33038]: from='client.14964 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:34:06.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:06 smithi032 ceph-mon[28331]: pgmap v393: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-30T08:34:06.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:06 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:34:06.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:06 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:34:06.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:06 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:34:07.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:06 smithi071 ceph-mon[33385]: pgmap v393: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-30T08:34:07.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:06 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:34:07.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:06 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:34:07.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:06 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:34:07.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:06 smithi189 ceph-mon[33038]: pgmap v393: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-30T08:34:07.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:06 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:34:07.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:06 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:34:07.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:06 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:34:08.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.565+0000 7fac58400700 1 -- 172.21.15.32:0/3733617890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac50110ad0 msgr2=0x7fac50112f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:08.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.565+0000 7fac58400700 1 --2- 172.21.15.32:0/3733617890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac50110ad0 0x7fac50112f30 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fac4800b0a0 tx=0x7fac4800b3b0 comp rx=0 tx=0).stop 2024-04-30T08:34:08.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.566+0000 7fac58400700 1 -- 172.21.15.32:0/3733617890 shutdown_connections 2024-04-30T08:34:08.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.566+0000 7fac58400700 1 --2- 172.21.15.32:0/3733617890 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac50110ad0 0x7fac50112f30 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:08.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.566+0000 7fac58400700 1 --2- 172.21.15.32:0/3733617890 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac50076d90 0x7fac500771a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:08.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.566+0000 7fac58400700 1 --2- 172.21.15.32:0/3733617890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac500776e0 0x7fac50110590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:08.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.566+0000 7fac58400700 1 -- 172.21.15.32:0/3733617890 >> 172.21.15.32:0/3733617890 conn(0x7fac500ffdf0 msgr2=0x7fac50102240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:08.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.566+0000 7fac58400700 1 -- 172.21.15.32:0/3733617890 shutdown_connections 2024-04-30T08:34:08.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.566+0000 7fac58400700 1 -- 172.21.15.32:0/3733617890 wait complete. 2024-04-30T08:34:08.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.567+0000 7fac58400700 1 Processor -- start 2024-04-30T08:34:08.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.567+0000 7fac58400700 1 -- start start 2024-04-30T08:34:08.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.567+0000 7fac58400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac50076d90 0x7fac5019cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:08.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.567+0000 7fac58400700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac500776e0 0x7fac5019d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:08.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.568+0000 7fac58400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac50110ad0 0x7fac501a1780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:08.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.568+0000 7fac58400700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fac50116600 con 0x7fac500776e0 2024-04-30T08:34:08.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.568+0000 7fac58400700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fac50116480 con 0x7fac50110ad0 2024-04-30T08:34:08.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.568+0000 7fac58400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fac50116780 con 0x7fac50076d90 2024-04-30T08:34:08.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.568+0000 7fac55a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac50076d90 0x7fac5019cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:08.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.568+0000 7fac55000700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac500776e0 0x7fac5019d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:08.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.568+0000 7fac55a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac50076d90 0x7fac5019cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:60274/0 (socket says 172.21.15.32:60274) 2024-04-30T08:34:08.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.568+0000 7fac55a00700 1 -- 172.21.15.32:0/3900783240 learned_addr learned my addr 172.21.15.32:0/3900783240 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:08.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.568+0000 7fac56400700 1 --2- 172.21.15.32:0/3900783240 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac50110ad0 0x7fac501a1780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:08.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.568+0000 7fac55a00700 1 -- 172.21.15.32:0/3900783240 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac50110ad0 msgr2=0x7fac501a1780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:08.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.568+0000 7fac55a00700 1 --2- 172.21.15.32:0/3900783240 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac50110ad0 0x7fac501a1780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:08.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.568+0000 7fac55a00700 1 -- 172.21.15.32:0/3900783240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac500776e0 msgr2=0x7fac5019d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:08.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.568+0000 7fac55a00700 1 --2- 172.21.15.32:0/3900783240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac500776e0 0x7fac5019d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:08.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.568+0000 7fac55a00700 1 -- 172.21.15.32:0/3900783240 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac501a1e20 con 0x7fac50076d90 2024-04-30T08:34:08.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.569+0000 7fac55a00700 1 --2- 172.21.15.32:0/3900783240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac50076d90 0x7fac5019cf50 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fac4400eb30 tx=0x7fac4400ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:08.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.569+0000 7fac4e000700 1 -- 172.21.15.32:0/3900783240 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac4400cbd0 con 0x7fac50076d90 2024-04-30T08:34:08.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.569+0000 7fac58400700 1 -- 172.21.15.32:0/3900783240 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac501a2110 con 0x7fac50076d90 2024-04-30T08:34:08.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.569+0000 7fac4e000700 1 -- 172.21.15.32:0/3900783240 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac440044d0 con 0x7fac50076d90 2024-04-30T08:34:08.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.569+0000 7fac4e000700 1 -- 172.21.15.32:0/3900783240 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac44003e30 con 0x7fac50076d90 2024-04-30T08:34:08.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.569+0000 7fac58400700 1 -- 172.21.15.32:0/3900783240 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac501a9a30 con 0x7fac50076d90 2024-04-30T08:34:08.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.570+0000 7fac58400700 1 -- 172.21.15.32:0/3900783240 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac50051210 con 0x7fac50076d90 2024-04-30T08:34:08.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.573+0000 7fac4e000700 1 -- 172.21.15.32:0/3900783240 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fac44004000 con 0x7fac50076d90 2024-04-30T08:34:08.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.575+0000 7fac4e000700 1 --2- 172.21.15.32:0/3900783240 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fac380afc00 0x7fac380b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:08.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.575+0000 7fac4e000700 1 -- 172.21.15.32:0/3900783240 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fac440df490 con 0x7fac50076d90 2024-04-30T08:34:08.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.575+0000 7fac55000700 1 --2- 172.21.15.32:0/3900783240 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fac380afc00 0x7fac380b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:08.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.576+0000 7fac55000700 1 --2- 172.21.15.32:0/3900783240 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fac380afc00 0x7fac380b2110 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fac3c009680 tx=0x7fac3c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:08.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.577+0000 7fac4e000700 1 -- 172.21.15.32:0/3900783240 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fac44052ff0 con 0x7fac50076d90 2024-04-30T08:34:08.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.863+0000 7fac58400700 1 -- 172.21.15.32:0/3900783240 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fac5019e2b0 con 0x7fac50076d90 2024-04-30T08:34:08.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.864+0000 7fac4e000700 1 -- 172.21.15.32:0/3900783240 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fac440042f0 con 0x7fac50076d90 2024-04-30T08:34:08.865 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:34:08.865 INFO:teuthology.orchestra.run.smithi032.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-30T08:34:08.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.867+0000 7fac58400700 1 -- 172.21.15.32:0/3900783240 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fac380afc00 msgr2=0x7fac380b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:08.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.867+0000 7fac58400700 1 --2- 172.21.15.32:0/3900783240 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fac380afc00 0x7fac380b2110 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fac3c009680 tx=0x7fac3c009260 comp rx=0 tx=0).stop 2024-04-30T08:34:08.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.867+0000 7fac58400700 1 -- 172.21.15.32:0/3900783240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac50076d90 msgr2=0x7fac5019cf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:08.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.867+0000 7fac58400700 1 --2- 172.21.15.32:0/3900783240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac50076d90 0x7fac5019cf50 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fac4400eb30 tx=0x7fac4400ee40 comp rx=0 tx=0).stop 2024-04-30T08:34:08.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.867+0000 7fac58400700 1 -- 172.21.15.32:0/3900783240 shutdown_connections 2024-04-30T08:34:08.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.867+0000 7fac58400700 1 --2- 172.21.15.32:0/3900783240 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fac380afc00 0x7fac380b2110 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:08.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.867+0000 7fac58400700 1 --2- 172.21.15.32:0/3900783240 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac50110ad0 0x7fac501a1780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:08.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.867+0000 7fac58400700 1 --2- 172.21.15.32:0/3900783240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac50076d90 0x7fac5019cf50 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:08.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.867+0000 7fac58400700 1 --2- 172.21.15.32:0/3900783240 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac500776e0 0x7fac5019d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:08.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.867+0000 7fac58400700 1 -- 172.21.15.32:0/3900783240 >> 172.21.15.32:0/3900783240 conn(0x7fac500ffdf0 msgr2=0x7fac50100200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:08.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.867+0000 7fac58400700 1 -- 172.21.15.32:0/3900783240 shutdown_connections 2024-04-30T08:34:08.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:08.867+0000 7fac58400700 1 -- 172.21.15.32:0/3900783240 wait complete. 2024-04-30T08:34:09.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:08 smithi071 ceph-mon[33385]: pgmap v394: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 597 B/s wr, 9 op/s 2024-04-30T08:34:09.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:08 smithi032 ceph-mon[28331]: pgmap v394: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 597 B/s wr, 9 op/s 2024-04-30T08:34:09.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:08 smithi189 ceph-mon[33038]: pgmap v394: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 597 B/s wr, 9 op/s 2024-04-30T08:34:09.676 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-30T08:34:09.677 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-30T08:34:09.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:09 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3900783240' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-30T08:34:09.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.971+0000 7f53c3e00640 1 -- 172.21.15.32:0/4268272197 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53bc109ad0 msgr2=0x7f53bc10bef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:09.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.971+0000 7f53c3e00640 1 --2- 172.21.15.32:0/4268272197 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53bc109ad0 0x7f53bc10bef0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f53b4009900 tx=0x7f53b402a410 comp rx=0 tx=0).stop 2024-04-30T08:34:09.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.971+0000 7f53c3e00640 1 -- 172.21.15.32:0/4268272197 shutdown_connections 2024-04-30T08:34:09.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.971+0000 7f53c3e00640 1 --2- 172.21.15.32:0/4268272197 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53bc109ad0 0x7f53bc10bef0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:09.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.971+0000 7f53c3e00640 1 --2- 172.21.15.32:0/4268272197 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53bc104dd0 0x7f53bc1051e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:09.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.971+0000 7f53c3e00640 1 --2- 172.21.15.32:0/4268272197 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53bc103b90 0x7f53bc104890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:09.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.971+0000 7f53c3e00640 1 -- 172.21.15.32:0/4268272197 >> 172.21.15.32:0/4268272197 conn(0x7f53bc0fedc0 msgr2=0x7f53bc101230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:09.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.971+0000 7f53c3e00640 1 -- 172.21.15.32:0/4268272197 shutdown_connections 2024-04-30T08:34:09.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.971+0000 7f53c3e00640 1 -- 172.21.15.32:0/4268272197 wait complete. 2024-04-30T08:34:09.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.972+0000 7f53c3e00640 1 Processor -- start 2024-04-30T08:34:09.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.972+0000 7f53c3e00640 1 -- start start 2024-04-30T08:34:09.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.973+0000 7f53c3e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53bc103b90 0x7f53bc19a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:09.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.973+0000 7f53c3e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53bc104dd0 0x7f53bc19a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:09.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.973+0000 7f53c3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53bc109ad0 0x7f53bc1a17b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:09.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.973+0000 7f53c3e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f53bc10f710 con 0x7f53bc109ad0 2024-04-30T08:34:09.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.973+0000 7f53c3e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f53bc10f590 con 0x7f53bc104dd0 2024-04-30T08:34:09.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.973+0000 7f53c3e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f53bc10f890 con 0x7f53bc103b90 2024-04-30T08:34:09.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.973+0000 7f53c1400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53bc103b90 0x7f53bc19a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:09.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.973+0000 7f53c1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53bc109ad0 0x7f53bc1a17b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:09.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.973+0000 7f53c1400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53bc103b90 0x7f53bc19a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:60292/0 (socket says 172.21.15.32:60292) 2024-04-30T08:34:09.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.973+0000 7f53c1400640 1 -- 172.21.15.32:0/1991472297 learned_addr learned my addr 172.21.15.32:0/1991472297 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:09.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.973+0000 7f53c0a00640 1 --2- 172.21.15.32:0/1991472297 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53bc104dd0 0x7f53bc19a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:09.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.974+0000 7f53c1400640 1 -- 172.21.15.32:0/1991472297 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53bc104dd0 msgr2=0x7f53bc19a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:09.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.974+0000 7f53c1400640 1 --2- 172.21.15.32:0/1991472297 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53bc104dd0 0x7f53bc19a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:09.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.974+0000 7f53c1400640 1 -- 172.21.15.32:0/1991472297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53bc109ad0 msgr2=0x7f53bc1a17b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:09.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.974+0000 7f53c1400640 1 --2- 172.21.15.32:0/1991472297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53bc109ad0 0x7f53bc1a17b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:09.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.974+0000 7f53c1400640 1 -- 172.21.15.32:0/1991472297 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53bc1a1e20 con 0x7f53bc103b90 2024-04-30T08:34:09.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.974+0000 7f53c1400640 1 --2- 172.21.15.32:0/1991472297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53bc103b90 0x7f53bc19a2e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f53b000eb90 tx=0x7f53b000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:09.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.974+0000 7f53ba000640 1 -- 172.21.15.32:0/1991472297 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b0003b70 con 0x7f53bc103b90 2024-04-30T08:34:09.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.974+0000 7f53ba000640 1 -- 172.21.15.32:0/1991472297 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53b000ee90 con 0x7f53bc103b90 2024-04-30T08:34:09.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.974+0000 7f53ba000640 1 -- 172.21.15.32:0/1991472297 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53b00197b0 con 0x7f53bc103b90 2024-04-30T08:34:09.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.974+0000 7f53c3e00640 1 -- 172.21.15.32:0/1991472297 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53bc1a2110 con 0x7f53bc103b90 2024-04-30T08:34:09.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.975+0000 7f53c3e00640 1 -- 172.21.15.32:0/1991472297 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53bc1a2610 con 0x7f53bc103b90 2024-04-30T08:34:09.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.976+0000 7f53c3e00640 1 -- 172.21.15.32:0/1991472297 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f53bc1081d0 con 0x7f53bc103b90 2024-04-30T08:34:09.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.978+0000 7f53ba000640 1 -- 172.21.15.32:0/1991472297 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f53b0004030 con 0x7f53bc103b90 2024-04-30T08:34:09.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.979+0000 7f53ba000640 1 --2- 172.21.15.32:0/1991472297 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f53a40afaf0 0x7f53a40b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:09.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.980+0000 7f53ba000640 1 -- 172.21.15.32:0/1991472297 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f53b00df850 con 0x7f53bc103b90 2024-04-30T08:34:09.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.980+0000 7f53c0a00640 1 --2- 172.21.15.32:0/1991472297 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f53a40afaf0 0x7f53a40b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:09.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.981+0000 7f53c0a00640 1 --2- 172.21.15.32:0/1991472297 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f53a40afaf0 0x7f53a40b2000 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f53bc19b380 tx=0x7f53a8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:09.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:09.984+0000 7f53ba000640 1 -- 172.21.15.32:0/1991472297 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f53b00a8020 con 0x7f53bc103b90 2024-04-30T08:34:10.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.195+0000 7f53c3e00640 1 -- 172.21.15.32:0/1991472297 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f53bc063820 con 0x7f53bc103b90 2024-04-30T08:34:10.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.198+0000 7f53ba000640 1 -- 172.21.15.32:0/1991472297 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+24730 (secure 0 0 0) 0x7f53b00b00a0 con 0x7f53bc103b90 2024-04-30T08:34:10.200 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:34:10.200 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":85,"fsid":"98c0a670-06ca-11ef-bc93-c7b262605968","created":"2024-04-30T08:22:11.122875+0000","modified":"2024-04-30T08:33:31.872704+0000","last_up_change":"2024-04-30T08:30:50.350439+0000","last_in_change":"2024-04-30T08:30:30.210620+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-30T08:26:19.728331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-30T08:31:49.800542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-30T08:31:51.211320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-30T08:31:55.716625+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a7e060c8-1cbb-418f-9a03-06693f6b3f52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6803","nonce":2624700402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6805","nonce":2624700402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6809","nonce":2624700402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6807","nonce":2624700402}]},"public_addr":"172.21.15.32:6803/2624700402","cluster_addr":"172.21.15.32:6805/2624700402","heartbeat_back_addr":"172.21.15.32:6809/2624700402","heartbeat_front_addr":"172.21.15.32:6807/2624700402","state":["exists","up"]},{"osd":1,"uuid":"72ef2fb1-e5ca-48ef-aa3a-4796a64b922f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6801","nonce":444649711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6803","nonce":444649711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6807","nonce":444649711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6805","nonce":444649711}]},"public_addr":"172.21.15.71:6801/444649711","cluster_addr":"172.21.15.71:6803/444649711","heartbeat_back_addr":"172.21.15.71:6807/444649711","heartbeat_front_addr":"172.21.15.71:6805/444649711","state":["exists","up"]},{"osd":2,"uuid":"704e5e8a-935e-47c7-ae28-27119876ac09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6801","nonce":3255528671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6803","nonce":3255528671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6807","nonce":3255528671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6805","nonce":3255528671}]},"public_addr":"172.21.15.189:6801/3255528671","cluster_addr":"172.21.15.189:6803/3255528671","heartbeat_back_addr":"172.21.15.189:6807/3255528671","heartbeat_front_addr":"172.21.15.189:6805/3255528671","state":["exists","up"]},{"osd":3,"uuid":"575004f7-6559-4a62-863e-e33078b0eebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6811","nonce":1384702112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6813","nonce":1384702112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6817","nonce":1384702112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6815","nonce":1384702112}]},"public_addr":"172.21.15.32:6811/1384702112","cluster_addr":"172.21.15.32:6813/1384702112","heartbeat_back_addr":"172.21.15.32:6817/1384702112","heartbeat_front_addr":"172.21.15.32:6815/1384702112","state":["exists","up"]},{"osd":4,"uuid":"e1c1e894-074d-4c78-89ef-9e5a114268bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6809","nonce":4250534574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6811","nonce":4250534574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6815","nonce":4250534574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6813","nonce":4250534574}]},"public_addr":"172.21.15.71:6809/4250534574","cluster_addr":"172.21.15.71:6811/4250534574","heartbeat_back_addr":"172.21.15.71:6815/4250534574","heartbeat_front_addr":"172.21.15.71:6813/4250534574","state":["exists","up"]},{"osd":5,"uuid":"12c022d4-fac6-40c4-ae7a-691571bee5d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6809","nonce":4245346198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6811","nonce":4245346198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6815","nonce":4245346198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6813","nonce":4245346198}]},"public_addr":"172.21.15.189:6809/4245346198","cluster_addr":"172.21.15.189:6811/4245346198","heartbeat_back_addr":"172.21.15.189:6815/4245346198","heartbeat_front_addr":"172.21.15.189:6813/4245346198","state":["exists","up"]},{"osd":6,"uuid":"b413f8ee-3d4a-415a-a2ba-d68437b7330d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6819","nonce":1910389190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6821","nonce":1910389190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6825","nonce":1910389190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6823","nonce":1910389190}]},"public_addr":"172.21.15.32:6819/1910389190","cluster_addr":"172.21.15.32:6821/1910389190","heartbeat_back_addr":"172.21.15.32:6825/1910389190","heartbeat_front_addr":"172.21.15.32:6823/1910389190","state":["exists","up"]},{"osd":7,"uuid":"aac64fb3-fc13-4126-b842-7b06ad5ce238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6817","nonce":692607380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6819","nonce":692607380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6823","nonce":692607380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6821","nonce":692607380}]},"public_addr":"172.21.15.71:6817/692607380","cluster_addr":"172.21.15.71:6819/692607380","heartbeat_back_addr":"172.21.15.71:6823/692607380","heartbeat_front_addr":"172.21.15.71:6821/692607380","state":["exists","up"]},{"osd":8,"uuid":"cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6817","nonce":3973697069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6819","nonce":3973697069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6823","nonce":3973697069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6821","nonce":3973697069}]},"public_addr":"172.21.15.189:6817/3973697069","cluster_addr":"172.21.15.189:6819/3973697069","heartbeat_back_addr":"172.21.15.189:6823/3973697069","heartbeat_front_addr":"172.21.15.189:6821/3973697069","state":["exists","up"]},{"osd":9,"uuid":"ef4a1645-ae6e-45c3-82ad-161510e62aa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6827","nonce":46066685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6829","nonce":46066685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6833","nonce":46066685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6831","nonce":46066685}]},"public_addr":"172.21.15.32:6827/46066685","cluster_addr":"172.21.15.32:6829/46066685","heartbeat_back_addr":"172.21.15.32:6833/46066685","heartbeat_front_addr":"172.21.15.32:6831/46066685","state":["exists","up"]},{"osd":10,"uuid":"25b26a38-a644-409f-b941-59efdf04a3ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6825","nonce":1886290293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6827","nonce":1886290293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6831","nonce":1886290293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6829","nonce":1886290293}]},"public_addr":"172.21.15.71:6825/1886290293","cluster_addr":"172.21.15.71:6827/1886290293","heartbeat_back_addr":"172.21.15.71:6831/1886290293","heartbeat_front_addr":"172.21.15.71:6829/1886290293","state":["exists","up"]},{"osd":11,"uuid":"ab96b3ff-3688-455b-af64-60226c5bfbff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6825","nonce":850185001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6827","nonce":850185001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6831","nonce":850185001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6829","nonce":850185001}]},"public_addr":"172.21.15.189:6825/850185001","cluster_addr":"172.21.15.189:6827/850185001","heartbeat_back_addr":"172.21.15.189:6831/850185001","heartbeat_front_addr":"172.21.15.189:6829/850185001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:27.220308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:53.230783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:17.190642+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:52.603799+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:20.568671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:46.158363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:22.706933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:50.665173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:16.602410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:53.814698+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:22.435189+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:48.651949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.e","mappings":[{"from":4,"to":7}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.28","mappings":[{"from":3,"to":6},{"from":5,"to":8}]},{"pgid":"3.30","mappings":[{"from":1,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1266897222":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/2915689533":"2024-05-01T08:23:29.526664+0000","172.21.15.32:6801/4019351306":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/425836193":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/1134075722":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6801/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/21059253":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/3245550894":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/4019351306":"2024-05-01T08:23:29.526664+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-30T08:34:10.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.201+0000 7f53c3e00640 1 -- 172.21.15.32:0/1991472297 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f53a40afaf0 msgr2=0x7f53a40b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:10.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.201+0000 7f53c3e00640 1 --2- 172.21.15.32:0/1991472297 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f53a40afaf0 0x7f53a40b2000 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f53bc19b380 tx=0x7f53a8005f90 comp rx=0 tx=0).stop 2024-04-30T08:34:10.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.201+0000 7f53c3e00640 1 -- 172.21.15.32:0/1991472297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53bc103b90 msgr2=0x7f53bc19a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:10.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.201+0000 7f53c3e00640 1 --2- 172.21.15.32:0/1991472297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53bc103b90 0x7f53bc19a2e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f53b000eb90 tx=0x7f53b000c6a0 comp rx=0 tx=0).stop 2024-04-30T08:34:10.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.201+0000 7f53c3e00640 1 -- 172.21.15.32:0/1991472297 shutdown_connections 2024-04-30T08:34:10.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.201+0000 7f53c3e00640 1 --2- 172.21.15.32:0/1991472297 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f53a40afaf0 0x7f53a40b2000 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:10.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.201+0000 7f53c3e00640 1 --2- 172.21.15.32:0/1991472297 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53bc109ad0 0x7f53bc1a17b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:10.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.201+0000 7f53c3e00640 1 --2- 172.21.15.32:0/1991472297 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53bc104dd0 0x7f53bc19a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:10.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.201+0000 7f53c3e00640 1 --2- 172.21.15.32:0/1991472297 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53bc103b90 0x7f53bc19a2e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:10.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.201+0000 7f53c3e00640 1 -- 172.21.15.32:0/1991472297 >> 172.21.15.32:0/1991472297 conn(0x7f53bc0fedc0 msgr2=0x7f53bc102230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:10.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.201+0000 7f53c3e00640 1 -- 172.21.15.32:0/1991472297 shutdown_connections 2024-04-30T08:34:10.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.201+0000 7f53c3e00640 1 -- 172.21.15.32:0/1991472297 wait complete. 2024-04-30T08:34:10.234 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-30T08:26:19.728331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-30T08:31:49.800542+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-30T08:31:51.211320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-30T08:31:55.716625+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-30T08:34:10.235 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-30T08:34:10.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:09 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3900783240' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-30T08:34:10.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:09 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3900783240' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-30T08:34:10.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.529+0000 7f5deb200640 1 -- 172.21.15.32:0/2747069657 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dec105bb0 msgr2=0x7f5dec1068b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:10.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.529+0000 7f5deb200640 1 --2- 172.21.15.32:0/2747069657 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dec105bb0 0x7f5dec1068b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc009be0 tx=0x7f5ddc02a6c0 comp rx=0 tx=0).stop 2024-04-30T08:34:10.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.529+0000 7f5deb200640 1 -- 172.21.15.32:0/2747069657 shutdown_connections 2024-04-30T08:34:10.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.529+0000 7f5deb200640 1 --2- 172.21.15.32:0/2747069657 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dec10baf0 0x7f5dec10df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:10.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.529+0000 7f5deb200640 1 --2- 172.21.15.32:0/2747069657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dec106df0 0x7f5dec107200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:10.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.529+0000 7f5deb200640 1 --2- 172.21.15.32:0/2747069657 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dec105bb0 0x7f5dec1068b0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:10.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.529+0000 7f5deb200640 1 -- 172.21.15.32:0/2747069657 >> 172.21.15.32:0/2747069657 conn(0x7f5dec100da0 msgr2=0x7f5dec103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:10.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.529+0000 7f5deb200640 1 -- 172.21.15.32:0/2747069657 shutdown_connections 2024-04-30T08:34:10.531 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.529+0000 7f5deb200640 1 -- 172.21.15.32:0/2747069657 wait complete. 2024-04-30T08:34:10.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.530+0000 7f5deb200640 1 Processor -- start 2024-04-30T08:34:10.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.530+0000 7f5deb200640 1 -- start start 2024-04-30T08:34:10.532 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.531+0000 7f5deb200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dec105bb0 0x7f5dec19c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:10.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.531+0000 7f5deb200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dec106df0 0x7f5dec19c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:10.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.531+0000 7f5deb200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dec10baf0 0x7f5dec1a3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:10.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.531+0000 7f5deb200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5dec111690 con 0x7f5dec106df0 2024-04-30T08:34:10.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.531+0000 7f5deb200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5dec111510 con 0x7f5dec10baf0 2024-04-30T08:34:10.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.531+0000 7f5deb200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5dec111810 con 0x7f5dec105bb0 2024-04-30T08:34:10.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.531+0000 7f5de9400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dec106df0 0x7f5dec19c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:10.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.531+0000 7f5de9400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dec106df0 0x7f5dec19c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45598/0 (socket says 172.21.15.32:45598) 2024-04-30T08:34:10.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.531+0000 7f5de9400640 1 -- 172.21.15.32:0/2009195554 learned_addr learned my addr 172.21.15.32:0/2009195554 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:10.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.531+0000 7f5de9e00640 1 --2- 172.21.15.32:0/2009195554 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dec105bb0 0x7f5dec19c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:10.533 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.531+0000 7f5dea800640 1 --2- 172.21.15.32:0/2009195554 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dec10baf0 0x7f5dec1a3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:10.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.531+0000 7f5de9400640 1 -- 172.21.15.32:0/2009195554 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dec105bb0 msgr2=0x7f5dec19c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:10.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.531+0000 7f5de9400640 1 --2- 172.21.15.32:0/2009195554 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dec105bb0 0x7f5dec19c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:10.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.531+0000 7f5de9400640 1 -- 172.21.15.32:0/2009195554 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dec10baf0 msgr2=0x7f5dec1a3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:10.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.531+0000 7f5de9400640 1 --2- 172.21.15.32:0/2009195554 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dec10baf0 0x7f5dec1a3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:10.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.531+0000 7f5de9400640 1 -- 172.21.15.32:0/2009195554 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5dec1a3e00 con 0x7f5dec106df0 2024-04-30T08:34:10.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.532+0000 7f5de9400640 1 --2- 172.21.15.32:0/2009195554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dec106df0 0x7f5dec19c800 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5dd400e9c0 tx=0x7f5dd400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:10.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.532+0000 7f5de2a00640 1 -- 172.21.15.32:0/2009195554 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd400ccc0 con 0x7f5dec106df0 2024-04-30T08:34:10.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.532+0000 7f5deb200640 1 -- 172.21.15.32:0/2009195554 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5dec1a40f0 con 0x7f5dec106df0 2024-04-30T08:34:10.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.532+0000 7f5de2a00640 1 -- 172.21.15.32:0/2009195554 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5dd40054d0 con 0x7f5dec106df0 2024-04-30T08:34:10.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.532+0000 7f5de2a00640 1 -- 172.21.15.32:0/2009195554 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dd4010430 con 0x7f5dec106df0 2024-04-30T08:34:10.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.532+0000 7f5deb200640 1 -- 172.21.15.32:0/2009195554 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5dec1a4640 con 0x7f5dec106df0 2024-04-30T08:34:10.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.533+0000 7f5deb200640 1 -- 172.21.15.32:0/2009195554 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5dec10a900 con 0x7f5dec106df0 2024-04-30T08:34:10.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.536+0000 7f5de2a00640 1 -- 172.21.15.32:0/2009195554 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f5dd4004f80 con 0x7f5dec106df0 2024-04-30T08:34:10.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.537+0000 7f5de2a00640 1 --2- 172.21.15.32:0/2009195554 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f5dd00afae0 0x7f5dd00b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:10.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.537+0000 7f5de2a00640 1 -- 172.21.15.32:0/2009195554 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f5dd40df430 con 0x7f5dec106df0 2024-04-30T08:34:10.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.538+0000 7f5de9e00640 1 --2- 172.21.15.32:0/2009195554 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f5dd00afae0 0x7f5dd00b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:10.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.538+0000 7f5de9e00640 1 --2- 172.21.15.32:0/2009195554 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f5dd00afae0 0x7f5dd00b1ff0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc02abe0 tx=0x7f5ddc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:10.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.542+0000 7f5de2a00640 1 -- 172.21.15.32:0/2009195554 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5dd40a7c00 con 0x7f5dec106df0 2024-04-30T08:34:10.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.752+0000 7f5deb200640 1 -- 172.21.15.32:0/2009195554 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f5dec063820 con 0x7f5dec106df0 2024-04-30T08:34:10.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.754+0000 7f5de2a00640 1 -- 172.21.15.32:0/2009195554 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) ==== 93+0+10 (secure 0 0 0) 0x7f5dd40a7500 con 0x7f5dec106df0 2024-04-30T08:34:10.755 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-30T08:34:10.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.757+0000 7f5deb200640 1 -- 172.21.15.32:0/2009195554 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f5dd00afae0 msgr2=0x7f5dd00b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:10.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.757+0000 7f5deb200640 1 --2- 172.21.15.32:0/2009195554 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f5dd00afae0 0x7f5dd00b1ff0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc02abe0 tx=0x7f5ddc005fd0 comp rx=0 tx=0).stop 2024-04-30T08:34:10.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.757+0000 7f5deb200640 1 -- 172.21.15.32:0/2009195554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dec106df0 msgr2=0x7f5dec19c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:10.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.757+0000 7f5deb200640 1 --2- 172.21.15.32:0/2009195554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dec106df0 0x7f5dec19c800 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5dd400e9c0 tx=0x7f5dd400ee80 comp rx=0 tx=0).stop 2024-04-30T08:34:10.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.757+0000 7f5deb200640 1 -- 172.21.15.32:0/2009195554 shutdown_connections 2024-04-30T08:34:10.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.757+0000 7f5deb200640 1 --2- 172.21.15.32:0/2009195554 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f5dd00afae0 0x7f5dd00b1ff0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:10.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.757+0000 7f5deb200640 1 --2- 172.21.15.32:0/2009195554 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5dec10baf0 0x7f5dec1a3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:10.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.757+0000 7f5deb200640 1 --2- 172.21.15.32:0/2009195554 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5dec106df0 0x7f5dec19c800 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:10.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.757+0000 7f5deb200640 1 --2- 172.21.15.32:0/2009195554 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5dec105bb0 0x7f5dec19c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:10.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.757+0000 7f5deb200640 1 -- 172.21.15.32:0/2009195554 >> 172.21.15.32:0/2009195554 conn(0x7f5dec100da0 msgr2=0x7f5dec107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:10.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.758+0000 7f5deb200640 1 -- 172.21.15.32:0/2009195554 shutdown_connections 2024-04-30T08:34:10.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:10.758+0000 7f5deb200640 1 -- 172.21.15.32:0/2009195554 wait complete. 2024-04-30T08:34:10.786 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-30T08:34:11.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.073+0000 7fca14a00640 1 -- 172.21.15.32:0/927579380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca1010baf0 msgr2=0x7fca1010df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:11.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.073+0000 7fca14a00640 1 --2- 172.21.15.32:0/927579380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca1010baf0 0x7fca1010df10 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fca08009ca0 tx=0x7fca0802a8b0 comp rx=0 tx=0).stop 2024-04-30T08:34:11.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.073+0000 7fca14a00640 1 -- 172.21.15.32:0/927579380 shutdown_connections 2024-04-30T08:34:11.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.073+0000 7fca14a00640 1 --2- 172.21.15.32:0/927579380 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca1010baf0 0x7fca1010df10 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:11.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.073+0000 7fca14a00640 1 --2- 172.21.15.32:0/927579380 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca10106df0 0x7fca10107200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:11.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.073+0000 7fca14a00640 1 --2- 172.21.15.32:0/927579380 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca10105bb0 0x7fca101068b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:11.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.073+0000 7fca14a00640 1 -- 172.21.15.32:0/927579380 >> 172.21.15.32:0/927579380 conn(0x7fca10100da0 msgr2=0x7fca10103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:11.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.073+0000 7fca14a00640 1 -- 172.21.15.32:0/927579380 shutdown_connections 2024-04-30T08:34:11.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.073+0000 7fca14a00640 1 -- 172.21.15.32:0/927579380 wait complete. 2024-04-30T08:34:11.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.075+0000 7fca14a00640 1 Processor -- start 2024-04-30T08:34:11.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.075+0000 7fca14a00640 1 -- start start 2024-04-30T08:34:11.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.075+0000 7fca14a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca10105bb0 0x7fca1019c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:11.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.075+0000 7fca14a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca10106df0 0x7fca1019c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:11.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.075+0000 7fca14a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca1010baf0 0x7fca101a3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:11.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.075+0000 7fca14a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fca10111710 con 0x7fca10105bb0 2024-04-30T08:34:11.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.075+0000 7fca14a00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fca10111590 con 0x7fca1010baf0 2024-04-30T08:34:11.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.075+0000 7fca14a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fca10111890 con 0x7fca10106df0 2024-04-30T08:34:11.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.076+0000 7fca0f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca10105bb0 0x7fca1019c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:11.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.076+0000 7fca0ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca10106df0 0x7fca1019c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:11.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.076+0000 7fca0f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca10105bb0 0x7fca1019c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45610/0 (socket says 172.21.15.32:45610) 2024-04-30T08:34:11.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.076+0000 7fca0f400640 1 -- 172.21.15.32:0/3082541752 learned_addr learned my addr 172.21.15.32:0/3082541752 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:11.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.076+0000 7fca0ea00640 1 --2- 172.21.15.32:0/3082541752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca10106df0 0x7fca1019c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:60330/0 (socket says 172.21.15.32:60330) 2024-04-30T08:34:11.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.076+0000 7fca0fe00640 1 --2- 172.21.15.32:0/3082541752 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca1010baf0 0x7fca101a3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:11.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.076+0000 7fca0ea00640 1 -- 172.21.15.32:0/3082541752 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca1010baf0 msgr2=0x7fca101a3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:11.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.076+0000 7fca0ea00640 1 --2- 172.21.15.32:0/3082541752 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca1010baf0 0x7fca101a3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:11.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.076+0000 7fca0ea00640 1 -- 172.21.15.32:0/3082541752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca10105bb0 msgr2=0x7fca1019c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:11.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.076+0000 7fca0ea00640 1 --2- 172.21.15.32:0/3082541752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca10105bb0 0x7fca1019c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:11.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.076+0000 7fca0ea00640 1 -- 172.21.15.32:0/3082541752 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca101a3d90 con 0x7fca10106df0 2024-04-30T08:34:11.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.077+0000 7fca0ea00640 1 --2- 172.21.15.32:0/3082541752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca10106df0 0x7fca1019c790 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc00e9f0 tx=0x7fc9fc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:11.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.077+0000 7fca07e00640 1 -- 172.21.15.32:0/3082541752 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9fc00ccc0 con 0x7fca10106df0 2024-04-30T08:34:11.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.077+0000 7fca07e00640 1 -- 172.21.15.32:0/3082541752 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc9fc0108e0 con 0x7fca10106df0 2024-04-30T08:34:11.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.077+0000 7fca07e00640 1 -- 172.21.15.32:0/3082541752 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9fc018880 con 0x7fca10106df0 2024-04-30T08:34:11.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.077+0000 7fca14a00640 1 -- 172.21.15.32:0/3082541752 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca101a4080 con 0x7fca10106df0 2024-04-30T08:34:11.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.077+0000 7fca14a00640 1 -- 172.21.15.32:0/3082541752 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca101a4580 con 0x7fca10106df0 2024-04-30T08:34:11.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.078+0000 7fca14a00640 1 -- 172.21.15.32:0/3082541752 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca1010a900 con 0x7fca10106df0 2024-04-30T08:34:11.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.082+0000 7fca07e00640 1 -- 172.21.15.32:0/3082541752 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fc9fc010430 con 0x7fca10106df0 2024-04-30T08:34:11.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.083+0000 7fca07e00640 1 --2- 172.21.15.32:0/3082541752 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc9f40afae0 0x7fc9f40b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:11.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.083+0000 7fca07e00640 1 -- 172.21.15.32:0/3082541752 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fc9fc0df380 con 0x7fca10106df0 2024-04-30T08:34:11.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.084+0000 7fca0f400640 1 --2- 172.21.15.32:0/3082541752 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc9f40afae0 0x7fc9f40b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:11.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.084+0000 7fca0f400640 1 --2- 172.21.15.32:0/3082541752 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc9f40afae0 0x7fc9f40b1ff0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fca000095f0 tx=0x7fca00005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:11.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.088+0000 7fca07e00640 1 -- 172.21.15.32:0/3082541752 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc9fc0a7500 con 0x7fca10106df0 2024-04-30T08:34:11.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:10 smithi032 ceph-mon[28331]: pgmap v395: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 12 op/s 2024-04-30T08:34:11.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:10 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1991472297' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:34:11.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:10 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2009195554' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:34:11.201 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:10 smithi189 ceph-mon[33038]: pgmap v395: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 12 op/s 2024-04-30T08:34:11.202 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:10 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1991472297' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:34:11.202 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:10 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2009195554' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:34:11.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:10 smithi071 ceph-mon[33385]: pgmap v395: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 12 op/s 2024-04-30T08:34:11.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:10 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1991472297' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:34:11.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:10 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2009195554' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:34:11.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.295+0000 7fca14a00640 1 -- 172.21.15.32:0/3082541752 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fca10063820 con 0x7fca10106df0 2024-04-30T08:34:11.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.296+0000 7fca07e00640 1 -- 172.21.15.32:0/3082541752 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) ==== 104+0+11 (secure 0 0 0) 0x7fc9fc0ab800 con 0x7fca10106df0 2024-04-30T08:34:11.297 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-30T08:34:11.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.298+0000 7fca14a00640 1 -- 172.21.15.32:0/3082541752 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc9f40afae0 msgr2=0x7fc9f40b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:11.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.298+0000 7fca14a00640 1 --2- 172.21.15.32:0/3082541752 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc9f40afae0 0x7fc9f40b1ff0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fca000095f0 tx=0x7fca00005f90 comp rx=0 tx=0).stop 2024-04-30T08:34:11.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.298+0000 7fca14a00640 1 -- 172.21.15.32:0/3082541752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca10106df0 msgr2=0x7fca1019c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:11.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.298+0000 7fca14a00640 1 --2- 172.21.15.32:0/3082541752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca10106df0 0x7fca1019c790 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fc9fc00e9f0 tx=0x7fc9fc00eeb0 comp rx=0 tx=0).stop 2024-04-30T08:34:11.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.298+0000 7fca14a00640 1 -- 172.21.15.32:0/3082541752 shutdown_connections 2024-04-30T08:34:11.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.298+0000 7fca14a00640 1 --2- 172.21.15.32:0/3082541752 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc9f40afae0 0x7fc9f40b1ff0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:11.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.298+0000 7fca14a00640 1 --2- 172.21.15.32:0/3082541752 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fca1010baf0 0x7fca101a3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:11.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.298+0000 7fca14a00640 1 --2- 172.21.15.32:0/3082541752 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fca10106df0 0x7fca1019c790 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:11.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.298+0000 7fca14a00640 1 --2- 172.21.15.32:0/3082541752 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fca10105bb0 0x7fca1019c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:11.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.298+0000 7fca14a00640 1 -- 172.21.15.32:0/3082541752 >> 172.21.15.32:0/3082541752 conn(0x7fca10100da0 msgr2=0x7fca10107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:11.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.298+0000 7fca14a00640 1 -- 172.21.15.32:0/3082541752 shutdown_connections 2024-04-30T08:34:11.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.298+0000 7fca14a00640 1 -- 172.21.15.32:0/3082541752 wait complete. 2024-04-30T08:34:11.327 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-30T08:34:11.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.619+0000 7f3823e00640 1 -- 172.21.15.32:0/3688426080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f382410ff00 msgr2=0x7f3824112320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:11.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.619+0000 7f3823e00640 1 --2- 172.21.15.32:0/3688426080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f382410ff00 0x7f3824112320 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f381c009ca0 tx=0x7f381c02a8c0 comp rx=0 tx=0).stop 2024-04-30T08:34:11.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.619+0000 7f3823e00640 1 -- 172.21.15.32:0/3688426080 shutdown_connections 2024-04-30T08:34:11.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.619+0000 7f3823e00640 1 --2- 172.21.15.32:0/3688426080 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f382410ff00 0x7f3824112320 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:11.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.619+0000 7f3823e00640 1 --2- 172.21.15.32:0/3688426080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f382410b200 0x7f382410b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:11.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.619+0000 7f3823e00640 1 --2- 172.21.15.32:0/3688426080 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3824109fc0 0x7f382410acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:11.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.619+0000 7f3823e00640 1 -- 172.21.15.32:0/3688426080 >> 172.21.15.32:0/3688426080 conn(0x7f3824078bb0 msgr2=0x7f3824107a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:11.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.619+0000 7f3823e00640 1 -- 172.21.15.32:0/3688426080 shutdown_connections 2024-04-30T08:34:11.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.620+0000 7f3823e00640 1 -- 172.21.15.32:0/3688426080 wait complete. 2024-04-30T08:34:11.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.621+0000 7f3823e00640 1 Processor -- start 2024-04-30T08:34:11.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.621+0000 7f3823e00640 1 -- start start 2024-04-30T08:34:11.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.621+0000 7f3823e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3824109fc0 0x7f38241a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:11.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.621+0000 7f3823e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f382410b200 0x7f38241a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:11.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.621+0000 7f3823e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f382410ff00 0x7f38241a7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:11.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.621+0000 7f3823e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3824115ae0 con 0x7f382410ff00 2024-04-30T08:34:11.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.621+0000 7f3823e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f3824115960 con 0x7f382410b200 2024-04-30T08:34:11.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.621+0000 7f3823e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3824115c60 con 0x7f3824109fc0 2024-04-30T08:34:11.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.621+0000 7f3822000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f382410b200 0x7f38241a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:11.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.621+0000 7f3822a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3824109fc0 0x7f38241a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:11.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.621+0000 7f3823400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f382410ff00 0x7f38241a7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:11.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.621+0000 7f3822000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f382410b200 0x7f38241a0b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:47292/0 (socket says 172.21.15.32:47292) 2024-04-30T08:34:11.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.622+0000 7f3822000640 1 -- 172.21.15.32:0/128318078 learned_addr learned my addr 172.21.15.32:0/128318078 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:11.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.622+0000 7f3822a00640 1 -- 172.21.15.32:0/128318078 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f382410b200 msgr2=0x7f38241a0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:11.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.622+0000 7f3822a00640 1 --2- 172.21.15.32:0/128318078 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f382410b200 0x7f38241a0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:11.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.622+0000 7f3822a00640 1 -- 172.21.15.32:0/128318078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f382410ff00 msgr2=0x7f38241a7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:11.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.622+0000 7f3822a00640 1 --2- 172.21.15.32:0/128318078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f382410ff00 0x7f38241a7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:11.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.622+0000 7f3822a00640 1 -- 172.21.15.32:0/128318078 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38241a8180 con 0x7f3824109fc0 2024-04-30T08:34:11.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.622+0000 7f3822a00640 1 --2- 172.21.15.32:0/128318078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3824109fc0 0x7f38241a0640 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3824109890 tx=0x7f381400ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:11.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.622+0000 7f3828a00640 1 -- 172.21.15.32:0/128318078 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f381400cd00 con 0x7f3824109fc0 2024-04-30T08:34:11.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.623+0000 7f3828a00640 1 -- 172.21.15.32:0/128318078 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f38140108e0 con 0x7f3824109fc0 2024-04-30T08:34:11.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.623+0000 7f3823e00640 1 -- 172.21.15.32:0/128318078 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38241a8470 con 0x7f3824109fc0 2024-04-30T08:34:11.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.623+0000 7f3828a00640 1 -- 172.21.15.32:0/128318078 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38140188c0 con 0x7f3824109fc0 2024-04-30T08:34:11.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.623+0000 7f3823e00640 1 -- 172.21.15.32:0/128318078 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38241a8a10 con 0x7f3824109fc0 2024-04-30T08:34:11.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.624+0000 7f3823e00640 1 -- 172.21.15.32:0/128318078 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f382410ed10 con 0x7f3824109fc0 2024-04-30T08:34:11.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.626+0000 7f3828a00640 1 -- 172.21.15.32:0/128318078 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f381401f020 con 0x7f3824109fc0 2024-04-30T08:34:11.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.628+0000 7f3828a00640 1 --2- 172.21.15.32:0/128318078 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f38080afb60 0x7f38080b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:11.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.628+0000 7f3828a00640 1 -- 172.21.15.32:0/128318078 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f38140defe0 con 0x7f3824109fc0 2024-04-30T08:34:11.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.628+0000 7f3822000640 1 --2- 172.21.15.32:0/128318078 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f38080afb60 0x7f38080b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:11.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.629+0000 7f3822000640 1 --2- 172.21.15.32:0/128318078 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f38080afb60 0x7f38080b2070 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f38241a16e0 tx=0x7f3810005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:11.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.632+0000 7f3828a00640 1 -- 172.21.15.32:0/128318078 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3814053ff0 con 0x7f3824109fc0 2024-04-30T08:34:11.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.844+0000 7f3823e00640 1 -- 172.21.15.32:0/128318078 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3824063820 con 0x7f3824109fc0 2024-04-30T08:34:11.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.845+0000 7f3828a00640 1 -- 172.21.15.32:0/128318078 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) ==== 100+0+11 (secure 0 0 0) 0x7f38140a77f0 con 0x7f3824109fc0 2024-04-30T08:34:11.847 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-04-30T08:34:11.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.848+0000 7f3823e00640 1 -- 172.21.15.32:0/128318078 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f38080afb60 msgr2=0x7f38080b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:11.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.848+0000 7f3823e00640 1 --2- 172.21.15.32:0/128318078 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f38080afb60 0x7f38080b2070 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f38241a16e0 tx=0x7f3810005f90 comp rx=0 tx=0).stop 2024-04-30T08:34:11.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.848+0000 7f3823e00640 1 -- 172.21.15.32:0/128318078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3824109fc0 msgr2=0x7f38241a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:11.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.848+0000 7f3823e00640 1 --2- 172.21.15.32:0/128318078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3824109fc0 0x7f38241a0640 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f3824109890 tx=0x7f381400ee50 comp rx=0 tx=0).stop 2024-04-30T08:34:11.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.848+0000 7f3823e00640 1 -- 172.21.15.32:0/128318078 shutdown_connections 2024-04-30T08:34:11.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.848+0000 7f3823e00640 1 --2- 172.21.15.32:0/128318078 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f38080afb60 0x7f38080b2070 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:11.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.848+0000 7f3823e00640 1 --2- 172.21.15.32:0/128318078 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f382410ff00 0x7f38241a7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:11.850 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.848+0000 7f3823e00640 1 --2- 172.21.15.32:0/128318078 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f382410b200 0x7f38241a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:11.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.848+0000 7f3823e00640 1 --2- 172.21.15.32:0/128318078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3824109fc0 0x7f38241a0640 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:11.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.848+0000 7f3823e00640 1 -- 172.21.15.32:0/128318078 >> 172.21.15.32:0/128318078 conn(0x7f3824078bb0 msgr2=0x7f382410bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:11.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.848+0000 7f3823e00640 1 -- 172.21.15.32:0/128318078 shutdown_connections 2024-04-30T08:34:11.851 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:11.849+0000 7f3823e00640 1 -- 172.21.15.32:0/128318078 wait complete. 2024-04-30T08:34:11.877 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-30T08:34:12.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.171+0000 7f16e5200640 1 -- 172.21.15.32:0/3532630688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16e0075960 msgr2=0x7f16e010fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:12.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.171+0000 7f16e5200640 1 --2- 172.21.15.32:0/3532630688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16e0075960 0x7f16e010fbb0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f16c8009900 tx=0x7f16c802a3f0 comp rx=0 tx=0).stop 2024-04-30T08:34:12.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.172+0000 7f16e5200640 1 -- 172.21.15.32:0/3532630688 shutdown_connections 2024-04-30T08:34:12.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.172+0000 7f16e5200640 1 --2- 172.21.15.32:0/3532630688 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f16e01100f0 0x7f16e0112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:12.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.172+0000 7f16e5200640 1 --2- 172.21.15.32:0/3532630688 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16e0075960 0x7f16e010fbb0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:12.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.172+0000 7f16e5200640 1 --2- 172.21.15.32:0/3532630688 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16e0075010 0x7f16e0075420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:12.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.172+0000 7f16e5200640 1 -- 172.21.15.32:0/3532630688 >> 172.21.15.32:0/3532630688 conn(0x7f16e00feba0 msgr2=0x7f16e0101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:12.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.172+0000 7f16e5200640 1 -- 172.21.15.32:0/3532630688 shutdown_connections 2024-04-30T08:34:12.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.172+0000 7f16e5200640 1 -- 172.21.15.32:0/3532630688 wait complete. 2024-04-30T08:34:12.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.173+0000 7f16e5200640 1 Processor -- start 2024-04-30T08:34:12.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.173+0000 7f16e5200640 1 -- start start 2024-04-30T08:34:12.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.173+0000 7f16e5200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16e0075010 0x7f16e01a0780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:12.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.173+0000 7f16e5200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f16e0075960 0x7f16e01a0cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:12.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.173+0000 7f16e5200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16e01100f0 0x7f16e01a7be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:12.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.174+0000 7f16e5200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f16e0115b20 con 0x7f16e0075010 2024-04-30T08:34:12.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.174+0000 7f16e5200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f16e01159a0 con 0x7f16e0075960 2024-04-30T08:34:12.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.174+0000 7f16e5200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f16e0115ca0 con 0x7f16e01100f0 2024-04-30T08:34:12.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.174+0000 7f16df400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16e0075010 0x7f16e01a0780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:12.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.174+0000 7f16df400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16e0075010 0x7f16e01a0780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45650/0 (socket says 172.21.15.32:45650) 2024-04-30T08:34:12.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.174+0000 7f16df400640 1 -- 172.21.15.32:0/4156424402 learned_addr learned my addr 172.21.15.32:0/4156424402 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:12.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.174+0000 7f16d6a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f16e0075960 0x7f16e01a0cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:12.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.174+0000 7f16dfe00640 1 --2- 172.21.15.32:0/4156424402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16e01100f0 0x7f16e01a7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:12.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.174+0000 7f16df400640 1 -- 172.21.15.32:0/4156424402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16e01100f0 msgr2=0x7f16e01a7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:12.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.174+0000 7f16df400640 1 --2- 172.21.15.32:0/4156424402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16e01100f0 0x7f16e01a7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:12.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.174+0000 7f16df400640 1 -- 172.21.15.32:0/4156424402 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f16e0075960 msgr2=0x7f16e01a0cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:12.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.174+0000 7f16df400640 1 --2- 172.21.15.32:0/4156424402 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f16e0075960 0x7f16e01a0cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:12.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.174+0000 7f16df400640 1 -- 172.21.15.32:0/4156424402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16e01a8250 con 0x7f16e0075010 2024-04-30T08:34:12.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.175+0000 7f16df400640 1 --2- 172.21.15.32:0/4156424402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16e0075010 0x7f16e01a0780 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f16d000e9f0 tx=0x7f16d000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:12.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.175+0000 7f16dcc00640 1 -- 172.21.15.32:0/4156424402 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d000ccc0 con 0x7f16e0075010 2024-04-30T08:34:12.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.175+0000 7f16e5200640 1 -- 172.21.15.32:0/4156424402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16e01a8540 con 0x7f16e0075010 2024-04-30T08:34:12.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.175+0000 7f16dcc00640 1 -- 172.21.15.32:0/4156424402 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f16d00054d0 con 0x7f16e0075010 2024-04-30T08:34:12.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.175+0000 7f16dcc00640 1 -- 172.21.15.32:0/4156424402 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16d0010430 con 0x7f16e0075010 2024-04-30T08:34:12.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.175+0000 7f16e5200640 1 -- 172.21.15.32:0/4156424402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16e01a8a10 con 0x7f16e0075010 2024-04-30T08:34:12.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.176+0000 7f16e5200640 1 -- 172.21.15.32:0/4156424402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16e0105d80 con 0x7f16e0075010 2024-04-30T08:34:12.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.179+0000 7f16dcc00640 1 -- 172.21.15.32:0/4156424402 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f16d00046c0 con 0x7f16e0075010 2024-04-30T08:34:12.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.180+0000 7f16dcc00640 1 --2- 172.21.15.32:0/4156424402 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f16c40afae0 0x7f16c40b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:12.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.180+0000 7f16dcc00640 1 -- 172.21.15.32:0/4156424402 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f16d00df570 con 0x7f16e0075010 2024-04-30T08:34:12.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.180+0000 7f16d6a00640 1 --2- 172.21.15.32:0/4156424402 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f16c40afae0 0x7f16c40b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:12.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.181+0000 7f16d6a00640 1 --2- 172.21.15.32:0/4156424402 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f16c40afae0 0x7f16c40b1ff0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f16c8004770 tx=0x7f16c80314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:12.184 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.183+0000 7f16dcc00640 1 -- 172.21.15.32:0/4156424402 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f16d0053ff0 con 0x7f16e0075010 2024-04-30T08:34:12.190 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:11 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3082541752' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-30T08:34:12.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:11 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/128318078' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-30T08:34:12.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:11 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3082541752' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-30T08:34:12.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:11 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/128318078' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-30T08:34:12.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:11 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3082541752' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-30T08:34:12.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:11 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/128318078' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-30T08:34:12.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.392+0000 7f16e5200640 1 -- 172.21.15.32:0/4156424402 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f16e0063820 con 0x7f16e0075010 2024-04-30T08:34:12.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.393+0000 7f16dcc00640 1 -- 172.21.15.32:0/4156424402 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v85) ==== 103+0+11 (secure 0 0 0) 0x7f16d00a7d40 con 0x7f16e0075010 2024-04-30T08:34:12.394 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-30T08:34:12.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.395+0000 7f16e5200640 1 -- 172.21.15.32:0/4156424402 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f16c40afae0 msgr2=0x7f16c40b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:12.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.395+0000 7f16e5200640 1 --2- 172.21.15.32:0/4156424402 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f16c40afae0 0x7f16c40b1ff0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f16c8004770 tx=0x7f16c80314a0 comp rx=0 tx=0).stop 2024-04-30T08:34:12.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.395+0000 7f16e5200640 1 -- 172.21.15.32:0/4156424402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16e0075010 msgr2=0x7f16e01a0780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:12.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.395+0000 7f16e5200640 1 --2- 172.21.15.32:0/4156424402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16e0075010 0x7f16e01a0780 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f16d000e9f0 tx=0x7f16d000eeb0 comp rx=0 tx=0).stop 2024-04-30T08:34:12.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.396+0000 7f16e5200640 1 -- 172.21.15.32:0/4156424402 shutdown_connections 2024-04-30T08:34:12.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.396+0000 7f16e5200640 1 --2- 172.21.15.32:0/4156424402 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f16c40afae0 0x7f16c40b1ff0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:12.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.396+0000 7f16e5200640 1 --2- 172.21.15.32:0/4156424402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f16e01100f0 0x7f16e01a7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:12.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.396+0000 7f16e5200640 1 --2- 172.21.15.32:0/4156424402 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f16e0075960 0x7f16e01a0cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:12.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.396+0000 7f16e5200640 1 --2- 172.21.15.32:0/4156424402 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f16e0075010 0x7f16e01a0780 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:12.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.396+0000 7f16e5200640 1 -- 172.21.15.32:0/4156424402 >> 172.21.15.32:0/4156424402 conn(0x7f16e00feba0 msgr2=0x7f16e0102f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:12.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.396+0000 7f16e5200640 1 -- 172.21.15.32:0/4156424402 shutdown_connections 2024-04-30T08:34:12.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.396+0000 7f16e5200640 1 -- 172.21.15.32:0/4156424402 wait complete. 2024-04-30T08:34:12.423 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-30T08:34:12.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.706+0000 7f079b200640 1 -- 172.21.15.32:0/2512198339 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f079c109610 msgr2=0x7f079c10bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:12.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.706+0000 7f079b200640 1 --2- 172.21.15.32:0/2512198339 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f079c109610 0x7f079c10bb20 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f0794009900 tx=0x7f079402a410 comp rx=0 tx=0).stop 2024-04-30T08:34:12.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.706+0000 7f079b200640 1 -- 172.21.15.32:0/2512198339 shutdown_connections 2024-04-30T08:34:12.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.706+0000 7f079b200640 1 --2- 172.21.15.32:0/2512198339 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f079c109610 0x7f079c10bb20 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:12.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.706+0000 7f079b200640 1 --2- 172.21.15.32:0/2512198339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f079c106ae0 0x7f079c108f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:12.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.706+0000 7f079b200640 1 --2- 172.21.15.32:0/2512198339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f079c1040f0 0x7f079c106510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:12.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.706+0000 7f079b200640 1 -- 172.21.15.32:0/2512198339 >> 172.21.15.32:0/2512198339 conn(0x7f079c0fda70 msgr2=0x7f079c0fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:12.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.706+0000 7f079b200640 1 -- 172.21.15.32:0/2512198339 shutdown_connections 2024-04-30T08:34:12.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.707+0000 7f079b200640 1 -- 172.21.15.32:0/2512198339 wait complete. 2024-04-30T08:34:12.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.708+0000 7f079b200640 1 Processor -- start 2024-04-30T08:34:12.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.708+0000 7f079b200640 1 -- start start 2024-04-30T08:34:12.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.708+0000 7f079b200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f079c1040f0 0x7f079c103960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:12.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.708+0000 7f079b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f079c106ae0 0x7f079c102000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:12.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.708+0000 7f079b200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f079c109610 0x7f079c102540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:12.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.708+0000 7f079b200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f079c10f4a0 con 0x7f079c106ae0 2024-04-30T08:34:12.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.708+0000 7f079b200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f079c10f320 con 0x7f079c1040f0 2024-04-30T08:34:12.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.708+0000 7f079b200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f079c10f620 con 0x7f079c109610 2024-04-30T08:34:12.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.709+0000 7f0799400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f079c106ae0 0x7f079c102000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:12.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.709+0000 7f0799e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f079c1040f0 0x7f079c103960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:12.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.709+0000 7f079a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f079c109610 0x7f079c102540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:12.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.709+0000 7f0799400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f079c106ae0 0x7f079c102000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45668/0 (socket says 172.21.15.32:45668) 2024-04-30T08:34:12.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.709+0000 7f079a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f079c109610 0x7f079c102540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:60380/0 (socket says 172.21.15.32:60380) 2024-04-30T08:34:12.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.709+0000 7f0799400640 1 -- 172.21.15.32:0/1673851406 learned_addr learned my addr 172.21.15.32:0/1673851406 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:12.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.709+0000 7f0799e00640 1 --2- 172.21.15.32:0/1673851406 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f079c1040f0 0x7f079c103960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:47332/0 (socket says 172.21.15.32:47332) 2024-04-30T08:34:12.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.709+0000 7f0799400640 1 -- 172.21.15.32:0/1673851406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f079c109610 msgr2=0x7f079c102540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:12.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.709+0000 7f0799400640 1 --2- 172.21.15.32:0/1673851406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f079c109610 0x7f079c102540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:12.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.709+0000 7f0799400640 1 -- 172.21.15.32:0/1673851406 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f079c1040f0 msgr2=0x7f079c103960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:12.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.709+0000 7f0799400640 1 --2- 172.21.15.32:0/1673851406 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f079c1040f0 0x7f079c103960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:12.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.709+0000 7f0799400640 1 -- 172.21.15.32:0/1673851406 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f079c102c70 con 0x7f079c106ae0 2024-04-30T08:34:12.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.710+0000 7f0799400640 1 --2- 172.21.15.32:0/1673851406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f079c106ae0 0x7f079c102000 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f078800e9f0 tx=0x7f078800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:12.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.710+0000 7f0792a00640 1 -- 172.21.15.32:0/1673851406 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078800cd80 con 0x7f079c106ae0 2024-04-30T08:34:12.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.710+0000 7f079b200640 1 -- 172.21.15.32:0/1673851406 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f079c1a4110 con 0x7f079c106ae0 2024-04-30T08:34:12.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.710+0000 7f0792a00640 1 -- 172.21.15.32:0/1673851406 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f07880108e0 con 0x7f079c106ae0 2024-04-30T08:34:12.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.710+0000 7f079b200640 1 -- 172.21.15.32:0/1673851406 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f079c1a45b0 con 0x7f079c106ae0 2024-04-30T08:34:12.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.710+0000 7f0792a00640 1 -- 172.21.15.32:0/1673851406 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0788026060 con 0x7f079c106ae0 2024-04-30T08:34:12.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.711+0000 7f079b200640 1 -- 172.21.15.32:0/1673851406 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f079c069c60 con 0x7f079c106ae0 2024-04-30T08:34:12.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.715+0000 7f0792a00640 1 -- 172.21.15.32:0/1673851406 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f0788010430 con 0x7f079c106ae0 2024-04-30T08:34:12.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.716+0000 7f0792a00640 1 --2- 172.21.15.32:0/1673851406 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f07800afae0 0x7f07800b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:12.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.716+0000 7f0792a00640 1 -- 172.21.15.32:0/1673851406 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f07880dfa80 con 0x7f079c106ae0 2024-04-30T08:34:12.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.716+0000 7f0799e00640 1 --2- 172.21.15.32:0/1673851406 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f07800afae0 0x7f07800b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:12.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.718+0000 7f0799e00640 1 --2- 172.21.15.32:0/1673851406 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f07800afae0 0x7f07800b1ff0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f079c06a890 tx=0x7f078c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:12.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.718+0000 7f0792a00640 1 -- 172.21.15.32:0/1673851406 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f07880a7c00 con 0x7f079c106ae0 2024-04-30T08:34:12.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.970+0000 7f079b200640 1 -- 172.21.15.32:0/1673851406 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f079c02df80 con 0x7f079c106ae0 2024-04-30T08:34:12.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.972+0000 7f0792a00640 1 -- 172.21.15.32:0/1673851406 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7f079c02df80 con 0x7f079c106ae0 2024-04-30T08:34:12.974 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:34:12.974 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2235842676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2235842676},{"type":"v1","addr":"172.21.15.32:6835","nonce":2235842676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/3577009355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":3577009355},{"type":"v1","addr":"172.21.15.32:6841","nonce":3577009355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-30T08:31:52.236668+0000","modified":"2024-04-30T08:32:30.541714+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24541,"mds_3":24533,"mds_4":24517},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/463203038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":463203038},{"type":"v1","addr":"172.21.15.189:6833","nonce":463203038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6833/3635143591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3635143591},{"type":"v1","addr":"172.21.15.71:6833","nonce":3635143591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6837/2569449994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2569449994},{"type":"v1","addr":"172.21.15.32:6837","nonce":2569449994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/278686565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":278686565},{"type":"v1","addr":"172.21.15.189:6835","nonce":278686565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6835/1558303056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1558303056},{"type":"v1","addr":"172.21.15.71:6835","nonce":1558303056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.189:6837/686520873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":686520873},{"type":"v1","addr":"172.21.15.189:6837","nonce":686520873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.71:6837/2324039932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2324039932},{"type":"v1","addr":"172.21.15.71:6837","nonce":2324039932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6839/352970107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":352970107},{"type":"v1","addr":"172.21.15.189:6839","nonce":352970107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.32:6839/4160386524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4160386524},{"type":"v1","addr":"172.21.15.32:6839","nonce":4160386524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.71:6839/3517409993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3517409993},{"type":"v1","addr":"172.21.15.71:6839","nonce":3517409993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24590,24541,24560,24533,24517]},"id":1}]} 2024-04-30T08:34:12.974 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 27 2024-04-30T08:34:12.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.974+0000 7f079b200640 1 -- 172.21.15.32:0/1673851406 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f07800afae0 msgr2=0x7f07800b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:12.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.974+0000 7f079b200640 1 --2- 172.21.15.32:0/1673851406 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f07800afae0 0x7f07800b1ff0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f079c06a890 tx=0x7f078c009500 comp rx=0 tx=0).stop 2024-04-30T08:34:12.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.974+0000 7f079b200640 1 -- 172.21.15.32:0/1673851406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f079c106ae0 msgr2=0x7f079c102000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:12.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.974+0000 7f079b200640 1 --2- 172.21.15.32:0/1673851406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f079c106ae0 0x7f079c102000 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f078800e9f0 tx=0x7f078800eeb0 comp rx=0 tx=0).stop 2024-04-30T08:34:12.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.975+0000 7f079b200640 1 -- 172.21.15.32:0/1673851406 shutdown_connections 2024-04-30T08:34:12.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.975+0000 7f079b200640 1 --2- 172.21.15.32:0/1673851406 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f07800afae0 0x7f07800b1ff0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:12.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.975+0000 7f079b200640 1 --2- 172.21.15.32:0/1673851406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f079c109610 0x7f079c102540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:12.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.975+0000 7f079b200640 1 --2- 172.21.15.32:0/1673851406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f079c106ae0 0x7f079c102000 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:12.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.975+0000 7f079b200640 1 --2- 172.21.15.32:0/1673851406 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f079c1040f0 0x7f079c103960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:12.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.975+0000 7f079b200640 1 -- 172.21.15.32:0/1673851406 >> 172.21.15.32:0/1673851406 conn(0x7f079c0fda70 msgr2=0x7f079c104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:12.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.975+0000 7f079b200640 1 -- 172.21.15.32:0/1673851406 shutdown_connections 2024-04-30T08:34:12.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:12.975+0000 7f079b200640 1 -- 172.21.15.32:0/1673851406 wait complete. 2024-04-30T08:34:13.008 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-30T08:34:13.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:12 smithi032 ceph-mon[28331]: pgmap v396: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-30T08:34:13.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:12 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/4156424402' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-30T08:34:13.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:12 smithi071 ceph-mon[33385]: pgmap v396: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-30T08:34:13.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:12 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/4156424402' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-30T08:34:13.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.295+0000 7f76e0600640 1 -- 172.21.15.32:0/242335488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76d8103b80 msgr2=0x7f76d8104880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:13.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.295+0000 7f76e0600640 1 --2- 172.21.15.32:0/242335488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76d8103b80 0x7f76d8104880 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f76cc009900 tx=0x7f76cc02a3f0 comp rx=0 tx=0).stop 2024-04-30T08:34:13.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.295+0000 7f76e0600640 1 -- 172.21.15.32:0/242335488 shutdown_connections 2024-04-30T08:34:13.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.295+0000 7f76e0600640 1 --2- 172.21.15.32:0/242335488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76d8109ac0 0x7f76d810bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:13.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.295+0000 7f76e0600640 1 --2- 172.21.15.32:0/242335488 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76d8104dc0 0x7f76d81051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:13.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.295+0000 7f76e0600640 1 --2- 172.21.15.32:0/242335488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76d8103b80 0x7f76d8104880 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:13.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.295+0000 7f76e0600640 1 -- 172.21.15.32:0/242335488 >> 172.21.15.32:0/242335488 conn(0x7f76d80fed90 msgr2=0x7f76d8101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:13.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.295+0000 7f76e0600640 1 -- 172.21.15.32:0/242335488 shutdown_connections 2024-04-30T08:34:13.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.295+0000 7f76e0600640 1 -- 172.21.15.32:0/242335488 wait complete. 2024-04-30T08:34:13.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.296+0000 7f76e0600640 1 Processor -- start 2024-04-30T08:34:13.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.297+0000 7f76e0600640 1 -- start start 2024-04-30T08:34:13.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.297+0000 7f76e0600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76d8103b80 0x7f76d819c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:13.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.297+0000 7f76e0600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76d8104dc0 0x7f76d819ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:13.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.297+0000 7f76e0600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76d8109ac0 0x7f76d81a3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:13.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.297+0000 7f76e0600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f76d810f6b0 con 0x7f76d8104dc0 2024-04-30T08:34:13.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.297+0000 7f76e0600640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f76d810f530 con 0x7f76d8103b80 2024-04-30T08:34:13.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.297+0000 7f76e0600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f76d810f830 con 0x7f76d8109ac0 2024-04-30T08:34:13.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.297+0000 7f76ddc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76d8103b80 0x7f76d819c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:13.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.297+0000 7f76dd200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76d8104dc0 0x7f76d819ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:13.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.297+0000 7f76ddc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76d8103b80 0x7f76d819c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:47358/0 (socket says 172.21.15.32:47358) 2024-04-30T08:34:13.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.297+0000 7f76ddc00640 1 -- 172.21.15.32:0/1479793854 learned_addr learned my addr 172.21.15.32:0/1479793854 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:13.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.297+0000 7f76de600640 1 --2- 172.21.15.32:0/1479793854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76d8109ac0 0x7f76d81a3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:13.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.298+0000 7f76de600640 1 -- 172.21.15.32:0/1479793854 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76d8103b80 msgr2=0x7f76d819c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:13.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.298+0000 7f76de600640 1 --2- 172.21.15.32:0/1479793854 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76d8103b80 0x7f76d819c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:13.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.298+0000 7f76de600640 1 -- 172.21.15.32:0/1479793854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76d8104dc0 msgr2=0x7f76d819ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:13.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.298+0000 7f76de600640 1 --2- 172.21.15.32:0/1479793854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76d8104dc0 0x7f76d819ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:13.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.298+0000 7f76de600640 1 -- 172.21.15.32:0/1479793854 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76d81a4000 con 0x7f76d8109ac0 2024-04-30T08:34:13.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.298+0000 7f76dd200640 1 --2- 172.21.15.32:0/1479793854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76d8104dc0 0x7f76d819ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:34:13.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.298+0000 7f76de600640 1 --2- 172.21.15.32:0/1479793854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76d8109ac0 0x7f76d81a3990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f76d0009fd0 tx=0x7f76d000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:13.302 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:12 smithi189 ceph-mon[33038]: pgmap v396: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-30T08:34:13.302 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:12 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/4156424402' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-30T08:34:13.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.299+0000 7f76d6000640 1 -- 172.21.15.32:0/1479793854 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d000ce20 con 0x7f76d8109ac0 2024-04-30T08:34:13.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.299+0000 7f76d6000640 1 -- 172.21.15.32:0/1479793854 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f76d0005020 con 0x7f76d8109ac0 2024-04-30T08:34:13.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.299+0000 7f76d6000640 1 -- 172.21.15.32:0/1479793854 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d0010690 con 0x7f76d8109ac0 2024-04-30T08:34:13.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.299+0000 7f76e0600640 1 -- 172.21.15.32:0/1479793854 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76d81a42f0 con 0x7f76d8109ac0 2024-04-30T08:34:13.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.299+0000 7f76e0600640 1 -- 172.21.15.32:0/1479793854 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76d81a4840 con 0x7f76d8109ac0 2024-04-30T08:34:13.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.300+0000 7f76e0600640 1 -- 172.21.15.32:0/1479793854 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76d81a4b00 con 0x7f76d8109ac0 2024-04-30T08:34:13.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.303+0000 7f76d6000640 1 -- 172.21.15.32:0/1479793854 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f76d00054d0 con 0x7f76d8109ac0 2024-04-30T08:34:13.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.305+0000 7f76d6000640 1 --2- 172.21.15.32:0/1479793854 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f76c00afb70 0x7f76c00b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:13.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.305+0000 7f76d6000640 1 -- 172.21.15.32:0/1479793854 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f76d00df650 con 0x7f76d8109ac0 2024-04-30T08:34:13.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.305+0000 7f76ddc00640 1 --2- 172.21.15.32:0/1479793854 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f76c00afb70 0x7f76c00b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:13.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.306+0000 7f76ddc00640 1 --2- 172.21.15.32:0/1479793854 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f76c00afb70 0x7f76c00b2080 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f76cc000c00 tx=0x7f76cc0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:13.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.310+0000 7f76d6000640 1 -- 172.21.15.32:0/1479793854 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f76d00a7e60 con 0x7f76d8109ac0 2024-04-30T08:34:13.519 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.518+0000 7f76e0600640 1 -- 172.21.15.32:0/1479793854 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f76d81a4cd0 con 0x7f76d8109ac0 2024-04-30T08:34:13.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.520+0000 7f76d6000640 1 -- 172.21.15.32:0/1479793854 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+24730 (secure 0 0 0) 0x7f76d00a7760 con 0x7f76d8109ac0 2024-04-30T08:34:13.523 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:34:13.523 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":85,"fsid":"98c0a670-06ca-11ef-bc93-c7b262605968","created":"2024-04-30T08:22:11.122875+0000","modified":"2024-04-30T08:33:31.872704+0000","last_up_change":"2024-04-30T08:30:50.350439+0000","last_in_change":"2024-04-30T08:30:30.210620+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-30T08:26:19.728331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-30T08:31:49.800542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-30T08:31:51.211320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-30T08:31:55.716625+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a7e060c8-1cbb-418f-9a03-06693f6b3f52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6803","nonce":2624700402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6805","nonce":2624700402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6809","nonce":2624700402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6807","nonce":2624700402}]},"public_addr":"172.21.15.32:6803/2624700402","cluster_addr":"172.21.15.32:6805/2624700402","heartbeat_back_addr":"172.21.15.32:6809/2624700402","heartbeat_front_addr":"172.21.15.32:6807/2624700402","state":["exists","up"]},{"osd":1,"uuid":"72ef2fb1-e5ca-48ef-aa3a-4796a64b922f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6801","nonce":444649711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6803","nonce":444649711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6807","nonce":444649711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6805","nonce":444649711}]},"public_addr":"172.21.15.71:6801/444649711","cluster_addr":"172.21.15.71:6803/444649711","heartbeat_back_addr":"172.21.15.71:6807/444649711","heartbeat_front_addr":"172.21.15.71:6805/444649711","state":["exists","up"]},{"osd":2,"uuid":"704e5e8a-935e-47c7-ae28-27119876ac09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6801","nonce":3255528671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6803","nonce":3255528671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6807","nonce":3255528671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6805","nonce":3255528671}]},"public_addr":"172.21.15.189:6801/3255528671","cluster_addr":"172.21.15.189:6803/3255528671","heartbeat_back_addr":"172.21.15.189:6807/3255528671","heartbeat_front_addr":"172.21.15.189:6805/3255528671","state":["exists","up"]},{"osd":3,"uuid":"575004f7-6559-4a62-863e-e33078b0eebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6811","nonce":1384702112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6813","nonce":1384702112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6817","nonce":1384702112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6815","nonce":1384702112}]},"public_addr":"172.21.15.32:6811/1384702112","cluster_addr":"172.21.15.32:6813/1384702112","heartbeat_back_addr":"172.21.15.32:6817/1384702112","heartbeat_front_addr":"172.21.15.32:6815/1384702112","state":["exists","up"]},{"osd":4,"uuid":"e1c1e894-074d-4c78-89ef-9e5a114268bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6809","nonce":4250534574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6811","nonce":4250534574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6815","nonce":4250534574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6813","nonce":4250534574}]},"public_addr":"172.21.15.71:6809/4250534574","cluster_addr":"172.21.15.71:6811/4250534574","heartbeat_back_addr":"172.21.15.71:6815/4250534574","heartbeat_front_addr":"172.21.15.71:6813/4250534574","state":["exists","up"]},{"osd":5,"uuid":"12c022d4-fac6-40c4-ae7a-691571bee5d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6809","nonce":4245346198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6811","nonce":4245346198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6815","nonce":4245346198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6813","nonce":4245346198}]},"public_addr":"172.21.15.189:6809/4245346198","cluster_addr":"172.21.15.189:6811/4245346198","heartbeat_back_addr":"172.21.15.189:6815/4245346198","heartbeat_front_addr":"172.21.15.189:6813/4245346198","state":["exists","up"]},{"osd":6,"uuid":"b413f8ee-3d4a-415a-a2ba-d68437b7330d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6819","nonce":1910389190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6821","nonce":1910389190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6825","nonce":1910389190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6823","nonce":1910389190}]},"public_addr":"172.21.15.32:6819/1910389190","cluster_addr":"172.21.15.32:6821/1910389190","heartbeat_back_addr":"172.21.15.32:6825/1910389190","heartbeat_front_addr":"172.21.15.32:6823/1910389190","state":["exists","up"]},{"osd":7,"uuid":"aac64fb3-fc13-4126-b842-7b06ad5ce238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6817","nonce":692607380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6819","nonce":692607380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6823","nonce":692607380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6821","nonce":692607380}]},"public_addr":"172.21.15.71:6817/692607380","cluster_addr":"172.21.15.71:6819/692607380","heartbeat_back_addr":"172.21.15.71:6823/692607380","heartbeat_front_addr":"172.21.15.71:6821/692607380","state":["exists","up"]},{"osd":8,"uuid":"cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6817","nonce":3973697069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6819","nonce":3973697069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6823","nonce":3973697069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6821","nonce":3973697069}]},"public_addr":"172.21.15.189:6817/3973697069","cluster_addr":"172.21.15.189:6819/3973697069","heartbeat_back_addr":"172.21.15.189:6823/3973697069","heartbeat_front_addr":"172.21.15.189:6821/3973697069","state":["exists","up"]},{"osd":9,"uuid":"ef4a1645-ae6e-45c3-82ad-161510e62aa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6827","nonce":46066685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6829","nonce":46066685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6833","nonce":46066685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6831","nonce":46066685}]},"public_addr":"172.21.15.32:6827/46066685","cluster_addr":"172.21.15.32:6829/46066685","heartbeat_back_addr":"172.21.15.32:6833/46066685","heartbeat_front_addr":"172.21.15.32:6831/46066685","state":["exists","up"]},{"osd":10,"uuid":"25b26a38-a644-409f-b941-59efdf04a3ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6825","nonce":1886290293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6827","nonce":1886290293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6831","nonce":1886290293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6829","nonce":1886290293}]},"public_addr":"172.21.15.71:6825/1886290293","cluster_addr":"172.21.15.71:6827/1886290293","heartbeat_back_addr":"172.21.15.71:6831/1886290293","heartbeat_front_addr":"172.21.15.71:6829/1886290293","state":["exists","up"]},{"osd":11,"uuid":"ab96b3ff-3688-455b-af64-60226c5bfbff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6825","nonce":850185001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6827","nonce":850185001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6831","nonce":850185001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6829","nonce":850185001}]},"public_addr":"172.21.15.189:6825/850185001","cluster_addr":"172.21.15.189:6827/850185001","heartbeat_back_addr":"172.21.15.189:6831/850185001","heartbeat_front_addr":"172.21.15.189:6829/850185001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:27.220308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:53.230783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:17.190642+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:52.603799+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:20.568671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:46.158363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:22.706933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:50.665173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:16.602410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:53.814698+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:22.435189+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:48.651949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.e","mappings":[{"from":4,"to":7}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.28","mappings":[{"from":3,"to":6},{"from":5,"to":8}]},{"pgid":"3.30","mappings":[{"from":1,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1266897222":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/2915689533":"2024-05-01T08:23:29.526664+0000","172.21.15.32:6801/4019351306":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/425836193":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/1134075722":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6801/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/21059253":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/3245550894":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/4019351306":"2024-05-01T08:23:29.526664+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-30T08:34:13.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.523+0000 7f76e0600640 1 -- 172.21.15.32:0/1479793854 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f76c00afb70 msgr2=0x7f76c00b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:13.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.523+0000 7f76e0600640 1 --2- 172.21.15.32:0/1479793854 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f76c00afb70 0x7f76c00b2080 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f76cc000c00 tx=0x7f76cc0314a0 comp rx=0 tx=0).stop 2024-04-30T08:34:13.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.523+0000 7f76e0600640 1 -- 172.21.15.32:0/1479793854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76d8109ac0 msgr2=0x7f76d81a3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:13.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.523+0000 7f76e0600640 1 --2- 172.21.15.32:0/1479793854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76d8109ac0 0x7f76d81a3990 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f76d0009fd0 tx=0x7f76d000ef10 comp rx=0 tx=0).stop 2024-04-30T08:34:13.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.523+0000 7f76e0600640 1 -- 172.21.15.32:0/1479793854 shutdown_connections 2024-04-30T08:34:13.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.523+0000 7f76e0600640 1 --2- 172.21.15.32:0/1479793854 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f76c00afb70 0x7f76c00b2080 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:13.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.523+0000 7f76e0600640 1 --2- 172.21.15.32:0/1479793854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f76d8109ac0 0x7f76d81a3990 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:13.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.523+0000 7f76e0600640 1 --2- 172.21.15.32:0/1479793854 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f76d8104dc0 0x7f76d819ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:13.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.523+0000 7f76e0600640 1 --2- 172.21.15.32:0/1479793854 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f76d8103b80 0x7f76d819c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:13.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.523+0000 7f76e0600640 1 -- 172.21.15.32:0/1479793854 >> 172.21.15.32:0/1479793854 conn(0x7f76d80fed90 msgr2=0x7f76d80ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:13.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.523+0000 7f76e0600640 1 -- 172.21.15.32:0/1479793854 shutdown_connections 2024-04-30T08:34:13.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.523+0000 7f76e0600640 1 -- 172.21.15.32:0/1479793854 wait complete. 2024-04-30T08:34:13.552 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-30T08:26:19.728331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-30T08:31:49.800542+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-30T08:31:51.211320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-30T08:31:55.716625+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-30T08:34:13.553 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-30T08:34:13.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.853+0000 7f079fe00640 1 -- 172.21.15.32:0/2869782737 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07a0075900 msgr2=0x7f07a010b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:13.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.853+0000 7f079fe00640 1 --2- 172.21.15.32:0/2869782737 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07a0075900 0x7f07a010b420 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f078c009900 tx=0x7f078c02a3f0 comp rx=0 tx=0).stop 2024-04-30T08:34:13.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.854+0000 7f079fe00640 1 -- 172.21.15.32:0/2869782737 shutdown_connections 2024-04-30T08:34:13.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.854+0000 7f079fe00640 1 --2- 172.21.15.32:0/2869782737 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07a010b960 0x7f07a010dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:13.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.854+0000 7f079fe00640 1 --2- 172.21.15.32:0/2869782737 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07a0075900 0x7f07a010b420 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:13.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.854+0000 7f079fe00640 1 --2- 172.21.15.32:0/2869782737 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f07a0074fb0 0x7f07a00753c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:13.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.854+0000 7f079fe00640 1 -- 172.21.15.32:0/2869782737 >> 172.21.15.32:0/2869782737 conn(0x7f07a00feba0 msgr2=0x7f07a0101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:13.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.854+0000 7f079fe00640 1 -- 172.21.15.32:0/2869782737 shutdown_connections 2024-04-30T08:34:13.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.854+0000 7f079fe00640 1 -- 172.21.15.32:0/2869782737 wait complete. 2024-04-30T08:34:13.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.855+0000 7f079fe00640 1 Processor -- start 2024-04-30T08:34:13.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.855+0000 7f079fe00640 1 -- start start 2024-04-30T08:34:13.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.855+0000 7f079fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07a0074fb0 0x7f07a019c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:13.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.855+0000 7f079fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07a0075900 0x7f07a019c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:13.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.855+0000 7f079fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f07a010b960 0x7f07a01a3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:13.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.855+0000 7f079fe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f07a0111700 con 0x7f07a0075900 2024-04-30T08:34:13.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.855+0000 7f079fe00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f07a0111580 con 0x7f07a010b960 2024-04-30T08:34:13.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.855+0000 7f079fe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f07a0111880 con 0x7f07a0074fb0 2024-04-30T08:34:13.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.856+0000 7f079e000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07a0075900 0x7f07a019c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:13.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.856+0000 7f079ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07a0074fb0 0x7f07a019c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:13.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.856+0000 7f079f400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f07a010b960 0x7f07a01a3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:13.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.856+0000 7f079e000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07a0075900 0x7f07a019c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45696/0 (socket says 172.21.15.32:45696) 2024-04-30T08:34:13.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.856+0000 7f079e000640 1 -- 172.21.15.32:0/615360310 learned_addr learned my addr 172.21.15.32:0/615360310 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:13.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.856+0000 7f079ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07a0074fb0 0x7f07a019c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:60434/0 (socket says 172.21.15.32:60434) 2024-04-30T08:34:13.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.856+0000 7f079f400640 1 -- 172.21.15.32:0/615360310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07a0074fb0 msgr2=0x7f07a019c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:13.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.856+0000 7f079f400640 1 --2- 172.21.15.32:0/615360310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07a0074fb0 0x7f07a019c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:13.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.856+0000 7f079f400640 1 -- 172.21.15.32:0/615360310 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07a0075900 msgr2=0x7f07a019c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:13.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.856+0000 7f079f400640 1 --2- 172.21.15.32:0/615360310 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07a0075900 0x7f07a019c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:13.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.856+0000 7f079f400640 1 -- 172.21.15.32:0/615360310 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07a01a3df0 con 0x7f07a010b960 2024-04-30T08:34:13.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.856+0000 7f079e000640 1 --2- 172.21.15.32:0/615360310 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07a0075900 0x7f07a019c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:34:13.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.857+0000 7f079f400640 1 --2- 172.21.15.32:0/615360310 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f07a010b960 0x7f07a01a3780 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0798009fd0 tx=0x7f079800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:13.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.857+0000 7f0797400640 1 -- 172.21.15.32:0/615360310 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f079800ce20 con 0x7f07a010b960 2024-04-30T08:34:13.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.857+0000 7f0797400640 1 -- 172.21.15.32:0/615360310 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0798005020 con 0x7f07a010b960 2024-04-30T08:34:13.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.857+0000 7f0797400640 1 -- 172.21.15.32:0/615360310 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0798010690 con 0x7f07a010b960 2024-04-30T08:34:13.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.857+0000 7f079fe00640 1 -- 172.21.15.32:0/615360310 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07a01a40e0 con 0x7f07a010b960 2024-04-30T08:34:13.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.857+0000 7f079fe00640 1 -- 172.21.15.32:0/615360310 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07a01a4680 con 0x7f07a010b960 2024-04-30T08:34:13.861 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:13 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1673851406' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:34:13.861 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:13 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1479793854' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:34:13.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.859+0000 7f079fe00640 1 -- 172.21.15.32:0/615360310 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f07a010ab20 con 0x7f07a010b960 2024-04-30T08:34:13.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.861+0000 7f0797400640 1 -- 172.21.15.32:0/615360310 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f0798004b20 con 0x7f07a010b960 2024-04-30T08:34:13.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.862+0000 7f0797400640 1 --2- 172.21.15.32:0/615360310 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f07840afb60 0x7f07840b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:13.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.863+0000 7f0797400640 1 -- 172.21.15.32:0/615360310 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f07980e13b0 con 0x7f07a010b960 2024-04-30T08:34:13.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.863+0000 7f079ea00640 1 --2- 172.21.15.32:0/615360310 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f07840afb60 0x7f07840b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:13.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.864+0000 7f079ea00640 1 --2- 172.21.15.32:0/615360310 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f07840afb60 0x7f07840b2070 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f07900095f0 tx=0x7f0790005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:13.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:13.867+0000 7f0797400640 1 -- 172.21.15.32:0/615360310 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f07980a7e00 con 0x7f07a010b960 2024-04-30T08:34:14.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.078+0000 7f079fe00640 1 -- 172.21.15.32:0/615360310 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f07a0063820 con 0x7f07a010b960 2024-04-30T08:34:14.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.079+0000 7f0797400640 1 -- 172.21.15.32:0/615360310 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) ==== 93+0+10 (secure 0 0 0) 0x7f07980a7700 con 0x7f07a010b960 2024-04-30T08:34:14.081 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-30T08:34:14.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.082+0000 7f079fe00640 1 -- 172.21.15.32:0/615360310 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f07840afb60 msgr2=0x7f07840b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:14.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.082+0000 7f079fe00640 1 --2- 172.21.15.32:0/615360310 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f07840afb60 0x7f07840b2070 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f07900095f0 tx=0x7f0790005f90 comp rx=0 tx=0).stop 2024-04-30T08:34:14.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.082+0000 7f079fe00640 1 -- 172.21.15.32:0/615360310 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f07a010b960 msgr2=0x7f07a01a3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:14.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.082+0000 7f079fe00640 1 --2- 172.21.15.32:0/615360310 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f07a010b960 0x7f07a01a3780 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0798009fd0 tx=0x7f079800ef10 comp rx=0 tx=0).stop 2024-04-30T08:34:14.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.082+0000 7f079fe00640 1 -- 172.21.15.32:0/615360310 shutdown_connections 2024-04-30T08:34:14.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.082+0000 7f079fe00640 1 --2- 172.21.15.32:0/615360310 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f07840afb60 0x7f07840b2070 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:14.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.082+0000 7f079fe00640 1 --2- 172.21.15.32:0/615360310 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f07a010b960 0x7f07a01a3780 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:14.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.082+0000 7f079fe00640 1 --2- 172.21.15.32:0/615360310 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f07a0075900 0x7f07a019c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:14.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.082+0000 7f079fe00640 1 --2- 172.21.15.32:0/615360310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f07a0074fb0 0x7f07a019c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:14.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.082+0000 7f079fe00640 1 -- 172.21.15.32:0/615360310 >> 172.21.15.32:0/615360310 conn(0x7f07a00feba0 msgr2=0x7f07a00ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:14.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.083+0000 7f079fe00640 1 -- 172.21.15.32:0/615360310 shutdown_connections 2024-04-30T08:34:14.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.083+0000 7f079fe00640 1 -- 172.21.15.32:0/615360310 wait complete. 2024-04-30T08:34:14.112 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-30T08:34:14.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:13 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1673851406' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:34:14.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:13 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1479793854' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:34:14.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:13 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1673851406' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:34:14.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:13 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1479793854' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:34:14.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.403+0000 7f3e4ee00640 1 -- 172.21.15.32:0/1321490440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e48106df0 msgr2=0x7f3e48107200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:14.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.403+0000 7f3e4ee00640 1 --2- 172.21.15.32:0/1321490440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e48106df0 0x7f3e48107200 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f3e34009900 tx=0x7f3e3402a3f0 comp rx=0 tx=0).stop 2024-04-30T08:34:14.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.403+0000 7f3e4ee00640 1 -- 172.21.15.32:0/1321490440 shutdown_connections 2024-04-30T08:34:14.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.403+0000 7f3e4ee00640 1 --2- 172.21.15.32:0/1321490440 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e4810baf0 0x7f3e4810df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:14.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.403+0000 7f3e4ee00640 1 --2- 172.21.15.32:0/1321490440 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e48106df0 0x7f3e48107200 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:14.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.403+0000 7f3e4ee00640 1 --2- 172.21.15.32:0/1321490440 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3e48105bb0 0x7f3e481068b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:14.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.403+0000 7f3e4ee00640 1 -- 172.21.15.32:0/1321490440 >> 172.21.15.32:0/1321490440 conn(0x7f3e48100da0 msgr2=0x7f3e48103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:14.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.403+0000 7f3e4ee00640 1 -- 172.21.15.32:0/1321490440 shutdown_connections 2024-04-30T08:34:14.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.403+0000 7f3e4ee00640 1 -- 172.21.15.32:0/1321490440 wait complete. 2024-04-30T08:34:14.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.404+0000 7f3e4ee00640 1 Processor -- start 2024-04-30T08:34:14.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.404+0000 7f3e4ee00640 1 -- start start 2024-04-30T08:34:14.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.405+0000 7f3e4ee00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e48105bb0 0x7f3e4819c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:14.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.405+0000 7f3e4ee00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3e48106df0 0x7f3e4819c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:14.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.405+0000 7f3e4ee00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e4810baf0 0x7f3e481a3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:14.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.405+0000 7f3e4ee00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3e481116d0 con 0x7f3e48105bb0 2024-04-30T08:34:14.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.405+0000 7f3e4ee00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f3e48111550 con 0x7f3e48106df0 2024-04-30T08:34:14.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.405+0000 7f3e4ee00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3e48111850 con 0x7f3e4810baf0 2024-04-30T08:34:14.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.405+0000 7f3e4da00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e48105bb0 0x7f3e4819c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:14.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.405+0000 7f3e4da00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e48105bb0 0x7f3e4819c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45714/0 (socket says 172.21.15.32:45714) 2024-04-30T08:34:14.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.405+0000 7f3e4da00640 1 -- 172.21.15.32:0/3064545631 learned_addr learned my addr 172.21.15.32:0/3064545631 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:14.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.405+0000 7f3e4e400640 1 --2- 172.21.15.32:0/3064545631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e4810baf0 0x7f3e481a3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:14.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.405+0000 7f3e4d000640 1 --2- 172.21.15.32:0/3064545631 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3e48106df0 0x7f3e4819c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:14.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.406+0000 7f3e4e400640 1 -- 172.21.15.32:0/3064545631 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3e48106df0 msgr2=0x7f3e4819c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:14.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.406+0000 7f3e4e400640 1 --2- 172.21.15.32:0/3064545631 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3e48106df0 0x7f3e4819c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:14.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.406+0000 7f3e4e400640 1 -- 172.21.15.32:0/3064545631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e48105bb0 msgr2=0x7f3e4819c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:14.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.406+0000 7f3e4e400640 1 --2- 172.21.15.32:0/3064545631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e48105bb0 0x7f3e4819c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:14.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.406+0000 7f3e4e400640 1 -- 172.21.15.32:0/3064545631 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e481a3df0 con 0x7f3e4810baf0 2024-04-30T08:34:14.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.406+0000 7f3e4da00640 1 --2- 172.21.15.32:0/3064545631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e48105bb0 0x7f3e4819c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:34:14.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.406+0000 7f3e4e400640 1 --2- 172.21.15.32:0/3064545631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e4810baf0 0x7f3e481a3780 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3e4000e940 tx=0x7f3e4000ee00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:14.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.407+0000 7f3e46000640 1 -- 172.21.15.32:0/3064545631 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e40004d20 con 0x7f3e4810baf0 2024-04-30T08:34:14.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.407+0000 7f3e46000640 1 -- 172.21.15.32:0/3064545631 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e40005020 con 0x7f3e4810baf0 2024-04-30T08:34:14.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.407+0000 7f3e4ee00640 1 -- 172.21.15.32:0/3064545631 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e481a40e0 con 0x7f3e4810baf0 2024-04-30T08:34:14.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.407+0000 7f3e46000640 1 -- 172.21.15.32:0/3064545631 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e400116d0 con 0x7f3e4810baf0 2024-04-30T08:34:14.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.407+0000 7f3e4ee00640 1 -- 172.21.15.32:0/3064545631 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e481a45e0 con 0x7f3e4810baf0 2024-04-30T08:34:14.410 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.408+0000 7f3e4ee00640 1 -- 172.21.15.32:0/3064545631 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e4810a900 con 0x7f3e4810baf0 2024-04-30T08:34:14.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.410+0000 7f3e46000640 1 -- 172.21.15.32:0/3064545631 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f3e40005460 con 0x7f3e4810baf0 2024-04-30T08:34:14.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.412+0000 7f3e46000640 1 --2- 172.21.15.32:0/3064545631 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3e300afae0 0x7f3e300b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:14.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.412+0000 7f3e46000640 1 -- 172.21.15.32:0/3064545631 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f3e400df460 con 0x7f3e4810baf0 2024-04-30T08:34:14.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.412+0000 7f3e4da00640 1 --2- 172.21.15.32:0/3064545631 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3e300afae0 0x7f3e300b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:14.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.413+0000 7f3e4da00640 1 --2- 172.21.15.32:0/3064545631 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3e300afae0 0x7f3e300b1ff0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c009fd0 tx=0x7f3e3c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:14.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.414+0000 7f3e46000640 1 -- 172.21.15.32:0/3064545631 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3e40016030 con 0x7f3e4810baf0 2024-04-30T08:34:14.612 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.611+0000 7f3e4ee00640 1 -- 172.21.15.32:0/3064545631 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f3e48063820 con 0x7f3e4810baf0 2024-04-30T08:34:14.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.612+0000 7f3e46000640 1 -- 172.21.15.32:0/3064545631 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) ==== 104+0+11 (secure 0 0 0) 0x7f3e400a7620 con 0x7f3e4810baf0 2024-04-30T08:34:14.613 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-30T08:34:14.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.615+0000 7f3e4ee00640 1 -- 172.21.15.32:0/3064545631 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3e300afae0 msgr2=0x7f3e300b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:14.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.615+0000 7f3e4ee00640 1 --2- 172.21.15.32:0/3064545631 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3e300afae0 0x7f3e300b1ff0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c009fd0 tx=0x7f3e3c005f90 comp rx=0 tx=0).stop 2024-04-30T08:34:14.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.615+0000 7f3e4ee00640 1 -- 172.21.15.32:0/3064545631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e4810baf0 msgr2=0x7f3e481a3780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:14.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.615+0000 7f3e4ee00640 1 --2- 172.21.15.32:0/3064545631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e4810baf0 0x7f3e481a3780 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f3e4000e940 tx=0x7f3e4000ee00 comp rx=0 tx=0).stop 2024-04-30T08:34:14.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.616+0000 7f3e4ee00640 1 -- 172.21.15.32:0/3064545631 shutdown_connections 2024-04-30T08:34:14.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.616+0000 7f3e4ee00640 1 --2- 172.21.15.32:0/3064545631 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3e300afae0 0x7f3e300b1ff0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:14.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.616+0000 7f3e4ee00640 1 --2- 172.21.15.32:0/3064545631 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e4810baf0 0x7f3e481a3780 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:14.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.616+0000 7f3e4ee00640 1 --2- 172.21.15.32:0/3064545631 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3e48106df0 0x7f3e4819c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:14.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.616+0000 7f3e4ee00640 1 --2- 172.21.15.32:0/3064545631 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3e48105bb0 0x7f3e4819c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:14.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.616+0000 7f3e4ee00640 1 -- 172.21.15.32:0/3064545631 >> 172.21.15.32:0/3064545631 conn(0x7f3e48100da0 msgr2=0x7f3e48107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:14.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.616+0000 7f3e4ee00640 1 -- 172.21.15.32:0/3064545631 shutdown_connections 2024-04-30T08:34:14.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.616+0000 7f3e4ee00640 1 -- 172.21.15.32:0/3064545631 wait complete. 2024-04-30T08:34:14.647 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-30T08:34:14.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.930+0000 7f163ac00640 1 -- 172.21.15.32:0/4211576726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1634106df0 msgr2=0x7f1634107200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:14.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.930+0000 7f163ac00640 1 --2- 172.21.15.32:0/4211576726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1634106df0 0x7f1634107200 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f1624009900 tx=0x7f162402a3f0 comp rx=0 tx=0).stop 2024-04-30T08:34:14.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.930+0000 7f163ac00640 1 -- 172.21.15.32:0/4211576726 shutdown_connections 2024-04-30T08:34:14.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.930+0000 7f163ac00640 1 --2- 172.21.15.32:0/4211576726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f163410baf0 0x7f163410df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:14.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.930+0000 7f163ac00640 1 --2- 172.21.15.32:0/4211576726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1634106df0 0x7f1634107200 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:14.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.930+0000 7f163ac00640 1 --2- 172.21.15.32:0/4211576726 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1634105bb0 0x7f16341068b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:14.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.930+0000 7f163ac00640 1 -- 172.21.15.32:0/4211576726 >> 172.21.15.32:0/4211576726 conn(0x7f1634100da0 msgr2=0x7f1634103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:14.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.930+0000 7f163ac00640 1 -- 172.21.15.32:0/4211576726 shutdown_connections 2024-04-30T08:34:14.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.930+0000 7f163ac00640 1 -- 172.21.15.32:0/4211576726 wait complete. 2024-04-30T08:34:14.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.931+0000 7f163ac00640 1 Processor -- start 2024-04-30T08:34:14.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.931+0000 7f163ac00640 1 -- start start 2024-04-30T08:34:14.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.932+0000 7f163ac00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1634105bb0 0x7f163419c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:14.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.932+0000 7f163ac00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1634106df0 0x7f163419c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:14.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.932+0000 7f163ac00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f163410baf0 0x7f16341a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:14.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.932+0000 7f163ac00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f16341116d0 con 0x7f1634106df0 2024-04-30T08:34:14.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.932+0000 7f163ac00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1634111550 con 0x7f163410baf0 2024-04-30T08:34:14.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.932+0000 7f163ac00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1634111850 con 0x7f1634105bb0 2024-04-30T08:34:14.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.932+0000 7f1638e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1634106df0 0x7f163419c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:14.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.932+0000 7f1638e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1634106df0 0x7f163419c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45736/0 (socket says 172.21.15.32:45736) 2024-04-30T08:34:14.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.932+0000 7f1638e00640 1 -- 172.21.15.32:0/1474622550 learned_addr learned my addr 172.21.15.32:0/1474622550 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:14.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.932+0000 7f1639800640 1 --2- 172.21.15.32:0/1474622550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1634105bb0 0x7f163419c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:14.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.932+0000 7f163a200640 1 --2- 172.21.15.32:0/1474622550 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f163410baf0 0x7f16341a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:14.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.932+0000 7f1639800640 1 -- 172.21.15.32:0/1474622550 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f163410baf0 msgr2=0x7f16341a3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:14.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.932+0000 7f1639800640 1 --2- 172.21.15.32:0/1474622550 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f163410baf0 0x7f16341a3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:14.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.932+0000 7f1639800640 1 -- 172.21.15.32:0/1474622550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1634106df0 msgr2=0x7f163419c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:14.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.932+0000 7f1639800640 1 --2- 172.21.15.32:0/1474622550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1634106df0 0x7f163419c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:14.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.933+0000 7f1639800640 1 -- 172.21.15.32:0/1474622550 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16341a3de0 con 0x7f1634105bb0 2024-04-30T08:34:14.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.933+0000 7f1639800640 1 --2- 172.21.15.32:0/1474622550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1634105bb0 0x7f163419c2a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1634105480 tx=0x7f162800ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:14.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.933+0000 7f1632000640 1 -- 172.21.15.32:0/1474622550 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f162800cc40 con 0x7f1634105bb0 2024-04-30T08:34:14.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.933+0000 7f163ac00640 1 -- 172.21.15.32:0/1474622550 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16341a40d0 con 0x7f1634105bb0 2024-04-30T08:34:14.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.933+0000 7f1632000640 1 -- 172.21.15.32:0/1474622550 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f162800cde0 con 0x7f1634105bb0 2024-04-30T08:34:14.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.933+0000 7f1632000640 1 -- 172.21.15.32:0/1474622550 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1628010430 con 0x7f1634105bb0 2024-04-30T08:34:14.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.933+0000 7f163ac00640 1 -- 172.21.15.32:0/1474622550 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16341a45d0 con 0x7f1634105bb0 2024-04-30T08:34:14.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.934+0000 7f163ac00640 1 -- 172.21.15.32:0/1474622550 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f163410a900 con 0x7f1634105bb0 2024-04-30T08:34:14.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.937+0000 7f1632000640 1 -- 172.21.15.32:0/1474622550 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f1628003680 con 0x7f1634105bb0 2024-04-30T08:34:14.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.938+0000 7f1632000640 1 --2- 172.21.15.32:0/1474622550 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f161c0afae0 0x7f161c0b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:14.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.938+0000 7f1632000640 1 -- 172.21.15.32:0/1474622550 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f16280df330 con 0x7f1634105bb0 2024-04-30T08:34:14.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.939+0000 7f1638e00640 1 --2- 172.21.15.32:0/1474622550 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f161c0afae0 0x7f161c0b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:14.940 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:14 smithi032 ceph-mon[28331]: pgmap v397: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 511 B/s wr, 10 op/s 2024-04-30T08:34:14.940 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:14 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/615360310' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:34:14.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:14 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3064545631' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-30T08:34:14.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.939+0000 7f1638e00640 1 --2- 172.21.15.32:0/1474622550 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f161c0afae0 0x7f161c0b1ff0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1624031440 tx=0x7f16240314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:14.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:14.942+0000 7f1632000640 1 -- 172.21.15.32:0/1474622550 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f16280a7b00 con 0x7f1634105bb0 2024-04-30T08:34:15.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.152+0000 7f163ac00640 1 -- 172.21.15.32:0/1474622550 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f1634063820 con 0x7f1634105bb0 2024-04-30T08:34:15.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.153+0000 7f1632000640 1 -- 172.21.15.32:0/1474622550 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) ==== 100+0+11 (secure 0 0 0) 0x7f16280a7400 con 0x7f1634105bb0 2024-04-30T08:34:15.155 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-04-30T08:34:15.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.156+0000 7f163ac00640 1 -- 172.21.15.32:0/1474622550 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f161c0afae0 msgr2=0x7f161c0b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:15.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.156+0000 7f163ac00640 1 --2- 172.21.15.32:0/1474622550 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f161c0afae0 0x7f161c0b1ff0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f1624031440 tx=0x7f16240314a0 comp rx=0 tx=0).stop 2024-04-30T08:34:15.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.156+0000 7f163ac00640 1 -- 172.21.15.32:0/1474622550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1634105bb0 msgr2=0x7f163419c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:15.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.156+0000 7f163ac00640 1 --2- 172.21.15.32:0/1474622550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1634105bb0 0x7f163419c2a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f1634105480 tx=0x7f162800ee50 comp rx=0 tx=0).stop 2024-04-30T08:34:15.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.156+0000 7f163ac00640 1 -- 172.21.15.32:0/1474622550 shutdown_connections 2024-04-30T08:34:15.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.156+0000 7f163ac00640 1 --2- 172.21.15.32:0/1474622550 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f161c0afae0 0x7f161c0b1ff0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:15.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.156+0000 7f163ac00640 1 --2- 172.21.15.32:0/1474622550 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f163410baf0 0x7f16341a3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:15.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.156+0000 7f163ac00640 1 --2- 172.21.15.32:0/1474622550 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1634106df0 0x7f163419c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:15.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.156+0000 7f163ac00640 1 --2- 172.21.15.32:0/1474622550 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1634105bb0 0x7f163419c2a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:15.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.156+0000 7f163ac00640 1 -- 172.21.15.32:0/1474622550 >> 172.21.15.32:0/1474622550 conn(0x7f1634100da0 msgr2=0x7f1634107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:15.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.156+0000 7f163ac00640 1 -- 172.21.15.32:0/1474622550 shutdown_connections 2024-04-30T08:34:15.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.156+0000 7f163ac00640 1 -- 172.21.15.32:0/1474622550 wait complete. 2024-04-30T08:34:15.187 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-30T08:34:15.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:14 smithi071 ceph-mon[33385]: pgmap v397: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 511 B/s wr, 10 op/s 2024-04-30T08:34:15.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:14 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/615360310' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:34:15.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:14 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3064545631' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-30T08:34:15.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:14 smithi189 ceph-mon[33038]: pgmap v397: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 511 B/s wr, 10 op/s 2024-04-30T08:34:15.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:14 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/615360310' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:34:15.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:14 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3064545631' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-30T08:34:15.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.474+0000 7f1513e00640 1 -- 172.21.15.32:0/2941924150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f150c10a4a0 msgr2=0x7f150c10ae50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:15.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.474+0000 7f1513e00640 1 --2- 172.21.15.32:0/2941924150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f150c10a4a0 0x7f150c10ae50 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f14fc009900 tx=0x7f14fc02a3f0 comp rx=0 tx=0).stop 2024-04-30T08:34:15.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.474+0000 7f1513e00640 1 -- 172.21.15.32:0/2941924150 shutdown_connections 2024-04-30T08:34:15.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.474+0000 7f1513e00640 1 --2- 172.21.15.32:0/2941924150 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f150c10c090 0x7f150c10c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:15.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.474+0000 7f1513e00640 1 --2- 172.21.15.32:0/2941924150 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f150c10b390 0x7f150c10b7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:15.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.474+0000 7f1513e00640 1 --2- 172.21.15.32:0/2941924150 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f150c10a4a0 0x7f150c10ae50 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:15.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.474+0000 7f1513e00640 1 -- 172.21.15.32:0/2941924150 >> 172.21.15.32:0/2941924150 conn(0x7f150c0709c0 msgr2=0x7f150c072e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:15.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.475+0000 7f1513e00640 1 -- 172.21.15.32:0/2941924150 shutdown_connections 2024-04-30T08:34:15.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.475+0000 7f1513e00640 1 -- 172.21.15.32:0/2941924150 wait complete. 2024-04-30T08:34:15.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.476+0000 7f1513e00640 1 Processor -- start 2024-04-30T08:34:15.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.476+0000 7f1513e00640 1 -- start start 2024-04-30T08:34:15.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.476+0000 7f1513e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f150c10a4a0 0x7f150c1a4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:15.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.476+0000 7f1513e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f150c10b390 0x7f150c1a52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:15.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.476+0000 7f1513e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f150c10c090 0x7f150c1ac210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:15.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.476+0000 7f1513e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f150c11a130 con 0x7f150c10a4a0 2024-04-30T08:34:15.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.476+0000 7f1513e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f150c119fb0 con 0x7f150c10b390 2024-04-30T08:34:15.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.476+0000 7f1513e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f150c11a2b0 con 0x7f150c10c090 2024-04-30T08:34:15.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.476+0000 7f1511400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f150c10a4a0 0x7f150c1a4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:15.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.476+0000 7f1511400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f150c10a4a0 0x7f150c1a4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45750/0 (socket says 172.21.15.32:45750) 2024-04-30T08:34:15.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.477+0000 7f1511400640 1 -- 172.21.15.32:0/2077225059 learned_addr learned my addr 172.21.15.32:0/2077225059 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:15.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.477+0000 7f1511e00640 1 --2- 172.21.15.32:0/2077225059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f150c10c090 0x7f150c1ac210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:15.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.477+0000 7f1510a00640 1 --2- 172.21.15.32:0/2077225059 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f150c10b390 0x7f150c1a52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:15.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.477+0000 7f1511400640 1 -- 172.21.15.32:0/2077225059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f150c10c090 msgr2=0x7f150c1ac210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:15.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.477+0000 7f1511400640 1 --2- 172.21.15.32:0/2077225059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f150c10c090 0x7f150c1ac210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:15.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.477+0000 7f1511400640 1 -- 172.21.15.32:0/2077225059 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f150c10b390 msgr2=0x7f150c1a52c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:15.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.477+0000 7f1511400640 1 --2- 172.21.15.32:0/2077225059 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f150c10b390 0x7f150c1a52c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:15.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.477+0000 7f1511400640 1 -- 172.21.15.32:0/2077225059 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f150c1ac8b0 con 0x7f150c10a4a0 2024-04-30T08:34:15.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.477+0000 7f1511400640 1 --2- 172.21.15.32:0/2077225059 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f150c10a4a0 0x7f150c1a4d80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f14fc031440 tx=0x7f14fc003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:15.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.478+0000 7f150a000640 1 -- 172.21.15.32:0/2077225059 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14fc03f070 con 0x7f150c10a4a0 2024-04-30T08:34:15.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.478+0000 7f150a000640 1 -- 172.21.15.32:0/2077225059 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14fc031c50 con 0x7f150c10a4a0 2024-04-30T08:34:15.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.478+0000 7f1513e00640 1 -- 172.21.15.32:0/2077225059 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f150c1acb40 con 0x7f150c10a4a0 2024-04-30T08:34:15.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.478+0000 7f150a000640 1 -- 172.21.15.32:0/2077225059 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14fc035030 con 0x7f150c10a4a0 2024-04-30T08:34:15.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.478+0000 7f1513e00640 1 -- 172.21.15.32:0/2077225059 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f150c1ad080 con 0x7f150c10a4a0 2024-04-30T08:34:15.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.479+0000 7f1513e00640 1 -- 172.21.15.32:0/2077225059 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f150c116a20 con 0x7f150c10a4a0 2024-04-30T08:34:15.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.481+0000 7f150a000640 1 -- 172.21.15.32:0/2077225059 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f14fc042070 con 0x7f150c10a4a0 2024-04-30T08:34:15.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.483+0000 7f150a000640 1 --2- 172.21.15.32:0/2077225059 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f14f40afb70 0x7f14f40b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:15.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.483+0000 7f150a000640 1 -- 172.21.15.32:0/2077225059 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f14fc0ffea0 con 0x7f150c10a4a0 2024-04-30T08:34:15.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.483+0000 7f1510a00640 1 --2- 172.21.15.32:0/2077225059 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f14f40afb70 0x7f14f40b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:15.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.484+0000 7f1510a00640 1 --2- 172.21.15.32:0/2077225059 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f14f40afb70 0x7f14f40b2080 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f15000099a0 tx=0x7f1500009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:15.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.486+0000 7f150a000640 1 -- 172.21.15.32:0/2077225059 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f14fc0c8020 con 0x7f150c10a4a0 2024-04-30T08:34:15.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.693+0000 7f1513e00640 1 -- 172.21.15.32:0/2077225059 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f150c063820 con 0x7f150c10a4a0 2024-04-30T08:34:15.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.695+0000 7f150a000640 1 -- 172.21.15.32:0/2077225059 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v85) ==== 103+0+11 (secure 0 0 0) 0x7f14fc0cc320 con 0x7f150c10a4a0 2024-04-30T08:34:15.696 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-30T08:34:15.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.697+0000 7f1513e00640 1 -- 172.21.15.32:0/2077225059 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f14f40afb70 msgr2=0x7f14f40b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:15.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.698+0000 7f1513e00640 1 --2- 172.21.15.32:0/2077225059 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f14f40afb70 0x7f14f40b2080 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f15000099a0 tx=0x7f1500009500 comp rx=0 tx=0).stop 2024-04-30T08:34:15.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.698+0000 7f1513e00640 1 -- 172.21.15.32:0/2077225059 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f150c10a4a0 msgr2=0x7f150c1a4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:15.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.698+0000 7f1513e00640 1 --2- 172.21.15.32:0/2077225059 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f150c10a4a0 0x7f150c1a4d80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f14fc031440 tx=0x7f14fc003c10 comp rx=0 tx=0).stop 2024-04-30T08:34:15.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.698+0000 7f1513e00640 1 -- 172.21.15.32:0/2077225059 shutdown_connections 2024-04-30T08:34:15.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.698+0000 7f1513e00640 1 --2- 172.21.15.32:0/2077225059 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f14f40afb70 0x7f14f40b2080 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:15.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.698+0000 7f1513e00640 1 --2- 172.21.15.32:0/2077225059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f150c10c090 0x7f150c1ac210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:15.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.698+0000 7f1513e00640 1 --2- 172.21.15.32:0/2077225059 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f150c10b390 0x7f150c1a52c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:15.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.698+0000 7f1513e00640 1 --2- 172.21.15.32:0/2077225059 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f150c10a4a0 0x7f150c1a4d80 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:15.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.698+0000 7f1513e00640 1 -- 172.21.15.32:0/2077225059 >> 172.21.15.32:0/2077225059 conn(0x7f150c0709c0 msgr2=0x7f150c113bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:15.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.698+0000 7f1513e00640 1 -- 172.21.15.32:0/2077225059 shutdown_connections 2024-04-30T08:34:15.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:15.698+0000 7f1513e00640 1 -- 172.21.15.32:0/2077225059 wait complete. 2024-04-30T08:34:15.728 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-30T08:34:16.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.011+0000 7f0784a00640 1 -- 172.21.15.32:0/1097103727 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0780105bb0 msgr2=0x7f07801068b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:16.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.011+0000 7f0784a00640 1 --2- 172.21.15.32:0/1097103727 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0780105bb0 0x7f07801068b0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f0768009900 tx=0x7f076802a3f0 comp rx=0 tx=0).stop 2024-04-30T08:34:16.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.011+0000 7f0784a00640 1 -- 172.21.15.32:0/1097103727 shutdown_connections 2024-04-30T08:34:16.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.011+0000 7f0784a00640 1 --2- 172.21.15.32:0/1097103727 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f078010baf0 0x7f078010df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:16.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.011+0000 7f0784a00640 1 --2- 172.21.15.32:0/1097103727 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0780106df0 0x7f0780107200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:16.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.011+0000 7f0784a00640 1 --2- 172.21.15.32:0/1097103727 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0780105bb0 0x7f07801068b0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:16.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.011+0000 7f0784a00640 1 -- 172.21.15.32:0/1097103727 >> 172.21.15.32:0/1097103727 conn(0x7f0780100da0 msgr2=0x7f0780103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:16.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.011+0000 7f0784a00640 1 -- 172.21.15.32:0/1097103727 shutdown_connections 2024-04-30T08:34:16.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.011+0000 7f0784a00640 1 -- 172.21.15.32:0/1097103727 wait complete. 2024-04-30T08:34:16.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.012+0000 7f0784a00640 1 Processor -- start 2024-04-30T08:34:16.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.012+0000 7f0784a00640 1 -- start start 2024-04-30T08:34:16.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.013+0000 7f0784a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0780105bb0 0x7f078019c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:16.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.013+0000 7f0784a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0780106df0 0x7f078019c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:16.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.013+0000 7f0784a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f078010baf0 0x7f07801a37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:16.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.013+0000 7f0784a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0780111700 con 0x7f0780105bb0 2024-04-30T08:34:16.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.013+0000 7f0784a00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f0780111580 con 0x7f0780106df0 2024-04-30T08:34:16.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.013+0000 7f0784a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0780111880 con 0x7f078010baf0 2024-04-30T08:34:16.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.013+0000 7f077e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0780105bb0 0x7f078019c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:16.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.013+0000 7f077e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0780105bb0 0x7f078019c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45766/0 (socket says 172.21.15.32:45766) 2024-04-30T08:34:16.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.013+0000 7f077e800640 1 -- 172.21.15.32:0/3409291012 learned_addr learned my addr 172.21.15.32:0/3409291012 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:16.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.013+0000 7f077de00640 1 --2- 172.21.15.32:0/3409291012 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0780106df0 0x7f078019c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:16.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.013+0000 7f077f200640 1 --2- 172.21.15.32:0/3409291012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f078010baf0 0x7f07801a37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:16.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.013+0000 7f077e800640 1 -- 172.21.15.32:0/3409291012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f078010baf0 msgr2=0x7f07801a37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:16.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.013+0000 7f077e800640 1 --2- 172.21.15.32:0/3409291012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f078010baf0 0x7f07801a37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:16.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.013+0000 7f077e800640 1 -- 172.21.15.32:0/3409291012 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0780106df0 msgr2=0x7f078019c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:16.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.013+0000 7f077e800640 1 --2- 172.21.15.32:0/3409291012 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0780106df0 0x7f078019c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:16.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.013+0000 7f077e800640 1 -- 172.21.15.32:0/3409291012 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f07801a3d70 con 0x7f0780105bb0 2024-04-30T08:34:16.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.014+0000 7f077e800640 1 --2- 172.21.15.32:0/3409291012 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0780105bb0 0x7f078019c2e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0768031440 tx=0x7f0768003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:16.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.014+0000 7f0777400640 1 -- 172.21.15.32:0/3409291012 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f076803f070 con 0x7f0780105bb0 2024-04-30T08:34:16.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.014+0000 7f0777400640 1 -- 172.21.15.32:0/3409291012 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0768031c50 con 0x7f0780105bb0 2024-04-30T08:34:16.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.014+0000 7f0777400640 1 -- 172.21.15.32:0/3409291012 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0768035030 con 0x7f0780105bb0 2024-04-30T08:34:16.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.014+0000 7f0784a00640 1 -- 172.21.15.32:0/3409291012 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f07801a4000 con 0x7f0780105bb0 2024-04-30T08:34:16.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.014+0000 7f0784a00640 1 -- 172.21.15.32:0/3409291012 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f07801a44f0 con 0x7f0780105bb0 2024-04-30T08:34:16.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.016+0000 7f0784a00640 1 -- 172.21.15.32:0/3409291012 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f078010a900 con 0x7f0780105bb0 2024-04-30T08:34:16.020 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.018+0000 7f0777400640 1 -- 172.21.15.32:0/3409291012 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f0768042070 con 0x7f0780105bb0 2024-04-30T08:34:16.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.020+0000 7f0777400640 1 --2- 172.21.15.32:0/3409291012 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f07640afae0 0x7f07640b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:16.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.020+0000 7f0777400640 1 -- 172.21.15.32:0/3409291012 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f07680fff50 con 0x7f0780105bb0 2024-04-30T08:34:16.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.021+0000 7f077de00640 1 --2- 172.21.15.32:0/3409291012 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f07640afae0 0x7f07640b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:16.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.021+0000 7f077de00640 1 --2- 172.21.15.32:0/3409291012 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f07640afae0 0x7f07640b1ff0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f078019d380 tx=0x7f0770009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:16.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.024+0000 7f0777400640 1 -- 172.21.15.32:0/3409291012 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f07680c80d0 con 0x7f0780105bb0 2024-04-30T08:34:16.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:15 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1474622550' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-30T08:34:16.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:15 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2077225059' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-30T08:34:16.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:15 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1474622550' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-30T08:34:16.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:15 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2077225059' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-30T08:34:16.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.283+0000 7f0784a00640 1 -- 172.21.15.32:0/3409291012 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f078002dbd0 con 0x7f0780105bb0 2024-04-30T08:34:16.286 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.285+0000 7f0777400640 1 -- 172.21.15.32:0/3409291012 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7f07680cc3d0 con 0x7f0780105bb0 2024-04-30T08:34:16.286 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:34:16.287 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2235842676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2235842676},{"type":"v1","addr":"172.21.15.32:6835","nonce":2235842676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/3577009355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":3577009355},{"type":"v1","addr":"172.21.15.32:6841","nonce":3577009355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-30T08:31:52.236668+0000","modified":"2024-04-30T08:32:30.541714+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24541,"mds_3":24533,"mds_4":24517},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/463203038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":463203038},{"type":"v1","addr":"172.21.15.189:6833","nonce":463203038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6833/3635143591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3635143591},{"type":"v1","addr":"172.21.15.71:6833","nonce":3635143591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6837/2569449994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2569449994},{"type":"v1","addr":"172.21.15.32:6837","nonce":2569449994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/278686565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":278686565},{"type":"v1","addr":"172.21.15.189:6835","nonce":278686565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6835/1558303056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1558303056},{"type":"v1","addr":"172.21.15.71:6835","nonce":1558303056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.189:6837/686520873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":686520873},{"type":"v1","addr":"172.21.15.189:6837","nonce":686520873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.71:6837/2324039932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2324039932},{"type":"v1","addr":"172.21.15.71:6837","nonce":2324039932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6839/352970107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":352970107},{"type":"v1","addr":"172.21.15.189:6839","nonce":352970107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.32:6839/4160386524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4160386524},{"type":"v1","addr":"172.21.15.32:6839","nonce":4160386524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.71:6839/3517409993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3517409993},{"type":"v1","addr":"172.21.15.71:6839","nonce":3517409993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24590,24541,24560,24533,24517]},"id":1}]} 2024-04-30T08:34:16.287 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 27 2024-04-30T08:34:16.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.287+0000 7f0784a00640 1 -- 172.21.15.32:0/3409291012 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f07640afae0 msgr2=0x7f07640b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:16.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.287+0000 7f0784a00640 1 --2- 172.21.15.32:0/3409291012 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f07640afae0 0x7f07640b1ff0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f078019d380 tx=0x7f0770009500 comp rx=0 tx=0).stop 2024-04-30T08:34:16.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.287+0000 7f0784a00640 1 -- 172.21.15.32:0/3409291012 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0780105bb0 msgr2=0x7f078019c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:16.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.287+0000 7f0784a00640 1 --2- 172.21.15.32:0/3409291012 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0780105bb0 0x7f078019c2e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f0768031440 tx=0x7f0768003c10 comp rx=0 tx=0).stop 2024-04-30T08:34:16.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.287+0000 7f0784a00640 1 -- 172.21.15.32:0/3409291012 shutdown_connections 2024-04-30T08:34:16.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.287+0000 7f0784a00640 1 --2- 172.21.15.32:0/3409291012 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f07640afae0 0x7f07640b1ff0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:16.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.287+0000 7f0784a00640 1 --2- 172.21.15.32:0/3409291012 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f078010baf0 0x7f07801a37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:16.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.287+0000 7f0784a00640 1 --2- 172.21.15.32:0/3409291012 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0780106df0 0x7f078019c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:16.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.287+0000 7f0784a00640 1 --2- 172.21.15.32:0/3409291012 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0780105bb0 0x7f078019c2e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:16.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.287+0000 7f0784a00640 1 -- 172.21.15.32:0/3409291012 >> 172.21.15.32:0/3409291012 conn(0x7f0780100da0 msgr2=0x7f0780107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:16.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.288+0000 7f0784a00640 1 -- 172.21.15.32:0/3409291012 shutdown_connections 2024-04-30T08:34:16.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.288+0000 7f0784a00640 1 -- 172.21.15.32:0/3409291012 wait complete. 2024-04-30T08:34:16.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:15 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1474622550' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-30T08:34:16.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:15 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2077225059' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-30T08:34:16.314 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-30T08:34:16.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.595+0000 7feeeea00640 1 -- 172.21.15.32:0/3493997982 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feee810baf0 msgr2=0x7feee810df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:16.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.595+0000 7feeeea00640 1 --2- 172.21.15.32:0/3493997982 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feee810baf0 0x7feee810df10 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7feee0009ca0 tx=0x7feee002a8b0 comp rx=0 tx=0).stop 2024-04-30T08:34:16.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.595+0000 7feeeea00640 1 -- 172.21.15.32:0/3493997982 shutdown_connections 2024-04-30T08:34:16.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.595+0000 7feeeea00640 1 --2- 172.21.15.32:0/3493997982 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feee810baf0 0x7feee810df10 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:16.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.595+0000 7feeeea00640 1 --2- 172.21.15.32:0/3493997982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feee8106df0 0x7feee8107200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:16.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.595+0000 7feeeea00640 1 --2- 172.21.15.32:0/3493997982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feee8105bb0 0x7feee81068b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:16.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.595+0000 7feeeea00640 1 -- 172.21.15.32:0/3493997982 >> 172.21.15.32:0/3493997982 conn(0x7feee8100da0 msgr2=0x7feee8103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:16.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.595+0000 7feeeea00640 1 -- 172.21.15.32:0/3493997982 shutdown_connections 2024-04-30T08:34:16.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.596+0000 7feeeea00640 1 -- 172.21.15.32:0/3493997982 wait complete. 2024-04-30T08:34:16.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.597+0000 7feeeea00640 1 Processor -- start 2024-04-30T08:34:16.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.597+0000 7feeeea00640 1 -- start start 2024-04-30T08:34:16.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.597+0000 7feeeea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feee8105bb0 0x7feee819c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:16.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.597+0000 7feeeea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feee8106df0 0x7feee819c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:16.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.597+0000 7feeeea00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feee810baf0 0x7feee81a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:16.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.597+0000 7feeeea00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7feee8111730 con 0x7feee8105bb0 2024-04-30T08:34:16.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.597+0000 7feeeea00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7feee81115b0 con 0x7feee810baf0 2024-04-30T08:34:16.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.597+0000 7feeeea00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7feee81118b0 con 0x7feee8106df0 2024-04-30T08:34:16.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.597+0000 7feeed600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feee8105bb0 0x7feee819c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:16.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.597+0000 7feeecc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feee8106df0 0x7feee819c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:16.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.597+0000 7feeed600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feee8105bb0 0x7feee819c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:45792/0 (socket says 172.21.15.32:45792) 2024-04-30T08:34:16.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.597+0000 7feeee000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feee810baf0 0x7feee81a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:16.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.597+0000 7feeecc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feee8106df0 0x7feee819c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:60512/0 (socket says 172.21.15.32:60512) 2024-04-30T08:34:16.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.597+0000 7feeed600640 1 -- 172.21.15.32:0/1296706621 learned_addr learned my addr 172.21.15.32:0/1296706621 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:16.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.598+0000 7feeecc00640 1 -- 172.21.15.32:0/1296706621 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feee810baf0 msgr2=0x7feee81a36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:16.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.598+0000 7feeecc00640 1 --2- 172.21.15.32:0/1296706621 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feee810baf0 0x7feee81a36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:16.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.598+0000 7feeecc00640 1 -- 172.21.15.32:0/1296706621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feee8105bb0 msgr2=0x7feee819c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.598+0000 7feeecc00640 1 --2- 172.21.15.32:0/1296706621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feee8105bb0 0x7feee819c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.598+0000 7feeecc00640 1 -- 172.21.15.32:0/1296706621 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feee81a3d60 con 0x7feee8106df0 2024-04-30T08:34:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.598+0000 7feeed600640 1 --2- 172.21.15.32:0/1296706621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feee8105bb0 0x7feee819c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:34:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.598+0000 7feeecc00640 1 --2- 172.21.15.32:0/1296706621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feee8106df0 0x7feee819c7b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7feed800e9f0 tx=0x7feed800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.599+0000 7feee6000640 1 -- 172.21.15.32:0/1296706621 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed800ccc0 con 0x7feee8106df0 2024-04-30T08:34:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.599+0000 7feee6000640 1 -- 172.21.15.32:0/1296706621 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7feed80108e0 con 0x7feee8106df0 2024-04-30T08:34:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.599+0000 7feeeea00640 1 -- 172.21.15.32:0/1296706621 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feee81a3ff0 con 0x7feee8106df0 2024-04-30T08:34:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.599+0000 7feee6000640 1 -- 172.21.15.32:0/1296706621 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feed8018880 con 0x7feee8106df0 2024-04-30T08:34:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.599+0000 7feeeea00640 1 -- 172.21.15.32:0/1296706621 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feee81a44e0 con 0x7feee8106df0 2024-04-30T08:34:16.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.600+0000 7feeeea00640 1 -- 172.21.15.32:0/1296706621 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feee810a900 con 0x7feee8106df0 2024-04-30T08:34:16.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.603+0000 7feee6000640 1 -- 172.21.15.32:0/1296706621 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7feed8010430 con 0x7feee8106df0 2024-04-30T08:34:16.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.604+0000 7feee6000640 1 --2- 172.21.15.32:0/1296706621 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7feed00afae0 0x7feed00b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:16.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.604+0000 7feee6000640 1 -- 172.21.15.32:0/1296706621 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7feed80df7c0 con 0x7feee8106df0 2024-04-30T08:34:16.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.605+0000 7feeed600640 1 --2- 172.21.15.32:0/1296706621 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7feed00afae0 0x7feed00b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:16.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.605+0000 7feeed600640 1 --2- 172.21.15.32:0/1296706621 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7feed00afae0 0x7feed00b1ff0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7feedc009fd0 tx=0x7feedc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:16.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.606+0000 7feee6000640 1 -- 172.21.15.32:0/1296706621 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7feed80a7940 con 0x7feee8106df0 2024-04-30T08:34:16.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.824+0000 7feeeea00640 1 -- 172.21.15.32:0/1296706621 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7feee802dda0 con 0x7feee8106df0 2024-04-30T08:34:16.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.827+0000 7feee6000640 1 -- 172.21.15.32:0/1296706621 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+24730 (secure 0 0 0) 0x7feed80abc40 con 0x7feee8106df0 2024-04-30T08:34:16.829 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:34:16.829 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":85,"fsid":"98c0a670-06ca-11ef-bc93-c7b262605968","created":"2024-04-30T08:22:11.122875+0000","modified":"2024-04-30T08:33:31.872704+0000","last_up_change":"2024-04-30T08:30:50.350439+0000","last_in_change":"2024-04-30T08:30:30.210620+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-30T08:26:19.728331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-30T08:31:49.800542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-30T08:31:51.211320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-30T08:31:55.716625+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a7e060c8-1cbb-418f-9a03-06693f6b3f52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6803","nonce":2624700402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6805","nonce":2624700402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6809","nonce":2624700402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6807","nonce":2624700402}]},"public_addr":"172.21.15.32:6803/2624700402","cluster_addr":"172.21.15.32:6805/2624700402","heartbeat_back_addr":"172.21.15.32:6809/2624700402","heartbeat_front_addr":"172.21.15.32:6807/2624700402","state":["exists","up"]},{"osd":1,"uuid":"72ef2fb1-e5ca-48ef-aa3a-4796a64b922f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6801","nonce":444649711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6803","nonce":444649711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6807","nonce":444649711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6805","nonce":444649711}]},"public_addr":"172.21.15.71:6801/444649711","cluster_addr":"172.21.15.71:6803/444649711","heartbeat_back_addr":"172.21.15.71:6807/444649711","heartbeat_front_addr":"172.21.15.71:6805/444649711","state":["exists","up"]},{"osd":2,"uuid":"704e5e8a-935e-47c7-ae28-27119876ac09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6801","nonce":3255528671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6803","nonce":3255528671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6807","nonce":3255528671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6805","nonce":3255528671}]},"public_addr":"172.21.15.189:6801/3255528671","cluster_addr":"172.21.15.189:6803/3255528671","heartbeat_back_addr":"172.21.15.189:6807/3255528671","heartbeat_front_addr":"172.21.15.189:6805/3255528671","state":["exists","up"]},{"osd":3,"uuid":"575004f7-6559-4a62-863e-e33078b0eebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6811","nonce":1384702112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6813","nonce":1384702112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6817","nonce":1384702112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6815","nonce":1384702112}]},"public_addr":"172.21.15.32:6811/1384702112","cluster_addr":"172.21.15.32:6813/1384702112","heartbeat_back_addr":"172.21.15.32:6817/1384702112","heartbeat_front_addr":"172.21.15.32:6815/1384702112","state":["exists","up"]},{"osd":4,"uuid":"e1c1e894-074d-4c78-89ef-9e5a114268bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6809","nonce":4250534574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6811","nonce":4250534574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6815","nonce":4250534574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6813","nonce":4250534574}]},"public_addr":"172.21.15.71:6809/4250534574","cluster_addr":"172.21.15.71:6811/4250534574","heartbeat_back_addr":"172.21.15.71:6815/4250534574","heartbeat_front_addr":"172.21.15.71:6813/4250534574","state":["exists","up"]},{"osd":5,"uuid":"12c022d4-fac6-40c4-ae7a-691571bee5d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6809","nonce":4245346198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6811","nonce":4245346198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6815","nonce":4245346198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6813","nonce":4245346198}]},"public_addr":"172.21.15.189:6809/4245346198","cluster_addr":"172.21.15.189:6811/4245346198","heartbeat_back_addr":"172.21.15.189:6815/4245346198","heartbeat_front_addr":"172.21.15.189:6813/4245346198","state":["exists","up"]},{"osd":6,"uuid":"b413f8ee-3d4a-415a-a2ba-d68437b7330d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6819","nonce":1910389190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6821","nonce":1910389190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6825","nonce":1910389190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6823","nonce":1910389190}]},"public_addr":"172.21.15.32:6819/1910389190","cluster_addr":"172.21.15.32:6821/1910389190","heartbeat_back_addr":"172.21.15.32:6825/1910389190","heartbeat_front_addr":"172.21.15.32:6823/1910389190","state":["exists","up"]},{"osd":7,"uuid":"aac64fb3-fc13-4126-b842-7b06ad5ce238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6817","nonce":692607380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6819","nonce":692607380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6823","nonce":692607380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6821","nonce":692607380}]},"public_addr":"172.21.15.71:6817/692607380","cluster_addr":"172.21.15.71:6819/692607380","heartbeat_back_addr":"172.21.15.71:6823/692607380","heartbeat_front_addr":"172.21.15.71:6821/692607380","state":["exists","up"]},{"osd":8,"uuid":"cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6817","nonce":3973697069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6819","nonce":3973697069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6823","nonce":3973697069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6821","nonce":3973697069}]},"public_addr":"172.21.15.189:6817/3973697069","cluster_addr":"172.21.15.189:6819/3973697069","heartbeat_back_addr":"172.21.15.189:6823/3973697069","heartbeat_front_addr":"172.21.15.189:6821/3973697069","state":["exists","up"]},{"osd":9,"uuid":"ef4a1645-ae6e-45c3-82ad-161510e62aa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6827","nonce":46066685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6829","nonce":46066685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6833","nonce":46066685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6831","nonce":46066685}]},"public_addr":"172.21.15.32:6827/46066685","cluster_addr":"172.21.15.32:6829/46066685","heartbeat_back_addr":"172.21.15.32:6833/46066685","heartbeat_front_addr":"172.21.15.32:6831/46066685","state":["exists","up"]},{"osd":10,"uuid":"25b26a38-a644-409f-b941-59efdf04a3ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6825","nonce":1886290293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6827","nonce":1886290293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6831","nonce":1886290293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6829","nonce":1886290293}]},"public_addr":"172.21.15.71:6825/1886290293","cluster_addr":"172.21.15.71:6827/1886290293","heartbeat_back_addr":"172.21.15.71:6831/1886290293","heartbeat_front_addr":"172.21.15.71:6829/1886290293","state":["exists","up"]},{"osd":11,"uuid":"ab96b3ff-3688-455b-af64-60226c5bfbff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6825","nonce":850185001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6827","nonce":850185001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6831","nonce":850185001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6829","nonce":850185001}]},"public_addr":"172.21.15.189:6825/850185001","cluster_addr":"172.21.15.189:6827/850185001","heartbeat_back_addr":"172.21.15.189:6831/850185001","heartbeat_front_addr":"172.21.15.189:6829/850185001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:27.220308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:53.230783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:17.190642+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:52.603799+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:20.568671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:46.158363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:22.706933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:50.665173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:16.602410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:53.814698+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:22.435189+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:48.651949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.e","mappings":[{"from":4,"to":7}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.28","mappings":[{"from":3,"to":6},{"from":5,"to":8}]},{"pgid":"3.30","mappings":[{"from":1,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1266897222":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/2915689533":"2024-05-01T08:23:29.526664+0000","172.21.15.32:6801/4019351306":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/425836193":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/1134075722":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6801/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/21059253":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/3245550894":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/4019351306":"2024-05-01T08:23:29.526664+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-30T08:34:16.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.830+0000 7feeeea00640 1 -- 172.21.15.32:0/1296706621 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7feed00afae0 msgr2=0x7feed00b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:16.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.830+0000 7feeeea00640 1 --2- 172.21.15.32:0/1296706621 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7feed00afae0 0x7feed00b1ff0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7feedc009fd0 tx=0x7feedc005f90 comp rx=0 tx=0).stop 2024-04-30T08:34:16.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.830+0000 7feeeea00640 1 -- 172.21.15.32:0/1296706621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feee8106df0 msgr2=0x7feee819c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:16.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.830+0000 7feeeea00640 1 --2- 172.21.15.32:0/1296706621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feee8106df0 0x7feee819c7b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7feed800e9f0 tx=0x7feed800eeb0 comp rx=0 tx=0).stop 2024-04-30T08:34:16.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.830+0000 7feeeea00640 1 -- 172.21.15.32:0/1296706621 shutdown_connections 2024-04-30T08:34:16.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.830+0000 7feeeea00640 1 --2- 172.21.15.32:0/1296706621 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7feed00afae0 0x7feed00b1ff0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:16.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.830+0000 7feeeea00640 1 --2- 172.21.15.32:0/1296706621 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7feee810baf0 0x7feee81a36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:16.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.830+0000 7feeeea00640 1 --2- 172.21.15.32:0/1296706621 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feee8106df0 0x7feee819c7b0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:16.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.830+0000 7feeeea00640 1 --2- 172.21.15.32:0/1296706621 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7feee8105bb0 0x7feee819c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:16.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.830+0000 7feeeea00640 1 -- 172.21.15.32:0/1296706621 >> 172.21.15.32:0/1296706621 conn(0x7feee8100da0 msgr2=0x7feee8107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:16.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.830+0000 7feeeea00640 1 -- 172.21.15.32:0/1296706621 shutdown_connections 2024-04-30T08:34:16.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:16.830+0000 7feeeea00640 1 -- 172.21.15.32:0/1296706621 wait complete. 2024-04-30T08:34:16.861 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-30T08:34:17.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.145+0000 7fb3f7200640 1 -- 172.21.15.32:0/3356977523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3f010baf0 msgr2=0x7fb3f010df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:17.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.145+0000 7fb3f7200640 1 --2- 172.21.15.32:0/3356977523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3f010baf0 0x7fb3f010df10 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec009ca0 tx=0x7fb3ec02a8c0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.146+0000 7fb3f7200640 1 -- 172.21.15.32:0/3356977523 shutdown_connections 2024-04-30T08:34:17.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.146+0000 7fb3f7200640 1 --2- 172.21.15.32:0/3356977523 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3f010baf0 0x7fb3f010df10 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.146+0000 7fb3f7200640 1 --2- 172.21.15.32:0/3356977523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3f0106df0 0x7fb3f0107200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.146+0000 7fb3f7200640 1 --2- 172.21.15.32:0/3356977523 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb3f0105bb0 0x7fb3f01068b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.146+0000 7fb3f7200640 1 -- 172.21.15.32:0/3356977523 >> 172.21.15.32:0/3356977523 conn(0x7fb3f0100da0 msgr2=0x7fb3f0103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:17.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.146+0000 7fb3f7200640 1 -- 172.21.15.32:0/3356977523 shutdown_connections 2024-04-30T08:34:17.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.146+0000 7fb3f7200640 1 -- 172.21.15.32:0/3356977523 wait complete. 2024-04-30T08:34:17.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.147+0000 7fb3f7200640 1 Processor -- start 2024-04-30T08:34:17.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.147+0000 7fb3f7200640 1 -- start start 2024-04-30T08:34:17.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.147+0000 7fb3f7200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3f0105bb0 0x7fb3f019c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:17.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.147+0000 7fb3f7200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3f0106df0 0x7fb3f019c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:17.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.147+0000 7fb3f7200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb3f010baf0 0x7fb3f01a3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:17.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.147+0000 7fb3f7200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb3f01116e0 con 0x7fb3f0106df0 2024-04-30T08:34:17.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.147+0000 7fb3f7200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb3f0111560 con 0x7fb3f010baf0 2024-04-30T08:34:17.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.147+0000 7fb3f7200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb3f0111860 con 0x7fb3f0105bb0 2024-04-30T08:34:17.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.148+0000 7fb3f5400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3f0106df0 0x7fb3f019c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:17.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.148+0000 7fb3f5e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3f0105bb0 0x7fb3f019c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:17.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.148+0000 7fb3f5e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3f0105bb0 0x7fb3f019c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:59442/0 (socket says 172.21.15.32:59442) 2024-04-30T08:34:17.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.148+0000 7fb3f5e00640 1 -- 172.21.15.32:0/1290817206 learned_addr learned my addr 172.21.15.32:0/1290817206 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:17.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.148+0000 7fb3f5400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3f0106df0 0x7fb3f019c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56872/0 (socket says 172.21.15.32:56872) 2024-04-30T08:34:17.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.148+0000 7fb3f6800640 1 --2- 172.21.15.32:0/1290817206 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb3f010baf0 0x7fb3f01a3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:17.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.148+0000 7fb3f5e00640 1 -- 172.21.15.32:0/1290817206 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb3f010baf0 msgr2=0x7fb3f01a3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:17.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.148+0000 7fb3f5e00640 1 --2- 172.21.15.32:0/1290817206 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb3f010baf0 0x7fb3f01a3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.148+0000 7fb3f5e00640 1 -- 172.21.15.32:0/1290817206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3f0106df0 msgr2=0x7fb3f019c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:17.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.148+0000 7fb3f5e00640 1 --2- 172.21.15.32:0/1290817206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3f0106df0 0x7fb3f019c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.148+0000 7fb3f5e00640 1 -- 172.21.15.32:0/1290817206 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3f01a3d90 con 0x7fb3f0105bb0 2024-04-30T08:34:17.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.149+0000 7fb3f5e00640 1 --2- 172.21.15.32:0/1290817206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3f0105bb0 0x7fb3f019c250 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb3e400e9f0 tx=0x7fb3e400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:17.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.149+0000 7fb3eaa00640 1 -- 172.21.15.32:0/1290817206 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3e400ccc0 con 0x7fb3f0105bb0 2024-04-30T08:34:17.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.149+0000 7fb3f7200640 1 -- 172.21.15.32:0/1290817206 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3f01a4080 con 0x7fb3f0105bb0 2024-04-30T08:34:17.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.149+0000 7fb3eaa00640 1 -- 172.21.15.32:0/1290817206 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb3e40044d0 con 0x7fb3f0105bb0 2024-04-30T08:34:17.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.149+0000 7fb3f7200640 1 -- 172.21.15.32:0/1290817206 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3f01a4620 con 0x7fb3f0105bb0 2024-04-30T08:34:17.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.149+0000 7fb3eaa00640 1 -- 172.21.15.32:0/1290817206 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3e4010430 con 0x7fb3f0105bb0 2024-04-30T08:34:17.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.150+0000 7fb3f7200640 1 -- 172.21.15.32:0/1290817206 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb3d0005290 con 0x7fb3f0105bb0 2024-04-30T08:34:17.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.153+0000 7fb3eaa00640 1 -- 172.21.15.32:0/1290817206 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fb3e40035f0 con 0x7fb3f0105bb0 2024-04-30T08:34:17.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.154+0000 7fb3eaa00640 1 --2- 172.21.15.32:0/1290817206 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb3d80afb60 0x7fb3d80b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:17.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.154+0000 7fb3eaa00640 1 -- 172.21.15.32:0/1290817206 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fb3e40df4e0 con 0x7fb3f0105bb0 2024-04-30T08:34:17.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.155+0000 7fb3f5400640 1 --2- 172.21.15.32:0/1290817206 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb3d80afb60 0x7fb3d80b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:17.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.155+0000 7fb3f5400640 1 --2- 172.21.15.32:0/1290817206 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb3d80afb60 0x7fb3d80b2070 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb3f019d2f0 tx=0x7fb3dc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:17.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.156+0000 7fb3eaa00640 1 -- 172.21.15.32:0/1290817206 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb3e40a7d40 con 0x7fb3f0105bb0 2024-04-30T08:34:17.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:16 smithi032 ceph-mon[28331]: pgmap v398: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-30T08:34:17.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:16 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3409291012' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:34:17.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:16 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1296706621' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:34:17.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:16 smithi071 ceph-mon[33385]: pgmap v398: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-30T08:34:17.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:16 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3409291012' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:34:17.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:16 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1296706621' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:34:17.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:16 smithi189 ceph-mon[33038]: pgmap v398: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-30T08:34:17.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:16 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3409291012' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:34:17.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:16 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1296706621' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:34:17.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.392+0000 7fb3f7200640 1 -- 172.21.15.32:0/1290817206 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb3d0005580 con 0x7fb3f0105bb0 2024-04-30T08:34:17.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.393+0000 7fb3eaa00640 1 -- 172.21.15.32:0/1290817206 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7fb3e40a7640 con 0x7fb3f0105bb0 2024-04-30T08:34:17.395 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:34:17.396 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2235842676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2235842676},{"type":"v1","addr":"172.21.15.32:6835","nonce":2235842676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/3577009355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":3577009355},{"type":"v1","addr":"172.21.15.32:6841","nonce":3577009355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-30T08:31:52.236668+0000","modified":"2024-04-30T08:32:30.541714+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24541,"mds_3":24533,"mds_4":24517},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/463203038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":463203038},{"type":"v1","addr":"172.21.15.189:6833","nonce":463203038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6833/3635143591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3635143591},{"type":"v1","addr":"172.21.15.71:6833","nonce":3635143591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6837/2569449994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2569449994},{"type":"v1","addr":"172.21.15.32:6837","nonce":2569449994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/278686565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":278686565},{"type":"v1","addr":"172.21.15.189:6835","nonce":278686565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6835/1558303056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1558303056},{"type":"v1","addr":"172.21.15.71:6835","nonce":1558303056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.189:6837/686520873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":686520873},{"type":"v1","addr":"172.21.15.189:6837","nonce":686520873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.71:6837/2324039932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2324039932},{"type":"v1","addr":"172.21.15.71:6837","nonce":2324039932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6839/352970107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":352970107},{"type":"v1","addr":"172.21.15.189:6839","nonce":352970107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.32:6839/4160386524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4160386524},{"type":"v1","addr":"172.21.15.32:6839","nonce":4160386524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.71:6839/3517409993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3517409993},{"type":"v1","addr":"172.21.15.71:6839","nonce":3517409993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24590,24541,24560,24533,24517]},"id":1}]} 2024-04-30T08:34:17.396 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 27 2024-04-30T08:34:17.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.396+0000 7fb3f7200640 1 -- 172.21.15.32:0/1290817206 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb3d80afb60 msgr2=0x7fb3d80b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:17.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.396+0000 7fb3f7200640 1 --2- 172.21.15.32:0/1290817206 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb3d80afb60 0x7fb3d80b2070 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fb3f019d2f0 tx=0x7fb3dc005f90 comp rx=0 tx=0).stop 2024-04-30T08:34:17.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.396+0000 7fb3f7200640 1 -- 172.21.15.32:0/1290817206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3f0105bb0 msgr2=0x7fb3f019c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:17.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.396+0000 7fb3f7200640 1 --2- 172.21.15.32:0/1290817206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3f0105bb0 0x7fb3f019c250 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fb3e400e9f0 tx=0x7fb3e400eeb0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.396+0000 7fb3f7200640 1 -- 172.21.15.32:0/1290817206 shutdown_connections 2024-04-30T08:34:17.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.396+0000 7fb3f7200640 1 --2- 172.21.15.32:0/1290817206 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb3d80afb60 0x7fb3d80b2070 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.396+0000 7fb3f7200640 1 --2- 172.21.15.32:0/1290817206 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb3f010baf0 0x7fb3f01a3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.396+0000 7fb3f7200640 1 --2- 172.21.15.32:0/1290817206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb3f0106df0 0x7fb3f019c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.396+0000 7fb3f7200640 1 --2- 172.21.15.32:0/1290817206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3f0105bb0 0x7fb3f019c250 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.396+0000 7fb3f7200640 1 -- 172.21.15.32:0/1290817206 >> 172.21.15.32:0/1290817206 conn(0x7fb3f0100da0 msgr2=0x7fb3f0107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:17.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.396+0000 7fb3f7200640 1 -- 172.21.15.32:0/1290817206 shutdown_connections 2024-04-30T08:34:17.399 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.397+0000 7fb3f7200640 1 -- 172.21.15.32:0/1290817206 wait complete. 2024-04-30T08:34:17.423 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-30T08:31:52.236668+0000', 'modified': '2024-04-30T08:32:30.541714+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24590, 'mds_1': 24560, 'mds_2': 24541, 'mds_3': 24533, 'mds_4': 24517}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24442': {'gid': 24442, 'name': 'l', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.189:6833/463203038', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6832', 'nonce': 463203038}, {'type': 'v1', 'addr': '172.21.15.189:6833', 'nonce': 463203038}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24458': {'gid': 24458, 'name': 'k', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.71:6833/3635143591', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6832', 'nonce': 3635143591}, {'type': 'v1', 'addr': '172.21.15.71:6833', 'nonce': 3635143591}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24484': {'gid': 24484, 'name': 'd', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.32:6837/2569449994', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 2569449994}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 2569449994}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24505': {'gid': 24505, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.189:6835/278686565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6834', 'nonce': 278686565}, {'type': 'v1', 'addr': '172.21.15.189:6835', 'nonce': 278686565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24506': {'gid': 24506, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.71:6835/1558303056', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6834', 'nonce': 1558303056}, {'type': 'v1', 'addr': '172.21.15.71:6835', 'nonce': 1558303056}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24517': {'gid': 24517, 'name': 'c', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.189:6837/686520873', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6836', 'nonce': 686520873}, {'type': 'v1', 'addr': '172.21.15.189:6837', 'nonce': 686520873}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24533': {'gid': 24533, 'name': 'h', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.71:6837/2324039932', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6836', 'nonce': 2324039932}, {'type': 'v1', 'addr': '172.21.15.71:6837', 'nonce': 2324039932}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24541': {'gid': 24541, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.189:6839/352970107', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6838', 'nonce': 352970107}, {'type': 'v1', 'addr': '172.21.15.189:6839', 'nonce': 352970107}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24560': {'gid': 24560, 'name': 'g', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.32:6839/4160386524', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4160386524}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4160386524}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24590': {'gid': 24590, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.71:6839/3517409993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6838', 'nonce': 3517409993}, {'type': 'v1', 'addr': '172.21.15.71:6839', 'nonce': 3517409993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24590, 'qdb_cluster': [24590, 24541, 24560, 24533, 24517]} 2024-04-30T08:34:17.423 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-30T08:34:17.424 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-30T08:34:17.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.708+0000 7ff644600640 1 -- 172.21.15.32:0/623978128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff63c103b90 msgr2=0x7ff63c104890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:17.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.708+0000 7ff644600640 1 --2- 172.21.15.32:0/623978128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff63c103b90 0x7ff63c104890 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff630009ca0 tx=0x7ff63002a8c0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.708+0000 7ff644600640 1 -- 172.21.15.32:0/623978128 shutdown_connections 2024-04-30T08:34:17.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.708+0000 7ff644600640 1 --2- 172.21.15.32:0/623978128 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff63c109ad0 0x7ff63c10bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.708+0000 7ff644600640 1 --2- 172.21.15.32:0/623978128 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff63c104dd0 0x7ff63c1051e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.708+0000 7ff644600640 1 --2- 172.21.15.32:0/623978128 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff63c103b90 0x7ff63c104890 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.708+0000 7ff644600640 1 -- 172.21.15.32:0/623978128 >> 172.21.15.32:0/623978128 conn(0x7ff63c0fedc0 msgr2=0x7ff63c101230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:17.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.708+0000 7ff644600640 1 -- 172.21.15.32:0/623978128 shutdown_connections 2024-04-30T08:34:17.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.708+0000 7ff644600640 1 -- 172.21.15.32:0/623978128 wait complete. 2024-04-30T08:34:17.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.709+0000 7ff644600640 1 Processor -- start 2024-04-30T08:34:17.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.710+0000 7ff644600640 1 -- start start 2024-04-30T08:34:17.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.710+0000 7ff644600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff63c103b90 0x7ff63c19a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:17.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.710+0000 7ff644600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff63c104dd0 0x7ff63c19a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:17.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.710+0000 7ff644600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff63c109ad0 0x7ff63c1a1830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:17.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.710+0000 7ff644600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff63c10f6c0 con 0x7ff63c109ad0 2024-04-30T08:34:17.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.710+0000 7ff644600640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff63c10f540 con 0x7ff63c103b90 2024-04-30T08:34:17.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.710+0000 7ff644600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff63c10f840 con 0x7ff63c104dd0 2024-04-30T08:34:17.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.710+0000 7ff641c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff63c103b90 0x7ff63c19a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:17.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.710+0000 7ff641c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff63c103b90 0x7ff63c19a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:55432/0 (socket says 172.21.15.32:55432) 2024-04-30T08:34:17.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.710+0000 7ff642600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff63c109ad0 0x7ff63c1a1830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:17.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.710+0000 7ff641c00640 1 -- 172.21.15.32:0/671012905 learned_addr learned my addr 172.21.15.32:0/671012905 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:17.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.711+0000 7ff641200640 1 --2- 172.21.15.32:0/671012905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff63c104dd0 0x7ff63c19a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:17.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.711+0000 7ff641c00640 1 -- 172.21.15.32:0/671012905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff63c104dd0 msgr2=0x7ff63c19a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:17.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.711+0000 7ff641c00640 1 --2- 172.21.15.32:0/671012905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff63c104dd0 0x7ff63c19a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.711+0000 7ff641c00640 1 -- 172.21.15.32:0/671012905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff63c109ad0 msgr2=0x7ff63c1a1830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:17.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.711+0000 7ff641c00640 1 --2- 172.21.15.32:0/671012905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff63c109ad0 0x7ff63c1a1830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.711+0000 7ff641c00640 1 -- 172.21.15.32:0/671012905 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff63c1a1ea0 con 0x7ff63c103b90 2024-04-30T08:34:17.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.711+0000 7ff641c00640 1 --2- 172.21.15.32:0/671012905 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff63c103b90 0x7ff63c19a360 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff630005e30 tx=0x7ff6300317e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:17.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.712+0000 7ff63a000640 1 -- 172.21.15.32:0/671012905 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff630003b70 con 0x7ff63c103b90 2024-04-30T08:34:17.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.712+0000 7ff63a000640 1 -- 172.21.15.32:0/671012905 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff630003d10 con 0x7ff63c103b90 2024-04-30T08:34:17.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.712+0000 7ff63a000640 1 -- 172.21.15.32:0/671012905 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff63002fa00 con 0x7ff63c103b90 2024-04-30T08:34:17.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.712+0000 7ff644600640 1 -- 172.21.15.32:0/671012905 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff63c1a2130 con 0x7ff63c103b90 2024-04-30T08:34:17.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.712+0000 7ff644600640 1 -- 172.21.15.32:0/671012905 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff63c1a2680 con 0x7ff63c103b90 2024-04-30T08:34:17.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.716+0000 7ff64b59e740 1 Processor -- start 2024-04-30T08:34:17.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.716+0000 7ff64b59e740 1 -- start start 2024-04-30T08:34:17.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.717+0000 7ff64b59e740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f320d83270 0x55f321129750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:17.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.717+0000 7ff64b59e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f320d5a3b0 0x55f321129cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:17.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.717+0000 7ff64b59e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f320d62590 0x55f32112a960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:17.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.717+0000 7ff64b59e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f321089790 con 0x55f320d5a3b0 2024-04-30T08:34:17.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.717+0000 7ff64b59e740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55f32102ef20 con 0x55f320d83270 2024-04-30T08:34:17.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.717+0000 7ff64b59e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f321085750 con 0x55f320d62590 2024-04-30T08:34:17.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.717+0000 7ff642600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f320d83270 0x55f321129750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:17.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.717+0000 7ff642600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f320d83270 0x55f321129750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:55446/0 (socket says 172.21.15.32:55446) 2024-04-30T08:34:17.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.717+0000 7ff642600640 1 -- 172.21.15.32:0/3100518014 learned_addr learned my addr 172.21.15.32:0/3100518014 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:17.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.717+0000 7ff641200640 1 --2- 172.21.15.32:0/3100518014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f320d62590 0x55f32112a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:17.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.717+0000 7ff63a000640 1 -- 172.21.15.32:0/671012905 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ff630004580 con 0x7ff63c103b90 2024-04-30T08:34:17.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.717+0000 7ff642600640 1 -- 172.21.15.32:0/3100518014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f320d62590 msgr2=0x55f32112a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:17.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.718+0000 7ff642600640 1 --2- 172.21.15.32:0/3100518014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f320d62590 0x55f32112a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.718+0000 7ff641c00640 1 --2- 172.21.15.32:0/3100518014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f320d5a3b0 0x55f321129cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:17.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.718+0000 7ff642600640 1 -- 172.21.15.32:0/3100518014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f320d5a3b0 msgr2=0x55f321129cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:17.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.718+0000 7ff642600640 1 --2- 172.21.15.32:0/3100518014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f320d5a3b0 0x55f321129cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.718+0000 7ff642600640 1 -- 172.21.15.32:0/3100518014 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f320d9b2d0 con 0x55f320d83270 2024-04-30T08:34:17.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.718+0000 7ff641200640 1 --2- 172.21.15.32:0/3100518014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f320d62590 0x55f32112a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:34:17.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.718+0000 7ff641c00640 1 --2- 172.21.15.32:0/3100518014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f320d5a3b0 0x55f321129cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-30T08:34:17.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.718+0000 7ff642600640 1 --2- 172.21.15.32:0/3100518014 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f320d83270 0x55f321129750 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff634009c90 tx=0x7ff634011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87dca9a6309b52e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:17.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.718+0000 7ff62ea00640 1 -- 172.21.15.32:0/3100518014 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff634004230 con 0x55f320d83270 2024-04-30T08:34:17.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.718+0000 7ff62ea00640 1 -- 172.21.15.32:0/3100518014 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff634009e40 con 0x55f320d83270 2024-04-30T08:34:17.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.718+0000 7ff62ea00640 1 -- 172.21.15.32:0/3100518014 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff634019760 con 0x55f320d83270 2024-04-30T08:34:17.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.719+0000 7ff63a000640 1 --2- 172.21.15.32:0/671012905 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff6240afaf0 0x7ff6240b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:17.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.719+0000 7ff63a000640 1 -- 172.21.15.32:0/671012905 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7ff630100280 con 0x7ff63c103b90 2024-04-30T08:34:17.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.719+0000 7ff64b59e740 1 -- 172.21.15.32:0/3100518014 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f320d83270 msgr2=0x55f321129750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:17.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.719+0000 7ff64b59e740 1 --2- 172.21.15.32:0/3100518014 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f320d83270 0x55f321129750 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff634009c90 tx=0x7ff634011810 comp rx=0 tx=0).stop 2024-04-30T08:34:17.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.719+0000 7ff641c00640 1 --2- 172.21.15.32:0/671012905 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff6240afaf0 0x7ff6240b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:17.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.719+0000 7ff64b59e740 1 -- 172.21.15.32:0/3100518014 shutdown_connections 2024-04-30T08:34:17.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.719+0000 7ff64b59e740 1 --2- 172.21.15.32:0/3100518014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f320d62590 0x55f32112a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.719+0000 7ff64b59e740 1 --2- 172.21.15.32:0/3100518014 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f320d5a3b0 0x55f321129cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.719+0000 7ff64b59e740 1 --2- 172.21.15.32:0/3100518014 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f320d83270 0x55f321129750 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.719+0000 7ff64b59e740 1 -- 172.21.15.32:0/3100518014 >> 172.21.15.32:0/3100518014 conn(0x55f320d773c0 msgr2=0x55f321120b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:17.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.719+0000 7ff64b59e740 1 -- 172.21.15.32:0/3100518014 shutdown_connections 2024-04-30T08:34:17.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.719+0000 7ff64b59e740 1 -- 172.21.15.32:0/3100518014 wait complete. 2024-04-30T08:34:17.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.720+0000 7ff641c00640 1 --2- 172.21.15.32:0/671012905 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff6240afaf0 0x7ff6240b2000 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff630000f80 tx=0x7ff630031b10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:17.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.720+0000 7ff64b59e740 1 -- start start 2024-04-30T08:34:17.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.720+0000 7ff64b59e740 1 Processor -- start 2024-04-30T08:34:17.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.721+0000 7ff64b59e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f320d5a3b0 0x55f32112ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:17.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.721+0000 7ff64b59e740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f320d62590 0x55f32112f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:17.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.721+0000 7ff64b59e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f320d83270 0x55f321130160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:17.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.721+0000 7ff64b59e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f32112b470 con 0x55f320d83270 2024-04-30T08:34:17.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.721+0000 7ff64b59e740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55f3210429c0 con 0x55f320d62590 2024-04-30T08:34:17.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.721+0000 7ff64b59e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f32112b5f0 con 0x55f320d5a3b0 2024-04-30T08:34:17.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.721+0000 7ff64b59e740 10 client.0 _ll_register_callbacks cb 0x55f32100f120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:34:17.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.721+0000 7ff642600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f320d83270 0x55f321130160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:17.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.721+0000 7ff642600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f320d83270 0x55f321130160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56916/0 (socket says 172.21.15.32:56916) 2024-04-30T08:34:17.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.721+0000 7ff642600640 1 -- 172.21.15.32:0/3020097762 learned_addr learned my addr 172.21.15.32:0/3020097762 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:17.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.721+0000 7ff641200640 1 --2- 172.21.15.32:0/3020097762 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f320d62590 0x55f32112f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:17.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.721+0000 7ff642600640 1 --2- 172.21.15.32:0/3020097762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f320d5a3b0 0x55f32112ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:17.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.722+0000 7ff641200640 1 -- 172.21.15.32:0/3020097762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f320d5a3b0 msgr2=0x55f32112ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:17.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.722+0000 7ff641200640 1 --2- 172.21.15.32:0/3020097762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f320d5a3b0 0x55f32112ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.722+0000 7ff641200640 1 -- 172.21.15.32:0/3020097762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f320d83270 msgr2=0x55f321130160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:17.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.722+0000 7ff641200640 1 --2- 172.21.15.32:0/3020097762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f320d83270 0x55f321130160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.722+0000 7ff641200640 1 -- 172.21.15.32:0/3020097762 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f320e656b0 con 0x55f320d62590 2024-04-30T08:34:17.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.722+0000 7ff641200640 1 --2- 172.21.15.32:0/3020097762 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f320d62590 0x55f32112f230 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff63c19b400 tx=0x7ff628012b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d6ef3bb28cb7060e server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:17.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.722+0000 7ff623400640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:34:17.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.722+0000 7ff623400640 1 -- 172.21.15.32:0/3020097762 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff628004030 con 0x55f320d62590 2024-04-30T08:34:17.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.722+0000 7ff64b59e740 1 -- 172.21.15.32:0/3020097762 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55f3210973a0 con 0x55f320d62590 2024-04-30T08:34:17.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.722+0000 7ff623400640 1 -- 172.21.15.32:0/3020097762 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7ff6280041d0 con 0x55f320d62590 2024-04-30T08:34:17.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.723+0000 7ff623400640 1 -- 172.21.15.32:0/3020097762 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff628025050 con 0x55f320d62590 2024-04-30T08:34:17.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.723+0000 7ff64b59e740 10 client.25184 fetch_fsmap learned FSMap version 27 2024-04-30T08:34:17.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.723+0000 7ff64b59e740 1 -- 172.21.15.32:0/3020097762 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f321141b70 con 0x55f320d62590 2024-04-30T08:34:17.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.723+0000 7ff623400640 1 -- 172.21.15.32:0/3020097762 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55f3210973a0 con 0x55f320d62590 2024-04-30T08:34:17.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.723+0000 7ff623400640 1 -- 172.21.15.32:0/3020097762 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7ff628003900 con 0x55f320d62590 2024-04-30T08:34:17.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.723+0000 7ff64b59e740 10 client.25184 fetch_fsmap finished waiting for FSMap version 27 2024-04-30T08:34:17.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.723+0000 7ff64b59e740 10 client.25184 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-30T08:34:17.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.723+0000 7ff64b59e740 20 client.25184 populate_metadata read hostname 'smithi032' 2024-04-30T08:34:17.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.723+0000 7ff64b59e740 1 --2- 172.21.15.32:0/3020097762 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x55f320d41e40 0x55f3211455f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:17.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.723+0000 7ff64b59e740 4 client.25184 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:34:17.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.723+0000 7ff64b59e740 1 -- 172.21.15.32:0/3020097762 --> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f321091210 con 0x55f320d41e40 2024-04-30T08:34:17.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.724+0000 7ff642600640 1 --2- 172.21.15.32:0/3020097762 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x55f320d41e40 0x55f3211455f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:17.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.724+0000 7ff642600640 1 --2- 172.21.15.32:0/3020097762 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x55f320d41e40 0x55f3211455f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ab1566e9e7dc3a52 server_cookie=377ad66d653cc05c in_seq=0 out_seq=0 2024-04-30T08:34:17.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.724+0000 7ff623400640 10 client.25184 ms_handle_connect on v2:172.21.15.189:6836/686520873 2024-04-30T08:34:17.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.725+0000 7ff623400640 1 -- 172.21.15.32:0/3020097762 <== mds.4 v2:172.21.15.189:6836/686520873 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f321091210 con 0x55f320d41e40 2024-04-30T08:34:17.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.725+0000 7ff623400640 10 client.25184 handle_command_reply: tid=0 2024-04-30T08:34:17.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.725+0000 7ff64b59e740 1 client.25184 shutdown 2024-04-30T08:34:17.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.726+0000 7ff64b59e740 1 -- 172.21.15.32:0/3020097762 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f320d62590 msgr2=0x55f32112f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:17.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.726+0000 7ff64b59e740 1 --2- 172.21.15.32:0/3020097762 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f320d62590 0x55f32112f230 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff63c19b400 tx=0x7ff628012b60 comp rx=0 tx=0).stop 2024-04-30T08:34:17.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.726+0000 7ff64b59e740 1 -- 172.21.15.32:0/3020097762 shutdown_connections 2024-04-30T08:34:17.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.726+0000 7ff64b59e740 1 --2- 172.21.15.32:0/3020097762 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x55f320d41e40 0x55f3211455f0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.726+0000 7ff64b59e740 1 --2- 172.21.15.32:0/3020097762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f320d83270 0x55f321130160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.726+0000 7ff64b59e740 1 --2- 172.21.15.32:0/3020097762 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f320d62590 0x55f32112f230 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.726+0000 7ff64b59e740 1 --2- 172.21.15.32:0/3020097762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f320d5a3b0 0x55f32112ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.726+0000 7ff64b59e740 1 -- 172.21.15.32:0/3020097762 >> 172.21.15.32:0/3020097762 conn(0x55f320d773c0 msgr2=0x55f32111fbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:17.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.726+0000 7ff623400640 0 client.25184 ms_handle_reset on v2:172.21.15.189:6836/686520873 2024-04-30T08:34:17.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.726+0000 7ff64b59e740 1 -- 172.21.15.32:0/3020097762 shutdown_connections 2024-04-30T08:34:17.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.726+0000 7ff64b59e740 1 -- 172.21.15.32:0/3020097762 wait complete. 2024-04-30T08:34:17.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.726+0000 7ff64b59e740 20 client.25184 trim_cache size 0 max 16384 2024-04-30T08:34:17.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.747+0000 7ff64b59e740 1 Processor -- start 2024-04-30T08:34:17.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.747+0000 7ff64b59e740 1 -- start start 2024-04-30T08:34:17.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.747+0000 7ff64b59e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f320d5a3b0 0x55f321128eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:17.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.747+0000 7ff64b59e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f320d62590 0x55f32112d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:17.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.747+0000 7ff64b59e740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f320d83270 0x55f321129770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:17.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.747+0000 7ff64b59e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f321143930 con 0x55f320d62590 2024-04-30T08:34:17.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.747+0000 7ff64b59e740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55f321144000 con 0x55f320d83270 2024-04-30T08:34:17.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.747+0000 7ff64b59e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f3211446d0 con 0x55f320d5a3b0 2024-04-30T08:34:17.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.748+0000 7ff641200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f320d62590 0x55f32112d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:17.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.748+0000 7ff641200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f320d62590 0x55f32112d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56922/0 (socket says 172.21.15.32:56922) 2024-04-30T08:34:17.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.748+0000 7ff641200640 1 -- 172.21.15.32:0/3534853170 learned_addr learned my addr 172.21.15.32:0/3534853170 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:17.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.748+0000 7ff642600640 1 --2- 172.21.15.32:0/3534853170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f320d5a3b0 0x55f321128eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:17.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.748+0000 7ff642600640 1 --2- 172.21.15.32:0/3534853170 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f320d83270 0x55f321129770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:17.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.748+0000 7ff641200640 1 -- 172.21.15.32:0/3534853170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f320d5a3b0 msgr2=0x55f321128eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:17.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.748+0000 7ff641200640 1 --2- 172.21.15.32:0/3534853170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f320d5a3b0 0x55f321128eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.748+0000 7ff641200640 1 -- 172.21.15.32:0/3534853170 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f320d83270 msgr2=0x55f321129770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:17.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.748+0000 7ff641200640 1 --2- 172.21.15.32:0/3534853170 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f320d83270 0x55f321129770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.748+0000 7ff641200640 1 -- 172.21.15.32:0/3534853170 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f32112a750 con 0x55f320d62590 2024-04-30T08:34:17.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.748+0000 7ff642600640 1 --2- 172.21.15.32:0/3534853170 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f320d83270 0x55f321129770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-30T08:34:17.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.748+0000 7ff641200640 1 --2- 172.21.15.32:0/3534853170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f320d62590 0x55f32112d980 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff6280059f0 tx=0x7ff628010660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b1dc201f0e167a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:17.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.749+0000 7ff62e000640 1 -- 172.21.15.32:0/3534853170 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff628022040 con 0x55f320d62590 2024-04-30T08:34:17.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.749+0000 7ff62e000640 1 -- 172.21.15.32:0/3534853170 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff628003aa0 con 0x55f320d62590 2024-04-30T08:34:17.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.749+0000 7ff62e000640 1 -- 172.21.15.32:0/3534853170 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff628005360 con 0x55f320d62590 2024-04-30T08:34:17.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.749+0000 7ff64b59e740 1 -- 172.21.15.32:0/3534853170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f320d62590 msgr2=0x55f32112d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:17.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.749+0000 7ff64b59e740 1 --2- 172.21.15.32:0/3534853170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f320d62590 0x55f32112d980 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff6280059f0 tx=0x7ff628010660 comp rx=0 tx=0).stop 2024-04-30T08:34:17.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.750+0000 7ff64b59e740 1 -- 172.21.15.32:0/3534853170 shutdown_connections 2024-04-30T08:34:17.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.750+0000 7ff64b59e740 1 --2- 172.21.15.32:0/3534853170 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f320d83270 0x55f321129770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.750+0000 7ff64b59e740 1 --2- 172.21.15.32:0/3534853170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f320d62590 0x55f32112d980 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.750+0000 7ff64b59e740 1 --2- 172.21.15.32:0/3534853170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f320d5a3b0 0x55f321128eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.750+0000 7ff64b59e740 1 -- 172.21.15.32:0/3534853170 >> 172.21.15.32:0/3534853170 conn(0x55f320d773c0 msgr2=0x55f32112bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:17.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.750+0000 7ff64b59e740 1 -- 172.21.15.32:0/3534853170 shutdown_connections 2024-04-30T08:34:17.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.750+0000 7ff64b59e740 1 -- 172.21.15.32:0/3534853170 wait complete. 2024-04-30T08:34:17.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.750+0000 7ff64b59e740 1 -- start start 2024-04-30T08:34:17.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.750+0000 7ff64b59e740 1 Processor -- start 2024-04-30T08:34:17.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.751+0000 7ff64b59e740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f320d5a3b0 0x55f32113d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:17.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.751+0000 7ff64b59e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f320d62590 0x55f32113d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:17.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.751+0000 7ff64b59e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f320d83270 0x55f32113e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:17.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.751+0000 7ff64b59e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f321145470 con 0x55f320d83270 2024-04-30T08:34:17.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.751+0000 7ff64b59e740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55f321144da0 con 0x55f320d5a3b0 2024-04-30T08:34:17.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.751+0000 7ff64b59e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f321145b40 con 0x55f320d62590 2024-04-30T08:34:17.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.751+0000 7ff64b59e740 10 client.0 _ll_register_callbacks cb 0x7ff628009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:34:17.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.751+0000 7ff642600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f320d83270 0x55f32113e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:17.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.751+0000 7ff642600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f320d5a3b0 0x55f32113d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:17.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.751+0000 7ff642600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f320d5a3b0 0x55f32113d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:55464/0 (socket says 172.21.15.32:55464) 2024-04-30T08:34:17.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.751+0000 7ff642600640 1 -- 172.21.15.32:0/2926420414 learned_addr learned my addr 172.21.15.32:0/2926420414 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:17.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.751+0000 7ff641200640 1 --2- 172.21.15.32:0/2926420414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f320d62590 0x55f32113d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:17.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.752+0000 7ff641200640 1 -- 172.21.15.32:0/2926420414 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f320d5a3b0 msgr2=0x55f32113d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:17.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.752+0000 7ff641200640 1 --2- 172.21.15.32:0/2926420414 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f320d5a3b0 0x55f32113d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.752+0000 7ff641200640 1 -- 172.21.15.32:0/2926420414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f320d83270 msgr2=0x55f32113e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:17.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.752+0000 7ff641200640 1 --2- 172.21.15.32:0/2926420414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f320d83270 0x55f32113e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.752+0000 7ff641200640 1 -- 172.21.15.32:0/2926420414 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f32113ecd0 con 0x55f320d62590 2024-04-30T08:34:17.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.752+0000 7ff641200640 1 --2- 172.21.15.32:0/2926420414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f320d62590 0x55f32113d830 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x55f321145440 tx=0x7ff628003800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8c794d9aadc8d9cb server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:17.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.752+0000 7ff623400640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-30T08:34:17.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.752+0000 7ff623400640 1 -- 172.21.15.32:0/2926420414 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff628009ea0 con 0x55f320d62590 2024-04-30T08:34:17.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.752+0000 7ff64b59e740 1 -- 172.21.15.32:0/2926420414 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff628003aa0 con 0x55f320d62590 2024-04-30T08:34:17.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.752+0000 7ff623400640 1 -- 172.21.15.32:0/2926420414 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7ff62800fb30 con 0x55f320d62590 2024-04-30T08:34:17.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.753+0000 7ff623400640 1 -- 172.21.15.32:0/2926420414 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7ff628010430 con 0x55f320d62590 2024-04-30T08:34:17.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.753+0000 7ff64b59e740 10 client.25189 fetch_fsmap learned FSMap version 27 2024-04-30T08:34:17.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.753+0000 7ff64b59e740 1 -- 172.21.15.32:0/2926420414 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55f32113fec0 con 0x55f320d62590 2024-04-30T08:34:17.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.753+0000 7ff623400640 1 -- 172.21.15.32:0/2926420414 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff628003aa0 con 0x55f320d62590 2024-04-30T08:34:17.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.753+0000 7ff623400640 1 -- 172.21.15.32:0/2926420414 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7ff628010b00 con 0x55f320d62590 2024-04-30T08:34:17.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.753+0000 7ff64b59e740 10 client.25189 fetch_fsmap finished waiting for FSMap version 27 2024-04-30T08:34:17.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.753+0000 7ff64b59e740 10 client.25189 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-30T08:34:17.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.753+0000 7ff64b59e740 20 client.25189 populate_metadata read hostname 'smithi032' 2024-04-30T08:34:17.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.753+0000 7ff64b59e740 1 --2- 172.21.15.32:0/2926420414 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x55f320d17280 0x55f3211ae470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:17.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.753+0000 7ff64b59e740 4 client.25189 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-30T08:34:17.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.753+0000 7ff64b59e740 1 -- 172.21.15.32:0/2926420414 --> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] -- command(tid 0: {"prefix": "status"}) -- 0x55f32100c860 con 0x55f320d17280 2024-04-30T08:34:17.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.753+0000 7ff642600640 1 --2- 172.21.15.32:0/2926420414 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x55f320d17280 0x55f3211ae470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:17.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.754+0000 7ff642600640 1 --2- 172.21.15.32:0/2926420414 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x55f320d17280 0x55f3211ae470 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=dd422c0551aafd89 server_cookie=be5f30123d202d50 in_seq=0 out_seq=0 2024-04-30T08:34:17.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.754+0000 7ff623400640 10 client.25189 ms_handle_connect on v2:172.21.15.189:6836/686520873 2024-04-30T08:34:17.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.754+0000 7ff623400640 1 -- 172.21.15.32:0/2926420414 <== mds.4 v2:172.21.15.189:6836/686520873 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x55f32100c860 con 0x55f320d17280 2024-04-30T08:34:17.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.754+0000 7ff623400640 10 client.25189 handle_command_reply: tid=0 2024-04-30T08:34:17.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.754+0000 7ff64b59e740 1 client.25189 shutdown 2024-04-30T08:34:17.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.755+0000 7ff64b59e740 1 -- 172.21.15.32:0/2926420414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f320d62590 msgr2=0x55f32113d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:17.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.755+0000 7ff64b59e740 1 --2- 172.21.15.32:0/2926420414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f320d62590 0x55f32113d830 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x55f321145440 tx=0x7ff628003800 comp rx=0 tx=0).stop 2024-04-30T08:34:17.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.755+0000 7ff64b59e740 1 -- 172.21.15.32:0/2926420414 shutdown_connections 2024-04-30T08:34:17.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.755+0000 7ff64b59e740 1 --2- 172.21.15.32:0/2926420414 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x55f320d17280 0x55f3211ae470 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.755+0000 7ff64b59e740 1 --2- 172.21.15.32:0/2926420414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f320d83270 0x55f32113e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.755+0000 7ff64b59e740 1 --2- 172.21.15.32:0/2926420414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f320d62590 0x55f32113d830 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.755+0000 7ff64b59e740 1 --2- 172.21.15.32:0/2926420414 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f320d5a3b0 0x55f32113d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.755+0000 7ff623400640 0 client.25189 ms_handle_reset on v2:172.21.15.189:6836/686520873 2024-04-30T08:34:17.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.755+0000 7ff64b59e740 1 -- 172.21.15.32:0/2926420414 >> 172.21.15.32:0/2926420414 conn(0x55f320d773c0 msgr2=0x55f321124380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:17.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.755+0000 7ff64b59e740 1 -- 172.21.15.32:0/2926420414 shutdown_connections 2024-04-30T08:34:17.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.755+0000 7ff64b59e740 1 -- 172.21.15.32:0/2926420414 wait complete. 2024-04-30T08:34:17.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.755+0000 7ff64b59e740 20 client.25189 trim_cache size 0 max 16384 2024-04-30T08:34:17.761 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-30T08:34:17.761 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "98c0a670-06ca-11ef-bc93-c7b262605968", 2024-04-30T08:34:17.761 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 4, 2024-04-30T08:34:17.761 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 24517, 2024-04-30T08:34:17.762 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-04-30T08:34:17.762 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-04-30T08:34:17.762 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-04-30T08:34:17.762 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 108.222554231, 2024-04-30T08:34:17.762 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 27, 2024-04-30T08:34:17.762 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 85, 2024-04-30T08:34:17.762 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 81, 2024-04-30T08:34:17.762 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 170.18147903400001 2024-04-30T08:34:17.762 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-30T08:34:17.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.756+0000 7ff644600640 1 -- 172.21.15.32:0/671012905 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff6240afaf0 msgr2=0x7ff6240b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:17.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.756+0000 7ff644600640 1 --2- 172.21.15.32:0/671012905 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff6240afaf0 0x7ff6240b2000 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff630000f80 tx=0x7ff630031b10 comp rx=0 tx=0).stop 2024-04-30T08:34:17.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.756+0000 7ff644600640 1 -- 172.21.15.32:0/671012905 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff63c103b90 msgr2=0x7ff63c19a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:17.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.756+0000 7ff644600640 1 --2- 172.21.15.32:0/671012905 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff63c103b90 0x7ff63c19a360 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff630005e30 tx=0x7ff6300317e0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.756+0000 7ff644600640 1 -- 172.21.15.32:0/671012905 shutdown_connections 2024-04-30T08:34:17.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.756+0000 7ff644600640 1 --2- 172.21.15.32:0/671012905 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff6240afaf0 0x7ff6240b2000 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.756+0000 7ff644600640 1 --2- 172.21.15.32:0/671012905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff63c109ad0 0x7ff63c1a1830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.756+0000 7ff644600640 1 --2- 172.21.15.32:0/671012905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff63c104dd0 0x7ff63c19a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.756+0000 7ff644600640 1 --2- 172.21.15.32:0/671012905 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff63c103b90 0x7ff63c19a360 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:17.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.756+0000 7ff644600640 1 -- 172.21.15.32:0/671012905 >> 172.21.15.32:0/671012905 conn(0x7ff63c0fedc0 msgr2=0x7ff63c105ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:17.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.756+0000 7ff644600640 1 -- 172.21.15.32:0/671012905 shutdown_connections 2024-04-30T08:34:17.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:17.756+0000 7ff644600640 1 -- 172.21.15.32:0/671012905 wait complete. 2024-04-30T08:34:17.787 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-30T08:34:18.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.066+0000 7faeb9600640 1 -- 172.21.15.32:0/2593535696 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faeb4076a70 msgr2=0x7faeb410b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:18.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.066+0000 7faeb9600640 1 --2- 172.21.15.32:0/2593535696 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faeb4076a70 0x7faeb410b5a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fae9c009860 tx=0x7fae9c02a340 comp rx=0 tx=0).stop 2024-04-30T08:34:18.068 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.066+0000 7faeb9600640 1 -- 172.21.15.32:0/2593535696 shutdown_connections 2024-04-30T08:34:18.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.066+0000 7faeb9600640 1 --2- 172.21.15.32:0/2593535696 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faeb410bae0 0x7faeb410df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.066+0000 7faeb9600640 1 --2- 172.21.15.32:0/2593535696 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faeb4076a70 0x7faeb410b5a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.066+0000 7faeb9600640 1 --2- 172.21.15.32:0/2593535696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faeb4076120 0x7faeb4076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.066+0000 7faeb9600640 1 -- 172.21.15.32:0/2593535696 >> 172.21.15.32:0/2593535696 conn(0x7faeb40fed60 msgr2=0x7faeb4101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:18.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.066+0000 7faeb9600640 1 -- 172.21.15.32:0/2593535696 shutdown_connections 2024-04-30T08:34:18.069 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.067+0000 7faeb9600640 1 -- 172.21.15.32:0/2593535696 wait complete. 2024-04-30T08:34:18.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.068+0000 7faeb9600640 1 Processor -- start 2024-04-30T08:34:18.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.068+0000 7faeb9600640 1 -- start start 2024-04-30T08:34:18.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.068+0000 7faeb9600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faeb4076120 0x7faeb419c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.068+0000 7faeb9600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faeb4076a70 0x7faeb419c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.068+0000 7faeb9600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faeb410bae0 0x7faeb41a3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.068+0000 7faeb9600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7faeb41118a0 con 0x7faeb4076120 2024-04-30T08:34:18.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.068+0000 7faeb9600640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7faeb4111720 con 0x7faeb410bae0 2024-04-30T08:34:18.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.068+0000 7faeb9600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7faeb4111a20 con 0x7faeb4076a70 2024-04-30T08:34:18.071 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.069+0000 7faeb2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faeb4076120 0x7faeb419c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.069+0000 7faeb2000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faeb4076a70 0x7faeb419c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.069+0000 7faeb2000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faeb4076a70 0x7faeb419c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:59514/0 (socket says 172.21.15.32:59514) 2024-04-30T08:34:18.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.069+0000 7faeb2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faeb4076120 0x7faeb419c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56954/0 (socket says 172.21.15.32:56954) 2024-04-30T08:34:18.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.069+0000 7faeb2000640 1 -- 172.21.15.32:0/2825565172 learned_addr learned my addr 172.21.15.32:0/2825565172 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:18.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.069+0000 7faeb3400640 1 --2- 172.21.15.32:0/2825565172 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faeb410bae0 0x7faeb41a3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.069+0000 7faeb2a00640 1 -- 172.21.15.32:0/2825565172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faeb4076a70 msgr2=0x7faeb419c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:18.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.069+0000 7faeb2a00640 1 --2- 172.21.15.32:0/2825565172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faeb4076a70 0x7faeb419c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.069+0000 7faeb2a00640 1 -- 172.21.15.32:0/2825565172 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faeb410bae0 msgr2=0x7faeb41a3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:18.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.069+0000 7faeb2a00640 1 --2- 172.21.15.32:0/2825565172 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faeb410bae0 0x7faeb41a3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.069+0000 7faeb2a00640 1 -- 172.21.15.32:0/2825565172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faeb41a3ff0 con 0x7faeb4076120 2024-04-30T08:34:18.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.069+0000 7faeb2000640 1 --2- 172.21.15.32:0/2825565172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faeb4076a70 0x7faeb419c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:34:18.072 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.070+0000 7faeb2a00640 1 --2- 172.21.15.32:0/2825565172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faeb4076120 0x7faeb419c4b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7faea400e9c0 tx=0x7faea400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:18.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.070+0000 7faeab400640 1 -- 172.21.15.32:0/2825565172 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea400ccc0 con 0x7faeb4076120 2024-04-30T08:34:18.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.070+0000 7faeb9600640 1 -- 172.21.15.32:0/2825565172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faeb41a42e0 con 0x7faeb4076120 2024-04-30T08:34:18.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.070+0000 7faeab400640 1 -- 172.21.15.32:0/2825565172 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faea40108e0 con 0x7faeb4076120 2024-04-30T08:34:18.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.070+0000 7faeab400640 1 -- 172.21.15.32:0/2825565172 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faea40187a0 con 0x7faeb4076120 2024-04-30T08:34:18.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.070+0000 7faeb9600640 1 -- 172.21.15.32:0/2825565172 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faeb41a4830 con 0x7faeb4076120 2024-04-30T08:34:18.075 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.074+0000 7faec05f1740 1 Processor -- start 2024-04-30T08:34:18.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.075+0000 7faec05f1740 1 -- start start 2024-04-30T08:34:18.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.075+0000 7faec05f1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e1d972270 0x564e1dd14340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.075+0000 7faec05f1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e1d9493b0 0x564e1dd148c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.075+0000 7faec05f1740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564e1d951590 0x564e1dd15550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.075+0000 7faec05f1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564e1dc78790 con 0x564e1d972270 2024-04-30T08:34:18.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.075+0000 7faec05f1740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x564e1dc1df20 con 0x564e1d951590 2024-04-30T08:34:18.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.075+0000 7faec05f1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564e1dc74750 con 0x564e1d9493b0 2024-04-30T08:34:18.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.075+0000 7faeb3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e1d972270 0x564e1dd14340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.076+0000 7faeb3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e1d972270 0x564e1dd14340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56970/0 (socket says 172.21.15.32:56970) 2024-04-30T08:34:18.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.076+0000 7faeb3400640 1 -- 172.21.15.32:0/1215645413 learned_addr learned my addr 172.21.15.32:0/1215645413 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:18.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.076+0000 7faeab400640 1 -- 172.21.15.32:0/2825565172 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7faea4010430 con 0x7faeb4076120 2024-04-30T08:34:18.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.076+0000 7faeb2000640 1 --2- 172.21.15.32:0/1215645413 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564e1d951590 0x564e1dd15550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.076+0000 7faeb3400640 1 -- 172.21.15.32:0/1215645413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e1d9493b0 msgr2=0x564e1dd148c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:18.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.076+0000 7faeb3400640 1 --2- 172.21.15.32:0/1215645413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e1d9493b0 0x564e1dd148c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.076+0000 7faeb3400640 1 -- 172.21.15.32:0/1215645413 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564e1d951590 msgr2=0x564e1dd15550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:18.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.076+0000 7faeb3400640 1 --2- 172.21.15.32:0/1215645413 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564e1d951590 0x564e1dd15550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.076+0000 7faeb3400640 1 -- 172.21.15.32:0/1215645413 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e1d98a2d0 con 0x564e1d972270 2024-04-30T08:34:18.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.077+0000 7faeb3400640 1 --2- 172.21.15.32:0/1215645413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e1d972270 0x564e1dd14340 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7faeac00e4d0 tx=0x7faeac0095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b00b7ff2c5cf1703 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:18.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.077+0000 7faea3400640 1 -- 172.21.15.32:0/1215645413 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeac003970 con 0x564e1d972270 2024-04-30T08:34:18.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.077+0000 7faea3400640 1 -- 172.21.15.32:0/1215645413 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faeac00d770 con 0x564e1d972270 2024-04-30T08:34:18.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.077+0000 7faea3400640 1 -- 172.21.15.32:0/1215645413 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeac004420 con 0x564e1d972270 2024-04-30T08:34:18.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.078+0000 7faeab400640 1 --2- 172.21.15.32:0/2825565172 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fae980afaf0 0x7fae980b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.078+0000 7faec05f1740 1 -- 172.21.15.32:0/1215645413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e1d972270 msgr2=0x564e1dd14340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:18.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.078+0000 7faec05f1740 1 --2- 172.21.15.32:0/1215645413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e1d972270 0x564e1dd14340 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7faeac00e4d0 tx=0x7faeac0095c0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.078+0000 7faec05f1740 1 -- 172.21.15.32:0/1215645413 shutdown_connections 2024-04-30T08:34:18.086 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.078+0000 7faec05f1740 1 --2- 172.21.15.32:0/1215645413 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564e1d951590 0x564e1dd15550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.078+0000 7faec05f1740 1 --2- 172.21.15.32:0/1215645413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e1d9493b0 0x564e1dd148c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.078+0000 7faec05f1740 1 --2- 172.21.15.32:0/1215645413 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e1d972270 0x564e1dd14340 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.078+0000 7faec05f1740 1 -- 172.21.15.32:0/1215645413 >> 172.21.15.32:0/1215645413 conn(0x564e1d9663c0 msgr2=0x564e1dd0fb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:18.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.078+0000 7faeab400640 1 -- 172.21.15.32:0/2825565172 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7faea40df350 con 0x7faeb4076120 2024-04-30T08:34:18.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.078+0000 7faec05f1740 1 -- 172.21.15.32:0/1215645413 shutdown_connections 2024-04-30T08:34:18.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.078+0000 7faec05f1740 1 -- 172.21.15.32:0/1215645413 wait complete. 2024-04-30T08:34:18.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.078+0000 7faeb2a00640 1 --2- 172.21.15.32:0/2825565172 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fae980afaf0 0x7fae980b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.079+0000 7faec05f1740 1 -- start start 2024-04-30T08:34:18.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.079+0000 7faeb2a00640 1 --2- 172.21.15.32:0/2825565172 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fae980afaf0 0x7fae980b2000 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7faea40df5f0 tx=0x7faea4009700 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:18.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.079+0000 7faec05f1740 1 Processor -- start 2024-04-30T08:34:18.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.080+0000 7faec05f1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e1d9493b0 0x564e1dd1de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.080+0000 7faec05f1740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564e1d951590 0x564e1dd1e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.080+0000 7faec05f1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e1d972270 0x564e1dd252b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.080+0000 7faec05f1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564e1dd160b0 con 0x564e1d9493b0 2024-04-30T08:34:18.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.080+0000 7faec05f1740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x564e1dc319c0 con 0x564e1d951590 2024-04-30T08:34:18.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.080+0000 7faec05f1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564e1dd16230 con 0x564e1d972270 2024-04-30T08:34:18.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.080+0000 7faec05f1740 10 client.0 _ll_register_callbacks cb 0x564e1dbfe120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:34:18.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.080+0000 7faeb3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e1d9493b0 0x564e1dd1de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.080+0000 7faeb3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e1d9493b0 0x564e1dd1de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56982/0 (socket says 172.21.15.32:56982) 2024-04-30T08:34:18.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.080+0000 7faeb3400640 1 -- 172.21.15.32:0/1402566011 learned_addr learned my addr 172.21.15.32:0/1402566011 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:18.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.080+0000 7faeb2000640 1 --2- 172.21.15.32:0/1402566011 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564e1d951590 0x564e1dd1e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.080+0000 7faeb3400640 1 --2- 172.21.15.32:0/1402566011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e1d972270 0x564e1dd252b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.080+0000 7faeb3400640 1 -- 172.21.15.32:0/1402566011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e1d972270 msgr2=0x564e1dd252b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:18.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.080+0000 7faeb3400640 1 --2- 172.21.15.32:0/1402566011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e1d972270 0x564e1dd252b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.080+0000 7faeb3400640 1 -- 172.21.15.32:0/1402566011 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564e1d951590 msgr2=0x564e1dd1e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:18.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.080+0000 7faeb3400640 1 --2- 172.21.15.32:0/1402566011 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564e1d951590 0x564e1dd1e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.080+0000 7faeb3400640 1 -- 172.21.15.32:0/1402566011 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e1da546b0 con 0x564e1d9493b0 2024-04-30T08:34:18.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.081+0000 7faeb3400640 1 --2- 172.21.15.32:0/1402566011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e1d9493b0 0x564e1dd1de80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7faeac0097f0 tx=0x7faeac005a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e4a4063806b510c server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:18.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.081+0000 7fae97e00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:34:18.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.081+0000 7fae97e00640 1 -- 172.21.15.32:0/1402566011 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faeac00ddb0 con 0x564e1d9493b0 2024-04-30T08:34:18.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.081+0000 7fae97e00640 1 -- 172.21.15.32:0/1402566011 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7faeac0041a0 con 0x564e1d9493b0 2024-04-30T08:34:18.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.081+0000 7faec05f1740 1 -- 172.21.15.32:0/1402566011 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faeac00d770 con 0x564e1d9493b0 2024-04-30T08:34:18.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.081+0000 7fae97e00640 1 -- 172.21.15.32:0/1402566011 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7faeac0043a0 con 0x564e1d9493b0 2024-04-30T08:34:18.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.081+0000 7faec05f1740 10 client.15078 fetch_fsmap learned FSMap version 27 2024-04-30T08:34:18.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.081+0000 7faec05f1740 1 -- 172.21.15.32:0/1402566011 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564e1dd281e0 con 0x564e1d9493b0 2024-04-30T08:34:18.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.082+0000 7fae97e00640 1 -- 172.21.15.32:0/1402566011 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faeac00d770 con 0x564e1d9493b0 2024-04-30T08:34:18.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.082+0000 7fae97e00640 1 -- 172.21.15.32:0/1402566011 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7faeac00b040 con 0x564e1d9493b0 2024-04-30T08:34:18.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.082+0000 7faec05f1740 10 client.15078 fetch_fsmap finished waiting for FSMap version 27 2024-04-30T08:34:18.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.082+0000 7faec05f1740 10 client.15078 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-30T08:34:18.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.082+0000 7faec05f1740 20 client.15078 populate_metadata read hostname 'smithi032' 2024-04-30T08:34:18.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.082+0000 7faec05f1740 1 --2- 172.21.15.32:0/1402566011 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x564e1d930e40 0x564e1dd2ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.082+0000 7faec05f1740 4 client.15078 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:34:18.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.082+0000 7faec05f1740 1 -- 172.21.15.32:0/1402566011 --> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e1dc863a0 con 0x564e1d930e40 2024-04-30T08:34:18.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.083+0000 7faeb2000640 1 --2- 172.21.15.32:0/1402566011 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x564e1d930e40 0x564e1dd2ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.083+0000 7faeb2000640 1 --2- 172.21.15.32:0/1402566011 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x564e1d930e40 0x564e1dd2ba50 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=5aa3f87d1b8f705a server_cookie=b2bdd68ccfacf04b in_seq=0 out_seq=0 2024-04-30T08:34:18.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.083+0000 7fae97e00640 10 client.15078 ms_handle_connect on v2:172.21.15.71:6836/2324039932 2024-04-30T08:34:18.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.084+0000 7fae97e00640 1 -- 172.21.15.32:0/1402566011 <== mds.3 v2:172.21.15.71:6836/2324039932 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564e1dc863a0 con 0x564e1d930e40 2024-04-30T08:34:18.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.084+0000 7fae97e00640 10 client.15078 handle_command_reply: tid=0 2024-04-30T08:34:18.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.084+0000 7faec05f1740 1 client.15078 shutdown 2024-04-30T08:34:18.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.084+0000 7faec05f1740 1 -- 172.21.15.32:0/1402566011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e1d9493b0 msgr2=0x564e1dd1de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:18.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.084+0000 7faec05f1740 1 --2- 172.21.15.32:0/1402566011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e1d9493b0 0x564e1dd1de80 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7faeac0097f0 tx=0x7faeac005a70 comp rx=0 tx=0).stop 2024-04-30T08:34:18.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.085+0000 7faec05f1740 1 -- 172.21.15.32:0/1402566011 shutdown_connections 2024-04-30T08:34:18.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.085+0000 7faec05f1740 1 --2- 172.21.15.32:0/1402566011 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x564e1d930e40 0x564e1dd2ba50 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.085+0000 7faec05f1740 1 --2- 172.21.15.32:0/1402566011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e1d972270 0x564e1dd252b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.085+0000 7faec05f1740 1 --2- 172.21.15.32:0/1402566011 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564e1d951590 0x564e1dd1e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.085+0000 7faec05f1740 1 --2- 172.21.15.32:0/1402566011 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e1d9493b0 0x564e1dd1de80 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.085+0000 7fae97e00640 0 client.15078 ms_handle_reset on v2:172.21.15.71:6836/2324039932 2024-04-30T08:34:18.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.085+0000 7faec05f1740 1 -- 172.21.15.32:0/1402566011 >> 172.21.15.32:0/1402566011 conn(0x564e1d9663c0 msgr2=0x564e1dd0ebf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:18.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.085+0000 7faec05f1740 1 -- 172.21.15.32:0/1402566011 shutdown_connections 2024-04-30T08:34:18.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.085+0000 7faec05f1740 1 -- 172.21.15.32:0/1402566011 wait complete. 2024-04-30T08:34:18.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.085+0000 7faec05f1740 20 client.15078 trim_cache size 0 max 16384 2024-04-30T08:34:18.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.105+0000 7faec05f1740 1 Processor -- start 2024-04-30T08:34:18.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.106+0000 7faec05f1740 1 -- start start 2024-04-30T08:34:18.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.106+0000 7faec05f1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e1d9493b0 0x564e1dd12a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.106+0000 7faec05f1740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564e1d951590 0x564e1dd12f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.106+0000 7faec05f1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e1d972270 0x564e1dd13480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.106+0000 7faec05f1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564e1dd29d90 con 0x564e1d972270 2024-04-30T08:34:18.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.106+0000 7faec05f1740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x564e1dd2a460 con 0x564e1d951590 2024-04-30T08:34:18.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.106+0000 7faec05f1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564e1dd2ab30 con 0x564e1d9493b0 2024-04-30T08:34:18.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.106+0000 7faeb3400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e1d9493b0 0x564e1dd12a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.106+0000 7faeb2000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564e1d951590 0x564e1dd12f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.106+0000 7faeb3400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e1d9493b0 0x564e1dd12a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:59538/0 (socket says 172.21.15.32:59538) 2024-04-30T08:34:18.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.106+0000 7faeb3400640 1 -- 172.21.15.32:0/183404333 learned_addr learned my addr 172.21.15.32:0/183404333 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:18.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.106+0000 7faeb3400640 1 --2- 172.21.15.32:0/183404333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e1d972270 0x564e1dd13480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.107+0000 7faeb2000640 1 -- 172.21.15.32:0/183404333 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e1d9493b0 msgr2=0x564e1dd12a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:18.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.107+0000 7faeb2000640 1 --2- 172.21.15.32:0/183404333 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e1d9493b0 0x564e1dd12a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.107+0000 7faeb2000640 1 -- 172.21.15.32:0/183404333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e1d972270 msgr2=0x564e1dd13480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:18.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.107+0000 7faeb2000640 1 --2- 172.21.15.32:0/183404333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e1d972270 0x564e1dd13480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.107+0000 7faeb2000640 1 -- 172.21.15.32:0/183404333 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e1dd2a910 con 0x564e1d951590 2024-04-30T08:34:18.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.107+0000 7faeb2000640 1 --2- 172.21.15.32:0/183404333 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564e1d951590 0x564e1dd12f40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fae9c009830 tx=0x7fae9c030470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f02e56452390d9a server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:18.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.107+0000 7faea2a00640 1 -- 172.21.15.32:0/183404333 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae9c04e070 con 0x564e1d951590 2024-04-30T08:34:18.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.107+0000 7faea2a00640 1 -- 172.21.15.32:0/183404333 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae9c005d30 con 0x564e1d951590 2024-04-30T08:34:18.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.107+0000 7faea2a00640 1 -- 172.21.15.32:0/183404333 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae9c037040 con 0x564e1d951590 2024-04-30T08:34:18.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.108+0000 7faec05f1740 1 -- 172.21.15.32:0/183404333 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564e1d951590 msgr2=0x564e1dd12f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:18.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.108+0000 7faec05f1740 1 --2- 172.21.15.32:0/183404333 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564e1d951590 0x564e1dd12f40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fae9c009830 tx=0x7fae9c030470 comp rx=0 tx=0).stop 2024-04-30T08:34:18.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.108+0000 7faec05f1740 1 -- 172.21.15.32:0/183404333 shutdown_connections 2024-04-30T08:34:18.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.108+0000 7faec05f1740 1 --2- 172.21.15.32:0/183404333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e1d972270 0x564e1dd13480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.108+0000 7faec05f1740 1 --2- 172.21.15.32:0/183404333 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564e1d951590 0x564e1dd12f40 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.108+0000 7faec05f1740 1 --2- 172.21.15.32:0/183404333 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e1d9493b0 0x564e1dd12a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.108+0000 7faec05f1740 1 -- 172.21.15.32:0/183404333 >> 172.21.15.32:0/183404333 conn(0x564e1d9663c0 msgr2=0x564e1dd0a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:18.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.108+0000 7faec05f1740 1 -- 172.21.15.32:0/183404333 shutdown_connections 2024-04-30T08:34:18.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.108+0000 7faec05f1740 1 -- 172.21.15.32:0/183404333 wait complete. 2024-04-30T08:34:18.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.109+0000 7faec05f1740 1 -- start start 2024-04-30T08:34:18.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.109+0000 7faec05f1740 1 Processor -- start 2024-04-30T08:34:18.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.109+0000 7faec05f1740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564e1d9493b0 0x564e1dd20c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.110+0000 7faec05f1740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e1d951590 0x564e1dd21140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.110+0000 7faec05f1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e1d972270 0x564e1dd24010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.110+0000 7faec05f1740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564e1dd2b8d0 con 0x564e1d951590 2024-04-30T08:34:18.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.110+0000 7faec05f1740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x564e1dd2b200 con 0x564e1d9493b0 2024-04-30T08:34:18.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.110+0000 7faec05f1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564e1dd2bfa0 con 0x564e1d972270 2024-04-30T08:34:18.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.110+0000 7faec05f1740 10 client.0 _ll_register_callbacks cb 0x7faeac009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:34:18.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.110+0000 7faeb2000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e1d951590 0x564e1dd21140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.110+0000 7faeb2000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e1d951590 0x564e1dd21140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57002/0 (socket says 172.21.15.32:57002) 2024-04-30T08:34:18.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.110+0000 7faeb2000640 1 -- 172.21.15.32:0/752485912 learned_addr learned my addr 172.21.15.32:0/752485912 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:18.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.110+0000 7faeb3400640 1 --2- 172.21.15.32:0/752485912 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564e1d9493b0 0x564e1dd20c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.110+0000 7faeb3400640 1 --2- 172.21.15.32:0/752485912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e1d972270 0x564e1dd24010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.110+0000 7faeb2000640 1 -- 172.21.15.32:0/752485912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e1d972270 msgr2=0x564e1dd24010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:18.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.110+0000 7faeb2000640 1 --2- 172.21.15.32:0/752485912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e1d972270 0x564e1dd24010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.110+0000 7faeb2000640 1 -- 172.21.15.32:0/752485912 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564e1d9493b0 msgr2=0x564e1dd20c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:18.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.110+0000 7faeb2000640 1 --2- 172.21.15.32:0/752485912 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564e1d9493b0 0x564e1dd20c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.110+0000 7faeb2000640 1 -- 172.21.15.32:0/752485912 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e1dd245d0 con 0x564e1d951590 2024-04-30T08:34:18.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.110+0000 7faeb3400640 1 --2- 172.21.15.32:0/752485912 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564e1d9493b0 0x564e1dd20c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:34:18.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.111+0000 7faeb2000640 1 --2- 172.21.15.32:0/752485912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e1d951590 0x564e1dd21140 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fae9c02a380 tx=0x7fae9c0041a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa79f5ab2ffa407c server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:18.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.111+0000 7fae97e00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:34:18.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.111+0000 7fae97e00640 1 -- 172.21.15.32:0/752485912 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae9c04e050 con 0x564e1d951590 2024-04-30T08:34:18.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.111+0000 7faec05f1740 1 -- 172.21.15.32:0/752485912 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fae9c005d30 con 0x564e1d951590 2024-04-30T08:34:18.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.111+0000 7fae97e00640 1 -- 172.21.15.32:0/752485912 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fae9c030610 con 0x564e1d951590 2024-04-30T08:34:18.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.111+0000 7fae97e00640 1 -- 172.21.15.32:0/752485912 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fae9c05d050 con 0x564e1d951590 2024-04-30T08:34:18.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.111+0000 7faec05f1740 10 client.15084 fetch_fsmap learned FSMap version 27 2024-04-30T08:34:18.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.111+0000 7faec05f1740 1 -- 172.21.15.32:0/752485912 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x564e1dd25850 con 0x564e1d951590 2024-04-30T08:34:18.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.112+0000 7fae97e00640 1 -- 172.21.15.32:0/752485912 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fae9c005d30 con 0x564e1d951590 2024-04-30T08:34:18.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.112+0000 7fae97e00640 1 -- 172.21.15.32:0/752485912 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fae9c003780 con 0x564e1d951590 2024-04-30T08:34:18.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.112+0000 7faec05f1740 10 client.15084 fetch_fsmap finished waiting for FSMap version 27 2024-04-30T08:34:18.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.112+0000 7faec05f1740 10 client.15084 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-30T08:34:18.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.112+0000 7faec05f1740 20 client.15084 populate_metadata read hostname 'smithi032' 2024-04-30T08:34:18.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.112+0000 7faec05f1740 1 --2- 172.21.15.32:0/752485912 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x564e1d906280 0x564e1dd975a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.112+0000 7faec05f1740 4 client.15084 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-30T08:34:18.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.112+0000 7faec05f1740 1 -- 172.21.15.32:0/752485912 --> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] -- command(tid 0: {"prefix": "status"}) -- 0x564e1dc80210 con 0x564e1d906280 2024-04-30T08:34:18.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.112+0000 7faeb3400640 1 --2- 172.21.15.32:0/752485912 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x564e1d906280 0x564e1dd975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.113+0000 7faeb3400640 1 --2- 172.21.15.32:0/752485912 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x564e1d906280 0x564e1dd975a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=11858487e36a3af0 server_cookie=17ca2a9cd43f0270 in_seq=0 out_seq=0 2024-04-30T08:34:18.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.113+0000 7fae97e00640 10 client.15084 ms_handle_connect on v2:172.21.15.71:6836/2324039932 2024-04-30T08:34:18.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.113+0000 7fae97e00640 1 -- 172.21.15.32:0/752485912 <== mds.3 v2:172.21.15.71:6836/2324039932 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x564e1dc80210 con 0x564e1d906280 2024-04-30T08:34:18.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.113+0000 7fae97e00640 10 client.15084 handle_command_reply: tid=0 2024-04-30T08:34:18.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.113+0000 7faec05f1740 1 client.15084 shutdown 2024-04-30T08:34:18.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.113+0000 7faec05f1740 1 -- 172.21.15.32:0/752485912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e1d951590 msgr2=0x564e1dd21140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:18.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.113+0000 7faec05f1740 1 --2- 172.21.15.32:0/752485912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e1d951590 0x564e1dd21140 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fae9c02a380 tx=0x7fae9c0041a0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.113+0000 7faec05f1740 1 -- 172.21.15.32:0/752485912 shutdown_connections 2024-04-30T08:34:18.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.114+0000 7faec05f1740 1 --2- 172.21.15.32:0/752485912 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x564e1d906280 0x564e1dd975a0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.114+0000 7faec05f1740 1 --2- 172.21.15.32:0/752485912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564e1d972270 0x564e1dd24010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.114+0000 7faec05f1740 1 --2- 172.21.15.32:0/752485912 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564e1d951590 0x564e1dd21140 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.114+0000 7fae97e00640 0 client.15084 ms_handle_reset on v2:172.21.15.71:6836/2324039932 2024-04-30T08:34:18.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.114+0000 7faec05f1740 1 --2- 172.21.15.32:0/752485912 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564e1d9493b0 0x564e1dd20c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.114+0000 7faec05f1740 1 -- 172.21.15.32:0/752485912 >> 172.21.15.32:0/752485912 conn(0x564e1d9663c0 msgr2=0x564e1dd0a8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:18.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.114+0000 7faec05f1740 1 -- 172.21.15.32:0/752485912 shutdown_connections 2024-04-30T08:34:18.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.114+0000 7faec05f1740 1 -- 172.21.15.32:0/752485912 wait complete. 2024-04-30T08:34:18.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.114+0000 7faec05f1740 20 client.15084 trim_cache size 0 max 16384 2024-04-30T08:34:18.120 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-30T08:34:18.120 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "98c0a670-06ca-11ef-bc93-c7b262605968", 2024-04-30T08:34:18.121 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 3, 2024-04-30T08:34:18.121 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 24533, 2024-04-30T08:34:18.121 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-04-30T08:34:18.121 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-04-30T08:34:18.121 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-04-30T08:34:18.121 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 109.595371308, 2024-04-30T08:34:18.121 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 27, 2024-04-30T08:34:18.121 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 85, 2024-04-30T08:34:18.121 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 81, 2024-04-30T08:34:18.121 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 167.38682979699999 2024-04-30T08:34:18.121 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-30T08:34:18.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.114+0000 7faeb9600640 1 -- 172.21.15.32:0/2825565172 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fae980afaf0 msgr2=0x7fae980b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:18.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.114+0000 7faeb9600640 1 --2- 172.21.15.32:0/2825565172 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fae980afaf0 0x7fae980b2000 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7faea40df5f0 tx=0x7faea4009700 comp rx=0 tx=0).stop 2024-04-30T08:34:18.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.114+0000 7faeb9600640 1 -- 172.21.15.32:0/2825565172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faeb4076120 msgr2=0x7faeb419c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:18.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.114+0000 7faeb9600640 1 --2- 172.21.15.32:0/2825565172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faeb4076120 0x7faeb419c4b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7faea400e9c0 tx=0x7faea400ee80 comp rx=0 tx=0).stop 2024-04-30T08:34:18.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.115+0000 7faeb9600640 1 -- 172.21.15.32:0/2825565172 shutdown_connections 2024-04-30T08:34:18.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.115+0000 7faeb9600640 1 --2- 172.21.15.32:0/2825565172 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fae980afaf0 0x7fae980b2000 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.115+0000 7faeb9600640 1 --2- 172.21.15.32:0/2825565172 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faeb410bae0 0x7faeb41a3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.115+0000 7faeb9600640 1 --2- 172.21.15.32:0/2825565172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faeb4076a70 0x7faeb419c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.115+0000 7faeb9600640 1 --2- 172.21.15.32:0/2825565172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faeb4076120 0x7faeb419c4b0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.115+0000 7faeb9600640 1 -- 172.21.15.32:0/2825565172 >> 172.21.15.32:0/2825565172 conn(0x7faeb40fed60 msgr2=0x7faeb40ffba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:18.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.115+0000 7faeb9600640 1 -- 172.21.15.32:0/2825565172 shutdown_connections 2024-04-30T08:34:18.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.115+0000 7faeb9600640 1 -- 172.21.15.32:0/2825565172 wait complete. 2024-04-30T08:34:18.141 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:17 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1290817206' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:34:18.145 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-30T08:34:18.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:17 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1290817206' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:34:18.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:17 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1290817206' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:34:18.435 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.433+0000 7f6b14600640 1 -- 172.21.15.32:0/4045384969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b0c06f240 msgr2=0x7f6b0c06ff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:18.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.433+0000 7f6b14600640 1 --2- 172.21.15.32:0/4045384969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b0c06f240 0x7f6b0c06ff90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6afc009900 tx=0x7f6afc02a410 comp rx=0 tx=0).stop 2024-04-30T08:34:18.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.434+0000 7f6b14600640 1 -- 172.21.15.32:0/4045384969 shutdown_connections 2024-04-30T08:34:18.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.434+0000 7f6b14600640 1 --2- 172.21.15.32:0/4045384969 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b0c0711d0 0x7f6b0c0715e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.434+0000 7f6b14600640 1 --2- 172.21.15.32:0/4045384969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b0c0704d0 0x7f6b0c0708e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.434+0000 7f6b14600640 1 --2- 172.21.15.32:0/4045384969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b0c06f240 0x7f6b0c06ff90 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.434+0000 7f6b14600640 1 -- 172.21.15.32:0/4045384969 >> 172.21.15.32:0/4045384969 conn(0x7f6b0c06e440 msgr2=0x7f6b0c10bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:18.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.434+0000 7f6b14600640 1 -- 172.21.15.32:0/4045384969 shutdown_connections 2024-04-30T08:34:18.436 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.434+0000 7f6b14600640 1 -- 172.21.15.32:0/4045384969 wait complete. 2024-04-30T08:34:18.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.435+0000 7f6b14600640 1 Processor -- start 2024-04-30T08:34:18.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.435+0000 7f6b14600640 1 -- start start 2024-04-30T08:34:18.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.435+0000 7f6b14600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b0c06f240 0x7f6b0c1a90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.435+0000 7f6b14600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b0c0704d0 0x7f6b0c1a9610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.435+0000 7f6b14600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b0c0711d0 0x7f6b0c1b05a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.435+0000 7f6b14600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6b0c11e480 con 0x7f6b0c06f240 2024-04-30T08:34:18.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.435+0000 7f6b14600640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6b0c11e300 con 0x7f6b0c0704d0 2024-04-30T08:34:18.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.435+0000 7f6b14600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6b0c11e600 con 0x7f6b0c0711d0 2024-04-30T08:34:18.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.436+0000 7f6b11c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b0c06f240 0x7f6b0c1a90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.436+0000 7f6b11200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b0c0704d0 0x7f6b0c1a9610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.436+0000 7f6b11c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b0c06f240 0x7f6b0c1a90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57024/0 (socket says 172.21.15.32:57024) 2024-04-30T08:34:18.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.436+0000 7f6b11c00640 1 -- 172.21.15.32:0/1893582829 learned_addr learned my addr 172.21.15.32:0/1893582829 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:18.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.436+0000 7f6b12600640 1 --2- 172.21.15.32:0/1893582829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b0c0711d0 0x7f6b0c1b05a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.436+0000 7f6b11200640 1 -- 172.21.15.32:0/1893582829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b0c0711d0 msgr2=0x7f6b0c1b05a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:18.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.436+0000 7f6b11200640 1 --2- 172.21.15.32:0/1893582829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b0c0711d0 0x7f6b0c1b05a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.436+0000 7f6b11200640 1 -- 172.21.15.32:0/1893582829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b0c06f240 msgr2=0x7f6b0c1a90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:18.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.436+0000 7f6b11200640 1 --2- 172.21.15.32:0/1893582829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b0c06f240 0x7f6b0c1a90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.436+0000 7f6b11200640 1 -- 172.21.15.32:0/1893582829 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b0c1b0c10 con 0x7f6b0c0704d0 2024-04-30T08:34:18.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.436+0000 7f6b11c00640 1 --2- 172.21.15.32:0/1893582829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b0c06f240 0x7f6b0c1a90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:34:18.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.437+0000 7f6b11200640 1 --2- 172.21.15.32:0/1893582829 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b0c0704d0 0x7f6b0c1a9610 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6af800eb00 tx=0x7f6af800ef00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:18.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.437+0000 7f6b0a000640 1 -- 172.21.15.32:0/1893582829 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6af800ce40 con 0x7f6b0c0704d0 2024-04-30T08:34:18.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.437+0000 7f6b0a000640 1 -- 172.21.15.32:0/1893582829 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6af80108e0 con 0x7f6b0c0704d0 2024-04-30T08:34:18.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.437+0000 7f6b14600640 1 -- 172.21.15.32:0/1893582829 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b0c1b0f00 con 0x7f6b0c0704d0 2024-04-30T08:34:18.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.437+0000 7f6b0a000640 1 -- 172.21.15.32:0/1893582829 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6af80179f0 con 0x7f6b0c0704d0 2024-04-30T08:34:18.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.437+0000 7f6b14600640 1 -- 172.21.15.32:0/1893582829 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b0c1b1370 con 0x7f6b0c0704d0 2024-04-30T08:34:18.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.440+0000 7f6b1b64a740 1 Processor -- start 2024-04-30T08:34:18.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.441+0000 7f6b1b64a740 1 -- start start 2024-04-30T08:34:18.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.441+0000 7f6b1b64a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565545982270 0x565545d24340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.441+0000 7f6b1b64a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5655459593b0 0x565545d248c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.441+0000 7f6b1b64a740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x565545961590 0x565545d25550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.441+0000 7f6b1b64a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x565545c88790 con 0x5655459593b0 2024-04-30T08:34:18.443 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.441+0000 7f6b1b64a740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x565545c2df20 con 0x565545961590 2024-04-30T08:34:18.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.441+0000 7f6b1b64a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x565545c84750 con 0x565545982270 2024-04-30T08:34:18.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.441+0000 7f6b0a000640 1 -- 172.21.15.32:0/1893582829 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f6af8017b90 con 0x7f6b0c0704d0 2024-04-30T08:34:18.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.441+0000 7f6b11c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5655459593b0 0x565545d248c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.441+0000 7f6b11c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5655459593b0 0x565545d248c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57032/0 (socket says 172.21.15.32:57032) 2024-04-30T08:34:18.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.441+0000 7f6b11c00640 1 -- 172.21.15.32:0/3513802778 learned_addr learned my addr 172.21.15.32:0/3513802778 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:18.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.441+0000 7f6b12600640 1 --2- 172.21.15.32:0/3513802778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565545982270 0x565545d24340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.441+0000 7f6b11200640 1 --2- 172.21.15.32:0/3513802778 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x565545961590 0x565545d25550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.442+0000 7f6b11c00640 1 -- 172.21.15.32:0/3513802778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565545982270 msgr2=0x565545d24340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:18.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.442+0000 7f6b11c00640 1 --2- 172.21.15.32:0/3513802778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565545982270 0x565545d24340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.442+0000 7f6b11c00640 1 -- 172.21.15.32:0/3513802778 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x565545961590 msgr2=0x565545d25550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:18.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.442+0000 7f6b11c00640 1 --2- 172.21.15.32:0/3513802778 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x565545961590 0x565545d25550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.442+0000 7f6b11c00640 1 -- 172.21.15.32:0/3513802778 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56554599a2d0 con 0x5655459593b0 2024-04-30T08:34:18.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.442+0000 7f6b11c00640 1 --2- 172.21.15.32:0/3513802778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5655459593b0 0x565545d248c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6afc005ab0 tx=0x7f6afc0095a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f13eada8cf210f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:18.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.443+0000 7f6b02a00640 1 -- 172.21.15.32:0/3513802778 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6afc035070 con 0x5655459593b0 2024-04-30T08:34:18.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.443+0000 7f6b02a00640 1 -- 172.21.15.32:0/3513802778 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6afc0318a0 con 0x5655459593b0 2024-04-30T08:34:18.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.443+0000 7f6b0a000640 1 --2- 172.21.15.32:0/1893582829 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f6af00afaf0 0x7f6af00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.443+0000 7f6b02a00640 1 -- 172.21.15.32:0/3513802778 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6afc03c440 con 0x5655459593b0 2024-04-30T08:34:18.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.443+0000 7f6b0a000640 1 -- 172.21.15.32:0/1893582829 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f6af80df820 con 0x7f6b0c0704d0 2024-04-30T08:34:18.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.443+0000 7f6b12600640 1 --2- 172.21.15.32:0/1893582829 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f6af00afaf0 0x7f6af00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.444+0000 7f6b1b64a740 1 -- 172.21.15.32:0/3513802778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5655459593b0 msgr2=0x565545d248c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:18.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.444+0000 7f6b1b64a740 1 --2- 172.21.15.32:0/3513802778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5655459593b0 0x565545d248c0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f6afc005ab0 tx=0x7f6afc0095a0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.444+0000 7f6b1b64a740 1 -- 172.21.15.32:0/3513802778 shutdown_connections 2024-04-30T08:34:18.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.444+0000 7f6b1b64a740 1 --2- 172.21.15.32:0/3513802778 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x565545961590 0x565545d25550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.444+0000 7f6b1b64a740 1 --2- 172.21.15.32:0/3513802778 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5655459593b0 0x565545d248c0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.444+0000 7f6b1b64a740 1 --2- 172.21.15.32:0/3513802778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565545982270 0x565545d24340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.444+0000 7f6b1b64a740 1 -- 172.21.15.32:0/3513802778 >> 172.21.15.32:0/3513802778 conn(0x5655459763c0 msgr2=0x565545d1fb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:18.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.444+0000 7f6b1b64a740 1 -- 172.21.15.32:0/3513802778 shutdown_connections 2024-04-30T08:34:18.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.444+0000 7f6b12600640 1 --2- 172.21.15.32:0/1893582829 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f6af00afaf0 0x7f6af00b2000 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x565545d25390 tx=0x7f6b04005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:18.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.444+0000 7f6b1b64a740 1 -- 172.21.15.32:0/3513802778 wait complete. 2024-04-30T08:34:18.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.445+0000 7f6b1b64a740 1 -- start start 2024-04-30T08:34:18.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.445+0000 7f6b1b64a740 1 Processor -- start 2024-04-30T08:34:18.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.445+0000 7f6b1b64a740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5655459593b0 0x565545d2ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.445+0000 7f6b1b64a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565545961590 0x565545d2e2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.445+0000 7f6b1b64a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565545982270 0x565545d35250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.445+0000 7f6b1b64a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x565545d26000 con 0x565545982270 2024-04-30T08:34:18.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.445+0000 7f6b1b64a740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x565545c419c0 con 0x5655459593b0 2024-04-30T08:34:18.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.445+0000 7f6b1b64a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x565545d26180 con 0x565545961590 2024-04-30T08:34:18.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.445+0000 7f6b1b64a740 10 client.0 _ll_register_callbacks cb 0x565545c0e120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:34:18.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.446+0000 7f6b12600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565545982270 0x565545d35250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.446+0000 7f6b12600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565545982270 0x565545d35250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57038/0 (socket says 172.21.15.32:57038) 2024-04-30T08:34:18.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.446+0000 7f6b11c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5655459593b0 0x565545d2ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.446+0000 7f6b12600640 1 -- 172.21.15.32:0/1956906411 learned_addr learned my addr 172.21.15.32:0/1956906411 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:18.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.446+0000 7f6b11200640 1 --2- 172.21.15.32:0/1956906411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565545961590 0x565545d2e2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.446+0000 7f6b12600640 1 -- 172.21.15.32:0/1956906411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565545961590 msgr2=0x565545d2e2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:18.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.446+0000 7f6b12600640 1 --2- 172.21.15.32:0/1956906411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565545961590 0x565545d2e2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.446+0000 7f6b12600640 1 -- 172.21.15.32:0/1956906411 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5655459593b0 msgr2=0x565545d2ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:18.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.446+0000 7f6b12600640 1 --2- 172.21.15.32:0/1956906411 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5655459593b0 0x565545d2ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.446+0000 7f6b12600640 1 -- 172.21.15.32:0/1956906411 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565545a646b0 con 0x565545982270 2024-04-30T08:34:18.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.447+0000 7f6b12600640 1 --2- 172.21.15.32:0/1956906411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565545982270 0x565545d35250 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6b040045e0 tx=0x7f6b04004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f8f29c241c1b048 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:18.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.447+0000 7f6af7400640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:34:18.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.447+0000 7f6af7400640 1 -- 172.21.15.32:0/1956906411 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b0401e680 con 0x565545982270 2024-04-30T08:34:18.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.447+0000 7f6af7400640 1 -- 172.21.15.32:0/1956906411 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f6b0401e820 con 0x565545982270 2024-04-30T08:34:18.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.447+0000 7f6b1b64a740 1 -- 172.21.15.32:0/1956906411 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6afc0318a0 con 0x565545982270 2024-04-30T08:34:18.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.447+0000 7f6af7400640 1 -- 172.21.15.32:0/1956906411 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6b0401ea20 con 0x565545982270 2024-04-30T08:34:18.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.447+0000 7f6b1b64a740 10 client.15096 fetch_fsmap learned FSMap version 27 2024-04-30T08:34:18.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.447+0000 7f6b1b64a740 1 -- 172.21.15.32:0/1956906411 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x565545d38250 con 0x565545982270 2024-04-30T08:34:18.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.448+0000 7f6af7400640 1 -- 172.21.15.32:0/1956906411 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6afc0318a0 con 0x565545982270 2024-04-30T08:34:18.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.448+0000 7f6af7400640 1 -- 172.21.15.32:0/1956906411 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f6b04027070 con 0x565545982270 2024-04-30T08:34:18.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.448+0000 7f6b1b64a740 10 client.15096 fetch_fsmap finished waiting for FSMap version 27 2024-04-30T08:34:18.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.448+0000 7f6b1b64a740 10 client.15096 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-30T08:34:18.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.448+0000 7f6b1b64a740 20 client.15096 populate_metadata read hostname 'smithi032' 2024-04-30T08:34:18.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.448+0000 7f6b1b64a740 1 --2- 172.21.15.32:0/1956906411 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x565545940e40 0x565545d3bd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.448+0000 7f6b1b64a740 4 client.15096 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:34:18.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.448+0000 7f6b1b64a740 1 -- 172.21.15.32:0/1956906411 --> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x565545c963a0 con 0x565545940e40 2024-04-30T08:34:18.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.448+0000 7f6b11c00640 1 --2- 172.21.15.32:0/1956906411 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x565545940e40 0x565545d3bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.449+0000 7f6b11c00640 1 --2- 172.21.15.32:0/1956906411 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x565545940e40 0x565545d3bd20 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=76bc3cdba58feda2 server_cookie=33e745f859a91bdf in_seq=0 out_seq=0 2024-04-30T08:34:18.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.449+0000 7f6af7400640 10 client.15096 ms_handle_connect on v2:172.21.15.189:6838/352970107 2024-04-30T08:34:18.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.450+0000 7f6af7400640 1 -- 172.21.15.32:0/1956906411 <== mds.2 v2:172.21.15.189:6838/352970107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x565545c963a0 con 0x565545940e40 2024-04-30T08:34:18.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.450+0000 7f6af7400640 10 client.15096 handle_command_reply: tid=0 2024-04-30T08:34:18.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.450+0000 7f6b1b64a740 1 client.15096 shutdown 2024-04-30T08:34:18.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.450+0000 7f6b1b64a740 1 -- 172.21.15.32:0/1956906411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565545982270 msgr2=0x565545d35250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:18.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.450+0000 7f6b1b64a740 1 --2- 172.21.15.32:0/1956906411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565545982270 0x565545d35250 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6b040045e0 tx=0x7f6b04004730 comp rx=0 tx=0).stop 2024-04-30T08:34:18.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.451+0000 7f6b1b64a740 1 -- 172.21.15.32:0/1956906411 shutdown_connections 2024-04-30T08:34:18.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.451+0000 7f6b1b64a740 1 --2- 172.21.15.32:0/1956906411 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x565545940e40 0x565545d3bd20 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.451+0000 7f6b1b64a740 1 --2- 172.21.15.32:0/1956906411 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565545982270 0x565545d35250 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.451+0000 7f6b1b64a740 1 --2- 172.21.15.32:0/1956906411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x565545961590 0x565545d2e2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.451+0000 7f6b1b64a740 1 --2- 172.21.15.32:0/1956906411 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5655459593b0 0x565545d2ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.451+0000 7f6b1b64a740 1 -- 172.21.15.32:0/1956906411 >> 172.21.15.32:0/1956906411 conn(0x5655459763c0 msgr2=0x565545d22a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:18.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.451+0000 7f6af7400640 0 client.15096 ms_handle_reset on v2:172.21.15.189:6838/352970107 2024-04-30T08:34:18.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.451+0000 7f6b1b64a740 1 -- 172.21.15.32:0/1956906411 shutdown_connections 2024-04-30T08:34:18.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.451+0000 7f6b1b64a740 1 -- 172.21.15.32:0/1956906411 wait complete. 2024-04-30T08:34:18.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.451+0000 7f6b1b64a740 20 client.15096 trim_cache size 0 max 16384 2024-04-30T08:34:18.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.471+0000 7f6b1b64a740 1 Processor -- start 2024-04-30T08:34:18.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.472+0000 7f6b1b64a740 1 -- start start 2024-04-30T08:34:18.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.472+0000 7f6b1b64a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5655459593b0 0x565545d1a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.472+0000 7f6b1b64a740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x565545961590 0x565545d35170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.472+0000 7f6b1b64a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565545982270 0x565545d2d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.472+0000 7f6b1b64a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x565545d3a060 con 0x565545982270 2024-04-30T08:34:18.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.472+0000 7f6b1b64a740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x565545d3a730 con 0x565545961590 2024-04-30T08:34:18.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.472+0000 7f6b1b64a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x565545d3ae00 con 0x5655459593b0 2024-04-30T08:34:18.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.472+0000 7f6b12600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565545982270 0x565545d2d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.472+0000 7f6b11c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5655459593b0 0x565545d1a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.472+0000 7f6b11c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5655459593b0 0x565545d1a8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:59576/0 (socket says 172.21.15.32:59576) 2024-04-30T08:34:18.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.472+0000 7f6b11c00640 1 -- 172.21.15.32:0/3555264801 learned_addr learned my addr 172.21.15.32:0/3555264801 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:18.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.472+0000 7f6b11200640 1 --2- 172.21.15.32:0/3555264801 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x565545961590 0x565545d35170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.473+0000 7f6b11c00640 1 -- 172.21.15.32:0/3555264801 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x565545961590 msgr2=0x565545d35170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:18.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.473+0000 7f6b11c00640 1 --2- 172.21.15.32:0/3555264801 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x565545961590 0x565545d35170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.473+0000 7f6b11c00640 1 -- 172.21.15.32:0/3555264801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565545982270 msgr2=0x565545d2d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:18.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.473+0000 7f6b11c00640 1 --2- 172.21.15.32:0/3555264801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565545982270 0x565545d2d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.473+0000 7f6b11c00640 1 -- 172.21.15.32:0/3555264801 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565545d206e0 con 0x5655459593b0 2024-04-30T08:34:18.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.473+0000 7f6b12600640 1 --2- 172.21.15.32:0/3555264801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565545982270 0x565545d2d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:34:18.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.473+0000 7f6b11c00640 1 --2- 172.21.15.32:0/3555264801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5655459593b0 0x565545d1a8e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6afc034040 tx=0x7f6afc0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac1b8fe45a9c4348 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:18.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.474+0000 7f6b02000640 1 -- 172.21.15.32:0/3555264801 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6afc057020 con 0x5655459593b0 2024-04-30T08:34:18.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.474+0000 7f6b02000640 1 -- 172.21.15.32:0/3555264801 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6afc003870 con 0x5655459593b0 2024-04-30T08:34:18.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.474+0000 7f6b02000640 1 -- 172.21.15.32:0/3555264801 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6afc035040 con 0x5655459593b0 2024-04-30T08:34:18.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.474+0000 7f6b1b64a740 1 -- 172.21.15.32:0/3555264801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5655459593b0 msgr2=0x565545d1a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:18.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.474+0000 7f6b1b64a740 1 --2- 172.21.15.32:0/3555264801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5655459593b0 0x565545d1a8e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f6afc034040 tx=0x7f6afc0046c0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.475+0000 7f6b1b64a740 1 -- 172.21.15.32:0/3555264801 shutdown_connections 2024-04-30T08:34:18.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.475+0000 7f6b1b64a740 1 --2- 172.21.15.32:0/3555264801 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565545982270 0x565545d2d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.475+0000 7f6b1b64a740 1 --2- 172.21.15.32:0/3555264801 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x565545961590 0x565545d35170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.475+0000 7f6b1b64a740 1 --2- 172.21.15.32:0/3555264801 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5655459593b0 0x565545d1a8e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.475+0000 7f6b1b64a740 1 -- 172.21.15.32:0/3555264801 >> 172.21.15.32:0/3555264801 conn(0x5655459763c0 msgr2=0x565545d242c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:18.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.475+0000 7f6b1b64a740 1 -- 172.21.15.32:0/3555264801 shutdown_connections 2024-04-30T08:34:18.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.475+0000 7f6b1b64a740 1 -- 172.21.15.32:0/3555264801 wait complete. 2024-04-30T08:34:18.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.475+0000 7f6b1b64a740 1 -- start start 2024-04-30T08:34:18.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.475+0000 7f6b1b64a740 1 Processor -- start 2024-04-30T08:34:18.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.476+0000 7f6b1b64a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5655459593b0 0x565545d32150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.476+0000 7f6b1b64a740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x565545961590 0x565545d32690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.476+0000 7f6b1b64a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565545982270 0x565545da2b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.476+0000 7f6b1b64a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x565545d3bba0 con 0x565545982270 2024-04-30T08:34:18.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.476+0000 7f6b1b64a740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x565545d3b4d0 con 0x565545961590 2024-04-30T08:34:18.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.476+0000 7f6b1b64a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x565545d3c270 con 0x5655459593b0 2024-04-30T08:34:18.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.476+0000 7f6b1b64a740 10 client.0 _ll_register_callbacks cb 0x7f6b04009a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:34:18.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.476+0000 7f6b12600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565545982270 0x565545da2b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.476+0000 7f6b12600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565545982270 0x565545da2b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57052/0 (socket says 172.21.15.32:57052) 2024-04-30T08:34:18.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.476+0000 7f6b12600640 1 -- 172.21.15.32:0/776623253 learned_addr learned my addr 172.21.15.32:0/776623253 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:18.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.476+0000 7f6b11200640 1 --2- 172.21.15.32:0/776623253 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x565545961590 0x565545d32690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.476+0000 7f6b11c00640 1 --2- 172.21.15.32:0/776623253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5655459593b0 0x565545d32150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.477+0000 7f6b12600640 1 -- 172.21.15.32:0/776623253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5655459593b0 msgr2=0x565545d32150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:18.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.477+0000 7f6b12600640 1 --2- 172.21.15.32:0/776623253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5655459593b0 0x565545d32150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.477+0000 7f6b12600640 1 -- 172.21.15.32:0/776623253 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x565545961590 msgr2=0x565545d32690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:18.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.477+0000 7f6b12600640 1 --2- 172.21.15.32:0/776623253 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x565545961590 0x565545d32690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.477+0000 7f6b12600640 1 -- 172.21.15.32:0/776623253 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x565545da30d0 con 0x565545982270 2024-04-30T08:34:18.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.477+0000 7f6b11200640 1 --2- 172.21.15.32:0/776623253 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x565545961590 0x565545d32690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:34:18.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.477+0000 7f6b12600640 1 --2- 172.21.15.32:0/776623253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565545982270 0x565545da2b90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6b0402aff0 tx=0x7f6b0401e930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a98b074d26ef3e5b server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:18.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.477+0000 7f6af7400640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:34:18.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.477+0000 7f6af7400640 1 -- 172.21.15.32:0/776623253 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b0401edd0 con 0x565545982270 2024-04-30T08:34:18.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.477+0000 7f6af7400640 1 -- 172.21.15.32:0/776623253 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f6b04018050 con 0x565545982270 2024-04-30T08:34:18.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.477+0000 7f6b1b64a740 1 -- 172.21.15.32:0/776623253 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6afc003870 con 0x565545982270 2024-04-30T08:34:18.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.478+0000 7f6af7400640 1 -- 172.21.15.32:0/776623253 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f6b0401f730 con 0x565545982270 2024-04-30T08:34:18.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.478+0000 7f6b1b64a740 10 client.15102 fetch_fsmap learned FSMap version 27 2024-04-30T08:34:18.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.478+0000 7f6b1b64a740 1 -- 172.21.15.32:0/776623253 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x565545da4220 con 0x565545982270 2024-04-30T08:34:18.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.478+0000 7f6af7400640 1 -- 172.21.15.32:0/776623253 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6afc003870 con 0x565545982270 2024-04-30T08:34:18.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.478+0000 7f6af7400640 1 -- 172.21.15.32:0/776623253 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f6b04027040 con 0x565545982270 2024-04-30T08:34:18.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.478+0000 7f6b1b64a740 10 client.15102 fetch_fsmap finished waiting for FSMap version 27 2024-04-30T08:34:18.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.478+0000 7f6b1b64a740 10 client.15102 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-30T08:34:18.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.478+0000 7f6b1b64a740 20 client.15102 populate_metadata read hostname 'smithi032' 2024-04-30T08:34:18.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.478+0000 7f6b1b64a740 1 --2- 172.21.15.32:0/776623253 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x565545916280 0x565545da7af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.478+0000 7f6b1b64a740 4 client.15102 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-30T08:34:18.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.478+0000 7f6b1b64a740 1 -- 172.21.15.32:0/776623253 --> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] -- command(tid 0: {"prefix": "status"}) -- 0x565545c90210 con 0x565545916280 2024-04-30T08:34:18.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.479+0000 7f6b11c00640 1 --2- 172.21.15.32:0/776623253 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x565545916280 0x565545da7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.479+0000 7f6b11c00640 1 --2- 172.21.15.32:0/776623253 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x565545916280 0x565545da7af0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2136a9949294a7cb server_cookie=9ea061b0ec5c3e6b in_seq=0 out_seq=0 2024-04-30T08:34:18.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.479+0000 7f6af7400640 10 client.15102 ms_handle_connect on v2:172.21.15.189:6838/352970107 2024-04-30T08:34:18.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.480+0000 7f6af7400640 1 -- 172.21.15.32:0/776623253 <== mds.2 v2:172.21.15.189:6838/352970107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x565545c90210 con 0x565545916280 2024-04-30T08:34:18.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.480+0000 7f6af7400640 10 client.15102 handle_command_reply: tid=0 2024-04-30T08:34:18.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.480+0000 7f6b1b64a740 1 client.15102 shutdown 2024-04-30T08:34:18.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.480+0000 7f6b1b64a740 1 -- 172.21.15.32:0/776623253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565545982270 msgr2=0x565545da2b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:18.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.480+0000 7f6b1b64a740 1 --2- 172.21.15.32:0/776623253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565545982270 0x565545da2b90 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f6b0402aff0 tx=0x7f6b0401e930 comp rx=0 tx=0).stop 2024-04-30T08:34:18.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.480+0000 7f6b1b64a740 1 -- 172.21.15.32:0/776623253 shutdown_connections 2024-04-30T08:34:18.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.480+0000 7f6b1b64a740 1 --2- 172.21.15.32:0/776623253 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x565545916280 0x565545da7af0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.480+0000 7f6b1b64a740 1 --2- 172.21.15.32:0/776623253 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x565545982270 0x565545da2b90 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.480+0000 7f6b1b64a740 1 --2- 172.21.15.32:0/776623253 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x565545961590 0x565545d32690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.480+0000 7f6b1b64a740 1 --2- 172.21.15.32:0/776623253 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5655459593b0 0x565545d32150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.480+0000 7f6af7400640 0 client.15102 ms_handle_reset on v2:172.21.15.189:6838/352970107 2024-04-30T08:34:18.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.480+0000 7f6b1b64a740 1 -- 172.21.15.32:0/776623253 >> 172.21.15.32:0/776623253 conn(0x5655459763c0 msgr2=0x565545d22950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:18.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.480+0000 7f6b1b64a740 1 -- 172.21.15.32:0/776623253 shutdown_connections 2024-04-30T08:34:18.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.480+0000 7f6b1b64a740 1 -- 172.21.15.32:0/776623253 wait complete. 2024-04-30T08:34:18.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.480+0000 7f6b1b64a740 20 client.15102 trim_cache size 0 max 16384 2024-04-30T08:34:18.485 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-30T08:34:18.486 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "98c0a670-06ca-11ef-bc93-c7b262605968", 2024-04-30T08:34:18.486 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 2, 2024-04-30T08:34:18.486 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 24541, 2024-04-30T08:34:18.486 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-04-30T08:34:18.486 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-04-30T08:34:18.486 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-04-30T08:34:18.486 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 110.972168694, 2024-04-30T08:34:18.486 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 27, 2024-04-30T08:34:18.486 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 85, 2024-04-30T08:34:18.486 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 81, 2024-04-30T08:34:18.486 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 164.729968305 2024-04-30T08:34:18.486 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-30T08:34:18.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.481+0000 7f6b14600640 1 -- 172.21.15.32:0/1893582829 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f6af00afaf0 msgr2=0x7f6af00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:18.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.481+0000 7f6b14600640 1 --2- 172.21.15.32:0/1893582829 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f6af00afaf0 0x7f6af00b2000 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x565545d25390 tx=0x7f6b04005f90 comp rx=0 tx=0).stop 2024-04-30T08:34:18.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.481+0000 7f6b14600640 1 -- 172.21.15.32:0/1893582829 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b0c0704d0 msgr2=0x7f6b0c1a9610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:18.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.481+0000 7f6b14600640 1 --2- 172.21.15.32:0/1893582829 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b0c0704d0 0x7f6b0c1a9610 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f6af800eb00 tx=0x7f6af800ef00 comp rx=0 tx=0).stop 2024-04-30T08:34:18.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.481+0000 7f6b14600640 1 -- 172.21.15.32:0/1893582829 shutdown_connections 2024-04-30T08:34:18.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.481+0000 7f6b14600640 1 --2- 172.21.15.32:0/1893582829 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f6af00afaf0 0x7f6af00b2000 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.481+0000 7f6b14600640 1 --2- 172.21.15.32:0/1893582829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6b0c0711d0 0x7f6b0c1b05a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.481+0000 7f6b14600640 1 --2- 172.21.15.32:0/1893582829 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6b0c0704d0 0x7f6b0c1a9610 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.481+0000 7f6b14600640 1 --2- 172.21.15.32:0/1893582829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6b0c06f240 0x7f6b0c1a90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.481+0000 7f6b14600640 1 -- 172.21.15.32:0/1893582829 >> 172.21.15.32:0/1893582829 conn(0x7f6b0c06e440 msgr2=0x7f6b0c118410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:18.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.481+0000 7f6b14600640 1 -- 172.21.15.32:0/1893582829 shutdown_connections 2024-04-30T08:34:18.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.481+0000 7f6b14600640 1 -- 172.21.15.32:0/1893582829 wait complete. 2024-04-30T08:34:18.510 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.g status 2024-04-30T08:34:18.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.790+0000 7f2336e00640 1 -- 172.21.15.32:0/2235365254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2330075960 msgr2=0x7f233010b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:18.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.790+0000 7f2336e00640 1 --2- 172.21.15.32:0/2235365254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2330075960 0x7f233010b3c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f231c009900 tx=0x7f231c02a3f0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.790+0000 7f2336e00640 1 -- 172.21.15.32:0/2235365254 shutdown_connections 2024-04-30T08:34:18.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.790+0000 7f2336e00640 1 --2- 172.21.15.32:0/2235365254 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f233010b900 0x7f233010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.790+0000 7f2336e00640 1 --2- 172.21.15.32:0/2235365254 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2330075960 0x7f233010b3c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.790+0000 7f2336e00640 1 --2- 172.21.15.32:0/2235365254 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2330075010 0x7f2330075420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.790+0000 7f2336e00640 1 -- 172.21.15.32:0/2235365254 >> 172.21.15.32:0/2235365254 conn(0x7f23300feba0 msgr2=0x7f2330101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:18.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.791+0000 7f2336e00640 1 -- 172.21.15.32:0/2235365254 shutdown_connections 2024-04-30T08:34:18.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.791+0000 7f2336e00640 1 -- 172.21.15.32:0/2235365254 wait complete. 2024-04-30T08:34:18.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.792+0000 7f2336e00640 1 Processor -- start 2024-04-30T08:34:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.792+0000 7f2336e00640 1 -- start start 2024-04-30T08:34:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.792+0000 7f2336e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2330075010 0x7f233019c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.792+0000 7f2336e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2330075960 0x7f233019c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.792+0000 7f2336e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f233010b900 0x7f23301a37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.792+0000 7f2336e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2330111730 con 0x7f2330075010 2024-04-30T08:34:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.792+0000 7f2336e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f23301115b0 con 0x7f2330075960 2024-04-30T08:34:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.792+0000 7f2336e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f23301118b0 con 0x7f233010b900 2024-04-30T08:34:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.793+0000 7f2335a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2330075010 0x7f233019c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.793+0000 7f2335000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2330075960 0x7f233019c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.793+0000 7f2335000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2330075960 0x7f233019c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:55608/0 (socket says 172.21.15.32:55608) 2024-04-30T08:34:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.793+0000 7f2335000640 1 -- 172.21.15.32:0/3041264336 learned_addr learned my addr 172.21.15.32:0/3041264336 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:18.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.793+0000 7f2336400640 1 --2- 172.21.15.32:0/3041264336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f233010b900 0x7f23301a37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.793+0000 7f2335a00640 1 -- 172.21.15.32:0/3041264336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f233010b900 msgr2=0x7f23301a37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:18.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.793+0000 7f2335a00640 1 --2- 172.21.15.32:0/3041264336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f233010b900 0x7f23301a37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.793+0000 7f2335a00640 1 -- 172.21.15.32:0/3041264336 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2330075960 msgr2=0x7f233019c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:18.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.793+0000 7f2335a00640 1 --2- 172.21.15.32:0/3041264336 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2330075960 0x7f233019c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.793+0000 7f2335a00640 1 -- 172.21.15.32:0/3041264336 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23301a3e10 con 0x7f2330075010 2024-04-30T08:34:18.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.794+0000 7f2335a00640 1 --2- 172.21.15.32:0/3041264336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2330075010 0x7f233019c2d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2320005fd0 tx=0x7f232000ef90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:18.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.794+0000 7f232e000640 1 -- 172.21.15.32:0/3041264336 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2320010430 con 0x7f2330075010 2024-04-30T08:34:18.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.794+0000 7f2336e00640 1 -- 172.21.15.32:0/3041264336 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23301a4100 con 0x7f2330075010 2024-04-30T08:34:18.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.794+0000 7f232e000640 1 -- 172.21.15.32:0/3041264336 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2320010a80 con 0x7f2330075010 2024-04-30T08:34:18.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.794+0000 7f232e000640 1 -- 172.21.15.32:0/3041264336 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2320017a50 con 0x7f2330075010 2024-04-30T08:34:18.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.794+0000 7f2336e00640 1 -- 172.21.15.32:0/3041264336 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23301a4650 con 0x7f2330075010 2024-04-30T08:34:18.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.797+0000 7f233f390740 1 Processor -- start 2024-04-30T08:34:18.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.798+0000 7f233f390740 1 -- start start 2024-04-30T08:34:18.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.798+0000 7f233f390740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5559c1d88270 0x5559c212a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.798+0000 7f233f390740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559c1d5f3b0 0x5559c212aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.798+0000 7f233f390740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559c1d67590 0x5559c212b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.798+0000 7f233f390740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5559c208e790 con 0x5559c1d5f3b0 2024-04-30T08:34:18.800 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.798+0000 7f233f390740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x5559c2033f20 con 0x5559c1d88270 2024-04-30T08:34:18.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.798+0000 7f233f390740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5559c208a750 con 0x5559c1d67590 2024-04-30T08:34:18.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.798+0000 7f2336400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5559c1d88270 0x5559c212a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.798+0000 7f2336400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5559c1d88270 0x5559c212a530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:55614/0 (socket says 172.21.15.32:55614) 2024-04-30T08:34:18.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.798+0000 7f2336400640 1 -- 172.21.15.32:0/3823721073 learned_addr learned my addr 172.21.15.32:0/3823721073 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:18.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.798+0000 7f2335000640 1 --2- 172.21.15.32:0/3823721073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559c1d67590 0x5559c212b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.799+0000 7f2336400640 1 -- 172.21.15.32:0/3823721073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559c1d67590 msgr2=0x5559c212b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:18.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.799+0000 7f2336400640 1 --2- 172.21.15.32:0/3823721073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559c1d67590 0x5559c212b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.799+0000 7f2336400640 1 -- 172.21.15.32:0/3823721073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559c1d5f3b0 msgr2=0x5559c212aab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-30T08:34:18.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.799+0000 7f2336400640 1 --2- 172.21.15.32:0/3823721073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559c1d5f3b0 0x5559c212aab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.799+0000 7f2336400640 1 -- 172.21.15.32:0/3823721073 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559c1da02d0 con 0x5559c1d88270 2024-04-30T08:34:18.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.799+0000 7f232e000640 1 -- 172.21.15.32:0/3041264336 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f2320017c30 con 0x7f2330075010 2024-04-30T08:34:18.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.799+0000 7f2336400640 1 --2- 172.21.15.32:0/3823721073 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5559c1d88270 0x5559c212a530 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2328009c20 tx=0x7f2328011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=722cc504e9c43fce server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:18.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.799+0000 7f2326a00640 1 -- 172.21.15.32:0/3823721073 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f232801e070 con 0x5559c1d88270 2024-04-30T08:34:18.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.799+0000 7f2326a00640 1 -- 172.21.15.32:0/3823721073 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2328019ae0 con 0x5559c1d88270 2024-04-30T08:34:18.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.799+0000 7f2326a00640 1 -- 172.21.15.32:0/3823721073 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f232800e920 con 0x5559c1d88270 2024-04-30T08:34:18.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.800+0000 7f233f390740 1 -- 172.21.15.32:0/3823721073 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5559c1d88270 msgr2=0x5559c212a530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:18.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.800+0000 7f233f390740 1 --2- 172.21.15.32:0/3823721073 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5559c1d88270 0x5559c212a530 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2328009c20 tx=0x7f2328011810 comp rx=0 tx=0).stop 2024-04-30T08:34:18.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.800+0000 7f233f390740 1 -- 172.21.15.32:0/3823721073 shutdown_connections 2024-04-30T08:34:18.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.800+0000 7f233f390740 1 --2- 172.21.15.32:0/3823721073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559c1d67590 0x5559c212b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.800+0000 7f233f390740 1 --2- 172.21.15.32:0/3823721073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559c1d5f3b0 0x5559c212aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.800+0000 7f233f390740 1 --2- 172.21.15.32:0/3823721073 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5559c1d88270 0x5559c212a530 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.800+0000 7f233f390740 1 -- 172.21.15.32:0/3823721073 >> 172.21.15.32:0/3823721073 conn(0x5559c1d7c3c0 msgr2=0x5559c2125d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:18.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.801+0000 7f233f390740 1 -- 172.21.15.32:0/3823721073 shutdown_connections 2024-04-30T08:34:18.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.801+0000 7f233f390740 1 -- 172.21.15.32:0/3823721073 wait complete. 2024-04-30T08:34:18.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.801+0000 7f232e000640 1 --2- 172.21.15.32:0/3041264336 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f23140afa00 0x7f23140b1f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.801+0000 7f233f390740 1 -- start start 2024-04-30T08:34:18.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.801+0000 7f232e000640 1 -- 172.21.15.32:0/3041264336 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f23200dfa40 con 0x7f2330075010 2024-04-30T08:34:18.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.801+0000 7f2336400640 1 --2- 172.21.15.32:0/3041264336 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f23140afa00 0x7f23140b1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.802+0000 7f233f390740 1 Processor -- start 2024-04-30T08:34:18.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.802+0000 7f233f390740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5559c1d5f3b0 0x5559c21340d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.802+0000 7f233f390740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559c1d67590 0x5559c2134610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.802+0000 7f233f390740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559c1d88270 0x5559c213b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.802+0000 7f233f390740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5559c212c250 con 0x5559c1d88270 2024-04-30T08:34:18.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.802+0000 7f233f390740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x5559c20479c0 con 0x5559c1d5f3b0 2024-04-30T08:34:18.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.802+0000 7f233f390740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5559c212c3d0 con 0x5559c1d67590 2024-04-30T08:34:18.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.802+0000 7f233f390740 10 client.0 _ll_register_callbacks cb 0x5559c2014120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:34:18.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.802+0000 7f2336400640 1 --2- 172.21.15.32:0/3041264336 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f23140afa00 0x7f23140b1f10 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f232800a010 tx=0x7f2328009830 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:18.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.803+0000 7f2335000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559c1d67590 0x5559c2134610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.803+0000 7f2336400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559c1d88270 0x5559c213b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.803+0000 7f2335000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559c1d67590 0x5559c2134610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:59624/0 (socket says 172.21.15.32:59624) 2024-04-30T08:34:18.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.803+0000 7f2335a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5559c1d5f3b0 0x5559c21340d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.803+0000 7f2335000640 1 -- 172.21.15.32:0/603097811 learned_addr learned my addr 172.21.15.32:0/603097811 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:18.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.803+0000 7f2335a00640 1 -- 172.21.15.32:0/603097811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559c1d67590 msgr2=0x5559c2134610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:18.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.803+0000 7f2335a00640 1 --2- 172.21.15.32:0/603097811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559c1d67590 0x5559c2134610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.808 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.803+0000 7f2335a00640 1 -- 172.21.15.32:0/603097811 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559c1d88270 msgr2=0x5559c213b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:18.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.803+0000 7f2335a00640 1 --2- 172.21.15.32:0/603097811 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559c1d88270 0x5559c213b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.803+0000 7f2335a00640 1 -- 172.21.15.32:0/603097811 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559c1e6a6b0 con 0x5559c1d5f3b0 2024-04-30T08:34:18.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.803+0000 7f2335000640 1 --2- 172.21.15.32:0/603097811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559c1d67590 0x5559c2134610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:34:18.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.803+0000 7f2335a00640 1 --2- 172.21.15.32:0/603097811 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5559c1d5f3b0 0x5559c21340d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x5559c212a250 tx=0x7f23200a79d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c05e35fd50df67f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:18.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.803+0000 7f231b400640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:34:18.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.804+0000 7f231b400640 1 -- 172.21.15.32:0/603097811 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23200a8210 con 0x5559c1d5f3b0 2024-04-30T08:34:18.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.804+0000 7f231b400640 1 -- 172.21.15.32:0/603097811 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f23200a3da0 con 0x5559c1d5f3b0 2024-04-30T08:34:18.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.804+0000 7f233f390740 1 -- 172.21.15.32:0/603097811 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2328019ae0 con 0x5559c1d5f3b0 2024-04-30T08:34:18.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.804+0000 7f231b400640 1 -- 172.21.15.32:0/603097811 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f23200a7b10 con 0x5559c1d5f3b0 2024-04-30T08:34:18.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.804+0000 7f233f390740 10 client.25235 fetch_fsmap learned FSMap version 27 2024-04-30T08:34:18.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.804+0000 7f233f390740 1 -- 172.21.15.32:0/603097811 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5559c213e570 con 0x5559c1d5f3b0 2024-04-30T08:34:18.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.804+0000 7f231b400640 1 -- 172.21.15.32:0/603097811 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2328019ae0 con 0x5559c1d5f3b0 2024-04-30T08:34:18.809 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.804+0000 7f231b400640 1 -- 172.21.15.32:0/603097811 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f23200a6e90 con 0x5559c1d5f3b0 2024-04-30T08:34:18.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.804+0000 7f233f390740 10 client.25235 fetch_fsmap finished waiting for FSMap version 27 2024-04-30T08:34:18.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.805+0000 7f233f390740 10 client.25235 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-30T08:34:18.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.805+0000 7f233f390740 20 client.25235 populate_metadata read hostname 'smithi032' 2024-04-30T08:34:18.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.805+0000 7f233f390740 1 --2- 172.21.15.32:0/603097811 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x5559c1d46e40 0x5559c2141f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.805+0000 7f233f390740 4 client.25235 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:34:18.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.805+0000 7f233f390740 1 -- 172.21.15.32:0/603097811 --> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5559c209c3a0 con 0x5559c1d46e40 2024-04-30T08:34:18.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.805+0000 7f2335000640 1 --2- 172.21.15.32:0/603097811 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x5559c1d46e40 0x5559c2141f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.806+0000 7f2335000640 1 --2- 172.21.15.32:0/603097811 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x5559c1d46e40 0x5559c2141f90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=73017ffc02eeb10e server_cookie=b92599ab4283f92e in_seq=0 out_seq=0 2024-04-30T08:34:18.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.806+0000 7f231b400640 10 client.25235 ms_handle_connect on v2:172.21.15.32:6838/4160386524 2024-04-30T08:34:18.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.806+0000 7f231b400640 1 -- 172.21.15.32:0/603097811 <== mds.1 v2:172.21.15.32:6838/4160386524 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5559c209c3a0 con 0x5559c1d46e40 2024-04-30T08:34:18.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.806+0000 7f231b400640 10 client.25235 handle_command_reply: tid=0 2024-04-30T08:34:18.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.807+0000 7f233f390740 1 client.25235 shutdown 2024-04-30T08:34:18.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.807+0000 7f233f390740 1 -- 172.21.15.32:0/603097811 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5559c1d5f3b0 msgr2=0x5559c21340d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:18.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.807+0000 7f233f390740 1 --2- 172.21.15.32:0/603097811 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5559c1d5f3b0 0x5559c21340d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x5559c212a250 tx=0x7f23200a79d0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.807+0000 7f233f390740 1 -- 172.21.15.32:0/603097811 shutdown_connections 2024-04-30T08:34:18.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.807+0000 7f233f390740 1 --2- 172.21.15.32:0/603097811 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x5559c1d46e40 0x5559c2141f90 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.807+0000 7f233f390740 1 --2- 172.21.15.32:0/603097811 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559c1d88270 0x5559c213b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.807+0000 7f233f390740 1 --2- 172.21.15.32:0/603097811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559c1d67590 0x5559c2134610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.807+0000 7f233f390740 1 --2- 172.21.15.32:0/603097811 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5559c1d5f3b0 0x5559c21340d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.807+0000 7f231b400640 0 client.25235 ms_handle_reset on v2:172.21.15.32:6838/4160386524 2024-04-30T08:34:18.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.807+0000 7f233f390740 1 -- 172.21.15.32:0/603097811 >> 172.21.15.32:0/603097811 conn(0x5559c1d7c3c0 msgr2=0x5559c2124d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:18.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.807+0000 7f233f390740 1 -- 172.21.15.32:0/603097811 shutdown_connections 2024-04-30T08:34:18.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.807+0000 7f233f390740 1 -- 172.21.15.32:0/603097811 wait complete. 2024-04-30T08:34:18.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.807+0000 7f233f390740 20 client.25235 trim_cache size 0 max 16384 2024-04-30T08:34:18.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.828+0000 7f233f390740 1 Processor -- start 2024-04-30T08:34:18.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.828+0000 7f233f390740 1 -- start start 2024-04-30T08:34:18.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.829+0000 7f233f390740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559c1d5f3b0 0x5559c213fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.829+0000 7f233f390740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5559c1d67590 0x5559c21292d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.829+0000 7f233f390740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559c1d88270 0x5559c213b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.829+0000 7f233f390740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5559c21402d0 con 0x5559c1d88270 2024-04-30T08:34:18.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.829+0000 7f233f390740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x5559c21409a0 con 0x5559c1d67590 2024-04-30T08:34:18.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.829+0000 7f233f390740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5559c2141070 con 0x5559c1d5f3b0 2024-04-30T08:34:18.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.829+0000 7f2335a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559c1d5f3b0 0x5559c213fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.829+0000 7f2335a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559c1d5f3b0 0x5559c213fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:59632/0 (socket says 172.21.15.32:59632) 2024-04-30T08:34:18.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.829+0000 7f2335a00640 1 -- 172.21.15.32:0/562946350 learned_addr learned my addr 172.21.15.32:0/562946350 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:18.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.829+0000 7f2336400640 1 --2- 172.21.15.32:0/562946350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559c1d88270 0x5559c213b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.829+0000 7f2335000640 1 --2- 172.21.15.32:0/562946350 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5559c1d67590 0x5559c21292d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.830+0000 7f2335a00640 1 -- 172.21.15.32:0/562946350 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5559c1d67590 msgr2=0x5559c21292d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:18.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.830+0000 7f2335a00640 1 --2- 172.21.15.32:0/562946350 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5559c1d67590 0x5559c21292d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.830+0000 7f2335a00640 1 -- 172.21.15.32:0/562946350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559c1d88270 msgr2=0x5559c213b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:18.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.830+0000 7f2335a00640 1 --2- 172.21.15.32:0/562946350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559c1d88270 0x5559c213b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.830+0000 7f2335a00640 1 -- 172.21.15.32:0/562946350 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5559c2135230 con 0x5559c1d5f3b0 2024-04-30T08:34:18.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.830+0000 7f2335a00640 1 --2- 172.21.15.32:0/562946350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559c1d5f3b0 0x5559c213fa30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f23200a77c0 tx=0x7f2320000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69c2ce954207aacc server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:18.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.830+0000 7f2326000640 1 -- 172.21.15.32:0/562946350 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23200a7280 con 0x5559c1d5f3b0 2024-04-30T08:34:18.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.830+0000 7f2326000640 1 -- 172.21.15.32:0/562946350 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f23200c6070 con 0x5559c1d5f3b0 2024-04-30T08:34:18.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.830+0000 7f2326000640 1 -- 172.21.15.32:0/562946350 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23200a3da0 con 0x5559c1d5f3b0 2024-04-30T08:34:18.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.831+0000 7f233f390740 1 -- 172.21.15.32:0/562946350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559c1d5f3b0 msgr2=0x5559c213fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:18.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.831+0000 7f233f390740 1 --2- 172.21.15.32:0/562946350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559c1d5f3b0 0x5559c213fa30 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f23200a77c0 tx=0x7f2320000f40 comp rx=0 tx=0).stop 2024-04-30T08:34:18.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.831+0000 7f233f390740 1 -- 172.21.15.32:0/562946350 shutdown_connections 2024-04-30T08:34:18.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.831+0000 7f233f390740 1 --2- 172.21.15.32:0/562946350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559c1d88270 0x5559c213b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.831+0000 7f233f390740 1 --2- 172.21.15.32:0/562946350 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5559c1d67590 0x5559c21292d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.831+0000 7f233f390740 1 --2- 172.21.15.32:0/562946350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559c1d5f3b0 0x5559c213fa30 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.831+0000 7f233f390740 1 -- 172.21.15.32:0/562946350 >> 172.21.15.32:0/562946350 conn(0x5559c1d7c3c0 msgr2=0x5559c2128bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:18.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.832+0000 7f233f390740 1 -- 172.21.15.32:0/562946350 shutdown_connections 2024-04-30T08:34:18.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.832+0000 7f233f390740 1 -- 172.21.15.32:0/562946350 wait complete. 2024-04-30T08:34:18.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.832+0000 7f233f390740 1 -- start start 2024-04-30T08:34:18.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.832+0000 7f233f390740 1 Processor -- start 2024-04-30T08:34:18.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.833+0000 7f233f390740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559c1d5f3b0 0x5559c21371f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.833+0000 7f233f390740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559c1d67590 0x5559c2137730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.833+0000 7f233f390740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5559c1d88270 0x5559c213a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.833+0000 7f233f390740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5559c2141e10 con 0x5559c1d67590 2024-04-30T08:34:18.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.833+0000 7f233f390740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x5559c2141740 con 0x5559c1d88270 2024-04-30T08:34:18.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.833+0000 7f233f390740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5559c21424e0 con 0x5559c1d5f3b0 2024-04-30T08:34:18.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.833+0000 7f233f390740 10 client.0 _ll_register_callbacks cb 0x7f23200a3ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:34:18.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.833+0000 7f2335000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559c1d67590 0x5559c2137730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.833+0000 7f2336400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5559c1d88270 0x5559c213a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.833+0000 7f2335a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559c1d5f3b0 0x5559c21371f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.833+0000 7f2335000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559c1d67590 0x5559c2137730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57094/0 (socket says 172.21.15.32:57094) 2024-04-30T08:34:18.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.833+0000 7f2335000640 1 -- 172.21.15.32:0/3562798425 learned_addr learned my addr 172.21.15.32:0/3562798425 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:18.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.834+0000 7f2335000640 1 -- 172.21.15.32:0/3562798425 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559c1d5f3b0 msgr2=0x5559c21371f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:18.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.834+0000 7f2335000640 1 --2- 172.21.15.32:0/3562798425 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559c1d5f3b0 0x5559c21371f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.834+0000 7f2335000640 1 -- 172.21.15.32:0/3562798425 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5559c1d88270 msgr2=0x5559c213a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:18.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.834+0000 7f2335000640 1 --2- 172.21.15.32:0/3562798425 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5559c1d88270 0x5559c213a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.834+0000 7f2335000640 1 -- 172.21.15.32:0/3562798425 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5559c213abe0 con 0x5559c1d67590 2024-04-30T08:34:18.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.834+0000 7f2335a00640 1 --2- 172.21.15.32:0/3562798425 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559c1d5f3b0 0x5559c21371f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:34:18.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.834+0000 7f2336400640 1 --2- 172.21.15.32:0/3562798425 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5559c1d88270 0x5559c213a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:34:18.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.834+0000 7f2335000640 1 --2- 172.21.15.32:0/3562798425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559c1d67590 0x5559c2137730 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f231c009340 tx=0x7f231c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f5cfb75858f2878 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:18.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.834+0000 7f231b400640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:34:18.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.834+0000 7f231b400640 1 -- 172.21.15.32:0/3562798425 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f231c035040 con 0x5559c1d67590 2024-04-30T08:34:18.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.834+0000 7f231b400640 1 -- 172.21.15.32:0/3562798425 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f231c0467e0 con 0x5559c1d67590 2024-04-30T08:34:18.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.834+0000 7f233f390740 1 -- 172.21.15.32:0/3562798425 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23200c6070 con 0x5559c1d67590 2024-04-30T08:34:18.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.835+0000 7f231b400640 1 -- 172.21.15.32:0/3562798425 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f231c0469e0 con 0x5559c1d67590 2024-04-30T08:34:18.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.835+0000 7f233f390740 10 client.15117 fetch_fsmap learned FSMap version 27 2024-04-30T08:34:18.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.835+0000 7f233f390740 1 -- 172.21.15.32:0/3562798425 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5559c213bf60 con 0x5559c1d67590 2024-04-30T08:34:18.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.835+0000 7f231b400640 1 -- 172.21.15.32:0/3562798425 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f23200c6070 con 0x5559c1d67590 2024-04-30T08:34:18.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.835+0000 7f231b400640 1 -- 172.21.15.32:0/3562798425 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f231c031730 con 0x5559c1d67590 2024-04-30T08:34:18.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.835+0000 7f233f390740 10 client.15117 fetch_fsmap finished waiting for FSMap version 27 2024-04-30T08:34:18.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.835+0000 7f233f390740 10 client.15117 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-30T08:34:18.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.835+0000 7f233f390740 20 client.15117 populate_metadata read hostname 'smithi032' 2024-04-30T08:34:18.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.835+0000 7f233f390740 1 --2- 172.21.15.32:0/3562798425 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x5559c1d1c280 0x5559c21ae410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:18.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.835+0000 7f233f390740 4 client.15117 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-30T08:34:18.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.835+0000 7f233f390740 1 -- 172.21.15.32:0/3562798425 --> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] -- command(tid 0: {"prefix": "status"}) -- 0x5559c2096210 con 0x5559c1d1c280 2024-04-30T08:34:18.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.836+0000 7f2335a00640 1 --2- 172.21.15.32:0/3562798425 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x5559c1d1c280 0x5559c21ae410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:18.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.836+0000 7f2335a00640 1 --2- 172.21.15.32:0/3562798425 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x5559c1d1c280 0x5559c21ae410 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=82ddef6265a8aa1b server_cookie=ac7b90c916d17337 in_seq=0 out_seq=0 2024-04-30T08:34:18.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.836+0000 7f231b400640 10 client.15117 ms_handle_connect on v2:172.21.15.32:6838/4160386524 2024-04-30T08:34:18.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.837+0000 7f231b400640 1 -- 172.21.15.32:0/3562798425 <== mds.1 v2:172.21.15.32:6838/4160386524 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5559c2096210 con 0x5559c1d1c280 2024-04-30T08:34:18.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.837+0000 7f231b400640 10 client.15117 handle_command_reply: tid=0 2024-04-30T08:34:18.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.837+0000 7f233f390740 1 client.15117 shutdown 2024-04-30T08:34:18.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.837+0000 7f233f390740 1 -- 172.21.15.32:0/3562798425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559c1d67590 msgr2=0x5559c2137730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:18.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.837+0000 7f233f390740 1 --2- 172.21.15.32:0/3562798425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559c1d67590 0x5559c2137730 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f231c009340 tx=0x7f231c004470 comp rx=0 tx=0).stop 2024-04-30T08:34:18.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.837+0000 7f233f390740 1 -- 172.21.15.32:0/3562798425 shutdown_connections 2024-04-30T08:34:18.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.837+0000 7f233f390740 1 --2- 172.21.15.32:0/3562798425 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x5559c1d1c280 0x5559c21ae410 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.837+0000 7f233f390740 1 --2- 172.21.15.32:0/3562798425 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5559c1d88270 0x5559c213a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.837+0000 7f233f390740 1 --2- 172.21.15.32:0/3562798425 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5559c1d67590 0x5559c2137730 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.837+0000 7f231b400640 0 client.15117 ms_handle_reset on v2:172.21.15.32:6838/4160386524 2024-04-30T08:34:18.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.837+0000 7f233f390740 1 --2- 172.21.15.32:0/3562798425 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559c1d5f3b0 0x5559c21371f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.837+0000 7f233f390740 1 -- 172.21.15.32:0/3562798425 >> 172.21.15.32:0/3562798425 conn(0x5559c1d7c3c0 msgr2=0x5559c2128bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:18.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.837+0000 7f233f390740 1 -- 172.21.15.32:0/3562798425 shutdown_connections 2024-04-30T08:34:18.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.837+0000 7f233f390740 1 -- 172.21.15.32:0/3562798425 wait complete. 2024-04-30T08:34:18.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.837+0000 7f233f390740 20 client.15117 trim_cache size 0 max 16384 2024-04-30T08:34:18.844 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-30T08:34:18.845 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "98c0a670-06ca-11ef-bc93-c7b262605968", 2024-04-30T08:34:18.845 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 1, 2024-04-30T08:34:18.845 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 24560, 2024-04-30T08:34:18.845 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-04-30T08:34:18.845 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-04-30T08:34:18.845 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-04-30T08:34:18.845 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 112.34449189199999, 2024-04-30T08:34:18.845 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 27, 2024-04-30T08:34:18.845 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 85, 2024-04-30T08:34:18.845 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 81, 2024-04-30T08:34:18.845 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 160.877365357 2024-04-30T08:34:18.845 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-30T08:34:18.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.838+0000 7f2336e00640 1 -- 172.21.15.32:0/3041264336 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f23140afa00 msgr2=0x7f23140b1f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:18.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.838+0000 7f2336e00640 1 --2- 172.21.15.32:0/3041264336 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f23140afa00 0x7f23140b1f10 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f232800a010 tx=0x7f2328009830 comp rx=0 tx=0).stop 2024-04-30T08:34:18.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.838+0000 7f2336e00640 1 -- 172.21.15.32:0/3041264336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2330075010 msgr2=0x7f233019c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:18.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.838+0000 7f2336e00640 1 --2- 172.21.15.32:0/3041264336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2330075010 0x7f233019c2d0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f2320005fd0 tx=0x7f232000ef90 comp rx=0 tx=0).stop 2024-04-30T08:34:18.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.838+0000 7f2336e00640 1 -- 172.21.15.32:0/3041264336 shutdown_connections 2024-04-30T08:34:18.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.838+0000 7f2336e00640 1 --2- 172.21.15.32:0/3041264336 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f23140afa00 0x7f23140b1f10 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.838+0000 7f2336e00640 1 --2- 172.21.15.32:0/3041264336 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f233010b900 0x7f23301a37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.838+0000 7f2336e00640 1 --2- 172.21.15.32:0/3041264336 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2330075960 0x7f233019c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.838+0000 7f2336e00640 1 --2- 172.21.15.32:0/3041264336 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2330075010 0x7f233019c2d0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:18.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.838+0000 7f2336e00640 1 -- 172.21.15.32:0/3041264336 >> 172.21.15.32:0/3041264336 conn(0x7f23300feba0 msgr2=0x7f23300ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:18.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.838+0000 7f2336e00640 1 -- 172.21.15.32:0/3041264336 shutdown_connections 2024-04-30T08:34:18.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:18.838+0000 7f2336e00640 1 -- 172.21.15.32:0/3041264336 wait complete. 2024-04-30T08:34:18.869 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-30T08:34:19.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.151+0000 7f17f6200640 1 -- 172.21.15.32:0/915257601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f0075960 msgr2=0x7f17f010b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:19.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.151+0000 7f17f6200640 1 --2- 172.21.15.32:0/915257601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f0075960 0x7f17f010b3c0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f17dc009900 tx=0x7f17dc02a3f0 comp rx=0 tx=0).stop 2024-04-30T08:34:19.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.151+0000 7f17f6200640 1 -- 172.21.15.32:0/915257601 shutdown_connections 2024-04-30T08:34:19.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.151+0000 7f17f6200640 1 --2- 172.21.15.32:0/915257601 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17f010b900 0x7f17f010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:19.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.151+0000 7f17f6200640 1 --2- 172.21.15.32:0/915257601 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f0075960 0x7f17f010b3c0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:19.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.151+0000 7f17f6200640 1 --2- 172.21.15.32:0/915257601 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17f0075010 0x7f17f0075420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:19.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.151+0000 7f17f6200640 1 -- 172.21.15.32:0/915257601 >> 172.21.15.32:0/915257601 conn(0x7f17f00feba0 msgr2=0x7f17f0101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:19.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.151+0000 7f17f6200640 1 -- 172.21.15.32:0/915257601 shutdown_connections 2024-04-30T08:34:19.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.151+0000 7f17f6200640 1 -- 172.21.15.32:0/915257601 wait complete. 2024-04-30T08:34:19.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.152+0000 7f17f6200640 1 Processor -- start 2024-04-30T08:34:19.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.153+0000 7f17f6200640 1 -- start start 2024-04-30T08:34:19.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.153+0000 7f17f6200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f0075010 0x7f17f019c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:19.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.153+0000 7f17f6200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17f0075960 0x7f17f019c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:19.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.153+0000 7f17f6200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17f010b900 0x7f17f01a37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:19.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.153+0000 7f17f6200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f17f01116c0 con 0x7f17f0075010 2024-04-30T08:34:19.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.153+0000 7f17f6200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f17f0111540 con 0x7f17f010b900 2024-04-30T08:34:19.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.153+0000 7f17f6200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f17f0111840 con 0x7f17f0075960 2024-04-30T08:34:19.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.153+0000 7f17f4e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f0075010 0x7f17f019c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:19.159 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.153+0000 7f17e7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17f0075960 0x7f17f019c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:19.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.153+0000 7f17f5800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17f010b900 0x7f17f01a37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:19.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.153+0000 7f17f5800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17f010b900 0x7f17f01a37a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:55670/0 (socket says 172.21.15.32:55670) 2024-04-30T08:34:19.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.153+0000 7f17f4e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f0075010 0x7f17f019c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57112/0 (socket says 172.21.15.32:57112) 2024-04-30T08:34:19.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.153+0000 7f17e7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17f0075960 0x7f17f019c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:59668/0 (socket says 172.21.15.32:59668) 2024-04-30T08:34:19.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.153+0000 7f17f5800640 1 -- 172.21.15.32:0/441997049 learned_addr learned my addr 172.21.15.32:0/441997049 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:19.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.154+0000 7f17f5800640 1 -- 172.21.15.32:0/441997049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17f0075960 msgr2=0x7f17f019c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:19.160 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.154+0000 7f17f5800640 1 --2- 172.21.15.32:0/441997049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17f0075960 0x7f17f019c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:19.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.154+0000 7f17f5800640 1 -- 172.21.15.32:0/441997049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f0075010 msgr2=0x7f17f019c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:19.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.154+0000 7f17f5800640 1 --2- 172.21.15.32:0/441997049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f0075010 0x7f17f019c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:19.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.154+0000 7f17f5800640 1 -- 172.21.15.32:0/441997049 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17f01a3e10 con 0x7f17f010b900 2024-04-30T08:34:19.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.154+0000 7f17f5800640 1 --2- 172.21.15.32:0/441997049 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17f010b900 0x7f17f01a37a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f17e800eb40 tx=0x7f17e800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:19.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.154+0000 7f17ee000640 1 -- 172.21.15.32:0/441997049 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e8004b20 con 0x7f17f010b900 2024-04-30T08:34:19.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.154+0000 7f17f6200640 1 -- 172.21.15.32:0/441997049 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17f01a4100 con 0x7f17f010b900 2024-04-30T08:34:19.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.154+0000 7f17ee000640 1 -- 172.21.15.32:0/441997049 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17e8004cc0 con 0x7f17f010b900 2024-04-30T08:34:19.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.154+0000 7f17ee000640 1 -- 172.21.15.32:0/441997049 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e8010720 con 0x7f17f010b900 2024-04-30T08:34:19.162 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.155+0000 7f17f6200640 1 -- 172.21.15.32:0/441997049 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17f01a4650 con 0x7f17f010b900 2024-04-30T08:34:19.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.158+0000 7f17fe784740 1 Processor -- start 2024-04-30T08:34:19.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.158+0000 7f17ee000640 1 -- 172.21.15.32:0/441997049 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f17e8005560 con 0x7f17f010b900 2024-04-30T08:34:19.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.158+0000 7f17fe784740 1 -- start start 2024-04-30T08:34:19.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.159+0000 7f17fe784740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ae14c71270 0x55ae1501bd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:19.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.159+0000 7f17fe784740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae14c483b0 0x55ae1501c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:19.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.159+0000 7f17fe784740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae14c50590 0x55ae1501cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:19.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.159+0000 7f17fe784740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ae14f77790 con 0x55ae14c50590 2024-04-30T08:34:19.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.159+0000 7f17fe784740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55ae14f1cf20 con 0x55ae14c71270 2024-04-30T08:34:19.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.159+0000 7f17fe784740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ae14f73750 con 0x55ae14c483b0 2024-04-30T08:34:19.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.159+0000 7f17e7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae14c50590 0x55ae1501cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:19.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.159+0000 7f17e7e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ae14c71270 0x55ae1501bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:19.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.159+0000 7f17f4e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae14c483b0 0x55ae1501c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:19.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.159+0000 7f17f4e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae14c483b0 0x55ae1501c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:59678/0 (socket says 172.21.15.32:59678) 2024-04-30T08:34:19.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.159+0000 7f17f4e00640 1 -- 172.21.15.32:0/2222172283 learned_addr learned my addr 172.21.15.32:0/2222172283 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:19.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.160+0000 7f17f4e00640 1 -- 172.21.15.32:0/2222172283 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ae14c71270 msgr2=0x55ae1501bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:19.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.160+0000 7f17f4e00640 1 --2- 172.21.15.32:0/2222172283 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ae14c71270 0x55ae1501bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:19.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.160+0000 7f17f4e00640 1 -- 172.21.15.32:0/2222172283 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae14c50590 msgr2=0x55ae1501cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:19.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.160+0000 7f17f4e00640 1 --2- 172.21.15.32:0/2222172283 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae14c50590 0x55ae1501cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:19.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.160+0000 7f17f4e00640 1 -- 172.21.15.32:0/2222172283 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae14c892d0 con 0x55ae14c483b0 2024-04-30T08:34:19.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.160+0000 7f17ee000640 1 --2- 172.21.15.32:0/441997049 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f17cc0afae0 0x7f17cc0b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:19.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.160+0000 7f17f4e00640 1 --2- 172.21.15.32:0/2222172283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae14c483b0 0x55ae1501c2c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f17d4009c90 tx=0x7f17d4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=be5d0584a0b977e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:19.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.160+0000 7f17ee000640 1 -- 172.21.15.32:0/441997049 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f17e80df7c0 con 0x7f17f010b900 2024-04-30T08:34:19.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.160+0000 7f17e7e00640 1 --2- 172.21.15.32:0/441997049 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f17cc0afae0 0x7f17cc0b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:19.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.161+0000 7f17e6000640 1 -- 172.21.15.32:0/2222172283 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d401e070 con 0x55ae14c483b0 2024-04-30T08:34:19.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.161+0000 7f17e6000640 1 -- 172.21.15.32:0/2222172283 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17d4019ae0 con 0x55ae14c483b0 2024-04-30T08:34:19.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.161+0000 7f17e6000640 1 -- 172.21.15.32:0/2222172283 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d400e8d0 con 0x55ae14c483b0 2024-04-30T08:34:19.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.161+0000 7f17fe784740 1 -- 172.21.15.32:0/2222172283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae14c483b0 msgr2=0x55ae1501c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:19.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.161+0000 7f17e7e00640 1 --2- 172.21.15.32:0/441997049 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f17cc0afae0 0x7f17cc0b1ff0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f17dc009230 tx=0x7f17dc000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:19.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.161+0000 7f17fe784740 1 --2- 172.21.15.32:0/2222172283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae14c483b0 0x55ae1501c2c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f17d4009c90 tx=0x7f17d4011810 comp rx=0 tx=0).stop 2024-04-30T08:34:19.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.162+0000 7f17fe784740 1 -- 172.21.15.32:0/2222172283 shutdown_connections 2024-04-30T08:34:19.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.162+0000 7f17fe784740 1 --2- 172.21.15.32:0/2222172283 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae14c50590 0x55ae1501cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:19.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.162+0000 7f17fe784740 1 --2- 172.21.15.32:0/2222172283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae14c483b0 0x55ae1501c2c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:19.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.162+0000 7f17fe784740 1 --2- 172.21.15.32:0/2222172283 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ae14c71270 0x55ae1501bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:19.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.162+0000 7f17fe784740 1 -- 172.21.15.32:0/2222172283 >> 172.21.15.32:0/2222172283 conn(0x55ae14c653c0 msgr2=0x55ae15017550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:19.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.162+0000 7f17fe784740 1 -- 172.21.15.32:0/2222172283 shutdown_connections 2024-04-30T08:34:19.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.162+0000 7f17fe784740 1 -- 172.21.15.32:0/2222172283 wait complete. 2024-04-30T08:34:19.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.162+0000 7f17fe784740 1 -- start start 2024-04-30T08:34:19.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.163+0000 7f17fe784740 1 Processor -- start 2024-04-30T08:34:19.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.163+0000 7f17fe784740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae14c483b0 0x55ae15025880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:19.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.163+0000 7f17fe784740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ae14c50590 0x55ae15025dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:19.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.163+0000 7f17fe784740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae14c71270 0x55ae1502cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:19.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.163+0000 7f17fe784740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ae1501da60 con 0x55ae14c71270 2024-04-30T08:34:19.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.163+0000 7f17fe784740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55ae14f309c0 con 0x55ae14c50590 2024-04-30T08:34:19.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.163+0000 7f17fe784740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ae1501dbe0 con 0x55ae14c483b0 2024-04-30T08:34:19.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.163+0000 7f17fe784740 10 client.0 _ll_register_callbacks cb 0x55ae14efd120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:34:19.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.163+0000 7f17f4e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae14c483b0 0x55ae15025880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:19.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.163+0000 7f17f5800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae14c71270 0x55ae1502cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:19.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.163+0000 7f17f4e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae14c483b0 0x55ae15025880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:59682/0 (socket says 172.21.15.32:59682) 2024-04-30T08:34:19.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.163+0000 7f17f4e00640 1 -- 172.21.15.32:0/1197069701 learned_addr learned my addr 172.21.15.32:0/1197069701 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:19.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.163+0000 7f17e7e00640 1 --2- 172.21.15.32:0/1197069701 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ae14c50590 0x55ae15025dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:19.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.164+0000 7f17f4e00640 1 -- 172.21.15.32:0/1197069701 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ae14c50590 msgr2=0x55ae15025dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:19.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.164+0000 7f17f4e00640 1 --2- 172.21.15.32:0/1197069701 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ae14c50590 0x55ae15025dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:19.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.164+0000 7f17f4e00640 1 -- 172.21.15.32:0/1197069701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae14c71270 msgr2=0x55ae1502cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:19.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.164+0000 7f17f4e00640 1 --2- 172.21.15.32:0/1197069701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae14c71270 0x55ae1502cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:19.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.164+0000 7f17f4e00640 1 -- 172.21.15.32:0/1197069701 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae14d536b0 con 0x55ae14c483b0 2024-04-30T08:34:19.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.164+0000 7f17f5800640 1 --2- 172.21.15.32:0/1197069701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae14c71270 0x55ae1502cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:34:19.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.164+0000 7f17f4e00640 1 --2- 172.21.15.32:0/1197069701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae14c483b0 0x55ae15025880 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f17d401c040 tx=0x7f17d4009f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=364787f8cb4273f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:19.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.164+0000 7f17e2e00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-30T08:34:19.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.165+0000 7f17e2e00640 1 -- 172.21.15.32:0/1197069701 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d401e050 con 0x55ae14c483b0 2024-04-30T08:34:19.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.165+0000 7f17fe784740 1 -- 172.21.15.32:0/1197069701 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17d4019ae0 con 0x55ae14c483b0 2024-04-30T08:34:19.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.165+0000 7f17e2e00640 1 -- 172.21.15.32:0/1197069701 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f17d4005450 con 0x55ae14c483b0 2024-04-30T08:34:19.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.165+0000 7f17e2e00640 1 -- 172.21.15.32:0/1197069701 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f17d4032050 con 0x55ae14c483b0 2024-04-30T08:34:19.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.165+0000 7f17fe784740 10 client.25252 fetch_fsmap learned FSMap version 27 2024-04-30T08:34:19.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.165+0000 7f17fe784740 1 -- 172.21.15.32:0/1197069701 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ae1502fca0 con 0x55ae14c483b0 2024-04-30T08:34:19.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.165+0000 7f17e2e00640 1 -- 172.21.15.32:0/1197069701 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17d4019ae0 con 0x55ae14c483b0 2024-04-30T08:34:19.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.166+0000 7f17e2e00640 1 -- 172.21.15.32:0/1197069701 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f17d402d030 con 0x55ae14c483b0 2024-04-30T08:34:19.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.166+0000 7f17fe784740 10 client.25252 fetch_fsmap finished waiting for FSMap version 27 2024-04-30T08:34:19.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.166+0000 7f17fe784740 10 client.25252 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-30T08:34:19.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.166+0000 7f17fe784740 20 client.25252 populate_metadata read hostname 'smithi032' 2024-04-30T08:34:19.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.166+0000 7f17fe784740 1 --2- 172.21.15.32:0/1197069701 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55ae14c2fe40 0x55ae150334b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:19.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.166+0000 7f17fe784740 4 client.25252 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:34:19.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.166+0000 7f17fe784740 1 -- 172.21.15.32:0/1197069701 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ae14f853a0 con 0x55ae14c2fe40 2024-04-30T08:34:19.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.166+0000 7f17e7e00640 1 --2- 172.21.15.32:0/1197069701 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55ae14c2fe40 0x55ae150334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:19.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.167+0000 7f17e7e00640 1 --2- 172.21.15.32:0/1197069701 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55ae14c2fe40 0x55ae150334b0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d91c83e959c20d30 server_cookie=745654e0ce656b9e in_seq=0 out_seq=0 2024-04-30T08:34:19.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.167+0000 7f17e2e00640 10 client.25252 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:34:19.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.168+0000 7f17e2e00640 1 -- 172.21.15.32:0/1197069701 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ae14f853a0 con 0x55ae14c2fe40 2024-04-30T08:34:19.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.168+0000 7f17e2e00640 10 client.25252 handle_command_reply: tid=0 2024-04-30T08:34:19.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.168+0000 7f17fe784740 1 client.25252 shutdown 2024-04-30T08:34:19.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.168+0000 7f17fe784740 1 -- 172.21.15.32:0/1197069701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae14c483b0 msgr2=0x55ae15025880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:19.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.168+0000 7f17fe784740 1 --2- 172.21.15.32:0/1197069701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae14c483b0 0x55ae15025880 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f17d401c040 tx=0x7f17d4009f00 comp rx=0 tx=0).stop 2024-04-30T08:34:19.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.168+0000 7f17fe784740 1 -- 172.21.15.32:0/1197069701 shutdown_connections 2024-04-30T08:34:19.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.168+0000 7f17fe784740 1 --2- 172.21.15.32:0/1197069701 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55ae14c2fe40 0x55ae150334b0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:19.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.168+0000 7f17fe784740 1 --2- 172.21.15.32:0/1197069701 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae14c71270 0x55ae1502cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:19.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.168+0000 7f17fe784740 1 --2- 172.21.15.32:0/1197069701 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ae14c50590 0x55ae15025dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:19.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.168+0000 7f17fe784740 1 --2- 172.21.15.32:0/1197069701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae14c483b0 0x55ae15025880 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:19.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.168+0000 7f17e2e00640 0 client.25252 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:34:19.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.168+0000 7f17fe784740 1 -- 172.21.15.32:0/1197069701 >> 172.21.15.32:0/1197069701 conn(0x55ae14c653c0 msgr2=0x55ae1501a460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:19.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.168+0000 7f17fe784740 1 -- 172.21.15.32:0/1197069701 shutdown_connections 2024-04-30T08:34:19.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.168+0000 7f17fe784740 1 -- 172.21.15.32:0/1197069701 wait complete. 2024-04-30T08:34:19.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.168+0000 7f17fe784740 20 client.25252 trim_cache size 0 max 16384 2024-04-30T08:34:19.190 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.189+0000 7f17fe784740 1 Processor -- start 2024-04-30T08:34:19.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:18 smithi032 ceph-mon[28331]: pgmap v399: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-30T08:34:19.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.189+0000 7f17fe784740 1 -- start start 2024-04-30T08:34:19.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.189+0000 7f17fe784740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae14c653c0 0x55ae1501aa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:19.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.190+0000 7f17fe784740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ae14c483b0 0x55ae1502cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:19.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.190+0000 7f17fe784740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae14c50590 0x55ae1502d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:19.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.190+0000 7f17fe784740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ae150317f0 con 0x55ae14c653c0 2024-04-30T08:34:19.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.190+0000 7f17fe784740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55ae15031ec0 con 0x55ae14c483b0 2024-04-30T08:34:19.192 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.190+0000 7f17fe784740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ae15032590 con 0x55ae14c50590 2024-04-30T08:34:19.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.190+0000 7f17f4e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae14c653c0 0x55ae1501aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:19.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.190+0000 7f17f4e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae14c653c0 0x55ae1501aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57144/0 (socket says 172.21.15.32:57144) 2024-04-30T08:34:19.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.190+0000 7f17f4e00640 1 -- 172.21.15.32:0/1198218295 learned_addr learned my addr 172.21.15.32:0/1198218295 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:19.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.190+0000 7f17e7e00640 1 --2- 172.21.15.32:0/1198218295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ae14c483b0 0x55ae1502cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:19.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.190+0000 7f17f5800640 1 --2- 172.21.15.32:0/1198218295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae14c50590 0x55ae1502d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:19.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.190+0000 7f17f4e00640 1 -- 172.21.15.32:0/1198218295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae14c50590 msgr2=0x55ae1502d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:19.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.190+0000 7f17f4e00640 1 --2- 172.21.15.32:0/1198218295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae14c50590 0x55ae1502d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:19.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.190+0000 7f17f4e00640 1 -- 172.21.15.32:0/1198218295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ae14c483b0 msgr2=0x55ae1502cc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:19.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.190+0000 7f17f4e00640 1 --2- 172.21.15.32:0/1198218295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ae14c483b0 0x55ae1502cc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:19.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.190+0000 7f17f4e00640 1 -- 172.21.15.32:0/1198218295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae1501c3a0 con 0x55ae14c653c0 2024-04-30T08:34:19.193 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.191+0000 7f17f4e00640 1 --2- 172.21.15.32:0/1198218295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae14c653c0 0x55ae1501aa60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f17d4012fd0 tx=0x7f17d4005df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=223be39d0ac9c8cc server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:19.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.191+0000 7f17e5600640 1 -- 172.21.15.32:0/1198218295 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d401e040 con 0x55ae14c653c0 2024-04-30T08:34:19.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.191+0000 7f17e5600640 1 -- 172.21.15.32:0/1198218295 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17d401f040 con 0x55ae14c653c0 2024-04-30T08:34:19.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.191+0000 7f17e5600640 1 -- 172.21.15.32:0/1198218295 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17d4025450 con 0x55ae14c653c0 2024-04-30T08:34:19.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.192+0000 7f17fe784740 1 -- 172.21.15.32:0/1198218295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae14c653c0 msgr2=0x55ae1501aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:19.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.192+0000 7f17fe784740 1 --2- 172.21.15.32:0/1198218295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae14c653c0 0x55ae1501aa60 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f17d4012fd0 tx=0x7f17d4005df0 comp rx=0 tx=0).stop 2024-04-30T08:34:19.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.192+0000 7f17fe784740 1 -- 172.21.15.32:0/1198218295 shutdown_connections 2024-04-30T08:34:19.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.192+0000 7f17fe784740 1 --2- 172.21.15.32:0/1198218295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae14c50590 0x55ae1502d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:19.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.192+0000 7f17fe784740 1 --2- 172.21.15.32:0/1198218295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ae14c483b0 0x55ae1502cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:19.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.192+0000 7f17fe784740 1 --2- 172.21.15.32:0/1198218295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae14c653c0 0x55ae1501aa60 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:19.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.192+0000 7f17fe784740 1 -- 172.21.15.32:0/1198218295 >> 172.21.15.32:0/1198218295 conn(0x55ae15009e90 msgr2=0x55ae1501a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:19.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.192+0000 7f17fe784740 1 -- 172.21.15.32:0/1198218295 shutdown_connections 2024-04-30T08:34:19.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.192+0000 7f17fe784740 1 -- 172.21.15.32:0/1198218295 wait complete. 2024-04-30T08:34:19.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.193+0000 7f17fe784740 1 -- start start 2024-04-30T08:34:19.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.193+0000 7f17fe784740 1 Processor -- start 2024-04-30T08:34:19.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.193+0000 7f17fe784740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ae14c483b0 0x55ae150285e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:19.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.193+0000 7f17fe784740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae14c50590 0x55ae15028b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:19.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.193+0000 7f17fe784740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae14c653c0 0x55ae1502ba60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:19.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.193+0000 7f17fe784740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ae15033330 con 0x55ae14c653c0 2024-04-30T08:34:19.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.193+0000 7f17fe784740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55ae15032c60 con 0x55ae14c483b0 2024-04-30T08:34:19.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.193+0000 7f17fe784740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ae15033a00 con 0x55ae14c50590 2024-04-30T08:34:19.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.194+0000 7f17fe784740 10 client.0 _ll_register_callbacks cb 0x7f17d401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:34:19.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.194+0000 7f17f5800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae14c653c0 0x55ae1502ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:19.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.194+0000 7f17e7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae14c50590 0x55ae15028b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:19.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.194+0000 7f17f5800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae14c653c0 0x55ae1502ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:57148/0 (socket says 172.21.15.32:57148) 2024-04-30T08:34:19.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.194+0000 7f17f5800640 1 -- 172.21.15.32:0/1830431428 learned_addr learned my addr 172.21.15.32:0/1830431428 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:19.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.194+0000 7f17f4e00640 1 --2- 172.21.15.32:0/1830431428 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ae14c483b0 0x55ae150285e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:19.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.194+0000 7f17f5800640 1 -- 172.21.15.32:0/1830431428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae14c50590 msgr2=0x55ae15028b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:19.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.194+0000 7f17f5800640 1 --2- 172.21.15.32:0/1830431428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae14c50590 0x55ae15028b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:19.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.194+0000 7f17f5800640 1 -- 172.21.15.32:0/1830431428 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ae14c483b0 msgr2=0x55ae150285e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:19.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.194+0000 7f17f5800640 1 --2- 172.21.15.32:0/1830431428 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ae14c483b0 0x55ae150285e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:19.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.194+0000 7f17f5800640 1 -- 172.21.15.32:0/1830431428 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae1502c020 con 0x55ae14c653c0 2024-04-30T08:34:19.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.194+0000 7f17e7e00640 1 --2- 172.21.15.32:0/1830431428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae14c50590 0x55ae15028b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:34:19.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.195+0000 7f17f5800640 1 --2- 172.21.15.32:0/1830431428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae14c653c0 0x55ae1502ba60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f17e800cfa0 tx=0x7f17e80a7d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e96aee4730daf716 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:19.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.195+0000 7f17e2e00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:34:19.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.195+0000 7f17e2e00640 1 -- 172.21.15.32:0/1830431428 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17e80c0020 con 0x55ae14c653c0 2024-04-30T08:34:19.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.195+0000 7f17e2e00640 1 -- 172.21.15.32:0/1830431428 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f17e80a8bc0 con 0x55ae14c653c0 2024-04-30T08:34:19.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.195+0000 7f17fe784740 1 -- 172.21.15.32:0/1830431428 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17d401f040 con 0x55ae14c653c0 2024-04-30T08:34:19.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.195+0000 7f17e2e00640 1 -- 172.21.15.32:0/1830431428 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f17e80a73a0 con 0x55ae14c653c0 2024-04-30T08:34:19.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.195+0000 7f17fe784740 10 client.15135 fetch_fsmap learned FSMap version 27 2024-04-30T08:34:19.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.195+0000 7f17fe784740 1 -- 172.21.15.32:0/1830431428 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55ae1502d370 con 0x55ae14c653c0 2024-04-30T08:34:19.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.195+0000 7f17e2e00640 1 -- 172.21.15.32:0/1830431428 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f17d401f040 con 0x55ae14c653c0 2024-04-30T08:34:19.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.196+0000 7f17e2e00640 1 -- 172.21.15.32:0/1830431428 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f17e80bcce0 con 0x55ae14c653c0 2024-04-30T08:34:19.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.196+0000 7f17fe784740 10 client.15135 fetch_fsmap finished waiting for FSMap version 27 2024-04-30T08:34:19.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.196+0000 7f17fe784740 10 client.15135 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-30T08:34:19.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.196+0000 7f17fe784740 20 client.15135 populate_metadata read hostname 'smithi032' 2024-04-30T08:34:19.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.196+0000 7f17fe784740 1 --2- 172.21.15.32:0/1830431428 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55ae14c71270 0x55ae1509cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:19.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.196+0000 7f17fe784740 4 client.15135 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-30T08:34:19.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.196+0000 7f17fe784740 1 -- 172.21.15.32:0/1830431428 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "status"}) -- 0x55ae14f7f210 con 0x55ae14c71270 2024-04-30T08:34:19.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.196+0000 7f17f4e00640 1 --2- 172.21.15.32:0/1830431428 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55ae14c71270 0x55ae1509cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:19.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.197+0000 7f17f4e00640 1 --2- 172.21.15.32:0/1830431428 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55ae14c71270 0x55ae1509cce0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dec979fe7fadf414 server_cookie=74beff05fbb789bb in_seq=0 out_seq=0 2024-04-30T08:34:19.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.197+0000 7f17e2e00640 10 client.15135 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:34:19.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.197+0000 7f17e2e00640 1 -- 172.21.15.32:0/1830431428 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55ae14f7f210 con 0x55ae14c71270 2024-04-30T08:34:19.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.197+0000 7f17e2e00640 10 client.15135 handle_command_reply: tid=0 2024-04-30T08:34:19.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.197+0000 7f17fe784740 1 client.15135 shutdown 2024-04-30T08:34:19.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.197+0000 7f17fe784740 1 -- 172.21.15.32:0/1830431428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae14c653c0 msgr2=0x55ae1502ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:19.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.197+0000 7f17fe784740 1 --2- 172.21.15.32:0/1830431428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae14c653c0 0x55ae1502ba60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f17e800cfa0 tx=0x7f17e80a7d80 comp rx=0 tx=0).stop 2024-04-30T08:34:19.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.197+0000 7f17fe784740 1 -- 172.21.15.32:0/1830431428 shutdown_connections 2024-04-30T08:34:19.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.197+0000 7f17fe784740 1 --2- 172.21.15.32:0/1830431428 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55ae14c71270 0x55ae1509cce0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:19.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.197+0000 7f17fe784740 1 --2- 172.21.15.32:0/1830431428 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ae14c653c0 0x55ae1502ba60 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:19.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.197+0000 7f17fe784740 1 --2- 172.21.15.32:0/1830431428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ae14c50590 0x55ae15028b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:19.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.197+0000 7f17fe784740 1 --2- 172.21.15.32:0/1830431428 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ae14c483b0 0x55ae150285e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:19.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.197+0000 7f17fe784740 1 -- 172.21.15.32:0/1830431428 >> 172.21.15.32:0/1830431428 conn(0x55ae15009e90 msgr2=0x55ae1501a350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:19.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.197+0000 7f17e2e00640 0 client.15135 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:34:19.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.197+0000 7f17fe784740 1 -- 172.21.15.32:0/1830431428 shutdown_connections 2024-04-30T08:34:19.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.197+0000 7f17fe784740 1 -- 172.21.15.32:0/1830431428 wait complete. 2024-04-30T08:34:19.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.198+0000 7f17fe784740 20 client.15135 trim_cache size 0 max 16384 2024-04-30T08:34:19.203 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-30T08:34:19.203 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "98c0a670-06ca-11ef-bc93-c7b262605968", 2024-04-30T08:34:19.203 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-04-30T08:34:19.203 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 24590, 2024-04-30T08:34:19.203 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-04-30T08:34:19.203 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-04-30T08:34:19.203 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-04-30T08:34:19.203 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 145.96622024600001, 2024-04-30T08:34:19.204 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 27, 2024-04-30T08:34:19.204 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 85, 2024-04-30T08:34:19.204 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 81, 2024-04-30T08:34:19.204 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 154.40885512200001 2024-04-30T08:34:19.204 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-30T08:34:19.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.198+0000 7f17f6200640 1 -- 172.21.15.32:0/441997049 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f17cc0afae0 msgr2=0x7f17cc0b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:19.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.198+0000 7f17f6200640 1 --2- 172.21.15.32:0/441997049 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f17cc0afae0 0x7f17cc0b1ff0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f17dc009230 tx=0x7f17dc000bc0 comp rx=0 tx=0).stop 2024-04-30T08:34:19.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.198+0000 7f17f6200640 1 -- 172.21.15.32:0/441997049 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17f010b900 msgr2=0x7f17f01a37a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:19.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.198+0000 7f17f6200640 1 --2- 172.21.15.32:0/441997049 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17f010b900 0x7f17f01a37a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f17e800eb40 tx=0x7f17e800c2d0 comp rx=0 tx=0).stop 2024-04-30T08:34:19.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.198+0000 7f17f6200640 1 -- 172.21.15.32:0/441997049 shutdown_connections 2024-04-30T08:34:19.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.198+0000 7f17f6200640 1 --2- 172.21.15.32:0/441997049 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f17cc0afae0 0x7f17cc0b1ff0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:19.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.198+0000 7f17f6200640 1 --2- 172.21.15.32:0/441997049 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f17f010b900 0x7f17f01a37a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:19.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.198+0000 7f17f6200640 1 --2- 172.21.15.32:0/441997049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f17f0075960 0x7f17f019c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:19.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.198+0000 7f17f6200640 1 --2- 172.21.15.32:0/441997049 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f17f0075010 0x7f17f019c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:19.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.198+0000 7f17f6200640 1 -- 172.21.15.32:0/441997049 >> 172.21.15.32:0/441997049 conn(0x7f17f00feba0 msgr2=0x7f17f00ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:19.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.199+0000 7f17f6200640 1 -- 172.21.15.32:0/441997049 shutdown_connections 2024-04-30T08:34:19.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:19.199+0000 7f17f6200640 1 -- 172.21.15.32:0/441997049 wait complete. 2024-04-30T08:34:19.231 INFO:tasks.cephadm:Setup complete, yielding 2024-04-30T08:34:19.231 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-30T08:34:19.247 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi032.front.sepia.ceph.com 2024-04-30T08:34:19.247 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-30T08:34:19.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:18 smithi071 ceph-mon[33385]: pgmap v399: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-30T08:34:19.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:18 smithi189 ceph-mon[33038]: pgmap v399: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-30T08:34:19.719 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:34:21.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:20 smithi032 ceph-mon[28331]: pgmap v400: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-30T08:34:21.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:20 smithi071 ceph-mon[33385]: pgmap v400: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-30T08:34:21.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:20 smithi189 ceph-mon[33038]: pgmap v400: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-30T08:34:22.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.138+0000 7f539d800700 1 -- 172.21.15.32:0/2020380862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53981042c0 msgr2=0x7f53981046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:22.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.138+0000 7f539d800700 1 --2- 172.21.15.32:0/2020380862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53981042c0 0x7f53981046d0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f5388009990 tx=0x7f5388009ca0 comp rx=0 tx=0).stop 2024-04-30T08:34:22.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.138+0000 7f539d800700 1 -- 172.21.15.32:0/2020380862 shutdown_connections 2024-04-30T08:34:22.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.138+0000 7f539d800700 1 --2- 172.21.15.32:0/2020380862 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5398105d90 0x7f539810c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:22.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.138+0000 7f539d800700 1 --2- 172.21.15.32:0/2020380862 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53981042c0 0x7f53981046d0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:22.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.138+0000 7f539d800700 1 --2- 172.21.15.32:0/2020380862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53981053f0 0x7f5398105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:22.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.138+0000 7f539d800700 1 -- 172.21.15.32:0/2020380862 >> 172.21.15.32:0/2020380862 conn(0x7f53980ff920 msgr2=0x7f5398101db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:22.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.138+0000 7f539d800700 1 -- 172.21.15.32:0/2020380862 shutdown_connections 2024-04-30T08:34:22.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.138+0000 7f539d800700 1 -- 172.21.15.32:0/2020380862 wait complete. 2024-04-30T08:34:22.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.139+0000 7f539d800700 1 Processor -- start 2024-04-30T08:34:22.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.140+0000 7f539d800700 1 -- start start 2024-04-30T08:34:22.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.140+0000 7f539d800700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53981042c0 0x7f539819a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:22.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.140+0000 7f539d800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53981053f0 0x7f539819ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:22.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.140+0000 7f539d800700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5398105d90 0x7f53981a1db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:22.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.140+0000 7f539d800700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f539810fd60 con 0x7f5398105d90 2024-04-30T08:34:22.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.140+0000 7f539d800700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f539810fbe0 con 0x7f53981042c0 2024-04-30T08:34:22.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.140+0000 7f539d800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f539810fee0 con 0x7f53981053f0 2024-04-30T08:34:22.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.140+0000 7f5396a00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53981042c0 0x7f539819a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:22.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.140+0000 7f5396a00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53981042c0 0x7f539819a8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:55732/0 (socket says 172.21.15.32:55732) 2024-04-30T08:34:22.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.140+0000 7f5396a00700 1 -- 172.21.15.32:0/3553134176 learned_addr learned my addr 172.21.15.32:0/3553134176 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:22.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.140+0000 7f5396000700 1 --2- 172.21.15.32:0/3553134176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53981053f0 0x7f539819ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:22.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.140+0000 7f5397400700 1 --2- 172.21.15.32:0/3553134176 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5398105d90 0x7f53981a1db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:22.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.141+0000 7f5396a00700 1 -- 172.21.15.32:0/3553134176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53981053f0 msgr2=0x7f539819ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:22.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.141+0000 7f5396a00700 1 --2- 172.21.15.32:0/3553134176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53981053f0 0x7f539819ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:22.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.141+0000 7f5396a00700 1 -- 172.21.15.32:0/3553134176 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5398105d90 msgr2=0x7f53981a1db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:22.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.141+0000 7f5396a00700 1 --2- 172.21.15.32:0/3553134176 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5398105d90 0x7f53981a1db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:22.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.141+0000 7f5396a00700 1 -- 172.21.15.32:0/3553134176 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53981a2430 con 0x7f53981042c0 2024-04-30T08:34:22.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.141+0000 7f5396a00700 1 --2- 172.21.15.32:0/3553134176 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53981042c0 0x7f539819a8d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5388000c00 tx=0x7f53880057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:22.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.141+0000 7f538f400700 1 -- 172.21.15.32:0/3553134176 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5388026070 con 0x7f53981042c0 2024-04-30T08:34:22.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.142+0000 7f538f400700 1 -- 172.21.15.32:0/3553134176 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5388004000 con 0x7f53981042c0 2024-04-30T08:34:22.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.142+0000 7f539d800700 1 -- 172.21.15.32:0/3553134176 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5398076a50 con 0x7f53981042c0 2024-04-30T08:34:22.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.142+0000 7f538f400700 1 -- 172.21.15.32:0/3553134176 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f538800fca0 con 0x7f53981042c0 2024-04-30T08:34:22.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.142+0000 7f539d800700 1 -- 172.21.15.32:0/3553134176 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5398076f80 con 0x7f53981042c0 2024-04-30T08:34:22.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.143+0000 7f539d800700 1 -- 172.21.15.32:0/3553134176 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5398051210 con 0x7f53981042c0 2024-04-30T08:34:22.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.145+0000 7f538f400700 1 -- 172.21.15.32:0/3553134176 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f5388029070 con 0x7f53981042c0 2024-04-30T08:34:22.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.147+0000 7f538f400700 1 --2- 172.21.15.32:0/3553134176 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f537c0afc00 0x7f537c0b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:22.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.147+0000 7f538f400700 1 -- 172.21.15.32:0/3553134176 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f53880df9d0 con 0x7f53981042c0 2024-04-30T08:34:22.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.147+0000 7f5396000700 1 --2- 172.21.15.32:0/3553134176 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f537c0afc00 0x7f537c0b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:22.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.148+0000 7f5396000700 1 --2- 172.21.15.32:0/3553134176 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f537c0afc00 0x7f537c0b2110 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f539819b980 tx=0x7f53840073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:22.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.151+0000 7f538f400700 1 -- 172.21.15.32:0/3553134176 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5388013070 con 0x7f53981042c0 2024-04-30T08:34:22.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.409+0000 7f539d800700 1 -- 172.21.15.32:0/3553134176 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f5398063b50 con 0x7f537c0afc00 2024-04-30T08:34:22.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.411+0000 7f538f400700 1 -- 172.21.15.32:0/3553134176 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f5398063b50 con 0x7f537c0afc00 2024-04-30T08:34:22.413 INFO:teuthology.orchestra.run.smithi032.stdout:Backend: cephadm 2024-04-30T08:34:22.413 INFO:teuthology.orchestra.run.smithi032.stdout:Available: Yes 2024-04-30T08:34:22.413 INFO:teuthology.orchestra.run.smithi032.stdout:Paused: No 2024-04-30T08:34:22.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.416+0000 7f539d800700 1 -- 172.21.15.32:0/3553134176 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f537c0afc00 msgr2=0x7f537c0b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:22.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.416+0000 7f539d800700 1 --2- 172.21.15.32:0/3553134176 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f537c0afc00 0x7f537c0b2110 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f539819b980 tx=0x7f53840073d0 comp rx=0 tx=0).stop 2024-04-30T08:34:22.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.416+0000 7f539d800700 1 -- 172.21.15.32:0/3553134176 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53981042c0 msgr2=0x7f539819a8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:22.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.416+0000 7f539d800700 1 --2- 172.21.15.32:0/3553134176 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53981042c0 0x7f539819a8d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5388000c00 tx=0x7f53880057c0 comp rx=0 tx=0).stop 2024-04-30T08:34:22.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.416+0000 7f539d800700 1 -- 172.21.15.32:0/3553134176 shutdown_connections 2024-04-30T08:34:22.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.416+0000 7f539d800700 1 --2- 172.21.15.32:0/3553134176 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f537c0afc00 0x7f537c0b2110 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:22.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.416+0000 7f539d800700 1 --2- 172.21.15.32:0/3553134176 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5398105d90 0x7f53981a1db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:22.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.416+0000 7f539d800700 1 --2- 172.21.15.32:0/3553134176 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53981042c0 0x7f539819a8d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:22.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.416+0000 7f539d800700 1 --2- 172.21.15.32:0/3553134176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53981053f0 0x7f539819ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:22.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.416+0000 7f539d800700 1 -- 172.21.15.32:0/3553134176 >> 172.21.15.32:0/3553134176 conn(0x7f53980ff920 msgr2=0x7f5398108620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:22.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.416+0000 7f539d800700 1 -- 172.21.15.32:0/3553134176 shutdown_connections 2024-04-30T08:34:22.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:22.416+0000 7f539d800700 1 -- 172.21.15.32:0/3553134176 wait complete. 2024-04-30T08:34:23.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:22 smithi032 ceph-mon[28331]: pgmap v401: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-30T08:34:23.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:22 smithi071 ceph-mon[33385]: pgmap v401: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-30T08:34:23.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:22 smithi189 ceph-mon[33038]: pgmap v401: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-30T08:34:23.338 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-30T08:34:23.796 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:34:24.156 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:23 smithi032 ceph-mon[28331]: from='client.25265 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:34:24.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:23 smithi071 ceph-mon[33385]: from='client.25265 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:34:24.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:23 smithi189 ceph-mon[33038]: from='client.25265 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:34:25.166 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:24 smithi032 ceph-mon[28331]: pgmap v402: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:34:25.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:24 smithi071 ceph-mon[33385]: pgmap v402: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:34:25.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:24 smithi189 ceph-mon[33038]: pgmap v402: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:34:26.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.176+0000 7f4acb200700 1 -- 172.21.15.32:0/4039722974 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ac41053c0 msgr2=0x7f4ac41057d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:26.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.176+0000 7f4acb200700 1 --2- 172.21.15.32:0/4039722974 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ac41053c0 0x7f4ac41057d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f4ab00098f0 tx=0x7f4ab0009c00 comp rx=0 tx=0).stop 2024-04-30T08:34:26.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.176+0000 7f4acb200700 1 -- 172.21.15.32:0/4039722974 shutdown_connections 2024-04-30T08:34:26.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.176+0000 7f4acb200700 1 --2- 172.21.15.32:0/4039722974 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ac4105d60 0x7f4ac410c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:26.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.176+0000 7f4acb200700 1 --2- 172.21.15.32:0/4039722974 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ac4104290 0x7f4ac41046a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:26.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.176+0000 7f4acb200700 1 --2- 172.21.15.32:0/4039722974 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ac41053c0 0x7f4ac41057d0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:26.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.176+0000 7f4acb200700 1 -- 172.21.15.32:0/4039722974 >> 172.21.15.32:0/4039722974 conn(0x7f4ac40ff910 msgr2=0x7f4ac4101d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:26.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.176+0000 7f4acb200700 1 -- 172.21.15.32:0/4039722974 shutdown_connections 2024-04-30T08:34:26.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.176+0000 7f4acb200700 1 -- 172.21.15.32:0/4039722974 wait complete. 2024-04-30T08:34:26.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.178+0000 7f4acb200700 1 Processor -- start 2024-04-30T08:34:26.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.178+0000 7f4acb200700 1 -- start start 2024-04-30T08:34:26.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.178+0000 7f4acb200700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ac4104290 0x7f4ac419a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:26.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.178+0000 7f4acb200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ac41053c0 0x7f4ac419ade0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:26.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.178+0000 7f4acb200700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ac4105d60 0x7f4ac41a1d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:26.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.178+0000 7f4acb200700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac410fd00 con 0x7f4ac4105d60 2024-04-30T08:34:26.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.178+0000 7f4acb200700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac410fb80 con 0x7f4ac4104290 2024-04-30T08:34:26.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.178+0000 7f4acb200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac410fe80 con 0x7f4ac41053c0 2024-04-30T08:34:26.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.179+0000 7f4ac3e00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ac4104290 0x7f4ac419a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:26.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.179+0000 7f4ac3400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ac41053c0 0x7f4ac419ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:26.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.179+0000 7f4ac3e00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ac4104290 0x7f4ac419a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:55742/0 (socket says 172.21.15.32:55742) 2024-04-30T08:34:26.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.179+0000 7f4ac3e00700 1 -- 172.21.15.32:0/2750247893 learned_addr learned my addr 172.21.15.32:0/2750247893 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:26.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.179+0000 7f4ac9200700 1 --2- 172.21.15.32:0/2750247893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ac4105d60 0x7f4ac41a1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:26.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.179+0000 7f4ac3e00700 1 -- 172.21.15.32:0/2750247893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ac41053c0 msgr2=0x7f4ac419ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:26.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.179+0000 7f4ac3e00700 1 --2- 172.21.15.32:0/2750247893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ac41053c0 0x7f4ac419ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:26.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.179+0000 7f4ac3e00700 1 -- 172.21.15.32:0/2750247893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ac4105d60 msgr2=0x7f4ac41a1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:26.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.179+0000 7f4ac3e00700 1 --2- 172.21.15.32:0/2750247893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ac4105d60 0x7f4ac41a1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:26.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.179+0000 7f4ac3e00700 1 -- 172.21.15.32:0/2750247893 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ac41a2400 con 0x7f4ac4104290 2024-04-30T08:34:26.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.179+0000 7f4ac3e00700 1 --2- 172.21.15.32:0/2750247893 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ac4104290 0x7f4ac419a8a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4ab400ebc0 tx=0x7f4ab400eed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:26.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.180+0000 7f4ac0c00700 1 -- 172.21.15.32:0/2750247893 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab400cc90 con 0x7f4ac4104290 2024-04-30T08:34:26.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.180+0000 7f4ac0c00700 1 -- 172.21.15.32:0/2750247893 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4ab400ce30 con 0x7f4ac4104290 2024-04-30T08:34:26.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.180+0000 7f4acb200700 1 -- 172.21.15.32:0/2750247893 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ac41a26f0 con 0x7f4ac4104290 2024-04-30T08:34:26.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.180+0000 7f4ac0c00700 1 -- 172.21.15.32:0/2750247893 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab4018820 con 0x7f4ac4104290 2024-04-30T08:34:26.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.180+0000 7f4acb200700 1 -- 172.21.15.32:0/2750247893 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ac41a2c30 con 0x7f4ac4104290 2024-04-30T08:34:26.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.181+0000 7f4acb200700 1 -- 172.21.15.32:0/2750247893 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ac4051210 con 0x7f4ac4104290 2024-04-30T08:34:26.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.184+0000 7f4ac0c00700 1 -- 172.21.15.32:0/2750247893 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f4ab401f020 con 0x7f4ac4104290 2024-04-30T08:34:26.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.185+0000 7f4ac0c00700 1 --2- 172.21.15.32:0/2750247893 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4aa80afb80 0x7f4aa80b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:26.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.185+0000 7f4ac0c00700 1 -- 172.21.15.32:0/2750247893 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f4ab40df980 con 0x7f4ac4104290 2024-04-30T08:34:26.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.186+0000 7f4ac3400700 1 --2- 172.21.15.32:0/2750247893 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4aa80afb80 0x7f4aa80b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:26.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.186+0000 7f4ac3400700 1 --2- 172.21.15.32:0/2750247893 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4aa80afb80 0x7f4aa80b2090 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4ab00098c0 tx=0x7f4ab0019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:26.188 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.187+0000 7f4ac0c00700 1 -- 172.21.15.32:0/2750247893 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4ab40a8150 con 0x7f4ac4104290 2024-04-30T08:34:26.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.430+0000 7f4acb200700 1 -- 172.21.15.32:0/2750247893 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f4ac4063b50 con 0x7f4aa80afb80 2024-04-30T08:34:26.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.446+0000 7f4ac0c00700 1 -- 172.21.15.32:0/2750247893 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4123 (secure 0 0 0) 0x7f4ac4063b50 con 0x7f4aa80afb80 2024-04-30T08:34:26.448 INFO:teuthology.orchestra.run.smithi032.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-30T08:34:26.449 INFO:teuthology.orchestra.run.smithi032.stdout:mds.a smithi032 running (2m) 21s ago 2m 80.4M - 19.0.0-3348-g8feaf7d9 16056ace0edc 5234a090b9bb 2024-04-30T08:34:26.449 INFO:teuthology.orchestra.run.smithi032.stdout:mds.b smithi071 running (2m) 26s ago 2m 94.1M - 19.0.0-3348-g8feaf7d9 16056ace0edc 562a91b76601 2024-04-30T08:34:26.449 INFO:teuthology.orchestra.run.smithi032.stdout:mds.c smithi189 running (2m) 26s ago 2m 88.9M - 19.0.0-3348-g8feaf7d9 16056ace0edc 2d6e1472fdad 2024-04-30T08:34:26.449 INFO:teuthology.orchestra.run.smithi032.stdout:mds.d smithi032 running (3m) 21s ago 3m 85.1M - 19.0.0-3348-g8feaf7d9 16056ace0edc c92cfdfac4b5 2024-04-30T08:34:26.449 INFO:teuthology.orchestra.run.smithi032.stdout:mds.e smithi071 running (3m) 26s ago 3m 87.7M - 19.0.0-3348-g8feaf7d9 16056ace0edc fe42b55cc699 2024-04-30T08:34:26.449 INFO:teuthology.orchestra.run.smithi032.stdout:mds.f smithi189 running (3m) 26s ago 3m 88.9M - 19.0.0-3348-g8feaf7d9 16056ace0edc c9190cc367e4 2024-04-30T08:34:26.449 INFO:teuthology.orchestra.run.smithi032.stdout:mds.g smithi032 running (2m) 21s ago 2m 86.1M - 19.0.0-3348-g8feaf7d9 16056ace0edc 42dc38ed484f 2024-04-30T08:34:26.449 INFO:teuthology.orchestra.run.smithi032.stdout:mds.h smithi071 running (2m) 26s ago 2m 87.8M - 19.0.0-3348-g8feaf7d9 16056ace0edc 478762dd0b2a 2024-04-30T08:34:26.449 INFO:teuthology.orchestra.run.smithi032.stdout:mds.i smithi189 running (2m) 26s ago 2m 88.8M - 19.0.0-3348-g8feaf7d9 16056ace0edc 7800c09b8935 2024-04-30T08:34:26.449 INFO:teuthology.orchestra.run.smithi032.stdout:mds.j smithi032 running (3m) 21s ago 3m 80.5M - 19.0.0-3348-g8feaf7d9 16056ace0edc e7e3c5f5b723 2024-04-30T08:34:26.449 INFO:teuthology.orchestra.run.smithi032.stdout:mds.k smithi071 running (3m) 26s ago 3m 88.5M - 19.0.0-3348-g8feaf7d9 16056ace0edc 188e029109ea 2024-04-30T08:34:26.449 INFO:teuthology.orchestra.run.smithi032.stdout:mds.l smithi189 running (3m) 26s ago 3m 86.5M - 19.0.0-3348-g8feaf7d9 16056ace0edc 40bea0c1edd3 2024-04-30T08:34:26.449 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.x smithi032 *:9283,8765 running (11m) 21s ago 11m 586M - 19.0.0-3348-g8feaf7d9 16056ace0edc 55eff9d8db7c 2024-04-30T08:34:26.450 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.y smithi071 *:8443,8765 running (9m) 26s ago 9m 524M - 19.0.0-3348-g8feaf7d9 16056ace0edc b69f6e6706d9 2024-04-30T08:34:26.450 INFO:teuthology.orchestra.run.smithi032.stdout:mgr.z smithi189 *:8443,8765 running (9m) 26s ago 9m 522M - 19.0.0-3348-g8feaf7d9 16056ace0edc b90455e4a748 2024-04-30T08:34:26.450 INFO:teuthology.orchestra.run.smithi032.stdout:mon.a smithi032 running (12m) 21s ago 12m 120M 2048M 19.0.0-3348-g8feaf7d9 16056ace0edc 9a3de84cb1e9 2024-04-30T08:34:26.450 INFO:teuthology.orchestra.run.smithi032.stdout:mon.b smithi071 running (9m) 26s ago 9m 112M 2048M 19.0.0-3348-g8feaf7d9 16056ace0edc 7615c446928d 2024-04-30T08:34:26.450 INFO:teuthology.orchestra.run.smithi032.stdout:mon.c smithi189 running (10m) 26s ago 9m 110M 2048M 19.0.0-3348-g8feaf7d9 16056ace0edc bf88bed5fe55 2024-04-30T08:34:26.450 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 smithi032 running (9m) 21s ago 9m 125M 4273M 19.0.0-3348-g8feaf7d9 16056ace0edc db5407ef6167 2024-04-30T08:34:26.450 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 smithi071 running (8m) 26s ago 8m 126M 4272M 19.0.0-3348-g8feaf7d9 16056ace0edc ddbbffda8919 2024-04-30T08:34:26.450 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 smithi189 running (8m) 26s ago 8m 124M 4272M 19.0.0-3348-g8feaf7d9 16056ace0edc ba827a097097 2024-04-30T08:34:26.450 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 smithi032 running (7m) 21s ago 7m 112M 4273M 19.0.0-3348-g8feaf7d9 16056ace0edc dd27e0c3eec1 2024-04-30T08:34:26.450 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 smithi071 running (7m) 26s ago 7m 110M 4272M 19.0.0-3348-g8feaf7d9 16056ace0edc 5624c66ec459 2024-04-30T08:34:26.450 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 smithi189 running (6m) 26s ago 6m 114M 4272M 19.0.0-3348-g8feaf7d9 16056ace0edc c4e2487a1068 2024-04-30T08:34:26.450 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 smithi032 running (6m) 21s ago 6m 123M 4273M 19.0.0-3348-g8feaf7d9 16056ace0edc 9dfc7b047d10 2024-04-30T08:34:26.450 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 smithi071 running (5m) 26s ago 5m 122M 4272M 19.0.0-3348-g8feaf7d9 16056ace0edc 330b77d31c1e 2024-04-30T08:34:26.451 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8 smithi189 running (5m) 26s ago 5m 121M 4272M 19.0.0-3348-g8feaf7d9 16056ace0edc 034b9a88b593 2024-04-30T08:34:26.451 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9 smithi032 running (4m) 21s ago 4m 110M 4273M 19.0.0-3348-g8feaf7d9 16056ace0edc 7a2dccf97f0e 2024-04-30T08:34:26.451 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10 smithi071 running (4m) 26s ago 4m 110M 4272M 19.0.0-3348-g8feaf7d9 16056ace0edc 9779d2ada491 2024-04-30T08:34:26.451 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11 smithi189 running (3m) 26s ago 3m 121M 4272M 19.0.0-3348-g8feaf7d9 16056ace0edc b149d40573b8 2024-04-30T08:34:26.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.449+0000 7f4acb200700 1 -- 172.21.15.32:0/2750247893 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4aa80afb80 msgr2=0x7f4aa80b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:26.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.449+0000 7f4acb200700 1 --2- 172.21.15.32:0/2750247893 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4aa80afb80 0x7f4aa80b2090 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f4ab00098c0 tx=0x7f4ab0019040 comp rx=0 tx=0).stop 2024-04-30T08:34:26.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.449+0000 7f4acb200700 1 -- 172.21.15.32:0/2750247893 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ac4104290 msgr2=0x7f4ac419a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:26.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.449+0000 7f4acb200700 1 --2- 172.21.15.32:0/2750247893 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ac4104290 0x7f4ac419a8a0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f4ab400ebc0 tx=0x7f4ab400eed0 comp rx=0 tx=0).stop 2024-04-30T08:34:26.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.449+0000 7f4acb200700 1 -- 172.21.15.32:0/2750247893 shutdown_connections 2024-04-30T08:34:26.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.449+0000 7f4acb200700 1 --2- 172.21.15.32:0/2750247893 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4aa80afb80 0x7f4aa80b2090 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:26.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.449+0000 7f4acb200700 1 --2- 172.21.15.32:0/2750247893 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ac4105d60 0x7f4ac41a1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:26.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.449+0000 7f4acb200700 1 --2- 172.21.15.32:0/2750247893 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ac4104290 0x7f4ac419a8a0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:26.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.449+0000 7f4acb200700 1 --2- 172.21.15.32:0/2750247893 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ac41053c0 0x7f4ac419ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:26.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.449+0000 7f4acb200700 1 -- 172.21.15.32:0/2750247893 >> 172.21.15.32:0/2750247893 conn(0x7f4ac40ff910 msgr2=0x7f4ac41085f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:26.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.449+0000 7f4acb200700 1 -- 172.21.15.32:0/2750247893 shutdown_connections 2024-04-30T08:34:26.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:26.449+0000 7f4acb200700 1 -- 172.21.15.32:0/2750247893 wait complete. 2024-04-30T08:34:27.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:26 smithi032 ceph-mon[28331]: pgmap v403: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-30T08:34:27.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:26 smithi071 ceph-mon[33385]: pgmap v403: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-30T08:34:27.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:26 smithi189 ceph-mon[33038]: pgmap v403: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-30T08:34:27.415 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-30T08:34:27.890 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:34:28.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:27 smithi032 ceph-mon[28331]: from='client.25271 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:34:28.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:27 smithi071 ceph-mon[33385]: from='client.25271 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:34:28.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:27 smithi189 ceph-mon[33038]: from='client.25271 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:34:29.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:28 smithi032 ceph-mon[28331]: pgmap v404: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-30T08:34:29.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:28 smithi071 ceph-mon[33385]: pgmap v404: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-30T08:34:29.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:28 smithi189 ceph-mon[33038]: pgmap v404: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-30T08:34:30.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.304+0000 7f2fe5a00700 1 -- 172.21.15.32:0/383498764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fe00786c0 msgr2=0x7f2fe0076b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:30.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.304+0000 7f2fe5a00700 1 --2- 172.21.15.32:0/383498764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fe00786c0 0x7f2fe0076b90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0009990 tx=0x7f2fd0009ca0 comp rx=0 tx=0).stop 2024-04-30T08:34:30.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.304+0000 7f2fe5a00700 1 -- 172.21.15.32:0/383498764 shutdown_connections 2024-04-30T08:34:30.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.304+0000 7f2fe5a00700 1 --2- 172.21.15.32:0/383498764 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fe010c1a0 0x7f2fe010e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:30.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.304+0000 7f2fe5a00700 1 --2- 172.21.15.32:0/383498764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fe00786c0 0x7f2fe0076b90 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:30.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.304+0000 7f2fe5a00700 1 --2- 172.21.15.32:0/383498764 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2fe00770d0 0x7f2fe00774e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:30.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.304+0000 7f2fe5a00700 1 -- 172.21.15.32:0/383498764 >> 172.21.15.32:0/383498764 conn(0x7f2fe00ff920 msgr2=0x7f2fe0101db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:30.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.304+0000 7f2fe5a00700 1 -- 172.21.15.32:0/383498764 shutdown_connections 2024-04-30T08:34:30.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.304+0000 7f2fe5a00700 1 -- 172.21.15.32:0/383498764 wait complete. 2024-04-30T08:34:30.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.306+0000 7f2fe5a00700 1 Processor -- start 2024-04-30T08:34:30.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.306+0000 7f2fe5a00700 1 -- start start 2024-04-30T08:34:30.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.307+0000 7f2fe5a00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fe00770d0 0x7f2fe019ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:30.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.307+0000 7f2fe5a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fe00786c0 0x7f2fe019cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:30.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.307+0000 7f2fe5a00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2fe010c1a0 0x7f2fe01a3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:30.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.307+0000 7f2fe5a00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe0111f10 con 0x7f2fe00770d0 2024-04-30T08:34:30.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.307+0000 7f2fe5a00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe0111d90 con 0x7f2fe010c1a0 2024-04-30T08:34:30.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.307+0000 7f2fe5a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f2fe0112090 con 0x7f2fe00786c0 2024-04-30T08:34:30.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.307+0000 7f2fdea00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fe00770d0 0x7f2fe019ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:30.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.307+0000 7f2fd7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fe00786c0 0x7f2fe019cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:30.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.307+0000 7f2fdf400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2fe010c1a0 0x7f2fe01a3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:30.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.307+0000 7f2fd7e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fe00786c0 0x7f2fe019cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:38036/0 (socket says 172.21.15.32:38036) 2024-04-30T08:34:30.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.307+0000 7f2fdf400700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2fe010c1a0 0x7f2fe01a3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:46288/0 (socket says 172.21.15.32:46288) 2024-04-30T08:34:30.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.307+0000 7f2fd7e00700 1 -- 172.21.15.32:0/3387792438 learned_addr learned my addr 172.21.15.32:0/3387792438 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:30.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.308+0000 7f2fd7e00700 1 -- 172.21.15.32:0/3387792438 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2fe010c1a0 msgr2=0x7f2fe01a3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:30.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.308+0000 7f2fd7e00700 1 --2- 172.21.15.32:0/3387792438 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2fe010c1a0 0x7f2fe01a3ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:30.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.308+0000 7f2fd7e00700 1 -- 172.21.15.32:0/3387792438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fe00770d0 msgr2=0x7f2fe019ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:30.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.308+0000 7f2fd7e00700 1 --2- 172.21.15.32:0/3387792438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fe00770d0 0x7f2fe019ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:30.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.308+0000 7f2fd7e00700 1 -- 172.21.15.32:0/3387792438 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fe01a4560 con 0x7f2fe00786c0 2024-04-30T08:34:30.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.308+0000 7f2fd7e00700 1 --2- 172.21.15.32:0/3387792438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fe00786c0 0x7f2fe019cf90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2fc800d950 tx=0x7f2fc800dc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:30.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.308+0000 7f2fd7400700 1 -- 172.21.15.32:0/3387792438 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc8016070 con 0x7f2fe00786c0 2024-04-30T08:34:30.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.309+0000 7f2fd7400700 1 -- 172.21.15.32:0/3387792438 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f2fc800fd40 con 0x7f2fe00786c0 2024-04-30T08:34:30.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.309+0000 7f2fe5a00700 1 -- 172.21.15.32:0/3387792438 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fe01a4850 con 0x7f2fe00786c0 2024-04-30T08:34:30.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.309+0000 7f2fd7400700 1 -- 172.21.15.32:0/3387792438 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc800e3b0 con 0x7f2fe00786c0 2024-04-30T08:34:30.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.309+0000 7f2fe5a00700 1 -- 172.21.15.32:0/3387792438 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fe01a4de0 con 0x7f2fe00786c0 2024-04-30T08:34:30.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.310+0000 7f2fe5a00700 1 -- 172.21.15.32:0/3387792438 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fe0051210 con 0x7f2fe00786c0 2024-04-30T08:34:30.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.312+0000 7f2fd7400700 1 -- 172.21.15.32:0/3387792438 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f2fc8019070 con 0x7f2fe00786c0 2024-04-30T08:34:30.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.314+0000 7f2fd7400700 1 --2- 172.21.15.32:0/3387792438 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f2fc40afc00 0x7f2fc40b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:30.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.315+0000 7f2fd7400700 1 -- 172.21.15.32:0/3387792438 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f2fc80dea10 con 0x7f2fe00786c0 2024-04-30T08:34:30.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.315+0000 7f2fdea00700 1 --2- 172.21.15.32:0/3387792438 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f2fc40afc00 0x7f2fc40b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:30.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.315+0000 7f2fdea00700 1 --2- 172.21.15.32:0/3387792438 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f2fc40afc00 0x7f2fc40b2110 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0006010 tx=0x7f2fd000b720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:30.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.316+0000 7f2fd7400700 1 -- 172.21.15.32:0/3387792438 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2fc80a71e0 con 0x7f2fe00786c0 2024-04-30T08:34:30.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.556+0000 7f2fe5a00700 1 -- 172.21.15.32:0/3387792438 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f2fe0063b50 con 0x7f2fc40afc00 2024-04-30T08:34:30.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.573+0000 7f2fd7400700 1 -- 172.21.15.32:0/3387792438 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f2fe0063b50 con 0x7f2fc40afc00 2024-04-30T08:34:30.574 INFO:teuthology.orchestra.run.smithi032.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-30T08:34:30.574 INFO:teuthology.orchestra.run.smithi032.stdout:mds.all 12/12 30s ago 3m smithi032=a;smithi032=d;smithi032=g;smithi032=j;smithi071=b;smithi071=e;smithi071=h;smithi071=k;smithi189=c;smithi189=f;smithi189=i;smithi189=l;count:12 2024-04-30T08:34:30.574 INFO:teuthology.orchestra.run.smithi032.stdout:mgr 3/3 30s ago 9m smithi032=x;smithi071=y;smithi189=z;count:3 2024-04-30T08:34:30.574 INFO:teuthology.orchestra.run.smithi032.stdout:mon 3/3 30s ago 10m smithi032:172.21.15.32=a;smithi071:172.21.15.71=b;smithi189:172.21.15.189=c;count:3 2024-04-30T08:34:30.574 INFO:teuthology.orchestra.run.smithi032.stdout:osd 12 30s ago - 2024-04-30T08:34:30.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.576+0000 7f2fe5a00700 1 -- 172.21.15.32:0/3387792438 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f2fc40afc00 msgr2=0x7f2fc40b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:30.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.576+0000 7f2fe5a00700 1 --2- 172.21.15.32:0/3387792438 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f2fc40afc00 0x7f2fc40b2110 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f2fd0006010 tx=0x7f2fd000b720 comp rx=0 tx=0).stop 2024-04-30T08:34:30.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.576+0000 7f2fe5a00700 1 -- 172.21.15.32:0/3387792438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fe00786c0 msgr2=0x7f2fe019cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:30.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.576+0000 7f2fe5a00700 1 --2- 172.21.15.32:0/3387792438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fe00786c0 0x7f2fe019cf90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f2fc800d950 tx=0x7f2fc800dc60 comp rx=0 tx=0).stop 2024-04-30T08:34:30.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.576+0000 7f2fe5a00700 1 -- 172.21.15.32:0/3387792438 shutdown_connections 2024-04-30T08:34:30.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.576+0000 7f2fe5a00700 1 --2- 172.21.15.32:0/3387792438 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f2fc40afc00 0x7f2fc40b2110 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:30.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.576+0000 7f2fe5a00700 1 --2- 172.21.15.32:0/3387792438 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2fe010c1a0 0x7f2fe01a3ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:30.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.577+0000 7f2fe5a00700 1 --2- 172.21.15.32:0/3387792438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2fe00770d0 0x7f2fe019ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:30.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.577+0000 7f2fe5a00700 1 --2- 172.21.15.32:0/3387792438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2fe00786c0 0x7f2fe019cf90 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:30.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.577+0000 7f2fe5a00700 1 -- 172.21.15.32:0/3387792438 >> 172.21.15.32:0/3387792438 conn(0x7f2fe00ff920 msgr2=0x7f2fe0108180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:30.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.577+0000 7f2fe5a00700 1 -- 172.21.15.32:0/3387792438 shutdown_connections 2024-04-30T08:34:30.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:30.577+0000 7f2fe5a00700 1 -- 172.21.15.32:0/3387792438 wait complete. 2024-04-30T08:34:31.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:30 smithi071 ceph-mon[33385]: pgmap v405: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-30T08:34:31.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:30 smithi189 ceph-mon[33038]: pgmap v405: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-30T08:34:31.379 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:30 smithi032 ceph-mon[28331]: pgmap v405: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-30T08:34:31.459 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-30T08:34:31.914 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:34:32.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:31 smithi071 ceph-mon[33385]: from='client.25276 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:34:32.278 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:31 smithi032 ceph-mon[28331]: from='client.25276 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:34:32.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:31 smithi189 ceph-mon[33038]: from='client.25276 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:34:33.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:32 smithi032 ceph-mon[28331]: pgmap v406: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-30T08:34:33.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:32 smithi071 ceph-mon[33385]: pgmap v406: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-30T08:34:33.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:32 smithi189 ceph-mon[33038]: pgmap v406: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-30T08:34:34.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.298+0000 7f0ad3c00700 1 -- 172.21.15.32:0/2261698905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0acc105d90 msgr2=0x7f0acc10c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:34.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.298+0000 7f0ad3c00700 1 --2- 172.21.15.32:0/2261698905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0acc105d90 0x7f0acc10c640 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f0ac400b0a0 tx=0x7f0ac400b3b0 comp rx=0 tx=0).stop 2024-04-30T08:34:34.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.298+0000 7f0ad3c00700 1 -- 172.21.15.32:0/2261698905 shutdown_connections 2024-04-30T08:34:34.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.298+0000 7f0ad3c00700 1 --2- 172.21.15.32:0/2261698905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0acc105d90 0x7f0acc10c640 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:34.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.298+0000 7f0ad3c00700 1 --2- 172.21.15.32:0/2261698905 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0acc1042c0 0x7f0acc1046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:34.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.298+0000 7f0ad3c00700 1 --2- 172.21.15.32:0/2261698905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0acc1053f0 0x7f0acc105800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:34.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.298+0000 7f0ad3c00700 1 -- 172.21.15.32:0/2261698905 >> 172.21.15.32:0/2261698905 conn(0x7f0acc0ff920 msgr2=0x7f0acc101db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:34.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.298+0000 7f0ad3c00700 1 -- 172.21.15.32:0/2261698905 shutdown_connections 2024-04-30T08:34:34.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.298+0000 7f0ad3c00700 1 -- 172.21.15.32:0/2261698905 wait complete. 2024-04-30T08:34:34.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.299+0000 7f0ad3c00700 1 Processor -- start 2024-04-30T08:34:34.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.299+0000 7f0ad3c00700 1 -- start start 2024-04-30T08:34:34.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.299+0000 7f0ad3c00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0acc1042c0 0x7f0acc19a7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:34.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.299+0000 7f0ad3c00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0acc1053f0 0x7f0acc19ad00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:34.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.299+0000 7f0ad3c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0acc105d90 0x7f0acc1a1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:34.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.299+0000 7f0ad3c00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0acc10fd30 con 0x7f0acc1053f0 2024-04-30T08:34:34.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.299+0000 7f0ad3c00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f0acc10fbb0 con 0x7f0acc1042c0 2024-04-30T08:34:34.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.299+0000 7f0ad3c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0acc10feb0 con 0x7f0acc105d90 2024-04-30T08:34:34.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.300+0000 7f0acbe00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0acc1053f0 0x7f0acc19ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:34.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.300+0000 7f0acbe00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0acc1053f0 0x7f0acc19ad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:35442/0 (socket says 172.21.15.32:35442) 2024-04-30T08:34:34.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.300+0000 7f0acbe00700 1 -- 172.21.15.32:0/316336771 learned_addr learned my addr 172.21.15.32:0/316336771 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:34.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.300+0000 7f0ad1c00700 1 --2- 172.21.15.32:0/316336771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0acc105d90 0x7f0acc1a1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:34.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.300+0000 7f0ad1200700 1 --2- 172.21.15.32:0/316336771 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0acc1042c0 0x7f0acc19a7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:34.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.300+0000 7f0acbe00700 1 -- 172.21.15.32:0/316336771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0acc105d90 msgr2=0x7f0acc1a1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:34.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.300+0000 7f0acbe00700 1 --2- 172.21.15.32:0/316336771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0acc105d90 0x7f0acc1a1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:34.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.300+0000 7f0acbe00700 1 -- 172.21.15.32:0/316336771 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0acc1042c0 msgr2=0x7f0acc19a7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:34.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.300+0000 7f0acbe00700 1 --2- 172.21.15.32:0/316336771 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0acc1042c0 0x7f0acc19a7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:34.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.300+0000 7f0acbe00700 1 -- 172.21.15.32:0/316336771 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0acc1a2320 con 0x7f0acc1053f0 2024-04-30T08:34:34.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.300+0000 7f0ad1c00700 1 --2- 172.21.15.32:0/316336771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0acc105d90 0x7f0acc1a1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:34:34.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.300+0000 7f0ad1200700 1 --2- 172.21.15.32:0/316336771 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0acc1042c0 0x7f0acc19a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:34:34.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.301+0000 7f0acbe00700 1 --2- 172.21.15.32:0/316336771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0acc1053f0 0x7f0acc19ad00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0ab800e9a0 tx=0x7f0ab800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:34.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.301+0000 7f0ac9600700 1 -- 172.21.15.32:0/316336771 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab800ca10 con 0x7f0acc1053f0 2024-04-30T08:34:34.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.301+0000 7f0ac9600700 1 -- 172.21.15.32:0/316336771 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0ab800cbb0 con 0x7f0acc1053f0 2024-04-30T08:34:34.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.301+0000 7f0ac9600700 1 -- 172.21.15.32:0/316336771 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ab8018680 con 0x7f0acc1053f0 2024-04-30T08:34:34.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.301+0000 7f0ad3c00700 1 -- 172.21.15.32:0/316336771 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0acc1a2610 con 0x7f0acc1053f0 2024-04-30T08:34:34.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.301+0000 7f0ad3c00700 1 -- 172.21.15.32:0/316336771 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0acc1a2ba0 con 0x7f0acc1053f0 2024-04-30T08:34:34.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.303+0000 7f0ad3c00700 1 -- 172.21.15.32:0/316336771 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0acc051210 con 0x7f0acc1053f0 2024-04-30T08:34:34.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.305+0000 7f0ac9600700 1 -- 172.21.15.32:0/316336771 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f0ab8010430 con 0x7f0acc1053f0 2024-04-30T08:34:34.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.307+0000 7f0ac9600700 1 --2- 172.21.15.32:0/316336771 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f0ab40afc00 0x7f0ab40b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:34.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.307+0000 7f0ac9600700 1 -- 172.21.15.32:0/316336771 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f0ab80df450 con 0x7f0acc1053f0 2024-04-30T08:34:34.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.307+0000 7f0ad1200700 1 --2- 172.21.15.32:0/316336771 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f0ab40afc00 0x7f0ab40b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:34.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.308+0000 7f0ad1200700 1 --2- 172.21.15.32:0/316336771 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f0ab40afc00 0x7f0ab40b2110 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f0acc1052f0 tx=0x7f0ac0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:34.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.310+0000 7f0ac9600700 1 -- 172.21.15.32:0/316336771 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0ab80a7c20 con 0x7f0acc1053f0 2024-04-30T08:34:34.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.551+0000 7f0ad3c00700 1 -- 172.21.15.32:0/316336771 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f0acc063b50 con 0x7f0ab40afc00 2024-04-30T08:34:34.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.555+0000 7f0ac9600700 1 -- 172.21.15.32:0/316336771 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f0acc063b50 con 0x7f0ab40afc00 2024-04-30T08:34:34.558 INFO:teuthology.orchestra.run.smithi032.stdout:HOST ADDR LABELS STATUS 2024-04-30T08:34:34.558 INFO:teuthology.orchestra.run.smithi032.stdout:smithi032 172.21.15.32 2024-04-30T08:34:34.559 INFO:teuthology.orchestra.run.smithi032.stdout:smithi071 172.21.15.71 2024-04-30T08:34:34.559 INFO:teuthology.orchestra.run.smithi032.stdout:smithi189 172.21.15.189 2024-04-30T08:34:34.559 INFO:teuthology.orchestra.run.smithi032.stdout:3 hosts in cluster 2024-04-30T08:34:34.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.558+0000 7f0ad3c00700 1 -- 172.21.15.32:0/316336771 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f0ab40afc00 msgr2=0x7f0ab40b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:34.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.558+0000 7f0ad3c00700 1 --2- 172.21.15.32:0/316336771 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f0ab40afc00 0x7f0ab40b2110 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f0acc1052f0 tx=0x7f0ac0009260 comp rx=0 tx=0).stop 2024-04-30T08:34:34.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.558+0000 7f0ad3c00700 1 -- 172.21.15.32:0/316336771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0acc1053f0 msgr2=0x7f0acc19ad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:34.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.558+0000 7f0ad3c00700 1 --2- 172.21.15.32:0/316336771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0acc1053f0 0x7f0acc19ad00 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f0ab800e9a0 tx=0x7f0ab800ecb0 comp rx=0 tx=0).stop 2024-04-30T08:34:34.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.558+0000 7f0ad3c00700 1 -- 172.21.15.32:0/316336771 shutdown_connections 2024-04-30T08:34:34.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.558+0000 7f0ad3c00700 1 --2- 172.21.15.32:0/316336771 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f0ab40afc00 0x7f0ab40b2110 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:34.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.558+0000 7f0ad3c00700 1 --2- 172.21.15.32:0/316336771 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0acc105d90 0x7f0acc1a1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:34.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.558+0000 7f0ad3c00700 1 --2- 172.21.15.32:0/316336771 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0acc1042c0 0x7f0acc19a7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:34.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.558+0000 7f0ad3c00700 1 --2- 172.21.15.32:0/316336771 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0acc1053f0 0x7f0acc19ad00 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:34.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.558+0000 7f0ad3c00700 1 -- 172.21.15.32:0/316336771 >> 172.21.15.32:0/316336771 conn(0x7f0acc0ff920 msgr2=0x7f0acc108620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:34.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.558+0000 7f0ad3c00700 1 -- 172.21.15.32:0/316336771 shutdown_connections 2024-04-30T08:34:34.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:34.558+0000 7f0ad3c00700 1 -- 172.21.15.32:0/316336771 wait complete. 2024-04-30T08:34:35.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:34 smithi071 ceph-mon[33385]: pgmap v407: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:34:35.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:34 smithi189 ceph-mon[33038]: pgmap v407: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:34:35.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:34 smithi032 ceph-mon[28331]: pgmap v407: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:34:35.541 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-30T08:34:36.011 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:34:36.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:35 smithi071 ceph-mon[33385]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:34:36.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:35 smithi189 ceph-mon[33038]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:34:36.379 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:35 smithi032 ceph-mon[28331]: from='client.15159 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:34:37.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:36 smithi071 ceph-mon[33385]: pgmap v408: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-30T08:34:37.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:36 smithi189 ceph-mon[33038]: pgmap v408: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-30T08:34:37.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:36 smithi032 ceph-mon[28331]: pgmap v408: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-30T08:34:38.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.689+0000 7f613fc00700 1 -- 172.21.15.32:0/2738288665 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61381064c0 msgr2=0x7f61381068d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:38.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.689+0000 7f613fc00700 1 --2- 172.21.15.32:0/2738288665 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61381064c0 0x7f61381068d0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f61280099a0 tx=0x7f6128009cb0 comp rx=0 tx=0).stop 2024-04-30T08:34:38.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.689+0000 7f613fc00700 1 -- 172.21.15.32:0/2738288665 shutdown_connections 2024-04-30T08:34:38.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.689+0000 7f613fc00700 1 --2- 172.21.15.32:0/2738288665 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6138107f90 0x7f613810e840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:38.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.689+0000 7f613fc00700 1 --2- 172.21.15.32:0/2738288665 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61381064c0 0x7f61381068d0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:38.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.689+0000 7f613fc00700 1 --2- 172.21.15.32:0/2738288665 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f61381075f0 0x7f6138107a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:38.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.689+0000 7f613fc00700 1 -- 172.21.15.32:0/2738288665 >> 172.21.15.32:0/2738288665 conn(0x7f6138101b20 msgr2=0x7f6138103fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:38.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.689+0000 7f613fc00700 1 -- 172.21.15.32:0/2738288665 shutdown_connections 2024-04-30T08:34:38.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.689+0000 7f613fc00700 1 -- 172.21.15.32:0/2738288665 wait complete. 2024-04-30T08:34:38.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.691+0000 7f613fc00700 1 Processor -- start 2024-04-30T08:34:38.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.691+0000 7f613fc00700 1 -- start start 2024-04-30T08:34:38.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.691+0000 7f613fc00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f61381064c0 0x7f613819caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:38.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.691+0000 7f613fc00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61381075f0 0x7f613819cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:38.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.691+0000 7f613fc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6138107f90 0x7f61381a3f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:38.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.691+0000 7f613fc00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6138111f80 con 0x7f61381075f0 2024-04-30T08:34:38.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.691+0000 7f613fc00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6138111e00 con 0x7f61381064c0 2024-04-30T08:34:38.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.691+0000 7f613fc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6138112100 con 0x7f6138107f90 2024-04-30T08:34:38.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.692+0000 7f612fe00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61381075f0 0x7f613819cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:38.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.692+0000 7f613d200700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f61381064c0 0x7f613819caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:38.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.692+0000 7f612fe00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61381075f0 0x7f613819cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:50780/0 (socket says 172.21.15.32:50780) 2024-04-30T08:34:38.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.692+0000 7f612fe00700 1 -- 172.21.15.32:0/2008949109 learned_addr learned my addr 172.21.15.32:0/2008949109 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:38.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.692+0000 7f613dc00700 1 --2- 172.21.15.32:0/2008949109 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6138107f90 0x7f61381a3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:38.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.692+0000 7f613d200700 1 -- 172.21.15.32:0/2008949109 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6138107f90 msgr2=0x7f61381a3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:38.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.692+0000 7f613d200700 1 --2- 172.21.15.32:0/2008949109 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6138107f90 0x7f61381a3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:38.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.692+0000 7f613d200700 1 -- 172.21.15.32:0/2008949109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61381075f0 msgr2=0x7f613819cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:38.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.692+0000 7f613d200700 1 --2- 172.21.15.32:0/2008949109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61381075f0 0x7f613819cfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:38.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.692+0000 7f613d200700 1 -- 172.21.15.32:0/2008949109 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61381a4600 con 0x7f61381064c0 2024-04-30T08:34:38.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.693+0000 7f613d200700 1 --2- 172.21.15.32:0/2008949109 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f61381064c0 0x7f613819caa0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6128000c00 tx=0x7f6128004c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:38.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.693+0000 7f6136000700 1 -- 172.21.15.32:0/2008949109 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6128026070 con 0x7f61381064c0 2024-04-30T08:34:38.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.693+0000 7f6136000700 1 -- 172.21.15.32:0/2008949109 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f612800bdd0 con 0x7f61381064c0 2024-04-30T08:34:38.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.693+0000 7f6136000700 1 -- 172.21.15.32:0/2008949109 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6128013070 con 0x7f61381064c0 2024-04-30T08:34:38.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.693+0000 7f613fc00700 1 -- 172.21.15.32:0/2008949109 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61381a4890 con 0x7f61381064c0 2024-04-30T08:34:38.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.693+0000 7f613fc00700 1 -- 172.21.15.32:0/2008949109 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61381a4d40 con 0x7f61381064c0 2024-04-30T08:34:38.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.694+0000 7f613fc00700 1 -- 172.21.15.32:0/2008949109 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6138051210 con 0x7f61381064c0 2024-04-30T08:34:38.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.697+0000 7f6136000700 1 -- 172.21.15.32:0/2008949109 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f6128029070 con 0x7f61381064c0 2024-04-30T08:34:38.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.698+0000 7f6136000700 1 --2- 172.21.15.32:0/2008949109 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f61180afc00 0x7f61180b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:38.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.698+0000 7f6136000700 1 -- 172.21.15.32:0/2008949109 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f61280dfca0 con 0x7f61381064c0 2024-04-30T08:34:38.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.699+0000 7f612fe00700 1 --2- 172.21.15.32:0/2008949109 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f61180afc00 0x7f61180b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:38.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.699+0000 7f612fe00700 1 --2- 172.21.15.32:0/2008949109 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f61180afc00 0x7f61180b2110 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f613819db50 tx=0x7f6120009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:38.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.701+0000 7f6136000700 1 -- 172.21.15.32:0/2008949109 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f61280a8500 con 0x7f61381064c0 2024-04-30T08:34:38.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.943+0000 7f613fc00700 1 -- 172.21.15.32:0/2008949109 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f6138063b50 con 0x7f61180afc00 2024-04-30T08:34:38.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.947+0000 7f6136000700 1 -- 172.21.15.32:0/2008949109 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f6138063b50 con 0x7f61180afc00 2024-04-30T08:34:38.948 INFO:teuthology.orchestra.run.smithi032.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-30T08:34:38.948 INFO:teuthology.orchestra.run.smithi032.stdout:smithi032 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB747402DN480DGN 447G No 4m ago Has a FileSystem, LVM detected 2024-04-30T08:34:38.948 INFO:teuthology.orchestra.run.smithi032.stdout:smithi071 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT62040172400BGN 372G No 4m ago Has a FileSystem, LVM detected 2024-04-30T08:34:38.948 INFO:teuthology.orchestra.run.smithi032.stdout:smithi189 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6232004L400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-30T08:34:38.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.950+0000 7f613fc00700 1 -- 172.21.15.32:0/2008949109 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f61180afc00 msgr2=0x7f61180b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:38.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.950+0000 7f613fc00700 1 --2- 172.21.15.32:0/2008949109 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f61180afc00 0x7f61180b2110 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f613819db50 tx=0x7f6120009260 comp rx=0 tx=0).stop 2024-04-30T08:34:38.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.950+0000 7f613fc00700 1 -- 172.21.15.32:0/2008949109 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f61381064c0 msgr2=0x7f613819caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:38.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.950+0000 7f613fc00700 1 --2- 172.21.15.32:0/2008949109 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f61381064c0 0x7f613819caa0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f6128000c00 tx=0x7f6128004c70 comp rx=0 tx=0).stop 2024-04-30T08:34:38.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.950+0000 7f613fc00700 1 -- 172.21.15.32:0/2008949109 shutdown_connections 2024-04-30T08:34:38.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.950+0000 7f613fc00700 1 --2- 172.21.15.32:0/2008949109 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f61180afc00 0x7f61180b2110 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:38.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.950+0000 7f613fc00700 1 --2- 172.21.15.32:0/2008949109 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6138107f90 0x7f61381a3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:38.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.950+0000 7f613fc00700 1 --2- 172.21.15.32:0/2008949109 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f61381064c0 0x7f613819caa0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:38.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.950+0000 7f613fc00700 1 --2- 172.21.15.32:0/2008949109 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61381075f0 0x7f613819cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:38.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.950+0000 7f613fc00700 1 -- 172.21.15.32:0/2008949109 >> 172.21.15.32:0/2008949109 conn(0x7f6138101b20 msgr2=0x7f613810a820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:38.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.951+0000 7f613fc00700 1 -- 172.21.15.32:0/2008949109 shutdown_connections 2024-04-30T08:34:38.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:38.951+0000 7f613fc00700 1 -- 172.21.15.32:0/2008949109 wait complete. 2024-04-30T08:34:39.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:38 smithi071 ceph-mon[33385]: pgmap v409: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-30T08:34:39.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:38 smithi189 ceph-mon[33038]: pgmap v409: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-30T08:34:39.349 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:38 smithi032 ceph-mon[28331]: pgmap v409: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-30T08:34:39.802 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-30T08:34:39.814 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi032.front.sepia.ceph.com 2024-04-30T08:34:39.814 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-30T08:34:40.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:39 smithi071 ceph-mon[33385]: from='client.25289 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:34:40.284 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:34:40.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:39 smithi189 ceph-mon[33038]: from='client.25289 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:34:40.430 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:39 smithi032 ceph-mon[28331]: from='client.25289 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:34:41.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:40 smithi071 ceph-mon[33385]: pgmap v410: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-30T08:34:41.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:40 smithi189 ceph-mon[33038]: pgmap v410: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-30T08:34:41.401 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:40 smithi032 ceph-mon[28331]: pgmap v410: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-30T08:34:42.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.749+0000 7f8251000700 1 -- 172.21.15.32:0/4103473232 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f824c10b9d0 msgr2=0x7f824c10bde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:42.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.749+0000 7f8251000700 1 --2- 172.21.15.32:0/4103473232 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f824c10b9d0 0x7f824c10bde0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f82380098f0 tx=0x7f8238009c00 comp rx=0 tx=0).stop 2024-04-30T08:34:42.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.749+0000 7f8251000700 1 -- 172.21.15.32:0/4103473232 shutdown_connections 2024-04-30T08:34:42.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.749+0000 7f8251000700 1 --2- 172.21.15.32:0/4103473232 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f824c10c370 0x7f824c112c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:42.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.749+0000 7f8251000700 1 --2- 172.21.15.32:0/4103473232 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f824c10aa60 0x7f824c10ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:42.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.749+0000 7f8251000700 1 --2- 172.21.15.32:0/4103473232 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f824c10b9d0 0x7f824c10bde0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:42.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.749+0000 7f8251000700 1 -- 172.21.15.32:0/4103473232 >> 172.21.15.32:0/4103473232 conn(0x7f824c079910 msgr2=0x7f824c07bdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:42.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.749+0000 7f8251000700 1 -- 172.21.15.32:0/4103473232 shutdown_connections 2024-04-30T08:34:42.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.749+0000 7f8251000700 1 -- 172.21.15.32:0/4103473232 wait complete. 2024-04-30T08:34:42.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.751+0000 7f8251000700 1 Processor -- start 2024-04-30T08:34:42.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.751+0000 7f8251000700 1 -- start start 2024-04-30T08:34:42.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.751+0000 7f8251000700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f824c10aa60 0x7f824c1a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:42.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.751+0000 7f8251000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f824c10b9d0 0x7f824c1a13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:42.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.751+0000 7f8251000700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f824c10c370 0x7f824c1a8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:42.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.751+0000 7f8251000700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f824c116310 con 0x7f824c10c370 2024-04-30T08:34:42.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.751+0000 7f8251000700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f824c116190 con 0x7f824c10aa60 2024-04-30T08:34:42.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.751+0000 7f8251000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f824c116490 con 0x7f824c10b9d0 2024-04-30T08:34:42.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.752+0000 7f8249e00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f824c10aa60 0x7f824c1a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:42.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.752+0000 7f8249e00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f824c10aa60 0x7f824c1a0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:33700/0 (socket says 172.21.15.32:33700) 2024-04-30T08:34:42.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.752+0000 7f8249e00700 1 -- 172.21.15.32:0/1157948426 learned_addr learned my addr 172.21.15.32:0/1157948426 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:42.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.752+0000 7f824a800700 1 --2- 172.21.15.32:0/1157948426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f824c10c370 0x7f824c1a8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:42.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.752+0000 7f8249400700 1 --2- 172.21.15.32:0/1157948426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f824c10b9d0 0x7f824c1a13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:42.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.752+0000 7f8249e00700 1 -- 172.21.15.32:0/1157948426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f824c10b9d0 msgr2=0x7f824c1a13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:42.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.752+0000 7f8249e00700 1 --2- 172.21.15.32:0/1157948426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f824c10b9d0 0x7f824c1a13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:42.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.752+0000 7f8249e00700 1 -- 172.21.15.32:0/1157948426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f824c10c370 msgr2=0x7f824c1a8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:42.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.752+0000 7f8249e00700 1 --2- 172.21.15.32:0/1157948426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f824c10c370 0x7f824c1a8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:42.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.752+0000 7f8249e00700 1 -- 172.21.15.32:0/1157948426 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f824c1a89e0 con 0x7f824c10aa60 2024-04-30T08:34:42.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.753+0000 7f8249e00700 1 --2- 172.21.15.32:0/1157948426 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f824c10aa60 0x7f824c1a0e80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f823c00d950 tx=0x7f823c00dc60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:42.754 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.753+0000 7f8242a00700 1 -- 172.21.15.32:0/1157948426 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f823c016070 con 0x7f824c10aa60 2024-04-30T08:34:42.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.753+0000 7f8242a00700 1 -- 172.21.15.32:0/1157948426 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f823c005020 con 0x7f824c10aa60 2024-04-30T08:34:42.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.753+0000 7f8242a00700 1 -- 172.21.15.32:0/1157948426 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f823c00f9d0 con 0x7f824c10aa60 2024-04-30T08:34:42.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.753+0000 7f8251000700 1 -- 172.21.15.32:0/1157948426 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f824c1a8cd0 con 0x7f824c10aa60 2024-04-30T08:34:42.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.753+0000 7f8251000700 1 -- 172.21.15.32:0/1157948426 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f824c1a91c0 con 0x7f824c10aa60 2024-04-30T08:34:42.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.754+0000 7f8251000700 1 -- 172.21.15.32:0/1157948426 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f824c051210 con 0x7f824c10aa60 2024-04-30T08:34:42.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.757+0000 7f8242a00700 1 -- 172.21.15.32:0/1157948426 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f823c0051d0 con 0x7f824c10aa60 2024-04-30T08:34:42.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.758+0000 7f8242a00700 1 --2- 172.21.15.32:0/1157948426 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f82300afb80 0x7f82300b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:42.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.759+0000 7f8242a00700 1 -- 172.21.15.32:0/1157948426 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f823c0dea40 con 0x7f824c10aa60 2024-04-30T08:34:42.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.759+0000 7f8249400700 1 --2- 172.21.15.32:0/1157948426 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f82300afb80 0x7f82300b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:42.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.760+0000 7f8249400700 1 --2- 172.21.15.32:0/1157948426 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f82300afb80 0x7f82300b2090 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f82380098c0 tx=0x7f8238019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:42.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:42.763+0000 7f8242a00700 1 -- 172.21.15.32:0/1157948426 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f823c0a7210 con 0x7f824c10aa60 2024-04-30T08:34:43.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:43.047+0000 7f8251000700 1 -- 172.21.15.32:0/1157948426 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7f824c064320 con 0x7f824c10aa60 2024-04-30T08:34:43.048 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:42 smithi032 ceph-mon[28331]: pgmap v411: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-30T08:34:43.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:43.048+0000 7f8242a00700 1 -- 172.21.15.32:0/1157948426 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) ==== 76+0+3088 (secure 0 0 0) 0x7f823c0a6b10 con 0x7f824c10aa60 2024-04-30T08:34:43.049 INFO:teuthology.orchestra.run.smithi032.stdout:e27 2024-04-30T08:34:43.049 INFO:teuthology.orchestra.run.smithi032.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-30T08:34:43.049 INFO:teuthology.orchestra.run.smithi032.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-30T08:34:43.049 INFO:teuthology.orchestra.run.smithi032.stdout:legacy client fscid: 1 2024-04-30T08:34:43.049 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:34:43.049 INFO:teuthology.orchestra.run.smithi032.stdout:Filesystem 'cephfs' (1) 2024-04-30T08:34:43.049 INFO:teuthology.orchestra.run.smithi032.stdout:fs_name cephfs 2024-04-30T08:34:43.050 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 27 2024-04-30T08:34:43.050 INFO:teuthology.orchestra.run.smithi032.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-30T08:34:43.050 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-30T08:31:52.236668+0000 2024-04-30T08:34:43.050 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-30T08:32:30.541714+0000 2024-04-30T08:34:43.050 INFO:teuthology.orchestra.run.smithi032.stdout:tableserver 0 2024-04-30T08:34:43.050 INFO:teuthology.orchestra.run.smithi032.stdout:root 0 2024-04-30T08:34:43.050 INFO:teuthology.orchestra.run.smithi032.stdout:session_timeout 300 2024-04-30T08:34:43.050 INFO:teuthology.orchestra.run.smithi032.stdout:session_autoclose 300 2024-04-30T08:34:43.050 INFO:teuthology.orchestra.run.smithi032.stdout:max_file_size 1099511627776 2024-04-30T08:34:43.050 INFO:teuthology.orchestra.run.smithi032.stdout:max_xattr_size 65536 2024-04-30T08:34:43.050 INFO:teuthology.orchestra.run.smithi032.stdout:required_client_features {} 2024-04-30T08:34:43.050 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure 0 2024-04-30T08:34:43.050 INFO:teuthology.orchestra.run.smithi032.stdout:last_failure_osd_epoch 0 2024-04-30T08:34:43.051 INFO:teuthology.orchestra.run.smithi032.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-30T08:34:43.051 INFO:teuthology.orchestra.run.smithi032.stdout:max_mds 5 2024-04-30T08:34:43.051 INFO:teuthology.orchestra.run.smithi032.stdout:in 0,1,2,3,4 2024-04-30T08:34:43.051 INFO:teuthology.orchestra.run.smithi032.stdout:up {0=24590,1=24560,2=24541,3=24533,4=24517} 2024-04-30T08:34:43.051 INFO:teuthology.orchestra.run.smithi032.stdout:failed 2024-04-30T08:34:43.051 INFO:teuthology.orchestra.run.smithi032.stdout:damaged 2024-04-30T08:34:43.051 INFO:teuthology.orchestra.run.smithi032.stdout:stopped 2024-04-30T08:34:43.051 INFO:teuthology.orchestra.run.smithi032.stdout:data_pools [3,4] 2024-04-30T08:34:43.051 INFO:teuthology.orchestra.run.smithi032.stdout:metadata_pool 2 2024-04-30T08:34:43.051 INFO:teuthology.orchestra.run.smithi032.stdout:inline_data disabled 2024-04-30T08:34:43.051 INFO:teuthology.orchestra.run.smithi032.stdout:balancer 2024-04-30T08:34:43.051 INFO:teuthology.orchestra.run.smithi032.stdout:bal_rank_mask -1 2024-04-30T08:34:43.051 INFO:teuthology.orchestra.run.smithi032.stdout:standby_count_wanted 0 2024-04-30T08:34:43.051 INFO:teuthology.orchestra.run.smithi032.stdout:qdb_cluster leader: 24590 members: 24590,24541,24560,24533,24517 2024-04-30T08:34:43.052 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.b{0:24590} state up:active seq 4 addr [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] compat {c=[1],r=[1],i=[1fff]}] 2024-04-30T08:34:43.052 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.e{0:24506} state up:standby-replay seq 1 addr [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] compat {c=[1],r=[1],i=[1fff]}] 2024-04-30T08:34:43.052 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.g{1:24560} state up:active seq 14 addr [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] compat {c=[1],r=[1],i=[1fff]}] 2024-04-30T08:34:43.052 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.f{1:24505} state up:standby-replay seq 1 addr [v2:172.21.15.189:6834/278686565,v1:172.21.15.189:6835/278686565] compat {c=[1],r=[1],i=[1fff]}] 2024-04-30T08:34:43.052 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.i{2:24541} state up:active seq 15 addr [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] compat {c=[1],r=[1],i=[1fff]}] 2024-04-30T08:34:43.052 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.d{2:24484} state up:standby-replay seq 1 addr [v2:172.21.15.32:6836/2569449994,v1:172.21.15.32:6837/2569449994] compat {c=[1],r=[1],i=[1fff]}] 2024-04-30T08:34:43.052 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.h{3:24533} state up:active seq 16 addr [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] compat {c=[1],r=[1],i=[1fff]}] 2024-04-30T08:34:43.052 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.k{3:24458} state up:standby-replay seq 1 addr [v2:172.21.15.71:6832/3635143591,v1:172.21.15.71:6833/3635143591] compat {c=[1],r=[1],i=[1fff]}] 2024-04-30T08:34:43.052 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.c{4:24517} state up:active seq 17 addr [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] compat {c=[1],r=[1],i=[1fff]}] 2024-04-30T08:34:43.052 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.l{4:24442} state up:standby-replay seq 1 addr [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] compat {c=[1],r=[1],i=[1fff]}] 2024-04-30T08:34:43.052 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:34:43.052 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:34:43.053 INFO:teuthology.orchestra.run.smithi032.stdout:Standby daemons: 2024-04-30T08:34:43.053 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:34:43.053 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.j{-1:14379} state up:standby seq 1 addr [v2:172.21.15.32:6834/2235842676,v1:172.21.15.32:6835/2235842676] compat {c=[1],r=[1],i=[1fff]}] 2024-04-30T08:34:43.053 INFO:teuthology.orchestra.run.smithi032.stdout:[mds.a{-1:14457} state up:standby seq 1 addr [v2:172.21.15.32:6840/3577009355,v1:172.21.15.32:6841/3577009355] compat {c=[1],r=[1],i=[1fff]}] 2024-04-30T08:34:43.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:43.051+0000 7f8251000700 1 -- 172.21.15.32:0/1157948426 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f82300afb80 msgr2=0x7f82300b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:43.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:43.051+0000 7f8251000700 1 --2- 172.21.15.32:0/1157948426 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f82300afb80 0x7f82300b2090 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f82380098c0 tx=0x7f8238019040 comp rx=0 tx=0).stop 2024-04-30T08:34:43.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:43.051+0000 7f8251000700 1 -- 172.21.15.32:0/1157948426 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f824c10aa60 msgr2=0x7f824c1a0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:43.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:43.052+0000 7f8251000700 1 --2- 172.21.15.32:0/1157948426 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f824c10aa60 0x7f824c1a0e80 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f823c00d950 tx=0x7f823c00dc60 comp rx=0 tx=0).stop 2024-04-30T08:34:43.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:43.052+0000 7f8251000700 1 -- 172.21.15.32:0/1157948426 shutdown_connections 2024-04-30T08:34:43.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:43.052+0000 7f8251000700 1 --2- 172.21.15.32:0/1157948426 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f82300afb80 0x7f82300b2090 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:43.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:43.052+0000 7f8251000700 1 --2- 172.21.15.32:0/1157948426 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f824c10c370 0x7f824c1a8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:43.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:43.052+0000 7f8251000700 1 --2- 172.21.15.32:0/1157948426 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f824c10aa60 0x7f824c1a0e80 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:43.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:43.052+0000 7f8251000700 1 --2- 172.21.15.32:0/1157948426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f824c10b9d0 0x7f824c1a13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:43.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:43.052+0000 7f8251000700 1 -- 172.21.15.32:0/1157948426 >> 172.21.15.32:0/1157948426 conn(0x7f824c079910 msgr2=0x7f824c10ec00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:43.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:43.052+0000 7f8251000700 1 -- 172.21.15.32:0/1157948426 shutdown_connections 2024-04-30T08:34:43.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:43.052+0000 7f8251000700 1 -- 172.21.15.32:0/1157948426 wait complete. 2024-04-30T08:34:43.055 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 27 2024-04-30T08:34:43.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:42 smithi071 ceph-mon[33385]: pgmap v411: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-30T08:34:43.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:42 smithi189 ceph-mon[33038]: pgmap v411: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-30T08:34:44.010 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-30T08:34:44.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:43 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1157948426' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-30T08:34:44.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:43 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1157948426' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-30T08:34:44.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:43 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1157948426' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-30T08:34:44.502 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:34:45.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:45 smithi071 ceph-mon[33385]: pgmap v412: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:34:45.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:45 smithi189 ceph-mon[33038]: pgmap v412: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:34:45.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:45 smithi032 ceph-mon[28331]: pgmap v412: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:34:46.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.764+0000 7ff640c00700 1 -- 172.21.15.32:0/2409002843 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff63c077630 msgr2=0x7ff63c075b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:46.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.764+0000 7ff640c00700 1 --2- 172.21.15.32:0/2409002843 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff63c077630 0x7ff63c075b00 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff62c009990 tx=0x7ff62c009ca0 comp rx=0 tx=0).stop 2024-04-30T08:34:46.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.764+0000 7ff640c00700 1 -- 172.21.15.32:0/2409002843 shutdown_connections 2024-04-30T08:34:46.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.764+0000 7ff640c00700 1 --2- 172.21.15.32:0/2409002843 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff63c10c140 0x7ff63c10e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:46.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.764+0000 7ff640c00700 1 --2- 172.21.15.32:0/2409002843 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff63c077630 0x7ff63c075b00 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:46.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.764+0000 7ff640c00700 1 --2- 172.21.15.32:0/2409002843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff63c076040 0x7ff63c0764d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:46.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.764+0000 7ff640c00700 1 -- 172.21.15.32:0/2409002843 >> 172.21.15.32:0/2409002843 conn(0x7ff63c0ff910 msgr2=0x7ff63c101d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:46.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.765+0000 7ff640c00700 1 -- 172.21.15.32:0/2409002843 shutdown_connections 2024-04-30T08:34:46.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.765+0000 7ff640c00700 1 -- 172.21.15.32:0/2409002843 wait complete. 2024-04-30T08:34:46.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.766+0000 7ff640c00700 1 Processor -- start 2024-04-30T08:34:46.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.766+0000 7ff640c00700 1 -- start start 2024-04-30T08:34:46.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.767+0000 7ff640c00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff63c076040 0x7ff63c19caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:46.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.767+0000 7ff640c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff63c077630 0x7ff63c19d030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:46.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.767+0000 7ff640c00700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff63c10c140 0x7ff63c1a3f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:46.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.767+0000 7ff640c00700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff63c111f20 con 0x7ff63c076040 2024-04-30T08:34:46.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.767+0000 7ff640c00700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff63c111da0 con 0x7ff63c10c140 2024-04-30T08:34:46.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.767+0000 7ff640c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff63c1120a0 con 0x7ff63c077630 2024-04-30T08:34:46.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.767+0000 7ff639400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff63c077630 0x7ff63c19d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:46.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.767+0000 7ff639e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff63c076040 0x7ff63c19caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:46.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.767+0000 7ff639400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff63c077630 0x7ff63c19d030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:48766/0 (socket says 172.21.15.32:48766) 2024-04-30T08:34:46.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.767+0000 7ff639400700 1 -- 172.21.15.32:0/3126364931 learned_addr learned my addr 172.21.15.32:0/3126364931 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:46.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.767+0000 7ff63a800700 1 --2- 172.21.15.32:0/3126364931 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff63c10c140 0x7ff63c1a3f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:46.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.768+0000 7ff639400700 1 -- 172.21.15.32:0/3126364931 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff63c10c140 msgr2=0x7ff63c1a3f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:46.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.768+0000 7ff639400700 1 --2- 172.21.15.32:0/3126364931 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff63c10c140 0x7ff63c1a3f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:46.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.768+0000 7ff639400700 1 -- 172.21.15.32:0/3126364931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff63c076040 msgr2=0x7ff63c19caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:46.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.768+0000 7ff639400700 1 --2- 172.21.15.32:0/3126364931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff63c076040 0x7ff63c19caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:46.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.768+0000 7ff639400700 1 -- 172.21.15.32:0/3126364931 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff63c1a45e0 con 0x7ff63c077630 2024-04-30T08:34:46.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.768+0000 7ff63a800700 1 --2- 172.21.15.32:0/3126364931 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff63c10c140 0x7ff63c1a3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:34:46.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.768+0000 7ff639400700 1 --2- 172.21.15.32:0/3126364931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff63c077630 0x7ff63c19d030 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff62800e9a0 tx=0x7ff62800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:46.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.768+0000 7ff632a00700 1 -- 172.21.15.32:0/3126364931 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff62800ca10 con 0x7ff63c077630 2024-04-30T08:34:46.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.768+0000 7ff640c00700 1 -- 172.21.15.32:0/3126364931 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff63c1a48d0 con 0x7ff63c077630 2024-04-30T08:34:46.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.768+0000 7ff632a00700 1 -- 172.21.15.32:0/3126364931 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff62800cbb0 con 0x7ff63c077630 2024-04-30T08:34:46.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.768+0000 7ff632a00700 1 -- 172.21.15.32:0/3126364931 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff628018680 con 0x7ff63c077630 2024-04-30T08:34:46.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.768+0000 7ff640c00700 1 -- 172.21.15.32:0/3126364931 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff63c1a4e10 con 0x7ff63c077630 2024-04-30T08:34:46.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.770+0000 7ff640c00700 1 -- 172.21.15.32:0/3126364931 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff63c051210 con 0x7ff63c077630 2024-04-30T08:34:46.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.772+0000 7ff632a00700 1 -- 172.21.15.32:0/3126364931 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7ff628010430 con 0x7ff63c077630 2024-04-30T08:34:46.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.774+0000 7ff632a00700 1 --2- 172.21.15.32:0/3126364931 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff6200afb80 0x7ff6200b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:46.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.774+0000 7ff632a00700 1 -- 172.21.15.32:0/3126364931 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7ff6280df460 con 0x7ff63c077630 2024-04-30T08:34:46.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.774+0000 7ff639e00700 1 --2- 172.21.15.32:0/3126364931 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff6200afb80 0x7ff6200b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:46.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.775+0000 7ff639e00700 1 --2- 172.21.15.32:0/3126364931 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff6200afb80 0x7ff6200b2090 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff62c006010 tx=0x7ff62c00b2a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:46.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:46.776+0000 7ff632a00700 1 -- 172.21.15.32:0/3126364931 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff6280a7c70 con 0x7ff63c077630 2024-04-30T08:34:47.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:47.005+0000 7ff640c00700 1 -- 172.21.15.32:0/3126364931 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7ff63c02e720 con 0x7ff63c077630 2024-04-30T08:34:47.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:47.015+0000 7ff632a00700 1 -- 172.21.15.32:0/3126364931 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v85) ==== 56+0+5683 (secure 0 0 0) 0x7ff6280a7570 con 0x7ff63c077630 2024-04-30T08:34:47.017 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:47 smithi032 ceph-mon[28331]: pgmap v413: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-30T08:34:47.017 INFO:teuthology.orchestra.run.smithi032.stdout:epoch 85 2024-04-30T08:34:47.018 INFO:teuthology.orchestra.run.smithi032.stdout:fsid 98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:34:47.018 INFO:teuthology.orchestra.run.smithi032.stdout:created 2024-04-30T08:22:11.122875+0000 2024-04-30T08:34:47.018 INFO:teuthology.orchestra.run.smithi032.stdout:modified 2024-04-30T08:33:31.872704+0000 2024-04-30T08:34:47.018 INFO:teuthology.orchestra.run.smithi032.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-30T08:34:47.018 INFO:teuthology.orchestra.run.smithi032.stdout:crush_version 27 2024-04-30T08:34:47.018 INFO:teuthology.orchestra.run.smithi032.stdout:full_ratio 0.95 2024-04-30T08:34:47.018 INFO:teuthology.orchestra.run.smithi032.stdout:backfillfull_ratio 0.9 2024-04-30T08:34:47.018 INFO:teuthology.orchestra.run.smithi032.stdout:nearfull_ratio 0.85 2024-04-30T08:34:47.018 INFO:teuthology.orchestra.run.smithi032.stdout:require_min_compat_client luminous 2024-04-30T08:34:47.018 INFO:teuthology.orchestra.run.smithi032.stdout:min_compat_client luminous 2024-04-30T08:34:47.018 INFO:teuthology.orchestra.run.smithi032.stdout:require_osd_release squid 2024-04-30T08:34:47.019 INFO:teuthology.orchestra.run.smithi032.stdout:stretch_mode_enabled false 2024-04-30T08:34:47.019 INFO:teuthology.orchestra.run.smithi032.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-30T08:34:47.019 INFO:teuthology.orchestra.run.smithi032.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 75 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2024-04-30T08:34:47.019 INFO:teuthology.orchestra.run.smithi032.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 76 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-04-30T08:34:47.019 INFO:teuthology.orchestra.run.smithi032.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 81 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-04-30T08:34:47.019 INFO:teuthology.orchestra.run.smithi032.stdout:max_osd 12 2024-04-30T08:34:47.019 INFO:teuthology.orchestra.run.smithi032.stdout:osd.0 up in weight 1 up_from 8 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.32:6802/2624700402,v1:172.21.15.32:6803/2624700402] [v2:172.21.15.32:6804/2624700402,v1:172.21.15.32:6805/2624700402] exists,up a7e060c8-1cbb-418f-9a03-06693f6b3f52 2024-04-30T08:34:47.019 INFO:teuthology.orchestra.run.smithi032.stdout:osd.1 up in weight 1 up_from 13 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.71:6800/444649711,v1:172.21.15.71:6801/444649711] [v2:172.21.15.71:6802/444649711,v1:172.21.15.71:6803/444649711] exists,up 72ef2fb1-e5ca-48ef-aa3a-4796a64b922f 2024-04-30T08:34:47.019 INFO:teuthology.orchestra.run.smithi032.stdout:osd.2 up in weight 1 up_from 18 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.189:6800/3255528671,v1:172.21.15.189:6801/3255528671] [v2:172.21.15.189:6802/3255528671,v1:172.21.15.189:6803/3255528671] exists,up 704e5e8a-935e-47c7-ae28-27119876ac09 2024-04-30T08:34:47.019 INFO:teuthology.orchestra.run.smithi032.stdout:osd.3 up in weight 1 up_from 25 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.32:6810/1384702112,v1:172.21.15.32:6811/1384702112] [v2:172.21.15.32:6812/1384702112,v1:172.21.15.32:6813/1384702112] exists,up 575004f7-6559-4a62-863e-e33078b0eebf 2024-04-30T08:34:47.019 INFO:teuthology.orchestra.run.smithi032.stdout:osd.4 up in weight 1 up_from 31 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.71:6808/4250534574,v1:172.21.15.71:6809/4250534574] [v2:172.21.15.71:6810/4250534574,v1:172.21.15.71:6811/4250534574] exists,up e1c1e894-074d-4c78-89ef-9e5a114268bd 2024-04-30T08:34:47.019 INFO:teuthology.orchestra.run.smithi032.stdout:osd.5 up in weight 1 up_from 36 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.189:6808/4245346198,v1:172.21.15.189:6809/4245346198] [v2:172.21.15.189:6810/4245346198,v1:172.21.15.189:6811/4245346198] exists,up 12c022d4-fac6-40c4-ae7a-691571bee5d0 2024-04-30T08:34:47.019 INFO:teuthology.orchestra.run.smithi032.stdout:osd.6 up in weight 1 up_from 42 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.32:6818/1910389190,v1:172.21.15.32:6819/1910389190] [v2:172.21.15.32:6820/1910389190,v1:172.21.15.32:6821/1910389190] exists,up b413f8ee-3d4a-415a-a2ba-d68437b7330d 2024-04-30T08:34:47.019 INFO:teuthology.orchestra.run.smithi032.stdout:osd.7 up in weight 1 up_from 48 up_thru 79 down_at 0 last_clean_interval [0,0) [v2:172.21.15.71:6816/692607380,v1:172.21.15.71:6817/692607380] [v2:172.21.15.71:6818/692607380,v1:172.21.15.71:6819/692607380] exists,up aac64fb3-fc13-4126-b842-7b06ad5ce238 2024-04-30T08:34:47.019 INFO:teuthology.orchestra.run.smithi032.stdout:osd.8 up in weight 1 up_from 54 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.189:6816/3973697069,v1:172.21.15.189:6817/3973697069] [v2:172.21.15.189:6818/3973697069,v1:172.21.15.189:6819/3973697069] exists,up cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5 2024-04-30T08:34:47.020 INFO:teuthology.orchestra.run.smithi032.stdout:osd.9 up in weight 1 up_from 60 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.32:6826/46066685,v1:172.21.15.32:6827/46066685] [v2:172.21.15.32:6828/46066685,v1:172.21.15.32:6829/46066685] exists,up ef4a1645-ae6e-45c3-82ad-161510e62aa9 2024-04-30T08:34:47.020 INFO:teuthology.orchestra.run.smithi032.stdout:osd.10 up in weight 1 up_from 65 up_thru 82 down_at 0 last_clean_interval [0,0) [v2:172.21.15.71:6824/1886290293,v1:172.21.15.71:6825/1886290293] [v2:172.21.15.71:6826/1886290293,v1:172.21.15.71:6827/1886290293] exists,up 25b26a38-a644-409f-b941-59efdf04a3ae 2024-04-30T08:34:47.020 INFO:teuthology.orchestra.run.smithi032.stdout:osd.11 up in weight 1 up_from 71 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.189:6824/850185001,v1:172.21.15.189:6825/850185001] [v2:172.21.15.189:6826/850185001,v1:172.21.15.189:6827/850185001] exists,up ab96b3ff-3688-455b-af64-60226c5bfbff 2024-04-30T08:34:47.020 INFO:teuthology.orchestra.run.smithi032.stdout:pg_upmap_items 2.a [2,11] 2024-04-30T08:34:47.020 INFO:teuthology.orchestra.run.smithi032.stdout:pg_upmap_items 2.13 [2,11] 2024-04-30T08:34:47.020 INFO:teuthology.orchestra.run.smithi032.stdout:pg_upmap_items 3.0 [4,8] 2024-04-30T08:34:47.020 INFO:teuthology.orchestra.run.smithi032.stdout:pg_upmap_items 3.e [4,7] 2024-04-30T08:34:47.020 INFO:teuthology.orchestra.run.smithi032.stdout:pg_upmap_items 3.17 [4,6] 2024-04-30T08:34:47.020 INFO:teuthology.orchestra.run.smithi032.stdout:pg_upmap_items 3.18 [3,9] 2024-04-30T08:34:47.020 INFO:teuthology.orchestra.run.smithi032.stdout:pg_upmap_items 3.1a [5,7] 2024-04-30T08:34:47.020 INFO:teuthology.orchestra.run.smithi032.stdout:pg_upmap_items 3.28 [3,6,5,8] 2024-04-30T08:34:47.021 INFO:teuthology.orchestra.run.smithi032.stdout:pg_upmap_items 3.30 [1,9] 2024-04-30T08:34:47.021 INFO:teuthology.orchestra.run.smithi032.stdout:pg_upmap_items 3.36 [4,9] 2024-04-30T08:34:47.021 INFO:teuthology.orchestra.run.smithi032.stdout:pg_upmap_items 4.8 [5,8] 2024-04-30T08:34:47.021 INFO:teuthology.orchestra.run.smithi032.stdout:pg_upmap_items 4.12 [9,10] 2024-04-30T08:34:47.021 INFO:teuthology.orchestra.run.smithi032.stdout:pg_upmap_items 4.13 [9,8] 2024-04-30T08:34:47.021 INFO:teuthology.orchestra.run.smithi032.stdout:pg_upmap_items 4.14 [9,8] 2024-04-30T08:34:47.021 INFO:teuthology.orchestra.run.smithi032.stdout:blocklist 172.21.15.32:0/1266897222 expires 2024-05-01T08:23:29.526664+0000 2024-04-30T08:34:47.021 INFO:teuthology.orchestra.run.smithi032.stdout:blocklist 172.21.15.32:0/2915689533 expires 2024-05-01T08:23:29.526664+0000 2024-04-30T08:34:47.021 INFO:teuthology.orchestra.run.smithi032.stdout:blocklist 172.21.15.32:6801/4019351306 expires 2024-05-01T08:23:29.526664+0000 2024-04-30T08:34:47.021 INFO:teuthology.orchestra.run.smithi032.stdout:blocklist 172.21.15.32:0/425836193 expires 2024-05-01T08:23:29.526664+0000 2024-04-30T08:34:47.021 INFO:teuthology.orchestra.run.smithi032.stdout:blocklist 172.21.15.32:0/1134075722 expires 2024-05-01T08:22:53.639893+0000 2024-04-30T08:34:47.021 INFO:teuthology.orchestra.run.smithi032.stdout:blocklist 172.21.15.32:6800/464872591 expires 2024-05-01T08:22:53.639893+0000 2024-04-30T08:34:47.021 INFO:teuthology.orchestra.run.smithi032.stdout:blocklist 172.21.15.32:6801/464872591 expires 2024-05-01T08:22:53.639893+0000 2024-04-30T08:34:47.021 INFO:teuthology.orchestra.run.smithi032.stdout:blocklist 172.21.15.32:0/21059253 expires 2024-05-01T08:22:53.639893+0000 2024-04-30T08:34:47.021 INFO:teuthology.orchestra.run.smithi032.stdout:blocklist 172.21.15.32:0/3245550894 expires 2024-05-01T08:22:53.639893+0000 2024-04-30T08:34:47.021 INFO:teuthology.orchestra.run.smithi032.stdout:blocklist 172.21.15.32:6800/4019351306 expires 2024-05-01T08:23:29.526664+0000 2024-04-30T08:34:47.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:47.018+0000 7ff640c00700 1 -- 172.21.15.32:0/3126364931 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff6200afb80 msgr2=0x7ff6200b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:47.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:47.018+0000 7ff640c00700 1 --2- 172.21.15.32:0/3126364931 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff6200afb80 0x7ff6200b2090 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ff62c006010 tx=0x7ff62c00b2a0 comp rx=0 tx=0).stop 2024-04-30T08:34:47.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:47.018+0000 7ff640c00700 1 -- 172.21.15.32:0/3126364931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff63c077630 msgr2=0x7ff63c19d030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:47.022 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:47.018+0000 7ff640c00700 1 --2- 172.21.15.32:0/3126364931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff63c077630 0x7ff63c19d030 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7ff62800e9a0 tx=0x7ff62800ecb0 comp rx=0 tx=0).stop 2024-04-30T08:34:47.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:47.019+0000 7ff640c00700 1 -- 172.21.15.32:0/3126364931 shutdown_connections 2024-04-30T08:34:47.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:47.019+0000 7ff640c00700 1 --2- 172.21.15.32:0/3126364931 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff6200afb80 0x7ff6200b2090 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:47.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:47.019+0000 7ff640c00700 1 --2- 172.21.15.32:0/3126364931 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff63c10c140 0x7ff63c1a3f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:47.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:47.019+0000 7ff640c00700 1 --2- 172.21.15.32:0/3126364931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff63c076040 0x7ff63c19caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:47.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:47.019+0000 7ff640c00700 1 --2- 172.21.15.32:0/3126364931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff63c077630 0x7ff63c19d030 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:47.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:47.019+0000 7ff640c00700 1 -- 172.21.15.32:0/3126364931 >> 172.21.15.32:0/3126364931 conn(0x7ff63c0ff910 msgr2=0x7ff63c1080d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:47.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:47.019+0000 7ff640c00700 1 -- 172.21.15.32:0/3126364931 shutdown_connections 2024-04-30T08:34:47.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:47.019+0000 7ff640c00700 1 -- 172.21.15.32:0/3126364931 wait complete. 2024-04-30T08:34:47.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:47 smithi189 ceph-mon[33038]: pgmap v413: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-30T08:34:47.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:47 smithi071 ceph-mon[33385]: pgmap v413: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-30T08:34:47.862 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-30T08:34:47.873 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-30T08:34:48.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.175+0000 7fd857e00640 1 -- 172.21.15.32:0/2477226600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd858075960 msgr2=0x7fd85810b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:48.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.175+0000 7fd857e00640 1 --2- 172.21.15.32:0/2477226600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd858075960 0x7fd85810b3c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd840009900 tx=0x7fd84002a410 comp rx=0 tx=0).stop 2024-04-30T08:34:48.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.175+0000 7fd857e00640 1 -- 172.21.15.32:0/2477226600 shutdown_connections 2024-04-30T08:34:48.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.175+0000 7fd857e00640 1 --2- 172.21.15.32:0/2477226600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd85810b900 0x7fd85810dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:48.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.175+0000 7fd857e00640 1 --2- 172.21.15.32:0/2477226600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd858075960 0x7fd85810b3c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:48.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.175+0000 7fd857e00640 1 --2- 172.21.15.32:0/2477226600 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd858075010 0x7fd858075420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:48.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.175+0000 7fd857e00640 1 -- 172.21.15.32:0/2477226600 >> 172.21.15.32:0/2477226600 conn(0x7fd8580feba0 msgr2=0x7fd858101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:48.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.175+0000 7fd857e00640 1 -- 172.21.15.32:0/2477226600 shutdown_connections 2024-04-30T08:34:48.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.175+0000 7fd857e00640 1 -- 172.21.15.32:0/2477226600 wait complete. 2024-04-30T08:34:48.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.176+0000 7fd857e00640 1 Processor -- start 2024-04-30T08:34:48.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.177+0000 7fd857e00640 1 -- start start 2024-04-30T08:34:48.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.177+0000 7fd857e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd858075010 0x7fd85819c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:48.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.177+0000 7fd857e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd858075960 0x7fd85819c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:48.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.177+0000 7fd857e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd85810b900 0x7fd8581a37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:48.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.177+0000 7fd857e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd858111730 con 0x7fd85810b900 2024-04-30T08:34:48.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.177+0000 7fd857e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd8581115b0 con 0x7fd858075010 2024-04-30T08:34:48.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.177+0000 7fd857e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd8581118b0 con 0x7fd858075960 2024-04-30T08:34:48.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.177+0000 7fd856a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd858075010 0x7fd85819c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:48.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.177+0000 7fd856000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd858075960 0x7fd85819c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:48.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.177+0000 7fd856a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd858075010 0x7fd85819c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:46136/0 (socket says 172.21.15.32:46136) 2024-04-30T08:34:48.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.177+0000 7fd856a00640 1 -- 172.21.15.32:0/1571734817 learned_addr learned my addr 172.21.15.32:0/1571734817 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:48.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.178+0000 7fd857400640 1 --2- 172.21.15.32:0/1571734817 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd85810b900 0x7fd8581a37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:48.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.178+0000 7fd856a00640 1 -- 172.21.15.32:0/1571734817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd858075960 msgr2=0x7fd85819c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:48.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.178+0000 7fd856a00640 1 --2- 172.21.15.32:0/1571734817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd858075960 0x7fd85819c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:48.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.178+0000 7fd856a00640 1 -- 172.21.15.32:0/1571734817 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd85810b900 msgr2=0x7fd8581a37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:48.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.178+0000 7fd856a00640 1 --2- 172.21.15.32:0/1571734817 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd85810b900 0x7fd8581a37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:48.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.178+0000 7fd856a00640 1 -- 172.21.15.32:0/1571734817 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8581a3e20 con 0x7fd858075010 2024-04-30T08:34:48.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.178+0000 7fd856a00640 1 --2- 172.21.15.32:0/1571734817 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd858075010 0x7fd85819c330 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd84800e9a0 tx=0x7fd84800ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:48.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.178+0000 7fd84f400640 1 -- 172.21.15.32:0/1571734817 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd84800cd80 con 0x7fd858075010 2024-04-30T08:34:48.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.179+0000 7fd84f400640 1 -- 172.21.15.32:0/1571734817 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd8480035f0 con 0x7fd858075010 2024-04-30T08:34:48.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.179+0000 7fd84f400640 1 -- 172.21.15.32:0/1571734817 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd848010690 con 0x7fd858075010 2024-04-30T08:34:48.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.179+0000 7fd857e00640 1 -- 172.21.15.32:0/1571734817 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8581a4110 con 0x7fd858075010 2024-04-30T08:34:48.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.179+0000 7fd857e00640 1 -- 172.21.15.32:0/1571734817 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8581a4630 con 0x7fd858075010 2024-04-30T08:34:48.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.180+0000 7fd857e00640 1 -- 172.21.15.32:0/1571734817 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd85810aac0 con 0x7fd858075010 2024-04-30T08:34:48.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.182+0000 7fd84f400640 1 -- 172.21.15.32:0/1571734817 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fd848004000 con 0x7fd858075010 2024-04-30T08:34:48.185 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.184+0000 7fd84f400640 1 --2- 172.21.15.32:0/1571734817 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd83c0afae0 0x7fd83c0b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:48.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.184+0000 7fd84f400640 1 -- 172.21.15.32:0/1571734817 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fd8480df560 con 0x7fd858075010 2024-04-30T08:34:48.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.184+0000 7fd856000640 1 --2- 172.21.15.32:0/1571734817 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd83c0afae0 0x7fd83c0b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:48.186 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.185+0000 7fd856000640 1 --2- 172.21.15.32:0/1571734817 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd83c0afae0 0x7fd83c0b1ff0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fd840004770 tx=0x7fd8400314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:48.187 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.186+0000 7fd84f400640 1 -- 172.21.15.32:0/1571734817 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd8480042f0 con 0x7fd858075010 2024-04-30T08:34:48.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:48 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3126364931' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-30T08:34:48.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:48 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3126364931' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-30T08:34:48.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.399+0000 7fd857e00640 1 -- 172.21.15.32:0/1571734817 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd858063820 con 0x7fd858075010 2024-04-30T08:34:48.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.402+0000 7fd84f400640 1 -- 172.21.15.32:0/1571734817 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+24730 (secure 0 0 0) 0x7fd8480a7d30 con 0x7fd858075010 2024-04-30T08:34:48.404 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:34:48.404 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":85,"fsid":"98c0a670-06ca-11ef-bc93-c7b262605968","created":"2024-04-30T08:22:11.122875+0000","modified":"2024-04-30T08:33:31.872704+0000","last_up_change":"2024-04-30T08:30:50.350439+0000","last_in_change":"2024-04-30T08:30:30.210620+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-30T08:26:19.728331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-30T08:31:49.800542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-30T08:31:51.211320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-30T08:31:55.716625+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a7e060c8-1cbb-418f-9a03-06693f6b3f52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6803","nonce":2624700402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6805","nonce":2624700402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6809","nonce":2624700402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6807","nonce":2624700402}]},"public_addr":"172.21.15.32:6803/2624700402","cluster_addr":"172.21.15.32:6805/2624700402","heartbeat_back_addr":"172.21.15.32:6809/2624700402","heartbeat_front_addr":"172.21.15.32:6807/2624700402","state":["exists","up"]},{"osd":1,"uuid":"72ef2fb1-e5ca-48ef-aa3a-4796a64b922f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6801","nonce":444649711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6803","nonce":444649711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6807","nonce":444649711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6805","nonce":444649711}]},"public_addr":"172.21.15.71:6801/444649711","cluster_addr":"172.21.15.71:6803/444649711","heartbeat_back_addr":"172.21.15.71:6807/444649711","heartbeat_front_addr":"172.21.15.71:6805/444649711","state":["exists","up"]},{"osd":2,"uuid":"704e5e8a-935e-47c7-ae28-27119876ac09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6801","nonce":3255528671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6803","nonce":3255528671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6807","nonce":3255528671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6805","nonce":3255528671}]},"public_addr":"172.21.15.189:6801/3255528671","cluster_addr":"172.21.15.189:6803/3255528671","heartbeat_back_addr":"172.21.15.189:6807/3255528671","heartbeat_front_addr":"172.21.15.189:6805/3255528671","state":["exists","up"]},{"osd":3,"uuid":"575004f7-6559-4a62-863e-e33078b0eebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6811","nonce":1384702112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6813","nonce":1384702112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6817","nonce":1384702112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6815","nonce":1384702112}]},"public_addr":"172.21.15.32:6811/1384702112","cluster_addr":"172.21.15.32:6813/1384702112","heartbeat_back_addr":"172.21.15.32:6817/1384702112","heartbeat_front_addr":"172.21.15.32:6815/1384702112","state":["exists","up"]},{"osd":4,"uuid":"e1c1e894-074d-4c78-89ef-9e5a114268bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6809","nonce":4250534574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6811","nonce":4250534574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6815","nonce":4250534574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6813","nonce":4250534574}]},"public_addr":"172.21.15.71:6809/4250534574","cluster_addr":"172.21.15.71:6811/4250534574","heartbeat_back_addr":"172.21.15.71:6815/4250534574","heartbeat_front_addr":"172.21.15.71:6813/4250534574","state":["exists","up"]},{"osd":5,"uuid":"12c022d4-fac6-40c4-ae7a-691571bee5d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6809","nonce":4245346198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6811","nonce":4245346198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6815","nonce":4245346198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6813","nonce":4245346198}]},"public_addr":"172.21.15.189:6809/4245346198","cluster_addr":"172.21.15.189:6811/4245346198","heartbeat_back_addr":"172.21.15.189:6815/4245346198","heartbeat_front_addr":"172.21.15.189:6813/4245346198","state":["exists","up"]},{"osd":6,"uuid":"b413f8ee-3d4a-415a-a2ba-d68437b7330d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6819","nonce":1910389190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6821","nonce":1910389190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6825","nonce":1910389190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6823","nonce":1910389190}]},"public_addr":"172.21.15.32:6819/1910389190","cluster_addr":"172.21.15.32:6821/1910389190","heartbeat_back_addr":"172.21.15.32:6825/1910389190","heartbeat_front_addr":"172.21.15.32:6823/1910389190","state":["exists","up"]},{"osd":7,"uuid":"aac64fb3-fc13-4126-b842-7b06ad5ce238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6817","nonce":692607380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6819","nonce":692607380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6823","nonce":692607380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6821","nonce":692607380}]},"public_addr":"172.21.15.71:6817/692607380","cluster_addr":"172.21.15.71:6819/692607380","heartbeat_back_addr":"172.21.15.71:6823/692607380","heartbeat_front_addr":"172.21.15.71:6821/692607380","state":["exists","up"]},{"osd":8,"uuid":"cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6817","nonce":3973697069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6819","nonce":3973697069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6823","nonce":3973697069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6821","nonce":3973697069}]},"public_addr":"172.21.15.189:6817/3973697069","cluster_addr":"172.21.15.189:6819/3973697069","heartbeat_back_addr":"172.21.15.189:6823/3973697069","heartbeat_front_addr":"172.21.15.189:6821/3973697069","state":["exists","up"]},{"osd":9,"uuid":"ef4a1645-ae6e-45c3-82ad-161510e62aa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6827","nonce":46066685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6829","nonce":46066685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6833","nonce":46066685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6831","nonce":46066685}]},"public_addr":"172.21.15.32:6827/46066685","cluster_addr":"172.21.15.32:6829/46066685","heartbeat_back_addr":"172.21.15.32:6833/46066685","heartbeat_front_addr":"172.21.15.32:6831/46066685","state":["exists","up"]},{"osd":10,"uuid":"25b26a38-a644-409f-b941-59efdf04a3ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6825","nonce":1886290293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6827","nonce":1886290293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6831","nonce":1886290293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6829","nonce":1886290293}]},"public_addr":"172.21.15.71:6825/1886290293","cluster_addr":"172.21.15.71:6827/1886290293","heartbeat_back_addr":"172.21.15.71:6831/1886290293","heartbeat_front_addr":"172.21.15.71:6829/1886290293","state":["exists","up"]},{"osd":11,"uuid":"ab96b3ff-3688-455b-af64-60226c5bfbff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6825","nonce":850185001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6827","nonce":850185001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6831","nonce":850185001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6829","nonce":850185001}]},"public_addr":"172.21.15.189:6825/850185001","cluster_addr":"172.21.15.189:6827/850185001","heartbeat_back_addr":"172.21.15.189:6831/850185001","heartbeat_front_addr":"172.21.15.189:6829/850185001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:27.220308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:53.230783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:17.190642+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:52.603799+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:20.568671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:46.158363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:22.706933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:50.665173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:16.602410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:53.814698+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:22.435189+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:48.651949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.e","mappings":[{"from":4,"to":7}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.28","mappings":[{"from":3,"to":6},{"from":5,"to":8}]},{"pgid":"3.30","mappings":[{"from":1,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1266897222":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/2915689533":"2024-05-01T08:23:29.526664+0000","172.21.15.32:6801/4019351306":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/425836193":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/1134075722":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6801/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/21059253":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/3245550894":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/4019351306":"2024-05-01T08:23:29.526664+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-30T08:34:48.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.404+0000 7fd857e00640 1 -- 172.21.15.32:0/1571734817 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd83c0afae0 msgr2=0x7fd83c0b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:48.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.404+0000 7fd857e00640 1 --2- 172.21.15.32:0/1571734817 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd83c0afae0 0x7fd83c0b1ff0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fd840004770 tx=0x7fd8400314a0 comp rx=0 tx=0).stop 2024-04-30T08:34:48.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.404+0000 7fd857e00640 1 -- 172.21.15.32:0/1571734817 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd858075010 msgr2=0x7fd85819c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:48.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.404+0000 7fd857e00640 1 --2- 172.21.15.32:0/1571734817 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd858075010 0x7fd85819c330 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fd84800e9a0 tx=0x7fd84800ee60 comp rx=0 tx=0).stop 2024-04-30T08:34:48.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.404+0000 7fd857e00640 1 -- 172.21.15.32:0/1571734817 shutdown_connections 2024-04-30T08:34:48.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.404+0000 7fd857e00640 1 --2- 172.21.15.32:0/1571734817 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd83c0afae0 0x7fd83c0b1ff0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:48.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.404+0000 7fd857e00640 1 --2- 172.21.15.32:0/1571734817 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd85810b900 0x7fd8581a37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:48.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.404+0000 7fd857e00640 1 --2- 172.21.15.32:0/1571734817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd858075960 0x7fd85819c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:48.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.404+0000 7fd857e00640 1 --2- 172.21.15.32:0/1571734817 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd858075010 0x7fd85819c330 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:48.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.405+0000 7fd857e00640 1 -- 172.21.15.32:0/1571734817 >> 172.21.15.32:0/1571734817 conn(0x7fd8580feba0 msgr2=0x7fd8580ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:48.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.405+0000 7fd857e00640 1 -- 172.21.15.32:0/1571734817 shutdown_connections 2024-04-30T08:34:48.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.405+0000 7fd857e00640 1 -- 172.21.15.32:0/1571734817 wait complete. 2024-04-30T08:34:48.436 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-30T08:26:19.728331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-30T08:31:49.800542+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-30T08:31:51.211320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-30T08:31:55.716625+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-30T08:34:48.437 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-30T08:34:48.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:48 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3126364931' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-30T08:34:48.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.730+0000 7fd0dd000640 1 -- 172.21.15.32:0/3777293530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0d8104db0 msgr2=0x7fd0d81051c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:48.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.730+0000 7fd0dd000640 1 --2- 172.21.15.32:0/3777293530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0d8104db0 0x7fd0d81051c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fd0c0009900 tx=0x7fd0c002a3f0 comp rx=0 tx=0).stop 2024-04-30T08:34:48.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.730+0000 7fd0dd000640 1 -- 172.21.15.32:0/3777293530 shutdown_connections 2024-04-30T08:34:48.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.730+0000 7fd0dd000640 1 --2- 172.21.15.32:0/3777293530 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0d8109ab0 0x7fd0d810bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:48.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.730+0000 7fd0dd000640 1 --2- 172.21.15.32:0/3777293530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0d8104db0 0x7fd0d81051c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:48.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.730+0000 7fd0dd000640 1 --2- 172.21.15.32:0/3777293530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0d8103b70 0x7fd0d8104870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:48.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.730+0000 7fd0dd000640 1 -- 172.21.15.32:0/3777293530 >> 172.21.15.32:0/3777293530 conn(0x7fd0d80fed60 msgr2=0x7fd0d8101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:48.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.730+0000 7fd0dd000640 1 -- 172.21.15.32:0/3777293530 shutdown_connections 2024-04-30T08:34:48.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.730+0000 7fd0dd000640 1 -- 172.21.15.32:0/3777293530 wait complete. 2024-04-30T08:34:48.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.732+0000 7fd0dd000640 1 Processor -- start 2024-04-30T08:34:48.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.732+0000 7fd0dd000640 1 -- start start 2024-04-30T08:34:48.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.732+0000 7fd0dd000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0d8103b70 0x7fd0d819c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:48.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.733+0000 7fd0dd000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0d8104db0 0x7fd0d819ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:48.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.733+0000 7fd0dd000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0d8109ab0 0x7fd0d81a39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:48.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.733+0000 7fd0dd000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d810f6c0 con 0x7fd0d8104db0 2024-04-30T08:34:48.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.733+0000 7fd0dd000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d810f540 con 0x7fd0d8103b70 2024-04-30T08:34:48.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.733+0000 7fd0dd000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd0d810f840 con 0x7fd0d8109ab0 2024-04-30T08:34:48.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.733+0000 7fd0d5400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0d8104db0 0x7fd0d819ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:48.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.733+0000 7fd0d5400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0d8104db0 0x7fd0d819ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43216/0 (socket says 172.21.15.32:43216) 2024-04-30T08:34:48.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.733+0000 7fd0d5400640 1 -- 172.21.15.32:0/3685128820 learned_addr learned my addr 172.21.15.32:0/3685128820 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:48.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.733+0000 7fd0d5e00640 1 --2- 172.21.15.32:0/3685128820 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0d8103b70 0x7fd0d819c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:48.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.733+0000 7fd0d6800640 1 --2- 172.21.15.32:0/3685128820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0d8109ab0 0x7fd0d81a39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:48.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.733+0000 7fd0d5400640 1 -- 172.21.15.32:0/3685128820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0d8109ab0 msgr2=0x7fd0d81a39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:48.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.733+0000 7fd0d5400640 1 --2- 172.21.15.32:0/3685128820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0d8109ab0 0x7fd0d81a39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:48.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.733+0000 7fd0d5400640 1 -- 172.21.15.32:0/3685128820 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0d8103b70 msgr2=0x7fd0d819c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:48.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.733+0000 7fd0d5400640 1 --2- 172.21.15.32:0/3685128820 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0d8103b70 0x7fd0d819c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:48.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.733+0000 7fd0d5400640 1 -- 172.21.15.32:0/3685128820 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd0d81a4010 con 0x7fd0d8104db0 2024-04-30T08:34:48.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.733+0000 7fd0d5e00640 1 --2- 172.21.15.32:0/3685128820 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0d8103b70 0x7fd0d819c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:34:48.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.734+0000 7fd0d5400640 1 --2- 172.21.15.32:0/3685128820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0d8104db0 0x7fd0d819ca10 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd0c0031440 tx=0x7fd0c0003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:48.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.734+0000 7fd0cea00640 1 -- 172.21.15.32:0/3685128820 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c003f070 con 0x7fd0d8104db0 2024-04-30T08:34:48.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.734+0000 7fd0cea00640 1 -- 172.21.15.32:0/3685128820 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd0c0031c50 con 0x7fd0d8104db0 2024-04-30T08:34:48.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.734+0000 7fd0dd000640 1 -- 172.21.15.32:0/3685128820 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd0d81a42a0 con 0x7fd0d8104db0 2024-04-30T08:34:48.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.734+0000 7fd0cea00640 1 -- 172.21.15.32:0/3685128820 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0c0035030 con 0x7fd0d8104db0 2024-04-30T08:34:48.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.734+0000 7fd0dd000640 1 -- 172.21.15.32:0/3685128820 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0d81a47e0 con 0x7fd0d8104db0 2024-04-30T08:34:48.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.735+0000 7fd0dd000640 1 -- 172.21.15.32:0/3685128820 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd0d81a4b00 con 0x7fd0d8104db0 2024-04-30T08:34:48.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.738+0000 7fd0cea00640 1 -- 172.21.15.32:0/3685128820 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fd0c0042070 con 0x7fd0d8104db0 2024-04-30T08:34:48.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.740+0000 7fd0cea00640 1 --2- 172.21.15.32:0/3685128820 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd0bc0afb70 0x7fd0bc0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:48.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.740+0000 7fd0cea00640 1 -- 172.21.15.32:0/3685128820 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fd0c00ffed0 con 0x7fd0d8104db0 2024-04-30T08:34:48.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.740+0000 7fd0d5e00640 1 --2- 172.21.15.32:0/3685128820 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd0bc0afb70 0x7fd0bc0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:48.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.741+0000 7fd0d5e00640 1 --2- 172.21.15.32:0/3685128820 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd0bc0afb70 0x7fd0bc0b2080 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8104770 tx=0x7fd0c8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:48.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.742+0000 7fd0cea00640 1 -- 172.21.15.32:0/3685128820 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd0c00c8050 con 0x7fd0d8104db0 2024-04-30T08:34:48.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.951+0000 7fd0dd000640 1 -- 172.21.15.32:0/3685128820 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fd0d81a4cd0 con 0x7fd0d8104db0 2024-04-30T08:34:48.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.952+0000 7fd0cea00640 1 -- 172.21.15.32:0/3685128820 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) ==== 93+0+10 (secure 0 0 0) 0x7fd0c00cc350 con 0x7fd0d8104db0 2024-04-30T08:34:48.954 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-30T08:34:48.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.955+0000 7fd0dd000640 1 -- 172.21.15.32:0/3685128820 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd0bc0afb70 msgr2=0x7fd0bc0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:48.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.955+0000 7fd0dd000640 1 --2- 172.21.15.32:0/3685128820 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd0bc0afb70 0x7fd0bc0b2080 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fd0d8104770 tx=0x7fd0c8005f90 comp rx=0 tx=0).stop 2024-04-30T08:34:48.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.955+0000 7fd0dd000640 1 -- 172.21.15.32:0/3685128820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0d8104db0 msgr2=0x7fd0d819ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:48.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.955+0000 7fd0dd000640 1 --2- 172.21.15.32:0/3685128820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0d8104db0 0x7fd0d819ca10 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd0c0031440 tx=0x7fd0c0003c10 comp rx=0 tx=0).stop 2024-04-30T08:34:48.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.955+0000 7fd0dd000640 1 -- 172.21.15.32:0/3685128820 shutdown_connections 2024-04-30T08:34:48.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.955+0000 7fd0dd000640 1 --2- 172.21.15.32:0/3685128820 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd0bc0afb70 0x7fd0bc0b2080 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:48.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.955+0000 7fd0dd000640 1 --2- 172.21.15.32:0/3685128820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0d8109ab0 0x7fd0d81a39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:48.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.955+0000 7fd0dd000640 1 --2- 172.21.15.32:0/3685128820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd0d8104db0 0x7fd0d819ca10 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:48.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.955+0000 7fd0dd000640 1 --2- 172.21.15.32:0/3685128820 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd0d8103b70 0x7fd0d819c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:48.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.955+0000 7fd0dd000640 1 -- 172.21.15.32:0/3685128820 >> 172.21.15.32:0/3685128820 conn(0x7fd0d80fed60 msgr2=0x7fd0d80ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:48.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.956+0000 7fd0dd000640 1 -- 172.21.15.32:0/3685128820 shutdown_connections 2024-04-30T08:34:48.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:48.956+0000 7fd0dd000640 1 -- 172.21.15.32:0/3685128820 wait complete. 2024-04-30T08:34:48.985 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-30T08:34:49.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.278+0000 7efc93e00640 1 -- 172.21.15.32:0/2414418010 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc94103990 msgr2=0x7efc94104690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:49.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.278+0000 7efc93e00640 1 --2- 172.21.15.32:0/2414418010 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc94103990 0x7efc94104690 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7efc84009900 tx=0x7efc8402a410 comp rx=0 tx=0).stop 2024-04-30T08:34:49.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.278+0000 7efc93e00640 1 -- 172.21.15.32:0/2414418010 shutdown_connections 2024-04-30T08:34:49.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.278+0000 7efc93e00640 1 --2- 172.21.15.32:0/2414418010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc941098d0 0x7efc9410bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:49.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.278+0000 7efc93e00640 1 --2- 172.21.15.32:0/2414418010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc94104bd0 0x7efc94104fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:49.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.278+0000 7efc93e00640 1 --2- 172.21.15.32:0/2414418010 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc94103990 0x7efc94104690 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:49.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.278+0000 7efc93e00640 1 -- 172.21.15.32:0/2414418010 >> 172.21.15.32:0/2414418010 conn(0x7efc940feba0 msgr2=0x7efc94101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:49.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.278+0000 7efc93e00640 1 -- 172.21.15.32:0/2414418010 shutdown_connections 2024-04-30T08:34:49.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.278+0000 7efc93e00640 1 -- 172.21.15.32:0/2414418010 wait complete. 2024-04-30T08:34:49.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.280+0000 7efc93e00640 1 Processor -- start 2024-04-30T08:34:49.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.280+0000 7efc93e00640 1 -- start start 2024-04-30T08:34:49.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.280+0000 7efc93e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc94103990 0x7efc9419a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:49.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.281+0000 7efc93e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc94104bd0 0x7efc9419a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:49.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.281+0000 7efc93e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc941098d0 0x7efc941a15c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:49.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.281+0000 7efc93e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7efc9410f490 con 0x7efc94104bd0 2024-04-30T08:34:49.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.281+0000 7efc93e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7efc9410f310 con 0x7efc94103990 2024-04-30T08:34:49.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.281+0000 7efc93e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7efc9410f610 con 0x7efc941098d0 2024-04-30T08:34:49.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.281+0000 7efc8be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc94104bd0 0x7efc9419a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:49.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.281+0000 7efc8be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc94104bd0 0x7efc9419a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43228/0 (socket says 172.21.15.32:43228) 2024-04-30T08:34:49.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.281+0000 7efc8be00640 1 -- 172.21.15.32:0/2298459706 learned_addr learned my addr 172.21.15.32:0/2298459706 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:49.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.281+0000 7efc92a00640 1 --2- 172.21.15.32:0/2298459706 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc94103990 0x7efc9419a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:49.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.281+0000 7efc93400640 1 --2- 172.21.15.32:0/2298459706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc941098d0 0x7efc941a15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:49.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.281+0000 7efc8be00640 1 -- 172.21.15.32:0/2298459706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc941098d0 msgr2=0x7efc941a15c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:49.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.281+0000 7efc8be00640 1 --2- 172.21.15.32:0/2298459706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc941098d0 0x7efc941a15c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:49.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.281+0000 7efc8be00640 1 -- 172.21.15.32:0/2298459706 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc94103990 msgr2=0x7efc9419a160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:49.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.281+0000 7efc8be00640 1 --2- 172.21.15.32:0/2298459706 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc94103990 0x7efc9419a160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:49.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.281+0000 7efc8be00640 1 -- 172.21.15.32:0/2298459706 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc941a1c30 con 0x7efc94104bd0 2024-04-30T08:34:49.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.281+0000 7efc92a00640 1 --2- 172.21.15.32:0/2298459706 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc94103990 0x7efc9419a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:34:49.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.282+0000 7efc8be00640 1 --2- 172.21.15.32:0/2298459706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc94104bd0 0x7efc9419a6a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7efc7c00e9f0 tx=0x7efc7c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:49.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.282+0000 7efc8b400640 1 -- 172.21.15.32:0/2298459706 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc7c00cd80 con 0x7efc94104bd0 2024-04-30T08:34:49.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.282+0000 7efc93e00640 1 -- 172.21.15.32:0/2298459706 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc941a1f20 con 0x7efc94104bd0 2024-04-30T08:34:49.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.282+0000 7efc8b400640 1 -- 172.21.15.32:0/2298459706 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7efc7c0054d0 con 0x7efc94104bd0 2024-04-30T08:34:49.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.282+0000 7efc8b400640 1 -- 172.21.15.32:0/2298459706 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc7c010690 con 0x7efc94104bd0 2024-04-30T08:34:49.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.282+0000 7efc93e00640 1 -- 172.21.15.32:0/2298459706 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc941a2380 con 0x7efc94104bd0 2024-04-30T08:34:49.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.283+0000 7efc93e00640 1 -- 172.21.15.32:0/2298459706 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc941086e0 con 0x7efc94104bd0 2024-04-30T08:34:49.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.286+0000 7efc8b400640 1 -- 172.21.15.32:0/2298459706 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7efc7c004e30 con 0x7efc94104bd0 2024-04-30T08:34:49.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.288+0000 7efc8b400640 1 --2- 172.21.15.32:0/2298459706 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7efc780afae0 0x7efc780b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:49.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.288+0000 7efc8b400640 1 -- 172.21.15.32:0/2298459706 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7efc7c0df690 con 0x7efc94104bd0 2024-04-30T08:34:49.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.289+0000 7efc92a00640 1 --2- 172.21.15.32:0/2298459706 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7efc780afae0 0x7efc780b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:49.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.290+0000 7efc92a00640 1 --2- 172.21.15.32:0/2298459706 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7efc780afae0 0x7efc780b1ff0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7efc84005ab0 tx=0x7efc84031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:49.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.290+0000 7efc8b400640 1 -- 172.21.15.32:0/2298459706 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efc7c053ff0 con 0x7efc94104bd0 2024-04-30T08:34:49.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:49 smithi189 ceph-mon[33038]: pgmap v414: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-30T08:34:49.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:49 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1571734817' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:34:49.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:49 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3685128820' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:34:49.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:49 smithi032 ceph-mon[28331]: pgmap v414: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-30T08:34:49.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:49 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1571734817' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:34:49.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:49 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3685128820' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:34:49.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.501+0000 7efc93e00640 1 -- 172.21.15.32:0/2298459706 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7efc94063820 con 0x7efc94104bd0 2024-04-30T08:34:49.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:49 smithi071 ceph-mon[33385]: pgmap v414: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-30T08:34:49.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:49 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1571734817' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:34:49.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:49 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3685128820' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:34:49.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.502+0000 7efc8b400640 1 -- 172.21.15.32:0/2298459706 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) ==== 104+0+11 (secure 0 0 0) 0x7efc7c0a7e60 con 0x7efc94104bd0 2024-04-30T08:34:49.504 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-30T08:34:49.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.505+0000 7efc93e00640 1 -- 172.21.15.32:0/2298459706 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7efc780afae0 msgr2=0x7efc780b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:49.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.505+0000 7efc93e00640 1 --2- 172.21.15.32:0/2298459706 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7efc780afae0 0x7efc780b1ff0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7efc84005ab0 tx=0x7efc84031400 comp rx=0 tx=0).stop 2024-04-30T08:34:49.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.505+0000 7efc93e00640 1 -- 172.21.15.32:0/2298459706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc94104bd0 msgr2=0x7efc9419a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:49.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.505+0000 7efc93e00640 1 --2- 172.21.15.32:0/2298459706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc94104bd0 0x7efc9419a6a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7efc7c00e9f0 tx=0x7efc7c00edf0 comp rx=0 tx=0).stop 2024-04-30T08:34:49.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.505+0000 7efc93e00640 1 -- 172.21.15.32:0/2298459706 shutdown_connections 2024-04-30T08:34:49.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.505+0000 7efc93e00640 1 --2- 172.21.15.32:0/2298459706 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7efc780afae0 0x7efc780b1ff0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:49.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.505+0000 7efc93e00640 1 --2- 172.21.15.32:0/2298459706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efc941098d0 0x7efc941a15c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:49.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.505+0000 7efc93e00640 1 --2- 172.21.15.32:0/2298459706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7efc94104bd0 0x7efc9419a6a0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:49.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.505+0000 7efc93e00640 1 --2- 172.21.15.32:0/2298459706 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7efc94103990 0x7efc9419a160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:49.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.505+0000 7efc93e00640 1 -- 172.21.15.32:0/2298459706 >> 172.21.15.32:0/2298459706 conn(0x7efc940feba0 msgr2=0x7efc941058b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:49.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.506+0000 7efc93e00640 1 -- 172.21.15.32:0/2298459706 shutdown_connections 2024-04-30T08:34:49.508 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.506+0000 7efc93e00640 1 -- 172.21.15.32:0/2298459706 wait complete. 2024-04-30T08:34:49.534 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-30T08:34:49.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.818+0000 7f0a77200640 1 -- 172.21.15.32:0/186416690 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a78075010 msgr2=0x7f0a78075420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:49.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.818+0000 7f0a77200640 1 --2- 172.21.15.32:0/186416690 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a78075010 0x7f0a78075420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f0a68009860 tx=0x7f0a6802a340 comp rx=0 tx=0).stop 2024-04-30T08:34:49.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.818+0000 7f0a77200640 1 -- 172.21.15.32:0/186416690 shutdown_connections 2024-04-30T08:34:49.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.818+0000 7f0a77200640 1 --2- 172.21.15.32:0/186416690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a7810b900 0x7f0a7810dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:49.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.818+0000 7f0a77200640 1 --2- 172.21.15.32:0/186416690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a78075960 0x7f0a7810b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:49.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.818+0000 7f0a77200640 1 --2- 172.21.15.32:0/186416690 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a78075010 0x7f0a78075420 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:49.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.818+0000 7f0a77200640 1 -- 172.21.15.32:0/186416690 >> 172.21.15.32:0/186416690 conn(0x7f0a780feba0 msgr2=0x7f0a78101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:49.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.818+0000 7f0a77200640 1 -- 172.21.15.32:0/186416690 shutdown_connections 2024-04-30T08:34:49.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.818+0000 7f0a77200640 1 -- 172.21.15.32:0/186416690 wait complete. 2024-04-30T08:34:49.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.819+0000 7f0a77200640 1 Processor -- start 2024-04-30T08:34:49.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.820+0000 7f0a77200640 1 -- start start 2024-04-30T08:34:49.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.820+0000 7f0a77200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a78075010 0x7f0a7819c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:49.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.820+0000 7f0a77200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a78075960 0x7f0a7819c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:49.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.820+0000 7f0a77200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a7810b900 0x7f0a781a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:49.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.820+0000 7f0a77200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0a781116a0 con 0x7f0a78075960 2024-04-30T08:34:49.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.820+0000 7f0a77200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f0a78111520 con 0x7f0a78075010 2024-04-30T08:34:49.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.820+0000 7f0a77200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0a78111820 con 0x7f0a7810b900 2024-04-30T08:34:49.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.820+0000 7f0a75e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a78075010 0x7f0a7819c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:49.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.820+0000 7f0a75400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a78075960 0x7f0a7819c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:49.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.820+0000 7f0a76800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a7810b900 0x7f0a781a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:49.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.820+0000 7f0a75e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a78075010 0x7f0a7819c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:46192/0 (socket says 172.21.15.32:46192) 2024-04-30T08:34:49.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.820+0000 7f0a75e00640 1 -- 172.21.15.32:0/761017329 learned_addr learned my addr 172.21.15.32:0/761017329 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:49.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.821+0000 7f0a76800640 1 -- 172.21.15.32:0/761017329 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a78075010 msgr2=0x7f0a7819c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:49.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.821+0000 7f0a76800640 1 --2- 172.21.15.32:0/761017329 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a78075010 0x7f0a7819c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:49.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.821+0000 7f0a76800640 1 -- 172.21.15.32:0/761017329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a78075960 msgr2=0x7f0a7819c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:49.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.821+0000 7f0a76800640 1 --2- 172.21.15.32:0/761017329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a78075960 0x7f0a7819c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:49.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.821+0000 7f0a76800640 1 -- 172.21.15.32:0/761017329 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a781a3e40 con 0x7f0a7810b900 2024-04-30T08:34:49.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.821+0000 7f0a76800640 1 --2- 172.21.15.32:0/761017329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a7810b900 0x7f0a781a37d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0a70005fd0 tx=0x7f0a7000c740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:49.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.821+0000 7f0a6ea00640 1 -- 172.21.15.32:0/761017329 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a70004b20 con 0x7f0a7810b900 2024-04-30T08:34:49.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.821+0000 7f0a6ea00640 1 -- 172.21.15.32:0/761017329 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a70003e20 con 0x7f0a7810b900 2024-04-30T08:34:49.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.821+0000 7f0a77200640 1 -- 172.21.15.32:0/761017329 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a781a4130 con 0x7f0a7810b900 2024-04-30T08:34:49.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.822+0000 7f0a6ea00640 1 -- 172.21.15.32:0/761017329 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a7000b570 con 0x7f0a7810b900 2024-04-30T08:34:49.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.822+0000 7f0a77200640 1 -- 172.21.15.32:0/761017329 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a781a4680 con 0x7f0a7810b900 2024-04-30T08:34:49.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.823+0000 7f0a77200640 1 -- 172.21.15.32:0/761017329 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a7810aac0 con 0x7f0a7810b900 2024-04-30T08:34:49.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.825+0000 7f0a6ea00640 1 -- 172.21.15.32:0/761017329 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f0a7001d020 con 0x7f0a7810b900 2024-04-30T08:34:49.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.827+0000 7f0a6ea00640 1 --2- 172.21.15.32:0/761017329 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f0a5c0afb60 0x7f0a5c0b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:49.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.827+0000 7f0a6ea00640 1 -- 172.21.15.32:0/761017329 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f0a700df840 con 0x7f0a7810b900 2024-04-30T08:34:49.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.827+0000 7f0a75e00640 1 --2- 172.21.15.32:0/761017329 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f0a5c0afb60 0x7f0a5c0b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:49.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.828+0000 7f0a75e00640 1 --2- 172.21.15.32:0/761017329 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f0a5c0afb60 0x7f0a5c0b2070 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0a680057c0 tx=0x7f0a68030450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:49.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:49.829+0000 7f0a6ea00640 1 -- 172.21.15.32:0/761017329 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0a70024030 con 0x7f0a7810b900 2024-04-30T08:34:50.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.039+0000 7f0a77200640 1 -- 172.21.15.32:0/761017329 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0a78063820 con 0x7f0a7810b900 2024-04-30T08:34:50.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.040+0000 7f0a6ea00640 1 -- 172.21.15.32:0/761017329 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) ==== 100+0+11 (secure 0 0 0) 0x7f0a70004cc0 con 0x7f0a7810b900 2024-04-30T08:34:50.042 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-04-30T08:34:50.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.043+0000 7f0a77200640 1 -- 172.21.15.32:0/761017329 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f0a5c0afb60 msgr2=0x7f0a5c0b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:50.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.043+0000 7f0a77200640 1 --2- 172.21.15.32:0/761017329 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f0a5c0afb60 0x7f0a5c0b2070 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f0a680057c0 tx=0x7f0a68030450 comp rx=0 tx=0).stop 2024-04-30T08:34:50.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.043+0000 7f0a77200640 1 -- 172.21.15.32:0/761017329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a7810b900 msgr2=0x7f0a781a37d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:50.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.043+0000 7f0a77200640 1 --2- 172.21.15.32:0/761017329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a7810b900 0x7f0a781a37d0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f0a70005fd0 tx=0x7f0a7000c740 comp rx=0 tx=0).stop 2024-04-30T08:34:50.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.043+0000 7f0a77200640 1 -- 172.21.15.32:0/761017329 shutdown_connections 2024-04-30T08:34:50.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.043+0000 7f0a77200640 1 --2- 172.21.15.32:0/761017329 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f0a5c0afb60 0x7f0a5c0b2070 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:50.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.043+0000 7f0a77200640 1 --2- 172.21.15.32:0/761017329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a7810b900 0x7f0a781a37d0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:50.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.043+0000 7f0a77200640 1 --2- 172.21.15.32:0/761017329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a78075960 0x7f0a7819c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:50.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.043+0000 7f0a77200640 1 --2- 172.21.15.32:0/761017329 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a78075010 0x7f0a7819c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:50.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.043+0000 7f0a77200640 1 -- 172.21.15.32:0/761017329 >> 172.21.15.32:0/761017329 conn(0x7f0a780feba0 msgr2=0x7f0a780ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:50.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.044+0000 7f0a77200640 1 -- 172.21.15.32:0/761017329 shutdown_connections 2024-04-30T08:34:50.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.044+0000 7f0a77200640 1 -- 172.21.15.32:0/761017329 wait complete. 2024-04-30T08:34:50.075 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-30T08:34:50.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:50 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2298459706' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-30T08:34:50.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.361+0000 7fa1cca00640 1 -- 172.21.15.32:0/2698711099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1c810b200 msgr2=0x7fa1c810b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:50.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.361+0000 7fa1cca00640 1 --2- 172.21.15.32:0/2698711099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1c810b200 0x7fa1c810b610 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0009900 tx=0x7fa1b002a410 comp rx=0 tx=0).stop 2024-04-30T08:34:50.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.361+0000 7fa1cca00640 1 -- 172.21.15.32:0/2698711099 shutdown_connections 2024-04-30T08:34:50.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.361+0000 7fa1cca00640 1 --2- 172.21.15.32:0/2698711099 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1c810ff00 0x7fa1c8112320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:50.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.361+0000 7fa1cca00640 1 --2- 172.21.15.32:0/2698711099 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1c810b200 0x7fa1c810b610 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:50.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.361+0000 7fa1cca00640 1 --2- 172.21.15.32:0/2698711099 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1c8109fc0 0x7fa1c810acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:50.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.361+0000 7fa1cca00640 1 -- 172.21.15.32:0/2698711099 >> 172.21.15.32:0/2698711099 conn(0x7fa1c8078bb0 msgr2=0x7fa1c8107a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:50.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.361+0000 7fa1cca00640 1 -- 172.21.15.32:0/2698711099 shutdown_connections 2024-04-30T08:34:50.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.361+0000 7fa1cca00640 1 -- 172.21.15.32:0/2698711099 wait complete. 2024-04-30T08:34:50.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.363+0000 7fa1cca00640 1 Processor -- start 2024-04-30T08:34:50.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.363+0000 7fa1cca00640 1 -- start start 2024-04-30T08:34:50.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.363+0000 7fa1cca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1c8109fc0 0x7fa1c81a06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:50.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.363+0000 7fa1cca00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1c810b200 0x7fa1c81a0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:50.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.363+0000 7fa1cca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1c810ff00 0x7fa1c81a7bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:50.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.363+0000 7fa1cca00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c8115ae0 con 0x7fa1c8109fc0 2024-04-30T08:34:50.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.363+0000 7fa1cca00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c8115960 con 0x7fa1c810b200 2024-04-30T08:34:50.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.363+0000 7fa1cca00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c8115c60 con 0x7fa1c810ff00 2024-04-30T08:34:50.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.364+0000 7fa1c6800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1c8109fc0 0x7fa1c81a06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:50.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.364+0000 7fa1c6800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1c8109fc0 0x7fa1c81a06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43278/0 (socket says 172.21.15.32:43278) 2024-04-30T08:34:50.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.364+0000 7fa1c6800640 1 -- 172.21.15.32:0/125452675 learned_addr learned my addr 172.21.15.32:0/125452675 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:50.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.364+0000 7fa1bfe00640 1 --2- 172.21.15.32:0/125452675 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1c810b200 0x7fa1c81a0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:50.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.364+0000 7fa1c7200640 1 --2- 172.21.15.32:0/125452675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1c810ff00 0x7fa1c81a7bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:50.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.364+0000 7fa1c6800640 1 -- 172.21.15.32:0/125452675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1c810ff00 msgr2=0x7fa1c81a7bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:50.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.364+0000 7fa1c6800640 1 --2- 172.21.15.32:0/125452675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1c810ff00 0x7fa1c81a7bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:50.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.364+0000 7fa1c6800640 1 -- 172.21.15.32:0/125452675 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1c810b200 msgr2=0x7fa1c81a0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:50.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.364+0000 7fa1c6800640 1 --2- 172.21.15.32:0/125452675 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1c810b200 0x7fa1c81a0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:50.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.364+0000 7fa1c6800640 1 -- 172.21.15.32:0/125452675 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1c81a8230 con 0x7fa1c8109fc0 2024-04-30T08:34:50.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.364+0000 7fa1c7200640 1 --2- 172.21.15.32:0/125452675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1c810ff00 0x7fa1c81a7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:34:50.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.365+0000 7fa1c6800640 1 --2- 172.21.15.32:0/125452675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1c8109fc0 0x7fa1c81a06f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa1b800e980 tx=0x7fa1b800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:50.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.365+0000 7fa1bf400640 1 -- 172.21.15.32:0/125452675 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b800cd80 con 0x7fa1c8109fc0 2024-04-30T08:34:50.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.365+0000 7fa1bf400640 1 -- 172.21.15.32:0/125452675 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1b80108e0 con 0x7fa1c8109fc0 2024-04-30T08:34:50.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.365+0000 7fa1cca00640 1 -- 172.21.15.32:0/125452675 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1c81a8520 con 0x7fa1c8109fc0 2024-04-30T08:34:50.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.365+0000 7fa1bf400640 1 -- 172.21.15.32:0/125452675 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1b8018990 con 0x7fa1c8109fc0 2024-04-30T08:34:50.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.365+0000 7fa1cca00640 1 -- 172.21.15.32:0/125452675 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1c81a8a70 con 0x7fa1c8109fc0 2024-04-30T08:34:50.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.366+0000 7fa1cca00640 1 -- 172.21.15.32:0/125452675 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1c810ed10 con 0x7fa1c8109fc0 2024-04-30T08:34:50.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.369+0000 7fa1bf400640 1 -- 172.21.15.32:0/125452675 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fa1b801f020 con 0x7fa1c8109fc0 2024-04-30T08:34:50.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.371+0000 7fa1bf400640 1 --2- 172.21.15.32:0/125452675 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa1ac0afae0 0x7fa1ac0b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:50.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.371+0000 7fa1bf400640 1 -- 172.21.15.32:0/125452675 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fa1b80df6a0 con 0x7fa1c8109fc0 2024-04-30T08:34:50.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.371+0000 7fa1bfe00640 1 --2- 172.21.15.32:0/125452675 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa1ac0afae0 0x7fa1ac0b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:50.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.372+0000 7fa1bfe00640 1 --2- 172.21.15.32:0/125452675 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa1ac0afae0 0x7fa1ac0b1ff0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0009230 tx=0x7fa1b0031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:50.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.374+0000 7fa1bf400640 1 -- 172.21.15.32:0/125452675 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa1b8026020 con 0x7fa1c8109fc0 2024-04-30T08:34:50.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:50 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2298459706' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-30T08:34:50.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:50 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2298459706' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-30T08:34:50.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.581+0000 7fa1cca00640 1 -- 172.21.15.32:0/125452675 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fa1c8063820 con 0x7fa1c8109fc0 2024-04-30T08:34:50.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.582+0000 7fa1bf400640 1 -- 172.21.15.32:0/125452675 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v85) ==== 103+0+11 (secure 0 0 0) 0x7fa1b80a7e70 con 0x7fa1c8109fc0 2024-04-30T08:34:50.584 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-30T08:34:50.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.584+0000 7fa1cca00640 1 -- 172.21.15.32:0/125452675 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa1ac0afae0 msgr2=0x7fa1ac0b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:50.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.585+0000 7fa1cca00640 1 --2- 172.21.15.32:0/125452675 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa1ac0afae0 0x7fa1ac0b1ff0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fa1b0009230 tx=0x7fa1b0031400 comp rx=0 tx=0).stop 2024-04-30T08:34:50.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.585+0000 7fa1cca00640 1 -- 172.21.15.32:0/125452675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1c8109fc0 msgr2=0x7fa1c81a06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:50.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.585+0000 7fa1cca00640 1 --2- 172.21.15.32:0/125452675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1c8109fc0 0x7fa1c81a06f0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fa1b800e980 tx=0x7fa1b800ee40 comp rx=0 tx=0).stop 2024-04-30T08:34:50.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.585+0000 7fa1cca00640 1 -- 172.21.15.32:0/125452675 shutdown_connections 2024-04-30T08:34:50.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.585+0000 7fa1cca00640 1 --2- 172.21.15.32:0/125452675 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa1ac0afae0 0x7fa1ac0b1ff0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:50.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.585+0000 7fa1cca00640 1 --2- 172.21.15.32:0/125452675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1c810ff00 0x7fa1c81a7bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:50.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.585+0000 7fa1cca00640 1 --2- 172.21.15.32:0/125452675 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1c810b200 0x7fa1c81a0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:50.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.585+0000 7fa1cca00640 1 --2- 172.21.15.32:0/125452675 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa1c8109fc0 0x7fa1c81a06f0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:50.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.585+0000 7fa1cca00640 1 -- 172.21.15.32:0/125452675 >> 172.21.15.32:0/125452675 conn(0x7fa1c8078bb0 msgr2=0x7fa1c810bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:50.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.585+0000 7fa1cca00640 1 -- 172.21.15.32:0/125452675 shutdown_connections 2024-04-30T08:34:50.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.585+0000 7fa1cca00640 1 -- 172.21.15.32:0/125452675 wait complete. 2024-04-30T08:34:50.615 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-30T08:34:50.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.905+0000 7f6c27800640 1 -- 172.21.15.32:0/3874935457 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c20110940 msgr2=0x7f6c20112d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:50.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.905+0000 7f6c27800640 1 --2- 172.21.15.32:0/3874935457 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c20110940 0x7f6c20112d60 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f6c18009be0 tx=0x7f6c1802a6d0 comp rx=0 tx=0).stop 2024-04-30T08:34:50.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.905+0000 7f6c27800640 1 -- 172.21.15.32:0/3874935457 shutdown_connections 2024-04-30T08:34:50.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.905+0000 7f6c27800640 1 --2- 172.21.15.32:0/3874935457 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c20110940 0x7f6c20112d60 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:50.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.905+0000 7f6c27800640 1 --2- 172.21.15.32:0/3874935457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c2010bbb0 0x7f6c2010bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:50.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.905+0000 7f6c27800640 1 --2- 172.21.15.32:0/3874935457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c2010aca0 0x7f6c2010b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:50.907 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.905+0000 7f6c27800640 1 -- 172.21.15.32:0/3874935457 >> 172.21.15.32:0/3874935457 conn(0x7f6c200709a0 msgr2=0x7f6c20072e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:50.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.905+0000 7f6c27800640 1 -- 172.21.15.32:0/3874935457 shutdown_connections 2024-04-30T08:34:50.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.905+0000 7f6c27800640 1 -- 172.21.15.32:0/3874935457 wait complete. 2024-04-30T08:34:50.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.906+0000 7f6c27800640 1 Processor -- start 2024-04-30T08:34:50.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.907+0000 7f6c27800640 1 -- start start 2024-04-30T08:34:50.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.907+0000 7f6c27800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c2010aca0 0x7f6c201a4c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:50.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.907+0000 7f6c27800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c2010bbb0 0x7f6c201a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:50.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.907+0000 7f6c27800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c20110940 0x7f6c201ac0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:50.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.907+0000 7f6c27800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6c20117eb0 con 0x7f6c2010bbb0 2024-04-30T08:34:50.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.907+0000 7f6c27800640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6c20117d30 con 0x7f6c2010aca0 2024-04-30T08:34:50.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.907+0000 7f6c27800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6c20118030 con 0x7f6c20110940 2024-04-30T08:34:50.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.907+0000 7f6c24e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c2010aca0 0x7f6c201a4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:50.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.907+0000 7f6c24e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c2010aca0 0x7f6c201a4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:46232/0 (socket says 172.21.15.32:46232) 2024-04-30T08:34:50.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.907+0000 7f6c1fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c2010bbb0 0x7f6c201a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:50.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.907+0000 7f6c24e00640 1 -- 172.21.15.32:0/964267485 learned_addr learned my addr 172.21.15.32:0/964267485 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:50.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.907+0000 7f6c25800640 1 --2- 172.21.15.32:0/964267485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c20110940 0x7f6c201ac0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:50.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.908+0000 7f6c24e00640 1 -- 172.21.15.32:0/964267485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c20110940 msgr2=0x7f6c201ac0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:50.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.908+0000 7f6c24e00640 1 --2- 172.21.15.32:0/964267485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c20110940 0x7f6c201ac0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:50.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.908+0000 7f6c24e00640 1 -- 172.21.15.32:0/964267485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c2010bbb0 msgr2=0x7f6c201a5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:50.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.908+0000 7f6c24e00640 1 --2- 172.21.15.32:0/964267485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c2010bbb0 0x7f6c201a5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:50.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.908+0000 7f6c24e00640 1 -- 172.21.15.32:0/964267485 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c201ac760 con 0x7f6c2010aca0 2024-04-30T08:34:50.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.908+0000 7f6c24e00640 1 --2- 172.21.15.32:0/964267485 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c2010aca0 0x7f6c201a4c20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6c140058c0 tx=0x7f6c1400ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:50.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.908+0000 7f6c1d600640 1 -- 172.21.15.32:0/964267485 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c1400cd30 con 0x7f6c2010aca0 2024-04-30T08:34:50.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.908+0000 7f6c1d600640 1 -- 172.21.15.32:0/964267485 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6c140035f0 con 0x7f6c2010aca0 2024-04-30T08:34:50.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.908+0000 7f6c1d600640 1 -- 172.21.15.32:0/964267485 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c140044e0 con 0x7f6c2010aca0 2024-04-30T08:34:50.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.908+0000 7f6c27800640 1 -- 172.21.15.32:0/964267485 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c201aca50 con 0x7f6c2010aca0 2024-04-30T08:34:50.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.909+0000 7f6c27800640 1 -- 172.21.15.32:0/964267485 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c201acfa0 con 0x7f6c2010aca0 2024-04-30T08:34:50.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.910+0000 7f6c27800640 1 -- 172.21.15.32:0/964267485 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c201ad1d0 con 0x7f6c2010aca0 2024-04-30T08:34:50.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.913+0000 7f6c1d600640 1 -- 172.21.15.32:0/964267485 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f6c14003aa0 con 0x7f6c2010aca0 2024-04-30T08:34:50.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.914+0000 7f6c1d600640 1 --2- 172.21.15.32:0/964267485 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f6c080afaf0 0x7f6c080b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:50.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.915+0000 7f6c1d600640 1 -- 172.21.15.32:0/964267485 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f6c140df330 con 0x7f6c2010aca0 2024-04-30T08:34:50.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.915+0000 7f6c1fe00640 1 --2- 172.21.15.32:0/964267485 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f6c080afaf0 0x7f6c080b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:50.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.916+0000 7f6c1d600640 1 -- 172.21.15.32:0/964267485 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6c140a7b00 con 0x7f6c2010aca0 2024-04-30T08:34:50.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:50.916+0000 7f6c1fe00640 1 --2- 172.21.15.32:0/964267485 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f6c080afaf0 0x7f6c080b2000 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6c201a5cc0 tx=0x7f6c0c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:51.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.166+0000 7f6c27800640 1 -- 172.21.15.32:0/964267485 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6c201ad3a0 con 0x7f6c2010aca0 2024-04-30T08:34:51.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.168+0000 7f6c1d600640 1 -- 172.21.15.32:0/964267485 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7f6c140a7400 con 0x7f6c2010aca0 2024-04-30T08:34:51.170 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:34:51.170 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2235842676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2235842676},{"type":"v1","addr":"172.21.15.32:6835","nonce":2235842676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/3577009355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":3577009355},{"type":"v1","addr":"172.21.15.32:6841","nonce":3577009355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-30T08:31:52.236668+0000","modified":"2024-04-30T08:32:30.541714+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24541,"mds_3":24533,"mds_4":24517},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/463203038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":463203038},{"type":"v1","addr":"172.21.15.189:6833","nonce":463203038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6833/3635143591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3635143591},{"type":"v1","addr":"172.21.15.71:6833","nonce":3635143591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6837/2569449994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2569449994},{"type":"v1","addr":"172.21.15.32:6837","nonce":2569449994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/278686565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":278686565},{"type":"v1","addr":"172.21.15.189:6835","nonce":278686565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6835/1558303056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1558303056},{"type":"v1","addr":"172.21.15.71:6835","nonce":1558303056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.189:6837/686520873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":686520873},{"type":"v1","addr":"172.21.15.189:6837","nonce":686520873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.71:6837/2324039932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2324039932},{"type":"v1","addr":"172.21.15.71:6837","nonce":2324039932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6839/352970107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":352970107},{"type":"v1","addr":"172.21.15.189:6839","nonce":352970107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.32:6839/4160386524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4160386524},{"type":"v1","addr":"172.21.15.32:6839","nonce":4160386524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.71:6839/3517409993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3517409993},{"type":"v1","addr":"172.21.15.71:6839","nonce":3517409993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24590,24541,24560,24533,24517]},"id":1}]} 2024-04-30T08:34:51.170 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 27 2024-04-30T08:34:51.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.170+0000 7f6c27800640 1 -- 172.21.15.32:0/964267485 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f6c080afaf0 msgr2=0x7f6c080b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:51.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.170+0000 7f6c27800640 1 --2- 172.21.15.32:0/964267485 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f6c080afaf0 0x7f6c080b2000 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f6c201a5cc0 tx=0x7f6c0c005f90 comp rx=0 tx=0).stop 2024-04-30T08:34:51.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.170+0000 7f6c27800640 1 -- 172.21.15.32:0/964267485 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c2010aca0 msgr2=0x7f6c201a4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:51.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.170+0000 7f6c27800640 1 --2- 172.21.15.32:0/964267485 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c2010aca0 0x7f6c201a4c20 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6c140058c0 tx=0x7f6c1400ee50 comp rx=0 tx=0).stop 2024-04-30T08:34:51.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.171+0000 7f6c27800640 1 -- 172.21.15.32:0/964267485 shutdown_connections 2024-04-30T08:34:51.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.171+0000 7f6c27800640 1 --2- 172.21.15.32:0/964267485 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f6c080afaf0 0x7f6c080b2000 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:51.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.171+0000 7f6c27800640 1 --2- 172.21.15.32:0/964267485 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c20110940 0x7f6c201ac0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:51.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.171+0000 7f6c27800640 1 --2- 172.21.15.32:0/964267485 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6c2010bbb0 0x7f6c201a5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:51.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.171+0000 7f6c27800640 1 --2- 172.21.15.32:0/964267485 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6c2010aca0 0x7f6c201a4c20 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:51.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.171+0000 7f6c27800640 1 -- 172.21.15.32:0/964267485 >> 172.21.15.32:0/964267485 conn(0x7f6c200709a0 msgr2=0x7f6c20070f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:51.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.171+0000 7f6c27800640 1 -- 172.21.15.32:0/964267485 shutdown_connections 2024-04-30T08:34:51.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.171+0000 7f6c27800640 1 -- 172.21.15.32:0/964267485 wait complete. 2024-04-30T08:34:51.200 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-30T08:34:51.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:51 smithi189 ceph-mon[33038]: pgmap v415: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-30T08:34:51.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:51 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/761017329' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-30T08:34:51.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:51 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/125452675' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-30T08:34:51.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:51 smithi032 ceph-mon[28331]: pgmap v415: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-30T08:34:51.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:51 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/761017329' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-30T08:34:51.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:51 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/125452675' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-30T08:34:51.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.485+0000 7f9800c00640 1 -- 172.21.15.32:0/1891833558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97fc109fc0 msgr2=0x7f97fc10acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:51.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.485+0000 7f9800c00640 1 --2- 172.21.15.32:0/1891833558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97fc109fc0 0x7f97fc10acc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f97ec009900 tx=0x7f97ec02a410 comp rx=0 tx=0).stop 2024-04-30T08:34:51.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.485+0000 7f9800c00640 1 -- 172.21.15.32:0/1891833558 shutdown_connections 2024-04-30T08:34:51.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.485+0000 7f9800c00640 1 --2- 172.21.15.32:0/1891833558 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97fc10ff00 0x7f97fc112320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:51.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.485+0000 7f9800c00640 1 --2- 172.21.15.32:0/1891833558 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97fc10b200 0x7f97fc10b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:51.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.485+0000 7f9800c00640 1 --2- 172.21.15.32:0/1891833558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97fc109fc0 0x7f97fc10acc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:51.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.485+0000 7f9800c00640 1 -- 172.21.15.32:0/1891833558 >> 172.21.15.32:0/1891833558 conn(0x7f97fc078bb0 msgr2=0x7f97fc107a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:51.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.485+0000 7f9800c00640 1 -- 172.21.15.32:0/1891833558 shutdown_connections 2024-04-30T08:34:51.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.485+0000 7f9800c00640 1 -- 172.21.15.32:0/1891833558 wait complete. 2024-04-30T08:34:51.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.486+0000 7f9800c00640 1 Processor -- start 2024-04-30T08:34:51.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.486+0000 7f9800c00640 1 -- start start 2024-04-30T08:34:51.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.487+0000 7f9800c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97fc109fc0 0x7f97fc1a0750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:51.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.487+0000 7f9800c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97fc10b200 0x7f97fc1a0c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:51.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.487+0000 7f9800c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97fc10ff00 0x7f97fc1a7c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:51.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.487+0000 7f9800c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f97fc115ae0 con 0x7f97fc10b200 2024-04-30T08:34:51.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.487+0000 7f9800c00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f97fc115960 con 0x7f97fc109fc0 2024-04-30T08:34:51.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.487+0000 7f9800c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f97fc115c60 con 0x7f97fc10ff00 2024-04-30T08:34:51.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.487+0000 7f97f3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97fc10b200 0x7f97fc1a0c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:51.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.487+0000 7f97f3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97fc10b200 0x7f97fc1a0c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43318/0 (socket says 172.21.15.32:43318) 2024-04-30T08:34:51.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.487+0000 7f97f3e00640 1 -- 172.21.15.32:0/3432396457 learned_addr learned my addr 172.21.15.32:0/3432396457 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:51.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.487+0000 7f97fb400640 1 --2- 172.21.15.32:0/3432396457 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97fc109fc0 0x7f97fc1a0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:51.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.487+0000 7f97fbe00640 1 --2- 172.21.15.32:0/3432396457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97fc10ff00 0x7f97fc1a7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:51.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.488+0000 7f97f3e00640 1 -- 172.21.15.32:0/3432396457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97fc10ff00 msgr2=0x7f97fc1a7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:51.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.488+0000 7f97f3e00640 1 --2- 172.21.15.32:0/3432396457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97fc10ff00 0x7f97fc1a7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:51.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.488+0000 7f97f3e00640 1 -- 172.21.15.32:0/3432396457 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97fc109fc0 msgr2=0x7f97fc1a0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:51.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.488+0000 7f97f3e00640 1 --2- 172.21.15.32:0/3432396457 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97fc109fc0 0x7f97fc1a0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:51.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.488+0000 7f97f3e00640 1 -- 172.21.15.32:0/3432396457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97fc1a8290 con 0x7f97fc10b200 2024-04-30T08:34:51.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.488+0000 7f97f3e00640 1 --2- 172.21.15.32:0/3432396457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97fc10b200 0x7f97fc1a0c90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f97e400d770 tx=0x7f97e400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:51.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.488+0000 7f97f8c00640 1 -- 172.21.15.32:0/3432396457 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e4017070 con 0x7f97fc10b200 2024-04-30T08:34:51.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.488+0000 7f9800c00640 1 -- 172.21.15.32:0/3432396457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97fc1a8580 con 0x7f97fc10b200 2024-04-30T08:34:51.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.488+0000 7f97f8c00640 1 -- 172.21.15.32:0/3432396457 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f97e40054d0 con 0x7f97fc10b200 2024-04-30T08:34:51.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.488+0000 7f97f8c00640 1 -- 172.21.15.32:0/3432396457 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97e4004c30 con 0x7f97fc10b200 2024-04-30T08:34:51.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.488+0000 7f9800c00640 1 -- 172.21.15.32:0/3432396457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97fc1a8ad0 con 0x7f97fc10b200 2024-04-30T08:34:51.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.490+0000 7f9800c00640 1 -- 172.21.15.32:0/3432396457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f97fc10ed10 con 0x7f97fc10b200 2024-04-30T08:34:51.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.492+0000 7f97f8c00640 1 -- 172.21.15.32:0/3432396457 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f97e4004dd0 con 0x7f97fc10b200 2024-04-30T08:34:51.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.494+0000 7f97f8c00640 1 --2- 172.21.15.32:0/3432396457 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f97e00afae0 0x7f97e00b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:51.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.494+0000 7f97f8c00640 1 -- 172.21.15.32:0/3432396457 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f97e40dee60 con 0x7f97fc10b200 2024-04-30T08:34:51.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.494+0000 7f97fb400640 1 --2- 172.21.15.32:0/3432396457 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f97e00afae0 0x7f97e00b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:51.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.495+0000 7f97fb400640 1 --2- 172.21.15.32:0/3432396457 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f97e00afae0 0x7f97e00b1ff0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f97ec005ab0 tx=0x7f97ec031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:51.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.496+0000 7f97f8c00640 1 -- 172.21.15.32:0/3432396457 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f97e40a6fe0 con 0x7f97fc10b200 2024-04-30T08:34:51.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:51 smithi071 ceph-mon[33385]: pgmap v415: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-30T08:34:51.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:51 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/761017329' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-30T08:34:51.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:51 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/125452675' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-30T08:34:51.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.702+0000 7f9800c00640 1 -- 172.21.15.32:0/3432396457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f97fc063820 con 0x7f97fc10b200 2024-04-30T08:34:51.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.707+0000 7f97f8c00640 1 -- 172.21.15.32:0/3432396457 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+24730 (secure 0 0 0) 0x7f97e40ab2e0 con 0x7f97fc10b200 2024-04-30T08:34:51.709 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:34:51.709 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":85,"fsid":"98c0a670-06ca-11ef-bc93-c7b262605968","created":"2024-04-30T08:22:11.122875+0000","modified":"2024-04-30T08:33:31.872704+0000","last_up_change":"2024-04-30T08:30:50.350439+0000","last_in_change":"2024-04-30T08:30:30.210620+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-30T08:26:19.728331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-30T08:31:49.800542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-30T08:31:51.211320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-30T08:31:55.716625+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a7e060c8-1cbb-418f-9a03-06693f6b3f52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6803","nonce":2624700402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6805","nonce":2624700402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6809","nonce":2624700402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6807","nonce":2624700402}]},"public_addr":"172.21.15.32:6803/2624700402","cluster_addr":"172.21.15.32:6805/2624700402","heartbeat_back_addr":"172.21.15.32:6809/2624700402","heartbeat_front_addr":"172.21.15.32:6807/2624700402","state":["exists","up"]},{"osd":1,"uuid":"72ef2fb1-e5ca-48ef-aa3a-4796a64b922f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6801","nonce":444649711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6803","nonce":444649711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6807","nonce":444649711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6805","nonce":444649711}]},"public_addr":"172.21.15.71:6801/444649711","cluster_addr":"172.21.15.71:6803/444649711","heartbeat_back_addr":"172.21.15.71:6807/444649711","heartbeat_front_addr":"172.21.15.71:6805/444649711","state":["exists","up"]},{"osd":2,"uuid":"704e5e8a-935e-47c7-ae28-27119876ac09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6801","nonce":3255528671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6803","nonce":3255528671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6807","nonce":3255528671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6805","nonce":3255528671}]},"public_addr":"172.21.15.189:6801/3255528671","cluster_addr":"172.21.15.189:6803/3255528671","heartbeat_back_addr":"172.21.15.189:6807/3255528671","heartbeat_front_addr":"172.21.15.189:6805/3255528671","state":["exists","up"]},{"osd":3,"uuid":"575004f7-6559-4a62-863e-e33078b0eebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6811","nonce":1384702112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6813","nonce":1384702112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6817","nonce":1384702112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6815","nonce":1384702112}]},"public_addr":"172.21.15.32:6811/1384702112","cluster_addr":"172.21.15.32:6813/1384702112","heartbeat_back_addr":"172.21.15.32:6817/1384702112","heartbeat_front_addr":"172.21.15.32:6815/1384702112","state":["exists","up"]},{"osd":4,"uuid":"e1c1e894-074d-4c78-89ef-9e5a114268bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6809","nonce":4250534574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6811","nonce":4250534574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6815","nonce":4250534574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6813","nonce":4250534574}]},"public_addr":"172.21.15.71:6809/4250534574","cluster_addr":"172.21.15.71:6811/4250534574","heartbeat_back_addr":"172.21.15.71:6815/4250534574","heartbeat_front_addr":"172.21.15.71:6813/4250534574","state":["exists","up"]},{"osd":5,"uuid":"12c022d4-fac6-40c4-ae7a-691571bee5d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6809","nonce":4245346198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6811","nonce":4245346198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6815","nonce":4245346198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6813","nonce":4245346198}]},"public_addr":"172.21.15.189:6809/4245346198","cluster_addr":"172.21.15.189:6811/4245346198","heartbeat_back_addr":"172.21.15.189:6815/4245346198","heartbeat_front_addr":"172.21.15.189:6813/4245346198","state":["exists","up"]},{"osd":6,"uuid":"b413f8ee-3d4a-415a-a2ba-d68437b7330d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6819","nonce":1910389190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6821","nonce":1910389190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6825","nonce":1910389190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6823","nonce":1910389190}]},"public_addr":"172.21.15.32:6819/1910389190","cluster_addr":"172.21.15.32:6821/1910389190","heartbeat_back_addr":"172.21.15.32:6825/1910389190","heartbeat_front_addr":"172.21.15.32:6823/1910389190","state":["exists","up"]},{"osd":7,"uuid":"aac64fb3-fc13-4126-b842-7b06ad5ce238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6817","nonce":692607380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6819","nonce":692607380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6823","nonce":692607380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6821","nonce":692607380}]},"public_addr":"172.21.15.71:6817/692607380","cluster_addr":"172.21.15.71:6819/692607380","heartbeat_back_addr":"172.21.15.71:6823/692607380","heartbeat_front_addr":"172.21.15.71:6821/692607380","state":["exists","up"]},{"osd":8,"uuid":"cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6817","nonce":3973697069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6819","nonce":3973697069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6823","nonce":3973697069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6821","nonce":3973697069}]},"public_addr":"172.21.15.189:6817/3973697069","cluster_addr":"172.21.15.189:6819/3973697069","heartbeat_back_addr":"172.21.15.189:6823/3973697069","heartbeat_front_addr":"172.21.15.189:6821/3973697069","state":["exists","up"]},{"osd":9,"uuid":"ef4a1645-ae6e-45c3-82ad-161510e62aa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6827","nonce":46066685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6829","nonce":46066685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6833","nonce":46066685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6831","nonce":46066685}]},"public_addr":"172.21.15.32:6827/46066685","cluster_addr":"172.21.15.32:6829/46066685","heartbeat_back_addr":"172.21.15.32:6833/46066685","heartbeat_front_addr":"172.21.15.32:6831/46066685","state":["exists","up"]},{"osd":10,"uuid":"25b26a38-a644-409f-b941-59efdf04a3ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6825","nonce":1886290293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6827","nonce":1886290293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6831","nonce":1886290293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6829","nonce":1886290293}]},"public_addr":"172.21.15.71:6825/1886290293","cluster_addr":"172.21.15.71:6827/1886290293","heartbeat_back_addr":"172.21.15.71:6831/1886290293","heartbeat_front_addr":"172.21.15.71:6829/1886290293","state":["exists","up"]},{"osd":11,"uuid":"ab96b3ff-3688-455b-af64-60226c5bfbff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6825","nonce":850185001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6827","nonce":850185001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6831","nonce":850185001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6829","nonce":850185001}]},"public_addr":"172.21.15.189:6825/850185001","cluster_addr":"172.21.15.189:6827/850185001","heartbeat_back_addr":"172.21.15.189:6831/850185001","heartbeat_front_addr":"172.21.15.189:6829/850185001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:27.220308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:53.230783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:17.190642+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:52.603799+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:20.568671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:46.158363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:22.706933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:50.665173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:16.602410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:53.814698+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:22.435189+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:48.651949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.e","mappings":[{"from":4,"to":7}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.28","mappings":[{"from":3,"to":6},{"from":5,"to":8}]},{"pgid":"3.30","mappings":[{"from":1,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1266897222":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/2915689533":"2024-05-01T08:23:29.526664+0000","172.21.15.32:6801/4019351306":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/425836193":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/1134075722":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6801/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/21059253":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/3245550894":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/4019351306":"2024-05-01T08:23:29.526664+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-30T08:34:51.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.709+0000 7f9800c00640 1 -- 172.21.15.32:0/3432396457 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f97e00afae0 msgr2=0x7f97e00b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.710+0000 7f9800c00640 1 --2- 172.21.15.32:0/3432396457 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f97e00afae0 0x7f97e00b1ff0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f97ec005ab0 tx=0x7f97ec031400 comp rx=0 tx=0).stop 2024-04-30T08:34:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.710+0000 7f9800c00640 1 -- 172.21.15.32:0/3432396457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97fc10b200 msgr2=0x7f97fc1a0c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.710+0000 7f9800c00640 1 --2- 172.21.15.32:0/3432396457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97fc10b200 0x7f97fc1a0c90 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f97e400d770 tx=0x7f97e400dc30 comp rx=0 tx=0).stop 2024-04-30T08:34:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.710+0000 7f9800c00640 1 -- 172.21.15.32:0/3432396457 shutdown_connections 2024-04-30T08:34:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.710+0000 7f9800c00640 1 --2- 172.21.15.32:0/3432396457 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f97e00afae0 0x7f97e00b1ff0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.710+0000 7f9800c00640 1 --2- 172.21.15.32:0/3432396457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97fc10ff00 0x7f97fc1a7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.710+0000 7f9800c00640 1 --2- 172.21.15.32:0/3432396457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f97fc10b200 0x7f97fc1a0c90 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.710+0000 7f9800c00640 1 --2- 172.21.15.32:0/3432396457 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f97fc109fc0 0x7f97fc1a0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.710+0000 7f9800c00640 1 -- 172.21.15.32:0/3432396457 >> 172.21.15.32:0/3432396457 conn(0x7f97fc078bb0 msgr2=0x7f97fc10bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.710+0000 7f9800c00640 1 -- 172.21.15.32:0/3432396457 shutdown_connections 2024-04-30T08:34:51.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:51.710+0000 7f9800c00640 1 -- 172.21.15.32:0/3432396457 wait complete. 2024-04-30T08:34:51.739 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-30T08:26:19.728331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-30T08:31:49.800542+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-30T08:31:51.211320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-30T08:31:55.716625+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-30T08:34:51.740 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-30T08:34:52.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.021+0000 7f79a3200640 1 -- 172.21.15.32:0/2264941309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79a4106df0 msgr2=0x7f79a4107200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:52.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.021+0000 7f79a3200640 1 --2- 172.21.15.32:0/2264941309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79a4106df0 0x7f79a4107200 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f798c009900 tx=0x7f798c02a410 comp rx=0 tx=0).stop 2024-04-30T08:34:52.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.022+0000 7f79a3200640 1 -- 172.21.15.32:0/2264941309 shutdown_connections 2024-04-30T08:34:52.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.022+0000 7f79a3200640 1 --2- 172.21.15.32:0/2264941309 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79a410baf0 0x7f79a410df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:52.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.022+0000 7f79a3200640 1 --2- 172.21.15.32:0/2264941309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79a4106df0 0x7f79a4107200 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:52.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.022+0000 7f79a3200640 1 --2- 172.21.15.32:0/2264941309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79a4105bb0 0x7f79a41068b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:52.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.022+0000 7f79a3200640 1 -- 172.21.15.32:0/2264941309 >> 172.21.15.32:0/2264941309 conn(0x7f79a4100da0 msgr2=0x7f79a4103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:52.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.022+0000 7f79a3200640 1 -- 172.21.15.32:0/2264941309 shutdown_connections 2024-04-30T08:34:52.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.022+0000 7f79a3200640 1 -- 172.21.15.32:0/2264941309 wait complete. 2024-04-30T08:34:52.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.023+0000 7f79a3200640 1 Processor -- start 2024-04-30T08:34:52.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.023+0000 7f79a3200640 1 -- start start 2024-04-30T08:34:52.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.023+0000 7f79a3200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79a4105bb0 0x7f79a419c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:52.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.023+0000 7f79a3200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79a4106df0 0x7f79a419c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:52.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.023+0000 7f79a3200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79a410baf0 0x7f79a41a37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:52.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.023+0000 7f79a3200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f79a4111700 con 0x7f79a4106df0 2024-04-30T08:34:52.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.023+0000 7f79a3200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f79a4111580 con 0x7f79a4105bb0 2024-04-30T08:34:52.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.023+0000 7f79a3200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f79a4111880 con 0x7f79a410baf0 2024-04-30T08:34:52.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.024+0000 7f799be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79a4106df0 0x7f79a419c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:52.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.024+0000 7f79a1e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79a4105bb0 0x7f79a419c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:52.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.024+0000 7f799be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79a4106df0 0x7f79a419c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43340/0 (socket says 172.21.15.32:43340) 2024-04-30T08:34:52.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.024+0000 7f799be00640 1 -- 172.21.15.32:0/2637382162 learned_addr learned my addr 172.21.15.32:0/2637382162 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:52.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.024+0000 7f79a2800640 1 --2- 172.21.15.32:0/2637382162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79a410baf0 0x7f79a41a37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:52.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.024+0000 7f79a1e00640 1 -- 172.21.15.32:0/2637382162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79a410baf0 msgr2=0x7f79a41a37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:52.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.024+0000 7f79a1e00640 1 --2- 172.21.15.32:0/2637382162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79a410baf0 0x7f79a41a37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:52.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.024+0000 7f79a1e00640 1 -- 172.21.15.32:0/2637382162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79a4106df0 msgr2=0x7f79a419c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:52.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.024+0000 7f79a1e00640 1 --2- 172.21.15.32:0/2637382162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79a4106df0 0x7f79a419c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:52.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.024+0000 7f79a1e00640 1 -- 172.21.15.32:0/2637382162 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79a41a3e10 con 0x7f79a4105bb0 2024-04-30T08:34:52.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.024+0000 7f799be00640 1 --2- 172.21.15.32:0/2637382162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79a4106df0 0x7f79a419c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:34:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.024+0000 7f79a1e00640 1 --2- 172.21.15.32:0/2637382162 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79a4105bb0 0x7f79a419c2d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f79a4105480 tx=0x7f799400ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.025+0000 7f799aa00640 1 -- 172.21.15.32:0/2637382162 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f799400cc40 con 0x7f79a4105bb0 2024-04-30T08:34:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.025+0000 7f799aa00640 1 -- 172.21.15.32:0/2637382162 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f799400cde0 con 0x7f79a4105bb0 2024-04-30T08:34:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.025+0000 7f79a3200640 1 -- 172.21.15.32:0/2637382162 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79a41a4100 con 0x7f79a4105bb0 2024-04-30T08:34:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.025+0000 7f799aa00640 1 -- 172.21.15.32:0/2637382162 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7994010430 con 0x7f79a4105bb0 2024-04-30T08:34:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.025+0000 7f79a3200640 1 -- 172.21.15.32:0/2637382162 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79a41a4650 con 0x7f79a4105bb0 2024-04-30T08:34:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.026+0000 7f79a3200640 1 -- 172.21.15.32:0/2637382162 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79a410a900 con 0x7f79a4105bb0 2024-04-30T08:34:52.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.029+0000 7f799aa00640 1 -- 172.21.15.32:0/2637382162 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f7994003eb0 con 0x7f79a4105bb0 2024-04-30T08:34:52.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.030+0000 7f799aa00640 1 --2- 172.21.15.32:0/2637382162 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f79880afae0 0x7f79880b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:52.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.030+0000 7f799aa00640 1 -- 172.21.15.32:0/2637382162 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f79940df4c0 con 0x7f79a4105bb0 2024-04-30T08:34:52.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.030+0000 7f799be00640 1 --2- 172.21.15.32:0/2637382162 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f79880afae0 0x7f79880b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:52.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.031+0000 7f799be00640 1 --2- 172.21.15.32:0/2637382162 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f79880afae0 0x7f79880b1ff0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f798c005f70 tx=0x7f798c0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:52.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.032+0000 7f799aa00640 1 -- 172.21.15.32:0/2637382162 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f79940a7d60 con 0x7f79a4105bb0 2024-04-30T08:34:52.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.239+0000 7f79a3200640 1 -- 172.21.15.32:0/2637382162 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f79a4063820 con 0x7f79a4105bb0 2024-04-30T08:34:52.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.240+0000 7f799aa00640 1 -- 172.21.15.32:0/2637382162 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) ==== 93+0+10 (secure 0 0 0) 0x7f79940a7660 con 0x7f79a4105bb0 2024-04-30T08:34:52.242 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-30T08:34:52.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.243+0000 7f79a3200640 1 -- 172.21.15.32:0/2637382162 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f79880afae0 msgr2=0x7f79880b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:52.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.243+0000 7f79a3200640 1 --2- 172.21.15.32:0/2637382162 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f79880afae0 0x7f79880b1ff0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f798c005f70 tx=0x7f798c0314a0 comp rx=0 tx=0).stop 2024-04-30T08:34:52.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.243+0000 7f79a3200640 1 -- 172.21.15.32:0/2637382162 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79a4105bb0 msgr2=0x7f79a419c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:52.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.243+0000 7f79a3200640 1 --2- 172.21.15.32:0/2637382162 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79a4105bb0 0x7f79a419c2d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f79a4105480 tx=0x7f799400ee50 comp rx=0 tx=0).stop 2024-04-30T08:34:52.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.243+0000 7f79a3200640 1 -- 172.21.15.32:0/2637382162 shutdown_connections 2024-04-30T08:34:52.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.243+0000 7f79a3200640 1 --2- 172.21.15.32:0/2637382162 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f79880afae0 0x7f79880b1ff0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:52.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.243+0000 7f79a3200640 1 --2- 172.21.15.32:0/2637382162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f79a410baf0 0x7f79a41a37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:52.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.243+0000 7f79a3200640 1 --2- 172.21.15.32:0/2637382162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f79a4106df0 0x7f79a419c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:52.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.243+0000 7f79a3200640 1 --2- 172.21.15.32:0/2637382162 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f79a4105bb0 0x7f79a419c2d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:52.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.243+0000 7f79a3200640 1 -- 172.21.15.32:0/2637382162 >> 172.21.15.32:0/2637382162 conn(0x7f79a4100da0 msgr2=0x7f79a4107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:52.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.244+0000 7f79a3200640 1 -- 172.21.15.32:0/2637382162 shutdown_connections 2024-04-30T08:34:52.246 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.244+0000 7f79a3200640 1 -- 172.21.15.32:0/2637382162 wait complete. 2024-04-30T08:34:52.271 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-30T08:34:52.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:52 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/964267485' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:34:52.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:52 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3432396457' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:34:52.344 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:52 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/964267485' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:34:52.344 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:52 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3432396457' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:34:52.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:52 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/964267485' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:34:52.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:52 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3432396457' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:34:52.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.559+0000 7fb0bce00640 1 -- 172.21.15.32:0/1383676374 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0b8077990 msgr2=0x7fb0b80786e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:52.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.559+0000 7fb0bce00640 1 --2- 172.21.15.32:0/1383676374 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0b8077990 0x7fb0b80786e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8009900 tx=0x7fb0a802a410 comp rx=0 tx=0).stop 2024-04-30T08:34:52.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.559+0000 7fb0bce00640 1 -- 172.21.15.32:0/1383676374 shutdown_connections 2024-04-30T08:34:52.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.559+0000 7fb0bce00640 1 --2- 172.21.15.32:0/1383676374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0b8079920 0x7fb0b8079d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:52.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.559+0000 7fb0bce00640 1 --2- 172.21.15.32:0/1383676374 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0b8078c20 0x7fb0b8079030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:52.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.559+0000 7fb0bce00640 1 --2- 172.21.15.32:0/1383676374 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0b8077990 0x7fb0b80786e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:52.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.559+0000 7fb0bce00640 1 -- 172.21.15.32:0/1383676374 >> 172.21.15.32:0/1383676374 conn(0x7fb0b8100da0 msgr2=0x7fb0b8103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:52.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.559+0000 7fb0bce00640 1 -- 172.21.15.32:0/1383676374 shutdown_connections 2024-04-30T08:34:52.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.559+0000 7fb0bce00640 1 -- 172.21.15.32:0/1383676374 wait complete. 2024-04-30T08:34:52.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.561+0000 7fb0bce00640 1 Processor -- start 2024-04-30T08:34:52.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.561+0000 7fb0bce00640 1 -- start start 2024-04-30T08:34:52.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.561+0000 7fb0bce00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0b8077990 0x7fb0b81a06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:52.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.561+0000 7fb0bce00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0b8078c20 0x7fb0b81a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:52.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.561+0000 7fb0bce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0b8079920 0x7fb0b81a7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:52.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.561+0000 7fb0bce00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b8115ad0 con 0x7fb0b8079920 2024-04-30T08:34:52.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.561+0000 7fb0bce00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b8115950 con 0x7fb0b8077990 2024-04-30T08:34:52.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.561+0000 7fb0bce00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb0b8115c50 con 0x7fb0b8078c20 2024-04-30T08:34:52.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.562+0000 7fb0b7400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0b8077990 0x7fb0b81a06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:52.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.562+0000 7fb0b7400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0b8077990 0x7fb0b81a06e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:46282/0 (socket says 172.21.15.32:46282) 2024-04-30T08:34:52.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.562+0000 7fb0b7400640 1 -- 172.21.15.32:0/3938210299 learned_addr learned my addr 172.21.15.32:0/3938210299 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:52.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.562+0000 7fb0b7e00640 1 --2- 172.21.15.32:0/3938210299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0b8079920 0x7fb0b81a7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:52.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.562+0000 7fb0b6a00640 1 --2- 172.21.15.32:0/3938210299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0b8078c20 0x7fb0b81a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:52.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.562+0000 7fb0b7400640 1 -- 172.21.15.32:0/3938210299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0b8078c20 msgr2=0x7fb0b81a0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:52.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.562+0000 7fb0b7400640 1 --2- 172.21.15.32:0/3938210299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0b8078c20 0x7fb0b81a0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:52.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.562+0000 7fb0b7400640 1 -- 172.21.15.32:0/3938210299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0b8079920 msgr2=0x7fb0b81a7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:52.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.562+0000 7fb0b7400640 1 --2- 172.21.15.32:0/3938210299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0b8079920 0x7fb0b81a7bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:52.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.562+0000 7fb0b7400640 1 -- 172.21.15.32:0/3938210299 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0b81a8220 con 0x7fb0b8077990 2024-04-30T08:34:52.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.562+0000 7fb0b7400640 1 --2- 172.21.15.32:0/3938210299 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0b8077990 0x7fb0b81a06e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8031440 tx=0x7fb0a8003d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:52.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.563+0000 7fb0afe00640 1 -- 172.21.15.32:0/3938210299 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a803f070 con 0x7fb0b8077990 2024-04-30T08:34:52.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.563+0000 7fb0afe00640 1 -- 172.21.15.32:0/3938210299 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb0a803b460 con 0x7fb0b8077990 2024-04-30T08:34:52.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.563+0000 7fb0bce00640 1 -- 172.21.15.32:0/3938210299 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0b81a84b0 con 0x7fb0b8077990 2024-04-30T08:34:52.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.563+0000 7fb0afe00640 1 -- 172.21.15.32:0/3938210299 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0a8035030 con 0x7fb0b8077990 2024-04-30T08:34:52.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.563+0000 7fb0bce00640 1 -- 172.21.15.32:0/3938210299 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0b81a89f0 con 0x7fb0b8077990 2024-04-30T08:34:52.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.564+0000 7fb0bce00640 1 -- 172.21.15.32:0/3938210299 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0b81123b0 con 0x7fb0b8077990 2024-04-30T08:34:52.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.567+0000 7fb0afe00640 1 -- 172.21.15.32:0/3938210299 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fb0a8042070 con 0x7fb0b8077990 2024-04-30T08:34:52.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.569+0000 7fb0afe00640 1 --2- 172.21.15.32:0/3938210299 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb09c0afb60 0x7fb09c0b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:52.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.569+0000 7fb0afe00640 1 -- 172.21.15.32:0/3938210299 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fb0a80ffd00 con 0x7fb0b8077990 2024-04-30T08:34:52.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.569+0000 7fb0b6a00640 1 --2- 172.21.15.32:0/3938210299 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb09c0afb60 0x7fb09c0b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:52.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.570+0000 7fb0b6a00640 1 --2- 172.21.15.32:0/3938210299 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb09c0afb60 0x7fb09c0b2070 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb0b81a1780 tx=0x7fb0a400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:52.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.571+0000 7fb0afe00640 1 -- 172.21.15.32:0/3938210299 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb0a80c7e80 con 0x7fb0b8077990 2024-04-30T08:34:52.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.778+0000 7fb0bce00640 1 -- 172.21.15.32:0/3938210299 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb0b8063820 con 0x7fb0b8077990 2024-04-30T08:34:52.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.780+0000 7fb0afe00640 1 -- 172.21.15.32:0/3938210299 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) ==== 104+0+11 (secure 0 0 0) 0x7fb0a80cc180 con 0x7fb0b8077990 2024-04-30T08:34:52.781 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-30T08:34:52.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.782+0000 7fb0bce00640 1 -- 172.21.15.32:0/3938210299 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb09c0afb60 msgr2=0x7fb09c0b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:52.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.782+0000 7fb0bce00640 1 --2- 172.21.15.32:0/3938210299 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb09c0afb60 0x7fb09c0b2070 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb0b81a1780 tx=0x7fb0a400b390 comp rx=0 tx=0).stop 2024-04-30T08:34:52.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.782+0000 7fb0bce00640 1 -- 172.21.15.32:0/3938210299 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0b8077990 msgr2=0x7fb0b81a06e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:52.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.782+0000 7fb0bce00640 1 --2- 172.21.15.32:0/3938210299 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0b8077990 0x7fb0b81a06e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fb0a8031440 tx=0x7fb0a8003d50 comp rx=0 tx=0).stop 2024-04-30T08:34:52.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.782+0000 7fb0bce00640 1 -- 172.21.15.32:0/3938210299 shutdown_connections 2024-04-30T08:34:52.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.782+0000 7fb0bce00640 1 --2- 172.21.15.32:0/3938210299 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb09c0afb60 0x7fb09c0b2070 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:52.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.782+0000 7fb0bce00640 1 --2- 172.21.15.32:0/3938210299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb0b8079920 0x7fb0b81a7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:52.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.782+0000 7fb0bce00640 1 --2- 172.21.15.32:0/3938210299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb0b8078c20 0x7fb0b81a0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:52.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.783+0000 7fb0bce00640 1 --2- 172.21.15.32:0/3938210299 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb0b8077990 0x7fb0b81a06e0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:52.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.783+0000 7fb0bce00640 1 -- 172.21.15.32:0/3938210299 >> 172.21.15.32:0/3938210299 conn(0x7fb0b8100da0 msgr2=0x7fb0b81024f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:52.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.783+0000 7fb0bce00640 1 -- 172.21.15.32:0/3938210299 shutdown_connections 2024-04-30T08:34:52.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:52.783+0000 7fb0bce00640 1 -- 172.21.15.32:0/3938210299 wait complete. 2024-04-30T08:34:52.812 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-30T08:34:53.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.110+0000 7fc719600640 1 -- 172.21.15.32:0/723327708 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc714103b90 msgr2=0x7fc714104890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:53.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.110+0000 7fc719600640 1 --2- 172.21.15.32:0/723327708 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc714103b90 0x7fc714104890 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc704009900 tx=0x7fc70402a410 comp rx=0 tx=0).stop 2024-04-30T08:34:53.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.110+0000 7fc719600640 1 -- 172.21.15.32:0/723327708 shutdown_connections 2024-04-30T08:34:53.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.110+0000 7fc719600640 1 --2- 172.21.15.32:0/723327708 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc714109ad0 0x7fc71410bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:53.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.110+0000 7fc719600640 1 --2- 172.21.15.32:0/723327708 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc714104dd0 0x7fc7141051e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:53.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.110+0000 7fc719600640 1 --2- 172.21.15.32:0/723327708 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc714103b90 0x7fc714104890 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:53.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.110+0000 7fc719600640 1 -- 172.21.15.32:0/723327708 >> 172.21.15.32:0/723327708 conn(0x7fc7140fedc0 msgr2=0x7fc714101230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:53.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.111+0000 7fc719600640 1 -- 172.21.15.32:0/723327708 shutdown_connections 2024-04-30T08:34:53.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.111+0000 7fc719600640 1 -- 172.21.15.32:0/723327708 wait complete. 2024-04-30T08:34:53.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.112+0000 7fc719600640 1 Processor -- start 2024-04-30T08:34:53.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.112+0000 7fc719600640 1 -- start start 2024-04-30T08:34:53.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.113+0000 7fc719600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc714103b90 0x7fc71419c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:53.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.113+0000 7fc719600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc714104dd0 0x7fc71419cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:53.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.113+0000 7fc719600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc714109ad0 0x7fc7141a3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:53.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.113+0000 7fc719600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc71410f6c0 con 0x7fc714104dd0 2024-04-30T08:34:53.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.113+0000 7fc719600640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc71410f540 con 0x7fc714103b90 2024-04-30T08:34:53.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.113+0000 7fc719600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc71410f840 con 0x7fc714109ad0 2024-04-30T08:34:53.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.113+0000 7fc712000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc714104dd0 0x7fc71419cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:53.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.113+0000 7fc712000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc714104dd0 0x7fc71419cad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43388/0 (socket says 172.21.15.32:43388) 2024-04-30T08:34:53.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.113+0000 7fc712000640 1 -- 172.21.15.32:0/3412606954 learned_addr learned my addr 172.21.15.32:0/3412606954 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:53.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.113+0000 7fc712a00640 1 --2- 172.21.15.32:0/3412606954 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc714103b90 0x7fc71419c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:53.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.113+0000 7fc713400640 1 --2- 172.21.15.32:0/3412606954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc714109ad0 0x7fc7141a3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:53.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.114+0000 7fc712000640 1 -- 172.21.15.32:0/3412606954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc714109ad0 msgr2=0x7fc7141a3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:53.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.114+0000 7fc712000640 1 --2- 172.21.15.32:0/3412606954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc714109ad0 0x7fc7141a3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:53.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.114+0000 7fc712000640 1 -- 172.21.15.32:0/3412606954 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc714103b90 msgr2=0x7fc71419c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:53.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.114+0000 7fc712000640 1 --2- 172.21.15.32:0/3412606954 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc714103b90 0x7fc71419c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:53.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.114+0000 7fc712000640 1 -- 172.21.15.32:0/3412606954 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc7141a40d0 con 0x7fc714104dd0 2024-04-30T08:34:53.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.114+0000 7fc712000640 1 --2- 172.21.15.32:0/3412606954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc714104dd0 0x7fc71419cad0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc70000ea60 tx=0x7fc70000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:53.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.114+0000 7fc70b400640 1 -- 172.21.15.32:0/3412606954 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc70000cde0 con 0x7fc714104dd0 2024-04-30T08:34:53.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.114+0000 7fc70b400640 1 -- 172.21.15.32:0/3412606954 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc7000035f0 con 0x7fc714104dd0 2024-04-30T08:34:53.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.114+0000 7fc719600640 1 -- 172.21.15.32:0/3412606954 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc7141a43c0 con 0x7fc714104dd0 2024-04-30T08:34:53.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.114+0000 7fc70b400640 1 -- 172.21.15.32:0/3412606954 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc700010690 con 0x7fc714104dd0 2024-04-30T08:34:53.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.115+0000 7fc719600640 1 -- 172.21.15.32:0/3412606954 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc7141a4910 con 0x7fc714104dd0 2024-04-30T08:34:53.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.116+0000 7fc719600640 1 -- 172.21.15.32:0/3412606954 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc7141a4bd0 con 0x7fc714104dd0 2024-04-30T08:34:53.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.119+0000 7fc70b400640 1 -- 172.21.15.32:0/3412606954 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fc7000037a0 con 0x7fc714104dd0 2024-04-30T08:34:53.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.121+0000 7fc70b400640 1 --2- 172.21.15.32:0/3412606954 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc6f80afaf0 0x7fc6f80b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:53.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.122+0000 7fc70b400640 1 -- 172.21.15.32:0/3412606954 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fc7000df500 con 0x7fc714104dd0 2024-04-30T08:34:53.123 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.122+0000 7fc712a00640 1 --2- 172.21.15.32:0/3412606954 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc6f80afaf0 0x7fc6f80b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:53.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.123+0000 7fc712a00640 1 --2- 172.21.15.32:0/3412606954 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc6f80afaf0 0x7fc6f80b2000 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc70402aa00 tx=0x7fc7040314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:53.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.124+0000 7fc70b400640 1 -- 172.21.15.32:0/3412606954 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc700053ff0 con 0x7fc714104dd0 2024-04-30T08:34:53.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:53 smithi032 ceph-mon[28331]: pgmap v416: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-30T08:34:53.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:53 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2637382162' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:34:53.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:53 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3938210299' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-30T08:34:53.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:53 smithi189 ceph-mon[33038]: pgmap v416: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-30T08:34:53.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:53 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2637382162' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:34:53.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:53 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3938210299' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-30T08:34:53.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.330+0000 7fc719600640 1 -- 172.21.15.32:0/3412606954 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fc7141a51e0 con 0x7fc714104dd0 2024-04-30T08:34:53.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.331+0000 7fc70b400640 1 -- 172.21.15.32:0/3412606954 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) ==== 100+0+11 (secure 0 0 0) 0x7fc7000a7cd0 con 0x7fc714104dd0 2024-04-30T08:34:53.333 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-04-30T08:34:53.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.334+0000 7fc719600640 1 -- 172.21.15.32:0/3412606954 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc6f80afaf0 msgr2=0x7fc6f80b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:53.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.334+0000 7fc719600640 1 --2- 172.21.15.32:0/3412606954 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc6f80afaf0 0x7fc6f80b2000 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fc70402aa00 tx=0x7fc7040314a0 comp rx=0 tx=0).stop 2024-04-30T08:34:53.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.334+0000 7fc719600640 1 -- 172.21.15.32:0/3412606954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc714104dd0 msgr2=0x7fc71419cad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:53.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.334+0000 7fc719600640 1 --2- 172.21.15.32:0/3412606954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc714104dd0 0x7fc71419cad0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fc70000ea60 tx=0x7fc70000ef20 comp rx=0 tx=0).stop 2024-04-30T08:34:53.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.334+0000 7fc719600640 1 -- 172.21.15.32:0/3412606954 shutdown_connections 2024-04-30T08:34:53.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.334+0000 7fc719600640 1 --2- 172.21.15.32:0/3412606954 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc6f80afaf0 0x7fc6f80b2000 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:53.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.334+0000 7fc719600640 1 --2- 172.21.15.32:0/3412606954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc714109ad0 0x7fc7141a3a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:53.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.334+0000 7fc719600640 1 --2- 172.21.15.32:0/3412606954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc714104dd0 0x7fc71419cad0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:53.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.334+0000 7fc719600640 1 --2- 172.21.15.32:0/3412606954 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc714103b90 0x7fc71419c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:53.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.334+0000 7fc719600640 1 -- 172.21.15.32:0/3412606954 >> 172.21.15.32:0/3412606954 conn(0x7fc7140fedc0 msgr2=0x7fc7140ff400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:53.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.334+0000 7fc719600640 1 -- 172.21.15.32:0/3412606954 shutdown_connections 2024-04-30T08:34:53.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.334+0000 7fc719600640 1 -- 172.21.15.32:0/3412606954 wait complete. 2024-04-30T08:34:53.365 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-30T08:34:53.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:53 smithi071 ceph-mon[33385]: pgmap v416: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-30T08:34:53.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:53 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2637382162' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:34:53.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:53 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3938210299' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-30T08:34:53.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.654+0000 7f4e75000640 1 -- 172.21.15.32:0/2376026567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e70104bd0 msgr2=0x7f4e70104fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:53.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.654+0000 7f4e75000640 1 --2- 172.21.15.32:0/2376026567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e70104bd0 0x7f4e70104fe0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c009860 tx=0x7f4e5c02a340 comp rx=0 tx=0).stop 2024-04-30T08:34:53.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.654+0000 7f4e75000640 1 -- 172.21.15.32:0/2376026567 shutdown_connections 2024-04-30T08:34:53.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.654+0000 7f4e75000640 1 --2- 172.21.15.32:0/2376026567 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e701098d0 0x7f4e7010bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:53.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.654+0000 7f4e75000640 1 --2- 172.21.15.32:0/2376026567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e70104bd0 0x7f4e70104fe0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:53.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.654+0000 7f4e75000640 1 --2- 172.21.15.32:0/2376026567 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e70103990 0x7f4e70104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:53.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.654+0000 7f4e75000640 1 -- 172.21.15.32:0/2376026567 >> 172.21.15.32:0/2376026567 conn(0x7f4e700feba0 msgr2=0x7f4e70101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:53.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.654+0000 7f4e75000640 1 -- 172.21.15.32:0/2376026567 shutdown_connections 2024-04-30T08:34:53.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.655+0000 7f4e75000640 1 -- 172.21.15.32:0/2376026567 wait complete. 2024-04-30T08:34:53.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.656+0000 7f4e75000640 1 Processor -- start 2024-04-30T08:34:53.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.656+0000 7f4e75000640 1 -- start start 2024-04-30T08:34:53.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.656+0000 7f4e75000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e70103990 0x7f4e7019c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:53.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.656+0000 7f4e75000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e70104bd0 0x7f4e7019c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:53.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.656+0000 7f4e75000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e701098d0 0x7f4e701a3780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:53.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.656+0000 7f4e75000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7010f4c0 con 0x7f4e701098d0 2024-04-30T08:34:53.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.656+0000 7f4e75000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7010f340 con 0x7f4e70104bd0 2024-04-30T08:34:53.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.656+0000 7f4e75000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7010f640 con 0x7f4e70103990 2024-04-30T08:34:53.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.656+0000 7f4e6f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e70103990 0x7f4e7019c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:53.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.656+0000 7f4e6fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e701098d0 0x7f4e701a3780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:53.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.656+0000 7f4e6f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e70103990 0x7f4e7019c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:58386/0 (socket says 172.21.15.32:58386) 2024-04-30T08:34:53.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.656+0000 7f4e6f400640 1 -- 172.21.15.32:0/2937572677 learned_addr learned my addr 172.21.15.32:0/2937572677 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:53.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.656+0000 7f4e6ea00640 1 --2- 172.21.15.32:0/2937572677 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e70104bd0 0x7f4e7019c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:53.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.657+0000 7f4e6f400640 1 -- 172.21.15.32:0/2937572677 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e70104bd0 msgr2=0x7f4e7019c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:53.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.657+0000 7f4e6f400640 1 --2- 172.21.15.32:0/2937572677 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e70104bd0 0x7f4e7019c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:53.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.657+0000 7f4e6f400640 1 -- 172.21.15.32:0/2937572677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e701098d0 msgr2=0x7f4e701a3780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:53.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.657+0000 7f4e6f400640 1 --2- 172.21.15.32:0/2937572677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e701098d0 0x7f4e701a3780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:53.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.657+0000 7f4e6f400640 1 -- 172.21.15.32:0/2937572677 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e701a3df0 con 0x7f4e70103990 2024-04-30T08:34:53.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.657+0000 7f4e6fe00640 1 --2- 172.21.15.32:0/2937572677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e701098d0 0x7f4e701a3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:34:53.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.657+0000 7f4e6f400640 1 --2- 172.21.15.32:0/2937572677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e70103990 0x7f4e7019c2b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4e6000e980 tx=0x7f4e6000ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:53.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.658+0000 7f4e67e00640 1 -- 172.21.15.32:0/2937572677 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e6000cd80 con 0x7f4e70103990 2024-04-30T08:34:53.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.658+0000 7f4e67e00640 1 -- 172.21.15.32:0/2937572677 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e600108e0 con 0x7f4e70103990 2024-04-30T08:34:53.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.658+0000 7f4e67e00640 1 -- 172.21.15.32:0/2937572677 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e60018940 con 0x7f4e70103990 2024-04-30T08:34:53.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.658+0000 7f4e75000640 1 -- 172.21.15.32:0/2937572677 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e701a40e0 con 0x7f4e70103990 2024-04-30T08:34:53.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.658+0000 7f4e75000640 1 -- 172.21.15.32:0/2937572677 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e701a4630 con 0x7f4e70103990 2024-04-30T08:34:53.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.659+0000 7f4e75000640 1 -- 172.21.15.32:0/2937572677 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e701a4890 con 0x7f4e70103990 2024-04-30T08:34:53.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.662+0000 7f4e67e00640 1 -- 172.21.15.32:0/2937572677 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f4e6001f020 con 0x7f4e70103990 2024-04-30T08:34:53.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.663+0000 7f4e67e00640 1 --2- 172.21.15.32:0/2937572677 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4e540afae0 0x7f4e540b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:53.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.664+0000 7f4e67e00640 1 -- 172.21.15.32:0/2937572677 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f4e600df720 con 0x7f4e70103990 2024-04-30T08:34:53.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.664+0000 7f4e6ea00640 1 --2- 172.21.15.32:0/2937572677 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4e540afae0 0x7f4e540b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:53.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.664+0000 7f4e6ea00640 1 --2- 172.21.15.32:0/2937572677 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4e540afae0 0x7f4e540b1ff0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c02a930 tx=0x7f4e5c030450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:53.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.665+0000 7f4e67e00640 1 -- 172.21.15.32:0/2937572677 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4e600a7ef0 con 0x7f4e70103990 2024-04-30T08:34:53.876 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.874+0000 7f4e75000640 1 -- 172.21.15.32:0/2937572677 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f4e701a4a60 con 0x7f4e70103990 2024-04-30T08:34:53.877 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.875+0000 7f4e67e00640 1 -- 172.21.15.32:0/2937572677 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v85) ==== 103+0+11 (secure 0 0 0) 0x7f4e600a77f0 con 0x7f4e70103990 2024-04-30T08:34:53.877 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-30T08:34:53.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.879+0000 7f4e75000640 1 -- 172.21.15.32:0/2937572677 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4e540afae0 msgr2=0x7f4e540b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:53.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.879+0000 7f4e75000640 1 --2- 172.21.15.32:0/2937572677 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4e540afae0 0x7f4e540b1ff0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f4e5c02a930 tx=0x7f4e5c030450 comp rx=0 tx=0).stop 2024-04-30T08:34:53.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.879+0000 7f4e75000640 1 -- 172.21.15.32:0/2937572677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e70103990 msgr2=0x7f4e7019c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:53.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.879+0000 7f4e75000640 1 --2- 172.21.15.32:0/2937572677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e70103990 0x7f4e7019c2b0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f4e6000e980 tx=0x7f4e6000ee40 comp rx=0 tx=0).stop 2024-04-30T08:34:53.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.879+0000 7f4e75000640 1 -- 172.21.15.32:0/2937572677 shutdown_connections 2024-04-30T08:34:53.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.879+0000 7f4e75000640 1 --2- 172.21.15.32:0/2937572677 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4e540afae0 0x7f4e540b1ff0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:53.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.879+0000 7f4e75000640 1 --2- 172.21.15.32:0/2937572677 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e701098d0 0x7f4e701a3780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:53.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.879+0000 7f4e75000640 1 --2- 172.21.15.32:0/2937572677 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e70104bd0 0x7f4e7019c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:53.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.879+0000 7f4e75000640 1 --2- 172.21.15.32:0/2937572677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e70103990 0x7f4e7019c2b0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:53.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.879+0000 7f4e75000640 1 -- 172.21.15.32:0/2937572677 >> 172.21.15.32:0/2937572677 conn(0x7f4e700feba0 msgr2=0x7f4e700fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:53.881 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.879+0000 7f4e75000640 1 -- 172.21.15.32:0/2937572677 shutdown_connections 2024-04-30T08:34:53.882 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:53.879+0000 7f4e75000640 1 -- 172.21.15.32:0/2937572677 wait complete. 2024-04-30T08:34:53.910 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-30T08:34:54.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:54 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3412606954' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-30T08:34:54.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:54 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2937572677' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-30T08:34:54.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.193+0000 7f7b3e000640 1 -- 172.21.15.32:0/3452055811 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b3810ff00 msgr2=0x7f7b38112320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:54.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.193+0000 7f7b3e000640 1 --2- 172.21.15.32:0/3452055811 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b3810ff00 0x7f7b38112320 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7b30009ca0 tx=0x7f7b3002a8c0 comp rx=0 tx=0).stop 2024-04-30T08:34:54.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.193+0000 7f7b3e000640 1 -- 172.21.15.32:0/3452055811 shutdown_connections 2024-04-30T08:34:54.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.193+0000 7f7b3e000640 1 --2- 172.21.15.32:0/3452055811 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b3810ff00 0x7f7b38112320 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:54.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.193+0000 7f7b3e000640 1 --2- 172.21.15.32:0/3452055811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b3810b200 0x7f7b3810b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:54.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.193+0000 7f7b3e000640 1 --2- 172.21.15.32:0/3452055811 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b38109fc0 0x7f7b3810acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:54.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.193+0000 7f7b3e000640 1 -- 172.21.15.32:0/3452055811 >> 172.21.15.32:0/3452055811 conn(0x7f7b38078bb0 msgr2=0x7f7b38107a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:54.195 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.193+0000 7f7b3e000640 1 -- 172.21.15.32:0/3452055811 shutdown_connections 2024-04-30T08:34:54.196 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.193+0000 7f7b3e000640 1 -- 172.21.15.32:0/3452055811 wait complete. 2024-04-30T08:34:54.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.194+0000 7f7b3e000640 1 Processor -- start 2024-04-30T08:34:54.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.195+0000 7f7b3e000640 1 -- start start 2024-04-30T08:34:54.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.195+0000 7f7b3e000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b38109fc0 0x7f7b381a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:54.197 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.195+0000 7f7b3e000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b3810b200 0x7f7b381a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:54.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.195+0000 7f7b3e000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b3810ff00 0x7f7b381a7af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:54.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.195+0000 7f7b3e000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7b38115ae0 con 0x7f7b38109fc0 2024-04-30T08:34:54.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.195+0000 7f7b3e000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f7b38115960 con 0x7f7b3810b200 2024-04-30T08:34:54.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.195+0000 7f7b3e000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7b38115c60 con 0x7f7b3810ff00 2024-04-30T08:34:54.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.195+0000 7f7b3cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b38109fc0 0x7f7b381a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:54.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.195+0000 7f7b2fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b3810b200 0x7f7b381a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:54.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.195+0000 7f7b3d600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b3810ff00 0x7f7b381a7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:54.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.195+0000 7f7b2fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b3810b200 0x7f7b381a0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:46328/0 (socket says 172.21.15.32:46328) 2024-04-30T08:34:54.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.195+0000 7f7b2fe00640 1 -- 172.21.15.32:0/464300092 learned_addr learned my addr 172.21.15.32:0/464300092 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:54.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.195+0000 7f7b3cc00640 1 --2- 172.21.15.32:0/464300092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b38109fc0 0x7f7b381a0620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43416/0 (socket says 172.21.15.32:43416) 2024-04-30T08:34:54.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.196+0000 7f7b2fe00640 1 -- 172.21.15.32:0/464300092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b3810ff00 msgr2=0x7f7b381a7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:54.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.196+0000 7f7b2fe00640 1 --2- 172.21.15.32:0/464300092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b3810ff00 0x7f7b381a7af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:54.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.196+0000 7f7b2fe00640 1 -- 172.21.15.32:0/464300092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b38109fc0 msgr2=0x7f7b381a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:54.198 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.196+0000 7f7b2fe00640 1 --2- 172.21.15.32:0/464300092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b38109fc0 0x7f7b381a0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:54.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.196+0000 7f7b2fe00640 1 -- 172.21.15.32:0/464300092 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b381a8160 con 0x7f7b3810b200 2024-04-30T08:34:54.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.196+0000 7f7b3cc00640 1 --2- 172.21.15.32:0/464300092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b38109fc0 0x7f7b381a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:34:54.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.196+0000 7f7b2fe00640 1 --2- 172.21.15.32:0/464300092 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b3810b200 0x7f7b381a0b60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7b2000e9f0 tx=0x7f7b2000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:54.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.196+0000 7f7b36000640 1 -- 172.21.15.32:0/464300092 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b2000cd80 con 0x7f7b3810b200 2024-04-30T08:34:54.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.196+0000 7f7b36000640 1 -- 172.21.15.32:0/464300092 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7b200054d0 con 0x7f7b3810b200 2024-04-30T08:34:54.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.197+0000 7f7b3e000640 1 -- 172.21.15.32:0/464300092 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b381a8450 con 0x7f7b3810b200 2024-04-30T08:34:54.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.197+0000 7f7b36000640 1 -- 172.21.15.32:0/464300092 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b20010690 con 0x7f7b3810b200 2024-04-30T08:34:54.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.197+0000 7f7b3e000640 1 -- 172.21.15.32:0/464300092 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b381a89a0 con 0x7f7b3810b200 2024-04-30T08:34:54.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.198+0000 7f7b3e000640 1 -- 172.21.15.32:0/464300092 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b3810ed10 con 0x7f7b3810b200 2024-04-30T08:34:54.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.200+0000 7f7b36000640 1 -- 172.21.15.32:0/464300092 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f7b20004e90 con 0x7f7b3810b200 2024-04-30T08:34:54.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.201+0000 7f7b36000640 1 --2- 172.21.15.32:0/464300092 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f7b180afb60 0x7f7b180b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:54.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.202+0000 7f7b36000640 1 -- 172.21.15.32:0/464300092 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f7b200df670 con 0x7f7b3810b200 2024-04-30T08:34:54.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.202+0000 7f7b3cc00640 1 --2- 172.21.15.32:0/464300092 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f7b180afb60 0x7f7b180b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:54.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.203+0000 7f7b3cc00640 1 --2- 172.21.15.32:0/464300092 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f7b180afb60 0x7f7b180b2070 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7b3810abc0 tx=0x7f7b28005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:54.205 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.204+0000 7f7b36000640 1 -- 172.21.15.32:0/464300092 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7b200a7e40 con 0x7f7b3810b200 2024-04-30T08:34:54.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.454+0000 7f7b3e000640 1 -- 172.21.15.32:0/464300092 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7b3802dbd0 con 0x7f7b3810b200 2024-04-30T08:34:54.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.455+0000 7f7b36000640 1 -- 172.21.15.32:0/464300092 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) ==== 94+0+11927 (secure 0 0 0) 0x7f7b200a7740 con 0x7f7b3810b200 2024-04-30T08:34:54.457 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:34:54.457 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2235842676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2235842676},{"type":"v1","addr":"172.21.15.32:6835","nonce":2235842676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/3577009355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":3577009355},{"type":"v1","addr":"172.21.15.32:6841","nonce":3577009355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-30T08:31:52.236668+0000","modified":"2024-04-30T08:32:30.541714+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24541,"mds_3":24533,"mds_4":24517},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/463203038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":463203038},{"type":"v1","addr":"172.21.15.189:6833","nonce":463203038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6833/3635143591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3635143591},{"type":"v1","addr":"172.21.15.71:6833","nonce":3635143591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6837/2569449994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2569449994},{"type":"v1","addr":"172.21.15.32:6837","nonce":2569449994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/278686565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":278686565},{"type":"v1","addr":"172.21.15.189:6835","nonce":278686565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6835/1558303056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1558303056},{"type":"v1","addr":"172.21.15.71:6835","nonce":1558303056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.189:6837/686520873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":686520873},{"type":"v1","addr":"172.21.15.189:6837","nonce":686520873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.71:6837/2324039932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2324039932},{"type":"v1","addr":"172.21.15.71:6837","nonce":2324039932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6839/352970107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":352970107},{"type":"v1","addr":"172.21.15.189:6839","nonce":352970107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.32:6839/4160386524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4160386524},{"type":"v1","addr":"172.21.15.32:6839","nonce":4160386524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.71:6839/3517409993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3517409993},{"type":"v1","addr":"172.21.15.71:6839","nonce":3517409993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24590,24541,24560,24533,24517]},"id":1}]} 2024-04-30T08:34:54.458 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 27 2024-04-30T08:34:54.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.458+0000 7f7b3e000640 1 -- 172.21.15.32:0/464300092 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f7b180afb60 msgr2=0x7f7b180b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:54.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.458+0000 7f7b3e000640 1 --2- 172.21.15.32:0/464300092 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f7b180afb60 0x7f7b180b2070 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f7b3810abc0 tx=0x7f7b28005f90 comp rx=0 tx=0).stop 2024-04-30T08:34:54.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.458+0000 7f7b3e000640 1 -- 172.21.15.32:0/464300092 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b3810b200 msgr2=0x7f7b381a0b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:54.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.458+0000 7f7b3e000640 1 --2- 172.21.15.32:0/464300092 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b3810b200 0x7f7b381a0b60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f7b2000e9f0 tx=0x7f7b2000edf0 comp rx=0 tx=0).stop 2024-04-30T08:34:54.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.458+0000 7f7b3e000640 1 -- 172.21.15.32:0/464300092 shutdown_connections 2024-04-30T08:34:54.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.458+0000 7f7b3e000640 1 --2- 172.21.15.32:0/464300092 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f7b180afb60 0x7f7b180b2070 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:54.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.458+0000 7f7b3e000640 1 --2- 172.21.15.32:0/464300092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b3810ff00 0x7f7b381a7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:54.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.458+0000 7f7b3e000640 1 --2- 172.21.15.32:0/464300092 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7b3810b200 0x7f7b381a0b60 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:54.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.458+0000 7f7b3e000640 1 --2- 172.21.15.32:0/464300092 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7b38109fc0 0x7f7b381a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:54.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.458+0000 7f7b3e000640 1 -- 172.21.15.32:0/464300092 >> 172.21.15.32:0/464300092 conn(0x7f7b38078bb0 msgr2=0x7f7b3810bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:54.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.459+0000 7f7b3e000640 1 -- 172.21.15.32:0/464300092 shutdown_connections 2024-04-30T08:34:54.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.459+0000 7f7b3e000640 1 -- 172.21.15.32:0/464300092 wait complete. 2024-04-30T08:34:54.489 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-30T08:34:54.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:54 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3412606954' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-30T08:34:54.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:54 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2937572677' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-30T08:34:54.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:54 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3412606954' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-30T08:34:54.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:54 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2937572677' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-30T08:34:54.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.772+0000 7f4ab4a00640 1 -- 172.21.15.32:0/1519795573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ab0079920 msgr2=0x7f4ab0079d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:54.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.772+0000 7f4ab4a00640 1 --2- 172.21.15.32:0/1519795573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ab0079920 0x7f4ab0079d30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4aa8009be0 tx=0x7f4aa802a6d0 comp rx=0 tx=0).stop 2024-04-30T08:34:54.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.773+0000 7f4ab4a00640 1 -- 172.21.15.32:0/1519795573 shutdown_connections 2024-04-30T08:34:54.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.773+0000 7f4ab4a00640 1 --2- 172.21.15.32:0/1519795573 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ab0079920 0x7f4ab0079d30 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:54.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.773+0000 7f4ab4a00640 1 --2- 172.21.15.32:0/1519795573 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ab0078c20 0x7f4ab0079030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:54.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.773+0000 7f4ab4a00640 1 --2- 172.21.15.32:0/1519795573 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ab0077990 0x7f4ab00786e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:54.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.773+0000 7f4ab4a00640 1 -- 172.21.15.32:0/1519795573 >> 172.21.15.32:0/1519795573 conn(0x7f4ab0100da0 msgr2=0x7f4ab0103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:54.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.773+0000 7f4ab4a00640 1 -- 172.21.15.32:0/1519795573 shutdown_connections 2024-04-30T08:34:54.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.773+0000 7f4ab4a00640 1 -- 172.21.15.32:0/1519795573 wait complete. 2024-04-30T08:34:54.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.774+0000 7f4ab4a00640 1 Processor -- start 2024-04-30T08:34:54.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.774+0000 7f4ab4a00640 1 -- start start 2024-04-30T08:34:54.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.774+0000 7f4ab4a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ab0077990 0x7f4ab01a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:54.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.774+0000 7f4ab4a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ab0078c20 0x7f4ab01a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:54.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.775+0000 7f4ab4a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ab0079920 0x7f4ab01a7b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:54.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.775+0000 7f4ab4a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab0115ac0 con 0x7f4ab0077990 2024-04-30T08:34:54.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.775+0000 7f4ab4a00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab0115940 con 0x7f4ab0078c20 2024-04-30T08:34:54.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.775+0000 7f4ab4a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4ab0115c40 con 0x7f4ab0079920 2024-04-30T08:34:54.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.775+0000 7f4aae800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ab0077990 0x7f4ab01a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:54.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.775+0000 7f4aae800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ab0077990 0x7f4ab01a0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43426/0 (socket says 172.21.15.32:43426) 2024-04-30T08:34:54.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.775+0000 7f4aae800640 1 -- 172.21.15.32:0/109944557 learned_addr learned my addr 172.21.15.32:0/109944557 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:54.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.775+0000 7f4aaf200640 1 --2- 172.21.15.32:0/109944557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ab0079920 0x7f4ab01a7b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:54.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.775+0000 7f4aade00640 1 --2- 172.21.15.32:0/109944557 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ab0078c20 0x7f4ab01a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:54.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.775+0000 7f4aae800640 1 -- 172.21.15.32:0/109944557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ab0079920 msgr2=0x7f4ab01a7b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:54.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.775+0000 7f4aae800640 1 --2- 172.21.15.32:0/109944557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ab0079920 0x7f4ab01a7b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:54.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.775+0000 7f4aae800640 1 -- 172.21.15.32:0/109944557 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ab0078c20 msgr2=0x7f4ab01a0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:54.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.775+0000 7f4aae800640 1 --2- 172.21.15.32:0/109944557 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ab0078c20 0x7f4ab01a0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:54.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.775+0000 7f4aae800640 1 -- 172.21.15.32:0/109944557 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ab01a8180 con 0x7f4ab0077990 2024-04-30T08:34:54.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.776+0000 7f4aae800640 1 --2- 172.21.15.32:0/109944557 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ab0077990 0x7f4ab01a0640 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4aa000e9f0 tx=0x7f4aa000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:54.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.776+0000 7f4aa7400640 1 -- 172.21.15.32:0/109944557 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa000ccc0 con 0x7f4ab0077990 2024-04-30T08:34:54.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.776+0000 7f4ab4a00640 1 -- 172.21.15.32:0/109944557 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ab01a8470 con 0x7f4ab0077990 2024-04-30T08:34:54.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.776+0000 7f4aa7400640 1 -- 172.21.15.32:0/109944557 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4aa00108e0 con 0x7f4ab0077990 2024-04-30T08:34:54.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.776+0000 7f4ab4a00640 1 -- 172.21.15.32:0/109944557 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ab01a89c0 con 0x7f4ab0077990 2024-04-30T08:34:54.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.776+0000 7f4aa7400640 1 -- 172.21.15.32:0/109944557 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4aa0018820 con 0x7f4ab0077990 2024-04-30T08:34:54.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.777+0000 7f4ab4a00640 1 -- 172.21.15.32:0/109944557 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ab0112170 con 0x7f4ab0077990 2024-04-30T08:34:54.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.780+0000 7f4aa7400640 1 -- 172.21.15.32:0/109944557 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f4aa0010430 con 0x7f4ab0077990 2024-04-30T08:34:54.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.781+0000 7f4aa7400640 1 --2- 172.21.15.32:0/109944557 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4a940afae0 0x7f4a940b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:54.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.781+0000 7f4aa7400640 1 -- 172.21.15.32:0/109944557 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f4aa00df410 con 0x7f4ab0077990 2024-04-30T08:34:54.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.782+0000 7f4aade00640 1 --2- 172.21.15.32:0/109944557 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4a940afae0 0x7f4a940b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:54.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.783+0000 7f4aade00640 1 --2- 172.21.15.32:0/109944557 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4a940afae0 0x7f4a940b1ff0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4ab01a16e0 tx=0x7f4a9c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:54.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.784+0000 7f4aa7400640 1 -- 172.21.15.32:0/109944557 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4aa00a7590 con 0x7f4ab0077990 2024-04-30T08:34:54.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.993+0000 7f4ab4a00640 1 -- 172.21.15.32:0/109944557 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4ab0063820 con 0x7f4ab0077990 2024-04-30T08:34:54.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:54.998+0000 7f4aa7400640 1 -- 172.21.15.32:0/109944557 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+24730 (secure 0 0 0) 0x7f4aa00ab890 con 0x7f4ab0077990 2024-04-30T08:34:55.000 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:34:55.000 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":85,"fsid":"98c0a670-06ca-11ef-bc93-c7b262605968","created":"2024-04-30T08:22:11.122875+0000","modified":"2024-04-30T08:33:31.872704+0000","last_up_change":"2024-04-30T08:30:50.350439+0000","last_in_change":"2024-04-30T08:30:30.210620+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-30T08:26:19.728331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-30T08:31:49.800542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-30T08:31:51.211320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-30T08:31:55.716625+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a7e060c8-1cbb-418f-9a03-06693f6b3f52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6803","nonce":2624700402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6805","nonce":2624700402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6809","nonce":2624700402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6807","nonce":2624700402}]},"public_addr":"172.21.15.32:6803/2624700402","cluster_addr":"172.21.15.32:6805/2624700402","heartbeat_back_addr":"172.21.15.32:6809/2624700402","heartbeat_front_addr":"172.21.15.32:6807/2624700402","state":["exists","up"]},{"osd":1,"uuid":"72ef2fb1-e5ca-48ef-aa3a-4796a64b922f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6801","nonce":444649711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6803","nonce":444649711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6807","nonce":444649711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6805","nonce":444649711}]},"public_addr":"172.21.15.71:6801/444649711","cluster_addr":"172.21.15.71:6803/444649711","heartbeat_back_addr":"172.21.15.71:6807/444649711","heartbeat_front_addr":"172.21.15.71:6805/444649711","state":["exists","up"]},{"osd":2,"uuid":"704e5e8a-935e-47c7-ae28-27119876ac09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6801","nonce":3255528671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6803","nonce":3255528671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6807","nonce":3255528671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6805","nonce":3255528671}]},"public_addr":"172.21.15.189:6801/3255528671","cluster_addr":"172.21.15.189:6803/3255528671","heartbeat_back_addr":"172.21.15.189:6807/3255528671","heartbeat_front_addr":"172.21.15.189:6805/3255528671","state":["exists","up"]},{"osd":3,"uuid":"575004f7-6559-4a62-863e-e33078b0eebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6811","nonce":1384702112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6813","nonce":1384702112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6817","nonce":1384702112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6815","nonce":1384702112}]},"public_addr":"172.21.15.32:6811/1384702112","cluster_addr":"172.21.15.32:6813/1384702112","heartbeat_back_addr":"172.21.15.32:6817/1384702112","heartbeat_front_addr":"172.21.15.32:6815/1384702112","state":["exists","up"]},{"osd":4,"uuid":"e1c1e894-074d-4c78-89ef-9e5a114268bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6809","nonce":4250534574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6811","nonce":4250534574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6815","nonce":4250534574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6813","nonce":4250534574}]},"public_addr":"172.21.15.71:6809/4250534574","cluster_addr":"172.21.15.71:6811/4250534574","heartbeat_back_addr":"172.21.15.71:6815/4250534574","heartbeat_front_addr":"172.21.15.71:6813/4250534574","state":["exists","up"]},{"osd":5,"uuid":"12c022d4-fac6-40c4-ae7a-691571bee5d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6809","nonce":4245346198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6811","nonce":4245346198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6815","nonce":4245346198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6813","nonce":4245346198}]},"public_addr":"172.21.15.189:6809/4245346198","cluster_addr":"172.21.15.189:6811/4245346198","heartbeat_back_addr":"172.21.15.189:6815/4245346198","heartbeat_front_addr":"172.21.15.189:6813/4245346198","state":["exists","up"]},{"osd":6,"uuid":"b413f8ee-3d4a-415a-a2ba-d68437b7330d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6819","nonce":1910389190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6821","nonce":1910389190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6825","nonce":1910389190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6823","nonce":1910389190}]},"public_addr":"172.21.15.32:6819/1910389190","cluster_addr":"172.21.15.32:6821/1910389190","heartbeat_back_addr":"172.21.15.32:6825/1910389190","heartbeat_front_addr":"172.21.15.32:6823/1910389190","state":["exists","up"]},{"osd":7,"uuid":"aac64fb3-fc13-4126-b842-7b06ad5ce238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6817","nonce":692607380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6819","nonce":692607380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6823","nonce":692607380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6821","nonce":692607380}]},"public_addr":"172.21.15.71:6817/692607380","cluster_addr":"172.21.15.71:6819/692607380","heartbeat_back_addr":"172.21.15.71:6823/692607380","heartbeat_front_addr":"172.21.15.71:6821/692607380","state":["exists","up"]},{"osd":8,"uuid":"cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6817","nonce":3973697069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6819","nonce":3973697069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6823","nonce":3973697069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6821","nonce":3973697069}]},"public_addr":"172.21.15.189:6817/3973697069","cluster_addr":"172.21.15.189:6819/3973697069","heartbeat_back_addr":"172.21.15.189:6823/3973697069","heartbeat_front_addr":"172.21.15.189:6821/3973697069","state":["exists","up"]},{"osd":9,"uuid":"ef4a1645-ae6e-45c3-82ad-161510e62aa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6827","nonce":46066685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6829","nonce":46066685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6833","nonce":46066685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6831","nonce":46066685}]},"public_addr":"172.21.15.32:6827/46066685","cluster_addr":"172.21.15.32:6829/46066685","heartbeat_back_addr":"172.21.15.32:6833/46066685","heartbeat_front_addr":"172.21.15.32:6831/46066685","state":["exists","up"]},{"osd":10,"uuid":"25b26a38-a644-409f-b941-59efdf04a3ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6825","nonce":1886290293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6827","nonce":1886290293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6831","nonce":1886290293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6829","nonce":1886290293}]},"public_addr":"172.21.15.71:6825/1886290293","cluster_addr":"172.21.15.71:6827/1886290293","heartbeat_back_addr":"172.21.15.71:6831/1886290293","heartbeat_front_addr":"172.21.15.71:6829/1886290293","state":["exists","up"]},{"osd":11,"uuid":"ab96b3ff-3688-455b-af64-60226c5bfbff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6825","nonce":850185001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6827","nonce":850185001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6831","nonce":850185001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6829","nonce":850185001}]},"public_addr":"172.21.15.189:6825/850185001","cluster_addr":"172.21.15.189:6827/850185001","heartbeat_back_addr":"172.21.15.189:6831/850185001","heartbeat_front_addr":"172.21.15.189:6829/850185001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:27.220308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:53.230783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:17.190642+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:52.603799+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:20.568671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:46.158363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:22.706933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:50.665173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:16.602410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:53.814698+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:22.435189+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:48.651949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.e","mappings":[{"from":4,"to":7}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.28","mappings":[{"from":3,"to":6},{"from":5,"to":8}]},{"pgid":"3.30","mappings":[{"from":1,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1266897222":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/2915689533":"2024-05-01T08:23:29.526664+0000","172.21.15.32:6801/4019351306":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/425836193":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/1134075722":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6801/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/21059253":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/3245550894":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/4019351306":"2024-05-01T08:23:29.526664+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-30T08:34:55.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.000+0000 7f4ab4a00640 1 -- 172.21.15.32:0/109944557 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4a940afae0 msgr2=0x7f4a940b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:55.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.000+0000 7f4ab4a00640 1 --2- 172.21.15.32:0/109944557 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4a940afae0 0x7f4a940b1ff0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4ab01a16e0 tx=0x7f4a9c005f90 comp rx=0 tx=0).stop 2024-04-30T08:34:55.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.000+0000 7f4ab4a00640 1 -- 172.21.15.32:0/109944557 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ab0077990 msgr2=0x7f4ab01a0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:55.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.000+0000 7f4ab4a00640 1 --2- 172.21.15.32:0/109944557 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ab0077990 0x7f4ab01a0640 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f4aa000e9f0 tx=0x7f4aa000eeb0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.001+0000 7f4ab4a00640 1 -- 172.21.15.32:0/109944557 shutdown_connections 2024-04-30T08:34:55.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.001+0000 7f4ab4a00640 1 --2- 172.21.15.32:0/109944557 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4a940afae0 0x7f4a940b1ff0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.001+0000 7f4ab4a00640 1 --2- 172.21.15.32:0/109944557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ab0079920 0x7f4ab01a7b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.001+0000 7f4ab4a00640 1 --2- 172.21.15.32:0/109944557 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4ab0078c20 0x7f4ab01a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.001+0000 7f4ab4a00640 1 --2- 172.21.15.32:0/109944557 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4ab0077990 0x7f4ab01a0640 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.001+0000 7f4ab4a00640 1 -- 172.21.15.32:0/109944557 >> 172.21.15.32:0/109944557 conn(0x7f4ab0100da0 msgr2=0x7f4ab01013d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:55.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.001+0000 7f4ab4a00640 1 -- 172.21.15.32:0/109944557 shutdown_connections 2024-04-30T08:34:55.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.001+0000 7f4ab4a00640 1 -- 172.21.15.32:0/109944557 wait complete. 2024-04-30T08:34:55.030 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-30T08:31:52.236668+0000', 'modified': '2024-04-30T08:32:30.541714+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24590, 'mds_1': 24560, 'mds_2': 24541, 'mds_3': 24533, 'mds_4': 24517}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24442': {'gid': 24442, 'name': 'l', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.189:6833/463203038', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6832', 'nonce': 463203038}, {'type': 'v1', 'addr': '172.21.15.189:6833', 'nonce': 463203038}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24458': {'gid': 24458, 'name': 'k', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.71:6833/3635143591', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6832', 'nonce': 3635143591}, {'type': 'v1', 'addr': '172.21.15.71:6833', 'nonce': 3635143591}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24484': {'gid': 24484, 'name': 'd', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.32:6837/2569449994', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6836', 'nonce': 2569449994}, {'type': 'v1', 'addr': '172.21.15.32:6837', 'nonce': 2569449994}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24505': {'gid': 24505, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.189:6835/278686565', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6834', 'nonce': 278686565}, {'type': 'v1', 'addr': '172.21.15.189:6835', 'nonce': 278686565}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24506': {'gid': 24506, 'name': 'e', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.71:6835/1558303056', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6834', 'nonce': 1558303056}, {'type': 'v1', 'addr': '172.21.15.71:6835', 'nonce': 1558303056}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24517': {'gid': 24517, 'name': 'c', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 17, 'addr': '172.21.15.189:6837/686520873', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6836', 'nonce': 686520873}, {'type': 'v1', 'addr': '172.21.15.189:6837', 'nonce': 686520873}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24533': {'gid': 24533, 'name': 'h', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 16, 'addr': '172.21.15.71:6837/2324039932', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6836', 'nonce': 2324039932}, {'type': 'v1', 'addr': '172.21.15.71:6837', 'nonce': 2324039932}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24541': {'gid': 24541, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 15, 'addr': '172.21.15.189:6839/352970107', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6838', 'nonce': 352970107}, {'type': 'v1', 'addr': '172.21.15.189:6839', 'nonce': 352970107}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24560': {'gid': 24560, 'name': 'g', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 14, 'addr': '172.21.15.32:6839/4160386524', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.32:6838', 'nonce': 4160386524}, {'type': 'v1', 'addr': '172.21.15.32:6839', 'nonce': 4160386524}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24590': {'gid': 24590, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.71:6839/3517409993', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.71:6838', 'nonce': 3517409993}, {'type': 'v1', 'addr': '172.21.15.71:6839', 'nonce': 3517409993}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24590, 'qdb_cluster': [24590, 24541, 24560, 24533, 24517]} 2024-04-30T08:34:55.030 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-04-30T08:34:55.031 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-04-30T08:34:55.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.317+0000 7fa657000640 1 -- 172.21.15.32:0/1536351705 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa6500760c0 msgr2=0x7fa6500764d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:55.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.317+0000 7fa657000640 1 --2- 172.21.15.32:0/1536351705 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa6500760c0 0x7fa6500764d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fa640009ca0 tx=0x7fa64002a8b0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.317+0000 7fa657000640 1 -- 172.21.15.32:0/1536351705 shutdown_connections 2024-04-30T08:34:55.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.317+0000 7fa657000640 1 --2- 172.21.15.32:0/1536351705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa65010bb40 0x7fa65010df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.317+0000 7fa657000640 1 --2- 172.21.15.32:0/1536351705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa650076a10 0x7fa65010b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.317+0000 7fa657000640 1 --2- 172.21.15.32:0/1536351705 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa6500760c0 0x7fa6500764d0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.317+0000 7fa657000640 1 -- 172.21.15.32:0/1536351705 >> 172.21.15.32:0/1536351705 conn(0x7fa6500fed60 msgr2=0x7fa650101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:55.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.317+0000 7fa657000640 1 -- 172.21.15.32:0/1536351705 shutdown_connections 2024-04-30T08:34:55.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.317+0000 7fa657000640 1 -- 172.21.15.32:0/1536351705 wait complete. 2024-04-30T08:34:55.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.319+0000 7fa657000640 1 Processor -- start 2024-04-30T08:34:55.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.319+0000 7fa657000640 1 -- start start 2024-04-30T08:34:55.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.319+0000 7fa657000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6500760c0 0x7fa65019c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.319+0000 7fa657000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa650076a10 0x7fa65019c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.319+0000 7fa657000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa65010bb40 0x7fa6501a3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.319+0000 7fa657000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa6501118a0 con 0x7fa6500760c0 2024-04-30T08:34:55.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.319+0000 7fa657000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa650111720 con 0x7fa650076a10 2024-04-30T08:34:55.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.319+0000 7fa657000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa650111a20 con 0x7fa65010bb40 2024-04-30T08:34:55.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.320+0000 7fa64fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6500760c0 0x7fa65019c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:55.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.320+0000 7fa655000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa65010bb40 0x7fa6501a3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:55.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.320+0000 7fa64fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6500760c0 0x7fa65019c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43450/0 (socket says 172.21.15.32:43450) 2024-04-30T08:34:55.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.320+0000 7fa64fe00640 1 -- 172.21.15.32:0/1037103887 learned_addr learned my addr 172.21.15.32:0/1037103887 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:55.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.320+0000 7fa64f400640 1 --2- 172.21.15.32:0/1037103887 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa650076a10 0x7fa65019c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:55.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.320+0000 7fa655000640 1 -- 172.21.15.32:0/1037103887 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa650076a10 msgr2=0x7fa65019c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:55.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.320+0000 7fa655000640 1 --2- 172.21.15.32:0/1037103887 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa650076a10 0x7fa65019c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.320+0000 7fa655000640 1 -- 172.21.15.32:0/1037103887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6500760c0 msgr2=0x7fa65019c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:55.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.320+0000 7fa655000640 1 --2- 172.21.15.32:0/1037103887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6500760c0 0x7fa65019c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.320+0000 7fa655000640 1 -- 172.21.15.32:0/1037103887 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6501a3fb0 con 0x7fa65010bb40 2024-04-30T08:34:55.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.321+0000 7fa655000640 1 --2- 172.21.15.32:0/1037103887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa65010bb40 0x7fa6501a3940 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa64800e970 tx=0x7fa64800ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:55.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.321+0000 7fa64cc00640 1 -- 172.21.15.32:0/1037103887 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa64800ccc0 con 0x7fa65010bb40 2024-04-30T08:34:55.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.321+0000 7fa657000640 1 -- 172.21.15.32:0/1037103887 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6501a42a0 con 0x7fa65010bb40 2024-04-30T08:34:55.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.321+0000 7fa657000640 1 -- 172.21.15.32:0/1037103887 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6501a47f0 con 0x7fa65010bb40 2024-04-30T08:34:55.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.321+0000 7fa64cc00640 1 -- 172.21.15.32:0/1037103887 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6480044d0 con 0x7fa65010bb40 2024-04-30T08:34:55.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.321+0000 7fa64cc00640 1 -- 172.21.15.32:0/1037103887 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa64801c070 con 0x7fa65010bb40 2024-04-30T08:34:55.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.325+0000 7fa65e184740 1 Processor -- start 2024-04-30T08:34:55.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.325+0000 7fa64cc00640 1 -- 172.21.15.32:0/1037103887 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fa648004020 con 0x7fa65010bb40 2024-04-30T08:34:55.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.325+0000 7fa65e184740 1 -- start start 2024-04-30T08:34:55.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.325+0000 7fa65e184740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555724651270 0x5557249f3340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.325+0000 7fa65e184740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557246283b0 0x5557249f38c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.325+0000 7fa65e184740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555724630590 0x5557249f4550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.326+0000 7fa65e184740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555724957790 con 0x555724651270 2024-04-30T08:34:55.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.326+0000 7fa65e184740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x5557248fcf20 con 0x555724630590 2024-04-30T08:34:55.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.326+0000 7fa65e184740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555724953750 con 0x5557246283b0 2024-04-30T08:34:55.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.326+0000 7fa64f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555724651270 0x5557249f3340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:55.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.326+0000 7fa64f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555724651270 0x5557249f3340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43460/0 (socket says 172.21.15.32:43460) 2024-04-30T08:34:55.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.326+0000 7fa64f400640 1 -- 172.21.15.32:0/175795753 learned_addr learned my addr 172.21.15.32:0/175795753 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:55.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.326+0000 7fa64fe00640 1 --2- 172.21.15.32:0/175795753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557246283b0 0x5557249f38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:55.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.326+0000 7fa64f400640 1 --2- 172.21.15.32:0/175795753 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555724630590 0x5557249f4550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:55.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.326+0000 7fa64f400640 1 -- 172.21.15.32:0/175795753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557246283b0 msgr2=0x5557249f38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:55.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.326+0000 7fa64f400640 1 --2- 172.21.15.32:0/175795753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557246283b0 0x5557249f38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.326+0000 7fa64f400640 1 -- 172.21.15.32:0/175795753 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555724630590 msgr2=0x5557249f4550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:55.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.326+0000 7fa64f400640 1 --2- 172.21.15.32:0/175795753 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555724630590 0x5557249f4550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.326+0000 7fa64f400640 1 -- 172.21.15.32:0/175795753 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557246692d0 con 0x555724651270 2024-04-30T08:34:55.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.327+0000 7fa64f400640 1 --2- 172.21.15.32:0/175795753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555724651270 0x5557249f3340 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa63c013620 tx=0x7fa63c009850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c159b996ce04e65 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.327+0000 7fa64cc00640 1 --2- 172.21.15.32:0/1037103887 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa6340afaf0 0x7fa6340b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.327+0000 7fa645600640 1 -- 172.21.15.32:0/175795753 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63c00f580 con 0x555724651270 2024-04-30T08:34:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.327+0000 7fa645600640 1 -- 172.21.15.32:0/175795753 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa63c00fbd0 con 0x555724651270 2024-04-30T08:34:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.327+0000 7fa645600640 1 -- 172.21.15.32:0/175795753 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa63c010b00 con 0x555724651270 2024-04-30T08:34:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.327+0000 7fa64cc00640 1 -- 172.21.15.32:0/1037103887 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fa6480deff0 con 0x7fa65010bb40 2024-04-30T08:34:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.327+0000 7fa64fe00640 1 --2- 172.21.15.32:0/1037103887 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa6340afaf0 0x7fa6340b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.328+0000 7fa65e184740 1 -- 172.21.15.32:0/175795753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555724651270 msgr2=0x5557249f3340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.328+0000 7fa65e184740 1 --2- 172.21.15.32:0/175795753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555724651270 0x5557249f3340 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fa63c013620 tx=0x7fa63c009850 comp rx=0 tx=0).stop 2024-04-30T08:34:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.328+0000 7fa65e184740 1 -- 172.21.15.32:0/175795753 shutdown_connections 2024-04-30T08:34:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.328+0000 7fa65e184740 1 --2- 172.21.15.32:0/175795753 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555724630590 0x5557249f4550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.328+0000 7fa65e184740 1 --2- 172.21.15.32:0/175795753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557246283b0 0x5557249f38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.328+0000 7fa65e184740 1 --2- 172.21.15.32:0/175795753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555724651270 0x5557249f3340 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.328+0000 7fa65e184740 1 -- 172.21.15.32:0/175795753 >> 172.21.15.32:0/175795753 conn(0x5557246453c0 msgr2=0x5557249eeb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.328+0000 7fa65e184740 1 -- 172.21.15.32:0/175795753 shutdown_connections 2024-04-30T08:34:55.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.328+0000 7fa65e184740 1 -- 172.21.15.32:0/175795753 wait complete. 2024-04-30T08:34:55.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.328+0000 7fa64fe00640 1 --2- 172.21.15.32:0/1037103887 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa6340afaf0 0x7fa6340b2000 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa640031040 tx=0x7fa640030760 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:55.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.329+0000 7fa65e184740 1 -- start start 2024-04-30T08:34:55.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.329+0000 7fa65e184740 1 Processor -- start 2024-04-30T08:34:55.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.329+0000 7fa65e184740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557246283b0 0x5557249fce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.329+0000 7fa65e184740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555724630590 0x5557249fd370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.329+0000 7fa65e184740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555724651270 0x555724a042d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.329+0000 7fa65e184740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5557249f5000 con 0x555724651270 2024-04-30T08:34:55.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.329+0000 7fa65e184740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x5557249109c0 con 0x555724630590 2024-04-30T08:34:55.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.329+0000 7fa65e184740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5557249f5180 con 0x5557246283b0 2024-04-30T08:34:55.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.330+0000 7fa65e184740 10 client.0 _ll_register_callbacks cb 0x5557248dd120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:34:55.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.330+0000 7fa655000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555724651270 0x555724a042d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:55.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.330+0000 7fa655000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555724651270 0x555724a042d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43474/0 (socket says 172.21.15.32:43474) 2024-04-30T08:34:55.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.330+0000 7fa655000640 1 -- 172.21.15.32:0/2305102457 learned_addr learned my addr 172.21.15.32:0/2305102457 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:55.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.330+0000 7fa64f400640 1 --2- 172.21.15.32:0/2305102457 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555724630590 0x5557249fd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:55.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.330+0000 7fa64fe00640 1 --2- 172.21.15.32:0/2305102457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557246283b0 0x5557249fce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:55.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.330+0000 7fa655000640 1 -- 172.21.15.32:0/2305102457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557246283b0 msgr2=0x5557249fce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:55.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.330+0000 7fa655000640 1 --2- 172.21.15.32:0/2305102457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557246283b0 0x5557249fce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.330+0000 7fa655000640 1 -- 172.21.15.32:0/2305102457 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555724630590 msgr2=0x5557249fd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:55.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.330+0000 7fa655000640 1 --2- 172.21.15.32:0/2305102457 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555724630590 0x5557249fd370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.330+0000 7fa655000640 1 -- 172.21.15.32:0/2305102457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5557247336b0 con 0x555724651270 2024-04-30T08:34:55.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.330+0000 7fa64f400640 1 --2- 172.21.15.32:0/2305102457 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555724630590 0x5557249fd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:34:55.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.331+0000 7fa655000640 1 --2- 172.21.15.32:0/2305102457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555724651270 0x555724a042d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa6480a3140 tx=0x7fa6480a68a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b847833bdcdb8e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:55.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.331+0000 7fa63a000640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:34:55.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.331+0000 7fa63a000640 1 -- 172.21.15.32:0/2305102457 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6480a77c0 con 0x555724651270 2024-04-30T08:34:55.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.331+0000 7fa63a000640 1 -- 172.21.15.32:0/2305102457 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fa6480afca0 con 0x555724651270 2024-04-30T08:34:55.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.331+0000 7fa65e184740 1 -- 172.21.15.32:0/2305102457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa63c00fbd0 con 0x555724651270 2024-04-30T08:34:55.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.331+0000 7fa63a000640 1 -- 172.21.15.32:0/2305102457 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa6480a70c0 con 0x555724651270 2024-04-30T08:34:55.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.331+0000 7fa65e184740 10 client.15267 fetch_fsmap learned FSMap version 27 2024-04-30T08:34:55.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.331+0000 7fa65e184740 1 -- 172.21.15.32:0/2305102457 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555724a07300 con 0x555724651270 2024-04-30T08:34:55.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.331+0000 7fa63a000640 1 -- 172.21.15.32:0/2305102457 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa63c00fbd0 con 0x555724651270 2024-04-30T08:34:55.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.332+0000 7fa63a000640 1 -- 172.21.15.32:0/2305102457 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fa6480dcba0 con 0x555724651270 2024-04-30T08:34:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.332+0000 7fa65e184740 10 client.15267 fetch_fsmap finished waiting for FSMap version 27 2024-04-30T08:34:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.332+0000 7fa65e184740 10 client.15267 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-30T08:34:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.332+0000 7fa65e184740 20 client.15267 populate_metadata read hostname 'smithi032' 2024-04-30T08:34:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.332+0000 7fa65e184740 1 --2- 172.21.15.32:0/2305102457 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x55572460fe40 0x555724a0ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.332+0000 7fa65e184740 4 client.15267 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:34:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.332+0000 7fa65e184740 1 -- 172.21.15.32:0/2305102457 --> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5557249653a0 con 0x55572460fe40 2024-04-30T08:34:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.332+0000 7fa64fe00640 1 --2- 172.21.15.32:0/2305102457 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x55572460fe40 0x555724a0ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.333+0000 7fa64fe00640 1 --2- 172.21.15.32:0/2305102457 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x55572460fe40 0x555724a0ae40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2bc9fcca04bf209b server_cookie=8b7183d2e5476e92 in_seq=0 out_seq=0 2024-04-30T08:34:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.333+0000 7fa63a000640 10 client.15267 ms_handle_connect on v2:172.21.15.189:6836/686520873 2024-04-30T08:34:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.334+0000 7fa63a000640 1 -- 172.21.15.32:0/2305102457 <== mds.4 v2:172.21.15.189:6836/686520873 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5557249653a0 con 0x55572460fe40 2024-04-30T08:34:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.334+0000 7fa63a000640 10 client.15267 handle_command_reply: tid=0 2024-04-30T08:34:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.334+0000 7fa65e184740 1 client.15267 shutdown 2024-04-30T08:34:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.334+0000 7fa65e184740 1 -- 172.21.15.32:0/2305102457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555724651270 msgr2=0x555724a042d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:55.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.334+0000 7fa65e184740 1 --2- 172.21.15.32:0/2305102457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555724651270 0x555724a042d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fa6480a3140 tx=0x7fa6480a68a0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.334+0000 7fa65e184740 1 -- 172.21.15.32:0/2305102457 shutdown_connections 2024-04-30T08:34:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.334+0000 7fa65e184740 1 --2- 172.21.15.32:0/2305102457 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x55572460fe40 0x555724a0ae40 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.334+0000 7fa65e184740 1 --2- 172.21.15.32:0/2305102457 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555724651270 0x555724a042d0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.334+0000 7fa65e184740 1 --2- 172.21.15.32:0/2305102457 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555724630590 0x5557249fd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.334+0000 7fa63a000640 0 client.15267 ms_handle_reset on v2:172.21.15.189:6836/686520873 2024-04-30T08:34:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.334+0000 7fa65e184740 1 --2- 172.21.15.32:0/2305102457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557246283b0 0x5557249fce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.334+0000 7fa65e184740 1 -- 172.21.15.32:0/2305102457 >> 172.21.15.32:0/2305102457 conn(0x5557246453c0 msgr2=0x5557249f1a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.334+0000 7fa65e184740 1 -- 172.21.15.32:0/2305102457 shutdown_connections 2024-04-30T08:34:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.334+0000 7fa65e184740 1 -- 172.21.15.32:0/2305102457 wait complete. 2024-04-30T08:34:55.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.335+0000 7fa65e184740 20 client.15267 trim_cache size 0 max 16384 2024-04-30T08:34:55.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.355+0000 7fa65e184740 1 Processor -- start 2024-04-30T08:34:55.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.355+0000 7fa65e184740 1 -- start start 2024-04-30T08:34:55.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.356+0000 7fa65e184740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557246283b0 0x555724a087c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.356+0000 7fa65e184740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555724630590 0x555724a041f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.356+0000 7fa65e184740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555724651270 0x5557249fca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.356+0000 7fa65e184740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555724a09180 con 0x555724651270 2024-04-30T08:34:55.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.356+0000 7fa65e184740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x555724a09850 con 0x555724630590 2024-04-30T08:34:55.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.356+0000 7fa65e184740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555724a09f20 con 0x5557246283b0 2024-04-30T08:34:55.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.356+0000 7fa655000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555724651270 0x5557249fca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:55.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.356+0000 7fa655000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555724651270 0x5557249fca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43484/0 (socket says 172.21.15.32:43484) 2024-04-30T08:34:55.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.356+0000 7fa64f400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555724630590 0x555724a041f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:55.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.356+0000 7fa655000640 1 -- 172.21.15.32:0/4244254537 learned_addr learned my addr 172.21.15.32:0/4244254537 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:55.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.356+0000 7fa64fe00640 1 --2- 172.21.15.32:0/4244254537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557246283b0 0x555724a087c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:55.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.357+0000 7fa655000640 1 -- 172.21.15.32:0/4244254537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557246283b0 msgr2=0x555724a087c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:55.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.357+0000 7fa655000640 1 --2- 172.21.15.32:0/4244254537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557246283b0 0x555724a087c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.357+0000 7fa655000640 1 -- 172.21.15.32:0/4244254537 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555724630590 msgr2=0x555724a041f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:55.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.357+0000 7fa655000640 1 --2- 172.21.15.32:0/4244254537 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555724630590 0x555724a041f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.357+0000 7fa655000640 1 -- 172.21.15.32:0/4244254537 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5557249faf20 con 0x555724651270 2024-04-30T08:34:55.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.357+0000 7fa64fe00640 1 --2- 172.21.15.32:0/4244254537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557246283b0 0x555724a087c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:34:55.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.357+0000 7fa655000640 1 --2- 172.21.15.32:0/4244254537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555724651270 0x5557249fca00 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa64801a040 tx=0x7fa6480a6d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e31c118683189f79 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:55.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.357+0000 7fa644c00640 1 -- 172.21.15.32:0/4244254537 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6480a77c0 con 0x555724651270 2024-04-30T08:34:55.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.357+0000 7fa644c00640 1 -- 172.21.15.32:0/4244254537 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6480ab470 con 0x555724651270 2024-04-30T08:34:55.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.357+0000 7fa644c00640 1 -- 172.21.15.32:0/4244254537 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6480a2a30 con 0x555724651270 2024-04-30T08:34:55.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.358+0000 7fa65e184740 1 -- 172.21.15.32:0/4244254537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555724651270 msgr2=0x5557249fca00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:55.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.358+0000 7fa65e184740 1 --2- 172.21.15.32:0/4244254537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555724651270 0x5557249fca00 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa64801a040 tx=0x7fa6480a6d10 comp rx=0 tx=0).stop 2024-04-30T08:34:55.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.358+0000 7fa65e184740 1 -- 172.21.15.32:0/4244254537 shutdown_connections 2024-04-30T08:34:55.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.358+0000 7fa65e184740 1 --2- 172.21.15.32:0/4244254537 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555724651270 0x5557249fca00 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.358+0000 7fa65e184740 1 --2- 172.21.15.32:0/4244254537 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555724630590 0x555724a041f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.358+0000 7fa65e184740 1 --2- 172.21.15.32:0/4244254537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557246283b0 0x555724a087c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.358+0000 7fa65e184740 1 -- 172.21.15.32:0/4244254537 >> 172.21.15.32:0/4244254537 conn(0x5557246453c0 msgr2=0x5557249f3300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:55.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.359+0000 7fa65e184740 1 -- 172.21.15.32:0/4244254537 shutdown_connections 2024-04-30T08:34:55.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.359+0000 7fa65e184740 1 -- 172.21.15.32:0/4244254537 wait complete. 2024-04-30T08:34:55.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.359+0000 7fa65e184740 1 -- start start 2024-04-30T08:34:55.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.359+0000 7fa65e184740 1 Processor -- start 2024-04-30T08:34:55.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.360+0000 7fa65e184740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557246283b0 0x5557249ff2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.360+0000 7fa65e184740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555724630590 0x5557249ff830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.360+0000 7fa65e184740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555724651270 0x555724a71cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.360+0000 7fa65e184740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555724a0acc0 con 0x555724651270 2024-04-30T08:34:55.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.360+0000 7fa65e184740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x555724a0a5f0 con 0x555724630590 2024-04-30T08:34:55.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.360+0000 7fa65e184740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555724a0b390 con 0x5557246283b0 2024-04-30T08:34:55.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.360+0000 7fa65e184740 10 client.0 _ll_register_callbacks cb 0x7fa64800ce60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:34:55.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.360+0000 7fa655000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555724651270 0x555724a71cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:55.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.360+0000 7fa655000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555724651270 0x555724a71cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43488/0 (socket says 172.21.15.32:43488) 2024-04-30T08:34:55.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.360+0000 7fa655000640 1 -- 172.21.15.32:0/3523592483 learned_addr learned my addr 172.21.15.32:0/3523592483 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:55.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.360+0000 7fa64f400640 1 --2- 172.21.15.32:0/3523592483 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555724630590 0x5557249ff830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:55.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.360+0000 7fa64fe00640 1 --2- 172.21.15.32:0/3523592483 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557246283b0 0x5557249ff2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:55.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.360+0000 7fa655000640 1 -- 172.21.15.32:0/3523592483 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557246283b0 msgr2=0x5557249ff2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:55.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.360+0000 7fa655000640 1 --2- 172.21.15.32:0/3523592483 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557246283b0 0x5557249ff2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.361+0000 7fa655000640 1 -- 172.21.15.32:0/3523592483 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555724630590 msgr2=0x5557249ff830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:55.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.361+0000 7fa655000640 1 --2- 172.21.15.32:0/3523592483 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555724630590 0x5557249ff830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.361+0000 7fa655000640 1 -- 172.21.15.32:0/3523592483 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555724a72220 con 0x555724651270 2024-04-30T08:34:55.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.361+0000 7fa655000640 1 --2- 172.21.15.32:0/3523592483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555724651270 0x555724a71cb0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa64800cf50 tx=0x7fa6480a6830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffe13aaeb8265c31 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:55.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.361+0000 7fa63a000640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:34:55.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.361+0000 7fa63a000640 1 -- 172.21.15.32:0/3523592483 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6480b1020 con 0x555724651270 2024-04-30T08:34:55.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.361+0000 7fa63a000640 1 -- 172.21.15.32:0/3523592483 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fa64808e120 con 0x555724651270 2024-04-30T08:34:55.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.361+0000 7fa65e184740 1 -- 172.21.15.32:0/3523592483 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa6480ab470 con 0x555724651270 2024-04-30T08:34:55.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.361+0000 7fa63a000640 1 -- 172.21.15.32:0/3523592483 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fa6480b9050 con 0x555724651270 2024-04-30T08:34:55.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.361+0000 7fa65e184740 10 client.15273 fetch_fsmap learned FSMap version 27 2024-04-30T08:34:55.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.361+0000 7fa65e184740 1 -- 172.21.15.32:0/3523592483 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x555724a73480 con 0x555724651270 2024-04-30T08:34:55.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.362+0000 7fa63a000640 1 -- 172.21.15.32:0/3523592483 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa6480ab470 con 0x555724651270 2024-04-30T08:34:55.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.362+0000 7fa63a000640 1 -- 172.21.15.32:0/3523592483 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fa6480a2a30 con 0x555724651270 2024-04-30T08:34:55.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.362+0000 7fa65e184740 10 client.15273 fetch_fsmap finished waiting for FSMap version 27 2024-04-30T08:34:55.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.362+0000 7fa65e184740 10 client.15273 resolve_mds: resolved name 'c' to daemon mds.c 2024-04-30T08:34:55.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.362+0000 7fa65e184740 20 client.15273 populate_metadata read hostname 'smithi032' 2024-04-30T08:34:55.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.362+0000 7fa65e184740 1 --2- 172.21.15.32:0/3523592483 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x5557245e5280 0x555724a745b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.362+0000 7fa65e184740 4 client.15273 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-30T08:34:55.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.362+0000 7fa65e184740 1 -- 172.21.15.32:0/3523592483 --> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] -- command(tid 0: {"prefix": "status"}) -- 0x55572495f210 con 0x5557245e5280 2024-04-30T08:34:55.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.362+0000 7fa64fe00640 1 --2- 172.21.15.32:0/3523592483 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x5557245e5280 0x555724a745b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:55.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.363+0000 7fa64fe00640 1 --2- 172.21.15.32:0/3523592483 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x5557245e5280 0x555724a745b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1343ae3833b79b3f server_cookie=cf7f9a44c6d74c47 in_seq=0 out_seq=0 2024-04-30T08:34:55.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.363+0000 7fa63a000640 10 client.15273 ms_handle_connect on v2:172.21.15.189:6836/686520873 2024-04-30T08:34:55.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.363+0000 7fa63a000640 1 -- 172.21.15.32:0/3523592483 <== mds.4 v2:172.21.15.189:6836/686520873 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55572495f210 con 0x5557245e5280 2024-04-30T08:34:55.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.363+0000 7fa63a000640 10 client.15273 handle_command_reply: tid=0 2024-04-30T08:34:55.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.363+0000 7fa65e184740 1 client.15273 shutdown 2024-04-30T08:34:55.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.363+0000 7fa65e184740 1 -- 172.21.15.32:0/3523592483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555724651270 msgr2=0x555724a71cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:55.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.363+0000 7fa65e184740 1 --2- 172.21.15.32:0/3523592483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555724651270 0x555724a71cb0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa64800cf50 tx=0x7fa6480a6830 comp rx=0 tx=0).stop 2024-04-30T08:34:55.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.363+0000 7fa65e184740 1 -- 172.21.15.32:0/3523592483 shutdown_connections 2024-04-30T08:34:55.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.363+0000 7fa65e184740 1 --2- 172.21.15.32:0/3523592483 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x5557245e5280 0x555724a745b0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.363+0000 7fa65e184740 1 --2- 172.21.15.32:0/3523592483 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555724651270 0x555724a71cb0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.363+0000 7fa65e184740 1 --2- 172.21.15.32:0/3523592483 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555724630590 0x5557249ff830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.363+0000 7fa65e184740 1 --2- 172.21.15.32:0/3523592483 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5557246283b0 0x5557249ff2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.363+0000 7fa63a000640 0 client.15273 ms_handle_reset on v2:172.21.15.189:6836/686520873 2024-04-30T08:34:55.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.363+0000 7fa65e184740 1 -- 172.21.15.32:0/3523592483 >> 172.21.15.32:0/3523592483 conn(0x5557246453c0 msgr2=0x555724a087c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:55.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.364+0000 7fa65e184740 1 -- 172.21.15.32:0/3523592483 shutdown_connections 2024-04-30T08:34:55.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.364+0000 7fa65e184740 1 -- 172.21.15.32:0/3523592483 wait complete. 2024-04-30T08:34:55.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.364+0000 7fa65e184740 20 client.15273 trim_cache size 0 max 16384 2024-04-30T08:34:55.369 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-30T08:34:55.369 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "98c0a670-06ca-11ef-bc93-c7b262605968", 2024-04-30T08:34:55.369 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 4, 2024-04-30T08:34:55.369 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 24517, 2024-04-30T08:34:55.369 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-04-30T08:34:55.369 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-04-30T08:34:55.369 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-04-30T08:34:55.370 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 145.83129540799999, 2024-04-30T08:34:55.370 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 27, 2024-04-30T08:34:55.370 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 85, 2024-04-30T08:34:55.370 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 81, 2024-04-30T08:34:55.370 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 207.79022040999999 2024-04-30T08:34:55.370 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-30T08:34:55.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.364+0000 7fa657000640 1 -- 172.21.15.32:0/1037103887 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa6340afaf0 msgr2=0x7fa6340b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:55.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.364+0000 7fa657000640 1 --2- 172.21.15.32:0/1037103887 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa6340afaf0 0x7fa6340b2000 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fa640031040 tx=0x7fa640030760 comp rx=0 tx=0).stop 2024-04-30T08:34:55.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.364+0000 7fa657000640 1 -- 172.21.15.32:0/1037103887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa65010bb40 msgr2=0x7fa6501a3940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:55.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.364+0000 7fa657000640 1 --2- 172.21.15.32:0/1037103887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa65010bb40 0x7fa6501a3940 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fa64800e970 tx=0x7fa64800ee30 comp rx=0 tx=0).stop 2024-04-30T08:34:55.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.365+0000 7fa657000640 1 -- 172.21.15.32:0/1037103887 shutdown_connections 2024-04-30T08:34:55.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.365+0000 7fa657000640 1 --2- 172.21.15.32:0/1037103887 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa6340afaf0 0x7fa6340b2000 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.365+0000 7fa657000640 1 --2- 172.21.15.32:0/1037103887 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa65010bb40 0x7fa6501a3940 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.365+0000 7fa657000640 1 --2- 172.21.15.32:0/1037103887 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa650076a10 0x7fa65019c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.365+0000 7fa657000640 1 --2- 172.21.15.32:0/1037103887 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa6500760c0 0x7fa65019c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.365+0000 7fa657000640 1 -- 172.21.15.32:0/1037103887 >> 172.21.15.32:0/1037103887 conn(0x7fa6500fed60 msgr2=0x7fa6500ffba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:55.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.365+0000 7fa657000640 1 -- 172.21.15.32:0/1037103887 shutdown_connections 2024-04-30T08:34:55.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.365+0000 7fa657000640 1 -- 172.21.15.32:0/1037103887 wait complete. 2024-04-30T08:34:55.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:55 smithi032 ceph-mon[28331]: pgmap v417: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:34:55.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:55 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/464300092' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:34:55.391 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:55 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/109944557' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:34:55.394 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-04-30T08:34:55.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:55 smithi071 ceph-mon[33385]: pgmap v417: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:34:55.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:55 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/464300092' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:34:55.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:55 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/109944557' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:34:55.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:55 smithi189 ceph-mon[33038]: pgmap v417: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:34:55.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:55 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/464300092' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:34:55.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:55 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/109944557' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:34:55.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.679+0000 7fe2a0c00640 1 -- 172.21.15.32:0/1624789586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe29c104bd0 msgr2=0x7fe29c104fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:55.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.679+0000 7fe2a0c00640 1 --2- 172.21.15.32:0/1624789586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe29c104bd0 0x7fe29c104fe0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fe284009900 tx=0x7fe28402a400 comp rx=0 tx=0).stop 2024-04-30T08:34:55.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.679+0000 7fe2a0c00640 1 -- 172.21.15.32:0/1624789586 shutdown_connections 2024-04-30T08:34:55.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.679+0000 7fe2a0c00640 1 --2- 172.21.15.32:0/1624789586 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe29c1098d0 0x7fe29c10bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.679+0000 7fe2a0c00640 1 --2- 172.21.15.32:0/1624789586 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe29c104bd0 0x7fe29c104fe0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.679+0000 7fe2a0c00640 1 --2- 172.21.15.32:0/1624789586 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe29c103990 0x7fe29c104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.679+0000 7fe2a0c00640 1 -- 172.21.15.32:0/1624789586 >> 172.21.15.32:0/1624789586 conn(0x7fe29c0feba0 msgr2=0x7fe29c101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:55.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.680+0000 7fe2a0c00640 1 -- 172.21.15.32:0/1624789586 shutdown_connections 2024-04-30T08:34:55.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.680+0000 7fe2a0c00640 1 -- 172.21.15.32:0/1624789586 wait complete. 2024-04-30T08:34:55.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.681+0000 7fe2a0c00640 1 Processor -- start 2024-04-30T08:34:55.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.681+0000 7fe2a0c00640 1 -- start start 2024-04-30T08:34:55.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.681+0000 7fe2a0c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe29c103990 0x7fe29c19c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.681+0000 7fe2a0c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe29c104bd0 0x7fe29c19c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.681+0000 7fe2a0c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe29c1098d0 0x7fe29c1a3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.681+0000 7fe2a0c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe29c10f490 con 0x7fe29c103990 2024-04-30T08:34:55.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.681+0000 7fe2a0c00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe29c10f310 con 0x7fe29c104bd0 2024-04-30T08:34:55.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.681+0000 7fe2a0c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe29c10f610 con 0x7fe29c1098d0 2024-04-30T08:34:55.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.682+0000 7fe292a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe29c104bd0 0x7fe29c19c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:55.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.682+0000 7fe29b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe29c103990 0x7fe29c19c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:55.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.682+0000 7fe292a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe29c104bd0 0x7fe29c19c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:46426/0 (socket says 172.21.15.32:46426) 2024-04-30T08:34:55.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.682+0000 7fe292a00640 1 -- 172.21.15.32:0/2924547784 learned_addr learned my addr 172.21.15.32:0/2924547784 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:55.684 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.682+0000 7fe29be00640 1 --2- 172.21.15.32:0/2924547784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe29c1098d0 0x7fe29c1a3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:55.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.682+0000 7fe292a00640 1 -- 172.21.15.32:0/2924547784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe29c1098d0 msgr2=0x7fe29c1a3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:55.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.682+0000 7fe292a00640 1 --2- 172.21.15.32:0/2924547784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe29c1098d0 0x7fe29c1a3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.682+0000 7fe292a00640 1 -- 172.21.15.32:0/2924547784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe29c103990 msgr2=0x7fe29c19c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:55.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.682+0000 7fe292a00640 1 --2- 172.21.15.32:0/2924547784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe29c103990 0x7fe29c19c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.682+0000 7fe292a00640 1 -- 172.21.15.32:0/2924547784 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe29c1a3e00 con 0x7fe29c104bd0 2024-04-30T08:34:55.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.682+0000 7fe29be00640 1 --2- 172.21.15.32:0/2924547784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe29c1098d0 0x7fe29c1a3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:34:55.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.683+0000 7fe292a00640 1 --2- 172.21.15.32:0/2924547784 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe29c104bd0 0x7fe29c19c800 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe284006010 tx=0x7fe284031a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:55.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.683+0000 7fe298c00640 1 -- 172.21.15.32:0/2924547784 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28403f030 con 0x7fe29c104bd0 2024-04-30T08:34:55.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.683+0000 7fe2a0c00640 1 -- 172.21.15.32:0/2924547784 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe29c1a4090 con 0x7fe29c104bd0 2024-04-30T08:34:55.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.683+0000 7fe2a0c00640 1 -- 172.21.15.32:0/2924547784 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe29c1a45d0 con 0x7fe29c104bd0 2024-04-30T08:34:55.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.683+0000 7fe298c00640 1 -- 172.21.15.32:0/2924547784 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe284003aa0 con 0x7fe29c104bd0 2024-04-30T08:34:55.685 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.683+0000 7fe298c00640 1 -- 172.21.15.32:0/2924547784 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe284049060 con 0x7fe29c104bd0 2024-04-30T08:34:55.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.686+0000 7fe2a921e740 1 Processor -- start 2024-04-30T08:34:55.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.687+0000 7fe2a921e740 1 -- start start 2024-04-30T08:34:55.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.687+0000 7fe2a921e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55badcb19270 0x55badcebb530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.687+0000 7fe2a921e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55badcaf03b0 0x55badcebbab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.687+0000 7fe2a921e740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55badcaf8590 0x55badcebc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.687+0000 7fe2a921e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55badce1f790 con 0x55badcb19270 2024-04-30T08:34:55.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.687+0000 7fe2a921e740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55badcdc4f20 con 0x55badcaf8590 2024-04-30T08:34:55.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.687+0000 7fe2a921e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55badce1b750 con 0x55badcaf03b0 2024-04-30T08:34:55.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.687+0000 7fe29be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55badcb19270 0x55badcebb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:55.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.687+0000 7fe298c00640 1 -- 172.21.15.32:0/2924547784 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fe284042030 con 0x7fe29c104bd0 2024-04-30T08:34:55.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.687+0000 7fe29b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55badcaf03b0 0x55badcebbab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:55.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.687+0000 7fe29be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55badcb19270 0x55badcebb530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43508/0 (socket says 172.21.15.32:43508) 2024-04-30T08:34:55.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.687+0000 7fe29be00640 1 -- 172.21.15.32:0/252994611 learned_addr learned my addr 172.21.15.32:0/252994611 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:55.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.688+0000 7fe29b400640 1 -- 172.21.15.32:0/252994611 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55badcaf8590 msgr2=0x55badcebc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:55.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.688+0000 7fe29b400640 1 --2- 172.21.15.32:0/252994611 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55badcaf8590 0x55badcebc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.688+0000 7fe29b400640 1 -- 172.21.15.32:0/252994611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55badcb19270 msgr2=0x55badcebb530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:55.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.688+0000 7fe29b400640 1 --2- 172.21.15.32:0/252994611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55badcb19270 0x55badcebb530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.688+0000 7fe29b400640 1 -- 172.21.15.32:0/252994611 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55badcb312d0 con 0x55badcaf03b0 2024-04-30T08:34:55.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.688+0000 7fe29b400640 1 --2- 172.21.15.32:0/252994611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55badcaf03b0 0x55badcebbab0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe29c104590 tx=0x7fe28c00bbe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5955c09bb45ae354 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:55.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.688+0000 7fe28be00640 1 -- 172.21.15.32:0/252994611 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28c003da0 con 0x55badcaf03b0 2024-04-30T08:34:55.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.689+0000 7fe28be00640 1 -- 172.21.15.32:0/252994611 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe28c015070 con 0x55badcaf03b0 2024-04-30T08:34:55.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.689+0000 7fe28be00640 1 -- 172.21.15.32:0/252994611 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28c0116a0 con 0x55badcaf03b0 2024-04-30T08:34:55.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.689+0000 7fe298c00640 1 --2- 172.21.15.32:0/2924547784 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe2800afb60 0x7fe2800b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.689+0000 7fe2a921e740 1 -- 172.21.15.32:0/252994611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55badcaf03b0 msgr2=0x55badcebbab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:55.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.689+0000 7fe2a921e740 1 --2- 172.21.15.32:0/252994611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55badcaf03b0 0x55badcebbab0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fe29c104590 tx=0x7fe28c00bbe0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.690+0000 7fe2a921e740 1 -- 172.21.15.32:0/252994611 shutdown_connections 2024-04-30T08:34:55.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.690+0000 7fe2a921e740 1 --2- 172.21.15.32:0/252994611 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55badcaf8590 0x55badcebc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.690+0000 7fe2a921e740 1 --2- 172.21.15.32:0/252994611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55badcaf03b0 0x55badcebbab0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.690+0000 7fe2a921e740 1 --2- 172.21.15.32:0/252994611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55badcb19270 0x55badcebb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.690+0000 7fe298c00640 1 -- 172.21.15.32:0/2924547784 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fe2840ff070 con 0x7fe29c104bd0 2024-04-30T08:34:55.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.690+0000 7fe2a921e740 1 -- 172.21.15.32:0/252994611 >> 172.21.15.32:0/252994611 conn(0x55badcb0d3c0 msgr2=0x55badceb6d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:55.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.690+0000 7fe29be00640 1 --2- 172.21.15.32:0/2924547784 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe2800afb60 0x7fe2800b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:55.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.690+0000 7fe2a921e740 1 -- 172.21.15.32:0/252994611 shutdown_connections 2024-04-30T08:34:55.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.690+0000 7fe2a921e740 1 -- 172.21.15.32:0/252994611 wait complete. 2024-04-30T08:34:55.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.691+0000 7fe29be00640 1 --2- 172.21.15.32:0/2924547784 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe2800afb60 0x7fe2800b2070 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe294009fd0 tx=0x7fe294009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:55.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.691+0000 7fe2a921e740 1 -- start start 2024-04-30T08:34:55.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.691+0000 7fe2a921e740 1 Processor -- start 2024-04-30T08:34:55.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.691+0000 7fe2a921e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55badcaf03b0 0x55badcec50e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.691+0000 7fe2a921e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55badcaf8590 0x55badcec5620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.691+0000 7fe2a921e740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55badcb19270 0x55badcecc580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.692+0000 7fe2a921e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55badcebd250 con 0x55badcaf8590 2024-04-30T08:34:55.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.692+0000 7fe2a921e740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55badcdd89c0 con 0x55badcb19270 2024-04-30T08:34:55.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.692+0000 7fe2a921e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55badcebd3d0 con 0x55badcaf03b0 2024-04-30T08:34:55.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.692+0000 7fe2a921e740 10 client.0 _ll_register_callbacks cb 0x55badcda5120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:34:55.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.692+0000 7fe292a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55badcaf8590 0x55badcec5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:55.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.692+0000 7fe292a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55badcaf8590 0x55badcec5620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43512/0 (socket says 172.21.15.32:43512) 2024-04-30T08:34:55.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.692+0000 7fe292a00640 1 -- 172.21.15.32:0/2104077831 learned_addr learned my addr 172.21.15.32:0/2104077831 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:55.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.692+0000 7fe29b400640 1 --2- 172.21.15.32:0/2104077831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55badcaf03b0 0x55badcec50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:55.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.692+0000 7fe29be00640 1 --2- 172.21.15.32:0/2104077831 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55badcb19270 0x55badcecc580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:55.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.692+0000 7fe292a00640 1 -- 172.21.15.32:0/2104077831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55badcaf03b0 msgr2=0x55badcec50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:55.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.692+0000 7fe292a00640 1 --2- 172.21.15.32:0/2104077831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55badcaf03b0 0x55badcec50e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.692+0000 7fe292a00640 1 -- 172.21.15.32:0/2104077831 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55badcb19270 msgr2=0x55badcecc580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:55.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.692+0000 7fe292a00640 1 --2- 172.21.15.32:0/2104077831 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55badcb19270 0x55badcecc580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.692+0000 7fe292a00640 1 -- 172.21.15.32:0/2104077831 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55badcbfb6b0 con 0x55badcaf8590 2024-04-30T08:34:55.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.693+0000 7fe292a00640 1 --2- 172.21.15.32:0/2104077831 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55badcaf8590 0x55badcec5620 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe2840c4460 tx=0x7fe2840c6a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0cda56e171c8277 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:55.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.693+0000 7fe288c00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:34:55.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.693+0000 7fe288c00640 1 -- 172.21.15.32:0/2104077831 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe284035040 con 0x55badcaf8590 2024-04-30T08:34:55.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.693+0000 7fe288c00640 1 -- 172.21.15.32:0/2104077831 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fe2841010e0 con 0x55badcaf8590 2024-04-30T08:34:55.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.693+0000 7fe2a921e740 1 -- 172.21.15.32:0/2104077831 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe28c015070 con 0x55badcaf8590 2024-04-30T08:34:55.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.693+0000 7fe288c00640 1 -- 172.21.15.32:0/2104077831 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe2841012e0 con 0x55badcaf8590 2024-04-30T08:34:55.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.693+0000 7fe2a921e740 10 client.15285 fetch_fsmap learned FSMap version 27 2024-04-30T08:34:55.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.693+0000 7fe2a921e740 1 -- 172.21.15.32:0/2104077831 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55badcecf4b0 con 0x55badcaf8590 2024-04-30T08:34:55.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.694+0000 7fe288c00640 1 -- 172.21.15.32:0/2104077831 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe28c015070 con 0x55badcaf8590 2024-04-30T08:34:55.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.694+0000 7fe288c00640 1 -- 172.21.15.32:0/2104077831 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fe2840c7450 con 0x55badcaf8590 2024-04-30T08:34:55.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.694+0000 7fe2a921e740 10 client.15285 fetch_fsmap finished waiting for FSMap version 27 2024-04-30T08:34:55.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.694+0000 7fe2a921e740 10 client.15285 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-30T08:34:55.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.694+0000 7fe2a921e740 20 client.15285 populate_metadata read hostname 'smithi032' 2024-04-30T08:34:55.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.694+0000 7fe2a921e740 1 --2- 172.21.15.32:0/2104077831 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x55badcad7e40 0x55badced2f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.694+0000 7fe2a921e740 4 client.15285 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:34:55.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.694+0000 7fe2a921e740 1 -- 172.21.15.32:0/2104077831 --> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55badce2d3a0 con 0x55badcad7e40 2024-04-30T08:34:55.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.694+0000 7fe29b400640 1 --2- 172.21.15.32:0/2104077831 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x55badcad7e40 0x55badced2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:55.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.695+0000 7fe29b400640 1 --2- 172.21.15.32:0/2104077831 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x55badcad7e40 0x55badced2f90 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=c4d6228bf996fadd server_cookie=d790349a1a9d701e in_seq=0 out_seq=0 2024-04-30T08:34:55.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.695+0000 7fe288c00640 10 client.15285 ms_handle_connect on v2:172.21.15.71:6836/2324039932 2024-04-30T08:34:55.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.696+0000 7fe288c00640 1 -- 172.21.15.32:0/2104077831 <== mds.3 v2:172.21.15.71:6836/2324039932 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55badce2d3a0 con 0x55badcad7e40 2024-04-30T08:34:55.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.696+0000 7fe288c00640 10 client.15285 handle_command_reply: tid=0 2024-04-30T08:34:55.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.696+0000 7fe2a921e740 1 client.15285 shutdown 2024-04-30T08:34:55.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.696+0000 7fe2a921e740 1 -- 172.21.15.32:0/2104077831 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55badcaf8590 msgr2=0x55badcec5620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:55.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.696+0000 7fe2a921e740 1 --2- 172.21.15.32:0/2104077831 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55badcaf8590 0x55badcec5620 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fe2840c4460 tx=0x7fe2840c6a20 comp rx=0 tx=0).stop 2024-04-30T08:34:55.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.696+0000 7fe2a921e740 1 -- 172.21.15.32:0/2104077831 shutdown_connections 2024-04-30T08:34:55.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.696+0000 7fe2a921e740 1 --2- 172.21.15.32:0/2104077831 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x55badcad7e40 0x55badced2f90 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.696+0000 7fe2a921e740 1 --2- 172.21.15.32:0/2104077831 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55badcb19270 0x55badcecc580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.696+0000 7fe2a921e740 1 --2- 172.21.15.32:0/2104077831 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55badcaf8590 0x55badcec5620 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.696+0000 7fe2a921e740 1 --2- 172.21.15.32:0/2104077831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55badcaf03b0 0x55badcec50e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.696+0000 7fe288c00640 0 client.15285 ms_handle_reset on v2:172.21.15.71:6836/2324039932 2024-04-30T08:34:55.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.696+0000 7fe2a921e740 1 -- 172.21.15.32:0/2104077831 >> 172.21.15.32:0/2104077831 conn(0x55badcb0d3c0 msgr2=0x55badceb9c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:55.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.696+0000 7fe2a921e740 1 -- 172.21.15.32:0/2104077831 shutdown_connections 2024-04-30T08:34:55.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.696+0000 7fe2a921e740 1 -- 172.21.15.32:0/2104077831 wait complete. 2024-04-30T08:34:55.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.697+0000 7fe2a921e740 20 client.15285 trim_cache size 0 max 16384 2024-04-30T08:34:55.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.717+0000 7fe2a921e740 1 Processor -- start 2024-04-30T08:34:55.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.717+0000 7fe2a921e740 1 -- start start 2024-04-30T08:34:55.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.717+0000 7fe2a921e740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55badcaf03b0 0x55badceba570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.717+0000 7fe2a921e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55badcaf8590 0x55badcebb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.718+0000 7fe2a921e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55badcb19270 0x55badceb6de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.718+0000 7fe2a921e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55badced12d0 con 0x55badcb19270 2024-04-30T08:34:55.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.718+0000 7fe2a921e740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55badced19a0 con 0x55badcaf03b0 2024-04-30T08:34:55.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.718+0000 7fe2a921e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55badced2070 con 0x55badcaf8590 2024-04-30T08:34:55.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.718+0000 7fe29be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55badcb19270 0x55badceb6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:55.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.718+0000 7fe29b400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55badcaf03b0 0x55badceba570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:55.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.718+0000 7fe29be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55badcb19270 0x55badceb6de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43518/0 (socket says 172.21.15.32:43518) 2024-04-30T08:34:55.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.718+0000 7fe29be00640 1 -- 172.21.15.32:0/224461709 learned_addr learned my addr 172.21.15.32:0/224461709 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:55.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.718+0000 7fe29b400640 1 --2- 172.21.15.32:0/224461709 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55badcaf03b0 0x55badceba570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:46450/0 (socket says 172.21.15.32:46450) 2024-04-30T08:34:55.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.718+0000 7fe292a00640 1 --2- 172.21.15.32:0/224461709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55badcaf8590 0x55badcebb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:55.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.718+0000 7fe29b400640 1 -- 172.21.15.32:0/224461709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55badcaf8590 msgr2=0x55badcebb4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:55.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.718+0000 7fe29b400640 1 --2- 172.21.15.32:0/224461709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55badcaf8590 0x55badcebb4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.718+0000 7fe29b400640 1 -- 172.21.15.32:0/224461709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55badcb19270 msgr2=0x55badceb6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:55.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.718+0000 7fe29b400640 1 --2- 172.21.15.32:0/224461709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55badcb19270 0x55badceb6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.718+0000 7fe29b400640 1 -- 172.21.15.32:0/224461709 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55badcec6220 con 0x55badcaf03b0 2024-04-30T08:34:55.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.719+0000 7fe29b400640 1 --2- 172.21.15.32:0/224461709 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55badcaf03b0 0x55badceba570 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe28c000c00 tx=0x7fe28c013040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7739dd4e92fae6bb server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:55.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.719+0000 7fe28b400640 1 -- 172.21.15.32:0/224461709 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28c02f070 con 0x55badcaf03b0 2024-04-30T08:34:55.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.719+0000 7fe28b400640 1 -- 172.21.15.32:0/224461709 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe28c015070 con 0x55badcaf03b0 2024-04-30T08:34:55.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.719+0000 7fe28b400640 1 -- 172.21.15.32:0/224461709 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe28c004c90 con 0x55badcaf03b0 2024-04-30T08:34:55.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.720+0000 7fe2a921e740 1 -- 172.21.15.32:0/224461709 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55badcaf03b0 msgr2=0x55badceba570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:55.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.720+0000 7fe2a921e740 1 --2- 172.21.15.32:0/224461709 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55badcaf03b0 0x55badceba570 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe28c000c00 tx=0x7fe28c013040 comp rx=0 tx=0).stop 2024-04-30T08:34:55.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.720+0000 7fe2a921e740 1 -- 172.21.15.32:0/224461709 shutdown_connections 2024-04-30T08:34:55.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.720+0000 7fe2a921e740 1 --2- 172.21.15.32:0/224461709 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55badcb19270 0x55badceb6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.720+0000 7fe2a921e740 1 --2- 172.21.15.32:0/224461709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55badcaf8590 0x55badcebb4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.720+0000 7fe2a921e740 1 --2- 172.21.15.32:0/224461709 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55badcaf03b0 0x55badceba570 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.720+0000 7fe2a921e740 1 -- 172.21.15.32:0/224461709 >> 172.21.15.32:0/224461709 conn(0x55badcb0d3c0 msgr2=0x55badcebf8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:55.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.720+0000 7fe2a921e740 1 -- 172.21.15.32:0/224461709 shutdown_connections 2024-04-30T08:34:55.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.720+0000 7fe2a921e740 1 -- 172.21.15.32:0/224461709 wait complete. 2024-04-30T08:34:55.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.721+0000 7fe2a921e740 1 -- start start 2024-04-30T08:34:55.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.721+0000 7fe2a921e740 1 Processor -- start 2024-04-30T08:34:55.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.721+0000 7fe2a921e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55badcaf03b0 0x55badcec87f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.721+0000 7fe2a921e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55badcaf8590 0x55badcec8d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.721+0000 7fe2a921e740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55badcb19270 0x55badcecbc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.721+0000 7fe2a921e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55badced2e10 con 0x55badcaf03b0 2024-04-30T08:34:55.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.721+0000 7fe2a921e740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55badced2740 con 0x55badcb19270 2024-04-30T08:34:55.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.721+0000 7fe2a921e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55badced34e0 con 0x55badcaf8590 2024-04-30T08:34:55.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.721+0000 7fe2a921e740 10 client.0 _ll_register_callbacks cb 0x7fe284009290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:34:55.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.722+0000 7fe29b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55badcaf03b0 0x55badcec87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:55.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.722+0000 7fe29be00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55badcb19270 0x55badcecbc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:55.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.722+0000 7fe292a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55badcaf8590 0x55badcec8d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:55.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.722+0000 7fe29be00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55badcb19270 0x55badcecbc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:46462/0 (socket says 172.21.15.32:46462) 2024-04-30T08:34:55.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.722+0000 7fe292a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55badcaf8590 0x55badcec8d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:58556/0 (socket says 172.21.15.32:58556) 2024-04-30T08:34:55.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.722+0000 7fe29b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55badcaf03b0 0x55badcec87f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43526/0 (socket says 172.21.15.32:43526) 2024-04-30T08:34:55.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.722+0000 7fe29be00640 1 -- 172.21.15.32:0/1848260651 learned_addr learned my addr 172.21.15.32:0/1848260651 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:55.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.722+0000 7fe29be00640 1 -- 172.21.15.32:0/1848260651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55badcaf8590 msgr2=0x55badcec8d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:55.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.722+0000 7fe29be00640 1 --2- 172.21.15.32:0/1848260651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55badcaf8590 0x55badcec8d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.722+0000 7fe29be00640 1 -- 172.21.15.32:0/1848260651 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55badcaf03b0 msgr2=0x55badcec87f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:55.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.722+0000 7fe29be00640 1 --2- 172.21.15.32:0/1848260651 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55badcaf03b0 0x55badcec87f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.722+0000 7fe29be00640 1 -- 172.21.15.32:0/1848260651 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55badcecc1e0 con 0x55badcb19270 2024-04-30T08:34:55.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.722+0000 7fe292a00640 1 --2- 172.21.15.32:0/1848260651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55badcaf8590 0x55badcec8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:34:55.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.722+0000 7fe29b400640 1 --2- 172.21.15.32:0/1848260651 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55badcaf03b0 0x55badcec87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:34:55.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.723+0000 7fe29be00640 1 --2- 172.21.15.32:0/1848260651 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55badcb19270 0x55badcecbc20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe29400fef0 tx=0x7fe294004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fa1de2599781b7f server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:55.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.723+0000 7fe288c00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:34:55.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.723+0000 7fe288c00640 1 -- 172.21.15.32:0/1848260651 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe29402f060 con 0x55badcb19270 2024-04-30T08:34:55.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.723+0000 7fe2a921e740 1 -- 172.21.15.32:0/1848260651 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe28c015070 con 0x55badcb19270 2024-04-30T08:34:55.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.723+0000 7fe288c00640 1 -- 172.21.15.32:0/1848260651 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fe294011bd0 con 0x55badcb19270 2024-04-30T08:34:55.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.723+0000 7fe288c00640 1 -- 172.21.15.32:0/1848260651 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7fe294036050 con 0x55badcb19270 2024-04-30T08:34:55.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.723+0000 7fe2a921e740 10 client.25412 fetch_fsmap learned FSMap version 27 2024-04-30T08:34:55.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.723+0000 7fe2a921e740 1 -- 172.21.15.32:0/1848260651 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55badcf3c0c0 con 0x55badcb19270 2024-04-30T08:34:55.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.724+0000 7fe288c00640 1 -- 172.21.15.32:0/1848260651 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe28c015070 con 0x55badcb19270 2024-04-30T08:34:55.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.724+0000 7fe288c00640 1 -- 172.21.15.32:0/1848260651 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7fe294004590 con 0x55badcb19270 2024-04-30T08:34:55.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.724+0000 7fe2a921e740 10 client.25412 fetch_fsmap finished waiting for FSMap version 27 2024-04-30T08:34:55.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.724+0000 7fe2a921e740 10 client.25412 resolve_mds: resolved name 'h' to daemon mds.h 2024-04-30T08:34:55.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.724+0000 7fe2a921e740 20 client.25412 populate_metadata read hostname 'smithi032' 2024-04-30T08:34:55.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.724+0000 7fe2a921e740 1 --2- 172.21.15.32:0/1848260651 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x55badcaad280 0x55badcf3fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:55.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.724+0000 7fe2a921e740 4 client.25412 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-30T08:34:55.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.724+0000 7fe2a921e740 1 -- 172.21.15.32:0/1848260651 --> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] -- command(tid 0: {"prefix": "status"}) -- 0x55badce27210 con 0x55badcaad280 2024-04-30T08:34:55.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.724+0000 7fe29b400640 1 --2- 172.21.15.32:0/1848260651 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x55badcaad280 0x55badcf3fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:55.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.725+0000 7fe29b400640 1 --2- 172.21.15.32:0/1848260651 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x55badcaad280 0x55badcf3fa20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d496a6df0af3c403 server_cookie=bd21e97c864b02ae in_seq=0 out_seq=0 2024-04-30T08:34:55.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.725+0000 7fe288c00640 10 client.25412 ms_handle_connect on v2:172.21.15.71:6836/2324039932 2024-04-30T08:34:55.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.725+0000 7fe288c00640 1 -- 172.21.15.32:0/1848260651 <== mds.3 v2:172.21.15.71:6836/2324039932 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55badce27210 con 0x55badcaad280 2024-04-30T08:34:55.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.725+0000 7fe288c00640 10 client.25412 handle_command_reply: tid=0 2024-04-30T08:34:55.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.725+0000 7fe2a921e740 1 client.25412 shutdown 2024-04-30T08:34:55.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.725+0000 7fe2a921e740 1 -- 172.21.15.32:0/1848260651 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55badcb19270 msgr2=0x55badcecbc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:55.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.725+0000 7fe2a921e740 1 --2- 172.21.15.32:0/1848260651 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55badcb19270 0x55badcecbc20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fe29400fef0 tx=0x7fe294004a40 comp rx=0 tx=0).stop 2024-04-30T08:34:55.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.725+0000 7fe2a921e740 1 -- 172.21.15.32:0/1848260651 shutdown_connections 2024-04-30T08:34:55.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.725+0000 7fe2a921e740 1 --2- 172.21.15.32:0/1848260651 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x55badcaad280 0x55badcf3fa20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.725+0000 7fe2a921e740 1 --2- 172.21.15.32:0/1848260651 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55badcb19270 0x55badcecbc20 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.725+0000 7fe2a921e740 1 --2- 172.21.15.32:0/1848260651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55badcaf8590 0x55badcec8d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.725+0000 7fe288c00640 0 client.25412 ms_handle_reset on v2:172.21.15.71:6836/2324039932 2024-04-30T08:34:55.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.725+0000 7fe2a921e740 1 --2- 172.21.15.32:0/1848260651 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55badcaf03b0 0x55badcec87f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.725+0000 7fe2a921e740 1 -- 172.21.15.32:0/1848260651 >> 172.21.15.32:0/1848260651 conn(0x55badcb0d3c0 msgr2=0x55badceba2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:55.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.726+0000 7fe2a921e740 1 -- 172.21.15.32:0/1848260651 shutdown_connections 2024-04-30T08:34:55.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.726+0000 7fe2a921e740 1 -- 172.21.15.32:0/1848260651 wait complete. 2024-04-30T08:34:55.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.726+0000 7fe2a921e740 20 client.25412 trim_cache size 0 max 16384 2024-04-30T08:34:55.730 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-30T08:34:55.730 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "98c0a670-06ca-11ef-bc93-c7b262605968", 2024-04-30T08:34:55.730 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 3, 2024-04-30T08:34:55.731 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 24533, 2024-04-30T08:34:55.731 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-04-30T08:34:55.731 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-04-30T08:34:55.731 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-04-30T08:34:55.731 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 147.20723289700001, 2024-04-30T08:34:55.731 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 27, 2024-04-30T08:34:55.731 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 85, 2024-04-30T08:34:55.731 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 81, 2024-04-30T08:34:55.731 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 204.99869154800001 2024-04-30T08:34:55.731 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-30T08:34:55.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.726+0000 7fe2a0c00640 1 -- 172.21.15.32:0/2924547784 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe2800afb60 msgr2=0x7fe2800b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:55.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.726+0000 7fe2a0c00640 1 --2- 172.21.15.32:0/2924547784 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe2800afb60 0x7fe2800b2070 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fe294009fd0 tx=0x7fe294009500 comp rx=0 tx=0).stop 2024-04-30T08:34:55.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.726+0000 7fe2a0c00640 1 -- 172.21.15.32:0/2924547784 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe29c104bd0 msgr2=0x7fe29c19c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:55.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.726+0000 7fe2a0c00640 1 --2- 172.21.15.32:0/2924547784 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe29c104bd0 0x7fe29c19c800 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fe284006010 tx=0x7fe284031a50 comp rx=0 tx=0).stop 2024-04-30T08:34:55.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.726+0000 7fe2a0c00640 1 -- 172.21.15.32:0/2924547784 shutdown_connections 2024-04-30T08:34:55.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.726+0000 7fe2a0c00640 1 --2- 172.21.15.32:0/2924547784 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe2800afb60 0x7fe2800b2070 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.727+0000 7fe2a0c00640 1 --2- 172.21.15.32:0/2924547784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe29c1098d0 0x7fe29c1a3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.727+0000 7fe2a0c00640 1 --2- 172.21.15.32:0/2924547784 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe29c104bd0 0x7fe29c19c800 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.727+0000 7fe2a0c00640 1 --2- 172.21.15.32:0/2924547784 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe29c103990 0x7fe29c19c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:55.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.727+0000 7fe2a0c00640 1 -- 172.21.15.32:0/2924547784 >> 172.21.15.32:0/2924547784 conn(0x7fe29c0feba0 msgr2=0x7fe29c0ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:55.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.727+0000 7fe2a0c00640 1 -- 172.21.15.32:0/2924547784 shutdown_connections 2024-04-30T08:34:55.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:55.727+0000 7fe2a0c00640 1 -- 172.21.15.32:0/2924547784 wait complete. 2024-04-30T08:34:55.754 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-30T08:34:56.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.041+0000 7f5635e00640 1 -- 172.21.15.32:0/591611157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5630105bb0 msgr2=0x7f56301068b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:56.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.041+0000 7f5635e00640 1 --2- 172.21.15.32:0/591611157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5630105bb0 0x7f56301068b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5624009900 tx=0x7f562402a3f0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.041+0000 7f5635e00640 1 -- 172.21.15.32:0/591611157 shutdown_connections 2024-04-30T08:34:56.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.041+0000 7f5635e00640 1 --2- 172.21.15.32:0/591611157 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f563010baf0 0x7f563010df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.041+0000 7f5635e00640 1 --2- 172.21.15.32:0/591611157 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5630106df0 0x7f5630107200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.041+0000 7f5635e00640 1 --2- 172.21.15.32:0/591611157 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5630105bb0 0x7f56301068b0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.041+0000 7f5635e00640 1 -- 172.21.15.32:0/591611157 >> 172.21.15.32:0/591611157 conn(0x7f5630100da0 msgr2=0x7f5630103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:56.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.041+0000 7f5635e00640 1 -- 172.21.15.32:0/591611157 shutdown_connections 2024-04-30T08:34:56.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.041+0000 7f5635e00640 1 -- 172.21.15.32:0/591611157 wait complete. 2024-04-30T08:34:56.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.043+0000 7f5635e00640 1 Processor -- start 2024-04-30T08:34:56.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.043+0000 7f5635e00640 1 -- start start 2024-04-30T08:34:56.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.043+0000 7f5635e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5630105bb0 0x7f563019c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.043+0000 7f5635e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5630106df0 0x7f563019c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.043+0000 7f5635e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f563010baf0 0x7f56301a3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.043+0000 7f5635e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f56301116e0 con 0x7f5630105bb0 2024-04-30T08:34:56.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.043+0000 7f5635e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5630111560 con 0x7f5630106df0 2024-04-30T08:34:56.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.043+0000 7f5635e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5630111860 con 0x7f563010baf0 2024-04-30T08:34:56.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.044+0000 7f5634a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5630105bb0 0x7f563019c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.044+0000 7f5634a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5630105bb0 0x7f563019c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43540/0 (socket says 172.21.15.32:43540) 2024-04-30T08:34:56.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.044+0000 7f5634a00640 1 -- 172.21.15.32:0/2884302835 learned_addr learned my addr 172.21.15.32:0/2884302835 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:56.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.044+0000 7f562fe00640 1 --2- 172.21.15.32:0/2884302835 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5630106df0 0x7f563019c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.044+0000 7f5635400640 1 --2- 172.21.15.32:0/2884302835 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f563010baf0 0x7f56301a3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.044+0000 7f5634a00640 1 -- 172.21.15.32:0/2884302835 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f563010baf0 msgr2=0x7f56301a3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:56.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.044+0000 7f5634a00640 1 --2- 172.21.15.32:0/2884302835 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f563010baf0 0x7f56301a3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.044+0000 7f5634a00640 1 -- 172.21.15.32:0/2884302835 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5630106df0 msgr2=0x7f563019c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:56.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.044+0000 7f5634a00640 1 --2- 172.21.15.32:0/2884302835 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5630106df0 0x7f563019c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.044+0000 7f5634a00640 1 -- 172.21.15.32:0/2884302835 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56301a3e00 con 0x7f5630105bb0 2024-04-30T08:34:56.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.044+0000 7f5635400640 1 --2- 172.21.15.32:0/2884302835 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f563010baf0 0x7f56301a3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:34:56.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.045+0000 7f5634a00640 1 --2- 172.21.15.32:0/2884302835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5630105bb0 0x7f563019c2c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5624031440 tx=0x7f5624003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:56.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.045+0000 7f562d600640 1 -- 172.21.15.32:0/2884302835 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f562403f070 con 0x7f5630105bb0 2024-04-30T08:34:56.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.045+0000 7f5635e00640 1 -- 172.21.15.32:0/2884302835 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56301a4090 con 0x7f5630105bb0 2024-04-30T08:34:56.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.045+0000 7f562d600640 1 -- 172.21.15.32:0/2884302835 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5624031c50 con 0x7f5630105bb0 2024-04-30T08:34:56.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.045+0000 7f562d600640 1 -- 172.21.15.32:0/2884302835 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5624035030 con 0x7f5630105bb0 2024-04-30T08:34:56.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.045+0000 7f5635e00640 1 -- 172.21.15.32:0/2884302835 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56301a45d0 con 0x7f5630105bb0 2024-04-30T08:34:56.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.049+0000 7f563e36f740 1 Processor -- start 2024-04-30T08:34:56.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.049+0000 7f563e36f740 1 -- start start 2024-04-30T08:34:56.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.049+0000 7f562d600640 1 -- 172.21.15.32:0/2884302835 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f5624042070 con 0x7f5630105bb0 2024-04-30T08:34:56.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.049+0000 7f563e36f740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5618e982c270 0x5618e9bce530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.049+0000 7f563e36f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618e98033b0 0x5618e9bceab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.049+0000 7f563e36f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618e980b590 0x5618e9bcf740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.049+0000 7f563e36f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5618e9b32790 con 0x5618e980b590 2024-04-30T08:34:56.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.049+0000 7f563e36f740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x5618e9ad7f20 con 0x5618e982c270 2024-04-30T08:34:56.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.049+0000 7f563e36f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5618e9b2e750 con 0x5618e98033b0 2024-04-30T08:34:56.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.050+0000 7f5635400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5618e982c270 0x5618e9bce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.050+0000 7f562fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618e980b590 0x5618e9bcf740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.050+0000 7f5635400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5618e982c270 0x5618e9bce530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:46482/0 (socket says 172.21.15.32:46482) 2024-04-30T08:34:56.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.050+0000 7f5635400640 1 -- 172.21.15.32:0/402197600 learned_addr learned my addr 172.21.15.32:0/402197600 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:56.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.050+0000 7f5634a00640 1 --2- 172.21.15.32:0/402197600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618e98033b0 0x5618e9bceab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.050+0000 7f5635400640 1 -- 172.21.15.32:0/402197600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618e98033b0 msgr2=0x5618e9bceab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:56.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.050+0000 7f5635400640 1 --2- 172.21.15.32:0/402197600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618e98033b0 0x5618e9bceab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.050+0000 7f5635400640 1 -- 172.21.15.32:0/402197600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618e980b590 msgr2=0x5618e9bcf740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:56.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.050+0000 7f5635400640 1 --2- 172.21.15.32:0/402197600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618e980b590 0x5618e9bcf740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.050+0000 7f5635400640 1 -- 172.21.15.32:0/402197600 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618e98442d0 con 0x5618e982c270 2024-04-30T08:34:56.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.050+0000 7f5635400640 1 --2- 172.21.15.32:0/402197600 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5618e982c270 0x5618e9bce530 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5628009fd0 tx=0x7f56280108e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b26987e6130ebeae server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:56.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.051+0000 7f5621c00640 1 -- 172.21.15.32:0/402197600 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5628011760 con 0x5618e982c270 2024-04-30T08:34:56.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.051+0000 7f5621c00640 1 -- 172.21.15.32:0/402197600 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5628011db0 con 0x5618e982c270 2024-04-30T08:34:56.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.051+0000 7f5621c00640 1 -- 172.21.15.32:0/402197600 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f562800e3c0 con 0x5618e982c270 2024-04-30T08:34:56.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.051+0000 7f562d600640 1 --2- 172.21.15.32:0/2884302835 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f56180afb60 0x7f56180b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.051+0000 7f562d600640 1 -- 172.21.15.32:0/2884302835 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f56240ff7d0 con 0x7f5630105bb0 2024-04-30T08:34:56.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.051+0000 7f5634a00640 1 --2- 172.21.15.32:0/2884302835 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f56180afb60 0x7f56180b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.052+0000 7f563e36f740 1 -- 172.21.15.32:0/402197600 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5618e982c270 msgr2=0x5618e9bce530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:56.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.052+0000 7f563e36f740 1 --2- 172.21.15.32:0/402197600 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5618e982c270 0x5618e9bce530 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5628009fd0 tx=0x7f56280108e0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.052+0000 7f563e36f740 1 -- 172.21.15.32:0/402197600 shutdown_connections 2024-04-30T08:34:56.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.052+0000 7f563e36f740 1 --2- 172.21.15.32:0/402197600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618e980b590 0x5618e9bcf740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.052+0000 7f563e36f740 1 --2- 172.21.15.32:0/402197600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618e98033b0 0x5618e9bceab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.052+0000 7f563e36f740 1 --2- 172.21.15.32:0/402197600 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5618e982c270 0x5618e9bce530 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.052+0000 7f563e36f740 1 -- 172.21.15.32:0/402197600 >> 172.21.15.32:0/402197600 conn(0x5618e98203c0 msgr2=0x5618e9bc9d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:56.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.052+0000 7f563e36f740 1 -- 172.21.15.32:0/402197600 shutdown_connections 2024-04-30T08:34:56.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.052+0000 7f563e36f740 1 -- 172.21.15.32:0/402197600 wait complete. 2024-04-30T08:34:56.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.052+0000 7f5634a00640 1 --2- 172.21.15.32:0/2884302835 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f56180afb60 0x7f56180b2070 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f56240fcec0 tx=0x7f5624105530 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:56.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.053+0000 7f563e36f740 1 -- start start 2024-04-30T08:34:56.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.053+0000 7f563e36f740 1 Processor -- start 2024-04-30T08:34:56.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.053+0000 7f563e36f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618e98033b0 0x5618e9bd8070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.053+0000 7f563e36f740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5618e980b590 0x5618e9bd85b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.053+0000 7f563e36f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618e982c270 0x5618e9bdf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.053+0000 7f563e36f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5618e9bd0250 con 0x5618e98033b0 2024-04-30T08:34:56.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.053+0000 7f563e36f740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x5618e9aeb9c0 con 0x5618e980b590 2024-04-30T08:34:56.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.053+0000 7f563e36f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5618e9bd03d0 con 0x5618e982c270 2024-04-30T08:34:56.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.054+0000 7f563e36f740 10 client.0 _ll_register_callbacks cb 0x5618e9ab8120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:34:56.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.054+0000 7f5635400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618e98033b0 0x5618e9bd8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.054+0000 7f5635400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618e98033b0 0x5618e9bd8070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43550/0 (socket says 172.21.15.32:43550) 2024-04-30T08:34:56.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.054+0000 7f5635400640 1 -- 172.21.15.32:0/3436912650 learned_addr learned my addr 172.21.15.32:0/3436912650 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:56.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.054+0000 7f562fe00640 1 --2- 172.21.15.32:0/3436912650 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5618e980b590 0x5618e9bd85b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.054+0000 7f5635400640 1 --2- 172.21.15.32:0/3436912650 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618e982c270 0x5618e9bdf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.054+0000 7f5635400640 1 -- 172.21.15.32:0/3436912650 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618e982c270 msgr2=0x5618e9bdf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:56.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.054+0000 7f5635400640 1 --2- 172.21.15.32:0/3436912650 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618e982c270 0x5618e9bdf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.054+0000 7f5635400640 1 -- 172.21.15.32:0/3436912650 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5618e980b590 msgr2=0x5618e9bd85b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:56.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.054+0000 7f5635400640 1 --2- 172.21.15.32:0/3436912650 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5618e980b590 0x5618e9bd85b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.054+0000 7f5635400640 1 -- 172.21.15.32:0/3436912650 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618e990e6b0 con 0x5618e98033b0 2024-04-30T08:34:56.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.055+0000 7f5635400640 1 --2- 172.21.15.32:0/3436912650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618e98033b0 0x5618e9bd8070 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5628005cc0 tx=0x7f5628009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce8842eceda603d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:56.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.055+0000 7f5616000640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:34:56.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.055+0000 7f5616000640 1 -- 172.21.15.32:0/3436912650 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5628011970 con 0x5618e98033b0 2024-04-30T08:34:56.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.055+0000 7f5616000640 1 -- 172.21.15.32:0/3436912650 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f5628011b10 con 0x5618e98033b0 2024-04-30T08:34:56.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.055+0000 7f563e36f740 1 -- 172.21.15.32:0/3436912650 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5628011db0 con 0x5618e98033b0 2024-04-30T08:34:56.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.055+0000 7f5616000640 1 -- 172.21.15.32:0/3436912650 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5628005040 con 0x5618e98033b0 2024-04-30T08:34:56.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.055+0000 7f563e36f740 10 client.15303 fetch_fsmap learned FSMap version 27 2024-04-30T08:34:56.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.055+0000 7f563e36f740 1 -- 172.21.15.32:0/3436912650 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5618e9be2440 con 0x5618e98033b0 2024-04-30T08:34:56.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.056+0000 7f5616000640 1 -- 172.21.15.32:0/3436912650 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5628011db0 con 0x5618e98033b0 2024-04-30T08:34:56.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.056+0000 7f5616000640 1 -- 172.21.15.32:0/3436912650 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f5628029020 con 0x5618e98033b0 2024-04-30T08:34:56.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.056+0000 7f563e36f740 10 client.15303 fetch_fsmap finished waiting for FSMap version 27 2024-04-30T08:34:56.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.056+0000 7f563e36f740 10 client.15303 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-30T08:34:56.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.056+0000 7f563e36f740 20 client.15303 populate_metadata read hostname 'smithi032' 2024-04-30T08:34:56.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.056+0000 7f563e36f740 1 --2- 172.21.15.32:0/3436912650 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x5618e97eae40 0x5618e9be5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.056+0000 7f563e36f740 4 client.15303 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:34:56.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.056+0000 7f563e36f740 1 -- 172.21.15.32:0/3436912650 --> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5618e9b403a0 con 0x5618e97eae40 2024-04-30T08:34:56.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.056+0000 7f562fe00640 1 --2- 172.21.15.32:0/3436912650 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x5618e97eae40 0x5618e9be5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.057+0000 7f562fe00640 1 --2- 172.21.15.32:0/3436912650 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x5618e97eae40 0x5618e9be5ac0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8d91cf49893ae481 server_cookie=6d58f5de21896494 in_seq=0 out_seq=0 2024-04-30T08:34:56.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.057+0000 7f5616000640 10 client.15303 ms_handle_connect on v2:172.21.15.189:6838/352970107 2024-04-30T08:34:56.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.058+0000 7f5616000640 1 -- 172.21.15.32:0/3436912650 <== mds.2 v2:172.21.15.189:6838/352970107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5618e9b403a0 con 0x5618e97eae40 2024-04-30T08:34:56.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.058+0000 7f5616000640 10 client.15303 handle_command_reply: tid=0 2024-04-30T08:34:56.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.058+0000 7f563e36f740 1 client.15303 shutdown 2024-04-30T08:34:56.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.058+0000 7f563e36f740 1 -- 172.21.15.32:0/3436912650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618e98033b0 msgr2=0x5618e9bd8070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:56.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.058+0000 7f563e36f740 1 --2- 172.21.15.32:0/3436912650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618e98033b0 0x5618e9bd8070 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5628005cc0 tx=0x7f5628009bb0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.058+0000 7f563e36f740 1 -- 172.21.15.32:0/3436912650 shutdown_connections 2024-04-30T08:34:56.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.058+0000 7f563e36f740 1 --2- 172.21.15.32:0/3436912650 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x5618e97eae40 0x5618e9be5ac0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.058+0000 7f563e36f740 1 --2- 172.21.15.32:0/3436912650 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618e982c270 0x5618e9bdf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.058+0000 7f563e36f740 1 --2- 172.21.15.32:0/3436912650 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5618e980b590 0x5618e9bd85b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.063 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.058+0000 7f563e36f740 1 --2- 172.21.15.32:0/3436912650 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618e98033b0 0x5618e9bd8070 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.058+0000 7f5616000640 0 client.15303 ms_handle_reset on v2:172.21.15.189:6838/352970107 2024-04-30T08:34:56.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.058+0000 7f563e36f740 1 -- 172.21.15.32:0/3436912650 >> 172.21.15.32:0/3436912650 conn(0x5618e98203c0 msgr2=0x5618e9bccc50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:56.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.059+0000 7f563e36f740 1 -- 172.21.15.32:0/3436912650 shutdown_connections 2024-04-30T08:34:56.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.059+0000 7f563e36f740 1 -- 172.21.15.32:0/3436912650 wait complete. 2024-04-30T08:34:56.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.059+0000 7f563e36f740 20 client.15303 trim_cache size 0 max 16384 2024-04-30T08:34:56.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.080+0000 7f563e36f740 1 Processor -- start 2024-04-30T08:34:56.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.080+0000 7f563e36f740 1 -- start start 2024-04-30T08:34:56.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.080+0000 7f563e36f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618e98033b0 0x5618e9bcf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.080+0000 7f563e36f740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5618e980b590 0x5618e9bdf4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.080+0000 7f563e36f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618e982c270 0x5618e9bdf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.080+0000 7f563e36f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5618e9be3e00 con 0x5618e98033b0 2024-04-30T08:34:56.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.080+0000 7f563e36f740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x5618e9be44d0 con 0x5618e980b590 2024-04-30T08:34:56.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.080+0000 7f563e36f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5618e9be4ba0 con 0x5618e982c270 2024-04-30T08:34:56.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.081+0000 7f5635400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618e98033b0 0x5618e9bcf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.081+0000 7f562fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5618e980b590 0x5618e9bdf4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.081+0000 7f5635400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618e98033b0 0x5618e9bcf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43564/0 (socket says 172.21.15.32:43564) 2024-04-30T08:34:56.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.081+0000 7f5635400640 1 -- 172.21.15.32:0/3813778710 learned_addr learned my addr 172.21.15.32:0/3813778710 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:56.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.081+0000 7f5635400640 1 --2- 172.21.15.32:0/3813778710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618e982c270 0x5618e9bdf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.081+0000 7f562fe00640 1 -- 172.21.15.32:0/3813778710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618e982c270 msgr2=0x5618e9bdf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:56.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.081+0000 7f562fe00640 1 --2- 172.21.15.32:0/3813778710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618e982c270 0x5618e9bdf9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.081+0000 7f562fe00640 1 -- 172.21.15.32:0/3813778710 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618e98033b0 msgr2=0x5618e9bcf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:56.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.081+0000 7f562fe00640 1 --2- 172.21.15.32:0/3813778710 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618e98033b0 0x5618e9bcf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.082+0000 7f562fe00640 1 -- 172.21.15.32:0/3813778710 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5618e9bd6160 con 0x5618e980b590 2024-04-30T08:34:56.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.082+0000 7f5635400640 1 --2- 172.21.15.32:0/3813778710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618e982c270 0x5618e9bdf9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:34:56.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.082+0000 7f562fe00640 1 --2- 172.21.15.32:0/3813778710 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5618e980b590 0x5618e9bdf4a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x5618e9be68d0 tx=0x7f561c005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78bf8473682527fd server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:56.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.082+0000 7f5621200640 1 -- 172.21.15.32:0/3813778710 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f561c020390 con 0x5618e980b590 2024-04-30T08:34:56.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.082+0000 7f5621200640 1 -- 172.21.15.32:0/3813778710 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f561c024030 con 0x5618e980b590 2024-04-30T08:34:56.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.082+0000 7f5621200640 1 -- 172.21.15.32:0/3813778710 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f561c01ec30 con 0x5618e980b590 2024-04-30T08:34:56.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.083+0000 7f563e36f740 1 -- 172.21.15.32:0/3813778710 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5618e980b590 msgr2=0x5618e9bdf4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:56.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.083+0000 7f563e36f740 1 --2- 172.21.15.32:0/3813778710 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5618e980b590 0x5618e9bdf4a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x5618e9be68d0 tx=0x7f561c005f90 comp rx=0 tx=0).stop 2024-04-30T08:34:56.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.083+0000 7f563e36f740 1 -- 172.21.15.32:0/3813778710 shutdown_connections 2024-04-30T08:34:56.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.083+0000 7f563e36f740 1 --2- 172.21.15.32:0/3813778710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618e982c270 0x5618e9bdf9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.083+0000 7f563e36f740 1 --2- 172.21.15.32:0/3813778710 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5618e980b590 0x5618e9bdf4a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.083+0000 7f563e36f740 1 --2- 172.21.15.32:0/3813778710 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618e98033b0 0x5618e9bcf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.083+0000 7f563e36f740 1 -- 172.21.15.32:0/3813778710 >> 172.21.15.32:0/3813778710 conn(0x5618e98203c0 msgr2=0x5618e9bce4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:56.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.083+0000 7f563e36f740 1 -- 172.21.15.32:0/3813778710 shutdown_connections 2024-04-30T08:34:56.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.084+0000 7f563e36f740 1 -- 172.21.15.32:0/3813778710 wait complete. 2024-04-30T08:34:56.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.084+0000 7f563e36f740 1 -- start start 2024-04-30T08:34:56.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.084+0000 7f563e36f740 1 Processor -- start 2024-04-30T08:34:56.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.085+0000 7f563e36f740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618e98033b0 0x5618e9bdb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.085+0000 7f563e36f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618e980b590 0x5618e9bdb630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.085+0000 7f563e36f740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5618e982c270 0x5618e9bde520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.085+0000 7f563e36f740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5618e9be5940 con 0x5618e98033b0 2024-04-30T08:34:56.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.085+0000 7f563e36f740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x5618e9be5270 con 0x5618e982c270 2024-04-30T08:34:56.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.085+0000 7f563e36f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5618e9be6010 con 0x5618e980b590 2024-04-30T08:34:56.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.085+0000 7f563e36f740 10 client.0 _ll_register_callbacks cb 0x7f561c009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:34:56.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.085+0000 7f5635400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618e98033b0 0x5618e9bdb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.085+0000 7f5635400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618e98033b0 0x5618e9bdb0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43580/0 (socket says 172.21.15.32:43580) 2024-04-30T08:34:56.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.085+0000 7f5635400640 1 -- 172.21.15.32:0/3995228058 learned_addr learned my addr 172.21.15.32:0/3995228058 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:56.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.085+0000 7f562fe00640 1 --2- 172.21.15.32:0/3995228058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618e980b590 0x5618e9bdb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.085+0000 7f5635400640 1 --2- 172.21.15.32:0/3995228058 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5618e982c270 0x5618e9bde520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.085+0000 7f5635400640 1 -- 172.21.15.32:0/3995228058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618e980b590 msgr2=0x5618e9bdb630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:56.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.085+0000 7f5635400640 1 --2- 172.21.15.32:0/3995228058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618e980b590 0x5618e9bdb630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.085+0000 7f5635400640 1 -- 172.21.15.32:0/3995228058 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5618e982c270 msgr2=0x5618e9bde520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:56.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.085+0000 7f5635400640 1 --2- 172.21.15.32:0/3995228058 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5618e982c270 0x5618e9bde520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.085+0000 7f5635400640 1 -- 172.21.15.32:0/3995228058 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5618e9bdeae0 con 0x5618e98033b0 2024-04-30T08:34:56.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.086+0000 7f5635400640 1 --2- 172.21.15.32:0/3995228058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618e98033b0 0x5618e9bdb0f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f562802b6c0 tx=0x7f5628004ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaa80eeed43b18ae server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:56.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.086+0000 7f5616000640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:34:56.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.086+0000 7f5616000640 1 -- 172.21.15.32:0/3995228058 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5628041020 con 0x5618e98033b0 2024-04-30T08:34:56.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.086+0000 7f5616000640 1 -- 172.21.15.32:0/3995228058 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f562802b9e0 con 0x5618e98033b0 2024-04-30T08:34:56.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.086+0000 7f563e36f740 1 -- 172.21.15.32:0/3995228058 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f561c024030 con 0x5618e98033b0 2024-04-30T08:34:56.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.086+0000 7f5616000640 1 -- 172.21.15.32:0/3995228058 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f5628049050 con 0x5618e98033b0 2024-04-30T08:34:56.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.086+0000 7f563e36f740 10 client.15309 fetch_fsmap learned FSMap version 27 2024-04-30T08:34:56.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.086+0000 7f563e36f740 1 -- 172.21.15.32:0/3995228058 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x5618e9bdfd60 con 0x5618e98033b0 2024-04-30T08:34:56.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.087+0000 7f5616000640 1 -- 172.21.15.32:0/3995228058 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f561c024030 con 0x5618e98033b0 2024-04-30T08:34:56.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.087+0000 7f5616000640 1 -- 172.21.15.32:0/3995228058 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f5628029040 con 0x5618e98033b0 2024-04-30T08:34:56.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.087+0000 7f563e36f740 10 client.15309 fetch_fsmap finished waiting for FSMap version 27 2024-04-30T08:34:56.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.087+0000 7f563e36f740 10 client.15309 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-30T08:34:56.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.087+0000 7f563e36f740 20 client.15309 populate_metadata read hostname 'smithi032' 2024-04-30T08:34:56.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.087+0000 7f563e36f740 1 --2- 172.21.15.32:0/3995228058 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x5618e97c0280 0x5618e9c51d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.087+0000 7f563e36f740 4 client.15309 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-30T08:34:56.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.087+0000 7f563e36f740 1 -- 172.21.15.32:0/3995228058 --> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] -- command(tid 0: {"prefix": "status"}) -- 0x5618e9b3a210 con 0x5618e97c0280 2024-04-30T08:34:56.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.087+0000 7f562fe00640 1 --2- 172.21.15.32:0/3995228058 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x5618e97c0280 0x5618e9c51d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.088+0000 7f5616000640 10 client.15309 ms_handle_connect on v2:172.21.15.189:6838/352970107 2024-04-30T08:34:56.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.088+0000 7f562fe00640 1 --2- 172.21.15.32:0/3995228058 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x5618e97c0280 0x5618e9c51d20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4e8295eb0477ee0f server_cookie=b2a92f28380deb4 in_seq=0 out_seq=0 2024-04-30T08:34:56.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.088+0000 7f5616000640 1 -- 172.21.15.32:0/3995228058 <== mds.2 v2:172.21.15.189:6838/352970107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5618e9b3a210 con 0x5618e97c0280 2024-04-30T08:34:56.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.088+0000 7f5616000640 10 client.15309 handle_command_reply: tid=0 2024-04-30T08:34:56.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.089+0000 7f563e36f740 1 client.15309 shutdown 2024-04-30T08:34:56.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.089+0000 7f563e36f740 1 -- 172.21.15.32:0/3995228058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618e98033b0 msgr2=0x5618e9bdb0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:56.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.089+0000 7f563e36f740 1 --2- 172.21.15.32:0/3995228058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618e98033b0 0x5618e9bdb0f0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f562802b6c0 tx=0x7f5628004ed0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.089+0000 7f563e36f740 1 -- 172.21.15.32:0/3995228058 shutdown_connections 2024-04-30T08:34:56.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.089+0000 7f563e36f740 1 --2- 172.21.15.32:0/3995228058 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x5618e97c0280 0x5618e9c51d20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.089+0000 7f563e36f740 1 --2- 172.21.15.32:0/3995228058 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5618e982c270 0x5618e9bde520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.089+0000 7f563e36f740 1 --2- 172.21.15.32:0/3995228058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5618e980b590 0x5618e9bdb630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.089+0000 7f563e36f740 1 --2- 172.21.15.32:0/3995228058 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5618e98033b0 0x5618e9bdb0f0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.089+0000 7f563e36f740 1 -- 172.21.15.32:0/3995228058 >> 172.21.15.32:0/3995228058 conn(0x5618e98203c0 msgr2=0x5618e9bd7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:56.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.089+0000 7f5616000640 0 client.15309 ms_handle_reset on v2:172.21.15.189:6838/352970107 2024-04-30T08:34:56.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.089+0000 7f563e36f740 1 -- 172.21.15.32:0/3995228058 shutdown_connections 2024-04-30T08:34:56.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.089+0000 7f563e36f740 1 -- 172.21.15.32:0/3995228058 wait complete. 2024-04-30T08:34:56.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.089+0000 7f563e36f740 20 client.15309 trim_cache size 0 max 16384 2024-04-30T08:34:56.097 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-30T08:34:56.097 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "98c0a670-06ca-11ef-bc93-c7b262605968", 2024-04-30T08:34:56.097 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 2, 2024-04-30T08:34:56.097 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 24541, 2024-04-30T08:34:56.097 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-04-30T08:34:56.097 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-04-30T08:34:56.097 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-04-30T08:34:56.097 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 148.58091318800001, 2024-04-30T08:34:56.097 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 27, 2024-04-30T08:34:56.097 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 85, 2024-04-30T08:34:56.097 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 81, 2024-04-30T08:34:56.098 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 202.33871334099999 2024-04-30T08:34:56.098 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-30T08:34:56.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.090+0000 7f5635e00640 1 -- 172.21.15.32:0/2884302835 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f56180afb60 msgr2=0x7f56180b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:56.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.090+0000 7f5635e00640 1 --2- 172.21.15.32:0/2884302835 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f56180afb60 0x7f56180b2070 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f56240fcec0 tx=0x7f5624105530 comp rx=0 tx=0).stop 2024-04-30T08:34:56.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.090+0000 7f5635e00640 1 -- 172.21.15.32:0/2884302835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5630105bb0 msgr2=0x7f563019c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:56.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.090+0000 7f5635e00640 1 --2- 172.21.15.32:0/2884302835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5630105bb0 0x7f563019c2c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f5624031440 tx=0x7f5624003c10 comp rx=0 tx=0).stop 2024-04-30T08:34:56.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.090+0000 7f5635e00640 1 -- 172.21.15.32:0/2884302835 shutdown_connections 2024-04-30T08:34:56.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.090+0000 7f5635e00640 1 --2- 172.21.15.32:0/2884302835 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f56180afb60 0x7f56180b2070 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.090+0000 7f5635e00640 1 --2- 172.21.15.32:0/2884302835 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f563010baf0 0x7f56301a3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.090+0000 7f5635e00640 1 --2- 172.21.15.32:0/2884302835 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5630106df0 0x7f563019c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.090+0000 7f5635e00640 1 --2- 172.21.15.32:0/2884302835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5630105bb0 0x7f563019c2c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.090+0000 7f5635e00640 1 -- 172.21.15.32:0/2884302835 >> 172.21.15.32:0/2884302835 conn(0x7f5630100da0 msgr2=0x7f5630107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:56.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.090+0000 7f5635e00640 1 -- 172.21.15.32:0/2884302835 shutdown_connections 2024-04-30T08:34:56.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.090+0000 7f5635e00640 1 -- 172.21.15.32:0/2884302835 wait complete. 2024-04-30T08:34:56.119 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.g status 2024-04-30T08:34:56.404 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.402+0000 7f63bec00640 1 -- 172.21.15.32:0/213119802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63b810e550 msgr2=0x7f63b810f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:56.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.402+0000 7f63bec00640 1 --2- 172.21.15.32:0/213119802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63b810e550 0x7f63b810f250 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f63a8009900 tx=0x7f63a802a3f0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.403+0000 7f63bec00640 1 -- 172.21.15.32:0/213119802 shutdown_connections 2024-04-30T08:34:56.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.403+0000 7f63bec00640 1 --2- 172.21.15.32:0/213119802 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f63b8114490 0x7f63b81168b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.403+0000 7f63bec00640 1 --2- 172.21.15.32:0/213119802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63b810f790 0x7f63b810fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.403+0000 7f63bec00640 1 --2- 172.21.15.32:0/213119802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63b810e550 0x7f63b810f250 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.403+0000 7f63bec00640 1 -- 172.21.15.32:0/213119802 >> 172.21.15.32:0/213119802 conn(0x7f63b8074990 msgr2=0x7f63b810bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:56.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.403+0000 7f63bec00640 1 -- 172.21.15.32:0/213119802 shutdown_connections 2024-04-30T08:34:56.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.403+0000 7f63bec00640 1 -- 172.21.15.32:0/213119802 wait complete. 2024-04-30T08:34:56.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.404+0000 7f63bec00640 1 Processor -- start 2024-04-30T08:34:56.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.404+0000 7f63bec00640 1 -- start start 2024-04-30T08:34:56.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.404+0000 7f63bec00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63b810e550 0x7f63b81a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.404+0000 7f63bec00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f63b810f790 0x7f63b81a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.404+0000 7f63bec00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63b8114490 0x7f63b81ac170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.404+0000 7f63bec00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f63b811a0a0 con 0x7f63b8114490 2024-04-30T08:34:56.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.404+0000 7f63bec00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f63b8119f20 con 0x7f63b810f790 2024-04-30T08:34:56.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.404+0000 7f63bec00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f63b811a220 con 0x7f63b810e550 2024-04-30T08:34:56.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.405+0000 7f63bcc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63b8114490 0x7f63b81ac170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.405+0000 7f63b7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63b810e550 0x7f63b81a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.405+0000 7f63b7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63b810e550 0x7f63b81a4cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:58636/0 (socket says 172.21.15.32:58636) 2024-04-30T08:34:56.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.405+0000 7f63b7e00640 1 -- 172.21.15.32:0/1838039393 learned_addr learned my addr 172.21.15.32:0/1838039393 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:56.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.405+0000 7f63afe00640 1 --2- 172.21.15.32:0/1838039393 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f63b810f790 0x7f63b81a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.405+0000 7f63b7e00640 1 -- 172.21.15.32:0/1838039393 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f63b810f790 msgr2=0x7f63b81a51f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:56.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.405+0000 7f63b7e00640 1 --2- 172.21.15.32:0/1838039393 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f63b810f790 0x7f63b81a51f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.405+0000 7f63b7e00640 1 -- 172.21.15.32:0/1838039393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63b8114490 msgr2=0x7f63b81ac170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:56.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.405+0000 7f63b7e00640 1 --2- 172.21.15.32:0/1838039393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63b8114490 0x7f63b81ac170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.405+0000 7f63b7e00640 1 -- 172.21.15.32:0/1838039393 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63b81ac7e0 con 0x7f63b810e550 2024-04-30T08:34:56.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.405+0000 7f63bcc00640 1 --2- 172.21.15.32:0/1838039393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63b8114490 0x7f63b81ac170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:34:56.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.405+0000 7f63afe00640 1 --2- 172.21.15.32:0/1838039393 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f63b810f790 0x7f63b81a51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:34:56.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.406+0000 7f63b7e00640 1 --2- 172.21.15.32:0/1838039393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63b810e550 0x7f63b81a4cb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f63a8031440 tx=0x7f63a8003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:56.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.406+0000 7f63b5600640 1 -- 172.21.15.32:0/1838039393 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a803f070 con 0x7f63b810e550 2024-04-30T08:34:56.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.406+0000 7f63b5600640 1 -- 172.21.15.32:0/1838039393 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63a8031c50 con 0x7f63b810e550 2024-04-30T08:34:56.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.406+0000 7f63bec00640 1 -- 172.21.15.32:0/1838039393 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63b81aca70 con 0x7f63b810e550 2024-04-30T08:34:56.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.406+0000 7f63b5600640 1 -- 172.21.15.32:0/1838039393 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a8035030 con 0x7f63b810e550 2024-04-30T08:34:56.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.406+0000 7f63bec00640 1 -- 172.21.15.32:0/1838039393 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63b81acfb0 con 0x7f63b810e550 2024-04-30T08:34:56.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.409+0000 7f63c5c0d740 1 Processor -- start 2024-04-30T08:34:56.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.410+0000 7f63c5c0d740 1 -- start start 2024-04-30T08:34:56.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.410+0000 7f63c5c0d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f73a54270 0x560f73df6340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.410+0000 7f63c5c0d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f73a2b3b0 0x560f73df68c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.410+0000 7f63c5c0d740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560f73a33590 0x560f73df7550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.410+0000 7f63c5c0d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560f73d5a790 con 0x560f73a54270 2024-04-30T08:34:56.412 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.410+0000 7f63c5c0d740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x560f73cfff20 con 0x560f73a33590 2024-04-30T08:34:56.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.410+0000 7f63c5c0d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560f73d56750 con 0x560f73a2b3b0 2024-04-30T08:34:56.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.410+0000 7f63b5600640 1 -- 172.21.15.32:0/1838039393 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f63a8042070 con 0x7f63b810e550 2024-04-30T08:34:56.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.410+0000 7f63bcc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f73a54270 0x560f73df6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.410+0000 7f63bcc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f73a54270 0x560f73df6340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43608/0 (socket says 172.21.15.32:43608) 2024-04-30T08:34:56.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.410+0000 7f63bcc00640 1 -- 172.21.15.32:0/22362344 learned_addr learned my addr 172.21.15.32:0/22362344 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:56.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.410+0000 7f63afe00640 1 --2- 172.21.15.32:0/22362344 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560f73a33590 0x560f73df7550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.411+0000 7f63b7e00640 1 --2- 172.21.15.32:0/22362344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f73a2b3b0 0x560f73df68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.411+0000 7f63bcc00640 1 -- 172.21.15.32:0/22362344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f73a2b3b0 msgr2=0x560f73df68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:56.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.411+0000 7f63bcc00640 1 --2- 172.21.15.32:0/22362344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f73a2b3b0 0x560f73df68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.411+0000 7f63bcc00640 1 -- 172.21.15.32:0/22362344 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560f73a33590 msgr2=0x560f73df7550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:56.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.411+0000 7f63bcc00640 1 --2- 172.21.15.32:0/22362344 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560f73a33590 0x560f73df7550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.411+0000 7f63bcc00640 1 -- 172.21.15.32:0/22362344 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f73a6c2d0 con 0x560f73a54270 2024-04-30T08:34:56.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.411+0000 7f63bcc00640 1 --2- 172.21.15.32:0/22362344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f73a54270 0x560f73df6340 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f63b00117e0 tx=0x7f63b0011ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d518553b0ccf332 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:56.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.411+0000 7f63ad200640 1 -- 172.21.15.32:0/22362344 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b0011310 con 0x560f73a54270 2024-04-30T08:34:56.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.412+0000 7f63ad200640 1 -- 172.21.15.32:0/22362344 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63b0003e20 con 0x560f73a54270 2024-04-30T08:34:56.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.412+0000 7f63ad200640 1 -- 172.21.15.32:0/22362344 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63b00044f0 con 0x560f73a54270 2024-04-30T08:34:56.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.412+0000 7f63b5600640 1 --2- 172.21.15.32:0/1838039393 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f639c0afbe0 0x7f639c0b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.412+0000 7f63b5600640 1 -- 172.21.15.32:0/1838039393 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f63a80ff6f0 con 0x7f63b810e550 2024-04-30T08:34:56.415 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.412+0000 7f63c5c0d740 1 -- 172.21.15.32:0/22362344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f73a54270 msgr2=0x560f73df6340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:56.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.412+0000 7f63c5c0d740 1 --2- 172.21.15.32:0/22362344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f73a54270 0x560f73df6340 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f63b00117e0 tx=0x7f63b0011ca0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.412+0000 7f63b7e00640 1 --2- 172.21.15.32:0/1838039393 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f639c0afbe0 0x7f639c0b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.413+0000 7f63c5c0d740 1 -- 172.21.15.32:0/22362344 shutdown_connections 2024-04-30T08:34:56.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.413+0000 7f63c5c0d740 1 --2- 172.21.15.32:0/22362344 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560f73a33590 0x560f73df7550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.413+0000 7f63c5c0d740 1 --2- 172.21.15.32:0/22362344 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f73a2b3b0 0x560f73df68c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.413+0000 7f63c5c0d740 1 --2- 172.21.15.32:0/22362344 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f73a54270 0x560f73df6340 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.413+0000 7f63c5c0d740 1 -- 172.21.15.32:0/22362344 >> 172.21.15.32:0/22362344 conn(0x560f73a483c0 msgr2=0x560f73df1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:56.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.413+0000 7f63c5c0d740 1 -- 172.21.15.32:0/22362344 shutdown_connections 2024-04-30T08:34:56.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.413+0000 7f63c5c0d740 1 -- 172.21.15.32:0/22362344 wait complete. 2024-04-30T08:34:56.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.413+0000 7f63b7e00640 1 --2- 172.21.15.32:0/1838039393 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f639c0afbe0 0x7f639c0b20f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f63a80c7570 tx=0x7f63a8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:56.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.413+0000 7f63c5c0d740 1 -- start start 2024-04-30T08:34:56.416 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.414+0000 7f63c5c0d740 1 Processor -- start 2024-04-30T08:34:56.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.414+0000 7f63c5c0d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f73a2b3b0 0x560f73dffeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.414+0000 7f63c5c0d740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560f73a33590 0x560f73e003f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.414+0000 7f63c5c0d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f73a54270 0x560f73e07350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.414+0000 7f63c5c0d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560f73df8060 con 0x560f73a54270 2024-04-30T08:34:56.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.414+0000 7f63c5c0d740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x560f73d139c0 con 0x560f73a33590 2024-04-30T08:34:56.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.414+0000 7f63c5c0d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560f73df81e0 con 0x560f73a2b3b0 2024-04-30T08:34:56.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.414+0000 7f63c5c0d740 10 client.0 _ll_register_callbacks cb 0x560f73ce0120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:34:56.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.415+0000 7f63bcc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f73a2b3b0 0x560f73dffeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.415+0000 7f63bcc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f73a54270 0x560f73e07350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.415+0000 7f63afe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560f73a33590 0x560f73e003f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.415+0000 7f63afe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560f73a33590 0x560f73e003f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:46560/0 (socket says 172.21.15.32:46560) 2024-04-30T08:34:56.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.415+0000 7f63afe00640 1 -- 172.21.15.32:0/784109763 learned_addr learned my addr 172.21.15.32:0/784109763 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:56.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.415+0000 7f63afe00640 1 -- 172.21.15.32:0/784109763 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f73a2b3b0 msgr2=0x560f73dffeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:56.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.415+0000 7f63afe00640 1 --2- 172.21.15.32:0/784109763 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f73a2b3b0 0x560f73dffeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.415+0000 7f63afe00640 1 -- 172.21.15.32:0/784109763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f73a54270 msgr2=0x560f73e07350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:56.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.415+0000 7f63afe00640 1 --2- 172.21.15.32:0/784109763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f73a54270 0x560f73e07350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.415+0000 7f63afe00640 1 -- 172.21.15.32:0/784109763 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f73b366b0 con 0x560f73a33590 2024-04-30T08:34:56.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.415+0000 7f63bcc00640 1 --2- 172.21.15.32:0/784109763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f73a54270 0x560f73e07350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:34:56.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.416+0000 7f63afe00640 1 --2- 172.21.15.32:0/784109763 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560f73a33590 0x560f73e003f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f63a0012d20 tx=0x7f63a000f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d23d4576a0451fe5 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:56.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.416+0000 7f63a5600640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:34:56.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.416+0000 7f63a5600640 1 -- 172.21.15.32:0/784109763 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a0012110 con 0x560f73a33590 2024-04-30T08:34:56.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.416+0000 7f63a5600640 1 -- 172.21.15.32:0/784109763 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f63a0004050 con 0x560f73a33590 2024-04-30T08:34:56.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.416+0000 7f63c5c0d740 1 -- 172.21.15.32:0/784109763 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63b0003e20 con 0x560f73a33590 2024-04-30T08:34:56.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.416+0000 7f63a5600640 1 -- 172.21.15.32:0/784109763 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f63a00122b0 con 0x560f73a33590 2024-04-30T08:34:56.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.416+0000 7f63c5c0d740 10 client.25442 fetch_fsmap learned FSMap version 27 2024-04-30T08:34:56.421 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.416+0000 7f63c5c0d740 1 -- 172.21.15.32:0/784109763 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560f73e0a3a0 con 0x560f73a33590 2024-04-30T08:34:56.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.417+0000 7f63a5600640 1 -- 172.21.15.32:0/784109763 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63b0003e20 con 0x560f73a33590 2024-04-30T08:34:56.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.417+0000 7f63a5600640 1 -- 172.21.15.32:0/784109763 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f63a0010430 con 0x560f73a33590 2024-04-30T08:34:56.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.417+0000 7f63c5c0d740 10 client.25442 fetch_fsmap finished waiting for FSMap version 27 2024-04-30T08:34:56.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.417+0000 7f63c5c0d740 10 client.25442 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-30T08:34:56.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.417+0000 7f63c5c0d740 20 client.25442 populate_metadata read hostname 'smithi032' 2024-04-30T08:34:56.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.417+0000 7f63c5c0d740 1 --2- 172.21.15.32:0/784109763 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x560f73a12e40 0x560f73e0de70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.417+0000 7f63c5c0d740 4 client.25442 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:34:56.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.417+0000 7f63c5c0d740 1 -- 172.21.15.32:0/784109763 --> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f73d683a0 con 0x560f73a12e40 2024-04-30T08:34:56.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.417+0000 7f63bcc00640 1 --2- 172.21.15.32:0/784109763 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x560f73a12e40 0x560f73e0de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.418+0000 7f63bcc00640 1 --2- 172.21.15.32:0/784109763 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x560f73a12e40 0x560f73e0de70 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=5bd1ed6ef7336b0 server_cookie=5ee6a7f0769e04f6 in_seq=0 out_seq=0 2024-04-30T08:34:56.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.418+0000 7f63a5600640 10 client.25442 ms_handle_connect on v2:172.21.15.32:6838/4160386524 2024-04-30T08:34:56.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.419+0000 7f63a5600640 1 -- 172.21.15.32:0/784109763 <== mds.1 v2:172.21.15.32:6838/4160386524 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560f73d683a0 con 0x560f73a12e40 2024-04-30T08:34:56.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.419+0000 7f63a5600640 10 client.25442 handle_command_reply: tid=0 2024-04-30T08:34:56.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.419+0000 7f63c5c0d740 1 client.25442 shutdown 2024-04-30T08:34:56.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.419+0000 7f63c5c0d740 1 -- 172.21.15.32:0/784109763 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560f73a33590 msgr2=0x560f73e003f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:56.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.419+0000 7f63c5c0d740 1 --2- 172.21.15.32:0/784109763 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560f73a33590 0x560f73e003f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f63a0012d20 tx=0x7f63a000f790 comp rx=0 tx=0).stop 2024-04-30T08:34:56.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.419+0000 7f63c5c0d740 1 -- 172.21.15.32:0/784109763 shutdown_connections 2024-04-30T08:34:56.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.419+0000 7f63c5c0d740 1 --2- 172.21.15.32:0/784109763 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x560f73a12e40 0x560f73e0de70 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.419+0000 7f63c5c0d740 1 --2- 172.21.15.32:0/784109763 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f73a54270 0x560f73e07350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.419+0000 7f63c5c0d740 1 --2- 172.21.15.32:0/784109763 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560f73a33590 0x560f73e003f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.419+0000 7f63c5c0d740 1 --2- 172.21.15.32:0/784109763 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f73a2b3b0 0x560f73dffeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.419+0000 7f63a5600640 0 client.25442 ms_handle_reset on v2:172.21.15.32:6838/4160386524 2024-04-30T08:34:56.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.419+0000 7f63c5c0d740 1 -- 172.21.15.32:0/784109763 >> 172.21.15.32:0/784109763 conn(0x560f73a483c0 msgr2=0x560f73df4a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:56.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.419+0000 7f63c5c0d740 1 -- 172.21.15.32:0/784109763 shutdown_connections 2024-04-30T08:34:56.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.419+0000 7f63c5c0d740 1 -- 172.21.15.32:0/784109763 wait complete. 2024-04-30T08:34:56.424 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.420+0000 7f63c5c0d740 20 client.25442 trim_cache size 0 max 16384 2024-04-30T08:34:56.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.444+0000 7f63c5c0d740 1 Processor -- start 2024-04-30T08:34:56.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.444+0000 7f63c5c0d740 1 -- start start 2024-04-30T08:34:56.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.444+0000 7f63c5c0d740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560f73a2b3b0 0x560f73df7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.444+0000 7f63c5c0d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f73a33590 0x560f73e07270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.444+0000 7f63c5c0d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f73a54270 0x560f73df1c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.444+0000 7f63c5c0d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560f73e0c1b0 con 0x560f73a33590 2024-04-30T08:34:56.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.444+0000 7f63c5c0d740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x560f73e0c880 con 0x560f73a2b3b0 2024-04-30T08:34:56.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.444+0000 7f63c5c0d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560f73e0cf50 con 0x560f73a54270 2024-04-30T08:34:56.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.445+0000 7f63afe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f73a33590 0x560f73e07270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.445+0000 7f63bcc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560f73a2b3b0 0x560f73df7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.445+0000 7f63afe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f73a33590 0x560f73e07270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43624/0 (socket says 172.21.15.32:43624) 2024-04-30T08:34:56.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.445+0000 7f63bcc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560f73a2b3b0 0x560f73df7f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:46576/0 (socket says 172.21.15.32:46576) 2024-04-30T08:34:56.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.445+0000 7f63afe00640 1 -- 172.21.15.32:0/3332535838 learned_addr learned my addr 172.21.15.32:0/3332535838 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:56.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.445+0000 7f63bcc00640 1 --2- 172.21.15.32:0/3332535838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f73a54270 0x560f73df1c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.445+0000 7f63afe00640 1 -- 172.21.15.32:0/3332535838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f73a54270 msgr2=0x560f73df1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:56.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.445+0000 7f63afe00640 1 --2- 172.21.15.32:0/3332535838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f73a54270 0x560f73df1c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.445+0000 7f63afe00640 1 -- 172.21.15.32:0/3332535838 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560f73a2b3b0 msgr2=0x560f73df7f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:56.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.445+0000 7f63afe00640 1 --2- 172.21.15.32:0/3332535838 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560f73a2b3b0 0x560f73df7f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.445+0000 7f63afe00640 1 -- 172.21.15.32:0/3332535838 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f73dff280 con 0x560f73a33590 2024-04-30T08:34:56.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.445+0000 7f63bcc00640 1 --2- 172.21.15.32:0/3332535838 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560f73a2b3b0 0x560f73df7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:34:56.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.446+0000 7f63afe00640 1 --2- 172.21.15.32:0/3332535838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f73a33590 0x560f73e07270 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f63a000a010 tx=0x7f63a0005870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f36f24b93def55f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:56.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.446+0000 7f63a7e00640 1 -- 172.21.15.32:0/3332535838 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a002e070 con 0x560f73a33590 2024-04-30T08:34:56.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.446+0000 7f63a7e00640 1 -- 172.21.15.32:0/3332535838 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63a0005590 con 0x560f73a33590 2024-04-30T08:34:56.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.446+0000 7f63a7e00640 1 -- 172.21.15.32:0/3332535838 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a0003850 con 0x560f73a33590 2024-04-30T08:34:56.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.447+0000 7f63c5c0d740 1 -- 172.21.15.32:0/3332535838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f73a33590 msgr2=0x560f73e07270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:56.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.447+0000 7f63c5c0d740 1 --2- 172.21.15.32:0/3332535838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f73a33590 0x560f73e07270 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f63a000a010 tx=0x7f63a0005870 comp rx=0 tx=0).stop 2024-04-30T08:34:56.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.447+0000 7f63c5c0d740 1 -- 172.21.15.32:0/3332535838 shutdown_connections 2024-04-30T08:34:56.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.447+0000 7f63c5c0d740 1 --2- 172.21.15.32:0/3332535838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f73a54270 0x560f73df1c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.447+0000 7f63c5c0d740 1 --2- 172.21.15.32:0/3332535838 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f73a33590 0x560f73e07270 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.447+0000 7f63c5c0d740 1 --2- 172.21.15.32:0/3332535838 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560f73a2b3b0 0x560f73df7f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.447+0000 7f63c5c0d740 1 -- 172.21.15.32:0/3332535838 >> 172.21.15.32:0/3332535838 conn(0x560f73a483c0 msgr2=0x560f73df1580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:56.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.447+0000 7f63c5c0d740 1 -- 172.21.15.32:0/3332535838 shutdown_connections 2024-04-30T08:34:56.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.447+0000 7f63c5c0d740 1 -- 172.21.15.32:0/3332535838 wait complete. 2024-04-30T08:34:56.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.448+0000 7f63c5c0d740 1 -- start start 2024-04-30T08:34:56.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.448+0000 7f63c5c0d740 1 Processor -- start 2024-04-30T08:34:56.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.449+0000 7f63c5c0d740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560f73a2b3b0 0x560f73e02280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.449+0000 7f63c5c0d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f73a33590 0x560f73e027c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.449+0000 7f63c5c0d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f73a54270 0x560f73e74ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.449+0000 7f63c5c0d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560f73e0dcf0 con 0x560f73a54270 2024-04-30T08:34:56.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.449+0000 7f63c5c0d740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x560f73e0d620 con 0x560f73a2b3b0 2024-04-30T08:34:56.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.449+0000 7f63c5c0d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560f73e0e3c0 con 0x560f73a33590 2024-04-30T08:34:56.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.449+0000 7f63c5c0d740 10 client.0 _ll_register_callbacks cb 0x7f63a0009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:34:56.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.449+0000 7f63bcc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f73a54270 0x560f73e74ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.449+0000 7f63afe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f73a33590 0x560f73e027c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.449+0000 7f63bcc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560f73a2b3b0 0x560f73e02280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.449+0000 7f63afe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f73a33590 0x560f73e027c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:58680/0 (socket says 172.21.15.32:58680) 2024-04-30T08:34:56.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.449+0000 7f63afe00640 1 -- 172.21.15.32:0/3240123119 learned_addr learned my addr 172.21.15.32:0/3240123119 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:56.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.450+0000 7f63afe00640 1 -- 172.21.15.32:0/3240123119 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560f73a2b3b0 msgr2=0x560f73e02280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:56.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.450+0000 7f63afe00640 1 --2- 172.21.15.32:0/3240123119 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560f73a2b3b0 0x560f73e02280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.450+0000 7f63afe00640 1 -- 172.21.15.32:0/3240123119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f73a54270 msgr2=0x560f73e74ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:56.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.450+0000 7f63afe00640 1 --2- 172.21.15.32:0/3240123119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f73a54270 0x560f73e74ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.450+0000 7f63afe00640 1 -- 172.21.15.32:0/3240123119 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f73e75220 con 0x560f73a33590 2024-04-30T08:34:56.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.450+0000 7f63bcc00640 1 --2- 172.21.15.32:0/3240123119 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560f73a2b3b0 0x560f73e02280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:34:56.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.450+0000 7f63afe00640 1 --2- 172.21.15.32:0/3240123119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f73a33590 0x560f73e027c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f63a00296b0 tx=0x7f63a00296e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49e6b48aeb14a395 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:56.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.450+0000 7f63a5600640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-30T08:34:56.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.450+0000 7f63a5600640 1 -- 172.21.15.32:0/3240123119 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63a0038060 con 0x560f73a33590 2024-04-30T08:34:56.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.450+0000 7f63c5c0d740 1 -- 172.21.15.32:0/3240123119 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63a0005590 con 0x560f73a33590 2024-04-30T08:34:56.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.450+0000 7f63a5600640 1 -- 172.21.15.32:0/3240123119 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f63a00052e0 con 0x560f73a33590 2024-04-30T08:34:56.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.451+0000 7f63a5600640 1 -- 172.21.15.32:0/3240123119 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f63a003f050 con 0x560f73a33590 2024-04-30T08:34:56.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.451+0000 7f63c5c0d740 10 client.25450 fetch_fsmap learned FSMap version 27 2024-04-30T08:34:56.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.451+0000 7f63c5c0d740 1 -- 172.21.15.32:0/3240123119 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x560f73e76430 con 0x560f73a33590 2024-04-30T08:34:56.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.451+0000 7f63a5600640 1 -- 172.21.15.32:0/3240123119 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63a0005590 con 0x560f73a33590 2024-04-30T08:34:56.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.451+0000 7f63a5600640 1 -- 172.21.15.32:0/3240123119 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f63a00036c0 con 0x560f73a33590 2024-04-30T08:34:56.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.451+0000 7f63c5c0d740 10 client.25450 fetch_fsmap finished waiting for FSMap version 27 2024-04-30T08:34:56.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.451+0000 7f63c5c0d740 10 client.25450 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-30T08:34:56.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.451+0000 7f63c5c0d740 20 client.25450 populate_metadata read hostname 'smithi032' 2024-04-30T08:34:56.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.451+0000 7f63c5c0d740 1 --2- 172.21.15.32:0/3240123119 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x560f739e8280 0x560f73e79910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.451+0000 7f63c5c0d740 4 client.25450 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-30T08:34:56.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.451+0000 7f63c5c0d740 1 -- 172.21.15.32:0/3240123119 --> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] -- command(tid 0: {"prefix": "status"}) -- 0x560f73d62210 con 0x560f739e8280 2024-04-30T08:34:56.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.452+0000 7f63bcc00640 1 --2- 172.21.15.32:0/3240123119 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x560f739e8280 0x560f73e79910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.452+0000 7f63bcc00640 1 --2- 172.21.15.32:0/3240123119 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x560f739e8280 0x560f73e79910 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6481c0ab731b236a server_cookie=f069f668b7bc7c72 in_seq=0 out_seq=0 2024-04-30T08:34:56.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.452+0000 7f63a5600640 10 client.25450 ms_handle_connect on v2:172.21.15.32:6838/4160386524 2024-04-30T08:34:56.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.452+0000 7f63a5600640 1 -- 172.21.15.32:0/3240123119 <== mds.1 v2:172.21.15.32:6838/4160386524 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x560f73d62210 con 0x560f739e8280 2024-04-30T08:34:56.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.452+0000 7f63a5600640 10 client.25450 handle_command_reply: tid=0 2024-04-30T08:34:56.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.453+0000 7f63c5c0d740 1 client.25450 shutdown 2024-04-30T08:34:56.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.453+0000 7f63c5c0d740 1 -- 172.21.15.32:0/3240123119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f73a33590 msgr2=0x560f73e027c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:56.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.453+0000 7f63c5c0d740 1 --2- 172.21.15.32:0/3240123119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f73a33590 0x560f73e027c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f63a00296b0 tx=0x7f63a00296e0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.453+0000 7f63c5c0d740 1 -- 172.21.15.32:0/3240123119 shutdown_connections 2024-04-30T08:34:56.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.453+0000 7f63c5c0d740 1 --2- 172.21.15.32:0/3240123119 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x560f739e8280 0x560f73e79910 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.453+0000 7f63c5c0d740 1 --2- 172.21.15.32:0/3240123119 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560f73a54270 0x560f73e74ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.453+0000 7f63c5c0d740 1 --2- 172.21.15.32:0/3240123119 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560f73a33590 0x560f73e027c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.453+0000 7f63c5c0d740 1 --2- 172.21.15.32:0/3240123119 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560f73a2b3b0 0x560f73e02280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.453+0000 7f63c5c0d740 1 -- 172.21.15.32:0/3240123119 >> 172.21.15.32:0/3240123119 conn(0x560f73a483c0 msgr2=0x560f73dfa6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:56.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.453+0000 7f63a5600640 0 client.25450 ms_handle_reset on v2:172.21.15.32:6838/4160386524 2024-04-30T08:34:56.457 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.453+0000 7f63c5c0d740 1 -- 172.21.15.32:0/3240123119 shutdown_connections 2024-04-30T08:34:56.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.453+0000 7f63c5c0d740 1 -- 172.21.15.32:0/3240123119 wait complete. 2024-04-30T08:34:56.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.453+0000 7f63c5c0d740 20 client.25450 trim_cache size 0 max 16384 2024-04-30T08:34:56.458 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-30T08:34:56.458 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "98c0a670-06ca-11ef-bc93-c7b262605968", 2024-04-30T08:34:56.458 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 1, 2024-04-30T08:34:56.458 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 24560, 2024-04-30T08:34:56.458 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-04-30T08:34:56.458 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-04-30T08:34:56.458 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-04-30T08:34:56.458 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 149.96044074400001, 2024-04-30T08:34:56.459 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 27, 2024-04-30T08:34:56.459 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 85, 2024-04-30T08:34:56.459 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 81, 2024-04-30T08:34:56.459 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 198.49331294800001 2024-04-30T08:34:56.459 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-30T08:34:56.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.454+0000 7f63bec00640 1 -- 172.21.15.32:0/1838039393 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f639c0afbe0 msgr2=0x7f639c0b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:56.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.454+0000 7f63bec00640 1 --2- 172.21.15.32:0/1838039393 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f639c0afbe0 0x7f639c0b20f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f63a80c7570 tx=0x7f63a8000bc0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.454+0000 7f63bec00640 1 -- 172.21.15.32:0/1838039393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63b810e550 msgr2=0x7f63b81a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:56.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.454+0000 7f63bec00640 1 --2- 172.21.15.32:0/1838039393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63b810e550 0x7f63b81a4cb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f63a8031440 tx=0x7f63a8003b90 comp rx=0 tx=0).stop 2024-04-30T08:34:56.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.454+0000 7f63bec00640 1 -- 172.21.15.32:0/1838039393 shutdown_connections 2024-04-30T08:34:56.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.454+0000 7f63bec00640 1 --2- 172.21.15.32:0/1838039393 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f639c0afbe0 0x7f639c0b20f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.454+0000 7f63bec00640 1 --2- 172.21.15.32:0/1838039393 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63b8114490 0x7f63b81ac170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.454+0000 7f63bec00640 1 --2- 172.21.15.32:0/1838039393 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f63b810f790 0x7f63b81a51f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.454+0000 7f63bec00640 1 --2- 172.21.15.32:0/1838039393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63b810e550 0x7f63b81a4cb0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.454+0000 7f63bec00640 1 -- 172.21.15.32:0/1838039393 >> 172.21.15.32:0/1838039393 conn(0x7f63b8074990 msgr2=0x7f63b8110470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:56.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.454+0000 7f63bec00640 1 -- 172.21.15.32:0/1838039393 shutdown_connections 2024-04-30T08:34:56.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.454+0000 7f63bec00640 1 -- 172.21.15.32:0/1838039393 wait complete. 2024-04-30T08:34:56.485 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-30T08:34:56.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.771+0000 7f414be00640 1 -- 172.21.15.32:0/4148380899 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4144114490 msgr2=0x7f41441168b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:56.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.771+0000 7f414be00640 1 --2- 172.21.15.32:0/4148380899 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4144114490 0x7f41441168b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f413c009900 tx=0x7f413c02a3f0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.771+0000 7f414be00640 1 -- 172.21.15.32:0/4148380899 shutdown_connections 2024-04-30T08:34:56.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.771+0000 7f414be00640 1 --2- 172.21.15.32:0/4148380899 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4144114490 0x7f41441168b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.771+0000 7f414be00640 1 --2- 172.21.15.32:0/4148380899 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f414410f790 0x7f414410fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.771+0000 7f414be00640 1 --2- 172.21.15.32:0/4148380899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f414410e550 0x7f414410f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.771+0000 7f414be00640 1 -- 172.21.15.32:0/4148380899 >> 172.21.15.32:0/4148380899 conn(0x7f4144074990 msgr2=0x7f414410bc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:56.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.771+0000 7f414be00640 1 -- 172.21.15.32:0/4148380899 shutdown_connections 2024-04-30T08:34:56.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.771+0000 7f414be00640 1 -- 172.21.15.32:0/4148380899 wait complete. 2024-04-30T08:34:56.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.772+0000 7f414be00640 1 Processor -- start 2024-04-30T08:34:56.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.773+0000 7f414be00640 1 -- start start 2024-04-30T08:34:56.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.773+0000 7f414be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f414410e550 0x7f41441a4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.773+0000 7f414be00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f414410f790 0x7f41441a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.773+0000 7f414be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4144114490 0x7f41441ac100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.773+0000 7f414be00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f414411a0c0 con 0x7f4144114490 2024-04-30T08:34:56.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.773+0000 7f414be00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4144119f40 con 0x7f414410f790 2024-04-30T08:34:56.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.773+0000 7f414be00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f414411a240 con 0x7f414410e550 2024-04-30T08:34:56.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.774+0000 7f4148a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f414410f790 0x7f41441a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.774+0000 7f4149400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f414410e550 0x7f41441a4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.774+0000 7f4148a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f414410f790 0x7f41441a5170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:46596/0 (socket says 172.21.15.32:46596) 2024-04-30T08:34:56.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.774+0000 7f4148a00640 1 -- 172.21.15.32:0/998489820 learned_addr learned my addr 172.21.15.32:0/998489820 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:56.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.774+0000 7f4149400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f414410e550 0x7f41441a4c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:58700/0 (socket says 172.21.15.32:58700) 2024-04-30T08:34:56.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.774+0000 7f4149e00640 1 --2- 172.21.15.32:0/998489820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4144114490 0x7f41441ac100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.774+0000 7f4148a00640 1 -- 172.21.15.32:0/998489820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f414410e550 msgr2=0x7f41441a4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:56.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.774+0000 7f4148a00640 1 --2- 172.21.15.32:0/998489820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f414410e550 0x7f41441a4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.774+0000 7f4148a00640 1 -- 172.21.15.32:0/998489820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4144114490 msgr2=0x7f41441ac100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:56.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.774+0000 7f4148a00640 1 --2- 172.21.15.32:0/998489820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4144114490 0x7f41441ac100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.774+0000 7f4148a00640 1 -- 172.21.15.32:0/998489820 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41441ac770 con 0x7f414410f790 2024-04-30T08:34:56.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.774+0000 7f4148a00640 1 --2- 172.21.15.32:0/998489820 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f414410f790 0x7f41441a5170 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f413400ebc0 tx=0x7f413400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:56.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.775+0000 7f4142000640 1 -- 172.21.15.32:0/998489820 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4134010630 con 0x7f414410f790 2024-04-30T08:34:56.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.775+0000 7f4142000640 1 -- 172.21.15.32:0/998489820 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4134010c80 con 0x7f414410f790 2024-04-30T08:34:56.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.775+0000 7f4142000640 1 -- 172.21.15.32:0/998489820 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4134018b60 con 0x7f414410f790 2024-04-30T08:34:56.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.775+0000 7f414be00640 1 -- 172.21.15.32:0/998489820 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41441aca60 con 0x7f414410f790 2024-04-30T08:34:56.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.775+0000 7f414be00640 1 -- 172.21.15.32:0/998489820 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41441ad000 con 0x7f414410f790 2024-04-30T08:34:56.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.778+0000 7f4152e69740 1 Processor -- start 2024-04-30T08:34:56.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.778+0000 7f4152e69740 1 -- start start 2024-04-30T08:34:56.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.779+0000 7f4152e69740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557619df0270 0x55761a192340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.779+0000 7f4152e69740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557619dc73b0 0x55761a1928c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.779+0000 7f4152e69740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557619dcf590 0x55761a193550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.779+0000 7f4152e69740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55761a0f6790 con 0x557619dcf590 2024-04-30T08:34:56.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.779+0000 7f4152e69740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55761a09bf20 con 0x557619dc73b0 2024-04-30T08:34:56.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.779+0000 7f4152e69740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55761a0f2750 con 0x557619df0270 2024-04-30T08:34:56.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.779+0000 7f4149e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557619df0270 0x55761a192340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.779+0000 7f4149400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557619dc73b0 0x55761a1928c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.779+0000 7f4149e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557619df0270 0x55761a192340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:58702/0 (socket says 172.21.15.32:58702) 2024-04-30T08:34:56.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.779+0000 7f4149e00640 1 -- 172.21.15.32:0/1441368219 learned_addr learned my addr 172.21.15.32:0/1441368219 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:56.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.780+0000 7f4149400640 1 -- 172.21.15.32:0/1441368219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557619df0270 msgr2=0x55761a192340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:56.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.780+0000 7f4149400640 1 --2- 172.21.15.32:0/1441368219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557619df0270 0x55761a192340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.780+0000 7f4149400640 1 -- 172.21.15.32:0/1441368219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557619dcf590 msgr2=0x55761a193550 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-30T08:34:56.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.780+0000 7f4149400640 1 --2- 172.21.15.32:0/1441368219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557619dcf590 0x55761a193550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.780+0000 7f4149400640 1 -- 172.21.15.32:0/1441368219 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557619e082d0 con 0x557619dc73b0 2024-04-30T08:34:56.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.780+0000 7f4149400640 1 --2- 172.21.15.32:0/1441368219 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557619dc73b0 0x55761a1928c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f414410de20 tx=0x7f4138011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94906df15a04c03e server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:56.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.780+0000 7f4132600640 1 -- 172.21.15.32:0/1441368219 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4138004230 con 0x557619dc73b0 2024-04-30T08:34:56.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.780+0000 7f4132600640 1 -- 172.21.15.32:0/1441368219 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4138009e10 con 0x557619dc73b0 2024-04-30T08:34:56.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.780+0000 7f4132600640 1 -- 172.21.15.32:0/1441368219 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4138019750 con 0x557619dc73b0 2024-04-30T08:34:56.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.780+0000 7f4142000640 1 -- 172.21.15.32:0/998489820 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f4134018d00 con 0x7f414410f790 2024-04-30T08:34:56.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.781+0000 7f4152e69740 1 -- 172.21.15.32:0/1441368219 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557619dc73b0 msgr2=0x55761a1928c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:56.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.781+0000 7f4152e69740 1 --2- 172.21.15.32:0/1441368219 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557619dc73b0 0x55761a1928c0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f414410de20 tx=0x7f4138011810 comp rx=0 tx=0).stop 2024-04-30T08:34:56.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.781+0000 7f4152e69740 1 -- 172.21.15.32:0/1441368219 shutdown_connections 2024-04-30T08:34:56.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.781+0000 7f4152e69740 1 --2- 172.21.15.32:0/1441368219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557619dcf590 0x55761a193550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.781+0000 7f4152e69740 1 --2- 172.21.15.32:0/1441368219 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557619dc73b0 0x55761a1928c0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.781+0000 7f4152e69740 1 --2- 172.21.15.32:0/1441368219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557619df0270 0x55761a192340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.781+0000 7f4152e69740 1 -- 172.21.15.32:0/1441368219 >> 172.21.15.32:0/1441368219 conn(0x557619de43c0 msgr2=0x55761a18db50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:56.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.781+0000 7f4152e69740 1 -- 172.21.15.32:0/1441368219 shutdown_connections 2024-04-30T08:34:56.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.782+0000 7f4152e69740 1 -- 172.21.15.32:0/1441368219 wait complete. 2024-04-30T08:34:56.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.782+0000 7f4152e69740 1 -- start start 2024-04-30T08:34:56.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.782+0000 7f4142000640 1 --2- 172.21.15.32:0/998489820 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f412c0afa80 0x7f412c0b1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.782+0000 7f4152e69740 1 Processor -- start 2024-04-30T08:34:56.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.783+0000 7f4142000640 1 -- 172.21.15.32:0/998489820 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f41340df5a0 con 0x7f414410f790 2024-04-30T08:34:56.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.783+0000 7f4149e00640 1 --2- 172.21.15.32:0/998489820 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f412c0afa80 0x7f412c0b1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.783+0000 7f4152e69740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557619dc73b0 0x55761a19bf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.783+0000 7f4152e69740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557619dcf590 0x55761a19c4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.783+0000 7f4152e69740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557619df0270 0x55761a1a33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.783+0000 7f4152e69740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55761a194060 con 0x557619df0270 2024-04-30T08:34:56.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.783+0000 7f4152e69740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55761a0af9c0 con 0x557619dcf590 2024-04-30T08:34:56.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.783+0000 7f4152e69740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55761a1941e0 con 0x557619dc73b0 2024-04-30T08:34:56.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.783+0000 7f4152e69740 10 client.0 _ll_register_callbacks cb 0x55761a07c120 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:34:56.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.784+0000 7f4148a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557619dcf590 0x55761a19c4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.784+0000 7f4149400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557619dc73b0 0x55761a19bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.784+0000 7f4148a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557619dcf590 0x55761a19c4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:46604/0 (socket says 172.21.15.32:46604) 2024-04-30T08:34:56.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.784+0000 7f4148a00640 1 -- 172.21.15.32:0/861050455 learned_addr learned my addr 172.21.15.32:0/861050455 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:56.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.784+0000 7f4149e00640 1 --2- 172.21.15.32:0/861050455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557619df0270 0x55761a1a33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.784+0000 7f4148a00640 1 -- 172.21.15.32:0/861050455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557619dc73b0 msgr2=0x55761a19bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:56.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.784+0000 7f4148a00640 1 --2- 172.21.15.32:0/861050455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557619dc73b0 0x55761a19bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.784+0000 7f4148a00640 1 -- 172.21.15.32:0/861050455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557619df0270 msgr2=0x55761a1a33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:56.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.784+0000 7f4148a00640 1 --2- 172.21.15.32:0/861050455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557619df0270 0x55761a1a33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.784+0000 7f4148a00640 1 -- 172.21.15.32:0/861050455 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557619ed26b0 con 0x557619dcf590 2024-04-30T08:34:56.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.784+0000 7f4149e00640 1 --2- 172.21.15.32:0/998489820 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f412c0afa80 0x7f412c0b1f90 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f413c006010 tx=0x7f413c005f30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:56.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.784+0000 7f4148a00640 1 --2- 172.21.15.32:0/861050455 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557619dcf590 0x55761a19c4b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f41340a76b0 tx=0x7f41340a74f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfd94c1ca6c5a95d server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:56.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.784+0000 7f4126a00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:34:56.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.784+0000 7f4126a00640 1 -- 172.21.15.32:0/861050455 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4134018420 con 0x557619dcf590 2024-04-30T08:34:56.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.785+0000 7f4126a00640 1 -- 172.21.15.32:0/861050455 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f41340900b0 con 0x557619dcf590 2024-04-30T08:34:56.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.785+0000 7f4152e69740 1 -- 172.21.15.32:0/861050455 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4138009e10 con 0x557619dcf590 2024-04-30T08:34:56.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.785+0000 7f4126a00640 1 -- 172.21.15.32:0/861050455 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f41340a7230 con 0x557619dcf590 2024-04-30T08:34:56.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.785+0000 7f4152e69740 10 client.25460 fetch_fsmap learned FSMap version 27 2024-04-30T08:34:56.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.785+0000 7f4152e69740 1 -- 172.21.15.32:0/861050455 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55761a1a6340 con 0x557619dcf590 2024-04-30T08:34:56.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.785+0000 7f4126a00640 1 -- 172.21.15.32:0/861050455 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4138009e10 con 0x557619dcf590 2024-04-30T08:34:56.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.785+0000 7f4126a00640 1 -- 172.21.15.32:0/861050455 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f41340a6980 con 0x557619dcf590 2024-04-30T08:34:56.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.785+0000 7f4152e69740 10 client.25460 fetch_fsmap finished waiting for FSMap version 27 2024-04-30T08:34:56.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.785+0000 7f4152e69740 10 client.25460 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-30T08:34:56.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.785+0000 7f4152e69740 20 client.25460 populate_metadata read hostname 'smithi032' 2024-04-30T08:34:56.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.785+0000 7f4152e69740 1 --2- 172.21.15.32:0/861050455 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x557619daee40 0x55761a1a9d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.785+0000 7f4152e69740 4 client.25460 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:34:56.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.785+0000 7f4152e69740 1 -- 172.21.15.32:0/861050455 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55761a1043a0 con 0x557619daee40 2024-04-30T08:34:56.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.786+0000 7f4149400640 1 --2- 172.21.15.32:0/861050455 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x557619daee40 0x55761a1a9d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.787+0000 7f4149400640 1 --2- 172.21.15.32:0/861050455 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x557619daee40 0x55761a1a9d90 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ebe162488497203 server_cookie=621fe5c82fcf70b5 in_seq=0 out_seq=0 2024-04-30T08:34:56.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.787+0000 7f4126a00640 10 client.25460 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:34:56.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.787+0000 7f4126a00640 1 -- 172.21.15.32:0/861050455 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55761a1043a0 con 0x557619daee40 2024-04-30T08:34:56.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.787+0000 7f4126a00640 10 client.25460 handle_command_reply: tid=0 2024-04-30T08:34:56.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.787+0000 7f4152e69740 1 client.25460 shutdown 2024-04-30T08:34:56.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.788+0000 7f4152e69740 1 -- 172.21.15.32:0/861050455 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557619dcf590 msgr2=0x55761a19c4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:56.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.788+0000 7f4152e69740 1 --2- 172.21.15.32:0/861050455 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557619dcf590 0x55761a19c4b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f41340a76b0 tx=0x7f41340a74f0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.788+0000 7f4152e69740 1 -- 172.21.15.32:0/861050455 shutdown_connections 2024-04-30T08:34:56.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.788+0000 7f4152e69740 1 --2- 172.21.15.32:0/861050455 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x557619daee40 0x55761a1a9d90 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.788+0000 7f4152e69740 1 --2- 172.21.15.32:0/861050455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557619df0270 0x55761a1a33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.788+0000 7f4152e69740 1 --2- 172.21.15.32:0/861050455 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557619dcf590 0x55761a19c4b0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.788+0000 7f4126a00640 0 client.25460 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:34:56.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.788+0000 7f4152e69740 1 --2- 172.21.15.32:0/861050455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557619dc73b0 0x55761a19bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.788+0000 7f4152e69740 1 -- 172.21.15.32:0/861050455 >> 172.21.15.32:0/861050455 conn(0x557619de43c0 msgr2=0x55761a190a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:56.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.788+0000 7f4152e69740 1 -- 172.21.15.32:0/861050455 shutdown_connections 2024-04-30T08:34:56.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.788+0000 7f4152e69740 1 -- 172.21.15.32:0/861050455 wait complete. 2024-04-30T08:34:56.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.788+0000 7f4152e69740 20 client.25460 trim_cache size 0 max 16384 2024-04-30T08:34:56.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.809+0000 7f4152e69740 1 Processor -- start 2024-04-30T08:34:56.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.809+0000 7f4152e69740 1 -- start start 2024-04-30T08:34:56.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.809+0000 7f4152e69740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557619dc73b0 0x55761a188930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.810+0000 7f4152e69740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557619dcf590 0x55761a1a32c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.810+0000 7f4152e69740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557619df0270 0x55761a1922c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.810+0000 7f4152e69740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55761a1a80d0 con 0x557619dc73b0 2024-04-30T08:34:56.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.810+0000 7f4152e69740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55761a1a87a0 con 0x557619df0270 2024-04-30T08:34:56.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.810+0000 7f4152e69740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55761a1a8e70 con 0x557619dcf590 2024-04-30T08:34:56.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.810+0000 7f4148a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557619dcf590 0x55761a1a32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.810+0000 7f4149400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557619dc73b0 0x55761a188930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.810+0000 7f4149e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557619df0270 0x55761a1922c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.810+0000 7f4148a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557619dcf590 0x55761a1a32c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:58716/0 (socket says 172.21.15.32:58716) 2024-04-30T08:34:56.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.810+0000 7f4148a00640 1 -- 172.21.15.32:0/2453231917 learned_addr learned my addr 172.21.15.32:0/2453231917 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:56.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.810+0000 7f4148a00640 1 -- 172.21.15.32:0/2453231917 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557619df0270 msgr2=0x55761a1922c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:56.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.810+0000 7f4148a00640 1 --2- 172.21.15.32:0/2453231917 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557619df0270 0x55761a1922c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.810+0000 7f4148a00640 1 -- 172.21.15.32:0/2453231917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557619dc73b0 msgr2=0x55761a188930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:56.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.810+0000 7f4148a00640 1 --2- 172.21.15.32:0/2453231917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557619dc73b0 0x55761a188930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.810+0000 7f4148a00640 1 -- 172.21.15.32:0/2453231917 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55761a1976d0 con 0x557619dcf590 2024-04-30T08:34:56.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.811+0000 7f4149400640 1 --2- 172.21.15.32:0/2453231917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557619dc73b0 0x55761a188930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:34:56.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.811+0000 7f4148a00640 1 --2- 172.21.15.32:0/2453231917 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557619dcf590 0x55761a1a32c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f413400cba0 tx=0x7f413400ccd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66a04322acafc0fc server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:56.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.811+0000 7f4131c00640 1 -- 172.21.15.32:0/2453231917 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41340a7070 con 0x557619dcf590 2024-04-30T08:34:56.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.811+0000 7f4131c00640 1 -- 172.21.15.32:0/2453231917 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41340c5020 con 0x557619dcf590 2024-04-30T08:34:56.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.811+0000 7f4131c00640 1 -- 172.21.15.32:0/2453231917 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41340a6980 con 0x557619dcf590 2024-04-30T08:34:56.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.812+0000 7f4152e69740 1 -- 172.21.15.32:0/2453231917 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557619dcf590 msgr2=0x55761a1a32c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:56.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.812+0000 7f4152e69740 1 --2- 172.21.15.32:0/2453231917 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557619dcf590 0x55761a1a32c0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f413400cba0 tx=0x7f413400ccd0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.812+0000 7f4152e69740 1 -- 172.21.15.32:0/2453231917 shutdown_connections 2024-04-30T08:34:56.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.812+0000 7f4152e69740 1 --2- 172.21.15.32:0/2453231917 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557619df0270 0x55761a1922c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.812+0000 7f4152e69740 1 --2- 172.21.15.32:0/2453231917 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557619dcf590 0x55761a1a32c0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.812+0000 7f4152e69740 1 --2- 172.21.15.32:0/2453231917 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557619dc73b0 0x55761a188930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.812+0000 7f4152e69740 1 -- 172.21.15.32:0/2453231917 >> 172.21.15.32:0/2453231917 conn(0x557619de43c0 msgr2=0x55761a191380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:56.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.812+0000 7f4152e69740 1 -- 172.21.15.32:0/2453231917 shutdown_connections 2024-04-30T08:34:56.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.812+0000 7f4152e69740 1 -- 172.21.15.32:0/2453231917 wait complete. 2024-04-30T08:34:56.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.813+0000 7f4152e69740 1 -- start start 2024-04-30T08:34:56.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.813+0000 7f4152e69740 1 Processor -- start 2024-04-30T08:34:56.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.813+0000 7f4152e69740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557619dc73b0 0x55761a1a0050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.813+0000 7f4152e69740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557619dcf590 0x55761a1a0590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.813+0000 7f4152e69740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557619df0270 0x55761a210c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.813+0000 7f4152e69740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55761a1a9c10 con 0x557619dc73b0 2024-04-30T08:34:56.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.813+0000 7f4152e69740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55761a1a9540 con 0x557619df0270 2024-04-30T08:34:56.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.813+0000 7f4152e69740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55761a1aa2e0 con 0x557619dcf590 2024-04-30T08:34:56.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.813+0000 7f4152e69740 10 client.0 _ll_register_callbacks cb 0x7f4134009a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:34:56.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.814+0000 7f4148a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557619dcf590 0x55761a1a0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.814+0000 7f4149400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557619dc73b0 0x55761a1a0050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.814+0000 7f4148a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557619dcf590 0x55761a1a0590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:58720/0 (socket says 172.21.15.32:58720) 2024-04-30T08:34:56.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.814+0000 7f4148a00640 1 -- 172.21.15.32:0/1567328441 learned_addr learned my addr 172.21.15.32:0/1567328441 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:34:56.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.814+0000 7f4149e00640 1 --2- 172.21.15.32:0/1567328441 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557619df0270 0x55761a210c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.814+0000 7f4148a00640 1 -- 172.21.15.32:0/1567328441 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557619df0270 msgr2=0x55761a210c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:56.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.814+0000 7f4148a00640 1 --2- 172.21.15.32:0/1567328441 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557619df0270 0x55761a210c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.814+0000 7f4148a00640 1 -- 172.21.15.32:0/1567328441 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557619dc73b0 msgr2=0x55761a1a0050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:34:56.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.814+0000 7f4148a00640 1 --2- 172.21.15.32:0/1567328441 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557619dc73b0 0x55761a1a0050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.814+0000 7f4148a00640 1 -- 172.21.15.32:0/1567328441 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55761a1a3500 con 0x557619dcf590 2024-04-30T08:34:56.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.814+0000 7f4148a00640 1 --2- 172.21.15.32:0/1567328441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557619dcf590 0x55761a1a0590 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4134010f90 tx=0x7f41340a74f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bb88481ef263842 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:34:56.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.814+0000 7f4126a00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-30T08:34:56.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.815+0000 7f4126a00640 1 -- 172.21.15.32:0/1567328441 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4134086070 con 0x557619dcf590 2024-04-30T08:34:56.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.815+0000 7f4152e69740 1 -- 172.21.15.32:0/1567328441 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41340c5020 con 0x557619dcf590 2024-04-30T08:34:56.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.815+0000 7f4126a00640 1 -- 172.21.15.32:0/1567328441 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f41340107d0 con 0x557619dcf590 2024-04-30T08:34:56.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.815+0000 7f4126a00640 1 -- 172.21.15.32:0/1567328441 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) ==== 24+0+0 (secure 0 0 0) 0x7f41340a7070 con 0x557619dcf590 2024-04-30T08:34:56.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.815+0000 7f4152e69740 10 client.25468 fetch_fsmap learned FSMap version 27 2024-04-30T08:34:56.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.815+0000 7f4152e69740 1 -- 172.21.15.32:0/1567328441 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) -- 0x55761a2120c0 con 0x557619dcf590 2024-04-30T08:34:56.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.815+0000 7f4126a00640 1 -- 172.21.15.32:0/1567328441 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f41340c5020 con 0x557619dcf590 2024-04-30T08:34:56.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.815+0000 7f4126a00640 1 -- 172.21.15.32:0/1567328441 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 27) ==== 8362+0+0 (secure 0 0 0) 0x7f41340a63f0 con 0x557619dcf590 2024-04-30T08:34:56.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.815+0000 7f4152e69740 10 client.25468 fetch_fsmap finished waiting for FSMap version 27 2024-04-30T08:34:56.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.816+0000 7f4152e69740 10 client.25468 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-30T08:34:56.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.816+0000 7f4152e69740 20 client.25468 populate_metadata read hostname 'smithi032' 2024-04-30T08:34:56.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.816+0000 7f4152e69740 1 --2- 172.21.15.32:0/1567328441 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x557619d84280 0x55761a215560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:34:56.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.816+0000 7f4152e69740 4 client.25468 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-30T08:34:56.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.816+0000 7f4152e69740 1 -- 172.21.15.32:0/1567328441 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "status"}) -- 0x55761a0fe210 con 0x557619d84280 2024-04-30T08:34:56.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.816+0000 7f4149400640 1 --2- 172.21.15.32:0/1567328441 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x557619d84280 0x55761a215560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:34:56.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.816+0000 7f4149400640 1 --2- 172.21.15.32:0/1567328441 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x557619d84280 0x55761a215560 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dac00ce78221a24a server_cookie=99541115d70d9b2e in_seq=0 out_seq=0 2024-04-30T08:34:56.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.816+0000 7f4126a00640 10 client.25468 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:34:56.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.817+0000 7f4126a00640 1 -- 172.21.15.32:0/1567328441 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55761a0fe210 con 0x557619d84280 2024-04-30T08:34:56.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.817+0000 7f4126a00640 10 client.25468 handle_command_reply: tid=0 2024-04-30T08:34:56.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.817+0000 7f4152e69740 1 client.25468 shutdown 2024-04-30T08:34:56.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.817+0000 7f4152e69740 1 -- 172.21.15.32:0/1567328441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557619dcf590 msgr2=0x55761a1a0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:56.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.817+0000 7f4152e69740 1 --2- 172.21.15.32:0/1567328441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557619dcf590 0x55761a1a0590 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f4134010f90 tx=0x7f41340a74f0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.817+0000 7f4152e69740 1 -- 172.21.15.32:0/1567328441 shutdown_connections 2024-04-30T08:34:56.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.817+0000 7f4152e69740 1 --2- 172.21.15.32:0/1567328441 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x557619d84280 0x55761a215560 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.817+0000 7f4152e69740 1 --2- 172.21.15.32:0/1567328441 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557619df0270 0x55761a210c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.817+0000 7f4152e69740 1 --2- 172.21.15.32:0/1567328441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557619dcf590 0x55761a1a0590 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.817+0000 7f4152e69740 1 --2- 172.21.15.32:0/1567328441 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557619dc73b0 0x55761a1a0050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.817+0000 7f4126a00640 0 client.25468 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:34:56.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.817+0000 7f4152e69740 1 -- 172.21.15.32:0/1567328441 >> 172.21.15.32:0/1567328441 conn(0x557619de43c0 msgr2=0x55761a190950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:56.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.817+0000 7f4152e69740 1 -- 172.21.15.32:0/1567328441 shutdown_connections 2024-04-30T08:34:56.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.817+0000 7f4152e69740 1 -- 172.21.15.32:0/1567328441 wait complete. 2024-04-30T08:34:56.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.817+0000 7f4152e69740 20 client.25468 trim_cache size 0 max 16384 2024-04-30T08:34:56.824 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-30T08:34:56.824 INFO:teuthology.orchestra.run.smithi032.stdout: "cluster_fsid": "98c0a670-06ca-11ef-bc93-c7b262605968", 2024-04-30T08:34:56.824 INFO:teuthology.orchestra.run.smithi032.stdout: "whoami": 0, 2024-04-30T08:34:56.824 INFO:teuthology.orchestra.run.smithi032.stdout: "id": 24590, 2024-04-30T08:34:56.824 INFO:teuthology.orchestra.run.smithi032.stdout: "want_state": "up:active", 2024-04-30T08:34:56.825 INFO:teuthology.orchestra.run.smithi032.stdout: "state": "up:active", 2024-04-30T08:34:56.825 INFO:teuthology.orchestra.run.smithi032.stdout: "fs_name": "cephfs", 2024-04-30T08:34:56.825 INFO:teuthology.orchestra.run.smithi032.stdout: "rank_uptime": 183.585976225, 2024-04-30T08:34:56.825 INFO:teuthology.orchestra.run.smithi032.stdout: "mdsmap_epoch": 27, 2024-04-30T08:34:56.825 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch": 85, 2024-04-30T08:34:56.825 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_epoch_barrier": 81, 2024-04-30T08:34:56.825 INFO:teuthology.orchestra.run.smithi032.stdout: "uptime": 192.028610593 2024-04-30T08:34:56.825 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-30T08:34:56.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.818+0000 7f414be00640 1 -- 172.21.15.32:0/998489820 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f412c0afa80 msgr2=0x7f412c0b1f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:56.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.818+0000 7f414be00640 1 --2- 172.21.15.32:0/998489820 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f412c0afa80 0x7f412c0b1f90 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f413c006010 tx=0x7f413c005f30 comp rx=0 tx=0).stop 2024-04-30T08:34:56.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.818+0000 7f414be00640 1 -- 172.21.15.32:0/998489820 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f414410f790 msgr2=0x7f41441a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:34:56.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.818+0000 7f414be00640 1 --2- 172.21.15.32:0/998489820 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f414410f790 0x7f41441a5170 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f413400ebc0 tx=0x7f413400c6a0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.818+0000 7f414be00640 1 -- 172.21.15.32:0/998489820 shutdown_connections 2024-04-30T08:34:56.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.818+0000 7f414be00640 1 --2- 172.21.15.32:0/998489820 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f412c0afa80 0x7f412c0b1f90 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.818+0000 7f414be00640 1 --2- 172.21.15.32:0/998489820 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4144114490 0x7f41441ac100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.818+0000 7f414be00640 1 --2- 172.21.15.32:0/998489820 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f414410f790 0x7f41441a5170 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.818+0000 7f414be00640 1 --2- 172.21.15.32:0/998489820 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f414410e550 0x7f41441a4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:34:56.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.818+0000 7f414be00640 1 -- 172.21.15.32:0/998489820 >> 172.21.15.32:0/998489820 conn(0x7f4144074990 msgr2=0x7f4144110470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:34:56.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.818+0000 7f414be00640 1 -- 172.21.15.32:0/998489820 shutdown_connections 2024-04-30T08:34:56.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:34:56.818+0000 7f414be00640 1 -- 172.21.15.32:0/998489820 wait complete. 2024-04-30T08:34:56.847 INFO:teuthology.run_tasks:Running task sequential... 2024-04-30T08:34:56.857 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-30T08:34:56.857 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-30T08:34:56.857 INFO:teuthology.task.print:Enabling mgr modules 2024-04-30T08:34:56.857 INFO:teuthology.run_tasks:Running task kclient... 2024-04-30T08:34:56.870 INFO:tasks.kclient:Mounting kernel clients... 2024-04-30T08:34:56.870 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-30T08:34:56.870 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-30T08:34:56.870 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-04-30T08:34:56.924 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-04-30T08:34:56.925 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-30T08:34:56.925 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-04-30T08:34:57.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-30T08:34:57.015 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link delete ceph-brx 2024-04-30T08:34:57.119 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:57 smithi032 ceph-mon[28331]: pgmap v418: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-30T08:34:57.137 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'nowsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-04-30T08:34:57.137 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-30T08:34:57.137 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-30T08:34:57.137 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-30T08:34:57.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:57 smithi071 ceph-mon[33385]: pgmap v418: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-30T08:34:57.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:57 smithi189 ceph-mon[33038]: pgmap v418: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-30T08:34:57.594 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:34:59.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:34:59 smithi032 ceph-mon[28331]: pgmap v419: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-30T08:34:59.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:34:59 smithi071 ceph-mon[33385]: pgmap v419: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-30T08:34:59.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:34:59 smithi189 ceph-mon[33038]: pgmap v419: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 8 op/s 2024-04-30T08:35:00.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.021+0000 7f9612200700 1 -- 172.21.15.32:0/1761097667 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f960c104350 msgr2=0x7f960c104760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:00.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.021+0000 7f9612200700 1 --2- 172.21.15.32:0/1761097667 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f960c104350 0x7f960c104760 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f96000099c0 tx=0x7f9600009cd0 comp rx=0 tx=0).stop 2024-04-30T08:35:00.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.021+0000 7f9612200700 1 -- 172.21.15.32:0/1761097667 shutdown_connections 2024-04-30T08:35:00.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.021+0000 7f9612200700 1 --2- 172.21.15.32:0/1761097667 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f960c1095d0 0x7f960c110f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:00.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.021+0000 7f9612200700 1 --2- 172.21.15.32:0/1761097667 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f960c104350 0x7f960c104760 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:00.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.021+0000 7f9612200700 1 --2- 172.21.15.32:0/1761097667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f960c108c80 0x7f960c109090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:00.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.021+0000 7f9612200700 1 -- 172.21.15.32:0/1761097667 >> 172.21.15.32:0/1761097667 conn(0x7f960c079c60 msgr2=0x7f960c07a0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:00.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.021+0000 7f9612200700 1 -- 172.21.15.32:0/1761097667 shutdown_connections 2024-04-30T08:35:00.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.021+0000 7f9612200700 1 -- 172.21.15.32:0/1761097667 wait complete. 2024-04-30T08:35:00.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.022+0000 7f9612200700 1 Processor -- start 2024-04-30T08:35:00.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.022+0000 7f9612200700 1 -- start start 2024-04-30T08:35:00.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.023+0000 7f9612200700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f960c104350 0x7f960c075a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:00.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.023+0000 7f9612200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f960c108c80 0x7f960c075f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:00.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.023+0000 7f9612200700 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f960c1095d0 0x7f960c076490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:00.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.023+0000 7f9612200700 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f960c114250 con 0x7f960c104350 2024-04-30T08:35:00.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.023+0000 7f9612200700 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f960c1140d0 con 0x7f960c1095d0 2024-04-30T08:35:00.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.023+0000 7f9612200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f960c1143d0 con 0x7f960c108c80 2024-04-30T08:35:00.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.023+0000 7f9610e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f960c104350 0x7f960c075a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:00.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.023+0000 7f9610e00700 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f960c104350 0x7f960c075a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37760/0 (socket says 172.21.15.32:37760) 2024-04-30T08:35:00.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.023+0000 7f9610e00700 1 -- 172.21.15.32:0/1768754230 learned_addr learned my addr 172.21.15.32:0/1768754230 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:00.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.023+0000 7f960be00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f960c108c80 0x7f960c075f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:00.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.023+0000 7f9611800700 1 --2- 172.21.15.32:0/1768754230 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f960c1095d0 0x7f960c076490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:00.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.024+0000 7f9610e00700 1 -- 172.21.15.32:0/1768754230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f960c108c80 msgr2=0x7f960c075f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:00.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.024+0000 7f9610e00700 1 --2- 172.21.15.32:0/1768754230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f960c108c80 0x7f960c075f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:00.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.024+0000 7f9610e00700 1 -- 172.21.15.32:0/1768754230 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f960c1095d0 msgr2=0x7f960c076490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:00.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.024+0000 7f9610e00700 1 --2- 172.21.15.32:0/1768754230 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f960c1095d0 0x7f960c076490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:00.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.024+0000 7f9610e00700 1 -- 172.21.15.32:0/1768754230 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f960c1a8f40 con 0x7f960c104350 2024-04-30T08:35:00.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.024+0000 7f960be00700 1 --2- 172.21.15.32:0/1768754230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f960c108c80 0x7f960c075f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:00.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.024+0000 7f9610e00700 1 --2- 172.21.15.32:0/1768754230 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f960c104350 0x7f960c075a10 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9600006010 tx=0x7f9600005280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:00.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.024+0000 7f9609600700 1 -- 172.21.15.32:0/1768754230 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9600026070 con 0x7f960c104350 2024-04-30T08:35:00.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.025+0000 7f9612200700 1 -- 172.21.15.32:0/1768754230 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f960c1a91d0 con 0x7f960c104350 2024-04-30T08:35:00.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.025+0000 7f9609600700 1 -- 172.21.15.32:0/1768754230 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9600005680 con 0x7f960c104350 2024-04-30T08:35:00.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.025+0000 7f9609600700 1 -- 172.21.15.32:0/1768754230 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f960000fb20 con 0x7f960c104350 2024-04-30T08:35:00.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.025+0000 7f9612200700 1 -- 172.21.15.32:0/1768754230 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f960c1a9620 con 0x7f960c104350 2024-04-30T08:35:00.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.026+0000 7f9612200700 1 -- 172.21.15.32:0/1768754230 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f960c105800 con 0x7f960c104350 2024-04-30T08:35:00.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.028+0000 7f9609600700 1 -- 172.21.15.32:0/1768754230 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f9600029070 con 0x7f960c104350 2024-04-30T08:35:00.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.030+0000 7f9609600700 1 --2- 172.21.15.32:0/1768754230 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f95f40afb80 0x7f95f40b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:00.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.030+0000 7f9609600700 1 -- 172.21.15.32:0/1768754230 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f96000dff90 con 0x7f960c104350 2024-04-30T08:35:00.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.030+0000 7f960be00700 1 --2- 172.21.15.32:0/1768754230 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f95f40afb80 0x7f95f40b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:00.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.031+0000 7f960be00700 1 --2- 172.21.15.32:0/1768754230 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f95f40afb80 0x7f95f40b2090 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f960c07a220 tx=0x7f95f8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:00.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.037+0000 7f9609600700 1 -- 172.21.15.32:0/1768754230 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9600030080 con 0x7f960c104350 2024-04-30T08:35:00.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.317+0000 7f9612200700 1 -- 172.21.15.32:0/1768754230 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f960c02e590 con 0x7f960c104350 2024-04-30T08:35:00.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.318+0000 7f9609600700 1 -- 172.21.15.32:0/1768754230 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) ==== 53+0+88 (secure 0 0 0) 0x7f9600013070 con 0x7f960c104350 2024-04-30T08:35:00.320 INFO:teuthology.orchestra.run.smithi032.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-04-30T08:35:00.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.322+0000 7f9612200700 1 -- 172.21.15.32:0/1768754230 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f95f40afb80 msgr2=0x7f95f40b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:00.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.322+0000 7f9612200700 1 --2- 172.21.15.32:0/1768754230 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f95f40afb80 0x7f95f40b2090 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f960c07a220 tx=0x7f95f8009260 comp rx=0 tx=0).stop 2024-04-30T08:35:00.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.322+0000 7f9612200700 1 -- 172.21.15.32:0/1768754230 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f960c104350 msgr2=0x7f960c075a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:00.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.322+0000 7f9612200700 1 --2- 172.21.15.32:0/1768754230 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f960c104350 0x7f960c075a10 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9600006010 tx=0x7f9600005280 comp rx=0 tx=0).stop 2024-04-30T08:35:00.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.322+0000 7f9612200700 1 -- 172.21.15.32:0/1768754230 shutdown_connections 2024-04-30T08:35:00.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.322+0000 7f9612200700 1 --2- 172.21.15.32:0/1768754230 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f95f40afb80 0x7f95f40b2090 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:00.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.322+0000 7f9612200700 1 --2- 172.21.15.32:0/1768754230 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f960c1095d0 0x7f960c076490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:00.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.322+0000 7f9612200700 1 --2- 172.21.15.32:0/1768754230 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f960c104350 0x7f960c075a10 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:00.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.322+0000 7f9612200700 1 --2- 172.21.15.32:0/1768754230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f960c108c80 0x7f960c075f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:00.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.322+0000 7f9612200700 1 -- 172.21.15.32:0/1768754230 >> 172.21.15.32:0/1768754230 conn(0x7f960c079c60 msgr2=0x7f960c101db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:00.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.322+0000 7f9612200700 1 -- 172.21.15.32:0/1768754230 shutdown_connections 2024-04-30T08:35:00.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:00.322+0000 7f9612200700 1 -- 172.21.15.32:0/1768754230 wait complete. 2024-04-30T08:35:01.282 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-30T08:35:01.282 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-30T08:35:01.282 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi032.front.sepia.ceph.com 2024-04-30T08:35:01.282 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-30T08:35:01.282 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-30T08:35:01.282 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-30T08:35:01.282 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-30T08:35:01.282 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-30T08:35:01.282 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-30T08:35:01.282 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-30T08:35:01.283 DEBUG:teuthology.orchestra.run.smithi032:> ip addr 2024-04-30T08:35:01.344 INFO:teuthology.orchestra.run.smithi032.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-30T08:35:01.345 INFO:teuthology.orchestra.run.smithi032.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-30T08:35:01.345 INFO:teuthology.orchestra.run.smithi032.stdout: inet 127.0.0.1/8 scope host lo 2024-04-30T08:35:01.345 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-04-30T08:35:01.345 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 ::1/128 scope host 2024-04-30T08:35:01.345 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-04-30T08:35:01.345 INFO:teuthology.orchestra.run.smithi032.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-30T08:35:01.345 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0c brd ff:ff:ff:ff:ff:ff 2024-04-30T08:35:01.345 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f0 2024-04-30T08:35:01.345 INFO:teuthology.orchestra.run.smithi032.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-30T08:35:01.345 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether 0c:c4:7a:6b:fd:0d brd ff:ff:ff:ff:ff:ff 2024-04-30T08:35:01.346 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp7s0f1 2024-04-30T08:35:01.346 INFO:teuthology.orchestra.run.smithi032.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-30T08:35:01.346 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:02 brd ff:ff:ff:ff:ff:ff 2024-04-30T08:35:01.346 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f0 2024-04-30T08:35:01.346 INFO:teuthology.orchestra.run.smithi032.stdout: inet 172.21.15.32/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-30T08:35:01.346 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft 41844sec preferred_lft 41844sec 2024-04-30T08:35:01.346 INFO:teuthology.orchestra.run.smithi032.stdout: inet6 fe80::ae1f:6bff:fef8:2602/64 scope link noprefixroute 2024-04-30T08:35:01.346 INFO:teuthology.orchestra.run.smithi032.stdout: valid_lft forever preferred_lft forever 2024-04-30T08:35:01.346 INFO:teuthology.orchestra.run.smithi032.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-30T08:35:01.346 INFO:teuthology.orchestra.run.smithi032.stdout: link/ether ac:1f:6b:f8:26:03 brd ff:ff:ff:ff:ff:ff 2024-04-30T08:35:01.346 INFO:teuthology.orchestra.run.smithi032.stdout: altname enp1s0f1 2024-04-30T08:35:01.347 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-30T08:35:01.347 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-30T08:35:01.348 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-04-30T08:35:01.348 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link add name ceph-brx type bridge 2024-04-30T08:35:01.348 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip addr flush dev ceph-brx 2024-04-30T08:35:01.348 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set ceph-brx up 2024-04-30T08:35:01.348 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-30T08:35:01.348 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-04-30T08:35:01.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:01 smithi032 ceph-mon[28331]: pgmap v420: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-30T08:35:01.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:01 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1768754230' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-30T08:35:01.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-30T08:35:01.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:01 smithi071 ceph-mon[33385]: pgmap v420: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-30T08:35:01.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:01 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1768754230' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-30T08:35:01.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:01 smithi189 ceph-mon[33038]: pgmap v420: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 11 op/s 2024-04-30T08:35:01.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:01 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1768754230' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-30T08:35:01.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-30T08:35:01.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-30T08:35:01.605 DEBUG:teuthology.orchestra.run.smithi032:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-30T08:35:01.670 INFO:teuthology.orchestra.run.smithi032.stdout:1 2024-04-30T08:35:01.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-30T08:35:01.675 DEBUG:teuthology.orchestra.run.smithi032:> route 2024-04-30T08:35:01.725 INFO:teuthology.orchestra.run.smithi032.stdout:Kernel IP routing table 2024-04-30T08:35:01.725 INFO:teuthology.orchestra.run.smithi032.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-30T08:35:01.725 INFO:teuthology.orchestra.run.smithi032.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-30T08:35:01.725 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-30T08:35:01.725 INFO:teuthology.orchestra.run.smithi032.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-30T08:35:01.726 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-30T08:35:01.726 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-04-30T08:35:01.726 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-30T08:35:01.726 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-30T08:35:01.726 DEBUG:teuthology.orchestra.run.smithi032:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-30T08:35:01.726 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-04-30T08:35:01.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:01 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-30T08:35:01.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:01 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-30T08:35:01.954 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-30T08:35:01.954 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-04-30T08:35:02.005 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-30T08:35:02.005 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list-id 2024-04-30T08:35:02.063 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-30T08:35:02.063 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-04-30T08:35:02.063 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-30T08:35:02.064 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-30T08:35:02.064 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-04-30T08:35:02.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-30T08:35:02.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-30T08:35:02.265 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-30T08:35:02.265 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-30T08:35:02.266 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-04-30T08:35:02.266 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-30T08:35:02.266 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-30T08:35:02.266 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-30T08:35:02.266 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-30T08:35:02.266 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-30T08:35:02.266 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-04-30T08:35:02.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-30T08:35:02.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-30T08:35:02.548 DEBUG:teuthology.orchestra.run.smithi032:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-30T08:35:02.548 DEBUG:teuthology.orchestra.run.smithi032:> set -e 2024-04-30T08:35:02.548 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set brx.0 up 2024-04-30T08:35:02.549 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link set dev brx.0 master ceph-brx 2024-04-30T08:35:02.549 DEBUG:teuthology.orchestra.run.smithi032:> ') 2024-04-30T08:35:02.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:02 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-30T08:35:02.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:02 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-30T08:35:02.760 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-30T08:35:02.760 DEBUG:teuthology.orchestra.run.smithi032:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-30T08:35:02.813 INFO:teuthology.orchestra.run.smithi032.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-30T08:35:02.814 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-30T08:35:02.814 DEBUG:teuthology.orchestra.run.smithi032:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-30T08:35:02.866 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-30T08:35:02.866 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-30T08:35:02.866 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/201e6199-10b1-41e5-a912-eef3ace842a3 2024-04-30T08:35:02.866 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-30T08:35:02.866 DEBUG:teuthology.orchestra.run.smithi032:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/201e6199-10b1-41e5-a912-eef3ace842a3 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-04-30T08:35:03.081 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-04-30T08:35:03.081 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync". 2024-04-30T08:35:03.082 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-04-30T08:35:03.332 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:03 smithi032 ceph-mon[28331]: pgmap v421: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-30T08:35:03.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:03 smithi071 ceph-mon[33385]: pgmap v421: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-30T08:35:03.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:03 smithi189 ceph-mon[33038]: pgmap v421: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-04-30T08:35:03.817 INFO:teuthology.orchestra.run.smithi032.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-04-30T08:35:03.817 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync". 2024-04-30T08:35:03.817 INFO:teuthology.orchestra.run.smithi032.stdout:invalid new device string format 2024-04-30T08:35:03.817 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: resolved to: "172.21.15.32:6789,172.21.15.71:6789,172.21.15.189:6789" 2024-04-30T08:35:03.818 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.32:6789,172.21.15.71:6789,172.21.15.189:6789:/volumes/qa/sv_1/201e6199-10b1-41e5-a912-eef3ace842a3 2024-04-30T08:35:03.818 INFO:teuthology.orchestra.run.smithi032.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync,key=0,fsid=98c0a670-06ca-11ef-bc93-c7b262605968" will pass to kernel 2024-04-30T08:35:03.821 INFO:teuthology.orchestra.run.smithi032.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-30T08:35:03.821 INFO:teuthology.orchestra.run.smithi032.stdout: You just mounted a file system that supports labels which does not 2024-04-30T08:35:03.821 INFO:teuthology.orchestra.run.smithi032.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-30T08:35:03.821 INFO:teuthology.orchestra.run.smithi032.stdout: applications will generate AVC messages and not be allowed access to 2024-04-30T08:35:03.821 INFO:teuthology.orchestra.run.smithi032.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-30T08:35:03.827 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-30T08:35:03.827 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-30T08:35:03.827 DEBUG:teuthology.orchestra.run.smithi032:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-30T08:35:03.920 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-30T08:35:04.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-30T08:35:04.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-30T08:35:04.051 INFO:teuthology.orchestra.run.smithi032.stdout:client25477 2024-04-30T08:35:04.052 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-30T08:35:04.052 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-30T08:35:04.107 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-30T08:35:04.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-30T08:35:04.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-30T08:35:04.235 INFO:teuthology.orchestra.run.smithi032.stdout:client25477 2024-04-30T08:35:04.235 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-30T08:35:04.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-30T08:35:04.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-30T08:35:04.362 INFO:teuthology.orchestra.run.smithi032.stdout:client25477 2024-04-30T08:35:04.363 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/98c0a670-06ca-11ef-bc93-c7b262605968.client25477/status') 2024-04-30T08:35:04.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-30T08:35:04.509 INFO:teuthology.orchestra.run.smithi032.stdout:instance: client.25477 (3)172.21.15.32:0/3381017268 2024-04-30T08:35:04.509 INFO:teuthology.orchestra.run.smithi032.stdout:blocklisted: false 2024-04-30T08:35:04.510 INFO:teuthology.orchestra.run.smithi032.stderr:0+1 records in 2024-04-30T08:35:04.510 INFO:teuthology.orchestra.run.smithi032.stderr:0+1 records out 2024-04-30T08:35:04.510 INFO:teuthology.orchestra.run.smithi032.stderr:71 bytes copied, 9.0277e-05 s, 786 kB/s 2024-04-30T08:35:04.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-30T08:35:04.524 INFO:teuthology.run_tasks:Running task exec... 2024-04-30T08:35:04.534 INFO:teuthology.task.exec:Executing custom commands... 2024-04-30T08:35:04.534 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi032.front.sepia.ceph.com 2024-04-30T08:35:04.535 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-04-30T08:35:04.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.825+0000 7f91a4a00640 1 -- 172.21.15.32:0/62328542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91a0104d20 msgr2=0x7f91a0105130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:04.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.825+0000 7f91a4a00640 1 --2- 172.21.15.32:0/62328542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91a0104d20 0x7f91a0105130 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f918c009900 tx=0x7f918c02a3f0 comp rx=0 tx=0).stop 2024-04-30T08:35:04.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.825+0000 7f91a4a00640 1 -- 172.21.15.32:0/62328542 shutdown_connections 2024-04-30T08:35:04.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.825+0000 7f91a4a00640 1 --2- 172.21.15.32:0/62328542 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91a0109a20 0x7f91a010be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:04.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.825+0000 7f91a4a00640 1 --2- 172.21.15.32:0/62328542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91a0104d20 0x7f91a0105130 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:04.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.825+0000 7f91a4a00640 1 --2- 172.21.15.32:0/62328542 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91a0103ae0 0x7f91a01047e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:04.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.825+0000 7f91a4a00640 1 -- 172.21.15.32:0/62328542 >> 172.21.15.32:0/62328542 conn(0x7f91a00fecd0 msgr2=0x7f91a0101180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:04.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.825+0000 7f91a4a00640 1 -- 172.21.15.32:0/62328542 shutdown_connections 2024-04-30T08:35:04.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.825+0000 7f91a4a00640 1 -- 172.21.15.32:0/62328542 wait complete. 2024-04-30T08:35:04.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.826+0000 7f91a4a00640 1 Processor -- start 2024-04-30T08:35:04.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.826+0000 7f91a4a00640 1 -- start start 2024-04-30T08:35:04.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.826+0000 7f91a4a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91a0103ae0 0x7f91a019c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:04.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.827+0000 7f91a4a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91a0104d20 0x7f91a019c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:04.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.827+0000 7f91a4a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91a0109a20 0x7f91a01a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:04.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.827+0000 7f91a4a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f91a010f530 con 0x7f91a0103ae0 2024-04-30T08:35:04.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.827+0000 7f91a4a00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f91a010f3b0 con 0x7f91a0109a20 2024-04-30T08:35:04.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.827+0000 7f91a4a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f91a010f6b0 con 0x7f91a0104d20 2024-04-30T08:35:04.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.827+0000 7f919f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91a0103ae0 0x7f91a019c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:04.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.827+0000 7f919f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91a0103ae0 0x7f91a019c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37784/0 (socket says 172.21.15.32:37784) 2024-04-30T08:35:04.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.827+0000 7f919f400640 1 -- 172.21.15.32:0/2752709077 learned_addr learned my addr 172.21.15.32:0/2752709077 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:04.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.827+0000 7f919ea00640 1 --2- 172.21.15.32:0/2752709077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91a0104d20 0x7f91a019c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:04.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.827+0000 7f919fe00640 1 --2- 172.21.15.32:0/2752709077 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91a0109a20 0x7f91a01a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:04.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.827+0000 7f919f400640 1 -- 172.21.15.32:0/2752709077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91a0104d20 msgr2=0x7f91a019c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:04.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.827+0000 7f919f400640 1 --2- 172.21.15.32:0/2752709077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91a0104d20 0x7f91a019c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:04.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.827+0000 7f919f400640 1 -- 172.21.15.32:0/2752709077 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91a0109a20 msgr2=0x7f91a01a37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:04.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.827+0000 7f919f400640 1 --2- 172.21.15.32:0/2752709077 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91a0109a20 0x7f91a01a37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:04.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.827+0000 7f919f400640 1 -- 172.21.15.32:0/2752709077 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91a01a3e60 con 0x7f91a0103ae0 2024-04-30T08:35:04.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.827+0000 7f919fe00640 1 --2- 172.21.15.32:0/2752709077 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91a0109a20 0x7f91a01a37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:35:04.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.828+0000 7f919f400640 1 --2- 172.21.15.32:0/2752709077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91a0103ae0 0x7f91a019c320 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f919000e9f0 tx=0x7f919000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:04.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.828+0000 7f9197e00640 1 -- 172.21.15.32:0/2752709077 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f919000ccc0 con 0x7f91a0103ae0 2024-04-30T08:35:04.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.828+0000 7f9197e00640 1 -- 172.21.15.32:0/2752709077 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f91900035f0 con 0x7f91a0103ae0 2024-04-30T08:35:04.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.828+0000 7f91a4a00640 1 -- 172.21.15.32:0/2752709077 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91a01a4150 con 0x7f91a0103ae0 2024-04-30T08:35:04.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.828+0000 7f9197e00640 1 -- 172.21.15.32:0/2752709077 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f919000ce60 con 0x7f91a0103ae0 2024-04-30T08:35:04.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.828+0000 7f91a4a00640 1 -- 172.21.15.32:0/2752709077 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91a01a46a0 con 0x7f91a0103ae0 2024-04-30T08:35:04.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.829+0000 7f91a4a00640 1 -- 172.21.15.32:0/2752709077 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f91a01a4960 con 0x7f91a0103ae0 2024-04-30T08:35:04.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.832+0000 7f9197e00640 1 -- 172.21.15.32:0/2752709077 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f9190003aa0 con 0x7f91a0103ae0 2024-04-30T08:35:04.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.833+0000 7f9197e00640 1 --2- 172.21.15.32:0/2752709077 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f91840afae0 0x7f91840b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:04.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.834+0000 7f9197e00640 1 -- 172.21.15.32:0/2752709077 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f91900df3d0 con 0x7f91a0103ae0 2024-04-30T08:35:04.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.834+0000 7f919ea00640 1 --2- 172.21.15.32:0/2752709077 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f91840afae0 0x7f91840b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:04.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.835+0000 7f919ea00640 1 --2- 172.21.15.32:0/2752709077 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f91840afae0 0x7f91840b1ff0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f918c009230 tx=0x7f918c031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:04.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:04.839+0000 7f9197e00640 1 -- 172.21.15.32:0/2752709077 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f91900a7ba0 con 0x7f91a0103ae0 2024-04-30T08:35:05.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:05.096+0000 7f91a4a00640 1 -- 172.21.15.32:0/2752709077 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7f91a01a4b30 con 0x7f91a0103ae0 2024-04-30T08:35:05.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:05 smithi032 ceph-mon[28331]: pgmap v422: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:35:05.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:05 smithi071 ceph-mon[33385]: pgmap v422: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:35:05.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:05 smithi189 ceph-mon[33038]: pgmap v422: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:35:06.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.094+0000 7f9197e00640 1 -- 172.21.15.32:0/2752709077 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v28) ==== 118+0+0 (secure 0 0 0) 0x7f91900a74a0 con 0x7f91a0103ae0 2024-04-30T08:35:06.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.098+0000 7f91a4a00640 1 -- 172.21.15.32:0/2752709077 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f91840afae0 msgr2=0x7f91840b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:06.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.098+0000 7f91a4a00640 1 --2- 172.21.15.32:0/2752709077 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f91840afae0 0x7f91840b1ff0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f918c009230 tx=0x7f918c031400 comp rx=0 tx=0).stop 2024-04-30T08:35:06.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.098+0000 7f91a4a00640 1 -- 172.21.15.32:0/2752709077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91a0103ae0 msgr2=0x7f91a019c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:06.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.098+0000 7f91a4a00640 1 --2- 172.21.15.32:0/2752709077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91a0103ae0 0x7f91a019c320 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f919000e9f0 tx=0x7f919000eeb0 comp rx=0 tx=0).stop 2024-04-30T08:35:06.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.098+0000 7f91a4a00640 1 -- 172.21.15.32:0/2752709077 shutdown_connections 2024-04-30T08:35:06.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.098+0000 7f91a4a00640 1 --2- 172.21.15.32:0/2752709077 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f91840afae0 0x7f91840b1ff0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:06.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.098+0000 7f91a4a00640 1 --2- 172.21.15.32:0/2752709077 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f91a0109a20 0x7f91a01a37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:06.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.098+0000 7f91a4a00640 1 --2- 172.21.15.32:0/2752709077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91a0104d20 0x7f91a019c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:06.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.098+0000 7f91a4a00640 1 --2- 172.21.15.32:0/2752709077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f91a0103ae0 0x7f91a019c320 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:06.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.098+0000 7f91a4a00640 1 -- 172.21.15.32:0/2752709077 >> 172.21.15.32:0/2752709077 conn(0x7f91a00fecd0 msgr2=0x7f91a00ff0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:06.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.099+0000 7f91a4a00640 1 -- 172.21.15.32:0/2752709077 shutdown_connections 2024-04-30T08:35:06.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.099+0000 7f91a4a00640 1 -- 172.21.15.32:0/2752709077 wait complete. 2024-04-30T08:35:06.133 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-04-30T08:35:06.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.420+0000 7f0ca3e00640 1 -- 172.21.15.32:0/514849465 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ca4106eb0 msgr2=0x7f0ca41072c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:06.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.420+0000 7f0ca3e00640 1 --2- 172.21.15.32:0/514849465 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ca4106eb0 0x7f0ca41072c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c009900 tx=0x7f0c8c02a3f0 comp rx=0 tx=0).stop 2024-04-30T08:35:06.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.421+0000 7f0ca3e00640 1 -- 172.21.15.32:0/514849465 shutdown_connections 2024-04-30T08:35:06.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.421+0000 7f0ca3e00640 1 --2- 172.21.15.32:0/514849465 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ca410bbb0 0x7f0ca410dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:06.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.421+0000 7f0ca3e00640 1 --2- 172.21.15.32:0/514849465 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ca4106eb0 0x7f0ca41072c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:06.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.421+0000 7f0ca3e00640 1 --2- 172.21.15.32:0/514849465 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ca4105c70 0x7f0ca4106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:06.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.421+0000 7f0ca3e00640 1 -- 172.21.15.32:0/514849465 >> 172.21.15.32:0/514849465 conn(0x7f0ca4100e60 msgr2=0x7f0ca4103310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:06.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.421+0000 7f0ca3e00640 1 -- 172.21.15.32:0/514849465 shutdown_connections 2024-04-30T08:35:06.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.421+0000 7f0ca3e00640 1 -- 172.21.15.32:0/514849465 wait complete. 2024-04-30T08:35:06.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.422+0000 7f0ca3e00640 1 Processor -- start 2024-04-30T08:35:06.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.422+0000 7f0ca3e00640 1 -- start start 2024-04-30T08:35:06.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.422+0000 7f0ca3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ca4105c70 0x7f0ca419c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:06.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.423+0000 7f0ca3e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ca4106eb0 0x7f0ca419c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:06.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.423+0000 7f0ca3e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ca410bbb0 0x7f0ca41a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:06.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.423+0000 7f0ca3e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0ca4111700 con 0x7f0ca4105c70 2024-04-30T08:35:06.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.423+0000 7f0ca3e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f0ca4111580 con 0x7f0ca410bbb0 2024-04-30T08:35:06.425 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.423+0000 7f0ca3e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0ca4111880 con 0x7f0ca4106eb0 2024-04-30T08:35:06.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.423+0000 7f0ca2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ca4105c70 0x7f0ca419c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:06.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.423+0000 7f0ca2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ca4105c70 0x7f0ca419c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37794/0 (socket says 172.21.15.32:37794) 2024-04-30T08:35:06.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.423+0000 7f0ca2a00640 1 -- 172.21.15.32:0/654533629 learned_addr learned my addr 172.21.15.32:0/654533629 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:06.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.423+0000 7f0ca2000640 1 --2- 172.21.15.32:0/654533629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ca4106eb0 0x7f0ca419c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:06.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.423+0000 7f0ca3400640 1 --2- 172.21.15.32:0/654533629 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ca410bbb0 0x7f0ca41a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:06.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.423+0000 7f0ca2a00640 1 -- 172.21.15.32:0/654533629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ca4106eb0 msgr2=0x7f0ca419c840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:06.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.423+0000 7f0ca2a00640 1 --2- 172.21.15.32:0/654533629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ca4106eb0 0x7f0ca419c840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:06.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.423+0000 7f0ca2a00640 1 -- 172.21.15.32:0/654533629 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ca410bbb0 msgr2=0x7f0ca41a37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:06.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.423+0000 7f0ca2a00640 1 --2- 172.21.15.32:0/654533629 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ca410bbb0 0x7f0ca41a37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:06.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.423+0000 7f0ca2a00640 1 -- 172.21.15.32:0/654533629 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ca41a3e40 con 0x7f0ca4105c70 2024-04-30T08:35:06.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.424+0000 7f0ca2a00640 1 --2- 172.21.15.32:0/654533629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ca4105c70 0x7f0ca419c300 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0c9400ed00 tx=0x7f0c9400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:06.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.424+0000 7f0c9b400640 1 -- 172.21.15.32:0/654533629 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c94010630 con 0x7f0ca4105c70 2024-04-30T08:35:06.427 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.424+0000 7f0ca3e00640 1 -- 172.21.15.32:0/654533629 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ca41a4130 con 0x7f0ca4105c70 2024-04-30T08:35:06.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.424+0000 7f0c9b400640 1 -- 172.21.15.32:0/654533629 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0c94010c80 con 0x7f0ca4105c70 2024-04-30T08:35:06.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.424+0000 7f0c9b400640 1 -- 172.21.15.32:0/654533629 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0c94018d50 con 0x7f0ca4105c70 2024-04-30T08:35:06.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.424+0000 7f0ca3e00640 1 -- 172.21.15.32:0/654533629 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ca41a4680 con 0x7f0ca4105c70 2024-04-30T08:35:06.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.425+0000 7f0ca3e00640 1 -- 172.21.15.32:0/654533629 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ca410a9c0 con 0x7f0ca4105c70 2024-04-30T08:35:06.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.428+0000 7f0c9b400640 1 -- 172.21.15.32:0/654533629 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f0c9401f020 con 0x7f0ca4105c70 2024-04-30T08:35:06.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.429+0000 7f0c9b400640 1 --2- 172.21.15.32:0/654533629 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f0c880afae0 0x7f0c880b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:06.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.429+0000 7f0c9b400640 1 -- 172.21.15.32:0/654533629 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f0c940df8b0 con 0x7f0ca4105c70 2024-04-30T08:35:06.431 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.430+0000 7f0ca2000640 1 --2- 172.21.15.32:0/654533629 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f0c880afae0 0x7f0c880b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:06.432 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.430+0000 7f0ca2000640 1 --2- 172.21.15.32:0/654533629 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f0c880afae0 0x7f0c880b1ff0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c009230 tx=0x7f0c8c031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:06.433 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.432+0000 7f0c9b400640 1 -- 172.21.15.32:0/654533629 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0c940a8080 con 0x7f0ca4105c70 2024-04-30T08:35:06.440 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:06 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2752709077' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-30T08:35:06.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:06 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2752709077' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-30T08:35:06.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:06 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2752709077' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-30T08:35:06.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.686+0000 7f0ca3e00640 1 -- 172.21.15.32:0/654533629 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f0ca4063100 con 0x7f0c880afae0 2024-04-30T08:35:06.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.710+0000 7f0c9b400640 1 -- 172.21.15.32:0/654533629 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7f0ca4063100 con 0x7f0c880afae0 2024-04-30T08:35:06.712 INFO:teuthology.orchestra.run.smithi032.stdout:{} 2024-04-30T08:35:06.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.713+0000 7f0ca3e00640 1 -- 172.21.15.32:0/654533629 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f0c880afae0 msgr2=0x7f0c880b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:06.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.713+0000 7f0ca3e00640 1 --2- 172.21.15.32:0/654533629 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f0c880afae0 0x7f0c880b1ff0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f0c8c009230 tx=0x7f0c8c031400 comp rx=0 tx=0).stop 2024-04-30T08:35:06.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.713+0000 7f0ca3e00640 1 -- 172.21.15.32:0/654533629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ca4105c70 msgr2=0x7f0ca419c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:06.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.713+0000 7f0ca3e00640 1 --2- 172.21.15.32:0/654533629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ca4105c70 0x7f0ca419c300 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0c9400ed00 tx=0x7f0c9400c6a0 comp rx=0 tx=0).stop 2024-04-30T08:35:06.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.713+0000 7f0ca3e00640 1 -- 172.21.15.32:0/654533629 shutdown_connections 2024-04-30T08:35:06.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.713+0000 7f0ca3e00640 1 --2- 172.21.15.32:0/654533629 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f0c880afae0 0x7f0c880b1ff0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:06.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.713+0000 7f0ca3e00640 1 --2- 172.21.15.32:0/654533629 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0ca410bbb0 0x7f0ca41a37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:06.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.713+0000 7f0ca3e00640 1 --2- 172.21.15.32:0/654533629 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0ca4106eb0 0x7f0ca419c840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:06.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.713+0000 7f0ca3e00640 1 --2- 172.21.15.32:0/654533629 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0ca4105c70 0x7f0ca419c300 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:06.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.713+0000 7f0ca3e00640 1 -- 172.21.15.32:0/654533629 >> 172.21.15.32:0/654533629 conn(0x7f0ca4100e60 msgr2=0x7f0ca4107b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:06.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.713+0000 7f0ca3e00640 1 -- 172.21.15.32:0/654533629 shutdown_connections 2024-04-30T08:35:06.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:06.713+0000 7f0ca3e00640 1 -- 172.21.15.32:0/654533629 wait complete. 2024-04-30T08:35:06.741 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-30T08:35:06.752 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-04-30T08:35:06.762 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-30T08:35:07.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.027+0000 7f3014a00640 1 -- 172.21.15.32:0/561930300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3010109640 msgr2=0x7f301010bb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:07.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.027+0000 7f3014a00640 1 --2- 172.21.15.32:0/561930300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3010109640 0x7f301010bb50 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f3008009be0 tx=0x7f300802a6c0 comp rx=0 tx=0).stop 2024-04-30T08:35:07.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.028+0000 7f3014a00640 1 -- 172.21.15.32:0/561930300 shutdown_connections 2024-04-30T08:35:07.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.028+0000 7f3014a00640 1 --2- 172.21.15.32:0/561930300 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3010109640 0x7f301010bb50 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:07.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.028+0000 7f3014a00640 1 --2- 172.21.15.32:0/561930300 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3010106ad0 0x7f3010108fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:07.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.028+0000 7f3014a00640 1 --2- 172.21.15.32:0/561930300 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f30101040e0 0x7f3010106500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:07.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.028+0000 7f3014a00640 1 -- 172.21.15.32:0/561930300 >> 172.21.15.32:0/561930300 conn(0x7f30100fdac0 msgr2=0x7f30100fff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:07.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.028+0000 7f3014a00640 1 -- 172.21.15.32:0/561930300 shutdown_connections 2024-04-30T08:35:07.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.028+0000 7f3014a00640 1 -- 172.21.15.32:0/561930300 wait complete. 2024-04-30T08:35:07.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.029+0000 7f3014a00640 1 Processor -- start 2024-04-30T08:35:07.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.029+0000 7f3014a00640 1 -- start start 2024-04-30T08:35:07.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.030+0000 7f3014a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30101040e0 0x7f301019c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:07.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.030+0000 7f3014a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3010106ad0 0x7f301019c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:07.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.030+0000 7f3014a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3010109640 0x7f30101a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:07.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.030+0000 7f3014a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f301010f490 con 0x7f3010106ad0 2024-04-30T08:35:07.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.030+0000 7f3014a00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f301010f310 con 0x7f3010109640 2024-04-30T08:35:07.032 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.030+0000 7f3014a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f301010f610 con 0x7f30101040e0 2024-04-30T08:35:07.033 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.030+0000 7f300f200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3010109640 0x7f30101a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:07.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.030+0000 7f300e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30101040e0 0x7f301019c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:07.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.030+0000 7f300f200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3010109640 0x7f30101a36f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:57068/0 (socket says 172.21.15.32:57068) 2024-04-30T08:35:07.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.030+0000 7f300f200640 1 -- 172.21.15.32:0/152456604 learned_addr learned my addr 172.21.15.32:0/152456604 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:07.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.030+0000 7f300e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30101040e0 0x7f301019c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:44178/0 (socket says 172.21.15.32:44178) 2024-04-30T08:35:07.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.030+0000 7f300de00640 1 --2- 172.21.15.32:0/152456604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3010106ad0 0x7f301019c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:07.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.031+0000 7f300f200640 1 -- 172.21.15.32:0/152456604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30101040e0 msgr2=0x7f301019c220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:07.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.031+0000 7f300f200640 1 --2- 172.21.15.32:0/152456604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30101040e0 0x7f301019c220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:07.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.031+0000 7f300f200640 1 -- 172.21.15.32:0/152456604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3010106ad0 msgr2=0x7f301019c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:07.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.031+0000 7f300f200640 1 --2- 172.21.15.32:0/152456604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3010106ad0 0x7f301019c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:07.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.031+0000 7f300f200640 1 -- 172.21.15.32:0/152456604 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30101a3d60 con 0x7f3010109640 2024-04-30T08:35:07.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.031+0000 7f300e800640 1 --2- 172.21.15.32:0/152456604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30101040e0 0x7f301019c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:07.034 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.031+0000 7f300f200640 1 --2- 172.21.15.32:0/152456604 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3010109640 0x7f30101a36f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3008034040 tx=0x7f3008030f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:07.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.031+0000 7f3007400640 1 -- 172.21.15.32:0/152456604 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3008041020 con 0x7f3010109640 2024-04-30T08:35:07.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.032+0000 7f3007400640 1 -- 172.21.15.32:0/152456604 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f30080318f0 con 0x7f3010109640 2024-04-30T08:35:07.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.032+0000 7f3014a00640 1 -- 172.21.15.32:0/152456604 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30101a3ff0 con 0x7f3010109640 2024-04-30T08:35:07.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.032+0000 7f3007400640 1 -- 172.21.15.32:0/152456604 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3008036040 con 0x7f3010109640 2024-04-30T08:35:07.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.032+0000 7f3014a00640 1 -- 172.21.15.32:0/152456604 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30101a4450 con 0x7f3010109640 2024-04-30T08:35:07.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.033+0000 7f3014a00640 1 -- 172.21.15.32:0/152456604 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f30101a4740 con 0x7f3010109640 2024-04-30T08:35:07.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.035+0000 7f3007400640 1 -- 172.21.15.32:0/152456604 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f3008044020 con 0x7f3010109640 2024-04-30T08:35:07.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.038+0000 7f3007400640 1 --2- 172.21.15.32:0/152456604 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f2ff40afb60 0x7f2ff40b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:07.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.038+0000 7f3007400640 1 -- 172.21.15.32:0/152456604 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f30080ffbc0 con 0x7f3010109640 2024-04-30T08:35:07.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.038+0000 7f300e800640 1 --2- 172.21.15.32:0/152456604 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f2ff40afb60 0x7f2ff40b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:07.040 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.039+0000 7f300e800640 1 --2- 172.21.15.32:0/152456604 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f2ff40afb60 0x7f2ff40b2070 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f301006ba80 tx=0x7f3000005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:07.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.041+0000 7f3007400640 1 -- 172.21.15.32:0/152456604 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f300804b020 con 0x7f3010109640 2024-04-30T08:35:07.098 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:35:06 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:35:06.694+0000 7fbb1a000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-30T08:35:07.098 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:35:06 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:35:06.694+0000 7fbb1a000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-30T08:35:07.098 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:35:06 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:35:06.694+0000 7fbb1a000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-30T08:35:07.099 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:35:06 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:35:06.694+0000 7fbb1a000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-30T08:35:07.099 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:35:06 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:35:06.694+0000 7fbb1a000700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-30T08:35:07.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.261+0000 7f3014a00640 1 -- 172.21.15.32:0/152456604 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f30101a4eb0 con 0x7f3010109640 2024-04-30T08:35:07.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.263+0000 7f3007400640 1 -- 172.21.15.32:0/152456604 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+24730 (secure 0 0 0) 0x7f30080c8390 con 0x7f3010109640 2024-04-30T08:35:07.265 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:35:07.265 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":85,"fsid":"98c0a670-06ca-11ef-bc93-c7b262605968","created":"2024-04-30T08:22:11.122875+0000","modified":"2024-04-30T08:33:31.872704+0000","last_up_change":"2024-04-30T08:30:50.350439+0000","last_in_change":"2024-04-30T08:30:30.210620+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-30T08:26:19.728331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-30T08:31:49.800542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-30T08:31:51.211320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-30T08:31:55.716625+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a7e060c8-1cbb-418f-9a03-06693f6b3f52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6803","nonce":2624700402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6805","nonce":2624700402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6809","nonce":2624700402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6807","nonce":2624700402}]},"public_addr":"172.21.15.32:6803/2624700402","cluster_addr":"172.21.15.32:6805/2624700402","heartbeat_back_addr":"172.21.15.32:6809/2624700402","heartbeat_front_addr":"172.21.15.32:6807/2624700402","state":["exists","up"]},{"osd":1,"uuid":"72ef2fb1-e5ca-48ef-aa3a-4796a64b922f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6801","nonce":444649711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6803","nonce":444649711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6807","nonce":444649711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6805","nonce":444649711}]},"public_addr":"172.21.15.71:6801/444649711","cluster_addr":"172.21.15.71:6803/444649711","heartbeat_back_addr":"172.21.15.71:6807/444649711","heartbeat_front_addr":"172.21.15.71:6805/444649711","state":["exists","up"]},{"osd":2,"uuid":"704e5e8a-935e-47c7-ae28-27119876ac09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6801","nonce":3255528671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6803","nonce":3255528671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6807","nonce":3255528671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6805","nonce":3255528671}]},"public_addr":"172.21.15.189:6801/3255528671","cluster_addr":"172.21.15.189:6803/3255528671","heartbeat_back_addr":"172.21.15.189:6807/3255528671","heartbeat_front_addr":"172.21.15.189:6805/3255528671","state":["exists","up"]},{"osd":3,"uuid":"575004f7-6559-4a62-863e-e33078b0eebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6811","nonce":1384702112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6813","nonce":1384702112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6817","nonce":1384702112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6815","nonce":1384702112}]},"public_addr":"172.21.15.32:6811/1384702112","cluster_addr":"172.21.15.32:6813/1384702112","heartbeat_back_addr":"172.21.15.32:6817/1384702112","heartbeat_front_addr":"172.21.15.32:6815/1384702112","state":["exists","up"]},{"osd":4,"uuid":"e1c1e894-074d-4c78-89ef-9e5a114268bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6809","nonce":4250534574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6811","nonce":4250534574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6815","nonce":4250534574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6813","nonce":4250534574}]},"public_addr":"172.21.15.71:6809/4250534574","cluster_addr":"172.21.15.71:6811/4250534574","heartbeat_back_addr":"172.21.15.71:6815/4250534574","heartbeat_front_addr":"172.21.15.71:6813/4250534574","state":["exists","up"]},{"osd":5,"uuid":"12c022d4-fac6-40c4-ae7a-691571bee5d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6809","nonce":4245346198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6811","nonce":4245346198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6815","nonce":4245346198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6813","nonce":4245346198}]},"public_addr":"172.21.15.189:6809/4245346198","cluster_addr":"172.21.15.189:6811/4245346198","heartbeat_back_addr":"172.21.15.189:6815/4245346198","heartbeat_front_addr":"172.21.15.189:6813/4245346198","state":["exists","up"]},{"osd":6,"uuid":"b413f8ee-3d4a-415a-a2ba-d68437b7330d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6819","nonce":1910389190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6821","nonce":1910389190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6825","nonce":1910389190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6823","nonce":1910389190}]},"public_addr":"172.21.15.32:6819/1910389190","cluster_addr":"172.21.15.32:6821/1910389190","heartbeat_back_addr":"172.21.15.32:6825/1910389190","heartbeat_front_addr":"172.21.15.32:6823/1910389190","state":["exists","up"]},{"osd":7,"uuid":"aac64fb3-fc13-4126-b842-7b06ad5ce238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6817","nonce":692607380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6819","nonce":692607380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6823","nonce":692607380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6821","nonce":692607380}]},"public_addr":"172.21.15.71:6817/692607380","cluster_addr":"172.21.15.71:6819/692607380","heartbeat_back_addr":"172.21.15.71:6823/692607380","heartbeat_front_addr":"172.21.15.71:6821/692607380","state":["exists","up"]},{"osd":8,"uuid":"cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6817","nonce":3973697069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6819","nonce":3973697069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6823","nonce":3973697069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6821","nonce":3973697069}]},"public_addr":"172.21.15.189:6817/3973697069","cluster_addr":"172.21.15.189:6819/3973697069","heartbeat_back_addr":"172.21.15.189:6823/3973697069","heartbeat_front_addr":"172.21.15.189:6821/3973697069","state":["exists","up"]},{"osd":9,"uuid":"ef4a1645-ae6e-45c3-82ad-161510e62aa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6827","nonce":46066685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6829","nonce":46066685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6833","nonce":46066685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6831","nonce":46066685}]},"public_addr":"172.21.15.32:6827/46066685","cluster_addr":"172.21.15.32:6829/46066685","heartbeat_back_addr":"172.21.15.32:6833/46066685","heartbeat_front_addr":"172.21.15.32:6831/46066685","state":["exists","up"]},{"osd":10,"uuid":"25b26a38-a644-409f-b941-59efdf04a3ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6825","nonce":1886290293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6827","nonce":1886290293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6831","nonce":1886290293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6829","nonce":1886290293}]},"public_addr":"172.21.15.71:6825/1886290293","cluster_addr":"172.21.15.71:6827/1886290293","heartbeat_back_addr":"172.21.15.71:6831/1886290293","heartbeat_front_addr":"172.21.15.71:6829/1886290293","state":["exists","up"]},{"osd":11,"uuid":"ab96b3ff-3688-455b-af64-60226c5bfbff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6825","nonce":850185001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6827","nonce":850185001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6831","nonce":850185001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6829","nonce":850185001}]},"public_addr":"172.21.15.189:6825/850185001","cluster_addr":"172.21.15.189:6827/850185001","heartbeat_back_addr":"172.21.15.189:6831/850185001","heartbeat_front_addr":"172.21.15.189:6829/850185001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:27.220308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:53.230783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:17.190642+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:52.603799+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:20.568671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:46.158363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:22.706933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:50.665173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:16.602410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:53.814698+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:22.435189+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:48.651949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.e","mappings":[{"from":4,"to":7}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.28","mappings":[{"from":3,"to":6},{"from":5,"to":8}]},{"pgid":"3.30","mappings":[{"from":1,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1266897222":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/2915689533":"2024-05-01T08:23:29.526664+0000","172.21.15.32:6801/4019351306":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/425836193":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/1134075722":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6801/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/21059253":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/3245550894":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/4019351306":"2024-05-01T08:23:29.526664+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-30T08:35:07.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.266+0000 7f3014a00640 1 -- 172.21.15.32:0/152456604 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f2ff40afb60 msgr2=0x7f2ff40b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:07.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.266+0000 7f3014a00640 1 --2- 172.21.15.32:0/152456604 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f2ff40afb60 0x7f2ff40b2070 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f301006ba80 tx=0x7f3000005f90 comp rx=0 tx=0).stop 2024-04-30T08:35:07.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.266+0000 7f3014a00640 1 -- 172.21.15.32:0/152456604 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3010109640 msgr2=0x7f30101a36f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:07.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.266+0000 7f3014a00640 1 --2- 172.21.15.32:0/152456604 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3010109640 0x7f30101a36f0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f3008034040 tx=0x7f3008030f60 comp rx=0 tx=0).stop 2024-04-30T08:35:07.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.266+0000 7f3014a00640 1 -- 172.21.15.32:0/152456604 shutdown_connections 2024-04-30T08:35:07.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.266+0000 7f3014a00640 1 --2- 172.21.15.32:0/152456604 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f2ff40afb60 0x7f2ff40b2070 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:07.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.266+0000 7f3014a00640 1 --2- 172.21.15.32:0/152456604 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3010109640 0x7f30101a36f0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:07.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.266+0000 7f3014a00640 1 --2- 172.21.15.32:0/152456604 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3010106ad0 0x7f301019c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:07.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.266+0000 7f3014a00640 1 --2- 172.21.15.32:0/152456604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f30101040e0 0x7f301019c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:07.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.267+0000 7f3014a00640 1 -- 172.21.15.32:0/152456604 >> 172.21.15.32:0/152456604 conn(0x7f30100fdac0 msgr2=0x7f3010101ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:07.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.267+0000 7f3014a00640 1 -- 172.21.15.32:0/152456604 shutdown_connections 2024-04-30T08:35:07.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.267+0000 7f3014a00640 1 -- 172.21.15.32:0/152456604 wait complete. 2024-04-30T08:35:07.294 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-30T08:26:19.728331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-30T08:31:49.800542+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-30T08:31:51.211320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-30T08:31:55.716625+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-30T08:35:07.295 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-30T08:35:07.366 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:07 smithi032 ceph-mon[28331]: pgmap v423: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-30T08:35:07.366 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:07 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2752709077' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-30T08:35:07.366 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:07 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:35:07.366 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:07 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:35:07.366 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:07 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:35:07.366 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:07 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:35:07.366 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:07 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:35:07.366 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:07 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:35:07.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:07 smithi071 ceph-mon[33385]: pgmap v423: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-30T08:35:07.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:07 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2752709077' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-30T08:35:07.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:07 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:35:07.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:07 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:35:07.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:07 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:35:07.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:07 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:35:07.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:07 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:35:07.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:07 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:35:07.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:07 smithi189 ceph-mon[33038]: pgmap v423: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-30T08:35:07.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:07 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2752709077' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-30T08:35:07.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:07 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:35:07.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:07 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:35:07.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:07 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:35:07.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:07 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:35:07.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:07 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:35:07.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:07 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:35:07.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.594+0000 7f5137200640 1 -- 172.21.15.32:0/2422923472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5138075960 msgr2=0x7f513810fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:07.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.594+0000 7f5137200640 1 --2- 172.21.15.32:0/2422923472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5138075960 0x7f513810fbb0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f5124009900 tx=0x7f512402a3f0 comp rx=0 tx=0).stop 2024-04-30T08:35:07.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.594+0000 7f5137200640 1 -- 172.21.15.32:0/2422923472 shutdown_connections 2024-04-30T08:35:07.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.594+0000 7f5137200640 1 --2- 172.21.15.32:0/2422923472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f51381100f0 0x7f5138112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:07.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.594+0000 7f5137200640 1 --2- 172.21.15.32:0/2422923472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5138075960 0x7f513810fbb0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:07.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.594+0000 7f5137200640 1 --2- 172.21.15.32:0/2422923472 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5138075010 0x7f5138075420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:07.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.594+0000 7f5137200640 1 -- 172.21.15.32:0/2422923472 >> 172.21.15.32:0/2422923472 conn(0x7f51380feba0 msgr2=0x7f5138101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:07.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.594+0000 7f5137200640 1 -- 172.21.15.32:0/2422923472 shutdown_connections 2024-04-30T08:35:07.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.594+0000 7f5137200640 1 -- 172.21.15.32:0/2422923472 wait complete. 2024-04-30T08:35:07.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.596+0000 7f5137200640 1 Processor -- start 2024-04-30T08:35:07.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.596+0000 7f5137200640 1 -- start start 2024-04-30T08:35:07.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.596+0000 7f5137200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5138075010 0x7f51381a0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:07.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.596+0000 7f5137200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5138075960 0x7f51381a0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:07.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.596+0000 7f5137200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f51381100f0 0x7f51381a7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:07.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.596+0000 7f5137200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5138115b20 con 0x7f5138075010 2024-04-30T08:35:07.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.596+0000 7f5137200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f51381159a0 con 0x7f51381100f0 2024-04-30T08:35:07.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.596+0000 7f5137200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5138115ca0 con 0x7f5138075960 2024-04-30T08:35:07.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.597+0000 7f5135e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5138075010 0x7f51381a0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:07.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.597+0000 7f5136800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f51381100f0 0x7f51381a7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:07.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.597+0000 7f5136800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f51381100f0 0x7f51381a7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:57090/0 (socket says 172.21.15.32:57090) 2024-04-30T08:35:07.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.597+0000 7f5135400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5138075960 0x7f51381a0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:07.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.597+0000 7f5135e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5138075010 0x7f51381a0700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53402/0 (socket says 172.21.15.32:53402) 2024-04-30T08:35:07.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.597+0000 7f5136800640 1 -- 172.21.15.32:0/729756702 learned_addr learned my addr 172.21.15.32:0/729756702 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:07.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.597+0000 7f5135400640 1 -- 172.21.15.32:0/729756702 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f51381100f0 msgr2=0x7f51381a7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:07.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.597+0000 7f5135400640 1 --2- 172.21.15.32:0/729756702 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f51381100f0 0x7f51381a7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:07.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.597+0000 7f5135400640 1 -- 172.21.15.32:0/729756702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5138075010 msgr2=0x7f51381a0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:07.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.597+0000 7f5135400640 1 --2- 172.21.15.32:0/729756702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5138075010 0x7f51381a0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:07.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.597+0000 7f5135400640 1 -- 172.21.15.32:0/729756702 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51381a8240 con 0x7f5138075960 2024-04-30T08:35:07.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.598+0000 7f5135400640 1 --2- 172.21.15.32:0/729756702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5138075960 0x7f51381a0c40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f512402a9e0 tx=0x7f5124003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:07.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.598+0000 7f512ea00640 1 -- 172.21.15.32:0/729756702 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f512403f070 con 0x7f5138075960 2024-04-30T08:35:07.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.598+0000 7f512ea00640 1 -- 172.21.15.32:0/729756702 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5124031c50 con 0x7f5138075960 2024-04-30T08:35:07.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.598+0000 7f5137200640 1 -- 172.21.15.32:0/729756702 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51381a84d0 con 0x7f5138075960 2024-04-30T08:35:07.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.598+0000 7f512ea00640 1 -- 172.21.15.32:0/729756702 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5124035030 con 0x7f5138075960 2024-04-30T08:35:07.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.598+0000 7f5137200640 1 -- 172.21.15.32:0/729756702 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51381a89e0 con 0x7f5138075960 2024-04-30T08:35:07.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.600+0000 7f5137200640 1 -- 172.21.15.32:0/729756702 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5138105c70 con 0x7f5138075960 2024-04-30T08:35:07.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.604+0000 7f512ea00640 1 -- 172.21.15.32:0/729756702 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f5124042070 con 0x7f5138075960 2024-04-30T08:35:07.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.605+0000 7f512ea00640 1 --2- 172.21.15.32:0/729756702 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f511c0afb60 0x7f511c0b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:07.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.606+0000 7f512ea00640 1 -- 172.21.15.32:0/729756702 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f51240ffc40 con 0x7f5138075960 2024-04-30T08:35:07.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.606+0000 7f5135e00640 1 --2- 172.21.15.32:0/729756702 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f511c0afb60 0x7f511c0b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:07.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.606+0000 7f512ea00640 1 -- 172.21.15.32:0/729756702 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f51240c7dc0 con 0x7f5138075960 2024-04-30T08:35:07.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.607+0000 7f5135e00640 1 --2- 172.21.15.32:0/729756702 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f511c0afb60 0x7f511c0b2070 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f5138076840 tx=0x7f5128005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:07.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.829+0000 7f5137200640 1 -- 172.21.15.32:0/729756702 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f513802dd60 con 0x7f5138075960 2024-04-30T08:35:07.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.830+0000 7f512ea00640 1 -- 172.21.15.32:0/729756702 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) ==== 93+0+10 (secure 0 0 0) 0x7f513802dd60 con 0x7f5138075960 2024-04-30T08:35:07.832 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-30T08:35:07.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.833+0000 7f5137200640 1 -- 172.21.15.32:0/729756702 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f511c0afb60 msgr2=0x7f511c0b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:07.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.833+0000 7f5137200640 1 --2- 172.21.15.32:0/729756702 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f511c0afb60 0x7f511c0b2070 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f5138076840 tx=0x7f5128005f90 comp rx=0 tx=0).stop 2024-04-30T08:35:07.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.833+0000 7f5137200640 1 -- 172.21.15.32:0/729756702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5138075960 msgr2=0x7f51381a0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:07.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.833+0000 7f5137200640 1 --2- 172.21.15.32:0/729756702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5138075960 0x7f51381a0c40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f512402a9e0 tx=0x7f5124003c10 comp rx=0 tx=0).stop 2024-04-30T08:35:07.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.833+0000 7f5137200640 1 -- 172.21.15.32:0/729756702 shutdown_connections 2024-04-30T08:35:07.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.833+0000 7f5137200640 1 --2- 172.21.15.32:0/729756702 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f511c0afb60 0x7f511c0b2070 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:07.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.833+0000 7f5137200640 1 --2- 172.21.15.32:0/729756702 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f51381100f0 0x7f51381a7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:07.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.833+0000 7f5137200640 1 --2- 172.21.15.32:0/729756702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5138075960 0x7f51381a0c40 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:07.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.833+0000 7f5137200640 1 --2- 172.21.15.32:0/729756702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5138075010 0x7f51381a0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:07.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.833+0000 7f5137200640 1 -- 172.21.15.32:0/729756702 >> 172.21.15.32:0/729756702 conn(0x7f51380feba0 msgr2=0x7f5138102e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:07.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.833+0000 7f5137200640 1 -- 172.21.15.32:0/729756702 shutdown_connections 2024-04-30T08:35:07.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:07.833+0000 7f5137200640 1 -- 172.21.15.32:0/729756702 wait complete. 2024-04-30T08:35:07.864 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-30T08:35:08.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.147+0000 7f0a1dc00640 1 -- 172.21.15.32:0/3336240047 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a18078c20 msgr2=0x7f0a18079030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:08.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.147+0000 7f0a1dc00640 1 --2- 172.21.15.32:0/3336240047 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a18078c20 0x7f0a18079030 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f0a04009860 tx=0x7f0a0402a340 comp rx=0 tx=0).stop 2024-04-30T08:35:08.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.147+0000 7f0a1dc00640 1 -- 172.21.15.32:0/3336240047 shutdown_connections 2024-04-30T08:35:08.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.147+0000 7f0a1dc00640 1 --2- 172.21.15.32:0/3336240047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a18079920 0x7f0a18079d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:08.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.147+0000 7f0a1dc00640 1 --2- 172.21.15.32:0/3336240047 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a18078c20 0x7f0a18079030 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:08.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.147+0000 7f0a1dc00640 1 --2- 172.21.15.32:0/3336240047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a18077990 0x7f0a180786e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:08.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.147+0000 7f0a1dc00640 1 -- 172.21.15.32:0/3336240047 >> 172.21.15.32:0/3336240047 conn(0x7f0a18100da0 msgr2=0x7f0a18103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:08.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.148+0000 7f0a1dc00640 1 -- 172.21.15.32:0/3336240047 shutdown_connections 2024-04-30T08:35:08.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.148+0000 7f0a1dc00640 1 -- 172.21.15.32:0/3336240047 wait complete. 2024-04-30T08:35:08.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.149+0000 7f0a1dc00640 1 Processor -- start 2024-04-30T08:35:08.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.149+0000 7f0a1dc00640 1 -- start start 2024-04-30T08:35:08.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.149+0000 7f0a1dc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a18077990 0x7f0a181a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:08.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.149+0000 7f0a1dc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a18078c20 0x7f0a181a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:08.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.150+0000 7f0a1dc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a18079920 0x7f0a181a7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:08.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.150+0000 7f0a1dc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0a18115ac0 con 0x7f0a18079920 2024-04-30T08:35:08.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.150+0000 7f0a1dc00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f0a18115940 con 0x7f0a18078c20 2024-04-30T08:35:08.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.150+0000 7f0a1dc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0a18115c40 con 0x7f0a18077990 2024-04-30T08:35:08.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.150+0000 7f0a17400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a18078c20 0x7f0a181a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:08.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.150+0000 7f0a17e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a18077990 0x7f0a181a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:08.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.150+0000 7f0a17400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a18078c20 0x7f0a181a0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:57110/0 (socket says 172.21.15.32:57110) 2024-04-30T08:35:08.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.150+0000 7f0a17e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a18077990 0x7f0a181a0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:44214/0 (socket says 172.21.15.32:44214) 2024-04-30T08:35:08.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.150+0000 7f0a17400640 1 -- 172.21.15.32:0/166117691 learned_addr learned my addr 172.21.15.32:0/166117691 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:08.153 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.150+0000 7f0a1d200640 1 --2- 172.21.15.32:0/166117691 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a18079920 0x7f0a181a7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:08.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.150+0000 7f0a17400640 1 -- 172.21.15.32:0/166117691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a18077990 msgr2=0x7f0a181a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:08.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.150+0000 7f0a17400640 1 --2- 172.21.15.32:0/166117691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a18077990 0x7f0a181a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:08.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.150+0000 7f0a17400640 1 -- 172.21.15.32:0/166117691 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a18079920 msgr2=0x7f0a181a7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:08.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.150+0000 7f0a17400640 1 --2- 172.21.15.32:0/166117691 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a18079920 0x7f0a181a7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:08.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.150+0000 7f0a17400640 1 -- 172.21.15.32:0/166117691 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a181a81b0 con 0x7f0a18078c20 2024-04-30T08:35:08.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.150+0000 7f0a1d200640 1 --2- 172.21.15.32:0/166117691 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a18079920 0x7f0a181a7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:35:08.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.151+0000 7f0a17400640 1 --2- 172.21.15.32:0/166117691 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a18078c20 0x7f0a181a0bb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0a04009980 tx=0x7f0a04032dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:08.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.151+0000 7f0a14c00640 1 -- 172.21.15.32:0/166117691 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a04040020 con 0x7f0a18078c20 2024-04-30T08:35:08.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.151+0000 7f0a1dc00640 1 -- 172.21.15.32:0/166117691 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a181a8440 con 0x7f0a18078c20 2024-04-30T08:35:08.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.151+0000 7f0a1dc00640 1 -- 172.21.15.32:0/166117691 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a181a8980 con 0x7f0a18078c20 2024-04-30T08:35:08.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.151+0000 7f0a14c00640 1 -- 172.21.15.32:0/166117691 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0a04030dd0 con 0x7f0a18078c20 2024-04-30T08:35:08.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.151+0000 7f0a14c00640 1 -- 172.21.15.32:0/166117691 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a04037040 con 0x7f0a18078c20 2024-04-30T08:35:08.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.152+0000 7f0a1dc00640 1 -- 172.21.15.32:0/166117691 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09f80052c0 con 0x7f0a18078c20 2024-04-30T08:35:08.156 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.154+0000 7f0a14c00640 1 -- 172.21.15.32:0/166117691 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7f0a04043020 con 0x7f0a18078c20 2024-04-30T08:35:08.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.156+0000 7f0a14c00640 1 --2- 172.21.15.32:0/166117691 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f09fc0afae0 0x7f09fc0b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:08.157 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.156+0000 7f0a14c00640 1 -- 172.21.15.32:0/166117691 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f0a040fea80 con 0x7f0a18078c20 2024-04-30T08:35:08.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.156+0000 7f0a17e00640 1 --2- 172.21.15.32:0/166117691 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f09fc0afae0 0x7f09fc0b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:08.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.157+0000 7f0a17e00640 1 --2- 172.21.15.32:0/166117691 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f09fc0afae0 0x7f09fc0b1ff0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0a18077260 tx=0x7f0a08009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:08.161 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.160+0000 7f0a14c00640 1 -- 172.21.15.32:0/166117691 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0a04100e20 con 0x7f0a18078c20 2024-04-30T08:35:08.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:08 smithi032 ceph-mon[28331]: from='client.15360 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:35:08.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:08 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/152456604' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:35:08.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:08 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:35:08.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:08 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:35:08.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:08 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:35:08.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:08 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/729756702' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:35:08.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.367+0000 7f0a1dc00640 1 -- 172.21.15.32:0/166117691 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f09f8005dc0 con 0x7f0a18078c20 2024-04-30T08:35:08.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.368+0000 7f0a14c00640 1 -- 172.21.15.32:0/166117691 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) ==== 104+0+11 (secure 0 0 0) 0x7f0a0404a070 con 0x7f0a18078c20 2024-04-30T08:35:08.370 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-30T08:35:08.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.371+0000 7f0a1dc00640 1 -- 172.21.15.32:0/166117691 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f09fc0afae0 msgr2=0x7f09fc0b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:08.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.371+0000 7f0a1dc00640 1 --2- 172.21.15.32:0/166117691 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f09fc0afae0 0x7f09fc0b1ff0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f0a18077260 tx=0x7f0a08009260 comp rx=0 tx=0).stop 2024-04-30T08:35:08.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.371+0000 7f0a1dc00640 1 -- 172.21.15.32:0/166117691 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a18078c20 msgr2=0x7f0a181a0bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:08.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.371+0000 7f0a1dc00640 1 --2- 172.21.15.32:0/166117691 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a18078c20 0x7f0a181a0bb0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f0a04009980 tx=0x7f0a04032dc0 comp rx=0 tx=0).stop 2024-04-30T08:35:08.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.371+0000 7f0a1dc00640 1 -- 172.21.15.32:0/166117691 shutdown_connections 2024-04-30T08:35:08.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.371+0000 7f0a1dc00640 1 --2- 172.21.15.32:0/166117691 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f09fc0afae0 0x7f09fc0b1ff0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:08.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.371+0000 7f0a1dc00640 1 --2- 172.21.15.32:0/166117691 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0a18079920 0x7f0a181a7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:08.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.371+0000 7f0a1dc00640 1 --2- 172.21.15.32:0/166117691 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0a18078c20 0x7f0a181a0bb0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:08.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.371+0000 7f0a1dc00640 1 --2- 172.21.15.32:0/166117691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a18077990 0x7f0a181a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:08.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.371+0000 7f0a1dc00640 1 -- 172.21.15.32:0/166117691 >> 172.21.15.32:0/166117691 conn(0x7f0a18100da0 msgr2=0x7f0a18101410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:08.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.371+0000 7f0a1dc00640 1 -- 172.21.15.32:0/166117691 shutdown_connections 2024-04-30T08:35:08.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.371+0000 7f0a1dc00640 1 -- 172.21.15.32:0/166117691 wait complete. 2024-04-30T08:35:08.402 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-30T08:35:08.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:08 smithi071 ceph-mon[33385]: from='client.15360 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:35:08.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:08 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/152456604' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:35:08.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:08 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:35:08.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:08 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:35:08.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:08 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:35:08.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:08 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/729756702' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:35:08.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:08 smithi189 ceph-mon[33038]: from='client.15360 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-30T08:35:08.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:08 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/152456604' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:35:08.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:08 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:35:08.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:08 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:35:08.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:08 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:35:08.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:08 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/729756702' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:35:08.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.689+0000 7fb65fe00640 1 -- 172.21.15.32:0/1733287233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb66010baf0 msgr2=0x7fb66010df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:08.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.689+0000 7fb65fe00640 1 --2- 172.21.15.32:0/1733287233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb66010baf0 0x7fb66010df10 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb658009ca0 tx=0x7fb65802a8c0 comp rx=0 tx=0).stop 2024-04-30T08:35:08.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.690+0000 7fb65fe00640 1 -- 172.21.15.32:0/1733287233 shutdown_connections 2024-04-30T08:35:08.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.690+0000 7fb65fe00640 1 --2- 172.21.15.32:0/1733287233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb66010baf0 0x7fb66010df10 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:08.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.690+0000 7fb65fe00640 1 --2- 172.21.15.32:0/1733287233 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb660106df0 0x7fb660107200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:08.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.690+0000 7fb65fe00640 1 --2- 172.21.15.32:0/1733287233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb660105bb0 0x7fb6601068b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:08.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.690+0000 7fb65fe00640 1 -- 172.21.15.32:0/1733287233 >> 172.21.15.32:0/1733287233 conn(0x7fb660100da0 msgr2=0x7fb660103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:08.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.690+0000 7fb65fe00640 1 -- 172.21.15.32:0/1733287233 shutdown_connections 2024-04-30T08:35:08.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.690+0000 7fb65fe00640 1 -- 172.21.15.32:0/1733287233 wait complete. 2024-04-30T08:35:08.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.691+0000 7fb65fe00640 1 Processor -- start 2024-04-30T08:35:08.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.691+0000 7fb65fe00640 1 -- start start 2024-04-30T08:35:08.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.691+0000 7fb65fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb660105bb0 0x7fb66019c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:08.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.691+0000 7fb65fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb660106df0 0x7fb66019c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:08.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.691+0000 7fb65fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb66010baf0 0x7fb6601a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:08.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.691+0000 7fb65fe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb660111730 con 0x7fb660105bb0 2024-04-30T08:35:08.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.691+0000 7fb65fe00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb6601115b0 con 0x7fb66010baf0 2024-04-30T08:35:08.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.692+0000 7fb65fe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb6601118b0 con 0x7fb660106df0 2024-04-30T08:35:08.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.692+0000 7fb65ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb660105bb0 0x7fb66019c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:08.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.692+0000 7fb65ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb660105bb0 0x7fb66019c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53448/0 (socket says 172.21.15.32:53448) 2024-04-30T08:35:08.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.692+0000 7fb65ea00640 1 -- 172.21.15.32:0/4038743863 learned_addr learned my addr 172.21.15.32:0/4038743863 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:08.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.692+0000 7fb65f400640 1 --2- 172.21.15.32:0/4038743863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb66010baf0 0x7fb6601a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:08.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.692+0000 7fb65e000640 1 --2- 172.21.15.32:0/4038743863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb660106df0 0x7fb66019c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:08.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.692+0000 7fb65ea00640 1 -- 172.21.15.32:0/4038743863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb660106df0 msgr2=0x7fb66019c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:08.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.692+0000 7fb65ea00640 1 --2- 172.21.15.32:0/4038743863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb660106df0 0x7fb66019c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:08.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.692+0000 7fb65ea00640 1 -- 172.21.15.32:0/4038743863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb66010baf0 msgr2=0x7fb6601a36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:08.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.692+0000 7fb65ea00640 1 --2- 172.21.15.32:0/4038743863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb66010baf0 0x7fb6601a36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:08.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.692+0000 7fb65ea00640 1 -- 172.21.15.32:0/4038743863 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6601a3d60 con 0x7fb660105bb0 2024-04-30T08:35:08.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.693+0000 7fb65ea00640 1 --2- 172.21.15.32:0/4038743863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb660105bb0 0x7fb66019c220 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb65000e9f0 tx=0x7fb65000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:08.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.693+0000 7fb657400640 1 -- 172.21.15.32:0/4038743863 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb65000ccc0 con 0x7fb660105bb0 2024-04-30T08:35:08.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.693+0000 7fb657400640 1 -- 172.21.15.32:0/4038743863 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb6500035f0 con 0x7fb660105bb0 2024-04-30T08:35:08.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.693+0000 7fb657400640 1 -- 172.21.15.32:0/4038743863 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb65000ce60 con 0x7fb660105bb0 2024-04-30T08:35:08.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.693+0000 7fb65fe00640 1 -- 172.21.15.32:0/4038743863 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6601a3ff0 con 0x7fb660105bb0 2024-04-30T08:35:08.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.693+0000 7fb65fe00640 1 -- 172.21.15.32:0/4038743863 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6601a4460 con 0x7fb660105bb0 2024-04-30T08:35:08.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.694+0000 7fb65fe00640 1 -- 172.21.15.32:0/4038743863 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb66010a900 con 0x7fb660105bb0 2024-04-30T08:35:08.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.699+0000 7fb657400640 1 -- 172.21.15.32:0/4038743863 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 18) ==== 147520+0+0 (secure 0 0 0) 0x7fb650003aa0 con 0x7fb660105bb0 2024-04-30T08:35:08.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.700+0000 7fb657400640 1 --2- 172.21.15.32:0/4038743863 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb6440afb60 0x7fb6440b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:08.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.700+0000 7fb657400640 1 -- 172.21.15.32:0/4038743863 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fb6500df3d0 con 0x7fb660105bb0 2024-04-30T08:35:08.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.700+0000 7fb65e000640 1 --2- 172.21.15.32:0/4038743863 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb6440afb60 0x7fb6440b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:08.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.701+0000 7fb65e000640 1 --2- 172.21.15.32:0/4038743863 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb6440afb60 0x7fb6440b2070 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb66019d2c0 tx=0x7fb64c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:08.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.703+0000 7fb657400640 1 -- 172.21.15.32:0/4038743863 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb6500a7ba0 con 0x7fb660105bb0 2024-04-30T08:35:08.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.800+0000 7fb657400640 1 -- 172.21.15.32:0/4038743863 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7fb6500a3730 con 0x7fb660105bb0 2024-04-30T08:35:08.922 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.921+0000 7fb65fe00640 1 -- 172.21.15.32:0/4038743863 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb66002dd60 con 0x7fb660105bb0 2024-04-30T08:35:08.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.922+0000 7fb657400640 1 -- 172.21.15.32:0/4038743863 <== mon.0 v2:172.21.15.32:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) ==== 100+0+11 (secure 0 0 0) 0x7fb6500d96a0 con 0x7fb660105bb0 2024-04-30T08:35:08.923 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-04-30T08:35:08.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.924+0000 7fb65fe00640 1 -- 172.21.15.32:0/4038743863 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb6440afb60 msgr2=0x7fb6440b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:08.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.924+0000 7fb65fe00640 1 --2- 172.21.15.32:0/4038743863 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb6440afb60 0x7fb6440b2070 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fb66019d2c0 tx=0x7fb64c005f90 comp rx=0 tx=0).stop 2024-04-30T08:35:08.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.924+0000 7fb65fe00640 1 -- 172.21.15.32:0/4038743863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb660105bb0 msgr2=0x7fb66019c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:08.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.924+0000 7fb65fe00640 1 --2- 172.21.15.32:0/4038743863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb660105bb0 0x7fb66019c220 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb65000e9f0 tx=0x7fb65000eeb0 comp rx=0 tx=0).stop 2024-04-30T08:35:08.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.924+0000 7fb65fe00640 1 -- 172.21.15.32:0/4038743863 shutdown_connections 2024-04-30T08:35:08.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.924+0000 7fb65fe00640 1 --2- 172.21.15.32:0/4038743863 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb6440afb60 0x7fb6440b2070 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:08.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.924+0000 7fb65fe00640 1 --2- 172.21.15.32:0/4038743863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb66010baf0 0x7fb6601a36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:08.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.924+0000 7fb65fe00640 1 --2- 172.21.15.32:0/4038743863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb660106df0 0x7fb66019c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:08.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.924+0000 7fb65fe00640 1 --2- 172.21.15.32:0/4038743863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb660105bb0 0x7fb66019c220 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:08.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.924+0000 7fb65fe00640 1 -- 172.21.15.32:0/4038743863 >> 172.21.15.32:0/4038743863 conn(0x7fb660100da0 msgr2=0x7fb660107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:08.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.925+0000 7fb65fe00640 1 -- 172.21.15.32:0/4038743863 shutdown_connections 2024-04-30T08:35:08.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:08.925+0000 7fb65fe00640 1 -- 172.21.15.32:0/4038743863 wait complete. 2024-04-30T08:35:08.954 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-30T08:35:09.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.249+0000 7f3beda00640 1 -- 172.21.15.32:0/274212449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3be8105bb0 msgr2=0x7f3be81068b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:09.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.249+0000 7f3beda00640 1 --2- 172.21.15.32:0/274212449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3be8105bb0 0x7f3be81068b0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f3bd8009900 tx=0x7f3bd802a410 comp rx=0 tx=0).stop 2024-04-30T08:35:09.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.249+0000 7f3beda00640 1 -- 172.21.15.32:0/274212449 shutdown_connections 2024-04-30T08:35:09.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.249+0000 7f3beda00640 1 --2- 172.21.15.32:0/274212449 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3be810baf0 0x7f3be810df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:09.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.249+0000 7f3beda00640 1 --2- 172.21.15.32:0/274212449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3be8106df0 0x7f3be8107200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:09.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.249+0000 7f3beda00640 1 --2- 172.21.15.32:0/274212449 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3be8105bb0 0x7f3be81068b0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:09.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.249+0000 7f3beda00640 1 -- 172.21.15.32:0/274212449 >> 172.21.15.32:0/274212449 conn(0x7f3be8100da0 msgr2=0x7f3be8103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:09.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.249+0000 7f3beda00640 1 -- 172.21.15.32:0/274212449 shutdown_connections 2024-04-30T08:35:09.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.249+0000 7f3beda00640 1 -- 172.21.15.32:0/274212449 wait complete. 2024-04-30T08:35:09.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.250+0000 7f3beda00640 1 Processor -- start 2024-04-30T08:35:09.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.250+0000 7f3beda00640 1 -- start start 2024-04-30T08:35:09.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.251+0000 7f3beda00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3be8105bb0 0x7f3be819c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:09.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.251+0000 7f3beda00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3be8106df0 0x7f3be819c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:09.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.251+0000 7f3beda00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3be810baf0 0x7f3be81a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:09.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.251+0000 7f3beda00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3be81116e0 con 0x7f3be8106df0 2024-04-30T08:35:09.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.251+0000 7f3beda00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f3be8111560 con 0x7f3be8105bb0 2024-04-30T08:35:09.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.251+0000 7f3beda00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3be8111860 con 0x7f3be810baf0 2024-04-30T08:35:09.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.251+0000 7f3bdfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3be8106df0 0x7f3be819c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:09.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.251+0000 7f3bdfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3be8106df0 0x7f3be819c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53466/0 (socket says 172.21.15.32:53466) 2024-04-30T08:35:09.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.251+0000 7f3bdfe00640 1 -- 172.21.15.32:0/2883860489 learned_addr learned my addr 172.21.15.32:0/2883860489 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:09.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.251+0000 7f3be7e00640 1 --2- 172.21.15.32:0/2883860489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3be8105bb0 0x7f3be819c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:09.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.251+0000 7f3bed000640 1 --2- 172.21.15.32:0/2883860489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3be810baf0 0x7f3be81a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:09.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.252+0000 7f3bdfe00640 1 -- 172.21.15.32:0/2883860489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3be810baf0 msgr2=0x7f3be81a37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:09.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.252+0000 7f3bdfe00640 1 --2- 172.21.15.32:0/2883860489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3be810baf0 0x7f3be81a37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:09.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.252+0000 7f3bdfe00640 1 -- 172.21.15.32:0/2883860489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3be8105bb0 msgr2=0x7f3be819c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:09.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.252+0000 7f3bdfe00640 1 --2- 172.21.15.32:0/2883860489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3be8105bb0 0x7f3be819c310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:09.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.252+0000 7f3bdfe00640 1 -- 172.21.15.32:0/2883860489 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3be81a3e50 con 0x7f3be8106df0 2024-04-30T08:35:09.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.252+0000 7f3bdfe00640 1 --2- 172.21.15.32:0/2883860489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3be8106df0 0x7f3be819c850 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3bd000e9f0 tx=0x7f3bd000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:09.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.252+0000 7f3be5600640 1 -- 172.21.15.32:0/2883860489 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd000ccc0 con 0x7f3be8106df0 2024-04-30T08:35:09.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.252+0000 7f3beda00640 1 -- 172.21.15.32:0/2883860489 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3be81a4140 con 0x7f3be8106df0 2024-04-30T08:35:09.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.252+0000 7f3be5600640 1 -- 172.21.15.32:0/2883860489 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3bd00108e0 con 0x7f3be8106df0 2024-04-30T08:35:09.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.252+0000 7f3be5600640 1 -- 172.21.15.32:0/2883860489 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3bd0018880 con 0x7f3be8106df0 2024-04-30T08:35:09.256 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.252+0000 7f3beda00640 1 -- 172.21.15.32:0/2883860489 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3be81a46e0 con 0x7f3be8106df0 2024-04-30T08:35:09.257 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.254+0000 7f3beda00640 1 -- 172.21.15.32:0/2883860489 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3be810a900 con 0x7f3be8106df0 2024-04-30T08:35:09.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.257+0000 7f3be5600640 1 -- 172.21.15.32:0/2883860489 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f3bd0010430 con 0x7f3be8106df0 2024-04-30T08:35:09.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.258+0000 7f3be5600640 1 --2- 172.21.15.32:0/2883860489 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3bcc0afbf0 0x7f3bcc0b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:09.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.259+0000 7f3be5600640 1 -- 172.21.15.32:0/2883860489 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f3bd00df960 con 0x7f3be8106df0 2024-04-30T08:35:09.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.259+0000 7f3be7e00640 1 --2- 172.21.15.32:0/2883860489 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3bcc0afbf0 0x7f3bcc0b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:09.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.260+0000 7f3be7e00640 1 --2- 172.21.15.32:0/2883860489 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3bcc0afbf0 0x7f3bcc0b2100 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f3bd802aa00 tx=0x7f3bd80314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:09.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.261+0000 7f3be5600640 1 -- 172.21.15.32:0/2883860489 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3bd00a8010 con 0x7f3be8106df0 2024-04-30T08:35:09.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:09 smithi032 ceph-mon[28331]: pgmap v424: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 341 B/s wr, 9 op/s 2024-04-30T08:35:09.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:09 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/166117691' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-30T08:35:09.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:09 smithi032 ceph-mon[28331]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-04-30T08:35:09.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:09 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/4038743863' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-30T08:35:09.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.467+0000 7f3beda00640 1 -- 172.21.15.32:0/2883860489 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f3be8063820 con 0x7f3be8106df0 2024-04-30T08:35:09.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.468+0000 7f3be5600640 1 -- 172.21.15.32:0/2883860489 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v85) ==== 103+0+11 (secure 0 0 0) 0x7f3bd00a7910 con 0x7f3be8106df0 2024-04-30T08:35:09.470 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-30T08:35:09.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.471+0000 7f3beda00640 1 -- 172.21.15.32:0/2883860489 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3bcc0afbf0 msgr2=0x7f3bcc0b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:09.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.471+0000 7f3beda00640 1 --2- 172.21.15.32:0/2883860489 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3bcc0afbf0 0x7f3bcc0b2100 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f3bd802aa00 tx=0x7f3bd80314a0 comp rx=0 tx=0).stop 2024-04-30T08:35:09.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.471+0000 7f3beda00640 1 -- 172.21.15.32:0/2883860489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3be8106df0 msgr2=0x7f3be819c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:09.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.471+0000 7f3beda00640 1 --2- 172.21.15.32:0/2883860489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3be8106df0 0x7f3be819c850 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f3bd000e9f0 tx=0x7f3bd000edf0 comp rx=0 tx=0).stop 2024-04-30T08:35:09.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.471+0000 7f3beda00640 1 -- 172.21.15.32:0/2883860489 shutdown_connections 2024-04-30T08:35:09.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.471+0000 7f3beda00640 1 --2- 172.21.15.32:0/2883860489 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3bcc0afbf0 0x7f3bcc0b2100 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:09.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.471+0000 7f3beda00640 1 --2- 172.21.15.32:0/2883860489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3be810baf0 0x7f3be81a37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:09.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.471+0000 7f3beda00640 1 --2- 172.21.15.32:0/2883860489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3be8106df0 0x7f3be819c850 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:09.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.471+0000 7f3beda00640 1 --2- 172.21.15.32:0/2883860489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3be8105bb0 0x7f3be819c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:09.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.471+0000 7f3beda00640 1 -- 172.21.15.32:0/2883860489 >> 172.21.15.32:0/2883860489 conn(0x7f3be8100da0 msgr2=0x7f3be8107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:09.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.471+0000 7f3beda00640 1 -- 172.21.15.32:0/2883860489 shutdown_connections 2024-04-30T08:35:09.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.471+0000 7f3beda00640 1 -- 172.21.15.32:0/2883860489 wait complete. 2024-04-30T08:35:09.499 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-30T08:35:09.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:09 smithi071 ceph-mon[33385]: pgmap v424: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 341 B/s wr, 9 op/s 2024-04-30T08:35:09.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:09 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/166117691' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-30T08:35:09.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:09 smithi071 ceph-mon[33385]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-04-30T08:35:09.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:09 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/4038743863' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-30T08:35:09.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:09 smithi189 ceph-mon[33038]: pgmap v424: 129 pgs: 129 active+clean; 645 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 341 B/s wr, 9 op/s 2024-04-30T08:35:09.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:09 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/166117691' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-30T08:35:09.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:09 smithi189 ceph-mon[33038]: mgrmap e19: x(active, since 11m), standbys: z, y 2024-04-30T08:35:09.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:09 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/4038743863' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-30T08:35:09.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.784+0000 7f8972a00640 1 -- 172.21.15.32:0/3497763756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f896c10baf0 msgr2=0x7f896c10df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:09.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.784+0000 7f8972a00640 1 --2- 172.21.15.32:0/3497763756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f896c10baf0 0x7f896c10df10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f8964009900 tx=0x7f896402a3f0 comp rx=0 tx=0).stop 2024-04-30T08:35:09.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.784+0000 7f8972a00640 1 -- 172.21.15.32:0/3497763756 shutdown_connections 2024-04-30T08:35:09.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.784+0000 7f8972a00640 1 --2- 172.21.15.32:0/3497763756 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f896c10baf0 0x7f896c10df10 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:09.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.784+0000 7f8972a00640 1 --2- 172.21.15.32:0/3497763756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f896c106df0 0x7f896c107200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:09.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.784+0000 7f8972a00640 1 --2- 172.21.15.32:0/3497763756 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f896c105bb0 0x7f896c1068b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:09.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.784+0000 7f8972a00640 1 -- 172.21.15.32:0/3497763756 >> 172.21.15.32:0/3497763756 conn(0x7f896c100da0 msgr2=0x7f896c103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:09.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.784+0000 7f8972a00640 1 -- 172.21.15.32:0/3497763756 shutdown_connections 2024-04-30T08:35:09.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.784+0000 7f8972a00640 1 -- 172.21.15.32:0/3497763756 wait complete. 2024-04-30T08:35:09.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.785+0000 7f8972a00640 1 Processor -- start 2024-04-30T08:35:09.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.786+0000 7f8972a00640 1 -- start start 2024-04-30T08:35:09.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.786+0000 7f8972a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f896c105bb0 0x7f896c19c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:09.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.786+0000 7f8972a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f896c106df0 0x7f896c19c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:09.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.786+0000 7f8972a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f896c10baf0 0x7f896c1a36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:09.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.786+0000 7f8972a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f896c111710 con 0x7f896c105bb0 2024-04-30T08:35:09.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.786+0000 7f8972a00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f896c111590 con 0x7f896c10baf0 2024-04-30T08:35:09.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.786+0000 7f8972a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f896c111890 con 0x7f896c106df0 2024-04-30T08:35:09.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.786+0000 7f8971600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f896c105bb0 0x7f896c19c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:09.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.786+0000 7f8971600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f896c105bb0 0x7f896c19c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53490/0 (socket says 172.21.15.32:53490) 2024-04-30T08:35:09.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.786+0000 7f8971600640 1 -- 172.21.15.32:0/603137226 learned_addr learned my addr 172.21.15.32:0/603137226 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:09.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.786+0000 7f8972000640 1 --2- 172.21.15.32:0/603137226 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f896c10baf0 0x7f896c1a36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:09.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.786+0000 7f8970c00640 1 --2- 172.21.15.32:0/603137226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f896c106df0 0x7f896c19c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:09.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.787+0000 7f8971600640 1 -- 172.21.15.32:0/603137226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f896c106df0 msgr2=0x7f896c19c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:09.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.787+0000 7f8971600640 1 --2- 172.21.15.32:0/603137226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f896c106df0 0x7f896c19c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:09.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.787+0000 7f8971600640 1 -- 172.21.15.32:0/603137226 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f896c10baf0 msgr2=0x7f896c1a36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:09.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.787+0000 7f8971600640 1 --2- 172.21.15.32:0/603137226 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f896c10baf0 0x7f896c1a36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:09.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.787+0000 7f8971600640 1 -- 172.21.15.32:0/603137226 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f896c1a3d60 con 0x7f896c105bb0 2024-04-30T08:35:09.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.787+0000 7f8971600640 1 --2- 172.21.15.32:0/603137226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f896c105bb0 0x7f896c19c220 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f896000ed00 tx=0x7f896000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:09.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.787+0000 7f896a000640 1 -- 172.21.15.32:0/603137226 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8960010630 con 0x7f896c105bb0 2024-04-30T08:35:09.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.787+0000 7f896a000640 1 -- 172.21.15.32:0/603137226 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8960010c80 con 0x7f896c105bb0 2024-04-30T08:35:09.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.787+0000 7f8972a00640 1 -- 172.21.15.32:0/603137226 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f896c1a3ff0 con 0x7f896c105bb0 2024-04-30T08:35:09.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.788+0000 7f896a000640 1 -- 172.21.15.32:0/603137226 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8960018d50 con 0x7f896c105bb0 2024-04-30T08:35:09.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.788+0000 7f8972a00640 1 -- 172.21.15.32:0/603137226 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f896c1a4500 con 0x7f896c105bb0 2024-04-30T08:35:09.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.789+0000 7f8972a00640 1 -- 172.21.15.32:0/603137226 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f896c10a900 con 0x7f896c105bb0 2024-04-30T08:35:09.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.791+0000 7f896a000640 1 -- 172.21.15.32:0/603137226 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f896001f020 con 0x7f896c105bb0 2024-04-30T08:35:09.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.793+0000 7f896a000640 1 --2- 172.21.15.32:0/603137226 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f89540afbf0 0x7f89540b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:09.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.793+0000 7f896a000640 1 -- 172.21.15.32:0/603137226 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f89600df4d0 con 0x7f896c105bb0 2024-04-30T08:35:09.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.794+0000 7f8970c00640 1 --2- 172.21.15.32:0/603137226 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f89540afbf0 0x7f89540b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:09.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.794+0000 7f8970c00640 1 --2- 172.21.15.32:0/603137226 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f89540afbf0 0x7f89540b2100 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f896c19d2c0 tx=0x7f8958005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:09.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:09.797+0000 7f896a000640 1 -- 172.21.15.32:0/603137226 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f89600a7b80 con 0x7f896c105bb0 2024-04-30T08:35:10.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.009+0000 7f8972a00640 1 -- 172.21.15.32:0/603137226 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f896c063820 con 0x7f896c105bb0 2024-04-30T08:35:10.013 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.012+0000 7f896a000640 1 -- 172.21.15.32:0/603137226 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+24730 (secure 0 0 0) 0x7f89600a7480 con 0x7f896c105bb0 2024-04-30T08:35:10.014 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:35:10.014 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":85,"fsid":"98c0a670-06ca-11ef-bc93-c7b262605968","created":"2024-04-30T08:22:11.122875+0000","modified":"2024-04-30T08:33:31.872704+0000","last_up_change":"2024-04-30T08:30:50.350439+0000","last_in_change":"2024-04-30T08:30:30.210620+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-30T08:26:19.728331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-30T08:31:49.800542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-30T08:31:51.211320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-30T08:31:55.716625+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a7e060c8-1cbb-418f-9a03-06693f6b3f52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6803","nonce":2624700402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6805","nonce":2624700402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6809","nonce":2624700402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6807","nonce":2624700402}]},"public_addr":"172.21.15.32:6803/2624700402","cluster_addr":"172.21.15.32:6805/2624700402","heartbeat_back_addr":"172.21.15.32:6809/2624700402","heartbeat_front_addr":"172.21.15.32:6807/2624700402","state":["exists","up"]},{"osd":1,"uuid":"72ef2fb1-e5ca-48ef-aa3a-4796a64b922f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6801","nonce":444649711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6803","nonce":444649711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6807","nonce":444649711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6805","nonce":444649711}]},"public_addr":"172.21.15.71:6801/444649711","cluster_addr":"172.21.15.71:6803/444649711","heartbeat_back_addr":"172.21.15.71:6807/444649711","heartbeat_front_addr":"172.21.15.71:6805/444649711","state":["exists","up"]},{"osd":2,"uuid":"704e5e8a-935e-47c7-ae28-27119876ac09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6801","nonce":3255528671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6803","nonce":3255528671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6807","nonce":3255528671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6805","nonce":3255528671}]},"public_addr":"172.21.15.189:6801/3255528671","cluster_addr":"172.21.15.189:6803/3255528671","heartbeat_back_addr":"172.21.15.189:6807/3255528671","heartbeat_front_addr":"172.21.15.189:6805/3255528671","state":["exists","up"]},{"osd":3,"uuid":"575004f7-6559-4a62-863e-e33078b0eebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6811","nonce":1384702112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6813","nonce":1384702112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6817","nonce":1384702112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6815","nonce":1384702112}]},"public_addr":"172.21.15.32:6811/1384702112","cluster_addr":"172.21.15.32:6813/1384702112","heartbeat_back_addr":"172.21.15.32:6817/1384702112","heartbeat_front_addr":"172.21.15.32:6815/1384702112","state":["exists","up"]},{"osd":4,"uuid":"e1c1e894-074d-4c78-89ef-9e5a114268bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6809","nonce":4250534574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6811","nonce":4250534574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6815","nonce":4250534574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6813","nonce":4250534574}]},"public_addr":"172.21.15.71:6809/4250534574","cluster_addr":"172.21.15.71:6811/4250534574","heartbeat_back_addr":"172.21.15.71:6815/4250534574","heartbeat_front_addr":"172.21.15.71:6813/4250534574","state":["exists","up"]},{"osd":5,"uuid":"12c022d4-fac6-40c4-ae7a-691571bee5d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6809","nonce":4245346198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6811","nonce":4245346198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6815","nonce":4245346198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6813","nonce":4245346198}]},"public_addr":"172.21.15.189:6809/4245346198","cluster_addr":"172.21.15.189:6811/4245346198","heartbeat_back_addr":"172.21.15.189:6815/4245346198","heartbeat_front_addr":"172.21.15.189:6813/4245346198","state":["exists","up"]},{"osd":6,"uuid":"b413f8ee-3d4a-415a-a2ba-d68437b7330d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6819","nonce":1910389190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6821","nonce":1910389190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6825","nonce":1910389190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6823","nonce":1910389190}]},"public_addr":"172.21.15.32:6819/1910389190","cluster_addr":"172.21.15.32:6821/1910389190","heartbeat_back_addr":"172.21.15.32:6825/1910389190","heartbeat_front_addr":"172.21.15.32:6823/1910389190","state":["exists","up"]},{"osd":7,"uuid":"aac64fb3-fc13-4126-b842-7b06ad5ce238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6817","nonce":692607380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6819","nonce":692607380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6823","nonce":692607380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6821","nonce":692607380}]},"public_addr":"172.21.15.71:6817/692607380","cluster_addr":"172.21.15.71:6819/692607380","heartbeat_back_addr":"172.21.15.71:6823/692607380","heartbeat_front_addr":"172.21.15.71:6821/692607380","state":["exists","up"]},{"osd":8,"uuid":"cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6817","nonce":3973697069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6819","nonce":3973697069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6823","nonce":3973697069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6821","nonce":3973697069}]},"public_addr":"172.21.15.189:6817/3973697069","cluster_addr":"172.21.15.189:6819/3973697069","heartbeat_back_addr":"172.21.15.189:6823/3973697069","heartbeat_front_addr":"172.21.15.189:6821/3973697069","state":["exists","up"]},{"osd":9,"uuid":"ef4a1645-ae6e-45c3-82ad-161510e62aa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6827","nonce":46066685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6829","nonce":46066685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6833","nonce":46066685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6831","nonce":46066685}]},"public_addr":"172.21.15.32:6827/46066685","cluster_addr":"172.21.15.32:6829/46066685","heartbeat_back_addr":"172.21.15.32:6833/46066685","heartbeat_front_addr":"172.21.15.32:6831/46066685","state":["exists","up"]},{"osd":10,"uuid":"25b26a38-a644-409f-b941-59efdf04a3ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6825","nonce":1886290293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6827","nonce":1886290293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6831","nonce":1886290293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6829","nonce":1886290293}]},"public_addr":"172.21.15.71:6825/1886290293","cluster_addr":"172.21.15.71:6827/1886290293","heartbeat_back_addr":"172.21.15.71:6831/1886290293","heartbeat_front_addr":"172.21.15.71:6829/1886290293","state":["exists","up"]},{"osd":11,"uuid":"ab96b3ff-3688-455b-af64-60226c5bfbff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6825","nonce":850185001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6827","nonce":850185001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6831","nonce":850185001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6829","nonce":850185001}]},"public_addr":"172.21.15.189:6825/850185001","cluster_addr":"172.21.15.189:6827/850185001","heartbeat_back_addr":"172.21.15.189:6831/850185001","heartbeat_front_addr":"172.21.15.189:6829/850185001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:27.220308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:53.230783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:17.190642+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:52.603799+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:20.568671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:46.158363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:22.706933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:50.665173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:16.602410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:53.814698+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:22.435189+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:48.651949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.e","mappings":[{"from":4,"to":7}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.28","mappings":[{"from":3,"to":6},{"from":5,"to":8}]},{"pgid":"3.30","mappings":[{"from":1,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1266897222":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/2915689533":"2024-05-01T08:23:29.526664+0000","172.21.15.32:6801/4019351306":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/425836193":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/1134075722":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6801/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/21059253":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/3245550894":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/4019351306":"2024-05-01T08:23:29.526664+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-30T08:35:10.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.015+0000 7f8972a00640 1 -- 172.21.15.32:0/603137226 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f89540afbf0 msgr2=0x7f89540b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:10.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.015+0000 7f8972a00640 1 --2- 172.21.15.32:0/603137226 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f89540afbf0 0x7f89540b2100 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f896c19d2c0 tx=0x7f8958005f90 comp rx=0 tx=0).stop 2024-04-30T08:35:10.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.015+0000 7f8972a00640 1 -- 172.21.15.32:0/603137226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f896c105bb0 msgr2=0x7f896c19c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:10.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.015+0000 7f8972a00640 1 --2- 172.21.15.32:0/603137226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f896c105bb0 0x7f896c19c220 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f896000ed00 tx=0x7f896000c6a0 comp rx=0 tx=0).stop 2024-04-30T08:35:10.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.015+0000 7f8972a00640 1 -- 172.21.15.32:0/603137226 shutdown_connections 2024-04-30T08:35:10.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.015+0000 7f8972a00640 1 --2- 172.21.15.32:0/603137226 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f89540afbf0 0x7f89540b2100 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:10.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.015+0000 7f8972a00640 1 --2- 172.21.15.32:0/603137226 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f896c10baf0 0x7f896c1a36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:10.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.015+0000 7f8972a00640 1 --2- 172.21.15.32:0/603137226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f896c106df0 0x7f896c19c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:10.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.015+0000 7f8972a00640 1 --2- 172.21.15.32:0/603137226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f896c105bb0 0x7f896c19c220 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:10.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.015+0000 7f8972a00640 1 -- 172.21.15.32:0/603137226 >> 172.21.15.32:0/603137226 conn(0x7f896c100da0 msgr2=0x7f896c107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:10.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.015+0000 7f8972a00640 1 -- 172.21.15.32:0/603137226 shutdown_connections 2024-04-30T08:35:10.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.015+0000 7f8972a00640 1 -- 172.21.15.32:0/603137226 wait complete. 2024-04-30T08:35:10.047 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-30T08:26:19.728331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-30T08:31:49.800542+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-30T08:31:51.211320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-30T08:31:55.716625+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-30T08:35:10.047 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-30T08:35:10.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.337+0000 7f8d48c00640 1 -- 172.21.15.32:0/2488949100 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d44105bb0 msgr2=0x7f8d441068b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:10.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.337+0000 7f8d48c00640 1 --2- 172.21.15.32:0/2488949100 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d44105bb0 0x7f8d441068b0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f8d34009900 tx=0x7f8d3402a3f0 comp rx=0 tx=0).stop 2024-04-30T08:35:10.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.337+0000 7f8d48c00640 1 -- 172.21.15.32:0/2488949100 shutdown_connections 2024-04-30T08:35:10.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.337+0000 7f8d48c00640 1 --2- 172.21.15.32:0/2488949100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d4410baf0 0x7f8d4410df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:10.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.337+0000 7f8d48c00640 1 --2- 172.21.15.32:0/2488949100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d44106df0 0x7f8d44107200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:10.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.337+0000 7f8d48c00640 1 --2- 172.21.15.32:0/2488949100 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d44105bb0 0x7f8d441068b0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:10.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.337+0000 7f8d48c00640 1 -- 172.21.15.32:0/2488949100 >> 172.21.15.32:0/2488949100 conn(0x7f8d44100da0 msgr2=0x7f8d44103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:10.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.337+0000 7f8d48c00640 1 -- 172.21.15.32:0/2488949100 shutdown_connections 2024-04-30T08:35:10.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.337+0000 7f8d48c00640 1 -- 172.21.15.32:0/2488949100 wait complete. 2024-04-30T08:35:10.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.338+0000 7f8d48c00640 1 Processor -- start 2024-04-30T08:35:10.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.339+0000 7f8d48c00640 1 -- start start 2024-04-30T08:35:10.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.339+0000 7f8d48c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d44105bb0 0x7f8d4419c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:10.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.339+0000 7f8d48c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d44106df0 0x7f8d4419c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:10.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.339+0000 7f8d48c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d4410baf0 0x7f8d441a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:10.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.339+0000 7f8d48c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8d441116b0 con 0x7f8d44105bb0 2024-04-30T08:35:10.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.339+0000 7f8d48c00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8d44111530 con 0x7f8d44106df0 2024-04-30T08:35:10.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.339+0000 7f8d48c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8d44111830 con 0x7f8d4410baf0 2024-04-30T08:35:10.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.339+0000 7f8d43400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d44105bb0 0x7f8d4419c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:10.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.339+0000 7f8d42a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d44106df0 0x7f8d4419c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:10.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.339+0000 7f8d42a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d44106df0 0x7f8d4419c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:57174/0 (socket says 172.21.15.32:57174) 2024-04-30T08:35:10.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.339+0000 7f8d43400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d44105bb0 0x7f8d4419c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53510/0 (socket says 172.21.15.32:53510) 2024-04-30T08:35:10.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.339+0000 7f8d42a00640 1 -- 172.21.15.32:0/3313251217 learned_addr learned my addr 172.21.15.32:0/3313251217 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:10.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.340+0000 7f8d43e00640 1 --2- 172.21.15.32:0/3313251217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d4410baf0 0x7f8d441a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:10.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.340+0000 7f8d42a00640 1 -- 172.21.15.32:0/3313251217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d4410baf0 msgr2=0x7f8d441a37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:10.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.340+0000 7f8d42a00640 1 --2- 172.21.15.32:0/3313251217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d4410baf0 0x7f8d441a37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:10.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.340+0000 7f8d42a00640 1 -- 172.21.15.32:0/3313251217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d44105bb0 msgr2=0x7f8d4419c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:10.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.340+0000 7f8d42a00640 1 --2- 172.21.15.32:0/3313251217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d44105bb0 0x7f8d4419c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:10.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.340+0000 7f8d42a00640 1 -- 172.21.15.32:0/3313251217 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d441a3e40 con 0x7f8d44106df0 2024-04-30T08:35:10.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.340+0000 7f8d42a00640 1 --2- 172.21.15.32:0/3313251217 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d44106df0 0x7f8d4419c8b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8d3000ea60 tx=0x7f8d3000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:10.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.340+0000 7f8d3be00640 1 -- 172.21.15.32:0/3313251217 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d3000cde0 con 0x7f8d44106df0 2024-04-30T08:35:10.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.341+0000 7f8d48c00640 1 -- 172.21.15.32:0/3313251217 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d441a4130 con 0x7f8d44106df0 2024-04-30T08:35:10.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.341+0000 7f8d3be00640 1 -- 172.21.15.32:0/3313251217 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8d300035f0 con 0x7f8d44106df0 2024-04-30T08:35:10.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.341+0000 7f8d3be00640 1 -- 172.21.15.32:0/3313251217 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d30010690 con 0x7f8d44106df0 2024-04-30T08:35:10.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.341+0000 7f8d48c00640 1 -- 172.21.15.32:0/3313251217 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d441a46d0 con 0x7f8d44106df0 2024-04-30T08:35:10.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.342+0000 7f8d48c00640 1 -- 172.21.15.32:0/3313251217 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d4410a900 con 0x7f8d44106df0 2024-04-30T08:35:10.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.345+0000 7f8d3be00640 1 -- 172.21.15.32:0/3313251217 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f8d300037a0 con 0x7f8d44106df0 2024-04-30T08:35:10.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.347+0000 7f8d3be00640 1 --2- 172.21.15.32:0/3313251217 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8d280afb70 0x7f8d280b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:10.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.347+0000 7f8d3be00640 1 -- 172.21.15.32:0/3313251217 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f8d300df8b0 con 0x7f8d44106df0 2024-04-30T08:35:10.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.347+0000 7f8d43400640 1 --2- 172.21.15.32:0/3313251217 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8d280afb70 0x7f8d280b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:10.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.348+0000 7f8d3be00640 1 -- 172.21.15.32:0/3313251217 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8d300a7f20 con 0x7f8d44106df0 2024-04-30T08:35:10.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.348+0000 7f8d43400640 1 --2- 172.21.15.32:0/3313251217 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8d280afb70 0x7f8d280b2080 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8d34005ab0 tx=0x7f8d34031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:10.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:10 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2883860489' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-30T08:35:10.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:10 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/603137226' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:35:10.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:10 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2883860489' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-30T08:35:10.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:10 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/603137226' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:35:10.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:10 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2883860489' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-30T08:35:10.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:10 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/603137226' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:35:10.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.557+0000 7f8d48c00640 1 -- 172.21.15.32:0/3313251217 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f8d44063820 con 0x7f8d44106df0 2024-04-30T08:35:10.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.558+0000 7f8d3be00640 1 -- 172.21.15.32:0/3313251217 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) ==== 93+0+10 (secure 0 0 0) 0x7f8d300a7820 con 0x7f8d44106df0 2024-04-30T08:35:10.560 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-30T08:35:10.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.561+0000 7f8d48c00640 1 -- 172.21.15.32:0/3313251217 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8d280afb70 msgr2=0x7f8d280b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:10.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.561+0000 7f8d48c00640 1 --2- 172.21.15.32:0/3313251217 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8d280afb70 0x7f8d280b2080 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8d34005ab0 tx=0x7f8d34031400 comp rx=0 tx=0).stop 2024-04-30T08:35:10.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.561+0000 7f8d48c00640 1 -- 172.21.15.32:0/3313251217 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d44106df0 msgr2=0x7f8d4419c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:10.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.561+0000 7f8d48c00640 1 --2- 172.21.15.32:0/3313251217 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d44106df0 0x7f8d4419c8b0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f8d3000ea60 tx=0x7f8d3000ef20 comp rx=0 tx=0).stop 2024-04-30T08:35:10.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.562+0000 7f8d48c00640 1 -- 172.21.15.32:0/3313251217 shutdown_connections 2024-04-30T08:35:10.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.562+0000 7f8d48c00640 1 --2- 172.21.15.32:0/3313251217 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8d280afb70 0x7f8d280b2080 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:10.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.562+0000 7f8d48c00640 1 --2- 172.21.15.32:0/3313251217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d4410baf0 0x7f8d441a37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:10.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.562+0000 7f8d48c00640 1 --2- 172.21.15.32:0/3313251217 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8d44106df0 0x7f8d4419c8b0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:10.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.562+0000 7f8d48c00640 1 --2- 172.21.15.32:0/3313251217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8d44105bb0 0x7f8d4419c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:10.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.562+0000 7f8d48c00640 1 -- 172.21.15.32:0/3313251217 >> 172.21.15.32:0/3313251217 conn(0x7f8d44100da0 msgr2=0x7f8d44107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:10.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.562+0000 7f8d48c00640 1 -- 172.21.15.32:0/3313251217 shutdown_connections 2024-04-30T08:35:10.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.562+0000 7f8d48c00640 1 -- 172.21.15.32:0/3313251217 wait complete. 2024-04-30T08:35:10.589 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-30T08:35:10.888 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.887+0000 7fd951200640 1 -- 172.21.15.32:0/1459555617 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd94c105d70 msgr2=0x7fd94c106a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:10.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.887+0000 7fd951200640 1 --2- 172.21.15.32:0/1459555617 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd94c105d70 0x7fd94c106a70 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fd93c009900 tx=0x7fd93c02a3f0 comp rx=0 tx=0).stop 2024-04-30T08:35:10.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.887+0000 7fd951200640 1 -- 172.21.15.32:0/1459555617 shutdown_connections 2024-04-30T08:35:10.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.887+0000 7fd951200640 1 --2- 172.21.15.32:0/1459555617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd94c10bcb0 0x7fd94c10e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:10.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.887+0000 7fd951200640 1 --2- 172.21.15.32:0/1459555617 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd94c106fb0 0x7fd94c1073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:10.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.887+0000 7fd951200640 1 --2- 172.21.15.32:0/1459555617 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd94c105d70 0x7fd94c106a70 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:10.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.887+0000 7fd951200640 1 -- 172.21.15.32:0/1459555617 >> 172.21.15.32:0/1459555617 conn(0x7fd94c100f80 msgr2=0x7fd94c103410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:10.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.887+0000 7fd951200640 1 -- 172.21.15.32:0/1459555617 shutdown_connections 2024-04-30T08:35:10.889 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.887+0000 7fd951200640 1 -- 172.21.15.32:0/1459555617 wait complete. 2024-04-30T08:35:10.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.888+0000 7fd951200640 1 Processor -- start 2024-04-30T08:35:10.890 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.889+0000 7fd951200640 1 -- start start 2024-04-30T08:35:10.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.889+0000 7fd951200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd94c105d70 0x7fd94c19c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:10.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.889+0000 7fd951200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd94c106fb0 0x7fd94c19c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:10.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.889+0000 7fd951200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd94c10bcb0 0x7fd94c1a3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:10.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.889+0000 7fd951200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fd94c111850 con 0x7fd94c10bcb0 2024-04-30T08:35:10.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.889+0000 7fd951200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fd94c1116d0 con 0x7fd94c105d70 2024-04-30T08:35:10.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.889+0000 7fd951200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fd94c1119d0 con 0x7fd94c106fb0 2024-04-30T08:35:10.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.889+0000 7fd949400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd94c106fb0 0x7fd94c19c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:10.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.889+0000 7fd949400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd94c106fb0 0x7fd94c19c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:44306/0 (socket says 172.21.15.32:44306) 2024-04-30T08:35:10.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.889+0000 7fd949400640 1 -- 172.21.15.32:0/2100323946 learned_addr learned my addr 172.21.15.32:0/2100323946 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:10.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.889+0000 7fd949e00640 1 --2- 172.21.15.32:0/2100323946 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd94c105d70 0x7fd94c19c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:10.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.890+0000 7fd94a800640 1 --2- 172.21.15.32:0/2100323946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd94c10bcb0 0x7fd94c1a3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:10.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.890+0000 7fd949400640 1 -- 172.21.15.32:0/2100323946 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd94c105d70 msgr2=0x7fd94c19c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:10.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.890+0000 7fd949400640 1 --2- 172.21.15.32:0/2100323946 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd94c105d70 0x7fd94c19c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:10.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.890+0000 7fd949400640 1 -- 172.21.15.32:0/2100323946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd94c10bcb0 msgr2=0x7fd94c1a3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:10.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.890+0000 7fd949400640 1 --2- 172.21.15.32:0/2100323946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd94c10bcb0 0x7fd94c1a3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:10.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.890+0000 7fd949400640 1 -- 172.21.15.32:0/2100323946 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd94c1a3fc0 con 0x7fd94c106fb0 2024-04-30T08:35:10.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.890+0000 7fd94a800640 1 --2- 172.21.15.32:0/2100323946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd94c10bcb0 0x7fd94c1a3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:35:10.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.890+0000 7fd949400640 1 --2- 172.21.15.32:0/2100323946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd94c106fb0 0x7fd94c19c9c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd93800e9f0 tx=0x7fd93800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:10.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.891+0000 7fd942a00640 1 -- 172.21.15.32:0/2100323946 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd93800cd80 con 0x7fd94c106fb0 2024-04-30T08:35:10.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.891+0000 7fd951200640 1 -- 172.21.15.32:0/2100323946 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd94c1a42b0 con 0x7fd94c106fb0 2024-04-30T08:35:10.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.891+0000 7fd942a00640 1 -- 172.21.15.32:0/2100323946 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fd9380108e0 con 0x7fd94c106fb0 2024-04-30T08:35:10.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.891+0000 7fd942a00640 1 -- 172.21.15.32:0/2100323946 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd938018980 con 0x7fd94c106fb0 2024-04-30T08:35:10.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.891+0000 7fd951200640 1 -- 172.21.15.32:0/2100323946 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd94c1a4850 con 0x7fd94c106fb0 2024-04-30T08:35:10.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.892+0000 7fd951200640 1 -- 172.21.15.32:0/2100323946 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd94c10aac0 con 0x7fd94c106fb0 2024-04-30T08:35:10.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.895+0000 7fd942a00640 1 -- 172.21.15.32:0/2100323946 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7fd938010430 con 0x7fd94c106fb0 2024-04-30T08:35:10.898 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.897+0000 7fd942a00640 1 --2- 172.21.15.32:0/2100323946 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd9300afb80 0x7fd9300b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:10.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.897+0000 7fd942a00640 1 -- 172.21.15.32:0/2100323946 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fd9380df5a0 con 0x7fd94c106fb0 2024-04-30T08:35:10.899 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.898+0000 7fd949e00640 1 --2- 172.21.15.32:0/2100323946 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd9300afb80 0x7fd9300b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:10.900 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.899+0000 7fd949e00640 1 --2- 172.21.15.32:0/2100323946 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd9300afb80 0x7fd9300b2090 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd93c005ab0 tx=0x7fd93c031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:10.902 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:10.901+0000 7fd942a00640 1 -- 172.21.15.32:0/2100323946 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd9380a7c10 con 0x7fd94c106fb0 2024-04-30T08:35:11.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.122+0000 7fd951200640 1 -- 172.21.15.32:0/2100323946 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd94c02dbd0 con 0x7fd94c106fb0 2024-04-30T08:35:11.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.124+0000 7fd942a00640 1 -- 172.21.15.32:0/2100323946 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) ==== 104+0+11 (secure 0 0 0) 0x7fd9380a7510 con 0x7fd94c106fb0 2024-04-30T08:35:11.125 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-30T08:35:11.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.126+0000 7fd951200640 1 -- 172.21.15.32:0/2100323946 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd9300afb80 msgr2=0x7fd9300b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:11.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.126+0000 7fd951200640 1 --2- 172.21.15.32:0/2100323946 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd9300afb80 0x7fd9300b2090 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fd93c005ab0 tx=0x7fd93c031400 comp rx=0 tx=0).stop 2024-04-30T08:35:11.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.126+0000 7fd951200640 1 -- 172.21.15.32:0/2100323946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd94c106fb0 msgr2=0x7fd94c19c9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:11.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.126+0000 7fd951200640 1 --2- 172.21.15.32:0/2100323946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd94c106fb0 0x7fd94c19c9c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fd93800e9f0 tx=0x7fd93800eeb0 comp rx=0 tx=0).stop 2024-04-30T08:35:11.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.126+0000 7fd951200640 1 -- 172.21.15.32:0/2100323946 shutdown_connections 2024-04-30T08:35:11.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.126+0000 7fd951200640 1 --2- 172.21.15.32:0/2100323946 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fd9300afb80 0x7fd9300b2090 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:11.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.126+0000 7fd951200640 1 --2- 172.21.15.32:0/2100323946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fd94c10bcb0 0x7fd94c1a3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:11.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.126+0000 7fd951200640 1 --2- 172.21.15.32:0/2100323946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd94c106fb0 0x7fd94c19c9c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:11.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.126+0000 7fd951200640 1 --2- 172.21.15.32:0/2100323946 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fd94c105d70 0x7fd94c19c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:11.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.126+0000 7fd951200640 1 -- 172.21.15.32:0/2100323946 >> 172.21.15.32:0/2100323946 conn(0x7fd94c100f80 msgr2=0x7fd94c107c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:11.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.127+0000 7fd951200640 1 -- 172.21.15.32:0/2100323946 shutdown_connections 2024-04-30T08:35:11.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.127+0000 7fd951200640 1 -- 172.21.15.32:0/2100323946 wait complete. 2024-04-30T08:35:11.155 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-30T08:35:11.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:11 smithi032 ceph-mon[28331]: pgmap v425: 129 pgs: 129 active+clean; 651 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-04-30T08:35:11.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:11 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3313251217' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:35:11.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.443+0000 7f6567e00640 1 -- 172.21.15.32:0/1446607521 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6568106df0 msgr2=0x7f6568107200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:11.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.443+0000 7f6567e00640 1 --2- 172.21.15.32:0/1446607521 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6568106df0 0x7f6568107200 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6554009900 tx=0x7f655402a410 comp rx=0 tx=0).stop 2024-04-30T08:35:11.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.444+0000 7f6567e00640 1 -- 172.21.15.32:0/1446607521 shutdown_connections 2024-04-30T08:35:11.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.444+0000 7f6567e00640 1 --2- 172.21.15.32:0/1446607521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f656810baf0 0x7f656810df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:11.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.444+0000 7f6567e00640 1 --2- 172.21.15.32:0/1446607521 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6568106df0 0x7f6568107200 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:11.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.444+0000 7f6567e00640 1 --2- 172.21.15.32:0/1446607521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6568105bb0 0x7f65681068b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:11.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.444+0000 7f6567e00640 1 -- 172.21.15.32:0/1446607521 >> 172.21.15.32:0/1446607521 conn(0x7f6568100da0 msgr2=0x7f6568103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:11.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.444+0000 7f6567e00640 1 -- 172.21.15.32:0/1446607521 shutdown_connections 2024-04-30T08:35:11.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.444+0000 7f6567e00640 1 -- 172.21.15.32:0/1446607521 wait complete. 2024-04-30T08:35:11.446 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.445+0000 7f6567e00640 1 Processor -- start 2024-04-30T08:35:11.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.445+0000 7f6567e00640 1 -- start start 2024-04-30T08:35:11.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.446+0000 7f6567e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6568105bb0 0x7f656807a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:11.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.446+0000 7f6567e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6568106df0 0x7f656807add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:11.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.446+0000 7f6567e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f656810baf0 0x7f65680773e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:11.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.446+0000 7f6567e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f65681116b0 con 0x7f656810baf0 2024-04-30T08:35:11.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.446+0000 7f6567e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6568111530 con 0x7f6568105bb0 2024-04-30T08:35:11.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.446+0000 7f6567e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6568111830 con 0x7f6568106df0 2024-04-30T08:35:11.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.446+0000 7f6566a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6568105bb0 0x7f656807a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:11.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.446+0000 7f6566a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6568105bb0 0x7f656807a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:57200/0 (socket says 172.21.15.32:57200) 2024-04-30T08:35:11.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.446+0000 7f6567400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f656810baf0 0x7f65680773e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:11.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.446+0000 7f6566a00640 1 -- 172.21.15.32:0/3186334562 learned_addr learned my addr 172.21.15.32:0/3186334562 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:11.448 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.446+0000 7f6566000640 1 --2- 172.21.15.32:0/3186334562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6568106df0 0x7f656807add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:11.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.447+0000 7f6566a00640 1 -- 172.21.15.32:0/3186334562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6568106df0 msgr2=0x7f656807add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:11.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.447+0000 7f6566a00640 1 --2- 172.21.15.32:0/3186334562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6568106df0 0x7f656807add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:11.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.447+0000 7f6566a00640 1 -- 172.21.15.32:0/3186334562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f656810baf0 msgr2=0x7f65680773e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:11.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.447+0000 7f6566a00640 1 --2- 172.21.15.32:0/3186334562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f656810baf0 0x7f65680773e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:11.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.447+0000 7f6566a00640 1 -- 172.21.15.32:0/3186334562 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6568077c00 con 0x7f6568105bb0 2024-04-30T08:35:11.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.447+0000 7f6566a00640 1 --2- 172.21.15.32:0/3186334562 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6568105bb0 0x7f656807a890 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f655800e9f0 tx=0x7f655800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:11.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.448+0000 7f655f400640 1 -- 172.21.15.32:0/3186334562 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655800ccc0 con 0x7f6568105bb0 2024-04-30T08:35:11.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.448+0000 7f6567e00640 1 -- 172.21.15.32:0/3186334562 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6568077ef0 con 0x7f6568105bb0 2024-04-30T08:35:11.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.448+0000 7f655f400640 1 -- 172.21.15.32:0/3186334562 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f65580035f0 con 0x7f6568105bb0 2024-04-30T08:35:11.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.448+0000 7f655f400640 1 -- 172.21.15.32:0/3186334562 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f655800ce60 con 0x7f6568105bb0 2024-04-30T08:35:11.450 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.448+0000 7f6567e00640 1 -- 172.21.15.32:0/3186334562 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65681a8a30 con 0x7f6568105bb0 2024-04-30T08:35:11.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.449+0000 7f6567e00640 1 -- 172.21.15.32:0/3186334562 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f656810a900 con 0x7f6568105bb0 2024-04-30T08:35:11.451 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:11 smithi189 ceph-mon[33038]: pgmap v425: 129 pgs: 129 active+clean; 651 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-04-30T08:35:11.451 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:11 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3313251217' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:35:11.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.451+0000 7f655f400640 1 -- 172.21.15.32:0/3186334562 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f6558003aa0 con 0x7f6568105bb0 2024-04-30T08:35:11.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.453+0000 7f655f400640 1 --2- 172.21.15.32:0/3186334562 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f654c0afb70 0x7f654c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:11.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.453+0000 7f655f400640 1 -- 172.21.15.32:0/3186334562 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f65580df5d0 con 0x7f6568105bb0 2024-04-30T08:35:11.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.453+0000 7f6566000640 1 --2- 172.21.15.32:0/3186334562 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f654c0afb70 0x7f654c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:11.455 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.454+0000 7f6566000640 1 --2- 172.21.15.32:0/3186334562 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f654c0afb70 0x7f654c0b2080 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f6554009230 tx=0x7f6554031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:11.456 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.455+0000 7f655f400640 1 -- 172.21.15.32:0/3186334562 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f65580a7c80 con 0x7f6568105bb0 2024-04-30T08:35:11.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:11 smithi071 ceph-mon[33385]: pgmap v425: 129 pgs: 129 active+clean; 651 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.2 KiB/s wr, 13 op/s 2024-04-30T08:35:11.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:11 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3313251217' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:35:11.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.663+0000 7f6567e00640 1 -- 172.21.15.32:0/3186334562 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6568063820 con 0x7f6568105bb0 2024-04-30T08:35:11.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.664+0000 7f655f400640 1 -- 172.21.15.32:0/3186334562 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) ==== 100+0+11 (secure 0 0 0) 0x7f65580a7580 con 0x7f6568105bb0 2024-04-30T08:35:11.666 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-04-30T08:35:11.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.667+0000 7f6567e00640 1 -- 172.21.15.32:0/3186334562 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f654c0afb70 msgr2=0x7f654c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:11.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.667+0000 7f6567e00640 1 --2- 172.21.15.32:0/3186334562 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f654c0afb70 0x7f654c0b2080 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f6554009230 tx=0x7f6554031400 comp rx=0 tx=0).stop 2024-04-30T08:35:11.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.667+0000 7f6567e00640 1 -- 172.21.15.32:0/3186334562 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6568105bb0 msgr2=0x7f656807a890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:11.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.667+0000 7f6567e00640 1 --2- 172.21.15.32:0/3186334562 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6568105bb0 0x7f656807a890 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f655800e9f0 tx=0x7f655800eeb0 comp rx=0 tx=0).stop 2024-04-30T08:35:11.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.667+0000 7f6567e00640 1 -- 172.21.15.32:0/3186334562 shutdown_connections 2024-04-30T08:35:11.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.667+0000 7f6567e00640 1 --2- 172.21.15.32:0/3186334562 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f654c0afb70 0x7f654c0b2080 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:11.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.667+0000 7f6567e00640 1 --2- 172.21.15.32:0/3186334562 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f656810baf0 0x7f65680773e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:11.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.667+0000 7f6567e00640 1 --2- 172.21.15.32:0/3186334562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6568106df0 0x7f656807add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:11.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.667+0000 7f6567e00640 1 --2- 172.21.15.32:0/3186334562 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6568105bb0 0x7f656807a890 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:11.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.667+0000 7f6567e00640 1 -- 172.21.15.32:0/3186334562 >> 172.21.15.32:0/3186334562 conn(0x7f6568100da0 msgr2=0x7f6568107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:11.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.667+0000 7f6567e00640 1 -- 172.21.15.32:0/3186334562 shutdown_connections 2024-04-30T08:35:11.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:11.667+0000 7f6567e00640 1 -- 172.21.15.32:0/3186334562 wait complete. 2024-04-30T08:35:11.700 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-30T08:35:12.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.039+0000 7f3a8e600640 1 -- 172.21.15.32:0/3574952335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3a88106df0 msgr2=0x7f3a88107200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:12.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.039+0000 7f3a8e600640 1 --2- 172.21.15.32:0/3574952335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3a88106df0 0x7f3a88107200 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3a70009900 tx=0x7f3a7002a410 comp rx=0 tx=0).stop 2024-04-30T08:35:12.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.039+0000 7f3a8e600640 1 -- 172.21.15.32:0/3574952335 shutdown_connections 2024-04-30T08:35:12.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.039+0000 7f3a8e600640 1 --2- 172.21.15.32:0/3574952335 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a8810baf0 0x7f3a8810df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:12.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.039+0000 7f3a8e600640 1 --2- 172.21.15.32:0/3574952335 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3a88106df0 0x7f3a88107200 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:12.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.039+0000 7f3a8e600640 1 --2- 172.21.15.32:0/3574952335 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a88105bb0 0x7f3a881068b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:12.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.039+0000 7f3a8e600640 1 -- 172.21.15.32:0/3574952335 >> 172.21.15.32:0/3574952335 conn(0x7f3a88100da0 msgr2=0x7f3a88103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:12.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.040+0000 7f3a8e600640 1 -- 172.21.15.32:0/3574952335 shutdown_connections 2024-04-30T08:35:12.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.040+0000 7f3a8e600640 1 -- 172.21.15.32:0/3574952335 wait complete. 2024-04-30T08:35:12.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.041+0000 7f3a8e600640 1 Processor -- start 2024-04-30T08:35:12.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.041+0000 7f3a8e600640 1 -- start start 2024-04-30T08:35:12.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.042+0000 7f3a8e600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a88105bb0 0x7f3a8819c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:12.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.042+0000 7f3a8e600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a88106df0 0x7f3a8819c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:12.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.042+0000 7f3a8e600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3a8810baf0 0x7f3a881a3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:12.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.042+0000 7f3a8e600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3a88111720 con 0x7f3a88105bb0 2024-04-30T08:35:12.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.042+0000 7f3a8e600640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f3a881115a0 con 0x7f3a8810baf0 2024-04-30T08:35:12.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.042+0000 7f3a8e600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3a881118a0 con 0x7f3a88106df0 2024-04-30T08:35:12.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.042+0000 7f3a8d200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a88105bb0 0x7f3a8819c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:12.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.042+0000 7f3a8d200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a88105bb0 0x7f3a8819c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53584/0 (socket says 172.21.15.32:53584) 2024-04-30T08:35:12.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.042+0000 7f3a8d200640 1 -- 172.21.15.32:0/2595876329 learned_addr learned my addr 172.21.15.32:0/2595876329 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:12.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.042+0000 7f3a8dc00640 1 --2- 172.21.15.32:0/2595876329 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3a8810baf0 0x7f3a881a3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:12.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.042+0000 7f3a7fe00640 1 --2- 172.21.15.32:0/2595876329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a88106df0 0x7f3a8819c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:12.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.043+0000 7f3a8dc00640 1 -- 172.21.15.32:0/2595876329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a88106df0 msgr2=0x7f3a8819c7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:12.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.043+0000 7f3a8dc00640 1 --2- 172.21.15.32:0/2595876329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a88106df0 0x7f3a8819c7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:12.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.043+0000 7f3a8dc00640 1 -- 172.21.15.32:0/2595876329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a88105bb0 msgr2=0x7f3a8819c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:12.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.043+0000 7f3a8dc00640 1 --2- 172.21.15.32:0/2595876329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a88105bb0 0x7f3a8819c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:12.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.043+0000 7f3a8dc00640 1 -- 172.21.15.32:0/2595876329 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a881a3db0 con 0x7f3a8810baf0 2024-04-30T08:35:12.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.043+0000 7f3a8d200640 1 --2- 172.21.15.32:0/2595876329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a88105bb0 0x7f3a8819c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:12.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.043+0000 7f3a8dc00640 1 --2- 172.21.15.32:0/2595876329 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3a8810baf0 0x7f3a881a3740 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3a80009fd0 tx=0x7f3a8000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:12.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.043+0000 7f3a86000640 1 -- 172.21.15.32:0/2595876329 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a8000ce20 con 0x7f3a8810baf0 2024-04-30T08:35:12.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.043+0000 7f3a86000640 1 -- 172.21.15.32:0/2595876329 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3a80005020 con 0x7f3a8810baf0 2024-04-30T08:35:12.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.044+0000 7f3a8e600640 1 -- 172.21.15.32:0/2595876329 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a881a40a0 con 0x7f3a8810baf0 2024-04-30T08:35:12.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.044+0000 7f3a86000640 1 -- 172.21.15.32:0/2595876329 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a80010690 con 0x7f3a8810baf0 2024-04-30T08:35:12.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.044+0000 7f3a8e600640 1 -- 172.21.15.32:0/2595876329 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a881a45c0 con 0x7f3a8810baf0 2024-04-30T08:35:12.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.045+0000 7f3a8e600640 1 -- 172.21.15.32:0/2595876329 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a8810a900 con 0x7f3a8810baf0 2024-04-30T08:35:12.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.047+0000 7f3a86000640 1 -- 172.21.15.32:0/2595876329 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f3a800054d0 con 0x7f3a8810baf0 2024-04-30T08:35:12.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.049+0000 7f3a86000640 1 --2- 172.21.15.32:0/2595876329 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3a6c0afbf0 0x7f3a6c0b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:12.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.049+0000 7f3a86000640 1 -- 172.21.15.32:0/2595876329 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f3a800df7e0 con 0x7f3a8810baf0 2024-04-30T08:35:12.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.049+0000 7f3a8d200640 1 --2- 172.21.15.32:0/2595876329 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3a6c0afbf0 0x7f3a6c0b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:12.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.050+0000 7f3a8d200640 1 --2- 172.21.15.32:0/2595876329 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3a6c0afbf0 0x7f3a6c0b2100 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f3a78009fd0 tx=0x7f3a78005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:12.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.051+0000 7f3a86000640 1 -- 172.21.15.32:0/2595876329 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3a800a7e50 con 0x7f3a8810baf0 2024-04-30T08:35:12.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.261+0000 7f3a8e600640 1 -- 172.21.15.32:0/2595876329 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f3a88063820 con 0x7f3a8810baf0 2024-04-30T08:35:12.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.262+0000 7f3a86000640 1 -- 172.21.15.32:0/2595876329 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v85) ==== 103+0+11 (secure 0 0 0) 0x7f3a800a7750 con 0x7f3a8810baf0 2024-04-30T08:35:12.264 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-30T08:35:12.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.265+0000 7f3a8e600640 1 -- 172.21.15.32:0/2595876329 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3a6c0afbf0 msgr2=0x7f3a6c0b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:12.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.265+0000 7f3a8e600640 1 --2- 172.21.15.32:0/2595876329 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3a6c0afbf0 0x7f3a6c0b2100 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f3a78009fd0 tx=0x7f3a78005f90 comp rx=0 tx=0).stop 2024-04-30T08:35:12.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.265+0000 7f3a8e600640 1 -- 172.21.15.32:0/2595876329 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3a8810baf0 msgr2=0x7f3a881a3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:12.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.265+0000 7f3a8e600640 1 --2- 172.21.15.32:0/2595876329 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3a8810baf0 0x7f3a881a3740 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3a80009fd0 tx=0x7f3a8000ef10 comp rx=0 tx=0).stop 2024-04-30T08:35:12.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.265+0000 7f3a8e600640 1 -- 172.21.15.32:0/2595876329 shutdown_connections 2024-04-30T08:35:12.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.265+0000 7f3a8e600640 1 --2- 172.21.15.32:0/2595876329 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3a6c0afbf0 0x7f3a6c0b2100 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:12.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.265+0000 7f3a8e600640 1 --2- 172.21.15.32:0/2595876329 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3a8810baf0 0x7f3a881a3740 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:12.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.265+0000 7f3a8e600640 1 --2- 172.21.15.32:0/2595876329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a88106df0 0x7f3a8819c7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:12.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.265+0000 7f3a8e600640 1 --2- 172.21.15.32:0/2595876329 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3a88105bb0 0x7f3a8819c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:12.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.265+0000 7f3a8e600640 1 -- 172.21.15.32:0/2595876329 >> 172.21.15.32:0/2595876329 conn(0x7f3a88100da0 msgr2=0x7f3a88107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:12.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.265+0000 7f3a8e600640 1 -- 172.21.15.32:0/2595876329 shutdown_connections 2024-04-30T08:35:12.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.265+0000 7f3a8e600640 1 -- 172.21.15.32:0/2595876329 wait complete. 2024-04-30T08:35:12.294 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-04-30T08:35:12.294 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-30T08:35:12.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:12 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2100323946' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-30T08:35:12.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:12 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3186334562' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-30T08:35:12.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:12 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2100323946' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-30T08:35:12.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:12 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3186334562' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-30T08:35:12.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:12 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2100323946' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-30T08:35:12.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:12 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3186334562' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-30T08:35:12.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.567+0000 7fb63f000640 1 -- 172.21.15.32:0/265912705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb638103990 msgr2=0x7fb638104690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:12.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.567+0000 7fb63f000640 1 --2- 172.21.15.32:0/265912705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb638103990 0x7fb638104690 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fb62c009900 tx=0x7fb62c02a410 comp rx=0 tx=0).stop 2024-04-30T08:35:12.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.567+0000 7fb63f000640 1 -- 172.21.15.32:0/265912705 shutdown_connections 2024-04-30T08:35:12.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.567+0000 7fb63f000640 1 --2- 172.21.15.32:0/265912705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6381098d0 0x7fb63810bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:12.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.567+0000 7fb63f000640 1 --2- 172.21.15.32:0/265912705 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb638104bd0 0x7fb638104fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:12.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.567+0000 7fb63f000640 1 --2- 172.21.15.32:0/265912705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb638103990 0x7fb638104690 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:12.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.567+0000 7fb63f000640 1 -- 172.21.15.32:0/265912705 >> 172.21.15.32:0/265912705 conn(0x7fb6380feba0 msgr2=0x7fb638101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:12.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.567+0000 7fb63f000640 1 -- 172.21.15.32:0/265912705 shutdown_connections 2024-04-30T08:35:12.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.567+0000 7fb63f000640 1 -- 172.21.15.32:0/265912705 wait complete. 2024-04-30T08:35:12.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.569+0000 7fb63f000640 1 Processor -- start 2024-04-30T08:35:12.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.569+0000 7fb63f000640 1 -- start start 2024-04-30T08:35:12.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.570+0000 7fb63f000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb638103990 0x7fb63819a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:12.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.570+0000 7fb63f000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb638104bd0 0x7fb63819a600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:12.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.570+0000 7fb63f000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6381098d0 0x7fb6381a1590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:12.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.570+0000 7fb63f000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb63810f470 con 0x7fb6381098d0 2024-04-30T08:35:12.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.570+0000 7fb63f000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb63810f2f0 con 0x7fb638104bd0 2024-04-30T08:35:12.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.570+0000 7fb63f000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb63810f5f0 con 0x7fb638103990 2024-04-30T08:35:12.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.570+0000 7fb63e600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6381098d0 0x7fb6381a1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:12.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.570+0000 7fb63dc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb638103990 0x7fb63819a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:12.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.570+0000 7fb63e600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6381098d0 0x7fb6381a1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53596/0 (socket says 172.21.15.32:53596) 2024-04-30T08:35:12.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.570+0000 7fb63e600640 1 -- 172.21.15.32:0/3347504630 learned_addr learned my addr 172.21.15.32:0/3347504630 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:12.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.570+0000 7fb63d200640 1 --2- 172.21.15.32:0/3347504630 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb638104bd0 0x7fb63819a600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:12.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.570+0000 7fb63e600640 1 -- 172.21.15.32:0/3347504630 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb638103990 msgr2=0x7fb63819a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:12.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.570+0000 7fb63e600640 1 --2- 172.21.15.32:0/3347504630 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb638103990 0x7fb63819a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:12.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.570+0000 7fb63e600640 1 -- 172.21.15.32:0/3347504630 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb638104bd0 msgr2=0x7fb63819a600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:12.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.570+0000 7fb63e600640 1 --2- 172.21.15.32:0/3347504630 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb638104bd0 0x7fb63819a600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:12.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.570+0000 7fb63e600640 1 -- 172.21.15.32:0/3347504630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6381a1c00 con 0x7fb6381098d0 2024-04-30T08:35:12.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.571+0000 7fb63e600640 1 --2- 172.21.15.32:0/3347504630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6381098d0 0x7fb6381a1590 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb63000eaa0 tx=0x7fb63000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:12.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.571+0000 7fb636000640 1 -- 172.21.15.32:0/3347504630 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb63000cdb0 con 0x7fb6381098d0 2024-04-30T08:35:12.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.571+0000 7fb636000640 1 -- 172.21.15.32:0/3347504630 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb630005020 con 0x7fb6381098d0 2024-04-30T08:35:12.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.571+0000 7fb63f000640 1 -- 172.21.15.32:0/3347504630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6381a1ef0 con 0x7fb6381098d0 2024-04-30T08:35:12.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.571+0000 7fb636000640 1 -- 172.21.15.32:0/3347504630 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6300116d0 con 0x7fb6381098d0 2024-04-30T08:35:12.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.571+0000 7fb63f000640 1 -- 172.21.15.32:0/3347504630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6381a2300 con 0x7fb6381098d0 2024-04-30T08:35:12.576 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.572+0000 7fb63f000640 1 -- 172.21.15.32:0/3347504630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6381086e0 con 0x7fb6381098d0 2024-04-30T08:35:12.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.575+0000 7fb636000640 1 -- 172.21.15.32:0/3347504630 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7fb6300051d0 con 0x7fb6381098d0 2024-04-30T08:35:12.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.577+0000 7fb636000640 1 --2- 172.21.15.32:0/3347504630 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb6200afb70 0x7fb6200b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:12.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.578+0000 7fb636000640 1 -- 172.21.15.32:0/3347504630 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fb6300df620 con 0x7fb6381098d0 2024-04-30T08:35:12.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.578+0000 7fb63dc00640 1 --2- 172.21.15.32:0/3347504630 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb6200afb70 0x7fb6200b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:12.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.578+0000 7fb63dc00640 1 --2- 172.21.15.32:0/3347504630 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb6200afb70 0x7fb6200b2080 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb62c005ab0 tx=0x7fb62c031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:12.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.579+0000 7fb636000640 1 -- 172.21.15.32:0/3347504630 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb630016030 con 0x7fb6381098d0 2024-04-30T08:35:12.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.828+0000 7fb63f000640 1 -- 172.21.15.32:0/3347504630 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb63802de20 con 0x7fb6381098d0 2024-04-30T08:35:12.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.830+0000 7fb636000640 1 -- 172.21.15.32:0/3347504630 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11927 (secure 0 0 0) 0x7fb6300a7640 con 0x7fb6381098d0 2024-04-30T08:35:12.833 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:35:12.833 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2235842676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2235842676},{"type":"v1","addr":"172.21.15.32:6835","nonce":2235842676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/3577009355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":3577009355},{"type":"v1","addr":"172.21.15.32:6841","nonce":3577009355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-30T08:31:52.236668+0000","modified":"2024-04-30T08:35:05.098418+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24541,"mds_3":24533,"mds_4":24517},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/463203038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":463203038},{"type":"v1","addr":"172.21.15.189:6833","nonce":463203038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6833/3635143591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3635143591},{"type":"v1","addr":"172.21.15.71:6833","nonce":3635143591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6837/2569449994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2569449994},{"type":"v1","addr":"172.21.15.32:6837","nonce":2569449994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/278686565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":278686565},{"type":"v1","addr":"172.21.15.189:6835","nonce":278686565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6835/1558303056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1558303056},{"type":"v1","addr":"172.21.15.71:6835","nonce":1558303056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.189:6837/686520873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":686520873},{"type":"v1","addr":"172.21.15.189:6837","nonce":686520873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.71:6837/2324039932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2324039932},{"type":"v1","addr":"172.21.15.71:6837","nonce":2324039932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6839/352970107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":352970107},{"type":"v1","addr":"172.21.15.189:6839","nonce":352970107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.32:6839/4160386524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4160386524},{"type":"v1","addr":"172.21.15.32:6839","nonce":4160386524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.71:6839/3517409993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3517409993},{"type":"v1","addr":"172.21.15.71:6839","nonce":3517409993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24517,24533,24560,24590,24541]},"id":1}]} 2024-04-30T08:35:12.833 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2024-04-30T08:35:12.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.833+0000 7fb63f000640 1 -- 172.21.15.32:0/3347504630 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb6200afb70 msgr2=0x7fb6200b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:12.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.833+0000 7fb63f000640 1 --2- 172.21.15.32:0/3347504630 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb6200afb70 0x7fb6200b2080 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fb62c005ab0 tx=0x7fb62c031400 comp rx=0 tx=0).stop 2024-04-30T08:35:12.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.833+0000 7fb63f000640 1 -- 172.21.15.32:0/3347504630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6381098d0 msgr2=0x7fb6381a1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:12.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.833+0000 7fb63f000640 1 --2- 172.21.15.32:0/3347504630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6381098d0 0x7fb6381a1590 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fb63000eaa0 tx=0x7fb63000ef60 comp rx=0 tx=0).stop 2024-04-30T08:35:12.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.833+0000 7fb63f000640 1 -- 172.21.15.32:0/3347504630 shutdown_connections 2024-04-30T08:35:12.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.833+0000 7fb63f000640 1 --2- 172.21.15.32:0/3347504630 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb6200afb70 0x7fb6200b2080 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:12.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.833+0000 7fb63f000640 1 --2- 172.21.15.32:0/3347504630 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb6381098d0 0x7fb6381a1590 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:12.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.833+0000 7fb63f000640 1 --2- 172.21.15.32:0/3347504630 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb638104bd0 0x7fb63819a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:12.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.833+0000 7fb63f000640 1 --2- 172.21.15.32:0/3347504630 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb638103990 0x7fb63819a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:12.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.833+0000 7fb63f000640 1 -- 172.21.15.32:0/3347504630 >> 172.21.15.32:0/3347504630 conn(0x7fb6380feba0 msgr2=0x7fb6381058b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:12.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.833+0000 7fb63f000640 1 -- 172.21.15.32:0/3347504630 shutdown_connections 2024-04-30T08:35:12.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:12.834+0000 7fb63f000640 1 -- 172.21.15.32:0/3347504630 wait complete. 2024-04-30T08:35:12.860 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-04-30T08:35:12.860 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-04-30T08:35:12.860 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-30T08:35:13.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.144+0000 7f8f8e800640 1 -- 172.21.15.32:0/4020554206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f88106df0 msgr2=0x7f8f88107200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:13.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.144+0000 7f8f8e800640 1 --2- 172.21.15.32:0/4020554206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f88106df0 0x7f8f88107200 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f8f74009900 tx=0x7f8f7402a3f0 comp rx=0 tx=0).stop 2024-04-30T08:35:13.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.144+0000 7f8f8e800640 1 -- 172.21.15.32:0/4020554206 shutdown_connections 2024-04-30T08:35:13.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.144+0000 7f8f8e800640 1 --2- 172.21.15.32:0/4020554206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f8810baf0 0x7f8f8810df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:13.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.144+0000 7f8f8e800640 1 --2- 172.21.15.32:0/4020554206 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f88106df0 0x7f8f88107200 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:13.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.144+0000 7f8f8e800640 1 --2- 172.21.15.32:0/4020554206 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f88105bb0 0x7f8f881068b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:13.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.144+0000 7f8f8e800640 1 -- 172.21.15.32:0/4020554206 >> 172.21.15.32:0/4020554206 conn(0x7f8f88100da0 msgr2=0x7f8f88103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:13.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.145+0000 7f8f8e800640 1 -- 172.21.15.32:0/4020554206 shutdown_connections 2024-04-30T08:35:13.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.145+0000 7f8f8e800640 1 -- 172.21.15.32:0/4020554206 wait complete. 2024-04-30T08:35:13.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.146+0000 7f8f8e800640 1 Processor -- start 2024-04-30T08:35:13.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.146+0000 7f8f8e800640 1 -- start start 2024-04-30T08:35:13.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.146+0000 7f8f8e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f88105bb0 0x7f8f8819c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:13.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.146+0000 7f8f8e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f88106df0 0x7f8f8819c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:13.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.146+0000 7f8f8e800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f8810baf0 0x7f8f881a3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:13.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.146+0000 7f8f8e800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8f88111700 con 0x7f8f88106df0 2024-04-30T08:35:13.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.146+0000 7f8f8e800640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8f88111580 con 0x7f8f8810baf0 2024-04-30T08:35:13.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.146+0000 7f8f8e800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8f88111880 con 0x7f8f88105bb0 2024-04-30T08:35:13.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.146+0000 7f8f8d400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f88105bb0 0x7f8f8819c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:13.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.146+0000 7f8f8ca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f88106df0 0x7f8f8819c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:13.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.146+0000 7f8f8d400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f88105bb0 0x7f8f8819c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:44394/0 (socket says 172.21.15.32:44394) 2024-04-30T08:35:13.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.146+0000 7f8f8d400640 1 -- 172.21.15.32:0/4209948490 learned_addr learned my addr 172.21.15.32:0/4209948490 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:13.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.146+0000 7f8f8de00640 1 --2- 172.21.15.32:0/4209948490 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f8810baf0 0x7f8f881a3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:13.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.147+0000 7f8f8d400640 1 -- 172.21.15.32:0/4209948490 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f8810baf0 msgr2=0x7f8f881a3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:13.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.147+0000 7f8f8d400640 1 --2- 172.21.15.32:0/4209948490 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f8810baf0 0x7f8f881a3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:13.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.147+0000 7f8f8d400640 1 -- 172.21.15.32:0/4209948490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f88106df0 msgr2=0x7f8f8819c790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:13.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.147+0000 7f8f8d400640 1 --2- 172.21.15.32:0/4209948490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f88106df0 0x7f8f8819c790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:13.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.147+0000 7f8f8d400640 1 -- 172.21.15.32:0/4209948490 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f881a3d90 con 0x7f8f88105bb0 2024-04-30T08:35:13.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.147+0000 7f8f8de00640 1 --2- 172.21.15.32:0/4209948490 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f8810baf0 0x7f8f881a3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:13.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.147+0000 7f8f8d400640 1 --2- 172.21.15.32:0/4209948490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f88105bb0 0x7f8f8819c250 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c00e9f0 tx=0x7f8f7c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:13.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.147+0000 7f8f86000640 1 -- 172.21.15.32:0/4209948490 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c00ccc0 con 0x7f8f88105bb0 2024-04-30T08:35:13.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.148+0000 7f8f86000640 1 -- 172.21.15.32:0/4209948490 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8f7c0035f0 con 0x7f8f88105bb0 2024-04-30T08:35:13.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.148+0000 7f8f8e800640 1 -- 172.21.15.32:0/4209948490 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f881a4080 con 0x7f8f88105bb0 2024-04-30T08:35:13.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.148+0000 7f8f86000640 1 -- 172.21.15.32:0/4209948490 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f7c00ce60 con 0x7f8f88105bb0 2024-04-30T08:35:13.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.148+0000 7f8f8e800640 1 -- 172.21.15.32:0/4209948490 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f881a45a0 con 0x7f8f88105bb0 2024-04-30T08:35:13.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.149+0000 7f8f8e800640 1 -- 172.21.15.32:0/4209948490 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f8806a0b0 con 0x7f8f88105bb0 2024-04-30T08:35:13.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.151+0000 7f8f86000640 1 -- 172.21.15.32:0/4209948490 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f8f7c004020 con 0x7f8f88105bb0 2024-04-30T08:35:13.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.152+0000 7f8f86000640 1 --2- 172.21.15.32:0/4209948490 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8f700afb70 0x7f8f700b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:13.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.153+0000 7f8f86000640 1 -- 172.21.15.32:0/4209948490 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f8f7c0df4d0 con 0x7f8f88105bb0 2024-04-30T08:35:13.154 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.153+0000 7f8f8ca00640 1 --2- 172.21.15.32:0/4209948490 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8f700afb70 0x7f8f700b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:13.155 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.153+0000 7f8f8ca00640 1 --2- 172.21.15.32:0/4209948490 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8f700afb70 0x7f8f700b2080 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8f74004770 tx=0x7f8f740314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:13.158 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.157+0000 7f8f86000640 1 -- 172.21.15.32:0/4209948490 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8f7c0a7b40 con 0x7f8f88105bb0 2024-04-30T08:35:13.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.352+0000 7f8f8e800640 1 -- 172.21.15.32:0/4209948490 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f8f88063060 con 0x7f8f700afb70 2024-04-30T08:35:13.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.386+0000 7f8f86000640 1 -- 172.21.15.32:0/4209948490 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340836 (secure 0 0 0) 0x7f8f88063060 con 0x7f8f700afb70 2024-04-30T08:35:13.393 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:35:13.395 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-30T08:35:13.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.393+0000 7f8f8e800640 1 -- 172.21.15.32:0/4209948490 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8f700afb70 msgr2=0x7f8f700b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:13.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.393+0000 7f8f8e800640 1 --2- 172.21.15.32:0/4209948490 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8f700afb70 0x7f8f700b2080 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8f74004770 tx=0x7f8f740314a0 comp rx=0 tx=0).stop 2024-04-30T08:35:13.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.393+0000 7f8f8e800640 1 -- 172.21.15.32:0/4209948490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f88105bb0 msgr2=0x7f8f8819c250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:13.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.393+0000 7f8f8e800640 1 --2- 172.21.15.32:0/4209948490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f88105bb0 0x7f8f8819c250 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f8f7c00e9f0 tx=0x7f8f7c00eeb0 comp rx=0 tx=0).stop 2024-04-30T08:35:13.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.394+0000 7f8f8e800640 1 -- 172.21.15.32:0/4209948490 shutdown_connections 2024-04-30T08:35:13.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.394+0000 7f8f8e800640 1 --2- 172.21.15.32:0/4209948490 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8f700afb70 0x7f8f700b2080 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:13.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.394+0000 7f8f8e800640 1 --2- 172.21.15.32:0/4209948490 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8f8810baf0 0x7f8f881a3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:13.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.394+0000 7f8f8e800640 1 --2- 172.21.15.32:0/4209948490 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8f88106df0 0x7f8f8819c790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:13.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.394+0000 7f8f8e800640 1 --2- 172.21.15.32:0/4209948490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8f88105bb0 0x7f8f8819c250 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:13.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.394+0000 7f8f8e800640 1 -- 172.21.15.32:0/4209948490 >> 172.21.15.32:0/4209948490 conn(0x7f8f88100da0 msgr2=0x7f8f88107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:13.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.394+0000 7f8f8e800640 1 -- 172.21.15.32:0/4209948490 shutdown_connections 2024-04-30T08:35:13.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.394+0000 7f8f8e800640 1 -- 172.21.15.32:0/4209948490 wait complete. 2024-04-30T08:35:13.424 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:13 smithi032 ceph-mon[28331]: pgmap v426: 129 pgs: 129 active+clean; 651 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-04-30T08:35:13.424 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:13 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2595876329' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-30T08:35:13.424 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:13 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3347504630' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:35:13.428 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":426,"stamp":"2024-04-30T08:35:11.794371+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":666516,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1707,"num_read_kb":966,"num_write":391,"num_write_kb":2315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":2951908,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":824,"ondisk_log_size":824,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":373172,"kb_used_data":11540,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124568652,"statfs":{"total":1151940427776,"available":1151558299648,"internally_reserved":0,"allocated":11816960,"data_stored":7720874,"data_compressed":434025,"data_compressed_allocated":589824,"data_compressed_original":2678784,"omap_allocated":19141,"internal_metadata":370193723},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6107,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":115,"num_read_kb":69,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005800"},"pg_stats":[{"pgid":"3.3f","version":"85'8","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.737715+0000","last_change":"2024-04-30T08:31:52.711443+0000","last_active":"2024-04-30T08:33:59.737715+0000","last_peered":"2024-04-30T08:33:59.737715+0000","last_clean":"2024-04-30T08:33:59.737715+0000","last_became_active":"2024-04-30T08:31:52.711029+0000","last_became_peered":"2024-04-30T08:31:52.711029+0000","last_unstale":"2024-04-30T08:33:59.737715+0000","last_undegraded":"2024-04-30T08:33:59.737715+0000","last_fullsized":"2024-04-30T08:33:59.737715+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:51:15.208528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"85'4","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.736806+0000","last_change":"2024-04-30T08:31:52.712324+0000","last_active":"2024-04-30T08:33:59.736806+0000","last_peered":"2024-04-30T08:33:59.736806+0000","last_clean":"2024-04-30T08:33:59.736806+0000","last_became_active":"2024-04-30T08:31:52.711923+0000","last_became_peered":"2024-04-30T08:31:52.711923+0000","last_unstale":"2024-04-30T08:33:59.736806+0000","last_undegraded":"2024-04-30T08:33:59.736806+0000","last_fullsized":"2024-04-30T08:33:59.736806+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:29:55.071459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"85'7","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.735218+0000","last_change":"2024-04-30T08:31:52.712936+0000","last_active":"2024-04-30T08:33:59.735218+0000","last_peered":"2024-04-30T08:33:59.735218+0000","last_clean":"2024-04-30T08:33:59.735218+0000","last_became_active":"2024-04-30T08:31:52.712585+0000","last_became_peered":"2024-04-30T08:31:52.712585+0000","last_unstale":"2024-04-30T08:33:59.735218+0000","last_undegraded":"2024-04-30T08:33:59.735218+0000","last_fullsized":"2024-04-30T08:33:59.735218+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:13:43.745561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"85'8","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.688422+0000","last_change":"2024-04-30T08:31:52.686880+0000","last_active":"2024-04-30T08:33:59.688422+0000","last_peered":"2024-04-30T08:33:59.688422+0000","last_clean":"2024-04-30T08:33:59.688422+0000","last_became_active":"2024-04-30T08:31:52.686506+0000","last_became_peered":"2024-04-30T08:31:52.686506+0000","last_unstale":"2024-04-30T08:33:59.688422+0000","last_undegraded":"2024-04-30T08:33:59.688422+0000","last_fullsized":"2024-04-30T08:33:59.688422+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:16:33.390951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"85'4","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.716502+0000","last_change":"2024-04-30T08:31:52.706014+0000","last_active":"2024-04-30T08:33:59.716502+0000","last_peered":"2024-04-30T08:33:59.716502+0000","last_clean":"2024-04-30T08:33:59.716502+0000","last_became_active":"2024-04-30T08:31:52.705582+0000","last_became_peered":"2024-04-30T08:31:52.705582+0000","last_unstale":"2024-04-30T08:33:59.716502+0000","last_undegraded":"2024-04-30T08:33:59.716502+0000","last_fullsized":"2024-04-30T08:33:59.716502+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:24:49.804592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"85'6","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.717350+0000","last_change":"2024-04-30T08:31:52.690462+0000","last_active":"2024-04-30T08:33:59.717350+0000","last_peered":"2024-04-30T08:33:59.717350+0000","last_clean":"2024-04-30T08:33:59.717350+0000","last_became_active":"2024-04-30T08:31:52.690092+0000","last_became_peered":"2024-04-30T08:31:52.690092+0000","last_unstale":"2024-04-30T08:33:59.717350+0000","last_undegraded":"2024-04-30T08:33:59.717350+0000","last_fullsized":"2024-04-30T08:33:59.717350+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:32:49.769383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"85'5","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.704595+0000","last_change":"2024-04-30T08:31:52.701786+0000","last_active":"2024-04-30T08:33:59.704595+0000","last_peered":"2024-04-30T08:33:59.704595+0000","last_clean":"2024-04-30T08:33:59.704595+0000","last_became_active":"2024-04-30T08:31:52.701402+0000","last_became_peered":"2024-04-30T08:31:52.701402+0000","last_unstale":"2024-04-30T08:33:59.704595+0000","last_undegraded":"2024-04-30T08:33:59.704595+0000","last_fullsized":"2024-04-30T08:33:59.704595+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:58:06.153479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"85'9","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.719088+0000","last_change":"2024-04-30T08:31:52.698037+0000","last_active":"2024-04-30T08:33:59.719088+0000","last_peered":"2024-04-30T08:33:59.719088+0000","last_clean":"2024-04-30T08:33:59.719088+0000","last_became_active":"2024-04-30T08:31:52.697646+0000","last_became_peered":"2024-04-30T08:31:52.697646+0000","last_unstale":"2024-04-30T08:33:59.719088+0000","last_undegraded":"2024-04-30T08:33:59.719088+0000","last_fullsized":"2024-04-30T08:33:59.719088+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:28:59.520091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"85'11","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.692082+0000","last_change":"2024-04-30T08:31:52.680377+0000","last_active":"2024-04-30T08:33:59.692082+0000","last_peered":"2024-04-30T08:33:59.692082+0000","last_clean":"2024-04-30T08:33:59.692082+0000","last_became_active":"2024-04-30T08:31:52.679977+0000","last_became_peered":"2024-04-30T08:31:52.679977+0000","last_unstale":"2024-04-30T08:33:59.692082+0000","last_undegraded":"2024-04-30T08:33:59.692082+0000","last_fullsized":"2024-04-30T08:33:59.692082+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:35:22.488450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"85'16","reported_seq":54,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.729297+0000","last_change":"2024-04-30T08:32:31.573546+0000","last_active":"2024-04-30T08:33:59.729297+0000","last_peered":"2024-04-30T08:33:59.729297+0000","last_clean":"2024-04-30T08:33:59.729297+0000","last_became_active":"2024-04-30T08:32:31.573289+0000","last_became_peered":"2024-04-30T08:32:31.573289+0000","last_unstale":"2024-04-30T08:33:59.729297+0000","last_undegraded":"2024-04-30T08:33:59.729297+0000","last_fullsized":"2024-04-30T08:33:59.729297+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:42:45.862326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"85'6","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.692001+0000","last_change":"2024-04-30T08:31:52.690344+0000","last_active":"2024-04-30T08:33:59.692001+0000","last_peered":"2024-04-30T08:33:59.692001+0000","last_clean":"2024-04-30T08:33:59.692001+0000","last_became_active":"2024-04-30T08:31:52.689857+0000","last_became_peered":"2024-04-30T08:31:52.689857+0000","last_unstale":"2024-04-30T08:33:59.692001+0000","last_undegraded":"2024-04-30T08:33:59.692001+0000","last_fullsized":"2024-04-30T08:33:59.692001+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:45:13.395589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"85'7","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.689051+0000","last_change":"2024-04-30T08:31:52.685151+0000","last_active":"2024-04-30T08:33:59.689051+0000","last_peered":"2024-04-30T08:33:59.689051+0000","last_clean":"2024-04-30T08:33:59.689051+0000","last_became_active":"2024-04-30T08:31:52.684722+0000","last_became_peered":"2024-04-30T08:31:52.684722+0000","last_unstale":"2024-04-30T08:33:59.689051+0000","last_undegraded":"2024-04-30T08:33:59.689051+0000","last_fullsized":"2024-04-30T08:33:59.689051+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:14:00.660478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"85'10","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.723211+0000","last_change":"2024-04-30T08:31:52.714294+0000","last_active":"2024-04-30T08:33:59.723211+0000","last_peered":"2024-04-30T08:33:59.723211+0000","last_clean":"2024-04-30T08:33:59.723211+0000","last_became_active":"2024-04-30T08:31:52.710094+0000","last_became_peered":"2024-04-30T08:31:52.710094+0000","last_unstale":"2024-04-30T08:33:59.723211+0000","last_undegraded":"2024-04-30T08:33:59.723211+0000","last_fullsized":"2024-04-30T08:33:59.723211+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:34:44.203093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"85'7","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.718132+0000","last_change":"2024-04-30T08:31:52.707992+0000","last_active":"2024-04-30T08:33:59.718132+0000","last_peered":"2024-04-30T08:33:59.718132+0000","last_clean":"2024-04-30T08:33:59.718132+0000","last_became_active":"2024-04-30T08:31:52.706456+0000","last_became_peered":"2024-04-30T08:31:52.706456+0000","last_unstale":"2024-04-30T08:33:59.718132+0000","last_undegraded":"2024-04-30T08:33:59.718132+0000","last_fullsized":"2024-04-30T08:33:59.718132+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:45:32.717842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"85'7","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.735414+0000","last_change":"2024-04-30T08:31:52.695605+0000","last_active":"2024-04-30T08:33:59.735414+0000","last_peered":"2024-04-30T08:33:59.735414+0000","last_clean":"2024-04-30T08:33:59.735414+0000","last_became_active":"2024-04-30T08:31:52.692439+0000","last_became_peered":"2024-04-30T08:31:52.692439+0000","last_unstale":"2024-04-30T08:33:59.735414+0000","last_undegraded":"2024-04-30T08:33:59.735414+0000","last_fullsized":"2024-04-30T08:33:59.735414+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:39:38.746567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"85'9","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.740363+0000","last_change":"2024-04-30T08:33:32.802311+0000","last_active":"2024-04-30T08:33:59.740363+0000","last_peered":"2024-04-30T08:33:59.740363+0000","last_clean":"2024-04-30T08:33:59.740363+0000","last_became_active":"2024-04-30T08:33:32.801601+0000","last_became_peered":"2024-04-30T08:33:32.801601+0000","last_unstale":"2024-04-30T08:33:59.740363+0000","last_undegraded":"2024-04-30T08:33:59.740363+0000","last_fullsized":"2024-04-30T08:33:59.740363+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:59:34.877824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0],"acting":[9,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"85'8","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.737797+0000","last_change":"2024-04-30T08:31:52.711439+0000","last_active":"2024-04-30T08:33:59.737797+0000","last_peered":"2024-04-30T08:33:59.737797+0000","last_clean":"2024-04-30T08:33:59.737797+0000","last_became_active":"2024-04-30T08:31:52.711023+0000","last_became_peered":"2024-04-30T08:31:52.711023+0000","last_unstale":"2024-04-30T08:33:59.737797+0000","last_undegraded":"2024-04-30T08:33:59.737797+0000","last_fullsized":"2024-04-30T08:33:59.737797+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:31:00.654348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"85'10","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.721531+0000","last_change":"2024-04-30T08:31:52.708476+0000","last_active":"2024-04-30T08:33:59.721531+0000","last_peered":"2024-04-30T08:33:59.721531+0000","last_clean":"2024-04-30T08:33:59.721531+0000","last_became_active":"2024-04-30T08:31:52.704735+0000","last_became_peered":"2024-04-30T08:31:52.704735+0000","last_unstale":"2024-04-30T08:33:59.721531+0000","last_undegraded":"2024-04-30T08:33:59.721531+0000","last_fullsized":"2024-04-30T08:33:59.721531+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:40:00.011783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"85'7","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.720093+0000","last_change":"2024-04-30T08:31:52.704901+0000","last_active":"2024-04-30T08:33:59.720093+0000","last_peered":"2024-04-30T08:33:59.720093+0000","last_clean":"2024-04-30T08:33:59.720093+0000","last_became_active":"2024-04-30T08:31:52.704636+0000","last_became_peered":"2024-04-30T08:31:52.704636+0000","last_unstale":"2024-04-30T08:33:59.720093+0000","last_undegraded":"2024-04-30T08:33:59.720093+0000","last_fullsized":"2024-04-30T08:33:59.720093+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:08:53.580809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.879602+0000","last_change":"2024-04-30T08:31:57.743342+0000","last_active":"2024-04-30T08:33:31.879602+0000","last_peered":"2024-04-30T08:33:31.879602+0000","last_clean":"2024-04-30T08:33:31.879602+0000","last_became_active":"2024-04-30T08:31:57.742850+0000","last_became_peered":"2024-04-30T08:31:57.742850+0000","last_unstale":"2024-04-30T08:33:31.879602+0000","last_undegraded":"2024-04-30T08:33:31.879602+0000","last_fullsized":"2024-04-30T08:33:31.879602+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:59:10.628822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'14","reported_seq":209,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:35:09.926274+0000","last_change":"2024-04-30T08:31:51.694984+0000","last_active":"2024-04-30T08:35:09.926274+0000","last_peered":"2024-04-30T08:35:09.926274+0000","last_clean":"2024-04-30T08:35:09.926274+0000","last_became_active":"2024-04-30T08:31:51.694514+0000","last_became_peered":"2024-04-30T08:31:51.694514+0000","last_unstale":"2024-04-30T08:35:09.926274+0000","last_undegraded":"2024-04-30T08:35:09.926274+0000","last_fullsized":"2024-04-30T08:35:09.926274+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:17:03.873090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":162,"num_read_kb":162,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"85'5","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.717378+0000","last_change":"2024-04-30T08:31:52.694276+0000","last_active":"2024-04-30T08:33:59.717378+0000","last_peered":"2024-04-30T08:33:59.717378+0000","last_clean":"2024-04-30T08:33:59.717378+0000","last_became_active":"2024-04-30T08:31:52.693839+0000","last_became_peered":"2024-04-30T08:31:52.693839+0000","last_unstale":"2024-04-30T08:33:59.717378+0000","last_undegraded":"2024-04-30T08:33:59.717378+0000","last_fullsized":"2024-04-30T08:33:59.717378+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:29:21.271040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.169742+0000","last_change":"2024-04-30T08:31:57.754828+0000","last_active":"2024-04-30T08:33:32.169742+0000","last_peered":"2024-04-30T08:33:32.169742+0000","last_clean":"2024-04-30T08:33:32.169742+0000","last_became_active":"2024-04-30T08:31:57.754322+0000","last_became_peered":"2024-04-30T08:31:57.754322+0000","last_unstale":"2024-04-30T08:33:32.169742+0000","last_undegraded":"2024-04-30T08:33:32.169742+0000","last_fullsized":"2024-04-30T08:33:32.169742+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:36:02.560134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"81'2","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.277887+0000","last_change":"2024-04-30T08:31:51.707637+0000","last_active":"2024-04-30T08:33:32.277887+0000","last_peered":"2024-04-30T08:33:32.277887+0000","last_clean":"2024-04-30T08:33:32.277887+0000","last_became_active":"2024-04-30T08:31:51.707233+0000","last_became_peered":"2024-04-30T08:31:51.707233+0000","last_unstale":"2024-04-30T08:33:32.277887+0000","last_undegraded":"2024-04-30T08:33:32.277887+0000","last_fullsized":"2024-04-30T08:33:32.277887+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:14:46.041169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"85'6","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.741653+0000","last_change":"2024-04-30T08:31:52.711442+0000","last_active":"2024-04-30T08:33:59.741653+0000","last_peered":"2024-04-30T08:33:59.741653+0000","last_clean":"2024-04-30T08:33:59.741653+0000","last_became_active":"2024-04-30T08:31:52.710942+0000","last_became_peered":"2024-04-30T08:31:52.710942+0000","last_unstale":"2024-04-30T08:33:59.741653+0000","last_undegraded":"2024-04-30T08:33:59.741653+0000","last_fullsized":"2024-04-30T08:33:59.741653+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:26:56.428770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.169895+0000","last_change":"2024-04-30T08:31:57.760535+0000","last_active":"2024-04-30T08:33:32.169895+0000","last_peered":"2024-04-30T08:33:32.169895+0000","last_clean":"2024-04-30T08:33:32.169895+0000","last_became_active":"2024-04-30T08:31:57.760073+0000","last_became_peered":"2024-04-30T08:31:57.760073+0000","last_unstale":"2024-04-30T08:33:32.169895+0000","last_undegraded":"2024-04-30T08:33:32.169895+0000","last_fullsized":"2024-04-30T08:33:32.169895+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:03:51.359474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"81'12","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.887094+0000","last_change":"2024-04-30T08:31:51.678575+0000","last_active":"2024-04-30T08:33:31.887094+0000","last_peered":"2024-04-30T08:33:31.887094+0000","last_clean":"2024-04-30T08:33:31.887094+0000","last_became_active":"2024-04-30T08:31:51.678114+0000","last_became_peered":"2024-04-30T08:31:51.678114+0000","last_unstale":"2024-04-30T08:33:31.887094+0000","last_undegraded":"2024-04-30T08:33:31.887094+0000","last_fullsized":"2024-04-30T08:33:31.887094+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:10:35.517183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"85'9","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.721346+0000","last_change":"2024-04-30T08:31:52.704137+0000","last_active":"2024-04-30T08:33:59.721346+0000","last_peered":"2024-04-30T08:33:59.721346+0000","last_clean":"2024-04-30T08:33:59.721346+0000","last_became_active":"2024-04-30T08:31:52.703695+0000","last_became_peered":"2024-04-30T08:31:52.703695+0000","last_unstale":"2024-04-30T08:33:59.721346+0000","last_undegraded":"2024-04-30T08:33:59.721346+0000","last_fullsized":"2024-04-30T08:33:59.721346+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:31:05.461106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.169818+0000","last_change":"2024-04-30T08:31:57.755237+0000","last_active":"2024-04-30T08:33:32.169818+0000","last_peered":"2024-04-30T08:33:32.169818+0000","last_clean":"2024-04-30T08:33:32.169818+0000","last_became_active":"2024-04-30T08:31:57.754438+0000","last_became_peered":"2024-04-30T08:31:57.754438+0000","last_unstale":"2024-04-30T08:33:32.169818+0000","last_undegraded":"2024-04-30T08:33:32.169818+0000","last_fullsized":"2024-04-30T08:33:32.169818+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:20:59.035110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"81'5","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886405+0000","last_change":"2024-04-30T08:31:51.673793+0000","last_active":"2024-04-30T08:33:31.886405+0000","last_peered":"2024-04-30T08:33:31.886405+0000","last_clean":"2024-04-30T08:33:31.886405+0000","last_became_active":"2024-04-30T08:31:51.672703+0000","last_became_peered":"2024-04-30T08:31:51.672703+0000","last_unstale":"2024-04-30T08:33:31.886405+0000","last_undegraded":"2024-04-30T08:33:31.886405+0000","last_fullsized":"2024-04-30T08:33:31.886405+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:13:18.959542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"85'15","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.686983+0000","last_change":"2024-04-30T08:31:52.694320+0000","last_active":"2024-04-30T08:33:59.686983+0000","last_peered":"2024-04-30T08:33:59.686983+0000","last_clean":"2024-04-30T08:33:59.686983+0000","last_became_active":"2024-04-30T08:31:52.694032+0000","last_became_peered":"2024-04-30T08:31:52.694032+0000","last_unstale":"2024-04-30T08:33:59.686983+0000","last_undegraded":"2024-04-30T08:33:59.686983+0000","last_fullsized":"2024-04-30T08:33:59.686983+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:33:12.347369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.880017+0000","last_change":"2024-04-30T08:31:57.739984+0000","last_active":"2024-04-30T08:33:31.880017+0000","last_peered":"2024-04-30T08:33:31.880017+0000","last_clean":"2024-04-30T08:33:31.880017+0000","last_became_active":"2024-04-30T08:31:57.739477+0000","last_became_peered":"2024-04-30T08:31:57.739477+0000","last_unstale":"2024-04-30T08:33:31.880017+0000","last_undegraded":"2024-04-30T08:33:31.880017+0000","last_fullsized":"2024-04-30T08:33:31.880017+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:10:11.325419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'7","reported_seq":202,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:35:06.810077+0000","last_change":"2024-04-30T08:31:51.678792+0000","last_active":"2024-04-30T08:35:06.810077+0000","last_peered":"2024-04-30T08:35:06.810077+0000","last_clean":"2024-04-30T08:35:06.810077+0000","last_became_active":"2024-04-30T08:31:51.678094+0000","last_became_peered":"2024-04-30T08:31:51.678094+0000","last_unstale":"2024-04-30T08:35:06.810077+0000","last_undegraded":"2024-04-30T08:35:06.810077+0000","last_fullsized":"2024-04-30T08:35:06.810077+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:03:17.526016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":161,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"85'7","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.700627+0000","last_change":"2024-04-30T08:32:31.572984+0000","last_active":"2024-04-30T08:33:59.700627+0000","last_peered":"2024-04-30T08:33:59.700627+0000","last_clean":"2024-04-30T08:33:59.700627+0000","last_became_active":"2024-04-30T08:32:31.572536+0000","last_became_peered":"2024-04-30T08:32:31.572536+0000","last_unstale":"2024-04-30T08:33:59.700627+0000","last_undegraded":"2024-04-30T08:33:59.700627+0000","last_fullsized":"2024-04-30T08:33:59.700627+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:46:15.884555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.879844+0000","last_change":"2024-04-30T08:31:57.765047+0000","last_active":"2024-04-30T08:33:31.879844+0000","last_peered":"2024-04-30T08:33:31.879844+0000","last_clean":"2024-04-30T08:33:31.879844+0000","last_became_active":"2024-04-30T08:31:57.764519+0000","last_became_peered":"2024-04-30T08:31:57.764519+0000","last_unstale":"2024-04-30T08:33:31.879844+0000","last_undegraded":"2024-04-30T08:33:31.879844+0000","last_fullsized":"2024-04-30T08:33:31.879844+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:06:16.126396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"81'7","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881627+0000","last_change":"2024-04-30T08:31:51.686403+0000","last_active":"2024-04-30T08:33:31.881627+0000","last_peered":"2024-04-30T08:33:31.881627+0000","last_clean":"2024-04-30T08:33:31.881627+0000","last_became_active":"2024-04-30T08:31:51.686096+0000","last_became_peered":"2024-04-30T08:31:51.686096+0000","last_unstale":"2024-04-30T08:33:31.881627+0000","last_undegraded":"2024-04-30T08:33:31.881627+0000","last_fullsized":"2024-04-30T08:33:31.881627+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:47:05.961930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"85'8","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.698959+0000","last_change":"2024-04-30T08:31:52.698512+0000","last_active":"2024-04-30T08:33:59.698959+0000","last_peered":"2024-04-30T08:33:59.698959+0000","last_clean":"2024-04-30T08:33:59.698959+0000","last_became_active":"2024-04-30T08:31:52.698168+0000","last_became_peered":"2024-04-30T08:31:52.698168+0000","last_unstale":"2024-04-30T08:33:59.698959+0000","last_undegraded":"2024-04-30T08:33:59.698959+0000","last_fullsized":"2024-04-30T08:33:59.698959+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:22:45.094365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886398+0000","last_change":"2024-04-30T08:31:57.761430+0000","last_active":"2024-04-30T08:33:31.886398+0000","last_peered":"2024-04-30T08:33:31.886398+0000","last_clean":"2024-04-30T08:33:31.886398+0000","last_became_active":"2024-04-30T08:31:57.761062+0000","last_became_peered":"2024-04-30T08:31:57.761062+0000","last_unstale":"2024-04-30T08:33:31.886398+0000","last_undegraded":"2024-04-30T08:33:31.886398+0000","last_fullsized":"2024-04-30T08:33:31.886398+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:32:05.403786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"81'3","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.879820+0000","last_change":"2024-04-30T08:31:51.686694+0000","last_active":"2024-04-30T08:33:31.879820+0000","last_peered":"2024-04-30T08:33:31.879820+0000","last_clean":"2024-04-30T08:33:31.879820+0000","last_became_active":"2024-04-30T08:31:51.685773+0000","last_became_peered":"2024-04-30T08:31:51.685773+0000","last_unstale":"2024-04-30T08:33:31.879820+0000","last_undegraded":"2024-04-30T08:33:31.879820+0000","last_fullsized":"2024-04-30T08:33:31.879820+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:44:25.327977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"85'8","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.734178+0000","last_change":"2024-04-30T08:33:31.888698+0000","last_active":"2024-04-30T08:33:59.734178+0000","last_peered":"2024-04-30T08:33:59.734178+0000","last_clean":"2024-04-30T08:33:59.734178+0000","last_became_active":"2024-04-30T08:33:31.887935+0000","last_became_peered":"2024-04-30T08:33:31.887935+0000","last_unstale":"2024-04-30T08:33:59.734178+0000","last_undegraded":"2024-04-30T08:33:59.734178+0000","last_fullsized":"2024-04-30T08:33:59.734178+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:15:40.989004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.799780+0000","last_change":"2024-04-30T08:31:57.757224+0000","last_active":"2024-04-30T08:33:32.799780+0000","last_peered":"2024-04-30T08:33:32.799780+0000","last_clean":"2024-04-30T08:33:32.799780+0000","last_became_active":"2024-04-30T08:31:57.756658+0000","last_became_peered":"2024-04-30T08:31:57.756658+0000","last_unstale":"2024-04-30T08:33:32.799780+0000","last_undegraded":"2024-04-30T08:33:32.799780+0000","last_fullsized":"2024-04-30T08:33:32.799780+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:01:21.309322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"81'3","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273042+0000","last_change":"2024-04-30T08:31:51.678944+0000","last_active":"2024-04-30T08:33:32.273042+0000","last_peered":"2024-04-30T08:33:32.273042+0000","last_clean":"2024-04-30T08:33:32.273042+0000","last_became_active":"2024-04-30T08:31:51.678493+0000","last_became_peered":"2024-04-30T08:31:51.678493+0000","last_unstale":"2024-04-30T08:33:32.273042+0000","last_undegraded":"2024-04-30T08:33:32.273042+0000","last_fullsized":"2024-04-30T08:33:32.273042+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:48:11.111466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"85'11","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.722707+0000","last_change":"2024-04-30T08:31:52.707627+0000","last_active":"2024-04-30T08:33:59.722707+0000","last_peered":"2024-04-30T08:33:59.722707+0000","last_clean":"2024-04-30T08:33:59.722707+0000","last_became_active":"2024-04-30T08:31:52.706441+0000","last_became_peered":"2024-04-30T08:31:52.706441+0000","last_unstale":"2024-04-30T08:33:59.722707+0000","last_undegraded":"2024-04-30T08:33:59.722707+0000","last_fullsized":"2024-04-30T08:33:59.722707+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:15:45.768927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.275247+0000","last_change":"2024-04-30T08:31:57.740324+0000","last_active":"2024-04-30T08:33:32.275247+0000","last_peered":"2024-04-30T08:33:32.275247+0000","last_clean":"2024-04-30T08:33:32.275247+0000","last_became_active":"2024-04-30T08:31:57.739464+0000","last_became_peered":"2024-04-30T08:31:57.739464+0000","last_unstale":"2024-04-30T08:33:32.275247+0000","last_undegraded":"2024-04-30T08:33:32.275247+0000","last_fullsized":"2024-04-30T08:33:32.275247+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:11:37.395545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"85'5","reported_seq":356,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:35:08.812181+0000","last_change":"2024-04-30T08:31:51.695283+0000","last_active":"2024-04-30T08:35:08.812181+0000","last_peered":"2024-04-30T08:35:08.812181+0000","last_clean":"2024-04-30T08:35:08.812181+0000","last_became_active":"2024-04-30T08:31:51.694848+0000","last_became_peered":"2024-04-30T08:31:51.694848+0000","last_unstale":"2024-04-30T08:35:08.812181+0000","last_undegraded":"2024-04-30T08:35:08.812181+0000","last_fullsized":"2024-04-30T08:35:08.812181+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:41:38.638034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":317,"num_read_kb":317,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"85'10","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.729523+0000","last_change":"2024-04-30T08:31:52.693594+0000","last_active":"2024-04-30T08:33:59.729523+0000","last_peered":"2024-04-30T08:33:59.729523+0000","last_clean":"2024-04-30T08:33:59.729523+0000","last_became_active":"2024-04-30T08:31:52.693199+0000","last_became_peered":"2024-04-30T08:31:52.693199+0000","last_unstale":"2024-04-30T08:33:59.729523+0000","last_undegraded":"2024-04-30T08:33:59.729523+0000","last_fullsized":"2024-04-30T08:33:59.729523+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:57:49.908617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.883697+0000","last_change":"2024-04-30T08:31:57.749727+0000","last_active":"2024-04-30T08:33:31.883697+0000","last_peered":"2024-04-30T08:33:31.883697+0000","last_clean":"2024-04-30T08:33:31.883697+0000","last_became_active":"2024-04-30T08:31:57.749267+0000","last_became_peered":"2024-04-30T08:31:57.749267+0000","last_unstale":"2024-04-30T08:33:31.883697+0000","last_undegraded":"2024-04-30T08:33:31.883697+0000","last_fullsized":"2024-04-30T08:33:31.883697+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:26:12.683008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"81'4","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.218677+0000","last_change":"2024-04-30T08:31:51.677398+0000","last_active":"2024-04-30T08:33:32.218677+0000","last_peered":"2024-04-30T08:33:32.218677+0000","last_clean":"2024-04-30T08:33:32.218677+0000","last_became_active":"2024-04-30T08:31:51.677031+0000","last_became_peered":"2024-04-30T08:31:51.677031+0000","last_unstale":"2024-04-30T08:33:32.218677+0000","last_undegraded":"2024-04-30T08:33:32.218677+0000","last_fullsized":"2024-04-30T08:33:32.218677+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:24:25.154229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"85'9","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.741480+0000","last_change":"2024-04-30T08:32:31.561554+0000","last_active":"2024-04-30T08:33:59.741480+0000","last_peered":"2024-04-30T08:33:59.741480+0000","last_clean":"2024-04-30T08:33:59.741480+0000","last_became_active":"2024-04-30T08:32:31.560253+0000","last_became_peered":"2024-04-30T08:32:31.560253+0000","last_unstale":"2024-04-30T08:33:59.741480+0000","last_undegraded":"2024-04-30T08:33:59.741480+0000","last_fullsized":"2024-04-30T08:33:59.741480+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:59:08.602282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.170073+0000","last_change":"2024-04-30T08:32:31.586746+0000","last_active":"2024-04-30T08:33:32.170073+0000","last_peered":"2024-04-30T08:33:32.170073+0000","last_clean":"2024-04-30T08:33:32.170073+0000","last_became_active":"2024-04-30T08:32:31.586356+0000","last_became_peered":"2024-04-30T08:32:31.586356+0000","last_unstale":"2024-04-30T08:33:32.170073+0000","last_undegraded":"2024-04-30T08:33:32.170073+0000","last_fullsized":"2024-04-30T08:33:32.170073+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:22:39.371508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"81'5","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273483+0000","last_change":"2024-04-30T08:31:51.694424+0000","last_active":"2024-04-30T08:33:32.273483+0000","last_peered":"2024-04-30T08:33:32.273483+0000","last_clean":"2024-04-30T08:33:32.273483+0000","last_became_active":"2024-04-30T08:31:51.694081+0000","last_became_peered":"2024-04-30T08:31:51.694081+0000","last_unstale":"2024-04-30T08:33:32.273483+0000","last_undegraded":"2024-04-30T08:33:32.273483+0000","last_fullsized":"2024-04-30T08:33:32.273483+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:27:33.328477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"85'11","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.692022+0000","last_change":"2024-04-30T08:31:52.684739+0000","last_active":"2024-04-30T08:33:59.692022+0000","last_peered":"2024-04-30T08:33:59.692022+0000","last_clean":"2024-04-30T08:33:59.692022+0000","last_became_active":"2024-04-30T08:31:52.684316+0000","last_became_peered":"2024-04-30T08:31:52.684316+0000","last_unstale":"2024-04-30T08:33:59.692022+0000","last_undegraded":"2024-04-30T08:33:59.692022+0000","last_fullsized":"2024-04-30T08:33:59.692022+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:12:05.437493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881452+0000","last_change":"2024-04-30T08:32:31.557578+0000","last_active":"2024-04-30T08:33:31.881452+0000","last_peered":"2024-04-30T08:33:31.881452+0000","last_clean":"2024-04-30T08:33:31.881452+0000","last_became_active":"2024-04-30T08:32:31.557115+0000","last_became_peered":"2024-04-30T08:32:31.557115+0000","last_unstale":"2024-04-30T08:33:31.881452+0000","last_undegraded":"2024-04-30T08:33:31.881452+0000","last_fullsized":"2024-04-30T08:33:31.881452+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:41:33.225879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"85'40","reported_seq":400,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:35:08.812186+0000","last_change":"2024-04-30T08:31:51.676223+0000","last_active":"2024-04-30T08:35:08.812186+0000","last_peered":"2024-04-30T08:35:08.812186+0000","last_clean":"2024-04-30T08:35:08.812186+0000","last_became_active":"2024-04-30T08:31:51.675830+0000","last_became_peered":"2024-04-30T08:31:51.675830+0000","last_unstale":"2024-04-30T08:35:08.812186+0000","last_undegraded":"2024-04-30T08:35:08.812186+0000","last_fullsized":"2024-04-30T08:35:08.812186+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:28:49.497719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67336,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":326,"num_read_kb":229,"num_write":42,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"85'6","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.684496+0000","last_change":"2024-04-30T08:31:52.686892+0000","last_active":"2024-04-30T08:33:59.684496+0000","last_peered":"2024-04-30T08:33:59.684496+0000","last_clean":"2024-04-30T08:33:59.684496+0000","last_became_active":"2024-04-30T08:31:52.686533+0000","last_became_peered":"2024-04-30T08:31:52.686533+0000","last_unstale":"2024-04-30T08:33:59.684496+0000","last_undegraded":"2024-04-30T08:33:59.684496+0000","last_fullsized":"2024-04-30T08:33:59.684496+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:50:18.836161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.274017+0000","last_change":"2024-04-30T08:31:57.752865+0000","last_active":"2024-04-30T08:33:32.274017+0000","last_peered":"2024-04-30T08:33:32.274017+0000","last_clean":"2024-04-30T08:33:32.274017+0000","last_became_active":"2024-04-30T08:31:57.752385+0000","last_became_peered":"2024-04-30T08:31:57.752385+0000","last_unstale":"2024-04-30T08:33:32.274017+0000","last_undegraded":"2024-04-30T08:33:32.274017+0000","last_fullsized":"2024-04-30T08:33:32.274017+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:15:15.315580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"81'9","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886273+0000","last_change":"2024-04-30T08:32:31.585282+0000","last_active":"2024-04-30T08:33:31.886273+0000","last_peered":"2024-04-30T08:33:31.886273+0000","last_clean":"2024-04-30T08:33:31.886273+0000","last_became_active":"2024-04-30T08:32:31.571676+0000","last_became_peered":"2024-04-30T08:32:31.571676+0000","last_unstale":"2024-04-30T08:33:31.886273+0000","last_undegraded":"2024-04-30T08:33:31.886273+0000","last_fullsized":"2024-04-30T08:33:31.886273+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:18:29.910201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"85'9","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.723531+0000","last_change":"2024-04-30T08:31:52.706259+0000","last_active":"2024-04-30T08:33:59.723531+0000","last_peered":"2024-04-30T08:33:59.723531+0000","last_clean":"2024-04-30T08:33:59.723531+0000","last_became_active":"2024-04-30T08:31:52.705620+0000","last_became_peered":"2024-04-30T08:31:52.705620+0000","last_unstale":"2024-04-30T08:33:59.723531+0000","last_undegraded":"2024-04-30T08:33:59.723531+0000","last_fullsized":"2024-04-30T08:33:59.723531+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:22:29.816950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886632+0000","last_change":"2024-04-30T08:32:31.562571+0000","last_active":"2024-04-30T08:33:31.886632+0000","last_peered":"2024-04-30T08:33:31.886632+0000","last_clean":"2024-04-30T08:33:31.886632+0000","last_became_active":"2024-04-30T08:32:31.562012+0000","last_became_peered":"2024-04-30T08:32:31.562012+0000","last_unstale":"2024-04-30T08:33:31.886632+0000","last_undegraded":"2024-04-30T08:33:31.886632+0000","last_fullsized":"2024-04-30T08:33:31.886632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:12:22.928286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"81'5","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273150+0000","last_change":"2024-04-30T08:31:51.701122+0000","last_active":"2024-04-30T08:33:32.273150+0000","last_peered":"2024-04-30T08:33:32.273150+0000","last_clean":"2024-04-30T08:33:32.273150+0000","last_became_active":"2024-04-30T08:31:51.700815+0000","last_became_peered":"2024-04-30T08:31:51.700815+0000","last_unstale":"2024-04-30T08:33:32.273150+0000","last_undegraded":"2024-04-30T08:33:32.273150+0000","last_fullsized":"2024-04-30T08:33:32.273150+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:33:12.714979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"85'15","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.721587+0000","last_change":"2024-04-30T08:31:52.716043+0000","last_active":"2024-04-30T08:33:59.721587+0000","last_peered":"2024-04-30T08:33:59.721587+0000","last_clean":"2024-04-30T08:33:59.721587+0000","last_became_active":"2024-04-30T08:31:52.715284+0000","last_became_peered":"2024-04-30T08:31:52.715284+0000","last_unstale":"2024-04-30T08:33:59.721587+0000","last_undegraded":"2024-04-30T08:33:59.721587+0000","last_fullsized":"2024-04-30T08:33:59.721587+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:00:01.863606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.885198+0000","last_change":"2024-04-30T08:31:57.765008+0000","last_active":"2024-04-30T08:33:31.885198+0000","last_peered":"2024-04-30T08:33:31.885198+0000","last_clean":"2024-04-30T08:33:31.885198+0000","last_became_active":"2024-04-30T08:31:57.762958+0000","last_became_peered":"2024-04-30T08:31:57.762958+0000","last_unstale":"2024-04-30T08:33:31.885198+0000","last_undegraded":"2024-04-30T08:33:31.885198+0000","last_fullsized":"2024-04-30T08:33:31.885198+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:48:47.116004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"85'9","reported_seq":206,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:35:06.944057+0000","last_change":"2024-04-30T08:31:51.676681+0000","last_active":"2024-04-30T08:35:06.944057+0000","last_peered":"2024-04-30T08:35:06.944057+0000","last_clean":"2024-04-30T08:35:06.944057+0000","last_became_active":"2024-04-30T08:31:51.676311+0000","last_became_peered":"2024-04-30T08:31:51.676311+0000","last_unstale":"2024-04-30T08:35:06.944057+0000","last_undegraded":"2024-04-30T08:35:06.944057+0000","last_fullsized":"2024-04-30T08:35:06.944057+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:24:49.757427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1821,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":164,"num_read_kb":6,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"85'6","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.687636+0000","last_change":"2024-04-30T08:31:52.689418+0000","last_active":"2024-04-30T08:33:59.687636+0000","last_peered":"2024-04-30T08:33:59.687636+0000","last_clean":"2024-04-30T08:33:59.687636+0000","last_became_active":"2024-04-30T08:31:52.688840+0000","last_became_peered":"2024-04-30T08:31:52.688840+0000","last_unstale":"2024-04-30T08:33:59.687636+0000","last_undegraded":"2024-04-30T08:33:59.687636+0000","last_fullsized":"2024-04-30T08:33:59.687636+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:40:05.781137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881271+0000","last_change":"2024-04-30T08:31:57.765875+0000","last_active":"2024-04-30T08:33:31.881271+0000","last_peered":"2024-04-30T08:33:31.881271+0000","last_clean":"2024-04-30T08:33:31.881271+0000","last_became_active":"2024-04-30T08:31:57.765432+0000","last_became_peered":"2024-04-30T08:31:57.765432+0000","last_unstale":"2024-04-30T08:33:31.881271+0000","last_undegraded":"2024-04-30T08:33:31.881271+0000","last_fullsized":"2024-04-30T08:33:31.881271+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:24:44.329532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"81'17","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.887076+0000","last_change":"2024-04-30T08:31:51.695007+0000","last_active":"2024-04-30T08:33:31.887076+0000","last_peered":"2024-04-30T08:33:31.887076+0000","last_clean":"2024-04-30T08:33:31.887076+0000","last_became_active":"2024-04-30T08:31:51.694533+0000","last_became_peered":"2024-04-30T08:31:51.694533+0000","last_unstale":"2024-04-30T08:33:31.887076+0000","last_undegraded":"2024-04-30T08:33:31.887076+0000","last_fullsized":"2024-04-30T08:33:31.887076+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:47:15.104172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"85'6","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.687169+0000","last_change":"2024-04-30T08:31:52.692274+0000","last_active":"2024-04-30T08:33:59.687169+0000","last_peered":"2024-04-30T08:33:59.687169+0000","last_clean":"2024-04-30T08:33:59.687169+0000","last_became_active":"2024-04-30T08:31:52.691809+0000","last_became_peered":"2024-04-30T08:31:52.691809+0000","last_unstale":"2024-04-30T08:33:59.687169+0000","last_undegraded":"2024-04-30T08:33:59.687169+0000","last_fullsized":"2024-04-30T08:33:59.687169+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:50:00.201782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273166+0000","last_change":"2024-04-30T08:31:57.732383+0000","last_active":"2024-04-30T08:33:32.273166+0000","last_peered":"2024-04-30T08:33:32.273166+0000","last_clean":"2024-04-30T08:33:32.273166+0000","last_became_active":"2024-04-30T08:31:57.731893+0000","last_became_peered":"2024-04-30T08:31:57.731893+0000","last_unstale":"2024-04-30T08:33:32.273166+0000","last_undegraded":"2024-04-30T08:33:32.273166+0000","last_fullsized":"2024-04-30T08:33:32.273166+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:08:55.243526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'4","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:35:04.537295+0000","last_change":"2024-04-30T08:31:51.671991+0000","last_active":"2024-04-30T08:35:04.537295+0000","last_peered":"2024-04-30T08:35:04.537295+0000","last_clean":"2024-04-30T08:35:04.537295+0000","last_became_active":"2024-04-30T08:31:51.671508+0000","last_became_peered":"2024-04-30T08:31:51.671508+0000","last_unstale":"2024-04-30T08:35:04.537295+0000","last_undegraded":"2024-04-30T08:35:04.537295+0000","last_fullsized":"2024-04-30T08:35:04.537295+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:54:30.106748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"85'5","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.689682+0000","last_change":"2024-04-30T08:33:32.274901+0000","last_active":"2024-04-30T08:33:59.689682+0000","last_peered":"2024-04-30T08:33:59.689682+0000","last_clean":"2024-04-30T08:33:59.689682+0000","last_became_active":"2024-04-30T08:33:32.274401+0000","last_became_peered":"2024-04-30T08:33:32.274401+0000","last_unstale":"2024-04-30T08:33:59.689682+0000","last_undegraded":"2024-04-30T08:33:59.689682+0000","last_fullsized":"2024-04-30T08:33:59.689682+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:23:08.703092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886092+0000","last_change":"2024-04-30T08:31:58.061077+0000","last_active":"2024-04-30T08:33:31.886092+0000","last_peered":"2024-04-30T08:33:31.886092+0000","last_clean":"2024-04-30T08:33:31.886092+0000","last_became_active":"2024-04-30T08:31:58.060486+0000","last_became_peered":"2024-04-30T08:31:58.060486+0000","last_unstale":"2024-04-30T08:33:31.886092+0000","last_undegraded":"2024-04-30T08:33:31.886092+0000","last_fullsized":"2024-04-30T08:33:31.886092+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:31:50.936560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"81'1","reported_seq":34,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.885199+0000","last_change":"2024-04-30T08:31:51.702869+0000","last_active":"2024-04-30T08:33:31.885199+0000","last_peered":"2024-04-30T08:33:31.885199+0000","last_clean":"2024-04-30T08:33:31.885199+0000","last_became_active":"2024-04-30T08:31:51.701150+0000","last_became_peered":"2024-04-30T08:31:51.701150+0000","last_unstale":"2024-04-30T08:33:31.885199+0000","last_undegraded":"2024-04-30T08:33:31.885199+0000","last_fullsized":"2024-04-30T08:33:31.885199+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:47:11.010310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"85'4","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.716233+0000","last_change":"2024-04-30T08:31:52.709159+0000","last_active":"2024-04-30T08:33:59.716233+0000","last_peered":"2024-04-30T08:33:59.716233+0000","last_clean":"2024-04-30T08:33:59.716233+0000","last_became_active":"2024-04-30T08:31:52.708799+0000","last_became_peered":"2024-04-30T08:31:52.708799+0000","last_unstale":"2024-04-30T08:33:59.716233+0000","last_undegraded":"2024-04-30T08:33:59.716233+0000","last_fullsized":"2024-04-30T08:33:59.716233+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:51:10.588599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881244+0000","last_change":"2024-04-30T08:31:57.751652+0000","last_active":"2024-04-30T08:33:31.881244+0000","last_peered":"2024-04-30T08:33:31.881244+0000","last_clean":"2024-04-30T08:33:31.881244+0000","last_became_active":"2024-04-30T08:31:57.751176+0000","last_became_peered":"2024-04-30T08:31:57.751176+0000","last_unstale":"2024-04-30T08:33:31.881244+0000","last_undegraded":"2024-04-30T08:33:31.881244+0000","last_fullsized":"2024-04-30T08:33:31.881244+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:13:24.528842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"85'6","reported_seq":200,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:35:06.921270+0000","last_change":"2024-04-30T08:31:51.700816+0000","last_active":"2024-04-30T08:35:06.921270+0000","last_peered":"2024-04-30T08:35:06.921270+0000","last_clean":"2024-04-30T08:35:06.921270+0000","last_became_active":"2024-04-30T08:31:51.700470+0000","last_became_peered":"2024-04-30T08:31:51.700470+0000","last_unstale":"2024-04-30T08:35:06.921270+0000","last_undegraded":"2024-04-30T08:35:06.921270+0000","last_fullsized":"2024-04-30T08:35:06.921270+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:49:11.793412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":160,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"85'7","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.726100+0000","last_change":"2024-04-30T08:31:52.715236+0000","last_active":"2024-04-30T08:33:59.726100+0000","last_peered":"2024-04-30T08:33:59.726100+0000","last_clean":"2024-04-30T08:33:59.726100+0000","last_became_active":"2024-04-30T08:31:52.714801+0000","last_became_peered":"2024-04-30T08:31:52.714801+0000","last_unstale":"2024-04-30T08:33:59.726100+0000","last_undegraded":"2024-04-30T08:33:59.726100+0000","last_fullsized":"2024-04-30T08:33:59.726100+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:43:09.650002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881702+0000","last_change":"2024-04-30T08:31:58.060927+0000","last_active":"2024-04-30T08:33:31.881702+0000","last_peered":"2024-04-30T08:33:31.881702+0000","last_clean":"2024-04-30T08:33:31.881702+0000","last_became_active":"2024-04-30T08:31:58.060182+0000","last_became_peered":"2024-04-30T08:31:58.060182+0000","last_unstale":"2024-04-30T08:33:31.881702+0000","last_undegraded":"2024-04-30T08:33:31.881702+0000","last_fullsized":"2024-04-30T08:33:31.881702+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:21:00.228064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"85'8","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.679643+0000","last_change":"2024-04-30T08:32:31.560485+0000","last_active":"2024-04-30T08:33:59.679643+0000","last_peered":"2024-04-30T08:33:59.679643+0000","last_clean":"2024-04-30T08:33:59.679643+0000","last_became_active":"2024-04-30T08:32:31.560066+0000","last_became_peered":"2024-04-30T08:32:31.560066+0000","last_unstale":"2024-04-30T08:33:59.679643+0000","last_undegraded":"2024-04-30T08:33:59.679643+0000","last_fullsized":"2024-04-30T08:33:59.679643+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:13:40.769761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'7","reported_seq":201,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:35:08.947184+0000","last_change":"2024-04-30T08:31:51.702808+0000","last_active":"2024-04-30T08:35:08.947184+0000","last_peered":"2024-04-30T08:35:08.947184+0000","last_clean":"2024-04-30T08:35:08.947184+0000","last_became_active":"2024-04-30T08:31:51.702326+0000","last_became_peered":"2024-04-30T08:31:51.702326+0000","last_unstale":"2024-04-30T08:35:08.947184+0000","last_undegraded":"2024-04-30T08:35:08.947184+0000","last_fullsized":"2024-04-30T08:35:08.947184+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:11:35.552670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":161,"num_read_kb":161,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"81'1","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.218503+0000","last_change":"2024-04-30T08:31:57.760359+0000","last_active":"2024-04-30T08:33:32.218503+0000","last_peered":"2024-04-30T08:33:32.218503+0000","last_clean":"2024-04-30T08:33:32.218503+0000","last_became_active":"2024-04-30T08:31:57.759821+0000","last_became_peered":"2024-04-30T08:31:57.759821+0000","last_unstale":"2024-04-30T08:33:32.218503+0000","last_undegraded":"2024-04-30T08:33:32.218503+0000","last_fullsized":"2024-04-30T08:33:32.218503+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:50:17.853791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"85'8","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.719174+0000","last_change":"2024-04-30T08:31:52.689281+0000","last_active":"2024-04-30T08:33:59.719174+0000","last_peered":"2024-04-30T08:33:59.719174+0000","last_clean":"2024-04-30T08:33:59.719174+0000","last_became_active":"2024-04-30T08:31:52.688834+0000","last_became_peered":"2024-04-30T08:31:52.688834+0000","last_unstale":"2024-04-30T08:33:59.719174+0000","last_undegraded":"2024-04-30T08:33:59.719174+0000","last_fullsized":"2024-04-30T08:33:59.719174+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:42:12.836649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'6","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.169972+0000","last_change":"2024-04-30T08:31:51.683712+0000","last_active":"2024-04-30T08:33:32.169972+0000","last_peered":"2024-04-30T08:33:32.169972+0000","last_clean":"2024-04-30T08:33:32.169972+0000","last_became_active":"2024-04-30T08:31:51.683298+0000","last_became_peered":"2024-04-30T08:31:51.683298+0000","last_unstale":"2024-04-30T08:33:32.169972+0000","last_undegraded":"2024-04-30T08:33:32.169972+0000","last_fullsized":"2024-04-30T08:33:32.169972+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:43:08.832104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":109,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.217885+0000","last_change":"2024-04-30T08:30:26.503221+0000","last_active":"2024-04-30T08:33:32.217885+0000","last_peered":"2024-04-30T08:33:32.217885+0000","last_clean":"2024-04-30T08:33:32.217885+0000","last_became_active":"2024-04-30T08:30:26.498399+0000","last_became_peered":"2024-04-30T08:30:26.498399+0000","last_unstale":"2024-04-30T08:33:32.217885+0000","last_undegraded":"2024-04-30T08:33:32.217885+0000","last_fullsized":"2024-04-30T08:33:32.217885+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:26:20.170075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:26:20.170075+0000","last_clean_scrub_stamp":"2024-04-30T08:26:20.170075+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:34:44.067560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273554+0000","last_change":"2024-04-30T08:31:58.061283+0000","last_active":"2024-04-30T08:33:32.273554+0000","last_peered":"2024-04-30T08:33:32.273554+0000","last_clean":"2024-04-30T08:33:32.273554+0000","last_became_active":"2024-04-30T08:31:58.060650+0000","last_became_peered":"2024-04-30T08:31:58.060650+0000","last_unstale":"2024-04-30T08:33:32.273554+0000","last_undegraded":"2024-04-30T08:33:32.273554+0000","last_fullsized":"2024-04-30T08:33:32.273554+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:13:01.811462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"85'5","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.728420+0000","last_change":"2024-04-30T08:31:52.699639+0000","last_active":"2024-04-30T08:33:59.728420+0000","last_peered":"2024-04-30T08:33:59.728420+0000","last_clean":"2024-04-30T08:33:59.728420+0000","last_became_active":"2024-04-30T08:31:52.699164+0000","last_became_peered":"2024-04-30T08:31:52.699164+0000","last_unstale":"2024-04-30T08:33:59.728420+0000","last_undegraded":"2024-04-30T08:33:59.728420+0000","last_fullsized":"2024-04-30T08:33:59.728420+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:22:18.873504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"81'4","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.217821+0000","last_change":"2024-04-30T08:31:51.679153+0000","last_active":"2024-04-30T08:33:32.217821+0000","last_peered":"2024-04-30T08:33:32.217821+0000","last_clean":"2024-04-30T08:33:32.217821+0000","last_became_active":"2024-04-30T08:31:51.678348+0000","last_became_peered":"2024-04-30T08:31:51.678348+0000","last_unstale":"2024-04-30T08:33:32.217821+0000","last_undegraded":"2024-04-30T08:33:32.217821+0000","last_fullsized":"2024-04-30T08:33:32.217821+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:02:56.240218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886616+0000","last_change":"2024-04-30T08:31:57.755423+0000","last_active":"2024-04-30T08:33:31.886616+0000","last_peered":"2024-04-30T08:33:31.886616+0000","last_clean":"2024-04-30T08:33:31.886616+0000","last_became_active":"2024-04-30T08:31:57.754835+0000","last_became_peered":"2024-04-30T08:31:57.754835+0000","last_unstale":"2024-04-30T08:33:31.886616+0000","last_undegraded":"2024-04-30T08:33:31.886616+0000","last_fullsized":"2024-04-30T08:33:31.886616+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:19:25.168920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"81'4","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886919+0000","last_change":"2024-04-30T08:31:51.693839+0000","last_active":"2024-04-30T08:33:31.886919+0000","last_peered":"2024-04-30T08:33:31.886919+0000","last_clean":"2024-04-30T08:33:31.886919+0000","last_became_active":"2024-04-30T08:31:51.693349+0000","last_became_peered":"2024-04-30T08:31:51.693349+0000","last_unstale":"2024-04-30T08:33:31.886919+0000","last_undegraded":"2024-04-30T08:33:31.886919+0000","last_fullsized":"2024-04-30T08:33:31.886919+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:41:51.330901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"85'9","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.733847+0000","last_change":"2024-04-30T08:31:52.712360+0000","last_active":"2024-04-30T08:33:59.733847+0000","last_peered":"2024-04-30T08:33:59.733847+0000","last_clean":"2024-04-30T08:33:59.733847+0000","last_became_active":"2024-04-30T08:31:52.711652+0000","last_became_peered":"2024-04-30T08:31:52.711652+0000","last_unstale":"2024-04-30T08:33:59.733847+0000","last_undegraded":"2024-04-30T08:33:59.733847+0000","last_fullsized":"2024-04-30T08:33:59.733847+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:21:10.285386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.276738+0000","last_change":"2024-04-30T08:31:58.061654+0000","last_active":"2024-04-30T08:33:32.276738+0000","last_peered":"2024-04-30T08:33:32.276738+0000","last_clean":"2024-04-30T08:33:32.276738+0000","last_became_active":"2024-04-30T08:31:58.060181+0000","last_became_peered":"2024-04-30T08:31:58.060181+0000","last_unstale":"2024-04-30T08:33:32.276738+0000","last_undegraded":"2024-04-30T08:33:32.276738+0000","last_fullsized":"2024-04-30T08:33:32.276738+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:46:03.230626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'15","reported_seq":48,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:35:04.265838+0000","last_change":"2024-04-30T08:31:51.706940+0000","last_active":"2024-04-30T08:35:04.265838+0000","last_peered":"2024-04-30T08:35:04.265838+0000","last_clean":"2024-04-30T08:35:04.265838+0000","last_became_active":"2024-04-30T08:31:51.706481+0000","last_became_peered":"2024-04-30T08:31:51.706481+0000","last_unstale":"2024-04-30T08:35:04.265838+0000","last_undegraded":"2024-04-30T08:35:04.265838+0000","last_fullsized":"2024-04-30T08:35:04.265838+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:53:31.955595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"85'5","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.714966+0000","last_change":"2024-04-30T08:31:52.696625+0000","last_active":"2024-04-30T08:33:59.714966+0000","last_peered":"2024-04-30T08:33:59.714966+0000","last_clean":"2024-04-30T08:33:59.714966+0000","last_became_active":"2024-04-30T08:31:52.696230+0000","last_became_peered":"2024-04-30T08:31:52.696230+0000","last_unstale":"2024-04-30T08:33:59.714966+0000","last_undegraded":"2024-04-30T08:33:59.714966+0000","last_fullsized":"2024-04-30T08:33:59.714966+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:11:46.404251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.879436+0000","last_change":"2024-04-30T08:31:57.738329+0000","last_active":"2024-04-30T08:33:31.879436+0000","last_peered":"2024-04-30T08:33:31.879436+0000","last_clean":"2024-04-30T08:33:31.879436+0000","last_became_active":"2024-04-30T08:31:57.737897+0000","last_became_peered":"2024-04-30T08:31:57.737897+0000","last_unstale":"2024-04-30T08:33:31.879436+0000","last_undegraded":"2024-04-30T08:33:31.879436+0000","last_fullsized":"2024-04-30T08:33:31.879436+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:02:43.074578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"81'7","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.170374+0000","last_change":"2024-04-30T08:31:51.684066+0000","last_active":"2024-04-30T08:33:32.170374+0000","last_peered":"2024-04-30T08:33:32.170374+0000","last_clean":"2024-04-30T08:33:32.170374+0000","last_became_active":"2024-04-30T08:31:51.683699+0000","last_became_peered":"2024-04-30T08:31:51.683699+0000","last_unstale":"2024-04-30T08:33:32.170374+0000","last_undegraded":"2024-04-30T08:33:32.170374+0000","last_fullsized":"2024-04-30T08:33:32.170374+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:36:20.900733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"85'7","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.740363+0000","last_change":"2024-04-30T08:31:52.715219+0000","last_active":"2024-04-30T08:33:59.740363+0000","last_peered":"2024-04-30T08:33:59.740363+0000","last_clean":"2024-04-30T08:33:59.740363+0000","last_became_active":"2024-04-30T08:31:52.714774+0000","last_became_peered":"2024-04-30T08:31:52.714774+0000","last_unstale":"2024-04-30T08:33:59.740363+0000","last_undegraded":"2024-04-30T08:33:59.740363+0000","last_fullsized":"2024-04-30T08:33:59.740363+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:42:41.082610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.170157+0000","last_change":"2024-04-30T08:31:58.060845+0000","last_active":"2024-04-30T08:33:32.170157+0000","last_peered":"2024-04-30T08:33:32.170157+0000","last_clean":"2024-04-30T08:33:32.170157+0000","last_became_active":"2024-04-30T08:31:58.060365+0000","last_became_peered":"2024-04-30T08:31:58.060365+0000","last_unstale":"2024-04-30T08:33:32.170157+0000","last_undegraded":"2024-04-30T08:33:32.170157+0000","last_fullsized":"2024-04-30T08:33:32.170157+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:21:39.583243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"81'2","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.272616+0000","last_change":"2024-04-30T08:31:51.694833+0000","last_active":"2024-04-30T08:33:32.272616+0000","last_peered":"2024-04-30T08:33:32.272616+0000","last_clean":"2024-04-30T08:33:32.272616+0000","last_became_active":"2024-04-30T08:31:51.694401+0000","last_became_peered":"2024-04-30T08:31:51.694401+0000","last_unstale":"2024-04-30T08:33:32.272616+0000","last_undegraded":"2024-04-30T08:33:32.272616+0000","last_fullsized":"2024-04-30T08:33:32.272616+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:38:52.605625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"85'2","reported_seq":33,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.676614+0000","last_change":"2024-04-30T08:31:52.686444+0000","last_active":"2024-04-30T08:33:59.676614+0000","last_peered":"2024-04-30T08:33:59.676614+0000","last_clean":"2024-04-30T08:33:59.676614+0000","last_became_active":"2024-04-30T08:31:52.686057+0000","last_became_peered":"2024-04-30T08:31:52.686057+0000","last_unstale":"2024-04-30T08:33:59.676614+0000","last_undegraded":"2024-04-30T08:33:59.676614+0000","last_fullsized":"2024-04-30T08:33:59.676614+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:54:29.824060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.885416+0000","last_change":"2024-04-30T08:31:57.766193+0000","last_active":"2024-04-30T08:33:31.885416+0000","last_peered":"2024-04-30T08:33:31.885416+0000","last_clean":"2024-04-30T08:33:31.885416+0000","last_became_active":"2024-04-30T08:31:57.765758+0000","last_became_peered":"2024-04-30T08:31:57.765758+0000","last_unstale":"2024-04-30T08:33:31.885416+0000","last_undegraded":"2024-04-30T08:33:31.885416+0000","last_fullsized":"2024-04-30T08:33:31.885416+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:59:57.282795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"81'7","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.275654+0000","last_change":"2024-04-30T08:31:51.704565+0000","last_active":"2024-04-30T08:33:32.275654+0000","last_peered":"2024-04-30T08:33:32.275654+0000","last_clean":"2024-04-30T08:33:32.275654+0000","last_became_active":"2024-04-30T08:31:51.703954+0000","last_became_peered":"2024-04-30T08:31:51.703954+0000","last_unstale":"2024-04-30T08:33:32.275654+0000","last_undegraded":"2024-04-30T08:33:32.275654+0000","last_fullsized":"2024-04-30T08:33:32.275654+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:03:00.517867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"85'9","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.717404+0000","last_change":"2024-04-30T08:31:52.706043+0000","last_active":"2024-04-30T08:33:59.717404+0000","last_peered":"2024-04-30T08:33:59.717404+0000","last_clean":"2024-04-30T08:33:59.717404+0000","last_became_active":"2024-04-30T08:31:52.705678+0000","last_became_peered":"2024-04-30T08:31:52.705678+0000","last_unstale":"2024-04-30T08:33:59.717404+0000","last_undegraded":"2024-04-30T08:33:59.717404+0000","last_fullsized":"2024-04-30T08:33:59.717404+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:44:56.787835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"82'1","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.170123+0000","last_change":"2024-04-30T08:31:57.764791+0000","last_active":"2024-04-30T08:33:32.170123+0000","last_peered":"2024-04-30T08:33:32.170123+0000","last_clean":"2024-04-30T08:33:32.170123+0000","last_became_active":"2024-04-30T08:31:57.764368+0000","last_became_peered":"2024-04-30T08:31:57.764368+0000","last_unstale":"2024-04-30T08:33:32.170123+0000","last_undegraded":"2024-04-30T08:33:32.170123+0000","last_fullsized":"2024-04-30T08:33:32.170123+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:19:28.545483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"81'3","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886461+0000","last_change":"2024-04-30T08:32:31.577309+0000","last_active":"2024-04-30T08:33:31.886461+0000","last_peered":"2024-04-30T08:33:31.886461+0000","last_clean":"2024-04-30T08:33:31.886461+0000","last_became_active":"2024-04-30T08:32:31.571515+0000","last_became_peered":"2024-04-30T08:32:31.571515+0000","last_unstale":"2024-04-30T08:33:31.886461+0000","last_undegraded":"2024-04-30T08:33:31.886461+0000","last_fullsized":"2024-04-30T08:33:31.886461+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:28:48.408414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"85'8","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.730222+0000","last_change":"2024-04-30T08:31:52.716707+0000","last_active":"2024-04-30T08:33:59.730222+0000","last_peered":"2024-04-30T08:33:59.730222+0000","last_clean":"2024-04-30T08:33:59.730222+0000","last_became_active":"2024-04-30T08:31:52.716293+0000","last_became_peered":"2024-04-30T08:31:52.716293+0000","last_unstale":"2024-04-30T08:33:59.730222+0000","last_undegraded":"2024-04-30T08:33:59.730222+0000","last_fullsized":"2024-04-30T08:33:59.730222+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:29:54.223472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.170159+0000","last_change":"2024-04-30T08:31:58.060640+0000","last_active":"2024-04-30T08:33:32.170159+0000","last_peered":"2024-04-30T08:33:32.170159+0000","last_clean":"2024-04-30T08:33:32.170159+0000","last_became_active":"2024-04-30T08:31:58.060061+0000","last_became_peered":"2024-04-30T08:31:58.060061+0000","last_unstale":"2024-04-30T08:33:32.170159+0000","last_undegraded":"2024-04-30T08:33:32.170159+0000","last_fullsized":"2024-04-30T08:33:32.170159+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:46:00.844191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"81'4","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886171+0000","last_change":"2024-04-30T08:31:51.679532+0000","last_active":"2024-04-30T08:33:31.886171+0000","last_peered":"2024-04-30T08:33:31.886171+0000","last_clean":"2024-04-30T08:33:31.886171+0000","last_became_active":"2024-04-30T08:31:51.679141+0000","last_became_peered":"2024-04-30T08:31:51.679141+0000","last_unstale":"2024-04-30T08:33:31.886171+0000","last_undegraded":"2024-04-30T08:33:31.886171+0000","last_fullsized":"2024-04-30T08:33:31.886171+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:19:50.526563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"85'5","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.720554+0000","last_change":"2024-04-30T08:31:52.703613+0000","last_active":"2024-04-30T08:33:59.720554+0000","last_peered":"2024-04-30T08:33:59.720554+0000","last_clean":"2024-04-30T08:33:59.720554+0000","last_became_active":"2024-04-30T08:31:52.703196+0000","last_became_peered":"2024-04-30T08:31:52.703196+0000","last_unstale":"2024-04-30T08:33:59.720554+0000","last_undegraded":"2024-04-30T08:33:59.720554+0000","last_fullsized":"2024-04-30T08:33:59.720554+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:33:11.136313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.272631+0000","last_change":"2024-04-30T08:31:57.755286+0000","last_active":"2024-04-30T08:33:32.272631+0000","last_peered":"2024-04-30T08:33:32.272631+0000","last_clean":"2024-04-30T08:33:32.272631+0000","last_became_active":"2024-04-30T08:31:57.754786+0000","last_became_peered":"2024-04-30T08:31:57.754786+0000","last_unstale":"2024-04-30T08:33:32.272631+0000","last_undegraded":"2024-04-30T08:33:32.272631+0000","last_fullsized":"2024-04-30T08:33:32.272631+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:02:50.931553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"81'5","reported_seq":198,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:35:07.811337+0000","last_change":"2024-04-30T08:31:51.693317+0000","last_active":"2024-04-30T08:35:07.811337+0000","last_peered":"2024-04-30T08:35:07.811337+0000","last_clean":"2024-04-30T08:35:07.811337+0000","last_became_active":"2024-04-30T08:31:51.692928+0000","last_became_peered":"2024-04-30T08:31:51.692928+0000","last_unstale":"2024-04-30T08:35:07.811337+0000","last_undegraded":"2024-04-30T08:35:07.811337+0000","last_fullsized":"2024-04-30T08:35:07.811337+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:35:09.851735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":160,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"85'7","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.692096+0000","last_change":"2024-04-30T08:31:52.694304+0000","last_active":"2024-04-30T08:33:59.692096+0000","last_peered":"2024-04-30T08:33:59.692096+0000","last_clean":"2024-04-30T08:33:59.692096+0000","last_became_active":"2024-04-30T08:31:52.693317+0000","last_became_peered":"2024-04-30T08:31:52.693317+0000","last_unstale":"2024-04-30T08:33:59.692096+0000","last_undegraded":"2024-04-30T08:33:59.692096+0000","last_fullsized":"2024-04-30T08:33:59.692096+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:31:39.294864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.799022+0000","last_change":"2024-04-30T08:31:57.763598+0000","last_active":"2024-04-30T08:33:32.799022+0000","last_peered":"2024-04-30T08:33:32.799022+0000","last_clean":"2024-04-30T08:33:32.799022+0000","last_became_active":"2024-04-30T08:31:57.762307+0000","last_became_peered":"2024-04-30T08:31:57.762307+0000","last_unstale":"2024-04-30T08:33:32.799022+0000","last_undegraded":"2024-04-30T08:33:32.799022+0000","last_fullsized":"2024-04-30T08:33:32.799022+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:24:13.390595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"81'3","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273810+0000","last_change":"2024-04-30T08:31:51.703476+0000","last_active":"2024-04-30T08:33:32.273810+0000","last_peered":"2024-04-30T08:33:32.273810+0000","last_clean":"2024-04-30T08:33:32.273810+0000","last_became_active":"2024-04-30T08:31:51.703072+0000","last_became_peered":"2024-04-30T08:31:51.703072+0000","last_unstale":"2024-04-30T08:33:32.273810+0000","last_undegraded":"2024-04-30T08:33:32.273810+0000","last_fullsized":"2024-04-30T08:33:32.273810+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:07:21.098223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"85'6","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.698707+0000","last_change":"2024-04-30T08:31:52.699223+0000","last_active":"2024-04-30T08:33:59.698707+0000","last_peered":"2024-04-30T08:33:59.698707+0000","last_clean":"2024-04-30T08:33:59.698707+0000","last_became_active":"2024-04-30T08:31:52.698837+0000","last_became_peered":"2024-04-30T08:31:52.698837+0000","last_unstale":"2024-04-30T08:33:59.698707+0000","last_undegraded":"2024-04-30T08:33:59.698707+0000","last_fullsized":"2024-04-30T08:33:59.698707+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:19:11.826470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886774+0000","last_change":"2024-04-30T08:32:31.586667+0000","last_active":"2024-04-30T08:33:31.886774+0000","last_peered":"2024-04-30T08:33:31.886774+0000","last_clean":"2024-04-30T08:33:31.886774+0000","last_became_active":"2024-04-30T08:32:31.586144+0000","last_became_peered":"2024-04-30T08:32:31.586144+0000","last_unstale":"2024-04-30T08:33:31.886774+0000","last_undegraded":"2024-04-30T08:33:31.886774+0000","last_fullsized":"2024-04-30T08:33:31.886774+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:50:39.752485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7,1],"acting":[8,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"81'5","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881669+0000","last_change":"2024-04-30T08:31:51.708943+0000","last_active":"2024-04-30T08:33:31.881669+0000","last_peered":"2024-04-30T08:33:31.881669+0000","last_clean":"2024-04-30T08:33:31.881669+0000","last_became_active":"2024-04-30T08:31:51.708376+0000","last_became_peered":"2024-04-30T08:31:51.708376+0000","last_unstale":"2024-04-30T08:33:31.881669+0000","last_undegraded":"2024-04-30T08:33:31.881669+0000","last_fullsized":"2024-04-30T08:33:31.881669+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:14:46.692192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"85'9","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.690244+0000","last_change":"2024-04-30T08:31:52.686634+0000","last_active":"2024-04-30T08:33:59.690244+0000","last_peered":"2024-04-30T08:33:59.690244+0000","last_clean":"2024-04-30T08:33:59.690244+0000","last_became_active":"2024-04-30T08:31:52.686231+0000","last_became_peered":"2024-04-30T08:31:52.686231+0000","last_unstale":"2024-04-30T08:33:59.690244+0000","last_undegraded":"2024-04-30T08:33:59.690244+0000","last_fullsized":"2024-04-30T08:33:59.690244+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:27:31.073041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"85'9","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.689734+0000","last_change":"2024-04-30T08:31:52.688695+0000","last_active":"2024-04-30T08:33:59.689734+0000","last_peered":"2024-04-30T08:33:59.689734+0000","last_clean":"2024-04-30T08:33:59.689734+0000","last_became_active":"2024-04-30T08:31:52.686556+0000","last_became_peered":"2024-04-30T08:31:52.686556+0000","last_unstale":"2024-04-30T08:33:59.689734+0000","last_undegraded":"2024-04-30T08:33:59.689734+0000","last_fullsized":"2024-04-30T08:33:59.689734+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:22:48.976440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"85'12","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.683087+0000","last_change":"2024-04-30T08:31:52.688324+0000","last_active":"2024-04-30T08:33:59.683087+0000","last_peered":"2024-04-30T08:33:59.683087+0000","last_clean":"2024-04-30T08:33:59.683087+0000","last_became_active":"2024-04-30T08:31:52.687966+0000","last_became_peered":"2024-04-30T08:31:52.687966+0000","last_unstale":"2024-04-30T08:33:59.683087+0000","last_undegraded":"2024-04-30T08:33:59.683087+0000","last_fullsized":"2024-04-30T08:33:59.683087+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:35:38.237750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"85'8","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.706258+0000","last_change":"2024-04-30T08:31:52.695638+0000","last_active":"2024-04-30T08:33:59.706258+0000","last_peered":"2024-04-30T08:33:59.706258+0000","last_clean":"2024-04-30T08:33:59.706258+0000","last_became_active":"2024-04-30T08:31:52.695283+0000","last_became_peered":"2024-04-30T08:31:52.695283+0000","last_unstale":"2024-04-30T08:33:59.706258+0000","last_undegraded":"2024-04-30T08:33:59.706258+0000","last_fullsized":"2024-04-30T08:33:59.706258+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:00:14.510381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"85'7","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.739539+0000","last_change":"2024-04-30T08:31:52.719112+0000","last_active":"2024-04-30T08:33:59.739539+0000","last_peered":"2024-04-30T08:33:59.739539+0000","last_clean":"2024-04-30T08:33:59.739539+0000","last_became_active":"2024-04-30T08:31:52.718207+0000","last_became_peered":"2024-04-30T08:31:52.718207+0000","last_unstale":"2024-04-30T08:33:59.739539+0000","last_undegraded":"2024-04-30T08:33:59.739539+0000","last_fullsized":"2024-04-30T08:33:59.739539+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:27:29.263962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"85'4","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.734901+0000","last_change":"2024-04-30T08:31:52.712008+0000","last_active":"2024-04-30T08:33:59.734901+0000","last_peered":"2024-04-30T08:33:59.734901+0000","last_clean":"2024-04-30T08:33:59.734901+0000","last_became_active":"2024-04-30T08:31:52.711064+0000","last_became_peered":"2024-04-30T08:31:52.711064+0000","last_unstale":"2024-04-30T08:33:59.734901+0000","last_undegraded":"2024-04-30T08:33:59.734901+0000","last_fullsized":"2024-04-30T08:33:59.734901+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:57:27.400602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"85'6","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.688657+0000","last_change":"2024-04-30T08:31:52.691210+0000","last_active":"2024-04-30T08:33:59.688657+0000","last_peered":"2024-04-30T08:33:59.688657+0000","last_clean":"2024-04-30T08:33:59.688657+0000","last_became_active":"2024-04-30T08:31:52.688852+0000","last_became_peered":"2024-04-30T08:31:52.688852+0000","last_unstale":"2024-04-30T08:33:59.688657+0000","last_undegraded":"2024-04-30T08:33:59.688657+0000","last_fullsized":"2024-04-30T08:33:59.688657+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:04:46.502743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"85'15","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.684401+0000","last_change":"2024-04-30T08:31:52.684501+0000","last_active":"2024-04-30T08:33:59.684401+0000","last_peered":"2024-04-30T08:33:59.684401+0000","last_clean":"2024-04-30T08:33:59.684401+0000","last_became_active":"2024-04-30T08:31:52.684122+0000","last_became_peered":"2024-04-30T08:31:52.684122+0000","last_unstale":"2024-04-30T08:33:59.684401+0000","last_undegraded":"2024-04-30T08:33:59.684401+0000","last_fullsized":"2024-04-30T08:33:59.684401+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:55:58.093269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"85'2","reported_seq":34,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.713723+0000","last_change":"2024-04-30T08:31:52.698016+0000","last_active":"2024-04-30T08:33:59.713723+0000","last_peered":"2024-04-30T08:33:59.713723+0000","last_clean":"2024-04-30T08:33:59.713723+0000","last_became_active":"2024-04-30T08:31:52.695037+0000","last_became_peered":"2024-04-30T08:31:52.695037+0000","last_unstale":"2024-04-30T08:33:59.713723+0000","last_undegraded":"2024-04-30T08:33:59.713723+0000","last_fullsized":"2024-04-30T08:33:59.713723+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:06:01.360947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"85'6","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.714903+0000","last_change":"2024-04-30T08:33:31.889445+0000","last_active":"2024-04-30T08:33:59.714903+0000","last_peered":"2024-04-30T08:33:59.714903+0000","last_clean":"2024-04-30T08:33:59.714903+0000","last_became_active":"2024-04-30T08:33:31.889110+0000","last_became_peered":"2024-04-30T08:33:31.889110+0000","last_unstale":"2024-04-30T08:33:59.714903+0000","last_undegraded":"2024-04-30T08:33:59.714903+0000","last_fullsized":"2024-04-30T08:33:59.714903+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:34:26.741854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.29","version":"85'8","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.699538+0000","last_change":"2024-04-30T08:31:52.694776+0000","last_active":"2024-04-30T08:33:59.699538+0000","last_peered":"2024-04-30T08:33:59.699538+0000","last_clean":"2024-04-30T08:33:59.699538+0000","last_became_active":"2024-04-30T08:31:52.693532+0000","last_became_peered":"2024-04-30T08:31:52.693532+0000","last_unstale":"2024-04-30T08:33:59.699538+0000","last_undegraded":"2024-04-30T08:33:59.699538+0000","last_fullsized":"2024-04-30T08:33:59.699538+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:31:10.186667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"85'10","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.706242+0000","last_change":"2024-04-30T08:31:52.688477+0000","last_active":"2024-04-30T08:33:59.706242+0000","last_peered":"2024-04-30T08:33:59.706242+0000","last_clean":"2024-04-30T08:33:59.706242+0000","last_became_active":"2024-04-30T08:31:52.688072+0000","last_became_peered":"2024-04-30T08:31:52.688072+0000","last_unstale":"2024-04-30T08:33:59.706242+0000","last_undegraded":"2024-04-30T08:33:59.706242+0000","last_fullsized":"2024-04-30T08:33:59.706242+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:35:42.659707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"85'11","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.738258+0000","last_change":"2024-04-30T08:31:52.716788+0000","last_active":"2024-04-30T08:33:59.738258+0000","last_peered":"2024-04-30T08:33:59.738258+0000","last_clean":"2024-04-30T08:33:59.738258+0000","last_became_active":"2024-04-30T08:31:52.716336+0000","last_became_peered":"2024-04-30T08:31:52.716336+0000","last_unstale":"2024-04-30T08:33:59.738258+0000","last_undegraded":"2024-04-30T08:33:59.738258+0000","last_fullsized":"2024-04-30T08:33:59.738258+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:23:23.477968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"85'8","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.681192+0000","last_change":"2024-04-30T08:31:52.684571+0000","last_active":"2024-04-30T08:33:59.681192+0000","last_peered":"2024-04-30T08:33:59.681192+0000","last_clean":"2024-04-30T08:33:59.681192+0000","last_became_active":"2024-04-30T08:31:52.684192+0000","last_became_peered":"2024-04-30T08:31:52.684192+0000","last_unstale":"2024-04-30T08:33:59.681192+0000","last_undegraded":"2024-04-30T08:33:59.681192+0000","last_fullsized":"2024-04-30T08:33:59.681192+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:05:57.520133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284927,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":75914,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1631,"num_read_kb":902,"num_write":266,"num_write_kb":163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":230262,"data_compressed":5523,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":230,"ondisk_log_size":230,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678071,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":946176,"data_stored":479251,"data_compressed":26652,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874299,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1089536,"data_stored":1134568,"data_compressed":64719,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037826,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":483209,"data_compressed":26652,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234058,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":479491,"data_compressed":26652,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430286,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":1077753,"data_compressed":62878,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626515,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":937984,"data_stored":475008,"data_compressed":26652,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574962,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":1007616,"data_stored":545894,"data_compressed":28493,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738486,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":476073,"data_compressed":26652,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411437,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1019904,"data_stored":543667,"data_compressed":28493,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182502,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":474325,"data_compressed":26652,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986272,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":479924,"data_compressed":26652,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822746,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":1071711,"data_compressed":62878,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34599,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34599,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34599,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":71999,"data_compressed":1841,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":69772,"data_compressed":1841,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":70305,"data_compressed":1841,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-30T08:35:13.433 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-30T08:35:13.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:13 smithi071 ceph-mon[33385]: pgmap v426: 129 pgs: 129 active+clean; 651 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-04-30T08:35:13.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:13 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2595876329' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-30T08:35:13.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:13 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3347504630' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:35:13.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:13 smithi189 ceph-mon[33038]: pgmap v426: 129 pgs: 129 active+clean; 651 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-04-30T08:35:13.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:13 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2595876329' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-30T08:35:13.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:13 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3347504630' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:35:13.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.759+0000 7f9d63200640 1 -- 172.21.15.32:0/3800691684 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d641100f0 msgr2=0x7f9d64112510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:13.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.759+0000 7f9d63200640 1 --2- 172.21.15.32:0/3800691684 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d641100f0 0x7f9d64112510 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c009be0 tx=0x7f9d5c02a6c0 comp rx=0 tx=0).stop 2024-04-30T08:35:13.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.759+0000 7f9d63200640 1 -- 172.21.15.32:0/3800691684 shutdown_connections 2024-04-30T08:35:13.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.759+0000 7f9d63200640 1 --2- 172.21.15.32:0/3800691684 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d641100f0 0x7f9d64112510 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:13.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.759+0000 7f9d63200640 1 --2- 172.21.15.32:0/3800691684 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d64075960 0x7f9d6410fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:13.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.759+0000 7f9d63200640 1 --2- 172.21.15.32:0/3800691684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d64075010 0x7f9d64075420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:13.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.759+0000 7f9d63200640 1 -- 172.21.15.32:0/3800691684 >> 172.21.15.32:0/3800691684 conn(0x7f9d640feba0 msgr2=0x7f9d64101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:13.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.759+0000 7f9d63200640 1 -- 172.21.15.32:0/3800691684 shutdown_connections 2024-04-30T08:35:13.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.759+0000 7f9d63200640 1 -- 172.21.15.32:0/3800691684 wait complete. 2024-04-30T08:35:13.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.760+0000 7f9d63200640 1 Processor -- start 2024-04-30T08:35:13.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.761+0000 7f9d63200640 1 -- start start 2024-04-30T08:35:13.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.761+0000 7f9d63200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d64075010 0x7f9d641a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:13.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.761+0000 7f9d63200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d64075960 0x7f9d641a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:13.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.761+0000 7f9d63200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d641100f0 0x7f9d641a7b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:13.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.761+0000 7f9d63200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f9d64115ad0 con 0x7f9d641100f0 2024-04-30T08:35:13.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.761+0000 7f9d63200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f9d64115950 con 0x7f9d64075960 2024-04-30T08:35:13.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.761+0000 7f9d63200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f9d64115c50 con 0x7f9d64075010 2024-04-30T08:35:13.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.762+0000 7f9d61400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d64075960 0x7f9d641a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:13.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.762+0000 7f9d61400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d64075960 0x7f9d641a0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:57276/0 (socket says 172.21.15.32:57276) 2024-04-30T08:35:13.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.762+0000 7f9d61e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d64075010 0x7f9d641a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:13.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.762+0000 7f9d62800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d641100f0 0x7f9d641a7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:13.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.762+0000 7f9d61400640 1 -- 172.21.15.32:0/876089231 learned_addr learned my addr 172.21.15.32:0/876089231 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:13.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.762+0000 7f9d61400640 1 -- 172.21.15.32:0/876089231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d64075010 msgr2=0x7f9d641a0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:13.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.762+0000 7f9d61400640 1 --2- 172.21.15.32:0/876089231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d64075010 0x7f9d641a0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:13.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.762+0000 7f9d61400640 1 -- 172.21.15.32:0/876089231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d641100f0 msgr2=0x7f9d641a7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:13.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.762+0000 7f9d61400640 1 --2- 172.21.15.32:0/876089231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d641100f0 0x7f9d641a7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:13.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.762+0000 7f9d61400640 1 -- 172.21.15.32:0/876089231 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d641a81a0 con 0x7f9d64075960 2024-04-30T08:35:13.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.762+0000 7f9d61400640 1 --2- 172.21.15.32:0/876089231 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d64075960 0x7f9d641a0ba0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9d5000e9f0 tx=0x7f9d5000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:13.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.763+0000 7f9d5aa00640 1 -- 172.21.15.32:0/876089231 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d5000cd80 con 0x7f9d64075960 2024-04-30T08:35:13.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.763+0000 7f9d63200640 1 -- 172.21.15.32:0/876089231 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d641a8490 con 0x7f9d64075960 2024-04-30T08:35:13.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.763+0000 7f9d5aa00640 1 -- 172.21.15.32:0/876089231 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9d500108e0 con 0x7f9d64075960 2024-04-30T08:35:13.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.763+0000 7f9d5aa00640 1 -- 172.21.15.32:0/876089231 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d50018980 con 0x7f9d64075960 2024-04-30T08:35:13.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.763+0000 7f9d63200640 1 -- 172.21.15.32:0/876089231 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d641a89e0 con 0x7f9d64075960 2024-04-30T08:35:13.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.764+0000 7f9d63200640 1 -- 172.21.15.32:0/876089231 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d64105c70 con 0x7f9d64075960 2024-04-30T08:35:13.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.767+0000 7f9d5aa00640 1 -- 172.21.15.32:0/876089231 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f9d50010430 con 0x7f9d64075960 2024-04-30T08:35:13.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.768+0000 7f9d5aa00640 1 --2- 172.21.15.32:0/876089231 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f9d480afb70 0x7f9d480b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:13.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.768+0000 7f9d5aa00640 1 -- 172.21.15.32:0/876089231 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f9d500df9a0 con 0x7f9d64075960 2024-04-30T08:35:13.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.769+0000 7f9d61e00640 1 --2- 172.21.15.32:0/876089231 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f9d480afb70 0x7f9d480b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:13.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.769+0000 7f9d61e00640 1 --2- 172.21.15.32:0/876089231 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f9d480afb70 0x7f9d480b2080 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9d54009530 tx=0x7f9d54009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:13.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.770+0000 7f9d5aa00640 1 -- 172.21.15.32:0/876089231 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9d500a79c0 con 0x7f9d64075960 2024-04-30T08:35:13.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:13.979+0000 7f9d63200640 1 -- 172.21.15.32:0/876089231 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f9d64063060 con 0x7f9d480afb70 2024-04-30T08:35:14.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.009+0000 7f9d5aa00640 1 -- 172.21.15.32:0/876089231 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340836 (secure 0 0 0) 0x7f9d64063060 con 0x7f9d480afb70 2024-04-30T08:35:14.015 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:35:14.017 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-30T08:35:14.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.016+0000 7f9d63200640 1 -- 172.21.15.32:0/876089231 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f9d480afb70 msgr2=0x7f9d480b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:14.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.016+0000 7f9d63200640 1 --2- 172.21.15.32:0/876089231 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f9d480afb70 0x7f9d480b2080 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f9d54009530 tx=0x7f9d54009260 comp rx=0 tx=0).stop 2024-04-30T08:35:14.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.016+0000 7f9d63200640 1 -- 172.21.15.32:0/876089231 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d64075960 msgr2=0x7f9d641a0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:14.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.016+0000 7f9d63200640 1 --2- 172.21.15.32:0/876089231 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d64075960 0x7f9d641a0ba0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f9d5000e9f0 tx=0x7f9d5000eeb0 comp rx=0 tx=0).stop 2024-04-30T08:35:14.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.016+0000 7f9d63200640 1 -- 172.21.15.32:0/876089231 shutdown_connections 2024-04-30T08:35:14.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.016+0000 7f9d63200640 1 --2- 172.21.15.32:0/876089231 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f9d480afb70 0x7f9d480b2080 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:14.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.016+0000 7f9d63200640 1 --2- 172.21.15.32:0/876089231 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f9d641100f0 0x7f9d641a7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:14.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.016+0000 7f9d63200640 1 --2- 172.21.15.32:0/876089231 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f9d64075960 0x7f9d641a0ba0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:14.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.016+0000 7f9d63200640 1 --2- 172.21.15.32:0/876089231 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d64075010 0x7f9d641a0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:14.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.016+0000 7f9d63200640 1 -- 172.21.15.32:0/876089231 >> 172.21.15.32:0/876089231 conn(0x7f9d640feba0 msgr2=0x7f9d64102e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:14.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.017+0000 7f9d63200640 1 -- 172.21.15.32:0/876089231 shutdown_connections 2024-04-30T08:35:14.019 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.017+0000 7f9d63200640 1 -- 172.21.15.32:0/876089231 wait complete. 2024-04-30T08:35:14.046 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":427,"stamp":"2024-04-30T08:35:13.795418+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":666516,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1722,"num_read_kb":966,"num_write":391,"num_write_kb":2315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":2951908,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":824,"ondisk_log_size":824,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":373172,"kb_used_data":11540,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124568652,"statfs":{"total":1151940427776,"available":1151558299648,"internally_reserved":0,"allocated":11816960,"data_stored":7720874,"data_compressed":434025,"data_compressed_allocated":589824,"data_compressed_original":2678784,"omap_allocated":19141,"internal_metadata":370193723},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6107,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":120,"num_read_kb":59,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005967"},"pg_stats":[{"pgid":"3.3f","version":"85'8","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.737715+0000","last_change":"2024-04-30T08:31:52.711443+0000","last_active":"2024-04-30T08:33:59.737715+0000","last_peered":"2024-04-30T08:33:59.737715+0000","last_clean":"2024-04-30T08:33:59.737715+0000","last_became_active":"2024-04-30T08:31:52.711029+0000","last_became_peered":"2024-04-30T08:31:52.711029+0000","last_unstale":"2024-04-30T08:33:59.737715+0000","last_undegraded":"2024-04-30T08:33:59.737715+0000","last_fullsized":"2024-04-30T08:33:59.737715+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:51:15.208528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"85'4","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.736806+0000","last_change":"2024-04-30T08:31:52.712324+0000","last_active":"2024-04-30T08:33:59.736806+0000","last_peered":"2024-04-30T08:33:59.736806+0000","last_clean":"2024-04-30T08:33:59.736806+0000","last_became_active":"2024-04-30T08:31:52.711923+0000","last_became_peered":"2024-04-30T08:31:52.711923+0000","last_unstale":"2024-04-30T08:33:59.736806+0000","last_undegraded":"2024-04-30T08:33:59.736806+0000","last_fullsized":"2024-04-30T08:33:59.736806+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:29:55.071459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"85'7","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.735218+0000","last_change":"2024-04-30T08:31:52.712936+0000","last_active":"2024-04-30T08:33:59.735218+0000","last_peered":"2024-04-30T08:33:59.735218+0000","last_clean":"2024-04-30T08:33:59.735218+0000","last_became_active":"2024-04-30T08:31:52.712585+0000","last_became_peered":"2024-04-30T08:31:52.712585+0000","last_unstale":"2024-04-30T08:33:59.735218+0000","last_undegraded":"2024-04-30T08:33:59.735218+0000","last_fullsized":"2024-04-30T08:33:59.735218+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:13:43.745561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"85'8","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.688422+0000","last_change":"2024-04-30T08:31:52.686880+0000","last_active":"2024-04-30T08:33:59.688422+0000","last_peered":"2024-04-30T08:33:59.688422+0000","last_clean":"2024-04-30T08:33:59.688422+0000","last_became_active":"2024-04-30T08:31:52.686506+0000","last_became_peered":"2024-04-30T08:31:52.686506+0000","last_unstale":"2024-04-30T08:33:59.688422+0000","last_undegraded":"2024-04-30T08:33:59.688422+0000","last_fullsized":"2024-04-30T08:33:59.688422+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:16:33.390951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"85'4","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.716502+0000","last_change":"2024-04-30T08:31:52.706014+0000","last_active":"2024-04-30T08:33:59.716502+0000","last_peered":"2024-04-30T08:33:59.716502+0000","last_clean":"2024-04-30T08:33:59.716502+0000","last_became_active":"2024-04-30T08:31:52.705582+0000","last_became_peered":"2024-04-30T08:31:52.705582+0000","last_unstale":"2024-04-30T08:33:59.716502+0000","last_undegraded":"2024-04-30T08:33:59.716502+0000","last_fullsized":"2024-04-30T08:33:59.716502+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:24:49.804592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"85'6","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.717350+0000","last_change":"2024-04-30T08:31:52.690462+0000","last_active":"2024-04-30T08:33:59.717350+0000","last_peered":"2024-04-30T08:33:59.717350+0000","last_clean":"2024-04-30T08:33:59.717350+0000","last_became_active":"2024-04-30T08:31:52.690092+0000","last_became_peered":"2024-04-30T08:31:52.690092+0000","last_unstale":"2024-04-30T08:33:59.717350+0000","last_undegraded":"2024-04-30T08:33:59.717350+0000","last_fullsized":"2024-04-30T08:33:59.717350+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:32:49.769383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"85'5","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.704595+0000","last_change":"2024-04-30T08:31:52.701786+0000","last_active":"2024-04-30T08:33:59.704595+0000","last_peered":"2024-04-30T08:33:59.704595+0000","last_clean":"2024-04-30T08:33:59.704595+0000","last_became_active":"2024-04-30T08:31:52.701402+0000","last_became_peered":"2024-04-30T08:31:52.701402+0000","last_unstale":"2024-04-30T08:33:59.704595+0000","last_undegraded":"2024-04-30T08:33:59.704595+0000","last_fullsized":"2024-04-30T08:33:59.704595+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:58:06.153479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"85'9","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.719088+0000","last_change":"2024-04-30T08:31:52.698037+0000","last_active":"2024-04-30T08:33:59.719088+0000","last_peered":"2024-04-30T08:33:59.719088+0000","last_clean":"2024-04-30T08:33:59.719088+0000","last_became_active":"2024-04-30T08:31:52.697646+0000","last_became_peered":"2024-04-30T08:31:52.697646+0000","last_unstale":"2024-04-30T08:33:59.719088+0000","last_undegraded":"2024-04-30T08:33:59.719088+0000","last_fullsized":"2024-04-30T08:33:59.719088+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:28:59.520091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"85'11","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.692082+0000","last_change":"2024-04-30T08:31:52.680377+0000","last_active":"2024-04-30T08:33:59.692082+0000","last_peered":"2024-04-30T08:33:59.692082+0000","last_clean":"2024-04-30T08:33:59.692082+0000","last_became_active":"2024-04-30T08:31:52.679977+0000","last_became_peered":"2024-04-30T08:31:52.679977+0000","last_unstale":"2024-04-30T08:33:59.692082+0000","last_undegraded":"2024-04-30T08:33:59.692082+0000","last_fullsized":"2024-04-30T08:33:59.692082+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:35:22.488450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"85'16","reported_seq":54,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.729297+0000","last_change":"2024-04-30T08:32:31.573546+0000","last_active":"2024-04-30T08:33:59.729297+0000","last_peered":"2024-04-30T08:33:59.729297+0000","last_clean":"2024-04-30T08:33:59.729297+0000","last_became_active":"2024-04-30T08:32:31.573289+0000","last_became_peered":"2024-04-30T08:32:31.573289+0000","last_unstale":"2024-04-30T08:33:59.729297+0000","last_undegraded":"2024-04-30T08:33:59.729297+0000","last_fullsized":"2024-04-30T08:33:59.729297+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:42:45.862326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"85'6","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.692001+0000","last_change":"2024-04-30T08:31:52.690344+0000","last_active":"2024-04-30T08:33:59.692001+0000","last_peered":"2024-04-30T08:33:59.692001+0000","last_clean":"2024-04-30T08:33:59.692001+0000","last_became_active":"2024-04-30T08:31:52.689857+0000","last_became_peered":"2024-04-30T08:31:52.689857+0000","last_unstale":"2024-04-30T08:33:59.692001+0000","last_undegraded":"2024-04-30T08:33:59.692001+0000","last_fullsized":"2024-04-30T08:33:59.692001+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:45:13.395589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"85'7","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.689051+0000","last_change":"2024-04-30T08:31:52.685151+0000","last_active":"2024-04-30T08:33:59.689051+0000","last_peered":"2024-04-30T08:33:59.689051+0000","last_clean":"2024-04-30T08:33:59.689051+0000","last_became_active":"2024-04-30T08:31:52.684722+0000","last_became_peered":"2024-04-30T08:31:52.684722+0000","last_unstale":"2024-04-30T08:33:59.689051+0000","last_undegraded":"2024-04-30T08:33:59.689051+0000","last_fullsized":"2024-04-30T08:33:59.689051+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:14:00.660478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"85'10","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.723211+0000","last_change":"2024-04-30T08:31:52.714294+0000","last_active":"2024-04-30T08:33:59.723211+0000","last_peered":"2024-04-30T08:33:59.723211+0000","last_clean":"2024-04-30T08:33:59.723211+0000","last_became_active":"2024-04-30T08:31:52.710094+0000","last_became_peered":"2024-04-30T08:31:52.710094+0000","last_unstale":"2024-04-30T08:33:59.723211+0000","last_undegraded":"2024-04-30T08:33:59.723211+0000","last_fullsized":"2024-04-30T08:33:59.723211+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:34:44.203093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"85'7","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.718132+0000","last_change":"2024-04-30T08:31:52.707992+0000","last_active":"2024-04-30T08:33:59.718132+0000","last_peered":"2024-04-30T08:33:59.718132+0000","last_clean":"2024-04-30T08:33:59.718132+0000","last_became_active":"2024-04-30T08:31:52.706456+0000","last_became_peered":"2024-04-30T08:31:52.706456+0000","last_unstale":"2024-04-30T08:33:59.718132+0000","last_undegraded":"2024-04-30T08:33:59.718132+0000","last_fullsized":"2024-04-30T08:33:59.718132+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:45:32.717842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"85'7","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.735414+0000","last_change":"2024-04-30T08:31:52.695605+0000","last_active":"2024-04-30T08:33:59.735414+0000","last_peered":"2024-04-30T08:33:59.735414+0000","last_clean":"2024-04-30T08:33:59.735414+0000","last_became_active":"2024-04-30T08:31:52.692439+0000","last_became_peered":"2024-04-30T08:31:52.692439+0000","last_unstale":"2024-04-30T08:33:59.735414+0000","last_undegraded":"2024-04-30T08:33:59.735414+0000","last_fullsized":"2024-04-30T08:33:59.735414+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:39:38.746567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"85'9","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.740363+0000","last_change":"2024-04-30T08:33:32.802311+0000","last_active":"2024-04-30T08:33:59.740363+0000","last_peered":"2024-04-30T08:33:59.740363+0000","last_clean":"2024-04-30T08:33:59.740363+0000","last_became_active":"2024-04-30T08:33:32.801601+0000","last_became_peered":"2024-04-30T08:33:32.801601+0000","last_unstale":"2024-04-30T08:33:59.740363+0000","last_undegraded":"2024-04-30T08:33:59.740363+0000","last_fullsized":"2024-04-30T08:33:59.740363+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:59:34.877824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0],"acting":[9,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"85'8","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.737797+0000","last_change":"2024-04-30T08:31:52.711439+0000","last_active":"2024-04-30T08:33:59.737797+0000","last_peered":"2024-04-30T08:33:59.737797+0000","last_clean":"2024-04-30T08:33:59.737797+0000","last_became_active":"2024-04-30T08:31:52.711023+0000","last_became_peered":"2024-04-30T08:31:52.711023+0000","last_unstale":"2024-04-30T08:33:59.737797+0000","last_undegraded":"2024-04-30T08:33:59.737797+0000","last_fullsized":"2024-04-30T08:33:59.737797+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:31:00.654348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"85'10","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.721531+0000","last_change":"2024-04-30T08:31:52.708476+0000","last_active":"2024-04-30T08:33:59.721531+0000","last_peered":"2024-04-30T08:33:59.721531+0000","last_clean":"2024-04-30T08:33:59.721531+0000","last_became_active":"2024-04-30T08:31:52.704735+0000","last_became_peered":"2024-04-30T08:31:52.704735+0000","last_unstale":"2024-04-30T08:33:59.721531+0000","last_undegraded":"2024-04-30T08:33:59.721531+0000","last_fullsized":"2024-04-30T08:33:59.721531+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:40:00.011783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"85'7","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.720093+0000","last_change":"2024-04-30T08:31:52.704901+0000","last_active":"2024-04-30T08:33:59.720093+0000","last_peered":"2024-04-30T08:33:59.720093+0000","last_clean":"2024-04-30T08:33:59.720093+0000","last_became_active":"2024-04-30T08:31:52.704636+0000","last_became_peered":"2024-04-30T08:31:52.704636+0000","last_unstale":"2024-04-30T08:33:59.720093+0000","last_undegraded":"2024-04-30T08:33:59.720093+0000","last_fullsized":"2024-04-30T08:33:59.720093+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:08:53.580809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.879602+0000","last_change":"2024-04-30T08:31:57.743342+0000","last_active":"2024-04-30T08:33:31.879602+0000","last_peered":"2024-04-30T08:33:31.879602+0000","last_clean":"2024-04-30T08:33:31.879602+0000","last_became_active":"2024-04-30T08:31:57.742850+0000","last_became_peered":"2024-04-30T08:31:57.742850+0000","last_unstale":"2024-04-30T08:33:31.879602+0000","last_undegraded":"2024-04-30T08:33:31.879602+0000","last_fullsized":"2024-04-30T08:33:31.879602+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:59:10.628822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'14","reported_seq":209,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:35:09.926274+0000","last_change":"2024-04-30T08:31:51.694984+0000","last_active":"2024-04-30T08:35:09.926274+0000","last_peered":"2024-04-30T08:35:09.926274+0000","last_clean":"2024-04-30T08:35:09.926274+0000","last_became_active":"2024-04-30T08:31:51.694514+0000","last_became_peered":"2024-04-30T08:31:51.694514+0000","last_unstale":"2024-04-30T08:35:09.926274+0000","last_undegraded":"2024-04-30T08:35:09.926274+0000","last_fullsized":"2024-04-30T08:35:09.926274+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:17:03.873090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":162,"num_read_kb":162,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"85'5","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.717378+0000","last_change":"2024-04-30T08:31:52.694276+0000","last_active":"2024-04-30T08:33:59.717378+0000","last_peered":"2024-04-30T08:33:59.717378+0000","last_clean":"2024-04-30T08:33:59.717378+0000","last_became_active":"2024-04-30T08:31:52.693839+0000","last_became_peered":"2024-04-30T08:31:52.693839+0000","last_unstale":"2024-04-30T08:33:59.717378+0000","last_undegraded":"2024-04-30T08:33:59.717378+0000","last_fullsized":"2024-04-30T08:33:59.717378+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:29:21.271040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.169742+0000","last_change":"2024-04-30T08:31:57.754828+0000","last_active":"2024-04-30T08:33:32.169742+0000","last_peered":"2024-04-30T08:33:32.169742+0000","last_clean":"2024-04-30T08:33:32.169742+0000","last_became_active":"2024-04-30T08:31:57.754322+0000","last_became_peered":"2024-04-30T08:31:57.754322+0000","last_unstale":"2024-04-30T08:33:32.169742+0000","last_undegraded":"2024-04-30T08:33:32.169742+0000","last_fullsized":"2024-04-30T08:33:32.169742+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:36:02.560134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"81'2","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.277887+0000","last_change":"2024-04-30T08:31:51.707637+0000","last_active":"2024-04-30T08:33:32.277887+0000","last_peered":"2024-04-30T08:33:32.277887+0000","last_clean":"2024-04-30T08:33:32.277887+0000","last_became_active":"2024-04-30T08:31:51.707233+0000","last_became_peered":"2024-04-30T08:31:51.707233+0000","last_unstale":"2024-04-30T08:33:32.277887+0000","last_undegraded":"2024-04-30T08:33:32.277887+0000","last_fullsized":"2024-04-30T08:33:32.277887+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:14:46.041169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"85'6","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.741653+0000","last_change":"2024-04-30T08:31:52.711442+0000","last_active":"2024-04-30T08:33:59.741653+0000","last_peered":"2024-04-30T08:33:59.741653+0000","last_clean":"2024-04-30T08:33:59.741653+0000","last_became_active":"2024-04-30T08:31:52.710942+0000","last_became_peered":"2024-04-30T08:31:52.710942+0000","last_unstale":"2024-04-30T08:33:59.741653+0000","last_undegraded":"2024-04-30T08:33:59.741653+0000","last_fullsized":"2024-04-30T08:33:59.741653+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:26:56.428770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.169895+0000","last_change":"2024-04-30T08:31:57.760535+0000","last_active":"2024-04-30T08:33:32.169895+0000","last_peered":"2024-04-30T08:33:32.169895+0000","last_clean":"2024-04-30T08:33:32.169895+0000","last_became_active":"2024-04-30T08:31:57.760073+0000","last_became_peered":"2024-04-30T08:31:57.760073+0000","last_unstale":"2024-04-30T08:33:32.169895+0000","last_undegraded":"2024-04-30T08:33:32.169895+0000","last_fullsized":"2024-04-30T08:33:32.169895+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:03:51.359474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"81'12","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.887094+0000","last_change":"2024-04-30T08:31:51.678575+0000","last_active":"2024-04-30T08:33:31.887094+0000","last_peered":"2024-04-30T08:33:31.887094+0000","last_clean":"2024-04-30T08:33:31.887094+0000","last_became_active":"2024-04-30T08:31:51.678114+0000","last_became_peered":"2024-04-30T08:31:51.678114+0000","last_unstale":"2024-04-30T08:33:31.887094+0000","last_undegraded":"2024-04-30T08:33:31.887094+0000","last_fullsized":"2024-04-30T08:33:31.887094+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:10:35.517183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"85'9","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.721346+0000","last_change":"2024-04-30T08:31:52.704137+0000","last_active":"2024-04-30T08:33:59.721346+0000","last_peered":"2024-04-30T08:33:59.721346+0000","last_clean":"2024-04-30T08:33:59.721346+0000","last_became_active":"2024-04-30T08:31:52.703695+0000","last_became_peered":"2024-04-30T08:31:52.703695+0000","last_unstale":"2024-04-30T08:33:59.721346+0000","last_undegraded":"2024-04-30T08:33:59.721346+0000","last_fullsized":"2024-04-30T08:33:59.721346+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:31:05.461106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.169818+0000","last_change":"2024-04-30T08:31:57.755237+0000","last_active":"2024-04-30T08:33:32.169818+0000","last_peered":"2024-04-30T08:33:32.169818+0000","last_clean":"2024-04-30T08:33:32.169818+0000","last_became_active":"2024-04-30T08:31:57.754438+0000","last_became_peered":"2024-04-30T08:31:57.754438+0000","last_unstale":"2024-04-30T08:33:32.169818+0000","last_undegraded":"2024-04-30T08:33:32.169818+0000","last_fullsized":"2024-04-30T08:33:32.169818+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:20:59.035110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"81'5","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886405+0000","last_change":"2024-04-30T08:31:51.673793+0000","last_active":"2024-04-30T08:33:31.886405+0000","last_peered":"2024-04-30T08:33:31.886405+0000","last_clean":"2024-04-30T08:33:31.886405+0000","last_became_active":"2024-04-30T08:31:51.672703+0000","last_became_peered":"2024-04-30T08:31:51.672703+0000","last_unstale":"2024-04-30T08:33:31.886405+0000","last_undegraded":"2024-04-30T08:33:31.886405+0000","last_fullsized":"2024-04-30T08:33:31.886405+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:13:18.959542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"85'15","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.686983+0000","last_change":"2024-04-30T08:31:52.694320+0000","last_active":"2024-04-30T08:33:59.686983+0000","last_peered":"2024-04-30T08:33:59.686983+0000","last_clean":"2024-04-30T08:33:59.686983+0000","last_became_active":"2024-04-30T08:31:52.694032+0000","last_became_peered":"2024-04-30T08:31:52.694032+0000","last_unstale":"2024-04-30T08:33:59.686983+0000","last_undegraded":"2024-04-30T08:33:59.686983+0000","last_fullsized":"2024-04-30T08:33:59.686983+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:33:12.347369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.880017+0000","last_change":"2024-04-30T08:31:57.739984+0000","last_active":"2024-04-30T08:33:31.880017+0000","last_peered":"2024-04-30T08:33:31.880017+0000","last_clean":"2024-04-30T08:33:31.880017+0000","last_became_active":"2024-04-30T08:31:57.739477+0000","last_became_peered":"2024-04-30T08:31:57.739477+0000","last_unstale":"2024-04-30T08:33:31.880017+0000","last_undegraded":"2024-04-30T08:33:31.880017+0000","last_fullsized":"2024-04-30T08:33:31.880017+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:10:11.325419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'7","reported_seq":207,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:35:11.817603+0000","last_change":"2024-04-30T08:31:51.678792+0000","last_active":"2024-04-30T08:35:11.817603+0000","last_peered":"2024-04-30T08:35:11.817603+0000","last_clean":"2024-04-30T08:35:11.817603+0000","last_became_active":"2024-04-30T08:31:51.678094+0000","last_became_peered":"2024-04-30T08:31:51.678094+0000","last_unstale":"2024-04-30T08:35:11.817603+0000","last_undegraded":"2024-04-30T08:35:11.817603+0000","last_fullsized":"2024-04-30T08:35:11.817603+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:03:17.526016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":166,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"85'7","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.700627+0000","last_change":"2024-04-30T08:32:31.572984+0000","last_active":"2024-04-30T08:33:59.700627+0000","last_peered":"2024-04-30T08:33:59.700627+0000","last_clean":"2024-04-30T08:33:59.700627+0000","last_became_active":"2024-04-30T08:32:31.572536+0000","last_became_peered":"2024-04-30T08:32:31.572536+0000","last_unstale":"2024-04-30T08:33:59.700627+0000","last_undegraded":"2024-04-30T08:33:59.700627+0000","last_fullsized":"2024-04-30T08:33:59.700627+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:46:15.884555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.879844+0000","last_change":"2024-04-30T08:31:57.765047+0000","last_active":"2024-04-30T08:33:31.879844+0000","last_peered":"2024-04-30T08:33:31.879844+0000","last_clean":"2024-04-30T08:33:31.879844+0000","last_became_active":"2024-04-30T08:31:57.764519+0000","last_became_peered":"2024-04-30T08:31:57.764519+0000","last_unstale":"2024-04-30T08:33:31.879844+0000","last_undegraded":"2024-04-30T08:33:31.879844+0000","last_fullsized":"2024-04-30T08:33:31.879844+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:06:16.126396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"81'7","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881627+0000","last_change":"2024-04-30T08:31:51.686403+0000","last_active":"2024-04-30T08:33:31.881627+0000","last_peered":"2024-04-30T08:33:31.881627+0000","last_clean":"2024-04-30T08:33:31.881627+0000","last_became_active":"2024-04-30T08:31:51.686096+0000","last_became_peered":"2024-04-30T08:31:51.686096+0000","last_unstale":"2024-04-30T08:33:31.881627+0000","last_undegraded":"2024-04-30T08:33:31.881627+0000","last_fullsized":"2024-04-30T08:33:31.881627+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:47:05.961930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"85'8","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.698959+0000","last_change":"2024-04-30T08:31:52.698512+0000","last_active":"2024-04-30T08:33:59.698959+0000","last_peered":"2024-04-30T08:33:59.698959+0000","last_clean":"2024-04-30T08:33:59.698959+0000","last_became_active":"2024-04-30T08:31:52.698168+0000","last_became_peered":"2024-04-30T08:31:52.698168+0000","last_unstale":"2024-04-30T08:33:59.698959+0000","last_undegraded":"2024-04-30T08:33:59.698959+0000","last_fullsized":"2024-04-30T08:33:59.698959+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:22:45.094365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886398+0000","last_change":"2024-04-30T08:31:57.761430+0000","last_active":"2024-04-30T08:33:31.886398+0000","last_peered":"2024-04-30T08:33:31.886398+0000","last_clean":"2024-04-30T08:33:31.886398+0000","last_became_active":"2024-04-30T08:31:57.761062+0000","last_became_peered":"2024-04-30T08:31:57.761062+0000","last_unstale":"2024-04-30T08:33:31.886398+0000","last_undegraded":"2024-04-30T08:33:31.886398+0000","last_fullsized":"2024-04-30T08:33:31.886398+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:32:05.403786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"81'3","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.879820+0000","last_change":"2024-04-30T08:31:51.686694+0000","last_active":"2024-04-30T08:33:31.879820+0000","last_peered":"2024-04-30T08:33:31.879820+0000","last_clean":"2024-04-30T08:33:31.879820+0000","last_became_active":"2024-04-30T08:31:51.685773+0000","last_became_peered":"2024-04-30T08:31:51.685773+0000","last_unstale":"2024-04-30T08:33:31.879820+0000","last_undegraded":"2024-04-30T08:33:31.879820+0000","last_fullsized":"2024-04-30T08:33:31.879820+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:44:25.327977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"85'8","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.734178+0000","last_change":"2024-04-30T08:33:31.888698+0000","last_active":"2024-04-30T08:33:59.734178+0000","last_peered":"2024-04-30T08:33:59.734178+0000","last_clean":"2024-04-30T08:33:59.734178+0000","last_became_active":"2024-04-30T08:33:31.887935+0000","last_became_peered":"2024-04-30T08:33:31.887935+0000","last_unstale":"2024-04-30T08:33:59.734178+0000","last_undegraded":"2024-04-30T08:33:59.734178+0000","last_fullsized":"2024-04-30T08:33:59.734178+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:15:40.989004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.799780+0000","last_change":"2024-04-30T08:31:57.757224+0000","last_active":"2024-04-30T08:33:32.799780+0000","last_peered":"2024-04-30T08:33:32.799780+0000","last_clean":"2024-04-30T08:33:32.799780+0000","last_became_active":"2024-04-30T08:31:57.756658+0000","last_became_peered":"2024-04-30T08:31:57.756658+0000","last_unstale":"2024-04-30T08:33:32.799780+0000","last_undegraded":"2024-04-30T08:33:32.799780+0000","last_fullsized":"2024-04-30T08:33:32.799780+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:01:21.309322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"81'3","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273042+0000","last_change":"2024-04-30T08:31:51.678944+0000","last_active":"2024-04-30T08:33:32.273042+0000","last_peered":"2024-04-30T08:33:32.273042+0000","last_clean":"2024-04-30T08:33:32.273042+0000","last_became_active":"2024-04-30T08:31:51.678493+0000","last_became_peered":"2024-04-30T08:31:51.678493+0000","last_unstale":"2024-04-30T08:33:32.273042+0000","last_undegraded":"2024-04-30T08:33:32.273042+0000","last_fullsized":"2024-04-30T08:33:32.273042+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:48:11.111466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"85'11","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.722707+0000","last_change":"2024-04-30T08:31:52.707627+0000","last_active":"2024-04-30T08:33:59.722707+0000","last_peered":"2024-04-30T08:33:59.722707+0000","last_clean":"2024-04-30T08:33:59.722707+0000","last_became_active":"2024-04-30T08:31:52.706441+0000","last_became_peered":"2024-04-30T08:31:52.706441+0000","last_unstale":"2024-04-30T08:33:59.722707+0000","last_undegraded":"2024-04-30T08:33:59.722707+0000","last_fullsized":"2024-04-30T08:33:59.722707+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:15:45.768927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.275247+0000","last_change":"2024-04-30T08:31:57.740324+0000","last_active":"2024-04-30T08:33:32.275247+0000","last_peered":"2024-04-30T08:33:32.275247+0000","last_clean":"2024-04-30T08:33:32.275247+0000","last_became_active":"2024-04-30T08:31:57.739464+0000","last_became_peered":"2024-04-30T08:31:57.739464+0000","last_unstale":"2024-04-30T08:33:32.275247+0000","last_undegraded":"2024-04-30T08:33:32.275247+0000","last_fullsized":"2024-04-30T08:33:32.275247+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:11:37.395545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"85'5","reported_seq":356,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:35:08.812181+0000","last_change":"2024-04-30T08:31:51.695283+0000","last_active":"2024-04-30T08:35:08.812181+0000","last_peered":"2024-04-30T08:35:08.812181+0000","last_clean":"2024-04-30T08:35:08.812181+0000","last_became_active":"2024-04-30T08:31:51.694848+0000","last_became_peered":"2024-04-30T08:31:51.694848+0000","last_unstale":"2024-04-30T08:35:08.812181+0000","last_undegraded":"2024-04-30T08:35:08.812181+0000","last_fullsized":"2024-04-30T08:35:08.812181+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:41:38.638034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":317,"num_read_kb":317,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"85'10","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.729523+0000","last_change":"2024-04-30T08:31:52.693594+0000","last_active":"2024-04-30T08:33:59.729523+0000","last_peered":"2024-04-30T08:33:59.729523+0000","last_clean":"2024-04-30T08:33:59.729523+0000","last_became_active":"2024-04-30T08:31:52.693199+0000","last_became_peered":"2024-04-30T08:31:52.693199+0000","last_unstale":"2024-04-30T08:33:59.729523+0000","last_undegraded":"2024-04-30T08:33:59.729523+0000","last_fullsized":"2024-04-30T08:33:59.729523+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:57:49.908617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.883697+0000","last_change":"2024-04-30T08:31:57.749727+0000","last_active":"2024-04-30T08:33:31.883697+0000","last_peered":"2024-04-30T08:33:31.883697+0000","last_clean":"2024-04-30T08:33:31.883697+0000","last_became_active":"2024-04-30T08:31:57.749267+0000","last_became_peered":"2024-04-30T08:31:57.749267+0000","last_unstale":"2024-04-30T08:33:31.883697+0000","last_undegraded":"2024-04-30T08:33:31.883697+0000","last_fullsized":"2024-04-30T08:33:31.883697+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:26:12.683008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"81'4","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.218677+0000","last_change":"2024-04-30T08:31:51.677398+0000","last_active":"2024-04-30T08:33:32.218677+0000","last_peered":"2024-04-30T08:33:32.218677+0000","last_clean":"2024-04-30T08:33:32.218677+0000","last_became_active":"2024-04-30T08:31:51.677031+0000","last_became_peered":"2024-04-30T08:31:51.677031+0000","last_unstale":"2024-04-30T08:33:32.218677+0000","last_undegraded":"2024-04-30T08:33:32.218677+0000","last_fullsized":"2024-04-30T08:33:32.218677+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:24:25.154229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"85'9","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.741480+0000","last_change":"2024-04-30T08:32:31.561554+0000","last_active":"2024-04-30T08:33:59.741480+0000","last_peered":"2024-04-30T08:33:59.741480+0000","last_clean":"2024-04-30T08:33:59.741480+0000","last_became_active":"2024-04-30T08:32:31.560253+0000","last_became_peered":"2024-04-30T08:32:31.560253+0000","last_unstale":"2024-04-30T08:33:59.741480+0000","last_undegraded":"2024-04-30T08:33:59.741480+0000","last_fullsized":"2024-04-30T08:33:59.741480+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:59:08.602282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.170073+0000","last_change":"2024-04-30T08:32:31.586746+0000","last_active":"2024-04-30T08:33:32.170073+0000","last_peered":"2024-04-30T08:33:32.170073+0000","last_clean":"2024-04-30T08:33:32.170073+0000","last_became_active":"2024-04-30T08:32:31.586356+0000","last_became_peered":"2024-04-30T08:32:31.586356+0000","last_unstale":"2024-04-30T08:33:32.170073+0000","last_undegraded":"2024-04-30T08:33:32.170073+0000","last_fullsized":"2024-04-30T08:33:32.170073+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:22:39.371508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"81'5","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273483+0000","last_change":"2024-04-30T08:31:51.694424+0000","last_active":"2024-04-30T08:33:32.273483+0000","last_peered":"2024-04-30T08:33:32.273483+0000","last_clean":"2024-04-30T08:33:32.273483+0000","last_became_active":"2024-04-30T08:31:51.694081+0000","last_became_peered":"2024-04-30T08:31:51.694081+0000","last_unstale":"2024-04-30T08:33:32.273483+0000","last_undegraded":"2024-04-30T08:33:32.273483+0000","last_fullsized":"2024-04-30T08:33:32.273483+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:27:33.328477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"85'11","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.692022+0000","last_change":"2024-04-30T08:31:52.684739+0000","last_active":"2024-04-30T08:33:59.692022+0000","last_peered":"2024-04-30T08:33:59.692022+0000","last_clean":"2024-04-30T08:33:59.692022+0000","last_became_active":"2024-04-30T08:31:52.684316+0000","last_became_peered":"2024-04-30T08:31:52.684316+0000","last_unstale":"2024-04-30T08:33:59.692022+0000","last_undegraded":"2024-04-30T08:33:59.692022+0000","last_fullsized":"2024-04-30T08:33:59.692022+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:12:05.437493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881452+0000","last_change":"2024-04-30T08:32:31.557578+0000","last_active":"2024-04-30T08:33:31.881452+0000","last_peered":"2024-04-30T08:33:31.881452+0000","last_clean":"2024-04-30T08:33:31.881452+0000","last_became_active":"2024-04-30T08:32:31.557115+0000","last_became_peered":"2024-04-30T08:32:31.557115+0000","last_unstale":"2024-04-30T08:33:31.881452+0000","last_undegraded":"2024-04-30T08:33:31.881452+0000","last_fullsized":"2024-04-30T08:33:31.881452+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:41:33.225879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"85'40","reported_seq":400,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:35:08.812186+0000","last_change":"2024-04-30T08:31:51.676223+0000","last_active":"2024-04-30T08:35:08.812186+0000","last_peered":"2024-04-30T08:35:08.812186+0000","last_clean":"2024-04-30T08:35:08.812186+0000","last_became_active":"2024-04-30T08:31:51.675830+0000","last_became_peered":"2024-04-30T08:31:51.675830+0000","last_unstale":"2024-04-30T08:35:08.812186+0000","last_undegraded":"2024-04-30T08:35:08.812186+0000","last_fullsized":"2024-04-30T08:35:08.812186+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:28:49.497719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67336,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":326,"num_read_kb":229,"num_write":42,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"85'6","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.684496+0000","last_change":"2024-04-30T08:31:52.686892+0000","last_active":"2024-04-30T08:33:59.684496+0000","last_peered":"2024-04-30T08:33:59.684496+0000","last_clean":"2024-04-30T08:33:59.684496+0000","last_became_active":"2024-04-30T08:31:52.686533+0000","last_became_peered":"2024-04-30T08:31:52.686533+0000","last_unstale":"2024-04-30T08:33:59.684496+0000","last_undegraded":"2024-04-30T08:33:59.684496+0000","last_fullsized":"2024-04-30T08:33:59.684496+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:50:18.836161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.274017+0000","last_change":"2024-04-30T08:31:57.752865+0000","last_active":"2024-04-30T08:33:32.274017+0000","last_peered":"2024-04-30T08:33:32.274017+0000","last_clean":"2024-04-30T08:33:32.274017+0000","last_became_active":"2024-04-30T08:31:57.752385+0000","last_became_peered":"2024-04-30T08:31:57.752385+0000","last_unstale":"2024-04-30T08:33:32.274017+0000","last_undegraded":"2024-04-30T08:33:32.274017+0000","last_fullsized":"2024-04-30T08:33:32.274017+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:15:15.315580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"81'9","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886273+0000","last_change":"2024-04-30T08:32:31.585282+0000","last_active":"2024-04-30T08:33:31.886273+0000","last_peered":"2024-04-30T08:33:31.886273+0000","last_clean":"2024-04-30T08:33:31.886273+0000","last_became_active":"2024-04-30T08:32:31.571676+0000","last_became_peered":"2024-04-30T08:32:31.571676+0000","last_unstale":"2024-04-30T08:33:31.886273+0000","last_undegraded":"2024-04-30T08:33:31.886273+0000","last_fullsized":"2024-04-30T08:33:31.886273+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:18:29.910201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"85'9","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.723531+0000","last_change":"2024-04-30T08:31:52.706259+0000","last_active":"2024-04-30T08:33:59.723531+0000","last_peered":"2024-04-30T08:33:59.723531+0000","last_clean":"2024-04-30T08:33:59.723531+0000","last_became_active":"2024-04-30T08:31:52.705620+0000","last_became_peered":"2024-04-30T08:31:52.705620+0000","last_unstale":"2024-04-30T08:33:59.723531+0000","last_undegraded":"2024-04-30T08:33:59.723531+0000","last_fullsized":"2024-04-30T08:33:59.723531+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:22:29.816950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886632+0000","last_change":"2024-04-30T08:32:31.562571+0000","last_active":"2024-04-30T08:33:31.886632+0000","last_peered":"2024-04-30T08:33:31.886632+0000","last_clean":"2024-04-30T08:33:31.886632+0000","last_became_active":"2024-04-30T08:32:31.562012+0000","last_became_peered":"2024-04-30T08:32:31.562012+0000","last_unstale":"2024-04-30T08:33:31.886632+0000","last_undegraded":"2024-04-30T08:33:31.886632+0000","last_fullsized":"2024-04-30T08:33:31.886632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:12:22.928286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"81'5","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273150+0000","last_change":"2024-04-30T08:31:51.701122+0000","last_active":"2024-04-30T08:33:32.273150+0000","last_peered":"2024-04-30T08:33:32.273150+0000","last_clean":"2024-04-30T08:33:32.273150+0000","last_became_active":"2024-04-30T08:31:51.700815+0000","last_became_peered":"2024-04-30T08:31:51.700815+0000","last_unstale":"2024-04-30T08:33:32.273150+0000","last_undegraded":"2024-04-30T08:33:32.273150+0000","last_fullsized":"2024-04-30T08:33:32.273150+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:33:12.714979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"85'15","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.721587+0000","last_change":"2024-04-30T08:31:52.716043+0000","last_active":"2024-04-30T08:33:59.721587+0000","last_peered":"2024-04-30T08:33:59.721587+0000","last_clean":"2024-04-30T08:33:59.721587+0000","last_became_active":"2024-04-30T08:31:52.715284+0000","last_became_peered":"2024-04-30T08:31:52.715284+0000","last_unstale":"2024-04-30T08:33:59.721587+0000","last_undegraded":"2024-04-30T08:33:59.721587+0000","last_fullsized":"2024-04-30T08:33:59.721587+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:00:01.863606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.885198+0000","last_change":"2024-04-30T08:31:57.765008+0000","last_active":"2024-04-30T08:33:31.885198+0000","last_peered":"2024-04-30T08:33:31.885198+0000","last_clean":"2024-04-30T08:33:31.885198+0000","last_became_active":"2024-04-30T08:31:57.762958+0000","last_became_peered":"2024-04-30T08:31:57.762958+0000","last_unstale":"2024-04-30T08:33:31.885198+0000","last_undegraded":"2024-04-30T08:33:31.885198+0000","last_fullsized":"2024-04-30T08:33:31.885198+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:48:47.116004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"85'9","reported_seq":211,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:35:11.955731+0000","last_change":"2024-04-30T08:31:51.676681+0000","last_active":"2024-04-30T08:35:11.955731+0000","last_peered":"2024-04-30T08:35:11.955731+0000","last_clean":"2024-04-30T08:35:11.955731+0000","last_became_active":"2024-04-30T08:31:51.676311+0000","last_became_peered":"2024-04-30T08:31:51.676311+0000","last_unstale":"2024-04-30T08:35:11.955731+0000","last_undegraded":"2024-04-30T08:35:11.955731+0000","last_fullsized":"2024-04-30T08:35:11.955731+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:24:49.757427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1821,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":169,"num_read_kb":6,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"85'6","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.687636+0000","last_change":"2024-04-30T08:31:52.689418+0000","last_active":"2024-04-30T08:33:59.687636+0000","last_peered":"2024-04-30T08:33:59.687636+0000","last_clean":"2024-04-30T08:33:59.687636+0000","last_became_active":"2024-04-30T08:31:52.688840+0000","last_became_peered":"2024-04-30T08:31:52.688840+0000","last_unstale":"2024-04-30T08:33:59.687636+0000","last_undegraded":"2024-04-30T08:33:59.687636+0000","last_fullsized":"2024-04-30T08:33:59.687636+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:40:05.781137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881271+0000","last_change":"2024-04-30T08:31:57.765875+0000","last_active":"2024-04-30T08:33:31.881271+0000","last_peered":"2024-04-30T08:33:31.881271+0000","last_clean":"2024-04-30T08:33:31.881271+0000","last_became_active":"2024-04-30T08:31:57.765432+0000","last_became_peered":"2024-04-30T08:31:57.765432+0000","last_unstale":"2024-04-30T08:33:31.881271+0000","last_undegraded":"2024-04-30T08:33:31.881271+0000","last_fullsized":"2024-04-30T08:33:31.881271+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:24:44.329532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"81'17","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.887076+0000","last_change":"2024-04-30T08:31:51.695007+0000","last_active":"2024-04-30T08:33:31.887076+0000","last_peered":"2024-04-30T08:33:31.887076+0000","last_clean":"2024-04-30T08:33:31.887076+0000","last_became_active":"2024-04-30T08:31:51.694533+0000","last_became_peered":"2024-04-30T08:31:51.694533+0000","last_unstale":"2024-04-30T08:33:31.887076+0000","last_undegraded":"2024-04-30T08:33:31.887076+0000","last_fullsized":"2024-04-30T08:33:31.887076+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:47:15.104172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"85'6","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.687169+0000","last_change":"2024-04-30T08:31:52.692274+0000","last_active":"2024-04-30T08:33:59.687169+0000","last_peered":"2024-04-30T08:33:59.687169+0000","last_clean":"2024-04-30T08:33:59.687169+0000","last_became_active":"2024-04-30T08:31:52.691809+0000","last_became_peered":"2024-04-30T08:31:52.691809+0000","last_unstale":"2024-04-30T08:33:59.687169+0000","last_undegraded":"2024-04-30T08:33:59.687169+0000","last_fullsized":"2024-04-30T08:33:59.687169+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:50:00.201782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273166+0000","last_change":"2024-04-30T08:31:57.732383+0000","last_active":"2024-04-30T08:33:32.273166+0000","last_peered":"2024-04-30T08:33:32.273166+0000","last_clean":"2024-04-30T08:33:32.273166+0000","last_became_active":"2024-04-30T08:31:57.731893+0000","last_became_peered":"2024-04-30T08:31:57.731893+0000","last_unstale":"2024-04-30T08:33:32.273166+0000","last_undegraded":"2024-04-30T08:33:32.273166+0000","last_fullsized":"2024-04-30T08:33:32.273166+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:08:55.243526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'4","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:35:04.537295+0000","last_change":"2024-04-30T08:31:51.671991+0000","last_active":"2024-04-30T08:35:04.537295+0000","last_peered":"2024-04-30T08:35:04.537295+0000","last_clean":"2024-04-30T08:35:04.537295+0000","last_became_active":"2024-04-30T08:31:51.671508+0000","last_became_peered":"2024-04-30T08:31:51.671508+0000","last_unstale":"2024-04-30T08:35:04.537295+0000","last_undegraded":"2024-04-30T08:35:04.537295+0000","last_fullsized":"2024-04-30T08:35:04.537295+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:54:30.106748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"85'5","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.689682+0000","last_change":"2024-04-30T08:33:32.274901+0000","last_active":"2024-04-30T08:33:59.689682+0000","last_peered":"2024-04-30T08:33:59.689682+0000","last_clean":"2024-04-30T08:33:59.689682+0000","last_became_active":"2024-04-30T08:33:32.274401+0000","last_became_peered":"2024-04-30T08:33:32.274401+0000","last_unstale":"2024-04-30T08:33:59.689682+0000","last_undegraded":"2024-04-30T08:33:59.689682+0000","last_fullsized":"2024-04-30T08:33:59.689682+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:23:08.703092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886092+0000","last_change":"2024-04-30T08:31:58.061077+0000","last_active":"2024-04-30T08:33:31.886092+0000","last_peered":"2024-04-30T08:33:31.886092+0000","last_clean":"2024-04-30T08:33:31.886092+0000","last_became_active":"2024-04-30T08:31:58.060486+0000","last_became_peered":"2024-04-30T08:31:58.060486+0000","last_unstale":"2024-04-30T08:33:31.886092+0000","last_undegraded":"2024-04-30T08:33:31.886092+0000","last_fullsized":"2024-04-30T08:33:31.886092+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:31:50.936560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"81'1","reported_seq":34,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.885199+0000","last_change":"2024-04-30T08:31:51.702869+0000","last_active":"2024-04-30T08:33:31.885199+0000","last_peered":"2024-04-30T08:33:31.885199+0000","last_clean":"2024-04-30T08:33:31.885199+0000","last_became_active":"2024-04-30T08:31:51.701150+0000","last_became_peered":"2024-04-30T08:31:51.701150+0000","last_unstale":"2024-04-30T08:33:31.885199+0000","last_undegraded":"2024-04-30T08:33:31.885199+0000","last_fullsized":"2024-04-30T08:33:31.885199+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:47:11.010310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"85'4","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.716233+0000","last_change":"2024-04-30T08:31:52.709159+0000","last_active":"2024-04-30T08:33:59.716233+0000","last_peered":"2024-04-30T08:33:59.716233+0000","last_clean":"2024-04-30T08:33:59.716233+0000","last_became_active":"2024-04-30T08:31:52.708799+0000","last_became_peered":"2024-04-30T08:31:52.708799+0000","last_unstale":"2024-04-30T08:33:59.716233+0000","last_undegraded":"2024-04-30T08:33:59.716233+0000","last_fullsized":"2024-04-30T08:33:59.716233+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:51:10.588599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881244+0000","last_change":"2024-04-30T08:31:57.751652+0000","last_active":"2024-04-30T08:33:31.881244+0000","last_peered":"2024-04-30T08:33:31.881244+0000","last_clean":"2024-04-30T08:33:31.881244+0000","last_became_active":"2024-04-30T08:31:57.751176+0000","last_became_peered":"2024-04-30T08:31:57.751176+0000","last_unstale":"2024-04-30T08:33:31.881244+0000","last_undegraded":"2024-04-30T08:33:31.881244+0000","last_fullsized":"2024-04-30T08:33:31.881244+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:13:24.528842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"85'6","reported_seq":205,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:35:11.931645+0000","last_change":"2024-04-30T08:31:51.700816+0000","last_active":"2024-04-30T08:35:11.931645+0000","last_peered":"2024-04-30T08:35:11.931645+0000","last_clean":"2024-04-30T08:35:11.931645+0000","last_became_active":"2024-04-30T08:31:51.700470+0000","last_became_peered":"2024-04-30T08:31:51.700470+0000","last_unstale":"2024-04-30T08:35:11.931645+0000","last_undegraded":"2024-04-30T08:35:11.931645+0000","last_fullsized":"2024-04-30T08:35:11.931645+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:49:11.793412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":165,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"85'7","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.726100+0000","last_change":"2024-04-30T08:31:52.715236+0000","last_active":"2024-04-30T08:33:59.726100+0000","last_peered":"2024-04-30T08:33:59.726100+0000","last_clean":"2024-04-30T08:33:59.726100+0000","last_became_active":"2024-04-30T08:31:52.714801+0000","last_became_peered":"2024-04-30T08:31:52.714801+0000","last_unstale":"2024-04-30T08:33:59.726100+0000","last_undegraded":"2024-04-30T08:33:59.726100+0000","last_fullsized":"2024-04-30T08:33:59.726100+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:43:09.650002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881702+0000","last_change":"2024-04-30T08:31:58.060927+0000","last_active":"2024-04-30T08:33:31.881702+0000","last_peered":"2024-04-30T08:33:31.881702+0000","last_clean":"2024-04-30T08:33:31.881702+0000","last_became_active":"2024-04-30T08:31:58.060182+0000","last_became_peered":"2024-04-30T08:31:58.060182+0000","last_unstale":"2024-04-30T08:33:31.881702+0000","last_undegraded":"2024-04-30T08:33:31.881702+0000","last_fullsized":"2024-04-30T08:33:31.881702+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:21:00.228064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"85'8","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.679643+0000","last_change":"2024-04-30T08:32:31.560485+0000","last_active":"2024-04-30T08:33:59.679643+0000","last_peered":"2024-04-30T08:33:59.679643+0000","last_clean":"2024-04-30T08:33:59.679643+0000","last_became_active":"2024-04-30T08:32:31.560066+0000","last_became_peered":"2024-04-30T08:32:31.560066+0000","last_unstale":"2024-04-30T08:33:59.679643+0000","last_undegraded":"2024-04-30T08:33:59.679643+0000","last_fullsized":"2024-04-30T08:33:59.679643+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:13:40.769761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'7","reported_seq":201,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:35:08.947184+0000","last_change":"2024-04-30T08:31:51.702808+0000","last_active":"2024-04-30T08:35:08.947184+0000","last_peered":"2024-04-30T08:35:08.947184+0000","last_clean":"2024-04-30T08:35:08.947184+0000","last_became_active":"2024-04-30T08:31:51.702326+0000","last_became_peered":"2024-04-30T08:31:51.702326+0000","last_unstale":"2024-04-30T08:35:08.947184+0000","last_undegraded":"2024-04-30T08:35:08.947184+0000","last_fullsized":"2024-04-30T08:35:08.947184+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:11:35.552670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":161,"num_read_kb":161,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"81'1","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.218503+0000","last_change":"2024-04-30T08:31:57.760359+0000","last_active":"2024-04-30T08:33:32.218503+0000","last_peered":"2024-04-30T08:33:32.218503+0000","last_clean":"2024-04-30T08:33:32.218503+0000","last_became_active":"2024-04-30T08:31:57.759821+0000","last_became_peered":"2024-04-30T08:31:57.759821+0000","last_unstale":"2024-04-30T08:33:32.218503+0000","last_undegraded":"2024-04-30T08:33:32.218503+0000","last_fullsized":"2024-04-30T08:33:32.218503+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:50:17.853791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"85'8","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.719174+0000","last_change":"2024-04-30T08:31:52.689281+0000","last_active":"2024-04-30T08:33:59.719174+0000","last_peered":"2024-04-30T08:33:59.719174+0000","last_clean":"2024-04-30T08:33:59.719174+0000","last_became_active":"2024-04-30T08:31:52.688834+0000","last_became_peered":"2024-04-30T08:31:52.688834+0000","last_unstale":"2024-04-30T08:33:59.719174+0000","last_undegraded":"2024-04-30T08:33:59.719174+0000","last_fullsized":"2024-04-30T08:33:59.719174+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:42:12.836649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'6","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.169972+0000","last_change":"2024-04-30T08:31:51.683712+0000","last_active":"2024-04-30T08:33:32.169972+0000","last_peered":"2024-04-30T08:33:32.169972+0000","last_clean":"2024-04-30T08:33:32.169972+0000","last_became_active":"2024-04-30T08:31:51.683298+0000","last_became_peered":"2024-04-30T08:31:51.683298+0000","last_unstale":"2024-04-30T08:33:32.169972+0000","last_undegraded":"2024-04-30T08:33:32.169972+0000","last_fullsized":"2024-04-30T08:33:32.169972+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:43:08.832104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":109,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.217885+0000","last_change":"2024-04-30T08:30:26.503221+0000","last_active":"2024-04-30T08:33:32.217885+0000","last_peered":"2024-04-30T08:33:32.217885+0000","last_clean":"2024-04-30T08:33:32.217885+0000","last_became_active":"2024-04-30T08:30:26.498399+0000","last_became_peered":"2024-04-30T08:30:26.498399+0000","last_unstale":"2024-04-30T08:33:32.217885+0000","last_undegraded":"2024-04-30T08:33:32.217885+0000","last_fullsized":"2024-04-30T08:33:32.217885+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:26:20.170075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:26:20.170075+0000","last_clean_scrub_stamp":"2024-04-30T08:26:20.170075+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:34:44.067560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273554+0000","last_change":"2024-04-30T08:31:58.061283+0000","last_active":"2024-04-30T08:33:32.273554+0000","last_peered":"2024-04-30T08:33:32.273554+0000","last_clean":"2024-04-30T08:33:32.273554+0000","last_became_active":"2024-04-30T08:31:58.060650+0000","last_became_peered":"2024-04-30T08:31:58.060650+0000","last_unstale":"2024-04-30T08:33:32.273554+0000","last_undegraded":"2024-04-30T08:33:32.273554+0000","last_fullsized":"2024-04-30T08:33:32.273554+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:13:01.811462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"85'5","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.728420+0000","last_change":"2024-04-30T08:31:52.699639+0000","last_active":"2024-04-30T08:33:59.728420+0000","last_peered":"2024-04-30T08:33:59.728420+0000","last_clean":"2024-04-30T08:33:59.728420+0000","last_became_active":"2024-04-30T08:31:52.699164+0000","last_became_peered":"2024-04-30T08:31:52.699164+0000","last_unstale":"2024-04-30T08:33:59.728420+0000","last_undegraded":"2024-04-30T08:33:59.728420+0000","last_fullsized":"2024-04-30T08:33:59.728420+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:22:18.873504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"81'4","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.217821+0000","last_change":"2024-04-30T08:31:51.679153+0000","last_active":"2024-04-30T08:33:32.217821+0000","last_peered":"2024-04-30T08:33:32.217821+0000","last_clean":"2024-04-30T08:33:32.217821+0000","last_became_active":"2024-04-30T08:31:51.678348+0000","last_became_peered":"2024-04-30T08:31:51.678348+0000","last_unstale":"2024-04-30T08:33:32.217821+0000","last_undegraded":"2024-04-30T08:33:32.217821+0000","last_fullsized":"2024-04-30T08:33:32.217821+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:02:56.240218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886616+0000","last_change":"2024-04-30T08:31:57.755423+0000","last_active":"2024-04-30T08:33:31.886616+0000","last_peered":"2024-04-30T08:33:31.886616+0000","last_clean":"2024-04-30T08:33:31.886616+0000","last_became_active":"2024-04-30T08:31:57.754835+0000","last_became_peered":"2024-04-30T08:31:57.754835+0000","last_unstale":"2024-04-30T08:33:31.886616+0000","last_undegraded":"2024-04-30T08:33:31.886616+0000","last_fullsized":"2024-04-30T08:33:31.886616+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:19:25.168920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"81'4","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886919+0000","last_change":"2024-04-30T08:31:51.693839+0000","last_active":"2024-04-30T08:33:31.886919+0000","last_peered":"2024-04-30T08:33:31.886919+0000","last_clean":"2024-04-30T08:33:31.886919+0000","last_became_active":"2024-04-30T08:31:51.693349+0000","last_became_peered":"2024-04-30T08:31:51.693349+0000","last_unstale":"2024-04-30T08:33:31.886919+0000","last_undegraded":"2024-04-30T08:33:31.886919+0000","last_fullsized":"2024-04-30T08:33:31.886919+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:41:51.330901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"85'9","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.733847+0000","last_change":"2024-04-30T08:31:52.712360+0000","last_active":"2024-04-30T08:33:59.733847+0000","last_peered":"2024-04-30T08:33:59.733847+0000","last_clean":"2024-04-30T08:33:59.733847+0000","last_became_active":"2024-04-30T08:31:52.711652+0000","last_became_peered":"2024-04-30T08:31:52.711652+0000","last_unstale":"2024-04-30T08:33:59.733847+0000","last_undegraded":"2024-04-30T08:33:59.733847+0000","last_fullsized":"2024-04-30T08:33:59.733847+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:21:10.285386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.276738+0000","last_change":"2024-04-30T08:31:58.061654+0000","last_active":"2024-04-30T08:33:32.276738+0000","last_peered":"2024-04-30T08:33:32.276738+0000","last_clean":"2024-04-30T08:33:32.276738+0000","last_became_active":"2024-04-30T08:31:58.060181+0000","last_became_peered":"2024-04-30T08:31:58.060181+0000","last_unstale":"2024-04-30T08:33:32.276738+0000","last_undegraded":"2024-04-30T08:33:32.276738+0000","last_fullsized":"2024-04-30T08:33:32.276738+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:46:03.230626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'15","reported_seq":48,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:35:04.265838+0000","last_change":"2024-04-30T08:31:51.706940+0000","last_active":"2024-04-30T08:35:04.265838+0000","last_peered":"2024-04-30T08:35:04.265838+0000","last_clean":"2024-04-30T08:35:04.265838+0000","last_became_active":"2024-04-30T08:31:51.706481+0000","last_became_peered":"2024-04-30T08:31:51.706481+0000","last_unstale":"2024-04-30T08:35:04.265838+0000","last_undegraded":"2024-04-30T08:35:04.265838+0000","last_fullsized":"2024-04-30T08:35:04.265838+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:53:31.955595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"85'5","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.714966+0000","last_change":"2024-04-30T08:31:52.696625+0000","last_active":"2024-04-30T08:33:59.714966+0000","last_peered":"2024-04-30T08:33:59.714966+0000","last_clean":"2024-04-30T08:33:59.714966+0000","last_became_active":"2024-04-30T08:31:52.696230+0000","last_became_peered":"2024-04-30T08:31:52.696230+0000","last_unstale":"2024-04-30T08:33:59.714966+0000","last_undegraded":"2024-04-30T08:33:59.714966+0000","last_fullsized":"2024-04-30T08:33:59.714966+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:11:46.404251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.879436+0000","last_change":"2024-04-30T08:31:57.738329+0000","last_active":"2024-04-30T08:33:31.879436+0000","last_peered":"2024-04-30T08:33:31.879436+0000","last_clean":"2024-04-30T08:33:31.879436+0000","last_became_active":"2024-04-30T08:31:57.737897+0000","last_became_peered":"2024-04-30T08:31:57.737897+0000","last_unstale":"2024-04-30T08:33:31.879436+0000","last_undegraded":"2024-04-30T08:33:31.879436+0000","last_fullsized":"2024-04-30T08:33:31.879436+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:02:43.074578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"81'7","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.170374+0000","last_change":"2024-04-30T08:31:51.684066+0000","last_active":"2024-04-30T08:33:32.170374+0000","last_peered":"2024-04-30T08:33:32.170374+0000","last_clean":"2024-04-30T08:33:32.170374+0000","last_became_active":"2024-04-30T08:31:51.683699+0000","last_became_peered":"2024-04-30T08:31:51.683699+0000","last_unstale":"2024-04-30T08:33:32.170374+0000","last_undegraded":"2024-04-30T08:33:32.170374+0000","last_fullsized":"2024-04-30T08:33:32.170374+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:36:20.900733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"85'7","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.740363+0000","last_change":"2024-04-30T08:31:52.715219+0000","last_active":"2024-04-30T08:33:59.740363+0000","last_peered":"2024-04-30T08:33:59.740363+0000","last_clean":"2024-04-30T08:33:59.740363+0000","last_became_active":"2024-04-30T08:31:52.714774+0000","last_became_peered":"2024-04-30T08:31:52.714774+0000","last_unstale":"2024-04-30T08:33:59.740363+0000","last_undegraded":"2024-04-30T08:33:59.740363+0000","last_fullsized":"2024-04-30T08:33:59.740363+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:42:41.082610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.170157+0000","last_change":"2024-04-30T08:31:58.060845+0000","last_active":"2024-04-30T08:33:32.170157+0000","last_peered":"2024-04-30T08:33:32.170157+0000","last_clean":"2024-04-30T08:33:32.170157+0000","last_became_active":"2024-04-30T08:31:58.060365+0000","last_became_peered":"2024-04-30T08:31:58.060365+0000","last_unstale":"2024-04-30T08:33:32.170157+0000","last_undegraded":"2024-04-30T08:33:32.170157+0000","last_fullsized":"2024-04-30T08:33:32.170157+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:21:39.583243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"81'2","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.272616+0000","last_change":"2024-04-30T08:31:51.694833+0000","last_active":"2024-04-30T08:33:32.272616+0000","last_peered":"2024-04-30T08:33:32.272616+0000","last_clean":"2024-04-30T08:33:32.272616+0000","last_became_active":"2024-04-30T08:31:51.694401+0000","last_became_peered":"2024-04-30T08:31:51.694401+0000","last_unstale":"2024-04-30T08:33:32.272616+0000","last_undegraded":"2024-04-30T08:33:32.272616+0000","last_fullsized":"2024-04-30T08:33:32.272616+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:38:52.605625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"85'2","reported_seq":33,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.676614+0000","last_change":"2024-04-30T08:31:52.686444+0000","last_active":"2024-04-30T08:33:59.676614+0000","last_peered":"2024-04-30T08:33:59.676614+0000","last_clean":"2024-04-30T08:33:59.676614+0000","last_became_active":"2024-04-30T08:31:52.686057+0000","last_became_peered":"2024-04-30T08:31:52.686057+0000","last_unstale":"2024-04-30T08:33:59.676614+0000","last_undegraded":"2024-04-30T08:33:59.676614+0000","last_fullsized":"2024-04-30T08:33:59.676614+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:54:29.824060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.885416+0000","last_change":"2024-04-30T08:31:57.766193+0000","last_active":"2024-04-30T08:33:31.885416+0000","last_peered":"2024-04-30T08:33:31.885416+0000","last_clean":"2024-04-30T08:33:31.885416+0000","last_became_active":"2024-04-30T08:31:57.765758+0000","last_became_peered":"2024-04-30T08:31:57.765758+0000","last_unstale":"2024-04-30T08:33:31.885416+0000","last_undegraded":"2024-04-30T08:33:31.885416+0000","last_fullsized":"2024-04-30T08:33:31.885416+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:59:57.282795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"81'7","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.275654+0000","last_change":"2024-04-30T08:31:51.704565+0000","last_active":"2024-04-30T08:33:32.275654+0000","last_peered":"2024-04-30T08:33:32.275654+0000","last_clean":"2024-04-30T08:33:32.275654+0000","last_became_active":"2024-04-30T08:31:51.703954+0000","last_became_peered":"2024-04-30T08:31:51.703954+0000","last_unstale":"2024-04-30T08:33:32.275654+0000","last_undegraded":"2024-04-30T08:33:32.275654+0000","last_fullsized":"2024-04-30T08:33:32.275654+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:03:00.517867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"85'9","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.717404+0000","last_change":"2024-04-30T08:31:52.706043+0000","last_active":"2024-04-30T08:33:59.717404+0000","last_peered":"2024-04-30T08:33:59.717404+0000","last_clean":"2024-04-30T08:33:59.717404+0000","last_became_active":"2024-04-30T08:31:52.705678+0000","last_became_peered":"2024-04-30T08:31:52.705678+0000","last_unstale":"2024-04-30T08:33:59.717404+0000","last_undegraded":"2024-04-30T08:33:59.717404+0000","last_fullsized":"2024-04-30T08:33:59.717404+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:44:56.787835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"82'1","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.170123+0000","last_change":"2024-04-30T08:31:57.764791+0000","last_active":"2024-04-30T08:33:32.170123+0000","last_peered":"2024-04-30T08:33:32.170123+0000","last_clean":"2024-04-30T08:33:32.170123+0000","last_became_active":"2024-04-30T08:31:57.764368+0000","last_became_peered":"2024-04-30T08:31:57.764368+0000","last_unstale":"2024-04-30T08:33:32.170123+0000","last_undegraded":"2024-04-30T08:33:32.170123+0000","last_fullsized":"2024-04-30T08:33:32.170123+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:19:28.545483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"81'3","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886461+0000","last_change":"2024-04-30T08:32:31.577309+0000","last_active":"2024-04-30T08:33:31.886461+0000","last_peered":"2024-04-30T08:33:31.886461+0000","last_clean":"2024-04-30T08:33:31.886461+0000","last_became_active":"2024-04-30T08:32:31.571515+0000","last_became_peered":"2024-04-30T08:32:31.571515+0000","last_unstale":"2024-04-30T08:33:31.886461+0000","last_undegraded":"2024-04-30T08:33:31.886461+0000","last_fullsized":"2024-04-30T08:33:31.886461+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:28:48.408414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"85'8","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.730222+0000","last_change":"2024-04-30T08:31:52.716707+0000","last_active":"2024-04-30T08:33:59.730222+0000","last_peered":"2024-04-30T08:33:59.730222+0000","last_clean":"2024-04-30T08:33:59.730222+0000","last_became_active":"2024-04-30T08:31:52.716293+0000","last_became_peered":"2024-04-30T08:31:52.716293+0000","last_unstale":"2024-04-30T08:33:59.730222+0000","last_undegraded":"2024-04-30T08:33:59.730222+0000","last_fullsized":"2024-04-30T08:33:59.730222+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:29:54.223472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.170159+0000","last_change":"2024-04-30T08:31:58.060640+0000","last_active":"2024-04-30T08:33:32.170159+0000","last_peered":"2024-04-30T08:33:32.170159+0000","last_clean":"2024-04-30T08:33:32.170159+0000","last_became_active":"2024-04-30T08:31:58.060061+0000","last_became_peered":"2024-04-30T08:31:58.060061+0000","last_unstale":"2024-04-30T08:33:32.170159+0000","last_undegraded":"2024-04-30T08:33:32.170159+0000","last_fullsized":"2024-04-30T08:33:32.170159+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:46:00.844191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"81'4","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886171+0000","last_change":"2024-04-30T08:31:51.679532+0000","last_active":"2024-04-30T08:33:31.886171+0000","last_peered":"2024-04-30T08:33:31.886171+0000","last_clean":"2024-04-30T08:33:31.886171+0000","last_became_active":"2024-04-30T08:31:51.679141+0000","last_became_peered":"2024-04-30T08:31:51.679141+0000","last_unstale":"2024-04-30T08:33:31.886171+0000","last_undegraded":"2024-04-30T08:33:31.886171+0000","last_fullsized":"2024-04-30T08:33:31.886171+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:19:50.526563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"85'5","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.720554+0000","last_change":"2024-04-30T08:31:52.703613+0000","last_active":"2024-04-30T08:33:59.720554+0000","last_peered":"2024-04-30T08:33:59.720554+0000","last_clean":"2024-04-30T08:33:59.720554+0000","last_became_active":"2024-04-30T08:31:52.703196+0000","last_became_peered":"2024-04-30T08:31:52.703196+0000","last_unstale":"2024-04-30T08:33:59.720554+0000","last_undegraded":"2024-04-30T08:33:59.720554+0000","last_fullsized":"2024-04-30T08:33:59.720554+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:33:11.136313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.272631+0000","last_change":"2024-04-30T08:31:57.755286+0000","last_active":"2024-04-30T08:33:32.272631+0000","last_peered":"2024-04-30T08:33:32.272631+0000","last_clean":"2024-04-30T08:33:32.272631+0000","last_became_active":"2024-04-30T08:31:57.754786+0000","last_became_peered":"2024-04-30T08:31:57.754786+0000","last_unstale":"2024-04-30T08:33:32.272631+0000","last_undegraded":"2024-04-30T08:33:32.272631+0000","last_fullsized":"2024-04-30T08:33:32.272631+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:02:50.931553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"81'5","reported_seq":198,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:35:07.811337+0000","last_change":"2024-04-30T08:31:51.693317+0000","last_active":"2024-04-30T08:35:07.811337+0000","last_peered":"2024-04-30T08:35:07.811337+0000","last_clean":"2024-04-30T08:35:07.811337+0000","last_became_active":"2024-04-30T08:31:51.692928+0000","last_became_peered":"2024-04-30T08:31:51.692928+0000","last_unstale":"2024-04-30T08:35:07.811337+0000","last_undegraded":"2024-04-30T08:35:07.811337+0000","last_fullsized":"2024-04-30T08:35:07.811337+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:35:09.851735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":160,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"85'7","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.692096+0000","last_change":"2024-04-30T08:31:52.694304+0000","last_active":"2024-04-30T08:33:59.692096+0000","last_peered":"2024-04-30T08:33:59.692096+0000","last_clean":"2024-04-30T08:33:59.692096+0000","last_became_active":"2024-04-30T08:31:52.693317+0000","last_became_peered":"2024-04-30T08:31:52.693317+0000","last_unstale":"2024-04-30T08:33:59.692096+0000","last_undegraded":"2024-04-30T08:33:59.692096+0000","last_fullsized":"2024-04-30T08:33:59.692096+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:31:39.294864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.799022+0000","last_change":"2024-04-30T08:31:57.763598+0000","last_active":"2024-04-30T08:33:32.799022+0000","last_peered":"2024-04-30T08:33:32.799022+0000","last_clean":"2024-04-30T08:33:32.799022+0000","last_became_active":"2024-04-30T08:31:57.762307+0000","last_became_peered":"2024-04-30T08:31:57.762307+0000","last_unstale":"2024-04-30T08:33:32.799022+0000","last_undegraded":"2024-04-30T08:33:32.799022+0000","last_fullsized":"2024-04-30T08:33:32.799022+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:24:13.390595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"81'3","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273810+0000","last_change":"2024-04-30T08:31:51.703476+0000","last_active":"2024-04-30T08:33:32.273810+0000","last_peered":"2024-04-30T08:33:32.273810+0000","last_clean":"2024-04-30T08:33:32.273810+0000","last_became_active":"2024-04-30T08:31:51.703072+0000","last_became_peered":"2024-04-30T08:31:51.703072+0000","last_unstale":"2024-04-30T08:33:32.273810+0000","last_undegraded":"2024-04-30T08:33:32.273810+0000","last_fullsized":"2024-04-30T08:33:32.273810+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:07:21.098223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"85'6","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.698707+0000","last_change":"2024-04-30T08:31:52.699223+0000","last_active":"2024-04-30T08:33:59.698707+0000","last_peered":"2024-04-30T08:33:59.698707+0000","last_clean":"2024-04-30T08:33:59.698707+0000","last_became_active":"2024-04-30T08:31:52.698837+0000","last_became_peered":"2024-04-30T08:31:52.698837+0000","last_unstale":"2024-04-30T08:33:59.698707+0000","last_undegraded":"2024-04-30T08:33:59.698707+0000","last_fullsized":"2024-04-30T08:33:59.698707+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:19:11.826470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886774+0000","last_change":"2024-04-30T08:32:31.586667+0000","last_active":"2024-04-30T08:33:31.886774+0000","last_peered":"2024-04-30T08:33:31.886774+0000","last_clean":"2024-04-30T08:33:31.886774+0000","last_became_active":"2024-04-30T08:32:31.586144+0000","last_became_peered":"2024-04-30T08:32:31.586144+0000","last_unstale":"2024-04-30T08:33:31.886774+0000","last_undegraded":"2024-04-30T08:33:31.886774+0000","last_fullsized":"2024-04-30T08:33:31.886774+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:50:39.752485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7,1],"acting":[8,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"81'5","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881669+0000","last_change":"2024-04-30T08:31:51.708943+0000","last_active":"2024-04-30T08:33:31.881669+0000","last_peered":"2024-04-30T08:33:31.881669+0000","last_clean":"2024-04-30T08:33:31.881669+0000","last_became_active":"2024-04-30T08:31:51.708376+0000","last_became_peered":"2024-04-30T08:31:51.708376+0000","last_unstale":"2024-04-30T08:33:31.881669+0000","last_undegraded":"2024-04-30T08:33:31.881669+0000","last_fullsized":"2024-04-30T08:33:31.881669+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:14:46.692192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"85'9","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.690244+0000","last_change":"2024-04-30T08:31:52.686634+0000","last_active":"2024-04-30T08:33:59.690244+0000","last_peered":"2024-04-30T08:33:59.690244+0000","last_clean":"2024-04-30T08:33:59.690244+0000","last_became_active":"2024-04-30T08:31:52.686231+0000","last_became_peered":"2024-04-30T08:31:52.686231+0000","last_unstale":"2024-04-30T08:33:59.690244+0000","last_undegraded":"2024-04-30T08:33:59.690244+0000","last_fullsized":"2024-04-30T08:33:59.690244+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:27:31.073041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"85'9","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.689734+0000","last_change":"2024-04-30T08:31:52.688695+0000","last_active":"2024-04-30T08:33:59.689734+0000","last_peered":"2024-04-30T08:33:59.689734+0000","last_clean":"2024-04-30T08:33:59.689734+0000","last_became_active":"2024-04-30T08:31:52.686556+0000","last_became_peered":"2024-04-30T08:31:52.686556+0000","last_unstale":"2024-04-30T08:33:59.689734+0000","last_undegraded":"2024-04-30T08:33:59.689734+0000","last_fullsized":"2024-04-30T08:33:59.689734+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:22:48.976440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"85'12","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.683087+0000","last_change":"2024-04-30T08:31:52.688324+0000","last_active":"2024-04-30T08:33:59.683087+0000","last_peered":"2024-04-30T08:33:59.683087+0000","last_clean":"2024-04-30T08:33:59.683087+0000","last_became_active":"2024-04-30T08:31:52.687966+0000","last_became_peered":"2024-04-30T08:31:52.687966+0000","last_unstale":"2024-04-30T08:33:59.683087+0000","last_undegraded":"2024-04-30T08:33:59.683087+0000","last_fullsized":"2024-04-30T08:33:59.683087+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:35:38.237750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"85'8","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.706258+0000","last_change":"2024-04-30T08:31:52.695638+0000","last_active":"2024-04-30T08:33:59.706258+0000","last_peered":"2024-04-30T08:33:59.706258+0000","last_clean":"2024-04-30T08:33:59.706258+0000","last_became_active":"2024-04-30T08:31:52.695283+0000","last_became_peered":"2024-04-30T08:31:52.695283+0000","last_unstale":"2024-04-30T08:33:59.706258+0000","last_undegraded":"2024-04-30T08:33:59.706258+0000","last_fullsized":"2024-04-30T08:33:59.706258+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:00:14.510381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"85'7","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.739539+0000","last_change":"2024-04-30T08:31:52.719112+0000","last_active":"2024-04-30T08:33:59.739539+0000","last_peered":"2024-04-30T08:33:59.739539+0000","last_clean":"2024-04-30T08:33:59.739539+0000","last_became_active":"2024-04-30T08:31:52.718207+0000","last_became_peered":"2024-04-30T08:31:52.718207+0000","last_unstale":"2024-04-30T08:33:59.739539+0000","last_undegraded":"2024-04-30T08:33:59.739539+0000","last_fullsized":"2024-04-30T08:33:59.739539+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:27:29.263962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"85'4","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.734901+0000","last_change":"2024-04-30T08:31:52.712008+0000","last_active":"2024-04-30T08:33:59.734901+0000","last_peered":"2024-04-30T08:33:59.734901+0000","last_clean":"2024-04-30T08:33:59.734901+0000","last_became_active":"2024-04-30T08:31:52.711064+0000","last_became_peered":"2024-04-30T08:31:52.711064+0000","last_unstale":"2024-04-30T08:33:59.734901+0000","last_undegraded":"2024-04-30T08:33:59.734901+0000","last_fullsized":"2024-04-30T08:33:59.734901+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:57:27.400602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"85'6","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.688657+0000","last_change":"2024-04-30T08:31:52.691210+0000","last_active":"2024-04-30T08:33:59.688657+0000","last_peered":"2024-04-30T08:33:59.688657+0000","last_clean":"2024-04-30T08:33:59.688657+0000","last_became_active":"2024-04-30T08:31:52.688852+0000","last_became_peered":"2024-04-30T08:31:52.688852+0000","last_unstale":"2024-04-30T08:33:59.688657+0000","last_undegraded":"2024-04-30T08:33:59.688657+0000","last_fullsized":"2024-04-30T08:33:59.688657+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:04:46.502743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"85'15","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.684401+0000","last_change":"2024-04-30T08:31:52.684501+0000","last_active":"2024-04-30T08:33:59.684401+0000","last_peered":"2024-04-30T08:33:59.684401+0000","last_clean":"2024-04-30T08:33:59.684401+0000","last_became_active":"2024-04-30T08:31:52.684122+0000","last_became_peered":"2024-04-30T08:31:52.684122+0000","last_unstale":"2024-04-30T08:33:59.684401+0000","last_undegraded":"2024-04-30T08:33:59.684401+0000","last_fullsized":"2024-04-30T08:33:59.684401+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:55:58.093269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"85'2","reported_seq":34,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.713723+0000","last_change":"2024-04-30T08:31:52.698016+0000","last_active":"2024-04-30T08:33:59.713723+0000","last_peered":"2024-04-30T08:33:59.713723+0000","last_clean":"2024-04-30T08:33:59.713723+0000","last_became_active":"2024-04-30T08:31:52.695037+0000","last_became_peered":"2024-04-30T08:31:52.695037+0000","last_unstale":"2024-04-30T08:33:59.713723+0000","last_undegraded":"2024-04-30T08:33:59.713723+0000","last_fullsized":"2024-04-30T08:33:59.713723+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:06:01.360947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"85'6","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.714903+0000","last_change":"2024-04-30T08:33:31.889445+0000","last_active":"2024-04-30T08:33:59.714903+0000","last_peered":"2024-04-30T08:33:59.714903+0000","last_clean":"2024-04-30T08:33:59.714903+0000","last_became_active":"2024-04-30T08:33:31.889110+0000","last_became_peered":"2024-04-30T08:33:31.889110+0000","last_unstale":"2024-04-30T08:33:59.714903+0000","last_undegraded":"2024-04-30T08:33:59.714903+0000","last_fullsized":"2024-04-30T08:33:59.714903+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:34:26.741854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.29","version":"85'8","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.699538+0000","last_change":"2024-04-30T08:31:52.694776+0000","last_active":"2024-04-30T08:33:59.699538+0000","last_peered":"2024-04-30T08:33:59.699538+0000","last_clean":"2024-04-30T08:33:59.699538+0000","last_became_active":"2024-04-30T08:31:52.693532+0000","last_became_peered":"2024-04-30T08:31:52.693532+0000","last_unstale":"2024-04-30T08:33:59.699538+0000","last_undegraded":"2024-04-30T08:33:59.699538+0000","last_fullsized":"2024-04-30T08:33:59.699538+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:31:10.186667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"85'10","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.706242+0000","last_change":"2024-04-30T08:31:52.688477+0000","last_active":"2024-04-30T08:33:59.706242+0000","last_peered":"2024-04-30T08:33:59.706242+0000","last_clean":"2024-04-30T08:33:59.706242+0000","last_became_active":"2024-04-30T08:31:52.688072+0000","last_became_peered":"2024-04-30T08:31:52.688072+0000","last_unstale":"2024-04-30T08:33:59.706242+0000","last_undegraded":"2024-04-30T08:33:59.706242+0000","last_fullsized":"2024-04-30T08:33:59.706242+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:35:42.659707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"85'11","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.738258+0000","last_change":"2024-04-30T08:31:52.716788+0000","last_active":"2024-04-30T08:33:59.738258+0000","last_peered":"2024-04-30T08:33:59.738258+0000","last_clean":"2024-04-30T08:33:59.738258+0000","last_became_active":"2024-04-30T08:31:52.716336+0000","last_became_peered":"2024-04-30T08:31:52.716336+0000","last_unstale":"2024-04-30T08:33:59.738258+0000","last_undegraded":"2024-04-30T08:33:59.738258+0000","last_fullsized":"2024-04-30T08:33:59.738258+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:23:23.477968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"85'8","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.681192+0000","last_change":"2024-04-30T08:31:52.684571+0000","last_active":"2024-04-30T08:33:59.681192+0000","last_peered":"2024-04-30T08:33:59.681192+0000","last_clean":"2024-04-30T08:33:59.681192+0000","last_became_active":"2024-04-30T08:31:52.684192+0000","last_became_peered":"2024-04-30T08:31:52.684192+0000","last_unstale":"2024-04-30T08:33:59.681192+0000","last_undegraded":"2024-04-30T08:33:59.681192+0000","last_fullsized":"2024-04-30T08:33:59.681192+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:05:57.520133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284927,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":75914,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1646,"num_read_kb":902,"num_write":266,"num_write_kb":163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":230262,"data_compressed":5523,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":230,"ondisk_log_size":230,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678071,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":946176,"data_stored":479251,"data_compressed":26652,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874299,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1089536,"data_stored":1134568,"data_compressed":64719,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037826,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":483209,"data_compressed":26652,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234058,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":479491,"data_compressed":26652,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430287,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":1077753,"data_compressed":62878,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626515,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":937984,"data_stored":475008,"data_compressed":26652,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574962,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":1007616,"data_stored":545894,"data_compressed":28493,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738486,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":476073,"data_compressed":26652,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411437,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1019904,"data_stored":543667,"data_compressed":28493,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182502,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":474325,"data_compressed":26652,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986273,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":479924,"data_compressed":26652,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822747,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":1071711,"data_compressed":62878,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34599,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34599,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34599,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":71999,"data_compressed":1841,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":69772,"data_compressed":1841,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":70305,"data_compressed":1841,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-30T08:35:14.051 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-04-30T08:35:14.052 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-30T08:35:14.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.369+0000 7fe8c2200640 1 -- 172.21.15.32:0/940051724 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe8bc106df0 msgr2=0x7fe8bc107200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:14.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.369+0000 7fe8c2200640 1 --2- 172.21.15.32:0/940051724 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe8bc106df0 0x7fe8bc107200 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fe8ac009900 tx=0x7fe8ac02a3f0 comp rx=0 tx=0).stop 2024-04-30T08:35:14.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.370+0000 7fe8c2200640 1 -- 172.21.15.32:0/940051724 shutdown_connections 2024-04-30T08:35:14.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.370+0000 7fe8c2200640 1 --2- 172.21.15.32:0/940051724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8bc10baf0 0x7fe8bc10df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:14.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.370+0000 7fe8c2200640 1 --2- 172.21.15.32:0/940051724 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe8bc106df0 0x7fe8bc107200 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:14.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.370+0000 7fe8c2200640 1 --2- 172.21.15.32:0/940051724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8bc105bb0 0x7fe8bc1068b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:14.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.370+0000 7fe8c2200640 1 -- 172.21.15.32:0/940051724 >> 172.21.15.32:0/940051724 conn(0x7fe8bc100da0 msgr2=0x7fe8bc103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:14.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.370+0000 7fe8c2200640 1 -- 172.21.15.32:0/940051724 shutdown_connections 2024-04-30T08:35:14.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.370+0000 7fe8c2200640 1 -- 172.21.15.32:0/940051724 wait complete. 2024-04-30T08:35:14.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.371+0000 7fe8c2200640 1 Processor -- start 2024-04-30T08:35:14.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.371+0000 7fe8c2200640 1 -- start start 2024-04-30T08:35:14.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.371+0000 7fe8c2200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe8bc105bb0 0x7fe8bc19c290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:14.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.371+0000 7fe8c2200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8bc106df0 0x7fe8bc19c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:14.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.371+0000 7fe8c2200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8bc10baf0 0x7fe8bc1a3760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:14.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.371+0000 7fe8c2200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe8bc1116e0 con 0x7fe8bc10baf0 2024-04-30T08:35:14.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.371+0000 7fe8c2200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe8bc111560 con 0x7fe8bc105bb0 2024-04-30T08:35:14.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.371+0000 7fe8c2200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe8bc111860 con 0x7fe8bc106df0 2024-04-30T08:35:14.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.372+0000 7fe8c1800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8bc10baf0 0x7fe8bc1a3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:14.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.372+0000 7fe8bbe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8bc106df0 0x7fe8bc19c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:14.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.372+0000 7fe8c1800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8bc10baf0 0x7fe8bc1a3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53656/0 (socket says 172.21.15.32:53656) 2024-04-30T08:35:14.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.372+0000 7fe8c1800640 1 -- 172.21.15.32:0/508874663 learned_addr learned my addr 172.21.15.32:0/508874663 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:14.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.372+0000 7fe8c0e00640 1 --2- 172.21.15.32:0/508874663 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe8bc105bb0 0x7fe8bc19c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:14.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.372+0000 7fe8c1800640 1 -- 172.21.15.32:0/508874663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8bc106df0 msgr2=0x7fe8bc19c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:14.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.372+0000 7fe8c1800640 1 --2- 172.21.15.32:0/508874663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8bc106df0 0x7fe8bc19c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:14.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.372+0000 7fe8c1800640 1 -- 172.21.15.32:0/508874663 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe8bc105bb0 msgr2=0x7fe8bc19c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:14.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.372+0000 7fe8c1800640 1 --2- 172.21.15.32:0/508874663 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe8bc105bb0 0x7fe8bc19c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:14.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.372+0000 7fe8c1800640 1 -- 172.21.15.32:0/508874663 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe8bc1a3dd0 con 0x7fe8bc10baf0 2024-04-30T08:35:14.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.373+0000 7fe8c1800640 1 --2- 172.21.15.32:0/508874663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8bc10baf0 0x7fe8bc1a3760 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe8b400eb10 tx=0x7fe8b400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:14.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.373+0000 7fe8b9600640 1 -- 172.21.15.32:0/508874663 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b4004b20 con 0x7fe8bc10baf0 2024-04-30T08:35:14.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.373+0000 7fe8b9600640 1 -- 172.21.15.32:0/508874663 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe8b4004cc0 con 0x7fe8bc10baf0 2024-04-30T08:35:14.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.373+0000 7fe8b9600640 1 -- 172.21.15.32:0/508874663 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe8b4010670 con 0x7fe8bc10baf0 2024-04-30T08:35:14.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.373+0000 7fe8c2200640 1 -- 172.21.15.32:0/508874663 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe8bc1a40c0 con 0x7fe8bc10baf0 2024-04-30T08:35:14.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.373+0000 7fe8c2200640 1 -- 172.21.15.32:0/508874663 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe8bc1a4660 con 0x7fe8bc10baf0 2024-04-30T08:35:14.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.375+0000 7fe8c2200640 1 -- 172.21.15.32:0/508874663 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe8bc10a900 con 0x7fe8bc10baf0 2024-04-30T08:35:14.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.377+0000 7fe8b9600640 1 -- 172.21.15.32:0/508874663 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7fe8b4005000 con 0x7fe8bc10baf0 2024-04-30T08:35:14.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.379+0000 7fe8b9600640 1 --2- 172.21.15.32:0/508874663 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe8a40afbf0 0x7fe8a40b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:14.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.379+0000 7fe8b9600640 1 -- 172.21.15.32:0/508874663 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fe8b40df8a0 con 0x7fe8bc10baf0 2024-04-30T08:35:14.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.379+0000 7fe8c0e00640 1 --2- 172.21.15.32:0/508874663 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe8a40afbf0 0x7fe8a40b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:14.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.380+0000 7fe8c0e00640 1 --2- 172.21.15.32:0/508874663 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe8a40afbf0 0x7fe8a40b2100 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc1067b0 tx=0x7fe8b0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:14.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.381+0000 7fe8b9600640 1 -- 172.21.15.32:0/508874663 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe8b4053ff0 con 0x7fe8bc10baf0 2024-04-30T08:35:14.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:14 smithi032 ceph-mon[28331]: from='client.25567 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:35:14.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:14 smithi071 ceph-mon[33385]: from='client.25567 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:35:14.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:14 smithi189 ceph-mon[33038]: from='client.25567 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:35:14.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.586+0000 7fe8c2200640 1 -- 172.21.15.32:0/508874663 --> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe8bc063060 con 0x7fe8a40afbf0 2024-04-30T08:35:14.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.603+0000 7fe8b9600640 1 -- 172.21.15.32:0/508874663 <== mgr.14152 v2:172.21.15.32:6800/2930853529 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340836 (secure 0 0 0) 0x7fe8bc063060 con 0x7fe8a40afbf0 2024-04-30T08:35:14.609 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:35:14.611 INFO:teuthology.orchestra.run.smithi032.stderr:dumped all 2024-04-30T08:35:14.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.608+0000 7fe8c2200640 1 -- 172.21.15.32:0/508874663 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe8a40afbf0 msgr2=0x7fe8a40b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:14.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.608+0000 7fe8c2200640 1 --2- 172.21.15.32:0/508874663 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe8a40afbf0 0x7fe8a40b2100 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe8bc1067b0 tx=0x7fe8b0005f90 comp rx=0 tx=0).stop 2024-04-30T08:35:14.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.608+0000 7fe8c2200640 1 -- 172.21.15.32:0/508874663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8bc10baf0 msgr2=0x7fe8bc1a3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:14.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.608+0000 7fe8c2200640 1 --2- 172.21.15.32:0/508874663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8bc10baf0 0x7fe8bc1a3760 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe8b400eb10 tx=0x7fe8b400c2d0 comp rx=0 tx=0).stop 2024-04-30T08:35:14.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.609+0000 7fe8c2200640 1 -- 172.21.15.32:0/508874663 shutdown_connections 2024-04-30T08:35:14.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.609+0000 7fe8c2200640 1 --2- 172.21.15.32:0/508874663 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe8a40afbf0 0x7fe8a40b2100 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:14.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.609+0000 7fe8c2200640 1 --2- 172.21.15.32:0/508874663 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe8bc10baf0 0x7fe8bc1a3760 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:14.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.609+0000 7fe8c2200640 1 --2- 172.21.15.32:0/508874663 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8bc106df0 0x7fe8bc19c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:14.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.609+0000 7fe8c2200640 1 --2- 172.21.15.32:0/508874663 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe8bc105bb0 0x7fe8bc19c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:14.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.609+0000 7fe8c2200640 1 -- 172.21.15.32:0/508874663 >> 172.21.15.32:0/508874663 conn(0x7fe8bc100da0 msgr2=0x7fe8bc107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:14.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.609+0000 7fe8c2200640 1 -- 172.21.15.32:0/508874663 shutdown_connections 2024-04-30T08:35:14.614 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.609+0000 7fe8c2200640 1 -- 172.21.15.32:0/508874663 wait complete. 2024-04-30T08:35:14.637 INFO:teuthology.orchestra.run.smithi032.stdout:{"pg_ready":true,"pg_map":{"version":427,"stamp":"2024-04-30T08:35:13.795418+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":666516,"num_objects":102,"num_object_clones":0,"num_object_copies":308,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1722,"num_read_kb":966,"num_write":391,"num_write_kb":2315,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":2951908,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":824,"ondisk_log_size":824,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":373172,"kb_used_data":11540,"kb_used_omap":18,"kb_used_meta":361517,"kb_avail":1124568652,"statfs":{"total":1151940427776,"available":1151558299648,"internally_reserved":0,"allocated":11816960,"data_stored":7720874,"data_compressed":434025,"data_compressed_allocated":589824,"data_compressed_original":2678784,"omap_allocated":19141,"internal_metadata":370193723},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":6107,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":120,"num_read_kb":59,"num_write":17,"num_write_kb":17,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005967"},"pg_stats":[{"pgid":"3.3f","version":"85'8","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.737715+0000","last_change":"2024-04-30T08:31:52.711443+0000","last_active":"2024-04-30T08:33:59.737715+0000","last_peered":"2024-04-30T08:33:59.737715+0000","last_clean":"2024-04-30T08:33:59.737715+0000","last_became_active":"2024-04-30T08:31:52.711029+0000","last_became_peered":"2024-04-30T08:31:52.711029+0000","last_unstale":"2024-04-30T08:33:59.737715+0000","last_undegraded":"2024-04-30T08:33:59.737715+0000","last_fullsized":"2024-04-30T08:33:59.737715+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:51:15.208528+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"85'4","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.736806+0000","last_change":"2024-04-30T08:31:52.712324+0000","last_active":"2024-04-30T08:33:59.736806+0000","last_peered":"2024-04-30T08:33:59.736806+0000","last_clean":"2024-04-30T08:33:59.736806+0000","last_became_active":"2024-04-30T08:31:52.711923+0000","last_became_peered":"2024-04-30T08:31:52.711923+0000","last_unstale":"2024-04-30T08:33:59.736806+0000","last_undegraded":"2024-04-30T08:33:59.736806+0000","last_fullsized":"2024-04-30T08:33:59.736806+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:29:55.071459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"85'7","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.735218+0000","last_change":"2024-04-30T08:31:52.712936+0000","last_active":"2024-04-30T08:33:59.735218+0000","last_peered":"2024-04-30T08:33:59.735218+0000","last_clean":"2024-04-30T08:33:59.735218+0000","last_became_active":"2024-04-30T08:31:52.712585+0000","last_became_peered":"2024-04-30T08:31:52.712585+0000","last_unstale":"2024-04-30T08:33:59.735218+0000","last_undegraded":"2024-04-30T08:33:59.735218+0000","last_fullsized":"2024-04-30T08:33:59.735218+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:13:43.745561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"85'8","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.688422+0000","last_change":"2024-04-30T08:31:52.686880+0000","last_active":"2024-04-30T08:33:59.688422+0000","last_peered":"2024-04-30T08:33:59.688422+0000","last_clean":"2024-04-30T08:33:59.688422+0000","last_became_active":"2024-04-30T08:31:52.686506+0000","last_became_peered":"2024-04-30T08:31:52.686506+0000","last_unstale":"2024-04-30T08:33:59.688422+0000","last_undegraded":"2024-04-30T08:33:59.688422+0000","last_fullsized":"2024-04-30T08:33:59.688422+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:16:33.390951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"85'4","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.716502+0000","last_change":"2024-04-30T08:31:52.706014+0000","last_active":"2024-04-30T08:33:59.716502+0000","last_peered":"2024-04-30T08:33:59.716502+0000","last_clean":"2024-04-30T08:33:59.716502+0000","last_became_active":"2024-04-30T08:31:52.705582+0000","last_became_peered":"2024-04-30T08:31:52.705582+0000","last_unstale":"2024-04-30T08:33:59.716502+0000","last_undegraded":"2024-04-30T08:33:59.716502+0000","last_fullsized":"2024-04-30T08:33:59.716502+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:24:49.804592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"85'6","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.717350+0000","last_change":"2024-04-30T08:31:52.690462+0000","last_active":"2024-04-30T08:33:59.717350+0000","last_peered":"2024-04-30T08:33:59.717350+0000","last_clean":"2024-04-30T08:33:59.717350+0000","last_became_active":"2024-04-30T08:31:52.690092+0000","last_became_peered":"2024-04-30T08:31:52.690092+0000","last_unstale":"2024-04-30T08:33:59.717350+0000","last_undegraded":"2024-04-30T08:33:59.717350+0000","last_fullsized":"2024-04-30T08:33:59.717350+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:32:49.769383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"85'5","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.704595+0000","last_change":"2024-04-30T08:31:52.701786+0000","last_active":"2024-04-30T08:33:59.704595+0000","last_peered":"2024-04-30T08:33:59.704595+0000","last_clean":"2024-04-30T08:33:59.704595+0000","last_became_active":"2024-04-30T08:31:52.701402+0000","last_became_peered":"2024-04-30T08:31:52.701402+0000","last_unstale":"2024-04-30T08:33:59.704595+0000","last_undegraded":"2024-04-30T08:33:59.704595+0000","last_fullsized":"2024-04-30T08:33:59.704595+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:58:06.153479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"85'9","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.719088+0000","last_change":"2024-04-30T08:31:52.698037+0000","last_active":"2024-04-30T08:33:59.719088+0000","last_peered":"2024-04-30T08:33:59.719088+0000","last_clean":"2024-04-30T08:33:59.719088+0000","last_became_active":"2024-04-30T08:31:52.697646+0000","last_became_peered":"2024-04-30T08:31:52.697646+0000","last_unstale":"2024-04-30T08:33:59.719088+0000","last_undegraded":"2024-04-30T08:33:59.719088+0000","last_fullsized":"2024-04-30T08:33:59.719088+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:28:59.520091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"85'11","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.692082+0000","last_change":"2024-04-30T08:31:52.680377+0000","last_active":"2024-04-30T08:33:59.692082+0000","last_peered":"2024-04-30T08:33:59.692082+0000","last_clean":"2024-04-30T08:33:59.692082+0000","last_became_active":"2024-04-30T08:31:52.679977+0000","last_became_peered":"2024-04-30T08:31:52.679977+0000","last_unstale":"2024-04-30T08:33:59.692082+0000","last_undegraded":"2024-04-30T08:33:59.692082+0000","last_fullsized":"2024-04-30T08:33:59.692082+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:35:22.488450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"85'16","reported_seq":54,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.729297+0000","last_change":"2024-04-30T08:32:31.573546+0000","last_active":"2024-04-30T08:33:59.729297+0000","last_peered":"2024-04-30T08:33:59.729297+0000","last_clean":"2024-04-30T08:33:59.729297+0000","last_became_active":"2024-04-30T08:32:31.573289+0000","last_became_peered":"2024-04-30T08:32:31.573289+0000","last_unstale":"2024-04-30T08:33:59.729297+0000","last_undegraded":"2024-04-30T08:33:59.729297+0000","last_fullsized":"2024-04-30T08:33:59.729297+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:42:45.862326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,9],"acting":[10,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"85'6","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.692001+0000","last_change":"2024-04-30T08:31:52.690344+0000","last_active":"2024-04-30T08:33:59.692001+0000","last_peered":"2024-04-30T08:33:59.692001+0000","last_clean":"2024-04-30T08:33:59.692001+0000","last_became_active":"2024-04-30T08:31:52.689857+0000","last_became_peered":"2024-04-30T08:31:52.689857+0000","last_unstale":"2024-04-30T08:33:59.692001+0000","last_undegraded":"2024-04-30T08:33:59.692001+0000","last_fullsized":"2024-04-30T08:33:59.692001+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:45:13.395589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"85'7","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.689051+0000","last_change":"2024-04-30T08:31:52.685151+0000","last_active":"2024-04-30T08:33:59.689051+0000","last_peered":"2024-04-30T08:33:59.689051+0000","last_clean":"2024-04-30T08:33:59.689051+0000","last_became_active":"2024-04-30T08:31:52.684722+0000","last_became_peered":"2024-04-30T08:31:52.684722+0000","last_unstale":"2024-04-30T08:33:59.689051+0000","last_undegraded":"2024-04-30T08:33:59.689051+0000","last_fullsized":"2024-04-30T08:33:59.689051+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:14:00.660478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"85'10","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.723211+0000","last_change":"2024-04-30T08:31:52.714294+0000","last_active":"2024-04-30T08:33:59.723211+0000","last_peered":"2024-04-30T08:33:59.723211+0000","last_clean":"2024-04-30T08:33:59.723211+0000","last_became_active":"2024-04-30T08:31:52.710094+0000","last_became_peered":"2024-04-30T08:31:52.710094+0000","last_unstale":"2024-04-30T08:33:59.723211+0000","last_undegraded":"2024-04-30T08:33:59.723211+0000","last_fullsized":"2024-04-30T08:33:59.723211+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:34:44.203093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"85'7","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.718132+0000","last_change":"2024-04-30T08:31:52.707992+0000","last_active":"2024-04-30T08:33:59.718132+0000","last_peered":"2024-04-30T08:33:59.718132+0000","last_clean":"2024-04-30T08:33:59.718132+0000","last_became_active":"2024-04-30T08:31:52.706456+0000","last_became_peered":"2024-04-30T08:31:52.706456+0000","last_unstale":"2024-04-30T08:33:59.718132+0000","last_undegraded":"2024-04-30T08:33:59.718132+0000","last_fullsized":"2024-04-30T08:33:59.718132+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:45:32.717842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"85'7","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.735414+0000","last_change":"2024-04-30T08:31:52.695605+0000","last_active":"2024-04-30T08:33:59.735414+0000","last_peered":"2024-04-30T08:33:59.735414+0000","last_clean":"2024-04-30T08:33:59.735414+0000","last_became_active":"2024-04-30T08:31:52.692439+0000","last_became_peered":"2024-04-30T08:31:52.692439+0000","last_unstale":"2024-04-30T08:33:59.735414+0000","last_undegraded":"2024-04-30T08:33:59.735414+0000","last_fullsized":"2024-04-30T08:33:59.735414+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:39:38.746567+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"85'9","reported_seq":20,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.740363+0000","last_change":"2024-04-30T08:33:32.802311+0000","last_active":"2024-04-30T08:33:59.740363+0000","last_peered":"2024-04-30T08:33:59.740363+0000","last_clean":"2024-04-30T08:33:59.740363+0000","last_became_active":"2024-04-30T08:33:32.801601+0000","last_became_peered":"2024-04-30T08:33:32.801601+0000","last_unstale":"2024-04-30T08:33:59.740363+0000","last_undegraded":"2024-04-30T08:33:59.740363+0000","last_fullsized":"2024-04-30T08:33:59.740363+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:59:34.877824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0],"acting":[9,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2f","version":"85'8","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.737797+0000","last_change":"2024-04-30T08:31:52.711439+0000","last_active":"2024-04-30T08:33:59.737797+0000","last_peered":"2024-04-30T08:33:59.737797+0000","last_clean":"2024-04-30T08:33:59.737797+0000","last_became_active":"2024-04-30T08:31:52.711023+0000","last_became_peered":"2024-04-30T08:31:52.711023+0000","last_unstale":"2024-04-30T08:33:59.737797+0000","last_undegraded":"2024-04-30T08:33:59.737797+0000","last_fullsized":"2024-04-30T08:33:59.737797+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:31:00.654348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"85'10","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.721531+0000","last_change":"2024-04-30T08:31:52.708476+0000","last_active":"2024-04-30T08:33:59.721531+0000","last_peered":"2024-04-30T08:33:59.721531+0000","last_clean":"2024-04-30T08:33:59.721531+0000","last_became_active":"2024-04-30T08:31:52.704735+0000","last_became_peered":"2024-04-30T08:31:52.704735+0000","last_unstale":"2024-04-30T08:33:59.721531+0000","last_undegraded":"2024-04-30T08:33:59.721531+0000","last_fullsized":"2024-04-30T08:33:59.721531+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:40:00.011783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"85'7","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.720093+0000","last_change":"2024-04-30T08:31:52.704901+0000","last_active":"2024-04-30T08:33:59.720093+0000","last_peered":"2024-04-30T08:33:59.720093+0000","last_clean":"2024-04-30T08:33:59.720093+0000","last_became_active":"2024-04-30T08:31:52.704636+0000","last_became_peered":"2024-04-30T08:31:52.704636+0000","last_unstale":"2024-04-30T08:33:59.720093+0000","last_undegraded":"2024-04-30T08:33:59.720093+0000","last_fullsized":"2024-04-30T08:33:59.720093+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:08:53.580809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.879602+0000","last_change":"2024-04-30T08:31:57.743342+0000","last_active":"2024-04-30T08:33:31.879602+0000","last_peered":"2024-04-30T08:33:31.879602+0000","last_clean":"2024-04-30T08:33:31.879602+0000","last_became_active":"2024-04-30T08:31:57.742850+0000","last_became_peered":"2024-04-30T08:31:57.742850+0000","last_unstale":"2024-04-30T08:33:31.879602+0000","last_undegraded":"2024-04-30T08:33:31.879602+0000","last_fullsized":"2024-04-30T08:33:31.879602+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:59:10.628822+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"85'14","reported_seq":209,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:35:09.926274+0000","last_change":"2024-04-30T08:31:51.694984+0000","last_active":"2024-04-30T08:35:09.926274+0000","last_peered":"2024-04-30T08:35:09.926274+0000","last_clean":"2024-04-30T08:35:09.926274+0000","last_became_active":"2024-04-30T08:31:51.694514+0000","last_became_peered":"2024-04-30T08:31:51.694514+0000","last_unstale":"2024-04-30T08:35:09.926274+0000","last_undegraded":"2024-04-30T08:35:09.926274+0000","last_fullsized":"2024-04-30T08:35:09.926274+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:17:03.873090+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":162,"num_read_kb":162,"num_write":15,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"85'5","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.717378+0000","last_change":"2024-04-30T08:31:52.694276+0000","last_active":"2024-04-30T08:33:59.717378+0000","last_peered":"2024-04-30T08:33:59.717378+0000","last_clean":"2024-04-30T08:33:59.717378+0000","last_became_active":"2024-04-30T08:31:52.693839+0000","last_became_peered":"2024-04-30T08:31:52.693839+0000","last_unstale":"2024-04-30T08:33:59.717378+0000","last_undegraded":"2024-04-30T08:33:59.717378+0000","last_fullsized":"2024-04-30T08:33:59.717378+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:29:21.271040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.169742+0000","last_change":"2024-04-30T08:31:57.754828+0000","last_active":"2024-04-30T08:33:32.169742+0000","last_peered":"2024-04-30T08:33:32.169742+0000","last_clean":"2024-04-30T08:33:32.169742+0000","last_became_active":"2024-04-30T08:31:57.754322+0000","last_became_peered":"2024-04-30T08:31:57.754322+0000","last_unstale":"2024-04-30T08:33:32.169742+0000","last_undegraded":"2024-04-30T08:33:32.169742+0000","last_fullsized":"2024-04-30T08:33:32.169742+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:36:02.560134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"81'2","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.277887+0000","last_change":"2024-04-30T08:31:51.707637+0000","last_active":"2024-04-30T08:33:32.277887+0000","last_peered":"2024-04-30T08:33:32.277887+0000","last_clean":"2024-04-30T08:33:32.277887+0000","last_became_active":"2024-04-30T08:31:51.707233+0000","last_became_peered":"2024-04-30T08:31:51.707233+0000","last_unstale":"2024-04-30T08:33:32.277887+0000","last_undegraded":"2024-04-30T08:33:32.277887+0000","last_fullsized":"2024-04-30T08:33:32.277887+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:14:46.041169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"85'6","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.741653+0000","last_change":"2024-04-30T08:31:52.711442+0000","last_active":"2024-04-30T08:33:59.741653+0000","last_peered":"2024-04-30T08:33:59.741653+0000","last_clean":"2024-04-30T08:33:59.741653+0000","last_became_active":"2024-04-30T08:31:52.710942+0000","last_became_peered":"2024-04-30T08:31:52.710942+0000","last_unstale":"2024-04-30T08:33:59.741653+0000","last_undegraded":"2024-04-30T08:33:59.741653+0000","last_fullsized":"2024-04-30T08:33:59.741653+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:26:56.428770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.169895+0000","last_change":"2024-04-30T08:31:57.760535+0000","last_active":"2024-04-30T08:33:32.169895+0000","last_peered":"2024-04-30T08:33:32.169895+0000","last_clean":"2024-04-30T08:33:32.169895+0000","last_became_active":"2024-04-30T08:31:57.760073+0000","last_became_peered":"2024-04-30T08:31:57.760073+0000","last_unstale":"2024-04-30T08:33:32.169895+0000","last_undegraded":"2024-04-30T08:33:32.169895+0000","last_fullsized":"2024-04-30T08:33:32.169895+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:03:51.359474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"81'12","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.887094+0000","last_change":"2024-04-30T08:31:51.678575+0000","last_active":"2024-04-30T08:33:31.887094+0000","last_peered":"2024-04-30T08:33:31.887094+0000","last_clean":"2024-04-30T08:33:31.887094+0000","last_became_active":"2024-04-30T08:31:51.678114+0000","last_became_peered":"2024-04-30T08:31:51.678114+0000","last_unstale":"2024-04-30T08:33:31.887094+0000","last_undegraded":"2024-04-30T08:33:31.887094+0000","last_fullsized":"2024-04-30T08:33:31.887094+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:10:35.517183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"85'9","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.721346+0000","last_change":"2024-04-30T08:31:52.704137+0000","last_active":"2024-04-30T08:33:59.721346+0000","last_peered":"2024-04-30T08:33:59.721346+0000","last_clean":"2024-04-30T08:33:59.721346+0000","last_became_active":"2024-04-30T08:31:52.703695+0000","last_became_peered":"2024-04-30T08:31:52.703695+0000","last_unstale":"2024-04-30T08:33:59.721346+0000","last_undegraded":"2024-04-30T08:33:59.721346+0000","last_fullsized":"2024-04-30T08:33:59.721346+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:31:05.461106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.169818+0000","last_change":"2024-04-30T08:31:57.755237+0000","last_active":"2024-04-30T08:33:32.169818+0000","last_peered":"2024-04-30T08:33:32.169818+0000","last_clean":"2024-04-30T08:33:32.169818+0000","last_became_active":"2024-04-30T08:31:57.754438+0000","last_became_peered":"2024-04-30T08:31:57.754438+0000","last_unstale":"2024-04-30T08:33:32.169818+0000","last_undegraded":"2024-04-30T08:33:32.169818+0000","last_fullsized":"2024-04-30T08:33:32.169818+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:20:59.035110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"81'5","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886405+0000","last_change":"2024-04-30T08:31:51.673793+0000","last_active":"2024-04-30T08:33:31.886405+0000","last_peered":"2024-04-30T08:33:31.886405+0000","last_clean":"2024-04-30T08:33:31.886405+0000","last_became_active":"2024-04-30T08:31:51.672703+0000","last_became_peered":"2024-04-30T08:31:51.672703+0000","last_unstale":"2024-04-30T08:33:31.886405+0000","last_undegraded":"2024-04-30T08:33:31.886405+0000","last_fullsized":"2024-04-30T08:33:31.886405+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:13:18.959542+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"85'15","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.686983+0000","last_change":"2024-04-30T08:31:52.694320+0000","last_active":"2024-04-30T08:33:59.686983+0000","last_peered":"2024-04-30T08:33:59.686983+0000","last_clean":"2024-04-30T08:33:59.686983+0000","last_became_active":"2024-04-30T08:31:52.694032+0000","last_became_peered":"2024-04-30T08:31:52.694032+0000","last_unstale":"2024-04-30T08:33:59.686983+0000","last_undegraded":"2024-04-30T08:33:59.686983+0000","last_fullsized":"2024-04-30T08:33:59.686983+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:33:12.347369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.880017+0000","last_change":"2024-04-30T08:31:57.739984+0000","last_active":"2024-04-30T08:33:31.880017+0000","last_peered":"2024-04-30T08:33:31.880017+0000","last_clean":"2024-04-30T08:33:31.880017+0000","last_became_active":"2024-04-30T08:31:57.739477+0000","last_became_peered":"2024-04-30T08:31:57.739477+0000","last_unstale":"2024-04-30T08:33:31.880017+0000","last_undegraded":"2024-04-30T08:33:31.880017+0000","last_fullsized":"2024-04-30T08:33:31.880017+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:10:11.325419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"85'7","reported_seq":207,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:35:11.817603+0000","last_change":"2024-04-30T08:31:51.678792+0000","last_active":"2024-04-30T08:35:11.817603+0000","last_peered":"2024-04-30T08:35:11.817603+0000","last_clean":"2024-04-30T08:35:11.817603+0000","last_became_active":"2024-04-30T08:31:51.678094+0000","last_became_peered":"2024-04-30T08:31:51.678094+0000","last_unstale":"2024-04-30T08:35:11.817603+0000","last_undegraded":"2024-04-30T08:35:11.817603+0000","last_fullsized":"2024-04-30T08:35:11.817603+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:03:17.526016+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":166,"num_read_kb":3,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"85'7","reported_seq":44,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.700627+0000","last_change":"2024-04-30T08:32:31.572984+0000","last_active":"2024-04-30T08:33:59.700627+0000","last_peered":"2024-04-30T08:33:59.700627+0000","last_clean":"2024-04-30T08:33:59.700627+0000","last_became_active":"2024-04-30T08:32:31.572536+0000","last_became_peered":"2024-04-30T08:32:31.572536+0000","last_unstale":"2024-04-30T08:33:59.700627+0000","last_undegraded":"2024-04-30T08:33:59.700627+0000","last_fullsized":"2024-04-30T08:33:59.700627+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:46:15.884555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,7],"acting":[8,9,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.879844+0000","last_change":"2024-04-30T08:31:57.765047+0000","last_active":"2024-04-30T08:33:31.879844+0000","last_peered":"2024-04-30T08:33:31.879844+0000","last_clean":"2024-04-30T08:33:31.879844+0000","last_became_active":"2024-04-30T08:31:57.764519+0000","last_became_peered":"2024-04-30T08:31:57.764519+0000","last_unstale":"2024-04-30T08:33:31.879844+0000","last_undegraded":"2024-04-30T08:33:31.879844+0000","last_fullsized":"2024-04-30T08:33:31.879844+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:06:16.126396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"81'7","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881627+0000","last_change":"2024-04-30T08:31:51.686403+0000","last_active":"2024-04-30T08:33:31.881627+0000","last_peered":"2024-04-30T08:33:31.881627+0000","last_clean":"2024-04-30T08:33:31.881627+0000","last_became_active":"2024-04-30T08:31:51.686096+0000","last_became_peered":"2024-04-30T08:31:51.686096+0000","last_unstale":"2024-04-30T08:33:31.881627+0000","last_undegraded":"2024-04-30T08:33:31.881627+0000","last_fullsized":"2024-04-30T08:33:31.881627+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:47:05.961930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"85'8","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.698959+0000","last_change":"2024-04-30T08:31:52.698512+0000","last_active":"2024-04-30T08:33:59.698959+0000","last_peered":"2024-04-30T08:33:59.698959+0000","last_clean":"2024-04-30T08:33:59.698959+0000","last_became_active":"2024-04-30T08:31:52.698168+0000","last_became_peered":"2024-04-30T08:31:52.698168+0000","last_unstale":"2024-04-30T08:33:59.698959+0000","last_undegraded":"2024-04-30T08:33:59.698959+0000","last_fullsized":"2024-04-30T08:33:59.698959+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:22:45.094365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886398+0000","last_change":"2024-04-30T08:31:57.761430+0000","last_active":"2024-04-30T08:33:31.886398+0000","last_peered":"2024-04-30T08:33:31.886398+0000","last_clean":"2024-04-30T08:33:31.886398+0000","last_became_active":"2024-04-30T08:31:57.761062+0000","last_became_peered":"2024-04-30T08:31:57.761062+0000","last_unstale":"2024-04-30T08:33:31.886398+0000","last_undegraded":"2024-04-30T08:33:31.886398+0000","last_fullsized":"2024-04-30T08:33:31.886398+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:32:05.403786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"81'3","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.879820+0000","last_change":"2024-04-30T08:31:51.686694+0000","last_active":"2024-04-30T08:33:31.879820+0000","last_peered":"2024-04-30T08:33:31.879820+0000","last_clean":"2024-04-30T08:33:31.879820+0000","last_became_active":"2024-04-30T08:31:51.685773+0000","last_became_peered":"2024-04-30T08:31:51.685773+0000","last_unstale":"2024-04-30T08:33:31.879820+0000","last_undegraded":"2024-04-30T08:33:31.879820+0000","last_fullsized":"2024-04-30T08:33:31.879820+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:44:25.327977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"85'8","reported_seq":19,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.734178+0000","last_change":"2024-04-30T08:33:31.888698+0000","last_active":"2024-04-30T08:33:59.734178+0000","last_peered":"2024-04-30T08:33:59.734178+0000","last_clean":"2024-04-30T08:33:59.734178+0000","last_became_active":"2024-04-30T08:33:31.887935+0000","last_became_peered":"2024-04-30T08:33:31.887935+0000","last_unstale":"2024-04-30T08:33:59.734178+0000","last_undegraded":"2024-04-30T08:33:59.734178+0000","last_fullsized":"2024-04-30T08:33:59.734178+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:15:40.989004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,6],"acting":[9,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.799780+0000","last_change":"2024-04-30T08:31:57.757224+0000","last_active":"2024-04-30T08:33:32.799780+0000","last_peered":"2024-04-30T08:33:32.799780+0000","last_clean":"2024-04-30T08:33:32.799780+0000","last_became_active":"2024-04-30T08:31:57.756658+0000","last_became_peered":"2024-04-30T08:31:57.756658+0000","last_unstale":"2024-04-30T08:33:32.799780+0000","last_undegraded":"2024-04-30T08:33:32.799780+0000","last_fullsized":"2024-04-30T08:33:32.799780+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:01:21.309322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"81'3","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273042+0000","last_change":"2024-04-30T08:31:51.678944+0000","last_active":"2024-04-30T08:33:32.273042+0000","last_peered":"2024-04-30T08:33:32.273042+0000","last_clean":"2024-04-30T08:33:32.273042+0000","last_became_active":"2024-04-30T08:31:51.678493+0000","last_became_peered":"2024-04-30T08:31:51.678493+0000","last_unstale":"2024-04-30T08:33:32.273042+0000","last_undegraded":"2024-04-30T08:33:32.273042+0000","last_fullsized":"2024-04-30T08:33:32.273042+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:48:11.111466+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"85'11","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.722707+0000","last_change":"2024-04-30T08:31:52.707627+0000","last_active":"2024-04-30T08:33:59.722707+0000","last_peered":"2024-04-30T08:33:59.722707+0000","last_clean":"2024-04-30T08:33:59.722707+0000","last_became_active":"2024-04-30T08:31:52.706441+0000","last_became_peered":"2024-04-30T08:31:52.706441+0000","last_unstale":"2024-04-30T08:33:59.722707+0000","last_undegraded":"2024-04-30T08:33:59.722707+0000","last_fullsized":"2024-04-30T08:33:59.722707+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:15:45.768927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.275247+0000","last_change":"2024-04-30T08:31:57.740324+0000","last_active":"2024-04-30T08:33:32.275247+0000","last_peered":"2024-04-30T08:33:32.275247+0000","last_clean":"2024-04-30T08:33:32.275247+0000","last_became_active":"2024-04-30T08:31:57.739464+0000","last_became_peered":"2024-04-30T08:31:57.739464+0000","last_unstale":"2024-04-30T08:33:32.275247+0000","last_undegraded":"2024-04-30T08:33:32.275247+0000","last_fullsized":"2024-04-30T08:33:32.275247+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:11:37.395545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"85'5","reported_seq":356,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:35:08.812181+0000","last_change":"2024-04-30T08:31:51.695283+0000","last_active":"2024-04-30T08:35:08.812181+0000","last_peered":"2024-04-30T08:35:08.812181+0000","last_clean":"2024-04-30T08:35:08.812181+0000","last_became_active":"2024-04-30T08:31:51.694848+0000","last_became_peered":"2024-04-30T08:31:51.694848+0000","last_unstale":"2024-04-30T08:35:08.812181+0000","last_undegraded":"2024-04-30T08:35:08.812181+0000","last_fullsized":"2024-04-30T08:35:08.812181+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:41:38.638034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":317,"num_read_kb":317,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"85'10","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.729523+0000","last_change":"2024-04-30T08:31:52.693594+0000","last_active":"2024-04-30T08:33:59.729523+0000","last_peered":"2024-04-30T08:33:59.729523+0000","last_clean":"2024-04-30T08:33:59.729523+0000","last_became_active":"2024-04-30T08:31:52.693199+0000","last_became_peered":"2024-04-30T08:31:52.693199+0000","last_unstale":"2024-04-30T08:33:59.729523+0000","last_undegraded":"2024-04-30T08:33:59.729523+0000","last_fullsized":"2024-04-30T08:33:59.729523+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:57:49.908617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.883697+0000","last_change":"2024-04-30T08:31:57.749727+0000","last_active":"2024-04-30T08:33:31.883697+0000","last_peered":"2024-04-30T08:33:31.883697+0000","last_clean":"2024-04-30T08:33:31.883697+0000","last_became_active":"2024-04-30T08:31:57.749267+0000","last_became_peered":"2024-04-30T08:31:57.749267+0000","last_unstale":"2024-04-30T08:33:31.883697+0000","last_undegraded":"2024-04-30T08:33:31.883697+0000","last_fullsized":"2024-04-30T08:33:31.883697+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:26:12.683008+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"81'4","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.218677+0000","last_change":"2024-04-30T08:31:51.677398+0000","last_active":"2024-04-30T08:33:32.218677+0000","last_peered":"2024-04-30T08:33:32.218677+0000","last_clean":"2024-04-30T08:33:32.218677+0000","last_became_active":"2024-04-30T08:31:51.677031+0000","last_became_peered":"2024-04-30T08:31:51.677031+0000","last_unstale":"2024-04-30T08:33:32.218677+0000","last_undegraded":"2024-04-30T08:33:32.218677+0000","last_fullsized":"2024-04-30T08:33:32.218677+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:24:25.154229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"85'9","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.741480+0000","last_change":"2024-04-30T08:32:31.561554+0000","last_active":"2024-04-30T08:33:59.741480+0000","last_peered":"2024-04-30T08:33:59.741480+0000","last_clean":"2024-04-30T08:33:59.741480+0000","last_became_active":"2024-04-30T08:32:31.560253+0000","last_became_peered":"2024-04-30T08:32:31.560253+0000","last_unstale":"2024-04-30T08:33:59.741480+0000","last_undegraded":"2024-04-30T08:33:59.741480+0000","last_fullsized":"2024-04-30T08:33:59.741480+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:59:08.602282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.170073+0000","last_change":"2024-04-30T08:32:31.586746+0000","last_active":"2024-04-30T08:33:32.170073+0000","last_peered":"2024-04-30T08:33:32.170073+0000","last_clean":"2024-04-30T08:33:32.170073+0000","last_became_active":"2024-04-30T08:32:31.586356+0000","last_became_peered":"2024-04-30T08:32:31.586356+0000","last_unstale":"2024-04-30T08:33:32.170073+0000","last_undegraded":"2024-04-30T08:33:32.170073+0000","last_fullsized":"2024-04-30T08:33:32.170073+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:22:39.371508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,8],"acting":[4,1,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"81'5","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273483+0000","last_change":"2024-04-30T08:31:51.694424+0000","last_active":"2024-04-30T08:33:32.273483+0000","last_peered":"2024-04-30T08:33:32.273483+0000","last_clean":"2024-04-30T08:33:32.273483+0000","last_became_active":"2024-04-30T08:31:51.694081+0000","last_became_peered":"2024-04-30T08:31:51.694081+0000","last_unstale":"2024-04-30T08:33:32.273483+0000","last_undegraded":"2024-04-30T08:33:32.273483+0000","last_fullsized":"2024-04-30T08:33:32.273483+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:27:33.328477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"85'11","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.692022+0000","last_change":"2024-04-30T08:31:52.684739+0000","last_active":"2024-04-30T08:33:59.692022+0000","last_peered":"2024-04-30T08:33:59.692022+0000","last_clean":"2024-04-30T08:33:59.692022+0000","last_became_active":"2024-04-30T08:31:52.684316+0000","last_became_peered":"2024-04-30T08:31:52.684316+0000","last_unstale":"2024-04-30T08:33:59.692022+0000","last_undegraded":"2024-04-30T08:33:59.692022+0000","last_fullsized":"2024-04-30T08:33:59.692022+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:12:05.437493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881452+0000","last_change":"2024-04-30T08:32:31.557578+0000","last_active":"2024-04-30T08:33:31.881452+0000","last_peered":"2024-04-30T08:33:31.881452+0000","last_clean":"2024-04-30T08:33:31.881452+0000","last_became_active":"2024-04-30T08:32:31.557115+0000","last_became_peered":"2024-04-30T08:32:31.557115+0000","last_unstale":"2024-04-30T08:33:31.881452+0000","last_undegraded":"2024-04-30T08:33:31.881452+0000","last_fullsized":"2024-04-30T08:33:31.881452+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:41:33.225879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"85'40","reported_seq":400,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:35:08.812186+0000","last_change":"2024-04-30T08:31:51.676223+0000","last_active":"2024-04-30T08:35:08.812186+0000","last_peered":"2024-04-30T08:35:08.812186+0000","last_clean":"2024-04-30T08:35:08.812186+0000","last_became_active":"2024-04-30T08:31:51.675830+0000","last_became_peered":"2024-04-30T08:31:51.675830+0000","last_unstale":"2024-04-30T08:35:08.812186+0000","last_undegraded":"2024-04-30T08:35:08.812186+0000","last_fullsized":"2024-04-30T08:35:08.812186+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:28:49.497719+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67336,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":326,"num_read_kb":229,"num_write":42,"num_write_kb":86,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"85'6","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.684496+0000","last_change":"2024-04-30T08:31:52.686892+0000","last_active":"2024-04-30T08:33:59.684496+0000","last_peered":"2024-04-30T08:33:59.684496+0000","last_clean":"2024-04-30T08:33:59.684496+0000","last_became_active":"2024-04-30T08:31:52.686533+0000","last_became_peered":"2024-04-30T08:31:52.686533+0000","last_unstale":"2024-04-30T08:33:59.684496+0000","last_undegraded":"2024-04-30T08:33:59.684496+0000","last_fullsized":"2024-04-30T08:33:59.684496+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:50:18.836161+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.274017+0000","last_change":"2024-04-30T08:31:57.752865+0000","last_active":"2024-04-30T08:33:32.274017+0000","last_peered":"2024-04-30T08:33:32.274017+0000","last_clean":"2024-04-30T08:33:32.274017+0000","last_became_active":"2024-04-30T08:31:57.752385+0000","last_became_peered":"2024-04-30T08:31:57.752385+0000","last_unstale":"2024-04-30T08:33:32.274017+0000","last_undegraded":"2024-04-30T08:33:32.274017+0000","last_fullsized":"2024-04-30T08:33:32.274017+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:15:15.315580+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"81'9","reported_seq":32,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886273+0000","last_change":"2024-04-30T08:32:31.585282+0000","last_active":"2024-04-30T08:33:31.886273+0000","last_peered":"2024-04-30T08:33:31.886273+0000","last_clean":"2024-04-30T08:33:31.886273+0000","last_became_active":"2024-04-30T08:32:31.571676+0000","last_became_peered":"2024-04-30T08:32:31.571676+0000","last_unstale":"2024-04-30T08:33:31.886273+0000","last_undegraded":"2024-04-30T08:33:31.886273+0000","last_fullsized":"2024-04-30T08:33:31.886273+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:18:29.910201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"85'9","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.723531+0000","last_change":"2024-04-30T08:31:52.706259+0000","last_active":"2024-04-30T08:33:59.723531+0000","last_peered":"2024-04-30T08:33:59.723531+0000","last_clean":"2024-04-30T08:33:59.723531+0000","last_became_active":"2024-04-30T08:31:52.705620+0000","last_became_peered":"2024-04-30T08:31:52.705620+0000","last_unstale":"2024-04-30T08:33:59.723531+0000","last_undegraded":"2024-04-30T08:33:59.723531+0000","last_fullsized":"2024-04-30T08:33:59.723531+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:22:29.816950+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":30,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886632+0000","last_change":"2024-04-30T08:32:31.562571+0000","last_active":"2024-04-30T08:33:31.886632+0000","last_peered":"2024-04-30T08:33:31.886632+0000","last_clean":"2024-04-30T08:33:31.886632+0000","last_became_active":"2024-04-30T08:32:31.562012+0000","last_became_peered":"2024-04-30T08:32:31.562012+0000","last_unstale":"2024-04-30T08:33:31.886632+0000","last_undegraded":"2024-04-30T08:33:31.886632+0000","last_fullsized":"2024-04-30T08:33:31.886632+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:12:22.928286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,7,6],"acting":[11,8,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"81'5","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273150+0000","last_change":"2024-04-30T08:31:51.701122+0000","last_active":"2024-04-30T08:33:32.273150+0000","last_peered":"2024-04-30T08:33:32.273150+0000","last_clean":"2024-04-30T08:33:32.273150+0000","last_became_active":"2024-04-30T08:31:51.700815+0000","last_became_peered":"2024-04-30T08:31:51.700815+0000","last_unstale":"2024-04-30T08:33:32.273150+0000","last_undegraded":"2024-04-30T08:33:32.273150+0000","last_fullsized":"2024-04-30T08:33:32.273150+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:33:12.714979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"85'15","reported_seq":47,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.721587+0000","last_change":"2024-04-30T08:31:52.716043+0000","last_active":"2024-04-30T08:33:59.721587+0000","last_peered":"2024-04-30T08:33:59.721587+0000","last_clean":"2024-04-30T08:33:59.721587+0000","last_became_active":"2024-04-30T08:31:52.715284+0000","last_became_peered":"2024-04-30T08:31:52.715284+0000","last_unstale":"2024-04-30T08:33:59.721587+0000","last_undegraded":"2024-04-30T08:33:59.721587+0000","last_fullsized":"2024-04-30T08:33:59.721587+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:00:01.863606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.885198+0000","last_change":"2024-04-30T08:31:57.765008+0000","last_active":"2024-04-30T08:33:31.885198+0000","last_peered":"2024-04-30T08:33:31.885198+0000","last_clean":"2024-04-30T08:33:31.885198+0000","last_became_active":"2024-04-30T08:31:57.762958+0000","last_became_peered":"2024-04-30T08:31:57.762958+0000","last_unstale":"2024-04-30T08:33:31.885198+0000","last_undegraded":"2024-04-30T08:33:31.885198+0000","last_fullsized":"2024-04-30T08:33:31.885198+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:48:47.116004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"85'9","reported_seq":211,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:35:11.955731+0000","last_change":"2024-04-30T08:31:51.676681+0000","last_active":"2024-04-30T08:35:11.955731+0000","last_peered":"2024-04-30T08:35:11.955731+0000","last_clean":"2024-04-30T08:35:11.955731+0000","last_became_active":"2024-04-30T08:31:51.676311+0000","last_became_peered":"2024-04-30T08:31:51.676311+0000","last_unstale":"2024-04-30T08:35:11.955731+0000","last_undegraded":"2024-04-30T08:35:11.955731+0000","last_fullsized":"2024-04-30T08:35:11.955731+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:24:49.757427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1821,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":169,"num_read_kb":6,"num_write":9,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"85'6","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.687636+0000","last_change":"2024-04-30T08:31:52.689418+0000","last_active":"2024-04-30T08:33:59.687636+0000","last_peered":"2024-04-30T08:33:59.687636+0000","last_clean":"2024-04-30T08:33:59.687636+0000","last_became_active":"2024-04-30T08:31:52.688840+0000","last_became_peered":"2024-04-30T08:31:52.688840+0000","last_unstale":"2024-04-30T08:33:59.687636+0000","last_undegraded":"2024-04-30T08:33:59.687636+0000","last_fullsized":"2024-04-30T08:33:59.687636+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:40:05.781137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881271+0000","last_change":"2024-04-30T08:31:57.765875+0000","last_active":"2024-04-30T08:33:31.881271+0000","last_peered":"2024-04-30T08:33:31.881271+0000","last_clean":"2024-04-30T08:33:31.881271+0000","last_became_active":"2024-04-30T08:31:57.765432+0000","last_became_peered":"2024-04-30T08:31:57.765432+0000","last_unstale":"2024-04-30T08:33:31.881271+0000","last_undegraded":"2024-04-30T08:33:31.881271+0000","last_fullsized":"2024-04-30T08:33:31.881271+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:24:44.329532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"81'17","reported_seq":51,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.887076+0000","last_change":"2024-04-30T08:31:51.695007+0000","last_active":"2024-04-30T08:33:31.887076+0000","last_peered":"2024-04-30T08:33:31.887076+0000","last_clean":"2024-04-30T08:33:31.887076+0000","last_became_active":"2024-04-30T08:31:51.694533+0000","last_became_peered":"2024-04-30T08:31:51.694533+0000","last_unstale":"2024-04-30T08:33:31.887076+0000","last_undegraded":"2024-04-30T08:33:31.887076+0000","last_fullsized":"2024-04-30T08:33:31.887076+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:47:15.104172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"85'6","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.687169+0000","last_change":"2024-04-30T08:31:52.692274+0000","last_active":"2024-04-30T08:33:59.687169+0000","last_peered":"2024-04-30T08:33:59.687169+0000","last_clean":"2024-04-30T08:33:59.687169+0000","last_became_active":"2024-04-30T08:31:52.691809+0000","last_became_peered":"2024-04-30T08:31:52.691809+0000","last_unstale":"2024-04-30T08:33:59.687169+0000","last_undegraded":"2024-04-30T08:33:59.687169+0000","last_fullsized":"2024-04-30T08:33:59.687169+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:50:00.201782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273166+0000","last_change":"2024-04-30T08:31:57.732383+0000","last_active":"2024-04-30T08:33:32.273166+0000","last_peered":"2024-04-30T08:33:32.273166+0000","last_clean":"2024-04-30T08:33:32.273166+0000","last_became_active":"2024-04-30T08:31:57.731893+0000","last_became_peered":"2024-04-30T08:31:57.731893+0000","last_unstale":"2024-04-30T08:33:32.273166+0000","last_undegraded":"2024-04-30T08:33:32.273166+0000","last_fullsized":"2024-04-30T08:33:32.273166+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:08:55.243526+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'4","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:35:04.537295+0000","last_change":"2024-04-30T08:31:51.671991+0000","last_active":"2024-04-30T08:35:04.537295+0000","last_peered":"2024-04-30T08:35:04.537295+0000","last_clean":"2024-04-30T08:35:04.537295+0000","last_became_active":"2024-04-30T08:31:51.671508+0000","last_became_peered":"2024-04-30T08:31:51.671508+0000","last_unstale":"2024-04-30T08:35:04.537295+0000","last_undegraded":"2024-04-30T08:35:04.537295+0000","last_fullsized":"2024-04-30T08:35:04.537295+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:54:30.106748+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"85'5","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.689682+0000","last_change":"2024-04-30T08:33:32.274901+0000","last_active":"2024-04-30T08:33:59.689682+0000","last_peered":"2024-04-30T08:33:59.689682+0000","last_clean":"2024-04-30T08:33:59.689682+0000","last_became_active":"2024-04-30T08:33:32.274401+0000","last_became_peered":"2024-04-30T08:33:32.274401+0000","last_unstale":"2024-04-30T08:33:59.689682+0000","last_undegraded":"2024-04-30T08:33:59.689682+0000","last_fullsized":"2024-04-30T08:33:59.689682+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:23:08.703092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7],"acting":[11,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886092+0000","last_change":"2024-04-30T08:31:58.061077+0000","last_active":"2024-04-30T08:33:31.886092+0000","last_peered":"2024-04-30T08:33:31.886092+0000","last_clean":"2024-04-30T08:33:31.886092+0000","last_became_active":"2024-04-30T08:31:58.060486+0000","last_became_peered":"2024-04-30T08:31:58.060486+0000","last_unstale":"2024-04-30T08:33:31.886092+0000","last_undegraded":"2024-04-30T08:33:31.886092+0000","last_fullsized":"2024-04-30T08:33:31.886092+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:31:50.936560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"81'1","reported_seq":34,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.885199+0000","last_change":"2024-04-30T08:31:51.702869+0000","last_active":"2024-04-30T08:33:31.885199+0000","last_peered":"2024-04-30T08:33:31.885199+0000","last_clean":"2024-04-30T08:33:31.885199+0000","last_became_active":"2024-04-30T08:31:51.701150+0000","last_became_peered":"2024-04-30T08:31:51.701150+0000","last_unstale":"2024-04-30T08:33:31.885199+0000","last_undegraded":"2024-04-30T08:33:31.885199+0000","last_fullsized":"2024-04-30T08:33:31.885199+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:47:11.010310+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"85'4","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.716233+0000","last_change":"2024-04-30T08:31:52.709159+0000","last_active":"2024-04-30T08:33:59.716233+0000","last_peered":"2024-04-30T08:33:59.716233+0000","last_clean":"2024-04-30T08:33:59.716233+0000","last_became_active":"2024-04-30T08:31:52.708799+0000","last_became_peered":"2024-04-30T08:31:52.708799+0000","last_unstale":"2024-04-30T08:33:59.716233+0000","last_undegraded":"2024-04-30T08:33:59.716233+0000","last_fullsized":"2024-04-30T08:33:59.716233+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:51:10.588599+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881244+0000","last_change":"2024-04-30T08:31:57.751652+0000","last_active":"2024-04-30T08:33:31.881244+0000","last_peered":"2024-04-30T08:33:31.881244+0000","last_clean":"2024-04-30T08:33:31.881244+0000","last_became_active":"2024-04-30T08:31:57.751176+0000","last_became_peered":"2024-04-30T08:31:57.751176+0000","last_unstale":"2024-04-30T08:33:31.881244+0000","last_undegraded":"2024-04-30T08:33:31.881244+0000","last_fullsized":"2024-04-30T08:33:31.881244+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:13:24.528842+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"85'6","reported_seq":205,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:35:11.931645+0000","last_change":"2024-04-30T08:31:51.700816+0000","last_active":"2024-04-30T08:35:11.931645+0000","last_peered":"2024-04-30T08:35:11.931645+0000","last_clean":"2024-04-30T08:35:11.931645+0000","last_became_active":"2024-04-30T08:31:51.700470+0000","last_became_peered":"2024-04-30T08:31:51.700470+0000","last_unstale":"2024-04-30T08:35:11.931645+0000","last_undegraded":"2024-04-30T08:35:11.931645+0000","last_fullsized":"2024-04-30T08:35:11.931645+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T08:49:11.793412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":165,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"85'7","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.726100+0000","last_change":"2024-04-30T08:31:52.715236+0000","last_active":"2024-04-30T08:33:59.726100+0000","last_peered":"2024-04-30T08:33:59.726100+0000","last_clean":"2024-04-30T08:33:59.726100+0000","last_became_active":"2024-04-30T08:31:52.714801+0000","last_became_peered":"2024-04-30T08:31:52.714801+0000","last_unstale":"2024-04-30T08:33:59.726100+0000","last_undegraded":"2024-04-30T08:33:59.726100+0000","last_fullsized":"2024-04-30T08:33:59.726100+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:43:09.650002+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881702+0000","last_change":"2024-04-30T08:31:58.060927+0000","last_active":"2024-04-30T08:33:31.881702+0000","last_peered":"2024-04-30T08:33:31.881702+0000","last_clean":"2024-04-30T08:33:31.881702+0000","last_became_active":"2024-04-30T08:31:58.060182+0000","last_became_peered":"2024-04-30T08:31:58.060182+0000","last_unstale":"2024-04-30T08:33:31.881702+0000","last_undegraded":"2024-04-30T08:33:31.881702+0000","last_fullsized":"2024-04-30T08:33:31.881702+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:21:00.228064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"85'8","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.679643+0000","last_change":"2024-04-30T08:32:31.560485+0000","last_active":"2024-04-30T08:33:59.679643+0000","last_peered":"2024-04-30T08:33:59.679643+0000","last_clean":"2024-04-30T08:33:59.679643+0000","last_became_active":"2024-04-30T08:32:31.560066+0000","last_became_peered":"2024-04-30T08:32:31.560066+0000","last_unstale":"2024-04-30T08:33:59.679643+0000","last_undegraded":"2024-04-30T08:33:59.679643+0000","last_fullsized":"2024-04-30T08:33:59.679643+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:13:40.769761+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,8],"acting":[5,2,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"85'7","reported_seq":201,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:35:08.947184+0000","last_change":"2024-04-30T08:31:51.702808+0000","last_active":"2024-04-30T08:35:08.947184+0000","last_peered":"2024-04-30T08:35:08.947184+0000","last_clean":"2024-04-30T08:35:08.947184+0000","last_became_active":"2024-04-30T08:31:51.702326+0000","last_became_peered":"2024-04-30T08:31:51.702326+0000","last_unstale":"2024-04-30T08:35:08.947184+0000","last_undegraded":"2024-04-30T08:35:08.947184+0000","last_fullsized":"2024-04-30T08:35:08.947184+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:11:35.552670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":161,"num_read_kb":161,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"81'1","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.218503+0000","last_change":"2024-04-30T08:31:57.760359+0000","last_active":"2024-04-30T08:33:32.218503+0000","last_peered":"2024-04-30T08:33:32.218503+0000","last_clean":"2024-04-30T08:33:32.218503+0000","last_became_active":"2024-04-30T08:31:57.759821+0000","last_became_peered":"2024-04-30T08:31:57.759821+0000","last_unstale":"2024-04-30T08:33:32.218503+0000","last_undegraded":"2024-04-30T08:33:32.218503+0000","last_fullsized":"2024-04-30T08:33:32.218503+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:50:17.853791+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"85'8","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.719174+0000","last_change":"2024-04-30T08:31:52.689281+0000","last_active":"2024-04-30T08:33:59.719174+0000","last_peered":"2024-04-30T08:33:59.719174+0000","last_clean":"2024-04-30T08:33:59.719174+0000","last_became_active":"2024-04-30T08:31:52.688834+0000","last_became_peered":"2024-04-30T08:31:52.688834+0000","last_unstale":"2024-04-30T08:33:59.719174+0000","last_undegraded":"2024-04-30T08:33:59.719174+0000","last_fullsized":"2024-04-30T08:33:59.719174+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:42:12.836649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"81'6","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.169972+0000","last_change":"2024-04-30T08:31:51.683712+0000","last_active":"2024-04-30T08:33:32.169972+0000","last_peered":"2024-04-30T08:33:32.169972+0000","last_clean":"2024-04-30T08:33:32.169972+0000","last_became_active":"2024-04-30T08:31:51.683298+0000","last_became_peered":"2024-04-30T08:31:51.683298+0000","last_unstale":"2024-04-30T08:33:32.169972+0000","last_undegraded":"2024-04-30T08:33:32.169972+0000","last_fullsized":"2024-04-30T08:33:32.169972+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:43:08.832104+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"20'92","reported_seq":109,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.217885+0000","last_change":"2024-04-30T08:30:26.503221+0000","last_active":"2024-04-30T08:33:32.217885+0000","last_peered":"2024-04-30T08:33:32.217885+0000","last_clean":"2024-04-30T08:33:32.217885+0000","last_became_active":"2024-04-30T08:30:26.498399+0000","last_became_peered":"2024-04-30T08:30:26.498399+0000","last_unstale":"2024-04-30T08:33:32.217885+0000","last_undegraded":"2024-04-30T08:33:32.217885+0000","last_fullsized":"2024-04-30T08:33:32.217885+0000","mapping_epoch":66,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":67,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:26:20.170075+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:26:20.170075+0000","last_clean_scrub_stamp":"2024-04-30T08:26:20.170075+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:34:44.067560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273554+0000","last_change":"2024-04-30T08:31:58.061283+0000","last_active":"2024-04-30T08:33:32.273554+0000","last_peered":"2024-04-30T08:33:32.273554+0000","last_clean":"2024-04-30T08:33:32.273554+0000","last_became_active":"2024-04-30T08:31:58.060650+0000","last_became_peered":"2024-04-30T08:31:58.060650+0000","last_unstale":"2024-04-30T08:33:32.273554+0000","last_undegraded":"2024-04-30T08:33:32.273554+0000","last_fullsized":"2024-04-30T08:33:32.273554+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:13:01.811462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"85'5","reported_seq":36,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.728420+0000","last_change":"2024-04-30T08:31:52.699639+0000","last_active":"2024-04-30T08:33:59.728420+0000","last_peered":"2024-04-30T08:33:59.728420+0000","last_clean":"2024-04-30T08:33:59.728420+0000","last_became_active":"2024-04-30T08:31:52.699164+0000","last_became_peered":"2024-04-30T08:31:52.699164+0000","last_unstale":"2024-04-30T08:33:59.728420+0000","last_undegraded":"2024-04-30T08:33:59.728420+0000","last_fullsized":"2024-04-30T08:33:59.728420+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:22:18.873504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"81'4","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.217821+0000","last_change":"2024-04-30T08:31:51.679153+0000","last_active":"2024-04-30T08:33:32.217821+0000","last_peered":"2024-04-30T08:33:32.217821+0000","last_clean":"2024-04-30T08:33:32.217821+0000","last_became_active":"2024-04-30T08:31:51.678348+0000","last_became_peered":"2024-04-30T08:31:51.678348+0000","last_unstale":"2024-04-30T08:33:32.217821+0000","last_undegraded":"2024-04-30T08:33:32.217821+0000","last_fullsized":"2024-04-30T08:33:32.217821+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:02:56.240218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886616+0000","last_change":"2024-04-30T08:31:57.755423+0000","last_active":"2024-04-30T08:33:31.886616+0000","last_peered":"2024-04-30T08:33:31.886616+0000","last_clean":"2024-04-30T08:33:31.886616+0000","last_became_active":"2024-04-30T08:31:57.754835+0000","last_became_peered":"2024-04-30T08:31:57.754835+0000","last_unstale":"2024-04-30T08:33:31.886616+0000","last_undegraded":"2024-04-30T08:33:31.886616+0000","last_fullsized":"2024-04-30T08:33:31.886616+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:19:25.168920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"81'4","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886919+0000","last_change":"2024-04-30T08:31:51.693839+0000","last_active":"2024-04-30T08:33:31.886919+0000","last_peered":"2024-04-30T08:33:31.886919+0000","last_clean":"2024-04-30T08:33:31.886919+0000","last_became_active":"2024-04-30T08:31:51.693349+0000","last_became_peered":"2024-04-30T08:31:51.693349+0000","last_unstale":"2024-04-30T08:33:31.886919+0000","last_undegraded":"2024-04-30T08:33:31.886919+0000","last_fullsized":"2024-04-30T08:33:31.886919+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:41:51.330901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"85'9","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.733847+0000","last_change":"2024-04-30T08:31:52.712360+0000","last_active":"2024-04-30T08:33:59.733847+0000","last_peered":"2024-04-30T08:33:59.733847+0000","last_clean":"2024-04-30T08:33:59.733847+0000","last_became_active":"2024-04-30T08:31:52.711652+0000","last_became_peered":"2024-04-30T08:31:52.711652+0000","last_unstale":"2024-04-30T08:33:59.733847+0000","last_undegraded":"2024-04-30T08:33:59.733847+0000","last_fullsized":"2024-04-30T08:33:59.733847+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:21:10.285386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.276738+0000","last_change":"2024-04-30T08:31:58.061654+0000","last_active":"2024-04-30T08:33:32.276738+0000","last_peered":"2024-04-30T08:33:32.276738+0000","last_clean":"2024-04-30T08:33:32.276738+0000","last_became_active":"2024-04-30T08:31:58.060181+0000","last_became_peered":"2024-04-30T08:31:58.060181+0000","last_unstale":"2024-04-30T08:33:32.276738+0000","last_undegraded":"2024-04-30T08:33:32.276738+0000","last_fullsized":"2024-04-30T08:33:32.276738+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:46:03.230626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'15","reported_seq":48,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:35:04.265838+0000","last_change":"2024-04-30T08:31:51.706940+0000","last_active":"2024-04-30T08:35:04.265838+0000","last_peered":"2024-04-30T08:35:04.265838+0000","last_clean":"2024-04-30T08:35:04.265838+0000","last_became_active":"2024-04-30T08:31:51.706481+0000","last_became_peered":"2024-04-30T08:31:51.706481+0000","last_unstale":"2024-04-30T08:35:04.265838+0000","last_undegraded":"2024-04-30T08:35:04.265838+0000","last_fullsized":"2024-04-30T08:35:04.265838+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:53:31.955595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"85'5","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.714966+0000","last_change":"2024-04-30T08:31:52.696625+0000","last_active":"2024-04-30T08:33:59.714966+0000","last_peered":"2024-04-30T08:33:59.714966+0000","last_clean":"2024-04-30T08:33:59.714966+0000","last_became_active":"2024-04-30T08:31:52.696230+0000","last_became_peered":"2024-04-30T08:31:52.696230+0000","last_unstale":"2024-04-30T08:33:59.714966+0000","last_undegraded":"2024-04-30T08:33:59.714966+0000","last_fullsized":"2024-04-30T08:33:59.714966+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:11:46.404251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.879436+0000","last_change":"2024-04-30T08:31:57.738329+0000","last_active":"2024-04-30T08:33:31.879436+0000","last_peered":"2024-04-30T08:33:31.879436+0000","last_clean":"2024-04-30T08:33:31.879436+0000","last_became_active":"2024-04-30T08:31:57.737897+0000","last_became_peered":"2024-04-30T08:31:57.737897+0000","last_unstale":"2024-04-30T08:33:31.879436+0000","last_undegraded":"2024-04-30T08:33:31.879436+0000","last_fullsized":"2024-04-30T08:33:31.879436+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T17:02:43.074578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"81'7","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.170374+0000","last_change":"2024-04-30T08:31:51.684066+0000","last_active":"2024-04-30T08:33:32.170374+0000","last_peered":"2024-04-30T08:33:32.170374+0000","last_clean":"2024-04-30T08:33:32.170374+0000","last_became_active":"2024-04-30T08:31:51.683699+0000","last_became_peered":"2024-04-30T08:31:51.683699+0000","last_unstale":"2024-04-30T08:33:32.170374+0000","last_undegraded":"2024-04-30T08:33:32.170374+0000","last_fullsized":"2024-04-30T08:33:32.170374+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:36:20.900733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"85'7","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.740363+0000","last_change":"2024-04-30T08:31:52.715219+0000","last_active":"2024-04-30T08:33:59.740363+0000","last_peered":"2024-04-30T08:33:59.740363+0000","last_clean":"2024-04-30T08:33:59.740363+0000","last_became_active":"2024-04-30T08:31:52.714774+0000","last_became_peered":"2024-04-30T08:31:52.714774+0000","last_unstale":"2024-04-30T08:33:59.740363+0000","last_undegraded":"2024-04-30T08:33:59.740363+0000","last_fullsized":"2024-04-30T08:33:59.740363+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:42:41.082610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.170157+0000","last_change":"2024-04-30T08:31:58.060845+0000","last_active":"2024-04-30T08:33:32.170157+0000","last_peered":"2024-04-30T08:33:32.170157+0000","last_clean":"2024-04-30T08:33:32.170157+0000","last_became_active":"2024-04-30T08:31:58.060365+0000","last_became_peered":"2024-04-30T08:31:58.060365+0000","last_unstale":"2024-04-30T08:33:32.170157+0000","last_undegraded":"2024-04-30T08:33:32.170157+0000","last_fullsized":"2024-04-30T08:33:32.170157+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:21:39.583243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"81'2","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.272616+0000","last_change":"2024-04-30T08:31:51.694833+0000","last_active":"2024-04-30T08:33:32.272616+0000","last_peered":"2024-04-30T08:33:32.272616+0000","last_clean":"2024-04-30T08:33:32.272616+0000","last_became_active":"2024-04-30T08:31:51.694401+0000","last_became_peered":"2024-04-30T08:31:51.694401+0000","last_unstale":"2024-04-30T08:33:32.272616+0000","last_undegraded":"2024-04-30T08:33:32.272616+0000","last_fullsized":"2024-04-30T08:33:32.272616+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T18:38:52.605625+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"85'2","reported_seq":33,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.676614+0000","last_change":"2024-04-30T08:31:52.686444+0000","last_active":"2024-04-30T08:33:59.676614+0000","last_peered":"2024-04-30T08:33:59.676614+0000","last_clean":"2024-04-30T08:33:59.676614+0000","last_became_active":"2024-04-30T08:31:52.686057+0000","last_became_peered":"2024-04-30T08:31:52.686057+0000","last_unstale":"2024-04-30T08:33:59.676614+0000","last_undegraded":"2024-04-30T08:33:59.676614+0000","last_fullsized":"2024-04-30T08:33:59.676614+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:54:29.824060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.885416+0000","last_change":"2024-04-30T08:31:57.766193+0000","last_active":"2024-04-30T08:33:31.885416+0000","last_peered":"2024-04-30T08:33:31.885416+0000","last_clean":"2024-04-30T08:33:31.885416+0000","last_became_active":"2024-04-30T08:31:57.765758+0000","last_became_peered":"2024-04-30T08:31:57.765758+0000","last_unstale":"2024-04-30T08:33:31.885416+0000","last_undegraded":"2024-04-30T08:33:31.885416+0000","last_fullsized":"2024-04-30T08:33:31.885416+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:59:57.282795+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"81'7","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.275654+0000","last_change":"2024-04-30T08:31:51.704565+0000","last_active":"2024-04-30T08:33:32.275654+0000","last_peered":"2024-04-30T08:33:32.275654+0000","last_clean":"2024-04-30T08:33:32.275654+0000","last_became_active":"2024-04-30T08:31:51.703954+0000","last_became_peered":"2024-04-30T08:31:51.703954+0000","last_unstale":"2024-04-30T08:33:32.275654+0000","last_undegraded":"2024-04-30T08:33:32.275654+0000","last_fullsized":"2024-04-30T08:33:32.275654+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:03:00.517867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"85'9","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.717404+0000","last_change":"2024-04-30T08:31:52.706043+0000","last_active":"2024-04-30T08:33:59.717404+0000","last_peered":"2024-04-30T08:33:59.717404+0000","last_clean":"2024-04-30T08:33:59.717404+0000","last_became_active":"2024-04-30T08:31:52.705678+0000","last_became_peered":"2024-04-30T08:31:52.705678+0000","last_unstale":"2024-04-30T08:33:59.717404+0000","last_undegraded":"2024-04-30T08:33:59.717404+0000","last_fullsized":"2024-04-30T08:33:59.717404+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:44:56.787835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"82'1","reported_seq":24,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.170123+0000","last_change":"2024-04-30T08:31:57.764791+0000","last_active":"2024-04-30T08:33:32.170123+0000","last_peered":"2024-04-30T08:33:32.170123+0000","last_clean":"2024-04-30T08:33:32.170123+0000","last_became_active":"2024-04-30T08:31:57.764368+0000","last_became_peered":"2024-04-30T08:31:57.764368+0000","last_unstale":"2024-04-30T08:33:32.170123+0000","last_undegraded":"2024-04-30T08:33:32.170123+0000","last_fullsized":"2024-04-30T08:33:32.170123+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:19:28.545483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"81'3","reported_seq":22,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886461+0000","last_change":"2024-04-30T08:32:31.577309+0000","last_active":"2024-04-30T08:33:31.886461+0000","last_peered":"2024-04-30T08:33:31.886461+0000","last_clean":"2024-04-30T08:33:31.886461+0000","last_became_active":"2024-04-30T08:32:31.571515+0000","last_became_peered":"2024-04-30T08:32:31.571515+0000","last_unstale":"2024-04-30T08:33:31.886461+0000","last_undegraded":"2024-04-30T08:33:31.886461+0000","last_fullsized":"2024-04-30T08:33:31.886461+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:28:48.408414+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"85'8","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.730222+0000","last_change":"2024-04-30T08:31:52.716707+0000","last_active":"2024-04-30T08:33:59.730222+0000","last_peered":"2024-04-30T08:33:59.730222+0000","last_clean":"2024-04-30T08:33:59.730222+0000","last_became_active":"2024-04-30T08:31:52.716293+0000","last_became_peered":"2024-04-30T08:31:52.716293+0000","last_unstale":"2024-04-30T08:33:59.730222+0000","last_undegraded":"2024-04-30T08:33:59.730222+0000","last_fullsized":"2024-04-30T08:33:59.730222+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T09:29:54.223472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.170159+0000","last_change":"2024-04-30T08:31:58.060640+0000","last_active":"2024-04-30T08:33:32.170159+0000","last_peered":"2024-04-30T08:33:32.170159+0000","last_clean":"2024-04-30T08:33:32.170159+0000","last_became_active":"2024-04-30T08:31:58.060061+0000","last_became_peered":"2024-04-30T08:31:58.060061+0000","last_unstale":"2024-04-30T08:33:32.170159+0000","last_undegraded":"2024-04-30T08:33:32.170159+0000","last_fullsized":"2024-04-30T08:33:32.170159+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:46:00.844191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"81'4","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886171+0000","last_change":"2024-04-30T08:31:51.679532+0000","last_active":"2024-04-30T08:33:31.886171+0000","last_peered":"2024-04-30T08:33:31.886171+0000","last_clean":"2024-04-30T08:33:31.886171+0000","last_became_active":"2024-04-30T08:31:51.679141+0000","last_became_peered":"2024-04-30T08:31:51.679141+0000","last_unstale":"2024-04-30T08:33:31.886171+0000","last_undegraded":"2024-04-30T08:33:31.886171+0000","last_fullsized":"2024-04-30T08:33:31.886171+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:19:50.526563+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"85'5","reported_seq":37,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.720554+0000","last_change":"2024-04-30T08:31:52.703613+0000","last_active":"2024-04-30T08:33:59.720554+0000","last_peered":"2024-04-30T08:33:59.720554+0000","last_clean":"2024-04-30T08:33:59.720554+0000","last_became_active":"2024-04-30T08:31:52.703196+0000","last_became_peered":"2024-04-30T08:31:52.703196+0000","last_unstale":"2024-04-30T08:33:59.720554+0000","last_undegraded":"2024-04-30T08:33:59.720554+0000","last_fullsized":"2024-04-30T08:33:59.720554+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:33:11.136313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.272631+0000","last_change":"2024-04-30T08:31:57.755286+0000","last_active":"2024-04-30T08:33:32.272631+0000","last_peered":"2024-04-30T08:33:32.272631+0000","last_clean":"2024-04-30T08:33:32.272631+0000","last_became_active":"2024-04-30T08:31:57.754786+0000","last_became_peered":"2024-04-30T08:31:57.754786+0000","last_unstale":"2024-04-30T08:33:32.272631+0000","last_undegraded":"2024-04-30T08:33:32.272631+0000","last_fullsized":"2024-04-30T08:33:32.272631+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:02:50.931553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"81'5","reported_seq":198,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:35:07.811337+0000","last_change":"2024-04-30T08:31:51.693317+0000","last_active":"2024-04-30T08:35:07.811337+0000","last_peered":"2024-04-30T08:35:07.811337+0000","last_clean":"2024-04-30T08:35:07.811337+0000","last_became_active":"2024-04-30T08:31:51.692928+0000","last_became_peered":"2024-04-30T08:31:51.692928+0000","last_unstale":"2024-04-30T08:35:07.811337+0000","last_undegraded":"2024-04-30T08:35:07.811337+0000","last_fullsized":"2024-04-30T08:35:07.811337+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:35:09.851735+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":160,"num_read_kb":2,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"85'7","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.692096+0000","last_change":"2024-04-30T08:31:52.694304+0000","last_active":"2024-04-30T08:33:59.692096+0000","last_peered":"2024-04-30T08:33:59.692096+0000","last_clean":"2024-04-30T08:33:59.692096+0000","last_became_active":"2024-04-30T08:31:52.693317+0000","last_became_peered":"2024-04-30T08:31:52.693317+0000","last_unstale":"2024-04-30T08:33:59.692096+0000","last_undegraded":"2024-04-30T08:33:59.692096+0000","last_fullsized":"2024-04-30T08:33:59.692096+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:31:39.294864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.799022+0000","last_change":"2024-04-30T08:31:57.763598+0000","last_active":"2024-04-30T08:33:32.799022+0000","last_peered":"2024-04-30T08:33:32.799022+0000","last_clean":"2024-04-30T08:33:32.799022+0000","last_became_active":"2024-04-30T08:31:57.762307+0000","last_became_peered":"2024-04-30T08:31:57.762307+0000","last_unstale":"2024-04-30T08:33:32.799022+0000","last_undegraded":"2024-04-30T08:33:32.799022+0000","last_fullsized":"2024-04-30T08:33:32.799022+0000","mapping_epoch":79,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":80,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:24:13.390595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"81'3","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:32.273810+0000","last_change":"2024-04-30T08:31:51.703476+0000","last_active":"2024-04-30T08:33:32.273810+0000","last_peered":"2024-04-30T08:33:32.273810+0000","last_clean":"2024-04-30T08:33:32.273810+0000","last_became_active":"2024-04-30T08:31:51.703072+0000","last_became_peered":"2024-04-30T08:31:51.703072+0000","last_unstale":"2024-04-30T08:33:32.273810+0000","last_undegraded":"2024-04-30T08:33:32.273810+0000","last_fullsized":"2024-04-30T08:33:32.273810+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:07:21.098223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"85'6","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.698707+0000","last_change":"2024-04-30T08:31:52.699223+0000","last_active":"2024-04-30T08:33:59.698707+0000","last_peered":"2024-04-30T08:33:59.698707+0000","last_clean":"2024-04-30T08:33:59.698707+0000","last_became_active":"2024-04-30T08:31:52.698837+0000","last_became_peered":"2024-04-30T08:31:52.698837+0000","last_unstale":"2024-04-30T08:33:59.698707+0000","last_undegraded":"2024-04-30T08:33:59.698707+0000","last_fullsized":"2024-04-30T08:33:59.698707+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:19:11.826470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.886774+0000","last_change":"2024-04-30T08:32:31.586667+0000","last_active":"2024-04-30T08:33:31.886774+0000","last_peered":"2024-04-30T08:33:31.886774+0000","last_clean":"2024-04-30T08:33:31.886774+0000","last_became_active":"2024-04-30T08:32:31.586144+0000","last_became_peered":"2024-04-30T08:32:31.586144+0000","last_unstale":"2024-04-30T08:33:31.886774+0000","last_undegraded":"2024-04-30T08:33:31.886774+0000","last_fullsized":"2024-04-30T08:33:31.886774+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":79,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:56.685471+0000","last_clean_scrub_stamp":"2024-04-30T08:31:56.685471+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:50:39.752485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7,1],"acting":[8,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.e","version":"81'5","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:31.881669+0000","last_change":"2024-04-30T08:31:51.708943+0000","last_active":"2024-04-30T08:33:31.881669+0000","last_peered":"2024-04-30T08:33:31.881669+0000","last_clean":"2024-04-30T08:33:31.881669+0000","last_became_active":"2024-04-30T08:31:51.708376+0000","last_became_peered":"2024-04-30T08:31:51.708376+0000","last_unstale":"2024-04-30T08:33:31.881669+0000","last_undegraded":"2024-04-30T08:33:31.881669+0000","last_fullsized":"2024-04-30T08:33:31.881669+0000","mapping_epoch":73,"log_start":"0'0","ondisk_log_start":"0'0","created":73,"last_epoch_clean":74,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:50.645568+0000","last_clean_scrub_stamp":"2024-04-30T08:31:50.645568+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:14:46.692192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"85'9","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.690244+0000","last_change":"2024-04-30T08:31:52.686634+0000","last_active":"2024-04-30T08:33:59.690244+0000","last_peered":"2024-04-30T08:33:59.690244+0000","last_clean":"2024-04-30T08:33:59.690244+0000","last_became_active":"2024-04-30T08:31:52.686231+0000","last_became_peered":"2024-04-30T08:31:52.686231+0000","last_unstale":"2024-04-30T08:33:59.690244+0000","last_undegraded":"2024-04-30T08:33:59.690244+0000","last_fullsized":"2024-04-30T08:33:59.690244+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:27:31.073041+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"85'9","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.689734+0000","last_change":"2024-04-30T08:31:52.688695+0000","last_active":"2024-04-30T08:33:59.689734+0000","last_peered":"2024-04-30T08:33:59.689734+0000","last_clean":"2024-04-30T08:33:59.689734+0000","last_became_active":"2024-04-30T08:31:52.686556+0000","last_became_peered":"2024-04-30T08:31:52.686556+0000","last_unstale":"2024-04-30T08:33:59.689734+0000","last_undegraded":"2024-04-30T08:33:59.689734+0000","last_fullsized":"2024-04-30T08:33:59.689734+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:22:48.976440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"85'12","reported_seq":43,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.683087+0000","last_change":"2024-04-30T08:31:52.688324+0000","last_active":"2024-04-30T08:33:59.683087+0000","last_peered":"2024-04-30T08:33:59.683087+0000","last_clean":"2024-04-30T08:33:59.683087+0000","last_became_active":"2024-04-30T08:31:52.687966+0000","last_became_peered":"2024-04-30T08:31:52.687966+0000","last_unstale":"2024-04-30T08:33:59.683087+0000","last_undegraded":"2024-04-30T08:33:59.683087+0000","last_fullsized":"2024-04-30T08:33:59.683087+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T15:35:38.237750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"85'8","reported_seq":39,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.706258+0000","last_change":"2024-04-30T08:31:52.695638+0000","last_active":"2024-04-30T08:33:59.706258+0000","last_peered":"2024-04-30T08:33:59.706258+0000","last_clean":"2024-04-30T08:33:59.706258+0000","last_became_active":"2024-04-30T08:31:52.695283+0000","last_became_peered":"2024-04-30T08:31:52.695283+0000","last_unstale":"2024-04-30T08:33:59.706258+0000","last_undegraded":"2024-04-30T08:33:59.706258+0000","last_fullsized":"2024-04-30T08:33:59.706258+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T10:00:14.510381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"85'7","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.739539+0000","last_change":"2024-04-30T08:31:52.719112+0000","last_active":"2024-04-30T08:33:59.739539+0000","last_peered":"2024-04-30T08:33:59.739539+0000","last_clean":"2024-04-30T08:33:59.739539+0000","last_became_active":"2024-04-30T08:31:52.718207+0000","last_became_peered":"2024-04-30T08:31:52.718207+0000","last_unstale":"2024-04-30T08:33:59.739539+0000","last_undegraded":"2024-04-30T08:33:59.739539+0000","last_fullsized":"2024-04-30T08:33:59.739539+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:27:29.263962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"85'4","reported_seq":35,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.734901+0000","last_change":"2024-04-30T08:31:52.712008+0000","last_active":"2024-04-30T08:33:59.734901+0000","last_peered":"2024-04-30T08:33:59.734901+0000","last_clean":"2024-04-30T08:33:59.734901+0000","last_became_active":"2024-04-30T08:31:52.711064+0000","last_became_peered":"2024-04-30T08:31:52.711064+0000","last_unstale":"2024-04-30T08:33:59.734901+0000","last_undegraded":"2024-04-30T08:33:59.734901+0000","last_fullsized":"2024-04-30T08:33:59.734901+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T11:57:27.400602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"85'6","reported_seq":38,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.688657+0000","last_change":"2024-04-30T08:31:52.691210+0000","last_active":"2024-04-30T08:33:59.688657+0000","last_peered":"2024-04-30T08:33:59.688657+0000","last_clean":"2024-04-30T08:33:59.688657+0000","last_became_active":"2024-04-30T08:31:52.688852+0000","last_became_peered":"2024-04-30T08:31:52.688852+0000","last_unstale":"2024-04-30T08:33:59.688657+0000","last_undegraded":"2024-04-30T08:33:59.688657+0000","last_fullsized":"2024-04-30T08:33:59.688657+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T14:04:46.502743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"85'15","reported_seq":46,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.684401+0000","last_change":"2024-04-30T08:31:52.684501+0000","last_active":"2024-04-30T08:33:59.684401+0000","last_peered":"2024-04-30T08:33:59.684401+0000","last_clean":"2024-04-30T08:33:59.684401+0000","last_became_active":"2024-04-30T08:31:52.684122+0000","last_became_peered":"2024-04-30T08:31:52.684122+0000","last_unstale":"2024-04-30T08:33:59.684401+0000","last_undegraded":"2024-04-30T08:33:59.684401+0000","last_fullsized":"2024-04-30T08:33:59.684401+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:55:58.093269+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"85'2","reported_seq":34,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.713723+0000","last_change":"2024-04-30T08:31:52.698016+0000","last_active":"2024-04-30T08:33:59.713723+0000","last_peered":"2024-04-30T08:33:59.713723+0000","last_clean":"2024-04-30T08:33:59.713723+0000","last_became_active":"2024-04-30T08:31:52.695037+0000","last_became_peered":"2024-04-30T08:31:52.695037+0000","last_unstale":"2024-04-30T08:33:59.713723+0000","last_undegraded":"2024-04-30T08:33:59.713723+0000","last_fullsized":"2024-04-30T08:33:59.713723+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T20:06:01.360947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"85'6","reported_seq":17,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.714903+0000","last_change":"2024-04-30T08:33:31.889445+0000","last_active":"2024-04-30T08:33:59.714903+0000","last_peered":"2024-04-30T08:33:59.714903+0000","last_clean":"2024-04-30T08:33:59.714903+0000","last_became_active":"2024-04-30T08:33:31.889110+0000","last_became_peered":"2024-04-30T08:33:31.889110+0000","last_unstale":"2024-04-30T08:33:59.714903+0000","last_undegraded":"2024-04-30T08:33:59.714903+0000","last_fullsized":"2024-04-30T08:33:59.714903+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T12:34:26.741854+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,6],"acting":[8,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.29","version":"85'8","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.699538+0000","last_change":"2024-04-30T08:31:52.694776+0000","last_active":"2024-04-30T08:33:59.699538+0000","last_peered":"2024-04-30T08:33:59.699538+0000","last_clean":"2024-04-30T08:33:59.699538+0000","last_became_active":"2024-04-30T08:31:52.693532+0000","last_became_peered":"2024-04-30T08:31:52.693532+0000","last_unstale":"2024-04-30T08:33:59.699538+0000","last_undegraded":"2024-04-30T08:33:59.699538+0000","last_fullsized":"2024-04-30T08:33:59.699538+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T16:31:10.186667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"85'10","reported_seq":41,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.706242+0000","last_change":"2024-04-30T08:31:52.688477+0000","last_active":"2024-04-30T08:33:59.706242+0000","last_peered":"2024-04-30T08:33:59.706242+0000","last_clean":"2024-04-30T08:33:59.706242+0000","last_became_active":"2024-04-30T08:31:52.688072+0000","last_became_peered":"2024-04-30T08:31:52.688072+0000","last_unstale":"2024-04-30T08:33:59.706242+0000","last_undegraded":"2024-04-30T08:33:59.706242+0000","last_fullsized":"2024-04-30T08:33:59.706242+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T19:35:42.659707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"85'11","reported_seq":42,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.738258+0000","last_change":"2024-04-30T08:31:52.716788+0000","last_active":"2024-04-30T08:33:59.738258+0000","last_peered":"2024-04-30T08:33:59.738258+0000","last_clean":"2024-04-30T08:33:59.738258+0000","last_became_active":"2024-04-30T08:31:52.716336+0000","last_became_peered":"2024-04-30T08:31:52.716336+0000","last_unstale":"2024-04-30T08:33:59.738258+0000","last_undegraded":"2024-04-30T08:33:59.738258+0000","last_fullsized":"2024-04-30T08:33:59.738258+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:23:23.477968+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"85'8","reported_seq":40,"reported_epoch":85,"state":"active+clean","last_fresh":"2024-04-30T08:33:59.681192+0000","last_change":"2024-04-30T08:31:52.684571+0000","last_active":"2024-04-30T08:33:59.681192+0000","last_peered":"2024-04-30T08:33:59.681192+0000","last_clean":"2024-04-30T08:33:59.681192+0000","last_became_active":"2024-04-30T08:31:52.684192+0000","last_became_peered":"2024-04-30T08:31:52.684192+0000","last_unstale":"2024-04-30T08:33:59.681192+0000","last_undegraded":"2024-04-30T08:33:59.681192+0000","last_fullsized":"2024-04-30T08:33:59.681192+0000","mapping_epoch":74,"log_start":"0'0","ondisk_log_start":"0'0","created":74,"last_epoch_clean":75,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-30T08:31:51.652740+0000","last_clean_scrub_stamp":"2024-04-30T08:31:51.652740+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-01T13:05:57.520133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":64,"num_write":123,"num_write_kb":2150,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284927,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":92,"ondisk_log_size":92,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":75914,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1646,"num_read_kb":902,"num_write":266,"num_write_kb":163,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":68,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":577536,"data_stored":230262,"data_compressed":5523,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":230,"ondisk_log_size":230,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":71,"seq":304942678071,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32100,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713052,"statfs":{"total":95995035648,"available":95962165248,"internally_reserved":0,"allocated":946176,"data_stored":479251,"data_compressed":26652,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1596,"internal_metadata":31914436},"hb_peers":[0,1,2,3,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":65,"seq":279172874299,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28080,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717072,"statfs":{"total":95995035648,"available":95966281728,"internally_reserved":0,"allocated":1089536,"data_stored":1134568,"data_compressed":64719,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":60,"seq":257698037826,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":483209,"data_compressed":26652,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":54,"seq":231928234058,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32112,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713040,"statfs":{"total":95995035648,"available":95962152960,"internally_reserved":0,"allocated":958464,"data_stored":479491,"data_compressed":26652,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":48,"seq":206158430287,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1011712,"data_stored":1077753,"data_compressed":62878,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":42,"seq":180388626515,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":937984,"data_stored":475008,"data_compressed":26652,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574962,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36320,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708832,"statfs":{"total":95995035648,"available":95957843968,"internally_reserved":0,"allocated":1007616,"data_stored":545894,"data_compressed":28493,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738486,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32116,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713036,"statfs":{"total":95995035648,"available":95962148864,"internally_reserved":0,"allocated":962560,"data_stored":476073,"data_compressed":26652,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411437,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36332,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708820,"statfs":{"total":95995035648,"available":95957831680,"internally_reserved":0,"allocated":1019904,"data_stored":543667,"data_compressed":28493,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1594,"internal_metadata":36174278},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182502,"num_pgs":38,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27944,"kb_used_data":928,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717208,"statfs":{"total":95995035648,"available":95966420992,"internally_reserved":0,"allocated":950272,"data_stored":474325,"data_compressed":26652,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986273,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":924,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":946176,"data_stored":479924,"data_compressed":26652,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":36,"seq":154618822747,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":1071711,"data_compressed":62878,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34599,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34599,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34599,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36226,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":71999,"data_compressed":1841,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":69772,"data_compressed":1841,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1933,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1113,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":5298,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1500,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":118784,"data_stored":70305,"data_compressed":1841,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-30T08:35:14.642 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-30T08:35:14.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.975+0000 7fe323200640 1 -- 172.21.15.32:0/3447938973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe324103990 msgr2=0x7fe324104690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:14.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.975+0000 7fe323200640 1 --2- 172.21.15.32:0/3447938973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe324103990 0x7fe324104690 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe314009900 tx=0x7fe31402a410 comp rx=0 tx=0).stop 2024-04-30T08:35:14.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.976+0000 7fe323200640 1 -- 172.21.15.32:0/3447938973 shutdown_connections 2024-04-30T08:35:14.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.976+0000 7fe323200640 1 --2- 172.21.15.32:0/3447938973 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3241098d0 0x7fe32410bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:14.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.976+0000 7fe323200640 1 --2- 172.21.15.32:0/3447938973 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe324104bd0 0x7fe324104fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:14.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.976+0000 7fe323200640 1 --2- 172.21.15.32:0/3447938973 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe324103990 0x7fe324104690 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:14.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.976+0000 7fe323200640 1 -- 172.21.15.32:0/3447938973 >> 172.21.15.32:0/3447938973 conn(0x7fe3240feba0 msgr2=0x7fe324101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:14.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.976+0000 7fe323200640 1 -- 172.21.15.32:0/3447938973 shutdown_connections 2024-04-30T08:35:14.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.976+0000 7fe323200640 1 -- 172.21.15.32:0/3447938973 wait complete. 2024-04-30T08:35:14.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.977+0000 7fe323200640 1 Processor -- start 2024-04-30T08:35:14.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.977+0000 7fe323200640 1 -- start start 2024-04-30T08:35:14.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.977+0000 7fe323200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe324103990 0x7fe32419c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:14.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.977+0000 7fe323200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe324104bd0 0x7fe32419c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:14.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.977+0000 7fe323200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3241098d0 0x7fe3241a37f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:14.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.977+0000 7fe323200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe32410f490 con 0x7fe324103990 2024-04-30T08:35:14.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.977+0000 7fe323200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe32410f310 con 0x7fe3241098d0 2024-04-30T08:35:14.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.977+0000 7fe323200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe32410f610 con 0x7fe324104bd0 2024-04-30T08:35:14.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.978+0000 7fe321e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe324103990 0x7fe32419c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:14.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.978+0000 7fe321e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe324103990 0x7fe32419c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53666/0 (socket says 172.21.15.32:53666) 2024-04-30T08:35:14.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.978+0000 7fe321e00640 1 -- 172.21.15.32:0/2199028548 learned_addr learned my addr 172.21.15.32:0/2199028548 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:14.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.978+0000 7fe321400640 1 --2- 172.21.15.32:0/2199028548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe324104bd0 0x7fe32419c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:14.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.978+0000 7fe322800640 1 --2- 172.21.15.32:0/2199028548 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3241098d0 0x7fe3241a37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:14.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.978+0000 7fe321e00640 1 -- 172.21.15.32:0/2199028548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe324104bd0 msgr2=0x7fe32419c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:14.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.978+0000 7fe321e00640 1 --2- 172.21.15.32:0/2199028548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe324104bd0 0x7fe32419c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:14.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.978+0000 7fe321e00640 1 -- 172.21.15.32:0/2199028548 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3241098d0 msgr2=0x7fe3241a37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:14.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.978+0000 7fe321e00640 1 --2- 172.21.15.32:0/2199028548 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3241098d0 0x7fe3241a37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:14.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.978+0000 7fe321e00640 1 -- 172.21.15.32:0/2199028548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3241a3e60 con 0x7fe324103990 2024-04-30T08:35:14.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.978+0000 7fe321400640 1 --2- 172.21.15.32:0/2199028548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe324104bd0 0x7fe32419c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:14.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.979+0000 7fe321e00640 1 --2- 172.21.15.32:0/2199028548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe324103990 0x7fe32419c320 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe314006010 tx=0x7fe314004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:14.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.979+0000 7fe31aa00640 1 -- 172.21.15.32:0/2199028548 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe31403f030 con 0x7fe324103990 2024-04-30T08:35:14.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.979+0000 7fe31aa00640 1 -- 172.21.15.32:0/2199028548 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe314031910 con 0x7fe324103990 2024-04-30T08:35:14.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.979+0000 7fe323200640 1 -- 172.21.15.32:0/2199028548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3241a40f0 con 0x7fe324103990 2024-04-30T08:35:14.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.979+0000 7fe31aa00640 1 -- 172.21.15.32:0/2199028548 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe314035070 con 0x7fe324103990 2024-04-30T08:35:14.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.979+0000 7fe323200640 1 -- 172.21.15.32:0/2199028548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3241a45e0 con 0x7fe324103990 2024-04-30T08:35:14.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.981+0000 7fe323200640 1 -- 172.21.15.32:0/2199028548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe3241a4900 con 0x7fe324103990 2024-04-30T08:35:14.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.983+0000 7fe31aa00640 1 -- 172.21.15.32:0/2199028548 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7fe314042030 con 0x7fe324103990 2024-04-30T08:35:14.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.985+0000 7fe31aa00640 1 --2- 172.21.15.32:0/2199028548 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe3080afb70 0x7fe3080b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:14.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.985+0000 7fe31aa00640 1 -- 172.21.15.32:0/2199028548 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fe3140ff4b0 con 0x7fe324103990 2024-04-30T08:35:14.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.985+0000 7fe321400640 1 --2- 172.21.15.32:0/2199028548 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe3080afb70 0x7fe3080b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:14.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.986+0000 7fe321400640 1 --2- 172.21.15.32:0/2199028548 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe3080afb70 0x7fe3080b2080 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe32419d3c0 tx=0x7fe310005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:14.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:14.989+0000 7fe31aa00640 1 -- 172.21.15.32:0/2199028548 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe3140c74d0 con 0x7fe324103990 2024-04-30T08:35:15.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.200+0000 7fe323200640 1 -- 172.21.15.32:0/2199028548 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe3241a4ad0 con 0x7fe324103990 2024-04-30T08:35:15.204 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.203+0000 7fe31aa00640 1 -- 172.21.15.32:0/2199028548 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+24730 (secure 0 0 0) 0x7fe3140cb7d0 con 0x7fe324103990 2024-04-30T08:35:15.205 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:35:15.205 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":85,"fsid":"98c0a670-06ca-11ef-bc93-c7b262605968","created":"2024-04-30T08:22:11.122875+0000","modified":"2024-04-30T08:33:31.872704+0000","last_up_change":"2024-04-30T08:30:50.350439+0000","last_in_change":"2024-04-30T08:30:30.210620+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-30T08:26:19.728331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-30T08:31:49.800542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-30T08:31:51.211320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-30T08:31:55.716625+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a7e060c8-1cbb-418f-9a03-06693f6b3f52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6803","nonce":2624700402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6805","nonce":2624700402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6809","nonce":2624700402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6807","nonce":2624700402}]},"public_addr":"172.21.15.32:6803/2624700402","cluster_addr":"172.21.15.32:6805/2624700402","heartbeat_back_addr":"172.21.15.32:6809/2624700402","heartbeat_front_addr":"172.21.15.32:6807/2624700402","state":["exists","up"]},{"osd":1,"uuid":"72ef2fb1-e5ca-48ef-aa3a-4796a64b922f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6801","nonce":444649711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6803","nonce":444649711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6807","nonce":444649711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6805","nonce":444649711}]},"public_addr":"172.21.15.71:6801/444649711","cluster_addr":"172.21.15.71:6803/444649711","heartbeat_back_addr":"172.21.15.71:6807/444649711","heartbeat_front_addr":"172.21.15.71:6805/444649711","state":["exists","up"]},{"osd":2,"uuid":"704e5e8a-935e-47c7-ae28-27119876ac09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6801","nonce":3255528671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6803","nonce":3255528671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6807","nonce":3255528671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6805","nonce":3255528671}]},"public_addr":"172.21.15.189:6801/3255528671","cluster_addr":"172.21.15.189:6803/3255528671","heartbeat_back_addr":"172.21.15.189:6807/3255528671","heartbeat_front_addr":"172.21.15.189:6805/3255528671","state":["exists","up"]},{"osd":3,"uuid":"575004f7-6559-4a62-863e-e33078b0eebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6811","nonce":1384702112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6813","nonce":1384702112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6817","nonce":1384702112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6815","nonce":1384702112}]},"public_addr":"172.21.15.32:6811/1384702112","cluster_addr":"172.21.15.32:6813/1384702112","heartbeat_back_addr":"172.21.15.32:6817/1384702112","heartbeat_front_addr":"172.21.15.32:6815/1384702112","state":["exists","up"]},{"osd":4,"uuid":"e1c1e894-074d-4c78-89ef-9e5a114268bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6809","nonce":4250534574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6811","nonce":4250534574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6815","nonce":4250534574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6813","nonce":4250534574}]},"public_addr":"172.21.15.71:6809/4250534574","cluster_addr":"172.21.15.71:6811/4250534574","heartbeat_back_addr":"172.21.15.71:6815/4250534574","heartbeat_front_addr":"172.21.15.71:6813/4250534574","state":["exists","up"]},{"osd":5,"uuid":"12c022d4-fac6-40c4-ae7a-691571bee5d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6809","nonce":4245346198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6811","nonce":4245346198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6815","nonce":4245346198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6813","nonce":4245346198}]},"public_addr":"172.21.15.189:6809/4245346198","cluster_addr":"172.21.15.189:6811/4245346198","heartbeat_back_addr":"172.21.15.189:6815/4245346198","heartbeat_front_addr":"172.21.15.189:6813/4245346198","state":["exists","up"]},{"osd":6,"uuid":"b413f8ee-3d4a-415a-a2ba-d68437b7330d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6819","nonce":1910389190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6821","nonce":1910389190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6825","nonce":1910389190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6823","nonce":1910389190}]},"public_addr":"172.21.15.32:6819/1910389190","cluster_addr":"172.21.15.32:6821/1910389190","heartbeat_back_addr":"172.21.15.32:6825/1910389190","heartbeat_front_addr":"172.21.15.32:6823/1910389190","state":["exists","up"]},{"osd":7,"uuid":"aac64fb3-fc13-4126-b842-7b06ad5ce238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6817","nonce":692607380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6819","nonce":692607380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6823","nonce":692607380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6821","nonce":692607380}]},"public_addr":"172.21.15.71:6817/692607380","cluster_addr":"172.21.15.71:6819/692607380","heartbeat_back_addr":"172.21.15.71:6823/692607380","heartbeat_front_addr":"172.21.15.71:6821/692607380","state":["exists","up"]},{"osd":8,"uuid":"cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6817","nonce":3973697069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6819","nonce":3973697069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6823","nonce":3973697069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6821","nonce":3973697069}]},"public_addr":"172.21.15.189:6817/3973697069","cluster_addr":"172.21.15.189:6819/3973697069","heartbeat_back_addr":"172.21.15.189:6823/3973697069","heartbeat_front_addr":"172.21.15.189:6821/3973697069","state":["exists","up"]},{"osd":9,"uuid":"ef4a1645-ae6e-45c3-82ad-161510e62aa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6827","nonce":46066685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6829","nonce":46066685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6833","nonce":46066685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6831","nonce":46066685}]},"public_addr":"172.21.15.32:6827/46066685","cluster_addr":"172.21.15.32:6829/46066685","heartbeat_back_addr":"172.21.15.32:6833/46066685","heartbeat_front_addr":"172.21.15.32:6831/46066685","state":["exists","up"]},{"osd":10,"uuid":"25b26a38-a644-409f-b941-59efdf04a3ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6825","nonce":1886290293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6827","nonce":1886290293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6831","nonce":1886290293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6829","nonce":1886290293}]},"public_addr":"172.21.15.71:6825/1886290293","cluster_addr":"172.21.15.71:6827/1886290293","heartbeat_back_addr":"172.21.15.71:6831/1886290293","heartbeat_front_addr":"172.21.15.71:6829/1886290293","state":["exists","up"]},{"osd":11,"uuid":"ab96b3ff-3688-455b-af64-60226c5bfbff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6825","nonce":850185001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6827","nonce":850185001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6831","nonce":850185001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6829","nonce":850185001}]},"public_addr":"172.21.15.189:6825/850185001","cluster_addr":"172.21.15.189:6827/850185001","heartbeat_back_addr":"172.21.15.189:6831/850185001","heartbeat_front_addr":"172.21.15.189:6829/850185001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:27.220308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:53.230783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:17.190642+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:52.603799+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:20.568671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:46.158363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:22.706933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:50.665173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:16.602410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:53.814698+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:22.435189+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:48.651949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.e","mappings":[{"from":4,"to":7}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.28","mappings":[{"from":3,"to":6},{"from":5,"to":8}]},{"pgid":"3.30","mappings":[{"from":1,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1266897222":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/2915689533":"2024-05-01T08:23:29.526664+0000","172.21.15.32:6801/4019351306":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/425836193":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/1134075722":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6801/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/21059253":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/3245550894":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/4019351306":"2024-05-01T08:23:29.526664+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-30T08:35:15.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.206+0000 7fe323200640 1 -- 172.21.15.32:0/2199028548 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe3080afb70 msgr2=0x7fe3080b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:15.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.206+0000 7fe323200640 1 --2- 172.21.15.32:0/2199028548 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe3080afb70 0x7fe3080b2080 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe32419d3c0 tx=0x7fe310005f90 comp rx=0 tx=0).stop 2024-04-30T08:35:15.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.206+0000 7fe323200640 1 -- 172.21.15.32:0/2199028548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe324103990 msgr2=0x7fe32419c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:15.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.206+0000 7fe323200640 1 --2- 172.21.15.32:0/2199028548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe324103990 0x7fe32419c320 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe314006010 tx=0x7fe314004270 comp rx=0 tx=0).stop 2024-04-30T08:35:15.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.206+0000 7fe323200640 1 -- 172.21.15.32:0/2199028548 shutdown_connections 2024-04-30T08:35:15.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.206+0000 7fe323200640 1 --2- 172.21.15.32:0/2199028548 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe3080afb70 0x7fe3080b2080 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:15.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.206+0000 7fe323200640 1 --2- 172.21.15.32:0/2199028548 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe3241098d0 0x7fe3241a37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:15.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.206+0000 7fe323200640 1 --2- 172.21.15.32:0/2199028548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe324104bd0 0x7fe32419c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:15.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.206+0000 7fe323200640 1 --2- 172.21.15.32:0/2199028548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe324103990 0x7fe32419c320 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:15.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.206+0000 7fe323200640 1 -- 172.21.15.32:0/2199028548 >> 172.21.15.32:0/2199028548 conn(0x7fe3240feba0 msgr2=0x7fe3240fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:15.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.206+0000 7fe323200640 1 -- 172.21.15.32:0/2199028548 shutdown_connections 2024-04-30T08:35:15.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.206+0000 7fe323200640 1 -- 172.21.15.32:0/2199028548 wait complete. 2024-04-30T08:35:15.236 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-30T08:26:19.728331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-30T08:31:49.800542+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-30T08:31:51.211320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-30T08:31:55.716625+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-30T08:35:15.236 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-30T08:35:15.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:15 smithi032 ceph-mon[28331]: pgmap v427: 129 pgs: 129 active+clean; 651 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-04-30T08:35:15.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:15 smithi032 ceph-mon[28331]: from='client.25568 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:35:15.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:15 smithi071 ceph-mon[33385]: pgmap v427: 129 pgs: 129 active+clean; 651 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-04-30T08:35:15.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:15 smithi071 ceph-mon[33385]: from='client.25568 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:35:15.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.514+0000 7f638f200640 1 -- 172.21.15.32:0/3332870986 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f638810ff00 msgr2=0x7f6388112320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:15.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.514+0000 7f638f200640 1 --2- 172.21.15.32:0/3332870986 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f638810ff00 0x7f6388112320 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f6384009900 tx=0x7f638402a3f0 comp rx=0 tx=0).stop 2024-04-30T08:35:15.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.514+0000 7f638f200640 1 -- 172.21.15.32:0/3332870986 shutdown_connections 2024-04-30T08:35:15.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.514+0000 7f638f200640 1 --2- 172.21.15.32:0/3332870986 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f638810ff00 0x7f6388112320 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:15.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.514+0000 7f638f200640 1 --2- 172.21.15.32:0/3332870986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f638810b200 0x7f638810b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:15.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.514+0000 7f638f200640 1 --2- 172.21.15.32:0/3332870986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6388109fc0 0x7f638810acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:15.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.514+0000 7f638f200640 1 -- 172.21.15.32:0/3332870986 >> 172.21.15.32:0/3332870986 conn(0x7f6388078bb0 msgr2=0x7f6388107a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:15.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.515+0000 7f638f200640 1 -- 172.21.15.32:0/3332870986 shutdown_connections 2024-04-30T08:35:15.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.515+0000 7f638f200640 1 -- 172.21.15.32:0/3332870986 wait complete. 2024-04-30T08:35:15.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.516+0000 7f638f200640 1 Processor -- start 2024-04-30T08:35:15.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.516+0000 7f638f200640 1 -- start start 2024-04-30T08:35:15.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.516+0000 7f638f200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6388109fc0 0x7f63881a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:15.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.516+0000 7f638f200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f638810b200 0x7f63881a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:15.521 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.516+0000 7f638f200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f638810ff00 0x7f63881a7b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:15.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.516+0000 7f638f200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6388115b30 con 0x7f638810ff00 2024-04-30T08:35:15.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.516+0000 7f638f200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f63881159b0 con 0x7f6388109fc0 2024-04-30T08:35:15.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.516+0000 7f638f200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6388115cb0 con 0x7f638810b200 2024-04-30T08:35:15.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.517+0000 7f638de00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6388109fc0 0x7f63881a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:15.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.517+0000 7f638e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f638810ff00 0x7f63881a7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:15.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.517+0000 7f638de00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6388109fc0 0x7f63881a06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:57328/0 (socket says 172.21.15.32:57328) 2024-04-30T08:35:15.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.517+0000 7f638de00640 1 -- 172.21.15.32:0/3553988494 learned_addr learned my addr 172.21.15.32:0/3553988494 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:15.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.517+0000 7f638d400640 1 --2- 172.21.15.32:0/3553988494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f638810b200 0x7f63881a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:15.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.517+0000 7f638de00640 1 -- 172.21.15.32:0/3553988494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f638810b200 msgr2=0x7f63881a0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:15.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.517+0000 7f638de00640 1 --2- 172.21.15.32:0/3553988494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f638810b200 0x7f63881a0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:15.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.517+0000 7f638de00640 1 -- 172.21.15.32:0/3553988494 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f638810ff00 msgr2=0x7f63881a7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:15.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.517+0000 7f638de00640 1 --2- 172.21.15.32:0/3553988494 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f638810ff00 0x7f63881a7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:15.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.517+0000 7f638de00640 1 -- 172.21.15.32:0/3553988494 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63881a81a0 con 0x7f6388109fc0 2024-04-30T08:35:15.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.517+0000 7f638d400640 1 --2- 172.21.15.32:0/3553988494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f638810b200 0x7f63881a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:15.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.518+0000 7f638de00640 1 --2- 172.21.15.32:0/3553988494 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6388109fc0 0x7f63881a06b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f638000eb90 tx=0x7f638000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:15.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.518+0000 7f637ea00640 1 -- 172.21.15.32:0/3553988494 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6380003b70 con 0x7f6388109fc0 2024-04-30T08:35:15.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.518+0000 7f637ea00640 1 -- 172.21.15.32:0/3553988494 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f638000ee90 con 0x7f6388109fc0 2024-04-30T08:35:15.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.518+0000 7f637ea00640 1 -- 172.21.15.32:0/3553988494 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63800197b0 con 0x7f6388109fc0 2024-04-30T08:35:15.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.518+0000 7f638f200640 1 -- 172.21.15.32:0/3553988494 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63881a8490 con 0x7f6388109fc0 2024-04-30T08:35:15.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.518+0000 7f638f200640 1 -- 172.21.15.32:0/3553988494 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63881a8990 con 0x7f6388109fc0 2024-04-30T08:35:15.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.519+0000 7f638f200640 1 -- 172.21.15.32:0/3553988494 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f638810ed10 con 0x7f6388109fc0 2024-04-30T08:35:15.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.522+0000 7f637ea00640 1 -- 172.21.15.32:0/3553988494 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f6380004030 con 0x7f6388109fc0 2024-04-30T08:35:15.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.523+0000 7f637ea00640 1 --2- 172.21.15.32:0/3553988494 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f63740afbf0 0x7f63740b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:15.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.523+0000 7f637ea00640 1 -- 172.21.15.32:0/3553988494 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f63800dfa90 con 0x7f6388109fc0 2024-04-30T08:35:15.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.524+0000 7f638d400640 1 --2- 172.21.15.32:0/3553988494 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f63740afbf0 0x7f63740b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:15.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.524+0000 7f638d400640 1 --2- 172.21.15.32:0/3553988494 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f63740afbf0 0x7f63740b2100 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f63881a1700 tx=0x7f6378005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:15.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.525+0000 7f637ea00640 1 -- 172.21.15.32:0/3553988494 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f63800a8190 con 0x7f6388109fc0 2024-04-30T08:35:15.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:15 smithi189 ceph-mon[33038]: pgmap v427: 129 pgs: 129 active+clean; 651 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.4 KiB/s wr, 11 op/s 2024-04-30T08:35:15.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:15 smithi189 ceph-mon[33038]: from='client.25568 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:35:15.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.733+0000 7f638f200640 1 -- 172.21.15.32:0/3553988494 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6388063850 con 0x7f6388109fc0 2024-04-30T08:35:15.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.734+0000 7f637ea00640 1 -- 172.21.15.32:0/3553988494 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) ==== 93+0+10 (secure 0 0 0) 0x7f63800b0180 con 0x7f6388109fc0 2024-04-30T08:35:15.735 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-30T08:35:15.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.736+0000 7f638f200640 1 -- 172.21.15.32:0/3553988494 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f63740afbf0 msgr2=0x7f63740b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:15.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.736+0000 7f638f200640 1 --2- 172.21.15.32:0/3553988494 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f63740afbf0 0x7f63740b2100 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f63881a1700 tx=0x7f6378005f90 comp rx=0 tx=0).stop 2024-04-30T08:35:15.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.737+0000 7f638f200640 1 -- 172.21.15.32:0/3553988494 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6388109fc0 msgr2=0x7f63881a06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:15.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.737+0000 7f638f200640 1 --2- 172.21.15.32:0/3553988494 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6388109fc0 0x7f63881a06b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f638000eb90 tx=0x7f638000c6a0 comp rx=0 tx=0).stop 2024-04-30T08:35:15.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.737+0000 7f638f200640 1 -- 172.21.15.32:0/3553988494 shutdown_connections 2024-04-30T08:35:15.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.737+0000 7f638f200640 1 --2- 172.21.15.32:0/3553988494 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f63740afbf0 0x7f63740b2100 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:15.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.737+0000 7f638f200640 1 --2- 172.21.15.32:0/3553988494 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f638810ff00 0x7f63881a7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:15.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.737+0000 7f638f200640 1 --2- 172.21.15.32:0/3553988494 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f638810b200 0x7f63881a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:15.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.737+0000 7f638f200640 1 --2- 172.21.15.32:0/3553988494 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6388109fc0 0x7f63881a06b0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:15.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.737+0000 7f638f200640 1 -- 172.21.15.32:0/3553988494 >> 172.21.15.32:0/3553988494 conn(0x7f6388078bb0 msgr2=0x7f638810bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:15.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.737+0000 7f638f200640 1 -- 172.21.15.32:0/3553988494 shutdown_connections 2024-04-30T08:35:15.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:15.737+0000 7f638f200640 1 -- 172.21.15.32:0/3553988494 wait complete. 2024-04-30T08:35:15.768 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-30T08:35:16.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.054+0000 7f95a2400640 1 -- 172.21.15.32:0/4251292227 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f959c107200 msgr2=0x7f959c105380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:16.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.054+0000 7f95a2400640 1 --2- 172.21.15.32:0/4251292227 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f959c107200 0x7f959c105380 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f958c009900 tx=0x7f958c02a3f0 comp rx=0 tx=0).stop 2024-04-30T08:35:16.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.054+0000 7f95a2400640 1 -- 172.21.15.32:0/4251292227 shutdown_connections 2024-04-30T08:35:16.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.054+0000 7f95a2400640 1 --2- 172.21.15.32:0/4251292227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f959c10bb00 0x7f959c10df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:16.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.054+0000 7f95a2400640 1 --2- 172.21.15.32:0/4251292227 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f959c1058c0 0x7f959c105cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:16.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.054+0000 7f95a2400640 1 --2- 172.21.15.32:0/4251292227 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f959c107200 0x7f959c105380 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:16.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.054+0000 7f95a2400640 1 -- 172.21.15.32:0/4251292227 >> 172.21.15.32:0/4251292227 conn(0x7f959c100df0 msgr2=0x7f959c103260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:16.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.054+0000 7f95a2400640 1 -- 172.21.15.32:0/4251292227 shutdown_connections 2024-04-30T08:35:16.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.054+0000 7f95a2400640 1 -- 172.21.15.32:0/4251292227 wait complete. 2024-04-30T08:35:16.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.055+0000 7f95a2400640 1 Processor -- start 2024-04-30T08:35:16.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.055+0000 7f95a2400640 1 -- start start 2024-04-30T08:35:16.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.056+0000 7f95a2400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f959c1058c0 0x7f959c19c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:16.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.056+0000 7f95a2400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f959c107200 0x7f959c19c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:16.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.056+0000 7f95a2400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f959c10bb00 0x7f959c1a38a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:16.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.056+0000 7f95a2400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f959c1117d0 con 0x7f959c107200 2024-04-30T08:35:16.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.056+0000 7f95a2400640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f959c111650 con 0x7f959c10bb00 2024-04-30T08:35:16.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.056+0000 7f95a2400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f959c111950 con 0x7f959c1058c0 2024-04-30T08:35:16.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.056+0000 7f959be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f959c107200 0x7f959c19c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:16.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.056+0000 7f959be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f959c107200 0x7f959c19c910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53720/0 (socket says 172.21.15.32:53720) 2024-04-30T08:35:16.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.056+0000 7f959be00640 1 -- 172.21.15.32:0/3422005547 learned_addr learned my addr 172.21.15.32:0/3422005547 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:16.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.056+0000 7f95a1a00640 1 --2- 172.21.15.32:0/3422005547 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f959c10bb00 0x7f959c1a38a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:16.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.056+0000 7f95a1000640 1 --2- 172.21.15.32:0/3422005547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f959c1058c0 0x7f959c19c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:16.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.056+0000 7f95a1a00640 1 -- 172.21.15.32:0/3422005547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f959c1058c0 msgr2=0x7f959c19c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:16.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.056+0000 7f95a1a00640 1 --2- 172.21.15.32:0/3422005547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f959c1058c0 0x7f959c19c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:16.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.056+0000 7f95a1a00640 1 -- 172.21.15.32:0/3422005547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f959c107200 msgr2=0x7f959c19c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:16.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.056+0000 7f95a1a00640 1 --2- 172.21.15.32:0/3422005547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f959c107200 0x7f959c19c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:16.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.056+0000 7f95a1a00640 1 -- 172.21.15.32:0/3422005547 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f959c1a3f10 con 0x7f959c10bb00 2024-04-30T08:35:16.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.057+0000 7f95a1a00640 1 --2- 172.21.15.32:0/3422005547 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f959c10bb00 0x7f959c1a38a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9594009fd0 tx=0x7f959400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:16.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.057+0000 7f9599600640 1 -- 172.21.15.32:0/3422005547 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f959400ce20 con 0x7f959c10bb00 2024-04-30T08:35:16.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.057+0000 7f9599600640 1 -- 172.21.15.32:0/3422005547 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9594005020 con 0x7f959c10bb00 2024-04-30T08:35:16.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.057+0000 7f95a2400640 1 -- 172.21.15.32:0/3422005547 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f959c1a4200 con 0x7f959c10bb00 2024-04-30T08:35:16.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.057+0000 7f9599600640 1 -- 172.21.15.32:0/3422005547 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9594010690 con 0x7f959c10bb00 2024-04-30T08:35:16.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.057+0000 7f95a2400640 1 -- 172.21.15.32:0/3422005547 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f959c1a4750 con 0x7f959c10bb00 2024-04-30T08:35:16.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.058+0000 7f95a2400640 1 -- 172.21.15.32:0/3422005547 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f959c069230 con 0x7f959c10bb00 2024-04-30T08:35:16.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.061+0000 7f9599600640 1 -- 172.21.15.32:0/3422005547 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f95940054d0 con 0x7f959c10bb00 2024-04-30T08:35:16.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.062+0000 7f9599600640 1 --2- 172.21.15.32:0/3422005547 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f95800afb70 0x7f95800b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:16.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.063+0000 7f9599600640 1 -- 172.21.15.32:0/3422005547 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f95940df830 con 0x7f959c10bb00 2024-04-30T08:35:16.064 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.063+0000 7f95a1000640 1 --2- 172.21.15.32:0/3422005547 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f95800afb70 0x7f95800b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:16.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.064+0000 7f95a1000640 1 --2- 172.21.15.32:0/3422005547 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f95800afb70 0x7f95800b2080 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f958c005ab0 tx=0x7f958c031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:16.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.064+0000 7f9599600640 1 -- 172.21.15.32:0/3422005547 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f95940a7ea0 con 0x7f959c10bb00 2024-04-30T08:35:16.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.271+0000 7f95a2400640 1 -- 172.21.15.32:0/3422005547 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f959c063820 con 0x7f959c10bb00 2024-04-30T08:35:16.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.272+0000 7f9599600640 1 -- 172.21.15.32:0/3422005547 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) ==== 104+0+11 (secure 0 0 0) 0x7f95940a77a0 con 0x7f959c10bb00 2024-04-30T08:35:16.273 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-30T08:35:16.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.275+0000 7f95a2400640 1 -- 172.21.15.32:0/3422005547 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f95800afb70 msgr2=0x7f95800b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:16.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.275+0000 7f95a2400640 1 --2- 172.21.15.32:0/3422005547 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f95800afb70 0x7f95800b2080 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f958c005ab0 tx=0x7f958c031400 comp rx=0 tx=0).stop 2024-04-30T08:35:16.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.275+0000 7f95a2400640 1 -- 172.21.15.32:0/3422005547 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f959c10bb00 msgr2=0x7f959c1a38a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:16.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.275+0000 7f95a2400640 1 --2- 172.21.15.32:0/3422005547 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f959c10bb00 0x7f959c1a38a0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f9594009fd0 tx=0x7f959400ef10 comp rx=0 tx=0).stop 2024-04-30T08:35:16.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.275+0000 7f95a2400640 1 -- 172.21.15.32:0/3422005547 shutdown_connections 2024-04-30T08:35:16.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.275+0000 7f95a2400640 1 --2- 172.21.15.32:0/3422005547 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f95800afb70 0x7f95800b2080 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:16.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.275+0000 7f95a2400640 1 --2- 172.21.15.32:0/3422005547 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f959c10bb00 0x7f959c1a38a0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:16.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.275+0000 7f95a2400640 1 --2- 172.21.15.32:0/3422005547 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f959c107200 0x7f959c19c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:16.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.275+0000 7f95a2400640 1 --2- 172.21.15.32:0/3422005547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f959c1058c0 0x7f959c19c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:16.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.275+0000 7f95a2400640 1 -- 172.21.15.32:0/3422005547 >> 172.21.15.32:0/3422005547 conn(0x7f959c100df0 msgr2=0x7f959c107ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:16.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.275+0000 7f95a2400640 1 -- 172.21.15.32:0/3422005547 shutdown_connections 2024-04-30T08:35:16.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.275+0000 7f95a2400640 1 -- 172.21.15.32:0/3422005547 wait complete. 2024-04-30T08:35:16.307 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-30T08:35:16.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:16 smithi032 ceph-mon[28331]: from='client.15450 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:35:16.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:16 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2199028548' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:35:16.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:16 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3553988494' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:35:16.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:16 smithi071 ceph-mon[33385]: from='client.15450 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:35:16.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:16 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2199028548' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:35:16.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:16 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3553988494' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:35:16.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:16 smithi189 ceph-mon[33038]: from='client.15450 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-30T08:35:16.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:16 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2199028548' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:35:16.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:16 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3553988494' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:35:16.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.597+0000 7faa17600640 1 -- 172.21.15.32:0/991512592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa100769f0 msgr2=0x7faa1010b640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:16.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.597+0000 7faa17600640 1 --2- 172.21.15.32:0/991512592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa100769f0 0x7faa1010b640 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc009900 tx=0x7fa9fc02a400 comp rx=0 tx=0).stop 2024-04-30T08:35:16.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.597+0000 7faa17600640 1 -- 172.21.15.32:0/991512592 shutdown_connections 2024-04-30T08:35:16.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.597+0000 7faa17600640 1 --2- 172.21.15.32:0/991512592 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa1010bb80 0x7faa1010dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:16.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.597+0000 7faa17600640 1 --2- 172.21.15.32:0/991512592 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa100769f0 0x7faa1010b640 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:16.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.597+0000 7faa17600640 1 --2- 172.21.15.32:0/991512592 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa100760a0 0x7faa100764b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:16.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.597+0000 7faa17600640 1 -- 172.21.15.32:0/991512592 >> 172.21.15.32:0/991512592 conn(0x7faa100fedc0 msgr2=0x7faa10101230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:16.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.597+0000 7faa17600640 1 -- 172.21.15.32:0/991512592 shutdown_connections 2024-04-30T08:35:16.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.598+0000 7faa17600640 1 -- 172.21.15.32:0/991512592 wait complete. 2024-04-30T08:35:16.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.599+0000 7faa17600640 1 Processor -- start 2024-04-30T08:35:16.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.599+0000 7faa17600640 1 -- start start 2024-04-30T08:35:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.599+0000 7faa17600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa100760a0 0x7faa1019c520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.599+0000 7faa17600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa100769f0 0x7faa1019ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.599+0000 7faa17600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa1010bb80 0x7faa101a39f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.599+0000 7faa17600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7faa101118e0 con 0x7faa100760a0 2024-04-30T08:35:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.599+0000 7faa17600640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7faa10111760 con 0x7faa100769f0 2024-04-30T08:35:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.599+0000 7faa17600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7faa10111a60 con 0x7faa1010bb80 2024-04-30T08:35:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.599+0000 7faa14c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa100760a0 0x7faa1019c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.599+0000 7faa14c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa100760a0 0x7faa1019c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53730/0 (socket says 172.21.15.32:53730) 2024-04-30T08:35:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.599+0000 7faa14c00640 1 -- 172.21.15.32:0/2061837740 learned_addr learned my addr 172.21.15.32:0/2061837740 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.600+0000 7faa0fe00640 1 --2- 172.21.15.32:0/2061837740 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa100769f0 0x7faa1019ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.600+0000 7faa15600640 1 --2- 172.21.15.32:0/2061837740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa1010bb80 0x7faa101a39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:16.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.600+0000 7faa14c00640 1 -- 172.21.15.32:0/2061837740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa1010bb80 msgr2=0x7faa101a39f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:16.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.600+0000 7faa14c00640 1 --2- 172.21.15.32:0/2061837740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa1010bb80 0x7faa101a39f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:16.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.600+0000 7faa14c00640 1 -- 172.21.15.32:0/2061837740 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa100769f0 msgr2=0x7faa1019ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:16.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.600+0000 7faa14c00640 1 --2- 172.21.15.32:0/2061837740 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa100769f0 0x7faa1019ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:16.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.600+0000 7faa14c00640 1 -- 172.21.15.32:0/2061837740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa101a4060 con 0x7faa100760a0 2024-04-30T08:35:16.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.600+0000 7faa14c00640 1 --2- 172.21.15.32:0/2061837740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa100760a0 0x7faa1019c520 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7faa0400e9a0 tx=0x7faa0400eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:16.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.601+0000 7faa0d600640 1 -- 172.21.15.32:0/2061837740 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa0400cd80 con 0x7faa100760a0 2024-04-30T08:35:16.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.601+0000 7faa0d600640 1 -- 172.21.15.32:0/2061837740 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa040035f0 con 0x7faa100760a0 2024-04-30T08:35:16.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.601+0000 7faa17600640 1 -- 172.21.15.32:0/2061837740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa101a4350 con 0x7faa100760a0 2024-04-30T08:35:16.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.601+0000 7faa0d600640 1 -- 172.21.15.32:0/2061837740 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa04010690 con 0x7faa100760a0 2024-04-30T08:35:16.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.601+0000 7faa17600640 1 -- 172.21.15.32:0/2061837740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa101a4870 con 0x7faa100760a0 2024-04-30T08:35:16.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.602+0000 7faa17600640 1 -- 172.21.15.32:0/2061837740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa1010ad40 con 0x7faa100760a0 2024-04-30T08:35:16.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.605+0000 7faa0d600640 1 -- 172.21.15.32:0/2061837740 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7faa04003aa0 con 0x7faa100760a0 2024-04-30T08:35:16.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.606+0000 7faa0d600640 1 --2- 172.21.15.32:0/2061837740 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa9f80afb80 0x7fa9f80b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:16.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.606+0000 7faa0d600640 1 -- 172.21.15.32:0/2061837740 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7faa040df5f0 con 0x7faa100760a0 2024-04-30T08:35:16.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.606+0000 7faa0fe00640 1 --2- 172.21.15.32:0/2061837740 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa9f80afb80 0x7fa9f80b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:16.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.607+0000 7faa0fe00640 1 --2- 172.21.15.32:0/2061837740 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa9f80afb80 0x7fa9f80b2090 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc004770 tx=0x7fa9fc0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:16.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.609+0000 7faa0d600640 1 -- 172.21.15.32:0/2061837740 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faa040a7c60 con 0x7faa100760a0 2024-04-30T08:35:16.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.819+0000 7faa17600640 1 -- 172.21.15.32:0/2061837740 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7faa1002dd60 con 0x7faa100760a0 2024-04-30T08:35:16.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.820+0000 7faa0d600640 1 -- 172.21.15.32:0/2061837740 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) ==== 100+0+11 (secure 0 0 0) 0x7faa040a7560 con 0x7faa100760a0 2024-04-30T08:35:16.822 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-04-30T08:35:16.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.823+0000 7faa17600640 1 -- 172.21.15.32:0/2061837740 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa9f80afb80 msgr2=0x7fa9f80b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:16.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.823+0000 7faa17600640 1 --2- 172.21.15.32:0/2061837740 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa9f80afb80 0x7fa9f80b2090 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fa9fc004770 tx=0x7fa9fc0314a0 comp rx=0 tx=0).stop 2024-04-30T08:35:16.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.823+0000 7faa17600640 1 -- 172.21.15.32:0/2061837740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa100760a0 msgr2=0x7faa1019c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:16.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.823+0000 7faa17600640 1 --2- 172.21.15.32:0/2061837740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa100760a0 0x7faa1019c520 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7faa0400e9a0 tx=0x7faa0400eda0 comp rx=0 tx=0).stop 2024-04-30T08:35:16.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.823+0000 7faa17600640 1 -- 172.21.15.32:0/2061837740 shutdown_connections 2024-04-30T08:35:16.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.823+0000 7faa17600640 1 --2- 172.21.15.32:0/2061837740 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa9f80afb80 0x7fa9f80b2090 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:16.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.823+0000 7faa17600640 1 --2- 172.21.15.32:0/2061837740 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa1010bb80 0x7faa101a39f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:16.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.823+0000 7faa17600640 1 --2- 172.21.15.32:0/2061837740 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa100769f0 0x7faa1019ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:16.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.823+0000 7faa17600640 1 --2- 172.21.15.32:0/2061837740 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa100760a0 0x7faa1019c520 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:16.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.823+0000 7faa17600640 1 -- 172.21.15.32:0/2061837740 >> 172.21.15.32:0/2061837740 conn(0x7faa100fedc0 msgr2=0x7faa100ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:16.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.823+0000 7faa17600640 1 -- 172.21.15.32:0/2061837740 shutdown_connections 2024-04-30T08:35:16.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:16.823+0000 7faa17600640 1 -- 172.21.15.32:0/2061837740 wait complete. 2024-04-30T08:35:16.856 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-30T08:35:17.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.135+0000 7f63a8e00640 1 -- 172.21.15.32:0/3876735622 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f63a4103b90 msgr2=0x7f63a4104890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:17.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.135+0000 7f63a8e00640 1 --2- 172.21.15.32:0/3876735622 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f63a4103b90 0x7f63a4104890 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f6394009900 tx=0x7f639402a410 comp rx=0 tx=0).stop 2024-04-30T08:35:17.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.135+0000 7f63a8e00640 1 -- 172.21.15.32:0/3876735622 shutdown_connections 2024-04-30T08:35:17.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.135+0000 7f63a8e00640 1 --2- 172.21.15.32:0/3876735622 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63a4109ad0 0x7f63a410bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:17.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.135+0000 7f63a8e00640 1 --2- 172.21.15.32:0/3876735622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63a4104dd0 0x7f63a41051e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:17.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.135+0000 7f63a8e00640 1 --2- 172.21.15.32:0/3876735622 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f63a4103b90 0x7f63a4104890 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:17.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.135+0000 7f63a8e00640 1 -- 172.21.15.32:0/3876735622 >> 172.21.15.32:0/3876735622 conn(0x7f63a40fedc0 msgr2=0x7f63a4101230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:17.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.136+0000 7f63a8e00640 1 -- 172.21.15.32:0/3876735622 shutdown_connections 2024-04-30T08:35:17.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.136+0000 7f63a8e00640 1 -- 172.21.15.32:0/3876735622 wait complete. 2024-04-30T08:35:17.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.137+0000 7f63a8e00640 1 Processor -- start 2024-04-30T08:35:17.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.137+0000 7f63a8e00640 1 -- start start 2024-04-30T08:35:17.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.137+0000 7f63a8e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63a4103b90 0x7f63a419c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:17.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.137+0000 7f63a8e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63a4104dd0 0x7f63a419cad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:17.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.137+0000 7f63a8e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f63a4109ad0 0x7f63a41a3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:17.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.137+0000 7f63a8e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f63a410f6e0 con 0x7f63a4103b90 2024-04-30T08:35:17.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.138+0000 7f63a8e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f63a410f560 con 0x7f63a4109ad0 2024-04-30T08:35:17.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.138+0000 7f63a8e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f63a410f860 con 0x7f63a4104dd0 2024-04-30T08:35:17.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.138+0000 7f63a1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63a4103b90 0x7f63a419c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:17.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.138+0000 7f63a2800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f63a4109ad0 0x7f63a41a3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:17.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.138+0000 7f63a1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63a4103b90 0x7f63a419c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39466/0 (socket says 172.21.15.32:39466) 2024-04-30T08:35:17.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.138+0000 7f63a1e00640 1 -- 172.21.15.32:0/1511509074 learned_addr learned my addr 172.21.15.32:0/1511509074 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:17.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.138+0000 7f63a1400640 1 --2- 172.21.15.32:0/1511509074 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63a4104dd0 0x7f63a419cad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:17.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.138+0000 7f63a2800640 1 -- 172.21.15.32:0/1511509074 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63a4104dd0 msgr2=0x7f63a419cad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:17.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.138+0000 7f63a2800640 1 --2- 172.21.15.32:0/1511509074 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63a4104dd0 0x7f63a419cad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:17.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.138+0000 7f63a2800640 1 -- 172.21.15.32:0/1511509074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63a4103b90 msgr2=0x7f63a419c590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:17.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.138+0000 7f63a2800640 1 --2- 172.21.15.32:0/1511509074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63a4103b90 0x7f63a419c590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:17.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.138+0000 7f63a2800640 1 -- 172.21.15.32:0/1511509074 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63a41a40d0 con 0x7f63a4109ad0 2024-04-30T08:35:17.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.138+0000 7f63a1e00640 1 --2- 172.21.15.32:0/1511509074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63a4103b90 0x7f63a419c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:17.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.139+0000 7f63a2800640 1 --2- 172.21.15.32:0/1511509074 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f63a4109ad0 0x7f63a41a3a60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f639c009fd0 tx=0x7f639c00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:17.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.139+0000 7f639aa00640 1 -- 172.21.15.32:0/1511509074 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f639c00ce20 con 0x7f63a4109ad0 2024-04-30T08:35:17.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.139+0000 7f639aa00640 1 -- 172.21.15.32:0/1511509074 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f639c005020 con 0x7f63a4109ad0 2024-04-30T08:35:17.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.139+0000 7f63a8e00640 1 -- 172.21.15.32:0/1511509074 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63a41a43c0 con 0x7f63a4109ad0 2024-04-30T08:35:17.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.139+0000 7f639aa00640 1 -- 172.21.15.32:0/1511509074 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f639c010690 con 0x7f63a4109ad0 2024-04-30T08:35:17.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.139+0000 7f63a8e00640 1 -- 172.21.15.32:0/1511509074 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63a41a48c0 con 0x7f63a4109ad0 2024-04-30T08:35:17.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.140+0000 7f63a8e00640 1 -- 172.21.15.32:0/1511509074 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63a41a4b80 con 0x7f63a4109ad0 2024-04-30T08:35:17.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.143+0000 7f639aa00640 1 -- 172.21.15.32:0/1511509074 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f639c0054d0 con 0x7f63a4109ad0 2024-04-30T08:35:17.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.145+0000 7f639aa00640 1 --2- 172.21.15.32:0/1511509074 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f63880afb80 0x7f63880b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:17.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.145+0000 7f639aa00640 1 -- 172.21.15.32:0/1511509074 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f639c0df890 con 0x7f63a4109ad0 2024-04-30T08:35:17.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.145+0000 7f63a1e00640 1 --2- 172.21.15.32:0/1511509074 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f63880afb80 0x7f63880b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:17.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.146+0000 7f63a1e00640 1 --2- 172.21.15.32:0/1511509074 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f63880afb80 0x7f63880b2090 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6394000c00 tx=0x7f63940314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:17.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.146+0000 7f639aa00640 1 -- 172.21.15.32:0/1511509074 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f639c0a7f00 con 0x7f63a4109ad0 2024-04-30T08:35:17.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.356+0000 7f63a8e00640 1 -- 172.21.15.32:0/1511509074 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f63a41a4d50 con 0x7f63a4109ad0 2024-04-30T08:35:17.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.357+0000 7f639aa00640 1 -- 172.21.15.32:0/1511509074 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v85) ==== 103+0+11 (secure 0 0 0) 0x7f639c0a7800 con 0x7f63a4109ad0 2024-04-30T08:35:17.359 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-30T08:35:17.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.362+0000 7f63a8e00640 1 -- 172.21.15.32:0/1511509074 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f63880afb80 msgr2=0x7f63880b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:17.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.362+0000 7f63a8e00640 1 --2- 172.21.15.32:0/1511509074 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f63880afb80 0x7f63880b2090 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f6394000c00 tx=0x7f63940314a0 comp rx=0 tx=0).stop 2024-04-30T08:35:17.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.362+0000 7f63a8e00640 1 -- 172.21.15.32:0/1511509074 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f63a4109ad0 msgr2=0x7f63a41a3a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:17.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.362+0000 7f63a8e00640 1 --2- 172.21.15.32:0/1511509074 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f63a4109ad0 0x7f63a41a3a60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f639c009fd0 tx=0x7f639c00ef10 comp rx=0 tx=0).stop 2024-04-30T08:35:17.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.362+0000 7f63a8e00640 1 -- 172.21.15.32:0/1511509074 shutdown_connections 2024-04-30T08:35:17.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.362+0000 7f63a8e00640 1 --2- 172.21.15.32:0/1511509074 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f63880afb80 0x7f63880b2090 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:17.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.362+0000 7f63a8e00640 1 --2- 172.21.15.32:0/1511509074 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f63a4109ad0 0x7f63a41a3a60 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:17.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.362+0000 7f63a8e00640 1 --2- 172.21.15.32:0/1511509074 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f63a4104dd0 0x7f63a419cad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:17.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.362+0000 7f63a8e00640 1 --2- 172.21.15.32:0/1511509074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f63a4103b90 0x7f63a419c590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:17.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.362+0000 7f63a8e00640 1 -- 172.21.15.32:0/1511509074 >> 172.21.15.32:0/1511509074 conn(0x7f63a40fedc0 msgr2=0x7f63a40ff3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:17.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.362+0000 7f63a8e00640 1 -- 172.21.15.32:0/1511509074 shutdown_connections 2024-04-30T08:35:17.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.362+0000 7f63a8e00640 1 -- 172.21.15.32:0/1511509074 wait complete. 2024-04-30T08:35:17.391 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-30T08:35:17.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:17 smithi032 ceph-mon[28331]: pgmap v428: 129 pgs: 129 active+clean; 653 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-04-30T08:35:17.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:17 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3422005547' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-30T08:35:17.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:17 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2061837740' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-30T08:35:17.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:17 smithi071 ceph-mon[33385]: pgmap v428: 129 pgs: 129 active+clean; 653 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-04-30T08:35:17.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:17 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3422005547' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-30T08:35:17.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:17 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2061837740' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-30T08:35:17.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:17 smithi189 ceph-mon[33038]: pgmap v428: 129 pgs: 129 active+clean; 653 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-04-30T08:35:17.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:17 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3422005547' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-30T08:35:17.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:17 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2061837740' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-30T08:35:17.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.686+0000 7fef49200640 1 -- 172.21.15.32:0/2111667941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef44106df0 msgr2=0x7fef44107200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:17.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.686+0000 7fef49200640 1 --2- 172.21.15.32:0/2111667941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef44106df0 0x7fef44107200 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fef2c009900 tx=0x7fef2c02a3f0 comp rx=0 tx=0).stop 2024-04-30T08:35:17.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.686+0000 7fef49200640 1 -- 172.21.15.32:0/2111667941 shutdown_connections 2024-04-30T08:35:17.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.686+0000 7fef49200640 1 --2- 172.21.15.32:0/2111667941 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fef4410baf0 0x7fef4410df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:17.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.686+0000 7fef49200640 1 --2- 172.21.15.32:0/2111667941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef44106df0 0x7fef44107200 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:17.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.686+0000 7fef49200640 1 --2- 172.21.15.32:0/2111667941 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef44105bb0 0x7fef441068b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:17.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.687+0000 7fef49200640 1 -- 172.21.15.32:0/2111667941 >> 172.21.15.32:0/2111667941 conn(0x7fef44100da0 msgr2=0x7fef44103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:17.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.687+0000 7fef49200640 1 -- 172.21.15.32:0/2111667941 shutdown_connections 2024-04-30T08:35:17.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.687+0000 7fef49200640 1 -- 172.21.15.32:0/2111667941 wait complete. 2024-04-30T08:35:17.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.688+0000 7fef49200640 1 Processor -- start 2024-04-30T08:35:17.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.688+0000 7fef49200640 1 -- start start 2024-04-30T08:35:17.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.689+0000 7fef49200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef44105bb0 0x7fef4419c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:17.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.689+0000 7fef49200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef44106df0 0x7fef4419c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:17.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.689+0000 7fef49200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fef4410baf0 0x7fef441a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:17.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.689+0000 7fef49200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fef441116d0 con 0x7fef44106df0 2024-04-30T08:35:17.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.689+0000 7fef49200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fef44111550 con 0x7fef4410baf0 2024-04-30T08:35:17.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.689+0000 7fef49200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fef44111850 con 0x7fef44105bb0 2024-04-30T08:35:17.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.689+0000 7fef3aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef44106df0 0x7fef4419c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:17.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.689+0000 7fef43400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef44105bb0 0x7fef4419c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:17.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.689+0000 7fef3aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef44106df0 0x7fef4419c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39484/0 (socket says 172.21.15.32:39484) 2024-04-30T08:35:17.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.689+0000 7fef3aa00640 1 -- 172.21.15.32:0/1275922222 learned_addr learned my addr 172.21.15.32:0/1275922222 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:17.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.689+0000 7fef43e00640 1 --2- 172.21.15.32:0/1275922222 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fef4410baf0 0x7fef441a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:17.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.689+0000 7fef43400640 1 -- 172.21.15.32:0/1275922222 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fef4410baf0 msgr2=0x7fef441a3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:17.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.689+0000 7fef43400640 1 --2- 172.21.15.32:0/1275922222 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fef4410baf0 0x7fef441a3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:17.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.689+0000 7fef43400640 1 -- 172.21.15.32:0/1275922222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef44106df0 msgr2=0x7fef4419c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:17.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.689+0000 7fef43400640 1 --2- 172.21.15.32:0/1275922222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef44106df0 0x7fef4419c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:17.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.689+0000 7fef43400640 1 -- 172.21.15.32:0/1275922222 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef441a3dc0 con 0x7fef44105bb0 2024-04-30T08:35:17.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.689+0000 7fef43400640 1 --2- 172.21.15.32:0/1275922222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef44105bb0 0x7fef4419c280 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fef44105480 tx=0x7fef3400ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:17.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.689+0000 7fef40c00640 1 -- 172.21.15.32:0/1275922222 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef3400cc40 con 0x7fef44105bb0 2024-04-30T08:35:17.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.689+0000 7fef40c00640 1 -- 172.21.15.32:0/1275922222 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fef3400cde0 con 0x7fef44105bb0 2024-04-30T08:35:17.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.689+0000 7fef49200640 1 -- 172.21.15.32:0/1275922222 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef441a40b0 con 0x7fef44105bb0 2024-04-30T08:35:17.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.691+0000 7fef40c00640 1 -- 172.21.15.32:0/1275922222 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef34010430 con 0x7fef44105bb0 2024-04-30T08:35:17.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.691+0000 7fef49200640 1 -- 172.21.15.32:0/1275922222 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef441a45b0 con 0x7fef44105bb0 2024-04-30T08:35:17.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.692+0000 7fef49200640 1 -- 172.21.15.32:0/1275922222 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef4410a900 con 0x7fef44105bb0 2024-04-30T08:35:17.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.693+0000 7fef40c00640 1 -- 172.21.15.32:0/1275922222 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7fef34003680 con 0x7fef44105bb0 2024-04-30T08:35:17.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.696+0000 7fef40c00640 1 --2- 172.21.15.32:0/1275922222 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fef280afb70 0x7fef280b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:17.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.696+0000 7fef40c00640 1 -- 172.21.15.32:0/1275922222 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fef340df640 con 0x7fef44105bb0 2024-04-30T08:35:17.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.696+0000 7fef3aa00640 1 --2- 172.21.15.32:0/1275922222 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fef280afb70 0x7fef280b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:17.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.697+0000 7fef3aa00640 1 --2- 172.21.15.32:0/1275922222 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fef280afb70 0x7fef280b2080 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fef2c031440 tx=0x7fef2c0314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:17.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.698+0000 7fef40c00640 1 -- 172.21.15.32:0/1275922222 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fef340a7cb0 con 0x7fef44105bb0 2024-04-30T08:35:17.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.948+0000 7fef49200640 1 -- 172.21.15.32:0/1275922222 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fef4402dbd0 con 0x7fef44105bb0 2024-04-30T08:35:17.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.949+0000 7fef40c00640 1 -- 172.21.15.32:0/1275922222 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11927 (secure 0 0 0) 0x7fef340a75b0 con 0x7fef44105bb0 2024-04-30T08:35:17.950 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:35:17.950 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2235842676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2235842676},{"type":"v1","addr":"172.21.15.32:6835","nonce":2235842676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/3577009355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":3577009355},{"type":"v1","addr":"172.21.15.32:6841","nonce":3577009355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-30T08:31:52.236668+0000","modified":"2024-04-30T08:35:05.098418+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24541,"mds_3":24533,"mds_4":24517},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/463203038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":463203038},{"type":"v1","addr":"172.21.15.189:6833","nonce":463203038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6833/3635143591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3635143591},{"type":"v1","addr":"172.21.15.71:6833","nonce":3635143591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6837/2569449994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2569449994},{"type":"v1","addr":"172.21.15.32:6837","nonce":2569449994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/278686565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":278686565},{"type":"v1","addr":"172.21.15.189:6835","nonce":278686565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6835/1558303056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1558303056},{"type":"v1","addr":"172.21.15.71:6835","nonce":1558303056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.189:6837/686520873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":686520873},{"type":"v1","addr":"172.21.15.189:6837","nonce":686520873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.71:6837/2324039932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2324039932},{"type":"v1","addr":"172.21.15.71:6837","nonce":2324039932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6839/352970107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":352970107},{"type":"v1","addr":"172.21.15.189:6839","nonce":352970107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.32:6839/4160386524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4160386524},{"type":"v1","addr":"172.21.15.32:6839","nonce":4160386524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.71:6839/3517409993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3517409993},{"type":"v1","addr":"172.21.15.71:6839","nonce":3517409993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24517,24533,24560,24590,24541]},"id":1}]} 2024-04-30T08:35:17.951 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2024-04-30T08:35:17.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.952+0000 7fef49200640 1 -- 172.21.15.32:0/1275922222 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fef280afb70 msgr2=0x7fef280b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:17.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.952+0000 7fef49200640 1 --2- 172.21.15.32:0/1275922222 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fef280afb70 0x7fef280b2080 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fef2c031440 tx=0x7fef2c0314a0 comp rx=0 tx=0).stop 2024-04-30T08:35:17.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.952+0000 7fef49200640 1 -- 172.21.15.32:0/1275922222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef44105bb0 msgr2=0x7fef4419c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:17.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.952+0000 7fef49200640 1 --2- 172.21.15.32:0/1275922222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef44105bb0 0x7fef4419c280 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fef44105480 tx=0x7fef3400ee50 comp rx=0 tx=0).stop 2024-04-30T08:35:17.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.953+0000 7fef49200640 1 -- 172.21.15.32:0/1275922222 shutdown_connections 2024-04-30T08:35:17.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.953+0000 7fef49200640 1 --2- 172.21.15.32:0/1275922222 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fef280afb70 0x7fef280b2080 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:17.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.953+0000 7fef49200640 1 --2- 172.21.15.32:0/1275922222 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fef4410baf0 0x7fef441a3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:17.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.953+0000 7fef49200640 1 --2- 172.21.15.32:0/1275922222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fef44106df0 0x7fef4419c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:17.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.953+0000 7fef49200640 1 --2- 172.21.15.32:0/1275922222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fef44105bb0 0x7fef4419c280 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:17.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.953+0000 7fef49200640 1 -- 172.21.15.32:0/1275922222 >> 172.21.15.32:0/1275922222 conn(0x7fef44100da0 msgr2=0x7fef44107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:17.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.953+0000 7fef49200640 1 -- 172.21.15.32:0/1275922222 shutdown_connections 2024-04-30T08:35:17.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:17.953+0000 7fef49200640 1 -- 172.21.15.32:0/1275922222 wait complete. 2024-04-30T08:35:17.986 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-30T08:35:18.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.270+0000 7f333b000640 1 -- 172.21.15.32:0/946439656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f333410b960 msgr2=0x7f333410dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:18.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.270+0000 7f333b000640 1 --2- 172.21.15.32:0/946439656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f333410b960 0x7f333410dd80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f332c009be0 tx=0x7f332c02a6c0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.271+0000 7f333b000640 1 -- 172.21.15.32:0/946439656 shutdown_connections 2024-04-30T08:35:18.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.271+0000 7f333b000640 1 --2- 172.21.15.32:0/946439656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f333410b960 0x7f333410dd80 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.271+0000 7f333b000640 1 --2- 172.21.15.32:0/946439656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3334075900 0x7f333410b420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.271+0000 7f333b000640 1 --2- 172.21.15.32:0/946439656 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3334074fb0 0x7f33340753c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.271+0000 7f333b000640 1 -- 172.21.15.32:0/946439656 >> 172.21.15.32:0/946439656 conn(0x7f33340feba0 msgr2=0x7f3334101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:18.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.271+0000 7f333b000640 1 -- 172.21.15.32:0/946439656 shutdown_connections 2024-04-30T08:35:18.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.271+0000 7f333b000640 1 -- 172.21.15.32:0/946439656 wait complete. 2024-04-30T08:35:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.272+0000 7f333b000640 1 Processor -- start 2024-04-30T08:35:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.272+0000 7f333b000640 1 -- start start 2024-04-30T08:35:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.272+0000 7f333b000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3334074fb0 0x7f333419c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.272+0000 7f333b000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3334075900 0x7f333419c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.272+0000 7f333b000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f333410b960 0x7f33341a3730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.272+0000 7f333b000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f33341116b0 con 0x7f3334075900 2024-04-30T08:35:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.272+0000 7f333b000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f3334111530 con 0x7f333410b960 2024-04-30T08:35:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.272+0000 7f333b000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3334111830 con 0x7f3334074fb0 2024-04-30T08:35:18.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.273+0000 7f3339200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3334075900 0x7f333419c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:18.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.273+0000 7f3339200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3334075900 0x7f333419c7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39496/0 (socket says 172.21.15.32:39496) 2024-04-30T08:35:18.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.273+0000 7f3339200640 1 -- 172.21.15.32:0/3667559219 learned_addr learned my addr 172.21.15.32:0/3667559219 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:18.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.273+0000 7f333a600640 1 --2- 172.21.15.32:0/3667559219 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f333410b960 0x7f33341a3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:18.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.273+0000 7f3339c00640 1 --2- 172.21.15.32:0/3667559219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3334074fb0 0x7f333419c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:18.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.273+0000 7f3339200640 1 -- 172.21.15.32:0/3667559219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3334074fb0 msgr2=0x7f333419c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:18.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.273+0000 7f3339200640 1 --2- 172.21.15.32:0/3667559219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3334074fb0 0x7f333419c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.273+0000 7f3339200640 1 -- 172.21.15.32:0/3667559219 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f333410b960 msgr2=0x7f33341a3730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:18.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.273+0000 7f3339200640 1 --2- 172.21.15.32:0/3667559219 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f333410b960 0x7f33341a3730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.273+0000 7f3339200640 1 -- 172.21.15.32:0/3667559219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33341a3da0 con 0x7f3334075900 2024-04-30T08:35:18.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.273+0000 7f333a600640 1 --2- 172.21.15.32:0/3667559219 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f333410b960 0x7f33341a3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:18.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.274+0000 7f3339200640 1 --2- 172.21.15.32:0/3667559219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3334075900 0x7f333419c7f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f332000e9c0 tx=0x7f332000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:18.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.274+0000 7f3332000640 1 -- 172.21.15.32:0/3667559219 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f332000ccc0 con 0x7f3334075900 2024-04-30T08:35:18.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.274+0000 7f3332000640 1 -- 172.21.15.32:0/3667559219 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f33200054d0 con 0x7f3334075900 2024-04-30T08:35:18.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.274+0000 7f333b000640 1 -- 172.21.15.32:0/3667559219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33341a4090 con 0x7f3334075900 2024-04-30T08:35:18.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.274+0000 7f3332000640 1 -- 172.21.15.32:0/3667559219 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3320010430 con 0x7f3334075900 2024-04-30T08:35:18.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.274+0000 7f333b000640 1 -- 172.21.15.32:0/3667559219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33341a45e0 con 0x7f3334075900 2024-04-30T08:35:18.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.276+0000 7f333b000640 1 -- 172.21.15.32:0/3667559219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f333410ab20 con 0x7f3334075900 2024-04-30T08:35:18.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.278+0000 7f3332000640 1 -- 172.21.15.32:0/3667559219 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f3320005020 con 0x7f3334075900 2024-04-30T08:35:18.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.279+0000 7f3332000640 1 --2- 172.21.15.32:0/3667559219 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f331c0afb70 0x7f331c0b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:18.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.279+0000 7f3332000640 1 -- 172.21.15.32:0/3667559219 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f33200df5c0 con 0x7f3334075900 2024-04-30T08:35:18.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.280+0000 7f3339c00640 1 --2- 172.21.15.32:0/3667559219 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f331c0afb70 0x7f331c0b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:18.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.281+0000 7f3339c00640 1 --2- 172.21.15.32:0/3667559219 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f331c0afb70 0x7f331c0b2080 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f33340767e0 tx=0x7f3328005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:18.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.284+0000 7f3332000640 1 -- 172.21.15.32:0/3667559219 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f33200a7c30 con 0x7f3334075900 2024-04-30T08:35:18.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:18 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1511509074' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-30T08:35:18.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:18 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1275922222' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:35:18.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.492+0000 7f333b000640 1 -- 172.21.15.32:0/3667559219 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3334063820 con 0x7f3334075900 2024-04-30T08:35:18.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.497+0000 7f3332000640 1 -- 172.21.15.32:0/3667559219 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+24730 (secure 0 0 0) 0x7f33200a7530 con 0x7f3334075900 2024-04-30T08:35:18.498 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:35:18.499 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":85,"fsid":"98c0a670-06ca-11ef-bc93-c7b262605968","created":"2024-04-30T08:22:11.122875+0000","modified":"2024-04-30T08:33:31.872704+0000","last_up_change":"2024-04-30T08:30:50.350439+0000","last_in_change":"2024-04-30T08:30:30.210620+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-30T08:26:19.728331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-30T08:31:49.800542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-30T08:31:51.211320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-30T08:31:55.716625+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a7e060c8-1cbb-418f-9a03-06693f6b3f52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6803","nonce":2624700402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6805","nonce":2624700402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6809","nonce":2624700402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6807","nonce":2624700402}]},"public_addr":"172.21.15.32:6803/2624700402","cluster_addr":"172.21.15.32:6805/2624700402","heartbeat_back_addr":"172.21.15.32:6809/2624700402","heartbeat_front_addr":"172.21.15.32:6807/2624700402","state":["exists","up"]},{"osd":1,"uuid":"72ef2fb1-e5ca-48ef-aa3a-4796a64b922f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6801","nonce":444649711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6803","nonce":444649711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6807","nonce":444649711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6805","nonce":444649711}]},"public_addr":"172.21.15.71:6801/444649711","cluster_addr":"172.21.15.71:6803/444649711","heartbeat_back_addr":"172.21.15.71:6807/444649711","heartbeat_front_addr":"172.21.15.71:6805/444649711","state":["exists","up"]},{"osd":2,"uuid":"704e5e8a-935e-47c7-ae28-27119876ac09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6801","nonce":3255528671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6803","nonce":3255528671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6807","nonce":3255528671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6805","nonce":3255528671}]},"public_addr":"172.21.15.189:6801/3255528671","cluster_addr":"172.21.15.189:6803/3255528671","heartbeat_back_addr":"172.21.15.189:6807/3255528671","heartbeat_front_addr":"172.21.15.189:6805/3255528671","state":["exists","up"]},{"osd":3,"uuid":"575004f7-6559-4a62-863e-e33078b0eebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6811","nonce":1384702112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6813","nonce":1384702112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6817","nonce":1384702112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6815","nonce":1384702112}]},"public_addr":"172.21.15.32:6811/1384702112","cluster_addr":"172.21.15.32:6813/1384702112","heartbeat_back_addr":"172.21.15.32:6817/1384702112","heartbeat_front_addr":"172.21.15.32:6815/1384702112","state":["exists","up"]},{"osd":4,"uuid":"e1c1e894-074d-4c78-89ef-9e5a114268bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6809","nonce":4250534574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6811","nonce":4250534574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6815","nonce":4250534574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6813","nonce":4250534574}]},"public_addr":"172.21.15.71:6809/4250534574","cluster_addr":"172.21.15.71:6811/4250534574","heartbeat_back_addr":"172.21.15.71:6815/4250534574","heartbeat_front_addr":"172.21.15.71:6813/4250534574","state":["exists","up"]},{"osd":5,"uuid":"12c022d4-fac6-40c4-ae7a-691571bee5d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6809","nonce":4245346198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6811","nonce":4245346198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6815","nonce":4245346198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6813","nonce":4245346198}]},"public_addr":"172.21.15.189:6809/4245346198","cluster_addr":"172.21.15.189:6811/4245346198","heartbeat_back_addr":"172.21.15.189:6815/4245346198","heartbeat_front_addr":"172.21.15.189:6813/4245346198","state":["exists","up"]},{"osd":6,"uuid":"b413f8ee-3d4a-415a-a2ba-d68437b7330d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6819","nonce":1910389190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6821","nonce":1910389190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6825","nonce":1910389190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6823","nonce":1910389190}]},"public_addr":"172.21.15.32:6819/1910389190","cluster_addr":"172.21.15.32:6821/1910389190","heartbeat_back_addr":"172.21.15.32:6825/1910389190","heartbeat_front_addr":"172.21.15.32:6823/1910389190","state":["exists","up"]},{"osd":7,"uuid":"aac64fb3-fc13-4126-b842-7b06ad5ce238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6817","nonce":692607380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6819","nonce":692607380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6823","nonce":692607380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6821","nonce":692607380}]},"public_addr":"172.21.15.71:6817/692607380","cluster_addr":"172.21.15.71:6819/692607380","heartbeat_back_addr":"172.21.15.71:6823/692607380","heartbeat_front_addr":"172.21.15.71:6821/692607380","state":["exists","up"]},{"osd":8,"uuid":"cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6817","nonce":3973697069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6819","nonce":3973697069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6823","nonce":3973697069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6821","nonce":3973697069}]},"public_addr":"172.21.15.189:6817/3973697069","cluster_addr":"172.21.15.189:6819/3973697069","heartbeat_back_addr":"172.21.15.189:6823/3973697069","heartbeat_front_addr":"172.21.15.189:6821/3973697069","state":["exists","up"]},{"osd":9,"uuid":"ef4a1645-ae6e-45c3-82ad-161510e62aa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6827","nonce":46066685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6829","nonce":46066685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6833","nonce":46066685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6831","nonce":46066685}]},"public_addr":"172.21.15.32:6827/46066685","cluster_addr":"172.21.15.32:6829/46066685","heartbeat_back_addr":"172.21.15.32:6833/46066685","heartbeat_front_addr":"172.21.15.32:6831/46066685","state":["exists","up"]},{"osd":10,"uuid":"25b26a38-a644-409f-b941-59efdf04a3ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6825","nonce":1886290293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6827","nonce":1886290293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6831","nonce":1886290293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6829","nonce":1886290293}]},"public_addr":"172.21.15.71:6825/1886290293","cluster_addr":"172.21.15.71:6827/1886290293","heartbeat_back_addr":"172.21.15.71:6831/1886290293","heartbeat_front_addr":"172.21.15.71:6829/1886290293","state":["exists","up"]},{"osd":11,"uuid":"ab96b3ff-3688-455b-af64-60226c5bfbff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6825","nonce":850185001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6827","nonce":850185001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6831","nonce":850185001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6829","nonce":850185001}]},"public_addr":"172.21.15.189:6825/850185001","cluster_addr":"172.21.15.189:6827/850185001","heartbeat_back_addr":"172.21.15.189:6831/850185001","heartbeat_front_addr":"172.21.15.189:6829/850185001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:27.220308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:53.230783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:17.190642+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:52.603799+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:20.568671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:46.158363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:22.706933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:50.665173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:16.602410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:53.814698+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:22.435189+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:48.651949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.e","mappings":[{"from":4,"to":7}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.28","mappings":[{"from":3,"to":6},{"from":5,"to":8}]},{"pgid":"3.30","mappings":[{"from":1,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1266897222":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/2915689533":"2024-05-01T08:23:29.526664+0000","172.21.15.32:6801/4019351306":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/425836193":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/1134075722":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6801/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/21059253":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/3245550894":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/4019351306":"2024-05-01T08:23:29.526664+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-30T08:35:18.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.500+0000 7f333b000640 1 -- 172.21.15.32:0/3667559219 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f331c0afb70 msgr2=0x7f331c0b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:18.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.500+0000 7f333b000640 1 --2- 172.21.15.32:0/3667559219 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f331c0afb70 0x7f331c0b2080 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f33340767e0 tx=0x7f3328005f90 comp rx=0 tx=0).stop 2024-04-30T08:35:18.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.500+0000 7f333b000640 1 -- 172.21.15.32:0/3667559219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3334075900 msgr2=0x7f333419c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:18.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.500+0000 7f333b000640 1 --2- 172.21.15.32:0/3667559219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3334075900 0x7f333419c7f0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f332000e9c0 tx=0x7f332000ee80 comp rx=0 tx=0).stop 2024-04-30T08:35:18.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.500+0000 7f333b000640 1 -- 172.21.15.32:0/3667559219 shutdown_connections 2024-04-30T08:35:18.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.500+0000 7f333b000640 1 --2- 172.21.15.32:0/3667559219 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f331c0afb70 0x7f331c0b2080 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.500+0000 7f333b000640 1 --2- 172.21.15.32:0/3667559219 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f333410b960 0x7f33341a3730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.500+0000 7f333b000640 1 --2- 172.21.15.32:0/3667559219 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3334075900 0x7f333419c7f0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.500+0000 7f333b000640 1 --2- 172.21.15.32:0/3667559219 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3334074fb0 0x7f333419c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.500+0000 7f333b000640 1 -- 172.21.15.32:0/3667559219 >> 172.21.15.32:0/3667559219 conn(0x7f33340feba0 msgr2=0x7f33340ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:18.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.500+0000 7f333b000640 1 -- 172.21.15.32:0/3667559219 shutdown_connections 2024-04-30T08:35:18.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.500+0000 7f333b000640 1 -- 172.21.15.32:0/3667559219 wait complete. 2024-04-30T08:35:18.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:18 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1511509074' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-30T08:35:18.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:18 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1275922222' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:35:18.528 DEBUG:tasks.fwd_scrub:Yielding 2024-04-30T08:35:18.528 INFO:teuthology.run_tasks:Running task background_exec... 2024-04-30T08:35:18.539 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi032.front.sepia.ceph.com 2024-04-30T08:35:18.539 DEBUG:teuthology.orchestra.run.smithi032:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-04-30T08:35:18.540 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-04-30T08:35:18.540 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-30T08:35:18.540 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-30T08:35:18.540 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-30T08:35:18.545 INFO:teuthology.run_tasks:Running task workunit... 2024-04-30T08:35:18.561 INFO:tasks.workunit:Pulling workunits from ref 8feaf7d9864e980041b042fd5d964ac18adfd20f 2024-04-30T08:35:18.561 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-30T08:35:18.561 INFO:tasks.workunit:timeout=3h 2024-04-30T08:35:18.561 INFO:tasks.workunit:cleanup=True 2024-04-30T08:35:18.562 DEBUG:teuthology.orchestra.run.smithi032:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-30T08:35:18.563 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:18 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1511509074' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-30T08:35:18.563 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:18 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1275922222' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:35:18.606 INFO:teuthology.orchestra.run.smithi032.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-30T08:35:18.606 INFO:teuthology.orchestra.run.smithi032.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-30T08:35:18.606 INFO:teuthology.orchestra.run.smithi032.stdout:Device: 72h/114d Inode: 1099511628283 Links: 2 2024-04-30T08:35:18.606 INFO:teuthology.orchestra.run.smithi032.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-30T08:35:18.606 INFO:teuthology.orchestra.run.smithi032.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-30T08:35:18.606 INFO:teuthology.orchestra.run.smithi032.stdout:Access: 2024-04-30 08:32:31.302703222 +0000 2024-04-30T08:35:18.606 INFO:teuthology.orchestra.run.smithi032.stdout:Modify: 2024-04-30 08:32:31.302703222 +0000 2024-04-30T08:35:18.606 INFO:teuthology.orchestra.run.smithi032.stdout:Change: 2024-04-30 08:35:03.908656197 +0000 2024-04-30T08:35:18.606 INFO:teuthology.orchestra.run.smithi032.stdout: Birth: 2024-04-30 08:32:31.302703222 +0000 2024-04-30T08:35:18.607 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-30T08:35:18.607 DEBUG:teuthology.orchestra.run.smithi032:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-30T08:35:18.741 DEBUG:teuthology.orchestra.run.smithi032:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 8feaf7d9864e980041b042fd5d964ac18adfd20f 2024-04-30T08:35:18.795 INFO:tasks.workunit.client.0.smithi032.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-30T08:35:18.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.813+0000 7f4e7b600640 1 -- 172.21.15.32:0/2759491841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e74103ca0 msgr2=0x7f4e741040b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:18.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.813+0000 7f4e7b600640 1 --2- 172.21.15.32:0/2759491841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e74103ca0 0x7f4e741040b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4e64009900 tx=0x7f4e6402a3f0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.813+0000 7f4e7b600640 1 -- 172.21.15.32:0/2759491841 shutdown_connections 2024-04-30T08:35:18.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.813+0000 7f4e7b600640 1 --2- 172.21.15.32:0/2759491841 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e741089a0 0x7f4e7410adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.813+0000 7f4e7b600640 1 --2- 172.21.15.32:0/2759491841 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e74103ca0 0x7f4e741040b0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.813+0000 7f4e7b600640 1 --2- 172.21.15.32:0/2759491841 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e74102a60 0x7f4e74103760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.813+0000 7f4e7b600640 1 -- 172.21.15.32:0/2759491841 >> 172.21.15.32:0/2759491841 conn(0x7f4e740fdc50 msgr2=0x7f4e74100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:18.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.813+0000 7f4e7b600640 1 -- 172.21.15.32:0/2759491841 shutdown_connections 2024-04-30T08:35:18.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.813+0000 7f4e7b600640 1 -- 172.21.15.32:0/2759491841 wait complete. 2024-04-30T08:35:18.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.815+0000 7f4e7b600640 1 Processor -- start 2024-04-30T08:35:18.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.815+0000 7f4e7b600640 1 -- start start 2024-04-30T08:35:18.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.815+0000 7f4e7b600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e74102a60 0x7f4e7410d330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:18.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.815+0000 7f4e7b600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e74103ca0 0x7f4e7410b9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:18.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.815+0000 7f4e7b600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e741089a0 0x7f4e7410bf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:18.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.815+0000 7f4e7b600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7410f690 con 0x7f4e74103ca0 2024-04-30T08:35:18.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.815+0000 7f4e7b600640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7410f510 con 0x7f4e74102a60 2024-04-30T08:35:18.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.815+0000 7f4e7b600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4e7410f810 con 0x7f4e741089a0 2024-04-30T08:35:18.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.816+0000 7f4e78c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e74102a60 0x7f4e7410d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:18.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.816+0000 7f4e78c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e74102a60 0x7f4e7410d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:57024/0 (socket says 172.21.15.32:57024) 2024-04-30T08:35:18.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.816+0000 7f4e78c00640 1 -- 172.21.15.32:0/123087706 learned_addr learned my addr 172.21.15.32:0/123087706 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:18.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.816+0000 7f4e79600640 1 --2- 172.21.15.32:0/123087706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e741089a0 0x7f4e7410bf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:18.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.816+0000 7f4e73e00640 1 --2- 172.21.15.32:0/123087706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e74103ca0 0x7f4e7410b9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:18.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.816+0000 7f4e78c00640 1 -- 172.21.15.32:0/123087706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e741089a0 msgr2=0x7f4e7410bf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:18.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.816+0000 7f4e78c00640 1 --2- 172.21.15.32:0/123087706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e741089a0 0x7f4e7410bf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.816+0000 7f4e78c00640 1 -- 172.21.15.32:0/123087706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e74103ca0 msgr2=0x7f4e7410b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:18.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.816+0000 7f4e78c00640 1 --2- 172.21.15.32:0/123087706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e74103ca0 0x7f4e7410b9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.816+0000 7f4e78c00640 1 -- 172.21.15.32:0/123087706 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e7410c640 con 0x7f4e74102a60 2024-04-30T08:35:18.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.816+0000 7f4e79600640 1 --2- 172.21.15.32:0/123087706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e741089a0 0x7f4e7410bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:35:18.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.816+0000 7f4e78c00640 1 --2- 172.21.15.32:0/123087706 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e74102a60 0x7f4e7410d330 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4e6800e980 tx=0x7f4e6800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:18.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.817+0000 7f4e71600640 1 -- 172.21.15.32:0/123087706 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e6800cd80 con 0x7f4e74102a60 2024-04-30T08:35:18.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.817+0000 7f4e7b600640 1 -- 172.21.15.32:0/123087706 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e741a4380 con 0x7f4e74102a60 2024-04-30T08:35:18.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.817+0000 7f4e7b600640 1 -- 172.21.15.32:0/123087706 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e741a4870 con 0x7f4e74102a60 2024-04-30T08:35:18.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.817+0000 7f4e71600640 1 -- 172.21.15.32:0/123087706 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e680108e0 con 0x7f4e74102a60 2024-04-30T08:35:18.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.817+0000 7f4e71600640 1 -- 172.21.15.32:0/123087706 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e68018890 con 0x7f4e74102a60 2024-04-30T08:35:18.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.821+0000 7f4e8264b740 1 Processor -- start 2024-04-30T08:35:18.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.821+0000 7f4e8264b740 1 -- start start 2024-04-30T08:35:18.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.821+0000 7f4e8264b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a79f3cf410 0x55a79f7711f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:18.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.822+0000 7f4e8264b740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a79f3a6550 0x55a79f771770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:18.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.822+0000 7f4e8264b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a79f3ae730 0x55a79f778670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:18.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.822+0000 7f4e8264b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a79f6d5b50 con 0x55a79f3cf410 2024-04-30T08:35:18.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.822+0000 7f4e8264b740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55a79f67b080 con 0x55a79f3a6550 2024-04-30T08:35:18.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.822+0000 7f4e8264b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a79f6d1b10 con 0x55a79f3ae730 2024-04-30T08:35:18.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.822+0000 7f4e79600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a79f3cf410 0x55a79f7711f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:18.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.822+0000 7f4e79600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a79f3cf410 0x55a79f7711f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39528/0 (socket says 172.21.15.32:39528) 2024-04-30T08:35:18.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.822+0000 7f4e79600640 1 -- 172.21.15.32:0/3122204179 learned_addr learned my addr 172.21.15.32:0/3122204179 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:18.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.822+0000 7f4e73e00640 1 --2- 172.21.15.32:0/3122204179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a79f3ae730 0x55a79f778670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:18.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.822+0000 7f4e79600640 1 -- 172.21.15.32:0/3122204179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a79f3ae730 msgr2=0x55a79f778670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:18.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.822+0000 7f4e79600640 1 --2- 172.21.15.32:0/3122204179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a79f3ae730 0x55a79f778670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.822+0000 7f4e79600640 1 -- 172.21.15.32:0/3122204179 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a79f3a6550 msgr2=0x55a79f771770 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-30T08:35:18.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.822+0000 7f4e79600640 1 --2- 172.21.15.32:0/3122204179 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a79f3a6550 0x55a79f771770 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.822+0000 7f4e79600640 1 -- 172.21.15.32:0/3122204179 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a79f69b0c0 con 0x55a79f3cf410 2024-04-30T08:35:18.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.823+0000 7f4e79600640 1 --2- 172.21.15.32:0/3122204179 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a79f3cf410 0x55a79f7711f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c009fd0 tx=0x7f4e6c0108e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c86522ed2e1ebd8d server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:18.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.823+0000 7f4e61c00640 1 -- 172.21.15.32:0/3122204179 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e6c011760 con 0x55a79f3cf410 2024-04-30T08:35:18.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.823+0000 7f4e61c00640 1 -- 172.21.15.32:0/3122204179 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e6c011db0 con 0x55a79f3cf410 2024-04-30T08:35:18.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.823+0000 7f4e61c00640 1 -- 172.21.15.32:0/3122204179 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e6c00e3c0 con 0x55a79f3cf410 2024-04-30T08:35:18.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.823+0000 7f4e71600640 1 -- 172.21.15.32:0/123087706 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f4e6801f020 con 0x7f4e74102a60 2024-04-30T08:35:18.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.824+0000 7f4e8264b740 1 -- 172.21.15.32:0/3122204179 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a79f3cf410 msgr2=0x55a79f7711f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:18.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.824+0000 7f4e8264b740 1 --2- 172.21.15.32:0/3122204179 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a79f3cf410 0x55a79f7711f0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c009fd0 tx=0x7f4e6c0108e0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.824+0000 7f4e8264b740 1 -- 172.21.15.32:0/3122204179 shutdown_connections 2024-04-30T08:35:18.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.824+0000 7f4e8264b740 1 --2- 172.21.15.32:0/3122204179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a79f3ae730 0x55a79f778670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.824+0000 7f4e8264b740 1 --2- 172.21.15.32:0/3122204179 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a79f3a6550 0x55a79f771770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.824+0000 7f4e8264b740 1 --2- 172.21.15.32:0/3122204179 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a79f3cf410 0x55a79f7711f0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.824+0000 7f4e8264b740 1 -- 172.21.15.32:0/3122204179 >> 172.21.15.32:0/3122204179 conn(0x55a79f3c3560 msgr2=0x55a79f76cb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:18.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.825+0000 7f4e8264b740 1 -- 172.21.15.32:0/3122204179 shutdown_connections 2024-04-30T08:35:18.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.825+0000 7f4e8264b740 1 -- 172.21.15.32:0/3122204179 wait complete. 2024-04-30T08:35:18.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.825+0000 7f4e8264b740 1 -- start start 2024-04-30T08:35:18.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.825+0000 7f4e71600640 1 --2- 172.21.15.32:0/123087706 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4e5c0afb10 0x7f4e5c0b2020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:18.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.825+0000 7f4e8264b740 1 Processor -- start 2024-04-30T08:35:18.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.826+0000 7f4e71600640 1 -- 172.21.15.32:0/123087706 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f4e680df300 con 0x7f4e74102a60 2024-04-30T08:35:18.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.826+0000 7f4e79600640 1 --2- 172.21.15.32:0/123087706 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4e5c0afb10 0x7f4e5c0b2020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:18.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.826+0000 7f4e8264b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a79f3a6550 0x55a79f77ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:18.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.826+0000 7f4e8264b740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a79f3ae730 0x55a79f77b340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:18.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.826+0000 7f4e8264b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a79f3cf410 0x55a79f782250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:18.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.826+0000 7f4e8264b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a79f7790c0 con 0x55a79f3cf410 2024-04-30T08:35:18.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.826+0000 7f4e8264b740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55a79f68eb20 con 0x55a79f3ae730 2024-04-30T08:35:18.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.826+0000 7f4e8264b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a79f779240 con 0x55a79f3a6550 2024-04-30T08:35:18.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.826+0000 7f4e8264b740 10 client.0 _ll_register_callbacks cb 0x55a79f6cca90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:35:18.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.827+0000 7f4e78c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a79f3a6550 0x55a79f77ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:18.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.827+0000 7f4e78c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a79f3a6550 0x55a79f77ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:58548/0 (socket says 172.21.15.32:58548) 2024-04-30T08:35:18.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.827+0000 7f4e78c00640 1 -- 172.21.15.32:0/1761842814 learned_addr learned my addr 172.21.15.32:0/1761842814 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:18.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.827+0000 7f4e73e00640 1 --2- 172.21.15.32:0/1761842814 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a79f3ae730 0x55a79f77b340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:18.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.827+0000 7f4e79600640 1 --2- 172.21.15.32:0/1761842814 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a79f3cf410 0x55a79f782250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:18.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.827+0000 7f4e79600640 1 --2- 172.21.15.32:0/123087706 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4e5c0afb10 0x7f4e5c0b2020 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c005cc0 tx=0x7f4e6c010f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:18.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.827+0000 7f4e78c00640 1 -- 172.21.15.32:0/1761842814 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a79f3ae730 msgr2=0x55a79f77b340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:18.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.827+0000 7f4e78c00640 1 --2- 172.21.15.32:0/1761842814 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a79f3ae730 0x55a79f77b340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.827+0000 7f4e78c00640 1 -- 172.21.15.32:0/1761842814 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a79f3cf410 msgr2=0x55a79f782250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:18.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.827+0000 7f4e78c00640 1 --2- 172.21.15.32:0/1761842814 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a79f3cf410 0x55a79f782250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.827+0000 7f4e78c00640 1 -- 172.21.15.32:0/1761842814 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a79f4af7e0 con 0x55a79f3a6550 2024-04-30T08:35:18.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.827+0000 7f4e78c00640 1 --2- 172.21.15.32:0/1761842814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a79f3a6550 0x55a79f77ae00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4e680a4520 tx=0x7f4e68010f70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5cf0c13b907af547 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:18.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.828+0000 7f4e56000640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-30T08:35:18.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.828+0000 7f4e56000640 1 -- 172.21.15.32:0/1761842814 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e680a72b0 con 0x55a79f3a6550 2024-04-30T08:35:18.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.828+0000 7f4e56000640 1 -- 172.21.15.32:0/1761842814 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f4e680afc80 con 0x55a79f3a6550 2024-04-30T08:35:18.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.828+0000 7f4e8264b740 1 -- 172.21.15.32:0/1761842814 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e6c011db0 con 0x55a79f3a6550 2024-04-30T08:35:18.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.828+0000 7f4e56000640 1 -- 172.21.15.32:0/1761842814 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4e680ab660 con 0x55a79f3a6550 2024-04-30T08:35:18.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.828+0000 7f4e8264b740 10 client.25633 fetch_fsmap learned FSMap version 28 2024-04-30T08:35:18.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.828+0000 7f4e8264b740 1 -- 172.21.15.32:0/1761842814 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a79f50ff70 con 0x55a79f3a6550 2024-04-30T08:35:18.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.828+0000 7f4e56000640 1 -- 172.21.15.32:0/1761842814 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e6c011db0 con 0x55a79f3a6550 2024-04-30T08:35:18.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.829+0000 7f4e56000640 1 -- 172.21.15.32:0/1761842814 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f4e680ab130 con 0x55a79f3a6550 2024-04-30T08:35:18.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.829+0000 7f4e8264b740 10 client.25633 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:35:18.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.829+0000 7f4e8264b740 10 client.25633 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:35:18.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.829+0000 7f4e8264b740 20 client.25633 populate_metadata read hostname 'smithi032' 2024-04-30T08:35:18.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.829+0000 7f4e8264b740 1 --2- 172.21.15.32:0/1761842814 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55a79f38e1f0 0x55a79f788620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:18.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.829+0000 7f4e8264b740 4 client.25633 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:35:18.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.829+0000 7f4e8264b740 1 -- 172.21.15.32:0/1761842814 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a79f6e3760 con 0x55a79f38e1f0 2024-04-30T08:35:18.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.829+0000 7f4e73e00640 1 --2- 172.21.15.32:0/1761842814 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55a79f38e1f0 0x55a79f788620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:18.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.830+0000 7f4e73e00640 1 --2- 172.21.15.32:0/1761842814 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55a79f38e1f0 0x55a79f788620 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ac1e7dbb5b45fc2 server_cookie=40708535b4b22aa9 in_seq=0 out_seq=0 2024-04-30T08:35:18.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.830+0000 7f4e56000640 10 client.25633 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:18.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.831+0000 7f4e56000640 1 -- 172.21.15.32:0/1761842814 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55a79f6e3760 con 0x55a79f38e1f0 2024-04-30T08:35:18.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.831+0000 7f4e56000640 10 client.25633 handle_command_reply: tid=0 2024-04-30T08:35:18.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.831+0000 7f4e8264b740 1 client.25633 shutdown 2024-04-30T08:35:18.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.831+0000 7f4e8264b740 1 -- 172.21.15.32:0/1761842814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a79f3a6550 msgr2=0x55a79f77ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:18.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.831+0000 7f4e8264b740 1 --2- 172.21.15.32:0/1761842814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a79f3a6550 0x55a79f77ae00 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f4e680a4520 tx=0x7f4e68010f70 comp rx=0 tx=0).stop 2024-04-30T08:35:18.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.831+0000 7f4e8264b740 1 -- 172.21.15.32:0/1761842814 shutdown_connections 2024-04-30T08:35:18.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.831+0000 7f4e8264b740 1 --2- 172.21.15.32:0/1761842814 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55a79f38e1f0 0x55a79f788620 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.831+0000 7f4e8264b740 1 --2- 172.21.15.32:0/1761842814 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a79f3cf410 0x55a79f782250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.831+0000 7f4e56000640 0 client.25633 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:18.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.831+0000 7f4e8264b740 1 --2- 172.21.15.32:0/1761842814 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a79f3ae730 0x55a79f77b340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.831+0000 7f4e8264b740 1 --2- 172.21.15.32:0/1761842814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a79f3a6550 0x55a79f77ae00 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.831+0000 7f4e8264b740 1 -- 172.21.15.32:0/1761842814 >> 172.21.15.32:0/1761842814 conn(0x55a79f3c3560 msgr2=0x55a79f76bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:18.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.831+0000 7f4e8264b740 1 -- 172.21.15.32:0/1761842814 shutdown_connections 2024-04-30T08:35:18.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.831+0000 7f4e8264b740 1 -- 172.21.15.32:0/1761842814 wait complete. 2024-04-30T08:35:18.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.831+0000 7f4e8264b740 20 client.25633 trim_cache size 0 max 16384 2024-04-30T08:35:18.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.852+0000 7f4e8264b740 1 Processor -- start 2024-04-30T08:35:18.852 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.852+0000 7f4e8264b740 1 -- start start 2024-04-30T08:35:18.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.852+0000 7f4e8264b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a79f3a6550 0x55a79f7758a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:18.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.852+0000 7f4e8264b740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a79f3ae730 0x55a79f782210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:18.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.852+0000 7f4e8264b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a79f3cf410 0x55a79f770090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:18.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.852+0000 7f4e8264b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4e741aee00 con 0x55a79f3a6550 2024-04-30T08:35:18.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.852+0000 7f4e8264b740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4e741b0940 con 0x55a79f3ae730 2024-04-30T08:35:18.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.852+0000 7f4e8264b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4e741af4d0 con 0x55a79f3cf410 2024-04-30T08:35:18.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.853+0000 7f4e78c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a79f3a6550 0x55a79f7758a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:18.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.853+0000 7f4e78c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a79f3a6550 0x55a79f7758a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39538/0 (socket says 172.21.15.32:39538) 2024-04-30T08:35:18.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.853+0000 7f4e78c00640 1 -- 172.21.15.32:0/666838714 learned_addr learned my addr 172.21.15.32:0/666838714 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:18.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.853+0000 7f4e73e00640 1 --2- 172.21.15.32:0/666838714 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a79f3ae730 0x55a79f782210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:18.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.853+0000 7f4e79600640 1 --2- 172.21.15.32:0/666838714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a79f3cf410 0x55a79f770090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:18.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.853+0000 7f4e78c00640 1 -- 172.21.15.32:0/666838714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a79f3cf410 msgr2=0x55a79f770090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:18.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.853+0000 7f4e78c00640 1 --2- 172.21.15.32:0/666838714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a79f3cf410 0x55a79f770090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.853+0000 7f4e78c00640 1 -- 172.21.15.32:0/666838714 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a79f3ae730 msgr2=0x55a79f782210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:18.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.853+0000 7f4e78c00640 1 --2- 172.21.15.32:0/666838714 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a79f3ae730 0x55a79f782210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.853+0000 7f4e78c00640 1 -- 172.21.15.32:0/666838714 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a79f3a7170 con 0x55a79f3a6550 2024-04-30T08:35:18.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.853+0000 7f4e73e00640 1 --2- 172.21.15.32:0/666838714 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a79f3ae730 0x55a79f782210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:18.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.854+0000 7f4e78c00640 1 --2- 172.21.15.32:0/666838714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a79f3a6550 0x55a79f7758a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4e6800c6a0 tx=0x7f4e680093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c47368d3d803e1a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:18.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.854+0000 7f4e61200640 1 -- 172.21.15.32:0/666838714 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e680a3170 con 0x55a79f3a6550 2024-04-30T08:35:18.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.854+0000 7f4e61200640 1 -- 172.21.15.32:0/666838714 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e680a72b0 con 0x55a79f3a6550 2024-04-30T08:35:18.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.854+0000 7f4e61200640 1 -- 172.21.15.32:0/666838714 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e680afc80 con 0x55a79f3a6550 2024-04-30T08:35:18.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.855+0000 7f4e8264b740 1 -- 172.21.15.32:0/666838714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a79f3a6550 msgr2=0x55a79f7758a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:18.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.855+0000 7f4e8264b740 1 --2- 172.21.15.32:0/666838714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a79f3a6550 0x55a79f7758a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4e6800c6a0 tx=0x7f4e680093f0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.855+0000 7f4e8264b740 1 -- 172.21.15.32:0/666838714 shutdown_connections 2024-04-30T08:35:18.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.855+0000 7f4e8264b740 1 --2- 172.21.15.32:0/666838714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a79f3cf410 0x55a79f770090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.855+0000 7f4e8264b740 1 --2- 172.21.15.32:0/666838714 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a79f3ae730 0x55a79f782210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.855+0000 7f4e8264b740 1 --2- 172.21.15.32:0/666838714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a79f3a6550 0x55a79f7758a0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.855+0000 7f4e8264b740 1 -- 172.21.15.32:0/666838714 >> 172.21.15.32:0/666838714 conn(0x55a79f3c3560 msgr2=0x55a79f76f980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:18.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.855+0000 7f4e8264b740 1 -- 172.21.15.32:0/666838714 shutdown_connections 2024-04-30T08:35:18.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.855+0000 7f4e8264b740 1 -- 172.21.15.32:0/666838714 wait complete. 2024-04-30T08:35:18.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.856+0000 7f4e8264b740 1 -- start start 2024-04-30T08:35:18.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.856+0000 7f4e8264b740 1 Processor -- start 2024-04-30T08:35:18.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.856+0000 7f4e8264b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a79f3a6550 0x55a79f77f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:18.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.856+0000 7f4e8264b740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a79f3ae730 0x55a79f77fe30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:18.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.856+0000 7f4e8264b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a79f3cf410 0x55a79f780d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:18.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.856+0000 7f4e8264b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4e741b0270 con 0x55a79f3cf410 2024-04-30T08:35:18.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.856+0000 7f4e8264b740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4e741afba0 con 0x55a79f3ae730 2024-04-30T08:35:18.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.856+0000 7f4e8264b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4e741acdc0 con 0x55a79f3a6550 2024-04-30T08:35:18.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.857+0000 7f4e8264b740 10 client.0 _ll_register_callbacks cb 0x7f4e6800ef70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:35:18.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.857+0000 7f4e79600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a79f3cf410 0x55a79f780d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:18.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.857+0000 7f4e79600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a79f3cf410 0x55a79f780d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39546/0 (socket says 172.21.15.32:39546) 2024-04-30T08:35:18.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.857+0000 7f4e79600640 1 -- 172.21.15.32:0/3411248347 learned_addr learned my addr 172.21.15.32:0/3411248347 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:18.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.857+0000 7f4e78c00640 1 --2- 172.21.15.32:0/3411248347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a79f3a6550 0x55a79f77f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:18.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.857+0000 7f4e73e00640 1 --2- 172.21.15.32:0/3411248347 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a79f3ae730 0x55a79f77fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:18.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.857+0000 7f4e78c00640 1 -- 172.21.15.32:0/3411248347 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a79f3ae730 msgr2=0x55a79f77fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:18.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.857+0000 7f4e78c00640 1 --2- 172.21.15.32:0/3411248347 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a79f3ae730 0x55a79f77fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.857+0000 7f4e78c00640 1 -- 172.21.15.32:0/3411248347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a79f3cf410 msgr2=0x55a79f780d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:18.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.857+0000 7f4e78c00640 1 --2- 172.21.15.32:0/3411248347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a79f3cf410 0x55a79f780d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.857+0000 7f4e78c00640 1 -- 172.21.15.32:0/3411248347 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a79f781320 con 0x55a79f3a6550 2024-04-30T08:35:18.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.857+0000 7f4e79600640 1 --2- 172.21.15.32:0/3411248347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a79f3cf410 0x55a79f780d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:18.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.858+0000 7f4e78c00640 1 --2- 172.21.15.32:0/3411248347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a79f3a6550 0x55a79f77f8f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x55a79f77b7c0 tx=0x7f4e680aff90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c339338ab3ec1e46 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:18.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.858+0000 7f4e56000640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-30T08:35:18.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.858+0000 7f4e56000640 1 -- 172.21.15.32:0/3411248347 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e680b0110 con 0x55a79f3a6550 2024-04-30T08:35:18.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.858+0000 7f4e8264b740 1 -- 172.21.15.32:0/3411248347 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e680a72b0 con 0x55a79f3a6550 2024-04-30T08:35:18.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.858+0000 7f4e56000640 1 -- 172.21.15.32:0/3411248347 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f4e680a38a0 con 0x55a79f3a6550 2024-04-30T08:35:18.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.858+0000 7f4e56000640 1 -- 172.21.15.32:0/3411248347 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4e680a6510 con 0x55a79f3a6550 2024-04-30T08:35:18.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.858+0000 7f4e8264b740 10 client.25639 fetch_fsmap learned FSMap version 28 2024-04-30T08:35:18.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.858+0000 7f4e8264b740 1 -- 172.21.15.32:0/3411248347 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a79f781320 con 0x55a79f3a6550 2024-04-30T08:35:18.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.859+0000 7f4e56000640 1 -- 172.21.15.32:0/3411248347 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4e680a72b0 con 0x55a79f3a6550 2024-04-30T08:35:18.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.859+0000 7f4e56000640 1 -- 172.21.15.32:0/3411248347 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f4e680a2c20 con 0x55a79f3a6550 2024-04-30T08:35:18.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.859+0000 7f4e8264b740 10 client.25639 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:35:18.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.859+0000 7f4e8264b740 10 client.25639 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:35:18.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.859+0000 7f4e8264b740 20 client.25639 populate_metadata read hostname 'smithi032' 2024-04-30T08:35:18.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.859+0000 7f4e8264b740 1 --2- 172.21.15.32:0/3411248347 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55a79f363630 0x55a79f7f2da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:18.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.859+0000 7f4e8264b740 4 client.25639 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-30T08:35:18.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.859+0000 7f4e8264b740 1 -- 172.21.15.32:0/3411248347 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a79f6dd5d0 con 0x55a79f363630 2024-04-30T08:35:18.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.859+0000 7f4e73e00640 1 --2- 172.21.15.32:0/3411248347 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55a79f363630 0x55a79f7f2da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:18.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.860+0000 7f4e73e00640 1 --2- 172.21.15.32:0/3411248347 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55a79f363630 0x55a79f7f2da0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4e889f229d16c4c server_cookie=343f91d72201e60a in_seq=0 out_seq=0 2024-04-30T08:35:18.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.860+0000 7f4e56000640 10 client.25639 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:18.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.861+0000 7f4e56000640 1 -- 172.21.15.32:0/3411248347 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a79f6dd5d0 con 0x55a79f363630 2024-04-30T08:35:18.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.861+0000 7f4e56000640 10 client.25639 handle_command_reply: tid=0 2024-04-30T08:35:18.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.861+0000 7f4e8264b740 1 client.25639 shutdown 2024-04-30T08:35:18.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.861+0000 7f4e8264b740 1 -- 172.21.15.32:0/3411248347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a79f3a6550 msgr2=0x55a79f77f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:18.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.861+0000 7f4e8264b740 1 --2- 172.21.15.32:0/3411248347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a79f3a6550 0x55a79f77f8f0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x55a79f77b7c0 tx=0x7f4e680aff90 comp rx=0 tx=0).stop 2024-04-30T08:35:18.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.861+0000 7f4e8264b740 1 -- 172.21.15.32:0/3411248347 shutdown_connections 2024-04-30T08:35:18.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.861+0000 7f4e8264b740 1 --2- 172.21.15.32:0/3411248347 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55a79f363630 0x55a79f7f2da0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.861+0000 7f4e8264b740 1 --2- 172.21.15.32:0/3411248347 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a79f3cf410 0x55a79f780d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.861+0000 7f4e8264b740 1 --2- 172.21.15.32:0/3411248347 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a79f3ae730 0x55a79f77fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.861+0000 7f4e8264b740 1 --2- 172.21.15.32:0/3411248347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a79f3a6550 0x55a79f77f8f0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.861+0000 7f4e56000640 0 client.25639 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:18.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.861+0000 7f4e8264b740 1 -- 172.21.15.32:0/3411248347 >> 172.21.15.32:0/3411248347 conn(0x55a79f3c3560 msgr2=0x55a79f7703b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:18.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.862+0000 7f4e8264b740 1 -- 172.21.15.32:0/3411248347 shutdown_connections 2024-04-30T08:35:18.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.862+0000 7f4e8264b740 1 -- 172.21.15.32:0/3411248347 wait complete. 2024-04-30T08:35:18.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.862+0000 7f4e8264b740 20 client.25639 trim_cache size 0 max 16384 2024-04-30T08:35:18.867 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-30T08:35:18.867 INFO:teuthology.orchestra.run.smithi032.stdout: "return_code": 0, 2024-04-30T08:35:18.867 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_tag": "60505704-69d7-4e42-ad1d-0b74325a4a28", 2024-04-30T08:35:18.867 INFO:teuthology.orchestra.run.smithi032.stdout: "mode": "asynchronous" 2024-04-30T08:35:18.867 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-30T08:35:18.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.862+0000 7f4e7b600640 1 -- 172.21.15.32:0/123087706 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4e5c0afb10 msgr2=0x7f4e5c0b2020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:18.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.862+0000 7f4e7b600640 1 --2- 172.21.15.32:0/123087706 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4e5c0afb10 0x7f4e5c0b2020 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f4e6c005cc0 tx=0x7f4e6c010f90 comp rx=0 tx=0).stop 2024-04-30T08:35:18.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.862+0000 7f4e7b600640 1 -- 172.21.15.32:0/123087706 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e74102a60 msgr2=0x7f4e7410d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:18.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.862+0000 7f4e7b600640 1 --2- 172.21.15.32:0/123087706 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e74102a60 0x7f4e7410d330 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f4e6800e980 tx=0x7f4e6800ee40 comp rx=0 tx=0).stop 2024-04-30T08:35:18.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.863+0000 7f4e7b600640 1 -- 172.21.15.32:0/123087706 shutdown_connections 2024-04-30T08:35:18.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.863+0000 7f4e7b600640 1 --2- 172.21.15.32:0/123087706 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4e5c0afb10 0x7f4e5c0b2020 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.863+0000 7f4e7b600640 1 --2- 172.21.15.32:0/123087706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e741089a0 0x7f4e7410bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.863+0000 7f4e7b600640 1 --2- 172.21.15.32:0/123087706 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4e74103ca0 0x7f4e7410b9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.863+0000 7f4e7b600640 1 --2- 172.21.15.32:0/123087706 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4e74102a60 0x7f4e7410d330 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:18.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.863+0000 7f4e7b600640 1 -- 172.21.15.32:0/123087706 >> 172.21.15.32:0/123087706 conn(0x7f4e740fdc50 msgr2=0x7f4e74104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:18.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.863+0000 7f4e7b600640 1 -- 172.21.15.32:0/123087706 shutdown_connections 2024-04-30T08:35:18.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:18.863+0000 7f4e7b600640 1 -- 172.21.15.32:0/123087706 wait complete. 2024-04-30T08:35:18.889 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-30T08:35:18.890 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-30T08:35:19.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.233+0000 7f99bac00640 1 -- 172.21.15.32:0/209720809 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99b4103990 msgr2=0x7f99b4104690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:19.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.233+0000 7f99bac00640 1 --2- 172.21.15.32:0/209720809 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99b4103990 0x7f99b4104690 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f99a8009900 tx=0x7f99a802a3f0 comp rx=0 tx=0).stop 2024-04-30T08:35:19.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.233+0000 7f99bac00640 1 -- 172.21.15.32:0/209720809 shutdown_connections 2024-04-30T08:35:19.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.233+0000 7f99bac00640 1 --2- 172.21.15.32:0/209720809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99b41098d0 0x7f99b410bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:19.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.233+0000 7f99bac00640 1 --2- 172.21.15.32:0/209720809 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99b4104bd0 0x7f99b4104fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:19.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.233+0000 7f99bac00640 1 --2- 172.21.15.32:0/209720809 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99b4103990 0x7f99b4104690 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:19.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.233+0000 7f99bac00640 1 -- 172.21.15.32:0/209720809 >> 172.21.15.32:0/209720809 conn(0x7f99b40feba0 msgr2=0x7f99b4101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:19.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.233+0000 7f99bac00640 1 -- 172.21.15.32:0/209720809 shutdown_connections 2024-04-30T08:35:19.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.233+0000 7f99bac00640 1 -- 172.21.15.32:0/209720809 wait complete. 2024-04-30T08:35:19.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.234+0000 7f99bac00640 1 Processor -- start 2024-04-30T08:35:19.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.235+0000 7f99bac00640 1 -- start start 2024-04-30T08:35:19.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.235+0000 7f99bac00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99b4103990 0x7f99b419a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:19.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.235+0000 7f99bac00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99b4104bd0 0x7f99b419a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:19.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.235+0000 7f99bac00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99b41098d0 0x7f99b41a15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:19.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.235+0000 7f99bac00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f99b410f4c0 con 0x7f99b4103990 2024-04-30T08:35:19.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.235+0000 7f99bac00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f99b410f340 con 0x7f99b4104bd0 2024-04-30T08:35:19.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.235+0000 7f99bac00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f99b410f640 con 0x7f99b41098d0 2024-04-30T08:35:19.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.235+0000 7f99b8e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99b4104bd0 0x7f99b419a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:19.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.235+0000 7f99b9800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99b4103990 0x7f99b419a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:19.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.236+0000 7f99b8e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99b4104bd0 0x7f99b419a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:57070/0 (socket says 172.21.15.32:57070) 2024-04-30T08:35:19.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.236+0000 7f99b8e00640 1 -- 172.21.15.32:0/4175217994 learned_addr learned my addr 172.21.15.32:0/4175217994 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:19.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.236+0000 7f99ba200640 1 --2- 172.21.15.32:0/4175217994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99b41098d0 0x7f99b41a15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:19.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.236+0000 7f99b8e00640 1 -- 172.21.15.32:0/4175217994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99b41098d0 msgr2=0x7f99b41a15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:19.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.236+0000 7f99b8e00640 1 --2- 172.21.15.32:0/4175217994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99b41098d0 0x7f99b41a15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:19.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.236+0000 7f99b8e00640 1 -- 172.21.15.32:0/4175217994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99b4103990 msgr2=0x7f99b419a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:19.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.236+0000 7f99b8e00640 1 --2- 172.21.15.32:0/4175217994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99b4103990 0x7f99b419a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:19.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.236+0000 7f99b8e00640 1 -- 172.21.15.32:0/4175217994 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99b41a1c10 con 0x7f99b4104bd0 2024-04-30T08:35:19.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.236+0000 7f99ba200640 1 --2- 172.21.15.32:0/4175217994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99b41098d0 0x7f99b41a15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:19.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.237+0000 7f99b8e00640 1 --2- 172.21.15.32:0/4175217994 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99b4104bd0 0x7f99b419a610 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f99a000e9f0 tx=0x7f99a000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:19.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.237+0000 7f99b2000640 1 -- 172.21.15.32:0/4175217994 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a000cd80 con 0x7f99b4104bd0 2024-04-30T08:35:19.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.237+0000 7f99b2000640 1 -- 172.21.15.32:0/4175217994 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99a00108e0 con 0x7f99b4104bd0 2024-04-30T08:35:19.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.237+0000 7f99b2000640 1 -- 172.21.15.32:0/4175217994 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a0018980 con 0x7f99b4104bd0 2024-04-30T08:35:19.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.237+0000 7f99bac00640 1 -- 172.21.15.32:0/4175217994 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99b41a1f00 con 0x7f99b4104bd0 2024-04-30T08:35:19.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.237+0000 7f99bac00640 1 -- 172.21.15.32:0/4175217994 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99b41a2310 con 0x7f99b4104bd0 2024-04-30T08:35:19.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.241+0000 7f99c318a740 1 Processor -- start 2024-04-30T08:35:19.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.241+0000 7f99c318a740 1 -- start start 2024-04-30T08:35:19.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.242+0000 7f99c318a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559b2a7f8390 0x559b2ab9d850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:19.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.242+0000 7f99c318a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559b2a7cf4d0 0x559b2ab9ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:19.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.242+0000 7f99c318a740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2a7d76b0 0x559b2aba4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:19.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.242+0000 7f99c318a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x559b2aafed50 con 0x559b2a7cf4d0 2024-04-30T08:35:19.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.242+0000 7f99c318a740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x559b2aaa3ef0 con 0x559b2a7d76b0 2024-04-30T08:35:19.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.242+0000 7f99c318a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559b2aafad10 con 0x559b2a7f8390 2024-04-30T08:35:19.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.242+0000 7f99b2000640 1 -- 172.21.15.32:0/4175217994 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f99a0010430 con 0x7f99b4104bd0 2024-04-30T08:35:19.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.242+0000 7f99b9800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559b2a7cf4d0 0x559b2ab9ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:19.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.242+0000 7f99b9800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559b2a7cf4d0 0x559b2ab9ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39576/0 (socket says 172.21.15.32:39576) 2024-04-30T08:35:19.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.242+0000 7f99b9800640 1 -- 172.21.15.32:0/2534944362 learned_addr learned my addr 172.21.15.32:0/2534944362 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:19.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.242+0000 7f99ba200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559b2a7f8390 0x559b2ab9d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:19.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.242+0000 7f99b8e00640 1 --2- 172.21.15.32:0/2534944362 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2a7d76b0 0x559b2aba4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:19.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.243+0000 7f99b9800640 1 -- 172.21.15.32:0/2534944362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559b2a7f8390 msgr2=0x559b2ab9d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:19.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.243+0000 7f99b9800640 1 --2- 172.21.15.32:0/2534944362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559b2a7f8390 0x559b2ab9d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:19.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.243+0000 7f99b9800640 1 -- 172.21.15.32:0/2534944362 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2a7d76b0 msgr2=0x559b2aba4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:19.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.243+0000 7f99b9800640 1 --2- 172.21.15.32:0/2534944362 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2a7d76b0 0x559b2aba4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:19.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.243+0000 7f99b9800640 1 -- 172.21.15.32:0/2534944362 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b2a8da7d0 con 0x559b2a7cf4d0 2024-04-30T08:35:19.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.243+0000 7f99ba200640 1 --2- 172.21.15.32:0/2534944362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559b2a7f8390 0x559b2ab9d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:19.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.243+0000 7f99b8e00640 1 --2- 172.21.15.32:0/2534944362 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2a7d76b0 0x559b2aba4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:35:19.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.243+0000 7f99b9800640 1 --2- 172.21.15.32:0/2534944362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559b2a7cf4d0 0x559b2ab9ddd0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f99a8009900 tx=0x7f99a8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5d56ea7c755c2dd server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.243+0000 7f99a6a00640 1 -- 172.21.15.32:0/2534944362 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a8035030 con 0x559b2a7cf4d0 2024-04-30T08:35:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.243+0000 7f99a6a00640 1 -- 172.21.15.32:0/2534944362 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99a8040070 con 0x559b2a7cf4d0 2024-04-30T08:35:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.243+0000 7f99a6a00640 1 -- 172.21.15.32:0/2534944362 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a8031920 con 0x559b2a7cf4d0 2024-04-30T08:35:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.244+0000 7f99b2000640 1 --2- 172.21.15.32:0/4175217994 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f999c0afbf0 0x7f999c0b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.244+0000 7f99b2000640 1 -- 172.21.15.32:0/4175217994 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f99a00df420 con 0x7f99b4104bd0 2024-04-30T08:35:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.244+0000 7f99ba200640 1 --2- 172.21.15.32:0/4175217994 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f999c0afbf0 0x7f999c0b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.244+0000 7f99c318a740 1 -- 172.21.15.32:0/2534944362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559b2a7cf4d0 msgr2=0x559b2ab9ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.244+0000 7f99c318a740 1 --2- 172.21.15.32:0/2534944362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559b2a7cf4d0 0x559b2ab9ddd0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f99a8009900 tx=0x7f99a8004040 comp rx=0 tx=0).stop 2024-04-30T08:35:19.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.245+0000 7f99c318a740 1 -- 172.21.15.32:0/2534944362 shutdown_connections 2024-04-30T08:35:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.245+0000 7f99c318a740 1 --2- 172.21.15.32:0/2534944362 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2a7d76b0 0x559b2aba4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.245+0000 7f99c318a740 1 --2- 172.21.15.32:0/2534944362 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559b2a7cf4d0 0x559b2ab9ddd0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.245+0000 7f99c318a740 1 --2- 172.21.15.32:0/2534944362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559b2a7f8390 0x559b2ab9d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.245+0000 7f99c318a740 1 -- 172.21.15.32:0/2534944362 >> 172.21.15.32:0/2534944362 conn(0x559b2a7ec4e0 msgr2=0x559b2ab94dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.245+0000 7f99c318a740 1 -- 172.21.15.32:0/2534944362 shutdown_connections 2024-04-30T08:35:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.245+0000 7f99c318a740 1 -- 172.21.15.32:0/2534944362 wait complete. 2024-04-30T08:35:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.245+0000 7f99ba200640 1 --2- 172.21.15.32:0/4175217994 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f999c0afbf0 0x7f999c0b2100 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x559b2ab9ea20 tx=0x7f99ac009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.245+0000 7f99c318a740 1 -- start start 2024-04-30T08:35:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.246+0000 7f99c318a740 1 Processor -- start 2024-04-30T08:35:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.246+0000 7f99c318a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559b2a7cf4d0 0x559b2aba7710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:19.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.246+0000 7f99c318a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559b2a7d76b0 0x559b2aba7c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:19.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.246+0000 7f99c318a740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2a7f8390 0x559b2abaebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:19.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.246+0000 7f99c318a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x559b2aba56c0 con 0x559b2a7cf4d0 2024-04-30T08:35:19.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.246+0000 7f99c318a740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x559b2aab7990 con 0x559b2a7f8390 2024-04-30T08:35:19.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.246+0000 7f99c318a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559b2aba5840 con 0x559b2a7d76b0 2024-04-30T08:35:19.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.246+0000 7f99c318a740 10 client.0 _ll_register_callbacks cb 0x559b2aab6110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:35:19.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.247+0000 7f99b9800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559b2a7cf4d0 0x559b2aba7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:19.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.247+0000 7f99ba200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2a7f8390 0x559b2abaebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:19.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.247+0000 7f99ba200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2a7f8390 0x559b2abaebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:57086/0 (socket says 172.21.15.32:57086) 2024-04-30T08:35:19.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.247+0000 7f99b8e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559b2a7d76b0 0x559b2aba7c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:19.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.247+0000 7f99ba200640 1 -- 172.21.15.32:0/1754533739 learned_addr learned my addr 172.21.15.32:0/1754533739 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:19.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.247+0000 7f99b9800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559b2a7cf4d0 0x559b2aba7710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:39586/0 (socket says 172.21.15.32:39586) 2024-04-30T08:35:19.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.247+0000 7f99ba200640 1 -- 172.21.15.32:0/1754533739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559b2a7d76b0 msgr2=0x559b2aba7c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:19.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.247+0000 7f99ba200640 1 --2- 172.21.15.32:0/1754533739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559b2a7d76b0 0x559b2aba7c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:19.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.247+0000 7f99ba200640 1 -- 172.21.15.32:0/1754533739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559b2a7cf4d0 msgr2=0x559b2aba7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:19.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.247+0000 7f99ba200640 1 --2- 172.21.15.32:0/1754533739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559b2a7cf4d0 0x559b2aba7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:19.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.247+0000 7f99ba200640 1 -- 172.21.15.32:0/1754533739 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b2a8103f0 con 0x559b2a7f8390 2024-04-30T08:35:19.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.247+0000 7f99b8e00640 1 --2- 172.21.15.32:0/1754533739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559b2a7d76b0 0x559b2aba7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:19.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.247+0000 7f99b9800640 1 --2- 172.21.15.32:0/1754533739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559b2a7cf4d0 0x559b2aba7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:19.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.247+0000 7f99ba200640 1 --2- 172.21.15.32:0/1754533739 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2a7f8390 0x559b2abaebb0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f99ac011b40 tx=0x7f99ac004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a43b7b3d69448c48 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:19.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.247+0000 7f999b400640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:35:19.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.248+0000 7f999b400640 1 -- 172.21.15.32:0/1754533739 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99ac02d060 con 0x559b2a7f8390 2024-04-30T08:35:19.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.248+0000 7f99c318a740 1 -- 172.21.15.32:0/1754533739 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99a8040070 con 0x559b2a7f8390 2024-04-30T08:35:19.251 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.248+0000 7f999b400640 1 -- 172.21.15.32:0/1754533739 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f99ac017030 con 0x559b2a7f8390 2024-04-30T08:35:19.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.248+0000 7f999b400640 1 -- 172.21.15.32:0/1754533739 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f99ac034050 con 0x559b2a7f8390 2024-04-30T08:35:19.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.248+0000 7f99c318a740 10 client.25643 fetch_fsmap learned FSMap version 28 2024-04-30T08:35:19.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.248+0000 7f99c318a740 1 -- 172.21.15.32:0/1754533739 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559b2a7d00f0 con 0x559b2a7f8390 2024-04-30T08:35:19.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.248+0000 7f999b400640 1 -- 172.21.15.32:0/1754533739 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99a8040070 con 0x559b2a7f8390 2024-04-30T08:35:19.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.248+0000 7f999b400640 1 -- 172.21.15.32:0/1754533739 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f99ac00f790 con 0x559b2a7f8390 2024-04-30T08:35:19.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.249+0000 7f99c318a740 10 client.25643 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:35:19.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.249+0000 7f99c318a740 10 client.25643 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:35:19.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.249+0000 7f99c318a740 20 client.25643 populate_metadata read hostname 'smithi032' 2024-04-30T08:35:19.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.249+0000 7f99c318a740 1 --2- 172.21.15.32:0/1754533739 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x559b2a7b6f60 0x559b2abb5110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:19.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.249+0000 7f99c318a740 4 client.25643 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:35:19.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.249+0000 7f99c318a740 1 -- 172.21.15.32:0/1754533739 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559b2ab12ef0 con 0x559b2a7b6f60 2024-04-30T08:35:19.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.249+0000 7f99b9800640 1 --2- 172.21.15.32:0/1754533739 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x559b2a7b6f60 0x559b2abb5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:19.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.250+0000 7f99b9800640 1 --2- 172.21.15.32:0/1754533739 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x559b2a7b6f60 0x559b2abb5110 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6e310ec4f4f56c6 server_cookie=28c266e9c2dbe91a in_seq=0 out_seq=0 2024-04-30T08:35:19.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.250+0000 7f999b400640 10 client.25643 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:19.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.251+0000 7f999b400640 1 -- 172.21.15.32:0/1754533739 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x559b2ab12ef0 con 0x559b2a7b6f60 2024-04-30T08:35:19.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.251+0000 7f999b400640 10 client.25643 handle_command_reply: tid=0 2024-04-30T08:35:19.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.251+0000 7f99c318a740 1 client.25643 shutdown 2024-04-30T08:35:19.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.251+0000 7f99c318a740 1 -- 172.21.15.32:0/1754533739 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2a7f8390 msgr2=0x559b2abaebb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:19.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.251+0000 7f99c318a740 1 --2- 172.21.15.32:0/1754533739 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2a7f8390 0x559b2abaebb0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f99ac011b40 tx=0x7f99ac004ab0 comp rx=0 tx=0).stop 2024-04-30T08:35:19.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.251+0000 7f99c318a740 1 -- 172.21.15.32:0/1754533739 shutdown_connections 2024-04-30T08:35:19.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.251+0000 7f99c318a740 1 --2- 172.21.15.32:0/1754533739 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x559b2a7b6f60 0x559b2abb5110 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:19.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.251+0000 7f99c318a740 1 --2- 172.21.15.32:0/1754533739 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2a7f8390 0x559b2abaebb0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:19.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.251+0000 7f99c318a740 1 --2- 172.21.15.32:0/1754533739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559b2a7d76b0 0x559b2aba7c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:19.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.251+0000 7f99c318a740 1 --2- 172.21.15.32:0/1754533739 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559b2a7cf4d0 0x559b2aba7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:19.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.251+0000 7f999b400640 0 client.25643 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:19.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.251+0000 7f99c318a740 1 -- 172.21.15.32:0/1754533739 >> 172.21.15.32:0/1754533739 conn(0x559b2a7ec4e0 msgr2=0x559b2ab93e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:19.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.251+0000 7f99c318a740 1 -- 172.21.15.32:0/1754533739 shutdown_connections 2024-04-30T08:35:19.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.251+0000 7f99c318a740 1 -- 172.21.15.32:0/1754533739 wait complete. 2024-04-30T08:35:19.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.251+0000 7f99c318a740 20 client.25643 trim_cache size 0 max 16384 2024-04-30T08:35:19.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.272+0000 7f99c318a740 1 Processor -- start 2024-04-30T08:35:19.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.272+0000 7f99c318a740 1 -- start start 2024-04-30T08:35:19.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.272+0000 7f99c318a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559b2a7cf4d0 0x559b2aba2230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:19.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.272+0000 7f99c318a740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2a7d76b0 0x559b2abaead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:19.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.272+0000 7f99c318a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559b2a7f8390 0x559b2aba7710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:19.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.272+0000 7f99c318a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f99b41aa7d0 con 0x559b2a7cf4d0 2024-04-30T08:35:19.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.272+0000 7f99c318a740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f99b41ac310 con 0x559b2a7d76b0 2024-04-30T08:35:19.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.272+0000 7f99c318a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f99b41aaea0 con 0x559b2a7f8390 2024-04-30T08:35:19.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.273+0000 7f99b8e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2a7d76b0 0x559b2abaead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:19.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.273+0000 7f99ba200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559b2a7f8390 0x559b2aba7710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:19.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.273+0000 7f99b8e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2a7d76b0 0x559b2abaead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:57088/0 (socket says 172.21.15.32:57088) 2024-04-30T08:35:19.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.273+0000 7f99b8e00640 1 -- 172.21.15.32:0/3919311222 learned_addr learned my addr 172.21.15.32:0/3919311222 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:19.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.273+0000 7f99b9800640 1 --2- 172.21.15.32:0/3919311222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559b2a7cf4d0 0x559b2aba2230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:19.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.273+0000 7f99b8e00640 1 -- 172.21.15.32:0/3919311222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559b2a7f8390 msgr2=0x559b2aba7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:19.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.273+0000 7f99b8e00640 1 --2- 172.21.15.32:0/3919311222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559b2a7f8390 0x559b2aba7710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:19.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.273+0000 7f99b8e00640 1 -- 172.21.15.32:0/3919311222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559b2a7cf4d0 msgr2=0x559b2aba2230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:19.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.273+0000 7f99b8e00640 1 --2- 172.21.15.32:0/3919311222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559b2a7cf4d0 0x559b2aba2230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:19.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.273+0000 7f99b8e00640 1 -- 172.21.15.32:0/3919311222 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b2aba0c70 con 0x559b2a7d76b0 2024-04-30T08:35:19.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.273+0000 7f99b9800640 1 --2- 172.21.15.32:0/3919311222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559b2a7cf4d0 0x559b2aba2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:35:19.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.274+0000 7f99b8e00640 1 --2- 172.21.15.32:0/3919311222 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2a7d76b0 0x559b2abaead0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f99a00a7410 tx=0x7f99a001f000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a563f303f447230 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:19.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.274+0000 7f99a6000640 1 -- 172.21.15.32:0/3919311222 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a00a6f90 con 0x559b2a7d76b0 2024-04-30T08:35:19.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.274+0000 7f99a6000640 1 -- 172.21.15.32:0/3919311222 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99a00e2e20 con 0x559b2a7d76b0 2024-04-30T08:35:19.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.274+0000 7f99a6000640 1 -- 172.21.15.32:0/3919311222 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a0010bb0 con 0x559b2a7d76b0 2024-04-30T08:35:19.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.275+0000 7f99c318a740 1 -- 172.21.15.32:0/3919311222 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2a7d76b0 msgr2=0x559b2abaead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:19.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.275+0000 7f99c318a740 1 --2- 172.21.15.32:0/3919311222 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2a7d76b0 0x559b2abaead0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f99a00a7410 tx=0x7f99a001f000 comp rx=0 tx=0).stop 2024-04-30T08:35:19.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.275+0000 7f99c318a740 1 -- 172.21.15.32:0/3919311222 shutdown_connections 2024-04-30T08:35:19.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.275+0000 7f99c318a740 1 --2- 172.21.15.32:0/3919311222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559b2a7f8390 0x559b2aba7710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:19.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.275+0000 7f99c318a740 1 --2- 172.21.15.32:0/3919311222 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2a7d76b0 0x559b2abaead0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:19.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.275+0000 7f99c318a740 1 --2- 172.21.15.32:0/3919311222 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559b2a7cf4d0 0x559b2aba2230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:19.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.275+0000 7f99c318a740 1 -- 172.21.15.32:0/3919311222 >> 172.21.15.32:0/3919311222 conn(0x559b2a7ec4e0 msgr2=0x559b2ab933d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:19.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.276+0000 7f99c318a740 1 -- 172.21.15.32:0/3919311222 shutdown_connections 2024-04-30T08:35:19.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.276+0000 7f99c318a740 1 -- 172.21.15.32:0/3919311222 wait complete. 2024-04-30T08:35:19.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.276+0000 7f99c318a740 1 -- start start 2024-04-30T08:35:19.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.276+0000 7f99c318a740 1 Processor -- start 2024-04-30T08:35:19.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.277+0000 7f99c318a740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2a7cf4d0 0x559b2ababc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:19.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.277+0000 7f99c318a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559b2a7d76b0 0x559b2abac1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:19.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.277+0000 7f99c318a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559b2a7f8390 0x559b2abad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:19.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.277+0000 7f99c318a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f99b41abc40 con 0x559b2a7f8390 2024-04-30T08:35:19.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.277+0000 7f99c318a740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f99b41ab570 con 0x559b2a7cf4d0 2024-04-30T08:35:19.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.277+0000 7f99c318a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f99b41a8790 con 0x559b2a7d76b0 2024-04-30T08:35:19.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.277+0000 7f99c318a740 10 client.0 _ll_register_callbacks cb 0x7f99ac0119d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:35:19.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.277+0000 7f99b9800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2a7cf4d0 0x559b2ababc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:19.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.277+0000 7f99ba200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559b2a7f8390 0x559b2abad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:19.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.277+0000 7f99b8e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559b2a7d76b0 0x559b2abac1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:19.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.277+0000 7f99b9800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2a7cf4d0 0x559b2ababc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:57104/0 (socket says 172.21.15.32:57104) 2024-04-30T08:35:19.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.277+0000 7f99b9800640 1 -- 172.21.15.32:0/1271813367 learned_addr learned my addr 172.21.15.32:0/1271813367 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:19.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.278+0000 7f99b9800640 1 -- 172.21.15.32:0/1271813367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559b2a7d76b0 msgr2=0x559b2abac1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:19.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.278+0000 7f99b9800640 1 --2- 172.21.15.32:0/1271813367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559b2a7d76b0 0x559b2abac1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:19.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.278+0000 7f99b9800640 1 -- 172.21.15.32:0/1271813367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559b2a7f8390 msgr2=0x559b2abad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:19.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.278+0000 7f99b9800640 1 --2- 172.21.15.32:0/1271813367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559b2a7f8390 0x559b2abad0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:19.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.278+0000 7f99b9800640 1 -- 172.21.15.32:0/1271813367 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b2abad690 con 0x559b2a7cf4d0 2024-04-30T08:35:19.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.278+0000 7f99b9800640 1 --2- 172.21.15.32:0/1271813367 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2a7cf4d0 0x559b2ababc60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f99a8009900 tx=0x7f99a802ab70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3c89114ea37b2f3b server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:19.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.278+0000 7f999b400640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:35:19.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.278+0000 7f999b400640 1 -- 172.21.15.32:0/1271813367 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a8054060 con 0x559b2a7cf4d0 2024-04-30T08:35:19.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.278+0000 7f999b400640 1 -- 172.21.15.32:0/1271813367 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f99a8003a00 con 0x559b2a7cf4d0 2024-04-30T08:35:19.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.278+0000 7f99c318a740 1 -- 172.21.15.32:0/1271813367 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f99a00e2e20 con 0x559b2a7cf4d0 2024-04-30T08:35:19.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.279+0000 7f999b400640 1 -- 172.21.15.32:0/1271813367 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f99a805b050 con 0x559b2a7cf4d0 2024-04-30T08:35:19.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.279+0000 7f99c318a740 10 client.25649 fetch_fsmap learned FSMap version 28 2024-04-30T08:35:19.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.279+0000 7f99c318a740 1 -- 172.21.15.32:0/1271813367 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559b2abae980 con 0x559b2a7cf4d0 2024-04-30T08:35:19.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.279+0000 7f999b400640 1 -- 172.21.15.32:0/1271813367 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99a00e2e20 con 0x559b2a7cf4d0 2024-04-30T08:35:19.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.279+0000 7f999b400640 1 -- 172.21.15.32:0/1271813367 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f99a80315a0 con 0x559b2a7cf4d0 2024-04-30T08:35:19.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.279+0000 7f99c318a740 10 client.25649 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:35:19.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.279+0000 7f99c318a740 10 client.25649 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:35:19.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.279+0000 7f99c318a740 20 client.25649 populate_metadata read hostname 'smithi032' 2024-04-30T08:35:19.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.279+0000 7f99c318a740 1 --2- 172.21.15.32:0/1271813367 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x559b2a78c3a0 0x559b2ac1f490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:19.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.279+0000 7f99c318a740 4 client.25649 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-30T08:35:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.279+0000 7f99c318a740 1 -- 172.21.15.32:0/1271813367 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "scrub status"}) -- 0x559b2ab0c960 con 0x559b2a78c3a0 2024-04-30T08:35:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.280+0000 7f99b8e00640 1 --2- 172.21.15.32:0/1271813367 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x559b2a78c3a0 0x559b2ac1f490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.280+0000 7f99b8e00640 1 --2- 172.21.15.32:0/1271813367 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x559b2a78c3a0 0x559b2ac1f490 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f8817e6719210fd0 server_cookie=8a5c94eb5ef9e80b in_seq=0 out_seq=0 2024-04-30T08:35:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.280+0000 7f999b400640 10 client.25649 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.280+0000 7f999b400640 1 -- 172.21.15.32:0/1271813367 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x559b2ab0c960 con 0x559b2a78c3a0 2024-04-30T08:35:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.280+0000 7f999b400640 10 client.25649 handle_command_reply: tid=0 2024-04-30T08:35:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.280+0000 7f99c318a740 1 client.25649 shutdown 2024-04-30T08:35:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.281+0000 7f99c318a740 1 -- 172.21.15.32:0/1271813367 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2a7cf4d0 msgr2=0x559b2ababc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.281+0000 7f99c318a740 1 --2- 172.21.15.32:0/1271813367 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2a7cf4d0 0x559b2ababc60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f99a8009900 tx=0x7f99a802ab70 comp rx=0 tx=0).stop 2024-04-30T08:35:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.281+0000 7f99c318a740 1 -- 172.21.15.32:0/1271813367 shutdown_connections 2024-04-30T08:35:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.281+0000 7f99c318a740 1 --2- 172.21.15.32:0/1271813367 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x559b2a78c3a0 0x559b2ac1f490 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.281+0000 7f99c318a740 1 --2- 172.21.15.32:0/1271813367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559b2a7f8390 0x559b2abad0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:19.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.281+0000 7f99c318a740 1 --2- 172.21.15.32:0/1271813367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559b2a7d76b0 0x559b2abac1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:19.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.281+0000 7f99c318a740 1 --2- 172.21.15.32:0/1271813367 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559b2a7cf4d0 0x559b2ababc60 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:19.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.281+0000 7f99c318a740 1 -- 172.21.15.32:0/1271813367 >> 172.21.15.32:0/1271813367 conn(0x559b2a7ec4e0 msgr2=0x559b2ab933d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:19.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.281+0000 7f999b400640 0 client.25649 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:19.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.281+0000 7f99c318a740 1 -- 172.21.15.32:0/1271813367 shutdown_connections 2024-04-30T08:35:19.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.281+0000 7f99c318a740 1 -- 172.21.15.32:0/1271813367 wait complete. 2024-04-30T08:35:19.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.281+0000 7f99c318a740 20 client.25649 trim_cache size 0 max 16384 2024-04-30T08:35:19.285 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-30T08:35:19.285 INFO:teuthology.orchestra.run.smithi032.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-30T08:35:19.285 INFO:teuthology.orchestra.run.smithi032.stdout: "scrubs": { 2024-04-30T08:35:19.286 INFO:teuthology.orchestra.run.smithi032.stdout: "60505704-69d7-4e42-ad1d-0b74325a4a28": { 2024-04-30T08:35:19.286 INFO:teuthology.orchestra.run.smithi032.stdout: "path": "/", 2024-04-30T08:35:19.286 INFO:teuthology.orchestra.run.smithi032.stdout: "tag": "60505704-69d7-4e42-ad1d-0b74325a4a28", 2024-04-30T08:35:19.286 INFO:teuthology.orchestra.run.smithi032.stdout: "options": "recursive,force" 2024-04-30T08:35:19.286 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:35:19.286 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:35:19.286 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-30T08:35:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.282+0000 7f99bac00640 1 -- 172.21.15.32:0/4175217994 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f999c0afbf0 msgr2=0x7f999c0b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.282+0000 7f99bac00640 1 --2- 172.21.15.32:0/4175217994 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f999c0afbf0 0x7f999c0b2100 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x559b2ab9ea20 tx=0x7f99ac009500 comp rx=0 tx=0).stop 2024-04-30T08:35:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.282+0000 7f99bac00640 1 -- 172.21.15.32:0/4175217994 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99b4104bd0 msgr2=0x7f99b419a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.282+0000 7f99bac00640 1 --2- 172.21.15.32:0/4175217994 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99b4104bd0 0x7f99b419a610 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f99a000e9f0 tx=0x7f99a000eeb0 comp rx=0 tx=0).stop 2024-04-30T08:35:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.282+0000 7f99bac00640 1 -- 172.21.15.32:0/4175217994 shutdown_connections 2024-04-30T08:35:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.282+0000 7f99bac00640 1 --2- 172.21.15.32:0/4175217994 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f999c0afbf0 0x7f999c0b2100 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.282+0000 7f99bac00640 1 --2- 172.21.15.32:0/4175217994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99b41098d0 0x7f99b41a15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.282+0000 7f99bac00640 1 --2- 172.21.15.32:0/4175217994 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99b4104bd0 0x7f99b419a610 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.282+0000 7f99bac00640 1 --2- 172.21.15.32:0/4175217994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99b4103990 0x7f99b419a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.282+0000 7f99bac00640 1 -- 172.21.15.32:0/4175217994 >> 172.21.15.32:0/4175217994 conn(0x7f99b40feba0 msgr2=0x7f99b41058b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:19.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.282+0000 7f99bac00640 1 -- 172.21.15.32:0/4175217994 shutdown_connections 2024-04-30T08:35:19.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:19.282+0000 7f99bac00640 1 -- 172.21.15.32:0/4175217994 wait complete. 2024-04-30T08:35:19.311 INFO:tasks.cephfs.filesystem:scrub status for tag:60505704-69d7-4e42-ad1d-0b74325a4a28 - {'path': '/', 'tag': '60505704-69d7-4e42-ad1d-0b74325a4a28', 'options': 'recursive,force'} 2024-04-30T08:35:19.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:19 smithi032 ceph-mon[28331]: pgmap v429: 129 pgs: 129 active+clean; 653 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-04-30T08:35:19.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:19 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3667559219' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:35:19.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:19 smithi071 ceph-mon[33385]: pgmap v429: 129 pgs: 129 active+clean; 653 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-04-30T08:35:19.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:19 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3667559219' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:35:19.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:19 smithi189 ceph-mon[33038]: pgmap v429: 129 pgs: 129 active+clean; 653 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.4 KiB/s wr, 10 op/s 2024-04-30T08:35:19.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:19 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3667559219' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:35:20.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:20 smithi032 ceph-mon[28331]: scrub queued for path: / 2024-04-30T08:35:20.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:20 smithi032 ceph-mon[28331]: scrub summary: idle+waiting paths [/] 2024-04-30T08:35:20.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:20 smithi032 ceph-mon[28331]: scrub summary: active paths [/] 2024-04-30T08:35:20.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:20 smithi032 ceph-mon[28331]: scrub summary: idle+waiting paths [/] 2024-04-30T08:35:20.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:20 smithi071 ceph-mon[33385]: scrub queued for path: / 2024-04-30T08:35:20.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:20 smithi071 ceph-mon[33385]: scrub summary: idle+waiting paths [/] 2024-04-30T08:35:20.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:20 smithi071 ceph-mon[33385]: scrub summary: active paths [/] 2024-04-30T08:35:20.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:20 smithi071 ceph-mon[33385]: scrub summary: idle+waiting paths [/] 2024-04-30T08:35:20.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:20 smithi189 ceph-mon[33038]: scrub queued for path: / 2024-04-30T08:35:20.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:20 smithi189 ceph-mon[33038]: scrub summary: idle+waiting paths [/] 2024-04-30T08:35:20.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:20 smithi189 ceph-mon[33038]: scrub summary: active paths [/] 2024-04-30T08:35:20.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:20 smithi189 ceph-mon[33038]: scrub summary: idle+waiting paths [/] 2024-04-30T08:35:21.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:21 smithi032 ceph-mon[28331]: pgmap v430: 129 pgs: 129 active+clean; 660 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.1 KiB/s wr, 13 op/s 2024-04-30T08:35:21.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:21 smithi071 ceph-mon[33385]: pgmap v430: 129 pgs: 129 active+clean; 660 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.1 KiB/s wr, 13 op/s 2024-04-30T08:35:21.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:21 smithi189 ceph-mon[33038]: pgmap v430: 129 pgs: 129 active+clean; 660 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 2.1 KiB/s wr, 13 op/s 2024-04-30T08:35:23.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:23 smithi032 ceph-mon[28331]: pgmap v431: 129 pgs: 129 active+clean; 660 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-30T08:35:23.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:23 smithi071 ceph-mon[33385]: pgmap v431: 129 pgs: 129 active+clean; 660 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-30T08:35:23.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:23 smithi189 ceph-mon[33038]: pgmap v431: 129 pgs: 129 active+clean; 660 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-30T08:35:25.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:25 smithi032 ceph-mon[28331]: pgmap v432: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-30T08:35:25.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:25 smithi071 ceph-mon[33385]: pgmap v432: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-30T08:35:25.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:25 smithi189 ceph-mon[33038]: pgmap v432: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-30T08:35:27.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:27 smithi071 ceph-mon[33385]: pgmap v433: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.3 KiB/s wr, 12 op/s 2024-04-30T08:35:27.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:27 smithi189 ceph-mon[33038]: pgmap v433: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.3 KiB/s wr, 12 op/s 2024-04-30T08:35:27.640 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:27 smithi032 ceph-mon[28331]: pgmap v433: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 7.1 KiB/s rd, 1.3 KiB/s wr, 12 op/s 2024-04-30T08:35:29.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:29 smithi071 ceph-mon[33385]: pgmap v434: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-04-30T08:35:29.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:29 smithi189 ceph-mon[33038]: pgmap v434: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-04-30T08:35:29.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:29 smithi032 ceph-mon[28331]: pgmap v434: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.1 KiB/s wr, 9 op/s 2024-04-30T08:35:30.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:30 smithi071 ceph-mon[33385]: scrub summary: idle 2024-04-30T08:35:30.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:30 smithi189 ceph-mon[33038]: scrub summary: idle 2024-04-30T08:35:30.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:30 smithi032 ceph-mon[28331]: scrub summary: idle 2024-04-30T08:35:31.505 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:31 smithi071 ceph-mon[33385]: pgmap v435: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-04-30T08:35:31.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:31 smithi189 ceph-mon[33038]: pgmap v435: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-04-30T08:35:31.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:31 smithi032 ceph-mon[28331]: pgmap v435: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 1.1 KiB/s wr, 12 op/s 2024-04-30T08:35:31.862 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.861+0000 7fbf1b200640 1 -- 172.21.15.32:0/3185236406 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf1c075960 msgr2=0x7fbf1c10b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.861+0000 7fbf1b200640 1 --2- 172.21.15.32:0/3185236406 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf1c075960 0x7fbf1c10b400 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbf04009900 tx=0x7fbf0402a410 comp rx=0 tx=0).stop 2024-04-30T08:35:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.861+0000 7fbf1b200640 1 -- 172.21.15.32:0/3185236406 shutdown_connections 2024-04-30T08:35:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.861+0000 7fbf1b200640 1 --2- 172.21.15.32:0/3185236406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf1c10b940 0x7fbf1c10dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.861+0000 7fbf1b200640 1 --2- 172.21.15.32:0/3185236406 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf1c075960 0x7fbf1c10b400 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:31.863 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.861+0000 7fbf1b200640 1 --2- 172.21.15.32:0/3185236406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf1c075010 0x7fbf1c075420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:31.864 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.861+0000 7fbf1b200640 1 -- 172.21.15.32:0/3185236406 >> 172.21.15.32:0/3185236406 conn(0x7fbf1c0febc0 msgr2=0x7fbf1c101070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:31.864 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.861+0000 7fbf1b200640 1 -- 172.21.15.32:0/3185236406 shutdown_connections 2024-04-30T08:35:31.864 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.861+0000 7fbf1b200640 1 -- 172.21.15.32:0/3185236406 wait complete. 2024-04-30T08:35:31.867 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.863+0000 7fbf1b200640 1 Processor -- start 2024-04-30T08:35:31.867 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.863+0000 7fbf1b200640 1 -- start start 2024-04-30T08:35:31.867 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.864+0000 7fbf1b200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf1c075010 0x7fbf1c19c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.864+0000 7fbf1b200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf1c075960 0x7fbf1c19c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.864+0000 7fbf1b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf1c10b940 0x7fbf1c1a3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.864+0000 7fbf1b200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1c111720 con 0x7fbf1c10b940 2024-04-30T08:35:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.864+0000 7fbf1b200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1c1115a0 con 0x7fbf1c075010 2024-04-30T08:35:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.864+0000 7fbf1b200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1c1118a0 con 0x7fbf1c075960 2024-04-30T08:35:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.864+0000 7fbf1a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf1c10b940 0x7fbf1c1a3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.864+0000 7fbf19400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf1c075960 0x7fbf1c19c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.864+0000 7fbf1a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf1c10b940 0x7fbf1c1a3790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43260/0 (socket says 172.21.15.32:43260) 2024-04-30T08:35:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.864+0000 7fbf1a800640 1 -- 172.21.15.32:0/621369529 learned_addr learned my addr 172.21.15.32:0/621369529 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:31.868 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.864+0000 7fbf19e00640 1 --2- 172.21.15.32:0/621369529 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf1c075010 0x7fbf1c19c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.865+0000 7fbf19400640 1 -- 172.21.15.32:0/621369529 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf1c075010 msgr2=0x7fbf1c19c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.865+0000 7fbf19400640 1 --2- 172.21.15.32:0/621369529 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf1c075010 0x7fbf1c19c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.865+0000 7fbf19400640 1 -- 172.21.15.32:0/621369529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf1c10b940 msgr2=0x7fbf1c1a3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.865+0000 7fbf19400640 1 --2- 172.21.15.32:0/621369529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf1c10b940 0x7fbf1c1a3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.865+0000 7fbf19400640 1 -- 172.21.15.32:0/621369529 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf1c1a3e00 con 0x7fbf1c075960 2024-04-30T08:35:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.865+0000 7fbf19e00640 1 --2- 172.21.15.32:0/621369529 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf1c075010 0x7fbf1c19c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.865+0000 7fbf1a800640 1 --2- 172.21.15.32:0/621369529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf1c10b940 0x7fbf1c1a3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.865+0000 7fbf19400640 1 --2- 172.21.15.32:0/621369529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf1c075960 0x7fbf1c19c800 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbf04006010 tx=0x7fbf04004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:31.869 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.865+0000 7fbf12a00640 1 -- 172.21.15.32:0/621369529 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf0403f070 con 0x7fbf1c075960 2024-04-30T08:35:31.870 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.865+0000 7fbf12a00640 1 -- 172.21.15.32:0/621369529 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf04031910 con 0x7fbf1c075960 2024-04-30T08:35:31.870 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.865+0000 7fbf1b200640 1 -- 172.21.15.32:0/621369529 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf1c1a4090 con 0x7fbf1c075960 2024-04-30T08:35:31.870 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.865+0000 7fbf12a00640 1 -- 172.21.15.32:0/621369529 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf04035070 con 0x7fbf1c075960 2024-04-30T08:35:31.870 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.866+0000 7fbf1b200640 1 -- 172.21.15.32:0/621369529 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf1c1a4580 con 0x7fbf1c075960 2024-04-30T08:35:31.880 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.869+0000 7fbf27e20740 1 Processor -- start 2024-04-30T08:35:31.880 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.869+0000 7fbf27e20740 1 -- start start 2024-04-30T08:35:31.881 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.869+0000 7fbf27e20740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c1c1031b0 0x559c1c4a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:31.881 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.869+0000 7fbf27e20740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559c1c0da2f0 0x559c1c4a5860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:31.881 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.869+0000 7fbf27e20740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c1c0e24d0 0x559c1c4ac760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:31.881 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.869+0000 7fbf12a00640 1 -- 172.21.15.32:0/621369529 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7fbf040037f0 con 0x7fbf1c075960 2024-04-30T08:35:31.881 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.869+0000 7fbf27e20740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x559c1c409550 con 0x559c1c0e24d0 2024-04-30T08:35:31.881 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.869+0000 7fbf27e20740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x559c1c3aebc0 con 0x559c1c0da2f0 2024-04-30T08:35:31.882 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.869+0000 7fbf27e20740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559c1c405510 con 0x559c1c1031b0 2024-04-30T08:35:31.882 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.870+0000 7fbf19e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559c1c0da2f0 0x559c1c4a5860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:31.882 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.870+0000 7fbf19e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559c1c0da2f0 0x559c1c4a5860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:52414/0 (socket says 172.21.15.32:52414) 2024-04-30T08:35:31.882 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.870+0000 7fbf19400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c1c0e24d0 0x559c1c4ac760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:31.882 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.870+0000 7fbf19e00640 1 -- 172.21.15.32:0/4086120351 learned_addr learned my addr 172.21.15.32:0/4086120351 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:31.882 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.870+0000 7fbf1a800640 1 --2- 172.21.15.32:0/4086120351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c1c1031b0 0x559c1c4a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:31.883 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.870+0000 7fbf19e00640 1 -- 172.21.15.32:0/4086120351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c1c1031b0 msgr2=0x559c1c4a52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:31.883 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.870+0000 7fbf19e00640 1 --2- 172.21.15.32:0/4086120351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c1c1031b0 0x559c1c4a52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:31.883 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.870+0000 7fbf19e00640 1 -- 172.21.15.32:0/4086120351 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c1c0e24d0 msgr2=0x559c1c4ac760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:31.883 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.870+0000 7fbf19e00640 1 --2- 172.21.15.32:0/4086120351 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c1c0e24d0 0x559c1c4ac760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:31.883 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.870+0000 7fbf19e00640 1 -- 172.21.15.32:0/4086120351 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c1c11b210 con 0x559c1c0da2f0 2024-04-30T08:35:31.884 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.871+0000 7fbf19e00640 1 --2- 172.21.15.32:0/4086120351 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559c1c0da2f0 0x559c1c4a5860 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c0114f0 tx=0x7fbf0c0118f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bb82d2fba5cabe60 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:31.884 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.871+0000 7fbf0b400640 1 -- 172.21.15.32:0/4086120351 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c01e070 con 0x559c1c0da2f0 2024-04-30T08:35:31.884 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.871+0000 7fbf0b400640 1 -- 172.21.15.32:0/4086120351 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf0c0035f0 con 0x559c1c0da2f0 2024-04-30T08:35:31.884 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.871+0000 7fbf0b400640 1 -- 172.21.15.32:0/4086120351 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c00f740 con 0x559c1c0da2f0 2024-04-30T08:35:31.884 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.871+0000 7fbf12a00640 1 --2- 172.21.15.32:0/621369529 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbf000afb70 0x7fbf000b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:31.884 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.872+0000 7fbf12a00640 1 -- 172.21.15.32:0/621369529 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fbf040ff6f0 con 0x7fbf1c075960 2024-04-30T08:35:31.885 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.872+0000 7fbf1a800640 1 --2- 172.21.15.32:0/621369529 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbf000afb70 0x7fbf000b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:31.885 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.872+0000 7fbf27e20740 1 -- 172.21.15.32:0/4086120351 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559c1c0da2f0 msgr2=0x559c1c4a5860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:31.885 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.872+0000 7fbf27e20740 1 --2- 172.21.15.32:0/4086120351 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559c1c0da2f0 0x559c1c4a5860 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c0114f0 tx=0x7fbf0c0118f0 comp rx=0 tx=0).stop 2024-04-30T08:35:31.885 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.872+0000 7fbf27e20740 1 -- 172.21.15.32:0/4086120351 shutdown_connections 2024-04-30T08:35:31.885 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.872+0000 7fbf27e20740 1 --2- 172.21.15.32:0/4086120351 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c1c0e24d0 0x559c1c4ac760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:31.885 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.872+0000 7fbf27e20740 1 --2- 172.21.15.32:0/4086120351 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559c1c0da2f0 0x559c1c4a5860 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:31.886 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.872+0000 7fbf27e20740 1 --2- 172.21.15.32:0/4086120351 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c1c1031b0 0x559c1c4a52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:31.886 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.872+0000 7fbf27e20740 1 -- 172.21.15.32:0/4086120351 >> 172.21.15.32:0/4086120351 conn(0x559c1c0f7300 msgr2=0x559c1c4a0c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:31.886 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.872+0000 7fbf27e20740 1 -- 172.21.15.32:0/4086120351 shutdown_connections 2024-04-30T08:35:31.886 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.872+0000 7fbf27e20740 1 -- 172.21.15.32:0/4086120351 wait complete. 2024-04-30T08:35:31.886 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.872+0000 7fbf1a800640 1 --2- 172.21.15.32:0/621369529 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbf000afb70 0x7fbf000b2080 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x559c1c4a64b0 tx=0x7fbf14009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:31.886 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.873+0000 7fbf27e20740 1 -- start start 2024-04-30T08:35:31.886 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.873+0000 7fbf27e20740 1 Processor -- start 2024-04-30T08:35:31.887 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.874+0000 7fbf27e20740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c1c0da2f0 0x559c1c4aeda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:31.887 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.874+0000 7fbf27e20740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559c1c0e24d0 0x559c1c4af2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:31.887 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.874+0000 7fbf27e20740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c1c1031b0 0x559c1c4b6240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:31.887 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.874+0000 7fbf27e20740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x559c1c4ad1b0 con 0x559c1c0da2f0 2024-04-30T08:35:31.887 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.874+0000 7fbf27e20740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x559c1c3c2660 con 0x559c1c0e24d0 2024-04-30T08:35:31.887 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.874+0000 7fbf27e20740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559c1c4ad330 con 0x559c1c1031b0 2024-04-30T08:35:31.887 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.874+0000 7fbf27e20740 10 client.0 _ll_register_callbacks cb 0x559c1c38c400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:35:31.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.874+0000 7fbf19e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c1c0da2f0 0x559c1c4aeda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:31.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.874+0000 7fbf19400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559c1c0e24d0 0x559c1c4af2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:31.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.874+0000 7fbf19e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c1c0da2f0 0x559c1c4aeda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43280/0 (socket says 172.21.15.32:43280) 2024-04-30T08:35:31.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.874+0000 7fbf19e00640 1 -- 172.21.15.32:0/2743997477 learned_addr learned my addr 172.21.15.32:0/2743997477 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:31.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.874+0000 7fbf1a800640 1 --2- 172.21.15.32:0/2743997477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c1c1031b0 0x559c1c4b6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:31.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.874+0000 7fbf19400640 1 -- 172.21.15.32:0/2743997477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c1c1031b0 msgr2=0x559c1c4b6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:31.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.874+0000 7fbf19400640 1 --2- 172.21.15.32:0/2743997477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c1c1031b0 0x559c1c4b6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:31.888 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.874+0000 7fbf19400640 1 -- 172.21.15.32:0/2743997477 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c1c0da2f0 msgr2=0x559c1c4aeda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:31.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.874+0000 7fbf19400640 1 --2- 172.21.15.32:0/2743997477 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c1c0da2f0 0x559c1c4aeda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:31.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.874+0000 7fbf19400640 1 -- 172.21.15.32:0/2743997477 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c1c1e7620 con 0x559c1c0e24d0 2024-04-30T08:35:31.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.875+0000 7fbf19400640 1 --2- 172.21.15.32:0/2743997477 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559c1c0e24d0 0x559c1c4af2e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fbf040fcc90 tx=0x7fbf0403b670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2a547c897850d7b server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:31.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.875+0000 7fbeffe00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:35:31.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.875+0000 7fbeffe00640 1 -- 172.21.15.32:0/2743997477 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf04048070 con 0x559c1c0e24d0 2024-04-30T08:35:31.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.875+0000 7fbeffe00640 1 -- 172.21.15.32:0/2743997477 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fbf040fcee0 con 0x559c1c0e24d0 2024-04-30T08:35:31.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.875+0000 7fbf27e20740 1 -- 172.21.15.32:0/2743997477 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf0c0035f0 con 0x559c1c0e24d0 2024-04-30T08:35:31.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.875+0000 7fbeffe00640 1 -- 172.21.15.32:0/2743997477 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbf040c74e0 con 0x559c1c0e24d0 2024-04-30T08:35:31.890 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.875+0000 7fbf27e20740 10 client.25661 fetch_fsmap learned FSMap version 28 2024-04-30T08:35:31.890 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.875+0000 7fbf27e20740 1 -- 172.21.15.32:0/2743997477 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559c1c4b9170 con 0x559c1c0e24d0 2024-04-30T08:35:31.890 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.876+0000 7fbeffe00640 1 -- 172.21.15.32:0/2743997477 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf0c0035f0 con 0x559c1c0e24d0 2024-04-30T08:35:31.890 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.876+0000 7fbeffe00640 1 -- 172.21.15.32:0/2743997477 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fbf0404b060 con 0x559c1c0e24d0 2024-04-30T08:35:31.890 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.876+0000 7fbf27e20740 10 client.25661 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:35:31.890 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.876+0000 7fbf27e20740 10 client.25661 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-30T08:35:31.890 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.876+0000 7fbf27e20740 20 client.25661 populate_metadata read hostname 'smithi032' 2024-04-30T08:35:31.891 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.876+0000 7fbf27e20740 1 --2- 172.21.15.32:0/2743997477 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x559c1c0c1d80 0x559c1c4bc930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:31.891 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.876+0000 7fbf27e20740 4 client.25661 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:35:31.891 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.876+0000 7fbf27e20740 1 -- 172.21.15.32:0/2743997477 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559c1c417160 con 0x559c1c0c1d80 2024-04-30T08:35:31.891 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.876+0000 7fbf19e00640 1 --2- 172.21.15.32:0/2743997477 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x559c1c0c1d80 0x559c1c4bc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:31.891 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.877+0000 7fbf19e00640 1 --2- 172.21.15.32:0/2743997477 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x559c1c0c1d80 0x559c1c4bc930 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f180c9dab6995d server_cookie=e1e794a24e0396af in_seq=0 out_seq=0 2024-04-30T08:35:31.891 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.877+0000 7fbeffe00640 10 client.25661 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:31.891 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.878+0000 7fbeffe00640 1 -- 172.21.15.32:0/2743997477 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x559c1c417160 con 0x559c1c0c1d80 2024-04-30T08:35:31.892 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.878+0000 7fbeffe00640 10 client.25661 handle_command_reply: tid=0 2024-04-30T08:35:31.892 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.878+0000 7fbf27e20740 1 client.25661 shutdown 2024-04-30T08:35:31.892 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.878+0000 7fbf27e20740 1 -- 172.21.15.32:0/2743997477 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559c1c0e24d0 msgr2=0x559c1c4af2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:31.892 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.878+0000 7fbf27e20740 1 --2- 172.21.15.32:0/2743997477 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559c1c0e24d0 0x559c1c4af2e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fbf040fcc90 tx=0x7fbf0403b670 comp rx=0 tx=0).stop 2024-04-30T08:35:31.892 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.878+0000 7fbf27e20740 1 -- 172.21.15.32:0/2743997477 shutdown_connections 2024-04-30T08:35:31.892 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.878+0000 7fbf27e20740 1 --2- 172.21.15.32:0/2743997477 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x559c1c0c1d80 0x559c1c4bc930 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:31.892 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.878+0000 7fbf27e20740 1 --2- 172.21.15.32:0/2743997477 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c1c1031b0 0x559c1c4b6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:31.892 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.878+0000 7fbf27e20740 1 --2- 172.21.15.32:0/2743997477 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559c1c0e24d0 0x559c1c4af2e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:31.892 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.878+0000 7fbf27e20740 1 --2- 172.21.15.32:0/2743997477 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c1c0da2f0 0x559c1c4aeda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:31.892 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.878+0000 7fbeffe00640 0 client.25661 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:31.892 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.878+0000 7fbf27e20740 1 -- 172.21.15.32:0/2743997477 >> 172.21.15.32:0/2743997477 conn(0x559c1c0f7300 msgr2=0x559c1c4a3b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:31.893 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.878+0000 7fbf27e20740 1 -- 172.21.15.32:0/2743997477 shutdown_connections 2024-04-30T08:35:31.893 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.878+0000 7fbf27e20740 1 -- 172.21.15.32:0/2743997477 wait complete. 2024-04-30T08:35:31.893 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.878+0000 7fbf27e20740 20 client.25661 trim_cache size 0 max 16384 2024-04-30T08:35:31.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.896+0000 7fbf27e20740 1 Processor -- start 2024-04-30T08:35:31.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.896+0000 7fbf27e20740 1 -- start start 2024-04-30T08:35:31.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.896+0000 7fbf27e20740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c1c0da2f0 0x559c1c4a0340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:31.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.896+0000 7fbf27e20740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c1c0e24d0 0x559c1c4b6160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:31.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.896+0000 7fbf27e20740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559c1c1031b0 0x559c1c4a0880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:31.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.896+0000 7fbf27e20740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1c1aeaf0 con 0x559c1c0da2f0 2024-04-30T08:35:31.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.896+0000 7fbf27e20740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1c1b0630 con 0x559c1c1031b0 2024-04-30T08:35:31.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.896+0000 7fbf27e20740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1c1af1c0 con 0x559c1c0e24d0 2024-04-30T08:35:31.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.896+0000 7fbf19e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c1c0da2f0 0x559c1c4a0340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:31.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.896+0000 7fbf19e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c1c0da2f0 0x559c1c4a0340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:43290/0 (socket says 172.21.15.32:43290) 2024-04-30T08:35:31.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.896+0000 7fbf19e00640 1 -- 172.21.15.32:0/2201416530 learned_addr learned my addr 172.21.15.32:0/2201416530 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:31.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.896+0000 7fbf1a800640 1 --2- 172.21.15.32:0/2201416530 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559c1c1031b0 0x559c1c4a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:31.897 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.896+0000 7fbf19400640 1 --2- 172.21.15.32:0/2201416530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c1c0e24d0 0x559c1c4b6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:31.899 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.897+0000 7fbf19e00640 1 -- 172.21.15.32:0/2201416530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c1c0e24d0 msgr2=0x559c1c4b6160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:31.899 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.897+0000 7fbf19e00640 1 --2- 172.21.15.32:0/2201416530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c1c0e24d0 0x559c1c4b6160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:31.899 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.897+0000 7fbf19e00640 1 -- 172.21.15.32:0/2201416530 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559c1c1031b0 msgr2=0x559c1c4a0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:31.899 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.897+0000 7fbf19e00640 1 --2- 172.21.15.32:0/2201416530 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559c1c1031b0 0x559c1c4a0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:31.899 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.897+0000 7fbf19e00640 1 -- 172.21.15.32:0/2201416530 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559c1c49c480 con 0x559c1c0da2f0 2024-04-30T08:35:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.897+0000 7fbf1a800640 1 --2- 172.21.15.32:0/2201416530 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559c1c1031b0 0x559c1c4a0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.897+0000 7fbf19e00640 1 --2- 172.21.15.32:0/2201416530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c1c0da2f0 0x559c1c4a0340 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c0114f0 tx=0x7fbf0c005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c14982e47d3e1a4e server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.897+0000 7fbf0aa00640 1 -- 172.21.15.32:0/2201416530 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c031070 con 0x559c1c0da2f0 2024-04-30T08:35:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.898+0000 7fbf0aa00640 1 -- 172.21.15.32:0/2201416530 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf0c01f040 con 0x559c1c0da2f0 2024-04-30T08:35:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.898+0000 7fbf0aa00640 1 -- 172.21.15.32:0/2201416530 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c01e050 con 0x559c1c0da2f0 2024-04-30T08:35:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.898+0000 7fbf27e20740 1 -- 172.21.15.32:0/2201416530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c1c0da2f0 msgr2=0x559c1c4a0340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.898+0000 7fbf27e20740 1 --2- 172.21.15.32:0/2201416530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c1c0da2f0 0x559c1c4a0340 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c0114f0 tx=0x7fbf0c005d90 comp rx=0 tx=0).stop 2024-04-30T08:35:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.898+0000 7fbf27e20740 1 -- 172.21.15.32:0/2201416530 shutdown_connections 2024-04-30T08:35:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.898+0000 7fbf27e20740 1 --2- 172.21.15.32:0/2201416530 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559c1c1031b0 0x559c1c4a0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.898+0000 7fbf27e20740 1 --2- 172.21.15.32:0/2201416530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c1c0e24d0 0x559c1c4b6160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.898+0000 7fbf27e20740 1 --2- 172.21.15.32:0/2201416530 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c1c0da2f0 0x559c1c4a0340 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.898+0000 7fbf27e20740 1 -- 172.21.15.32:0/2201416530 >> 172.21.15.32:0/2201416530 conn(0x559c1c0f7300 msgr2=0x559c1c4a5260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:31.900 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.899+0000 7fbf27e20740 1 -- 172.21.15.32:0/2201416530 shutdown_connections 2024-04-30T08:35:31.901 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.899+0000 7fbf27e20740 1 -- 172.21.15.32:0/2201416530 wait complete. 2024-04-30T08:35:31.901 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.899+0000 7fbf27e20740 1 -- start start 2024-04-30T08:35:31.903 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.899+0000 7fbf27e20740 1 Processor -- start 2024-04-30T08:35:31.903 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.900+0000 7fbf27e20740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559c1c0da2f0 0x559c1c4b3b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:31.904 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.900+0000 7fbf27e20740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c1c0e24d0 0x559c1c4b40a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:31.904 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.900+0000 7fbf27e20740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c1c1031b0 0x559c1c4b4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:31.904 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.900+0000 7fbf27e20740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1c1aff60 con 0x559c1c1031b0 2024-04-30T08:35:31.904 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.900+0000 7fbf27e20740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1c1af890 con 0x559c1c0da2f0 2024-04-30T08:35:31.904 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.900+0000 7fbf27e20740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1c1acab0 con 0x559c1c0e24d0 2024-04-30T08:35:31.904 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.900+0000 7fbf27e20740 10 client.0 _ll_register_callbacks cb 0x7fbf04005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:35:31.904 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.900+0000 7fbf19e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559c1c0da2f0 0x559c1c4b3b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:31.904 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.900+0000 7fbf19e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559c1c0da2f0 0x559c1c4b3b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:52436/0 (socket says 172.21.15.32:52436) 2024-04-30T08:35:31.904 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.900+0000 7fbf1a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c1c1031b0 0x559c1c4b4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:31.904 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.900+0000 7fbf19e00640 1 -- 172.21.15.32:0/4037417033 learned_addr learned my addr 172.21.15.32:0/4037417033 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:31.904 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.900+0000 7fbf19400640 1 --2- 172.21.15.32:0/4037417033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c1c0e24d0 0x559c1c4b40a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:31.904 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.901+0000 7fbf19e00640 1 -- 172.21.15.32:0/4037417033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c1c0e24d0 msgr2=0x559c1c4b40a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:31.904 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.901+0000 7fbf19e00640 1 --2- 172.21.15.32:0/4037417033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c1c0e24d0 0x559c1c4b40a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:31.904 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.901+0000 7fbf19e00640 1 -- 172.21.15.32:0/4037417033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c1c1031b0 msgr2=0x559c1c4b4fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:31.905 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.901+0000 7fbf19e00640 1 --2- 172.21.15.32:0/4037417033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c1c1031b0 0x559c1c4b4fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:31.905 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.901+0000 7fbf19e00640 1 -- 172.21.15.32:0/4037417033 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559c1c4a9bf0 con 0x559c1c0da2f0 2024-04-30T08:35:31.905 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.901+0000 7fbf19e00640 1 --2- 172.21.15.32:0/4037417033 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559c1c0da2f0 0x559c1c4b3b60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c005e00 tx=0x7fbf0c005700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65e80a9d32831928 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:31.905 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.901+0000 7fbeffe00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:35:31.905 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.901+0000 7fbeffe00640 1 -- 172.21.15.32:0/4037417033 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c039060 con 0x559c1c0da2f0 2024-04-30T08:35:31.905 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.901+0000 7fbf27e20740 1 -- 172.21.15.32:0/4037417033 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf0c01f040 con 0x559c1c0da2f0 2024-04-30T08:35:31.905 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.901+0000 7fbeffe00640 1 -- 172.21.15.32:0/4037417033 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fbf0c005040 con 0x559c1c0da2f0 2024-04-30T08:35:31.905 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.902+0000 7fbeffe00640 1 -- 172.21.15.32:0/4037417033 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fbf0c01e020 con 0x559c1c0da2f0 2024-04-30T08:35:31.905 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.902+0000 7fbf27e20740 10 client.25667 fetch_fsmap learned FSMap version 28 2024-04-30T08:35:31.905 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.902+0000 7fbf27e20740 1 -- 172.21.15.32:0/4037417033 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x559c1c524cb0 con 0x559c1c0da2f0 2024-04-30T08:35:31.905 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.902+0000 7fbeffe00640 1 -- 172.21.15.32:0/4037417033 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbf0c01f040 con 0x559c1c0da2f0 2024-04-30T08:35:31.905 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.902+0000 7fbeffe00640 1 -- 172.21.15.32:0/4037417033 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fbf0c00f4e0 con 0x559c1c0da2f0 2024-04-30T08:35:31.905 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.902+0000 7fbf27e20740 10 client.25667 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:35:31.905 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.902+0000 7fbf27e20740 10 client.25667 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-30T08:35:31.905 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.902+0000 7fbf27e20740 20 client.25667 populate_metadata read hostname 'smithi032' 2024-04-30T08:35:31.906 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.902+0000 7fbf27e20740 1 --2- 172.21.15.32:0/4037417033 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x559c1c0971c0 0x559c1c527480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:31.906 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.902+0000 7fbf27e20740 4 client.25667 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-30T08:35:31.906 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.902+0000 7fbf27e20740 1 -- 172.21.15.32:0/4037417033 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559c1c410fd0 con 0x559c1c0971c0 2024-04-30T08:35:31.906 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.902+0000 7fbf19400640 1 --2- 172.21.15.32:0/4037417033 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x559c1c0971c0 0x559c1c527480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:31.906 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.903+0000 7fbf19400640 1 --2- 172.21.15.32:0/4037417033 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x559c1c0971c0 0x559c1c527480 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3209e416869428a1 server_cookie=ad477f092caf1288 in_seq=0 out_seq=0 2024-04-30T08:35:31.906 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.903+0000 7fbeffe00640 10 client.25667 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:31.918 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.917+0000 7fbeffe00640 1 -- 172.21.15.32:0/4037417033 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559c1c410fd0 con 0x559c1c0971c0 2024-04-30T08:35:31.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.917+0000 7fbeffe00640 10 client.25667 handle_command_reply: tid=0 2024-04-30T08:35:31.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.917+0000 7fbf27e20740 1 client.25667 shutdown 2024-04-30T08:35:31.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.917+0000 7fbf27e20740 1 -- 172.21.15.32:0/4037417033 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559c1c0da2f0 msgr2=0x559c1c4b3b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:31.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.917+0000 7fbf27e20740 1 --2- 172.21.15.32:0/4037417033 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559c1c0da2f0 0x559c1c4b3b60 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c005e00 tx=0x7fbf0c005700 comp rx=0 tx=0).stop 2024-04-30T08:35:31.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.917+0000 7fbf27e20740 1 -- 172.21.15.32:0/4037417033 shutdown_connections 2024-04-30T08:35:31.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.917+0000 7fbf27e20740 1 --2- 172.21.15.32:0/4037417033 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x559c1c0971c0 0x559c1c527480 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:31.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.917+0000 7fbf27e20740 1 --2- 172.21.15.32:0/4037417033 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559c1c1031b0 0x559c1c4b4fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:31.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.917+0000 7fbf27e20740 1 --2- 172.21.15.32:0/4037417033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559c1c0e24d0 0x559c1c4b40a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:31.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.917+0000 7fbf27e20740 1 --2- 172.21.15.32:0/4037417033 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559c1c0da2f0 0x559c1c4b3b60 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:31.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.917+0000 7fbeffe00640 0 client.25667 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:31.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.917+0000 7fbf27e20740 1 -- 172.21.15.32:0/4037417033 >> 172.21.15.32:0/4037417033 conn(0x559c1c0f7300 msgr2=0x559c1c49f230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:31.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.917+0000 7fbf27e20740 1 -- 172.21.15.32:0/4037417033 shutdown_connections 2024-04-30T08:35:31.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.917+0000 7fbf27e20740 1 -- 172.21.15.32:0/4037417033 wait complete. 2024-04-30T08:35:31.921 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.918+0000 7fbf27e20740 20 client.25667 trim_cache size 0 max 16384 2024-04-30T08:35:31.921 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-30T08:35:31.921 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-30T08:35:31.921 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-30T08:35:31.922 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-30T08:35:31.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.918+0000 7fbf1b200640 1 -- 172.21.15.32:0/621369529 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbf000afb70 msgr2=0x7fbf000b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:31.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.918+0000 7fbf1b200640 1 --2- 172.21.15.32:0/621369529 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbf000afb70 0x7fbf000b2080 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x559c1c4a64b0 tx=0x7fbf14009500 comp rx=0 tx=0).stop 2024-04-30T08:35:31.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.918+0000 7fbf1b200640 1 -- 172.21.15.32:0/621369529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf1c075960 msgr2=0x7fbf1c19c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:31.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.918+0000 7fbf1b200640 1 --2- 172.21.15.32:0/621369529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf1c075960 0x7fbf1c19c800 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbf04006010 tx=0x7fbf04004270 comp rx=0 tx=0).stop 2024-04-30T08:35:31.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.918+0000 7fbf1b200640 1 -- 172.21.15.32:0/621369529 shutdown_connections 2024-04-30T08:35:31.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.918+0000 7fbf1b200640 1 --2- 172.21.15.32:0/621369529 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbf000afb70 0x7fbf000b2080 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:31.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.918+0000 7fbf1b200640 1 --2- 172.21.15.32:0/621369529 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbf1c10b940 0x7fbf1c1a3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:31.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.918+0000 7fbf1b200640 1 --2- 172.21.15.32:0/621369529 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbf1c075960 0x7fbf1c19c800 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:31.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.918+0000 7fbf1b200640 1 --2- 172.21.15.32:0/621369529 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbf1c075010 0x7fbf1c19c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:31.924 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.918+0000 7fbf1b200640 1 -- 172.21.15.32:0/621369529 >> 172.21.15.32:0/621369529 conn(0x7fbf1c0febc0 msgr2=0x7fbf1c0ffa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:31.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.919+0000 7fbf1b200640 1 -- 172.21.15.32:0/621369529 shutdown_connections 2024-04-30T08:35:31.925 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:31.919+0000 7fbf1b200640 1 -- 172.21.15.32:0/621369529 wait complete. 2024-04-30T08:35:33.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:33 smithi071 ceph-mon[33385]: pgmap v436: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 341 B/s wr, 9 op/s 2024-04-30T08:35:33.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:33 smithi189 ceph-mon[33038]: pgmap v436: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 341 B/s wr, 9 op/s 2024-04-30T08:35:33.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:33 smithi032 ceph-mon[28331]: pgmap v436: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 341 B/s wr, 9 op/s 2024-04-30T08:35:35.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:35 smithi071 ceph-mon[33385]: pgmap v437: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 596 B/s wr, 11 op/s 2024-04-30T08:35:35.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:35 smithi189 ceph-mon[33038]: pgmap v437: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 596 B/s wr, 11 op/s 2024-04-30T08:35:35.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:35 smithi032 ceph-mon[28331]: pgmap v437: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 596 B/s wr, 11 op/s 2024-04-30T08:35:37.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:37 smithi071 ceph-mon[33385]: pgmap v438: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-04-30T08:35:37.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:37 smithi189 ceph-mon[33038]: pgmap v438: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-04-30T08:35:37.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:37 smithi032 ceph-mon[28331]: pgmap v438: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-04-30T08:35:39.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:39 smithi032 ceph-mon[28331]: pgmap v439: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-04-30T08:35:39.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:39 smithi071 ceph-mon[33385]: pgmap v439: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-04-30T08:35:39.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:39 smithi189 ceph-mon[33038]: pgmap v439: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-04-30T08:35:41.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:41 smithi071 ceph-mon[33385]: pgmap v440: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.5 KiB/s wr, 16 op/s 2024-04-30T08:35:41.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:41 smithi189 ceph-mon[33038]: pgmap v440: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.5 KiB/s wr, 16 op/s 2024-04-30T08:35:41.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:41 smithi032 ceph-mon[28331]: pgmap v440: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.5 KiB/s wr, 16 op/s 2024-04-30T08:35:43.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:43 smithi071 ceph-mon[33385]: pgmap v441: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-04-30T08:35:43.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:43 smithi189 ceph-mon[33038]: pgmap v441: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-04-30T08:35:43.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:43 smithi032 ceph-mon[28331]: pgmap v441: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-04-30T08:35:45.147 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.145+0000 7fde46800640 1 -- 172.21.15.32:0/248839252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde40104cb0 msgr2=0x7fde401050c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:45.147 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.145+0000 7fde46800640 1 --2- 172.21.15.32:0/248839252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde40104cb0 0x7fde401050c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fde2c009900 tx=0x7fde2c02a3f0 comp rx=0 tx=0).stop 2024-04-30T08:35:45.147 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.145+0000 7fde46800640 1 -- 172.21.15.32:0/248839252 shutdown_connections 2024-04-30T08:35:45.147 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.145+0000 7fde46800640 1 --2- 172.21.15.32:0/248839252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde401099b0 0x7fde4010bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:45.147 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.145+0000 7fde46800640 1 --2- 172.21.15.32:0/248839252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde40104cb0 0x7fde401050c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:45.147 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.145+0000 7fde46800640 1 --2- 172.21.15.32:0/248839252 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde40103a70 0x7fde40104770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:45.147 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.145+0000 7fde46800640 1 -- 172.21.15.32:0/248839252 >> 172.21.15.32:0/248839252 conn(0x7fde400fec80 msgr2=0x7fde40101110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:45.147 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.145+0000 7fde46800640 1 -- 172.21.15.32:0/248839252 shutdown_connections 2024-04-30T08:35:45.148 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.145+0000 7fde46800640 1 -- 172.21.15.32:0/248839252 wait complete. 2024-04-30T08:35:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.146+0000 7fde46800640 1 Processor -- start 2024-04-30T08:35:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.146+0000 7fde46800640 1 -- start start 2024-04-30T08:35:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.147+0000 7fde46800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde40103a70 0x7fde400779e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.147+0000 7fde46800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde40104cb0 0x7fde40076080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.147+0000 7fde46800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde401099b0 0x7fde400765c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.147+0000 7fde46800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fde4010f5c0 con 0x7fde401099b0 2024-04-30T08:35:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.147+0000 7fde46800640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fde4010f440 con 0x7fde40104cb0 2024-04-30T08:35:45.150 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.147+0000 7fde46800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fde4010f740 con 0x7fde40103a70 2024-04-30T08:35:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.147+0000 7fde45400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde40103a70 0x7fde400779e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.147+0000 7fde45400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde40103a70 0x7fde400779e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:56466/0 (socket says 172.21.15.32:56466) 2024-04-30T08:35:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.147+0000 7fde45400640 1 -- 172.21.15.32:0/1441005398 learned_addr learned my addr 172.21.15.32:0/1441005398 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.147+0000 7fde44a00640 1 --2- 172.21.15.32:0/1441005398 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde40104cb0 0x7fde40076080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.147+0000 7fde45e00640 1 --2- 172.21.15.32:0/1441005398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde401099b0 0x7fde400765c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.148+0000 7fde45400640 1 -- 172.21.15.32:0/1441005398 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde40104cb0 msgr2=0x7fde40076080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.148+0000 7fde45400640 1 --2- 172.21.15.32:0/1441005398 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde40104cb0 0x7fde40076080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.148+0000 7fde45400640 1 -- 172.21.15.32:0/1441005398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde401099b0 msgr2=0x7fde400765c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.148+0000 7fde45400640 1 --2- 172.21.15.32:0/1441005398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde401099b0 0x7fde400765c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.148+0000 7fde45400640 1 -- 172.21.15.32:0/1441005398 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde40076cf0 con 0x7fde40103a70 2024-04-30T08:35:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.148+0000 7fde45e00640 1 --2- 172.21.15.32:0/1441005398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde401099b0 0x7fde400765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:35:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.148+0000 7fde45400640 1 --2- 172.21.15.32:0/1441005398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde40103a70 0x7fde400779e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fde3400e9f0 tx=0x7fde3400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.148+0000 7fde3e000640 1 -- 172.21.15.32:0/1441005398 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3400ccc0 con 0x7fde40103a70 2024-04-30T08:35:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.148+0000 7fde3e000640 1 -- 172.21.15.32:0/1441005398 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde340035f0 con 0x7fde40103a70 2024-04-30T08:35:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.148+0000 7fde3e000640 1 -- 172.21.15.32:0/1441005398 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde3400ce60 con 0x7fde40103a70 2024-04-30T08:35:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.148+0000 7fde46800640 1 -- 172.21.15.32:0/1441005398 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde401a4200 con 0x7fde40103a70 2024-04-30T08:35:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.149+0000 7fde46800640 1 -- 172.21.15.32:0/1441005398 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde401a46c0 con 0x7fde40103a70 2024-04-30T08:35:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.153+0000 7fde4edce740 1 Processor -- start 2024-04-30T08:35:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.153+0000 7fde4edce740 1 -- start start 2024-04-30T08:35:45.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.153+0000 7fde4edce740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55be30fba1b0 0x55be3135c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:45.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.154+0000 7fde4edce740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be30f912f0 0x55be3135c860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:45.155 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.154+0000 7fde4edce740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be30f994d0 0x55be31363760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.154+0000 7fde4edce740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55be312c0550 con 0x55be30f912f0 2024-04-30T08:35:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.154+0000 7fde4edce740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55be31265bc0 con 0x55be30fba1b0 2024-04-30T08:35:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.154+0000 7fde4edce740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55be312bc510 con 0x55be30f994d0 2024-04-30T08:35:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.154+0000 7fde45e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55be30fba1b0 0x55be3135c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.154+0000 7fde45e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55be30fba1b0 0x55be3135c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:45306/0 (socket says 172.21.15.32:45306) 2024-04-30T08:35:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.154+0000 7fde44a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be30f994d0 0x55be31363760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.154+0000 7fde45e00640 1 -- 172.21.15.32:0/3854213364 learned_addr learned my addr 172.21.15.32:0/3854213364 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:45.156 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.154+0000 7fde3e000640 1 -- 172.21.15.32:0/1441005398 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7fde34003aa0 con 0x7fde40103a70 2024-04-30T08:35:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.154+0000 7fde45e00640 1 -- 172.21.15.32:0/3854213364 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be30f994d0 msgr2=0x55be31363760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.154+0000 7fde45e00640 1 --2- 172.21.15.32:0/3854213364 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be30f994d0 0x55be31363760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.154+0000 7fde45e00640 1 -- 172.21.15.32:0/3854213364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be30f912f0 msgr2=0x55be3135c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.154+0000 7fde45e00640 1 --2- 172.21.15.32:0/3854213364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be30f912f0 0x55be3135c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.154+0000 7fde45e00640 1 -- 172.21.15.32:0/3854213364 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be30fd2210 con 0x55be30fba1b0 2024-04-30T08:35:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.155+0000 7fde45e00640 1 --2- 172.21.15.32:0/3854213364 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55be30fba1b0 0x55be3135c2e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fde3800f410 tx=0x7fde3800f8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7092723ffacd8237 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.155+0000 7fde32600640 1 -- 172.21.15.32:0/3854213364 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde38015070 con 0x55be30fba1b0 2024-04-30T08:35:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.155+0000 7fde32600640 1 -- 172.21.15.32:0/3854213364 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde38011bb0 con 0x55be30fba1b0 2024-04-30T08:35:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.155+0000 7fde32600640 1 -- 172.21.15.32:0/3854213364 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde38010c70 con 0x55be30fba1b0 2024-04-30T08:35:45.158 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.156+0000 7fde3e000640 1 --2- 172.21.15.32:0/1441005398 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fde280afbf0 0x7fde280b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.156+0000 7fde4edce740 1 -- 172.21.15.32:0/3854213364 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55be30fba1b0 msgr2=0x55be3135c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.156+0000 7fde4edce740 1 --2- 172.21.15.32:0/3854213364 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55be30fba1b0 0x55be3135c2e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fde3800f410 tx=0x7fde3800f8d0 comp rx=0 tx=0).stop 2024-04-30T08:35:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.156+0000 7fde3e000640 1 -- 172.21.15.32:0/1441005398 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fde340df530 con 0x7fde40103a70 2024-04-30T08:35:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.156+0000 7fde4edce740 1 -- 172.21.15.32:0/3854213364 shutdown_connections 2024-04-30T08:35:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.156+0000 7fde4edce740 1 --2- 172.21.15.32:0/3854213364 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be30f994d0 0x55be31363760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.156+0000 7fde4edce740 1 --2- 172.21.15.32:0/3854213364 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be30f912f0 0x55be3135c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.156+0000 7fde4edce740 1 --2- 172.21.15.32:0/3854213364 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55be30fba1b0 0x55be3135c2e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.156+0000 7fde45400640 1 --2- 172.21.15.32:0/1441005398 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fde280afbf0 0x7fde280b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.156+0000 7fde4edce740 1 -- 172.21.15.32:0/3854213364 >> 172.21.15.32:0/3854213364 conn(0x55be30fae300 msgr2=0x55be31357c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.156+0000 7fde4edce740 1 -- 172.21.15.32:0/3854213364 shutdown_connections 2024-04-30T08:35:45.159 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.156+0000 7fde4edce740 1 -- 172.21.15.32:0/3854213364 wait complete. 2024-04-30T08:35:45.162 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.157+0000 7fde45400640 1 --2- 172.21.15.32:0/1441005398 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fde280afbf0 0x7fde280b2100 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fde340a3520 tx=0x7fde340a3430 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:45.162 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.157+0000 7fde4edce740 1 -- start start 2024-04-30T08:35:45.162 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.157+0000 7fde4edce740 1 Processor -- start 2024-04-30T08:35:45.162 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.158+0000 7fde4edce740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be30f912f0 0x55be31365d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:45.162 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.158+0000 7fde4edce740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be30f994d0 0x55be313662b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:45.162 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.158+0000 7fde4edce740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55be30fba1b0 0x55be3136d210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:45.162 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.158+0000 7fde4edce740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55be313641b0 con 0x55be30f994d0 2024-04-30T08:35:45.162 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.158+0000 7fde4edce740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55be31279660 con 0x55be30fba1b0 2024-04-30T08:35:45.163 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.158+0000 7fde4edce740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55be31364330 con 0x55be30f912f0 2024-04-30T08:35:45.163 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.158+0000 7fde4edce740 10 client.0 _ll_register_callbacks cb 0x55be31243400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:35:45.163 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.158+0000 7fde44a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be30f994d0 0x55be313662b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:45.163 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.158+0000 7fde44a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be30f994d0 0x55be313662b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37484/0 (socket says 172.21.15.32:37484) 2024-04-30T08:35:45.163 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.158+0000 7fde44a00640 1 -- 172.21.15.32:0/3812755489 learned_addr learned my addr 172.21.15.32:0/3812755489 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:45.163 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.158+0000 7fde45e00640 1 --2- 172.21.15.32:0/3812755489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55be30fba1b0 0x55be3136d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:45.163 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.158+0000 7fde45e00640 1 --2- 172.21.15.32:0/3812755489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be30f912f0 0x55be31365d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:45.163 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.159+0000 7fde44a00640 1 -- 172.21.15.32:0/3812755489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be30f912f0 msgr2=0x55be31365d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:45.163 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.159+0000 7fde44a00640 1 --2- 172.21.15.32:0/3812755489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be30f912f0 0x55be31365d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:45.163 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.159+0000 7fde44a00640 1 -- 172.21.15.32:0/3812755489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55be30fba1b0 msgr2=0x55be3136d210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:45.163 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.159+0000 7fde44a00640 1 --2- 172.21.15.32:0/3812755489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55be30fba1b0 0x55be3136d210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:45.163 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.159+0000 7fde44a00640 1 -- 172.21.15.32:0/3812755489 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be3109e620 con 0x55be30f994d0 2024-04-30T08:35:45.163 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.159+0000 7fde45e00640 1 --2- 172.21.15.32:0/3812755489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55be30fba1b0 0x55be3136d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:45.164 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.159+0000 7fde44a00640 1 --2- 172.21.15.32:0/3812755489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be30f994d0 0x55be313662b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fde2c000c00 tx=0x7fde2c004510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2b33ea500decf1e server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:45.164 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.159+0000 7fde26a00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:35:45.164 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.159+0000 7fde26a00640 1 -- 172.21.15.32:0/3812755489 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2c035040 con 0x55be30f994d0 2024-04-30T08:35:45.164 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.159+0000 7fde26a00640 1 -- 172.21.15.32:0/3812755489 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fde2c003ac0 con 0x55be30f994d0 2024-04-30T08:35:45.164 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.159+0000 7fde4edce740 1 -- 172.21.15.32:0/3812755489 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde38011bb0 con 0x55be30f994d0 2024-04-30T08:35:45.164 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.160+0000 7fde26a00640 1 -- 172.21.15.32:0/3812755489 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fde2c04e050 con 0x55be30f994d0 2024-04-30T08:35:45.164 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.160+0000 7fde4edce740 10 client.15555 fetch_fsmap learned FSMap version 28 2024-04-30T08:35:45.164 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.160+0000 7fde4edce740 1 -- 172.21.15.32:0/3812755489 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55be313701b0 con 0x55be30f994d0 2024-04-30T08:35:45.164 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.160+0000 7fde26a00640 1 -- 172.21.15.32:0/3812755489 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde38011bb0 con 0x55be30f994d0 2024-04-30T08:35:45.164 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.160+0000 7fde26a00640 1 -- 172.21.15.32:0/3812755489 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fde2c03c440 con 0x55be30f994d0 2024-04-30T08:35:45.164 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.160+0000 7fde4edce740 10 client.15555 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:35:45.164 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.160+0000 7fde4edce740 10 client.15555 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-30T08:35:45.164 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.160+0000 7fde4edce740 20 client.15555 populate_metadata read hostname 'smithi032' 2024-04-30T08:35:45.164 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.160+0000 7fde4edce740 1 --2- 172.21.15.32:0/3812755489 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55be30f78d80 0x55be31373860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:45.165 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.160+0000 7fde4edce740 4 client.15555 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:35:45.165 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.160+0000 7fde4edce740 1 -- 172.21.15.32:0/3812755489 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55be312ce160 con 0x55be30f78d80 2024-04-30T08:35:45.165 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.161+0000 7fde45e00640 1 --2- 172.21.15.32:0/3812755489 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55be30f78d80 0x55be31373860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:45.166 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.161+0000 7fde45e00640 1 --2- 172.21.15.32:0/3812755489 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55be30f78d80 0x55be31373860 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f1e5e13b0eb81651 server_cookie=e030b37e39f11485 in_seq=0 out_seq=0 2024-04-30T08:35:45.166 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.161+0000 7fde26a00640 10 client.15555 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:45.166 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.162+0000 7fde26a00640 1 -- 172.21.15.32:0/3812755489 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55be312ce160 con 0x55be30f78d80 2024-04-30T08:35:45.166 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.162+0000 7fde26a00640 10 client.15555 handle_command_reply: tid=0 2024-04-30T08:35:45.166 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.162+0000 7fde4edce740 1 client.15555 shutdown 2024-04-30T08:35:45.166 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.163+0000 7fde4edce740 1 -- 172.21.15.32:0/3812755489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be30f994d0 msgr2=0x55be313662b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:45.166 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.163+0000 7fde4edce740 1 --2- 172.21.15.32:0/3812755489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be30f994d0 0x55be313662b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fde2c000c00 tx=0x7fde2c004510 comp rx=0 tx=0).stop 2024-04-30T08:35:45.167 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.163+0000 7fde4edce740 1 -- 172.21.15.32:0/3812755489 shutdown_connections 2024-04-30T08:35:45.167 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.163+0000 7fde4edce740 1 --2- 172.21.15.32:0/3812755489 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55be30f78d80 0x55be31373860 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:45.167 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.163+0000 7fde4edce740 1 --2- 172.21.15.32:0/3812755489 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55be30fba1b0 0x55be3136d210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:45.167 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.163+0000 7fde4edce740 1 --2- 172.21.15.32:0/3812755489 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be30f994d0 0x55be313662b0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:45.167 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.163+0000 7fde4edce740 1 --2- 172.21.15.32:0/3812755489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be30f912f0 0x55be31365d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:45.167 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.163+0000 7fde26a00640 0 client.15555 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:45.167 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.163+0000 7fde4edce740 1 -- 172.21.15.32:0/3812755489 >> 172.21.15.32:0/3812755489 conn(0x55be30fae300 msgr2=0x55be31356c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:45.167 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.163+0000 7fde4edce740 1 -- 172.21.15.32:0/3812755489 shutdown_connections 2024-04-30T08:35:45.167 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.163+0000 7fde4edce740 1 -- 172.21.15.32:0/3812755489 wait complete. 2024-04-30T08:35:45.167 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.163+0000 7fde4edce740 20 client.15555 trim_cache size 0 max 16384 2024-04-30T08:35:45.182 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.180+0000 7fde4edce740 1 Processor -- start 2024-04-30T08:35:45.182 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.180+0000 7fde4edce740 1 -- start start 2024-04-30T08:35:45.182 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.180+0000 7fde4edce740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be30f912f0 0x55be3135b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:45.182 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.180+0000 7fde4edce740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55be30f994d0 0x55be3135b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:45.182 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.180+0000 7fde4edce740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be30fba1b0 0x55be313790b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:45.182 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.180+0000 7fde4edce740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fde401aebb0 con 0x55be30fba1b0 2024-04-30T08:35:45.182 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.180+0000 7fde4edce740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fde401b06f0 con 0x55be30f994d0 2024-04-30T08:35:45.182 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.180+0000 7fde4edce740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fde401af280 con 0x55be30f912f0 2024-04-30T08:35:45.183 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.181+0000 7fde45e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be30fba1b0 0x55be313790b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:45.183 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.181+0000 7fde45e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be30fba1b0 0x55be313790b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37500/0 (socket says 172.21.15.32:37500) 2024-04-30T08:35:45.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.181+0000 7fde45e00640 1 -- 172.21.15.32:0/3494044611 learned_addr learned my addr 172.21.15.32:0/3494044611 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:45.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.181+0000 7fde44a00640 1 --2- 172.21.15.32:0/3494044611 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55be30f994d0 0x55be3135b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:45.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.181+0000 7fde45e00640 1 --2- 172.21.15.32:0/3494044611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be30f912f0 0x55be3135b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:45.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.181+0000 7fde44a00640 1 -- 172.21.15.32:0/3494044611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be30f912f0 msgr2=0x55be3135b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:45.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.181+0000 7fde44a00640 1 --2- 172.21.15.32:0/3494044611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be30f912f0 0x55be3135b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:45.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.181+0000 7fde44a00640 1 -- 172.21.15.32:0/3494044611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be30fba1b0 msgr2=0x55be313790b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:45.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.181+0000 7fde44a00640 1 --2- 172.21.15.32:0/3494044611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be30fba1b0 0x55be313790b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:45.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.181+0000 7fde44a00640 1 -- 172.21.15.32:0/3494044611 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be31360c20 con 0x55be30f994d0 2024-04-30T08:35:45.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.181+0000 7fde45e00640 1 --2- 172.21.15.32:0/3494044611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be30fba1b0 0x55be313790b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:45.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.182+0000 7fde44a00640 1 --2- 172.21.15.32:0/3494044611 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55be30f994d0 0x55be3135b6c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fde2c009850 tx=0x7fde2c02a3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fecbcad5efcbba5 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:45.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.182+0000 7fde31c00640 1 -- 172.21.15.32:0/3494044611 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2c057070 con 0x55be30f994d0 2024-04-30T08:35:45.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.182+0000 7fde31c00640 1 -- 172.21.15.32:0/3494044611 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde2c03cc80 con 0x55be30f994d0 2024-04-30T08:35:45.184 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.182+0000 7fde31c00640 1 -- 172.21.15.32:0/3494044611 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2c035030 con 0x55be30f994d0 2024-04-30T08:35:45.186 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.183+0000 7fde4edce740 1 -- 172.21.15.32:0/3494044611 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55be30f994d0 msgr2=0x55be3135b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:45.186 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.183+0000 7fde4edce740 1 --2- 172.21.15.32:0/3494044611 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55be30f994d0 0x55be3135b6c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fde2c009850 tx=0x7fde2c02a3f0 comp rx=0 tx=0).stop 2024-04-30T08:35:45.186 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.183+0000 7fde4edce740 1 -- 172.21.15.32:0/3494044611 shutdown_connections 2024-04-30T08:35:45.186 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.183+0000 7fde4edce740 1 --2- 172.21.15.32:0/3494044611 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be30fba1b0 0x55be313790b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:45.186 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.183+0000 7fde4edce740 1 --2- 172.21.15.32:0/3494044611 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55be30f994d0 0x55be3135b6c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:45.186 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.183+0000 7fde4edce740 1 --2- 172.21.15.32:0/3494044611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be30f912f0 0x55be3135b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:45.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.183+0000 7fde4edce740 1 -- 172.21.15.32:0/3494044611 >> 172.21.15.32:0/3494044611 conn(0x55be30fae300 msgr2=0x55be3135aa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:45.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.183+0000 7fde4edce740 1 -- 172.21.15.32:0/3494044611 shutdown_connections 2024-04-30T08:35:45.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.183+0000 7fde4edce740 1 -- 172.21.15.32:0/3494044611 wait complete. 2024-04-30T08:35:45.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.183+0000 7fde4edce740 1 -- start start 2024-04-30T08:35:45.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.184+0000 7fde4edce740 1 Processor -- start 2024-04-30T08:35:45.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.184+0000 7fde4edce740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be30f912f0 0x55be3136b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:45.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.184+0000 7fde4edce740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be30f994d0 0x55be3136b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:45.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.184+0000 7fde4edce740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55be30fba1b0 0x55be3136c880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:45.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.184+0000 7fde4edce740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fde401b0020 con 0x55be30f994d0 2024-04-30T08:35:45.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.184+0000 7fde4edce740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fde401af950 con 0x55be30fba1b0 2024-04-30T08:35:45.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.184+0000 7fde4edce740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fde401acb70 con 0x55be30f912f0 2024-04-30T08:35:45.187 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.184+0000 7fde4edce740 10 client.0 _ll_register_callbacks cb 0x7fde2c0041a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:35:45.188 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.184+0000 7fde44a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be30f994d0 0x55be3136b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:45.188 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.184+0000 7fde44a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be30f994d0 0x55be3136b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:37508/0 (socket says 172.21.15.32:37508) 2024-04-30T08:35:45.188 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.184+0000 7fde44a00640 1 -- 172.21.15.32:0/1430822195 learned_addr learned my addr 172.21.15.32:0/1430822195 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:45.188 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.184+0000 7fde45e00640 1 --2- 172.21.15.32:0/1430822195 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55be30fba1b0 0x55be3136c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:45.188 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.185+0000 7fde45e00640 1 --2- 172.21.15.32:0/1430822195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be30f912f0 0x55be3136b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:45.188 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.185+0000 7fde44a00640 1 -- 172.21.15.32:0/1430822195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be30f912f0 msgr2=0x55be3136b410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:45.188 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.185+0000 7fde44a00640 1 --2- 172.21.15.32:0/1430822195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be30f912f0 0x55be3136b410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:45.188 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.185+0000 7fde44a00640 1 -- 172.21.15.32:0/1430822195 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55be30fba1b0 msgr2=0x55be3136c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:45.188 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.185+0000 7fde44a00640 1 --2- 172.21.15.32:0/1430822195 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55be30fba1b0 0x55be3136c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:45.188 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.185+0000 7fde44a00640 1 -- 172.21.15.32:0/1430822195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be3136ce40 con 0x55be30f994d0 2024-04-30T08:35:45.190 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.185+0000 7fde44a00640 1 --2- 172.21.15.32:0/1430822195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be30f994d0 0x55be3136b950 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fde2c0042b0 tx=0x7fde2c009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bcbec86aafb77eba server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:45.190 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.185+0000 7fde26a00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:35:45.190 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.185+0000 7fde26a00640 1 -- 172.21.15.32:0/1430822195 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2c035070 con 0x55be30f994d0 2024-04-30T08:35:45.190 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.186+0000 7fde26a00640 1 -- 172.21.15.32:0/1430822195 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fde2c003f00 con 0x55be30f994d0 2024-04-30T08:35:45.190 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.186+0000 7fde4edce740 1 -- 172.21.15.32:0/1430822195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde2c03cc80 con 0x55be30f994d0 2024-04-30T08:35:45.190 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.186+0000 7fde26a00640 1 -- 172.21.15.32:0/1430822195 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fde2c064050 con 0x55be30f994d0 2024-04-30T08:35:45.190 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.186+0000 7fde4edce740 10 client.15561 fetch_fsmap learned FSMap version 28 2024-04-30T08:35:45.190 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.186+0000 7fde4edce740 1 -- 172.21.15.32:0/1430822195 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55be313dbe10 con 0x55be30f994d0 2024-04-30T08:35:45.190 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.186+0000 7fde26a00640 1 -- 172.21.15.32:0/1430822195 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fde2c03cc80 con 0x55be30f994d0 2024-04-30T08:35:45.190 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.186+0000 7fde26a00640 1 -- 172.21.15.32:0/1430822195 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fde2c0466d0 con 0x55be30f994d0 2024-04-30T08:35:45.190 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.186+0000 7fde4edce740 10 client.15561 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:35:45.190 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.186+0000 7fde4edce740 10 client.15561 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-30T08:35:45.191 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.186+0000 7fde4edce740 20 client.15561 populate_metadata read hostname 'smithi032' 2024-04-30T08:35:45.191 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.186+0000 7fde4edce740 1 --2- 172.21.15.32:0/1430822195 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55be30f4e1c0 0x55be313de7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:45.191 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.187+0000 7fde4edce740 4 client.15561 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-30T08:35:45.191 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.187+0000 7fde4edce740 1 -- 172.21.15.32:0/1430822195 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55be312c7fd0 con 0x55be30f4e1c0 2024-04-30T08:35:45.191 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.187+0000 7fde45e00640 1 --2- 172.21.15.32:0/1430822195 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55be30f4e1c0 0x55be313de7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:45.191 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.187+0000 7fde45e00640 1 --2- 172.21.15.32:0/1430822195 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55be30f4e1c0 0x55be313de7a0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f6012be7b5daadf server_cookie=eaaa9caca3815235 in_seq=0 out_seq=0 2024-04-30T08:35:45.191 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.187+0000 7fde26a00640 10 client.15561 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:45.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.191+0000 7fde26a00640 1 -- 172.21.15.32:0/1430822195 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55be312c7fd0 con 0x55be30f4e1c0 2024-04-30T08:35:45.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.191+0000 7fde26a00640 10 client.15561 handle_command_reply: tid=0 2024-04-30T08:35:45.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.191+0000 7fde4edce740 1 client.15561 shutdown 2024-04-30T08:35:45.193 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.191+0000 7fde4edce740 1 -- 172.21.15.32:0/1430822195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be30f994d0 msgr2=0x55be3136b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:45.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.191+0000 7fde4edce740 1 --2- 172.21.15.32:0/1430822195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be30f994d0 0x55be3136b950 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fde2c0042b0 tx=0x7fde2c009520 comp rx=0 tx=0).stop 2024-04-30T08:35:45.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.192+0000 7fde4edce740 1 -- 172.21.15.32:0/1430822195 shutdown_connections 2024-04-30T08:35:45.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.192+0000 7fde4edce740 1 --2- 172.21.15.32:0/1430822195 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55be30f4e1c0 0x55be313de7a0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:45.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.192+0000 7fde4edce740 1 --2- 172.21.15.32:0/1430822195 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55be30fba1b0 0x55be3136c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:45.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.192+0000 7fde4edce740 1 --2- 172.21.15.32:0/1430822195 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55be30f994d0 0x55be3136b950 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:45.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.192+0000 7fde4edce740 1 --2- 172.21.15.32:0/1430822195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be30f912f0 0x55be3136b410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:45.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.192+0000 7fde26a00640 0 client.15561 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:45.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.192+0000 7fde4edce740 1 -- 172.21.15.32:0/1430822195 >> 172.21.15.32:0/1430822195 conn(0x55be30fae300 msgr2=0x55be3135b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:45.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.192+0000 7fde4edce740 1 -- 172.21.15.32:0/1430822195 shutdown_connections 2024-04-30T08:35:45.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.192+0000 7fde4edce740 1 -- 172.21.15.32:0/1430822195 wait complete. 2024-04-30T08:35:45.194 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.192+0000 7fde4edce740 20 client.15561 trim_cache size 0 max 16384 2024-04-30T08:35:45.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-30T08:35:45.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-30T08:35:45.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-30T08:35:45.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-30T08:35:45.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.193+0000 7fde46800640 1 -- 172.21.15.32:0/1441005398 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fde280afbf0 msgr2=0x7fde280b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:45.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.193+0000 7fde46800640 1 --2- 172.21.15.32:0/1441005398 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fde280afbf0 0x7fde280b2100 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fde340a3520 tx=0x7fde340a3430 comp rx=0 tx=0).stop 2024-04-30T08:35:45.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.193+0000 7fde46800640 1 -- 172.21.15.32:0/1441005398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde40103a70 msgr2=0x7fde400779e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:45.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.193+0000 7fde46800640 1 --2- 172.21.15.32:0/1441005398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde40103a70 0x7fde400779e0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fde3400e9f0 tx=0x7fde3400eeb0 comp rx=0 tx=0).stop 2024-04-30T08:35:45.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.193+0000 7fde46800640 1 -- 172.21.15.32:0/1441005398 shutdown_connections 2024-04-30T08:35:45.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.193+0000 7fde46800640 1 --2- 172.21.15.32:0/1441005398 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fde280afbf0 0x7fde280b2100 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:45.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.193+0000 7fde46800640 1 --2- 172.21.15.32:0/1441005398 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fde401099b0 0x7fde400765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:45.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.193+0000 7fde46800640 1 --2- 172.21.15.32:0/1441005398 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fde40104cb0 0x7fde40076080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:45.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.193+0000 7fde46800640 1 --2- 172.21.15.32:0/1441005398 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde40103a70 0x7fde400779e0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:45.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.193+0000 7fde46800640 1 -- 172.21.15.32:0/1441005398 >> 172.21.15.32:0/1441005398 conn(0x7fde400fec80 msgr2=0x7fde40105990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:45.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.193+0000 7fde46800640 1 -- 172.21.15.32:0/1441005398 shutdown_connections 2024-04-30T08:35:45.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:45.193+0000 7fde46800640 1 -- 172.21.15.32:0/1441005398 wait complete. 2024-04-30T08:35:45.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:45 smithi071 ceph-mon[33385]: pgmap v442: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 14 op/s 2024-04-30T08:35:45.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:45 smithi189 ceph-mon[33038]: pgmap v442: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 14 op/s 2024-04-30T08:35:45.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:45 smithi032 ceph-mon[28331]: pgmap v442: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 14 op/s 2024-04-30T08:35:47.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:47 smithi189 ceph-mon[33038]: pgmap v443: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-04-30T08:35:47.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:47 smithi032 ceph-mon[28331]: pgmap v443: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-04-30T08:35:47.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:47 smithi071 ceph-mon[33385]: pgmap v443: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-04-30T08:35:49.312 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-30T08:35:49.313 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-30T08:35:49.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:49 smithi189 ceph-mon[33038]: pgmap v444: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-30T08:35:49.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.615+0000 7f67eca00640 1 -- 172.21.15.32:0/470097016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67e8103b70 msgr2=0x7f67e8104870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:49.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.615+0000 7f67eca00640 1 --2- 172.21.15.32:0/470097016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67e8103b70 0x7f67e8104870 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f67d8009860 tx=0x7f67d802a340 comp rx=0 tx=0).stop 2024-04-30T08:35:49.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.615+0000 7f67eca00640 1 -- 172.21.15.32:0/470097016 shutdown_connections 2024-04-30T08:35:49.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.615+0000 7f67eca00640 1 --2- 172.21.15.32:0/470097016 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f67e8109ab0 0x7f67e810bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.617 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.615+0000 7f67eca00640 1 --2- 172.21.15.32:0/470097016 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67e8104db0 0x7f67e81051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.615+0000 7f67eca00640 1 --2- 172.21.15.32:0/470097016 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67e8103b70 0x7f67e8104870 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.615+0000 7f67eca00640 1 -- 172.21.15.32:0/470097016 >> 172.21.15.32:0/470097016 conn(0x7f67e80fed60 msgr2=0x7f67e8101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:49.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.616+0000 7f67eca00640 1 -- 172.21.15.32:0/470097016 shutdown_connections 2024-04-30T08:35:49.618 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.616+0000 7f67eca00640 1 -- 172.21.15.32:0/470097016 wait complete. 2024-04-30T08:35:49.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.617+0000 7f67eca00640 1 Processor -- start 2024-04-30T08:35:49.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.617+0000 7f67eca00640 1 -- start start 2024-04-30T08:35:49.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.617+0000 7f67eca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67e8103b70 0x7f67e819c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:49.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.617+0000 7f67eca00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f67e8104db0 0x7f67e819ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:49.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.618+0000 7f67eca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67e8109ab0 0x7f67e81a3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:49.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.618+0000 7f67eca00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f67e810f670 con 0x7f67e8103b70 2024-04-30T08:35:49.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.618+0000 7f67eca00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f67e810f4f0 con 0x7f67e8104db0 2024-04-30T08:35:49.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.618+0000 7f67eca00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f67e810f7f0 con 0x7f67e8109ab0 2024-04-30T08:35:49.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.618+0000 7f67e5e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67e8103b70 0x7f67e819c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:49.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.618+0000 7f67e5e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67e8103b70 0x7f67e819c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53332/0 (socket says 172.21.15.32:53332) 2024-04-30T08:35:49.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.618+0000 7f67e5e00640 1 -- 172.21.15.32:0/1035083615 learned_addr learned my addr 172.21.15.32:0/1035083615 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:49.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.618+0000 7f67e5400640 1 --2- 172.21.15.32:0/1035083615 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f67e8104db0 0x7f67e819ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:49.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.618+0000 7f67e6800640 1 --2- 172.21.15.32:0/1035083615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67e8109ab0 0x7f67e81a3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:49.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.618+0000 7f67e5e00640 1 -- 172.21.15.32:0/1035083615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67e8109ab0 msgr2=0x7f67e81a3990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:49.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.618+0000 7f67e5e00640 1 --2- 172.21.15.32:0/1035083615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67e8109ab0 0x7f67e81a3990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.618+0000 7f67e5e00640 1 -- 172.21.15.32:0/1035083615 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f67e8104db0 msgr2=0x7f67e819ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:49.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.618+0000 7f67e5e00640 1 --2- 172.21.15.32:0/1035083615 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f67e8104db0 0x7f67e819ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.618+0000 7f67e5e00640 1 -- 172.21.15.32:0/1035083615 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67e81a4000 con 0x7f67e8103b70 2024-04-30T08:35:49.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.618+0000 7f67e6800640 1 --2- 172.21.15.32:0/1035083615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67e8109ab0 0x7f67e81a3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:35:49.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.619+0000 7f67e5e00640 1 --2- 172.21.15.32:0/1035083615 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67e8103b70 0x7f67e819c4c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f67d80304b0 tx=0x7f67d8032d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:49.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.619+0000 7f67dea00640 1 -- 172.21.15.32:0/1035083615 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d8037040 con 0x7f67e8103b70 2024-04-30T08:35:49.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.619+0000 7f67dea00640 1 -- 172.21.15.32:0/1035083615 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f67d8030c10 con 0x7f67e8103b70 2024-04-30T08:35:49.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.619+0000 7f67eca00640 1 -- 172.21.15.32:0/1035083615 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67e81a4290 con 0x7f67e8103b70 2024-04-30T08:35:49.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.619+0000 7f67dea00640 1 -- 172.21.15.32:0/1035083615 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d8003850 con 0x7f67e8103b70 2024-04-30T08:35:49.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.619+0000 7f67eca00640 1 -- 172.21.15.32:0/1035083615 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67e81a4720 con 0x7f67e8103b70 2024-04-30T08:35:49.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.623+0000 7f67f3a4e740 1 Processor -- start 2024-04-30T08:35:49.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.623+0000 7f67f3a4e740 1 -- start start 2024-04-30T08:35:49.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.623+0000 7f67f3a4e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8188b2390 0x55f818c53240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:49.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.623+0000 7f67f3a4e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8188894d0 0x55f818c537c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:49.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.623+0000 7f67f3a4e740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8188916b0 0x55f818c5a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:49.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.623+0000 7f67f3a4e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f818bb8d50 con 0x55f8188b2390 2024-04-30T08:35:49.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.623+0000 7f67f3a4e740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55f818b5def0 con 0x55f8188916b0 2024-04-30T08:35:49.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.623+0000 7f67f3a4e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f818bb4d10 con 0x55f8188894d0 2024-04-30T08:35:49.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.624+0000 7f67dea00640 1 -- 172.21.15.32:0/1035083615 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f67d8043020 con 0x7f67e8103b70 2024-04-30T08:35:49.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.624+0000 7f67e5400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8188916b0 0x55f818c5a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:49.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.624+0000 7f67e6800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8188b2390 0x55f818c53240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:49.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.624+0000 7f67e5400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8188916b0 0x55f818c5a6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:54556/0 (socket says 172.21.15.32:54556) 2024-04-30T08:35:49.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.624+0000 7f67e5400640 1 -- 172.21.15.32:0/1194015163 learned_addr learned my addr 172.21.15.32:0/1194015163 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:49.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.624+0000 7f67e5e00640 1 --2- 172.21.15.32:0/1194015163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8188894d0 0x55f818c537c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:49.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.625+0000 7f67e6800640 1 -- 172.21.15.32:0/1194015163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8188894d0 msgr2=0x55f818c537c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:49.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.625+0000 7f67e6800640 1 --2- 172.21.15.32:0/1194015163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8188894d0 0x55f818c537c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.625+0000 7f67e6800640 1 -- 172.21.15.32:0/1194015163 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8188916b0 msgr2=0x55f818c5a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:49.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.625+0000 7f67e6800640 1 --2- 172.21.15.32:0/1194015163 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8188916b0 0x55f818c5a6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.625+0000 7f67e6800640 1 -- 172.21.15.32:0/1194015163 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8189947d0 con 0x55f8188b2390 2024-04-30T08:35:49.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.625+0000 7f67e5400640 1 --2- 172.21.15.32:0/1194015163 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8188916b0 0x55f818c5a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:49.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.625+0000 7f67e6800640 1 --2- 172.21.15.32:0/1194015163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8188b2390 0x55f818c53240 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f67e0009f90 tx=0x7f67e000f890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b554e7aedd3fb05 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:49.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.625+0000 7f67d7000640 1 -- 172.21.15.32:0/1194015163 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e0003970 con 0x55f8188b2390 2024-04-30T08:35:49.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.625+0000 7f67d7000640 1 -- 172.21.15.32:0/1194015163 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f67e0004020 con 0x55f8188b2390 2024-04-30T08:35:49.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.625+0000 7f67d7000640 1 -- 172.21.15.32:0/1194015163 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e000c910 con 0x55f8188b2390 2024-04-30T08:35:49.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.625+0000 7f67dea00640 1 --2- 172.21.15.32:0/1035083615 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f67cc0afb80 0x7f67cc0b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:49.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.626+0000 7f67dea00640 1 -- 172.21.15.32:0/1035083615 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f67d80fe370 con 0x7f67e8103b70 2024-04-30T08:35:49.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.626+0000 7f67e5e00640 1 --2- 172.21.15.32:0/1035083615 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f67cc0afb80 0x7f67cc0b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:49.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.626+0000 7f67f3a4e740 1 -- 172.21.15.32:0/1194015163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8188b2390 msgr2=0x55f818c53240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:49.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.626+0000 7f67f3a4e740 1 --2- 172.21.15.32:0/1194015163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8188b2390 0x55f818c53240 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f67e0009f90 tx=0x7f67e000f890 comp rx=0 tx=0).stop 2024-04-30T08:35:49.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.626+0000 7f67f3a4e740 1 -- 172.21.15.32:0/1194015163 shutdown_connections 2024-04-30T08:35:49.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.626+0000 7f67f3a4e740 1 --2- 172.21.15.32:0/1194015163 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8188916b0 0x55f818c5a6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.626+0000 7f67f3a4e740 1 --2- 172.21.15.32:0/1194015163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8188894d0 0x55f818c537c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.626+0000 7f67f3a4e740 1 --2- 172.21.15.32:0/1194015163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8188b2390 0x55f818c53240 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.626+0000 7f67f3a4e740 1 -- 172.21.15.32:0/1194015163 >> 172.21.15.32:0/1194015163 conn(0x55f8188a64e0 msgr2=0x55f818c4ebd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:49.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.627+0000 7f67f3a4e740 1 -- 172.21.15.32:0/1194015163 shutdown_connections 2024-04-30T08:35:49.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.627+0000 7f67e5e00640 1 --2- 172.21.15.32:0/1035083615 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f67cc0afb80 0x7f67cc0b2090 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x55f818c54620 tx=0x7f67d8101e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:49.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.627+0000 7f67f3a4e740 1 -- 172.21.15.32:0/1194015163 wait complete. 2024-04-30T08:35:49.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.627+0000 7f67f3a4e740 1 -- start start 2024-04-30T08:35:49.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.627+0000 7f67f3a4e740 1 Processor -- start 2024-04-30T08:35:49.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.628+0000 7f67f3a4e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8188894d0 0x55f818c5d190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:49.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.628+0000 7f67f3a4e740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8188916b0 0x55f818c5d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:49.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.628+0000 7f67f3a4e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8188b2390 0x55f818c64630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:49.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.628+0000 7f67f3a4e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f818c5b110 con 0x55f8188b2390 2024-04-30T08:35:49.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.628+0000 7f67f3a4e740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55f818b71990 con 0x55f8188916b0 2024-04-30T08:35:49.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.628+0000 7f67f3a4e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f818c5b290 con 0x55f8188894d0 2024-04-30T08:35:49.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.628+0000 7f67f3a4e740 10 client.0 _ll_register_callbacks cb 0x55f818b70110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:35:49.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.628+0000 7f67e5400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8188916b0 0x55f818c5d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:49.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.628+0000 7f67e6800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8188b2390 0x55f818c64630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:49.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.628+0000 7f67e5400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8188916b0 0x55f818c5d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:54560/0 (socket says 172.21.15.32:54560) 2024-04-30T08:35:49.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.628+0000 7f67e5400640 1 -- 172.21.15.32:0/2933238488 learned_addr learned my addr 172.21.15.32:0/2933238488 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:49.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.628+0000 7f67e6800640 1 --2- 172.21.15.32:0/2933238488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8188894d0 0x55f818c5d190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:49.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.629+0000 7f67e5400640 1 -- 172.21.15.32:0/2933238488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8188894d0 msgr2=0x55f818c5d190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:49.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.629+0000 7f67e5400640 1 --2- 172.21.15.32:0/2933238488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8188894d0 0x55f818c5d190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.629+0000 7f67e5400640 1 -- 172.21.15.32:0/2933238488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8188b2390 msgr2=0x55f818c64630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:49.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.629+0000 7f67e5400640 1 --2- 172.21.15.32:0/2933238488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8188b2390 0x55f818c64630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.629+0000 7f67e5400640 1 -- 172.21.15.32:0/2933238488 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8188ca3f0 con 0x55f8188916b0 2024-04-30T08:35:49.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.629+0000 7f67e5400640 1 --2- 172.21.15.32:0/2933238488 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8188916b0 0x55f818c5d6d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f67d00126e0 tx=0x7f67d0012ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3fef8a680f4e00b server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:49.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.629+0000 7f67cb400640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:35:49.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.629+0000 7f67cb400640 1 -- 172.21.15.32:0/2933238488 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d00121d0 con 0x55f8188916b0 2024-04-30T08:35:49.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.629+0000 7f67cb400640 1 -- 172.21.15.32:0/2933238488 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f67d0004050 con 0x55f8188916b0 2024-04-30T08:35:49.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.629+0000 7f67f3a4e740 1 -- 172.21.15.32:0/2933238488 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67e0004020 con 0x55f8188916b0 2024-04-30T08:35:49.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.630+0000 7f67cb400640 1 -- 172.21.15.32:0/2933238488 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f67d0025050 con 0x55f8188916b0 2024-04-30T08:35:49.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.630+0000 7f67f3a4e740 10 client.25697 fetch_fsmap learned FSMap version 28 2024-04-30T08:35:49.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.630+0000 7f67f3a4e740 1 -- 172.21.15.32:0/2933238488 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f81888a0f0 con 0x55f8188916b0 2024-04-30T08:35:49.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.630+0000 7f67cb400640 1 -- 172.21.15.32:0/2933238488 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f67e0004020 con 0x55f8188916b0 2024-04-30T08:35:49.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.630+0000 7f67cb400640 1 -- 172.21.15.32:0/2933238488 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f67d00038c0 con 0x55f8188916b0 2024-04-30T08:35:49.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.630+0000 7f67f3a4e740 10 client.25697 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:35:49.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.630+0000 7f67f3a4e740 10 client.25697 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:35:49.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.630+0000 7f67f3a4e740 20 client.25697 populate_metadata read hostname 'smithi032' 2024-04-30T08:35:49.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.630+0000 7f67f3a4e740 1 --2- 172.21.15.32:0/2933238488 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55f818870f60 0x55f818c6aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:49.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.630+0000 7f67f3a4e740 4 client.25697 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:35:49.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.630+0000 7f67f3a4e740 1 -- 172.21.15.32:0/2933238488 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f818bccef0 con 0x55f818870f60 2024-04-30T08:35:49.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.631+0000 7f67e6800640 1 --2- 172.21.15.32:0/2933238488 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55f818870f60 0x55f818c6aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:49.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.631+0000 7f67e6800640 1 --2- 172.21.15.32:0/2933238488 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55f818870f60 0x55f818c6aae0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=caf62b20233d37ed server_cookie=40751d358fce1aae in_seq=0 out_seq=0 2024-04-30T08:35:49.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.631+0000 7f67cb400640 10 client.25697 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:49.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.632+0000 7f67cb400640 1 -- 172.21.15.32:0/2933238488 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f818bccef0 con 0x55f818870f60 2024-04-30T08:35:49.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.632+0000 7f67cb400640 10 client.25697 handle_command_reply: tid=0 2024-04-30T08:35:49.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.632+0000 7f67f3a4e740 1 client.25697 shutdown 2024-04-30T08:35:49.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.633+0000 7f67f3a4e740 1 -- 172.21.15.32:0/2933238488 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8188916b0 msgr2=0x55f818c5d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:49.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.633+0000 7f67f3a4e740 1 --2- 172.21.15.32:0/2933238488 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8188916b0 0x55f818c5d6d0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f67d00126e0 tx=0x7f67d0012ba0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.633+0000 7f67f3a4e740 1 -- 172.21.15.32:0/2933238488 shutdown_connections 2024-04-30T08:35:49.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.633+0000 7f67f3a4e740 1 --2- 172.21.15.32:0/2933238488 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55f818870f60 0x55f818c6aae0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.633+0000 7f67f3a4e740 1 --2- 172.21.15.32:0/2933238488 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8188b2390 0x55f818c64630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.633+0000 7f67f3a4e740 1 --2- 172.21.15.32:0/2933238488 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8188916b0 0x55f818c5d6d0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.633+0000 7f67cb400640 0 client.25697 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:49.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.633+0000 7f67f3a4e740 1 --2- 172.21.15.32:0/2933238488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8188894d0 0x55f818c5d190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.633+0000 7f67f3a4e740 1 -- 172.21.15.32:0/2933238488 >> 172.21.15.32:0/2933238488 conn(0x55f8188a64e0 msgr2=0x55f818c4dbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:49.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.633+0000 7f67f3a4e740 1 -- 172.21.15.32:0/2933238488 shutdown_connections 2024-04-30T08:35:49.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.633+0000 7f67f3a4e740 1 -- 172.21.15.32:0/2933238488 wait complete. 2024-04-30T08:35:49.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.633+0000 7f67f3a4e740 20 client.25697 trim_cache size 0 max 16384 2024-04-30T08:35:49.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.653+0000 7f67f3a4e740 1 Processor -- start 2024-04-30T08:35:49.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.653+0000 7f67f3a4e740 1 -- start start 2024-04-30T08:35:49.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.653+0000 7f67f3a4e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8188894d0 0x55f818c57c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:49.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.653+0000 7f67f3a4e740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8188916b0 0x55f818c5bec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:49.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.654+0000 7f67f3a4e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8188b2390 0x55f818c64550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:49.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.654+0000 7f67f3a4e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f67e81aed70 con 0x55f8188894d0 2024-04-30T08:35:49.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.654+0000 7f67f3a4e740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f67e81b08b0 con 0x55f8188916b0 2024-04-30T08:35:49.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.654+0000 7f67f3a4e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f67e81af440 con 0x55f8188b2390 2024-04-30T08:35:49.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.654+0000 7f67e5400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8188916b0 0x55f818c5bec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:49.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.654+0000 7f67e5400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8188916b0 0x55f818c5bec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:54570/0 (socket says 172.21.15.32:54570) 2024-04-30T08:35:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.654+0000 7f67e5400640 1 -- 172.21.15.32:0/1048946042 learned_addr learned my addr 172.21.15.32:0/1048946042 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.654+0000 7f67e6800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8188894d0 0x55f818c57c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.654+0000 7f67e5400640 1 -- 172.21.15.32:0/1048946042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8188b2390 msgr2=0x55f818c64550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.654+0000 7f67e5400640 1 --2- 172.21.15.32:0/1048946042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8188b2390 0x55f818c64550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.654+0000 7f67e5400640 1 -- 172.21.15.32:0/1048946042 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8188894d0 msgr2=0x55f818c57c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.654+0000 7f67e5400640 1 --2- 172.21.15.32:0/1048946042 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8188894d0 0x55f818c57c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.654+0000 7f67e5400640 1 -- 172.21.15.32:0/1048946042 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f818c56660 con 0x55f8188916b0 2024-04-30T08:35:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.655+0000 7f67e5400640 1 --2- 172.21.15.32:0/1048946042 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8188916b0 0x55f818c5bec0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f67d0000c00 tx=0x7f67d0009f00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85d32dcabf7c30d9 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.655+0000 7f67d6600640 1 -- 172.21.15.32:0/1048946042 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d002e070 con 0x55f8188916b0 2024-04-30T08:35:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.655+0000 7f67d6600640 1 -- 172.21.15.32:0/1048946042 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f67d0003aa0 con 0x55f8188916b0 2024-04-30T08:35:49.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.655+0000 7f67d6600640 1 -- 172.21.15.32:0/1048946042 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67d000fea0 con 0x55f8188916b0 2024-04-30T08:35:49.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.656+0000 7f67f3a4e740 1 -- 172.21.15.32:0/1048946042 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8188916b0 msgr2=0x55f818c5bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:49.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.656+0000 7f67f3a4e740 1 --2- 172.21.15.32:0/1048946042 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8188916b0 0x55f818c5bec0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f67d0000c00 tx=0x7f67d0009f00 comp rx=0 tx=0).stop 2024-04-30T08:35:49.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.656+0000 7f67f3a4e740 1 -- 172.21.15.32:0/1048946042 shutdown_connections 2024-04-30T08:35:49.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.656+0000 7f67f3a4e740 1 --2- 172.21.15.32:0/1048946042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8188b2390 0x55f818c64550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.656+0000 7f67f3a4e740 1 --2- 172.21.15.32:0/1048946042 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8188916b0 0x55f818c5bec0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.656+0000 7f67f3a4e740 1 --2- 172.21.15.32:0/1048946042 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8188894d0 0x55f818c57c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.656+0000 7f67f3a4e740 1 -- 172.21.15.32:0/1048946042 >> 172.21.15.32:0/1048946042 conn(0x55f8188a64e0 msgr2=0x55f818c5b7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:49.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.656+0000 7f67f3a4e740 1 -- 172.21.15.32:0/1048946042 shutdown_connections 2024-04-30T08:35:49.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.656+0000 7f67f3a4e740 1 -- 172.21.15.32:0/1048946042 wait complete. 2024-04-30T08:35:49.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.657+0000 7f67f3a4e740 1 -- start start 2024-04-30T08:35:49.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.657+0000 7f67f3a4e740 1 Processor -- start 2024-04-30T08:35:49.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.657+0000 7f67f3a4e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8188894d0 0x55f818c62200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:49.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.657+0000 7f67f3a4e740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8188916b0 0x55f818c62740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:49.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.657+0000 7f67f3a4e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8188b2390 0x55f818c63670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:49.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.657+0000 7f67f3a4e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f67e81b01e0 con 0x55f8188894d0 2024-04-30T08:35:49.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.657+0000 7f67f3a4e740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f67e81afb10 con 0x55f8188916b0 2024-04-30T08:35:49.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.657+0000 7f67f3a4e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f67e81acd30 con 0x55f8188b2390 2024-04-30T08:35:49.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.657+0000 7f67f3a4e740 10 client.0 _ll_register_callbacks cb 0x7f67d0009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:35:49.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.658+0000 7f67e6800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8188894d0 0x55f818c62200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:49.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.658+0000 7f67e5400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8188916b0 0x55f818c62740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:49.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.658+0000 7f67e6800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8188894d0 0x55f818c62200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53384/0 (socket says 172.21.15.32:53384) 2024-04-30T08:35:49.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.658+0000 7f67e6800640 1 -- 172.21.15.32:0/494279780 learned_addr learned my addr 172.21.15.32:0/494279780 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:49.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.658+0000 7f67e6800640 1 --2- 172.21.15.32:0/494279780 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8188b2390 0x55f818c63670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:49.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.658+0000 7f67e6800640 1 -- 172.21.15.32:0/494279780 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8188b2390 msgr2=0x55f818c63670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:49.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.658+0000 7f67e6800640 1 --2- 172.21.15.32:0/494279780 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8188b2390 0x55f818c63670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.658+0000 7f67e6800640 1 -- 172.21.15.32:0/494279780 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8188916b0 msgr2=0x55f818c62740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:49.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.658+0000 7f67e6800640 1 --2- 172.21.15.32:0/494279780 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8188916b0 0x55f818c62740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.658+0000 7f67e6800640 1 -- 172.21.15.32:0/494279780 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f818c63c30 con 0x55f8188894d0 2024-04-30T08:35:49.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.658+0000 7f67e5400640 1 --2- 172.21.15.32:0/494279780 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8188916b0 0x55f818c62740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:49.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.658+0000 7f67e6800640 1 --2- 172.21.15.32:0/494279780 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8188894d0 0x55f818c62200 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f67e000fe10 tx=0x7f67e000c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c5714f000e4a529 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:49.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.658+0000 7f67cb400640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:35:49.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.659+0000 7f67cb400640 1 -- 172.21.15.32:0/494279780 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67e0031070 con 0x55f8188894d0 2024-04-30T08:35:49.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.659+0000 7f67cb400640 1 -- 172.21.15.32:0/494279780 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f67e0004510 con 0x55f8188894d0 2024-04-30T08:35:49.663 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.659+0000 7f67f3a4e740 1 -- 172.21.15.32:0/494279780 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67d0003aa0 con 0x55f8188894d0 2024-04-30T08:35:49.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.659+0000 7f67cb400640 1 -- 172.21.15.32:0/494279780 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f67e00049e0 con 0x55f8188894d0 2024-04-30T08:35:49.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.659+0000 7f67f3a4e740 10 client.15579 fetch_fsmap learned FSMap version 28 2024-04-30T08:35:49.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.659+0000 7f67f3a4e740 1 -- 172.21.15.32:0/494279780 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f818c64ec0 con 0x55f8188894d0 2024-04-30T08:35:49.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.659+0000 7f67cb400640 1 -- 172.21.15.32:0/494279780 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f67d0003aa0 con 0x55f8188894d0 2024-04-30T08:35:49.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.660+0000 7f67cb400640 1 -- 172.21.15.32:0/494279780 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f67e0024040 con 0x55f8188894d0 2024-04-30T08:35:49.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.660+0000 7f67f3a4e740 10 client.15579 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:35:49.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.660+0000 7f67f3a4e740 10 client.15579 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:35:49.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.660+0000 7f67f3a4e740 20 client.15579 populate_metadata read hostname 'smithi032' 2024-04-30T08:35:49.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.660+0000 7f67f3a4e740 1 --2- 172.21.15.32:0/494279780 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55f8188463a0 0x55f818cd5250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:49.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.660+0000 7f67f3a4e740 4 client.15579 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-30T08:35:49.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.660+0000 7f67f3a4e740 1 -- 172.21.15.32:0/494279780 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f818bc6960 con 0x55f8188463a0 2024-04-30T08:35:49.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.660+0000 7f67e5400640 1 --2- 172.21.15.32:0/494279780 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55f8188463a0 0x55f818cd5250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:49.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.661+0000 7f67e5400640 1 --2- 172.21.15.32:0/494279780 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55f8188463a0 0x55f818cd5250 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da1e91913156f8c9 server_cookie=b647e26a79a6ba73 in_seq=0 out_seq=0 2024-04-30T08:35:49.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.661+0000 7f67cb400640 10 client.15579 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:49.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.661+0000 7f67cb400640 1 -- 172.21.15.32:0/494279780 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55f818bc6960 con 0x55f8188463a0 2024-04-30T08:35:49.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.661+0000 7f67cb400640 10 client.15579 handle_command_reply: tid=0 2024-04-30T08:35:49.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.661+0000 7f67f3a4e740 1 client.15579 shutdown 2024-04-30T08:35:49.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.662+0000 7f67f3a4e740 1 -- 172.21.15.32:0/494279780 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8188894d0 msgr2=0x55f818c62200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:49.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.662+0000 7f67f3a4e740 1 --2- 172.21.15.32:0/494279780 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8188894d0 0x55f818c62200 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f67e000fe10 tx=0x7f67e000c640 comp rx=0 tx=0).stop 2024-04-30T08:35:49.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.662+0000 7f67f3a4e740 1 -- 172.21.15.32:0/494279780 shutdown_connections 2024-04-30T08:35:49.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.662+0000 7f67f3a4e740 1 --2- 172.21.15.32:0/494279780 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55f8188463a0 0x55f818cd5250 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.662+0000 7f67f3a4e740 1 --2- 172.21.15.32:0/494279780 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8188b2390 0x55f818c63670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.662+0000 7f67f3a4e740 1 --2- 172.21.15.32:0/494279780 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8188916b0 0x55f818c62740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.662+0000 7f67f3a4e740 1 --2- 172.21.15.32:0/494279780 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8188894d0 0x55f818c62200 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.662+0000 7f67cb400640 0 client.15579 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:49.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.662+0000 7f67f3a4e740 1 -- 172.21.15.32:0/494279780 >> 172.21.15.32:0/494279780 conn(0x55f8188a64e0 msgr2=0x55f818c519d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:49.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.662+0000 7f67f3a4e740 1 -- 172.21.15.32:0/494279780 shutdown_connections 2024-04-30T08:35:49.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.662+0000 7f67f3a4e740 1 -- 172.21.15.32:0/494279780 wait complete. 2024-04-30T08:35:49.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.662+0000 7f67f3a4e740 20 client.15579 trim_cache size 0 max 16384 2024-04-30T08:35:49.668 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-30T08:35:49.668 INFO:teuthology.orchestra.run.smithi032.stdout: "status": "no active scrubs running", 2024-04-30T08:35:49.668 INFO:teuthology.orchestra.run.smithi032.stdout: "scrubs": {} 2024-04-30T08:35:49.668 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-30T08:35:49.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.663+0000 7f67eca00640 1 -- 172.21.15.32:0/1035083615 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f67cc0afb80 msgr2=0x7f67cc0b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:49.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.663+0000 7f67eca00640 1 --2- 172.21.15.32:0/1035083615 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f67cc0afb80 0x7f67cc0b2090 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x55f818c54620 tx=0x7f67d8101e20 comp rx=0 tx=0).stop 2024-04-30T08:35:49.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.663+0000 7f67eca00640 1 -- 172.21.15.32:0/1035083615 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67e8103b70 msgr2=0x7f67e819c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:49.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.663+0000 7f67eca00640 1 --2- 172.21.15.32:0/1035083615 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67e8103b70 0x7f67e819c4c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f67d80304b0 tx=0x7f67d8032d20 comp rx=0 tx=0).stop 2024-04-30T08:35:49.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.663+0000 7f67eca00640 1 -- 172.21.15.32:0/1035083615 shutdown_connections 2024-04-30T08:35:49.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.663+0000 7f67eca00640 1 --2- 172.21.15.32:0/1035083615 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f67cc0afb80 0x7f67cc0b2090 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.663+0000 7f67eca00640 1 --2- 172.21.15.32:0/1035083615 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67e8109ab0 0x7f67e81a3990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.663+0000 7f67eca00640 1 --2- 172.21.15.32:0/1035083615 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f67e8104db0 0x7f67e819ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.663+0000 7f67eca00640 1 --2- 172.21.15.32:0/1035083615 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f67e8103b70 0x7f67e819c4c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.663+0000 7f67eca00640 1 -- 172.21.15.32:0/1035083615 >> 172.21.15.32:0/1035083615 conn(0x7f67e80fed60 msgr2=0x7f67e80ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:49.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.663+0000 7f67eca00640 1 -- 172.21.15.32:0/1035083615 shutdown_connections 2024-04-30T08:35:49.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.663+0000 7f67eca00640 1 -- 172.21.15.32:0/1035083615 wait complete. 2024-04-30T08:35:49.690 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:49 smithi032 ceph-mon[28331]: pgmap v444: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-30T08:35:49.693 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-30T08:35:49.693 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-30T08:35:49.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:49 smithi071 ceph-mon[33385]: pgmap v444: 129 pgs: 129 active+clean; 661 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-30T08:35:49.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.978+0000 7fe595800640 1 -- 172.21.15.32:0/2478021236 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe590075010 msgr2=0x7fe590075420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:49.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.978+0000 7fe595800640 1 --2- 172.21.15.32:0/2478021236 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe590075010 0x7fe590075420 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fe580009860 tx=0x7fe58002a340 comp rx=0 tx=0).stop 2024-04-30T08:35:49.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.979+0000 7fe595800640 1 -- 172.21.15.32:0/2478021236 shutdown_connections 2024-04-30T08:35:49.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.979+0000 7fe595800640 1 --2- 172.21.15.32:0/2478021236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe59010b900 0x7fe59010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.979+0000 7fe595800640 1 --2- 172.21.15.32:0/2478021236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe590075960 0x7fe59010b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.979+0000 7fe595800640 1 --2- 172.21.15.32:0/2478021236 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe590075010 0x7fe590075420 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.979+0000 7fe595800640 1 -- 172.21.15.32:0/2478021236 >> 172.21.15.32:0/2478021236 conn(0x7fe5900feba0 msgr2=0x7fe590101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:49.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.979+0000 7fe595800640 1 -- 172.21.15.32:0/2478021236 shutdown_connections 2024-04-30T08:35:49.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.979+0000 7fe595800640 1 -- 172.21.15.32:0/2478021236 wait complete. 2024-04-30T08:35:49.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.980+0000 7fe595800640 1 Processor -- start 2024-04-30T08:35:49.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.980+0000 7fe595800640 1 -- start start 2024-04-30T08:35:49.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.981+0000 7fe595800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe590075010 0x7fe59019c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:49.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.981+0000 7fe595800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe590075960 0x7fe59019c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:49.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.981+0000 7fe595800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe59010b900 0x7fe5901a37c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:49.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.981+0000 7fe595800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe5901116c0 con 0x7fe59010b900 2024-04-30T08:35:49.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.981+0000 7fe595800640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe590111540 con 0x7fe590075960 2024-04-30T08:35:49.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.981+0000 7fe595800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe590111840 con 0x7fe590075010 2024-04-30T08:35:49.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.981+0000 7fe58f400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe590075960 0x7fe59019c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:49.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.981+0000 7fe594e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe59010b900 0x7fe5901a37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:49.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.981+0000 7fe58f400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe590075960 0x7fe59019c830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:54612/0 (socket says 172.21.15.32:54612) 2024-04-30T08:35:49.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.981+0000 7fe58f400640 1 -- 172.21.15.32:0/2934859828 learned_addr learned my addr 172.21.15.32:0/2934859828 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:49.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.981+0000 7fe58fe00640 1 --2- 172.21.15.32:0/2934859828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe590075010 0x7fe59019c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:49.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.982+0000 7fe58f400640 1 -- 172.21.15.32:0/2934859828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe590075010 msgr2=0x7fe59019c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:49.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.982+0000 7fe58f400640 1 --2- 172.21.15.32:0/2934859828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe590075010 0x7fe59019c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.982+0000 7fe58f400640 1 -- 172.21.15.32:0/2934859828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe59010b900 msgr2=0x7fe5901a37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:49.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.982+0000 7fe58f400640 1 --2- 172.21.15.32:0/2934859828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe59010b900 0x7fe5901a37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:49.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.982+0000 7fe58f400640 1 -- 172.21.15.32:0/2934859828 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5901a3e30 con 0x7fe590075960 2024-04-30T08:35:49.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.982+0000 7fe594e00640 1 --2- 172.21.15.32:0/2934859828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe59010b900 0x7fe5901a37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:49.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.982+0000 7fe58f400640 1 --2- 172.21.15.32:0/2934859828 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe590075960 0x7fe59019c830 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe57c00e9c0 tx=0x7fe57c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:49.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.982+0000 7fe58cc00640 1 -- 172.21.15.32:0/2934859828 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57c00ccc0 con 0x7fe590075960 2024-04-30T08:35:49.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.982+0000 7fe58cc00640 1 -- 172.21.15.32:0/2934859828 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe57c0054d0 con 0x7fe590075960 2024-04-30T08:35:49.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.982+0000 7fe58cc00640 1 -- 172.21.15.32:0/2934859828 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57c010430 con 0x7fe590075960 2024-04-30T08:35:49.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.982+0000 7fe595800640 1 -- 172.21.15.32:0/2934859828 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5901a4120 con 0x7fe590075960 2024-04-30T08:35:49.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.983+0000 7fe595800640 1 -- 172.21.15.32:0/2934859828 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5901a46c0 con 0x7fe590075960 2024-04-30T08:35:49.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.984+0000 7fe595800640 1 -- 172.21.15.32:0/2934859828 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe59010aac0 con 0x7fe590075960 2024-04-30T08:35:49.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.987+0000 7fe58cc00640 1 -- 172.21.15.32:0/2934859828 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7fe57c004f80 con 0x7fe590075960 2024-04-30T08:35:49.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.989+0000 7fe58cc00640 1 --2- 172.21.15.32:0/2934859828 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe5740afbf0 0x7fe5740b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:49.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.989+0000 7fe58cc00640 1 -- 172.21.15.32:0/2934859828 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fe57c0df890 con 0x7fe590075960 2024-04-30T08:35:49.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.989+0000 7fe58fe00640 1 --2- 172.21.15.32:0/2934859828 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe5740afbf0 0x7fe5740b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:49.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.990+0000 7fe58cc00640 1 -- 172.21.15.32:0/2934859828 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe57c0a7f00 con 0x7fe590075960 2024-04-30T08:35:49.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:49.990+0000 7fe58fe00640 1 --2- 172.21.15.32:0/2934859828 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe5740afbf0 0x7fe5740b2100 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe58002a930 tx=0x7fe580030450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:50.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.241+0000 7fe595800640 1 -- 172.21.15.32:0/2934859828 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe59002dbd0 con 0x7fe590075960 2024-04-30T08:35:50.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.242+0000 7fe58cc00640 1 -- 172.21.15.32:0/2934859828 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11927 (secure 0 0 0) 0x7fe57c0a7800 con 0x7fe590075960 2024-04-30T08:35:50.244 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:35:50.244 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2235842676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2235842676},{"type":"v1","addr":"172.21.15.32:6835","nonce":2235842676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/3577009355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":3577009355},{"type":"v1","addr":"172.21.15.32:6841","nonce":3577009355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-30T08:31:52.236668+0000","modified":"2024-04-30T08:35:05.098418+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24541,"mds_3":24533,"mds_4":24517},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/463203038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":463203038},{"type":"v1","addr":"172.21.15.189:6833","nonce":463203038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6833/3635143591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3635143591},{"type":"v1","addr":"172.21.15.71:6833","nonce":3635143591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6837/2569449994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2569449994},{"type":"v1","addr":"172.21.15.32:6837","nonce":2569449994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/278686565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":278686565},{"type":"v1","addr":"172.21.15.189:6835","nonce":278686565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6835/1558303056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1558303056},{"type":"v1","addr":"172.21.15.71:6835","nonce":1558303056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.189:6837/686520873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":686520873},{"type":"v1","addr":"172.21.15.189:6837","nonce":686520873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.71:6837/2324039932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2324039932},{"type":"v1","addr":"172.21.15.71:6837","nonce":2324039932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6839/352970107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":352970107},{"type":"v1","addr":"172.21.15.189:6839","nonce":352970107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.32:6839/4160386524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4160386524},{"type":"v1","addr":"172.21.15.32:6839","nonce":4160386524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.71:6839/3517409993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3517409993},{"type":"v1","addr":"172.21.15.71:6839","nonce":3517409993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24517,24533,24560,24590,24541]},"id":1}]} 2024-04-30T08:35:50.244 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2024-04-30T08:35:50.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.245+0000 7fe595800640 1 -- 172.21.15.32:0/2934859828 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe5740afbf0 msgr2=0x7fe5740b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:50.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.245+0000 7fe595800640 1 --2- 172.21.15.32:0/2934859828 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe5740afbf0 0x7fe5740b2100 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fe58002a930 tx=0x7fe580030450 comp rx=0 tx=0).stop 2024-04-30T08:35:50.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.245+0000 7fe595800640 1 -- 172.21.15.32:0/2934859828 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe590075960 msgr2=0x7fe59019c830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:50.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.245+0000 7fe595800640 1 --2- 172.21.15.32:0/2934859828 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe590075960 0x7fe59019c830 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fe57c00e9c0 tx=0x7fe57c00ee80 comp rx=0 tx=0).stop 2024-04-30T08:35:50.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.245+0000 7fe595800640 1 -- 172.21.15.32:0/2934859828 shutdown_connections 2024-04-30T08:35:50.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.245+0000 7fe595800640 1 --2- 172.21.15.32:0/2934859828 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe5740afbf0 0x7fe5740b2100 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.245+0000 7fe595800640 1 --2- 172.21.15.32:0/2934859828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe59010b900 0x7fe5901a37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.245+0000 7fe595800640 1 --2- 172.21.15.32:0/2934859828 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe590075960 0x7fe59019c830 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.245+0000 7fe595800640 1 --2- 172.21.15.32:0/2934859828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe590075010 0x7fe59019c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.245+0000 7fe595800640 1 -- 172.21.15.32:0/2934859828 >> 172.21.15.32:0/2934859828 conn(0x7fe5900feba0 msgr2=0x7fe5900ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:50.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.245+0000 7fe595800640 1 -- 172.21.15.32:0/2934859828 shutdown_connections 2024-04-30T08:35:50.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.246+0000 7fe595800640 1 -- 172.21.15.32:0/2934859828 wait complete. 2024-04-30T08:35:50.277 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-30T08:35:50.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:50 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2934859828' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:35:50.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.563+0000 7f87e5000640 1 -- 172.21.15.32:0/1195910934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87e0105bb0 msgr2=0x7f87e01068b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:50.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.563+0000 7f87e5000640 1 --2- 172.21.15.32:0/1195910934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87e0105bb0 0x7f87e01068b0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f87d0009900 tx=0x7f87d002a3f0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.564+0000 7f87e5000640 1 -- 172.21.15.32:0/1195910934 shutdown_connections 2024-04-30T08:35:50.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.564+0000 7f87e5000640 1 --2- 172.21.15.32:0/1195910934 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87e010baf0 0x7f87e010df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.564+0000 7f87e5000640 1 --2- 172.21.15.32:0/1195910934 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87e0106df0 0x7f87e0107200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.564+0000 7f87e5000640 1 --2- 172.21.15.32:0/1195910934 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87e0105bb0 0x7f87e01068b0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.564+0000 7f87e5000640 1 -- 172.21.15.32:0/1195910934 >> 172.21.15.32:0/1195910934 conn(0x7f87e0100da0 msgr2=0x7f87e0103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:50.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.564+0000 7f87e5000640 1 -- 172.21.15.32:0/1195910934 shutdown_connections 2024-04-30T08:35:50.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.564+0000 7f87e5000640 1 -- 172.21.15.32:0/1195910934 wait complete. 2024-04-30T08:35:50.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.565+0000 7f87e5000640 1 Processor -- start 2024-04-30T08:35:50.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.565+0000 7f87e5000640 1 -- start start 2024-04-30T08:35:50.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.565+0000 7f87e5000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87e0105bb0 0x7f87e019c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.565+0000 7f87e5000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87e0106df0 0x7f87e019c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.565+0000 7f87e5000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87e010baf0 0x7f87e01a37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.566+0000 7f87e5000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f87e01116b0 con 0x7f87e0106df0 2024-04-30T08:35:50.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.566+0000 7f87e5000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f87e0111530 con 0x7f87e0105bb0 2024-04-30T08:35:50.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.566+0000 7f87e5000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f87e0111830 con 0x7f87e010baf0 2024-04-30T08:35:50.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.566+0000 7f87dea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87e0106df0 0x7f87e019c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.566+0000 7f87df400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87e0105bb0 0x7f87e019c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.566+0000 7f87dfe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87e010baf0 0x7f87e01a37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.566+0000 7f87df400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87e0105bb0 0x7f87e019c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:54620/0 (socket says 172.21.15.32:54620) 2024-04-30T08:35:50.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.566+0000 7f87dea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87e0106df0 0x7f87e019c820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53420/0 (socket says 172.21.15.32:53420) 2024-04-30T08:35:50.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.566+0000 7f87df400640 1 -- 172.21.15.32:0/1816830863 learned_addr learned my addr 172.21.15.32:0/1816830863 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:50.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.566+0000 7f87dea00640 1 -- 172.21.15.32:0/1816830863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87e010baf0 msgr2=0x7f87e01a37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:50.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.566+0000 7f87dea00640 1 --2- 172.21.15.32:0/1816830863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87e010baf0 0x7f87e01a37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.566+0000 7f87dea00640 1 -- 172.21.15.32:0/1816830863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87e0105bb0 msgr2=0x7f87e019c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:50.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.566+0000 7f87dea00640 1 --2- 172.21.15.32:0/1816830863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87e0105bb0 0x7f87e019c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.566+0000 7f87dea00640 1 -- 172.21.15.32:0/1816830863 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f87e01a3e20 con 0x7f87e0106df0 2024-04-30T08:35:50.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.566+0000 7f87df400640 1 --2- 172.21.15.32:0/1816830863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87e0105bb0 0x7f87e019c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:50.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.567+0000 7f87dea00640 1 --2- 172.21.15.32:0/1816830863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87e0106df0 0x7f87e019c820 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f87cc00e9f0 tx=0x7f87cc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:50.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.567+0000 7f87d7e00640 1 -- 172.21.15.32:0/1816830863 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87cc00ccc0 con 0x7f87e0106df0 2024-04-30T08:35:50.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.567+0000 7f87d7e00640 1 -- 172.21.15.32:0/1816830863 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87cc0108e0 con 0x7f87e0106df0 2024-04-30T08:35:50.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.567+0000 7f87d7e00640 1 -- 172.21.15.32:0/1816830863 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87cc018880 con 0x7f87e0106df0 2024-04-30T08:35:50.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.567+0000 7f87e5000640 1 -- 172.21.15.32:0/1816830863 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f87e01a4110 con 0x7f87e0106df0 2024-04-30T08:35:50.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.567+0000 7f87e5000640 1 -- 172.21.15.32:0/1816830863 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87e01a4610 con 0x7f87e0106df0 2024-04-30T08:35:50.573 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.571+0000 7f87ed628740 1 Processor -- start 2024-04-30T08:35:50.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.571+0000 7f87ed628740 1 -- start start 2024-04-30T08:35:50.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.571+0000 7f87d7e00640 1 -- 172.21.15.32:0/1816830863 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f87cc010430 con 0x7f87e0106df0 2024-04-30T08:35:50.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.571+0000 7f87ed628740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557bcf004390 0x557bcf3a5730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.571+0000 7f87ed628740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557bcefdb4d0 0x557bcf3a5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.571+0000 7f87ed628740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557bcefe36b0 0x557bcf3acbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.571+0000 7f87ed628740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x557bcf30ad50 con 0x557bcf004390 2024-04-30T08:35:50.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.571+0000 7f87ed628740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x557bcf2b00c0 con 0x557bcefe36b0 2024-04-30T08:35:50.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.571+0000 7f87ed628740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x557bcf306d10 con 0x557bcefdb4d0 2024-04-30T08:35:50.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.572+0000 7f87dfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557bcf004390 0x557bcf3a5730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.572+0000 7f87dfe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557bcf004390 0x557bcf3a5730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53424/0 (socket says 172.21.15.32:53424) 2024-04-30T08:35:50.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.572+0000 7f87dfe00640 1 -- 172.21.15.32:0/3074036400 learned_addr learned my addr 172.21.15.32:0/3074036400 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:50.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.572+0000 7f87dea00640 1 --2- 172.21.15.32:0/3074036400 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557bcefe36b0 0x557bcf3acbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.574 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.572+0000 7f87df400640 1 --2- 172.21.15.32:0/3074036400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557bcefdb4d0 0x557bcf3a5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.572+0000 7f87dfe00640 1 -- 172.21.15.32:0/3074036400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557bcefdb4d0 msgr2=0x557bcf3a5cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:50.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.572+0000 7f87dfe00640 1 --2- 172.21.15.32:0/3074036400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557bcefdb4d0 0x557bcf3a5cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.572+0000 7f87dfe00640 1 -- 172.21.15.32:0/3074036400 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557bcefe36b0 msgr2=0x557bcf3acbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:50.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.572+0000 7f87dfe00640 1 --2- 172.21.15.32:0/3074036400 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557bcefe36b0 0x557bcf3acbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.572+0000 7f87dfe00640 1 -- 172.21.15.32:0/3074036400 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bcf01c3f0 con 0x557bcf004390 2024-04-30T08:35:50.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.572+0000 7f87df400640 1 --2- 172.21.15.32:0/3074036400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557bcefdb4d0 0x557bcf3a5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:35:50.575 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.572+0000 7f87dea00640 1 --2- 172.21.15.32:0/3074036400 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557bcefe36b0 0x557bcf3acbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:35:50.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.573+0000 7f87dfe00640 1 --2- 172.21.15.32:0/3074036400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557bcf004390 0x557bcf3a5730 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f87d8009fd0 tx=0x7f87d80118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ece5bbd8098de87 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:50.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.573+0000 7f87d4c00640 1 -- 172.21.15.32:0/3074036400 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d8004d20 con 0x557bcf004390 2024-04-30T08:35:50.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.573+0000 7f87d4c00640 1 -- 172.21.15.32:0/3074036400 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87d8004050 con 0x557bcf004390 2024-04-30T08:35:50.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.573+0000 7f87d4c00640 1 -- 172.21.15.32:0/3074036400 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d8005220 con 0x557bcf004390 2024-04-30T08:35:50.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.573+0000 7f87d7e00640 1 --2- 172.21.15.32:0/1816830863 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f87c40afbf0 0x7f87c40b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.573+0000 7f87d7e00640 1 -- 172.21.15.32:0/1816830863 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f87cc0df450 con 0x7f87e0106df0 2024-04-30T08:35:50.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.573+0000 7f87df400640 1 --2- 172.21.15.32:0/1816830863 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f87c40afbf0 0x7f87c40b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.574+0000 7f87ed628740 1 -- 172.21.15.32:0/3074036400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557bcf004390 msgr2=0x557bcf3a5730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:50.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.574+0000 7f87ed628740 1 --2- 172.21.15.32:0/3074036400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557bcf004390 0x557bcf3a5730 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f87d8009fd0 tx=0x7f87d80118d0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.574+0000 7f87ed628740 1 -- 172.21.15.32:0/3074036400 shutdown_connections 2024-04-30T08:35:50.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.574+0000 7f87ed628740 1 --2- 172.21.15.32:0/3074036400 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557bcefe36b0 0x557bcf3acbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.574+0000 7f87ed628740 1 --2- 172.21.15.32:0/3074036400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557bcefdb4d0 0x557bcf3a5cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.574+0000 7f87ed628740 1 --2- 172.21.15.32:0/3074036400 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557bcf004390 0x557bcf3a5730 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.574+0000 7f87ed628740 1 -- 172.21.15.32:0/3074036400 >> 172.21.15.32:0/3074036400 conn(0x557bceff84e0 msgr2=0x557bcf3a10c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:50.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.574+0000 7f87ed628740 1 -- 172.21.15.32:0/3074036400 shutdown_connections 2024-04-30T08:35:50.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.574+0000 7f87ed628740 1 -- 172.21.15.32:0/3074036400 wait complete. 2024-04-30T08:35:50.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.574+0000 7f87df400640 1 --2- 172.21.15.32:0/1816830863 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f87c40afbf0 0x7f87c40b2100 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f87d0031500 tx=0x7f87d00046c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:50.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.575+0000 7f87ed628740 1 -- start start 2024-04-30T08:35:50.579 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.575+0000 7f87ed628740 1 Processor -- start 2024-04-30T08:35:50.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.575+0000 7f87ed628740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557bcefdb4d0 0x557bcf3af620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.575+0000 7f87ed628740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557bcefe36b0 0x557bcf3afb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.575+0000 7f87ed628740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557bcf004390 0x557bcf3b6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.575+0000 7f87ed628740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x557bcf3ad600 con 0x557bcefdb4d0 2024-04-30T08:35:50.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.575+0000 7f87ed628740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x557bcf2c3b50 con 0x557bcf004390 2024-04-30T08:35:50.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.575+0000 7f87ed628740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x557bcf3ad780 con 0x557bcefe36b0 2024-04-30T08:35:50.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.576+0000 7f87ed628740 10 client.0 _ll_register_callbacks cb 0x557bcf28d420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:35:50.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.576+0000 7f87df400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557bcefdb4d0 0x557bcf3af620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.576+0000 7f87df400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557bcefdb4d0 0x557bcf3af620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53430/0 (socket says 172.21.15.32:53430) 2024-04-30T08:35:50.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.576+0000 7f87df400640 1 -- 172.21.15.32:0/2565049124 learned_addr learned my addr 172.21.15.32:0/2565049124 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:50.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.576+0000 7f87dfe00640 1 --2- 172.21.15.32:0/2565049124 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557bcf004390 0x557bcf3b6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.576+0000 7f87dea00640 1 --2- 172.21.15.32:0/2565049124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557bcefe36b0 0x557bcf3afb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.576+0000 7f87df400640 1 -- 172.21.15.32:0/2565049124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557bcefe36b0 msgr2=0x557bcf3afb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:50.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.576+0000 7f87df400640 1 --2- 172.21.15.32:0/2565049124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557bcefe36b0 0x557bcf3afb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.576+0000 7f87df400640 1 -- 172.21.15.32:0/2565049124 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557bcf004390 msgr2=0x557bcf3b6ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:50.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.576+0000 7f87df400640 1 --2- 172.21.15.32:0/2565049124 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557bcf004390 0x557bcf3b6ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.576+0000 7f87df400640 1 -- 172.21.15.32:0/2565049124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bcf0e67d0 con 0x557bcefdb4d0 2024-04-30T08:35:50.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.577+0000 7f87df400640 1 --2- 172.21.15.32:0/2565049124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557bcefdb4d0 0x557bcf3af620 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f87d002a910 tx=0x7f87d0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0503966c493b2f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:50.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.577+0000 7f87c9600640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:35:50.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.577+0000 7f87c9600640 1 -- 172.21.15.32:0/2565049124 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d0035070 con 0x557bcefdb4d0 2024-04-30T08:35:50.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.577+0000 7f87c9600640 1 -- 172.21.15.32:0/2565049124 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f87d0003a00 con 0x557bcefdb4d0 2024-04-30T08:35:50.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.577+0000 7f87ed628740 1 -- 172.21.15.32:0/2565049124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87d8004050 con 0x557bcefdb4d0 2024-04-30T08:35:50.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.577+0000 7f87c9600640 1 -- 172.21.15.32:0/2565049124 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f87d0003c00 con 0x557bcefdb4d0 2024-04-30T08:35:50.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.577+0000 7f87ed628740 10 client.15597 fetch_fsmap learned FSMap version 28 2024-04-30T08:35:50.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.577+0000 7f87ed628740 1 -- 172.21.15.32:0/2565049124 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557bcf3b9a60 con 0x557bcefdb4d0 2024-04-30T08:35:50.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.578+0000 7f87c9600640 1 -- 172.21.15.32:0/2565049124 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87d8004050 con 0x557bcefdb4d0 2024-04-30T08:35:50.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.578+0000 7f87c9600640 1 -- 172.21.15.32:0/2565049124 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f87d0043030 con 0x557bcefdb4d0 2024-04-30T08:35:50.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.578+0000 7f87ed628740 10 client.15597 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:35:50.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.578+0000 7f87ed628740 10 client.15597 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-30T08:35:50.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.578+0000 7f87ed628740 20 client.15597 populate_metadata read hostname 'smithi032' 2024-04-30T08:35:50.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.578+0000 7f87ed628740 1 --2- 172.21.15.32:0/2565049124 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x557bcefc2f60 0x557bcf3bd160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.578+0000 7f87ed628740 4 client.15597 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:35:50.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.578+0000 7f87ed628740 1 -- 172.21.15.32:0/2565049124 --> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557bcf318960 con 0x557bcefc2f60 2024-04-30T08:35:50.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.578+0000 7f87dea00640 1 --2- 172.21.15.32:0/2565049124 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x557bcefc2f60 0x557bcf3bd160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.579+0000 7f87dea00640 1 --2- 172.21.15.32:0/2565049124 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x557bcefc2f60 0x557bcf3bd160 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e27629a46d7aafd5 server_cookie=67251427ca09bbf6 in_seq=0 out_seq=0 2024-04-30T08:35:50.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.579+0000 7f87c9600640 10 client.15597 ms_handle_connect on v2:172.21.15.189:6836/686520873 2024-04-30T08:35:50.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.580+0000 7f87c9600640 1 -- 172.21.15.32:0/2565049124 <== mds.4 v2:172.21.15.189:6836/686520873 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x557bcf318960 con 0x557bcefc2f60 2024-04-30T08:35:50.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.580+0000 7f87c9600640 10 client.15597 handle_command_reply: tid=0 2024-04-30T08:35:50.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.580+0000 7f87ed628740 1 client.15597 shutdown 2024-04-30T08:35:50.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.580+0000 7f87ed628740 1 -- 172.21.15.32:0/2565049124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557bcefdb4d0 msgr2=0x557bcf3af620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:50.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.580+0000 7f87ed628740 1 --2- 172.21.15.32:0/2565049124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557bcefdb4d0 0x557bcf3af620 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f87d002a910 tx=0x7f87d0009260 comp rx=0 tx=0).stop 2024-04-30T08:35:50.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.580+0000 7f87ed628740 1 -- 172.21.15.32:0/2565049124 shutdown_connections 2024-04-30T08:35:50.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.580+0000 7f87ed628740 1 --2- 172.21.15.32:0/2565049124 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x557bcefc2f60 0x557bcf3bd160 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.580+0000 7f87ed628740 1 --2- 172.21.15.32:0/2565049124 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557bcf004390 0x557bcf3b6ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.580+0000 7f87ed628740 1 --2- 172.21.15.32:0/2565049124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557bcefe36b0 0x557bcf3afb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.580+0000 7f87ed628740 1 --2- 172.21.15.32:0/2565049124 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557bcefdb4d0 0x557bcf3af620 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.580+0000 7f87c9600640 0 client.15597 ms_handle_reset on v2:172.21.15.189:6836/686520873 2024-04-30T08:35:50.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.580+0000 7f87ed628740 1 -- 172.21.15.32:0/2565049124 >> 172.21.15.32:0/2565049124 conn(0x557bceff84e0 msgr2=0x557bcf3a0100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:50.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.580+0000 7f87ed628740 1 -- 172.21.15.32:0/2565049124 shutdown_connections 2024-04-30T08:35:50.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.580+0000 7f87ed628740 1 -- 172.21.15.32:0/2565049124 wait complete. 2024-04-30T08:35:50.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.580+0000 7f87ed628740 20 client.15597 trim_cache size 0 max 16384 2024-04-30T08:35:50.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.595+0000 7f87ed628740 1 Processor -- start 2024-04-30T08:35:50.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.596+0000 7f87ed628740 1 -- start start 2024-04-30T08:35:50.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.596+0000 7f87ed628740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557bcefdb4d0 0x557bcf3af900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.596+0000 7f87ed628740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557bcefe36b0 0x557bcf3b6a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.596+0000 7f87ed628740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557bcf004390 0x557bcf3b6f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.596+0000 7f87ed628740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f87e01aec70 con 0x557bcefe36b0 2024-04-30T08:35:50.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.596+0000 7f87ed628740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f87e01b07b0 con 0x557bcefdb4d0 2024-04-30T08:35:50.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.596+0000 7f87ed628740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f87e01af340 con 0x557bcf004390 2024-04-30T08:35:50.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.596+0000 7f87df400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557bcefdb4d0 0x557bcf3af900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.596+0000 7f87dea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557bcefe36b0 0x557bcf3b6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.596+0000 7f87df400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557bcefdb4d0 0x557bcf3af900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:54638/0 (socket says 172.21.15.32:54638) 2024-04-30T08:35:50.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.596+0000 7f87df400640 1 -- 172.21.15.32:0/4154492855 learned_addr learned my addr 172.21.15.32:0/4154492855 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:50.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.596+0000 7f87dfe00640 1 --2- 172.21.15.32:0/4154492855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557bcf004390 0x557bcf3b6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.597+0000 7f87df400640 1 -- 172.21.15.32:0/4154492855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557bcf004390 msgr2=0x557bcf3b6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:50.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.597+0000 7f87df400640 1 --2- 172.21.15.32:0/4154492855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557bcf004390 0x557bcf3b6f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.597+0000 7f87df400640 1 -- 172.21.15.32:0/4154492855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557bcefe36b0 msgr2=0x557bcf3b6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:50.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.597+0000 7f87df400640 1 --2- 172.21.15.32:0/4154492855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557bcefe36b0 0x557bcf3b6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.597+0000 7f87df400640 1 -- 172.21.15.32:0/4154492855 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557bcf3a1530 con 0x557bcefdb4d0 2024-04-30T08:35:50.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.597+0000 7f87dea00640 1 --2- 172.21.15.32:0/4154492855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557bcefe36b0 0x557bcf3b6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:50.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.597+0000 7f87df400640 1 --2- 172.21.15.32:0/4154492855 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557bcefdb4d0 0x557bcf3af900 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f87d00316c0 tx=0x7f87d0009590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68ec78a6561985da server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:50.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.597+0000 7f87cbe00640 1 -- 172.21.15.32:0/4154492855 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d003c700 con 0x557bcefdb4d0 2024-04-30T08:35:50.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.597+0000 7f87cbe00640 1 -- 172.21.15.32:0/4154492855 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87d004b070 con 0x557bcefdb4d0 2024-04-30T08:35:50.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.597+0000 7f87cbe00640 1 -- 172.21.15.32:0/4154492855 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d00478d0 con 0x557bcefdb4d0 2024-04-30T08:35:50.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.598+0000 7f87ed628740 1 -- 172.21.15.32:0/4154492855 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557bcefdb4d0 msgr2=0x557bcf3af900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:50.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.598+0000 7f87ed628740 1 --2- 172.21.15.32:0/4154492855 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557bcefdb4d0 0x557bcf3af900 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f87d00316c0 tx=0x7f87d0009590 comp rx=0 tx=0).stop 2024-04-30T08:35:50.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.598+0000 7f87ed628740 1 -- 172.21.15.32:0/4154492855 shutdown_connections 2024-04-30T08:35:50.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.598+0000 7f87ed628740 1 --2- 172.21.15.32:0/4154492855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557bcf004390 0x557bcf3b6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.598+0000 7f87ed628740 1 --2- 172.21.15.32:0/4154492855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557bcefe36b0 0x557bcf3b6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.598+0000 7f87ed628740 1 --2- 172.21.15.32:0/4154492855 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557bcefdb4d0 0x557bcf3af900 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.598+0000 7f87ed628740 1 -- 172.21.15.32:0/4154492855 >> 172.21.15.32:0/4154492855 conn(0x557bceff84e0 msgr2=0x557bcf3af1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:50.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.598+0000 7f87ed628740 1 -- 172.21.15.32:0/4154492855 shutdown_connections 2024-04-30T08:35:50.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.598+0000 7f87ed628740 1 -- 172.21.15.32:0/4154492855 wait complete. 2024-04-30T08:35:50.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.599+0000 7f87ed628740 1 -- start start 2024-04-30T08:35:50.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.599+0000 7f87ed628740 1 Processor -- start 2024-04-30T08:35:50.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.599+0000 7f87ed628740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557bcefdb4d0 0x557bcf3b4440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.599+0000 7f87ed628740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557bcefe36b0 0x557bcf3b4980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.599+0000 7f87ed628740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557bcf004390 0x557bcf3b58b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.599+0000 7f87ed628740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f87e01b00e0 con 0x557bcefdb4d0 2024-04-30T08:35:50.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.599+0000 7f87ed628740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f87e01afa10 con 0x557bcf004390 2024-04-30T08:35:50.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.599+0000 7f87ed628740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f87e01acc30 con 0x557bcefe36b0 2024-04-30T08:35:50.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.599+0000 7f87ed628740 10 client.0 _ll_register_callbacks cb 0x7f87d80093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:35:50.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.600+0000 7f87df400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557bcefdb4d0 0x557bcf3b4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.600+0000 7f87dfe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557bcf004390 0x557bcf3b58b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.600+0000 7f87df400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557bcefdb4d0 0x557bcf3b4440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53442/0 (socket says 172.21.15.32:53442) 2024-04-30T08:35:50.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.600+0000 7f87df400640 1 -- 172.21.15.32:0/4230545384 learned_addr learned my addr 172.21.15.32:0/4230545384 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:50.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.600+0000 7f87dea00640 1 --2- 172.21.15.32:0/4230545384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557bcefe36b0 0x557bcf3b4980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.600+0000 7f87df400640 1 -- 172.21.15.32:0/4230545384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557bcefe36b0 msgr2=0x557bcf3b4980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:50.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.600+0000 7f87df400640 1 --2- 172.21.15.32:0/4230545384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557bcefe36b0 0x557bcf3b4980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.600+0000 7f87df400640 1 -- 172.21.15.32:0/4230545384 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557bcf004390 msgr2=0x557bcf3b58b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:50.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.600+0000 7f87df400640 1 --2- 172.21.15.32:0/4230545384 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557bcf004390 0x557bcf3b58b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.600+0000 7f87df400640 1 -- 172.21.15.32:0/4230545384 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557bcf3b5e70 con 0x557bcefdb4d0 2024-04-30T08:35:50.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.600+0000 7f87dea00640 1 --2- 172.21.15.32:0/4230545384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557bcefe36b0 0x557bcf3b4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:35:50.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.600+0000 7f87df400640 1 --2- 172.21.15.32:0/4230545384 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557bcefdb4d0 0x557bcf3b4440 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f87d0004650 tx=0x7f87d0003880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b06a196fc9b26c86 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:50.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.600+0000 7f87c9600640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:35:50.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.601+0000 7f87c9600640 1 -- 172.21.15.32:0/4230545384 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87d0061020 con 0x557bcefdb4d0 2024-04-30T08:35:50.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.601+0000 7f87c9600640 1 -- 172.21.15.32:0/4230545384 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f87d0047650 con 0x557bcefdb4d0 2024-04-30T08:35:50.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.601+0000 7f87ed628740 1 -- 172.21.15.32:0/4230545384 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f87d004b070 con 0x557bcefdb4d0 2024-04-30T08:35:50.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.601+0000 7f87c9600640 1 -- 172.21.15.32:0/4230545384 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f87d006c050 con 0x557bcefdb4d0 2024-04-30T08:35:50.606 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.601+0000 7f87ed628740 10 client.15603 fetch_fsmap learned FSMap version 28 2024-04-30T08:35:50.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.601+0000 7f87ed628740 1 -- 172.21.15.32:0/4230545384 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x557bcf3b70f0 con 0x557bcefdb4d0 2024-04-30T08:35:50.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.601+0000 7f87c9600640 1 -- 172.21.15.32:0/4230545384 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f87d004b070 con 0x557bcefdb4d0 2024-04-30T08:35:50.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.601+0000 7f87c9600640 1 -- 172.21.15.32:0/4230545384 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f87d005c070 con 0x557bcefdb4d0 2024-04-30T08:35:50.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.602+0000 7f87ed628740 10 client.15603 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:35:50.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.602+0000 7f87ed628740 10 client.15603 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-30T08:35:50.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.602+0000 7f87ed628740 20 client.15603 populate_metadata read hostname 'smithi032' 2024-04-30T08:35:50.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.602+0000 7f87ed628740 1 --2- 172.21.15.32:0/4230545384 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x557bcef983a0 0x557bcf427790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.602+0000 7f87ed628740 4 client.15603 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-30T08:35:50.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.602+0000 7f87ed628740 1 -- 172.21.15.32:0/4230545384 --> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557bcf3127d0 con 0x557bcef983a0 2024-04-30T08:35:50.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.602+0000 7f87dea00640 1 --2- 172.21.15.32:0/4230545384 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x557bcef983a0 0x557bcf427790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.603+0000 7f87dea00640 1 --2- 172.21.15.32:0/4230545384 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x557bcef983a0 0x557bcf427790 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c159dd206024a211 server_cookie=ff02104fc4f6c72 in_seq=0 out_seq=0 2024-04-30T08:35:50.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.603+0000 7f87c9600640 10 client.15603 ms_handle_connect on v2:172.21.15.189:6836/686520873 2024-04-30T08:35:50.607 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.603+0000 7f87c9600640 1 -- 172.21.15.32:0/4230545384 <== mds.4 v2:172.21.15.189:6836/686520873 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557bcf3127d0 con 0x557bcef983a0 2024-04-30T08:35:50.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.603+0000 7f87c9600640 10 client.15603 handle_command_reply: tid=0 2024-04-30T08:35:50.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.603+0000 7f87ed628740 1 client.15603 shutdown 2024-04-30T08:35:50.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.603+0000 7f87ed628740 1 -- 172.21.15.32:0/4230545384 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557bcefdb4d0 msgr2=0x557bcf3b4440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:50.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.603+0000 7f87ed628740 1 --2- 172.21.15.32:0/4230545384 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557bcefdb4d0 0x557bcf3b4440 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f87d0004650 tx=0x7f87d0003880 comp rx=0 tx=0).stop 2024-04-30T08:35:50.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.603+0000 7f87ed628740 1 -- 172.21.15.32:0/4230545384 shutdown_connections 2024-04-30T08:35:50.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.603+0000 7f87ed628740 1 --2- 172.21.15.32:0/4230545384 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x557bcef983a0 0x557bcf427790 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.603+0000 7f87ed628740 1 --2- 172.21.15.32:0/4230545384 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557bcf004390 0x557bcf3b58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.603+0000 7f87ed628740 1 --2- 172.21.15.32:0/4230545384 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557bcefe36b0 0x557bcf3b4980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.603+0000 7f87c9600640 0 client.15603 ms_handle_reset on v2:172.21.15.189:6836/686520873 2024-04-30T08:35:50.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.603+0000 7f87ed628740 1 --2- 172.21.15.32:0/4230545384 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557bcefdb4d0 0x557bcf3b4440 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.603+0000 7f87ed628740 1 -- 172.21.15.32:0/4230545384 >> 172.21.15.32:0/4230545384 conn(0x557bceff84e0 msgr2=0x557bcf3af160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:50.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.603+0000 7f87ed628740 1 -- 172.21.15.32:0/4230545384 shutdown_connections 2024-04-30T08:35:50.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.603+0000 7f87ed628740 1 -- 172.21.15.32:0/4230545384 wait complete. 2024-04-30T08:35:50.608 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.603+0000 7f87ed628740 20 client.15603 trim_cache size 0 max 16384 2024-04-30T08:35:50.609 INFO:teuthology.orchestra.run.smithi032.stdout:[] 2024-04-30T08:35:50.609 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.604+0000 7f87e5000640 1 -- 172.21.15.32:0/1816830863 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f87c40afbf0 msgr2=0x7f87c40b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:50.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.604+0000 7f87e5000640 1 --2- 172.21.15.32:0/1816830863 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f87c40afbf0 0x7f87c40b2100 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f87d0031500 tx=0x7f87d00046c0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.604+0000 7f87e5000640 1 -- 172.21.15.32:0/1816830863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87e0106df0 msgr2=0x7f87e019c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:50.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.604+0000 7f87e5000640 1 --2- 172.21.15.32:0/1816830863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87e0106df0 0x7f87e019c820 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f87cc00e9f0 tx=0x7f87cc00eeb0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.604+0000 7f87e5000640 1 -- 172.21.15.32:0/1816830863 shutdown_connections 2024-04-30T08:35:50.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.604+0000 7f87e5000640 1 --2- 172.21.15.32:0/1816830863 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f87c40afbf0 0x7f87c40b2100 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.604+0000 7f87e5000640 1 --2- 172.21.15.32:0/1816830863 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f87e010baf0 0x7f87e01a37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.604+0000 7f87e5000640 1 --2- 172.21.15.32:0/1816830863 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f87e0106df0 0x7f87e019c820 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.604+0000 7f87e5000640 1 --2- 172.21.15.32:0/1816830863 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f87e0105bb0 0x7f87e019c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.604+0000 7f87e5000640 1 -- 172.21.15.32:0/1816830863 >> 172.21.15.32:0/1816830863 conn(0x7f87e0100da0 msgr2=0x7f87e0107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:50.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.605+0000 7f87e5000640 1 -- 172.21.15.32:0/1816830863 shutdown_connections 2024-04-30T08:35:50.610 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.605+0000 7f87e5000640 1 -- 172.21.15.32:0/1816830863 wait complete. 2024-04-30T08:35:50.630 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:50 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2934859828' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:35:50.632 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-30T08:35:50.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:50 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2934859828' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:35:50.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.906+0000 7f3eb6800640 1 -- 172.21.15.32:0/886437637 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3eb010b3b0 msgr2=0x7f3eb010b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:50.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.906+0000 7f3eb6800640 1 --2- 172.21.15.32:0/886437637 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3eb010b3b0 0x7f3eb010b7c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3e98009900 tx=0x7f3e9802a3f0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.906+0000 7f3eb6800640 1 -- 172.21.15.32:0/886437637 shutdown_connections 2024-04-30T08:35:50.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.906+0000 7f3eb6800640 1 --2- 172.21.15.32:0/886437637 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3eb01100b0 0x7f3eb01124d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.906+0000 7f3eb6800640 1 --2- 172.21.15.32:0/886437637 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3eb010b3b0 0x7f3eb010b7c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.906+0000 7f3eb6800640 1 --2- 172.21.15.32:0/886437637 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3eb010a450 0x7f3eb010ae70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.906+0000 7f3eb6800640 1 -- 172.21.15.32:0/886437637 >> 172.21.15.32:0/886437637 conn(0x7f3eb0078e60 msgr2=0x7f3eb007b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:50.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.906+0000 7f3eb6800640 1 -- 172.21.15.32:0/886437637 shutdown_connections 2024-04-30T08:35:50.908 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.906+0000 7f3eb6800640 1 -- 172.21.15.32:0/886437637 wait complete. 2024-04-30T08:35:50.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.907+0000 7f3eb6800640 1 Processor -- start 2024-04-30T08:35:50.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.907+0000 7f3eb6800640 1 -- start start 2024-04-30T08:35:50.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.908+0000 7f3eb6800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3eb010a450 0x7f3eb01a08f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.908+0000 7f3eb6800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3eb010b3b0 0x7f3eb01a0e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.908+0000 7f3eb6800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3eb01100b0 0x7f3eb01a7d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.908+0000 7f3eb6800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb0115ce0 con 0x7f3eb01100b0 2024-04-30T08:35:50.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.908+0000 7f3eb6800640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb0115b60 con 0x7f3eb010b3b0 2024-04-30T08:35:50.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.908+0000 7f3eb6800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb0115e60 con 0x7f3eb010a450 2024-04-30T08:35:50.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.908+0000 7f3eafe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3eb01100b0 0x7f3eb01a7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.908+0000 7f3eafe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3eb01100b0 0x7f3eb01a7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53462/0 (socket says 172.21.15.32:53462) 2024-04-30T08:35:50.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.908+0000 7f3eafe00640 1 -- 172.21.15.32:0/621926564 learned_addr learned my addr 172.21.15.32:0/621926564 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:50.910 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.908+0000 7f3ea7e00640 1 --2- 172.21.15.32:0/621926564 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3eb010b3b0 0x7f3eb01a0e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.908+0000 7f3eaf400640 1 --2- 172.21.15.32:0/621926564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3eb010a450 0x7f3eb01a08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.908+0000 7f3eafe00640 1 -- 172.21.15.32:0/621926564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3eb010a450 msgr2=0x7f3eb01a08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:50.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.908+0000 7f3eafe00640 1 --2- 172.21.15.32:0/621926564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3eb010a450 0x7f3eb01a08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.908+0000 7f3eafe00640 1 -- 172.21.15.32:0/621926564 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3eb010b3b0 msgr2=0x7f3eb01a0e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:50.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.908+0000 7f3eafe00640 1 --2- 172.21.15.32:0/621926564 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3eb010b3b0 0x7f3eb01a0e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.911 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.908+0000 7f3eafe00640 1 -- 172.21.15.32:0/621926564 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3eb01a83c0 con 0x7f3eb01100b0 2024-04-30T08:35:50.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.909+0000 7f3eafe00640 1 --2- 172.21.15.32:0/621926564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3eb01100b0 0x7f3eb01a7d50 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3ea800eb10 tx=0x7f3ea800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:50.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.909+0000 7f3eacc00640 1 -- 172.21.15.32:0/621926564 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea8004b20 con 0x7f3eb01100b0 2024-04-30T08:35:50.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.909+0000 7f3eacc00640 1 -- 172.21.15.32:0/621926564 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ea8004cc0 con 0x7f3eb01100b0 2024-04-30T08:35:50.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.909+0000 7f3eacc00640 1 -- 172.21.15.32:0/621926564 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea8010670 con 0x7f3eb01100b0 2024-04-30T08:35:50.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.909+0000 7f3eb6800640 1 -- 172.21.15.32:0/621926564 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3eb01a86b0 con 0x7f3eb01100b0 2024-04-30T08:35:50.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.909+0000 7f3eb6800640 1 -- 172.21.15.32:0/621926564 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3eb01a8b80 con 0x7f3eb01100b0 2024-04-30T08:35:50.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.913+0000 7f3ebd764740 1 Processor -- start 2024-04-30T08:35:50.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.913+0000 7f3eacc00640 1 -- 172.21.15.32:0/621926564 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f3ea8005020 con 0x7f3eb01100b0 2024-04-30T08:35:50.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.914+0000 7f3ebd764740 1 -- start start 2024-04-30T08:35:50.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.914+0000 7f3ebd764740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560b19bd9390 0x560b19f7a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.914+0000 7f3ebd764740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b19bb04d0 0x560b19f7aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.914+0000 7f3ebd764740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b19bb86b0 0x560b19f819f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.914+0000 7f3ebd764740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560b19edfd50 con 0x560b19bb04d0 2024-04-30T08:35:50.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.914+0000 7f3ebd764740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x560b19e850c0 con 0x560b19bd9390 2024-04-30T08:35:50.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.914+0000 7f3ebd764740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560b19edbd10 con 0x560b19bb86b0 2024-04-30T08:35:50.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.914+0000 7f3eaf400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b19bb04d0 0x560b19f7aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.923 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.914+0000 7f3ea7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b19bb86b0 0x560b19f819f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.914+0000 7f3eaf400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b19bb04d0 0x560b19f7aaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53470/0 (socket says 172.21.15.32:53470) 2024-04-30T08:35:50.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.914+0000 7f3eaf400640 1 -- 172.21.15.32:0/3703881034 learned_addr learned my addr 172.21.15.32:0/3703881034 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:50.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.915+0000 7f3ea7e00640 1 --2- 172.21.15.32:0/3703881034 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560b19bd9390 0x560b19f7a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.915+0000 7f3eaf400640 1 -- 172.21.15.32:0/3703881034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b19bb86b0 msgr2=0x560b19f819f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:50.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.915+0000 7f3eaf400640 1 --2- 172.21.15.32:0/3703881034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b19bb86b0 0x560b19f819f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.915+0000 7f3eaf400640 1 -- 172.21.15.32:0/3703881034 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560b19bd9390 msgr2=0x560b19f7a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:50.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.915+0000 7f3eaf400640 1 --2- 172.21.15.32:0/3703881034 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560b19bd9390 0x560b19f7a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.915+0000 7f3eaf400640 1 -- 172.21.15.32:0/3703881034 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b19bf13f0 con 0x560b19bb04d0 2024-04-30T08:35:50.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.915+0000 7f3eaf400640 1 --2- 172.21.15.32:0/3703881034 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b19bb04d0 0x560b19f7aaf0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f3ea000bcd0 tx=0x7f3ea00107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdf57d270803f250 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:50.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.916+0000 7f3eacc00640 1 --2- 172.21.15.32:0/621926564 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3e940afb80 0x7f3e940b2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.916+0000 7f3ea4c00640 1 -- 172.21.15.32:0/3703881034 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea0004020 con 0x560b19bb04d0 2024-04-30T08:35:50.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.916+0000 7f3ea4c00640 1 -- 172.21.15.32:0/3703881034 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ea00041c0 con 0x560b19bb04d0 2024-04-30T08:35:50.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.916+0000 7f3ea4c00640 1 -- 172.21.15.32:0/3703881034 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea0011760 con 0x560b19bb04d0 2024-04-30T08:35:50.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.916+0000 7f3eacc00640 1 -- 172.21.15.32:0/621926564 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f3ea80df740 con 0x7f3eb01100b0 2024-04-30T08:35:50.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.916+0000 7f3ea7e00640 1 --2- 172.21.15.32:0/621926564 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3e940afb80 0x7f3e940b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.917+0000 7f3ebd764740 1 -- 172.21.15.32:0/3703881034 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b19bb04d0 msgr2=0x560b19f7aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:50.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.917+0000 7f3ebd764740 1 --2- 172.21.15.32:0/3703881034 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b19bb04d0 0x560b19f7aaf0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f3ea000bcd0 tx=0x7f3ea00107a0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.917+0000 7f3ebd764740 1 -- 172.21.15.32:0/3703881034 shutdown_connections 2024-04-30T08:35:50.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.917+0000 7f3ebd764740 1 --2- 172.21.15.32:0/3703881034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b19bb86b0 0x560b19f819f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.917+0000 7f3ea7e00640 1 --2- 172.21.15.32:0/621926564 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3e940afb80 0x7f3e940b2090 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3e98004770 tx=0x7f3e9802a3f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:50.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.917+0000 7f3ebd764740 1 --2- 172.21.15.32:0/3703881034 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b19bb04d0 0x560b19f7aaf0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.917+0000 7f3ebd764740 1 --2- 172.21.15.32:0/3703881034 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560b19bd9390 0x560b19f7a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.917+0000 7f3ebd764740 1 -- 172.21.15.32:0/3703881034 >> 172.21.15.32:0/3703881034 conn(0x560b19bcd4e0 msgr2=0x560b19f75f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:50.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.917+0000 7f3ebd764740 1 -- 172.21.15.32:0/3703881034 shutdown_connections 2024-04-30T08:35:50.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.917+0000 7f3ebd764740 1 -- 172.21.15.32:0/3703881034 wait complete. 2024-04-30T08:35:50.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.918+0000 7f3ebd764740 1 -- start start 2024-04-30T08:35:50.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.918+0000 7f3ebd764740 1 Processor -- start 2024-04-30T08:35:50.925 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.918+0000 7f3ebd764740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b19bb04d0 0x560b19f843d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.918+0000 7f3ebd764740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560b19bb86b0 0x560b19f84910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.918+0000 7f3ebd764740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b19bd9390 0x560b19f8b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.918+0000 7f3ebd764740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560b19f82440 con 0x560b19bb04d0 2024-04-30T08:35:50.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.918+0000 7f3ebd764740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x560b19e98b50 con 0x560b19bb86b0 2024-04-30T08:35:50.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.918+0000 7f3ebd764740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560b19f825c0 con 0x560b19bd9390 2024-04-30T08:35:50.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.918+0000 7f3ebd764740 10 client.0 _ll_register_callbacks cb 0x560b19e62420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:35:50.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.919+0000 7f3eaf400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b19bb04d0 0x560b19f843d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.919+0000 7f3eaf400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b19bb04d0 0x560b19f843d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53472/0 (socket says 172.21.15.32:53472) 2024-04-30T08:35:50.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.919+0000 7f3eaf400640 1 -- 172.21.15.32:0/3722798176 learned_addr learned my addr 172.21.15.32:0/3722798176 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:50.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.919+0000 7f3ea7e00640 1 --2- 172.21.15.32:0/3722798176 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560b19bb86b0 0x560b19f84910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.919+0000 7f3eafe00640 1 --2- 172.21.15.32:0/3722798176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b19bd9390 0x560b19f8b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.919+0000 7f3eaf400640 1 -- 172.21.15.32:0/3722798176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b19bd9390 msgr2=0x560b19f8b870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:50.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.919+0000 7f3eaf400640 1 --2- 172.21.15.32:0/3722798176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b19bd9390 0x560b19f8b870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.919+0000 7f3eaf400640 1 -- 172.21.15.32:0/3722798176 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560b19bb86b0 msgr2=0x560b19f84910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:50.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.919+0000 7f3eaf400640 1 --2- 172.21.15.32:0/3722798176 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560b19bb86b0 0x560b19f84910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.919+0000 7f3eaf400640 1 -- 172.21.15.32:0/3722798176 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b19cbb7d0 con 0x560b19bb04d0 2024-04-30T08:35:50.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.919+0000 7f3eaf400640 1 --2- 172.21.15.32:0/3722798176 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b19bb04d0 0x560b19f843d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0003e10 tx=0x7f3ea000b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5648bc38298cbfa server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:50.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.919+0000 7f3e9d600640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:35:50.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.920+0000 7f3e9d600640 1 -- 172.21.15.32:0/3722798176 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea00035f0 con 0x560b19bb04d0 2024-04-30T08:35:50.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.920+0000 7f3e9d600640 1 -- 172.21.15.32:0/3722798176 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f3ea0003790 con 0x560b19bb04d0 2024-04-30T08:35:50.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.920+0000 7f3ebd764740 1 -- 172.21.15.32:0/3722798176 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3ea00041c0 con 0x560b19bb04d0 2024-04-30T08:35:50.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.920+0000 7f3e9d600640 1 -- 172.21.15.32:0/3722798176 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3ea0003990 con 0x560b19bb04d0 2024-04-30T08:35:50.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.920+0000 7f3ebd764740 10 client.15615 fetch_fsmap learned FSMap version 28 2024-04-30T08:35:50.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.920+0000 7f3ebd764740 1 -- 172.21.15.32:0/3722798176 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560b19f8e7a0 con 0x560b19bb04d0 2024-04-30T08:35:50.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.920+0000 7f3e9d600640 1 -- 172.21.15.32:0/3722798176 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3ea00041c0 con 0x560b19bb04d0 2024-04-30T08:35:50.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.920+0000 7f3e9d600640 1 -- 172.21.15.32:0/3722798176 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f3ea0004a60 con 0x560b19bb04d0 2024-04-30T08:35:50.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.921+0000 7f3ebd764740 10 client.15615 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:35:50.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.921+0000 7f3ebd764740 10 client.15615 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-30T08:35:50.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.921+0000 7f3ebd764740 20 client.15615 populate_metadata read hostname 'smithi032' 2024-04-30T08:35:50.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.921+0000 7f3ebd764740 1 --2- 172.21.15.32:0/3722798176 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x560b19b97f60 0x560b19f91c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.921+0000 7f3ebd764740 4 client.15615 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:35:50.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.921+0000 7f3ebd764740 1 -- 172.21.15.32:0/3722798176 --> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560b19eed960 con 0x560b19b97f60 2024-04-30T08:35:50.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.921+0000 7f3ea7e00640 1 --2- 172.21.15.32:0/3722798176 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x560b19b97f60 0x560b19f91c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.921+0000 7f3ea7e00640 1 --2- 172.21.15.32:0/3722798176 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x560b19b97f60 0x560b19f91c50 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=d3391e4ac9068bd5 server_cookie=446598edc2a109e8 in_seq=0 out_seq=0 2024-04-30T08:35:50.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.921+0000 7f3e9d600640 10 client.15615 ms_handle_connect on v2:172.21.15.71:6836/2324039932 2024-04-30T08:35:50.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.922+0000 7f3e9d600640 1 -- 172.21.15.32:0/3722798176 <== mds.3 v2:172.21.15.71:6836/2324039932 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560b19eed960 con 0x560b19b97f60 2024-04-30T08:35:50.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.922+0000 7f3e9d600640 10 client.15615 handle_command_reply: tid=0 2024-04-30T08:35:50.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.923+0000 7f3ebd764740 1 client.15615 shutdown 2024-04-30T08:35:50.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.923+0000 7f3ebd764740 1 -- 172.21.15.32:0/3722798176 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b19bb04d0 msgr2=0x560b19f843d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:50.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.923+0000 7f3ebd764740 1 --2- 172.21.15.32:0/3722798176 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b19bb04d0 0x560b19f843d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f3ea0003e10 tx=0x7f3ea000b930 comp rx=0 tx=0).stop 2024-04-30T08:35:50.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.923+0000 7f3ebd764740 1 -- 172.21.15.32:0/3722798176 shutdown_connections 2024-04-30T08:35:50.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.923+0000 7f3ebd764740 1 --2- 172.21.15.32:0/3722798176 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x560b19b97f60 0x560b19f91c50 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.923+0000 7f3ebd764740 1 --2- 172.21.15.32:0/3722798176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b19bd9390 0x560b19f8b870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.923+0000 7f3ebd764740 1 --2- 172.21.15.32:0/3722798176 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560b19bb86b0 0x560b19f84910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.923+0000 7f3ebd764740 1 --2- 172.21.15.32:0/3722798176 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b19bb04d0 0x560b19f843d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.923+0000 7f3ebd764740 1 -- 172.21.15.32:0/3722798176 >> 172.21.15.32:0/3722798176 conn(0x560b19bcd4e0 msgr2=0x560b19f74f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:50.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.923+0000 7f3e9d600640 0 client.15615 ms_handle_reset on v2:172.21.15.71:6836/2324039932 2024-04-30T08:35:50.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.923+0000 7f3ebd764740 1 -- 172.21.15.32:0/3722798176 shutdown_connections 2024-04-30T08:35:50.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.923+0000 7f3ebd764740 1 -- 172.21.15.32:0/3722798176 wait complete. 2024-04-30T08:35:50.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.923+0000 7f3ebd764740 20 client.15615 trim_cache size 0 max 16384 2024-04-30T08:35:50.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.939+0000 7f3ebd764740 1 Processor -- start 2024-04-30T08:35:50.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.939+0000 7f3ebd764740 1 -- start start 2024-04-30T08:35:50.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.939+0000 7f3ebd764740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b19bb04d0 0x560b19f82a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.939+0000 7f3ebd764740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b19bb86b0 0x560b19f8b790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.939+0000 7f3ebd764740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560b19bd9390 0x560b19f82f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.939+0000 7f3ebd764740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb01b3170 con 0x560b19bb04d0 2024-04-30T08:35:50.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.939+0000 7f3ebd764740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb01b4cb0 con 0x560b19bd9390 2024-04-30T08:35:50.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.939+0000 7f3ebd764740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb01b3840 con 0x560b19bb86b0 2024-04-30T08:35:50.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.939+0000 7f3ea7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b19bb86b0 0x560b19f8b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.939+0000 7f3eaf400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b19bb04d0 0x560b19f82a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.939+0000 7f3ea7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b19bb86b0 0x560b19f8b790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:40504/0 (socket says 172.21.15.32:40504) 2024-04-30T08:35:50.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.939+0000 7f3ea7e00640 1 -- 172.21.15.32:0/1866965560 learned_addr learned my addr 172.21.15.32:0/1866965560 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:50.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.940+0000 7f3eafe00640 1 --2- 172.21.15.32:0/1866965560 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560b19bd9390 0x560b19f82f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.940+0000 7f3ea7e00640 1 -- 172.21.15.32:0/1866965560 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560b19bd9390 msgr2=0x560b19f82f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:50.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.940+0000 7f3ea7e00640 1 --2- 172.21.15.32:0/1866965560 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560b19bd9390 0x560b19f82f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.940+0000 7f3ea7e00640 1 -- 172.21.15.32:0/1866965560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b19bb04d0 msgr2=0x560b19f82a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:50.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.940+0000 7f3ea7e00640 1 --2- 172.21.15.32:0/1866965560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b19bb04d0 0x560b19f82a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.940+0000 7f3ea7e00640 1 -- 172.21.15.32:0/1866965560 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560b19f71710 con 0x560b19bb86b0 2024-04-30T08:35:50.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.940+0000 7f3ea7e00640 1 --2- 172.21.15.32:0/1866965560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b19bb86b0 0x560b19f8b790 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3e98009230 tx=0x7f3e980314a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1b138656d19b5d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:50.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.940+0000 7f3e9fe00640 1 -- 172.21.15.32:0/1866965560 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e98035050 con 0x560b19bb86b0 2024-04-30T08:35:50.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.941+0000 7f3e9fe00640 1 -- 172.21.15.32:0/1866965560 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e9805c070 con 0x560b19bb86b0 2024-04-30T08:35:50.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.941+0000 7f3e9fe00640 1 -- 172.21.15.32:0/1866965560 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e98031800 con 0x560b19bb86b0 2024-04-30T08:35:50.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.941+0000 7f3ebd764740 1 -- 172.21.15.32:0/1866965560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b19bb86b0 msgr2=0x560b19f8b790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:50.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.941+0000 7f3ebd764740 1 --2- 172.21.15.32:0/1866965560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b19bb86b0 0x560b19f8b790 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3e98009230 tx=0x7f3e980314a0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.941+0000 7f3ebd764740 1 -- 172.21.15.32:0/1866965560 shutdown_connections 2024-04-30T08:35:50.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.941+0000 7f3ebd764740 1 --2- 172.21.15.32:0/1866965560 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560b19bd9390 0x560b19f82f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.941+0000 7f3ebd764740 1 --2- 172.21.15.32:0/1866965560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b19bb86b0 0x560b19f8b790 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.941+0000 7f3ebd764740 1 --2- 172.21.15.32:0/1866965560 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b19bb04d0 0x560b19f82a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.941+0000 7f3ebd764740 1 -- 172.21.15.32:0/1866965560 >> 172.21.15.32:0/1866965560 conn(0x560b19bcd4e0 msgr2=0x560b19f74f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:50.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.942+0000 7f3ebd764740 1 -- 172.21.15.32:0/1866965560 shutdown_connections 2024-04-30T08:35:50.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.942+0000 7f3ebd764740 1 -- 172.21.15.32:0/1866965560 wait complete. 2024-04-30T08:35:50.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.942+0000 7f3ebd764740 1 -- start start 2024-04-30T08:35:50.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.942+0000 7f3ebd764740 1 Processor -- start 2024-04-30T08:35:50.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.943+0000 7f3ebd764740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560b19bb04d0 0x560b19f88a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.943+0000 7f3ebd764740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b19bb86b0 0x560b19f88fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.943+0000 7f3ebd764740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b19bd9390 0x560b19f89ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.943+0000 7f3ebd764740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb01b45e0 con 0x560b19bd9390 2024-04-30T08:35:50.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.943+0000 7f3ebd764740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb01b3f10 con 0x560b19bb04d0 2024-04-30T08:35:50.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.943+0000 7f3ebd764740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3eb01b1130 con 0x560b19bb86b0 2024-04-30T08:35:50.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.943+0000 7f3ebd764740 10 client.0 _ll_register_callbacks cb 0x7f3e98004230 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:35:50.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.943+0000 7f3eafe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b19bd9390 0x560b19f89ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.943+0000 7f3eafe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b19bd9390 0x560b19f89ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53488/0 (socket says 172.21.15.32:53488) 2024-04-30T08:35:50.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.943+0000 7f3eafe00640 1 -- 172.21.15.32:0/1959269875 learned_addr learned my addr 172.21.15.32:0/1959269875 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:50.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.943+0000 7f3eaf400640 1 --2- 172.21.15.32:0/1959269875 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560b19bb04d0 0x560b19f88a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.943+0000 7f3ea7e00640 1 --2- 172.21.15.32:0/1959269875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b19bb86b0 0x560b19f88fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.943+0000 7f3eafe00640 1 -- 172.21.15.32:0/1959269875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b19bb86b0 msgr2=0x560b19f88fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:50.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.943+0000 7f3eafe00640 1 --2- 172.21.15.32:0/1959269875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b19bb86b0 0x560b19f88fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.943+0000 7f3eafe00640 1 -- 172.21.15.32:0/1959269875 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560b19bb04d0 msgr2=0x560b19f88a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:50.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.944+0000 7f3eafe00640 1 --2- 172.21.15.32:0/1959269875 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560b19bb04d0 0x560b19f88a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.944+0000 7f3eafe00640 1 -- 172.21.15.32:0/1959269875 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560b19f8a4b0 con 0x560b19bd9390 2024-04-30T08:35:50.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.944+0000 7f3eaf400640 1 --2- 172.21.15.32:0/1959269875 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560b19bb04d0 0x560b19f88a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:50.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.944+0000 7f3eafe00640 1 --2- 172.21.15.32:0/1959269875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b19bd9390 0x560b19f89ef0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3ea80dcc80 tx=0x7f3ea80a7000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a94deafbcfe86ec server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:50.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.944+0000 7f3e9d600640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:35:50.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.944+0000 7f3e9d600640 1 -- 172.21.15.32:0/1959269875 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ea80a7db0 con 0x560b19bd9390 2024-04-30T08:35:50.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.944+0000 7f3e9d600640 1 -- 172.21.15.32:0/1959269875 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f3ea80a7910 con 0x560b19bd9390 2024-04-30T08:35:50.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.944+0000 7f3ebd764740 1 -- 172.21.15.32:0/1959269875 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e9805c070 con 0x560b19bd9390 2024-04-30T08:35:50.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.944+0000 7f3e9d600640 1 -- 172.21.15.32:0/1959269875 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3ea80a76b0 con 0x560b19bd9390 2024-04-30T08:35:50.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.944+0000 7f3ebd764740 10 client.15621 fetch_fsmap learned FSMap version 28 2024-04-30T08:35:50.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.944+0000 7f3ebd764740 1 -- 172.21.15.32:0/1959269875 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560b19f8b830 con 0x560b19bd9390 2024-04-30T08:35:50.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.945+0000 7f3e9d600640 1 -- 172.21.15.32:0/1959269875 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3e9805c070 con 0x560b19bd9390 2024-04-30T08:35:50.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.945+0000 7f3e9d600640 1 -- 172.21.15.32:0/1959269875 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f3ea80de040 con 0x560b19bd9390 2024-04-30T08:35:50.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.945+0000 7f3ebd764740 10 client.15621 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:35:50.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.945+0000 7f3ebd764740 10 client.15621 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-30T08:35:50.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.945+0000 7f3ebd764740 20 client.15621 populate_metadata read hostname 'smithi032' 2024-04-30T08:35:50.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.945+0000 7f3ebd764740 1 --2- 172.21.15.32:0/1959269875 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x560b19b6d3a0 0x560b19ffc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:50.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.945+0000 7f3ebd764740 4 client.15621 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-30T08:35:50.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.945+0000 7f3ebd764740 1 -- 172.21.15.32:0/1959269875 --> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560b19ee77d0 con 0x560b19b6d3a0 2024-04-30T08:35:50.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.945+0000 7f3eaf400640 1 --2- 172.21.15.32:0/1959269875 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x560b19b6d3a0 0x560b19ffc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:50.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.946+0000 7f3eaf400640 1 --2- 172.21.15.32:0/1959269875 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x560b19b6d3a0 0x560b19ffc4e0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=6f6278a596354ad6 server_cookie=13c26554300b9c33 in_seq=0 out_seq=0 2024-04-30T08:35:50.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.946+0000 7f3e9d600640 10 client.15621 ms_handle_connect on v2:172.21.15.71:6836/2324039932 2024-04-30T08:35:50.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.946+0000 7f3e9d600640 1 -- 172.21.15.32:0/1959269875 <== mds.3 v2:172.21.15.71:6836/2324039932 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560b19ee77d0 con 0x560b19b6d3a0 2024-04-30T08:35:50.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.946+0000 7f3e9d600640 10 client.15621 handle_command_reply: tid=0 2024-04-30T08:35:50.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.946+0000 7f3ebd764740 1 client.15621 shutdown 2024-04-30T08:35:50.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.946+0000 7f3ebd764740 1 -- 172.21.15.32:0/1959269875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b19bd9390 msgr2=0x560b19f89ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:50.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.946+0000 7f3ebd764740 1 --2- 172.21.15.32:0/1959269875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b19bd9390 0x560b19f89ef0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f3ea80dcc80 tx=0x7f3ea80a7000 comp rx=0 tx=0).stop 2024-04-30T08:35:50.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.946+0000 7f3ebd764740 1 -- 172.21.15.32:0/1959269875 shutdown_connections 2024-04-30T08:35:50.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.946+0000 7f3ebd764740 1 --2- 172.21.15.32:0/1959269875 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x560b19b6d3a0 0x560b19ffc4e0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.946+0000 7f3ebd764740 1 --2- 172.21.15.32:0/1959269875 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560b19bd9390 0x560b19f89ef0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.946+0000 7f3ebd764740 1 --2- 172.21.15.32:0/1959269875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560b19bb86b0 0x560b19f88fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.946+0000 7f3ebd764740 1 --2- 172.21.15.32:0/1959269875 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560b19bb04d0 0x560b19f88a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.946+0000 7f3ebd764740 1 -- 172.21.15.32:0/1959269875 >> 172.21.15.32:0/1959269875 conn(0x560b19bcd4e0 msgr2=0x560b19f74f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:50.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.946+0000 7f3e9d600640 0 client.15621 ms_handle_reset on v2:172.21.15.71:6836/2324039932 2024-04-30T08:35:50.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.947+0000 7f3ebd764740 1 -- 172.21.15.32:0/1959269875 shutdown_connections 2024-04-30T08:35:50.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.947+0000 7f3ebd764740 1 -- 172.21.15.32:0/1959269875 wait complete. 2024-04-30T08:35:50.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.947+0000 7f3ebd764740 20 client.15621 trim_cache size 0 max 16384 2024-04-30T08:35:50.952 INFO:teuthology.orchestra.run.smithi032.stdout:[] 2024-04-30T08:35:50.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.947+0000 7f3eb6800640 1 -- 172.21.15.32:0/621926564 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3e940afb80 msgr2=0x7f3e940b2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:50.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.947+0000 7f3eb6800640 1 --2- 172.21.15.32:0/621926564 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3e940afb80 0x7f3e940b2090 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f3e98004770 tx=0x7f3e9802a3f0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.947+0000 7f3eb6800640 1 -- 172.21.15.32:0/621926564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3eb01100b0 msgr2=0x7f3eb01a7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:50.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.947+0000 7f3eb6800640 1 --2- 172.21.15.32:0/621926564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3eb01100b0 0x7f3eb01a7d50 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3ea800eb10 tx=0x7f3ea800c2d0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.947+0000 7f3eb6800640 1 -- 172.21.15.32:0/621926564 shutdown_connections 2024-04-30T08:35:50.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.947+0000 7f3eb6800640 1 --2- 172.21.15.32:0/621926564 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f3e940afb80 0x7f3e940b2090 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.948+0000 7f3eb6800640 1 --2- 172.21.15.32:0/621926564 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3eb01100b0 0x7f3eb01a7d50 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.948+0000 7f3eb6800640 1 --2- 172.21.15.32:0/621926564 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3eb010b3b0 0x7f3eb01a0e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.948+0000 7f3eb6800640 1 --2- 172.21.15.32:0/621926564 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3eb010a450 0x7f3eb01a08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:50.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.948+0000 7f3eb6800640 1 -- 172.21.15.32:0/621926564 >> 172.21.15.32:0/621926564 conn(0x7f3eb0078e60 msgr2=0x7f3eb010c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:50.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.948+0000 7f3eb6800640 1 -- 172.21.15.32:0/621926564 shutdown_connections 2024-04-30T08:35:50.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:50.948+0000 7f3eb6800640 1 -- 172.21.15.32:0/621926564 wait complete. 2024-04-30T08:35:50.976 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-30T08:35:51.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.259+0000 7f8eb3e00640 1 -- 172.21.15.32:0/4173159048 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8eb41098d0 msgr2=0x7f8eb410bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:51.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.259+0000 7f8eb3e00640 1 --2- 172.21.15.32:0/4173159048 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8eb41098d0 0x7f8eb410bcf0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f8eac009860 tx=0x7f8eac02a340 comp rx=0 tx=0).stop 2024-04-30T08:35:51.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.260+0000 7f8eb3e00640 1 -- 172.21.15.32:0/4173159048 shutdown_connections 2024-04-30T08:35:51.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.260+0000 7f8eb3e00640 1 --2- 172.21.15.32:0/4173159048 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8eb41098d0 0x7f8eb410bcf0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.260+0000 7f8eb3e00640 1 --2- 172.21.15.32:0/4173159048 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8eb4104bd0 0x7f8eb4104fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.260+0000 7f8eb3e00640 1 --2- 172.21.15.32:0/4173159048 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8eb4103990 0x7f8eb4104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.260+0000 7f8eb3e00640 1 -- 172.21.15.32:0/4173159048 >> 172.21.15.32:0/4173159048 conn(0x7f8eb40feba0 msgr2=0x7f8eb4101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:51.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.260+0000 7f8eb3e00640 1 -- 172.21.15.32:0/4173159048 shutdown_connections 2024-04-30T08:35:51.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.260+0000 7f8eb3e00640 1 -- 172.21.15.32:0/4173159048 wait complete. 2024-04-30T08:35:51.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.261+0000 7f8eb3e00640 1 Processor -- start 2024-04-30T08:35:51.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.261+0000 7f8eb3e00640 1 -- start start 2024-04-30T08:35:51.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.261+0000 7f8eb3e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8eb4103990 0x7f8eb419c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.261+0000 7f8eb3e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8eb4104bd0 0x7f8eb419c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.261+0000 7f8eb3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8eb41098d0 0x7f8eb41a37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.261+0000 7f8eb3e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8eb410f4c0 con 0x7f8eb41098d0 2024-04-30T08:35:51.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.261+0000 7f8eb3e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8eb410f340 con 0x7f8eb4103990 2024-04-30T08:35:51.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.261+0000 7f8eb3e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8eb410f640 con 0x7f8eb4104bd0 2024-04-30T08:35:51.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.262+0000 7f8eb2a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8eb4103990 0x7f8eb419c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.262+0000 7f8eb3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8eb41098d0 0x7f8eb41a37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.262+0000 7f8eb2a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8eb4103990 0x7f8eb419c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:54730/0 (socket says 172.21.15.32:54730) 2024-04-30T08:35:51.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.262+0000 7f8eb2a00640 1 -- 172.21.15.32:0/27352173 learned_addr learned my addr 172.21.15.32:0/27352173 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:51.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.262+0000 7f8eb2000640 1 --2- 172.21.15.32:0/27352173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8eb4104bd0 0x7f8eb419c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.262+0000 7f8eb3400640 1 -- 172.21.15.32:0/27352173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8eb4104bd0 msgr2=0x7f8eb419c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:51.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.262+0000 7f8eb3400640 1 --2- 172.21.15.32:0/27352173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8eb4104bd0 0x7f8eb419c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.262+0000 7f8eb3400640 1 -- 172.21.15.32:0/27352173 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8eb4103990 msgr2=0x7f8eb419c2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:51.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.262+0000 7f8eb3400640 1 --2- 172.21.15.32:0/27352173 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8eb4103990 0x7f8eb419c2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.262+0000 7f8eb3400640 1 -- 172.21.15.32:0/27352173 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8eb41a3e10 con 0x7f8eb41098d0 2024-04-30T08:35:51.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.262+0000 7f8eb2a00640 1 --2- 172.21.15.32:0/27352173 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8eb4103990 0x7f8eb419c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:51.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.262+0000 7f8eb3400640 1 --2- 172.21.15.32:0/27352173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8eb41098d0 0x7f8eb41a37a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8eac02a860 tx=0x7f8eac032d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:51.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.263+0000 7f8eab400640 1 -- 172.21.15.32:0/27352173 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eac037040 con 0x7f8eb41098d0 2024-04-30T08:35:51.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.263+0000 7f8eb3e00640 1 -- 172.21.15.32:0/27352173 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8eb41a40a0 con 0x7f8eb41098d0 2024-04-30T08:35:51.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.263+0000 7f8eab400640 1 -- 172.21.15.32:0/27352173 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8eac030550 con 0x7f8eb41098d0 2024-04-30T08:35:51.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.263+0000 7f8eab400640 1 -- 172.21.15.32:0/27352173 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eac003850 con 0x7f8eb41098d0 2024-04-30T08:35:51.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.263+0000 7f8eb3e00640 1 -- 172.21.15.32:0/27352173 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8eb41a4580 con 0x7f8eb41098d0 2024-04-30T08:35:51.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.266+0000 7f8ec0d67740 1 Processor -- start 2024-04-30T08:35:51.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.267+0000 7f8eab400640 1 -- 172.21.15.32:0/27352173 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f8eac043020 con 0x7f8eb41098d0 2024-04-30T08:35:51.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.267+0000 7f8ec0d67740 1 -- start start 2024-04-30T08:35:51.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.267+0000 7f8ec0d67740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a1375ee390 0x55a13798f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.267+0000 7f8ec0d67740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a1375c54d0 0x55a13798fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.267+0000 7f8ec0d67740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1375cd6b0 0x55a137996bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.267+0000 7f8ec0d67740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a1378f4d50 con 0x55a1375ee390 2024-04-30T08:35:51.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.268+0000 7f8ec0d67740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55a13789a0c0 con 0x55a1375c54d0 2024-04-30T08:35:51.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.268+0000 7f8ec0d67740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a1378f0d10 con 0x55a1375cd6b0 2024-04-30T08:35:51.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.268+0000 7f8eb2000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a1375ee390 0x55a13798f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.268+0000 7f8eb2000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a1375ee390 0x55a13798f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53516/0 (socket says 172.21.15.32:53516) 2024-04-30T08:35:51.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.268+0000 7f8eb2000640 1 -- 172.21.15.32:0/111816845 learned_addr learned my addr 172.21.15.32:0/111816845 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:51.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.268+0000 7f8eb2a00640 1 --2- 172.21.15.32:0/111816845 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a1375c54d0 0x55a13798fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.268+0000 7f8eb2000640 1 --2- 172.21.15.32:0/111816845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1375cd6b0 0x55a137996bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.268+0000 7f8eb2000640 1 -- 172.21.15.32:0/111816845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1375cd6b0 msgr2=0x55a137996bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:51.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.268+0000 7f8eb2000640 1 --2- 172.21.15.32:0/111816845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1375cd6b0 0x55a137996bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.268+0000 7f8eb2000640 1 -- 172.21.15.32:0/111816845 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a1375c54d0 msgr2=0x55a13798fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:51.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.268+0000 7f8eb2000640 1 --2- 172.21.15.32:0/111816845 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a1375c54d0 0x55a13798fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.268+0000 7f8eb2000640 1 -- 172.21.15.32:0/111816845 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a1376063f0 con 0x55a1375ee390 2024-04-30T08:35:51.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.268+0000 7f8eb2a00640 1 --2- 172.21.15.32:0/111816845 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a1375c54d0 0x55a13798fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:51.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.269+0000 7f8eb2000640 1 --2- 172.21.15.32:0/111816845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a1375ee390 0x55a13798f730 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c013920 tx=0x7f8e9c009be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66037b1ec503e333 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:51.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.269+0000 7f8eab400640 1 --2- 172.21.15.32:0/27352173 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8e940afb70 0x7f8e940b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.269+0000 7f8ea3e00640 1 -- 172.21.15.32:0/111816845 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c003970 con 0x55a1375ee390 2024-04-30T08:35:51.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.269+0000 7f8eab400640 1 -- 172.21.15.32:0/27352173 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f8eac0fe100 con 0x7f8eb41098d0 2024-04-30T08:35:51.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.269+0000 7f8eb2a00640 1 --2- 172.21.15.32:0/27352173 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8e940afb70 0x7f8e940b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.269+0000 7f8ea3e00640 1 -- 172.21.15.32:0/111816845 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e9c00fdf0 con 0x55a1375ee390 2024-04-30T08:35:51.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.269+0000 7f8ea3e00640 1 -- 172.21.15.32:0/111816845 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c010e00 con 0x55a1375ee390 2024-04-30T08:35:51.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.270+0000 7f8ec0d67740 1 -- 172.21.15.32:0/111816845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a1375ee390 msgr2=0x55a13798f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:51.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.270+0000 7f8eb2a00640 1 --2- 172.21.15.32:0/27352173 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8e940afb70 0x7f8e940b2080 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4009fd0 tx=0x7f8ea4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:51.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.270+0000 7f8ec0d67740 1 --2- 172.21.15.32:0/111816845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a1375ee390 0x55a13798f730 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c013920 tx=0x7f8e9c009be0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.270+0000 7f8ec0d67740 1 -- 172.21.15.32:0/111816845 shutdown_connections 2024-04-30T08:35:51.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.270+0000 7f8ec0d67740 1 --2- 172.21.15.32:0/111816845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1375cd6b0 0x55a137996bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.270+0000 7f8ec0d67740 1 --2- 172.21.15.32:0/111816845 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a1375c54d0 0x55a13798fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.270+0000 7f8ec0d67740 1 --2- 172.21.15.32:0/111816845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a1375ee390 0x55a13798f730 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.270+0000 7f8ec0d67740 1 -- 172.21.15.32:0/111816845 >> 172.21.15.32:0/111816845 conn(0x55a1375e24e0 msgr2=0x55a13798b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:51.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.270+0000 7f8ec0d67740 1 -- 172.21.15.32:0/111816845 shutdown_connections 2024-04-30T08:35:51.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.270+0000 7f8ec0d67740 1 -- 172.21.15.32:0/111816845 wait complete. 2024-04-30T08:35:51.274 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.271+0000 7f8ec0d67740 1 -- start start 2024-04-30T08:35:51.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.271+0000 7f8ec0d67740 1 Processor -- start 2024-04-30T08:35:51.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.272+0000 7f8ec0d67740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1375c54d0 0x55a137999700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.272+0000 7f8ec0d67740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a1375cd6b0 0x55a137999c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.272+0000 7f8ec0d67740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a1375ee390 0x55a1379a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.272+0000 7f8ec0d67740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55a137997600 con 0x55a1375ee390 2024-04-30T08:35:51.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.272+0000 7f8ec0d67740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55a1378adb50 con 0x55a1375cd6b0 2024-04-30T08:35:51.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.272+0000 7f8ec0d67740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55a137997780 con 0x55a1375c54d0 2024-04-30T08:35:51.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.272+0000 7f8ec0d67740 10 client.0 _ll_register_callbacks cb 0x55a137877420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:35:51.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.272+0000 7f8eb3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a1375ee390 0x55a1379a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.272+0000 7f8eb2000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a1375cd6b0 0x55a137999c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.272+0000 7f8eb3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a1375ee390 0x55a1379a0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53528/0 (socket says 172.21.15.32:53528) 2024-04-30T08:35:51.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.272+0000 7f8eb2000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a1375cd6b0 0x55a137999c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:54742/0 (socket says 172.21.15.32:54742) 2024-04-30T08:35:51.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.272+0000 7f8eb3400640 1 -- 172.21.15.32:0/599410690 learned_addr learned my addr 172.21.15.32:0/599410690 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:51.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.272+0000 7f8eb2a00640 1 --2- 172.21.15.32:0/599410690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1375c54d0 0x55a137999700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.273+0000 7f8eb2000640 1 -- 172.21.15.32:0/599410690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1375c54d0 msgr2=0x55a137999700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:51.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.273+0000 7f8eb2000640 1 --2- 172.21.15.32:0/599410690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1375c54d0 0x55a137999700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.273+0000 7f8eb2000640 1 -- 172.21.15.32:0/599410690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a1375ee390 msgr2=0x55a1379a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:51.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.273+0000 7f8eb2000640 1 --2- 172.21.15.32:0/599410690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a1375ee390 0x55a1379a0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.273+0000 7f8eb2000640 1 -- 172.21.15.32:0/599410690 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a1376d07d0 con 0x55a1375cd6b0 2024-04-30T08:35:51.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.273+0000 7f8eb3400640 1 --2- 172.21.15.32:0/599410690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a1375ee390 0x55a1379a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:51.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.274+0000 7f8eb2000640 1 --2- 172.21.15.32:0/599410690 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a1375cd6b0 0x55a137999c40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c0109d0 tx=0x7f8e9c00fb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7eae853aeca17e3e server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:51.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.274+0000 7f8ea0c00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:35:51.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.277+0000 7f8ea0c00640 1 -- 172.21.15.32:0/599410690 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c010d00 con 0x55a1375cd6b0 2024-04-30T08:35:51.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.277+0000 7f8ea0c00640 1 -- 172.21.15.32:0/599410690 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f8e9c0131b0 con 0x55a1375cd6b0 2024-04-30T08:35:51.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.277+0000 7f8ec0d67740 1 -- 172.21.15.32:0/599410690 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e9c00fdf0 con 0x55a1375cd6b0 2024-04-30T08:35:51.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.277+0000 7f8ea0c00640 1 -- 172.21.15.32:0/599410690 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8e9c0133b0 con 0x55a1375cd6b0 2024-04-30T08:35:51.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.277+0000 7f8ec0d67740 10 client.25757 fetch_fsmap learned FSMap version 28 2024-04-30T08:35:51.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.277+0000 7f8ec0d67740 1 -- 172.21.15.32:0/599410690 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a1379a3ad0 con 0x55a1375cd6b0 2024-04-30T08:35:51.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.277+0000 7f8ea0c00640 1 -- 172.21.15.32:0/599410690 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8e9c00fdf0 con 0x55a1375cd6b0 2024-04-30T08:35:51.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.278+0000 7f8ea0c00640 1 -- 172.21.15.32:0/599410690 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8e9c020bf0 con 0x55a1375cd6b0 2024-04-30T08:35:51.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.278+0000 7f8ec0d67740 10 client.25757 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:35:51.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.278+0000 7f8ec0d67740 10 client.25757 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-30T08:35:51.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.278+0000 7f8ec0d67740 20 client.25757 populate_metadata read hostname 'smithi032' 2024-04-30T08:35:51.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.278+0000 7f8ec0d67740 1 --2- 172.21.15.32:0/599410690 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x55a1375acf60 0x55a1379a6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.278+0000 7f8ec0d67740 4 client.25757 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:35:51.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.278+0000 7f8ec0d67740 1 -- 172.21.15.32:0/599410690 --> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a137902960 con 0x55a1375acf60 2024-04-30T08:35:51.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.278+0000 7f8eb2a00640 1 --2- 172.21.15.32:0/599410690 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x55a1375acf60 0x55a1379a6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.279+0000 7f8eb2a00640 1 --2- 172.21.15.32:0/599410690 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x55a1375acf60 0x55a1379a6fd0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ae736bdefd930b98 server_cookie=5090f9e679ca895e in_seq=0 out_seq=0 2024-04-30T08:35:51.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.279+0000 7f8ea0c00640 10 client.25757 ms_handle_connect on v2:172.21.15.189:6838/352970107 2024-04-30T08:35:51.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.280+0000 7f8ea0c00640 1 -- 172.21.15.32:0/599410690 <== mds.2 v2:172.21.15.189:6838/352970107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55a137902960 con 0x55a1375acf60 2024-04-30T08:35:51.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.280+0000 7f8ea0c00640 10 client.25757 handle_command_reply: tid=0 2024-04-30T08:35:51.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.280+0000 7f8ec0d67740 1 client.25757 shutdown 2024-04-30T08:35:51.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.280+0000 7f8ec0d67740 1 -- 172.21.15.32:0/599410690 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a1375cd6b0 msgr2=0x55a137999c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:51.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.280+0000 7f8ec0d67740 1 --2- 172.21.15.32:0/599410690 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a1375cd6b0 0x55a137999c40 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f8e9c0109d0 tx=0x7f8e9c00fb70 comp rx=0 tx=0).stop 2024-04-30T08:35:51.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.280+0000 7f8ec0d67740 1 -- 172.21.15.32:0/599410690 shutdown_connections 2024-04-30T08:35:51.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.280+0000 7f8ec0d67740 1 --2- 172.21.15.32:0/599410690 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x55a1375acf60 0x55a1379a6fd0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.280+0000 7f8ec0d67740 1 --2- 172.21.15.32:0/599410690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a1375ee390 0x55a1379a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.280+0000 7f8ec0d67740 1 --2- 172.21.15.32:0/599410690 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a1375cd6b0 0x55a137999c40 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.280+0000 7f8ea0c00640 0 client.25757 ms_handle_reset on v2:172.21.15.189:6838/352970107 2024-04-30T08:35:51.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.280+0000 7f8ec0d67740 1 --2- 172.21.15.32:0/599410690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1375c54d0 0x55a137999700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.280+0000 7f8ec0d67740 1 -- 172.21.15.32:0/599410690 >> 172.21.15.32:0/599410690 conn(0x55a1375e24e0 msgr2=0x55a13798a100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:51.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.281+0000 7f8ec0d67740 1 -- 172.21.15.32:0/599410690 shutdown_connections 2024-04-30T08:35:51.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.281+0000 7f8ec0d67740 1 -- 172.21.15.32:0/599410690 wait complete. 2024-04-30T08:35:51.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.281+0000 7f8ec0d67740 20 client.25757 trim_cache size 0 max 16384 2024-04-30T08:35:51.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.296+0000 7f8ec0d67740 1 Processor -- start 2024-04-30T08:35:51.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.296+0000 7f8ec0d67740 1 -- start start 2024-04-30T08:35:51.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.297+0000 7f8ec0d67740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a1375c54d0 0x55a13798ab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.297+0000 7f8ec0d67740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1375cd6b0 0x55a13798b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.297+0000 7f8ec0d67740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a1375ee390 0x55a137997d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.297+0000 7f8ec0d67740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8eb41aec10 con 0x55a1375ee390 2024-04-30T08:35:51.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.297+0000 7f8ec0d67740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8eb41b0750 con 0x55a1375c54d0 2024-04-30T08:35:51.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.297+0000 7f8ec0d67740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8eb41af2e0 con 0x55a1375cd6b0 2024-04-30T08:35:51.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.297+0000 7f8eb3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a1375ee390 0x55a137997d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.297+0000 7f8eb2a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a1375c54d0 0x55a13798ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.297+0000 7f8eb2a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a1375c54d0 0x55a13798ab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:54758/0 (socket says 172.21.15.32:54758) 2024-04-30T08:35:51.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.297+0000 7f8eb2a00640 1 -- 172.21.15.32:0/71697055 learned_addr learned my addr 172.21.15.32:0/71697055 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:51.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.297+0000 7f8eb2000640 1 --2- 172.21.15.32:0/71697055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1375cd6b0 0x55a13798b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.297+0000 7f8eb2a00640 1 -- 172.21.15.32:0/71697055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1375cd6b0 msgr2=0x55a13798b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:51.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.297+0000 7f8eb2a00640 1 --2- 172.21.15.32:0/71697055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1375cd6b0 0x55a13798b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.297+0000 7f8eb2a00640 1 -- 172.21.15.32:0/71697055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a1375ee390 msgr2=0x55a137997d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:51.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.297+0000 7f8eb2a00640 1 --2- 172.21.15.32:0/71697055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a1375ee390 0x55a137997d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.297+0000 7f8eb2a00640 1 -- 172.21.15.32:0/71697055 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a13798f2e0 con 0x55a1375c54d0 2024-04-30T08:35:51.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.297+0000 7f8eb2000640 1 --2- 172.21.15.32:0/71697055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1375cd6b0 0x55a13798b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:35:51.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.298+0000 7f8eb2a00640 1 --2- 172.21.15.32:0/71697055 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a1375c54d0 0x55a13798ab00 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x55a13798bef0 tx=0x7f8ea4019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5902e81837d217c server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:51.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.298+0000 7f8ea3400640 1 -- 172.21.15.32:0/71697055 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ea4031070 con 0x55a1375c54d0 2024-04-30T08:35:51.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.298+0000 7f8ea3400640 1 -- 172.21.15.32:0/71697055 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ea401db20 con 0x55a1375c54d0 2024-04-30T08:35:51.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.298+0000 7f8ea3400640 1 -- 172.21.15.32:0/71697055 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ea402ab60 con 0x55a1375c54d0 2024-04-30T08:35:51.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.299+0000 7f8ec0d67740 1 -- 172.21.15.32:0/71697055 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a1375c54d0 msgr2=0x55a13798ab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:51.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.299+0000 7f8ec0d67740 1 --2- 172.21.15.32:0/71697055 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a1375c54d0 0x55a13798ab00 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x55a13798bef0 tx=0x7f8ea4019000 comp rx=0 tx=0).stop 2024-04-30T08:35:51.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.299+0000 7f8ec0d67740 1 -- 172.21.15.32:0/71697055 shutdown_connections 2024-04-30T08:35:51.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.299+0000 7f8ec0d67740 1 --2- 172.21.15.32:0/71697055 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a1375ee390 0x55a137997d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.299+0000 7f8ec0d67740 1 --2- 172.21.15.32:0/71697055 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1375cd6b0 0x55a13798b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.299+0000 7f8ec0d67740 1 --2- 172.21.15.32:0/71697055 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a1375c54d0 0x55a13798ab00 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.299+0000 7f8ec0d67740 1 -- 172.21.15.32:0/71697055 >> 172.21.15.32:0/71697055 conn(0x55a1375e24e0 msgr2=0x55a137993b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:51.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.299+0000 7f8ec0d67740 1 -- 172.21.15.32:0/71697055 shutdown_connections 2024-04-30T08:35:51.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.299+0000 7f8ec0d67740 1 -- 172.21.15.32:0/71697055 wait complete. 2024-04-30T08:35:51.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.300+0000 7f8ec0d67740 1 -- start start 2024-04-30T08:35:51.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.300+0000 7f8ec0d67740 1 Processor -- start 2024-04-30T08:35:51.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.300+0000 7f8ec0d67740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a1375c54d0 0x55a13799d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.300+0000 7f8ec0d67740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a1375cd6b0 0x55a13799d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.300+0000 7f8ec0d67740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1375ee390 0x55a13799e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.300+0000 7f8ec0d67740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8eb41b0080 con 0x55a1375c54d0 2024-04-30T08:35:51.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.300+0000 7f8ec0d67740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8eb41af9b0 con 0x55a1375cd6b0 2024-04-30T08:35:51.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.300+0000 7f8ec0d67740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8eb41acbd0 con 0x55a1375ee390 2024-04-30T08:35:51.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.300+0000 7f8ec0d67740 10 client.0 _ll_register_callbacks cb 0x7f8ea4009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:35:51.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.301+0000 7f8eb2000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a1375cd6b0 0x55a13799d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.301+0000 7f8eb2000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a1375cd6b0 0x55a13799d8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:54764/0 (socket says 172.21.15.32:54764) 2024-04-30T08:35:51.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.301+0000 7f8eb2000640 1 -- 172.21.15.32:0/1118497179 learned_addr learned my addr 172.21.15.32:0/1118497179 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:51.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.301+0000 7f8eb2a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a1375c54d0 0x55a13799d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.301+0000 7f8eb3400640 1 --2- 172.21.15.32:0/1118497179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1375ee390 0x55a13799e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.301+0000 7f8eb2000640 1 -- 172.21.15.32:0/1118497179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1375ee390 msgr2=0x55a13799e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:51.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.301+0000 7f8eb2000640 1 --2- 172.21.15.32:0/1118497179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1375ee390 0x55a13799e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.301+0000 7f8eb2000640 1 -- 172.21.15.32:0/1118497179 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a1375c54d0 msgr2=0x55a13799d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:51.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.301+0000 7f8eb2000640 1 --2- 172.21.15.32:0/1118497179 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a1375c54d0 0x55a13799d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.301+0000 7f8eb2000640 1 -- 172.21.15.32:0/1118497179 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a13799ed70 con 0x55a1375cd6b0 2024-04-30T08:35:51.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.301+0000 7f8eb2a00640 1 --2- 172.21.15.32:0/1118497179 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a1375c54d0 0x55a13799d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:51.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.302+0000 7f8eb2000640 1 --2- 172.21.15.32:0/1118497179 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a1375cd6b0 0x55a13799d8d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x55a137990560 tx=0x7f8e9c005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cafc6ce058b1bb19 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:51.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.302+0000 7f8ea0c00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:35:51.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.302+0000 7f8ea0c00640 1 -- 172.21.15.32:0/1118497179 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e9c03d060 con 0x55a1375cd6b0 2024-04-30T08:35:51.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.302+0000 7f8ec0d67740 1 -- 172.21.15.32:0/1118497179 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ea401db20 con 0x55a1375cd6b0 2024-04-30T08:35:51.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.302+0000 7f8ea0c00640 1 -- 172.21.15.32:0/1118497179 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f8e9c009a50 con 0x55a1375cd6b0 2024-04-30T08:35:51.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.303+0000 7f8ea0c00640 1 -- 172.21.15.32:0/1118497179 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8e9c044050 con 0x55a1375cd6b0 2024-04-30T08:35:51.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.303+0000 7f8ec0d67740 10 client.25763 fetch_fsmap learned FSMap version 28 2024-04-30T08:35:51.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.303+0000 7f8ec0d67740 1 -- 172.21.15.32:0/1118497179 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a1379a0020 con 0x55a1375cd6b0 2024-04-30T08:35:51.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.303+0000 7f8ea0c00640 1 -- 172.21.15.32:0/1118497179 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8ea401db20 con 0x55a1375cd6b0 2024-04-30T08:35:51.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.303+0000 7f8ea0c00640 1 -- 172.21.15.32:0/1118497179 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f8e9c004a90 con 0x55a1375cd6b0 2024-04-30T08:35:51.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.303+0000 7f8ec0d67740 10 client.25763 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:35:51.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.303+0000 7f8ec0d67740 10 client.25763 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-30T08:35:51.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.303+0000 7f8ec0d67740 20 client.25763 populate_metadata read hostname 'smithi032' 2024-04-30T08:35:51.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.303+0000 7f8ec0d67740 1 --2- 172.21.15.32:0/1118497179 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x55a1375823a0 0x55a137a11600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.303+0000 7f8ec0d67740 4 client.25763 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-30T08:35:51.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.303+0000 7f8ec0d67740 1 -- 172.21.15.32:0/1118497179 --> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a1378fc7d0 con 0x55a1375823a0 2024-04-30T08:35:51.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.304+0000 7f8eb2a00640 1 --2- 172.21.15.32:0/1118497179 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x55a1375823a0 0x55a137a11600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.304+0000 7f8eb2a00640 1 --2- 172.21.15.32:0/1118497179 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x55a1375823a0 0x55a137a11600 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=5c2cdc081991591b server_cookie=c4dfe38d7a2c89d6 in_seq=0 out_seq=0 2024-04-30T08:35:51.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.304+0000 7f8ea0c00640 10 client.25763 ms_handle_connect on v2:172.21.15.189:6838/352970107 2024-04-30T08:35:51.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.305+0000 7f8ea0c00640 1 -- 172.21.15.32:0/1118497179 <== mds.2 v2:172.21.15.189:6838/352970107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a1378fc7d0 con 0x55a1375823a0 2024-04-30T08:35:51.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.305+0000 7f8ea0c00640 10 client.25763 handle_command_reply: tid=0 2024-04-30T08:35:51.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.305+0000 7f8ec0d67740 1 client.25763 shutdown 2024-04-30T08:35:51.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.305+0000 7f8ec0d67740 1 -- 172.21.15.32:0/1118497179 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a1375cd6b0 msgr2=0x55a13799d8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:51.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.305+0000 7f8ec0d67740 1 --2- 172.21.15.32:0/1118497179 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a1375cd6b0 0x55a13799d8d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x55a137990560 tx=0x7f8e9c005f90 comp rx=0 tx=0).stop 2024-04-30T08:35:51.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.305+0000 7f8ec0d67740 1 -- 172.21.15.32:0/1118497179 shutdown_connections 2024-04-30T08:35:51.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.305+0000 7f8ec0d67740 1 --2- 172.21.15.32:0/1118497179 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x55a1375823a0 0x55a137a11600 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.305+0000 7f8ec0d67740 1 --2- 172.21.15.32:0/1118497179 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1375ee390 0x55a13799e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.305+0000 7f8ec0d67740 1 --2- 172.21.15.32:0/1118497179 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55a1375cd6b0 0x55a13799d8d0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.305+0000 7f8ec0d67740 1 --2- 172.21.15.32:0/1118497179 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55a1375c54d0 0x55a13799d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.305+0000 7f8ea0c00640 0 client.25763 ms_handle_reset on v2:172.21.15.189:6838/352970107 2024-04-30T08:35:51.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.305+0000 7f8ec0d67740 1 -- 172.21.15.32:0/1118497179 >> 172.21.15.32:0/1118497179 conn(0x55a1375e24e0 msgr2=0x55a137993b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:51.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.305+0000 7f8ec0d67740 1 -- 172.21.15.32:0/1118497179 shutdown_connections 2024-04-30T08:35:51.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.305+0000 7f8ec0d67740 1 -- 172.21.15.32:0/1118497179 wait complete. 2024-04-30T08:35:51.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.305+0000 7f8ec0d67740 20 client.25763 trim_cache size 0 max 16384 2024-04-30T08:35:51.310 INFO:teuthology.orchestra.run.smithi032.stdout:[] 2024-04-30T08:35:51.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.306+0000 7f8eb3e00640 1 -- 172.21.15.32:0/27352173 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8e940afb70 msgr2=0x7f8e940b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:51.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.306+0000 7f8eb3e00640 1 --2- 172.21.15.32:0/27352173 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8e940afb70 0x7f8e940b2080 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8ea4009fd0 tx=0x7f8ea4009260 comp rx=0 tx=0).stop 2024-04-30T08:35:51.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.306+0000 7f8eb3e00640 1 -- 172.21.15.32:0/27352173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8eb41098d0 msgr2=0x7f8eb41a37a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:51.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.306+0000 7f8eb3e00640 1 --2- 172.21.15.32:0/27352173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8eb41098d0 0x7f8eb41a37a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f8eac02a860 tx=0x7f8eac032d40 comp rx=0 tx=0).stop 2024-04-30T08:35:51.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.306+0000 7f8eb3e00640 1 -- 172.21.15.32:0/27352173 shutdown_connections 2024-04-30T08:35:51.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.306+0000 7f8eb3e00640 1 --2- 172.21.15.32:0/27352173 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8e940afb70 0x7f8e940b2080 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.306+0000 7f8eb3e00640 1 --2- 172.21.15.32:0/27352173 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8eb41098d0 0x7f8eb41a37a0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.306+0000 7f8eb3e00640 1 --2- 172.21.15.32:0/27352173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8eb4104bd0 0x7f8eb419c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.306+0000 7f8eb3e00640 1 --2- 172.21.15.32:0/27352173 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8eb4103990 0x7f8eb419c2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.306+0000 7f8eb3e00640 1 -- 172.21.15.32:0/27352173 >> 172.21.15.32:0/27352173 conn(0x7f8eb40feba0 msgr2=0x7f8eb40fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:51.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.306+0000 7f8eb3e00640 1 -- 172.21.15.32:0/27352173 shutdown_connections 2024-04-30T08:35:51.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.306+0000 7f8eb3e00640 1 -- 172.21.15.32:0/27352173 wait complete. 2024-04-30T08:35:51.334 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-30T08:35:51.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:51 smithi189 ceph-mon[33038]: pgmap v445: 129 pgs: 129 active+clean; 662 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-30T08:35:51.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.629+0000 7fa478a00640 1 -- 172.21.15.32:0/727332840 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa474105bb0 msgr2=0x7fa4741068b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:51.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.629+0000 7fa478a00640 1 --2- 172.21.15.32:0/727332840 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa474105bb0 0x7fa4741068b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fa464009900 tx=0x7fa46402a3f0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.629+0000 7fa478a00640 1 -- 172.21.15.32:0/727332840 shutdown_connections 2024-04-30T08:35:51.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.629+0000 7fa478a00640 1 --2- 172.21.15.32:0/727332840 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa47410baf0 0x7fa47410df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.629+0000 7fa478a00640 1 --2- 172.21.15.32:0/727332840 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa474106df0 0x7fa474107200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.629+0000 7fa478a00640 1 --2- 172.21.15.32:0/727332840 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa474105bb0 0x7fa4741068b0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.629+0000 7fa478a00640 1 -- 172.21.15.32:0/727332840 >> 172.21.15.32:0/727332840 conn(0x7fa474100da0 msgr2=0x7fa474103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:51.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.629+0000 7fa478a00640 1 -- 172.21.15.32:0/727332840 shutdown_connections 2024-04-30T08:35:51.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.629+0000 7fa478a00640 1 -- 172.21.15.32:0/727332840 wait complete. 2024-04-30T08:35:51.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.631+0000 7fa478a00640 1 Processor -- start 2024-04-30T08:35:51.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.631+0000 7fa478a00640 1 -- start start 2024-04-30T08:35:51.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.631+0000 7fa478a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa474105bb0 0x7fa47419c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.632+0000 7fa478a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa474106df0 0x7fa47419c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.632+0000 7fa478a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa47410baf0 0x7fa4741a3770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.632+0000 7fa478a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa4741116e0 con 0x7fa47410baf0 2024-04-30T08:35:51.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.632+0000 7fa478a00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa474111560 con 0x7fa474105bb0 2024-04-30T08:35:51.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.632+0000 7fa478a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa474111860 con 0x7fa474106df0 2024-04-30T08:35:51.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.632+0000 7fa473e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa47410baf0 0x7fa4741a3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.632+0000 7fa473400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa474105bb0 0x7fa47419c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.632+0000 7fa473400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa474105bb0 0x7fa47419c2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:54786/0 (socket says 172.21.15.32:54786) 2024-04-30T08:35:51.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.632+0000 7fa473400640 1 -- 172.21.15.32:0/3982376855 learned_addr learned my addr 172.21.15.32:0/3982376855 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:51.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.632+0000 7fa473e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa47410baf0 0x7fa4741a3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53578/0 (socket says 172.21.15.32:53578) 2024-04-30T08:35:51.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.632+0000 7fa46aa00640 1 --2- 172.21.15.32:0/3982376855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa474106df0 0x7fa47419c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.632+0000 7fa473400640 1 -- 172.21.15.32:0/3982376855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa474106df0 msgr2=0x7fa47419c7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:51.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.632+0000 7fa473400640 1 --2- 172.21.15.32:0/3982376855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa474106df0 0x7fa47419c7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.632+0000 7fa473400640 1 -- 172.21.15.32:0/3982376855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa47410baf0 msgr2=0x7fa4741a3770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:51.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.632+0000 7fa473400640 1 --2- 172.21.15.32:0/3982376855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa47410baf0 0x7fa4741a3770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.632+0000 7fa473400640 1 -- 172.21.15.32:0/3982376855 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4741a3de0 con 0x7fa474105bb0 2024-04-30T08:35:51.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.633+0000 7fa473400640 1 --2- 172.21.15.32:0/3982376855 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa474105bb0 0x7fa47419c2a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa464031440 tx=0x7fa464003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:51.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.633+0000 7fa470c00640 1 -- 172.21.15.32:0/3982376855 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa46403f070 con 0x7fa474105bb0 2024-04-30T08:35:51.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.633+0000 7fa478a00640 1 -- 172.21.15.32:0/3982376855 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4741a4070 con 0x7fa474105bb0 2024-04-30T08:35:51.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.633+0000 7fa478a00640 1 -- 172.21.15.32:0/3982376855 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4741a45b0 con 0x7fa474105bb0 2024-04-30T08:35:51.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.633+0000 7fa470c00640 1 -- 172.21.15.32:0/3982376855 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa464031c50 con 0x7fa474105bb0 2024-04-30T08:35:51.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.634+0000 7fa470c00640 1 -- 172.21.15.32:0/3982376855 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa464049060 con 0x7fa474105bb0 2024-04-30T08:35:51.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.637+0000 7fa48106b740 1 Processor -- start 2024-04-30T08:35:51.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.637+0000 7fa48106b740 1 -- start start 2024-04-30T08:35:51.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.637+0000 7fa48106b740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560c421f9390 0x560c4259a730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.637+0000 7fa48106b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c421d04d0 0x560c4259acb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.637+0000 7fa470c00640 1 -- 172.21.15.32:0/3982376855 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7fa464042070 con 0x7fa474105bb0 2024-04-30T08:35:51.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.637+0000 7fa48106b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560c421d86b0 0x560c425a1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.637+0000 7fa48106b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560c424ffd50 con 0x560c421d86b0 2024-04-30T08:35:51.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.637+0000 7fa48106b740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x560c424a50c0 con 0x560c421f9390 2024-04-30T08:35:51.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.637+0000 7fa48106b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560c424fbd10 con 0x560c421d04d0 2024-04-30T08:35:51.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.638+0000 7fa473e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560c421f9390 0x560c4259a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.638+0000 7fa473e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560c421f9390 0x560c4259a730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:54792/0 (socket says 172.21.15.32:54792) 2024-04-30T08:35:51.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.638+0000 7fa473e00640 1 -- 172.21.15.32:0/3794552249 learned_addr learned my addr 172.21.15.32:0/3794552249 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:51.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.638+0000 7fa473400640 1 --2- 172.21.15.32:0/3794552249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c421d04d0 0x560c4259acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.638+0000 7fa46aa00640 1 --2- 172.21.15.32:0/3794552249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560c421d86b0 0x560c425a1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.638+0000 7fa473e00640 1 -- 172.21.15.32:0/3794552249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c421d04d0 msgr2=0x560c4259acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:51.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.638+0000 7fa473e00640 1 --2- 172.21.15.32:0/3794552249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c421d04d0 0x560c4259acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.638+0000 7fa473e00640 1 -- 172.21.15.32:0/3794552249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560c421d86b0 msgr2=0x560c425a1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:51.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.638+0000 7fa473e00640 1 --2- 172.21.15.32:0/3794552249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560c421d86b0 0x560c425a1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.638+0000 7fa473e00640 1 -- 172.21.15.32:0/3794552249 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c422113f0 con 0x560c421f9390 2024-04-30T08:35:51.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.638+0000 7fa473400640 1 --2- 172.21.15.32:0/3794552249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c421d04d0 0x560c4259acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:35:51.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.639+0000 7fa473e00640 1 --2- 172.21.15.32:0/3794552249 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560c421f9390 0x560c4259a730 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa46c00b700 tx=0x7fa46c00bbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=10b47206aa829c2a server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:51.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.639+0000 7fa463e00640 1 -- 172.21.15.32:0/3794552249 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa46c003dd0 con 0x560c421f9390 2024-04-30T08:35:51.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.639+0000 7fa463e00640 1 -- 172.21.15.32:0/3794552249 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa46c015070 con 0x560c421f9390 2024-04-30T08:35:51.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.639+0000 7fa463e00640 1 -- 172.21.15.32:0/3794552249 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa46c0106a0 con 0x560c421f9390 2024-04-30T08:35:51.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.639+0000 7fa470c00640 1 --2- 172.21.15.32:0/3982376855 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa4580afb70 0x7fa4580b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.639+0000 7fa470c00640 1 -- 172.21.15.32:0/3982376855 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fa4640ff690 con 0x7fa474105bb0 2024-04-30T08:35:51.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.640+0000 7fa473400640 1 --2- 172.21.15.32:0/3982376855 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa4580afb70 0x7fa4580b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.640+0000 7fa48106b740 1 -- 172.21.15.32:0/3794552249 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560c421f9390 msgr2=0x560c4259a730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:51.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.640+0000 7fa48106b740 1 --2- 172.21.15.32:0/3794552249 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560c421f9390 0x560c4259a730 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fa46c00b700 tx=0x7fa46c00bbc0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.640+0000 7fa48106b740 1 -- 172.21.15.32:0/3794552249 shutdown_connections 2024-04-30T08:35:51.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.640+0000 7fa48106b740 1 --2- 172.21.15.32:0/3794552249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560c421d86b0 0x560c425a1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.640+0000 7fa48106b740 1 --2- 172.21.15.32:0/3794552249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c421d04d0 0x560c4259acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.640+0000 7fa48106b740 1 --2- 172.21.15.32:0/3794552249 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560c421f9390 0x560c4259a730 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.640+0000 7fa48106b740 1 -- 172.21.15.32:0/3794552249 >> 172.21.15.32:0/3794552249 conn(0x560c421ed4e0 msgr2=0x560c425960c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:51.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.640+0000 7fa48106b740 1 -- 172.21.15.32:0/3794552249 shutdown_connections 2024-04-30T08:35:51.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.640+0000 7fa48106b740 1 -- 172.21.15.32:0/3794552249 wait complete. 2024-04-30T08:35:51.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.640+0000 7fa473400640 1 --2- 172.21.15.32:0/3982376855 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa4580afb70 0x7fa4580b2080 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa464006010 tx=0x7fa464009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:51.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.641+0000 7fa48106b740 1 -- start start 2024-04-30T08:35:51.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.641+0000 7fa48106b740 1 Processor -- start 2024-04-30T08:35:51.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.641+0000 7fa48106b740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560c421d04d0 0x560c425a4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.641+0000 7fa48106b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560c421d86b0 0x560c425a4bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.641+0000 7fa48106b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c421f9390 0x560c425abb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.641+0000 7fa48106b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560c425a2600 con 0x560c421d86b0 2024-04-30T08:35:51.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.641+0000 7fa48106b740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x560c424b8b50 con 0x560c421d04d0 2024-04-30T08:35:51.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.642+0000 7fa48106b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560c425a2780 con 0x560c421f9390 2024-04-30T08:35:51.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.642+0000 7fa48106b740 10 client.0 _ll_register_callbacks cb 0x560c42482420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:35:51.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.642+0000 7fa46aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560c421d86b0 0x560c425a4bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.642+0000 7fa46aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560c421d86b0 0x560c425a4bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53598/0 (socket says 172.21.15.32:53598) 2024-04-30T08:35:51.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.642+0000 7fa46aa00640 1 -- 172.21.15.32:0/1388757003 learned_addr learned my addr 172.21.15.32:0/1388757003 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:51.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.642+0000 7fa473e00640 1 --2- 172.21.15.32:0/1388757003 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560c421d04d0 0x560c425a4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.642+0000 7fa473e00640 1 --2- 172.21.15.32:0/1388757003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c421f9390 0x560c425abb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.642+0000 7fa46aa00640 1 -- 172.21.15.32:0/1388757003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c421f9390 msgr2=0x560c425abb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:51.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.642+0000 7fa46aa00640 1 --2- 172.21.15.32:0/1388757003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c421f9390 0x560c425abb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.642+0000 7fa46aa00640 1 -- 172.21.15.32:0/1388757003 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560c421d04d0 msgr2=0x560c425a4680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:51.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.642+0000 7fa46aa00640 1 --2- 172.21.15.32:0/1388757003 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560c421d04d0 0x560c425a4680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.642+0000 7fa46aa00640 1 -- 172.21.15.32:0/1388757003 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c422db7d0 con 0x560c421d86b0 2024-04-30T08:35:51.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.643+0000 7fa46aa00640 1 --2- 172.21.15.32:0/1388757003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560c421d86b0 0x560c425a4bc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa45c011d10 tx=0x7fa45c00f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c17620b1dee39ebd server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:51.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.643+0000 7fa460c00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:35:51.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.643+0000 7fa460c00640 1 -- 172.21.15.32:0/1388757003 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45c004b20 con 0x560c421d86b0 2024-04-30T08:35:51.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.643+0000 7fa460c00640 1 -- 172.21.15.32:0/1388757003 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fa45c017050 con 0x560c421d86b0 2024-04-30T08:35:51.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.643+0000 7fa48106b740 1 -- 172.21.15.32:0/1388757003 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa46c015070 con 0x560c421d86b0 2024-04-30T08:35:51.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.643+0000 7fa460c00640 1 -- 172.21.15.32:0/1388757003 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa45c025050 con 0x560c421d86b0 2024-04-30T08:35:51.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.643+0000 7fa48106b740 10 client.15651 fetch_fsmap learned FSMap version 28 2024-04-30T08:35:51.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.643+0000 7fa48106b740 1 -- 172.21.15.32:0/1388757003 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560c425aeb20 con 0x560c421d86b0 2024-04-30T08:35:51.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.644+0000 7fa460c00640 1 -- 172.21.15.32:0/1388757003 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa46c015070 con 0x560c421d86b0 2024-04-30T08:35:51.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.644+0000 7fa460c00640 1 -- 172.21.15.32:0/1388757003 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa45c019030 con 0x560c421d86b0 2024-04-30T08:35:51.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.644+0000 7fa48106b740 10 client.15651 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:35:51.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.644+0000 7fa48106b740 10 client.15651 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-30T08:35:51.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.644+0000 7fa48106b740 20 client.15651 populate_metadata read hostname 'smithi032' 2024-04-30T08:35:51.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.644+0000 7fa48106b740 1 --2- 172.21.15.32:0/1388757003 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x560c421b7f60 0x560c425b2330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.644+0000 7fa48106b740 4 client.15651 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:35:51.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.644+0000 7fa48106b740 1 -- 172.21.15.32:0/1388757003 --> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560c4250d960 con 0x560c421b7f60 2024-04-30T08:35:51.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.644+0000 7fa473e00640 1 --2- 172.21.15.32:0/1388757003 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x560c421b7f60 0x560c425b2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.645+0000 7fa473e00640 1 --2- 172.21.15.32:0/1388757003 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x560c421b7f60 0x560c425b2330 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e9e569650142e840 server_cookie=dcbeb6ad66a1dbc in_seq=0 out_seq=0 2024-04-30T08:35:51.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.645+0000 7fa460c00640 10 client.15651 ms_handle_connect on v2:172.21.15.32:6838/4160386524 2024-04-30T08:35:51.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.646+0000 7fa460c00640 1 -- 172.21.15.32:0/1388757003 <== mds.1 v2:172.21.15.32:6838/4160386524 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560c4250d960 con 0x560c421b7f60 2024-04-30T08:35:51.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.646+0000 7fa460c00640 10 client.15651 handle_command_reply: tid=0 2024-04-30T08:35:51.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.646+0000 7fa48106b740 1 client.15651 shutdown 2024-04-30T08:35:51.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.646+0000 7fa48106b740 1 -- 172.21.15.32:0/1388757003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560c421d86b0 msgr2=0x560c425a4bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:51.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.646+0000 7fa48106b740 1 --2- 172.21.15.32:0/1388757003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560c421d86b0 0x560c425a4bc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa45c011d10 tx=0x7fa45c00f790 comp rx=0 tx=0).stop 2024-04-30T08:35:51.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.646+0000 7fa48106b740 1 -- 172.21.15.32:0/1388757003 shutdown_connections 2024-04-30T08:35:51.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.646+0000 7fa48106b740 1 --2- 172.21.15.32:0/1388757003 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x560c421b7f60 0x560c425b2330 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.646+0000 7fa48106b740 1 --2- 172.21.15.32:0/1388757003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c421f9390 0x560c425abb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.646+0000 7fa48106b740 1 --2- 172.21.15.32:0/1388757003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560c421d86b0 0x560c425a4bc0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.646+0000 7fa48106b740 1 --2- 172.21.15.32:0/1388757003 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560c421d04d0 0x560c425a4680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.646+0000 7fa48106b740 1 -- 172.21.15.32:0/1388757003 >> 172.21.15.32:0/1388757003 conn(0x560c421ed4e0 msgr2=0x560c42595100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:51.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.646+0000 7fa460c00640 0 client.15651 ms_handle_reset on v2:172.21.15.32:6838/4160386524 2024-04-30T08:35:51.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.647+0000 7fa48106b740 1 -- 172.21.15.32:0/1388757003 shutdown_connections 2024-04-30T08:35:51.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.647+0000 7fa48106b740 1 -- 172.21.15.32:0/1388757003 wait complete. 2024-04-30T08:35:51.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.647+0000 7fa48106b740 20 client.15651 trim_cache size 0 max 16384 2024-04-30T08:35:51.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.663+0000 7fa48106b740 1 Processor -- start 2024-04-30T08:35:51.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.663+0000 7fa48106b740 1 -- start start 2024-04-30T08:35:51.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.663+0000 7fa48106b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560c421d04d0 0x560c425a33b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.663+0000 7fa48106b740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560c421d86b0 0x560c425aba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.663+0000 7fa48106b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c421f9390 0x560c425abf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.663+0000 7fa48106b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa4741aeb60 con 0x560c421d04d0 2024-04-30T08:35:51.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.663+0000 7fa48106b740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa4741b06a0 con 0x560c421d86b0 2024-04-30T08:35:51.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.663+0000 7fa48106b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa4741af230 con 0x560c421f9390 2024-04-30T08:35:51.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.664+0000 7fa473e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560c421d04d0 0x560c425a33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.664+0000 7fa46aa00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560c421d86b0 0x560c425aba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.664+0000 7fa473e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560c421d04d0 0x560c425a33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53614/0 (socket says 172.21.15.32:53614) 2024-04-30T08:35:51.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.664+0000 7fa473e00640 1 -- 172.21.15.32:0/4164072290 learned_addr learned my addr 172.21.15.32:0/4164072290 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:51.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.664+0000 7fa473e00640 1 --2- 172.21.15.32:0/4164072290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c421f9390 0x560c425abf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.664+0000 7fa46aa00640 1 -- 172.21.15.32:0/4164072290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c421f9390 msgr2=0x560c425abf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:51.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.664+0000 7fa46aa00640 1 --2- 172.21.15.32:0/4164072290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c421f9390 0x560c425abf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.664+0000 7fa46aa00640 1 -- 172.21.15.32:0/4164072290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560c421d04d0 msgr2=0x560c425a33b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:51.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.664+0000 7fa46aa00640 1 --2- 172.21.15.32:0/4164072290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560c421d04d0 0x560c425a33b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.664+0000 7fa46aa00640 1 -- 172.21.15.32:0/4164072290 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560c425918e0 con 0x560c421d86b0 2024-04-30T08:35:51.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.664+0000 7fa473e00640 1 --2- 172.21.15.32:0/4164072290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560c421d04d0 0x560c425a33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:51.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.664+0000 7fa46aa00640 1 --2- 172.21.15.32:0/4164072290 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560c421d86b0 0x560c425aba40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x560c4259bac0 tx=0x7fa45c013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8486922f2beab0c server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:51.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.665+0000 7fa463400640 1 -- 172.21.15.32:0/4164072290 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45c02f070 con 0x560c421d86b0 2024-04-30T08:35:51.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.665+0000 7fa463400640 1 -- 172.21.15.32:0/4164072290 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa45c003e20 con 0x560c421d86b0 2024-04-30T08:35:51.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.665+0000 7fa463400640 1 -- 172.21.15.32:0/4164072290 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa45c01a3f0 con 0x560c421d86b0 2024-04-30T08:35:51.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.666+0000 7fa48106b740 1 -- 172.21.15.32:0/4164072290 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560c421d86b0 msgr2=0x560c425aba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:51.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.666+0000 7fa48106b740 1 --2- 172.21.15.32:0/4164072290 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560c421d86b0 0x560c425aba40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x560c4259bac0 tx=0x7fa45c013000 comp rx=0 tx=0).stop 2024-04-30T08:35:51.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.666+0000 7fa48106b740 1 -- 172.21.15.32:0/4164072290 shutdown_connections 2024-04-30T08:35:51.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.666+0000 7fa48106b740 1 --2- 172.21.15.32:0/4164072290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c421f9390 0x560c425abf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.666+0000 7fa48106b740 1 --2- 172.21.15.32:0/4164072290 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560c421d86b0 0x560c425aba40 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.666+0000 7fa48106b740 1 --2- 172.21.15.32:0/4164072290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560c421d04d0 0x560c425a33b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.666+0000 7fa48106b740 1 -- 172.21.15.32:0/4164072290 >> 172.21.15.32:0/4164072290 conn(0x560c421ed4e0 msgr2=0x560c425a2ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:51.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.666+0000 7fa48106b740 1 -- 172.21.15.32:0/4164072290 shutdown_connections 2024-04-30T08:35:51.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.666+0000 7fa48106b740 1 -- 172.21.15.32:0/4164072290 wait complete. 2024-04-30T08:35:51.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.666+0000 7fa48106b740 1 -- start start 2024-04-30T08:35:51.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.666+0000 7fa48106b740 1 Processor -- start 2024-04-30T08:35:51.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.667+0000 7fa48106b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560c421d04d0 0x560c425a9810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.667+0000 7fa48106b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c421f9390 0x560c425a9d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.667+0000 7fa48106b740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560c4218d3a0 0x560c425aac80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.667+0000 7fa48106b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa4741affd0 con 0x560c421d04d0 2024-04-30T08:35:51.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.667+0000 7fa48106b740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa4741af900 con 0x560c4218d3a0 2024-04-30T08:35:51.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.667+0000 7fa48106b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa4741acb20 con 0x560c421f9390 2024-04-30T08:35:51.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.667+0000 7fa48106b740 10 client.0 _ll_register_callbacks cb 0x7fa45c009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:35:51.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.667+0000 7fa473e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560c421d04d0 0x560c425a9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.667+0000 7fa473e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560c421d04d0 0x560c425a9810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53630/0 (socket says 172.21.15.32:53630) 2024-04-30T08:35:51.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.667+0000 7fa473e00640 1 -- 172.21.15.32:0/2757406070 learned_addr learned my addr 172.21.15.32:0/2757406070 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:51.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.667+0000 7fa46aa00640 1 --2- 172.21.15.32:0/2757406070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c421f9390 0x560c425a9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.671 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.667+0000 7fa473e00640 1 --2- 172.21.15.32:0/2757406070 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560c4218d3a0 0x560c425aac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.668+0000 7fa473e00640 1 -- 172.21.15.32:0/2757406070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c421f9390 msgr2=0x560c425a9d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:51.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.668+0000 7fa473e00640 1 --2- 172.21.15.32:0/2757406070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c421f9390 0x560c425a9d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.668+0000 7fa473e00640 1 -- 172.21.15.32:0/2757406070 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560c4218d3a0 msgr2=0x560c425aac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:51.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.668+0000 7fa473e00640 1 --2- 172.21.15.32:0/2757406070 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560c4218d3a0 0x560c425aac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.668+0000 7fa473e00640 1 -- 172.21.15.32:0/2757406070 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560c425ab240 con 0x560c421d04d0 2024-04-30T08:35:51.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.668+0000 7fa46aa00640 1 --2- 172.21.15.32:0/2757406070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c421f9390 0x560c425a9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:51.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.668+0000 7fa473e00640 1 --2- 172.21.15.32:0/2757406070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560c421d04d0 0x560c425a9810 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa46c0044d0 tx=0x7fa46c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8b94225926a1031 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:51.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.668+0000 7fa460c00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:35:51.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.668+0000 7fa460c00640 1 -- 172.21.15.32:0/2757406070 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa46c005470 con 0x560c421d04d0 2024-04-30T08:35:51.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.668+0000 7fa460c00640 1 -- 172.21.15.32:0/2757406070 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fa46c004640 con 0x560c421d04d0 2024-04-30T08:35:51.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.668+0000 7fa48106b740 1 -- 172.21.15.32:0/2757406070 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa45c003e20 con 0x560c421d04d0 2024-04-30T08:35:51.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.669+0000 7fa460c00640 1 -- 172.21.15.32:0/2757406070 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa46c004840 con 0x560c421d04d0 2024-04-30T08:35:51.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.669+0000 7fa48106b740 10 client.15657 fetch_fsmap learned FSMap version 28 2024-04-30T08:35:51.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.669+0000 7fa48106b740 1 -- 172.21.15.32:0/2757406070 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x560c425ac4c0 con 0x560c421d04d0 2024-04-30T08:35:51.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.669+0000 7fa460c00640 1 -- 172.21.15.32:0/2757406070 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa45c003e20 con 0x560c421d04d0 2024-04-30T08:35:51.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.669+0000 7fa460c00640 1 -- 172.21.15.32:0/2757406070 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa46c01aa20 con 0x560c421d04d0 2024-04-30T08:35:51.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.669+0000 7fa48106b740 10 client.15657 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:35:51.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.669+0000 7fa48106b740 10 client.15657 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-30T08:35:51.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.669+0000 7fa48106b740 20 client.15657 populate_metadata read hostname 'smithi032' 2024-04-30T08:35:51.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.669+0000 7fa48106b740 1 --2- 172.21.15.32:0/2757406070 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x560c4261a9b0 0x560c4261cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.669+0000 7fa48106b740 4 client.15657 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-30T08:35:51.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.669+0000 7fa48106b740 1 -- 172.21.15.32:0/2757406070 --> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560c425077d0 con 0x560c4261a9b0 2024-04-30T08:35:51.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.670+0000 7fa46aa00640 1 --2- 172.21.15.32:0/2757406070 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x560c4261a9b0 0x560c4261cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.670+0000 7fa46aa00640 1 --2- 172.21.15.32:0/2757406070 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x560c4261a9b0 0x560c4261cdd0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cc49c5804d66f909 server_cookie=ebc495f8bf95474e in_seq=0 out_seq=0 2024-04-30T08:35:51.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.670+0000 7fa460c00640 10 client.15657 ms_handle_connect on v2:172.21.15.32:6838/4160386524 2024-04-30T08:35:51.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.670+0000 7fa460c00640 1 -- 172.21.15.32:0/2757406070 <== mds.1 v2:172.21.15.32:6838/4160386524 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560c425077d0 con 0x560c4261a9b0 2024-04-30T08:35:51.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.670+0000 7fa460c00640 10 client.15657 handle_command_reply: tid=0 2024-04-30T08:35:51.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.671+0000 7fa48106b740 1 client.15657 shutdown 2024-04-30T08:35:51.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.671+0000 7fa48106b740 1 -- 172.21.15.32:0/2757406070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560c421d04d0 msgr2=0x560c425a9810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:51.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.671+0000 7fa48106b740 1 --2- 172.21.15.32:0/2757406070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560c421d04d0 0x560c425a9810 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa46c0044d0 tx=0x7fa46c000bc0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.671+0000 7fa48106b740 1 -- 172.21.15.32:0/2757406070 shutdown_connections 2024-04-30T08:35:51.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.671+0000 7fa48106b740 1 --2- 172.21.15.32:0/2757406070 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x560c4261a9b0 0x560c4261cdd0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.671+0000 7fa48106b740 1 --2- 172.21.15.32:0/2757406070 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560c4218d3a0 0x560c425aac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.675 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.671+0000 7fa48106b740 1 --2- 172.21.15.32:0/2757406070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c421f9390 0x560c425a9d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.671+0000 7fa48106b740 1 --2- 172.21.15.32:0/2757406070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560c421d04d0 0x560c425a9810 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.671+0000 7fa460c00640 0 client.15657 ms_handle_reset on v2:172.21.15.32:6838/4160386524 2024-04-30T08:35:51.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.671+0000 7fa48106b740 1 -- 172.21.15.32:0/2757406070 >> 172.21.15.32:0/2757406070 conn(0x560c421ed4e0 msgr2=0x560c425946d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:51.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.671+0000 7fa48106b740 1 -- 172.21.15.32:0/2757406070 shutdown_connections 2024-04-30T08:35:51.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.671+0000 7fa48106b740 1 -- 172.21.15.32:0/2757406070 wait complete. 2024-04-30T08:35:51.676 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.671+0000 7fa48106b740 20 client.15657 trim_cache size 0 max 16384 2024-04-30T08:35:51.676 INFO:teuthology.orchestra.run.smithi032.stdout:[] 2024-04-30T08:35:51.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.672+0000 7fa478a00640 1 -- 172.21.15.32:0/3982376855 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa4580afb70 msgr2=0x7fa4580b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:51.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.672+0000 7fa478a00640 1 --2- 172.21.15.32:0/3982376855 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa4580afb70 0x7fa4580b2080 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa464006010 tx=0x7fa464009260 comp rx=0 tx=0).stop 2024-04-30T08:35:51.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.672+0000 7fa478a00640 1 -- 172.21.15.32:0/3982376855 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa474105bb0 msgr2=0x7fa47419c2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:51.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.672+0000 7fa478a00640 1 --2- 172.21.15.32:0/3982376855 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa474105bb0 0x7fa47419c2a0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fa464031440 tx=0x7fa464003c10 comp rx=0 tx=0).stop 2024-04-30T08:35:51.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.672+0000 7fa478a00640 1 -- 172.21.15.32:0/3982376855 shutdown_connections 2024-04-30T08:35:51.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.672+0000 7fa478a00640 1 --2- 172.21.15.32:0/3982376855 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa4580afb70 0x7fa4580b2080 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.672+0000 7fa478a00640 1 --2- 172.21.15.32:0/3982376855 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa47410baf0 0x7fa4741a3770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.672+0000 7fa478a00640 1 --2- 172.21.15.32:0/3982376855 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa474106df0 0x7fa47419c7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.677 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.672+0000 7fa478a00640 1 --2- 172.21.15.32:0/3982376855 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa474105bb0 0x7fa47419c2a0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.672+0000 7fa478a00640 1 -- 172.21.15.32:0/3982376855 >> 172.21.15.32:0/3982376855 conn(0x7fa474100da0 msgr2=0x7fa474107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:51.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.672+0000 7fa478a00640 1 -- 172.21.15.32:0/3982376855 shutdown_connections 2024-04-30T08:35:51.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.672+0000 7fa478a00640 1 -- 172.21.15.32:0/3982376855 wait complete. 2024-04-30T08:35:51.690 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:51 smithi032 ceph-mon[28331]: pgmap v445: 129 pgs: 129 active+clean; 662 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-30T08:35:51.700 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-30T08:35:51.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:51 smithi071 ceph-mon[33385]: pgmap v445: 129 pgs: 129 active+clean; 662 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-30T08:35:51.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.979+0000 7f089be00640 1 -- 172.21.15.32:0/2537096293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f089c109fc0 msgr2=0x7f089c10acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:51.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.979+0000 7f089be00640 1 --2- 172.21.15.32:0/2537096293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f089c109fc0 0x7f089c10acc0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f088c009900 tx=0x7f088c02a3f0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.979+0000 7f089be00640 1 -- 172.21.15.32:0/2537096293 shutdown_connections 2024-04-30T08:35:51.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.979+0000 7f089be00640 1 --2- 172.21.15.32:0/2537096293 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f089c10ff00 0x7f089c112320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.979+0000 7f089be00640 1 --2- 172.21.15.32:0/2537096293 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f089c10b200 0x7f089c10b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.979+0000 7f089be00640 1 --2- 172.21.15.32:0/2537096293 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f089c109fc0 0x7f089c10acc0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.979+0000 7f089be00640 1 -- 172.21.15.32:0/2537096293 >> 172.21.15.32:0/2537096293 conn(0x7f089c078bb0 msgr2=0x7f089c107a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:51.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.979+0000 7f089be00640 1 -- 172.21.15.32:0/2537096293 shutdown_connections 2024-04-30T08:35:51.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.979+0000 7f089be00640 1 -- 172.21.15.32:0/2537096293 wait complete. 2024-04-30T08:35:51.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.980+0000 7f089be00640 1 Processor -- start 2024-04-30T08:35:51.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.980+0000 7f089be00640 1 -- start start 2024-04-30T08:35:51.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.981+0000 7f089be00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f089c109fc0 0x7f089c1a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.981+0000 7f089be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f089c10b200 0x7f089c1a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.981+0000 7f089be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f089c10ff00 0x7f089c1a7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.981+0000 7f089be00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f089c115aa0 con 0x7f089c10ff00 2024-04-30T08:35:51.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.981+0000 7f089be00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f089c115920 con 0x7f089c109fc0 2024-04-30T08:35:51.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.981+0000 7f089be00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f089c115c20 con 0x7f089c10b200 2024-04-30T08:35:51.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.981+0000 7f089aa00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f089c109fc0 0x7f089c1a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.981+0000 7f089b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f089c10ff00 0x7f089c1a7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.981+0000 7f089aa00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f089c109fc0 0x7f089c1a06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:54848/0 (socket says 172.21.15.32:54848) 2024-04-30T08:35:51.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.981+0000 7f089aa00640 1 -- 172.21.15.32:0/1241266155 learned_addr learned my addr 172.21.15.32:0/1241266155 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:51.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.981+0000 7f089a000640 1 --2- 172.21.15.32:0/1241266155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f089c10b200 0x7f089c1a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.982+0000 7f089b400640 1 -- 172.21.15.32:0/1241266155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f089c10b200 msgr2=0x7f089c1a0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:51.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.982+0000 7f089b400640 1 --2- 172.21.15.32:0/1241266155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f089c10b200 0x7f089c1a0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.982+0000 7f089b400640 1 -- 172.21.15.32:0/1241266155 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f089c109fc0 msgr2=0x7f089c1a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:51.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.982+0000 7f089b400640 1 --2- 172.21.15.32:0/1241266155 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f089c109fc0 0x7f089c1a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.982+0000 7f089b400640 1 -- 172.21.15.32:0/1241266155 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f089c1a8140 con 0x7f089c10ff00 2024-04-30T08:35:51.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.982+0000 7f089a000640 1 --2- 172.21.15.32:0/1241266155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f089c10b200 0x7f089c1a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:51.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.982+0000 7f089aa00640 1 --2- 172.21.15.32:0/1241266155 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f089c109fc0 0x7f089c1a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:51.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.982+0000 7f089b400640 1 --2- 172.21.15.32:0/1241266155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f089c10ff00 0x7f089c1a7b80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f0894009fd0 tx=0x7f089400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:51.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.982+0000 7f0893400640 1 -- 172.21.15.32:0/1241266155 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f089400cd60 con 0x7f089c10ff00 2024-04-30T08:35:51.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.982+0000 7f089be00640 1 -- 172.21.15.32:0/1241266155 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f089c1a8430 con 0x7f089c10ff00 2024-04-30T08:35:51.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.982+0000 7f0893400640 1 -- 172.21.15.32:0/1241266155 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0894005020 con 0x7f089c10ff00 2024-04-30T08:35:51.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.982+0000 7f0893400640 1 -- 172.21.15.32:0/1241266155 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0894010690 con 0x7f089c10ff00 2024-04-30T08:35:51.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.983+0000 7f089be00640 1 -- 172.21.15.32:0/1241266155 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f089c1a8930 con 0x7f089c10ff00 2024-04-30T08:35:51.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.986+0000 7f08a8a05740 1 Processor -- start 2024-04-30T08:35:51.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.987+0000 7f0893400640 1 -- 172.21.15.32:0/1241266155 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f08940054d0 con 0x7f089c10ff00 2024-04-30T08:35:51.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.987+0000 7f08a8a05740 1 -- start start 2024-04-30T08:35:51.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.987+0000 7f08a8a05740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d7bfba6390 0x55d7bff47730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.988+0000 7f08a8a05740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d7bfb7d4d0 0x55d7bff47cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.988+0000 7f08a8a05740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d7bfb856b0 0x55d7bff4ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.988+0000 7f08a8a05740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d7bfeacd50 con 0x55d7bfb856b0 2024-04-30T08:35:51.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.988+0000 7f08a8a05740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55d7bfe520c0 con 0x55d7bfba6390 2024-04-30T08:35:51.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.988+0000 7f08a8a05740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d7bfea8d10 con 0x55d7bfb7d4d0 2024-04-30T08:35:51.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.988+0000 7f089aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d7bfb7d4d0 0x55d7bff47cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.988+0000 7f089aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d7bfb7d4d0 0x55d7bff47cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:40700/0 (socket says 172.21.15.32:40700) 2024-04-30T08:35:51.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.988+0000 7f089aa00640 1 -- 172.21.15.32:0/28148202 learned_addr learned my addr 172.21.15.32:0/28148202 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:51.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.988+0000 7f089a000640 1 --2- 172.21.15.32:0/28148202 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d7bfba6390 0x55d7bff47730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.988+0000 7f089a000640 1 --2- 172.21.15.32:0/28148202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d7bfb856b0 0x55d7bff4ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.988+0000 7f089aa00640 1 -- 172.21.15.32:0/28148202 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d7bfba6390 msgr2=0x55d7bff47730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:51.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.988+0000 7f089aa00640 1 --2- 172.21.15.32:0/28148202 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d7bfba6390 0x55d7bff47730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.988+0000 7f089aa00640 1 -- 172.21.15.32:0/28148202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d7bfb856b0 msgr2=0x55d7bff4ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:51.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.988+0000 7f089aa00640 1 --2- 172.21.15.32:0/28148202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d7bfb856b0 0x55d7bff4ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.988+0000 7f089aa00640 1 -- 172.21.15.32:0/28148202 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7bfbbe3f0 con 0x55d7bfb7d4d0 2024-04-30T08:35:51.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.989+0000 7f0893400640 1 --2- 172.21.15.32:0/1241266155 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f08800afb70 0x7f08800b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.989+0000 7f0893400640 1 -- 172.21.15.32:0/1241266155 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f08940df920 con 0x7f089c10ff00 2024-04-30T08:35:51.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.989+0000 7f089a000640 1 --2- 172.21.15.32:0/1241266155 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f08800afb70 0x7f08800b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.989+0000 7f089aa00640 1 --2- 172.21.15.32:0/28148202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d7bfb7d4d0 0x55d7bff47cb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f088c004730 tx=0x7f088c009680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=655109b84167a75b server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:51.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.990+0000 7f0887e00640 1 -- 172.21.15.32:0/28148202 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088c035070 con 0x55d7bfb7d4d0 2024-04-30T08:35:51.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.990+0000 7f0887e00640 1 -- 172.21.15.32:0/28148202 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f088c0035f0 con 0x55d7bfb7d4d0 2024-04-30T08:35:51.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.990+0000 7f0887e00640 1 -- 172.21.15.32:0/28148202 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088c03c6a0 con 0x55d7bfb7d4d0 2024-04-30T08:35:51.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.991+0000 7f08a8a05740 1 -- 172.21.15.32:0/28148202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d7bfb7d4d0 msgr2=0x55d7bff47cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:51.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.991+0000 7f08a8a05740 1 --2- 172.21.15.32:0/28148202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d7bfb7d4d0 0x55d7bff47cb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f088c004730 tx=0x7f088c009680 comp rx=0 tx=0).stop 2024-04-30T08:35:51.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.991+0000 7f089a000640 1 --2- 172.21.15.32:0/1241266155 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f08800afb70 0x7f08800b2080 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x55d7bff48900 tx=0x7f0888012040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:51.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.991+0000 7f08a8a05740 1 -- 172.21.15.32:0/28148202 shutdown_connections 2024-04-30T08:35:51.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.991+0000 7f08a8a05740 1 --2- 172.21.15.32:0/28148202 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d7bfb856b0 0x55d7bff4ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.991+0000 7f08a8a05740 1 --2- 172.21.15.32:0/28148202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d7bfb7d4d0 0x55d7bff47cb0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.991+0000 7f08a8a05740 1 --2- 172.21.15.32:0/28148202 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d7bfba6390 0x55d7bff47730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.991+0000 7f08a8a05740 1 -- 172.21.15.32:0/28148202 >> 172.21.15.32:0/28148202 conn(0x55d7bfb9a4e0 msgr2=0x55d7bff430c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:51.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.991+0000 7f08a8a05740 1 -- 172.21.15.32:0/28148202 shutdown_connections 2024-04-30T08:35:51.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.991+0000 7f08a8a05740 1 -- 172.21.15.32:0/28148202 wait complete. 2024-04-30T08:35:51.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.991+0000 7f08a8a05740 1 -- start start 2024-04-30T08:35:51.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.992+0000 7f08a8a05740 1 Processor -- start 2024-04-30T08:35:51.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.992+0000 7f08a8a05740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d7bfb7d4d0 0x55d7bff51580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.992+0000 7f08a8a05740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d7bfb856b0 0x55d7bff51ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.992+0000 7f08a8a05740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d7bfba6390 0x55d7bff58a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.992+0000 7f08a8a05740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d7bff4f5a0 con 0x55d7bfb856b0 2024-04-30T08:35:51.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.992+0000 7f08a8a05740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55d7bfe65b50 con 0x55d7bfb7d4d0 2024-04-30T08:35:51.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.992+0000 7f08a8a05740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d7bff4f720 con 0x55d7bfba6390 2024-04-30T08:35:51.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.992+0000 7f08a8a05740 10 client.0 _ll_register_callbacks cb 0x55d7bfe2f420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:35:51.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.993+0000 7f089a000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d7bfb856b0 0x55d7bff51ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.993+0000 7f089aa00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d7bfb7d4d0 0x55d7bff51580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.993+0000 7f089aa00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d7bfb7d4d0 0x55d7bff51580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:54856/0 (socket says 172.21.15.32:54856) 2024-04-30T08:35:51.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.993+0000 7f089aa00640 1 -- 172.21.15.32:0/611801903 learned_addr learned my addr 172.21.15.32:0/611801903 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:51.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.993+0000 7f089b400640 1 --2- 172.21.15.32:0/611801903 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d7bfba6390 0x55d7bff58a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:51.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.993+0000 7f089aa00640 1 -- 172.21.15.32:0/611801903 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d7bfba6390 msgr2=0x55d7bff58a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:51.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.993+0000 7f089aa00640 1 --2- 172.21.15.32:0/611801903 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d7bfba6390 0x55d7bff58a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.993+0000 7f089aa00640 1 -- 172.21.15.32:0/611801903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d7bfb856b0 msgr2=0x55d7bff51ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:51.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.993+0000 7f089aa00640 1 --2- 172.21.15.32:0/611801903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d7bfb856b0 0x55d7bff51ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:51.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.993+0000 7f089aa00640 1 -- 172.21.15.32:0/611801903 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7bfc887d0 con 0x55d7bfb7d4d0 2024-04-30T08:35:51.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.994+0000 7f089aa00640 1 --2- 172.21.15.32:0/611801903 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d7bfb7d4d0 0x55d7bff51580 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f088c031440 tx=0x7f088c005e40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f5b72f71c44a655 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:51.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.994+0000 7f0884c00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:35:51.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.994+0000 7f0884c00640 1 -- 172.21.15.32:0/611801903 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088c035070 con 0x55d7bfb7d4d0 2024-04-30T08:35:51.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.994+0000 7f0884c00640 1 -- 172.21.15.32:0/611801903 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f088c004300 con 0x55d7bfb7d4d0 2024-04-30T08:35:51.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.994+0000 7f08a8a05740 1 -- 172.21.15.32:0/611801903 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f088c0035f0 con 0x55d7bfb7d4d0 2024-04-30T08:35:51.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.994+0000 7f0884c00640 1 -- 172.21.15.32:0/611801903 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f088c05b050 con 0x55d7bfb7d4d0 2024-04-30T08:35:51.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.994+0000 7f08a8a05740 10 client.25790 fetch_fsmap learned FSMap version 28 2024-04-30T08:35:51.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.994+0000 7f08a8a05740 1 -- 172.21.15.32:0/611801903 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d7bff5b950 con 0x55d7bfb7d4d0 2024-04-30T08:35:51.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.995+0000 7f0884c00640 1 -- 172.21.15.32:0/611801903 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f088c0035f0 con 0x55d7bfb7d4d0 2024-04-30T08:35:51.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.995+0000 7f0884c00640 1 -- 172.21.15.32:0/611801903 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f088c056030 con 0x55d7bfb7d4d0 2024-04-30T08:35:51.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.995+0000 7f08a8a05740 10 client.25790 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:35:51.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.995+0000 7f08a8a05740 10 client.25790 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:35:51.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.995+0000 7f08a8a05740 20 client.25790 populate_metadata read hostname 'smithi032' 2024-04-30T08:35:51.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.995+0000 7f08a8a05740 1 --2- 172.21.15.32:0/611801903 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55d7bfb64f60 0x55d7bff5ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:51.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.995+0000 7f08a8a05740 4 client.25790 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:35:51.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.995+0000 7f08a8a05740 1 -- 172.21.15.32:0/611801903 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d7bfeba960 con 0x55d7bfb64f60 2024-04-30T08:35:51.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.995+0000 7f089a000640 1 --2- 172.21.15.32:0/611801903 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55d7bfb64f60 0x55d7bff5ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:52.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.996+0000 7f089a000640 1 --2- 172.21.15.32:0/611801903 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55d7bfb64f60 0x55d7bff5ecb0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=800bb0187e7f7dd2 server_cookie=e5ca2862df1dd1d0 in_seq=0 out_seq=0 2024-04-30T08:35:52.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.996+0000 7f0884c00640 10 client.25790 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:52.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.997+0000 7f0884c00640 1 -- 172.21.15.32:0/611801903 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d7bfeba960 con 0x55d7bfb64f60 2024-04-30T08:35:52.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.997+0000 7f0884c00640 10 client.25790 handle_command_reply: tid=0 2024-04-30T08:35:52.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.997+0000 7f08a8a05740 1 client.25790 shutdown 2024-04-30T08:35:52.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.997+0000 7f08a8a05740 1 -- 172.21.15.32:0/611801903 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d7bfb7d4d0 msgr2=0x55d7bff51580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:52.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.997+0000 7f08a8a05740 1 --2- 172.21.15.32:0/611801903 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d7bfb7d4d0 0x55d7bff51580 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f088c031440 tx=0x7f088c005e40 comp rx=0 tx=0).stop 2024-04-30T08:35:52.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.997+0000 7f08a8a05740 1 -- 172.21.15.32:0/611801903 shutdown_connections 2024-04-30T08:35:52.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.997+0000 7f08a8a05740 1 --2- 172.21.15.32:0/611801903 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55d7bfb64f60 0x55d7bff5ecb0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:52.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.998+0000 7f08a8a05740 1 --2- 172.21.15.32:0/611801903 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d7bfba6390 0x55d7bff58a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:52.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.998+0000 7f08a8a05740 1 --2- 172.21.15.32:0/611801903 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d7bfb856b0 0x55d7bff51ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:52.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.998+0000 7f08a8a05740 1 --2- 172.21.15.32:0/611801903 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d7bfb7d4d0 0x55d7bff51580 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:52.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.998+0000 7f0884c00640 0 client.25790 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:52.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.998+0000 7f08a8a05740 1 -- 172.21.15.32:0/611801903 >> 172.21.15.32:0/611801903 conn(0x55d7bfb9a4e0 msgr2=0x55d7bff42100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:52.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.998+0000 7f08a8a05740 1 -- 172.21.15.32:0/611801903 shutdown_connections 2024-04-30T08:35:52.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.998+0000 7f08a8a05740 1 -- 172.21.15.32:0/611801903 wait complete. 2024-04-30T08:35:52.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:51.998+0000 7f08a8a05740 20 client.25790 trim_cache size 0 max 16384 2024-04-30T08:35:52.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.013+0000 7f08a8a05740 1 Processor -- start 2024-04-30T08:35:52.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.013+0000 7f08a8a05740 1 -- start start 2024-04-30T08:35:52.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.013+0000 7f08a8a05740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d7bfb7d4d0 0x55d7bff501d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:52.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.013+0000 7f08a8a05740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d7bfb856b0 0x55d7bff58940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:52.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.014+0000 7f08a8a05740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d7bfba6390 0x55d7bff58e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:52.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.014+0000 7f08a8a05740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f089c1b2e70 con 0x55d7bfb856b0 2024-04-30T08:35:52.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.014+0000 7f08a8a05740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f089c1b49b0 con 0x55d7bfb7d4d0 2024-04-30T08:35:52.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.014+0000 7f08a8a05740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f089c1b3540 con 0x55d7bfba6390 2024-04-30T08:35:52.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.014+0000 7f089a000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d7bfb856b0 0x55d7bff58940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:52.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.014+0000 7f089aa00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d7bfb7d4d0 0x55d7bff501d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:52.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.014+0000 7f089b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d7bfba6390 0x55d7bff58e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:52.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.014+0000 7f089a000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d7bfb856b0 0x55d7bff58940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53670/0 (socket says 172.21.15.32:53670) 2024-04-30T08:35:52.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.014+0000 7f089a000640 1 -- 172.21.15.32:0/1078503945 learned_addr learned my addr 172.21.15.32:0/1078503945 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:52.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.014+0000 7f089aa00640 1 -- 172.21.15.32:0/1078503945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d7bfba6390 msgr2=0x55d7bff58e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:52.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.014+0000 7f089aa00640 1 --2- 172.21.15.32:0/1078503945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d7bfba6390 0x55d7bff58e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:52.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.014+0000 7f089aa00640 1 -- 172.21.15.32:0/1078503945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d7bfb856b0 msgr2=0x55d7bff58940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:52.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.014+0000 7f089aa00640 1 --2- 172.21.15.32:0/1078503945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d7bfb856b0 0x55d7bff58940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:52.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.014+0000 7f089aa00640 1 -- 172.21.15.32:0/1078503945 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7bff3e8e0 con 0x55d7bfb7d4d0 2024-04-30T08:35:52.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.014+0000 7f089b400640 1 --2- 172.21.15.32:0/1078503945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d7bfba6390 0x55d7bff58e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:52.017 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.014+0000 7f089a000640 1 --2- 172.21.15.32:0/1078503945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d7bfb856b0 0x55d7bff58940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:52.018 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.015+0000 7f089aa00640 1 --2- 172.21.15.32:0/1078503945 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d7bfb7d4d0 0x55d7bff501d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f088c03ca50 tx=0x7f088c009680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd0bd4d7d3597141 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:52.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.015+0000 7f0887400640 1 -- 172.21.15.32:0/1078503945 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088c035030 con 0x55d7bfb7d4d0 2024-04-30T08:35:52.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.015+0000 7f0887400640 1 -- 172.21.15.32:0/1078503945 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f088c004300 con 0x55d7bfb7d4d0 2024-04-30T08:35:52.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.015+0000 7f0887400640 1 -- 172.21.15.32:0/1078503945 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088c04a040 con 0x55d7bfb7d4d0 2024-04-30T08:35:52.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.016+0000 7f08a8a05740 1 -- 172.21.15.32:0/1078503945 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d7bfb7d4d0 msgr2=0x55d7bff501d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:52.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.016+0000 7f08a8a05740 1 --2- 172.21.15.32:0/1078503945 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d7bfb7d4d0 0x55d7bff501d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f088c03ca50 tx=0x7f088c009680 comp rx=0 tx=0).stop 2024-04-30T08:35:52.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.016+0000 7f08a8a05740 1 -- 172.21.15.32:0/1078503945 shutdown_connections 2024-04-30T08:35:52.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.016+0000 7f08a8a05740 1 --2- 172.21.15.32:0/1078503945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d7bfba6390 0x55d7bff58e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:52.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.016+0000 7f08a8a05740 1 --2- 172.21.15.32:0/1078503945 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d7bfb856b0 0x55d7bff58940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:52.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.016+0000 7f08a8a05740 1 --2- 172.21.15.32:0/1078503945 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d7bfb7d4d0 0x55d7bff501d0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:52.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.016+0000 7f08a8a05740 1 -- 172.21.15.32:0/1078503945 >> 172.21.15.32:0/1078503945 conn(0x55d7bfb9a4e0 msgr2=0x55d7bff4fac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:52.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.016+0000 7f08a8a05740 1 -- 172.21.15.32:0/1078503945 shutdown_connections 2024-04-30T08:35:52.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.016+0000 7f08a8a05740 1 -- 172.21.15.32:0/1078503945 wait complete. 2024-04-30T08:35:52.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.017+0000 7f08a8a05740 1 -- start start 2024-04-30T08:35:52.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.017+0000 7f08a8a05740 1 Processor -- start 2024-04-30T08:35:52.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.017+0000 7f08a8a05740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d7bfb856b0 0x55d7bff564a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:52.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.017+0000 7f08a8a05740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d7bfba6390 0x55d7bff569e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:52.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.018+0000 7f08a8a05740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d7bfb3a3a0 0x55d7bff57910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:52.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.018+0000 7f08a8a05740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f089c1b42e0 con 0x55d7bfb856b0 2024-04-30T08:35:52.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.018+0000 7f08a8a05740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f089c1b3c10 con 0x55d7bfb3a3a0 2024-04-30T08:35:52.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.018+0000 7f08a8a05740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f089c1b0e30 con 0x55d7bfba6390 2024-04-30T08:35:52.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.018+0000 7f08a8a05740 10 client.0 _ll_register_callbacks cb 0x7f088c0315a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:35:52.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.018+0000 7f089aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d7bfb856b0 0x55d7bff564a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:52.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.018+0000 7f089aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d7bfb856b0 0x55d7bff564a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53686/0 (socket says 172.21.15.32:53686) 2024-04-30T08:35:52.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.018+0000 7f089aa00640 1 -- 172.21.15.32:0/3406779732 learned_addr learned my addr 172.21.15.32:0/3406779732 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:52.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.018+0000 7f089b400640 1 --2- 172.21.15.32:0/3406779732 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d7bfb3a3a0 0x55d7bff57910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:52.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.018+0000 7f089a000640 1 --2- 172.21.15.32:0/3406779732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d7bfba6390 0x55d7bff569e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:52.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.018+0000 7f089aa00640 1 -- 172.21.15.32:0/3406779732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d7bfba6390 msgr2=0x55d7bff569e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:52.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.018+0000 7f089aa00640 1 --2- 172.21.15.32:0/3406779732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d7bfba6390 0x55d7bff569e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:52.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.018+0000 7f089aa00640 1 -- 172.21.15.32:0/3406779732 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d7bfb3a3a0 msgr2=0x55d7bff57910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:52.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.018+0000 7f089aa00640 1 --2- 172.21.15.32:0/3406779732 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d7bfb3a3a0 0x55d7bff57910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:52.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.018+0000 7f089aa00640 1 -- 172.21.15.32:0/3406779732 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7bff57ed0 con 0x55d7bfb856b0 2024-04-30T08:35:52.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.018+0000 7f089b400640 1 --2- 172.21.15.32:0/3406779732 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d7bfb3a3a0 0x55d7bff57910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:52.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.019+0000 7f089aa00640 1 --2- 172.21.15.32:0/3406779732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d7bfb856b0 0x55d7bff564a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x55d7bff427a0 tx=0x7f088c005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf1cd2719e084e3e server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:52.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.019+0000 7f0884c00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:35:52.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.019+0000 7f0884c00640 1 -- 172.21.15.32:0/3406779732 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f088c06c060 con 0x55d7bfb856b0 2024-04-30T08:35:52.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.019+0000 7f0884c00640 1 -- 172.21.15.32:0/3406779732 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f088c004100 con 0x55d7bfb856b0 2024-04-30T08:35:52.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.019+0000 7f08a8a05740 1 -- 172.21.15.32:0/3406779732 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f088c004300 con 0x55d7bfb856b0 2024-04-30T08:35:52.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.019+0000 7f0884c00640 1 -- 172.21.15.32:0/3406779732 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f088c073050 con 0x55d7bfb856b0 2024-04-30T08:35:52.027 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.019+0000 7f08a8a05740 10 client.15675 fetch_fsmap learned FSMap version 28 2024-04-30T08:35:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.019+0000 7f08a8a05740 1 -- 172.21.15.32:0/3406779732 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d7bff59180 con 0x55d7bfb856b0 2024-04-30T08:35:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.020+0000 7f0884c00640 1 -- 172.21.15.32:0/3406779732 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f088c004300 con 0x55d7bfb856b0 2024-04-30T08:35:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.020+0000 7f0884c00640 1 -- 172.21.15.32:0/3406779732 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f088c056040 con 0x55d7bfb856b0 2024-04-30T08:35:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.020+0000 7f08a8a05740 10 client.15675 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:35:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.020+0000 7f08a8a05740 10 client.15675 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:35:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.020+0000 7f08a8a05740 20 client.15675 populate_metadata read hostname 'smithi032' 2024-04-30T08:35:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.020+0000 7f08a8a05740 1 --2- 172.21.15.32:0/3406779732 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55d7bffc73e0 0x55d7bffc9800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.020+0000 7f08a8a05740 4 client.15675 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-30T08:35:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.020+0000 7f08a8a05740 1 -- 172.21.15.32:0/3406779732 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d7bfeb47d0 con 0x55d7bffc73e0 2024-04-30T08:35:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.020+0000 7f089a000640 1 --2- 172.21.15.32:0/3406779732 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55d7bffc73e0 0x55d7bffc9800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.021+0000 7f089a000640 1 --2- 172.21.15.32:0/3406779732 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55d7bffc73e0 0x55d7bffc9800 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6679c379ee3ba0a0 server_cookie=2b7b7bc9420a5300 in_seq=0 out_seq=0 2024-04-30T08:35:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.021+0000 7f0884c00640 10 client.15675 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.021+0000 7f0884c00640 1 -- 172.21.15.32:0/3406779732 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d7bfeb47d0 con 0x55d7bffc73e0 2024-04-30T08:35:52.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.021+0000 7f0884c00640 10 client.15675 handle_command_reply: tid=0 2024-04-30T08:35:52.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.021+0000 7f08a8a05740 1 client.15675 shutdown 2024-04-30T08:35:52.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.022+0000 7f08a8a05740 1 -- 172.21.15.32:0/3406779732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d7bfb856b0 msgr2=0x55d7bff564a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:52.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.022+0000 7f08a8a05740 1 --2- 172.21.15.32:0/3406779732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d7bfb856b0 0x55d7bff564a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x55d7bff427a0 tx=0x7f088c005e40 comp rx=0 tx=0).stop 2024-04-30T08:35:52.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.022+0000 7f08a8a05740 1 -- 172.21.15.32:0/3406779732 shutdown_connections 2024-04-30T08:35:52.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.022+0000 7f08a8a05740 1 --2- 172.21.15.32:0/3406779732 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55d7bffc73e0 0x55d7bffc9800 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:52.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.022+0000 7f08a8a05740 1 --2- 172.21.15.32:0/3406779732 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d7bfb3a3a0 0x55d7bff57910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:52.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.022+0000 7f08a8a05740 1 --2- 172.21.15.32:0/3406779732 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d7bfba6390 0x55d7bff569e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:52.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.022+0000 7f08a8a05740 1 --2- 172.21.15.32:0/3406779732 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d7bfb856b0 0x55d7bff564a0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:52.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.022+0000 7f0884c00640 0 client.15675 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:52.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.022+0000 7f08a8a05740 1 -- 172.21.15.32:0/3406779732 >> 172.21.15.32:0/3406779732 conn(0x55d7bfb9a4e0 msgr2=0x55d7bff50250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:52.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.022+0000 7f08a8a05740 1 -- 172.21.15.32:0/3406779732 shutdown_connections 2024-04-30T08:35:52.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.022+0000 7f08a8a05740 1 -- 172.21.15.32:0/3406779732 wait complete. 2024-04-30T08:35:52.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.022+0000 7f08a8a05740 20 client.15675 trim_cache size 0 max 16384 2024-04-30T08:35:52.030 INFO:teuthology.orchestra.run.smithi032.stdout:[] 2024-04-30T08:35:52.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.023+0000 7f089be00640 1 -- 172.21.15.32:0/1241266155 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f08800afb70 msgr2=0x7f08800b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:52.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.023+0000 7f089be00640 1 --2- 172.21.15.32:0/1241266155 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f08800afb70 0x7f08800b2080 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x55d7bff48900 tx=0x7f0888012040 comp rx=0 tx=0).stop 2024-04-30T08:35:52.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.023+0000 7f089be00640 1 -- 172.21.15.32:0/1241266155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f089c10ff00 msgr2=0x7f089c1a7b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:52.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.023+0000 7f089be00640 1 --2- 172.21.15.32:0/1241266155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f089c10ff00 0x7f089c1a7b80 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f0894009fd0 tx=0x7f089400ee50 comp rx=0 tx=0).stop 2024-04-30T08:35:52.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.023+0000 7f089be00640 1 -- 172.21.15.32:0/1241266155 shutdown_connections 2024-04-30T08:35:52.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.023+0000 7f089be00640 1 --2- 172.21.15.32:0/1241266155 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f08800afb70 0x7f08800b2080 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:52.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.023+0000 7f089be00640 1 --2- 172.21.15.32:0/1241266155 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f089c10ff00 0x7f089c1a7b80 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:52.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.023+0000 7f089be00640 1 --2- 172.21.15.32:0/1241266155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f089c10b200 0x7f089c1a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:52.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.023+0000 7f089be00640 1 --2- 172.21.15.32:0/1241266155 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f089c109fc0 0x7f089c1a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:52.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.023+0000 7f089be00640 1 -- 172.21.15.32:0/1241266155 >> 172.21.15.32:0/1241266155 conn(0x7f089c078bb0 msgr2=0x7f089c10bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:52.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.023+0000 7f089be00640 1 -- 172.21.15.32:0/1241266155 shutdown_connections 2024-04-30T08:35:52.031 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:52.023+0000 7f089be00640 1 -- 172.21.15.32:0/1241266155 wait complete. 2024-04-30T08:35:53.050 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-30T08:35:53.051 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-30T08:35:53.051 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-30T08:35:53.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.356+0000 7f4feca00640 1 -- 172.21.15.32:0/3572648209 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4fe8105d80 msgr2=0x7f4fe8106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:53.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.356+0000 7f4feca00640 1 --2- 172.21.15.32:0/3572648209 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4fe8105d80 0x7f4fe8106a80 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8009900 tx=0x7f4fd802a3f0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.356+0000 7f4feca00640 1 -- 172.21.15.32:0/3572648209 shutdown_connections 2024-04-30T08:35:53.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.356+0000 7f4feca00640 1 --2- 172.21.15.32:0/3572648209 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fe810bcc0 0x7f4fe810e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.356+0000 7f4feca00640 1 --2- 172.21.15.32:0/3572648209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fe8106fc0 0x7f4fe81073d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.356+0000 7f4feca00640 1 --2- 172.21.15.32:0/3572648209 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4fe8105d80 0x7f4fe8106a80 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.356+0000 7f4feca00640 1 -- 172.21.15.32:0/3572648209 >> 172.21.15.32:0/3572648209 conn(0x7f4fe8100f70 msgr2=0x7f4fe8103420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:53.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.356+0000 7f4feca00640 1 -- 172.21.15.32:0/3572648209 shutdown_connections 2024-04-30T08:35:53.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.356+0000 7f4feca00640 1 -- 172.21.15.32:0/3572648209 wait complete. 2024-04-30T08:35:53.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.357+0000 7f4feca00640 1 Processor -- start 2024-04-30T08:35:53.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.358+0000 7f4feca00640 1 -- start start 2024-04-30T08:35:53.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.358+0000 7f4feca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fe8105d80 0x7f4fe819c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.358+0000 7f4feca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fe8106fc0 0x7f4fe819ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.358+0000 7f4feca00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4fe810bcc0 0x7f4fe81a39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.360 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.358+0000 7f4feca00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4fe81118b0 con 0x7f4fe8106fc0 2024-04-30T08:35:53.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.358+0000 7f4feca00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4fe8111730 con 0x7f4fe810bcc0 2024-04-30T08:35:53.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.358+0000 7f4feca00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4fe8111a30 con 0x7f4fe8105d80 2024-04-30T08:35:53.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.358+0000 7f4fe5400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fe8106fc0 0x7f4fe819ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:53.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.358+0000 7f4fe5400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fe8106fc0 0x7f4fe819ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53704/0 (socket says 172.21.15.32:53704) 2024-04-30T08:35:53.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.358+0000 7f4fe5400640 1 -- 172.21.15.32:0/3735107279 learned_addr learned my addr 172.21.15.32:0/3735107279 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:53.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.358+0000 7f4fe5e00640 1 --2- 172.21.15.32:0/3735107279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fe8105d80 0x7f4fe819c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:53.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.359+0000 7f4fe6800640 1 --2- 172.21.15.32:0/3735107279 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4fe810bcc0 0x7f4fe81a39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:53.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.359+0000 7f4fe5400640 1 -- 172.21.15.32:0/3735107279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fe8105d80 msgr2=0x7f4fe819c540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:53.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.359+0000 7f4fe5400640 1 --2- 172.21.15.32:0/3735107279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fe8105d80 0x7f4fe819c540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.359+0000 7f4fe5400640 1 -- 172.21.15.32:0/3735107279 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4fe810bcc0 msgr2=0x7f4fe81a39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:53.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.359+0000 7f4fe5400640 1 --2- 172.21.15.32:0/3735107279 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4fe810bcc0 0x7f4fe81a39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.359+0000 7f4fe5400640 1 -- 172.21.15.32:0/3735107279 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fe81a4010 con 0x7f4fe8106fc0 2024-04-30T08:35:53.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.359+0000 7f4fe5400640 1 --2- 172.21.15.32:0/3735107279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fe8106fc0 0x7f4fe819ca80 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4fd0009d50 tx=0x7f4fd000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:53.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.360+0000 7f4fdea00640 1 -- 172.21.15.32:0/3735107279 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd0003b70 con 0x7f4fe8106fc0 2024-04-30T08:35:53.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.360+0000 7f4feca00640 1 -- 172.21.15.32:0/3735107279 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fe81a4300 con 0x7f4fe8106fc0 2024-04-30T08:35:53.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.360+0000 7f4fdea00640 1 -- 172.21.15.32:0/3735107279 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4fd0003d10 con 0x7f4fe8106fc0 2024-04-30T08:35:53.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.360+0000 7f4fdea00640 1 -- 172.21.15.32:0/3735107279 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd00108b0 con 0x7f4fe8106fc0 2024-04-30T08:35:53.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.360+0000 7f4feca00640 1 -- 172.21.15.32:0/3735107279 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fe81a48a0 con 0x7f4fe8106fc0 2024-04-30T08:35:53.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.363+0000 7f4ff3a35740 1 Processor -- start 2024-04-30T08:35:53.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.363+0000 7f4ff3a35740 1 -- start start 2024-04-30T08:35:53.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.364+0000 7f4ff3a35740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ddd813e410 0x55ddd84e01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.364+0000 7f4ff3a35740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ddd8115550 0x55ddd84e0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.364+0000 7f4ff3a35740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ddd811d730 0x55ddd84e7670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.364+0000 7f4ff3a35740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ddd8444b50 con 0x55ddd8115550 2024-04-30T08:35:53.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.364+0000 7f4ff3a35740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55ddd83ea080 con 0x55ddd813e410 2024-04-30T08:35:53.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.364+0000 7f4ff3a35740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ddd8440b10 con 0x55ddd811d730 2024-04-30T08:35:53.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.364+0000 7f4fdea00640 1 -- 172.21.15.32:0/3735107279 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f4fd0004000 con 0x7f4fe8106fc0 2024-04-30T08:35:53.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.364+0000 7f4fe5e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ddd8115550 0x55ddd84e0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:53.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.364+0000 7f4fe5e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ddd8115550 0x55ddd84e0770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53716/0 (socket says 172.21.15.32:53716) 2024-04-30T08:35:53.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.364+0000 7f4fe5e00640 1 -- 172.21.15.32:0/99885138 learned_addr learned my addr 172.21.15.32:0/99885138 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:53.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.364+0000 7f4fe6800640 1 --2- 172.21.15.32:0/99885138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ddd813e410 0x55ddd84e01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:53.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.364+0000 7f4fe5400640 1 --2- 172.21.15.32:0/99885138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ddd811d730 0x55ddd84e7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:53.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.365+0000 7f4fe5e00640 1 -- 172.21.15.32:0/99885138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ddd811d730 msgr2=0x55ddd84e7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:53.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.365+0000 7f4fe5e00640 1 --2- 172.21.15.32:0/99885138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ddd811d730 0x55ddd84e7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.365+0000 7f4fe5e00640 1 -- 172.21.15.32:0/99885138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ddd813e410 msgr2=0x55ddd84e01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:53.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.365+0000 7f4fe5e00640 1 --2- 172.21.15.32:0/99885138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ddd813e410 0x55ddd84e01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.365+0000 7f4fe5e00640 1 -- 172.21.15.32:0/99885138 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddd840a0c0 con 0x55ddd8115550 2024-04-30T08:35:53.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.365+0000 7f4fe5e00640 1 --2- 172.21.15.32:0/99885138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ddd8115550 0x55ddd84e0770 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8009900 tx=0x7f4fd8004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d6a849122a136cc server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:53.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.365+0000 7f4fd7000640 1 -- 172.21.15.32:0/99885138 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd8035030 con 0x55ddd8115550 2024-04-30T08:35:53.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.366+0000 7f4fd7000640 1 -- 172.21.15.32:0/99885138 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4fd8040070 con 0x55ddd8115550 2024-04-30T08:35:53.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.366+0000 7f4fd7000640 1 -- 172.21.15.32:0/99885138 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd8031920 con 0x55ddd8115550 2024-04-30T08:35:53.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.366+0000 7f4fdea00640 1 --2- 172.21.15.32:0/3735107279 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4fc80afc00 0x7f4fc80b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.366+0000 7f4fdea00640 1 -- 172.21.15.32:0/3735107279 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f4fd00df7e0 con 0x7f4fe8106fc0 2024-04-30T08:35:53.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.366+0000 7f4fe6800640 1 --2- 172.21.15.32:0/3735107279 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4fc80afc00 0x7f4fc80b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:53.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.366+0000 7f4ff3a35740 1 -- 172.21.15.32:0/99885138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ddd8115550 msgr2=0x55ddd84e0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:53.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.366+0000 7f4ff3a35740 1 --2- 172.21.15.32:0/99885138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ddd8115550 0x55ddd84e0770 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8009900 tx=0x7f4fd8004040 comp rx=0 tx=0).stop 2024-04-30T08:35:53.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.367+0000 7f4ff3a35740 1 -- 172.21.15.32:0/99885138 shutdown_connections 2024-04-30T08:35:53.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.367+0000 7f4ff3a35740 1 --2- 172.21.15.32:0/99885138 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ddd811d730 0x55ddd84e7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.367+0000 7f4ff3a35740 1 --2- 172.21.15.32:0/99885138 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ddd8115550 0x55ddd84e0770 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.367+0000 7f4ff3a35740 1 --2- 172.21.15.32:0/99885138 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ddd813e410 0x55ddd84e01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.367+0000 7f4ff3a35740 1 -- 172.21.15.32:0/99885138 >> 172.21.15.32:0/99885138 conn(0x55ddd8132560 msgr2=0x55ddd84dbb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:53.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.367+0000 7f4ff3a35740 1 -- 172.21.15.32:0/99885138 shutdown_connections 2024-04-30T08:35:53.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.367+0000 7f4fe6800640 1 --2- 172.21.15.32:0/3735107279 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4fc80afc00 0x7f4fc80b2110 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x55ddd84e13c0 tx=0x7f4fe0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:53.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.367+0000 7f4ff3a35740 1 -- 172.21.15.32:0/99885138 wait complete. 2024-04-30T08:35:53.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.367+0000 7f4ff3a35740 1 -- start start 2024-04-30T08:35:53.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.368+0000 7f4ff3a35740 1 Processor -- start 2024-04-30T08:35:53.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.368+0000 7f4ff3a35740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ddd8115550 0x55ddd84e9d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.368+0000 7f4ff3a35740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ddd811d730 0x55ddd84ea260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.368+0000 7f4ff3a35740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ddd813e410 0x55ddd84f11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.368+0000 7f4ff3a35740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55ddd84e8060 con 0x55ddd811d730 2024-04-30T08:35:53.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.368+0000 7f4ff3a35740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55ddd83fdb20 con 0x55ddd8115550 2024-04-30T08:35:53.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.368+0000 7f4ff3a35740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55ddd84e81e0 con 0x55ddd813e410 2024-04-30T08:35:53.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.368+0000 7f4ff3a35740 10 client.0 _ll_register_callbacks cb 0x55ddd843ba90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:35:53.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.369+0000 7f4fe5e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ddd8115550 0x55ddd84e9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:53.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.369+0000 7f4fe5e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ddd8115550 0x55ddd84e9d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:54900/0 (socket says 172.21.15.32:54900) 2024-04-30T08:35:53.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.369+0000 7f4fe5e00640 1 -- 172.21.15.32:0/2272287174 learned_addr learned my addr 172.21.15.32:0/2272287174 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:53.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.369+0000 7f4fe5400640 1 --2- 172.21.15.32:0/2272287174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ddd811d730 0x55ddd84ea260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:53.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.369+0000 7f4fe6800640 1 --2- 172.21.15.32:0/2272287174 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ddd813e410 0x55ddd84f11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:53.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.369+0000 7f4fe5e00640 1 -- 172.21.15.32:0/2272287174 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ddd813e410 msgr2=0x55ddd84f11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:53.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.369+0000 7f4fe5e00640 1 --2- 172.21.15.32:0/2272287174 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ddd813e410 0x55ddd84f11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.369+0000 7f4fe5e00640 1 -- 172.21.15.32:0/2272287174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ddd811d730 msgr2=0x55ddd84ea260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:53.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.369+0000 7f4fe5e00640 1 --2- 172.21.15.32:0/2272287174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ddd811d730 0x55ddd84ea260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.369+0000 7f4fe5e00640 1 -- 172.21.15.32:0/2272287174 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddd821e7e0 con 0x55ddd8115550 2024-04-30T08:35:53.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.369+0000 7f4fe5e00640 1 --2- 172.21.15.32:0/2272287174 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ddd8115550 0x55ddd84e9d20 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4fd802a430 tx=0x7f4fd8004520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfc78004a630b46b server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:53.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.369+0000 7f4fcf400640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:35:53.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.370+0000 7f4fcf400640 1 -- 172.21.15.32:0/2272287174 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd8035070 con 0x55ddd8115550 2024-04-30T08:35:53.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.370+0000 7f4fcf400640 1 -- 172.21.15.32:0/2272287174 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f4fd8003a00 con 0x55ddd8115550 2024-04-30T08:35:53.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.370+0000 7f4ff3a35740 1 -- 172.21.15.32:0/2272287174 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4fd8040070 con 0x55ddd8115550 2024-04-30T08:35:53.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.370+0000 7f4fcf400640 1 -- 172.21.15.32:0/2272287174 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4fd8003c00 con 0x55ddd8115550 2024-04-30T08:35:53.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.370+0000 7f4ff3a35740 10 client.25808 fetch_fsmap learned FSMap version 28 2024-04-30T08:35:53.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.370+0000 7f4ff3a35740 1 -- 172.21.15.32:0/2272287174 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ddd827ef70 con 0x55ddd8115550 2024-04-30T08:35:53.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.371+0000 7f4fcf400640 1 -- 172.21.15.32:0/2272287174 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4fd8040070 con 0x55ddd8115550 2024-04-30T08:35:53.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.371+0000 7f4fcf400640 1 -- 172.21.15.32:0/2272287174 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f4fd80456e0 con 0x55ddd8115550 2024-04-30T08:35:53.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.371+0000 7f4ff3a35740 10 client.25808 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:35:53.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.371+0000 7f4ff3a35740 10 client.25808 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:35:53.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.371+0000 7f4ff3a35740 20 client.25808 populate_metadata read hostname 'smithi032' 2024-04-30T08:35:53.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.371+0000 7f4ff3a35740 1 --2- 172.21.15.32:0/2272287174 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55ddd80fd1f0 0x55ddd84f7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.371+0000 7f4ff3a35740 4 client.25808 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:35:53.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.371+0000 7f4ff3a35740 1 -- 172.21.15.32:0/2272287174 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ddd8452760 con 0x55ddd80fd1f0 2024-04-30T08:35:53.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.372+0000 7f4fe5400640 1 --2- 172.21.15.32:0/2272287174 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55ddd80fd1f0 0x55ddd84f7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:53.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.372+0000 7f4fe5400640 1 --2- 172.21.15.32:0/2272287174 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55ddd80fd1f0 0x55ddd84f7280 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=977ef9c2448520aa server_cookie=3118a852cf43d6a7 in_seq=0 out_seq=0 2024-04-30T08:35:53.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.372+0000 7f4fcf400640 10 client.25808 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:53.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.373+0000 7f4fcf400640 1 -- 172.21.15.32:0/2272287174 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ddd8452760 con 0x55ddd80fd1f0 2024-04-30T08:35:53.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.373+0000 7f4fcf400640 10 client.25808 handle_command_reply: tid=0 2024-04-30T08:35:53.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.373+0000 7f4ff3a35740 1 client.25808 shutdown 2024-04-30T08:35:53.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.373+0000 7f4ff3a35740 1 -- 172.21.15.32:0/2272287174 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ddd8115550 msgr2=0x55ddd84e9d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:53.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.373+0000 7f4ff3a35740 1 --2- 172.21.15.32:0/2272287174 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ddd8115550 0x55ddd84e9d20 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4fd802a430 tx=0x7f4fd8004520 comp rx=0 tx=0).stop 2024-04-30T08:35:53.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.373+0000 7f4ff3a35740 1 -- 172.21.15.32:0/2272287174 shutdown_connections 2024-04-30T08:35:53.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.373+0000 7f4ff3a35740 1 --2- 172.21.15.32:0/2272287174 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55ddd80fd1f0 0x55ddd84f7280 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.373+0000 7f4ff3a35740 1 --2- 172.21.15.32:0/2272287174 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ddd813e410 0x55ddd84f11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.373+0000 7f4ff3a35740 1 --2- 172.21.15.32:0/2272287174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ddd811d730 0x55ddd84ea260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.373+0000 7f4ff3a35740 1 --2- 172.21.15.32:0/2272287174 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ddd8115550 0x55ddd84e9d20 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.373+0000 7f4ff3a35740 1 -- 172.21.15.32:0/2272287174 >> 172.21.15.32:0/2272287174 conn(0x55ddd8132560 msgr2=0x55ddd84dab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:53.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.373+0000 7f4fcf400640 0 client.25808 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:53.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.373+0000 7f4ff3a35740 1 -- 172.21.15.32:0/2272287174 shutdown_connections 2024-04-30T08:35:53.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.374+0000 7f4ff3a35740 1 -- 172.21.15.32:0/2272287174 wait complete. 2024-04-30T08:35:53.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.374+0000 7f4ff3a35740 20 client.25808 trim_cache size 0 max 16384 2024-04-30T08:35:53.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.394+0000 7f4ff3a35740 1 Processor -- start 2024-04-30T08:35:53.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.394+0000 7f4ff3a35740 1 -- start start 2024-04-30T08:35:53.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.394+0000 7f4ff3a35740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ddd8115550 0x55ddd84ea320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.394+0000 7f4ff3a35740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ddd811d730 0x55ddd84f10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.394+0000 7f4ff3a35740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ddd813e410 0x55ddd84da820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.394+0000 7f4ff3a35740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4fe81aedb0 con 0x55ddd813e410 2024-04-30T08:35:53.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.394+0000 7f4ff3a35740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4fe81b08f0 con 0x55ddd8115550 2024-04-30T08:35:53.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.394+0000 7f4ff3a35740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4fe81af480 con 0x55ddd811d730 2024-04-30T08:35:53.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.395+0000 7f4fe5e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ddd8115550 0x55ddd84ea320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:53.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.395+0000 7f4fe5e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ddd8115550 0x55ddd84ea320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:54912/0 (socket says 172.21.15.32:54912) 2024-04-30T08:35:53.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.395+0000 7f4fe5e00640 1 -- 172.21.15.32:0/3642732707 learned_addr learned my addr 172.21.15.32:0/3642732707 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:53.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.395+0000 7f4fe6800640 1 --2- 172.21.15.32:0/3642732707 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ddd813e410 0x55ddd84da820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:53.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.395+0000 7f4fe5400640 1 --2- 172.21.15.32:0/3642732707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ddd811d730 0x55ddd84f10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:53.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.395+0000 7f4fe5e00640 1 -- 172.21.15.32:0/3642732707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ddd811d730 msgr2=0x55ddd84f10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:53.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.395+0000 7f4fe5e00640 1 --2- 172.21.15.32:0/3642732707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ddd811d730 0x55ddd84f10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.395+0000 7f4fe5e00640 1 -- 172.21.15.32:0/3642732707 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ddd813e410 msgr2=0x55ddd84da820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:53.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.395+0000 7f4fe5e00640 1 --2- 172.21.15.32:0/3642732707 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ddd813e410 0x55ddd84da820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.395+0000 7f4fe5e00640 1 -- 172.21.15.32:0/3642732707 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddd8116170 con 0x55ddd8115550 2024-04-30T08:35:53.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.395+0000 7f4fe5e00640 1 --2- 172.21.15.32:0/3642732707 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ddd8115550 0x55ddd84ea320 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8045b30 tx=0x7f4fd8001e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e9be2e9be2ecdb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:53.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.395+0000 7f4fd6600640 1 -- 172.21.15.32:0/3642732707 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd8035040 con 0x55ddd8115550 2024-04-30T08:35:53.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.396+0000 7f4fd6600640 1 -- 172.21.15.32:0/3642732707 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4fd8040070 con 0x55ddd8115550 2024-04-30T08:35:53.398 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.396+0000 7f4fd6600640 1 -- 172.21.15.32:0/3642732707 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd8031bf0 con 0x55ddd8115550 2024-04-30T08:35:53.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.396+0000 7f4ff3a35740 1 -- 172.21.15.32:0/3642732707 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ddd8115550 msgr2=0x55ddd84ea320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:53.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.396+0000 7f4ff3a35740 1 --2- 172.21.15.32:0/3642732707 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ddd8115550 0x55ddd84ea320 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8045b30 tx=0x7f4fd8001e50 comp rx=0 tx=0).stop 2024-04-30T08:35:53.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.396+0000 7f4ff3a35740 1 -- 172.21.15.32:0/3642732707 shutdown_connections 2024-04-30T08:35:53.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.396+0000 7f4ff3a35740 1 --2- 172.21.15.32:0/3642732707 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ddd813e410 0x55ddd84da820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.400 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.396+0000 7f4ff3a35740 1 --2- 172.21.15.32:0/3642732707 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ddd811d730 0x55ddd84f10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.396+0000 7f4ff3a35740 1 --2- 172.21.15.32:0/3642732707 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ddd8115550 0x55ddd84ea320 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.396+0000 7f4ff3a35740 1 -- 172.21.15.32:0/3642732707 >> 172.21.15.32:0/3642732707 conn(0x55ddd8132560 msgr2=0x55ddd84da110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:53.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.397+0000 7f4ff3a35740 1 -- 172.21.15.32:0/3642732707 shutdown_connections 2024-04-30T08:35:53.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.397+0000 7f4ff3a35740 1 -- 172.21.15.32:0/3642732707 wait complete. 2024-04-30T08:35:53.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.397+0000 7f4ff3a35740 1 -- start start 2024-04-30T08:35:53.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.397+0000 7f4ff3a35740 1 Processor -- start 2024-04-30T08:35:53.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.398+0000 7f4ff3a35740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ddd8115550 0x55ddd84ee180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.398+0000 7f4ff3a35740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ddd811d730 0x55ddd84ee6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.398+0000 7f4ff3a35740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ddd813e410 0x55ddd855d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.398+0000 7f4ff3a35740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4fe81b0220 con 0x55ddd813e410 2024-04-30T08:35:53.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.398+0000 7f4ff3a35740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4fe81afb50 con 0x55ddd8115550 2024-04-30T08:35:53.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.398+0000 7f4ff3a35740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4fe81acd70 con 0x55ddd811d730 2024-04-30T08:35:53.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.398+0000 7f4ff3a35740 10 client.0 _ll_register_callbacks cb 0x7f4fe0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:35:53.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.398+0000 7f4fe6800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ddd813e410 0x55ddd855d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:53.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.398+0000 7f4fe6800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ddd813e410 0x55ddd855d4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53746/0 (socket says 172.21.15.32:53746) 2024-04-30T08:35:53.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.398+0000 7f4fe5e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ddd8115550 0x55ddd84ee180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:53.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.398+0000 7f4fe6800640 1 -- 172.21.15.32:0/878408307 learned_addr learned my addr 172.21.15.32:0/878408307 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:53.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.398+0000 7f4fe5400640 1 --2- 172.21.15.32:0/878408307 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ddd811d730 0x55ddd84ee6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:53.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.399+0000 7f4fe5e00640 1 -- 172.21.15.32:0/878408307 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ddd811d730 msgr2=0x55ddd84ee6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:53.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.399+0000 7f4fe5e00640 1 --2- 172.21.15.32:0/878408307 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ddd811d730 0x55ddd84ee6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.399+0000 7f4fe5e00640 1 -- 172.21.15.32:0/878408307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ddd813e410 msgr2=0x55ddd855d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:53.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.399+0000 7f4fe5e00640 1 --2- 172.21.15.32:0/878408307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ddd813e410 0x55ddd855d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.399+0000 7f4fe5e00640 1 -- 172.21.15.32:0/878408307 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddd855d9f0 con 0x55ddd8115550 2024-04-30T08:35:53.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.399+0000 7f4fe5400640 1 --2- 172.21.15.32:0/878408307 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ddd811d730 0x55ddd84ee6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:53.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.399+0000 7f4fe6800640 1 --2- 172.21.15.32:0/878408307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ddd813e410 0x55ddd855d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:53.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.399+0000 7f4fe5e00640 1 --2- 172.21.15.32:0/878408307 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ddd8115550 0x55ddd84ee180 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8045ab0 tx=0x7f4fd80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d81a24edd8c972 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:53.403 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.399+0000 7f4fcf400640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:35:53.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.399+0000 7f4fcf400640 1 -- 172.21.15.32:0/878408307 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fd8035040 con 0x55ddd8115550 2024-04-30T08:35:53.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.399+0000 7f4ff3a35740 1 -- 172.21.15.32:0/878408307 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4fd8040070 con 0x55ddd8115550 2024-04-30T08:35:53.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.399+0000 7f4fcf400640 1 -- 172.21.15.32:0/878408307 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f4fd803c440 con 0x55ddd8115550 2024-04-30T08:35:53.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.400+0000 7f4fcf400640 1 -- 172.21.15.32:0/878408307 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f4fd80315a0 con 0x55ddd8115550 2024-04-30T08:35:53.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.400+0000 7f4ff3a35740 10 client.25814 fetch_fsmap learned FSMap version 28 2024-04-30T08:35:53.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.400+0000 7f4ff3a35740 1 -- 172.21.15.32:0/878408307 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55ddd855ec60 con 0x55ddd8115550 2024-04-30T08:35:53.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.400+0000 7f4fcf400640 1 -- 172.21.15.32:0/878408307 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4fd8040070 con 0x55ddd8115550 2024-04-30T08:35:53.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.400+0000 7f4fcf400640 1 -- 172.21.15.32:0/878408307 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f4fd803cb30 con 0x55ddd8115550 2024-04-30T08:35:53.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.400+0000 7f4ff3a35740 10 client.25814 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:35:53.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.400+0000 7f4ff3a35740 10 client.25814 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:35:53.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.400+0000 7f4ff3a35740 20 client.25814 populate_metadata read hostname 'smithi032' 2024-04-30T08:35:53.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.400+0000 7f4ff3a35740 1 --2- 172.21.15.32:0/878408307 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55ddd80d2630 0x55ddd8561430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.400+0000 7f4ff3a35740 4 client.25814 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-30T08:35:53.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.400+0000 7f4ff3a35740 1 -- 172.21.15.32:0/878408307 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ddd844c5d0 con 0x55ddd80d2630 2024-04-30T08:35:53.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.400+0000 7f4fe5400640 1 --2- 172.21.15.32:0/878408307 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55ddd80d2630 0x55ddd8561430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:53.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.401+0000 7f4fe5400640 1 --2- 172.21.15.32:0/878408307 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55ddd80d2630 0x55ddd8561430 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dfb5a2311c83acc5 server_cookie=282410ff0f20132f in_seq=0 out_seq=0 2024-04-30T08:35:53.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.401+0000 7f4fcf400640 10 client.25814 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:53.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.402+0000 7f4fcf400640 1 -- 172.21.15.32:0/878408307 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ddd844c5d0 con 0x55ddd80d2630 2024-04-30T08:35:53.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.402+0000 7f4fcf400640 10 client.25814 handle_command_reply: tid=0 2024-04-30T08:35:53.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.402+0000 7f4ff3a35740 1 client.25814 shutdown 2024-04-30T08:35:53.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.402+0000 7f4ff3a35740 1 -- 172.21.15.32:0/878408307 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ddd8115550 msgr2=0x55ddd84ee180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:53.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.402+0000 7f4ff3a35740 1 --2- 172.21.15.32:0/878408307 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ddd8115550 0x55ddd84ee180 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4fd8045ab0 tx=0x7f4fd80046c0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.406 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.402+0000 7f4ff3a35740 1 -- 172.21.15.32:0/878408307 shutdown_connections 2024-04-30T08:35:53.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.402+0000 7f4ff3a35740 1 --2- 172.21.15.32:0/878408307 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55ddd80d2630 0x55ddd8561430 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.402+0000 7f4ff3a35740 1 --2- 172.21.15.32:0/878408307 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55ddd813e410 0x55ddd855d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.402+0000 7f4ff3a35740 1 --2- 172.21.15.32:0/878408307 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ddd811d730 0x55ddd84ee6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.403+0000 7f4fcf400640 0 client.25814 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:53.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.403+0000 7f4ff3a35740 1 --2- 172.21.15.32:0/878408307 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55ddd8115550 0x55ddd84ee180 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.403+0000 7f4ff3a35740 1 -- 172.21.15.32:0/878408307 >> 172.21.15.32:0/878408307 conn(0x55ddd8132560 msgr2=0x55ddd84e9ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:53.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.403+0000 7f4ff3a35740 1 -- 172.21.15.32:0/878408307 shutdown_connections 2024-04-30T08:35:53.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.403+0000 7f4ff3a35740 1 -- 172.21.15.32:0/878408307 wait complete. 2024-04-30T08:35:53.407 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.403+0000 7f4ff3a35740 20 client.25814 trim_cache size 0 max 16384 2024-04-30T08:35:53.407 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-30T08:35:53.408 INFO:teuthology.orchestra.run.smithi032.stdout: "return_code": 0, 2024-04-30T08:35:53.408 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_tag": "6e1b2a4c-dd72-4c81-a1df-67a660fa24a8", 2024-04-30T08:35:53.408 INFO:teuthology.orchestra.run.smithi032.stdout: "mode": "asynchronous" 2024-04-30T08:35:53.408 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-30T08:35:53.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.403+0000 7f4feca00640 1 -- 172.21.15.32:0/3735107279 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4fc80afc00 msgr2=0x7f4fc80b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:53.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.403+0000 7f4feca00640 1 --2- 172.21.15.32:0/3735107279 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4fc80afc00 0x7f4fc80b2110 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x55ddd84e13c0 tx=0x7f4fe0005f90 comp rx=0 tx=0).stop 2024-04-30T08:35:53.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.403+0000 7f4feca00640 1 -- 172.21.15.32:0/3735107279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fe8106fc0 msgr2=0x7f4fe819ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:53.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.403+0000 7f4feca00640 1 --2- 172.21.15.32:0/3735107279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fe8106fc0 0x7f4fe819ca80 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f4fd0009d50 tx=0x7f4fd000c6a0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.404+0000 7f4feca00640 1 -- 172.21.15.32:0/3735107279 shutdown_connections 2024-04-30T08:35:53.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.404+0000 7f4feca00640 1 --2- 172.21.15.32:0/3735107279 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4fc80afc00 0x7f4fc80b2110 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.404+0000 7f4feca00640 1 --2- 172.21.15.32:0/3735107279 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4fe810bcc0 0x7f4fe81a39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.404+0000 7f4feca00640 1 --2- 172.21.15.32:0/3735107279 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4fe8106fc0 0x7f4fe819ca80 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.404+0000 7f4feca00640 1 --2- 172.21.15.32:0/3735107279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4fe8105d80 0x7f4fe819c540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.404+0000 7f4feca00640 1 -- 172.21.15.32:0/3735107279 >> 172.21.15.32:0/3735107279 conn(0x7f4fe8100f70 msgr2=0x7f4fe8107ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:53.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.404+0000 7f4feca00640 1 -- 172.21.15.32:0/3735107279 shutdown_connections 2024-04-30T08:35:53.409 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.404+0000 7f4feca00640 1 -- 172.21.15.32:0/3735107279 wait complete. 2024-04-30T08:35:53.431 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:53 smithi032 ceph-mon[28331]: pgmap v446: 129 pgs: 129 active+clean; 662 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-30T08:35:53.436 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-30T08:35:53.436 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-30T08:35:53.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:53 smithi189 ceph-mon[33038]: pgmap v446: 129 pgs: 129 active+clean; 662 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-30T08:35:53.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.716+0000 7f362d800640 1 -- 172.21.15.32:0/2222222211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36281044a0 msgr2=0x7f36281068c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:53.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.716+0000 7f362d800640 1 --2- 172.21.15.32:0/2222222211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36281044a0 0x7f36281068c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f3618009900 tx=0x7f361802a3f0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.716+0000 7f362d800640 1 -- 172.21.15.32:0/2222222211 shutdown_connections 2024-04-30T08:35:53.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.716+0000 7f362d800640 1 --2- 172.21.15.32:0/2222222211 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3628109760 0x7f362810bb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.716+0000 7f362d800640 1 --2- 172.21.15.32:0/2222222211 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3628106e00 0x7f3628109220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.716+0000 7f362d800640 1 --2- 172.21.15.32:0/2222222211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f36281044a0 0x7f36281068c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.716+0000 7f362d800640 1 -- 172.21.15.32:0/2222222211 >> 172.21.15.32:0/2222222211 conn(0x7f36280fdac0 msgr2=0x7f36280fff50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:53.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.717+0000 7f362d800640 1 -- 172.21.15.32:0/2222222211 shutdown_connections 2024-04-30T08:35:53.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.717+0000 7f362d800640 1 -- 172.21.15.32:0/2222222211 wait complete. 2024-04-30T08:35:53.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.718+0000 7f362d800640 1 Processor -- start 2024-04-30T08:35:53.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.718+0000 7f362d800640 1 -- start start 2024-04-30T08:35:53.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.719+0000 7f362d800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36281044a0 0x7f362819a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.719+0000 7f362d800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3628106e00 0x7f362819a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.719+0000 7f362d800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3628109760 0x7f36281a15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.719+0000 7f362d800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f362810f490 con 0x7f36281044a0 2024-04-30T08:35:53.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.719+0000 7f362d800640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f362810f310 con 0x7f3628109760 2024-04-30T08:35:53.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.719+0000 7f362d800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f362810f610 con 0x7f3628106e00 2024-04-30T08:35:53.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.719+0000 7f3627400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3628106e00 0x7f362819a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:53.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.719+0000 7f3627e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36281044a0 0x7f362819a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:53.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.719+0000 7f3627400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3628106e00 0x7f362819a610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:40834/0 (socket says 172.21.15.32:40834) 2024-04-30T08:35:53.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.719+0000 7f3627400640 1 -- 172.21.15.32:0/3049331367 learned_addr learned my addr 172.21.15.32:0/3049331367 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:53.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.719+0000 7f362ce00640 1 --2- 172.21.15.32:0/3049331367 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3628109760 0x7f36281a15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:53.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.720+0000 7f3627400640 1 -- 172.21.15.32:0/3049331367 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3628109760 msgr2=0x7f36281a15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:53.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.720+0000 7f3627400640 1 --2- 172.21.15.32:0/3049331367 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3628109760 0x7f36281a15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.720+0000 7f3627400640 1 -- 172.21.15.32:0/3049331367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36281044a0 msgr2=0x7f362819a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:53.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.720+0000 7f3627400640 1 --2- 172.21.15.32:0/3049331367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36281044a0 0x7f362819a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.720+0000 7f3627400640 1 -- 172.21.15.32:0/3049331367 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36281a1c10 con 0x7f3628106e00 2024-04-30T08:35:53.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.720+0000 7f3627400640 1 --2- 172.21.15.32:0/3049331367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3628106e00 0x7f362819a610 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f361400ea60 tx=0x7f361400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:53.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.720+0000 7f3624c00640 1 -- 172.21.15.32:0/3049331367 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f361400cde0 con 0x7f3628106e00 2024-04-30T08:35:53.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.720+0000 7f3624c00640 1 -- 172.21.15.32:0/3049331367 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36140035f0 con 0x7f3628106e00 2024-04-30T08:35:53.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.720+0000 7f362d800640 1 -- 172.21.15.32:0/3049331367 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36281a1f00 con 0x7f3628106e00 2024-04-30T08:35:53.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.720+0000 7f3624c00640 1 -- 172.21.15.32:0/3049331367 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3614010690 con 0x7f3628106e00 2024-04-30T08:35:53.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.720+0000 7f362d800640 1 -- 172.21.15.32:0/3049331367 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36281a23c0 con 0x7f3628106e00 2024-04-30T08:35:53.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.724+0000 7f3635e83740 1 Processor -- start 2024-04-30T08:35:53.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.724+0000 7f3635e83740 1 -- start start 2024-04-30T08:35:53.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.724+0000 7f3635e83740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f892068390 0x55f892409440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.724+0000 7f3635e83740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f89203f4d0 0x55f8924099c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.724+0000 7f3635e83740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8920476b0 0x55f8924108c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.724+0000 7f3635e83740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f89236ed50 con 0x55f89203f4d0 2024-04-30T08:35:53.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.724+0000 7f3635e83740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55f892313ef0 con 0x55f892068390 2024-04-30T08:35:53.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.724+0000 7f3635e83740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f89236ad10 con 0x55f8920476b0 2024-04-30T08:35:53.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.725+0000 7f3627e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f89203f4d0 0x55f8924099c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:53.726 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.725+0000 7f362ce00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f892068390 0x55f892409440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:53.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.725+0000 7f362ce00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f892068390 0x55f892409440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:54940/0 (socket says 172.21.15.32:54940) 2024-04-30T08:35:53.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.725+0000 7f362ce00640 1 -- 172.21.15.32:0/3340641070 learned_addr learned my addr 172.21.15.32:0/3340641070 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:53.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.725+0000 7f362ce00640 1 -- 172.21.15.32:0/3340641070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8920476b0 msgr2=0x55f8924108c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-30T08:35:53.728 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.725+0000 7f362ce00640 1 --2- 172.21.15.32:0/3340641070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8920476b0 0x55f8924108c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.725+0000 7f362ce00640 1 -- 172.21.15.32:0/3340641070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f89203f4d0 msgr2=0x55f8924099c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:53.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.725+0000 7f362ce00640 1 --2- 172.21.15.32:0/3340641070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f89203f4d0 0x55f8924099c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.725+0000 7f362ce00640 1 -- 172.21.15.32:0/3340641070 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f89214a7d0 con 0x55f892068390 2024-04-30T08:35:53.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.725+0000 7f3624c00640 1 -- 172.21.15.32:0/3049331367 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f36140037a0 con 0x7f3628106e00 2024-04-30T08:35:53.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.725+0000 7f362ce00640 1 --2- 172.21.15.32:0/3340641070 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f892068390 0x55f892409440 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3620009f90 tx=0x7f36200118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ee1f160fbd4a155e server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:53.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.726+0000 7f361d600640 1 -- 172.21.15.32:0/3340641070 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3620004b20 con 0x55f892068390 2024-04-30T08:35:53.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.726+0000 7f361d600640 1 -- 172.21.15.32:0/3340641070 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3620004cc0 con 0x55f892068390 2024-04-30T08:35:53.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.726+0000 7f361d600640 1 -- 172.21.15.32:0/3340641070 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3620005020 con 0x55f892068390 2024-04-30T08:35:53.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.726+0000 7f3635e83740 1 -- 172.21.15.32:0/3340641070 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f892068390 msgr2=0x55f892409440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:53.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.726+0000 7f3635e83740 1 --2- 172.21.15.32:0/3340641070 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f892068390 0x55f892409440 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f3620009f90 tx=0x7f36200118d0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.727+0000 7f3635e83740 1 -- 172.21.15.32:0/3340641070 shutdown_connections 2024-04-30T08:35:53.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.727+0000 7f3635e83740 1 --2- 172.21.15.32:0/3340641070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8920476b0 0x55f8924108c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.727+0000 7f3635e83740 1 --2- 172.21.15.32:0/3340641070 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f89203f4d0 0x55f8924099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.727+0000 7f3635e83740 1 --2- 172.21.15.32:0/3340641070 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f892068390 0x55f892409440 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.727+0000 7f3635e83740 1 -- 172.21.15.32:0/3340641070 >> 172.21.15.32:0/3340641070 conn(0x55f89205c4e0 msgr2=0x55f892404dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:53.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.727+0000 7f3635e83740 1 -- 172.21.15.32:0/3340641070 shutdown_connections 2024-04-30T08:35:53.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.727+0000 7f3635e83740 1 -- 172.21.15.32:0/3340641070 wait complete. 2024-04-30T08:35:53.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.727+0000 7f3624c00640 1 --2- 172.21.15.32:0/3049331367 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f360c0afb00 0x7f360c0b2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.727+0000 7f3624c00640 1 -- 172.21.15.32:0/3049331367 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f36140df3c0 con 0x7f3628106e00 2024-04-30T08:35:53.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.727+0000 7f3627e00640 1 --2- 172.21.15.32:0/3049331367 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f360c0afb00 0x7f360c0b2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:53.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.727+0000 7f3635e83740 1 -- start start 2024-04-30T08:35:53.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.728+0000 7f3635e83740 1 Processor -- start 2024-04-30T08:35:53.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.728+0000 7f3627e00640 1 --2- 172.21.15.32:0/3049331367 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f360c0afb00 0x7f360c0b2010 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3618009230 tx=0x7f3618031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:53.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.728+0000 7f3635e83740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f89203f4d0 0x55f8924177a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.728+0000 7f3635e83740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8920476b0 0x55f892417ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.728+0000 7f3635e83740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f892068390 0x55f89241ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.728+0000 7f3635e83740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f892411310 con 0x55f8920476b0 2024-04-30T08:35:53.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.728+0000 7f3635e83740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55f892327990 con 0x55f892068390 2024-04-30T08:35:53.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.728+0000 7f3635e83740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f892411490 con 0x55f89203f4d0 2024-04-30T08:35:53.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.728+0000 7f3635e83740 10 client.0 _ll_register_callbacks cb 0x55f892326110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:35:53.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.729+0000 7f3627400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8920476b0 0x55f892417ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:53.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.729+0000 7f362ce00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f892068390 0x55f89241ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:53.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.729+0000 7f362ce00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f892068390 0x55f89241ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:54946/0 (socket says 172.21.15.32:54946) 2024-04-30T08:35:53.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.729+0000 7f3627400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8920476b0 0x55f892417ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53786/0 (socket says 172.21.15.32:53786) 2024-04-30T08:35:53.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.729+0000 7f362ce00640 1 -- 172.21.15.32:0/2127013773 learned_addr learned my addr 172.21.15.32:0/2127013773 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:53.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.729+0000 7f3627e00640 1 --2- 172.21.15.32:0/2127013773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f89203f4d0 0x55f8924177a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:53.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.729+0000 7f362ce00640 1 -- 172.21.15.32:0/2127013773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f89203f4d0 msgr2=0x55f8924177a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:53.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.729+0000 7f362ce00640 1 --2- 172.21.15.32:0/2127013773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f89203f4d0 0x55f8924177a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.729+0000 7f362ce00640 1 -- 172.21.15.32:0/2127013773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8920476b0 msgr2=0x55f892417ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:53.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.729+0000 7f362ce00640 1 --2- 172.21.15.32:0/2127013773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8920476b0 0x55f892417ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.729+0000 7f362ce00640 1 -- 172.21.15.32:0/2127013773 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8920803f0 con 0x55f892068390 2024-04-30T08:35:53.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.729+0000 7f3627400640 1 --2- 172.21.15.32:0/2127013773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8920476b0 0x55f892417ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:53.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.730+0000 7f362ce00640 1 --2- 172.21.15.32:0/2127013773 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f892068390 0x55f89241ec40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3620011dd0 tx=0x7f3620009d20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5b8afaca94b68aa server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:53.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.730+0000 7f3612000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:35:53.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.730+0000 7f3612000640 1 -- 172.21.15.32:0/2127013773 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3620028070 con 0x55f892068390 2024-04-30T08:35:53.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.730+0000 7f3612000640 1 -- 172.21.15.32:0/2127013773 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f3620003dc0 con 0x55f892068390 2024-04-30T08:35:53.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.730+0000 7f3635e83740 1 -- 172.21.15.32:0/2127013773 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3620004cc0 con 0x55f892068390 2024-04-30T08:35:53.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.730+0000 7f3612000640 1 -- 172.21.15.32:0/2127013773 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3620034050 con 0x55f892068390 2024-04-30T08:35:53.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.730+0000 7f3635e83740 10 client.25826 fetch_fsmap learned FSMap version 28 2024-04-30T08:35:53.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.730+0000 7f3635e83740 1 -- 172.21.15.32:0/2127013773 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f8920400f0 con 0x55f892068390 2024-04-30T08:35:53.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.731+0000 7f3612000640 1 -- 172.21.15.32:0/2127013773 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3620004cc0 con 0x55f892068390 2024-04-30T08:35:53.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.731+0000 7f3612000640 1 -- 172.21.15.32:0/2127013773 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f3620005000 con 0x55f892068390 2024-04-30T08:35:53.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.731+0000 7f3635e83740 10 client.25826 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:35:53.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.731+0000 7f3635e83740 10 client.25826 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:35:53.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.731+0000 7f3635e83740 20 client.25826 populate_metadata read hostname 'smithi032' 2024-04-30T08:35:53.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.731+0000 7f3635e83740 1 --2- 172.21.15.32:0/2127013773 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55f892026f60 0x55f892424e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.731+0000 7f3635e83740 4 client.25826 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:35:53.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.731+0000 7f3635e83740 1 -- 172.21.15.32:0/2127013773 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f892382ef0 con 0x55f892026f60 2024-04-30T08:35:53.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.731+0000 7f3627e00640 1 --2- 172.21.15.32:0/2127013773 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55f892026f60 0x55f892424e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:53.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.732+0000 7f3627e00640 1 --2- 172.21.15.32:0/2127013773 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55f892026f60 0x55f892424e30 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=667a3ec6ad776a3d server_cookie=c7530f7e9ef39303 in_seq=0 out_seq=0 2024-04-30T08:35:53.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.732+0000 7f3612000640 10 client.25826 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:53.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.733+0000 7f3612000640 1 -- 172.21.15.32:0/2127013773 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f892382ef0 con 0x55f892026f60 2024-04-30T08:35:53.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.733+0000 7f3612000640 10 client.25826 handle_command_reply: tid=0 2024-04-30T08:35:53.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.733+0000 7f3635e83740 1 client.25826 shutdown 2024-04-30T08:35:53.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.733+0000 7f3635e83740 1 -- 172.21.15.32:0/2127013773 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f892068390 msgr2=0x55f89241ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:53.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.733+0000 7f3635e83740 1 --2- 172.21.15.32:0/2127013773 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f892068390 0x55f89241ec40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f3620011dd0 tx=0x7f3620009d20 comp rx=0 tx=0).stop 2024-04-30T08:35:53.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.733+0000 7f3635e83740 1 -- 172.21.15.32:0/2127013773 shutdown_connections 2024-04-30T08:35:53.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.733+0000 7f3635e83740 1 --2- 172.21.15.32:0/2127013773 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55f892026f60 0x55f892424e30 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.733+0000 7f3635e83740 1 --2- 172.21.15.32:0/2127013773 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f892068390 0x55f89241ec40 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.733+0000 7f3635e83740 1 --2- 172.21.15.32:0/2127013773 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8920476b0 0x55f892417ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.733+0000 7f3635e83740 1 --2- 172.21.15.32:0/2127013773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f89203f4d0 0x55f8924177a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.733+0000 7f3635e83740 1 -- 172.21.15.32:0/2127013773 >> 172.21.15.32:0/2127013773 conn(0x55f89205c4e0 msgr2=0x55f892403e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:53.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.733+0000 7f3612000640 0 client.25826 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:53.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.733+0000 7f3635e83740 1 -- 172.21.15.32:0/2127013773 shutdown_connections 2024-04-30T08:35:53.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.733+0000 7f3635e83740 1 -- 172.21.15.32:0/2127013773 wait complete. 2024-04-30T08:35:53.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.734+0000 7f3635e83740 20 client.25826 trim_cache size 0 max 16384 2024-04-30T08:35:53.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:53 smithi071 ceph-mon[33385]: pgmap v446: 129 pgs: 129 active+clean; 662 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-30T08:35:53.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.753+0000 7f3635e83740 1 Processor -- start 2024-04-30T08:35:53.755 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.753+0000 7f3635e83740 1 -- start start 2024-04-30T08:35:53.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.754+0000 7f3635e83740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f89203f4d0 0x55f8924123e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.754+0000 7f3635e83740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8920476b0 0x55f89240de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.754+0000 7f3635e83740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f892068390 0x55f8924177a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.754+0000 7f3635e83740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f36281aa850 con 0x55f892068390 2024-04-30T08:35:53.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.754+0000 7f3635e83740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f36281ac390 con 0x55f89203f4d0 2024-04-30T08:35:53.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.754+0000 7f3635e83740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f36281aaf20 con 0x55f8920476b0 2024-04-30T08:35:53.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.754+0000 7f362ce00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f892068390 0x55f8924177a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:53.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.754+0000 7f3627400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8920476b0 0x55f89240de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:53.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.754+0000 7f3627400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8920476b0 0x55f89240de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:40858/0 (socket says 172.21.15.32:40858) 2024-04-30T08:35:53.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.754+0000 7f3627400640 1 -- 172.21.15.32:0/3757511470 learned_addr learned my addr 172.21.15.32:0/3757511470 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:53.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.754+0000 7f3627e00640 1 --2- 172.21.15.32:0/3757511470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f89203f4d0 0x55f8924123e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:53.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.754+0000 7f3627400640 1 -- 172.21.15.32:0/3757511470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f89203f4d0 msgr2=0x55f8924123e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:53.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.754+0000 7f3627400640 1 --2- 172.21.15.32:0/3757511470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f89203f4d0 0x55f8924123e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.754+0000 7f3627400640 1 -- 172.21.15.32:0/3757511470 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f892068390 msgr2=0x55f8924177a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:53.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.755+0000 7f3627400640 1 --2- 172.21.15.32:0/3757511470 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f892068390 0x55f8924177a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.755+0000 7f3627400640 1 -- 172.21.15.32:0/3757511470 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f892409e10 con 0x55f8920476b0 2024-04-30T08:35:53.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.755+0000 7f3627400640 1 --2- 172.21.15.32:0/3757511470 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8920476b0 0x55f89240de80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f361400c760 tx=0x7f36140a6cc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74a141d9bb5156e5 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:53.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.755+0000 7f361cc00640 1 -- 172.21.15.32:0/3757511470 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36140c0070 con 0x55f8920476b0 2024-04-30T08:35:53.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.755+0000 7f361cc00640 1 -- 172.21.15.32:0/3757511470 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36140a7a70 con 0x55f8920476b0 2024-04-30T08:35:53.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.755+0000 7f361cc00640 1 -- 172.21.15.32:0/3757511470 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36140a3600 con 0x55f8920476b0 2024-04-30T08:35:53.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.756+0000 7f3635e83740 1 -- 172.21.15.32:0/3757511470 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8920476b0 msgr2=0x55f89240de80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:53.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.756+0000 7f3635e83740 1 --2- 172.21.15.32:0/3757511470 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8920476b0 0x55f89240de80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f361400c760 tx=0x7f36140a6cc0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.756+0000 7f3635e83740 1 -- 172.21.15.32:0/3757511470 shutdown_connections 2024-04-30T08:35:53.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.756+0000 7f3635e83740 1 --2- 172.21.15.32:0/3757511470 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f892068390 0x55f8924177a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.756+0000 7f3635e83740 1 --2- 172.21.15.32:0/3757511470 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8920476b0 0x55f89240de80 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.756+0000 7f3635e83740 1 --2- 172.21.15.32:0/3757511470 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f89203f4d0 0x55f8924123e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.756+0000 7f3635e83740 1 -- 172.21.15.32:0/3757511470 >> 172.21.15.32:0/3757511470 conn(0x55f89205c4e0 msgr2=0x55f892407c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:53.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.756+0000 7f3635e83740 1 -- 172.21.15.32:0/3757511470 shutdown_connections 2024-04-30T08:35:53.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.756+0000 7f3635e83740 1 -- 172.21.15.32:0/3757511470 wait complete. 2024-04-30T08:35:53.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.757+0000 7f3635e83740 1 -- start start 2024-04-30T08:35:53.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.757+0000 7f3635e83740 1 Processor -- start 2024-04-30T08:35:53.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.757+0000 7f3635e83740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f89203f4d0 0x55f89241a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.757+0000 7f3635e83740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8920476b0 0x55f89241a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.757+0000 7f3635e83740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f892068390 0x55f89248f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.757+0000 7f3635e83740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f36281abcc0 con 0x55f89203f4d0 2024-04-30T08:35:53.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.757+0000 7f3635e83740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f36281ab5f0 con 0x55f8920476b0 2024-04-30T08:35:53.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.757+0000 7f3635e83740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f36281a8810 con 0x55f892068390 2024-04-30T08:35:53.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.757+0000 7f3635e83740 10 client.0 _ll_register_callbacks cb 0x7f36200093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:35:53.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.758+0000 7f3627e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f89203f4d0 0x55f89241a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:53.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.758+0000 7f362ce00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f892068390 0x55f89248f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:53.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.758+0000 7f3627e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f89203f4d0 0x55f89241a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:53804/0 (socket says 172.21.15.32:53804) 2024-04-30T08:35:53.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.758+0000 7f3627e00640 1 -- 172.21.15.32:0/3450583385 learned_addr learned my addr 172.21.15.32:0/3450583385 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:53.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.758+0000 7f3627400640 1 --2- 172.21.15.32:0/3450583385 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8920476b0 0x55f89241a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:53.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.758+0000 7f362ce00640 1 -- 172.21.15.32:0/3450583385 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8920476b0 msgr2=0x55f89241a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:53.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.758+0000 7f362ce00640 1 --2- 172.21.15.32:0/3450583385 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8920476b0 0x55f89241a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.758+0000 7f362ce00640 1 -- 172.21.15.32:0/3450583385 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f89203f4d0 msgr2=0x55f89241a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:53.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.758+0000 7f362ce00640 1 --2- 172.21.15.32:0/3450583385 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f89203f4d0 0x55f89241a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.758+0000 7f362ce00640 1 -- 172.21.15.32:0/3450583385 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f89241f650 con 0x55f892068390 2024-04-30T08:35:53.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.758+0000 7f3627e00640 1 --2- 172.21.15.32:0/3450583385 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f89203f4d0 0x55f89241a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:53.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.758+0000 7f362ce00640 1 --2- 172.21.15.32:0/3450583385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f892068390 0x55f89248f980 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3620005400 tx=0x7f3620003a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac8e5015b420e4e7 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:53.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.758+0000 7f3612000640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-30T08:35:53.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.759+0000 7f3612000640 1 -- 172.21.15.32:0/3450583385 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3620028040 con 0x55f892068390 2024-04-30T08:35:53.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.759+0000 7f3612000640 1 -- 172.21.15.32:0/3450583385 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f3620004fc0 con 0x55f892068390 2024-04-30T08:35:53.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.759+0000 7f3635e83740 1 -- 172.21.15.32:0/3450583385 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36140a7a70 con 0x55f892068390 2024-04-30T08:35:53.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.759+0000 7f3612000640 1 -- 172.21.15.32:0/3450583385 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f3620009e60 con 0x55f892068390 2024-04-30T08:35:53.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.759+0000 7f3635e83740 10 client.25834 fetch_fsmap learned FSMap version 28 2024-04-30T08:35:53.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.759+0000 7f3635e83740 1 -- 172.21.15.32:0/3450583385 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55f892490d70 con 0x55f892068390 2024-04-30T08:35:53.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.759+0000 7f3612000640 1 -- 172.21.15.32:0/3450583385 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f36140a7a70 con 0x55f892068390 2024-04-30T08:35:53.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.760+0000 7f3612000640 1 -- 172.21.15.32:0/3450583385 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f362000f570 con 0x55f892068390 2024-04-30T08:35:53.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.760+0000 7f3635e83740 10 client.25834 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:35:53.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.760+0000 7f3635e83740 10 client.25834 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:35:53.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.760+0000 7f3635e83740 20 client.25834 populate_metadata read hostname 'smithi032' 2024-04-30T08:35:53.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.760+0000 7f3635e83740 1 --2- 172.21.15.32:0/3450583385 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55f891ffc3a0 0x55f8924936a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:53.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.760+0000 7f3635e83740 4 client.25834 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-30T08:35:53.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.760+0000 7f3635e83740 1 -- 172.21.15.32:0/3450583385 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55f89237c960 con 0x55f891ffc3a0 2024-04-30T08:35:53.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.760+0000 7f3627e00640 1 --2- 172.21.15.32:0/3450583385 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55f891ffc3a0 0x55f8924936a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:53.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.761+0000 7f3627e00640 1 --2- 172.21.15.32:0/3450583385 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55f891ffc3a0 0x55f8924936a0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e7ed29ace6e5530b server_cookie=1715204e7ba45ee in_seq=0 out_seq=0 2024-04-30T08:35:53.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.761+0000 7f3612000640 10 client.25834 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:53.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.761+0000 7f3612000640 1 -- 172.21.15.32:0/3450583385 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55f89237c960 con 0x55f891ffc3a0 2024-04-30T08:35:53.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.761+0000 7f3612000640 10 client.25834 handle_command_reply: tid=0 2024-04-30T08:35:53.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.761+0000 7f3635e83740 1 client.25834 shutdown 2024-04-30T08:35:53.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.761+0000 7f3635e83740 1 -- 172.21.15.32:0/3450583385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f892068390 msgr2=0x55f89248f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:53.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.761+0000 7f3635e83740 1 --2- 172.21.15.32:0/3450583385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f892068390 0x55f89248f980 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f3620005400 tx=0x7f3620003a30 comp rx=0 tx=0).stop 2024-04-30T08:35:53.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.761+0000 7f3635e83740 1 -- 172.21.15.32:0/3450583385 shutdown_connections 2024-04-30T08:35:53.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.761+0000 7f3635e83740 1 --2- 172.21.15.32:0/3450583385 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55f891ffc3a0 0x55f8924936a0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.761+0000 7f3635e83740 1 --2- 172.21.15.32:0/3450583385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f892068390 0x55f89248f980 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.761+0000 7f3635e83740 1 --2- 172.21.15.32:0/3450583385 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8920476b0 0x55f89241a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.761+0000 7f3635e83740 1 --2- 172.21.15.32:0/3450583385 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f89203f4d0 0x55f89241a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.761+0000 7f3612000640 0 client.25834 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:53.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.761+0000 7f3635e83740 1 -- 172.21.15.32:0/3450583385 >> 172.21.15.32:0/3450583385 conn(0x55f89205c4e0 msgr2=0x55f892407c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:53.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.761+0000 7f3635e83740 1 -- 172.21.15.32:0/3450583385 shutdown_connections 2024-04-30T08:35:53.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.761+0000 7f3635e83740 1 -- 172.21.15.32:0/3450583385 wait complete. 2024-04-30T08:35:53.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.761+0000 7f3635e83740 20 client.25834 trim_cache size 0 max 16384 2024-04-30T08:35:53.769 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-30T08:35:53.769 INFO:teuthology.orchestra.run.smithi032.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-30T08:35:53.769 INFO:teuthology.orchestra.run.smithi032.stdout: "scrubs": { 2024-04-30T08:35:53.769 INFO:teuthology.orchestra.run.smithi032.stdout: "6e1b2a4c-dd72-4c81-a1df-67a660fa24a8": { 2024-04-30T08:35:53.769 INFO:teuthology.orchestra.run.smithi032.stdout: "path": "/", 2024-04-30T08:35:53.769 INFO:teuthology.orchestra.run.smithi032.stdout: "tag": "6e1b2a4c-dd72-4c81-a1df-67a660fa24a8", 2024-04-30T08:35:53.770 INFO:teuthology.orchestra.run.smithi032.stdout: "options": "recursive,force" 2024-04-30T08:35:53.770 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:35:53.770 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:35:53.770 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-30T08:35:53.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.762+0000 7f362d800640 1 -- 172.21.15.32:0/3049331367 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f360c0afb00 msgr2=0x7f360c0b2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:53.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.762+0000 7f362d800640 1 --2- 172.21.15.32:0/3049331367 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f360c0afb00 0x7f360c0b2010 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3618009230 tx=0x7f3618031400 comp rx=0 tx=0).stop 2024-04-30T08:35:53.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.762+0000 7f362d800640 1 -- 172.21.15.32:0/3049331367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3628106e00 msgr2=0x7f362819a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:53.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.762+0000 7f362d800640 1 --2- 172.21.15.32:0/3049331367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3628106e00 0x7f362819a610 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f361400ea60 tx=0x7f361400ef20 comp rx=0 tx=0).stop 2024-04-30T08:35:53.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.762+0000 7f362d800640 1 -- 172.21.15.32:0/3049331367 shutdown_connections 2024-04-30T08:35:53.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.762+0000 7f362d800640 1 --2- 172.21.15.32:0/3049331367 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f360c0afb00 0x7f360c0b2010 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.762+0000 7f362d800640 1 --2- 172.21.15.32:0/3049331367 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3628109760 0x7f36281a15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.762+0000 7f362d800640 1 --2- 172.21.15.32:0/3049331367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3628106e00 0x7f362819a610 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.762+0000 7f362d800640 1 --2- 172.21.15.32:0/3049331367 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f36281044a0 0x7f362819a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:53.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.762+0000 7f362d800640 1 -- 172.21.15.32:0/3049331367 >> 172.21.15.32:0/3049331367 conn(0x7f36280fdac0 msgr2=0x7f36281050f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:53.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.762+0000 7f362d800640 1 -- 172.21.15.32:0/3049331367 shutdown_connections 2024-04-30T08:35:53.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:35:53.762+0000 7f362d800640 1 -- 172.21.15.32:0/3049331367 wait complete. 2024-04-30T08:35:53.789 INFO:tasks.cephfs.filesystem:scrub status for tag:6e1b2a4c-dd72-4c81-a1df-67a660fa24a8 - {'path': '/', 'tag': '6e1b2a4c-dd72-4c81-a1df-67a660fa24a8', 'options': 'recursive,force'} 2024-04-30T08:35:54.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:54 smithi189 ceph-mon[33038]: scrub queued for path: / 2024-04-30T08:35:54.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:54 smithi189 ceph-mon[33038]: scrub summary: idle+waiting paths [/] 2024-04-30T08:35:54.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:54 smithi189 ceph-mon[33038]: scrub summary: active paths [/] 2024-04-30T08:35:54.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:54 smithi189 ceph-mon[33038]: scrub summary: idle+waiting paths [/] 2024-04-30T08:35:54.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:54 smithi032 ceph-mon[28331]: scrub queued for path: / 2024-04-30T08:35:54.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:54 smithi032 ceph-mon[28331]: scrub summary: idle+waiting paths [/] 2024-04-30T08:35:54.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:54 smithi032 ceph-mon[28331]: scrub summary: active paths [/] 2024-04-30T08:35:54.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:54 smithi032 ceph-mon[28331]: scrub summary: idle+waiting paths [/] 2024-04-30T08:35:54.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:54 smithi071 ceph-mon[33385]: scrub queued for path: / 2024-04-30T08:35:54.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:54 smithi071 ceph-mon[33385]: scrub summary: idle+waiting paths [/] 2024-04-30T08:35:54.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:54 smithi071 ceph-mon[33385]: scrub summary: active paths [/] 2024-04-30T08:35:54.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:54 smithi071 ceph-mon[33385]: scrub summary: idle+waiting paths [/] 2024-04-30T08:35:55.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:55 smithi189 ceph-mon[33038]: pgmap v447: 129 pgs: 129 active+clean; 662 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-30T08:35:55.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:55 smithi189 ceph-mon[33038]: scrub summary: idle 2024-04-30T08:35:55.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:55 smithi032 ceph-mon[28331]: pgmap v447: 129 pgs: 129 active+clean; 662 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-30T08:35:55.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:55 smithi032 ceph-mon[28331]: scrub summary: idle 2024-04-30T08:35:55.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:55 smithi071 ceph-mon[33385]: pgmap v447: 129 pgs: 129 active+clean; 662 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 255 B/s wr, 10 op/s 2024-04-30T08:35:55.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:55 smithi071 ceph-mon[33385]: scrub summary: idle 2024-04-30T08:35:57.556 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:57 smithi071 ceph-mon[33385]: pgmap v448: 129 pgs: 129 active+clean; 672 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-04-30T08:35:57.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:57 smithi032 ceph-mon[28331]: pgmap v448: 129 pgs: 129 active+clean; 672 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-04-30T08:35:57.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:57 smithi189 ceph-mon[33038]: pgmap v448: 129 pgs: 129 active+clean; 672 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-04-30T08:35:58.423 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.421+0000 7f53f6200640 1 -- 172.21.15.32:0/1563991728 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53f0104df0 msgr2=0x7f53f0105200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:58.423 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.421+0000 7f53f6200640 1 --2- 172.21.15.32:0/1563991728 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53f0104df0 0x7f53f0105200 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f53d8009860 tx=0x7f53d802a340 comp rx=0 tx=0).stop 2024-04-30T08:35:58.423 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.421+0000 7f53f6200640 1 -- 172.21.15.32:0/1563991728 shutdown_connections 2024-04-30T08:35:58.423 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.421+0000 7f53f6200640 1 --2- 172.21.15.32:0/1563991728 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53f0109af0 0x7f53f010bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:58.423 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.421+0000 7f53f6200640 1 --2- 172.21.15.32:0/1563991728 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53f0104df0 0x7f53f0105200 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:58.423 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.421+0000 7f53f6200640 1 --2- 172.21.15.32:0/1563991728 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53f0103bb0 0x7f53f01048b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:58.423 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.421+0000 7f53f6200640 1 -- 172.21.15.32:0/1563991728 >> 172.21.15.32:0/1563991728 conn(0x7f53f00fedc0 msgr2=0x7f53f0101250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.421+0000 7f53f6200640 1 -- 172.21.15.32:0/1563991728 shutdown_connections 2024-04-30T08:35:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.421+0000 7f53f6200640 1 -- 172.21.15.32:0/1563991728 wait complete. 2024-04-30T08:35:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.423+0000 7f53f6200640 1 Processor -- start 2024-04-30T08:35:58.424 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.423+0000 7f53f6200640 1 -- start start 2024-04-30T08:35:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.424+0000 7f53f6200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53f0103bb0 0x7f53f019c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.424+0000 7f53f6200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53f0104df0 0x7f53f019c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.424+0000 7f53f6200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53f0109af0 0x7f53f01a3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:58.427 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.424+0000 7f53f6200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f53f010f6b0 con 0x7f53f0109af0 2024-04-30T08:35:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.424+0000 7f53f6200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f53f010f530 con 0x7f53f0104df0 2024-04-30T08:35:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.424+0000 7f53f6200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f53f010f830 con 0x7f53f0103bb0 2024-04-30T08:35:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.424+0000 7f53ef400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53f0103bb0 0x7f53f019c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.424+0000 7f53eea00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53f0104df0 0x7f53f019c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.424+0000 7f53efe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53f0109af0 0x7f53f01a3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.424+0000 7f53eea00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53f0104df0 0x7f53f019c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:35150/0 (socket says 172.21.15.32:35150) 2024-04-30T08:35:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.424+0000 7f53eea00640 1 -- 172.21.15.32:0/502829495 learned_addr learned my addr 172.21.15.32:0/502829495 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.424+0000 7f53ef400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53f0103bb0 0x7f53f019c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:52600/0 (socket says 172.21.15.32:52600) 2024-04-30T08:35:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.425+0000 7f53eea00640 1 -- 172.21.15.32:0/502829495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53f0103bb0 msgr2=0x7f53f019c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.425+0000 7f53eea00640 1 --2- 172.21.15.32:0/502829495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53f0103bb0 0x7f53f019c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.425+0000 7f53eea00640 1 -- 172.21.15.32:0/502829495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53f0109af0 msgr2=0x7f53f01a3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.425+0000 7f53eea00640 1 --2- 172.21.15.32:0/502829495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53f0109af0 0x7f53f01a3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.425+0000 7f53eea00640 1 -- 172.21.15.32:0/502829495 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53f01a3fe0 con 0x7f53f0104df0 2024-04-30T08:35:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.425+0000 7f53efe00640 1 --2- 172.21.15.32:0/502829495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53f0109af0 0x7f53f01a3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:58.428 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.425+0000 7f53eea00640 1 --2- 172.21.15.32:0/502829495 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53f0104df0 0x7f53f019c9e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f53d8032d50 tx=0x7f53d8032d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.425+0000 7f53e7e00640 1 -- 172.21.15.32:0/502829495 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d8037040 con 0x7f53f0104df0 2024-04-30T08:35:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.425+0000 7f53e7e00640 1 -- 172.21.15.32:0/502829495 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53d8030550 con 0x7f53f0104df0 2024-04-30T08:35:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.425+0000 7f53e7e00640 1 -- 172.21.15.32:0/502829495 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d8003850 con 0x7f53f0104df0 2024-04-30T08:35:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.425+0000 7f53f6200640 1 -- 172.21.15.32:0/502829495 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53f01a4270 con 0x7f53f0104df0 2024-04-30T08:35:58.429 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.426+0000 7f53f6200640 1 -- 172.21.15.32:0/502829495 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53f01a46b0 con 0x7f53f0104df0 2024-04-30T08:35:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.429+0000 7f53fd1ff740 1 Processor -- start 2024-04-30T08:35:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.429+0000 7f53fd1ff740 1 -- start start 2024-04-30T08:35:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.429+0000 7f53fd1ff740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d4dc00c1b0 0x55d4dc3b68f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.429+0000 7f53fd1ff740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4dbfe32f0 0x55d4dc3b6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.429+0000 7f53fd1ff740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d4dbfeb4d0 0x55d4dc3bdd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.429+0000 7f53fd1ff740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d4dc312550 con 0x55d4dbfeb4d0 2024-04-30T08:35:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.429+0000 7f53fd1ff740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55d4dc2b7bc0 con 0x55d4dc00c1b0 2024-04-30T08:35:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.429+0000 7f53fd1ff740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d4dc30e510 con 0x55d4dbfe32f0 2024-04-30T08:35:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.429+0000 7f53e7e00640 1 -- 172.21.15.32:0/502829495 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f53d8043020 con 0x7f53f0104df0 2024-04-30T08:35:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.430+0000 7f53efe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d4dc00c1b0 0x55d4dc3b68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:58.432 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.430+0000 7f53ef400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4dbfe32f0 0x55d4dc3b6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.430+0000 7f53efe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d4dc00c1b0 0x55d4dc3b68f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:35156/0 (socket says 172.21.15.32:35156) 2024-04-30T08:35:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.430+0000 7f53efe00640 1 -- 172.21.15.32:0/1644221027 learned_addr learned my addr 172.21.15.32:0/1644221027 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.430+0000 7f53eea00640 1 --2- 172.21.15.32:0/1644221027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d4dbfeb4d0 0x55d4dc3bdd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.430+0000 7f53efe00640 1 -- 172.21.15.32:0/1644221027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4dbfe32f0 msgr2=0x55d4dc3b6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.430+0000 7f53efe00640 1 --2- 172.21.15.32:0/1644221027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4dbfe32f0 0x55d4dc3b6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.430+0000 7f53efe00640 1 -- 172.21.15.32:0/1644221027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d4dbfeb4d0 msgr2=0x55d4dc3bdd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.430+0000 7f53efe00640 1 --2- 172.21.15.32:0/1644221027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d4dbfeb4d0 0x55d4dc3bdd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.430+0000 7f53efe00640 1 -- 172.21.15.32:0/1644221027 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4dc024210 con 0x55d4dc00c1b0 2024-04-30T08:35:58.433 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.430+0000 7f53eea00640 1 --2- 172.21.15.32:0/1644221027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d4dbfeb4d0 0x55d4dc3bdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:35:58.436 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.431+0000 7f53efe00640 1 --2- 172.21.15.32:0/1644221027 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d4dc00c1b0 0x55d4dc3b68f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f53e8009fd0 tx=0x7f53e8011920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=13e2ef52e0134bf server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:58.436 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.431+0000 7f53dfe00640 1 -- 172.21.15.32:0/1644221027 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53e801e070 con 0x55d4dc00c1b0 2024-04-30T08:35:58.436 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.431+0000 7f53dfe00640 1 -- 172.21.15.32:0/1644221027 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53e800f9f0 con 0x55d4dc00c1b0 2024-04-30T08:35:58.436 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.431+0000 7f53dfe00640 1 -- 172.21.15.32:0/1644221027 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53e8019a80 con 0x55d4dc00c1b0 2024-04-30T08:35:58.436 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.431+0000 7f53e7e00640 1 --2- 172.21.15.32:0/502829495 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f53d40afb70 0x7f53d40b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:58.437 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.432+0000 7f53e7e00640 1 -- 172.21.15.32:0/502829495 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f53d80fe440 con 0x7f53f0104df0 2024-04-30T08:35:58.437 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.432+0000 7f53ef400640 1 --2- 172.21.15.32:0/502829495 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f53d40afb70 0x7f53d40b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:58.437 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.432+0000 7f53fd1ff740 1 -- 172.21.15.32:0/1644221027 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d4dc00c1b0 msgr2=0x55d4dc3b68f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:58.437 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.432+0000 7f53fd1ff740 1 --2- 172.21.15.32:0/1644221027 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d4dc00c1b0 0x55d4dc3b68f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f53e8009fd0 tx=0x7f53e8011920 comp rx=0 tx=0).stop 2024-04-30T08:35:58.437 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.432+0000 7f53fd1ff740 1 -- 172.21.15.32:0/1644221027 shutdown_connections 2024-04-30T08:35:58.437 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.432+0000 7f53fd1ff740 1 --2- 172.21.15.32:0/1644221027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d4dbfeb4d0 0x55d4dc3bdd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:58.437 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.432+0000 7f53fd1ff740 1 --2- 172.21.15.32:0/1644221027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4dbfe32f0 0x55d4dc3b6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:58.437 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.432+0000 7f53fd1ff740 1 --2- 172.21.15.32:0/1644221027 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d4dc00c1b0 0x55d4dc3b68f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:58.437 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.432+0000 7f53fd1ff740 1 -- 172.21.15.32:0/1644221027 >> 172.21.15.32:0/1644221027 conn(0x55d4dc000300 msgr2=0x55d4dc3b2280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:58.437 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.432+0000 7f53fd1ff740 1 -- 172.21.15.32:0/1644221027 shutdown_connections 2024-04-30T08:35:58.437 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.432+0000 7f53fd1ff740 1 -- 172.21.15.32:0/1644221027 wait complete. 2024-04-30T08:35:58.437 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.432+0000 7f53ef400640 1 --2- 172.21.15.32:0/502829495 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f53d40afb70 0x7f53d40b2080 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x55d4dc3b3010 tx=0x7f53e0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:58.437 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.433+0000 7f53fd1ff740 1 -- start start 2024-04-30T08:35:58.438 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.433+0000 7f53fd1ff740 1 Processor -- start 2024-04-30T08:35:58.438 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.434+0000 7f53fd1ff740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4dbfe32f0 0x55d4dc3c0450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:58.438 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.434+0000 7f53fd1ff740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d4dbfeb4d0 0x55d4dc3c0990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:58.438 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.434+0000 7f53fd1ff740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d4dc00c1b0 0x55d4dc3c78f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:58.438 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.434+0000 7f53fd1ff740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d4dc3be7c0 con 0x55d4dc00c1b0 2024-04-30T08:35:58.438 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.434+0000 7f53fd1ff740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55d4dc2cb660 con 0x55d4dbfeb4d0 2024-04-30T08:35:58.438 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.434+0000 7f53fd1ff740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d4dc3be940 con 0x55d4dbfe32f0 2024-04-30T08:35:58.438 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.434+0000 7f53fd1ff740 10 client.0 _ll_register_callbacks cb 0x55d4dc295400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:35:58.438 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.434+0000 7f53efe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d4dc00c1b0 0x55d4dc3c78f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:58.438 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.434+0000 7f53ef400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4dbfe32f0 0x55d4dc3c0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:58.438 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.434+0000 7f53efe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d4dc00c1b0 0x55d4dc3c78f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:60992/0 (socket says 172.21.15.32:60992) 2024-04-30T08:35:58.438 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.434+0000 7f53efe00640 1 -- 172.21.15.32:0/2765333848 learned_addr learned my addr 172.21.15.32:0/2765333848 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.434+0000 7f53eea00640 1 --2- 172.21.15.32:0/2765333848 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d4dbfeb4d0 0x55d4dc3c0990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.434+0000 7f53efe00640 1 -- 172.21.15.32:0/2765333848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4dbfe32f0 msgr2=0x55d4dc3c0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.434+0000 7f53efe00640 1 --2- 172.21.15.32:0/2765333848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4dbfe32f0 0x55d4dc3c0450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.434+0000 7f53efe00640 1 -- 172.21.15.32:0/2765333848 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d4dbfeb4d0 msgr2=0x55d4dc3c0990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.435+0000 7f53efe00640 1 --2- 172.21.15.32:0/2765333848 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d4dbfeb4d0 0x55d4dc3c0990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.435+0000 7f53efe00640 1 -- 172.21.15.32:0/2765333848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4dc0f0620 con 0x55d4dc00c1b0 2024-04-30T08:35:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.435+0000 7f53efe00640 1 --2- 172.21.15.32:0/2765333848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d4dc00c1b0 0x55d4dc3c78f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f53e8000c00 tx=0x7f53e800f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20986a1aea5adfe4 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:58.439 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.435+0000 7f53dcc00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:35:58.441 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.435+0000 7f53dcc00640 1 -- 172.21.15.32:0/2765333848 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53e801e040 con 0x55d4dc00c1b0 2024-04-30T08:35:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.435+0000 7f53fd1ff740 1 -- 172.21.15.32:0/2765333848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53e800f9f0 con 0x55d4dc00c1b0 2024-04-30T08:35:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.435+0000 7f53dcc00640 1 -- 172.21.15.32:0/2765333848 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f53e8005040 con 0x55d4dc00c1b0 2024-04-30T08:35:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.435+0000 7f53dcc00640 1 -- 172.21.15.32:0/2765333848 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f53e8032050 con 0x55d4dc00c1b0 2024-04-30T08:35:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.435+0000 7f53fd1ff740 10 client.15723 fetch_fsmap learned FSMap version 28 2024-04-30T08:35:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.436+0000 7f53fd1ff740 1 -- 172.21.15.32:0/2765333848 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d4dc3ca820 con 0x55d4dc00c1b0 2024-04-30T08:35:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.436+0000 7f53dcc00640 1 -- 172.21.15.32:0/2765333848 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53e800f9f0 con 0x55d4dc00c1b0 2024-04-30T08:35:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.436+0000 7f53dcc00640 1 -- 172.21.15.32:0/2765333848 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f53e80046c0 con 0x55d4dc00c1b0 2024-04-30T08:35:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.436+0000 7f53fd1ff740 10 client.15723 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:35:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.436+0000 7f53fd1ff740 10 client.15723 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-30T08:35:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.436+0000 7f53fd1ff740 20 client.15723 populate_metadata read hostname 'smithi032' 2024-04-30T08:35:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.436+0000 7f53fd1ff740 1 --2- 172.21.15.32:0/2765333848 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55d4dbfcad80 0x55d4dc3cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.436+0000 7f53fd1ff740 4 client.15723 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:35:58.442 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.436+0000 7f53fd1ff740 1 -- 172.21.15.32:0/2765333848 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d4dc320160 con 0x55d4dbfcad80 2024-04-30T08:35:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.437+0000 7f53ef400640 1 --2- 172.21.15.32:0/2765333848 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55d4dbfcad80 0x55d4dc3cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.437+0000 7f53ef400640 1 --2- 172.21.15.32:0/2765333848 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55d4dbfcad80 0x55d4dc3cdd00 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ddeeec2b6a13ad9 server_cookie=ded3f1afddf461ee in_seq=0 out_seq=0 2024-04-30T08:35:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.437+0000 7f53dcc00640 10 client.15723 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.438+0000 7f53dcc00640 1 -- 172.21.15.32:0/2765333848 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d4dc320160 con 0x55d4dbfcad80 2024-04-30T08:35:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.438+0000 7f53dcc00640 10 client.15723 handle_command_reply: tid=0 2024-04-30T08:35:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.438+0000 7f53fd1ff740 1 client.15723 shutdown 2024-04-30T08:35:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.438+0000 7f53fd1ff740 1 -- 172.21.15.32:0/2765333848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d4dc00c1b0 msgr2=0x55d4dc3c78f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.438+0000 7f53fd1ff740 1 --2- 172.21.15.32:0/2765333848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d4dc00c1b0 0x55d4dc3c78f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f53e8000c00 tx=0x7f53e800f7f0 comp rx=0 tx=0).stop 2024-04-30T08:35:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.438+0000 7f53fd1ff740 1 -- 172.21.15.32:0/2765333848 shutdown_connections 2024-04-30T08:35:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.438+0000 7f53fd1ff740 1 --2- 172.21.15.32:0/2765333848 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55d4dbfcad80 0x55d4dc3cdd00 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.438+0000 7f53fd1ff740 1 --2- 172.21.15.32:0/2765333848 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d4dc00c1b0 0x55d4dc3c78f0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:58.443 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.438+0000 7f53fd1ff740 1 --2- 172.21.15.32:0/2765333848 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d4dbfeb4d0 0x55d4dc3c0990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.438+0000 7f53dcc00640 0 client.15723 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.438+0000 7f53fd1ff740 1 --2- 172.21.15.32:0/2765333848 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4dbfe32f0 0x55d4dc3c0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.438+0000 7f53fd1ff740 1 -- 172.21.15.32:0/2765333848 >> 172.21.15.32:0/2765333848 conn(0x55d4dc000300 msgr2=0x55d4dc3b5190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.439+0000 7f53fd1ff740 1 -- 172.21.15.32:0/2765333848 shutdown_connections 2024-04-30T08:35:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.439+0000 7f53fd1ff740 1 -- 172.21.15.32:0/2765333848 wait complete. 2024-04-30T08:35:58.444 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.439+0000 7f53fd1ff740 20 client.15723 trim_cache size 0 max 16384 2024-04-30T08:35:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.456+0000 7f53fd1ff740 1 Processor -- start 2024-04-30T08:35:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.456+0000 7f53fd1ff740 1 -- start start 2024-04-30T08:35:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.456+0000 7f53fd1ff740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d4dbfe32f0 0x55d4dc3c0b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.456+0000 7f53fd1ff740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d4dbfeb4d0 0x55d4dc3c7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.456+0000 7f53fd1ff740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4dc00c1b0 0x55d4dc3b1b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.456+0000 7f53fd1ff740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f53f01aa9f0 con 0x55d4dbfe32f0 2024-04-30T08:35:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.456+0000 7f53fd1ff740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f53f01ac530 con 0x55d4dbfeb4d0 2024-04-30T08:35:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.456+0000 7f53fd1ff740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f53f01ab0c0 con 0x55d4dc00c1b0 2024-04-30T08:35:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.456+0000 7f53ef400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d4dbfe32f0 0x55d4dc3c0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.456+0000 7f53ef400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d4dbfe32f0 0x55d4dc3c0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32772/0 (socket says 172.21.15.32:32772) 2024-04-30T08:35:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.456+0000 7f53ef400640 1 -- 172.21.15.32:0/986093512 learned_addr learned my addr 172.21.15.32:0/986093512 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.456+0000 7f53efe00640 1 --2- 172.21.15.32:0/986093512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4dc00c1b0 0x55d4dc3b1b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.457+0000 7f53eea00640 1 --2- 172.21.15.32:0/986093512 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d4dbfeb4d0 0x55d4dc3c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.457+0000 7f53ef400640 1 -- 172.21.15.32:0/986093512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4dc00c1b0 msgr2=0x55d4dc3b1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.457+0000 7f53ef400640 1 --2- 172.21.15.32:0/986093512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4dc00c1b0 0x55d4dc3b1b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.457+0000 7f53ef400640 1 -- 172.21.15.32:0/986093512 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d4dbfeb4d0 msgr2=0x55d4dc3c7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.457+0000 7f53ef400640 1 --2- 172.21.15.32:0/986093512 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d4dbfeb4d0 0x55d4dc3c7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.457+0000 7f53ef400640 1 -- 172.21.15.32:0/986093512 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d4dc3be6a0 con 0x55d4dbfe32f0 2024-04-30T08:35:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.457+0000 7f53efe00640 1 --2- 172.21.15.32:0/986093512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4dc00c1b0 0x55d4dc3b1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:35:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.457+0000 7f53ef400640 1 --2- 172.21.15.32:0/986093512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d4dbfe32f0 0x55d4dc3c0b60 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f53e0012400 tx=0x7f53e0017000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9221fb63a178f6b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.458+0000 7f53df400640 1 -- 172.21.15.32:0/986093512 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53e00045e0 con 0x55d4dbfe32f0 2024-04-30T08:35:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.458+0000 7f53df400640 1 -- 172.21.15.32:0/986093512 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53e0004780 con 0x55d4dbfe32f0 2024-04-30T08:35:58.461 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.458+0000 7f53df400640 1 -- 172.21.15.32:0/986093512 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53e0005020 con 0x55d4dbfe32f0 2024-04-30T08:35:58.464 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.458+0000 7f53fd1ff740 1 -- 172.21.15.32:0/986093512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d4dbfe32f0 msgr2=0x55d4dc3c0b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:58.464 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.458+0000 7f53fd1ff740 1 --2- 172.21.15.32:0/986093512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d4dbfe32f0 0x55d4dc3c0b60 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f53e0012400 tx=0x7f53e0017000 comp rx=0 tx=0).stop 2024-04-30T08:35:58.464 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.459+0000 7f53fd1ff740 1 -- 172.21.15.32:0/986093512 shutdown_connections 2024-04-30T08:35:58.464 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.459+0000 7f53fd1ff740 1 --2- 172.21.15.32:0/986093512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4dc00c1b0 0x55d4dc3b1b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:58.464 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.459+0000 7f53fd1ff740 1 --2- 172.21.15.32:0/986093512 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d4dbfeb4d0 0x55d4dc3c7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:58.464 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.459+0000 7f53fd1ff740 1 --2- 172.21.15.32:0/986093512 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d4dbfe32f0 0x55d4dc3c0b60 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:58.465 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.459+0000 7f53fd1ff740 1 -- 172.21.15.32:0/986093512 >> 172.21.15.32:0/986093512 conn(0x55d4dc000300 msgr2=0x55d4dc3c0450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:58.465 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.459+0000 7f53fd1ff740 1 -- 172.21.15.32:0/986093512 shutdown_connections 2024-04-30T08:35:58.465 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.459+0000 7f53fd1ff740 1 -- 172.21.15.32:0/986093512 wait complete. 2024-04-30T08:35:58.465 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.459+0000 7f53fd1ff740 1 -- start start 2024-04-30T08:35:58.465 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.459+0000 7f53fd1ff740 1 Processor -- start 2024-04-30T08:35:58.465 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.460+0000 7f53fd1ff740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d4dbfe32f0 0x55d4dc3c4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:58.465 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.460+0000 7f53fd1ff740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d4dbfeb4d0 0x55d4dc3c4d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:58.465 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.460+0000 7f53fd1ff740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4dc00c1b0 0x55d4dc434080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:58.465 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.460+0000 7f53fd1ff740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f53f01abe60 con 0x55d4dbfeb4d0 2024-04-30T08:35:58.465 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.460+0000 7f53fd1ff740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f53f01ab790 con 0x55d4dbfe32f0 2024-04-30T08:35:58.465 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.460+0000 7f53fd1ff740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f53f01a89b0 con 0x55d4dc00c1b0 2024-04-30T08:35:58.465 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.460+0000 7f53fd1ff740 10 client.0 _ll_register_callbacks cb 0x7f53e801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:35:58.465 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.460+0000 7f53eea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d4dbfeb4d0 0x55d4dc3c4d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:58.465 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.460+0000 7f53eea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d4dbfeb4d0 0x55d4dc3c4d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:32786/0 (socket says 172.21.15.32:32786) 2024-04-30T08:35:58.465 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.460+0000 7f53eea00640 1 -- 172.21.15.32:0/725809156 learned_addr learned my addr 172.21.15.32:0/725809156 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:35:58.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.460+0000 7f53ef400640 1 --2- 172.21.15.32:0/725809156 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d4dbfe32f0 0x55d4dc3c4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:58.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.460+0000 7f53efe00640 1 --2- 172.21.15.32:0/725809156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4dc00c1b0 0x55d4dc434080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:58.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.461+0000 7f53eea00640 1 -- 172.21.15.32:0/725809156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4dc00c1b0 msgr2=0x55d4dc434080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:58.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.461+0000 7f53eea00640 1 --2- 172.21.15.32:0/725809156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4dc00c1b0 0x55d4dc434080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:58.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.461+0000 7f53eea00640 1 -- 172.21.15.32:0/725809156 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d4dbfe32f0 msgr2=0x55d4dc3c4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:35:58.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.461+0000 7f53eea00640 1 --2- 172.21.15.32:0/725809156 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d4dbfe32f0 0x55d4dc3c4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:58.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.461+0000 7f53eea00640 1 -- 172.21.15.32:0/725809156 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d4dc4345c0 con 0x55d4dbfeb4d0 2024-04-30T08:35:58.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.461+0000 7f53efe00640 1 --2- 172.21.15.32:0/725809156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4dc00c1b0 0x55d4dc434080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:35:58.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.461+0000 7f53eea00640 1 --2- 172.21.15.32:0/725809156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d4dbfeb4d0 0x55d4dc3c4d90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f53d802a930 tx=0x7f53d8005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3fd9bbb75fffee56 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:35:58.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.461+0000 7f53dcc00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:35:58.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.461+0000 7f53dcc00640 1 -- 172.21.15.32:0/725809156 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d80c5c30 con 0x55d4dbfeb4d0 2024-04-30T08:35:58.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.461+0000 7f53dcc00640 1 -- 172.21.15.32:0/725809156 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f53d80fe100 con 0x55d4dbfeb4d0 2024-04-30T08:35:58.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.461+0000 7f53fd1ff740 1 -- 172.21.15.32:0/725809156 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53e0004780 con 0x55d4dbfeb4d0 2024-04-30T08:35:58.466 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.461+0000 7f53dcc00640 1 -- 172.21.15.32:0/725809156 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f53d80c5fd0 con 0x55d4dbfeb4d0 2024-04-30T08:35:58.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.462+0000 7f53fd1ff740 10 client.15729 fetch_fsmap learned FSMap version 28 2024-04-30T08:35:58.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.462+0000 7f53fd1ff740 1 -- 172.21.15.32:0/725809156 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55d4dc435800 con 0x55d4dbfeb4d0 2024-04-30T08:35:58.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.462+0000 7f53dcc00640 1 -- 172.21.15.32:0/725809156 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53e0004780 con 0x55d4dbfeb4d0 2024-04-30T08:35:58.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.462+0000 7f53dcc00640 1 -- 172.21.15.32:0/725809156 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f53d80fbc30 con 0x55d4dbfeb4d0 2024-04-30T08:35:58.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.462+0000 7f53fd1ff740 10 client.15729 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:35:58.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.462+0000 7f53fd1ff740 10 client.15729 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-30T08:35:58.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.462+0000 7f53fd1ff740 20 client.15729 populate_metadata read hostname 'smithi032' 2024-04-30T08:35:58.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.462+0000 7f53fd1ff740 1 --2- 172.21.15.32:0/725809156 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55d4dbfa01c0 0x55d4dc4381f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:35:58.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.462+0000 7f53fd1ff740 4 client.15729 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-30T08:35:58.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.462+0000 7f53fd1ff740 1 -- 172.21.15.32:0/725809156 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d4dc319fd0 con 0x55d4dbfa01c0 2024-04-30T08:35:58.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.462+0000 7f53ef400640 1 --2- 172.21.15.32:0/725809156 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55d4dbfa01c0 0x55d4dc4381f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:35:58.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.463+0000 7f53ef400640 1 --2- 172.21.15.32:0/725809156 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55d4dbfa01c0 0x55d4dc4381f0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=230b60d373d6dee2 server_cookie=18cf339f334fe5b5 in_seq=0 out_seq=0 2024-04-30T08:35:58.467 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.463+0000 7f53dcc00640 10 client.15729 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:58.472 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.471+0000 7f53dcc00640 1 -- 172.21.15.32:0/725809156 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d4dc319fd0 con 0x55d4dbfa01c0 2024-04-30T08:35:58.473 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.471+0000 7f53dcc00640 10 client.15729 handle_command_reply: tid=0 2024-04-30T08:35:58.473 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.471+0000 7f53fd1ff740 1 client.15729 shutdown 2024-04-30T08:35:58.473 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.471+0000 7f53fd1ff740 1 -- 172.21.15.32:0/725809156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d4dbfeb4d0 msgr2=0x55d4dc3c4d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:58.473 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.471+0000 7f53fd1ff740 1 --2- 172.21.15.32:0/725809156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d4dbfeb4d0 0x55d4dc3c4d90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f53d802a930 tx=0x7f53d8005f90 comp rx=0 tx=0).stop 2024-04-30T08:35:58.473 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.471+0000 7f53fd1ff740 1 -- 172.21.15.32:0/725809156 shutdown_connections 2024-04-30T08:35:58.473 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.471+0000 7f53fd1ff740 1 --2- 172.21.15.32:0/725809156 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55d4dbfa01c0 0x55d4dc4381f0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:58.473 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.471+0000 7f53fd1ff740 1 --2- 172.21.15.32:0/725809156 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d4dc00c1b0 0x55d4dc434080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:58.473 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.471+0000 7f53fd1ff740 1 --2- 172.21.15.32:0/725809156 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d4dbfeb4d0 0x55d4dc3c4d90 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:58.473 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.471+0000 7f53fd1ff740 1 --2- 172.21.15.32:0/725809156 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d4dbfe32f0 0x55d4dc3c4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:58.473 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.471+0000 7f53fd1ff740 1 -- 172.21.15.32:0/725809156 >> 172.21.15.32:0/725809156 conn(0x55d4dc000300 msgr2=0x55d4dc3b1b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:58.473 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.471+0000 7f53dcc00640 0 client.15729 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:35:58.473 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.471+0000 7f53fd1ff740 1 -- 172.21.15.32:0/725809156 shutdown_connections 2024-04-30T08:35:58.474 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.471+0000 7f53fd1ff740 1 -- 172.21.15.32:0/725809156 wait complete. 2024-04-30T08:35:58.474 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.471+0000 7f53fd1ff740 20 client.15729 trim_cache size 0 max 16384 2024-04-30T08:35:58.474 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-30T08:35:58.474 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-30T08:35:58.474 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-30T08:35:58.474 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-30T08:35:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.472+0000 7f53f6200640 1 -- 172.21.15.32:0/502829495 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f53d40afb70 msgr2=0x7f53d40b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.472+0000 7f53f6200640 1 --2- 172.21.15.32:0/502829495 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f53d40afb70 0x7f53d40b2080 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x55d4dc3b3010 tx=0x7f53e0009500 comp rx=0 tx=0).stop 2024-04-30T08:35:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.472+0000 7f53f6200640 1 -- 172.21.15.32:0/502829495 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53f0104df0 msgr2=0x7f53f019c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:35:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.472+0000 7f53f6200640 1 --2- 172.21.15.32:0/502829495 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53f0104df0 0x7f53f019c9e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f53d8032d50 tx=0x7f53d8032d80 comp rx=0 tx=0).stop 2024-04-30T08:35:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.472+0000 7f53f6200640 1 -- 172.21.15.32:0/502829495 shutdown_connections 2024-04-30T08:35:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.472+0000 7f53f6200640 1 --2- 172.21.15.32:0/502829495 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f53d40afb70 0x7f53d40b2080 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.472+0000 7f53f6200640 1 --2- 172.21.15.32:0/502829495 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53f0109af0 0x7f53f01a3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.472+0000 7f53f6200640 1 --2- 172.21.15.32:0/502829495 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53f0104df0 0x7f53f019c9e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:58.475 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.472+0000 7f53f6200640 1 --2- 172.21.15.32:0/502829495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53f0103bb0 0x7f53f019c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:35:58.476 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.472+0000 7f53f6200640 1 -- 172.21.15.32:0/502829495 >> 172.21.15.32:0/502829495 conn(0x7f53f00fedc0 msgr2=0x7f53f00ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:35:58.476 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.472+0000 7f53f6200640 1 -- 172.21.15.32:0/502829495 shutdown_connections 2024-04-30T08:35:58.476 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:35:58.472+0000 7f53f6200640 1 -- 172.21.15.32:0/502829495 wait complete. 2024-04-30T08:35:59.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:35:59 smithi032 ceph-mon[28331]: pgmap v449: 129 pgs: 129 active+clean; 672 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-30T08:35:59.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:35:59 smithi071 ceph-mon[33385]: pgmap v449: 129 pgs: 129 active+clean; 672 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-30T08:35:59.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:35:59 smithi189 ceph-mon[33038]: pgmap v449: 129 pgs: 129 active+clean; 672 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-04-30T08:36:01.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:01 smithi032 ceph-mon[28331]: pgmap v450: 129 pgs: 129 active+clean; 673 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1.7 KiB/s wr, 14 op/s 2024-04-30T08:36:01.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:01 smithi071 ceph-mon[33385]: pgmap v450: 129 pgs: 129 active+clean; 673 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1.7 KiB/s wr, 14 op/s 2024-04-30T08:36:01.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:01 smithi189 ceph-mon[33038]: pgmap v450: 129 pgs: 129 active+clean; 673 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 7.5 KiB/s rd, 1.7 KiB/s wr, 14 op/s 2024-04-30T08:36:03.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:03 smithi032 ceph-mon[28331]: pgmap v451: 129 pgs: 129 active+clean; 673 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-04-30T08:36:03.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:03 smithi071 ceph-mon[33385]: pgmap v451: 129 pgs: 129 active+clean; 673 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-04-30T08:36:03.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:03 smithi189 ceph-mon[33038]: pgmap v451: 129 pgs: 129 active+clean; 673 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-04-30T08:36:05.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:05 smithi032 ceph-mon[28331]: pgmap v452: 129 pgs: 129 active+clean; 673 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-04-30T08:36:05.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:05 smithi071 ceph-mon[33385]: pgmap v452: 129 pgs: 129 active+clean; 673 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-04-30T08:36:05.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:05 smithi189 ceph-mon[33038]: pgmap v452: 129 pgs: 129 active+clean; 673 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.7 KiB/s wr, 12 op/s 2024-04-30T08:36:07.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:07 smithi032 ceph-mon[28331]: pgmap v453: 129 pgs: 129 active+clean; 673 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-04-30T08:36:07.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:07 smithi071 ceph-mon[33385]: pgmap v453: 129 pgs: 129 active+clean; 673 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-04-30T08:36:07.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:07 smithi189 ceph-mon[33038]: pgmap v453: 129 pgs: 129 active+clean; 673 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 7.7 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-04-30T08:36:08.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:08 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:36:08.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:08 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:36:08.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:08 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:36:09.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:09 smithi032 ceph-mon[28331]: pgmap v454: 129 pgs: 129 active+clean; 673 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 852 B/s wr, 11 op/s 2024-04-30T08:36:09.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:09 smithi071 ceph-mon[33385]: pgmap v454: 129 pgs: 129 active+clean; 673 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 852 B/s wr, 11 op/s 2024-04-30T08:36:09.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:09 smithi189 ceph-mon[33038]: pgmap v454: 129 pgs: 129 active+clean; 673 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 852 B/s wr, 11 op/s 2024-04-30T08:36:10.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:10 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:36:10.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:10 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:36:10.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:10 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:36:10.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:10 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:36:10.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:10 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:36:10.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:10 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:36:10.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:10 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:36:10.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:10 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:36:10.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:10 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:36:11.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:11 smithi032 ceph-mon[28331]: pgmap v455: 129 pgs: 129 active+clean; 673 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 767 B/s wr, 13 op/s 2024-04-30T08:36:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.697+0000 7fe11ea00640 1 -- 172.21.15.32:0/818421205 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe118103a00 msgr2=0x7fe118104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.697+0000 7fe11ea00640 1 --2- 172.21.15.32:0/818421205 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe118103a00 0x7fe118104700 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe10c009860 tx=0x7fe10c02a340 comp rx=0 tx=0).stop 2024-04-30T08:36:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.698+0000 7fe11ea00640 1 -- 172.21.15.32:0/818421205 shutdown_connections 2024-04-30T08:36:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.698+0000 7fe11ea00640 1 --2- 172.21.15.32:0/818421205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe118109940 0x7fe11810bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:11.698 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.698+0000 7fe11ea00640 1 --2- 172.21.15.32:0/818421205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe118104c40 0x7fe118105050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.698+0000 7fe11ea00640 1 --2- 172.21.15.32:0/818421205 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe118103a00 0x7fe118104700 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.698+0000 7fe11ea00640 1 -- 172.21.15.32:0/818421205 >> 172.21.15.32:0/818421205 conn(0x7fe1180fec10 msgr2=0x7fe1181010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.698+0000 7fe11ea00640 1 -- 172.21.15.32:0/818421205 shutdown_connections 2024-04-30T08:36:11.699 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.698+0000 7fe11ea00640 1 -- 172.21.15.32:0/818421205 wait complete. 2024-04-30T08:36:11.701 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.699+0000 7fe11ea00640 1 Processor -- start 2024-04-30T08:36:11.701 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.699+0000 7fe11ea00640 1 -- start start 2024-04-30T08:36:11.701 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.699+0000 7fe11ea00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe118103a00 0x7fe1180768d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:11.701 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.699+0000 7fe11ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe118104c40 0x7fe118074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:11.701 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.699+0000 7fe11ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe118109940 0x7fe1180754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:11.702 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.699+0000 7fe11ea00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe11810f4e0 con 0x7fe118104c40 2024-04-30T08:36:11.702 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.699+0000 7fe11ea00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe11810f360 con 0x7fe118103a00 2024-04-30T08:36:11.702 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.699+0000 7fe11ea00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe11810f660 con 0x7fe118109940 2024-04-30T08:36:11.702 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.699+0000 7fe11d600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe118103a00 0x7fe1180768d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:11.702 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.699+0000 7fe11cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe118104c40 0x7fe118074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:11.702 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.699+0000 7fe11d600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe118103a00 0x7fe1180768d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:41898/0 (socket says 172.21.15.32:41898) 2024-04-30T08:36:11.702 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.700+0000 7fe11d600640 1 -- 172.21.15.32:0/2160443905 learned_addr learned my addr 172.21.15.32:0/2160443905 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:11.702 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.700+0000 7fe11e000640 1 --2- 172.21.15.32:0/2160443905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe118109940 0x7fe1180754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:11.702 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.700+0000 7fe11d600640 1 -- 172.21.15.32:0/2160443905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe118109940 msgr2=0x7fe1180754b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:11.702 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.700+0000 7fe11d600640 1 --2- 172.21.15.32:0/2160443905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe118109940 0x7fe1180754b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:11.702 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.700+0000 7fe11d600640 1 -- 172.21.15.32:0/2160443905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe118104c40 msgr2=0x7fe118074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:11.702 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.700+0000 7fe11d600640 1 --2- 172.21.15.32:0/2160443905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe118104c40 0x7fe118074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:11.703 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.700+0000 7fe11d600640 1 -- 172.21.15.32:0/2160443905 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe118075be0 con 0x7fe118103a00 2024-04-30T08:36:11.703 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.700+0000 7fe11d600640 1 --2- 172.21.15.32:0/2160443905 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe118103a00 0x7fe1180768d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe10c0304b0 tx=0x7fe10c032e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:11.703 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.701+0000 7fe116000640 1 -- 172.21.15.32:0/2160443905 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe10c037040 con 0x7fe118103a00 2024-04-30T08:36:11.703 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.701+0000 7fe116000640 1 -- 172.21.15.32:0/2160443905 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe10c030d10 con 0x7fe118103a00 2024-04-30T08:36:11.703 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.701+0000 7fe11ea00640 1 -- 172.21.15.32:0/2160443905 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1181a41b0 con 0x7fe118103a00 2024-04-30T08:36:11.703 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.701+0000 7fe116000640 1 -- 172.21.15.32:0/2160443905 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe10c003850 con 0x7fe118103a00 2024-04-30T08:36:11.703 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.701+0000 7fe11ea00640 1 -- 172.21.15.32:0/2160443905 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1181a45b0 con 0x7fe118103a00 2024-04-30T08:36:11.704 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:11 smithi189 ceph-mon[33038]: pgmap v455: 129 pgs: 129 active+clean; 673 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 767 B/s wr, 13 op/s 2024-04-30T08:36:11.705 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.705+0000 7fe12703a740 1 Processor -- start 2024-04-30T08:36:11.706 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.705+0000 7fe12703a740 1 -- start start 2024-04-30T08:36:11.707 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.706+0000 7fe12703a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4868981b0 0x55c486c3a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:11.707 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.706+0000 7fe12703a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c48686f2f0 0x55c486c3a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:11.707 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.706+0000 7fe12703a740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c4868774d0 0x55c486c41760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:11.707 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.706+0000 7fe12703a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c486b9e550 con 0x55c48686f2f0 2024-04-30T08:36:11.707 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.706+0000 7fe12703a740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55c486b43bc0 con 0x55c4868774d0 2024-04-30T08:36:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.706+0000 7fe12703a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c486b9a510 con 0x55c4868981b0 2024-04-30T08:36:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.706+0000 7fe11cc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c4868774d0 0x55c486c41760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.706+0000 7fe11e000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4868981b0 0x55c486c3a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.706+0000 7fe11cc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c4868774d0 0x55c486c41760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:41912/0 (socket says 172.21.15.32:41912) 2024-04-30T08:36:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.706+0000 7fe11e000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4868981b0 0x55c486c3a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:56974/0 (socket says 172.21.15.32:56974) 2024-04-30T08:36:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.706+0000 7fe11cc00640 1 -- 172.21.15.32:0/1758615729 learned_addr learned my addr 172.21.15.32:0/1758615729 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.706+0000 7fe116000640 1 -- 172.21.15.32:0/2160443905 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7fe10c043020 con 0x7fe118103a00 2024-04-30T08:36:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.707+0000 7fe11cc00640 1 -- 172.21.15.32:0/1758615729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4868981b0 msgr2=0x55c486c3a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.707+0000 7fe11cc00640 1 --2- 172.21.15.32:0/1758615729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4868981b0 0x55c486c3a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.707+0000 7fe11d600640 1 --2- 172.21.15.32:0/1758615729 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c48686f2f0 0x55c486c3a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.707+0000 7fe11cc00640 1 -- 172.21.15.32:0/1758615729 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c48686f2f0 msgr2=0x55c486c3a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.707+0000 7fe11cc00640 1 --2- 172.21.15.32:0/1758615729 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c48686f2f0 0x55c486c3a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:11.708 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.707+0000 7fe11cc00640 1 -- 172.21.15.32:0/1758615729 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4868b0210 con 0x55c4868774d0 2024-04-30T08:36:11.709 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.707+0000 7fe11e000640 1 --2- 172.21.15.32:0/1758615729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4868981b0 0x55c486c3a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:36:11.709 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.707+0000 7fe11d600640 1 --2- 172.21.15.32:0/1758615729 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c48686f2f0 0x55c486c3a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-30T08:36:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.707+0000 7fe11cc00640 1 --2- 172.21.15.32:0/1758615729 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c4868774d0 0x55c486c41760 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe104009c80 tx=0x7fe104011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9927d30326803228 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:11.711 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.707+0000 7fe10a600640 1 -- 172.21.15.32:0/1758615729 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe104004220 con 0x55c4868774d0 2024-04-30T08:36:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.708+0000 7fe10a600640 1 -- 172.21.15.32:0/1758615729 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1040043c0 con 0x55c4868774d0 2024-04-30T08:36:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.708+0000 7fe10a600640 1 -- 172.21.15.32:0/1758615729 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1040197f0 con 0x55c4868774d0 2024-04-30T08:36:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.708+0000 7fe116000640 1 --2- 172.21.15.32:0/2160443905 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe1000afbf0 0x7fe1000b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.708+0000 7fe116000640 1 -- 172.21.15.32:0/2160443905 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fe10c0fe2a0 con 0x7fe118103a00 2024-04-30T08:36:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.708+0000 7fe11e000640 1 --2- 172.21.15.32:0/2160443905 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe1000afbf0 0x7fe1000b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.708+0000 7fe12703a740 1 -- 172.21.15.32:0/1758615729 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c4868774d0 msgr2=0x55c486c41760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.708+0000 7fe12703a740 1 --2- 172.21.15.32:0/1758615729 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c4868774d0 0x55c486c41760 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe104009c80 tx=0x7fe104011810 comp rx=0 tx=0).stop 2024-04-30T08:36:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.709+0000 7fe12703a740 1 -- 172.21.15.32:0/1758615729 shutdown_connections 2024-04-30T08:36:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.709+0000 7fe12703a740 1 --2- 172.21.15.32:0/1758615729 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c4868774d0 0x55c486c41760 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.709+0000 7fe12703a740 1 --2- 172.21.15.32:0/1758615729 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c48686f2f0 0x55c486c3a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.709+0000 7fe12703a740 1 --2- 172.21.15.32:0/1758615729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4868981b0 0x55c486c3a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.709+0000 7fe12703a740 1 -- 172.21.15.32:0/1758615729 >> 172.21.15.32:0/1758615729 conn(0x55c48688c300 msgr2=0x55c486c35c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:11.712 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.709+0000 7fe12703a740 1 -- 172.21.15.32:0/1758615729 shutdown_connections 2024-04-30T08:36:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.709+0000 7fe12703a740 1 -- 172.21.15.32:0/1758615729 wait complete. 2024-04-30T08:36:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.709+0000 7fe11e000640 1 --2- 172.21.15.32:0/2160443905 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe1000afbf0 0x7fe1000b2100 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe110009cf0 tx=0x7fe110009600 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.709+0000 7fe12703a740 1 -- start start 2024-04-30T08:36:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.709+0000 7fe12703a740 1 Processor -- start 2024-04-30T08:36:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.710+0000 7fe12703a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c48686f2f0 0x55c486c43db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.710+0000 7fe12703a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4868774d0 0x55c486c442f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.710+0000 7fe12703a740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c4868981b0 0x55c486c4b250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.710+0000 7fe12703a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55c486c421b0 con 0x55c4868774d0 2024-04-30T08:36:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.710+0000 7fe12703a740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55c486b57660 con 0x55c4868981b0 2024-04-30T08:36:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.710+0000 7fe12703a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55c486c42330 con 0x55c48686f2f0 2024-04-30T08:36:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.710+0000 7fe12703a740 10 client.0 _ll_register_callbacks cb 0x55c486b21400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:36:11.713 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.710+0000 7fe11d600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c48686f2f0 0x55c486c43db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:11.714 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.710+0000 7fe11d600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c48686f2f0 0x55c486c43db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:56984/0 (socket says 172.21.15.32:56984) 2024-04-30T08:36:11.714 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.710+0000 7fe11d600640 1 -- 172.21.15.32:0/36695339 learned_addr learned my addr 172.21.15.32:0/36695339 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:11.714 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.710+0000 7fe11e000640 1 --2- 172.21.15.32:0/36695339 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c4868981b0 0x55c486c4b250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:11.714 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.710+0000 7fe11cc00640 1 --2- 172.21.15.32:0/36695339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4868774d0 0x55c486c442f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:11.714 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.711+0000 7fe11d600640 1 -- 172.21.15.32:0/36695339 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c4868981b0 msgr2=0x55c486c4b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:11.714 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.711+0000 7fe11d600640 1 --2- 172.21.15.32:0/36695339 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c4868981b0 0x55c486c4b250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:11.714 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.711+0000 7fe11d600640 1 -- 172.21.15.32:0/36695339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4868774d0 msgr2=0x55c486c442f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:11.714 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.711+0000 7fe11d600640 1 --2- 172.21.15.32:0/36695339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4868774d0 0x55c486c442f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:11.714 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.711+0000 7fe11d600640 1 -- 172.21.15.32:0/36695339 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c48697c620 con 0x55c48686f2f0 2024-04-30T08:36:11.714 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.711+0000 7fe11d600640 1 --2- 172.21.15.32:0/36695339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c48686f2f0 0x55c486c43db0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe10c02a930 tx=0x7fe10c004350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=efb083fb1c935f8d server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:11.714 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.711+0000 7fe0fea00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-30T08:36:11.714 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.711+0000 7fe0fea00640 1 -- 172.21.15.32:0/36695339 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe10c0c5e30 con 0x55c48686f2f0 2024-04-30T08:36:11.714 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.711+0000 7fe0fea00640 1 -- 172.21.15.32:0/36695339 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fe10c0cef00 con 0x55c48686f2f0 2024-04-30T08:36:11.714 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.711+0000 7fe12703a740 1 -- 172.21.15.32:0/36695339 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1040043c0 con 0x55c48686f2f0 2024-04-30T08:36:11.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.712+0000 7fe0fea00640 1 -- 172.21.15.32:0/36695339 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe10c0c6a20 con 0x55c48686f2f0 2024-04-30T08:36:11.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.712+0000 7fe12703a740 10 client.25864 fetch_fsmap learned FSMap version 28 2024-04-30T08:36:11.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.712+0000 7fe12703a740 1 -- 172.21.15.32:0/36695339 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c486c42220 con 0x55c48686f2f0 2024-04-30T08:36:11.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.712+0000 7fe0fea00640 1 -- 172.21.15.32:0/36695339 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe1040043c0 con 0x55c48686f2f0 2024-04-30T08:36:11.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.712+0000 7fe0fea00640 1 -- 172.21.15.32:0/36695339 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe10c0c1db0 con 0x55c48686f2f0 2024-04-30T08:36:11.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.712+0000 7fe12703a740 10 client.25864 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:36:11.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.712+0000 7fe12703a740 10 client.25864 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-30T08:36:11.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.712+0000 7fe12703a740 20 client.25864 populate_metadata read hostname 'smithi032' 2024-04-30T08:36:11.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.712+0000 7fe12703a740 1 --2- 172.21.15.32:0/36695339 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55c486856d80 0x55c486c51780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:11.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.712+0000 7fe12703a740 4 client.25864 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:36:11.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.712+0000 7fe12703a740 1 -- 172.21.15.32:0/36695339 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c486bac160 con 0x55c486856d80 2024-04-30T08:36:11.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.713+0000 7fe11cc00640 1 --2- 172.21.15.32:0/36695339 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55c486856d80 0x55c486c51780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:11.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.713+0000 7fe11cc00640 1 --2- 172.21.15.32:0/36695339 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55c486856d80 0x55c486c51780 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6883bd439feff376 server_cookie=65f74d8d062d4298 in_seq=0 out_seq=0 2024-04-30T08:36:11.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.713+0000 7fe0fea00640 10 client.25864 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:11.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.714+0000 7fe0fea00640 1 -- 172.21.15.32:0/36695339 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55c486bac160 con 0x55c486856d80 2024-04-30T08:36:11.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.714+0000 7fe0fea00640 10 client.25864 handle_command_reply: tid=0 2024-04-30T08:36:11.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.714+0000 7fe12703a740 1 client.25864 shutdown 2024-04-30T08:36:11.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.714+0000 7fe12703a740 1 -- 172.21.15.32:0/36695339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c48686f2f0 msgr2=0x55c486c43db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:11.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.714+0000 7fe12703a740 1 --2- 172.21.15.32:0/36695339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c48686f2f0 0x55c486c43db0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fe10c02a930 tx=0x7fe10c004350 comp rx=0 tx=0).stop 2024-04-30T08:36:11.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.715+0000 7fe12703a740 1 -- 172.21.15.32:0/36695339 shutdown_connections 2024-04-30T08:36:11.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.715+0000 7fe12703a740 1 --2- 172.21.15.32:0/36695339 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55c486856d80 0x55c486c51780 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:11.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.715+0000 7fe12703a740 1 --2- 172.21.15.32:0/36695339 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c4868981b0 0x55c486c4b250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:11.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.715+0000 7fe12703a740 1 --2- 172.21.15.32:0/36695339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4868774d0 0x55c486c442f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:11.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.715+0000 7fe0fea00640 0 client.25864 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:11.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.715+0000 7fe12703a740 1 --2- 172.21.15.32:0/36695339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c48686f2f0 0x55c486c43db0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:11.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.715+0000 7fe12703a740 1 -- 172.21.15.32:0/36695339 >> 172.21.15.32:0/36695339 conn(0x55c48688c300 msgr2=0x55c486c34c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:11.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.715+0000 7fe12703a740 1 -- 172.21.15.32:0/36695339 shutdown_connections 2024-04-30T08:36:11.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.715+0000 7fe12703a740 1 -- 172.21.15.32:0/36695339 wait complete. 2024-04-30T08:36:11.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.715+0000 7fe12703a740 20 client.25864 trim_cache size 0 max 16384 2024-04-30T08:36:11.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.733+0000 7fe12703a740 1 Processor -- start 2024-04-30T08:36:11.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.733+0000 7fe12703a740 1 -- start start 2024-04-30T08:36:11.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.733+0000 7fe12703a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c48686f2f0 0x55c486c35d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:11.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.733+0000 7fe12703a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4868774d0 0x55c486c4b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:11.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.733+0000 7fe12703a740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c4868981b0 0x55c486c43980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:11.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.733+0000 7fe12703a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe1181aeb60 con 0x55c4868774d0 2024-04-30T08:36:11.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.733+0000 7fe12703a740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe1181b06a0 con 0x55c4868981b0 2024-04-30T08:36:11.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.733+0000 7fe12703a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe1181af230 con 0x55c48686f2f0 2024-04-30T08:36:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.733+0000 7fe11cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4868774d0 0x55c486c4b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.734+0000 7fe11d600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c48686f2f0 0x55c486c35d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.734+0000 7fe11cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4868774d0 0x55c486c4b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41958/0 (socket says 172.21.15.32:41958) 2024-04-30T08:36:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.734+0000 7fe11cc00640 1 -- 172.21.15.32:0/3538415406 learned_addr learned my addr 172.21.15.32:0/3538415406 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.734+0000 7fe11e000640 1 --2- 172.21.15.32:0/3538415406 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c4868981b0 0x55c486c43980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.734+0000 7fe11d600640 1 -- 172.21.15.32:0/3538415406 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c4868981b0 msgr2=0x55c486c43980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:11.736 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.734+0000 7fe11d600640 1 --2- 172.21.15.32:0/3538415406 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c4868981b0 0x55c486c43980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.734+0000 7fe11d600640 1 -- 172.21.15.32:0/3538415406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4868774d0 msgr2=0x55c486c4b1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.734+0000 7fe11d600640 1 --2- 172.21.15.32:0/3538415406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4868774d0 0x55c486c4b1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.734+0000 7fe11d600640 1 -- 172.21.15.32:0/3538415406 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c486c31480 con 0x55c48686f2f0 2024-04-30T08:36:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.734+0000 7fe11d600640 1 --2- 172.21.15.32:0/3538415406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c48686f2f0 0x55c486c35d40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe10c048000 tx=0x7fe10c0ca480 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2cce00385da9ab17 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.734+0000 7fe109c00640 1 -- 172.21.15.32:0/3538415406 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe10c0c5c70 con 0x55c48686f2f0 2024-04-30T08:36:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.735+0000 7fe109c00640 1 -- 172.21.15.32:0/3538415406 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe10c0fbe50 con 0x55c48686f2f0 2024-04-30T08:36:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.735+0000 7fe109c00640 1 -- 172.21.15.32:0/3538415406 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe10c0c6320 con 0x55c48686f2f0 2024-04-30T08:36:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.735+0000 7fe12703a740 1 -- 172.21.15.32:0/3538415406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c48686f2f0 msgr2=0x55c486c35d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.735+0000 7fe12703a740 1 --2- 172.21.15.32:0/3538415406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c48686f2f0 0x55c486c35d40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe10c048000 tx=0x7fe10c0ca480 comp rx=0 tx=0).stop 2024-04-30T08:36:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.735+0000 7fe12703a740 1 -- 172.21.15.32:0/3538415406 shutdown_connections 2024-04-30T08:36:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.735+0000 7fe12703a740 1 --2- 172.21.15.32:0/3538415406 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c4868981b0 0x55c486c43980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:11.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.735+0000 7fe12703a740 1 --2- 172.21.15.32:0/3538415406 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4868774d0 0x55c486c4b1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.735+0000 7fe12703a740 1 --2- 172.21.15.32:0/3538415406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c48686f2f0 0x55c486c35d40 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.735+0000 7fe12703a740 1 -- 172.21.15.32:0/3538415406 >> 172.21.15.32:0/3538415406 conn(0x55c48688c300 msgr2=0x55c486c35630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.736+0000 7fe12703a740 1 -- 172.21.15.32:0/3538415406 shutdown_connections 2024-04-30T08:36:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.736+0000 7fe12703a740 1 -- 172.21.15.32:0/3538415406 wait complete. 2024-04-30T08:36:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.736+0000 7fe12703a740 1 -- start start 2024-04-30T08:36:11.738 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.736+0000 7fe12703a740 1 Processor -- start 2024-04-30T08:36:11.740 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.737+0000 7fe12703a740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c48686f2f0 0x55c486c489f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:11.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.737+0000 7fe12703a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4868774d0 0x55c486c48f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:11.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.737+0000 7fe12703a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4868981b0 0x55c486c49e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:11.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.737+0000 7fe12703a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe1181affd0 con 0x55c4868774d0 2024-04-30T08:36:11.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.737+0000 7fe12703a740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe1181af900 con 0x55c48686f2f0 2024-04-30T08:36:11.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.737+0000 7fe12703a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe1181acb20 con 0x55c4868981b0 2024-04-30T08:36:11.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.737+0000 7fe12703a740 10 client.0 _ll_register_callbacks cb 0x7fe10c0097b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:36:11.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.737+0000 7fe11cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4868774d0 0x55c486c48f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:11.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.737+0000 7fe11cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4868774d0 0x55c486c48f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:41970/0 (socket says 172.21.15.32:41970) 2024-04-30T08:36:11.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.737+0000 7fe11cc00640 1 -- 172.21.15.32:0/3888748295 learned_addr learned my addr 172.21.15.32:0/3888748295 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:11.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.737+0000 7fe11e000640 1 --2- 172.21.15.32:0/3888748295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4868981b0 0x55c486c49e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:11.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.737+0000 7fe11d600640 1 --2- 172.21.15.32:0/3888748295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c48686f2f0 0x55c486c489f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:11.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.737+0000 7fe11cc00640 1 -- 172.21.15.32:0/3888748295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4868981b0 msgr2=0x55c486c49e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:11.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.737+0000 7fe11cc00640 1 --2- 172.21.15.32:0/3888748295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4868981b0 0x55c486c49e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:11.741 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.738+0000 7fe11cc00640 1 -- 172.21.15.32:0/3888748295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c48686f2f0 msgr2=0x55c486c489f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:11.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.738+0000 7fe11cc00640 1 --2- 172.21.15.32:0/3888748295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c48686f2f0 0x55c486c489f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:11.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.738+0000 7fe11cc00640 1 -- 172.21.15.32:0/3888748295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c486c3ec30 con 0x55c4868774d0 2024-04-30T08:36:11.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.738+0000 7fe11d600640 1 --2- 172.21.15.32:0/3888748295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c48686f2f0 0x55c486c489f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:36:11.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.738+0000 7fe11cc00640 1 --2- 172.21.15.32:0/3888748295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4868774d0 0x55c486c48f30 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe104009820 tx=0x7fe1040095a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feab77265b8cfbfe server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:11.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.738+0000 7fe0fea00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:36:11.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.738+0000 7fe0fea00640 1 -- 172.21.15.32:0/3888748295 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1040043e0 con 0x55c4868774d0 2024-04-30T08:36:11.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.738+0000 7fe0fea00640 1 -- 172.21.15.32:0/3888748295 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fe104005040 con 0x55c4868774d0 2024-04-30T08:36:11.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.738+0000 7fe12703a740 1 -- 172.21.15.32:0/3888748295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe10c0fbe50 con 0x55c4868774d0 2024-04-30T08:36:11.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.739+0000 7fe0fea00640 1 -- 172.21.15.32:0/3888748295 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe104005240 con 0x55c4868774d0 2024-04-30T08:36:11.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.739+0000 7fe12703a740 10 client.15744 fetch_fsmap learned FSMap version 28 2024-04-30T08:36:11.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.739+0000 7fe12703a740 1 -- 172.21.15.32:0/3888748295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55c486c4b580 con 0x55c4868774d0 2024-04-30T08:36:11.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.739+0000 7fe0fea00640 1 -- 172.21.15.32:0/3888748295 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe10c0fbe50 con 0x55c4868774d0 2024-04-30T08:36:11.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.739+0000 7fe0fea00640 1 -- 172.21.15.32:0/3888748295 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe1040035f0 con 0x55c4868774d0 2024-04-30T08:36:11.742 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.739+0000 7fe12703a740 10 client.15744 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:36:11.743 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.739+0000 7fe12703a740 10 client.15744 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-30T08:36:11.743 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.739+0000 7fe12703a740 20 client.15744 populate_metadata read hostname 'smithi032' 2024-04-30T08:36:11.743 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.739+0000 7fe12703a740 1 --2- 172.21.15.32:0/3888748295 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55c48682c1c0 0x55c486cbc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:11.743 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.739+0000 7fe12703a740 4 client.15744 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-30T08:36:11.743 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.739+0000 7fe12703a740 1 -- 172.21.15.32:0/3888748295 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c486c3abf0 con 0x55c48682c1c0 2024-04-30T08:36:11.743 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.740+0000 7fe11d600640 1 --2- 172.21.15.32:0/3888748295 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55c48682c1c0 0x55c486cbc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:11.743 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.740+0000 7fe11d600640 1 --2- 172.21.15.32:0/3888748295 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55c48682c1c0 0x55c486cbc130 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2499a2f532ff65d server_cookie=a55d740c9f68a7ed in_seq=0 out_seq=0 2024-04-30T08:36:11.743 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.740+0000 7fe0fea00640 10 client.15744 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:11.745 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.744+0000 7fe0fea00640 1 -- 172.21.15.32:0/3888748295 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c486c3abf0 con 0x55c48682c1c0 2024-04-30T08:36:11.745 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.744+0000 7fe0fea00640 10 client.15744 handle_command_reply: tid=0 2024-04-30T08:36:11.745 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.744+0000 7fe12703a740 1 client.15744 shutdown 2024-04-30T08:36:11.745 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.744+0000 7fe12703a740 1 -- 172.21.15.32:0/3888748295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4868774d0 msgr2=0x55c486c48f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:11.745 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.744+0000 7fe12703a740 1 --2- 172.21.15.32:0/3888748295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4868774d0 0x55c486c48f30 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe104009820 tx=0x7fe1040095a0 comp rx=0 tx=0).stop 2024-04-30T08:36:11.745 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.744+0000 7fe12703a740 1 -- 172.21.15.32:0/3888748295 shutdown_connections 2024-04-30T08:36:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.744+0000 7fe12703a740 1 --2- 172.21.15.32:0/3888748295 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55c48682c1c0 0x55c486cbc130 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.744+0000 7fe12703a740 1 --2- 172.21.15.32:0/3888748295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c4868981b0 0x55c486c49e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.744+0000 7fe12703a740 1 --2- 172.21.15.32:0/3888748295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55c4868774d0 0x55c486c48f30 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.744+0000 7fe0fea00640 0 client.15744 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.744+0000 7fe12703a740 1 --2- 172.21.15.32:0/3888748295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55c48686f2f0 0x55c486c489f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.744+0000 7fe12703a740 1 -- 172.21.15.32:0/3888748295 >> 172.21.15.32:0/3888748295 conn(0x55c48688c300 msgr2=0x55c486c38a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.745+0000 7fe12703a740 1 -- 172.21.15.32:0/3888748295 shutdown_connections 2024-04-30T08:36:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.745+0000 7fe12703a740 1 -- 172.21.15.32:0/3888748295 wait complete. 2024-04-30T08:36:11.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.745+0000 7fe12703a740 20 client.15744 trim_cache size 0 max 16384 2024-04-30T08:36:11.747 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-30T08:36:11.747 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-30T08:36:11.747 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-30T08:36:11.747 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-30T08:36:11.747 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.745+0000 7fe11ea00640 1 -- 172.21.15.32:0/2160443905 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe1000afbf0 msgr2=0x7fe1000b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:11.747 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.745+0000 7fe11ea00640 1 --2- 172.21.15.32:0/2160443905 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe1000afbf0 0x7fe1000b2100 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe110009cf0 tx=0x7fe110009600 comp rx=0 tx=0).stop 2024-04-30T08:36:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.745+0000 7fe11ea00640 1 -- 172.21.15.32:0/2160443905 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe118103a00 msgr2=0x7fe1180768d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.745+0000 7fe11ea00640 1 --2- 172.21.15.32:0/2160443905 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe118103a00 0x7fe1180768d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fe10c0304b0 tx=0x7fe10c032e30 comp rx=0 tx=0).stop 2024-04-30T08:36:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.746+0000 7fe11ea00640 1 -- 172.21.15.32:0/2160443905 shutdown_connections 2024-04-30T08:36:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.746+0000 7fe11ea00640 1 --2- 172.21.15.32:0/2160443905 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe1000afbf0 0x7fe1000b2100 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.746+0000 7fe11ea00640 1 --2- 172.21.15.32:0/2160443905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe118109940 0x7fe1180754b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.746+0000 7fe11ea00640 1 --2- 172.21.15.32:0/2160443905 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe118104c40 0x7fe118074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.746+0000 7fe11ea00640 1 --2- 172.21.15.32:0/2160443905 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe118103a00 0x7fe1180768d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.746+0000 7fe11ea00640 1 -- 172.21.15.32:0/2160443905 >> 172.21.15.32:0/2160443905 conn(0x7fe1180fec10 msgr2=0x7fe118105920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.746+0000 7fe11ea00640 1 -- 172.21.15.32:0/2160443905 shutdown_connections 2024-04-30T08:36:11.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:11.746+0000 7fe11ea00640 1 -- 172.21.15.32:0/2160443905 wait complete. 2024-04-30T08:36:11.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:11 smithi071 ceph-mon[33385]: pgmap v455: 129 pgs: 129 active+clean; 673 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 767 B/s wr, 13 op/s 2024-04-30T08:36:13.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:13 smithi032 ceph-mon[28331]: pgmap v456: 129 pgs: 129 active+clean; 673 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-30T08:36:13.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:13 smithi071 ceph-mon[33385]: pgmap v456: 129 pgs: 129 active+clean; 673 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-30T08:36:13.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:13 smithi189 ceph-mon[33038]: pgmap v456: 129 pgs: 129 active+clean; 673 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-30T08:36:15.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:15 smithi032 ceph-mon[28331]: pgmap v457: 129 pgs: 129 active+clean; 673 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:36:15.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:15 smithi071 ceph-mon[33385]: pgmap v457: 129 pgs: 129 active+clean; 673 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:36:15.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:15 smithi189 ceph-mon[33038]: pgmap v457: 129 pgs: 129 active+clean; 673 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:36:17.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:17 smithi032 ceph-mon[28331]: pgmap v458: 129 pgs: 129 active+clean; 674 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-30T08:36:17.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:17 smithi071 ceph-mon[33385]: pgmap v458: 129 pgs: 129 active+clean; 674 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-30T08:36:17.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:17 smithi189 ceph-mon[33038]: pgmap v458: 129 pgs: 129 active+clean; 674 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-30T08:36:19.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:19 smithi032 ceph-mon[28331]: pgmap v459: 129 pgs: 129 active+clean; 674 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-30T08:36:19.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:19 smithi071 ceph-mon[33385]: pgmap v459: 129 pgs: 129 active+clean; 674 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-30T08:36:19.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:19 smithi189 ceph-mon[33038]: pgmap v459: 129 pgs: 129 active+clean; 674 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-30T08:36:20.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:20 smithi071 ceph-mon[33385]: pgmap v460: 129 pgs: 129 active+clean; 674 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-30T08:36:20.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:20 smithi189 ceph-mon[33038]: pgmap v460: 129 pgs: 129 active+clean; 674 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-30T08:36:20.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:20 smithi032 ceph-mon[28331]: pgmap v460: 129 pgs: 129 active+clean; 674 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-30T08:36:23.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:22 smithi032 ceph-mon[28331]: pgmap v461: 129 pgs: 129 active+clean; 674 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-30T08:36:23.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:22 smithi071 ceph-mon[33385]: pgmap v461: 129 pgs: 129 active+clean; 674 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-30T08:36:23.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:22 smithi189 ceph-mon[33038]: pgmap v461: 129 pgs: 129 active+clean; 674 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-30T08:36:23.790 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-30T08:36:23.791 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-30T08:36:24.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.040+0000 7f29eca00640 1 -- 172.21.15.32:0/3209606333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29e8075010 msgr2=0x7f29e8075420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:24.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.040+0000 7f29eca00640 1 --2- 172.21.15.32:0/3209606333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29e8075010 0x7f29e8075420 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f29d8009900 tx=0x7f29d802a400 comp rx=0 tx=0).stop 2024-04-30T08:36:24.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.040+0000 7f29eca00640 1 -- 172.21.15.32:0/3209606333 shutdown_connections 2024-04-30T08:36:24.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.040+0000 7f29eca00640 1 --2- 172.21.15.32:0/3209606333 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e81100f0 0x7f29e8112510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.040+0000 7f29eca00640 1 --2- 172.21.15.32:0/3209606333 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29e8075960 0x7f29e810fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.040+0000 7f29eca00640 1 --2- 172.21.15.32:0/3209606333 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29e8075010 0x7f29e8075420 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.040+0000 7f29eca00640 1 -- 172.21.15.32:0/3209606333 >> 172.21.15.32:0/3209606333 conn(0x7f29e80feba0 msgr2=0x7f29e8101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:24.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.040+0000 7f29eca00640 1 -- 172.21.15.32:0/3209606333 shutdown_connections 2024-04-30T08:36:24.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.040+0000 7f29eca00640 1 -- 172.21.15.32:0/3209606333 wait complete. 2024-04-30T08:36:24.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.041+0000 7f29eca00640 1 Processor -- start 2024-04-30T08:36:24.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.042+0000 7f29eca00640 1 -- start start 2024-04-30T08:36:24.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.042+0000 7f29eca00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e8075010 0x7f29e81a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.042+0000 7f29eca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29e8075960 0x7f29e81a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.042+0000 7f29eca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29e81100f0 0x7f29e81a7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.042+0000 7f29eca00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f29e8115b20 con 0x7f29e81100f0 2024-04-30T08:36:24.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.042+0000 7f29eca00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f29e81159a0 con 0x7f29e8075010 2024-04-30T08:36:24.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.042+0000 7f29eca00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f29e8115ca0 con 0x7f29e8075960 2024-04-30T08:36:24.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.042+0000 7f29e7400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e8075010 0x7f29e81a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:24.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.042+0000 7f29e6a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29e8075960 0x7f29e81a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:24.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.043+0000 7f29e7400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e8075010 0x7f29e81a06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:60612/0 (socket says 172.21.15.32:60612) 2024-04-30T08:36:24.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.043+0000 7f29e7400640 1 -- 172.21.15.32:0/737151295 learned_addr learned my addr 172.21.15.32:0/737151295 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:24.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.043+0000 7f29e7e00640 1 --2- 172.21.15.32:0/737151295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29e81100f0 0x7f29e81a7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:24.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.043+0000 7f29e7400640 1 -- 172.21.15.32:0/737151295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29e8075960 msgr2=0x7f29e81a0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:24.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.043+0000 7f29e7400640 1 --2- 172.21.15.32:0/737151295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29e8075960 0x7f29e81a0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.043+0000 7f29e7400640 1 -- 172.21.15.32:0/737151295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29e81100f0 msgr2=0x7f29e81a7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:24.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.043+0000 7f29e7400640 1 --2- 172.21.15.32:0/737151295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29e81100f0 0x7f29e81a7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.043+0000 7f29e7400640 1 -- 172.21.15.32:0/737151295 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29e81a8200 con 0x7f29e8075010 2024-04-30T08:36:24.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.043+0000 7f29e7e00640 1 --2- 172.21.15.32:0/737151295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29e81100f0 0x7f29e81a7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:36:24.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.044+0000 7f29e7400640 1 --2- 172.21.15.32:0/737151295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e8075010 0x7f29e81a06c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f29d8031440 tx=0x7f29d8003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:24.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.044+0000 7f29dfe00640 1 -- 172.21.15.32:0/737151295 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d803f070 con 0x7f29e8075010 2024-04-30T08:36:24.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.044+0000 7f29eca00640 1 -- 172.21.15.32:0/737151295 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29e81a8490 con 0x7f29e8075010 2024-04-30T08:36:24.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.044+0000 7f29dfe00640 1 -- 172.21.15.32:0/737151295 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29d8031c50 con 0x7f29e8075010 2024-04-30T08:36:24.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.044+0000 7f29dfe00640 1 -- 172.21.15.32:0/737151295 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d8035030 con 0x7f29e8075010 2024-04-30T08:36:24.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.044+0000 7f29eca00640 1 -- 172.21.15.32:0/737151295 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29e81a89d0 con 0x7f29e8075010 2024-04-30T08:36:24.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.048+0000 7f29f4fa0740 1 Processor -- start 2024-04-30T08:36:24.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.048+0000 7f29f4fa0740 1 -- start start 2024-04-30T08:36:24.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.048+0000 7f29f4fa0740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fc8d379390 0x55fc8d71a440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.048+0000 7f29f4fa0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc8d3504d0 0x55fc8d71a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.048+0000 7f29f4fa0740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fc8d3586b0 0x55fc8d7218c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.048+0000 7f29f4fa0740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55fc8d67fd50 con 0x55fc8d3586b0 2024-04-30T08:36:24.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.048+0000 7f29f4fa0740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55fc8d624ef0 con 0x55fc8d379390 2024-04-30T08:36:24.049 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.048+0000 7f29f4fa0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55fc8d67bd10 con 0x55fc8d3504d0 2024-04-30T08:36:24.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.049+0000 7f29e7e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fc8d379390 0x55fc8d71a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:24.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.049+0000 7f29e7e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fc8d379390 0x55fc8d71a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:60614/0 (socket says 172.21.15.32:60614) 2024-04-30T08:36:24.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.049+0000 7f29e7e00640 1 -- 172.21.15.32:0/3565334310 learned_addr learned my addr 172.21.15.32:0/3565334310 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:24.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.049+0000 7f29e6a00640 1 --2- 172.21.15.32:0/3565334310 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fc8d3586b0 0x55fc8d7218c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:24.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.049+0000 7f29e7e00640 1 -- 172.21.15.32:0/3565334310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc8d3504d0 msgr2=0x55fc8d71a9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-30T08:36:24.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.049+0000 7f29e7e00640 1 --2- 172.21.15.32:0/3565334310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc8d3504d0 0x55fc8d71a9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.049+0000 7f29e7e00640 1 -- 172.21.15.32:0/3565334310 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fc8d3586b0 msgr2=0x55fc8d7218c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:24.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.049+0000 7f29e7e00640 1 --2- 172.21.15.32:0/3565334310 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fc8d3586b0 0x55fc8d7218c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.050 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.049+0000 7f29e7e00640 1 -- 172.21.15.32:0/3565334310 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc8d45b7d0 con 0x55fc8d379390 2024-04-30T08:36:24.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.049+0000 7f29e7e00640 1 --2- 172.21.15.32:0/3565334310 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fc8d379390 0x55fc8d71a440 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f29e00117a0 tx=0x7f29e0011c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11d35813cd2a05b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:24.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.049+0000 7f29dfe00640 1 -- 172.21.15.32:0/737151295 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f29d8042070 con 0x7f29e8075010 2024-04-30T08:36:24.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.049+0000 7f29d3e00640 1 -- 172.21.15.32:0/3565334310 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e0011220 con 0x55fc8d379390 2024-04-30T08:36:24.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.050+0000 7f29d3e00640 1 -- 172.21.15.32:0/3565334310 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29e000f9a0 con 0x55fc8d379390 2024-04-30T08:36:24.051 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.050+0000 7f29d3e00640 1 -- 172.21.15.32:0/3565334310 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e00045d0 con 0x55fc8d379390 2024-04-30T08:36:24.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.051+0000 7f29f4fa0740 1 -- 172.21.15.32:0/3565334310 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fc8d379390 msgr2=0x55fc8d71a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:24.052 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.051+0000 7f29f4fa0740 1 --2- 172.21.15.32:0/3565334310 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fc8d379390 0x55fc8d71a440 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f29e00117a0 tx=0x7f29e0011c60 comp rx=0 tx=0).stop 2024-04-30T08:36:24.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.051+0000 7f29f4fa0740 1 -- 172.21.15.32:0/3565334310 shutdown_connections 2024-04-30T08:36:24.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.051+0000 7f29f4fa0740 1 --2- 172.21.15.32:0/3565334310 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fc8d3586b0 0x55fc8d7218c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.051+0000 7f29f4fa0740 1 --2- 172.21.15.32:0/3565334310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc8d3504d0 0x55fc8d71a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.051+0000 7f29f4fa0740 1 --2- 172.21.15.32:0/3565334310 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fc8d379390 0x55fc8d71a440 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.051+0000 7f29f4fa0740 1 -- 172.21.15.32:0/3565334310 >> 172.21.15.32:0/3565334310 conn(0x55fc8d36d4e0 msgr2=0x55fc8d715dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:24.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.051+0000 7f29f4fa0740 1 -- 172.21.15.32:0/3565334310 shutdown_connections 2024-04-30T08:36:24.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.051+0000 7f29f4fa0740 1 -- 172.21.15.32:0/3565334310 wait complete. 2024-04-30T08:36:24.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.051+0000 7f29dfe00640 1 --2- 172.21.15.32:0/737151295 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f29cc0afb00 0x7f29cc0b2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.052+0000 7f29dfe00640 1 -- 172.21.15.32:0/737151295 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f29d80ff860 con 0x7f29e8075010 2024-04-30T08:36:24.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.052+0000 7f29e6a00640 1 --2- 172.21.15.32:0/737151295 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f29cc0afb00 0x7f29cc0b2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:24.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.052+0000 7f29f4fa0740 1 -- start start 2024-04-30T08:36:24.053 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.052+0000 7f29f4fa0740 1 Processor -- start 2024-04-30T08:36:24.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.053+0000 7f29e6a00640 1 --2- 172.21.15.32:0/737151295 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f29cc0afb00 0x7f29cc0b2010 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x55fc8d716b60 tx=0x7f29d4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:24.054 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.053+0000 7f29f4fa0740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fc8d3504d0 0x55fc8d7243d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.053+0000 7f29f4fa0740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fc8d3586b0 0x55fc8d724910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.053+0000 7f29f4fa0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc8d379390 0x55fc8d72b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.053+0000 7f29f4fa0740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55fc8d722360 con 0x55fc8d3504d0 2024-04-30T08:36:24.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.053+0000 7f29f4fa0740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55fc8d638990 con 0x55fc8d3586b0 2024-04-30T08:36:24.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.053+0000 7f29f4fa0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55fc8d7224e0 con 0x55fc8d379390 2024-04-30T08:36:24.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.053+0000 7f29f4fa0740 10 client.0 _ll_register_callbacks cb 0x55fc8d637110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:36:24.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.054+0000 7f29e7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc8d379390 0x55fc8d72b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:24.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.054+0000 7f29e7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc8d379390 0x55fc8d72b870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:42760/0 (socket says 172.21.15.32:42760) 2024-04-30T08:36:24.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.054+0000 7f29e7e00640 1 -- 172.21.15.32:0/1040462819 learned_addr learned my addr 172.21.15.32:0/1040462819 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:24.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.054+0000 7f29e7400640 1 --2- 172.21.15.32:0/1040462819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fc8d3504d0 0x55fc8d7243d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:24.055 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.054+0000 7f29e6a00640 1 --2- 172.21.15.32:0/1040462819 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fc8d3586b0 0x55fc8d724910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:24.056 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.054+0000 7f29e7e00640 1 -- 172.21.15.32:0/1040462819 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fc8d3586b0 msgr2=0x55fc8d724910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:24.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.054+0000 7f29e7e00640 1 --2- 172.21.15.32:0/1040462819 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fc8d3586b0 0x55fc8d724910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.054+0000 7f29e7e00640 1 -- 172.21.15.32:0/1040462819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fc8d3504d0 msgr2=0x55fc8d7243d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:24.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.054+0000 7f29e7e00640 1 --2- 172.21.15.32:0/1040462819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fc8d3504d0 0x55fc8d7243d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.054+0000 7f29e7e00640 1 -- 172.21.15.32:0/1040462819 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc8d3913f0 con 0x55fc8d379390 2024-04-30T08:36:24.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.055+0000 7f29e7e00640 1 --2- 172.21.15.32:0/1040462819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc8d379390 0x55fc8d72b870 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f29e0006010 tx=0x7f29e0011270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2f3d236e70ce7d32 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:24.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.055+0000 7f29d0c00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-30T08:36:24.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.055+0000 7f29d0c00640 1 -- 172.21.15.32:0/1040462819 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e001e040 con 0x55fc8d379390 2024-04-30T08:36:24.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.055+0000 7f29d0c00640 1 -- 172.21.15.32:0/1040462819 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f29e0004c40 con 0x55fc8d379390 2024-04-30T08:36:24.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.055+0000 7f29f4fa0740 1 -- 172.21.15.32:0/1040462819 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29e000f9a0 con 0x55fc8d379390 2024-04-30T08:36:24.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.056+0000 7f29d0c00640 1 -- 172.21.15.32:0/1040462819 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f29e0031050 con 0x55fc8d379390 2024-04-30T08:36:24.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.056+0000 7f29f4fa0740 10 client.25879 fetch_fsmap learned FSMap version 28 2024-04-30T08:36:24.057 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.056+0000 7f29f4fa0740 1 -- 172.21.15.32:0/1040462819 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fc8d3510f0 con 0x55fc8d379390 2024-04-30T08:36:24.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.056+0000 7f29d0c00640 1 -- 172.21.15.32:0/1040462819 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29e000f9a0 con 0x55fc8d379390 2024-04-30T08:36:24.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.057+0000 7f29d0c00640 1 -- 172.21.15.32:0/1040462819 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f29e0018070 con 0x55fc8d379390 2024-04-30T08:36:24.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.057+0000 7f29f4fa0740 10 client.25879 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:36:24.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.057+0000 7f29f4fa0740 10 client.25879 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:36:24.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.057+0000 7f29f4fa0740 20 client.25879 populate_metadata read hostname 'smithi032' 2024-04-30T08:36:24.058 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.057+0000 7f29f4fa0740 1 --2- 172.21.15.32:0/1040462819 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55fc8d337f60 0x55fc8d731b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.057+0000 7f29f4fa0740 4 client.25879 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:36:24.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.057+0000 7f29f4fa0740 1 -- 172.21.15.32:0/1040462819 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fc8d693ef0 con 0x55fc8d337f60 2024-04-30T08:36:24.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.058+0000 7f29e7400640 1 --2- 172.21.15.32:0/1040462819 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55fc8d337f60 0x55fc8d731b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:24.059 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.058+0000 7f29e7400640 1 --2- 172.21.15.32:0/1040462819 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55fc8d337f60 0x55fc8d731b00 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b83356c3843fe925 server_cookie=b8af1c5efe259b17 in_seq=0 out_seq=0 2024-04-30T08:36:24.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.058+0000 7f29d0c00640 10 client.25879 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:24.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.059+0000 7f29d0c00640 1 -- 172.21.15.32:0/1040462819 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55fc8d693ef0 con 0x55fc8d337f60 2024-04-30T08:36:24.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.059+0000 7f29d0c00640 10 client.25879 handle_command_reply: tid=0 2024-04-30T08:36:24.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.059+0000 7f29f4fa0740 1 client.25879 shutdown 2024-04-30T08:36:24.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.059+0000 7f29f4fa0740 1 -- 172.21.15.32:0/1040462819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc8d379390 msgr2=0x55fc8d72b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:24.060 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.059+0000 7f29f4fa0740 1 --2- 172.21.15.32:0/1040462819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc8d379390 0x55fc8d72b870 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f29e0006010 tx=0x7f29e0011270 comp rx=0 tx=0).stop 2024-04-30T08:36:24.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.060+0000 7f29f4fa0740 1 -- 172.21.15.32:0/1040462819 shutdown_connections 2024-04-30T08:36:24.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.060+0000 7f29f4fa0740 1 --2- 172.21.15.32:0/1040462819 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55fc8d337f60 0x55fc8d731b00 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.060+0000 7f29f4fa0740 1 --2- 172.21.15.32:0/1040462819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc8d379390 0x55fc8d72b870 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.060+0000 7f29f4fa0740 1 --2- 172.21.15.32:0/1040462819 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fc8d3586b0 0x55fc8d724910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.060+0000 7f29f4fa0740 1 --2- 172.21.15.32:0/1040462819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fc8d3504d0 0x55fc8d7243d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.060+0000 7f29f4fa0740 1 -- 172.21.15.32:0/1040462819 >> 172.21.15.32:0/1040462819 conn(0x55fc8d36d4e0 msgr2=0x55fc8d714e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:24.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.060+0000 7f29d0c00640 0 client.25879 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:24.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.060+0000 7f29f4fa0740 1 -- 172.21.15.32:0/1040462819 shutdown_connections 2024-04-30T08:36:24.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.060+0000 7f29f4fa0740 1 -- 172.21.15.32:0/1040462819 wait complete. 2024-04-30T08:36:24.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.060+0000 7f29f4fa0740 20 client.25879 trim_cache size 0 max 16384 2024-04-30T08:36:24.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.079+0000 7f29f4fa0740 1 Processor -- start 2024-04-30T08:36:24.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.079+0000 7f29f4fa0740 1 -- start start 2024-04-30T08:36:24.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.080+0000 7f29f4fa0740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fc8d3504d0 0x55fc8d71ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.080+0000 7f29f4fa0740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fc8d3586b0 0x55fc8d7243d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.080+0000 7f29f4fa0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc8d379390 0x55fc8d723140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.080+0000 7f29f4fa0740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f29e81b2f90 con 0x55fc8d3504d0 2024-04-30T08:36:24.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.080+0000 7f29f4fa0740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f29e81b4ad0 con 0x55fc8d3586b0 2024-04-30T08:36:24.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.080+0000 7f29f4fa0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f29e81b3660 con 0x55fc8d379390 2024-04-30T08:36:24.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.080+0000 7f29e7400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fc8d3504d0 0x55fc8d71ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:24.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.080+0000 7f29e7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc8d379390 0x55fc8d723140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:24.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.080+0000 7f29e6a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fc8d3586b0 0x55fc8d7243d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:24.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.080+0000 7f29e7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc8d379390 0x55fc8d723140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:42772/0 (socket says 172.21.15.32:42772) 2024-04-30T08:36:24.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.080+0000 7f29e7e00640 1 -- 172.21.15.32:0/677813651 learned_addr learned my addr 172.21.15.32:0/677813651 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:24.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.081+0000 7f29e7e00640 1 -- 172.21.15.32:0/677813651 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fc8d3586b0 msgr2=0x55fc8d7243d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:24.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.081+0000 7f29e7e00640 1 --2- 172.21.15.32:0/677813651 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fc8d3586b0 0x55fc8d7243d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.081+0000 7f29e7e00640 1 -- 172.21.15.32:0/677813651 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fc8d3504d0 msgr2=0x55fc8d71ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:24.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.081+0000 7f29e7e00640 1 --2- 172.21.15.32:0/677813651 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fc8d3504d0 0x55fc8d71ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.081+0000 7f29e7e00640 1 -- 172.21.15.32:0/677813651 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc8d71b230 con 0x55fc8d379390 2024-04-30T08:36:24.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.081+0000 7f29e7e00640 1 --2- 172.21.15.32:0/677813651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc8d379390 0x55fc8d723140 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f29e0000f80 tx=0x7f29e00115a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c8e5897187b22dde server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:24.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.081+0000 7f29d3400640 1 -- 172.21.15.32:0/677813651 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e0039020 con 0x55fc8d379390 2024-04-30T08:36:24.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.081+0000 7f29d3400640 1 -- 172.21.15.32:0/677813651 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29e0028070 con 0x55fc8d379390 2024-04-30T08:36:24.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.081+0000 7f29d3400640 1 -- 172.21.15.32:0/677813651 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29e00058d0 con 0x55fc8d379390 2024-04-30T08:36:24.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.082+0000 7f29f4fa0740 1 -- 172.21.15.32:0/677813651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc8d379390 msgr2=0x55fc8d723140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:24.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.082+0000 7f29f4fa0740 1 --2- 172.21.15.32:0/677813651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc8d379390 0x55fc8d723140 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f29e0000f80 tx=0x7f29e00115a0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.082+0000 7f29f4fa0740 1 -- 172.21.15.32:0/677813651 shutdown_connections 2024-04-30T08:36:24.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.082+0000 7f29f4fa0740 1 --2- 172.21.15.32:0/677813651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc8d379390 0x55fc8d723140 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.082+0000 7f29f4fa0740 1 --2- 172.21.15.32:0/677813651 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fc8d3586b0 0x55fc8d7243d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.082+0000 7f29f4fa0740 1 --2- 172.21.15.32:0/677813651 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fc8d3504d0 0x55fc8d71ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.082+0000 7f29f4fa0740 1 -- 172.21.15.32:0/677813651 >> 172.21.15.32:0/677813651 conn(0x55fc8d36d4e0 msgr2=0x55fc8d722a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:24.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.083+0000 7f29f4fa0740 1 -- 172.21.15.32:0/677813651 shutdown_connections 2024-04-30T08:36:24.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.083+0000 7f29f4fa0740 1 -- 172.21.15.32:0/677813651 wait complete. 2024-04-30T08:36:24.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.083+0000 7f29f4fa0740 1 -- start start 2024-04-30T08:36:24.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.083+0000 7f29f4fa0740 1 Processor -- start 2024-04-30T08:36:24.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.084+0000 7f29f4fa0740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fc8d3504d0 0x55fc8d729410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.084+0000 7f29f4fa0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc8d3586b0 0x55fc8d729950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.084+0000 7f29f4fa0740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fc8d379390 0x55fc8d72a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.084+0000 7f29f4fa0740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f29e81b4400 con 0x55fc8d379390 2024-04-30T08:36:24.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.084+0000 7f29f4fa0740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f29e81b3d30 con 0x55fc8d3504d0 2024-04-30T08:36:24.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.084+0000 7f29f4fa0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f29e81b0f50 con 0x55fc8d3586b0 2024-04-30T08:36:24.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.084+0000 7f29f4fa0740 10 client.0 _ll_register_callbacks cb 0x7f29d40097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:36:24.087 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.084+0000 7f29e6a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc8d3586b0 0x55fc8d729950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:24.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.084+0000 7f29e6a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc8d3586b0 0x55fc8d729950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:42784/0 (socket says 172.21.15.32:42784) 2024-04-30T08:36:24.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.084+0000 7f29e6a00640 1 -- 172.21.15.32:0/1279886484 learned_addr learned my addr 172.21.15.32:0/1279886484 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:24.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.084+0000 7f29e7e00640 1 --2- 172.21.15.32:0/1279886484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fc8d379390 0x55fc8d72a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:24.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.084+0000 7f29e7400640 1 --2- 172.21.15.32:0/1279886484 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fc8d3504d0 0x55fc8d729410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:24.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.085+0000 7f29e7400640 1 -- 172.21.15.32:0/1279886484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc8d3586b0 msgr2=0x55fc8d729950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:24.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.085+0000 7f29e7400640 1 --2- 172.21.15.32:0/1279886484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc8d3586b0 0x55fc8d729950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.085+0000 7f29e7400640 1 -- 172.21.15.32:0/1279886484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fc8d379390 msgr2=0x55fc8d72a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:24.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.085+0000 7f29e7400640 1 --2- 172.21.15.32:0/1279886484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fc8d379390 0x55fc8d72a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.085+0000 7f29e7400640 1 -- 172.21.15.32:0/1279886484 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc8d72ae40 con 0x55fc8d3504d0 2024-04-30T08:36:24.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.085+0000 7f29e6a00640 1 --2- 172.21.15.32:0/1279886484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc8d3586b0 0x55fc8d729950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:36:24.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.085+0000 7f29e7400640 1 --2- 172.21.15.32:0/1279886484 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fc8d3504d0 0x55fc8d729410 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f29d80fce40 tx=0x7f29d803b790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=737eb5f6b9bfbd1d server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:24.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.085+0000 7f29d0c00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:36:24.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.085+0000 7f29d0c00640 1 -- 172.21.15.32:0/1279886484 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d8049040 con 0x55fc8d3504d0 2024-04-30T08:36:24.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.086+0000 7f29d0c00640 1 -- 172.21.15.32:0/1279886484 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f29d803b460 con 0x55fc8d3504d0 2024-04-30T08:36:24.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.086+0000 7f29f4fa0740 1 -- 172.21.15.32:0/1279886484 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29e0028070 con 0x55fc8d3504d0 2024-04-30T08:36:24.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.086+0000 7f29d0c00640 1 -- 172.21.15.32:0/1279886484 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f29d80c73d0 con 0x55fc8d3504d0 2024-04-30T08:36:24.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.086+0000 7f29f4fa0740 10 client.25886 fetch_fsmap learned FSMap version 28 2024-04-30T08:36:24.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.086+0000 7f29f4fa0740 1 -- 172.21.15.32:0/1279886484 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fc8d72c1a0 con 0x55fc8d3504d0 2024-04-30T08:36:24.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.086+0000 7f29d0c00640 1 -- 172.21.15.32:0/1279886484 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f29e0028070 con 0x55fc8d3504d0 2024-04-30T08:36:24.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.087+0000 7f29d0c00640 1 -- 172.21.15.32:0/1279886484 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f29d80d01e0 con 0x55fc8d3504d0 2024-04-30T08:36:24.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.087+0000 7f29f4fa0740 10 client.25886 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:36:24.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.087+0000 7f29f4fa0740 10 client.25886 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:36:24.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.087+0000 7f29f4fa0740 20 client.25886 populate_metadata read hostname 'smithi032' 2024-04-30T08:36:24.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.087+0000 7f29f4fa0740 1 --2- 172.21.15.32:0/1279886484 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55fc8d30d3a0 0x55fc8d79c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.087+0000 7f29f4fa0740 4 client.25886 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-30T08:36:24.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.087+0000 7f29f4fa0740 1 -- 172.21.15.32:0/1279886484 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fc8d68d960 con 0x55fc8d30d3a0 2024-04-30T08:36:24.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.087+0000 7f29e6a00640 1 --2- 172.21.15.32:0/1279886484 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55fc8d30d3a0 0x55fc8d79c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:24.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.088+0000 7f29e6a00640 1 --2- 172.21.15.32:0/1279886484 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55fc8d30d3a0 0x55fc8d79c480 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2ceb2c2553cacdc server_cookie=29d2ecf4063a08bb in_seq=0 out_seq=0 2024-04-30T08:36:24.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.088+0000 7f29d0c00640 10 client.25886 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:24.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.089+0000 7f29d0c00640 1 -- 172.21.15.32:0/1279886484 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55fc8d68d960 con 0x55fc8d30d3a0 2024-04-30T08:36:24.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.089+0000 7f29d0c00640 10 client.25886 handle_command_reply: tid=0 2024-04-30T08:36:24.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.089+0000 7f29f4fa0740 1 client.25886 shutdown 2024-04-30T08:36:24.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.090+0000 7f29f4fa0740 1 -- 172.21.15.32:0/1279886484 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fc8d3504d0 msgr2=0x55fc8d729410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:24.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.090+0000 7f29f4fa0740 1 --2- 172.21.15.32:0/1279886484 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fc8d3504d0 0x55fc8d729410 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f29d80fce40 tx=0x7f29d803b790 comp rx=0 tx=0).stop 2024-04-30T08:36:24.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.090+0000 7f29f4fa0740 1 -- 172.21.15.32:0/1279886484 shutdown_connections 2024-04-30T08:36:24.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.090+0000 7f29f4fa0740 1 --2- 172.21.15.32:0/1279886484 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55fc8d30d3a0 0x55fc8d79c480 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.090+0000 7f29f4fa0740 1 --2- 172.21.15.32:0/1279886484 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fc8d379390 0x55fc8d72a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.090+0000 7f29f4fa0740 1 --2- 172.21.15.32:0/1279886484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fc8d3586b0 0x55fc8d729950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.090+0000 7f29f4fa0740 1 --2- 172.21.15.32:0/1279886484 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fc8d3504d0 0x55fc8d729410 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.090+0000 7f29d0c00640 0 client.25886 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:24.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.090+0000 7f29f4fa0740 1 -- 172.21.15.32:0/1279886484 >> 172.21.15.32:0/1279886484 conn(0x55fc8d36d4e0 msgr2=0x55fc8d7242e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:24.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.090+0000 7f29f4fa0740 1 -- 172.21.15.32:0/1279886484 shutdown_connections 2024-04-30T08:36:24.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.090+0000 7f29f4fa0740 1 -- 172.21.15.32:0/1279886484 wait complete. 2024-04-30T08:36:24.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.091+0000 7f29f4fa0740 20 client.25886 trim_cache size 0 max 16384 2024-04-30T08:36:24.096 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-30T08:36:24.096 INFO:teuthology.orchestra.run.smithi032.stdout: "status": "no active scrubs running", 2024-04-30T08:36:24.096 INFO:teuthology.orchestra.run.smithi032.stdout: "scrubs": {} 2024-04-30T08:36:24.096 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-30T08:36:24.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.091+0000 7f29eca00640 1 -- 172.21.15.32:0/737151295 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f29cc0afb00 msgr2=0x7f29cc0b2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:24.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.091+0000 7f29eca00640 1 --2- 172.21.15.32:0/737151295 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f29cc0afb00 0x7f29cc0b2010 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x55fc8d716b60 tx=0x7f29d4005f90 comp rx=0 tx=0).stop 2024-04-30T08:36:24.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.091+0000 7f29eca00640 1 -- 172.21.15.32:0/737151295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e8075010 msgr2=0x7f29e81a06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:24.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.091+0000 7f29eca00640 1 --2- 172.21.15.32:0/737151295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e8075010 0x7f29e81a06c0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f29d8031440 tx=0x7f29d8003c10 comp rx=0 tx=0).stop 2024-04-30T08:36:24.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.092+0000 7f29eca00640 1 -- 172.21.15.32:0/737151295 shutdown_connections 2024-04-30T08:36:24.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.092+0000 7f29eca00640 1 --2- 172.21.15.32:0/737151295 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f29cc0afb00 0x7f29cc0b2010 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.092+0000 7f29eca00640 1 --2- 172.21.15.32:0/737151295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f29e81100f0 0x7f29e81a7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.092+0000 7f29eca00640 1 --2- 172.21.15.32:0/737151295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f29e8075960 0x7f29e81a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.092+0000 7f29eca00640 1 --2- 172.21.15.32:0/737151295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f29e8075010 0x7f29e81a06c0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.092+0000 7f29eca00640 1 -- 172.21.15.32:0/737151295 >> 172.21.15.32:0/737151295 conn(0x7f29e80feba0 msgr2=0x7f29e8102f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:24.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.092+0000 7f29eca00640 1 -- 172.21.15.32:0/737151295 shutdown_connections 2024-04-30T08:36:24.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.092+0000 7f29eca00640 1 -- 172.21.15.32:0/737151295 wait complete. 2024-04-30T08:36:24.132 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-30T08:36:24.132 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-30T08:36:24.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.382+0000 7fa8ada00640 1 -- 172.21.15.32:0/3920476465 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8a8104bf0 msgr2=0x7fa8a8105000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:24.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.382+0000 7fa8ada00640 1 --2- 172.21.15.32:0/3920476465 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8a8104bf0 0x7fa8a8105000 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fa894009900 tx=0x7fa89402a3f0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.382+0000 7fa8ada00640 1 -- 172.21.15.32:0/3920476465 shutdown_connections 2024-04-30T08:36:24.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.382+0000 7fa8ada00640 1 --2- 172.21.15.32:0/3920476465 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8a81098f0 0x7fa8a810bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.382+0000 7fa8ada00640 1 --2- 172.21.15.32:0/3920476465 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8a8104bf0 0x7fa8a8105000 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.382+0000 7fa8ada00640 1 --2- 172.21.15.32:0/3920476465 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8a81039b0 0x7fa8a81046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.382+0000 7fa8ada00640 1 -- 172.21.15.32:0/3920476465 >> 172.21.15.32:0/3920476465 conn(0x7fa8a80feba0 msgr2=0x7fa8a8101050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:24.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.383+0000 7fa8ada00640 1 -- 172.21.15.32:0/3920476465 shutdown_connections 2024-04-30T08:36:24.384 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.383+0000 7fa8ada00640 1 -- 172.21.15.32:0/3920476465 wait complete. 2024-04-30T08:36:24.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.384+0000 7fa8ada00640 1 Processor -- start 2024-04-30T08:36:24.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.384+0000 7fa8ada00640 1 -- start start 2024-04-30T08:36:24.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.384+0000 7fa8ada00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8a81039b0 0x7fa8a819c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.384+0000 7fa8ada00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8a8104bf0 0x7fa8a819c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.384+0000 7fa8ada00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8a81098f0 0x7fa8a81a37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.384+0000 7fa8ada00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa8a810f530 con 0x7fa8a81098f0 2024-04-30T08:36:24.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.384+0000 7fa8ada00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa8a810f3b0 con 0x7fa8a8104bf0 2024-04-30T08:36:24.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.384+0000 7fa8ada00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa8a810f6b0 con 0x7fa8a81039b0 2024-04-30T08:36:24.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.385+0000 7fa8ad000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8a81098f0 0x7fa8a81a37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:24.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.385+0000 7fa8a7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8a81039b0 0x7fa8a819c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:24.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.385+0000 7fa8ad000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8a81098f0 0x7fa8a81a37b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34138/0 (socket says 172.21.15.32:34138) 2024-04-30T08:36:24.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.385+0000 7fa8a7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8a81039b0 0x7fa8a819c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:42792/0 (socket says 172.21.15.32:42792) 2024-04-30T08:36:24.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.385+0000 7fa8ad000640 1 -- 172.21.15.32:0/1001603528 learned_addr learned my addr 172.21.15.32:0/1001603528 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:24.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.385+0000 7fa8a7400640 1 --2- 172.21.15.32:0/1001603528 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8a8104bf0 0x7fa8a819c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:24.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.385+0000 7fa8a7400640 1 -- 172.21.15.32:0/1001603528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8a81039b0 msgr2=0x7fa8a819c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:24.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.385+0000 7fa8a7400640 1 --2- 172.21.15.32:0/1001603528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8a81039b0 0x7fa8a819c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.385+0000 7fa8a7400640 1 -- 172.21.15.32:0/1001603528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8a81098f0 msgr2=0x7fa8a81a37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:24.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.385+0000 7fa8a7400640 1 --2- 172.21.15.32:0/1001603528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8a81098f0 0x7fa8a81a37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.385+0000 7fa8a7400640 1 -- 172.21.15.32:0/1001603528 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8a81a3e20 con 0x7fa8a8104bf0 2024-04-30T08:36:24.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.386+0000 7fa8a7400640 1 --2- 172.21.15.32:0/1001603528 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8a8104bf0 0x7fa8a819c820 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa89402a9e0 tx=0x7fa894003d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:24.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.386+0000 7fa8a4c00640 1 -- 172.21.15.32:0/1001603528 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa89403f070 con 0x7fa8a8104bf0 2024-04-30T08:36:24.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.386+0000 7fa8a4c00640 1 -- 172.21.15.32:0/1001603528 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa89403b460 con 0x7fa8a8104bf0 2024-04-30T08:36:24.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.386+0000 7fa8ada00640 1 -- 172.21.15.32:0/1001603528 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8a81a40b0 con 0x7fa8a8104bf0 2024-04-30T08:36:24.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.386+0000 7fa8ada00640 1 -- 172.21.15.32:0/1001603528 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8a81a45c0 con 0x7fa8a8104bf0 2024-04-30T08:36:24.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.386+0000 7fa8a4c00640 1 -- 172.21.15.32:0/1001603528 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa894035030 con 0x7fa8a8104bf0 2024-04-30T08:36:24.388 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.387+0000 7fa8ada00640 1 -- 172.21.15.32:0/1001603528 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa888005290 con 0x7fa8a8104bf0 2024-04-30T08:36:24.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.390+0000 7fa8a4c00640 1 -- 172.21.15.32:0/1001603528 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7fa894042070 con 0x7fa8a8104bf0 2024-04-30T08:36:24.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.392+0000 7fa8a4c00640 1 --2- 172.21.15.32:0/1001603528 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa88c0afbf0 0x7fa88c0b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.392+0000 7fa8a4c00640 1 -- 172.21.15.32:0/1001603528 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fa894100050 con 0x7fa8a8104bf0 2024-04-30T08:36:24.393 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.393+0000 7fa8a7e00640 1 --2- 172.21.15.32:0/1001603528 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa88c0afbf0 0x7fa88c0b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:24.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.394+0000 7fa8a7e00640 1 --2- 172.21.15.32:0/1001603528 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa88c0afbf0 0x7fa88c0b2100 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa8a81045b0 tx=0x7fa89800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:24.394 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.394+0000 7fa8a4c00640 1 -- 172.21.15.32:0/1001603528 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa8940c8070 con 0x7fa8a8104bf0 2024-04-30T08:36:24.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.632+0000 7fa8ada00640 1 -- 172.21.15.32:0/1001603528 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa888005580 con 0x7fa8a8104bf0 2024-04-30T08:36:24.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.633+0000 7fa8a4c00640 1 -- 172.21.15.32:0/1001603528 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11927 (secure 0 0 0) 0x7fa8940cc370 con 0x7fa8a8104bf0 2024-04-30T08:36:24.634 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:36:24.634 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2235842676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2235842676},{"type":"v1","addr":"172.21.15.32:6835","nonce":2235842676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/3577009355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":3577009355},{"type":"v1","addr":"172.21.15.32:6841","nonce":3577009355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-30T08:31:52.236668+0000","modified":"2024-04-30T08:35:05.098418+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24541,"mds_3":24533,"mds_4":24517},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/463203038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":463203038},{"type":"v1","addr":"172.21.15.189:6833","nonce":463203038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6833/3635143591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3635143591},{"type":"v1","addr":"172.21.15.71:6833","nonce":3635143591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6837/2569449994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2569449994},{"type":"v1","addr":"172.21.15.32:6837","nonce":2569449994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/278686565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":278686565},{"type":"v1","addr":"172.21.15.189:6835","nonce":278686565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6835/1558303056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1558303056},{"type":"v1","addr":"172.21.15.71:6835","nonce":1558303056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.189:6837/686520873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":686520873},{"type":"v1","addr":"172.21.15.189:6837","nonce":686520873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.71:6837/2324039932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2324039932},{"type":"v1","addr":"172.21.15.71:6837","nonce":2324039932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6839/352970107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":352970107},{"type":"v1","addr":"172.21.15.189:6839","nonce":352970107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.32:6839/4160386524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4160386524},{"type":"v1","addr":"172.21.15.32:6839","nonce":4160386524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.71:6839/3517409993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3517409993},{"type":"v1","addr":"172.21.15.71:6839","nonce":3517409993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24517,24533,24560,24590,24541]},"id":1}]} 2024-04-30T08:36:24.634 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 28 2024-04-30T08:36:24.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.636+0000 7fa8ada00640 1 -- 172.21.15.32:0/1001603528 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa88c0afbf0 msgr2=0x7fa88c0b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:24.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.636+0000 7fa8ada00640 1 --2- 172.21.15.32:0/1001603528 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa88c0afbf0 0x7fa88c0b2100 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fa8a81045b0 tx=0x7fa89800b390 comp rx=0 tx=0).stop 2024-04-30T08:36:24.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.636+0000 7fa8ada00640 1 -- 172.21.15.32:0/1001603528 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8a8104bf0 msgr2=0x7fa8a819c820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:24.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.636+0000 7fa8ada00640 1 --2- 172.21.15.32:0/1001603528 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8a8104bf0 0x7fa8a819c820 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fa89402a9e0 tx=0x7fa894003d50 comp rx=0 tx=0).stop 2024-04-30T08:36:24.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.636+0000 7fa8ada00640 1 -- 172.21.15.32:0/1001603528 shutdown_connections 2024-04-30T08:36:24.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.636+0000 7fa8ada00640 1 --2- 172.21.15.32:0/1001603528 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa88c0afbf0 0x7fa88c0b2100 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.636+0000 7fa8ada00640 1 --2- 172.21.15.32:0/1001603528 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa8a81098f0 0x7fa8a81a37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.636+0000 7fa8ada00640 1 --2- 172.21.15.32:0/1001603528 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa8a8104bf0 0x7fa8a819c820 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.636+0000 7fa8ada00640 1 --2- 172.21.15.32:0/1001603528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8a81039b0 0x7fa8a819c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.636+0000 7fa8ada00640 1 -- 172.21.15.32:0/1001603528 >> 172.21.15.32:0/1001603528 conn(0x7fa8a80feba0 msgr2=0x7fa8a80fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:24.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.636+0000 7fa8ada00640 1 -- 172.21.15.32:0/1001603528 shutdown_connections 2024-04-30T08:36:24.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.636+0000 7fa8ada00640 1 -- 172.21.15.32:0/1001603528 wait complete. 2024-04-30T08:36:24.661 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-30T08:36:24.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:24 smithi032 ceph-mon[28331]: pgmap v462: 129 pgs: 129 active+clean; 674 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-30T08:36:24.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:24 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1001603528' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:36:24.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.943+0000 7ffa33200640 1 -- 172.21.15.32:0/410493341 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa2c1089a0 msgr2=0x7ffa2c10adc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:24.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.943+0000 7ffa33200640 1 --2- 172.21.15.32:0/410493341 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa2c1089a0 0x7ffa2c10adc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ffa24009ca0 tx=0x7ffa2402a8c0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.943+0000 7ffa33200640 1 -- 172.21.15.32:0/410493341 shutdown_connections 2024-04-30T08:36:24.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.943+0000 7ffa33200640 1 --2- 172.21.15.32:0/410493341 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa2c1089a0 0x7ffa2c10adc0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.943+0000 7ffa33200640 1 --2- 172.21.15.32:0/410493341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa2c103ca0 0x7ffa2c1040b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.943+0000 7ffa33200640 1 --2- 172.21.15.32:0/410493341 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa2c102a60 0x7ffa2c103760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.943+0000 7ffa33200640 1 -- 172.21.15.32:0/410493341 >> 172.21.15.32:0/410493341 conn(0x7ffa2c0fdc50 msgr2=0x7ffa2c100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:24.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.943+0000 7ffa33200640 1 -- 172.21.15.32:0/410493341 shutdown_connections 2024-04-30T08:36:24.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.943+0000 7ffa33200640 1 -- 172.21.15.32:0/410493341 wait complete. 2024-04-30T08:36:24.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.944+0000 7ffa33200640 1 Processor -- start 2024-04-30T08:36:24.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.944+0000 7ffa33200640 1 -- start start 2024-04-30T08:36:24.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.945+0000 7ffa33200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa2c102a60 0x7ffa2c19a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.945+0000 7ffa33200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa2c103ca0 0x7ffa2c19aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.948 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.945+0000 7ffa33200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa2c1089a0 0x7ffa2c1a1a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.945+0000 7ffa33200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2c10f680 con 0x7ffa2c102a60 2024-04-30T08:36:24.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.945+0000 7ffa33200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2c10f500 con 0x7ffa2c103ca0 2024-04-30T08:36:24.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.945+0000 7ffa33200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2c10f800 con 0x7ffa2c1089a0 2024-04-30T08:36:24.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.945+0000 7ffa31200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa2c1089a0 0x7ffa2c1a1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:24.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.945+0000 7ffa2be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa2c102a60 0x7ffa2c19a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:24.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.945+0000 7ffa2be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa2c102a60 0x7ffa2c19a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34156/0 (socket says 172.21.15.32:34156) 2024-04-30T08:36:24.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.945+0000 7ffa2be00640 1 -- 172.21.15.32:0/826293175 learned_addr learned my addr 172.21.15.32:0/826293175 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:24.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.946+0000 7ffa2b400640 1 --2- 172.21.15.32:0/826293175 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa2c103ca0 0x7ffa2c19aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:24.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.946+0000 7ffa2b400640 1 -- 172.21.15.32:0/826293175 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa2c1089a0 msgr2=0x7ffa2c1a1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:24.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.946+0000 7ffa2b400640 1 --2- 172.21.15.32:0/826293175 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa2c1089a0 0x7ffa2c1a1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.946+0000 7ffa2b400640 1 -- 172.21.15.32:0/826293175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa2c102a60 msgr2=0x7ffa2c19a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:24.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.946+0000 7ffa2b400640 1 --2- 172.21.15.32:0/826293175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa2c102a60 0x7ffa2c19a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.946+0000 7ffa2b400640 1 -- 172.21.15.32:0/826293175 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa2c10b730 con 0x7ffa2c103ca0 2024-04-30T08:36:24.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.947+0000 7ffa2b400640 1 --2- 172.21.15.32:0/826293175 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa2c103ca0 0x7ffa2c19aab0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ffa1800e9f0 tx=0x7ffa1800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:24.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.947+0000 7ffa28c00640 1 -- 172.21.15.32:0/826293175 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1800ccc0 con 0x7ffa2c103ca0 2024-04-30T08:36:24.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.947+0000 7ffa28c00640 1 -- 172.21.15.32:0/826293175 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa18005020 con 0x7ffa2c103ca0 2024-04-30T08:36:24.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.947+0000 7ffa28c00640 1 -- 172.21.15.32:0/826293175 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa18004a00 con 0x7ffa2c103ca0 2024-04-30T08:36:24.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.947+0000 7ffa33200640 1 -- 172.21.15.32:0/826293175 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa2c10ba20 con 0x7ffa2c103ca0 2024-04-30T08:36:24.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.947+0000 7ffa33200640 1 -- 172.21.15.32:0/826293175 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa2c10bf70 con 0x7ffa2c103ca0 2024-04-30T08:36:24.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.951+0000 7ffa28c00640 1 -- 172.21.15.32:0/826293175 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7ffa18004ba0 con 0x7ffa2c103ca0 2024-04-30T08:36:24.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.952+0000 7ffa3a36d740 1 Processor -- start 2024-04-30T08:36:24.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.952+0000 7ffa3a36d740 1 -- start start 2024-04-30T08:36:24.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.953+0000 7ffa3a36d740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55feeb862390 0x55feebc0bd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.953+0000 7ffa3a36d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feeb8394d0 0x55feebc0c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.953+0000 7ffa3a36d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feeb8416b0 0x55feebc13200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.953+0000 7ffa3a36d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55feebb68d50 con 0x55feeb8394d0 2024-04-30T08:36:24.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.953+0000 7ffa3a36d740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55feebb0e0c0 con 0x55feeb862390 2024-04-30T08:36:24.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.953+0000 7ffa3a36d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55feebb64d10 con 0x55feeb8416b0 2024-04-30T08:36:24.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.953+0000 7ffa28c00640 1 --2- 172.21.15.32:0/826293175 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ffa100afc00 0x7ffa100b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.954+0000 7ffa2b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feeb8416b0 0x55feebc13200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:24.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.954+0000 7ffa31200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55feeb862390 0x55feebc0bd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:24.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.954+0000 7ffa28c00640 1 -- 172.21.15.32:0/826293175 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7ffa180df120 con 0x7ffa2c103ca0 2024-04-30T08:36:24.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.954+0000 7ffa31200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55feeb862390 0x55feebc0bd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:60680/0 (socket says 172.21.15.32:60680) 2024-04-30T08:36:24.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.954+0000 7ffa31200640 1 -- 172.21.15.32:0/3227665100 learned_addr learned my addr 172.21.15.32:0/3227665100 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:24.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.954+0000 7ffa2b400640 1 --2- 172.21.15.32:0/3227665100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feeb8416b0 0x55feebc13200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:42822/0 (socket says 172.21.15.32:42822) 2024-04-30T08:36:24.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.954+0000 7ffa2be00640 1 --2- 172.21.15.32:0/3227665100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feeb8394d0 0x55feebc0c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:24.959 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.954+0000 7ffa31200640 1 --2- 172.21.15.32:0/826293175 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ffa100afc00 0x7ffa100b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:24.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.954+0000 7ffa31200640 1 -- 172.21.15.32:0/3227665100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feeb8416b0 msgr2=0x55feebc13200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:24.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.954+0000 7ffa31200640 1 --2- 172.21.15.32:0/3227665100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feeb8416b0 0x55feebc13200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.954+0000 7ffa31200640 1 -- 172.21.15.32:0/3227665100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feeb8394d0 msgr2=0x55feebc0c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:24.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.954+0000 7ffa31200640 1 --2- 172.21.15.32:0/3227665100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feeb8394d0 0x55feebc0c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.954+0000 7ffa31200640 1 -- 172.21.15.32:0/3227665100 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55feeb87a3f0 con 0x55feeb862390 2024-04-30T08:36:24.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.955+0000 7ffa2be00640 1 --2- 172.21.15.32:0/3227665100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feeb8394d0 0x55feebc0c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:36:24.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.955+0000 7ffa31200640 1 --2- 172.21.15.32:0/3227665100 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55feeb862390 0x55feebc0bd80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ffa24035000 tx=0x7ffa24032c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=871e1db7f7ec1554 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:24.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.955+0000 7ffa31200640 1 --2- 172.21.15.32:0/826293175 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ffa100afc00 0x7ffa100b2110 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ffa24041000 tx=0x7ffa24049000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:24.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.956+0000 7ffa21600640 1 -- 172.21.15.32:0/3227665100 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa24003a30 con 0x55feeb862390 2024-04-30T08:36:24.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.956+0000 7ffa21600640 1 -- 172.21.15.32:0/3227665100 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa2404d020 con 0x55feeb862390 2024-04-30T08:36:24.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.956+0000 7ffa21600640 1 -- 172.21.15.32:0/3227665100 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa2402f660 con 0x55feeb862390 2024-04-30T08:36:24.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.957+0000 7ffa3a36d740 1 -- 172.21.15.32:0/3227665100 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55feeb862390 msgr2=0x55feebc0bd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:24.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.957+0000 7ffa3a36d740 1 --2- 172.21.15.32:0/3227665100 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55feeb862390 0x55feebc0bd80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ffa24035000 tx=0x7ffa24032c80 comp rx=0 tx=0).stop 2024-04-30T08:36:24.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.957+0000 7ffa3a36d740 1 -- 172.21.15.32:0/3227665100 shutdown_connections 2024-04-30T08:36:24.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.957+0000 7ffa3a36d740 1 --2- 172.21.15.32:0/3227665100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feeb8416b0 0x55feebc13200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.957+0000 7ffa3a36d740 1 --2- 172.21.15.32:0/3227665100 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feeb8394d0 0x55feebc0c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.957+0000 7ffa3a36d740 1 --2- 172.21.15.32:0/3227665100 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55feeb862390 0x55feebc0bd80 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.957+0000 7ffa3a36d740 1 -- 172.21.15.32:0/3227665100 >> 172.21.15.32:0/3227665100 conn(0x55feeb8564e0 msgr2=0x55feebbfef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:24.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.957+0000 7ffa3a36d740 1 -- 172.21.15.32:0/3227665100 shutdown_connections 2024-04-30T08:36:24.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.957+0000 7ffa3a36d740 1 -- 172.21.15.32:0/3227665100 wait complete. 2024-04-30T08:36:24.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.958+0000 7ffa3a36d740 1 -- start start 2024-04-30T08:36:24.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.959+0000 7ffa3a36d740 1 Processor -- start 2024-04-30T08:36:24.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.959+0000 7ffa3a36d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feeb8394d0 0x55feebc15b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.959+0000 7ffa3a36d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feeb8416b0 0x55feebc160c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.960+0000 7ffa3a36d740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55feeb862390 0x55feebc1d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.960+0000 7ffa3a36d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55feebc13a70 con 0x55feeb8416b0 2024-04-30T08:36:24.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.960+0000 7ffa3a36d740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55feebb21b50 con 0x55feeb862390 2024-04-30T08:36:24.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.960+0000 7ffa3a36d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55feebc13bf0 con 0x55feeb8394d0 2024-04-30T08:36:24.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.960+0000 7ffa2be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feeb8394d0 0x55feebc15b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:24.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.960+0000 7ffa2be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feeb8394d0 0x55feebc15b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:42824/0 (socket says 172.21.15.32:42824) 2024-04-30T08:36:24.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.960+0000 7ffa2be00640 1 -- 172.21.15.32:0/1146599264 learned_addr learned my addr 172.21.15.32:0/1146599264 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:24.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.960+0000 7ffa3a36d740 10 client.0 _ll_register_callbacks cb 0x55feebaeb420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:36:24.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.960+0000 7ffa2b400640 1 --2- 172.21.15.32:0/1146599264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feeb8416b0 0x55feebc160c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:24.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.960+0000 7ffa31200640 1 --2- 172.21.15.32:0/1146599264 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55feeb862390 0x55feebc1d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:24.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.960+0000 7ffa2be00640 1 -- 172.21.15.32:0/1146599264 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55feeb862390 msgr2=0x55feebc1d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:24.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.960+0000 7ffa2be00640 1 --2- 172.21.15.32:0/1146599264 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55feeb862390 0x55feebc1d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.960+0000 7ffa2be00640 1 -- 172.21.15.32:0/1146599264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feeb8416b0 msgr2=0x55feebc160c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:24.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.960+0000 7ffa2be00640 1 --2- 172.21.15.32:0/1146599264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feeb8416b0 0x55feebc160c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.961+0000 7ffa2be00640 1 -- 172.21.15.32:0/1146599264 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55feeb9447d0 con 0x55feeb8394d0 2024-04-30T08:36:24.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.961+0000 7ffa31200640 1 --2- 172.21.15.32:0/1146599264 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55feeb862390 0x55feebc1d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-30T08:36:24.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.961+0000 7ffa2b400640 1 --2- 172.21.15.32:0/1146599264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feeb8416b0 0x55feebc160c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:36:24.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.961+0000 7ffa2be00640 1 --2- 172.21.15.32:0/1146599264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feeb8394d0 0x55feebc15b80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c012700 tx=0x7ffa1c012bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c40d3d5cf23dad79 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:24.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.961+0000 7ffa16000640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-30T08:36:24.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.961+0000 7ffa16000640 1 -- 172.21.15.32:0/1146599264 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c004030 con 0x55feeb8394d0 2024-04-30T08:36:24.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.962+0000 7ffa3a36d740 1 -- 172.21.15.32:0/1146599264 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa2404d020 con 0x55feeb8394d0 2024-04-30T08:36:24.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.962+0000 7ffa16000640 1 -- 172.21.15.32:0/1146599264 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7ffa1c0041d0 con 0x55feeb8394d0 2024-04-30T08:36:24.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.962+0000 7ffa16000640 1 -- 172.21.15.32:0/1146599264 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ffa1c025050 con 0x55feeb8394d0 2024-04-30T08:36:24.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.962+0000 7ffa3a36d740 10 client.25903 fetch_fsmap learned FSMap version 28 2024-04-30T08:36:24.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.962+0000 7ffa3a36d740 1 -- 172.21.15.32:0/1146599264 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55feebc20030 con 0x55feeb8394d0 2024-04-30T08:36:24.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.962+0000 7ffa16000640 1 -- 172.21.15.32:0/1146599264 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa2404d020 con 0x55feeb8394d0 2024-04-30T08:36:24.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.963+0000 7ffa16000640 1 -- 172.21.15.32:0/1146599264 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ffa1c0038a0 con 0x55feeb8394d0 2024-04-30T08:36:24.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.963+0000 7ffa3a36d740 10 client.25903 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:36:24.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.963+0000 7ffa3a36d740 10 client.25903 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-30T08:36:24.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.963+0000 7ffa3a36d740 20 client.25903 populate_metadata read hostname 'smithi032' 2024-04-30T08:36:24.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.963+0000 7ffa3a36d740 1 --2- 172.21.15.32:0/1146599264 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x55feeb820f60 0x55feebc23700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.963+0000 7ffa3a36d740 4 client.25903 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:36:24.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.963+0000 7ffa3a36d740 1 -- 172.21.15.32:0/1146599264 --> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55feebb76960 con 0x55feeb820f60 2024-04-30T08:36:24.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.963+0000 7ffa2b400640 1 --2- 172.21.15.32:0/1146599264 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x55feeb820f60 0x55feebc23700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:24.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.964+0000 7ffa2b400640 1 --2- 172.21.15.32:0/1146599264 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x55feeb820f60 0x55feebc23700 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=64ddb9631df6ef53 server_cookie=8fb5273e3d7470df in_seq=0 out_seq=0 2024-04-30T08:36:24.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.964+0000 7ffa16000640 10 client.25903 ms_handle_connect on v2:172.21.15.189:6836/686520873 2024-04-30T08:36:24.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.965+0000 7ffa16000640 1 -- 172.21.15.32:0/1146599264 <== mds.4 v2:172.21.15.189:6836/686520873 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55feebb76960 con 0x55feeb820f60 2024-04-30T08:36:24.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.965+0000 7ffa16000640 10 client.25903 handle_command_reply: tid=0 2024-04-30T08:36:24.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.965+0000 7ffa3a36d740 1 client.25903 shutdown 2024-04-30T08:36:24.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.965+0000 7ffa3a36d740 1 -- 172.21.15.32:0/1146599264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feeb8394d0 msgr2=0x55feebc15b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:24.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.965+0000 7ffa3a36d740 1 --2- 172.21.15.32:0/1146599264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feeb8394d0 0x55feebc15b80 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c012700 tx=0x7ffa1c012bc0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.965+0000 7ffa3a36d740 1 -- 172.21.15.32:0/1146599264 shutdown_connections 2024-04-30T08:36:24.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.965+0000 7ffa3a36d740 1 --2- 172.21.15.32:0/1146599264 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x55feeb820f60 0x55feebc23700 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.965+0000 7ffa3a36d740 1 --2- 172.21.15.32:0/1146599264 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55feeb862390 0x55feebc1d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.965+0000 7ffa3a36d740 1 --2- 172.21.15.32:0/1146599264 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feeb8416b0 0x55feebc160c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.965+0000 7ffa3a36d740 1 --2- 172.21.15.32:0/1146599264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feeb8394d0 0x55feebc15b80 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.965+0000 7ffa3a36d740 1 -- 172.21.15.32:0/1146599264 >> 172.21.15.32:0/1146599264 conn(0x55feeb8564e0 msgr2=0x55feebbfdf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:24.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.966+0000 7ffa16000640 0 client.25903 ms_handle_reset on v2:172.21.15.189:6836/686520873 2024-04-30T08:36:24.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.966+0000 7ffa3a36d740 1 -- 172.21.15.32:0/1146599264 shutdown_connections 2024-04-30T08:36:24.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.966+0000 7ffa3a36d740 1 -- 172.21.15.32:0/1146599264 wait complete. 2024-04-30T08:36:24.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.966+0000 7ffa3a36d740 20 client.25903 trim_cache size 0 max 16384 2024-04-30T08:36:24.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.995+0000 7ffa3a36d740 1 Processor -- start 2024-04-30T08:36:24.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.996+0000 7ffa3a36d740 1 -- start start 2024-04-30T08:36:24.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.996+0000 7ffa3a36d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feeb8416b0 0x55feebbff070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.996+0000 7ffa3a36d740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55feeb862390 0x55feebc1cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.996+0000 7ffa3a36d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feeb7f63a0 0x55feebc1d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:24.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.996+0000 7ffa3a36d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2c1aa950 con 0x55feeb8416b0 2024-04-30T08:36:24.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.996+0000 7ffa3a36d740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2c1ac490 con 0x55feeb862390 2024-04-30T08:36:24.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.996+0000 7ffa3a36d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2c1ab020 con 0x55feeb7f63a0 2024-04-30T08:36:24.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.997+0000 7ffa2be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feeb8416b0 0x55feebbff070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:24.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.997+0000 7ffa2be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feeb8416b0 0x55feebbff070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34174/0 (socket says 172.21.15.32:34174) 2024-04-30T08:36:24.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.997+0000 7ffa2be00640 1 -- 172.21.15.32:0/3227367416 learned_addr learned my addr 172.21.15.32:0/3227367416 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:24.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.997+0000 7ffa31200640 1 --2- 172.21.15.32:0/3227367416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feeb7f63a0 0x55feebc1d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:24.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.997+0000 7ffa2b400640 1 --2- 172.21.15.32:0/3227367416 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55feeb862390 0x55feebc1cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:24.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.997+0000 7ffa2be00640 1 -- 172.21.15.32:0/3227367416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feeb7f63a0 msgr2=0x55feebc1d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:24.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.997+0000 7ffa2be00640 1 --2- 172.21.15.32:0/3227367416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feeb7f63a0 0x55feebc1d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.997+0000 7ffa2be00640 1 -- 172.21.15.32:0/3227367416 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55feeb862390 msgr2=0x55feebc1cf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:24.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.997+0000 7ffa2be00640 1 --2- 172.21.15.32:0/3227367416 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55feeb862390 0x55feebc1cf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:24.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.998+0000 7ffa2be00640 1 -- 172.21.15.32:0/3227367416 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55feebc13f90 con 0x55feeb8416b0 2024-04-30T08:36:24.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.998+0000 7ffa31200640 1 --2- 172.21.15.32:0/3227367416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feeb7f63a0 0x55feebc1d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:36:24.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.998+0000 7ffa2be00640 1 --2- 172.21.15.32:0/3227367416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feeb8416b0 0x55feebbff070 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c003cf0 tx=0x7ffa1c0096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=160d0d780daaa4f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:25.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.998+0000 7ffa20c00640 1 -- 172.21.15.32:0/3227367416 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c022040 con 0x55feeb8416b0 2024-04-30T08:36:25.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.999+0000 7ffa20c00640 1 -- 172.21.15.32:0/3227367416 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa1c00f920 con 0x55feeb8416b0 2024-04-30T08:36:25.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:24.999+0000 7ffa20c00640 1 -- 172.21.15.32:0/3227367416 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c005550 con 0x55feeb8416b0 2024-04-30T08:36:25.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.000+0000 7ffa3a36d740 1 -- 172.21.15.32:0/3227367416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feeb8416b0 msgr2=0x55feebbff070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:25.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.000+0000 7ffa3a36d740 1 --2- 172.21.15.32:0/3227367416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feeb8416b0 0x55feebbff070 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c003cf0 tx=0x7ffa1c0096c0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.000+0000 7ffa3a36d740 1 -- 172.21.15.32:0/3227367416 shutdown_connections 2024-04-30T08:36:25.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.000+0000 7ffa3a36d740 1 --2- 172.21.15.32:0/3227367416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feeb7f63a0 0x55feebc1d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.000+0000 7ffa3a36d740 1 --2- 172.21.15.32:0/3227367416 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55feeb862390 0x55feebc1cf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.000+0000 7ffa3a36d740 1 --2- 172.21.15.32:0/3227367416 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feeb8416b0 0x55feebbff070 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.000+0000 7ffa3a36d740 1 -- 172.21.15.32:0/3227367416 >> 172.21.15.32:0/3227367416 conn(0x55feeb8564e0 msgr2=0x55feebbfe960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:25.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.000+0000 7ffa3a36d740 1 -- 172.21.15.32:0/3227367416 shutdown_connections 2024-04-30T08:36:25.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.000+0000 7ffa3a36d740 1 -- 172.21.15.32:0/3227367416 wait complete. 2024-04-30T08:36:25.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.001+0000 7ffa3a36d740 1 -- start start 2024-04-30T08:36:25.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.001+0000 7ffa3a36d740 1 Processor -- start 2024-04-30T08:36:25.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.001+0000 7ffa3a36d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feeb7f63a0 0x55feebc1a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.001+0000 7ffa3a36d740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55feeb8416b0 0x55feebc1aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.001+0000 7ffa3a36d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feeb862390 0x55feebc89ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.001+0000 7ffa3a36d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2c1abdc0 con 0x55feeb7f63a0 2024-04-30T08:36:25.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.001+0000 7ffa3a36d740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2c1ab6f0 con 0x55feeb8416b0 2024-04-30T08:36:25.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.001+0000 7ffa3a36d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2c1a8910 con 0x55feeb862390 2024-04-30T08:36:25.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.002+0000 7ffa3a36d740 10 client.0 _ll_register_callbacks cb 0x7ffa1c0098e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:36:25.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.002+0000 7ffa2b400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55feeb8416b0 0x55feebc1aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.002+0000 7ffa2b400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55feeb8416b0 0x55feebc1aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:60722/0 (socket says 172.21.15.32:60722) 2024-04-30T08:36:25.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.002+0000 7ffa2b400640 1 -- 172.21.15.32:0/1973955330 learned_addr learned my addr 172.21.15.32:0/1973955330 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:25.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.002+0000 7ffa31200640 1 --2- 172.21.15.32:0/1973955330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feeb862390 0x55feebc89ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.002+0000 7ffa2b400640 1 -- 172.21.15.32:0/1973955330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feeb862390 msgr2=0x55feebc89ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:25.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.002+0000 7ffa2b400640 1 --2- 172.21.15.32:0/1973955330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feeb862390 0x55feebc89ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.002+0000 7ffa2b400640 1 -- 172.21.15.32:0/1973955330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feeb7f63a0 msgr2=0x55feebc1a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:25.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.002+0000 7ffa2be00640 1 --2- 172.21.15.32:0/1973955330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feeb7f63a0 0x55feebc1a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.002+0000 7ffa2b400640 1 --2- 172.21.15.32:0/1973955330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feeb7f63a0 0x55feebc1a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.002+0000 7ffa2b400640 1 -- 172.21.15.32:0/1973955330 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55feebc10c10 con 0x55feeb8416b0 2024-04-30T08:36:25.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.003+0000 7ffa2be00640 1 --2- 172.21.15.32:0/1973955330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feeb7f63a0 0x55feebc1a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-30T08:36:25.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.003+0000 7ffa2b400640 1 --2- 172.21.15.32:0/1973955330 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55feeb8416b0 0x55feebc1aad0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ffa180a4300 tx=0x7ffa18014820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a5902da123094a8e server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:25.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.003+0000 7ffa16000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:36:25.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.003+0000 7ffa16000640 1 -- 172.21.15.32:0/1973955330 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa180a7790 con 0x55feeb8416b0 2024-04-30T08:36:25.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.003+0000 7ffa16000640 1 -- 172.21.15.32:0/1973955330 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7ffa180a3680 con 0x55feeb8416b0 2024-04-30T08:36:25.005 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.003+0000 7ffa3a36d740 1 -- 172.21.15.32:0/1973955330 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa1c00f920 con 0x55feeb8416b0 2024-04-30T08:36:25.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.003+0000 7ffa16000640 1 -- 172.21.15.32:0/1973955330 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ffa180a7090 con 0x55feeb8416b0 2024-04-30T08:36:25.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.003+0000 7ffa3a36d740 10 client.25907 fetch_fsmap learned FSMap version 28 2024-04-30T08:36:25.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.003+0000 7ffa3a36d740 1 -- 172.21.15.32:0/1973955330 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55feebc8ae70 con 0x55feeb8416b0 2024-04-30T08:36:25.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.004+0000 7ffa16000640 1 -- 172.21.15.32:0/1973955330 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffa180ab440 con 0x55feeb8416b0 2024-04-30T08:36:25.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.004+0000 7ffa16000640 1 -- 172.21.15.32:0/1973955330 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ffa180dc910 con 0x55feeb8416b0 2024-04-30T08:36:25.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.004+0000 7ffa3a36d740 10 client.25907 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:36:25.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.004+0000 7ffa3a36d740 10 client.25907 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-30T08:36:25.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.004+0000 7ffa3a36d740 20 client.25907 populate_metadata read hostname 'smithi032' 2024-04-30T08:36:25.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.004+0000 7ffa3a36d740 1 --2- 172.21.15.32:0/1973955330 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x55feebc8b780 0x55feebc8dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.004+0000 7ffa3a36d740 4 client.25907 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-30T08:36:25.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.004+0000 7ffa3a36d740 1 -- 172.21.15.32:0/1973955330 --> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7ffa1c00f920 con 0x55feebc8b780 2024-04-30T08:36:25.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.004+0000 7ffa2be00640 1 --2- 172.21.15.32:0/1973955330 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x55feebc8b780 0x55feebc8dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.005+0000 7ffa2be00640 1 --2- 172.21.15.32:0/1973955330 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x55feebc8b780 0x55feebc8dba0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=17ea4fae5081ae20 server_cookie=5356dd020a3544a9 in_seq=0 out_seq=0 2024-04-30T08:36:25.007 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.005+0000 7ffa16000640 10 client.25907 ms_handle_connect on v2:172.21.15.189:6836/686520873 2024-04-30T08:36:25.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.006+0000 7ffa16000640 1 -- 172.21.15.32:0/1973955330 <== mds.4 v2:172.21.15.189:6836/686520873 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7ffa1c00f920 con 0x55feebc8b780 2024-04-30T08:36:25.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.006+0000 7ffa16000640 10 client.25907 handle_command_reply: tid=0 2024-04-30T08:36:25.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.006+0000 7ffa3a36d740 1 client.25907 shutdown 2024-04-30T08:36:25.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.006+0000 7ffa3a36d740 1 -- 172.21.15.32:0/1973955330 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55feeb8416b0 msgr2=0x55feebc1aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:25.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.006+0000 7ffa3a36d740 1 --2- 172.21.15.32:0/1973955330 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55feeb8416b0 0x55feebc1aad0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ffa180a4300 tx=0x7ffa18014820 comp rx=0 tx=0).stop 2024-04-30T08:36:25.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.006+0000 7ffa3a36d740 1 -- 172.21.15.32:0/1973955330 shutdown_connections 2024-04-30T08:36:25.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.006+0000 7ffa3a36d740 1 --2- 172.21.15.32:0/1973955330 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x55feebc8b780 0x55feebc8dba0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.006+0000 7ffa3a36d740 1 --2- 172.21.15.32:0/1973955330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55feeb862390 0x55feebc89ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.006+0000 7ffa3a36d740 1 --2- 172.21.15.32:0/1973955330 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55feeb8416b0 0x55feebc1aad0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.006+0000 7ffa3a36d740 1 --2- 172.21.15.32:0/1973955330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55feeb7f63a0 0x55feebc1a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.006+0000 7ffa16000640 0 client.25907 ms_handle_reset on v2:172.21.15.189:6836/686520873 2024-04-30T08:36:25.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.006+0000 7ffa3a36d740 1 -- 172.21.15.32:0/1973955330 >> 172.21.15.32:0/1973955330 conn(0x55feeb8564e0 msgr2=0x55feebbfd510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:25.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.006+0000 7ffa3a36d740 1 -- 172.21.15.32:0/1973955330 shutdown_connections 2024-04-30T08:36:25.008 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.006+0000 7ffa3a36d740 1 -- 172.21.15.32:0/1973955330 wait complete. 2024-04-30T08:36:25.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.006+0000 7ffa3a36d740 20 client.25907 trim_cache size 0 max 16384 2024-04-30T08:36:25.009 INFO:teuthology.orchestra.run.smithi032.stdout:[] 2024-04-30T08:36:25.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.007+0000 7ffa33200640 1 -- 172.21.15.32:0/826293175 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ffa100afc00 msgr2=0x7ffa100b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:25.009 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.007+0000 7ffa33200640 1 --2- 172.21.15.32:0/826293175 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ffa100afc00 0x7ffa100b2110 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7ffa24041000 tx=0x7ffa24049000 comp rx=0 tx=0).stop 2024-04-30T08:36:25.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.007+0000 7ffa33200640 1 -- 172.21.15.32:0/826293175 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa2c103ca0 msgr2=0x7ffa2c19aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:25.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.007+0000 7ffa33200640 1 --2- 172.21.15.32:0/826293175 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa2c103ca0 0x7ffa2c19aab0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ffa1800e9f0 tx=0x7ffa1800eeb0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.007+0000 7ffa33200640 1 -- 172.21.15.32:0/826293175 shutdown_connections 2024-04-30T08:36:25.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.007+0000 7ffa33200640 1 --2- 172.21.15.32:0/826293175 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ffa100afc00 0x7ffa100b2110 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.007+0000 7ffa33200640 1 --2- 172.21.15.32:0/826293175 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffa2c1089a0 0x7ffa2c1a1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.007+0000 7ffa33200640 1 --2- 172.21.15.32:0/826293175 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffa2c103ca0 0x7ffa2c19aab0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.007+0000 7ffa33200640 1 --2- 172.21.15.32:0/826293175 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffa2c102a60 0x7ffa2c19a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.007+0000 7ffa33200640 1 -- 172.21.15.32:0/826293175 >> 172.21.15.32:0/826293175 conn(0x7ffa2c0fdc50 msgr2=0x7ffa2c104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:25.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.008+0000 7ffa33200640 1 -- 172.21.15.32:0/826293175 shutdown_connections 2024-04-30T08:36:25.010 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.008+0000 7ffa33200640 1 -- 172.21.15.32:0/826293175 wait complete. 2024-04-30T08:36:25.031 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-30T08:36:25.040 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.039+0000 7f285c400640 1 -- 172.21.15.32:0/2714034521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2854109950 msgr2=0x7f285410bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:25.040 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.039+0000 7f285c400640 1 --2- 172.21.15.32:0/2714034521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2854109950 0x7f285410bd70 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f284c009be0 tx=0x7f284c02a6c0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.040 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.039+0000 7f285c400640 1 -- 172.21.15.32:0/2714034521 shutdown_connections 2024-04-30T08:36:25.040 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.039+0000 7f285c400640 1 --2- 172.21.15.32:0/2714034521 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2854109950 0x7f285410bd70 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.040 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.040+0000 7f285c400640 1 --2- 172.21.15.32:0/2714034521 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2854106ff0 0x7f2854109410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.040 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.040+0000 7f285c400640 1 --2- 172.21.15.32:0/2714034521 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2854104690 0x7f2854106ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.040 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.040+0000 7f285c400640 1 -- 172.21.15.32:0/2714034521 >> 172.21.15.32:0/2714034521 conn(0x7f28540fdc90 msgr2=0x7f2854100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:25.040 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.040+0000 7f285c400640 1 -- 172.21.15.32:0/2714034521 shutdown_connections 2024-04-30T08:36:25.041 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.040+0000 7f285c400640 1 -- 172.21.15.32:0/2714034521 wait complete. 2024-04-30T08:36:25.041 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.041+0000 7f285c400640 1 Processor -- start 2024-04-30T08:36:25.041 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.041+0000 7f285c400640 1 -- start start 2024-04-30T08:36:25.042 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.042+0000 7f285c400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2854104690 0x7f285419c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.042 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.042+0000 7f285c400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2854106ff0 0x7f285419c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.042 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.042+0000 7f285c400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2854109950 0x7f28541a3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.042 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.042+0000 7f285c400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f285410f6b0 con 0x7f2854109950 2024-04-30T08:36:25.043 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.042+0000 7f285c400640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f285410f530 con 0x7f2854104690 2024-04-30T08:36:25.043 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.042+0000 7f285c400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f285410f830 con 0x7f2854106ff0 2024-04-30T08:36:25.043 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.042+0000 7f2859000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2854106ff0 0x7f285419c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.043 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.042+0000 7f2859000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2854106ff0 0x7f285419c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:42862/0 (socket says 172.21.15.32:42862) 2024-04-30T08:36:25.044 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.042+0000 7f2859000640 1 -- 172.21.15.32:0/2432228069 learned_addr learned my addr 172.21.15.32:0/2432228069 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:25.044 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.042+0000 7f2859a00640 1 --2- 172.21.15.32:0/2432228069 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2854104690 0x7f285419c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.044 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.042+0000 7f285a400640 1 --2- 172.21.15.32:0/2432228069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2854109950 0x7f28541a3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.044 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.043+0000 7f2859a00640 1 -- 172.21.15.32:0/2432228069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2854106ff0 msgr2=0x7f285419c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:25.044 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.043+0000 7f2859a00640 1 --2- 172.21.15.32:0/2432228069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2854106ff0 0x7f285419c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.044 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.043+0000 7f2859a00640 1 -- 172.21.15.32:0/2432228069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2854109950 msgr2=0x7f28541a3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:25.044 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.043+0000 7f2859a00640 1 --2- 172.21.15.32:0/2432228069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2854109950 0x7f28541a3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.044 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.043+0000 7f2859a00640 1 -- 172.21.15.32:0/2432228069 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28541a3fc0 con 0x7f2854104690 2024-04-30T08:36:25.044 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.043+0000 7f285a400640 1 --2- 172.21.15.32:0/2432228069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2854109950 0x7f28541a3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:36:25.044 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.043+0000 7f2859a00640 1 --2- 172.21.15.32:0/2432228069 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2854104690 0x7f285419c480 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2848005fd0 tx=0x7f284800ec20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:25.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.043+0000 7f2852000640 1 -- 172.21.15.32:0/2432228069 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f284800ccc0 con 0x7f2854104690 2024-04-30T08:36:25.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.044+0000 7f2852000640 1 -- 172.21.15.32:0/2432228069 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f284800ee70 con 0x7f2854104690 2024-04-30T08:36:25.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.044+0000 7f285c400640 1 -- 172.21.15.32:0/2432228069 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28541a42b0 con 0x7f2854104690 2024-04-30T08:36:25.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.044+0000 7f2852000640 1 -- 172.21.15.32:0/2432228069 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2848018750 con 0x7f2854104690 2024-04-30T08:36:25.045 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.044+0000 7f285c400640 1 -- 172.21.15.32:0/2432228069 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28541a4850 con 0x7f2854104690 2024-04-30T08:36:25.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.048+0000 7f2863453740 1 Processor -- start 2024-04-30T08:36:25.048 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.048+0000 7f2863453740 1 -- start start 2024-04-30T08:36:25.049 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.049+0000 7f2863453740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55de12f5e1b0 0x55de133000e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.049+0000 7f2863453740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55de12f352f0 0x55de13300660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.049+0000 7f2863453740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de12f3d4d0 0x55de13307560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.049+0000 7f2863453740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55de13264550 con 0x55de12f352f0 2024-04-30T08:36:25.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.049+0000 7f2863453740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55de13209bc0 con 0x55de12f5e1b0 2024-04-30T08:36:25.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.049+0000 7f2863453740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55de13260510 con 0x55de12f3d4d0 2024-04-30T08:36:25.050 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.050+0000 7f2859000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de12f3d4d0 0x55de13307560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.050+0000 7f2859000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de12f3d4d0 0x55de13307560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:42876/0 (socket says 172.21.15.32:42876) 2024-04-30T08:36:25.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.050+0000 7f2859000640 1 -- 172.21.15.32:0/596272988 learned_addr learned my addr 172.21.15.32:0/596272988 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:25.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.050+0000 7f285a400640 1 --2- 172.21.15.32:0/596272988 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55de12f5e1b0 0x55de133000e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.050+0000 7f2852000640 1 -- 172.21.15.32:0/2432228069 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f28480188f0 con 0x7f2854104690 2024-04-30T08:36:25.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.050+0000 7f285a400640 1 -- 172.21.15.32:0/596272988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de12f3d4d0 msgr2=0x55de13307560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:25.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.050+0000 7f285a400640 1 --2- 172.21.15.32:0/596272988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de12f3d4d0 0x55de13307560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.050+0000 7f285a400640 1 -- 172.21.15.32:0/596272988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55de12f352f0 msgr2=0x55de13300660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-30T08:36:25.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.051+0000 7f285a400640 1 --2- 172.21.15.32:0/596272988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55de12f352f0 0x55de13300660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.052 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.051+0000 7f285a400640 1 -- 172.21.15.32:0/596272988 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de12f76210 con 0x55de12f5e1b0 2024-04-30T08:36:25.052 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.051+0000 7f285a400640 1 --2- 172.21.15.32:0/596272988 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55de12f5e1b0 0x55de133000e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f284c034040 tx=0x7f284c030420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=308a7a01e476ffab server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:25.052 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.052+0000 7f2846a00640 1 -- 172.21.15.32:0/596272988 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f284c040030 con 0x55de12f5e1b0 2024-04-30T08:36:25.052 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.052+0000 7f2846a00640 1 -- 172.21.15.32:0/596272988 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f284c030610 con 0x55de12f5e1b0 2024-04-30T08:36:25.052 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.052+0000 7f2846a00640 1 -- 172.21.15.32:0/596272988 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f284c036040 con 0x55de12f5e1b0 2024-04-30T08:36:25.053 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.053+0000 7f2852000640 1 --2- 172.21.15.32:0/2432228069 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f283c0afbf0 0x7f283c0b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.053+0000 7f2863453740 1 -- 172.21.15.32:0/596272988 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55de12f5e1b0 msgr2=0x55de133000e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:25.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.053+0000 7f2863453740 1 --2- 172.21.15.32:0/596272988 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55de12f5e1b0 0x55de133000e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f284c034040 tx=0x7f284c030420 comp rx=0 tx=0).stop 2024-04-30T08:36:25.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.053+0000 7f2852000640 1 -- 172.21.15.32:0/2432228069 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f28480df430 con 0x7f2854104690 2024-04-30T08:36:25.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.053+0000 7f2863453740 1 -- 172.21.15.32:0/596272988 shutdown_connections 2024-04-30T08:36:25.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.053+0000 7f2863453740 1 --2- 172.21.15.32:0/596272988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de12f3d4d0 0x55de13307560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.053+0000 7f2863453740 1 --2- 172.21.15.32:0/596272988 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55de12f352f0 0x55de13300660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.053+0000 7f2863453740 1 --2- 172.21.15.32:0/596272988 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55de12f5e1b0 0x55de133000e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.053+0000 7f2863453740 1 -- 172.21.15.32:0/596272988 >> 172.21.15.32:0/596272988 conn(0x55de12f52300 msgr2=0x55de132fba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:25.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.053+0000 7f2859a00640 1 --2- 172.21.15.32:0/2432228069 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f283c0afbf0 0x7f283c0b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.054+0000 7f2863453740 1 -- 172.21.15.32:0/596272988 shutdown_connections 2024-04-30T08:36:25.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.054+0000 7f2863453740 1 -- 172.21.15.32:0/596272988 wait complete. 2024-04-30T08:36:25.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.054+0000 7f2859a00640 1 --2- 172.21.15.32:0/2432228069 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f283c0afbf0 0x7f283c0b2100 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f28480a3420 tx=0x7f284800ce60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:25.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.055+0000 7f2863453740 1 -- start start 2024-04-30T08:36:25.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.055+0000 7f2863453740 1 Processor -- start 2024-04-30T08:36:25.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.056+0000 7f2863453740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55de12f352f0 0x55de13312350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.056+0000 7f2863453740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de12f3d4d0 0x55de13312890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.056+0000 7f2863453740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55de12f5e1b0 0x55de133197f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.056+0000 7f2863453740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55de13307f50 con 0x55de12f5e1b0 2024-04-30T08:36:25.057 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.056+0000 7f2863453740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55de1321d660 con 0x55de12f352f0 2024-04-30T08:36:25.057 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.056+0000 7f2863453740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55de133080d0 con 0x55de12f3d4d0 2024-04-30T08:36:25.057 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.056+0000 7f2863453740 10 client.0 _ll_register_callbacks cb 0x55de131e7400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:36:25.057 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.056+0000 7f285a400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55de12f352f0 0x55de13312350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.056+0000 7f285a400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55de12f352f0 0x55de13312350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:60762/0 (socket says 172.21.15.32:60762) 2024-04-30T08:36:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.056+0000 7f285a400640 1 -- 172.21.15.32:0/3656573114 learned_addr learned my addr 172.21.15.32:0/3656573114 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.056+0000 7f2859000640 1 --2- 172.21.15.32:0/3656573114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de12f3d4d0 0x55de13312890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.057+0000 7f285a400640 1 --2- 172.21.15.32:0/3656573114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55de12f5e1b0 0x55de133197f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.057+0000 7f285a400640 1 -- 172.21.15.32:0/3656573114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de12f3d4d0 msgr2=0x55de13312890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.057+0000 7f285a400640 1 --2- 172.21.15.32:0/3656573114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de12f3d4d0 0x55de13312890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.057+0000 7f285a400640 1 -- 172.21.15.32:0/3656573114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55de12f5e1b0 msgr2=0x55de133197f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.057+0000 7f285a400640 1 --2- 172.21.15.32:0/3656573114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55de12f5e1b0 0x55de133197f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.058 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.057+0000 7f285a400640 1 -- 172.21.15.32:0/3656573114 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de13042620 con 0x55de12f352f0 2024-04-30T08:36:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.058+0000 7f285a400640 1 --2- 172.21.15.32:0/3656573114 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55de12f352f0 0x55de13312350 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f284c004010 tx=0x7f284c0307c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=92c8c479aa9559ba server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:25.059 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.058+0000 7f283b400640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:36:25.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.059+0000 7f283b400640 1 -- 172.21.15.32:0/3656573114 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f284c036040 con 0x55de12f352f0 2024-04-30T08:36:25.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.059+0000 7f283b400640 1 -- 172.21.15.32:0/3656573114 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f284c030c70 con 0x55de12f352f0 2024-04-30T08:36:25.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.059+0000 7f2863453740 1 -- 172.21.15.32:0/3656573114 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f284c030610 con 0x55de12f352f0 2024-04-30T08:36:25.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.059+0000 7f283b400640 1 -- 172.21.15.32:0/3656573114 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f284c030e70 con 0x55de12f352f0 2024-04-30T08:36:25.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.059+0000 7f2863453740 10 client.25919 fetch_fsmap learned FSMap version 28 2024-04-30T08:36:25.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.059+0000 7f2863453740 1 -- 172.21.15.32:0/3656573114 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55de1331c720 con 0x55de12f352f0 2024-04-30T08:36:25.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.060+0000 7f283b400640 1 -- 172.21.15.32:0/3656573114 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f284c030610 con 0x55de12f352f0 2024-04-30T08:36:25.062 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.061+0000 7f283b400640 1 -- 172.21.15.32:0/3656573114 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f284c031710 con 0x55de12f352f0 2024-04-30T08:36:25.062 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.061+0000 7f2863453740 10 client.25919 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:36:25.062 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.061+0000 7f2863453740 10 client.25919 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-30T08:36:25.062 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.061+0000 7f2863453740 20 client.25919 populate_metadata read hostname 'smithi032' 2024-04-30T08:36:25.062 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.061+0000 7f2863453740 1 --2- 172.21.15.32:0/3656573114 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55de12f1cd80 0x55de1331fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.062 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.061+0000 7f2863453740 4 client.25919 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:36:25.062 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.061+0000 7f2863453740 1 -- 172.21.15.32:0/3656573114 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55de13272160 con 0x55de12f1cd80 2024-04-30T08:36:25.062 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.061+0000 7f2859000640 1 --2- 172.21.15.32:0/3656573114 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55de12f1cd80 0x55de1331fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.063 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.062+0000 7f2859000640 1 --2- 172.21.15.32:0/3656573114 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55de12f1cd80 0x55de1331fa80 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6226574dfdc3d378 server_cookie=4dbcad6b854979e9 in_seq=0 out_seq=0 2024-04-30T08:36:25.063 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.062+0000 7f283b400640 10 client.25919 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:25.064 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.063+0000 7f283b400640 1 -- 172.21.15.32:0/3656573114 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55de13272160 con 0x55de12f1cd80 2024-04-30T08:36:25.064 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.063+0000 7f283b400640 10 client.25919 handle_command_reply: tid=0 2024-04-30T08:36:25.064 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.063+0000 7f2863453740 1 client.25919 shutdown 2024-04-30T08:36:25.064 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.063+0000 7f2863453740 1 -- 172.21.15.32:0/3656573114 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55de12f352f0 msgr2=0x55de13312350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:25.064 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.063+0000 7f2863453740 1 --2- 172.21.15.32:0/3656573114 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55de12f352f0 0x55de13312350 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f284c004010 tx=0x7f284c0307c0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.064 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.064+0000 7f2863453740 1 -- 172.21.15.32:0/3656573114 shutdown_connections 2024-04-30T08:36:25.065 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.064+0000 7f2863453740 1 --2- 172.21.15.32:0/3656573114 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55de12f1cd80 0x55de1331fa80 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.065 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.064+0000 7f283b400640 0 client.25919 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:25.065 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.064+0000 7f2863453740 1 --2- 172.21.15.32:0/3656573114 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55de12f5e1b0 0x55de133197f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.065 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.064+0000 7f2863453740 1 --2- 172.21.15.32:0/3656573114 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de12f3d4d0 0x55de13312890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.065 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.064+0000 7f2863453740 1 --2- 172.21.15.32:0/3656573114 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55de12f352f0 0x55de13312350 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.065 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.064+0000 7f2863453740 1 -- 172.21.15.32:0/3656573114 >> 172.21.15.32:0/3656573114 conn(0x55de12f52300 msgr2=0x55de132faa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:25.065 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.064+0000 7f2863453740 1 -- 172.21.15.32:0/3656573114 shutdown_connections 2024-04-30T08:36:25.065 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.064+0000 7f2863453740 1 -- 172.21.15.32:0/3656573114 wait complete. 2024-04-30T08:36:25.065 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.064+0000 7f2863453740 20 client.25919 trim_cache size 0 max 16384 2024-04-30T08:36:25.081 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.081+0000 7f2863453740 1 Processor -- start 2024-04-30T08:36:25.081 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.081+0000 7f2863453740 1 -- start start 2024-04-30T08:36:25.082 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.081+0000 7f2863453740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55de12f352f0 0x55de132fbb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.082 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.081+0000 7f2863453740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55de12f3d4d0 0x55de13319780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.082 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.081+0000 7f2863453740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de12f5e1b0 0x55de13319cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.082 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.081+0000 7f2863453740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f28541aaaa0 con 0x55de12f3d4d0 2024-04-30T08:36:25.083 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.081+0000 7f2863453740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f28541ac5e0 con 0x55de12f352f0 2024-04-30T08:36:25.083 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.081+0000 7f2863453740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f28541ab170 con 0x55de12f5e1b0 2024-04-30T08:36:25.083 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.082+0000 7f285a400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55de12f352f0 0x55de132fbb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.083 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.082+0000 7f285a400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55de12f352f0 0x55de132fbb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:60772/0 (socket says 172.21.15.32:60772) 2024-04-30T08:36:25.083 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.082+0000 7f285a400640 1 -- 172.21.15.32:0/1812434986 learned_addr learned my addr 172.21.15.32:0/1812434986 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:25.083 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.082+0000 7f2859000640 1 --2- 172.21.15.32:0/1812434986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55de12f3d4d0 0x55de13319780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.083 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.082+0000 7f285a400640 1 --2- 172.21.15.32:0/1812434986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de12f5e1b0 0x55de13319cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.084 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.082+0000 7f285a400640 1 -- 172.21.15.32:0/1812434986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de12f5e1b0 msgr2=0x55de13319cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:25.084 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.082+0000 7f285a400640 1 --2- 172.21.15.32:0/1812434986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de12f5e1b0 0x55de13319cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.084 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.082+0000 7f285a400640 1 -- 172.21.15.32:0/1812434986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55de12f3d4d0 msgr2=0x55de13319780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:25.084 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.082+0000 7f285a400640 1 --2- 172.21.15.32:0/1812434986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55de12f3d4d0 0x55de13319780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.084 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.082+0000 7f285a400640 1 -- 172.21.15.32:0/1812434986 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55de13304a40 con 0x55de12f352f0 2024-04-30T08:36:25.084 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.083+0000 7f285a400640 1 --2- 172.21.15.32:0/1812434986 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55de12f352f0 0x55de132fbb40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f284c004af0 tx=0x7f284c02a6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6afb230fef8b089b server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:25.084 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.083+0000 7f2846000640 1 -- 172.21.15.32:0/1812434986 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f284c036030 con 0x55de12f352f0 2024-04-30T08:36:25.084 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.083+0000 7f2846000640 1 -- 172.21.15.32:0/1812434986 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f284c041040 con 0x55de12f352f0 2024-04-30T08:36:25.084 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.083+0000 7f2846000640 1 -- 172.21.15.32:0/1812434986 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f284c040070 con 0x55de12f352f0 2024-04-30T08:36:25.085 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.084+0000 7f2863453740 1 -- 172.21.15.32:0/1812434986 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55de12f352f0 msgr2=0x55de132fbb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:25.085 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.084+0000 7f2863453740 1 --2- 172.21.15.32:0/1812434986 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55de12f352f0 0x55de132fbb40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f284c004af0 tx=0x7f284c02a6c0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.085 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.084+0000 7f2863453740 1 -- 172.21.15.32:0/1812434986 shutdown_connections 2024-04-30T08:36:25.085 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.084+0000 7f2863453740 1 --2- 172.21.15.32:0/1812434986 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de12f5e1b0 0x55de13319cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.085 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.084+0000 7f2863453740 1 --2- 172.21.15.32:0/1812434986 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55de12f3d4d0 0x55de13319780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.086 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.084+0000 7f2863453740 1 --2- 172.21.15.32:0/1812434986 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55de12f352f0 0x55de132fbb40 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.086 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.084+0000 7f2863453740 1 -- 172.21.15.32:0/1812434986 >> 172.21.15.32:0/1812434986 conn(0x55de12f52300 msgr2=0x55de132fb430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:25.086 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.085+0000 7f2863453740 1 -- 172.21.15.32:0/1812434986 shutdown_connections 2024-04-30T08:36:25.086 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.085+0000 7f2863453740 1 -- 172.21.15.32:0/1812434986 wait complete. 2024-04-30T08:36:25.086 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.085+0000 7f2863453740 1 -- start start 2024-04-30T08:36:25.087 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.085+0000 7f2863453740 1 Processor -- start 2024-04-30T08:36:25.087 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.086+0000 7f2863453740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55de12f352f0 0x55de133154a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.087 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.086+0000 7f2863453740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55de12f3d4d0 0x55de133159e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.087 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.086+0000 7f2863453740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de12f5e1b0 0x55de133188d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.087 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.086+0000 7f2863453740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f28541abf10 con 0x55de12f3d4d0 2024-04-30T08:36:25.087 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.086+0000 7f2863453740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f28541ab840 con 0x55de12f352f0 2024-04-30T08:36:25.087 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.086+0000 7f2863453740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f28541a8ae0 con 0x55de12f5e1b0 2024-04-30T08:36:25.087 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.086+0000 7f2863453740 10 client.0 _ll_register_callbacks cb 0x7f2848005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:36:25.088 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.087+0000 7f285a400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55de12f352f0 0x55de133154a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.088 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.087+0000 7f2859000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55de12f3d4d0 0x55de133159e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.089 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.087+0000 7f2859000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55de12f3d4d0 0x55de133159e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34236/0 (socket says 172.21.15.32:34236) 2024-04-30T08:36:25.089 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.087+0000 7f2859000640 1 -- 172.21.15.32:0/616895969 learned_addr learned my addr 172.21.15.32:0/616895969 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:25.089 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.087+0000 7f2859000640 1 -- 172.21.15.32:0/616895969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de12f5e1b0 msgr2=0x55de133188d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:25.089 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.087+0000 7f2859000640 1 --2- 172.21.15.32:0/616895969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de12f5e1b0 0x55de133188d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.089 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.087+0000 7f2859000640 1 -- 172.21.15.32:0/616895969 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55de12f352f0 msgr2=0x55de133154a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:25.089 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.087+0000 7f2859000640 1 --2- 172.21.15.32:0/616895969 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55de12f352f0 0x55de133154a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.089 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.087+0000 7f2859000640 1 -- 172.21.15.32:0/616895969 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55de13318e90 con 0x55de12f3d4d0 2024-04-30T08:36:25.089 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.088+0000 7f2859000640 1 --2- 172.21.15.32:0/616895969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55de12f3d4d0 0x55de133159e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2840009f20 tx=0x7f28400044f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9265d3e06d0689e server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:25.089 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.088+0000 7f283b400640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:36:25.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.088+0000 7f283b400640 1 -- 172.21.15.32:0/616895969 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2840026020 con 0x55de12f3d4d0 2024-04-30T08:36:25.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.089+0000 7f283b400640 1 -- 172.21.15.32:0/616895969 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f2840003d20 con 0x55de12f3d4d0 2024-04-30T08:36:25.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.089+0000 7f2863453740 1 -- 172.21.15.32:0/616895969 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f284c041040 con 0x55de12f3d4d0 2024-04-30T08:36:25.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.089+0000 7f283b400640 1 -- 172.21.15.32:0/616895969 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f28400124b0 con 0x55de12f3d4d0 2024-04-30T08:36:25.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.089+0000 7f2863453740 10 client.15795 fetch_fsmap learned FSMap version 28 2024-04-30T08:36:25.090 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.089+0000 7f2863453740 1 -- 172.21.15.32:0/616895969 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55de1331a1f0 con 0x55de12f3d4d0 2024-04-30T08:36:25.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.090+0000 7f283b400640 1 -- 172.21.15.32:0/616895969 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f284c041040 con 0x55de12f3d4d0 2024-04-30T08:36:25.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.090+0000 7f283b400640 1 -- 172.21.15.32:0/616895969 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f2840022400 con 0x55de12f3d4d0 2024-04-30T08:36:25.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.090+0000 7f2863453740 10 client.15795 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:36:25.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.090+0000 7f2863453740 10 client.15795 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-30T08:36:25.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.090+0000 7f2863453740 20 client.15795 populate_metadata read hostname 'smithi032' 2024-04-30T08:36:25.091 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.090+0000 7f2863453740 1 --2- 172.21.15.32:0/616895969 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55de12ef21c0 0x55de1338a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.092 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.090+0000 7f2863453740 4 client.15795 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-30T08:36:25.092 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.090+0000 7f2863453740 1 -- 172.21.15.32:0/616895969 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55de1326bfd0 con 0x55de12ef21c0 2024-04-30T08:36:25.092 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.091+0000 7f285a400640 1 --2- 172.21.15.32:0/616895969 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55de12ef21c0 0x55de1338a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.092 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.091+0000 7f285a400640 1 --2- 172.21.15.32:0/616895969 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55de12ef21c0 0x55de1338a3c0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd80867a0bfa5e7c server_cookie=2d78229c1e708708 in_seq=0 out_seq=0 2024-04-30T08:36:25.092 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.091+0000 7f283b400640 10 client.15795 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:25.098 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.098+0000 7f283b400640 1 -- 172.21.15.32:0/616895969 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55de1326bfd0 con 0x55de12ef21c0 2024-04-30T08:36:25.098 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.098+0000 7f283b400640 10 client.15795 handle_command_reply: tid=0 2024-04-30T08:36:25.098 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.098+0000 7f2863453740 1 client.15795 shutdown 2024-04-30T08:36:25.099 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.098+0000 7f2863453740 1 -- 172.21.15.32:0/616895969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55de12f3d4d0 msgr2=0x55de133159e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:25.099 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.098+0000 7f2863453740 1 --2- 172.21.15.32:0/616895969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55de12f3d4d0 0x55de133159e0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f2840009f20 tx=0x7f28400044f0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.099 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.098+0000 7f2863453740 1 -- 172.21.15.32:0/616895969 shutdown_connections 2024-04-30T08:36:25.099 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.098+0000 7f2863453740 1 --2- 172.21.15.32:0/616895969 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55de12ef21c0 0x55de1338a3c0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.099 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.098+0000 7f283b400640 0 client.15795 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:25.100 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.098+0000 7f2863453740 1 --2- 172.21.15.32:0/616895969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de12f5e1b0 0x55de133188d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.100 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.098+0000 7f2863453740 1 --2- 172.21.15.32:0/616895969 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55de12f3d4d0 0x55de133159e0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.100 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.098+0000 7f2863453740 1 --2- 172.21.15.32:0/616895969 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55de12f352f0 0x55de133154a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.100 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.098+0000 7f2863453740 1 -- 172.21.15.32:0/616895969 >> 172.21.15.32:0/616895969 conn(0x55de12f52300 msgr2=0x55de132fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:25.100 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.099+0000 7f2863453740 1 -- 172.21.15.32:0/616895969 shutdown_connections 2024-04-30T08:36:25.100 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.099+0000 7f2863453740 1 -- 172.21.15.32:0/616895969 wait complete. 2024-04-30T08:36:25.100 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.099+0000 7f2863453740 20 client.15795 trim_cache size 0 max 16384 2024-04-30T08:36:25.100 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-30T08:36:25.100 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-30T08:36:25.101 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-30T08:36:25.101 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-30T08:36:25.101 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.100+0000 7f285c400640 1 -- 172.21.15.32:0/2432228069 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f283c0afbf0 msgr2=0x7f283c0b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:25.101 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.100+0000 7f285c400640 1 --2- 172.21.15.32:0/2432228069 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f283c0afbf0 0x7f283c0b2100 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f28480a3420 tx=0x7f284800ce60 comp rx=0 tx=0).stop 2024-04-30T08:36:25.102 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.100+0000 7f285c400640 1 -- 172.21.15.32:0/2432228069 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2854104690 msgr2=0x7f285419c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:25.102 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.100+0000 7f285c400640 1 --2- 172.21.15.32:0/2432228069 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2854104690 0x7f285419c480 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2848005fd0 tx=0x7f284800ec20 comp rx=0 tx=0).stop 2024-04-30T08:36:25.102 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.100+0000 7f285c400640 1 -- 172.21.15.32:0/2432228069 shutdown_connections 2024-04-30T08:36:25.102 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.100+0000 7f285c400640 1 --2- 172.21.15.32:0/2432228069 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f283c0afbf0 0x7f283c0b2100 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.102 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.100+0000 7f285c400640 1 --2- 172.21.15.32:0/2432228069 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f2854109950 0x7f28541a3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.102 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.100+0000 7f285c400640 1 --2- 172.21.15.32:0/2432228069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2854106ff0 0x7f285419c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.102 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.100+0000 7f285c400640 1 --2- 172.21.15.32:0/2432228069 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f2854104690 0x7f285419c480 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.102 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.100+0000 7f285c400640 1 -- 172.21.15.32:0/2432228069 >> 172.21.15.32:0/2432228069 conn(0x7f28540fdc90 msgr2=0x7f2854102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:25.103 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.101+0000 7f285c400640 1 -- 172.21.15.32:0/2432228069 shutdown_connections 2024-04-30T08:36:25.103 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:25.101+0000 7f285c400640 1 -- 172.21.15.32:0/2432228069 wait complete. 2024-04-30T08:36:25.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:24 smithi071 ceph-mon[33385]: pgmap v462: 129 pgs: 129 active+clean; 674 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-30T08:36:25.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:24 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1001603528' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:36:25.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.286+0000 7fb75cc00640 1 -- 172.21.15.32:0/3457021376 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb758106dd0 msgr2=0x7fb7581091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:25.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.286+0000 7fb75cc00640 1 --2- 172.21.15.32:0/3457021376 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb758106dd0 0x7fb7581091f0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fb744009900 tx=0x7fb74402a3f0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.286+0000 7fb75cc00640 1 -- 172.21.15.32:0/3457021376 shutdown_connections 2024-04-30T08:36:25.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.286+0000 7fb75cc00640 1 --2- 172.21.15.32:0/3457021376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb758109730 0x7fb75810bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.286+0000 7fb75cc00640 1 --2- 172.21.15.32:0/3457021376 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb758106dd0 0x7fb7581091f0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.286+0000 7fb75cc00640 1 --2- 172.21.15.32:0/3457021376 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb758104470 0x7fb758106890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.286+0000 7fb75cc00640 1 -- 172.21.15.32:0/3457021376 >> 172.21.15.32:0/3457021376 conn(0x7fb7580fda70 msgr2=0x7fb7580fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:25.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.286+0000 7fb75cc00640 1 -- 172.21.15.32:0/3457021376 shutdown_connections 2024-04-30T08:36:25.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.286+0000 7fb75cc00640 1 -- 172.21.15.32:0/3457021376 wait complete. 2024-04-30T08:36:25.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.287+0000 7fb75cc00640 1 Processor -- start 2024-04-30T08:36:25.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.288+0000 7fb75cc00640 1 -- start start 2024-04-30T08:36:25.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.288+0000 7fb75cc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb758104470 0x7fb7581038c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.288+0000 7fb75cc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb758106dd0 0x7fb758101f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.288+0000 7fb75cc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb758109730 0x7fb7581024a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.288+0000 7fb75cc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb75810f4e0 con 0x7fb758104470 2024-04-30T08:36:25.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.288+0000 7fb75cc00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb75810f360 con 0x7fb758109730 2024-04-30T08:36:25.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.288+0000 7fb75cc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb75810f660 con 0x7fb758106dd0 2024-04-30T08:36:25.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.288+0000 7fb757400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb758104470 0x7fb7581038c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.289+0000 7fb756a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb758106dd0 0x7fb758101f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.289+0000 7fb757400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb758104470 0x7fb7581038c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34260/0 (socket says 172.21.15.32:34260) 2024-04-30T08:36:25.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.289+0000 7fb757400640 1 -- 172.21.15.32:0/3490038501 learned_addr learned my addr 172.21.15.32:0/3490038501 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:25.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.289+0000 7fb757e00640 1 --2- 172.21.15.32:0/3490038501 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb758109730 0x7fb7581024a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.289+0000 7fb756a00640 1 -- 172.21.15.32:0/3490038501 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb758109730 msgr2=0x7fb7581024a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:25.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.289+0000 7fb756a00640 1 --2- 172.21.15.32:0/3490038501 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb758109730 0x7fb7581024a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.289+0000 7fb756a00640 1 -- 172.21.15.32:0/3490038501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb758104470 msgr2=0x7fb7581038c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:25.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.289+0000 7fb756a00640 1 --2- 172.21.15.32:0/3490038501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb758104470 0x7fb7581038c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.289+0000 7fb756a00640 1 -- 172.21.15.32:0/3490038501 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb758102bd0 con 0x7fb758106dd0 2024-04-30T08:36:25.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.289+0000 7fb757400640 1 --2- 172.21.15.32:0/3490038501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb758104470 0x7fb7581038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:36:25.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.290+0000 7fb756a00640 1 --2- 172.21.15.32:0/3490038501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb758106dd0 0x7fb758101f60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb74402a9e0 tx=0x7fb744003d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:25.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.290+0000 7fb74fe00640 1 -- 172.21.15.32:0/3490038501 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb74403f070 con 0x7fb758106dd0 2024-04-30T08:36:25.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.290+0000 7fb74fe00640 1 -- 172.21.15.32:0/3490038501 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb74403b460 con 0x7fb758106dd0 2024-04-30T08:36:25.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.290+0000 7fb74fe00640 1 -- 172.21.15.32:0/3490038501 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb744035030 con 0x7fb758106dd0 2024-04-30T08:36:25.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.290+0000 7fb75cc00640 1 -- 172.21.15.32:0/3490038501 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7581a41a0 con 0x7fb758106dd0 2024-04-30T08:36:25.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.290+0000 7fb75cc00640 1 -- 172.21.15.32:0/3490038501 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7581a4550 con 0x7fb758106dd0 2024-04-30T08:36:25.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.295+0000 7fb74fe00640 1 -- 172.21.15.32:0/3490038501 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7fb744042070 con 0x7fb758106dd0 2024-04-30T08:36:25.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.295+0000 7fb76526b740 1 Processor -- start 2024-04-30T08:36:25.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.295+0000 7fb76526b740 1 -- start start 2024-04-30T08:36:25.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.296+0000 7fb76526b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55644a2b7390 0x55644a658730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.296+0000 7fb76526b740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55644a28e4d0 0x55644a658cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.296+0000 7fb76526b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55644a2966b0 0x55644a65fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.296+0000 7fb76526b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55644a5bdd50 con 0x55644a2b7390 2024-04-30T08:36:25.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.296+0000 7fb76526b740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55644a5630c0 con 0x55644a28e4d0 2024-04-30T08:36:25.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.296+0000 7fb76526b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55644a5b9d10 con 0x55644a2966b0 2024-04-30T08:36:25.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.296+0000 7fb757e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55644a2b7390 0x55644a658730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.296+0000 7fb756a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55644a2966b0 0x55644a65fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.296+0000 7fb757e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55644a2b7390 0x55644a658730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34268/0 (socket says 172.21.15.32:34268) 2024-04-30T08:36:25.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.296+0000 7fb757e00640 1 -- 172.21.15.32:0/2372963268 learned_addr learned my addr 172.21.15.32:0/2372963268 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:25.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.296+0000 7fb756a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55644a2966b0 0x55644a65fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:42946/0 (socket says 172.21.15.32:42946) 2024-04-30T08:36:25.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.296+0000 7fb757400640 1 --2- 172.21.15.32:0/2372963268 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55644a28e4d0 0x55644a658cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.297+0000 7fb757400640 1 -- 172.21.15.32:0/2372963268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55644a2966b0 msgr2=0x55644a65fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:25.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.297+0000 7fb757400640 1 --2- 172.21.15.32:0/2372963268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55644a2966b0 0x55644a65fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.297+0000 7fb757400640 1 -- 172.21.15.32:0/2372963268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55644a2b7390 msgr2=0x55644a658730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:25.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.297+0000 7fb757400640 1 --2- 172.21.15.32:0/2372963268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55644a2b7390 0x55644a658730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.297+0000 7fb757400640 1 -- 172.21.15.32:0/2372963268 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55644a2cf3f0 con 0x55644a28e4d0 2024-04-30T08:36:25.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.297+0000 7fb74fe00640 1 --2- 172.21.15.32:0/3490038501 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb73c0afbf0 0x7fb73c0b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.298 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.297+0000 7fb74fe00640 1 -- 172.21.15.32:0/3490038501 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fb7440ff930 con 0x7fb758106dd0 2024-04-30T08:36:25.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.297+0000 7fb757400640 1 --2- 172.21.15.32:0/2372963268 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55644a28e4d0 0x55644a658cb0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb748005fd0 tx=0x7fb74800bf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3823b4be7fd07773 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:25.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.297+0000 7fb757e00640 1 --2- 172.21.15.32:0/3490038501 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb73c0afbf0 0x7fb73c0b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.298+0000 7fb743e00640 1 -- 172.21.15.32:0/2372963268 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb748004230 con 0x55644a28e4d0 2024-04-30T08:36:25.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.298+0000 7fb743e00640 1 -- 172.21.15.32:0/2372963268 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7480043d0 con 0x55644a28e4d0 2024-04-30T08:36:25.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.298+0000 7fb743e00640 1 -- 172.21.15.32:0/2372963268 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7480035f0 con 0x55644a28e4d0 2024-04-30T08:36:25.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.299+0000 7fb76526b740 1 -- 172.21.15.32:0/2372963268 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55644a28e4d0 msgr2=0x55644a658cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:25.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.299+0000 7fb76526b740 1 --2- 172.21.15.32:0/2372963268 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55644a28e4d0 0x55644a658cb0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fb748005fd0 tx=0x7fb74800bf90 comp rx=0 tx=0).stop 2024-04-30T08:36:25.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.299+0000 7fb757e00640 1 --2- 172.21.15.32:0/3490038501 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb73c0afbf0 0x7fb73c0b2100 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x55644a659900 tx=0x7fb75000a4f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:25.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.299+0000 7fb76526b740 1 -- 172.21.15.32:0/2372963268 shutdown_connections 2024-04-30T08:36:25.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.299+0000 7fb76526b740 1 --2- 172.21.15.32:0/2372963268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55644a2966b0 0x55644a65fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.299+0000 7fb76526b740 1 --2- 172.21.15.32:0/2372963268 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55644a28e4d0 0x55644a658cb0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.299+0000 7fb76526b740 1 --2- 172.21.15.32:0/2372963268 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55644a2b7390 0x55644a658730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.299+0000 7fb76526b740 1 -- 172.21.15.32:0/2372963268 >> 172.21.15.32:0/2372963268 conn(0x55644a2ab4e0 msgr2=0x55644a6540c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:25.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.300+0000 7fb76526b740 1 -- 172.21.15.32:0/2372963268 shutdown_connections 2024-04-30T08:36:25.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.300+0000 7fb76526b740 1 -- 172.21.15.32:0/2372963268 wait complete. 2024-04-30T08:36:25.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:24 smithi189 ceph-mon[33038]: pgmap v462: 129 pgs: 129 active+clean; 674 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-30T08:36:25.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:24 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1001603528' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:36:25.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.300+0000 7fb76526b740 1 -- start start 2024-04-30T08:36:25.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.301+0000 7fb76526b740 1 Processor -- start 2024-04-30T08:36:25.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.301+0000 7fb76526b740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55644a28e4d0 0x55644a662580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.301+0000 7fb76526b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55644a2966b0 0x55644a662ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.301+0000 7fb76526b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55644a2b7390 0x55644a669a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.301+0000 7fb76526b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55644a660600 con 0x55644a2966b0 2024-04-30T08:36:25.302 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.301+0000 7fb76526b740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55644a576b50 con 0x55644a28e4d0 2024-04-30T08:36:25.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.301+0000 7fb76526b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55644a660780 con 0x55644a2b7390 2024-04-30T08:36:25.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.302+0000 7fb76526b740 10 client.0 _ll_register_callbacks cb 0x55644a540420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:36:25.303 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.302+0000 7fb757400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55644a28e4d0 0x55644a662580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.302+0000 7fb757400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55644a28e4d0 0x55644a662580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:60806/0 (socket says 172.21.15.32:60806) 2024-04-30T08:36:25.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.302+0000 7fb757400640 1 -- 172.21.15.32:0/2974010869 learned_addr learned my addr 172.21.15.32:0/2974010869 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:25.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.302+0000 7fb757e00640 1 --2- 172.21.15.32:0/2974010869 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55644a2b7390 0x55644a669a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.302+0000 7fb756a00640 1 --2- 172.21.15.32:0/2974010869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55644a2966b0 0x55644a662ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.302+0000 7fb757400640 1 -- 172.21.15.32:0/2974010869 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55644a2b7390 msgr2=0x55644a669a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:25.304 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.302+0000 7fb757400640 1 --2- 172.21.15.32:0/2974010869 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55644a2b7390 0x55644a669a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.302+0000 7fb757400640 1 -- 172.21.15.32:0/2974010869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55644a2966b0 msgr2=0x55644a662ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:25.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.302+0000 7fb757400640 1 --2- 172.21.15.32:0/2974010869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55644a2966b0 0x55644a662ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.302+0000 7fb757400640 1 -- 172.21.15.32:0/2974010869 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55644a3997d0 con 0x55644a28e4d0 2024-04-30T08:36:25.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.303+0000 7fb757400640 1 --2- 172.21.15.32:0/2974010869 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55644a28e4d0 0x55644a662580 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb748000c00 tx=0x7fb748004960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be9819c16d284720 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:25.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.303+0000 7fb740c00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:36:25.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.303+0000 7fb740c00640 1 -- 172.21.15.32:0/2974010869 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb74802b060 con 0x55644a28e4d0 2024-04-30T08:36:25.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.303+0000 7fb740c00640 1 -- 172.21.15.32:0/2974010869 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fb748004fb0 con 0x55644a28e4d0 2024-04-30T08:36:25.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.303+0000 7fb76526b740 1 -- 172.21.15.32:0/2974010869 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb7480043d0 con 0x55644a28e4d0 2024-04-30T08:36:25.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.303+0000 7fb740c00640 1 -- 172.21.15.32:0/2974010869 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb7480051b0 con 0x55644a28e4d0 2024-04-30T08:36:25.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.303+0000 7fb76526b740 10 client.25934 fetch_fsmap learned FSMap version 28 2024-04-30T08:36:25.305 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.303+0000 7fb76526b740 1 -- 172.21.15.32:0/2974010869 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55644a67d960 con 0x55644a28e4d0 2024-04-30T08:36:25.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.304+0000 7fb740c00640 1 -- 172.21.15.32:0/2974010869 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7480043d0 con 0x55644a28e4d0 2024-04-30T08:36:25.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.304+0000 7fb740c00640 1 -- 172.21.15.32:0/2974010869 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb74802e020 con 0x55644a28e4d0 2024-04-30T08:36:25.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.304+0000 7fb76526b740 10 client.25934 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:36:25.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.304+0000 7fb76526b740 10 client.25934 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-30T08:36:25.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.304+0000 7fb76526b740 20 client.25934 populate_metadata read hostname 'smithi032' 2024-04-30T08:36:25.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.305+0000 7fb76526b740 1 --2- 172.21.15.32:0/2974010869 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x55644a275f60 0x55644a680d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.305+0000 7fb76526b740 4 client.25934 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:36:25.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.305+0000 7fb76526b740 1 -- 172.21.15.32:0/2974010869 --> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55644a5cb960 con 0x55644a275f60 2024-04-30T08:36:25.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.305+0000 7fb756a00640 1 --2- 172.21.15.32:0/2974010869 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x55644a275f60 0x55644a680d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.306+0000 7fb756a00640 1 --2- 172.21.15.32:0/2974010869 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x55644a275f60 0x55644a680d20 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f78c0f813a6f8ffe server_cookie=8a325137321cf832 in_seq=0 out_seq=0 2024-04-30T08:36:25.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.306+0000 7fb740c00640 10 client.25934 ms_handle_connect on v2:172.21.15.71:6836/2324039932 2024-04-30T08:36:25.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.307+0000 7fb740c00640 1 -- 172.21.15.32:0/2974010869 <== mds.3 v2:172.21.15.71:6836/2324039932 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55644a5cb960 con 0x55644a275f60 2024-04-30T08:36:25.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.307+0000 7fb740c00640 10 client.25934 handle_command_reply: tid=0 2024-04-30T08:36:25.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.307+0000 7fb76526b740 1 client.25934 shutdown 2024-04-30T08:36:25.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.307+0000 7fb76526b740 1 -- 172.21.15.32:0/2974010869 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55644a28e4d0 msgr2=0x55644a662580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:25.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.307+0000 7fb76526b740 1 --2- 172.21.15.32:0/2974010869 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55644a28e4d0 0x55644a662580 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fb748000c00 tx=0x7fb748004960 comp rx=0 tx=0).stop 2024-04-30T08:36:25.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.307+0000 7fb76526b740 1 -- 172.21.15.32:0/2974010869 shutdown_connections 2024-04-30T08:36:25.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.307+0000 7fb76526b740 1 --2- 172.21.15.32:0/2974010869 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x55644a275f60 0x55644a680d20 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.307+0000 7fb76526b740 1 --2- 172.21.15.32:0/2974010869 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55644a2b7390 0x55644a669a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.307+0000 7fb740c00640 0 client.25934 ms_handle_reset on v2:172.21.15.71:6836/2324039932 2024-04-30T08:36:25.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.308+0000 7fb76526b740 1 --2- 172.21.15.32:0/2974010869 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55644a2966b0 0x55644a662ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.308+0000 7fb76526b740 1 --2- 172.21.15.32:0/2974010869 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55644a28e4d0 0x55644a662580 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.308+0000 7fb76526b740 1 -- 172.21.15.32:0/2974010869 >> 172.21.15.32:0/2974010869 conn(0x55644a2ab4e0 msgr2=0x55644a653100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:25.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.308+0000 7fb76526b740 1 -- 172.21.15.32:0/2974010869 shutdown_connections 2024-04-30T08:36:25.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.308+0000 7fb76526b740 1 -- 172.21.15.32:0/2974010869 wait complete. 2024-04-30T08:36:25.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.308+0000 7fb76526b740 20 client.25934 trim_cache size 0 max 16384 2024-04-30T08:36:25.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.337+0000 7fb76526b740 1 Processor -- start 2024-04-30T08:36:25.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.337+0000 7fb76526b740 1 -- start start 2024-04-30T08:36:25.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.337+0000 7fb76526b740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55644a275f60 0x55644a661230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.337+0000 7fb76526b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55644a28e4d0 0x55644a656f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.337+0000 7fb76526b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55644a2966b0 0x55644a6574a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.337+0000 7fb76526b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb7581a64c0 con 0x55644a28e4d0 2024-04-30T08:36:25.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.337+0000 7fb76526b740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb7581a8000 con 0x55644a275f60 2024-04-30T08:36:25.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.337+0000 7fb76526b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb7581a6b90 con 0x55644a2966b0 2024-04-30T08:36:25.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.338+0000 7fb757400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55644a275f60 0x55644a661230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.338+0000 7fb757400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55644a275f60 0x55644a661230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:60810/0 (socket says 172.21.15.32:60810) 2024-04-30T08:36:25.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.338+0000 7fb757400640 1 -- 172.21.15.32:0/682229460 learned_addr learned my addr 172.21.15.32:0/682229460 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:25.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.338+0000 7fb757e00640 1 --2- 172.21.15.32:0/682229460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55644a2966b0 0x55644a6574a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.338+0000 7fb756a00640 1 --2- 172.21.15.32:0/682229460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55644a28e4d0 0x55644a656f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.338+0000 7fb757400640 1 -- 172.21.15.32:0/682229460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55644a2966b0 msgr2=0x55644a6574a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:25.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.338+0000 7fb757400640 1 --2- 172.21.15.32:0/682229460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55644a2966b0 0x55644a6574a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.339+0000 7fb757400640 1 -- 172.21.15.32:0/682229460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55644a28e4d0 msgr2=0x55644a656f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:25.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.339+0000 7fb757400640 1 --2- 172.21.15.32:0/682229460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55644a28e4d0 0x55644a656f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.339+0000 7fb757400640 1 -- 172.21.15.32:0/682229460 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55644a64f8e0 con 0x55644a275f60 2024-04-30T08:36:25.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.339+0000 7fb757400640 1 --2- 172.21.15.32:0/682229460 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55644a275f60 0x55644a661230 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb748001eb0 tx=0x7fb748010ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b22cbd3065d37a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:25.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.339+0000 7fb743400640 1 -- 172.21.15.32:0/682229460 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb74802b070 con 0x55644a275f60 2024-04-30T08:36:25.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.340+0000 7fb743400640 1 -- 172.21.15.32:0/682229460 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb748004fb0 con 0x55644a275f60 2024-04-30T08:36:25.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.340+0000 7fb743400640 1 -- 172.21.15.32:0/682229460 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb7480117c0 con 0x55644a275f60 2024-04-30T08:36:25.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.341+0000 7fb76526b740 1 -- 172.21.15.32:0/682229460 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55644a275f60 msgr2=0x55644a661230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:25.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.341+0000 7fb76526b740 1 --2- 172.21.15.32:0/682229460 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55644a275f60 0x55644a661230 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fb748001eb0 tx=0x7fb748010ee0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.341+0000 7fb76526b740 1 -- 172.21.15.32:0/682229460 shutdown_connections 2024-04-30T08:36:25.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.341+0000 7fb76526b740 1 --2- 172.21.15.32:0/682229460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55644a2966b0 0x55644a6574a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.341+0000 7fb76526b740 1 --2- 172.21.15.32:0/682229460 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55644a28e4d0 0x55644a656f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.341+0000 7fb76526b740 1 --2- 172.21.15.32:0/682229460 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55644a275f60 0x55644a661230 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.341+0000 7fb76526b740 1 -- 172.21.15.32:0/682229460 >> 172.21.15.32:0/682229460 conn(0x55644a2ab4e0 msgr2=0x55644a660b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:25.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.341+0000 7fb76526b740 1 -- 172.21.15.32:0/682229460 shutdown_connections 2024-04-30T08:36:25.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.341+0000 7fb76526b740 1 -- 172.21.15.32:0/682229460 wait complete. 2024-04-30T08:36:25.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.342+0000 7fb76526b740 1 -- start start 2024-04-30T08:36:25.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.342+0000 7fb76526b740 1 Processor -- start 2024-04-30T08:36:25.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.343+0000 7fb76526b740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55644a275f60 0x55644a6662e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.343+0000 7fb76526b740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55644a28e4d0 0x55644a666820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.343+0000 7fb76526b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55644a2966b0 0x55644a667750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.343+0000 7fb76526b740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb7581a7930 con 0x55644a28e4d0 2024-04-30T08:36:25.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.343+0000 7fb76526b740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb7581a7260 con 0x55644a275f60 2024-04-30T08:36:25.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.343+0000 7fb76526b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb758074a80 con 0x55644a2966b0 2024-04-30T08:36:25.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.343+0000 7fb76526b740 10 client.0 _ll_register_callbacks cb 0x7fb74800be20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:36:25.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.343+0000 7fb757400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55644a275f60 0x55644a6662e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.343+0000 7fb757400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55644a275f60 0x55644a6662e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:60820/0 (socket says 172.21.15.32:60820) 2024-04-30T08:36:25.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.343+0000 7fb757400640 1 -- 172.21.15.32:0/860306870 learned_addr learned my addr 172.21.15.32:0/860306870 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:25.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.343+0000 7fb756a00640 1 --2- 172.21.15.32:0/860306870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55644a28e4d0 0x55644a666820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.344+0000 7fb757e00640 1 --2- 172.21.15.32:0/860306870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55644a2966b0 0x55644a667750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.344+0000 7fb756a00640 1 -- 172.21.15.32:0/860306870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55644a2966b0 msgr2=0x55644a667750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:25.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.344+0000 7fb756a00640 1 --2- 172.21.15.32:0/860306870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55644a2966b0 0x55644a667750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.344+0000 7fb756a00640 1 -- 172.21.15.32:0/860306870 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55644a275f60 msgr2=0x55644a6662e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:25.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.344+0000 7fb756a00640 1 --2- 172.21.15.32:0/860306870 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55644a275f60 0x55644a6662e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.344+0000 7fb756a00640 1 -- 172.21.15.32:0/860306870 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55644a65bab0 con 0x55644a28e4d0 2024-04-30T08:36:25.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.344+0000 7fb757e00640 1 --2- 172.21.15.32:0/860306870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55644a2966b0 0x55644a667750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:36:25.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.344+0000 7fb757400640 1 --2- 172.21.15.32:0/860306870 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55644a275f60 0x55644a6662e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:36:25.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.344+0000 7fb756a00640 1 --2- 172.21.15.32:0/860306870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55644a28e4d0 0x55644a666820 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb7440fced0 tx=0x7fb744101d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c202fc859c18a696 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:25.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.345+0000 7fb740c00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:36:25.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.345+0000 7fb740c00640 1 -- 172.21.15.32:0/860306870 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb744101e60 con 0x55644a28e4d0 2024-04-30T08:36:25.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.345+0000 7fb740c00640 1 -- 172.21.15.32:0/860306870 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fb7440ea0d0 con 0x55644a28e4d0 2024-04-30T08:36:25.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.345+0000 7fb76526b740 1 -- 172.21.15.32:0/860306870 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb748004fb0 con 0x55644a28e4d0 2024-04-30T08:36:25.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.345+0000 7fb740c00640 1 -- 172.21.15.32:0/860306870 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb744049040 con 0x55644a28e4d0 2024-04-30T08:36:25.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.345+0000 7fb76526b740 10 client.15813 fetch_fsmap learned FSMap version 28 2024-04-30T08:36:25.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.345+0000 7fb76526b740 1 -- 172.21.15.32:0/860306870 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55644a668e90 con 0x55644a28e4d0 2024-04-30T08:36:25.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.346+0000 7fb740c00640 1 -- 172.21.15.32:0/860306870 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb748004fb0 con 0x55644a28e4d0 2024-04-30T08:36:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.347+0000 7fb740c00640 1 -- 172.21.15.32:0/860306870 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb7440c3be0 con 0x55644a28e4d0 2024-04-30T08:36:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.347+0000 7fb76526b740 10 client.15813 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:36:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.347+0000 7fb76526b740 10 client.15813 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-30T08:36:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.347+0000 7fb76526b740 20 client.15813 populate_metadata read hostname 'smithi032' 2024-04-30T08:36:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.347+0000 7fb76526b740 1 --2- 172.21.15.32:0/860306870 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x55644a2b7390 0x55644a6f3a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.347+0000 7fb76526b740 4 client.15813 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-30T08:36:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.347+0000 7fb76526b740 1 -- 172.21.15.32:0/860306870 --> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55644a5c57d0 con 0x55644a2b7390 2024-04-30T08:36:25.348 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.347+0000 7fb757400640 1 --2- 172.21.15.32:0/860306870 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x55644a2b7390 0x55644a6f3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.348+0000 7fb757400640 1 --2- 172.21.15.32:0/860306870 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x55644a2b7390 0x55644a6f3a10 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=3834430b7d6b00c6 server_cookie=9c77b40c7bbedd27 in_seq=0 out_seq=0 2024-04-30T08:36:25.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.348+0000 7fb740c00640 10 client.15813 ms_handle_connect on v2:172.21.15.71:6836/2324039932 2024-04-30T08:36:25.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.348+0000 7fb740c00640 1 -- 172.21.15.32:0/860306870 <== mds.3 v2:172.21.15.71:6836/2324039932 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55644a5c57d0 con 0x55644a2b7390 2024-04-30T08:36:25.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.348+0000 7fb740c00640 10 client.15813 handle_command_reply: tid=0 2024-04-30T08:36:25.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.348+0000 7fb76526b740 1 client.15813 shutdown 2024-04-30T08:36:25.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.348+0000 7fb76526b740 1 -- 172.21.15.32:0/860306870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55644a28e4d0 msgr2=0x55644a666820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:25.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.348+0000 7fb76526b740 1 --2- 172.21.15.32:0/860306870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55644a28e4d0 0x55644a666820 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb7440fced0 tx=0x7fb744101d80 comp rx=0 tx=0).stop 2024-04-30T08:36:25.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.348+0000 7fb76526b740 1 -- 172.21.15.32:0/860306870 shutdown_connections 2024-04-30T08:36:25.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.349+0000 7fb76526b740 1 --2- 172.21.15.32:0/860306870 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x55644a2b7390 0x55644a6f3a10 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.349+0000 7fb76526b740 1 --2- 172.21.15.32:0/860306870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55644a2966b0 0x55644a667750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.349+0000 7fb76526b740 1 --2- 172.21.15.32:0/860306870 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55644a28e4d0 0x55644a666820 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.349+0000 7fb76526b740 1 --2- 172.21.15.32:0/860306870 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55644a275f60 0x55644a6662e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.349+0000 7fb740c00640 0 client.15813 ms_handle_reset on v2:172.21.15.71:6836/2324039932 2024-04-30T08:36:25.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.349+0000 7fb76526b740 1 -- 172.21.15.32:0/860306870 >> 172.21.15.32:0/860306870 conn(0x55644a2ab4e0 msgr2=0x55644a656f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:25.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.349+0000 7fb76526b740 1 -- 172.21.15.32:0/860306870 shutdown_connections 2024-04-30T08:36:25.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.349+0000 7fb76526b740 1 -- 172.21.15.32:0/860306870 wait complete. 2024-04-30T08:36:25.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.349+0000 7fb76526b740 20 client.15813 trim_cache size 0 max 16384 2024-04-30T08:36:25.351 INFO:teuthology.orchestra.run.smithi032.stdout:[] 2024-04-30T08:36:25.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.350+0000 7fb75cc00640 1 -- 172.21.15.32:0/3490038501 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb73c0afbf0 msgr2=0x7fb73c0b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:25.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.350+0000 7fb75cc00640 1 --2- 172.21.15.32:0/3490038501 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb73c0afbf0 0x7fb73c0b2100 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x55644a659900 tx=0x7fb75000a4f0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.350+0000 7fb75cc00640 1 -- 172.21.15.32:0/3490038501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb758106dd0 msgr2=0x7fb758101f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:25.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.350+0000 7fb75cc00640 1 --2- 172.21.15.32:0/3490038501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb758106dd0 0x7fb758101f60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb74402a9e0 tx=0x7fb744003d50 comp rx=0 tx=0).stop 2024-04-30T08:36:25.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.351+0000 7fb75cc00640 1 -- 172.21.15.32:0/3490038501 shutdown_connections 2024-04-30T08:36:25.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.351+0000 7fb75cc00640 1 --2- 172.21.15.32:0/3490038501 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb73c0afbf0 0x7fb73c0b2100 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.351+0000 7fb75cc00640 1 --2- 172.21.15.32:0/3490038501 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb758109730 0x7fb7581024a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.351+0000 7fb75cc00640 1 --2- 172.21.15.32:0/3490038501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb758106dd0 0x7fb758101f60 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.351+0000 7fb75cc00640 1 --2- 172.21.15.32:0/3490038501 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb758104470 0x7fb7581038c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.351+0000 7fb75cc00640 1 -- 172.21.15.32:0/3490038501 >> 172.21.15.32:0/3490038501 conn(0x7fb7580fda70 msgr2=0x7fb7581050e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:25.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.351+0000 7fb75cc00640 1 -- 172.21.15.32:0/3490038501 shutdown_connections 2024-04-30T08:36:25.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.351+0000 7fb75cc00640 1 -- 172.21.15.32:0/3490038501 wait complete. 2024-04-30T08:36:25.377 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-30T08:36:25.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.677+0000 7fa157200640 1 -- 172.21.15.32:0/3439141858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa158104bf0 msgr2=0x7fa158105000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:25.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.677+0000 7fa157200640 1 --2- 172.21.15.32:0/3439141858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa158104bf0 0x7fa158105000 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fa144009860 tx=0x7fa14402a340 comp rx=0 tx=0).stop 2024-04-30T08:36:25.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.677+0000 7fa157200640 1 -- 172.21.15.32:0/3439141858 shutdown_connections 2024-04-30T08:36:25.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.677+0000 7fa157200640 1 --2- 172.21.15.32:0/3439141858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1581098f0 0x7fa15810bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.677+0000 7fa157200640 1 --2- 172.21.15.32:0/3439141858 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa158104bf0 0x7fa158105000 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.677+0000 7fa157200640 1 --2- 172.21.15.32:0/3439141858 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1581039b0 0x7fa1581046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.677+0000 7fa157200640 1 -- 172.21.15.32:0/3439141858 >> 172.21.15.32:0/3439141858 conn(0x7fa1580feba0 msgr2=0x7fa158101050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:25.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.678+0000 7fa157200640 1 -- 172.21.15.32:0/3439141858 shutdown_connections 2024-04-30T08:36:25.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.678+0000 7fa157200640 1 -- 172.21.15.32:0/3439141858 wait complete. 2024-04-30T08:36:25.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.679+0000 7fa157200640 1 Processor -- start 2024-04-30T08:36:25.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.679+0000 7fa157200640 1 -- start start 2024-04-30T08:36:25.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.679+0000 7fa157200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1581039b0 0x7fa1581983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.679+0000 7fa157200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa158104bf0 0x7fa158198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.679+0000 7fa157200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1581098f0 0x7fa158194f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.679+0000 7fa157200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa15810f4e0 con 0x7fa158104bf0 2024-04-30T08:36:25.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.679+0000 7fa157200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa15810f360 con 0x7fa1581098f0 2024-04-30T08:36:25.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.679+0000 7fa157200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa15810f660 con 0x7fa1581039b0 2024-04-30T08:36:25.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.680+0000 7fa155e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1581039b0 0x7fa1581983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.680+0000 7fa155e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1581039b0 0x7fa1581983f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:43006/0 (socket says 172.21.15.32:43006) 2024-04-30T08:36:25.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.680+0000 7fa155e00640 1 -- 172.21.15.32:0/3053075882 learned_addr learned my addr 172.21.15.32:0/3053075882 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:25.681 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.680+0000 7fa156800640 1 --2- 172.21.15.32:0/3053075882 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1581098f0 0x7fa158194f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.680+0000 7fa155400640 1 --2- 172.21.15.32:0/3053075882 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa158104bf0 0x7fa158198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.680+0000 7fa155e00640 1 -- 172.21.15.32:0/3053075882 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1581098f0 msgr2=0x7fa158194f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:25.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.680+0000 7fa155e00640 1 --2- 172.21.15.32:0/3053075882 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1581098f0 0x7fa158194f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.680+0000 7fa155e00640 1 -- 172.21.15.32:0/3053075882 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa158104bf0 msgr2=0x7fa158198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:25.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.680+0000 7fa155e00640 1 --2- 172.21.15.32:0/3053075882 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa158104bf0 0x7fa158198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.680+0000 7fa155e00640 1 -- 172.21.15.32:0/3053075882 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa158195760 con 0x7fa1581039b0 2024-04-30T08:36:25.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.680+0000 7fa155400640 1 --2- 172.21.15.32:0/3053075882 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa158104bf0 0x7fa158198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-30T08:36:25.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.680+0000 7fa156800640 1 --2- 172.21.15.32:0/3053075882 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1581098f0 0x7fa158194f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:36:25.682 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.681+0000 7fa155e00640 1 --2- 172.21.15.32:0/3053075882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1581039b0 0x7fa1581983f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa14800e980 tx=0x7fa14800ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:25.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.681+0000 7fa14ea00640 1 -- 172.21.15.32:0/3053075882 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa14800cd80 con 0x7fa1581039b0 2024-04-30T08:36:25.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.681+0000 7fa157200640 1 -- 172.21.15.32:0/3053075882 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa158195a50 con 0x7fa1581039b0 2024-04-30T08:36:25.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.681+0000 7fa14ea00640 1 -- 172.21.15.32:0/3053075882 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1480108e0 con 0x7fa1581039b0 2024-04-30T08:36:25.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.681+0000 7fa157200640 1 -- 172.21.15.32:0/3053075882 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa15819c2b0 con 0x7fa1581039b0 2024-04-30T08:36:25.683 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.681+0000 7fa14ea00640 1 -- 172.21.15.32:0/3053075882 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1480188f0 con 0x7fa1581039b0 2024-04-30T08:36:25.686 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.686+0000 7fa163de4740 1 Processor -- start 2024-04-30T08:36:25.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.686+0000 7fa163de4740 1 -- start start 2024-04-30T08:36:25.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.686+0000 7fa163de4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da8f867390 0x55da8fc08730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.686+0000 7fa163de4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da8f83e4d0 0x55da8fc08cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.686+0000 7fa163de4740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55da8f8466b0 0x55da8fc0fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.686+0000 7fa163de4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55da8fb6dd50 con 0x55da8f83e4d0 2024-04-30T08:36:25.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.686+0000 7fa163de4740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55da8fb130c0 con 0x55da8f8466b0 2024-04-30T08:36:25.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.686+0000 7fa163de4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55da8fb69d10 con 0x55da8f867390 2024-04-30T08:36:25.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.687+0000 7fa155400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55da8f8466b0 0x55da8fc0fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.687+0000 7fa155400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55da8f8466b0 0x55da8fc0fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:60836/0 (socket says 172.21.15.32:60836) 2024-04-30T08:36:25.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.687+0000 7fa155400640 1 -- 172.21.15.32:0/1231597994 learned_addr learned my addr 172.21.15.32:0/1231597994 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:25.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.687+0000 7fa156800640 1 --2- 172.21.15.32:0/1231597994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da8f867390 0x55da8fc08730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.687+0000 7fa14ea00640 1 -- 172.21.15.32:0/3053075882 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7fa14801f020 con 0x7fa1581039b0 2024-04-30T08:36:25.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.688+0000 7fa155400640 1 -- 172.21.15.32:0/1231597994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da8f867390 msgr2=0x55da8fc08730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:25.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.688+0000 7fa155400640 1 --2- 172.21.15.32:0/1231597994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da8f867390 0x55da8fc08730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.688 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.688+0000 7fa155400640 1 -- 172.21.15.32:0/1231597994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da8f83e4d0 msgr2=0x55da8fc08cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:25.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.688+0000 7fa155400640 1 --2- 172.21.15.32:0/1231597994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da8f83e4d0 0x55da8fc08cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.688+0000 7fa155400640 1 -- 172.21.15.32:0/1231597994 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da8f87f3f0 con 0x55da8f8466b0 2024-04-30T08:36:25.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.689+0000 7fa155400640 1 --2- 172.21.15.32:0/1231597994 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55da8f8466b0 0x55da8fc0fbb0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa1440303f0 tx=0x7fa144030b50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c3d77acfe047a67 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:25.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.689+0000 7fa143400640 1 -- 172.21.15.32:0/1231597994 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa144037070 con 0x55da8f8466b0 2024-04-30T08:36:25.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.689+0000 7fa143400640 1 -- 172.21.15.32:0/1231597994 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa144038040 con 0x55da8f8466b0 2024-04-30T08:36:25.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.689+0000 7fa143400640 1 -- 172.21.15.32:0/1231597994 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa144005440 con 0x55da8f8466b0 2024-04-30T08:36:25.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.690+0000 7fa163de4740 1 -- 172.21.15.32:0/1231597994 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55da8f8466b0 msgr2=0x55da8fc0fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:25.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.690+0000 7fa163de4740 1 --2- 172.21.15.32:0/1231597994 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55da8f8466b0 0x55da8fc0fbb0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fa1440303f0 tx=0x7fa144030b50 comp rx=0 tx=0).stop 2024-04-30T08:36:25.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.690+0000 7fa14ea00640 1 --2- 172.21.15.32:0/3053075882 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa13c0afbf0 0x7fa13c0b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.691+0000 7fa14ea00640 1 -- 172.21.15.32:0/3053075882 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fa1480df2f0 con 0x7fa1581039b0 2024-04-30T08:36:25.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.691+0000 7fa156800640 1 --2- 172.21.15.32:0/3053075882 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa13c0afbf0 0x7fa13c0b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.691+0000 7fa163de4740 1 -- 172.21.15.32:0/1231597994 shutdown_connections 2024-04-30T08:36:25.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.691+0000 7fa163de4740 1 --2- 172.21.15.32:0/1231597994 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55da8f8466b0 0x55da8fc0fbb0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.691+0000 7fa163de4740 1 --2- 172.21.15.32:0/1231597994 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da8f83e4d0 0x55da8fc08cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.691+0000 7fa163de4740 1 --2- 172.21.15.32:0/1231597994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da8f867390 0x55da8fc08730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.691+0000 7fa163de4740 1 -- 172.21.15.32:0/1231597994 >> 172.21.15.32:0/1231597994 conn(0x55da8f85b4e0 msgr2=0x55da8fc040c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:25.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.692+0000 7fa163de4740 1 -- 172.21.15.32:0/1231597994 shutdown_connections 2024-04-30T08:36:25.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.692+0000 7fa163de4740 1 -- 172.21.15.32:0/1231597994 wait complete. 2024-04-30T08:36:25.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.692+0000 7fa156800640 1 --2- 172.21.15.32:0/3053075882 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa13c0afbf0 0x7fa13c0b2100 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x55da8fc09900 tx=0x7fa150009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:25.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.693+0000 7fa163de4740 1 -- start start 2024-04-30T08:36:25.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.693+0000 7fa163de4740 1 Processor -- start 2024-04-30T08:36:25.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.693+0000 7fa163de4740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55da8f83e4d0 0x55da8fc126c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.693+0000 7fa163de4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da8f8466b0 0x55da8fc12c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.693+0000 7fa163de4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da8f867390 0x55da8fc19b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.693+0000 7fa163de4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55da8fc105a0 con 0x55da8f8466b0 2024-04-30T08:36:25.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.693+0000 7fa163de4740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55da8fb26b50 con 0x55da8f83e4d0 2024-04-30T08:36:25.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.693+0000 7fa163de4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55da8fc10720 con 0x55da8f867390 2024-04-30T08:36:25.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.694+0000 7fa163de4740 10 client.0 _ll_register_callbacks cb 0x55da8faf0420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:36:25.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.694+0000 7fa155e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55da8f83e4d0 0x55da8fc126c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.694+0000 7fa155e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55da8f83e4d0 0x55da8fc126c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:60848/0 (socket says 172.21.15.32:60848) 2024-04-30T08:36:25.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.694+0000 7fa156800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da8f867390 0x55da8fc19b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.694+0000 7fa155e00640 1 -- 172.21.15.32:0/3349894006 learned_addr learned my addr 172.21.15.32:0/3349894006 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:25.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.694+0000 7fa155400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da8f8466b0 0x55da8fc12c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.694+0000 7fa155e00640 1 -- 172.21.15.32:0/3349894006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da8f867390 msgr2=0x55da8fc19b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:25.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.694+0000 7fa155e00640 1 --2- 172.21.15.32:0/3349894006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da8f867390 0x55da8fc19b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.694+0000 7fa155e00640 1 -- 172.21.15.32:0/3349894006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da8f8466b0 msgr2=0x55da8fc12c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:25.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.694+0000 7fa155e00640 1 --2- 172.21.15.32:0/3349894006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da8f8466b0 0x55da8fc12c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.694+0000 7fa155e00640 1 -- 172.21.15.32:0/3349894006 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da8f9497d0 con 0x55da8f83e4d0 2024-04-30T08:36:25.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.694+0000 7fa155400640 1 --2- 172.21.15.32:0/3349894006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da8f8466b0 0x55da8fc12c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:36:25.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.695+0000 7fa155e00640 1 --2- 172.21.15.32:0/3349894006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55da8f83e4d0 0x55da8fc126c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa1480a4510 tx=0x7fa148010f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e9b35bfa55ed87d server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:25.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.695+0000 7fa137e00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:36:25.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.695+0000 7fa137e00640 1 -- 172.21.15.32:0/3349894006 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1480a72a0 con 0x55da8f83e4d0 2024-04-30T08:36:25.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.695+0000 7fa137e00640 1 -- 172.21.15.32:0/3349894006 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fa1480afc70 con 0x55da8f83e4d0 2024-04-30T08:36:25.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.695+0000 7fa163de4740 1 -- 172.21.15.32:0/3349894006 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa144038040 con 0x55da8f83e4d0 2024-04-30T08:36:25.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.696+0000 7fa137e00640 1 -- 172.21.15.32:0/3349894006 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa1480ab650 con 0x55da8f83e4d0 2024-04-30T08:36:25.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.696+0000 7fa163de4740 10 client.25952 fetch_fsmap learned FSMap version 28 2024-04-30T08:36:25.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.696+0000 7fa163de4740 1 -- 172.21.15.32:0/3349894006 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55da8fc1cb00 con 0x55da8f83e4d0 2024-04-30T08:36:25.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.696+0000 7fa137e00640 1 -- 172.21.15.32:0/3349894006 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa144038040 con 0x55da8f83e4d0 2024-04-30T08:36:25.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.696+0000 7fa137e00640 1 -- 172.21.15.32:0/3349894006 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa1480ab120 con 0x55da8f83e4d0 2024-04-30T08:36:25.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.696+0000 7fa163de4740 10 client.25952 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:36:25.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.696+0000 7fa163de4740 10 client.25952 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-30T08:36:25.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.696+0000 7fa163de4740 20 client.25952 populate_metadata read hostname 'smithi032' 2024-04-30T08:36:25.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.696+0000 7fa163de4740 1 --2- 172.21.15.32:0/3349894006 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x55da8f825f60 0x55da8fc200f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.696+0000 7fa163de4740 4 client.25952 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:36:25.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.696+0000 7fa163de4740 1 -- 172.21.15.32:0/3349894006 --> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55da8fb7b960 con 0x55da8f825f60 2024-04-30T08:36:25.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.697+0000 7fa155400640 1 --2- 172.21.15.32:0/3349894006 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x55da8f825f60 0x55da8fc200f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.698+0000 7fa155400640 1 --2- 172.21.15.32:0/3349894006 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x55da8f825f60 0x55da8fc200f0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=95a7e19e2afdb85f server_cookie=412a41cd75e435b2 in_seq=0 out_seq=0 2024-04-30T08:36:25.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.698+0000 7fa137e00640 10 client.25952 ms_handle_connect on v2:172.21.15.189:6838/352970107 2024-04-30T08:36:25.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.698+0000 7fa137e00640 1 -- 172.21.15.32:0/3349894006 <== mds.2 v2:172.21.15.189:6838/352970107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55da8fb7b960 con 0x55da8f825f60 2024-04-30T08:36:25.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.698+0000 7fa137e00640 10 client.25952 handle_command_reply: tid=0 2024-04-30T08:36:25.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.699+0000 7fa163de4740 1 client.25952 shutdown 2024-04-30T08:36:25.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.699+0000 7fa163de4740 1 -- 172.21.15.32:0/3349894006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55da8f83e4d0 msgr2=0x55da8fc126c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:25.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.699+0000 7fa163de4740 1 --2- 172.21.15.32:0/3349894006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55da8f83e4d0 0x55da8fc126c0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fa1480a4510 tx=0x7fa148010f50 comp rx=0 tx=0).stop 2024-04-30T08:36:25.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.699+0000 7fa163de4740 1 -- 172.21.15.32:0/3349894006 shutdown_connections 2024-04-30T08:36:25.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.699+0000 7fa163de4740 1 --2- 172.21.15.32:0/3349894006 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x55da8f825f60 0x55da8fc200f0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.699+0000 7fa163de4740 1 --2- 172.21.15.32:0/3349894006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da8f867390 0x55da8fc19b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.699+0000 7fa163de4740 1 --2- 172.21.15.32:0/3349894006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da8f8466b0 0x55da8fc12c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.699+0000 7fa137e00640 0 client.25952 ms_handle_reset on v2:172.21.15.189:6838/352970107 2024-04-30T08:36:25.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.699+0000 7fa163de4740 1 --2- 172.21.15.32:0/3349894006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55da8f83e4d0 0x55da8fc126c0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.699+0000 7fa163de4740 1 -- 172.21.15.32:0/3349894006 >> 172.21.15.32:0/3349894006 conn(0x55da8f85b4e0 msgr2=0x55da8fc03100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:25.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.699+0000 7fa163de4740 1 -- 172.21.15.32:0/3349894006 shutdown_connections 2024-04-30T08:36:25.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.699+0000 7fa163de4740 1 -- 172.21.15.32:0/3349894006 wait complete. 2024-04-30T08:36:25.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.699+0000 7fa163de4740 20 client.25952 trim_cache size 0 max 16384 2024-04-30T08:36:25.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.728+0000 7fa163de4740 1 Processor -- start 2024-04-30T08:36:25.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.729+0000 7fa163de4740 1 -- start start 2024-04-30T08:36:25.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.729+0000 7fa163de4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da8f83e4d0 0x55da8fc113f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.729+0000 7fa163de4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da8f8466b0 0x55da8fc026d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.729+0000 7fa163de4740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55da8f867390 0x55da8fc02c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.729+0000 7fa163de4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa1581bb770 con 0x55da8f8466b0 2024-04-30T08:36:25.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.729+0000 7fa163de4740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa1581bd2b0 con 0x55da8f867390 2024-04-30T08:36:25.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.729+0000 7fa163de4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa1581bbe40 con 0x55da8f83e4d0 2024-04-30T08:36:25.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.730+0000 7fa156800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55da8f867390 0x55da8fc02c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.730+0000 7fa156800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55da8f867390 0x55da8fc02c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:60864/0 (socket says 172.21.15.32:60864) 2024-04-30T08:36:25.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.730+0000 7fa156800640 1 -- 172.21.15.32:0/2436685600 learned_addr learned my addr 172.21.15.32:0/2436685600 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:25.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.730+0000 7fa155e00640 1 --2- 172.21.15.32:0/2436685600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da8f83e4d0 0x55da8fc113f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.730+0000 7fa155400640 1 --2- 172.21.15.32:0/2436685600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da8f8466b0 0x55da8fc026d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.730+0000 7fa155e00640 1 -- 172.21.15.32:0/2436685600 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55da8f867390 msgr2=0x55da8fc02c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:25.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.730+0000 7fa155e00640 1 --2- 172.21.15.32:0/2436685600 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55da8f867390 0x55da8fc02c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.730+0000 7fa155e00640 1 -- 172.21.15.32:0/2436685600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da8f8466b0 msgr2=0x55da8fc026d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:25.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.730+0000 7fa155e00640 1 --2- 172.21.15.32:0/2436685600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da8f8466b0 0x55da8fc026d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.730+0000 7fa155e00640 1 -- 172.21.15.32:0/2436685600 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55da8fbff8e0 con 0x55da8f83e4d0 2024-04-30T08:36:25.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.731+0000 7fa155e00640 1 --2- 172.21.15.32:0/2436685600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da8f83e4d0 0x55da8fc113f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa1480a7230 tx=0x7fa14800cf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=daba39f905829dd3 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:25.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.731+0000 7fa142a00640 1 -- 172.21.15.32:0/2436685600 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1480b0100 con 0x55da8f83e4d0 2024-04-30T08:36:25.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.731+0000 7fa142a00640 1 -- 172.21.15.32:0/2436685600 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1480a6780 con 0x55da8f83e4d0 2024-04-30T08:36:25.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.731+0000 7fa142a00640 1 -- 172.21.15.32:0/2436685600 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1480c1b90 con 0x55da8f83e4d0 2024-04-30T08:36:25.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.732+0000 7fa163de4740 1 -- 172.21.15.32:0/2436685600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da8f83e4d0 msgr2=0x55da8fc113f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:25.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.732+0000 7fa163de4740 1 --2- 172.21.15.32:0/2436685600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da8f83e4d0 0x55da8fc113f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa1480a7230 tx=0x7fa14800cf20 comp rx=0 tx=0).stop 2024-04-30T08:36:25.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.732+0000 7fa163de4740 1 -- 172.21.15.32:0/2436685600 shutdown_connections 2024-04-30T08:36:25.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.732+0000 7fa163de4740 1 --2- 172.21.15.32:0/2436685600 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55da8f867390 0x55da8fc02c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.732+0000 7fa163de4740 1 --2- 172.21.15.32:0/2436685600 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da8f8466b0 0x55da8fc026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.732+0000 7fa163de4740 1 --2- 172.21.15.32:0/2436685600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da8f83e4d0 0x55da8fc113f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.732+0000 7fa163de4740 1 -- 172.21.15.32:0/2436685600 >> 172.21.15.32:0/2436685600 conn(0x55da8f85b4e0 msgr2=0x55da8fc10ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:25.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.732+0000 7fa163de4740 1 -- 172.21.15.32:0/2436685600 shutdown_connections 2024-04-30T08:36:25.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.732+0000 7fa163de4740 1 -- 172.21.15.32:0/2436685600 wait complete. 2024-04-30T08:36:25.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.733+0000 7fa163de4740 1 -- start start 2024-04-30T08:36:25.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.733+0000 7fa163de4740 1 Processor -- start 2024-04-30T08:36:25.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.734+0000 7fa163de4740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55da8f83e4d0 0x55da8fc164e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.734+0000 7fa163de4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da8f8466b0 0x55da8fc16a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.734+0000 7fa163de4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da8f867390 0x55da8fc17900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.734+0000 7fa163de4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa1581bcbe0 con 0x55da8f867390 2024-04-30T08:36:25.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.734+0000 7fa163de4740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa1581bc510 con 0x55da8f83e4d0 2024-04-30T08:36:25.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.734+0000 7fa163de4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa1581b97d0 con 0x55da8f8466b0 2024-04-30T08:36:25.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.734+0000 7fa163de4740 10 client.0 _ll_register_callbacks cb 0x7fa1480046c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:36:25.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.734+0000 7fa156800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da8f867390 0x55da8fc17900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.734+0000 7fa155e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55da8f83e4d0 0x55da8fc164e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.734+0000 7fa155400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da8f8466b0 0x55da8fc16a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.734+0000 7fa155400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da8f8466b0 0x55da8fc16a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:43048/0 (socket says 172.21.15.32:43048) 2024-04-30T08:36:25.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.734+0000 7fa156800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da8f867390 0x55da8fc17900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34354/0 (socket says 172.21.15.32:34354) 2024-04-30T08:36:25.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.734+0000 7fa155400640 1 -- 172.21.15.32:0/1211909447 learned_addr learned my addr 172.21.15.32:0/1211909447 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:25.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.735+0000 7fa155e00640 1 -- 172.21.15.32:0/1211909447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da8f8466b0 msgr2=0x55da8fc16a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:25.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.735+0000 7fa155e00640 1 --2- 172.21.15.32:0/1211909447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da8f8466b0 0x55da8fc16a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.735+0000 7fa155e00640 1 -- 172.21.15.32:0/1211909447 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da8f867390 msgr2=0x55da8fc17900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:25.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.735+0000 7fa155e00640 1 --2- 172.21.15.32:0/1211909447 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da8f867390 0x55da8fc17900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.735+0000 7fa155e00640 1 -- 172.21.15.32:0/1211909447 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55da8fc17ec0 con 0x55da8f83e4d0 2024-04-30T08:36:25.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.735+0000 7fa155e00640 1 --2- 172.21.15.32:0/1211909447 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55da8f83e4d0 0x55da8fc164e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa1480a3320 tx=0x7fa1480a7ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b814d13fdd1b0683 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:25.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.735+0000 7fa137e00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:36:25.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.735+0000 7fa137e00640 1 -- 172.21.15.32:0/1211909447 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1480ab290 con 0x55da8f83e4d0 2024-04-30T08:36:25.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.735+0000 7fa163de4740 1 -- 172.21.15.32:0/1211909447 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa1480a6780 con 0x55da8f83e4d0 2024-04-30T08:36:25.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.735+0000 7fa137e00640 1 -- 172.21.15.32:0/1211909447 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fa1480a3890 con 0x55da8f83e4d0 2024-04-30T08:36:25.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.736+0000 7fa137e00640 1 -- 172.21.15.32:0/1211909447 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fa1480a72a0 con 0x55da8f83e4d0 2024-04-30T08:36:25.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.736+0000 7fa163de4740 10 client.25958 fetch_fsmap learned FSMap version 28 2024-04-30T08:36:25.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.736+0000 7fa163de4740 1 -- 172.21.15.32:0/1211909447 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55da8fc19140 con 0x55da8f83e4d0 2024-04-30T08:36:25.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.736+0000 7fa137e00640 1 -- 172.21.15.32:0/1211909447 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa1480a6780 con 0x55da8f83e4d0 2024-04-30T08:36:25.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.736+0000 7fa137e00640 1 -- 172.21.15.32:0/1211909447 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fa1480c80d0 con 0x55da8f83e4d0 2024-04-30T08:36:25.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.736+0000 7fa163de4740 10 client.25958 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:36:25.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.736+0000 7fa163de4740 10 client.25958 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-30T08:36:25.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.736+0000 7fa163de4740 20 client.25958 populate_metadata read hostname 'smithi032' 2024-04-30T08:36:25.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.736+0000 7fa163de4740 1 --2- 172.21.15.32:0/1211909447 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x55da8f7fb3a0 0x55da8fc8a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:25.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.736+0000 7fa163de4740 4 client.25958 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-30T08:36:25.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.736+0000 7fa163de4740 1 -- 172.21.15.32:0/1211909447 --> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55da8fb757d0 con 0x55da8f7fb3a0 2024-04-30T08:36:25.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.737+0000 7fa155400640 1 --2- 172.21.15.32:0/1211909447 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x55da8f7fb3a0 0x55da8fc8a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:25.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.737+0000 7fa155400640 1 --2- 172.21.15.32:0/1211909447 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x55da8f7fb3a0 0x55da8fc8a5c0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=28f6d7d327cc094f server_cookie=fd3f103479896294 in_seq=0 out_seq=0 2024-04-30T08:36:25.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.737+0000 7fa137e00640 10 client.25958 ms_handle_connect on v2:172.21.15.189:6838/352970107 2024-04-30T08:36:25.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.738+0000 7fa137e00640 1 -- 172.21.15.32:0/1211909447 <== mds.2 v2:172.21.15.189:6838/352970107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55da8fb757d0 con 0x55da8f7fb3a0 2024-04-30T08:36:25.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.738+0000 7fa137e00640 10 client.25958 handle_command_reply: tid=0 2024-04-30T08:36:25.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.738+0000 7fa163de4740 1 client.25958 shutdown 2024-04-30T08:36:25.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.738+0000 7fa163de4740 1 -- 172.21.15.32:0/1211909447 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55da8f83e4d0 msgr2=0x55da8fc164e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:25.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.738+0000 7fa163de4740 1 --2- 172.21.15.32:0/1211909447 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55da8f83e4d0 0x55da8fc164e0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fa1480a3320 tx=0x7fa1480a7ae0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.738+0000 7fa163de4740 1 -- 172.21.15.32:0/1211909447 shutdown_connections 2024-04-30T08:36:25.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.738+0000 7fa163de4740 1 --2- 172.21.15.32:0/1211909447 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x55da8f7fb3a0 0x55da8fc8a5c0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.738+0000 7fa163de4740 1 --2- 172.21.15.32:0/1211909447 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55da8f867390 0x55da8fc17900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.738+0000 7fa163de4740 1 --2- 172.21.15.32:0/1211909447 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55da8f8466b0 0x55da8fc16a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.738+0000 7fa163de4740 1 --2- 172.21.15.32:0/1211909447 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55da8f83e4d0 0x55da8fc164e0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.738+0000 7fa163de4740 1 -- 172.21.15.32:0/1211909447 >> 172.21.15.32:0/1211909447 conn(0x55da8f85b4e0 msgr2=0x55da8fc026d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:25.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.738+0000 7fa137e00640 0 client.25958 ms_handle_reset on v2:172.21.15.189:6838/352970107 2024-04-30T08:36:25.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.739+0000 7fa163de4740 1 -- 172.21.15.32:0/1211909447 shutdown_connections 2024-04-30T08:36:25.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.739+0000 7fa163de4740 1 -- 172.21.15.32:0/1211909447 wait complete. 2024-04-30T08:36:25.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.739+0000 7fa163de4740 20 client.25958 trim_cache size 0 max 16384 2024-04-30T08:36:25.742 INFO:teuthology.orchestra.run.smithi032.stdout:[] 2024-04-30T08:36:25.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.740+0000 7fa157200640 1 -- 172.21.15.32:0/3053075882 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa13c0afbf0 msgr2=0x7fa13c0b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:25.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.740+0000 7fa157200640 1 --2- 172.21.15.32:0/3053075882 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa13c0afbf0 0x7fa13c0b2100 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x55da8fc09900 tx=0x7fa150009500 comp rx=0 tx=0).stop 2024-04-30T08:36:25.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.740+0000 7fa157200640 1 -- 172.21.15.32:0/3053075882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1581039b0 msgr2=0x7fa1581983f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:25.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.740+0000 7fa157200640 1 --2- 172.21.15.32:0/3053075882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1581039b0 0x7fa1581983f0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fa14800e980 tx=0x7fa14800ee40 comp rx=0 tx=0).stop 2024-04-30T08:36:25.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.740+0000 7fa157200640 1 -- 172.21.15.32:0/3053075882 shutdown_connections 2024-04-30T08:36:25.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.740+0000 7fa157200640 1 --2- 172.21.15.32:0/3053075882 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa13c0afbf0 0x7fa13c0b2100 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.740+0000 7fa157200640 1 --2- 172.21.15.32:0/3053075882 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa1581098f0 0x7fa158194f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.740+0000 7fa157200640 1 --2- 172.21.15.32:0/3053075882 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa158104bf0 0x7fa158198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.740+0000 7fa157200640 1 --2- 172.21.15.32:0/3053075882 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1581039b0 0x7fa1581983f0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:25.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.740+0000 7fa157200640 1 -- 172.21.15.32:0/3053075882 >> 172.21.15.32:0/3053075882 conn(0x7fa1580feba0 msgr2=0x7fa1580fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:25.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.740+0000 7fa157200640 1 -- 172.21.15.32:0/3053075882 shutdown_connections 2024-04-30T08:36:25.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:25.740+0000 7fa157200640 1 -- 172.21.15.32:0/3053075882 wait complete. 2024-04-30T08:36:25.767 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-30T08:36:26.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.129+0000 7f53e2e00640 1 -- 172.21.15.32:0/2465155755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53dc104bf0 msgr2=0x7f53dc105000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:26.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.129+0000 7f53e2e00640 1 --2- 172.21.15.32:0/2465155755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53dc104bf0 0x7f53dc105000 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f53c8009860 tx=0x7f53c802a340 comp rx=0 tx=0).stop 2024-04-30T08:36:26.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.130+0000 7f53e2e00640 1 -- 172.21.15.32:0/2465155755 shutdown_connections 2024-04-30T08:36:26.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.130+0000 7f53e2e00640 1 --2- 172.21.15.32:0/2465155755 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53dc1098f0 0x7f53dc10bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.130+0000 7f53e2e00640 1 --2- 172.21.15.32:0/2465155755 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53dc104bf0 0x7f53dc105000 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.130+0000 7f53e2e00640 1 --2- 172.21.15.32:0/2465155755 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53dc1039b0 0x7f53dc1046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.130+0000 7f53e2e00640 1 -- 172.21.15.32:0/2465155755 >> 172.21.15.32:0/2465155755 conn(0x7f53dc0feba0 msgr2=0x7f53dc101050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:26.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.130+0000 7f53e2e00640 1 -- 172.21.15.32:0/2465155755 shutdown_connections 2024-04-30T08:36:26.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.130+0000 7f53e2e00640 1 -- 172.21.15.32:0/2465155755 wait complete. 2024-04-30T08:36:26.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.131+0000 7f53e2e00640 1 Processor -- start 2024-04-30T08:36:26.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.131+0000 7f53e2e00640 1 -- start start 2024-04-30T08:36:26.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.131+0000 7f53e2e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53dc1039b0 0x7f53dc197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.131+0000 7f53e2e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53dc104bf0 0x7f53dc198450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.132+0000 7f53e2e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53dc1098f0 0x7f53dc19f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.132+0000 7f53e2e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f53dc10d300 con 0x7f53dc104bf0 2024-04-30T08:36:26.132 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.132+0000 7f53e2e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f53dc10d180 con 0x7f53dc1039b0 2024-04-30T08:36:26.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.132+0000 7f53e2e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f53dc10d480 con 0x7f53dc1098f0 2024-04-30T08:36:26.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.132+0000 7f53e1000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53dc104bf0 0x7f53dc198450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:26.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.132+0000 7f53e2400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53dc1098f0 0x7f53dc19f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:26.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.132+0000 7f53e1a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53dc1039b0 0x7f53dc197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:26.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.132+0000 7f53e2400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53dc1098f0 0x7f53dc19f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:43070/0 (socket says 172.21.15.32:43070) 2024-04-30T08:36:26.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.132+0000 7f53e2400640 1 -- 172.21.15.32:0/1120171829 learned_addr learned my addr 172.21.15.32:0/1120171829 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:26.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.132+0000 7f53e1a00640 1 -- 172.21.15.32:0/1120171829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53dc1098f0 msgr2=0x7f53dc19f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:26.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.132+0000 7f53e1a00640 1 --2- 172.21.15.32:0/1120171829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53dc1098f0 0x7f53dc19f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.132+0000 7f53e1a00640 1 -- 172.21.15.32:0/1120171829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53dc104bf0 msgr2=0x7f53dc198450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:26.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.132+0000 7f53e1a00640 1 --2- 172.21.15.32:0/1120171829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53dc104bf0 0x7f53dc198450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.132+0000 7f53e1a00640 1 -- 172.21.15.32:0/1120171829 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53dc19fa50 con 0x7f53dc1039b0 2024-04-30T08:36:26.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.133+0000 7f53e1a00640 1 --2- 172.21.15.32:0/1120171829 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53dc1039b0 0x7f53dc197f10 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f53dc103280 tx=0x7f53d0009ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:26.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.133+0000 7f53da000640 1 -- 172.21.15.32:0/1120171829 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d0007b00 con 0x7f53dc1039b0 2024-04-30T08:36:26.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.133+0000 7f53e2e00640 1 -- 172.21.15.32:0/1120171829 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53dc19fd40 con 0x7f53dc1039b0 2024-04-30T08:36:26.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.133+0000 7f53da000640 1 -- 172.21.15.32:0/1120171829 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53d0007ca0 con 0x7f53dc1039b0 2024-04-30T08:36:26.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.133+0000 7f53da000640 1 -- 172.21.15.32:0/1120171829 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d0003e10 con 0x7f53dc1039b0 2024-04-30T08:36:26.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.134+0000 7f53e2e00640 1 -- 172.21.15.32:0/1120171829 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53dc1a0220 con 0x7f53dc1039b0 2024-04-30T08:36:26.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.137+0000 7f53eb420740 1 Processor -- start 2024-04-30T08:36:26.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.137+0000 7f53eb420740 1 -- start start 2024-04-30T08:36:26.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.138+0000 7f53eb420740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56126f493390 0x56126f83cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.138+0000 7f53eb420740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56126f46a4d0 0x56126f83d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.138+0000 7f53eb420740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56126f4726b0 0x56126f8443c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.138+0000 7f53eb420740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56126f799d50 con 0x56126f493390 2024-04-30T08:36:26.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.138+0000 7f53eb420740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x56126f73f0c0 con 0x56126f46a4d0 2024-04-30T08:36:26.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.138+0000 7f53eb420740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56126f795d10 con 0x56126f4726b0 2024-04-30T08:36:26.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.138+0000 7f53e2400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56126f493390 0x56126f83cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:26.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.138+0000 7f53e2400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56126f493390 0x56126f83cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34384/0 (socket says 172.21.15.32:34384) 2024-04-30T08:36:26.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.138+0000 7f53e2400640 1 -- 172.21.15.32:0/1454867720 learned_addr learned my addr 172.21.15.32:0/1454867720 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:26.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.138+0000 7f53e1000640 1 --2- 172.21.15.32:0/1454867720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56126f4726b0 0x56126f8443c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:26.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.139+0000 7f53e2400640 1 -- 172.21.15.32:0/1454867720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56126f4726b0 msgr2=0x56126f8443c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:26.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.139+0000 7f53e2400640 1 --2- 172.21.15.32:0/1454867720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56126f4726b0 0x56126f8443c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.139+0000 7f53e2400640 1 -- 172.21.15.32:0/1454867720 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56126f46a4d0 msgr2=0x56126f83d4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-30T08:36:26.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.139+0000 7f53e2400640 1 --2- 172.21.15.32:0/1454867720 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56126f46a4d0 0x56126f83d4c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.139+0000 7f53e2400640 1 -- 172.21.15.32:0/1454867720 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56126f4ab3f0 con 0x56126f493390 2024-04-30T08:36:26.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.139+0000 7f53e2400640 1 --2- 172.21.15.32:0/1454867720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56126f493390 0x56126f83cf40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f53d40104d0 tx=0x7f53d4010990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7fcb902a1bd1fd9 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:26.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.139+0000 7f53da000640 1 -- 172.21.15.32:0/1120171829 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f53d0020030 con 0x7f53dc1039b0 2024-04-30T08:36:26.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.140+0000 7f53cea00640 1 -- 172.21.15.32:0/1454867720 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d4011760 con 0x56126f493390 2024-04-30T08:36:26.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.140+0000 7f53cea00640 1 -- 172.21.15.32:0/1454867720 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53d4011db0 con 0x56126f493390 2024-04-30T08:36:26.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.140+0000 7f53cea00640 1 -- 172.21.15.32:0/1454867720 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d40193d0 con 0x56126f493390 2024-04-30T08:36:26.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.141+0000 7f53eb420740 1 -- 172.21.15.32:0/1454867720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56126f493390 msgr2=0x56126f83cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:26.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.141+0000 7f53eb420740 1 --2- 172.21.15.32:0/1454867720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56126f493390 0x56126f83cf40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f53d40104d0 tx=0x7f53d4010990 comp rx=0 tx=0).stop 2024-04-30T08:36:26.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.141+0000 7f53eb420740 1 -- 172.21.15.32:0/1454867720 shutdown_connections 2024-04-30T08:36:26.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.141+0000 7f53eb420740 1 --2- 172.21.15.32:0/1454867720 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56126f4726b0 0x56126f8443c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.141+0000 7f53eb420740 1 --2- 172.21.15.32:0/1454867720 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56126f46a4d0 0x56126f83d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.141+0000 7f53eb420740 1 --2- 172.21.15.32:0/1454867720 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56126f493390 0x56126f83cf40 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.142 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.141+0000 7f53eb420740 1 -- 172.21.15.32:0/1454867720 >> 172.21.15.32:0/1454867720 conn(0x56126f4874e0 msgr2=0x56126f8388d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:26.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.142+0000 7f53eb420740 1 -- 172.21.15.32:0/1454867720 shutdown_connections 2024-04-30T08:36:26.143 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.142+0000 7f53eb420740 1 -- 172.21.15.32:0/1454867720 wait complete. 2024-04-30T08:36:26.144 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.142+0000 7f53da000640 1 --2- 172.21.15.32:0/1120171829 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f53c40afa80 0x7f53c40b1f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.142+0000 7f53da000640 1 -- 172.21.15.32:0/1120171829 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f53d00de180 con 0x7f53dc1039b0 2024-04-30T08:36:26.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.143+0000 7f53e1000640 1 --2- 172.21.15.32:0/1120171829 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f53c40afa80 0x7f53c40b1f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:26.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.143+0000 7f53eb420740 1 -- start start 2024-04-30T08:36:26.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.143+0000 7f53eb420740 1 Processor -- start 2024-04-30T08:36:26.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.144+0000 7f53eb420740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56126f46a4d0 0x56126f846e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.144+0000 7f53eb420740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56126f4726b0 0x56126f8473d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.144+0000 7f53eb420740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56126f493390 0x56126f85f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.144+0000 7f53eb420740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56126f844e10 con 0x56126f493390 2024-04-30T08:36:26.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.144+0000 7f53eb420740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x56126f752b50 con 0x56126f4726b0 2024-04-30T08:36:26.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.144+0000 7f53eb420740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56126f844f90 con 0x56126f46a4d0 2024-04-30T08:36:26.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.144+0000 7f53e1000640 1 --2- 172.21.15.32:0/1120171829 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f53c40afa80 0x7f53c40b1f90 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f53c802a930 tx=0x7f53c8030450 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:26.145 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.144+0000 7f53eb420740 10 client.0 _ll_register_callbacks cb 0x56126f71c420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:36:26.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.145+0000 7f53e1000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56126f4726b0 0x56126f8473d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:26.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.145+0000 7f53e1000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56126f4726b0 0x56126f8473d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:60894/0 (socket says 172.21.15.32:60894) 2024-04-30T08:36:26.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.145+0000 7f53e1000640 1 -- 172.21.15.32:0/3791812233 learned_addr learned my addr 172.21.15.32:0/3791812233 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:26.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.145+0000 7f53e1a00640 1 --2- 172.21.15.32:0/3791812233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56126f46a4d0 0x56126f846e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:26.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.145+0000 7f53e2400640 1 --2- 172.21.15.32:0/3791812233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56126f493390 0x56126f85f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:26.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.145+0000 7f53e1000640 1 -- 172.21.15.32:0/3791812233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56126f46a4d0 msgr2=0x56126f846e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:26.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.145+0000 7f53e1000640 1 --2- 172.21.15.32:0/3791812233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56126f46a4d0 0x56126f846e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.145+0000 7f53e1000640 1 -- 172.21.15.32:0/3791812233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56126f493390 msgr2=0x56126f85f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:26.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.145+0000 7f53e1000640 1 --2- 172.21.15.32:0/3791812233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56126f493390 0x56126f85f340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.148 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.145+0000 7f53e1000640 1 -- 172.21.15.32:0/3791812233 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56126f5757d0 con 0x56126f4726b0 2024-04-30T08:36:26.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.145+0000 7f53e1a00640 1 --2- 172.21.15.32:0/3791812233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56126f46a4d0 0x56126f846e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:36:26.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.145+0000 7f53e1000640 1 --2- 172.21.15.32:0/3791812233 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56126f4726b0 0x56126f8473d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f53c8009250 tx=0x7f53c802a340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0b5245425f62e51 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:26.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.145+0000 7f53c3400640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:36:26.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.146+0000 7f53c3400640 1 -- 172.21.15.32:0/3791812233 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c8037040 con 0x56126f4726b0 2024-04-30T08:36:26.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.146+0000 7f53c3400640 1 -- 172.21.15.32:0/3791812233 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f53c8003a00 con 0x56126f4726b0 2024-04-30T08:36:26.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.146+0000 7f53eb420740 1 -- 172.21.15.32:0/3791812233 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53d4011db0 con 0x56126f4726b0 2024-04-30T08:36:26.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.146+0000 7f53c3400640 1 -- 172.21.15.32:0/3791812233 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f53c8003c00 con 0x56126f4726b0 2024-04-30T08:36:26.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.146+0000 7f53eb420740 10 client.25967 fetch_fsmap learned FSMap version 28 2024-04-30T08:36:26.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.146+0000 7f53eb420740 1 -- 172.21.15.32:0/3791812233 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56126f8622e0 con 0x56126f4726b0 2024-04-30T08:36:26.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.147+0000 7f53c3400640 1 -- 172.21.15.32:0/3791812233 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53d4011db0 con 0x56126f4726b0 2024-04-30T08:36:26.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.147+0000 7f53c3400640 1 -- 172.21.15.32:0/3791812233 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f53c8049440 con 0x56126f4726b0 2024-04-30T08:36:26.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.147+0000 7f53eb420740 10 client.25967 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:36:26.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.147+0000 7f53eb420740 10 client.25967 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-30T08:36:26.149 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.147+0000 7f53eb420740 20 client.25967 populate_metadata read hostname 'smithi032' 2024-04-30T08:36:26.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.147+0000 7f53eb420740 1 --2- 172.21.15.32:0/3791812233 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x56126f451f60 0x56126f865920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.147+0000 7f53eb420740 4 client.25967 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:36:26.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.147+0000 7f53eb420740 1 -- 172.21.15.32:0/3791812233 --> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56126f7a7960 con 0x56126f451f60 2024-04-30T08:36:26.150 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.147+0000 7f53e1a00640 1 --2- 172.21.15.32:0/3791812233 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x56126f451f60 0x56126f865920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:26.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.148+0000 7f53c3400640 10 client.25967 ms_handle_connect on v2:172.21.15.32:6838/4160386524 2024-04-30T08:36:26.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.148+0000 7f53e1a00640 1 --2- 172.21.15.32:0/3791812233 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x56126f451f60 0x56126f865920 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e3b4a935c4d305ab server_cookie=99218e376f7beb29 in_seq=0 out_seq=0 2024-04-30T08:36:26.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.149+0000 7f53c3400640 1 -- 172.21.15.32:0/3791812233 <== mds.1 v2:172.21.15.32:6838/4160386524 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x56126f7a7960 con 0x56126f451f60 2024-04-30T08:36:26.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.149+0000 7f53c3400640 10 client.25967 handle_command_reply: tid=0 2024-04-30T08:36:26.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.149+0000 7f53eb420740 1 client.25967 shutdown 2024-04-30T08:36:26.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.149+0000 7f53eb420740 1 -- 172.21.15.32:0/3791812233 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56126f4726b0 msgr2=0x56126f8473d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:26.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.149+0000 7f53eb420740 1 --2- 172.21.15.32:0/3791812233 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56126f4726b0 0x56126f8473d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f53c8009250 tx=0x7f53c802a340 comp rx=0 tx=0).stop 2024-04-30T08:36:26.151 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.150+0000 7f53eb420740 1 -- 172.21.15.32:0/3791812233 shutdown_connections 2024-04-30T08:36:26.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.150+0000 7f53eb420740 1 --2- 172.21.15.32:0/3791812233 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x56126f451f60 0x56126f865920 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.150+0000 7f53eb420740 1 --2- 172.21.15.32:0/3791812233 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56126f493390 0x56126f85f340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.150+0000 7f53eb420740 1 --2- 172.21.15.32:0/3791812233 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56126f4726b0 0x56126f8473d0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.150+0000 7f53eb420740 1 --2- 172.21.15.32:0/3791812233 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56126f46a4d0 0x56126f846e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.150+0000 7f53eb420740 1 -- 172.21.15.32:0/3791812233 >> 172.21.15.32:0/3791812233 conn(0x56126f4874e0 msgr2=0x56126f82f100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:26.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.150+0000 7f53c3400640 0 client.25967 ms_handle_reset on v2:172.21.15.32:6838/4160386524 2024-04-30T08:36:26.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.150+0000 7f53eb420740 1 -- 172.21.15.32:0/3791812233 shutdown_connections 2024-04-30T08:36:26.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.150+0000 7f53eb420740 1 -- 172.21.15.32:0/3791812233 wait complete. 2024-04-30T08:36:26.152 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.150+0000 7f53eb420740 20 client.25967 trim_cache size 0 max 16384 2024-04-30T08:36:26.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.167+0000 7f53eb420740 1 Processor -- start 2024-04-30T08:36:26.167 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.167+0000 7f53eb420740 1 -- start start 2024-04-30T08:36:26.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.167+0000 7f53eb420740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56126f46a4d0 0x56126f838200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.167+0000 7f53eb420740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56126f4726b0 0x56126f838740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.167+0000 7f53eb420740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56126f493390 0x56126f8454b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.167+0000 7f53eb420740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f53dc1a20d0 con 0x56126f4726b0 2024-04-30T08:36:26.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.167+0000 7f53eb420740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f53dc1a3c10 con 0x56126f493390 2024-04-30T08:36:26.168 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.167+0000 7f53eb420740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f53dc1a27a0 con 0x56126f46a4d0 2024-04-30T08:36:26.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.168+0000 7f53e1000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56126f4726b0 0x56126f838740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:26.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.168+0000 7f53e1000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56126f4726b0 0x56126f838740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34394/0 (socket says 172.21.15.32:34394) 2024-04-30T08:36:26.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.168+0000 7f53e1000640 1 -- 172.21.15.32:0/2140208228 learned_addr learned my addr 172.21.15.32:0/2140208228 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:26.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.168+0000 7f53e1a00640 1 --2- 172.21.15.32:0/2140208228 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56126f46a4d0 0x56126f838200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:26.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.168+0000 7f53e2400640 1 --2- 172.21.15.32:0/2140208228 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56126f493390 0x56126f8454b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:26.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.169+0000 7f53e1000640 1 -- 172.21.15.32:0/2140208228 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56126f46a4d0 msgr2=0x56126f838200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:26.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.169+0000 7f53e1000640 1 --2- 172.21.15.32:0/2140208228 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56126f46a4d0 0x56126f838200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.169+0000 7f53e1000640 1 -- 172.21.15.32:0/2140208228 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56126f493390 msgr2=0x56126f8454b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:26.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.169+0000 7f53e1000640 1 --2- 172.21.15.32:0/2140208228 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56126f493390 0x56126f8454b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.169+0000 7f53e1000640 1 -- 172.21.15.32:0/2140208228 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56126f83db50 con 0x56126f4726b0 2024-04-30T08:36:26.170 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.169+0000 7f53e1000640 1 --2- 172.21.15.32:0/2140208228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56126f4726b0 0x56126f838740 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f53c8000c00 tx=0x7f53c80303b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bb78fdf91afa53f server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:26.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.169+0000 7f53ce000640 1 -- 172.21.15.32:0/2140208228 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c8060070 con 0x56126f4726b0 2024-04-30T08:36:26.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.169+0000 7f53ce000640 1 -- 172.21.15.32:0/2140208228 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53c804e070 con 0x56126f4726b0 2024-04-30T08:36:26.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.169+0000 7f53ce000640 1 -- 172.21.15.32:0/2140208228 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53c8003c00 con 0x56126f4726b0 2024-04-30T08:36:26.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.170+0000 7f53eb420740 1 -- 172.21.15.32:0/2140208228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56126f4726b0 msgr2=0x56126f838740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:26.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.170+0000 7f53eb420740 1 --2- 172.21.15.32:0/2140208228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56126f4726b0 0x56126f838740 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f53c8000c00 tx=0x7f53c80303b0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.170+0000 7f53eb420740 1 -- 172.21.15.32:0/2140208228 shutdown_connections 2024-04-30T08:36:26.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.170+0000 7f53eb420740 1 --2- 172.21.15.32:0/2140208228 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56126f493390 0x56126f8454b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.170+0000 7f53eb420740 1 --2- 172.21.15.32:0/2140208228 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56126f4726b0 0x56126f838740 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.170+0000 7f53eb420740 1 --2- 172.21.15.32:0/2140208228 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56126f46a4d0 0x56126f838200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.170+0000 7f53eb420740 1 -- 172.21.15.32:0/2140208228 >> 172.21.15.32:0/2140208228 conn(0x56126f4874e0 msgr2=0x56126f841320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:26.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.171+0000 7f53eb420740 1 -- 172.21.15.32:0/2140208228 shutdown_connections 2024-04-30T08:36:26.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.171+0000 7f53eb420740 1 -- 172.21.15.32:0/2140208228 wait complete. 2024-04-30T08:36:26.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.171+0000 7f53eb420740 1 -- start start 2024-04-30T08:36:26.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.171+0000 7f53eb420740 1 Processor -- start 2024-04-30T08:36:26.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.172+0000 7f53eb420740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56126f46a4d0 0x56126f874130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.172+0000 7f53eb420740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56126f4726b0 0x56126f874670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.172+0000 7f53eb420740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56126f493390 0x56126f8cb8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.172+0000 7f53eb420740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f53dc1a3540 con 0x56126f46a4d0 2024-04-30T08:36:26.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.172+0000 7f53eb420740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f53dc1a2e70 con 0x56126f493390 2024-04-30T08:36:26.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.172+0000 7f53eb420740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f53dc074a40 con 0x56126f4726b0 2024-04-30T08:36:26.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.172+0000 7f53eb420740 10 client.0 _ll_register_callbacks cb 0x7f53c80096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:36:26.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.172+0000 7f53e1a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56126f46a4d0 0x56126f874130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:26.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.172+0000 7f53e1000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56126f4726b0 0x56126f874670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:26.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.172+0000 7f53e1000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56126f4726b0 0x56126f874670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:43108/0 (socket says 172.21.15.32:43108) 2024-04-30T08:36:26.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.172+0000 7f53e1000640 1 -- 172.21.15.32:0/553866931 learned_addr learned my addr 172.21.15.32:0/553866931 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:26.174 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.172+0000 7f53e2400640 1 --2- 172.21.15.32:0/553866931 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56126f493390 0x56126f8cb8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:26.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.173+0000 7f53e1a00640 1 -- 172.21.15.32:0/553866931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56126f4726b0 msgr2=0x56126f874670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:26.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.173+0000 7f53e1a00640 1 --2- 172.21.15.32:0/553866931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56126f4726b0 0x56126f874670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.173+0000 7f53e1a00640 1 -- 172.21.15.32:0/553866931 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56126f493390 msgr2=0x56126f8cb8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:26.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.173+0000 7f53e1a00640 1 --2- 172.21.15.32:0/553866931 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56126f493390 0x56126f8cb8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.173+0000 7f53e1a00640 1 -- 172.21.15.32:0/553866931 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56126f8cbe60 con 0x56126f46a4d0 2024-04-30T08:36:26.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.173+0000 7f53e1a00640 1 --2- 172.21.15.32:0/553866931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56126f46a4d0 0x56126f874130 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f53d00db760 tx=0x7f53d00de460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53091e302b16a5a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:26.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.173+0000 7f53c3400640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:36:26.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.174+0000 7f53c3400640 1 -- 172.21.15.32:0/553866931 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53d00a61c0 con 0x56126f46a4d0 2024-04-30T08:36:26.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.174+0000 7f53c3400640 1 -- 172.21.15.32:0/553866931 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f53d00aa0b0 con 0x56126f46a4d0 2024-04-30T08:36:26.175 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.174+0000 7f53eb420740 1 -- 172.21.15.32:0/553866931 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53c804e070 con 0x56126f46a4d0 2024-04-30T08:36:26.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.174+0000 7f53c3400640 1 -- 172.21.15.32:0/553866931 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f53d00aa570 con 0x56126f46a4d0 2024-04-30T08:36:26.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.174+0000 7f53eb420740 10 client.15843 fetch_fsmap learned FSMap version 28 2024-04-30T08:36:26.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.175+0000 7f53eb420740 1 -- 172.21.15.32:0/553866931 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x56126f8cd150 con 0x56126f46a4d0 2024-04-30T08:36:26.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.175+0000 7f53c3400640 1 -- 172.21.15.32:0/553866931 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f53c804e070 con 0x56126f46a4d0 2024-04-30T08:36:26.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.175+0000 7f53c3400640 1 -- 172.21.15.32:0/553866931 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f53d00a4e90 con 0x56126f46a4d0 2024-04-30T08:36:26.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.175+0000 7f53eb420740 10 client.15843 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:36:26.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.175+0000 7f53eb420740 10 client.15843 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-30T08:36:26.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.175+0000 7f53eb420740 20 client.15843 populate_metadata read hostname 'smithi032' 2024-04-30T08:36:26.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.175+0000 7f53eb420740 1 --2- 172.21.15.32:0/553866931 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x56126f4273a0 0x56126f8cfb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.175+0000 7f53eb420740 4 client.15843 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-30T08:36:26.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.175+0000 7f53eb420740 1 -- 172.21.15.32:0/553866931 --> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56126f7a17d0 con 0x56126f4273a0 2024-04-30T08:36:26.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.176+0000 7f53e1000640 1 --2- 172.21.15.32:0/553866931 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x56126f4273a0 0x56126f8cfb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:26.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.177+0000 7f53e1000640 1 --2- 172.21.15.32:0/553866931 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x56126f4273a0 0x56126f8cfb40 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=eec3132957358d8b server_cookie=d8e31b6b68b15e89 in_seq=0 out_seq=0 2024-04-30T08:36:26.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.177+0000 7f53c3400640 10 client.15843 ms_handle_connect on v2:172.21.15.32:6838/4160386524 2024-04-30T08:36:26.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.177+0000 7f53c3400640 1 -- 172.21.15.32:0/553866931 <== mds.1 v2:172.21.15.32:6838/4160386524 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56126f7a17d0 con 0x56126f4273a0 2024-04-30T08:36:26.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.177+0000 7f53c3400640 10 client.15843 handle_command_reply: tid=0 2024-04-30T08:36:26.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.177+0000 7f53eb420740 1 client.15843 shutdown 2024-04-30T08:36:26.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.177+0000 7f53eb420740 1 -- 172.21.15.32:0/553866931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56126f46a4d0 msgr2=0x56126f874130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:26.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.177+0000 7f53eb420740 1 --2- 172.21.15.32:0/553866931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56126f46a4d0 0x56126f874130 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f53d00db760 tx=0x7f53d00de460 comp rx=0 tx=0).stop 2024-04-30T08:36:26.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.177+0000 7f53eb420740 1 -- 172.21.15.32:0/553866931 shutdown_connections 2024-04-30T08:36:26.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.177+0000 7f53eb420740 1 --2- 172.21.15.32:0/553866931 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x56126f4273a0 0x56126f8cfb40 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.178+0000 7f53c3400640 0 client.15843 ms_handle_reset on v2:172.21.15.32:6838/4160386524 2024-04-30T08:36:26.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.178+0000 7f53eb420740 1 --2- 172.21.15.32:0/553866931 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56126f493390 0x56126f8cb8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.178+0000 7f53eb420740 1 --2- 172.21.15.32:0/553866931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56126f4726b0 0x56126f874670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.178+0000 7f53eb420740 1 --2- 172.21.15.32:0/553866931 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56126f46a4d0 0x56126f874130 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.178+0000 7f53eb420740 1 -- 172.21.15.32:0/553866931 >> 172.21.15.32:0/553866931 conn(0x56126f4874e0 msgr2=0x56126f838200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:26.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.178+0000 7f53eb420740 1 -- 172.21.15.32:0/553866931 shutdown_connections 2024-04-30T08:36:26.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.178+0000 7f53eb420740 1 -- 172.21.15.32:0/553866931 wait complete. 2024-04-30T08:36:26.180 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.178+0000 7f53eb420740 20 client.15843 trim_cache size 0 max 16384 2024-04-30T08:36:26.181 INFO:teuthology.orchestra.run.smithi032.stdout:[] 2024-04-30T08:36:26.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.179+0000 7f53e2e00640 1 -- 172.21.15.32:0/1120171829 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f53c40afa80 msgr2=0x7f53c40b1f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:26.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.179+0000 7f53e2e00640 1 --2- 172.21.15.32:0/1120171829 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f53c40afa80 0x7f53c40b1f90 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f53c802a930 tx=0x7f53c8030450 comp rx=0 tx=0).stop 2024-04-30T08:36:26.181 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.179+0000 7f53e2e00640 1 -- 172.21.15.32:0/1120171829 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53dc1039b0 msgr2=0x7f53dc197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:26.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.179+0000 7f53e2e00640 1 --2- 172.21.15.32:0/1120171829 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53dc1039b0 0x7f53dc197f10 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f53dc103280 tx=0x7f53d0009ee0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.179+0000 7f53e2e00640 1 -- 172.21.15.32:0/1120171829 shutdown_connections 2024-04-30T08:36:26.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.179+0000 7f53e2e00640 1 --2- 172.21.15.32:0/1120171829 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f53c40afa80 0x7f53c40b1f90 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.179+0000 7f53e2e00640 1 --2- 172.21.15.32:0/1120171829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53dc1098f0 0x7f53dc19f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.179+0000 7f53e2e00640 1 --2- 172.21.15.32:0/1120171829 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f53dc104bf0 0x7f53dc198450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.179+0000 7f53e2e00640 1 --2- 172.21.15.32:0/1120171829 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f53dc1039b0 0x7f53dc197f10 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.179+0000 7f53e2e00640 1 -- 172.21.15.32:0/1120171829 >> 172.21.15.32:0/1120171829 conn(0x7f53dc0feba0 msgr2=0x7f53dc1058d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:26.182 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.179+0000 7f53e2e00640 1 -- 172.21.15.32:0/1120171829 shutdown_connections 2024-04-30T08:36:26.183 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.180+0000 7f53e2e00640 1 -- 172.21.15.32:0/1120171829 wait complete. 2024-04-30T08:36:26.221 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-30T08:36:26.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.481+0000 7fe6d6200640 1 -- 172.21.15.32:0/1807461845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6d010b900 msgr2=0x7fe6d010dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:26.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.481+0000 7fe6d6200640 1 --2- 172.21.15.32:0/1807461845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6d010b900 0x7fe6d010dd40 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8009ca0 tx=0x7fe6c802a8c0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.481+0000 7fe6d6200640 1 -- 172.21.15.32:0/1807461845 shutdown_connections 2024-04-30T08:36:26.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.481+0000 7fe6d6200640 1 --2- 172.21.15.32:0/1807461845 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6d010b900 0x7fe6d010dd40 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.481+0000 7fe6d6200640 1 --2- 172.21.15.32:0/1807461845 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6d0075960 0x7fe6d010b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.481+0000 7fe6d6200640 1 --2- 172.21.15.32:0/1807461845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6d0075010 0x7fe6d0075420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.481+0000 7fe6d6200640 1 -- 172.21.15.32:0/1807461845 >> 172.21.15.32:0/1807461845 conn(0x7fe6d00feba0 msgr2=0x7fe6d0101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:26.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.481+0000 7fe6d6200640 1 -- 172.21.15.32:0/1807461845 shutdown_connections 2024-04-30T08:36:26.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.481+0000 7fe6d6200640 1 -- 172.21.15.32:0/1807461845 wait complete. 2024-04-30T08:36:26.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.483+0000 7fe6d6200640 1 Processor -- start 2024-04-30T08:36:26.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.483+0000 7fe6d6200640 1 -- start start 2024-04-30T08:36:26.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.483+0000 7fe6d6200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6d0075010 0x7fe6d019c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.483+0000 7fe6d6200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6d0075960 0x7fe6d019c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.483+0000 7fe6d6200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6d010b900 0x7fe6d01a37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.483+0000 7fe6d6200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d01116a0 con 0x7fe6d0075010 2024-04-30T08:36:26.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.483+0000 7fe6d6200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d0111520 con 0x7fe6d010b900 2024-04-30T08:36:26.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.483+0000 7fe6d6200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d0111820 con 0x7fe6d0075960 2024-04-30T08:36:26.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.484+0000 7fe6d4e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6d0075010 0x7fe6d019c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:26.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.484+0000 7fe6d4e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6d0075010 0x7fe6d019c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34428/0 (socket says 172.21.15.32:34428) 2024-04-30T08:36:26.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.484+0000 7fe6d4e00640 1 -- 172.21.15.32:0/360733656 learned_addr learned my addr 172.21.15.32:0/360733656 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:26.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.484+0000 7fe6cfe00640 1 --2- 172.21.15.32:0/360733656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6d0075960 0x7fe6d019c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:26.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.484+0000 7fe6d5800640 1 --2- 172.21.15.32:0/360733656 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6d010b900 0x7fe6d01a37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:26.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.484+0000 7fe6d4e00640 1 -- 172.21.15.32:0/360733656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6d0075960 msgr2=0x7fe6d019c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:26.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.484+0000 7fe6d4e00640 1 --2- 172.21.15.32:0/360733656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6d0075960 0x7fe6d019c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.484+0000 7fe6d4e00640 1 -- 172.21.15.32:0/360733656 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6d010b900 msgr2=0x7fe6d01a37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:26.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.484+0000 7fe6d4e00640 1 --2- 172.21.15.32:0/360733656 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6d010b900 0x7fe6d01a37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.484+0000 7fe6d4e00640 1 -- 172.21.15.32:0/360733656 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6d01a3e10 con 0x7fe6d0075010 2024-04-30T08:36:26.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.485+0000 7fe6d4e00640 1 --2- 172.21.15.32:0/360733656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6d0075010 0x7fe6d019c2d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe6c400e9c0 tx=0x7fe6c400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:26.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.485+0000 7fe6cd600640 1 -- 172.21.15.32:0/360733656 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c400cd80 con 0x7fe6d0075010 2024-04-30T08:36:26.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.485+0000 7fe6cd600640 1 -- 172.21.15.32:0/360733656 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6c40054d0 con 0x7fe6d0075010 2024-04-30T08:36:26.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.485+0000 7fe6cd600640 1 -- 172.21.15.32:0/360733656 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c4010430 con 0x7fe6d0075010 2024-04-30T08:36:26.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.485+0000 7fe6d6200640 1 -- 172.21.15.32:0/360733656 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6d01a4100 con 0x7fe6d0075010 2024-04-30T08:36:26.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.485+0000 7fe6d6200640 1 -- 172.21.15.32:0/360733656 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6d01a4600 con 0x7fe6d0075010 2024-04-30T08:36:26.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.490+0000 7fe6de826740 1 Processor -- start 2024-04-30T08:36:26.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.491+0000 7fe6de826740 1 -- start start 2024-04-30T08:36:26.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.491+0000 7fe6de826740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b6c4516390 0x55b6c48bff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.491+0000 7fe6de826740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b6c44ed4d0 0x55b6c48c04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.491+0000 7fe6de826740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6c44f56b0 0x55b6c48c73c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.492+0000 7fe6de826740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b6c481cd50 con 0x55b6c44ed4d0 2024-04-30T08:36:26.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.492+0000 7fe6de826740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55b6c47c20c0 con 0x55b6c4516390 2024-04-30T08:36:26.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.492+0000 7fe6de826740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b6c4818d10 con 0x55b6c44f56b0 2024-04-30T08:36:26.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.492+0000 7fe6cfe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6c44f56b0 0x55b6c48c73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:26.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.492+0000 7fe6cfe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6c44f56b0 0x55b6c48c73c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:43130/0 (socket says 172.21.15.32:43130) 2024-04-30T08:36:26.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.492+0000 7fe6cfe00640 1 -- 172.21.15.32:0/1853935172 learned_addr learned my addr 172.21.15.32:0/1853935172 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:26.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.492+0000 7fe6d5800640 1 --2- 172.21.15.32:0/1853935172 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b6c4516390 0x55b6c48bff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:26.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.493+0000 7fe6d5800640 1 -- 172.21.15.32:0/1853935172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6c44f56b0 msgr2=0x55b6c48c73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:26.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.493+0000 7fe6d5800640 1 --2- 172.21.15.32:0/1853935172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6c44f56b0 0x55b6c48c73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.493+0000 7fe6d5800640 1 -- 172.21.15.32:0/1853935172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b6c44ed4d0 msgr2=0x55b6c48c04c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-30T08:36:26.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.493+0000 7fe6d5800640 1 --2- 172.21.15.32:0/1853935172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b6c44ed4d0 0x55b6c48c04c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.493+0000 7fe6d5800640 1 -- 172.21.15.32:0/1853935172 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6c452e3f0 con 0x55b6c4516390 2024-04-30T08:36:26.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.493+0000 7fe6d5800640 1 --2- 172.21.15.32:0/1853935172 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b6c4516390 0x55b6c48bff40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8006010 tx=0x7fe6c802f410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=76d8689518143c38 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:26.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.493+0000 7fe6c2000640 1 -- 172.21.15.32:0/1853935172 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c8031650 con 0x55b6c4516390 2024-04-30T08:36:26.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.493+0000 7fe6c2000640 1 -- 172.21.15.32:0/1853935172 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6c8036030 con 0x55b6c4516390 2024-04-30T08:36:26.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.493+0000 7fe6c2000640 1 -- 172.21.15.32:0/1853935172 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c80046d0 con 0x55b6c4516390 2024-04-30T08:36:26.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.494+0000 7fe6cd600640 1 -- 172.21.15.32:0/360733656 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7fe6c4004fd0 con 0x7fe6d0075010 2024-04-30T08:36:26.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.495+0000 7fe6de826740 1 -- 172.21.15.32:0/1853935172 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b6c4516390 msgr2=0x55b6c48bff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:26.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.495+0000 7fe6de826740 1 --2- 172.21.15.32:0/1853935172 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b6c4516390 0x55b6c48bff40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8006010 tx=0x7fe6c802f410 comp rx=0 tx=0).stop 2024-04-30T08:36:26.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.495+0000 7fe6de826740 1 -- 172.21.15.32:0/1853935172 shutdown_connections 2024-04-30T08:36:26.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.495+0000 7fe6de826740 1 --2- 172.21.15.32:0/1853935172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6c44f56b0 0x55b6c48c73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.495+0000 7fe6de826740 1 --2- 172.21.15.32:0/1853935172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b6c44ed4d0 0x55b6c48c04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.495+0000 7fe6de826740 1 --2- 172.21.15.32:0/1853935172 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b6c4516390 0x55b6c48bff40 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.495+0000 7fe6de826740 1 -- 172.21.15.32:0/1853935172 >> 172.21.15.32:0/1853935172 conn(0x55b6c450a4e0 msgr2=0x55b6c48bb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:26.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.495+0000 7fe6de826740 1 -- 172.21.15.32:0/1853935172 shutdown_connections 2024-04-30T08:36:26.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.495+0000 7fe6de826740 1 -- 172.21.15.32:0/1853935172 wait complete. 2024-04-30T08:36:26.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.496+0000 7fe6de826740 1 -- start start 2024-04-30T08:36:26.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.496+0000 7fe6de826740 1 Processor -- start 2024-04-30T08:36:26.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.498+0000 7fe6cd600640 1 --2- 172.21.15.32:0/360733656 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe6b80afb00 0x7fe6b80b2010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.498+0000 7fe6de826740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b6c44ed4d0 0x55b6c48c9cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.498+0000 7fe6de826740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6c44f56b0 0x55b6c48ca230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.498+0000 7fe6de826740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b6c4516390 0x55b6c48d1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.498+0000 7fe6de826740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b6c48c7c30 con 0x55b6c4516390 2024-04-30T08:36:26.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.498+0000 7fe6de826740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55b6c47d5b50 con 0x55b6c44ed4d0 2024-04-30T08:36:26.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.498+0000 7fe6de826740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b6c48c7db0 con 0x55b6c44f56b0 2024-04-30T08:36:26.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.498+0000 7fe6cd600640 1 -- 172.21.15.32:0/360733656 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fe6c40df5b0 con 0x7fe6d0075010 2024-04-30T08:36:26.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.498+0000 7fe6de826740 10 client.0 _ll_register_callbacks cb 0x55b6c479f420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:36:26.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.498+0000 7fe6d5800640 1 --2- 172.21.15.32:0/360733656 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe6b80afb00 0x7fe6b80b2010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:26.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.498+0000 7fe6cfe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6c44f56b0 0x55b6c48ca230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:26.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.498+0000 7fe6d4e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b6c44ed4d0 0x55b6c48c9cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:26.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.498+0000 7fe6cfe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6c44f56b0 0x55b6c48ca230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:43138/0 (socket says 172.21.15.32:43138) 2024-04-30T08:36:26.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.499+0000 7fe6d4e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b6c44ed4d0 0x55b6c48c9cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:60948/0 (socket says 172.21.15.32:60948) 2024-04-30T08:36:26.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.499+0000 7fe6d4e00640 1 -- 172.21.15.32:0/3127925256 learned_addr learned my addr 172.21.15.32:0/3127925256 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:26.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.499+0000 7fe6d5800640 1 --2- 172.21.15.32:0/3127925256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b6c4516390 0x55b6c48d1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:26.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.499+0000 7fe6d4e00640 1 -- 172.21.15.32:0/3127925256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6c44f56b0 msgr2=0x55b6c48ca230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:26.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.499+0000 7fe6d4e00640 1 --2- 172.21.15.32:0/3127925256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6c44f56b0 0x55b6c48ca230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.499+0000 7fe6d4e00640 1 -- 172.21.15.32:0/3127925256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b6c4516390 msgr2=0x55b6c48d1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:26.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.499+0000 7fe6d4e00640 1 --2- 172.21.15.32:0/3127925256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b6c4516390 0x55b6c48d1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.499+0000 7fe6d4e00640 1 -- 172.21.15.32:0/3127925256 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6c45f87d0 con 0x55b6c44ed4d0 2024-04-30T08:36:26.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.499+0000 7fe6d4e00640 1 --2- 172.21.15.32:0/3127925256 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b6c44ed4d0 0x55b6c48c9cf0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4000c00 tx=0x7fe6c40a6d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=713cd5c65cfc1b47 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:26.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.499+0000 7fe6b6a00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:36:26.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.500+0000 7fe6b6a00640 1 -- 172.21.15.32:0/3127925256 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c40a7c20 con 0x55b6c44ed4d0 2024-04-30T08:36:26.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.500+0000 7fe6b6a00640 1 -- 172.21.15.32:0/3127925256 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fe6c40dcda0 con 0x55b6c44ed4d0 2024-04-30T08:36:26.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.500+0000 7fe6d5800640 1 --2- 172.21.15.32:0/360733656 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe6b80afb00 0x7fe6b80b2010 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8009a60 tx=0x7fe6c80095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:26.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.500+0000 7fe6de826740 1 -- 172.21.15.32:0/3127925256 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6c8036030 con 0x55b6c44ed4d0 2024-04-30T08:36:26.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.500+0000 7fe6b6a00640 1 -- 172.21.15.32:0/3127925256 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe6c40a7520 con 0x55b6c44ed4d0 2024-04-30T08:36:26.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.500+0000 7fe6de826740 10 client.25985 fetch_fsmap learned FSMap version 28 2024-04-30T08:36:26.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.500+0000 7fe6de826740 1 -- 172.21.15.32:0/3127925256 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b6c48d4110 con 0x55b6c44ed4d0 2024-04-30T08:36:26.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.500+0000 7fe6b6a00640 1 -- 172.21.15.32:0/3127925256 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6c8036030 con 0x55b6c44ed4d0 2024-04-30T08:36:26.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.501+0000 7fe6b6a00640 1 -- 172.21.15.32:0/3127925256 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe6c40afef0 con 0x55b6c44ed4d0 2024-04-30T08:36:26.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.501+0000 7fe6de826740 10 client.25985 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:36:26.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.501+0000 7fe6de826740 10 client.25985 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:36:26.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.501+0000 7fe6de826740 20 client.25985 populate_metadata read hostname 'smithi032' 2024-04-30T08:36:26.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.501+0000 7fe6de826740 1 --2- 172.21.15.32:0/3127925256 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55b6c44d4f60 0x55b6c48d77f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.501+0000 7fe6de826740 4 client.25985 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:36:26.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.501+0000 7fe6de826740 1 -- 172.21.15.32:0/3127925256 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b6c482a960 con 0x55b6c44d4f60 2024-04-30T08:36:26.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.501+0000 7fe6cfe00640 1 --2- 172.21.15.32:0/3127925256 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55b6c44d4f60 0x55b6c48d77f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:26.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.502+0000 7fe6cfe00640 1 --2- 172.21.15.32:0/3127925256 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55b6c44d4f60 0x55b6c48d77f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc6ff044e0522523 server_cookie=2ad2c32e3ace27e in_seq=0 out_seq=0 2024-04-30T08:36:26.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.502+0000 7fe6b6a00640 10 client.25985 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:26.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.503+0000 7fe6b6a00640 1 -- 172.21.15.32:0/3127925256 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b6c482a960 con 0x55b6c44d4f60 2024-04-30T08:36:26.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.503+0000 7fe6b6a00640 10 client.25985 handle_command_reply: tid=0 2024-04-30T08:36:26.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.503+0000 7fe6de826740 1 client.25985 shutdown 2024-04-30T08:36:26.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.503+0000 7fe6de826740 1 -- 172.21.15.32:0/3127925256 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b6c44ed4d0 msgr2=0x55b6c48c9cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:26.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.503+0000 7fe6de826740 1 --2- 172.21.15.32:0/3127925256 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b6c44ed4d0 0x55b6c48c9cf0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4000c00 tx=0x7fe6c40a6d50 comp rx=0 tx=0).stop 2024-04-30T08:36:26.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.504+0000 7fe6de826740 1 -- 172.21.15.32:0/3127925256 shutdown_connections 2024-04-30T08:36:26.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.504+0000 7fe6de826740 1 --2- 172.21.15.32:0/3127925256 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55b6c44d4f60 0x55b6c48d77f0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.504+0000 7fe6de826740 1 --2- 172.21.15.32:0/3127925256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b6c4516390 0x55b6c48d1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.504+0000 7fe6de826740 1 --2- 172.21.15.32:0/3127925256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6c44f56b0 0x55b6c48ca230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.504+0000 7fe6de826740 1 --2- 172.21.15.32:0/3127925256 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b6c44ed4d0 0x55b6c48c9cf0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.504+0000 7fe6de826740 1 -- 172.21.15.32:0/3127925256 >> 172.21.15.32:0/3127925256 conn(0x55b6c450a4e0 msgr2=0x55b6c48b2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:26.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.504+0000 7fe6b6a00640 0 client.25985 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:26.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.504+0000 7fe6de826740 1 -- 172.21.15.32:0/3127925256 shutdown_connections 2024-04-30T08:36:26.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.504+0000 7fe6de826740 1 -- 172.21.15.32:0/3127925256 wait complete. 2024-04-30T08:36:26.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.505+0000 7fe6de826740 20 client.25985 trim_cache size 0 max 16384 2024-04-30T08:36:26.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.534+0000 7fe6de826740 1 Processor -- start 2024-04-30T08:36:26.534 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.534+0000 7fe6de826740 1 -- start start 2024-04-30T08:36:26.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.534+0000 7fe6de826740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b6c44ed4d0 0x55b6c48bba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.534+0000 7fe6de826740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6c44f56b0 0x55b6c48be770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.534+0000 7fe6de826740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b6c4516390 0x55b6c48becb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.535+0000 7fe6de826740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d01aa750 con 0x55b6c44ed4d0 2024-04-30T08:36:26.535 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.535+0000 7fe6de826740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d01ac290 con 0x55b6c4516390 2024-04-30T08:36:26.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.535+0000 7fe6de826740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d01aae20 con 0x55b6c44f56b0 2024-04-30T08:36:26.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.535+0000 7fe6cfe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6c44f56b0 0x55b6c48be770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:26.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.535+0000 7fe6cfe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6c44f56b0 0x55b6c48be770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:43154/0 (socket says 172.21.15.32:43154) 2024-04-30T08:36:26.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.535+0000 7fe6cfe00640 1 -- 172.21.15.32:0/704868249 learned_addr learned my addr 172.21.15.32:0/704868249 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:26.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.535+0000 7fe6d4e00640 1 --2- 172.21.15.32:0/704868249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b6c44ed4d0 0x55b6c48bba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:26.536 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.535+0000 7fe6d5800640 1 --2- 172.21.15.32:0/704868249 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b6c4516390 0x55b6c48becb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:26.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.535+0000 7fe6cfe00640 1 -- 172.21.15.32:0/704868249 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b6c4516390 msgr2=0x55b6c48becb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:26.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.535+0000 7fe6cfe00640 1 --2- 172.21.15.32:0/704868249 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b6c4516390 0x55b6c48becb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.535+0000 7fe6cfe00640 1 -- 172.21.15.32:0/704868249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b6c44ed4d0 msgr2=0x55b6c48bba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:26.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.535+0000 7fe6cfe00640 1 --2- 172.21.15.32:0/704868249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b6c44ed4d0 0x55b6c48bba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.535+0000 7fe6cfe00640 1 -- 172.21.15.32:0/704868249 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b6c48c4d80 con 0x55b6c44f56b0 2024-04-30T08:36:26.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.535+0000 7fe6d5800640 1 --2- 172.21.15.32:0/704868249 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b6c4516390 0x55b6c48becb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-30T08:36:26.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.536+0000 7fe6cfe00640 1 --2- 172.21.15.32:0/704868249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6c44f56b0 0x55b6c48be770 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x55b6c48cad60 tx=0x7fe6bc005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0922d06574f2048 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:26.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.536+0000 7fe6c1600640 1 -- 172.21.15.32:0/704868249 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc012190 con 0x55b6c44f56b0 2024-04-30T08:36:26.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.536+0000 7fe6c1600640 1 -- 172.21.15.32:0/704868249 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6bc024020 con 0x55b6c44f56b0 2024-04-30T08:36:26.537 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.536+0000 7fe6c1600640 1 -- 172.21.15.32:0/704868249 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6bc0203e0 con 0x55b6c44f56b0 2024-04-30T08:36:26.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.537+0000 7fe6de826740 1 -- 172.21.15.32:0/704868249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6c44f56b0 msgr2=0x55b6c48be770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:26.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.537+0000 7fe6de826740 1 --2- 172.21.15.32:0/704868249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6c44f56b0 0x55b6c48be770 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x55b6c48cad60 tx=0x7fe6bc005f90 comp rx=0 tx=0).stop 2024-04-30T08:36:26.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.537+0000 7fe6de826740 1 -- 172.21.15.32:0/704868249 shutdown_connections 2024-04-30T08:36:26.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.537+0000 7fe6de826740 1 --2- 172.21.15.32:0/704868249 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b6c4516390 0x55b6c48becb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.537+0000 7fe6de826740 1 --2- 172.21.15.32:0/704868249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6c44f56b0 0x55b6c48be770 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.537+0000 7fe6de826740 1 --2- 172.21.15.32:0/704868249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b6c44ed4d0 0x55b6c48bba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.537+0000 7fe6de826740 1 -- 172.21.15.32:0/704868249 >> 172.21.15.32:0/704868249 conn(0x55b6c450a4e0 msgr2=0x55b6c48bb330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:26.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.537+0000 7fe6de826740 1 -- 172.21.15.32:0/704868249 shutdown_connections 2024-04-30T08:36:26.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.537+0000 7fe6de826740 1 -- 172.21.15.32:0/704868249 wait complete. 2024-04-30T08:36:26.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.538+0000 7fe6de826740 1 -- start start 2024-04-30T08:36:26.539 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.538+0000 7fe6de826740 1 Processor -- start 2024-04-30T08:36:26.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.539+0000 7fe6de826740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b6c44ed4d0 0x55b6c48cd520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.539+0000 7fe6de826740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6c44f56b0 0x55b6c48cda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.539+0000 7fe6de826740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b6c4516390 0x55b6c493db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.539+0000 7fe6de826740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d01abbc0 con 0x55b6c44ed4d0 2024-04-30T08:36:26.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.539+0000 7fe6de826740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d01ab4f0 con 0x55b6c4516390 2024-04-30T08:36:26.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.539+0000 7fe6de826740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fe6d01a8710 con 0x55b6c44f56b0 2024-04-30T08:36:26.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.539+0000 7fe6de826740 10 client.0 _ll_register_callbacks cb 0x7fe6c400cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:36:26.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.539+0000 7fe6d4e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b6c44ed4d0 0x55b6c48cd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:26.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.539+0000 7fe6d4e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b6c44ed4d0 0x55b6c48cd520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34464/0 (socket says 172.21.15.32:34464) 2024-04-30T08:36:26.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.539+0000 7fe6d4e00640 1 -- 172.21.15.32:0/3511326006 learned_addr learned my addr 172.21.15.32:0/3511326006 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:26.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.539+0000 7fe6d5800640 1 --2- 172.21.15.32:0/3511326006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b6c4516390 0x55b6c493db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:26.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.539+0000 7fe6cfe00640 1 --2- 172.21.15.32:0/3511326006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6c44f56b0 0x55b6c48cda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:26.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.540+0000 7fe6d4e00640 1 -- 172.21.15.32:0/3511326006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6c44f56b0 msgr2=0x55b6c48cda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:26.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.540+0000 7fe6d4e00640 1 --2- 172.21.15.32:0/3511326006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6c44f56b0 0x55b6c48cda60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.540+0000 7fe6d4e00640 1 -- 172.21.15.32:0/3511326006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b6c4516390 msgr2=0x55b6c493db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:26.541 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.540+0000 7fe6d4e00640 1 --2- 172.21.15.32:0/3511326006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b6c4516390 0x55b6c493db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.540+0000 7fe6d4e00640 1 -- 172.21.15.32:0/3511326006 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b6c48d0a20 con 0x55b6c44ed4d0 2024-04-30T08:36:26.542 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.540+0000 7fe6d5800640 1 --2- 172.21.15.32:0/3511326006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b6c4516390 0x55b6c493db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:36:26.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.540+0000 7fe6d4e00640 1 --2- 172.21.15.32:0/3511326006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b6c44ed4d0 0x55b6c48cd520 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4009250 tx=0x7fe6c40a6dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ec18e9feff0820b server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:26.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.540+0000 7fe6b6a00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:36:26.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.540+0000 7fe6b6a00640 1 -- 172.21.15.32:0/3511326006 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6c40a72f0 con 0x55b6c44ed4d0 2024-04-30T08:36:26.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.541+0000 7fe6b6a00640 1 -- 172.21.15.32:0/3511326006 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fe6c40dca90 con 0x55b6c44ed4d0 2024-04-30T08:36:26.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.541+0000 7fe6de826740 1 -- 172.21.15.32:0/3511326006 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe6bc024020 con 0x55b6c44ed4d0 2024-04-30T08:36:26.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.541+0000 7fe6b6a00640 1 -- 172.21.15.32:0/3511326006 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fe6c40a7c20 con 0x55b6c44ed4d0 2024-04-30T08:36:26.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.541+0000 7fe6de826740 10 client.15858 fetch_fsmap learned FSMap version 28 2024-04-30T08:36:26.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.541+0000 7fe6de826740 1 -- 172.21.15.32:0/3511326006 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55b6c493ef80 con 0x55b6c44ed4d0 2024-04-30T08:36:26.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.541+0000 7fe6b6a00640 1 -- 172.21.15.32:0/3511326006 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fe6bc024020 con 0x55b6c44ed4d0 2024-04-30T08:36:26.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.542+0000 7fe6b6a00640 1 -- 172.21.15.32:0/3511326006 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fe6c40a61f0 con 0x55b6c44ed4d0 2024-04-30T08:36:26.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.542+0000 7fe6de826740 10 client.15858 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:36:26.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.542+0000 7fe6de826740 10 client.15858 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:36:26.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.542+0000 7fe6de826740 20 client.15858 populate_metadata read hostname 'smithi032' 2024-04-30T08:36:26.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.542+0000 7fe6de826740 1 --2- 172.21.15.32:0/3511326006 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55b6c44aa3a0 0x55b6c49419a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:26.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.542+0000 7fe6de826740 4 client.15858 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-30T08:36:26.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.542+0000 7fe6de826740 1 -- 172.21.15.32:0/3511326006 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b6c48247d0 con 0x55b6c44aa3a0 2024-04-30T08:36:26.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.542+0000 7fe6cfe00640 1 --2- 172.21.15.32:0/3511326006 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55b6c44aa3a0 0x55b6c49419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:26.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.543+0000 7fe6cfe00640 1 --2- 172.21.15.32:0/3511326006 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55b6c44aa3a0 0x55b6c49419a0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=baacc7946015398 server_cookie=98f7738ed7abfcc9 in_seq=0 out_seq=0 2024-04-30T08:36:26.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.543+0000 7fe6b6a00640 10 client.15858 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:26.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.543+0000 7fe6b6a00640 1 -- 172.21.15.32:0/3511326006 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b6c48247d0 con 0x55b6c44aa3a0 2024-04-30T08:36:26.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.543+0000 7fe6b6a00640 10 client.15858 handle_command_reply: tid=0 2024-04-30T08:36:26.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.543+0000 7fe6de826740 1 client.15858 shutdown 2024-04-30T08:36:26.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.544+0000 7fe6de826740 1 -- 172.21.15.32:0/3511326006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b6c44ed4d0 msgr2=0x55b6c48cd520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:26.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.544+0000 7fe6de826740 1 --2- 172.21.15.32:0/3511326006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b6c44ed4d0 0x55b6c48cd520 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fe6c4009250 tx=0x7fe6c40a6dc0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.544+0000 7fe6de826740 1 -- 172.21.15.32:0/3511326006 shutdown_connections 2024-04-30T08:36:26.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.544+0000 7fe6de826740 1 --2- 172.21.15.32:0/3511326006 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55b6c44aa3a0 0x55b6c49419a0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.544+0000 7fe6de826740 1 --2- 172.21.15.32:0/3511326006 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b6c4516390 0x55b6c493db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.544+0000 7fe6de826740 1 --2- 172.21.15.32:0/3511326006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b6c44f56b0 0x55b6c48cda60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.544+0000 7fe6de826740 1 --2- 172.21.15.32:0/3511326006 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b6c44ed4d0 0x55b6c48cd520 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.544+0000 7fe6b6a00640 0 client.15858 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:26.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.544+0000 7fe6de826740 1 -- 172.21.15.32:0/3511326006 >> 172.21.15.32:0/3511326006 conn(0x55b6c450a4e0 msgr2=0x55b6c48c8210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:26.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.544+0000 7fe6de826740 1 -- 172.21.15.32:0/3511326006 shutdown_connections 2024-04-30T08:36:26.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.544+0000 7fe6de826740 1 -- 172.21.15.32:0/3511326006 wait complete. 2024-04-30T08:36:26.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.545+0000 7fe6de826740 20 client.15858 trim_cache size 0 max 16384 2024-04-30T08:36:26.547 INFO:teuthology.orchestra.run.smithi032.stdout:[] 2024-04-30T08:36:26.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.546+0000 7fe6d6200640 1 -- 172.21.15.32:0/360733656 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe6b80afb00 msgr2=0x7fe6b80b2010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:26.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.546+0000 7fe6d6200640 1 --2- 172.21.15.32:0/360733656 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe6b80afb00 0x7fe6b80b2010 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe6c8009a60 tx=0x7fe6c80095c0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.546+0000 7fe6d6200640 1 -- 172.21.15.32:0/360733656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6d0075010 msgr2=0x7fe6d019c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:26.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.546+0000 7fe6d6200640 1 --2- 172.21.15.32:0/360733656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6d0075010 0x7fe6d019c2d0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fe6c400e9c0 tx=0x7fe6c400ee80 comp rx=0 tx=0).stop 2024-04-30T08:36:26.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.546+0000 7fe6d6200640 1 -- 172.21.15.32:0/360733656 shutdown_connections 2024-04-30T08:36:26.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.546+0000 7fe6d6200640 1 --2- 172.21.15.32:0/360733656 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fe6b80afb00 0x7fe6b80b2010 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.546+0000 7fe6d6200640 1 --2- 172.21.15.32:0/360733656 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fe6d010b900 0x7fe6d01a37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.546+0000 7fe6d6200640 1 --2- 172.21.15.32:0/360733656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe6d0075960 0x7fe6d019c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.546+0000 7fe6d6200640 1 --2- 172.21.15.32:0/360733656 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fe6d0075010 0x7fe6d019c2d0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:26.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.546+0000 7fe6d6200640 1 -- 172.21.15.32:0/360733656 >> 172.21.15.32:0/360733656 conn(0x7fe6d00feba0 msgr2=0x7fe6d00ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:26.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.546+0000 7fe6d6200640 1 -- 172.21.15.32:0/360733656 shutdown_connections 2024-04-30T08:36:26.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:26.547+0000 7fe6d6200640 1 -- 172.21.15.32:0/360733656 wait complete. 2024-04-30T08:36:27.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:26 smithi032 ceph-mon[28331]: pgmap v463: 129 pgs: 129 active+clean; 674 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-30T08:36:27.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:26 smithi071 ceph-mon[33385]: pgmap v463: 129 pgs: 129 active+clean; 674 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-30T08:36:27.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:26 smithi189 ceph-mon[33038]: pgmap v463: 129 pgs: 129 active+clean; 674 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-04-30T08:36:27.590 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-30T08:36:27.590 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-30T08:36:27.590 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-30T08:36:27.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.947+0000 7f1c69e00640 1 -- 172.21.15.32:0/3895370103 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c64103990 msgr2=0x7f1c64104690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:27.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.947+0000 7f1c69e00640 1 --2- 172.21.15.32:0/3895370103 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c64103990 0x7f1c64104690 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f1c58009900 tx=0x7f1c5802a410 comp rx=0 tx=0).stop 2024-04-30T08:36:27.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.947+0000 7f1c69e00640 1 -- 172.21.15.32:0/3895370103 shutdown_connections 2024-04-30T08:36:27.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.947+0000 7f1c69e00640 1 --2- 172.21.15.32:0/3895370103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c641098d0 0x7f1c6410bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:27.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.947+0000 7f1c69e00640 1 --2- 172.21.15.32:0/3895370103 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c64104bd0 0x7f1c64104fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:27.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.947+0000 7f1c69e00640 1 --2- 172.21.15.32:0/3895370103 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c64103990 0x7f1c64104690 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:27.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.947+0000 7f1c69e00640 1 -- 172.21.15.32:0/3895370103 >> 172.21.15.32:0/3895370103 conn(0x7f1c640feba0 msgr2=0x7f1c64101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:27.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.947+0000 7f1c69e00640 1 -- 172.21.15.32:0/3895370103 shutdown_connections 2024-04-30T08:36:27.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.947+0000 7f1c69e00640 1 -- 172.21.15.32:0/3895370103 wait complete. 2024-04-30T08:36:27.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.949+0000 7f1c69e00640 1 Processor -- start 2024-04-30T08:36:27.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.949+0000 7f1c69e00640 1 -- start start 2024-04-30T08:36:27.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.949+0000 7f1c69e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c64103990 0x7f1c6419a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:27.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.949+0000 7f1c69e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c64104bd0 0x7f1c6419a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:27.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.949+0000 7f1c69e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c641098d0 0x7f1c641a15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:27.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.949+0000 7f1c69e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1c6410f490 con 0x7f1c641098d0 2024-04-30T08:36:27.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.949+0000 7f1c69e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1c6410f310 con 0x7f1c64103990 2024-04-30T08:36:27.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.949+0000 7f1c69e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1c6410f610 con 0x7f1c64104bd0 2024-04-30T08:36:27.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.950+0000 7f1c68a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c64103990 0x7f1c6419a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:27.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.950+0000 7f1c68a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c64103990 0x7f1c6419a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:33426/0 (socket says 172.21.15.32:33426) 2024-04-30T08:36:27.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.950+0000 7f1c68a00640 1 -- 172.21.15.32:0/3557370270 learned_addr learned my addr 172.21.15.32:0/3557370270 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:27.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.950+0000 7f1c63e00640 1 --2- 172.21.15.32:0/3557370270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c64104bd0 0x7f1c6419a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:27.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.950+0000 7f1c69400640 1 --2- 172.21.15.32:0/3557370270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c641098d0 0x7f1c641a15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:27.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.950+0000 7f1c68a00640 1 -- 172.21.15.32:0/3557370270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c64104bd0 msgr2=0x7f1c6419a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:27.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.950+0000 7f1c68a00640 1 --2- 172.21.15.32:0/3557370270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c64104bd0 0x7f1c6419a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:27.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.950+0000 7f1c68a00640 1 -- 172.21.15.32:0/3557370270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c641098d0 msgr2=0x7f1c641a15a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:27.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.950+0000 7f1c68a00640 1 --2- 172.21.15.32:0/3557370270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c641098d0 0x7f1c641a15a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:27.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.950+0000 7f1c68a00640 1 -- 172.21.15.32:0/3557370270 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c641a1c10 con 0x7f1c64103990 2024-04-30T08:36:27.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.951+0000 7f1c68a00640 1 --2- 172.21.15.32:0/3557370270 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c64103990 0x7f1c6419a0d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1c58006010 tx=0x7f1c58004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:27.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.951+0000 7f1c61600640 1 -- 172.21.15.32:0/3557370270 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c5803f030 con 0x7f1c64103990 2024-04-30T08:36:27.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.951+0000 7f1c61600640 1 -- 172.21.15.32:0/3557370270 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c58031910 con 0x7f1c64103990 2024-04-30T08:36:27.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.951+0000 7f1c61600640 1 -- 172.21.15.32:0/3557370270 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c58035070 con 0x7f1c64103990 2024-04-30T08:36:27.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.951+0000 7f1c69e00640 1 -- 172.21.15.32:0/3557370270 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c641a1ea0 con 0x7f1c64103990 2024-04-30T08:36:27.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.951+0000 7f1c69e00640 1 -- 172.21.15.32:0/3557370270 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c641a2250 con 0x7f1c64103990 2024-04-30T08:36:27.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.955+0000 7f1c61600640 1 -- 172.21.15.32:0/3557370270 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f1c58042030 con 0x7f1c64103990 2024-04-30T08:36:27.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.956+0000 7f1c72444740 1 Processor -- start 2024-04-30T08:36:27.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.956+0000 7f1c72444740 1 -- start start 2024-04-30T08:36:27.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.956+0000 7f1c72444740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558788bb1410 0x558788f577e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:27.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.956+0000 7f1c72444740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558788b88550 0x558788f57d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:27.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.957+0000 7f1c72444740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558788b90730 0x558788f5ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:27.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.957+0000 7f1c72444740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558788eb7b50 con 0x558788bb1410 2024-04-30T08:36:27.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.957+0000 7f1c72444740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x558788e5d080 con 0x558788b88550 2024-04-30T08:36:27.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.957+0000 7f1c72444740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558788eb3b10 con 0x558788b90730 2024-04-30T08:36:27.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.957+0000 7f1c61600640 1 --2- 172.21.15.32:0/3557370270 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f1c4c0afbf0 0x7f1c4c0b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:27.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.957+0000 7f1c63e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558788b90730 0x558788f5ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:27.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.957+0000 7f1c63e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558788b90730 0x558788f5ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:40160/0 (socket says 172.21.15.32:40160) 2024-04-30T08:36:27.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.957+0000 7f1c63e00640 1 -- 172.21.15.32:0/2983224330 learned_addr learned my addr 172.21.15.32:0/2983224330 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:27.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.957+0000 7f1c61600640 1 -- 172.21.15.32:0/3557370270 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f1c580ff580 con 0x7f1c64103990 2024-04-30T08:36:27.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.957+0000 7f1c69400640 1 --2- 172.21.15.32:0/2983224330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558788bb1410 0x558788f577e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:27.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.958+0000 7f1c68a00640 1 --2- 172.21.15.32:0/2983224330 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558788b88550 0x558788f57d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:27.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.958+0000 7f1c69400640 1 -- 172.21.15.32:0/2983224330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558788b90730 msgr2=0x558788f5ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:27.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.958+0000 7f1c69400640 1 --2- 172.21.15.32:0/2983224330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558788b90730 0x558788f5ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:27.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.958+0000 7f1c69400640 1 -- 172.21.15.32:0/2983224330 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558788b88550 msgr2=0x558788f57d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:27.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.958+0000 7f1c69400640 1 --2- 172.21.15.32:0/2983224330 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558788b88550 0x558788f57d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:27.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.958+0000 7f1c69400640 1 -- 172.21.15.32:0/2983224330 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558788e7d0c0 con 0x558788bb1410 2024-04-30T08:36:27.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.958+0000 7f1c63e00640 1 --2- 172.21.15.32:0/2983224330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558788b90730 0x558788f5ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:36:27.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.958+0000 7f1c69400640 1 --2- 172.21.15.32:0/2983224330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558788bb1410 0x558788f577e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c009fa0 tx=0x7f1c5c0128e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5d1d3ff0e26201d server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:27.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.958+0000 7f1c69400640 1 --2- 172.21.15.32:0/3557370270 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f1c4c0afbf0 0x7f1c4c0b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:27.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.959+0000 7f1c55c00640 1 -- 172.21.15.32:0/2983224330 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c5c010990 con 0x558788bb1410 2024-04-30T08:36:27.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.959+0000 7f1c55c00640 1 -- 172.21.15.32:0/2983224330 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c5c0193b0 con 0x558788bb1410 2024-04-30T08:36:27.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.959+0000 7f1c55c00640 1 -- 172.21.15.32:0/2983224330 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c5c019e60 con 0x558788bb1410 2024-04-30T08:36:27.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.959+0000 7f1c69400640 1 --2- 172.21.15.32:0/3557370270 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f1c4c0afbf0 0x7f1c4c0b2100 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c020000 tx=0x7f1c5c01c030 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:27.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.960+0000 7f1c72444740 1 -- 172.21.15.32:0/2983224330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558788bb1410 msgr2=0x558788f577e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:27.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.960+0000 7f1c72444740 1 --2- 172.21.15.32:0/2983224330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558788bb1410 0x558788f577e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c009fa0 tx=0x7f1c5c0128e0 comp rx=0 tx=0).stop 2024-04-30T08:36:27.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.961+0000 7f1c72444740 1 -- 172.21.15.32:0/2983224330 shutdown_connections 2024-04-30T08:36:27.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.961+0000 7f1c72444740 1 --2- 172.21.15.32:0/2983224330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558788b90730 0x558788f5ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:27.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.961+0000 7f1c72444740 1 --2- 172.21.15.32:0/2983224330 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558788b88550 0x558788f57d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:27.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.961+0000 7f1c72444740 1 --2- 172.21.15.32:0/2983224330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558788bb1410 0x558788f577e0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:27.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.961+0000 7f1c72444740 1 -- 172.21.15.32:0/2983224330 >> 172.21.15.32:0/2983224330 conn(0x558788ba5560 msgr2=0x558788f4ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:27.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.961+0000 7f1c72444740 1 -- 172.21.15.32:0/2983224330 shutdown_connections 2024-04-30T08:36:27.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.961+0000 7f1c72444740 1 -- 172.21.15.32:0/2983224330 wait complete. 2024-04-30T08:36:27.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.962+0000 7f1c72444740 1 -- start start 2024-04-30T08:36:27.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.962+0000 7f1c72444740 1 Processor -- start 2024-04-30T08:36:27.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.963+0000 7f1c72444740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558788b88550 0x558788f61310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:27.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.963+0000 7f1c72444740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558788b90730 0x558788f61850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:27.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.963+0000 7f1c72444740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558788bb1410 0x558788f687b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:27.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.963+0000 7f1c72444740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558788f5f6b0 con 0x558788b88550 2024-04-30T08:36:27.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.963+0000 7f1c72444740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x558788e70b20 con 0x558788bb1410 2024-04-30T08:36:27.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.963+0000 7f1c72444740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558788f5f830 con 0x558788b90730 2024-04-30T08:36:27.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.963+0000 7f1c72444740 10 client.0 _ll_register_callbacks cb 0x558788eaea90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:36:27.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.963+0000 7f1c68a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558788b88550 0x558788f61310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:27.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.963+0000 7f1c63e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558788b90730 0x558788f61850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:27.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.963+0000 7f1c68a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558788b88550 0x558788f61310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34408/0 (socket says 172.21.15.32:34408) 2024-04-30T08:36:27.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.963+0000 7f1c68a00640 1 -- 172.21.15.32:0/312456290 learned_addr learned my addr 172.21.15.32:0/312456290 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:27.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.963+0000 7f1c69400640 1 --2- 172.21.15.32:0/312456290 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558788bb1410 0x558788f687b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:27.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.964+0000 7f1c63e00640 1 -- 172.21.15.32:0/312456290 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558788bb1410 msgr2=0x558788f687b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:27.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.964+0000 7f1c63e00640 1 --2- 172.21.15.32:0/312456290 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558788bb1410 0x558788f687b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:27.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.964+0000 7f1c63e00640 1 -- 172.21.15.32:0/312456290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558788b88550 msgr2=0x558788f61310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:27.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.964+0000 7f1c63e00640 1 --2- 172.21.15.32:0/312456290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558788b88550 0x558788f61310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:27.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.964+0000 7f1c63e00640 1 -- 172.21.15.32:0/312456290 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558788c917e0 con 0x558788b90730 2024-04-30T08:36:27.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.964+0000 7f1c63e00640 1 --2- 172.21.15.32:0/312456290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558788b90730 0x558788f61850 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1c50009560 tx=0x7f1c50012b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa03a53a229e8500 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:27.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.964+0000 7f1c4a000640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-30T08:36:27.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.964+0000 7f1c4a000640 1 -- 172.21.15.32:0/312456290 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c50004050 con 0x558788b90730 2024-04-30T08:36:27.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.964+0000 7f1c72444740 1 -- 172.21.15.32:0/312456290 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c5c0193b0 con 0x558788b90730 2024-04-30T08:36:27.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.964+0000 7f1c4a000640 1 -- 172.21.15.32:0/312456290 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f1c500041f0 con 0x558788b90730 2024-04-30T08:36:27.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.965+0000 7f1c4a000640 1 -- 172.21.15.32:0/312456290 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1c50025050 con 0x558788b90730 2024-04-30T08:36:27.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.965+0000 7f1c72444740 10 client.26005 fetch_fsmap learned FSMap version 28 2024-04-30T08:36:27.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.965+0000 7f1c72444740 1 -- 172.21.15.32:0/312456290 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558788cf1f70 con 0x558788b90730 2024-04-30T08:36:27.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.965+0000 7f1c4a000640 1 -- 172.21.15.32:0/312456290 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c5c0193b0 con 0x558788b90730 2024-04-30T08:36:27.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.965+0000 7f1c4a000640 1 -- 172.21.15.32:0/312456290 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1c50003990 con 0x558788b90730 2024-04-30T08:36:27.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.965+0000 7f1c72444740 10 client.26005 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:36:27.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.965+0000 7f1c72444740 10 client.26005 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:36:27.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.965+0000 7f1c72444740 20 client.26005 populate_metadata read hostname 'smithi032' 2024-04-30T08:36:27.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.965+0000 7f1c72444740 1 --2- 172.21.15.32:0/312456290 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x558788b701f0 0x558788f6ec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:27.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.965+0000 7f1c72444740 4 client.26005 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:36:27.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.965+0000 7f1c72444740 1 -- 172.21.15.32:0/312456290 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558788ec5760 con 0x558788b701f0 2024-04-30T08:36:27.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.966+0000 7f1c68a00640 1 --2- 172.21.15.32:0/312456290 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x558788b701f0 0x558788f6ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:27.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.966+0000 7f1c4a000640 10 client.26005 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:27.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.966+0000 7f1c68a00640 1 --2- 172.21.15.32:0/312456290 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x558788b701f0 0x558788f6ec60 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2159bade8e908553 server_cookie=ecb5fc2d0ca9b0f3 in_seq=0 out_seq=0 2024-04-30T08:36:27.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.967+0000 7f1c4a000640 1 -- 172.21.15.32:0/312456290 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x558788ec5760 con 0x558788b701f0 2024-04-30T08:36:27.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.967+0000 7f1c4a000640 10 client.26005 handle_command_reply: tid=0 2024-04-30T08:36:27.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.967+0000 7f1c72444740 1 client.26005 shutdown 2024-04-30T08:36:27.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.968+0000 7f1c72444740 1 -- 172.21.15.32:0/312456290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558788b90730 msgr2=0x558788f61850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:27.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.968+0000 7f1c72444740 1 --2- 172.21.15.32:0/312456290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558788b90730 0x558788f61850 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f1c50009560 tx=0x7f1c50012b60 comp rx=0 tx=0).stop 2024-04-30T08:36:27.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.968+0000 7f1c72444740 1 -- 172.21.15.32:0/312456290 shutdown_connections 2024-04-30T08:36:27.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.968+0000 7f1c72444740 1 --2- 172.21.15.32:0/312456290 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x558788b701f0 0x558788f6ec60 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:27.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.968+0000 7f1c72444740 1 --2- 172.21.15.32:0/312456290 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558788bb1410 0x558788f687b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:27.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.968+0000 7f1c72444740 1 --2- 172.21.15.32:0/312456290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558788b90730 0x558788f61850 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:27.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.968+0000 7f1c4a000640 0 client.26005 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:27.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.968+0000 7f1c72444740 1 --2- 172.21.15.32:0/312456290 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558788b88550 0x558788f61310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:27.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.968+0000 7f1c72444740 1 -- 172.21.15.32:0/312456290 >> 172.21.15.32:0/312456290 conn(0x558788ba5560 msgr2=0x558788f4dd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:27.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.968+0000 7f1c72444740 1 -- 172.21.15.32:0/312456290 shutdown_connections 2024-04-30T08:36:27.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.968+0000 7f1c72444740 1 -- 172.21.15.32:0/312456290 wait complete. 2024-04-30T08:36:27.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.968+0000 7f1c72444740 20 client.26005 trim_cache size 0 max 16384 2024-04-30T08:36:27.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.988+0000 7f1c72444740 1 Processor -- start 2024-04-30T08:36:27.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.988+0000 7f1c72444740 1 -- start start 2024-04-30T08:36:27.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.988+0000 7f1c72444740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558788b88550 0x558788f5bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:27.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.988+0000 7f1c72444740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558788b90730 0x558788f52270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:27.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.988+0000 7f1c72444740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558788bb1410 0x558788f527b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:27.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.988+0000 7f1c72444740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1c641aa720 con 0x558788b90730 2024-04-30T08:36:27.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.988+0000 7f1c72444740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1c641ac260 con 0x558788b88550 2024-04-30T08:36:27.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.988+0000 7f1c72444740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1c641aadf0 con 0x558788bb1410 2024-04-30T08:36:27.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.989+0000 7f1c68a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558788b88550 0x558788f5bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:27.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.989+0000 7f1c63e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558788b90730 0x558788f52270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:27.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.989+0000 7f1c68a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558788b88550 0x558788f5bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:33446/0 (socket says 172.21.15.32:33446) 2024-04-30T08:36:27.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.989+0000 7f1c68a00640 1 -- 172.21.15.32:0/833390059 learned_addr learned my addr 172.21.15.32:0/833390059 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:27.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.989+0000 7f1c69400640 1 --2- 172.21.15.32:0/833390059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558788bb1410 0x558788f527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:27.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.989+0000 7f1c68a00640 1 -- 172.21.15.32:0/833390059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558788bb1410 msgr2=0x558788f527b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:27.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.989+0000 7f1c68a00640 1 --2- 172.21.15.32:0/833390059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558788bb1410 0x558788f527b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:27.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.989+0000 7f1c68a00640 1 -- 172.21.15.32:0/833390059 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558788b90730 msgr2=0x558788f52270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:27.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.989+0000 7f1c68a00640 1 --2- 172.21.15.32:0/833390059 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558788b90730 0x558788f52270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:27.992 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.989+0000 7f1c68a00640 1 -- 172.21.15.32:0/833390059 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558788b89170 con 0x558788b88550 2024-04-30T08:36:27.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.989+0000 7f1c69400640 1 --2- 172.21.15.32:0/833390059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558788bb1410 0x558788f527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:36:27.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.989+0000 7f1c63e00640 1 --2- 172.21.15.32:0/833390059 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558788b90730 0x558788f52270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:36:27.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.990+0000 7f1c68a00640 1 --2- 172.21.15.32:0/833390059 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558788b88550 0x558788f5bda0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1c580c71f0 tx=0x7f1c5802a410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11c6bbb1d57eab57 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:27.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.990+0000 7f1c55200640 1 -- 172.21.15.32:0/833390059 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c58049040 con 0x558788b88550 2024-04-30T08:36:27.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.990+0000 7f1c55200640 1 -- 172.21.15.32:0/833390059 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c580e1070 con 0x558788b88550 2024-04-30T08:36:27.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.990+0000 7f1c55200640 1 -- 172.21.15.32:0/833390059 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c580f01c0 con 0x558788b88550 2024-04-30T08:36:27.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.991+0000 7f1c72444740 1 -- 172.21.15.32:0/833390059 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558788b88550 msgr2=0x558788f5bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:27.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.991+0000 7f1c72444740 1 --2- 172.21.15.32:0/833390059 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558788b88550 0x558788f5bda0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f1c580c71f0 tx=0x7f1c5802a410 comp rx=0 tx=0).stop 2024-04-30T08:36:27.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.991+0000 7f1c72444740 1 -- 172.21.15.32:0/833390059 shutdown_connections 2024-04-30T08:36:27.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.991+0000 7f1c72444740 1 --2- 172.21.15.32:0/833390059 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558788bb1410 0x558788f527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:27.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.991+0000 7f1c72444740 1 --2- 172.21.15.32:0/833390059 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558788b90730 0x558788f52270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:27.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.991+0000 7f1c72444740 1 --2- 172.21.15.32:0/833390059 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558788b88550 0x558788f5bda0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:27.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.991+0000 7f1c72444740 1 -- 172.21.15.32:0/833390059 >> 172.21.15.32:0/833390059 conn(0x558788ba5560 msgr2=0x558788f51b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:27.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.991+0000 7f1c72444740 1 -- 172.21.15.32:0/833390059 shutdown_connections 2024-04-30T08:36:27.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.991+0000 7f1c72444740 1 -- 172.21.15.32:0/833390059 wait complete. 2024-04-30T08:36:27.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.992+0000 7f1c72444740 1 -- start start 2024-04-30T08:36:27.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.992+0000 7f1c72444740 1 Processor -- start 2024-04-30T08:36:27.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.992+0000 7f1c72444740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558788b88550 0x558788f66c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:27.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.992+0000 7f1c72444740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558788b90730 0x558788f671d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:27.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.992+0000 7f1c72444740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558788bb1410 0x558788f68100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:27.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.992+0000 7f1c72444740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1c641abb90 con 0x558788b88550 2024-04-30T08:36:27.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.992+0000 7f1c72444740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1c641ab4c0 con 0x558788b90730 2024-04-30T08:36:27.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.992+0000 7f1c72444740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1c641a86e0 con 0x558788bb1410 2024-04-30T08:36:27.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.993+0000 7f1c72444740 10 client.0 _ll_register_callbacks cb 0x7f1c50009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:36:27.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.993+0000 7f1c63e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558788b90730 0x558788f671d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:27.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.993+0000 7f1c63e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558788b90730 0x558788f671d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:33448/0 (socket says 172.21.15.32:33448) 2024-04-30T08:36:27.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.993+0000 7f1c63e00640 1 -- 172.21.15.32:0/1049646047 learned_addr learned my addr 172.21.15.32:0/1049646047 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:27.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.993+0000 7f1c68a00640 1 --2- 172.21.15.32:0/1049646047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558788b88550 0x558788f66c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:27.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.993+0000 7f1c69400640 1 --2- 172.21.15.32:0/1049646047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558788bb1410 0x558788f68100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:27.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.993+0000 7f1c63e00640 1 -- 172.21.15.32:0/1049646047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558788bb1410 msgr2=0x558788f68100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:27.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.993+0000 7f1c63e00640 1 --2- 172.21.15.32:0/1049646047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558788bb1410 0x558788f68100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:27.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.993+0000 7f1c63e00640 1 -- 172.21.15.32:0/1049646047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558788b88550 msgr2=0x558788f66c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:27.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.993+0000 7f1c63e00640 1 --2- 172.21.15.32:0/1049646047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558788b88550 0x558788f66c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:27.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.993+0000 7f1c63e00640 1 -- 172.21.15.32:0/1049646047 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558788f686c0 con 0x558788b90730 2024-04-30T08:36:27.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.993+0000 7f1c68a00640 1 --2- 172.21.15.32:0/1049646047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558788b88550 0x558788f66c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:36:27.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.994+0000 7f1c63e00640 1 --2- 172.21.15.32:0/1049646047 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558788b90730 0x558788f671d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1c50009560 tx=0x7f1c50010660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d39daeb3dbfe0596 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:27.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.994+0000 7f1c4a000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:36:27.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.994+0000 7f1c4a000640 1 -- 172.21.15.32:0/1049646047 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c5002d070 con 0x558788b90730 2024-04-30T08:36:27.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.994+0000 7f1c4a000640 1 -- 172.21.15.32:0/1049646047 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f1c500039c0 con 0x558788b90730 2024-04-30T08:36:27.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.994+0000 7f1c72444740 1 -- 172.21.15.32:0/1049646047 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1c580e1070 con 0x558788b90730 2024-04-30T08:36:27.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.994+0000 7f1c4a000640 1 -- 172.21.15.32:0/1049646047 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f1c5003a050 con 0x558788b90730 2024-04-30T08:36:27.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.994+0000 7f1c72444740 10 client.26006 fetch_fsmap learned FSMap version 28 2024-04-30T08:36:27.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.994+0000 7f1c72444740 1 -- 172.21.15.32:0/1049646047 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558788fd7530 con 0x558788b90730 2024-04-30T08:36:27.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.994+0000 7f1c4a000640 1 -- 172.21.15.32:0/1049646047 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1c580e1070 con 0x558788b90730 2024-04-30T08:36:27.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.995+0000 7f1c4a000640 1 -- 172.21.15.32:0/1049646047 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1c5000f920 con 0x558788b90730 2024-04-30T08:36:27.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.995+0000 7f1c72444740 10 client.26006 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:36:27.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.995+0000 7f1c72444740 10 client.26006 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:36:27.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.995+0000 7f1c72444740 20 client.26006 populate_metadata read hostname 'smithi032' 2024-04-30T08:36:27.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.995+0000 7f1c72444740 1 --2- 172.21.15.32:0/1049646047 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x558788b45630 0x558788fd9ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:27.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.995+0000 7f1c72444740 4 client.26006 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-30T08:36:27.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.995+0000 7f1c72444740 1 -- 172.21.15.32:0/1049646047 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x558788ebf5d0 con 0x558788b45630 2024-04-30T08:36:27.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.996+0000 7f1c68a00640 1 --2- 172.21.15.32:0/1049646047 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x558788b45630 0x558788fd9ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:27.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.996+0000 7f1c68a00640 1 --2- 172.21.15.32:0/1049646047 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x558788b45630 0x558788fd9ec0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41991733cd215945 server_cookie=d6aa5dc33b3d91b2 in_seq=0 out_seq=0 2024-04-30T08:36:27.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.996+0000 7f1c4a000640 10 client.26006 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:28.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.998+0000 7f1c4a000640 1 -- 172.21.15.32:0/1049646047 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x558788ebf5d0 con 0x558788b45630 2024-04-30T08:36:28.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.998+0000 7f1c4a000640 10 client.26006 handle_command_reply: tid=0 2024-04-30T08:36:28.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.998+0000 7f1c72444740 1 client.26006 shutdown 2024-04-30T08:36:28.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.998+0000 7f1c72444740 1 -- 172.21.15.32:0/1049646047 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558788b90730 msgr2=0x558788f671d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:28.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.998+0000 7f1c72444740 1 --2- 172.21.15.32:0/1049646047 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558788b90730 0x558788f671d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f1c50009560 tx=0x7f1c50010660 comp rx=0 tx=0).stop 2024-04-30T08:36:28.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.998+0000 7f1c72444740 1 -- 172.21.15.32:0/1049646047 shutdown_connections 2024-04-30T08:36:28.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.998+0000 7f1c72444740 1 --2- 172.21.15.32:0/1049646047 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x558788b45630 0x558788fd9ec0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.998+0000 7f1c72444740 1 --2- 172.21.15.32:0/1049646047 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558788bb1410 0x558788f68100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.998+0000 7f1c72444740 1 --2- 172.21.15.32:0/1049646047 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558788b90730 0x558788f671d0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.998+0000 7f1c72444740 1 --2- 172.21.15.32:0/1049646047 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558788b88550 0x558788f66c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.998+0000 7f1c72444740 1 -- 172.21.15.32:0/1049646047 >> 172.21.15.32:0/1049646047 conn(0x558788ba5560 msgr2=0x558788f52590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:28.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.998+0000 7f1c4a000640 0 client.26006 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:28.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.999+0000 7f1c72444740 1 -- 172.21.15.32:0/1049646047 shutdown_connections 2024-04-30T08:36:28.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.999+0000 7f1c72444740 1 -- 172.21.15.32:0/1049646047 wait complete. 2024-04-30T08:36:28.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:27.999+0000 7f1c72444740 20 client.26006 trim_cache size 0 max 16384 2024-04-30T08:36:28.002 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-30T08:36:28.002 INFO:teuthology.orchestra.run.smithi032.stdout: "return_code": 0, 2024-04-30T08:36:28.002 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_tag": "a93545b2-02cb-49e3-a6cc-339a8df34046", 2024-04-30T08:36:28.002 INFO:teuthology.orchestra.run.smithi032.stdout: "mode": "asynchronous" 2024-04-30T08:36:28.002 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-30T08:36:28.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.000+0000 7f1c69e00640 1 -- 172.21.15.32:0/3557370270 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f1c4c0afbf0 msgr2=0x7f1c4c0b2100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:28.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.000+0000 7f1c69e00640 1 --2- 172.21.15.32:0/3557370270 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f1c4c0afbf0 0x7f1c4c0b2100 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f1c5c020000 tx=0x7f1c5c01c030 comp rx=0 tx=0).stop 2024-04-30T08:36:28.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.000+0000 7f1c69e00640 1 -- 172.21.15.32:0/3557370270 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c64103990 msgr2=0x7f1c6419a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:28.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.000+0000 7f1c69e00640 1 --2- 172.21.15.32:0/3557370270 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c64103990 0x7f1c6419a0d0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f1c58006010 tx=0x7f1c58004270 comp rx=0 tx=0).stop 2024-04-30T08:36:28.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.000+0000 7f1c69e00640 1 -- 172.21.15.32:0/3557370270 shutdown_connections 2024-04-30T08:36:28.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.000+0000 7f1c69e00640 1 --2- 172.21.15.32:0/3557370270 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f1c4c0afbf0 0x7f1c4c0b2100 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.000+0000 7f1c69e00640 1 --2- 172.21.15.32:0/3557370270 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1c641098d0 0x7f1c641a15a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.000+0000 7f1c69e00640 1 --2- 172.21.15.32:0/3557370270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1c64104bd0 0x7f1c6419a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.003 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.000+0000 7f1c69e00640 1 --2- 172.21.15.32:0/3557370270 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1c64103990 0x7f1c6419a0d0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.000+0000 7f1c69e00640 1 -- 172.21.15.32:0/3557370270 >> 172.21.15.32:0/3557370270 conn(0x7f1c640feba0 msgr2=0x7f1c641058b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:28.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.000+0000 7f1c69e00640 1 -- 172.21.15.32:0/3557370270 shutdown_connections 2024-04-30T08:36:28.004 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.001+0000 7f1c69e00640 1 -- 172.21.15.32:0/3557370270 wait complete. 2024-04-30T08:36:28.037 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-30T08:36:28.037 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-30T08:36:28.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.304+0000 7f1576000640 1 -- 172.21.15.32:0/1417071515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1570075900 msgr2=0x7f157010b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:28.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.304+0000 7f1576000640 1 --2- 172.21.15.32:0/1417071515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1570075900 0x7f157010b610 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f155c009900 tx=0x7f155c02a410 comp rx=0 tx=0).stop 2024-04-30T08:36:28.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.305+0000 7f1576000640 1 -- 172.21.15.32:0/1417071515 shutdown_connections 2024-04-30T08:36:28.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.305+0000 7f1576000640 1 --2- 172.21.15.32:0/1417071515 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f157010bb50 0x7f157010df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.305+0000 7f1576000640 1 --2- 172.21.15.32:0/1417071515 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1570075900 0x7f157010b610 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.305+0000 7f1576000640 1 --2- 172.21.15.32:0/1417071515 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1570074fb0 0x7f15700753c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.305+0000 7f1576000640 1 -- 172.21.15.32:0/1417071515 >> 172.21.15.32:0/1417071515 conn(0x7f15700fed70 msgr2=0x7f1570101220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:28.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.305+0000 7f1576000640 1 -- 172.21.15.32:0/1417071515 shutdown_connections 2024-04-30T08:36:28.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.305+0000 7f1576000640 1 -- 172.21.15.32:0/1417071515 wait complete. 2024-04-30T08:36:28.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.306+0000 7f1576000640 1 Processor -- start 2024-04-30T08:36:28.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.306+0000 7f1576000640 1 -- start start 2024-04-30T08:36:28.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.307+0000 7f1576000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1570074fb0 0x7f157019c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:28.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.307+0000 7f1576000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1570075900 0x7f157019ca40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:28.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.307+0000 7f1576000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f157010bb50 0x7f15701a39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:28.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.307+0000 7f1576000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f15701118f0 con 0x7f157010bb50 2024-04-30T08:36:28.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.307+0000 7f1576000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1570111770 con 0x7f1570074fb0 2024-04-30T08:36:28.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.307+0000 7f1576000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1570111a70 con 0x7f1570075900 2024-04-30T08:36:28.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.307+0000 7f156f400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1570074fb0 0x7f157019c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:28.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.307+0000 7f156f400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1570074fb0 0x7f157019c500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:33472/0 (socket says 172.21.15.32:33472) 2024-04-30T08:36:28.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.307+0000 7f156f400640 1 -- 172.21.15.32:0/3002531966 learned_addr learned my addr 172.21.15.32:0/3002531966 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:28.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.307+0000 7f156ea00640 1 --2- 172.21.15.32:0/3002531966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1570075900 0x7f157019ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:28.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.307+0000 7f156fe00640 1 --2- 172.21.15.32:0/3002531966 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f157010bb50 0x7f15701a39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:28.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.308+0000 7f156f400640 1 -- 172.21.15.32:0/3002531966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1570075900 msgr2=0x7f157019ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:28.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.308+0000 7f156f400640 1 --2- 172.21.15.32:0/3002531966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1570075900 0x7f157019ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.308+0000 7f156f400640 1 -- 172.21.15.32:0/3002531966 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f157010bb50 msgr2=0x7f15701a39d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:28.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.308+0000 7f156f400640 1 --2- 172.21.15.32:0/3002531966 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f157010bb50 0x7f15701a39d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.308+0000 7f156f400640 1 -- 172.21.15.32:0/3002531966 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15701a4040 con 0x7f1570074fb0 2024-04-30T08:36:28.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.308+0000 7f156f400640 1 --2- 172.21.15.32:0/3002531966 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1570074fb0 0x7f157019c500 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f156000e9f0 tx=0x7f156000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:28.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.308+0000 7f1567e00640 1 -- 172.21.15.32:0/3002531966 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f156000ccc0 con 0x7f1570074fb0 2024-04-30T08:36:28.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.308+0000 7f1576000640 1 -- 172.21.15.32:0/3002531966 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15701a4330 con 0x7f1570074fb0 2024-04-30T08:36:28.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.309+0000 7f1567e00640 1 -- 172.21.15.32:0/3002531966 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f15600054d0 con 0x7f1570074fb0 2024-04-30T08:36:28.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.309+0000 7f1567e00640 1 -- 172.21.15.32:0/3002531966 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1560010430 con 0x7f1570074fb0 2024-04-30T08:36:28.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.309+0000 7f1576000640 1 -- 172.21.15.32:0/3002531966 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15701a4850 con 0x7f1570074fb0 2024-04-30T08:36:28.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.313+0000 7f1567e00640 1 -- 172.21.15.32:0/3002531966 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 19) ==== 147571+0+0 (secure 0 0 0) 0x7f15600046c0 con 0x7f1570074fb0 2024-04-30T08:36:28.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.314+0000 7f157d035740 1 Processor -- start 2024-04-30T08:36:28.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.314+0000 7f157d035740 1 -- start start 2024-04-30T08:36:28.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.314+0000 7f157d035740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ae4e53390 0x563ae51f4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:28.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.314+0000 7f157d035740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ae4e2a4d0 0x563ae51f47c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:28.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.314+0000 7f157d035740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ae4e326b0 0x563ae51fb6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:28.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.314+0000 7f157d035740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x563ae5159d50 con 0x563ae4e53390 2024-04-30T08:36:28.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.314+0000 7f157d035740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x563ae50feef0 con 0x563ae4e326b0 2024-04-30T08:36:28.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.314+0000 7f157d035740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563ae5155d10 con 0x563ae4e2a4d0 2024-04-30T08:36:28.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.315+0000 7f1567e00640 1 --2- 172.21.15.32:0/3002531966 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f15540afc00 0x7f15540b2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:28.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.315+0000 7f156f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ae4e2a4d0 0x563ae51f47c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:28.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.315+0000 7f156ea00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ae4e326b0 0x563ae51fb6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:28.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.315+0000 7f156ea00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ae4e326b0 0x563ae51fb6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:33474/0 (socket says 172.21.15.32:33474) 2024-04-30T08:36:28.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.315+0000 7f156ea00640 1 -- 172.21.15.32:0/3075404417 learned_addr learned my addr 172.21.15.32:0/3075404417 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:28.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.315+0000 7f156fe00640 1 --2- 172.21.15.32:0/3002531966 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f15540afc00 0x7f15540b2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:28.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.315+0000 7f156fe00640 1 --2- 172.21.15.32:0/3075404417 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ae4e53390 0x563ae51f4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:28.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.316+0000 7f156ea00640 1 -- 172.21.15.32:0/3075404417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ae4e2a4d0 msgr2=0x563ae51f47c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:28.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.316+0000 7f156ea00640 1 --2- 172.21.15.32:0/3075404417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ae4e2a4d0 0x563ae51f47c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.316+0000 7f156ea00640 1 -- 172.21.15.32:0/3075404417 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ae4e53390 msgr2=0x563ae51f4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:28.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.316+0000 7f156ea00640 1 --2- 172.21.15.32:0/3075404417 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ae4e53390 0x563ae51f4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.316+0000 7f156ea00640 1 -- 172.21.15.32:0/3075404417 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ae4f357d0 con 0x563ae4e326b0 2024-04-30T08:36:28.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.316+0000 7f1567e00640 1 -- 172.21.15.32:0/3002531966 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f15600df580 con 0x7f1570074fb0 2024-04-30T08:36:28.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.316+0000 7f156ea00640 1 --2- 172.21.15.32:0/3075404417 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ae4e326b0 0x563ae51fb6c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f155c000c00 tx=0x7f155c0040f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8b1222c57b0b77c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:28.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.317+0000 7f156fe00640 1 --2- 172.21.15.32:0/3002531966 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f15540afc00 0x7f15540b2110 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f1568005fd0 tx=0x7f15680099f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:28.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.317+0000 7f155be00640 1 -- 172.21.15.32:0/3075404417 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155c035030 con 0x563ae4e326b0 2024-04-30T08:36:28.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.317+0000 7f155be00640 1 -- 172.21.15.32:0/3075404417 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f155c040070 con 0x563ae4e326b0 2024-04-30T08:36:28.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.317+0000 7f155be00640 1 -- 172.21.15.32:0/3075404417 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155c031960 con 0x563ae4e326b0 2024-04-30T08:36:28.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.318+0000 7f157d035740 1 -- 172.21.15.32:0/3075404417 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ae4e326b0 msgr2=0x563ae51fb6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:28.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.318+0000 7f157d035740 1 --2- 172.21.15.32:0/3075404417 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ae4e326b0 0x563ae51fb6c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f155c000c00 tx=0x7f155c0040f0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.318+0000 7f157d035740 1 -- 172.21.15.32:0/3075404417 shutdown_connections 2024-04-30T08:36:28.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.318+0000 7f157d035740 1 --2- 172.21.15.32:0/3075404417 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ae4e326b0 0x563ae51fb6c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.319+0000 7f157d035740 1 --2- 172.21.15.32:0/3075404417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ae4e2a4d0 0x563ae51f47c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.319+0000 7f157d035740 1 --2- 172.21.15.32:0/3075404417 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ae4e53390 0x563ae51f4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.319+0000 7f157d035740 1 -- 172.21.15.32:0/3075404417 >> 172.21.15.32:0/3075404417 conn(0x563ae4e474e0 msgr2=0x563ae51efbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:28.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.319+0000 7f157d035740 1 -- 172.21.15.32:0/3075404417 shutdown_connections 2024-04-30T08:36:28.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.319+0000 7f157d035740 1 -- 172.21.15.32:0/3075404417 wait complete. 2024-04-30T08:36:28.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.320+0000 7f157d035740 1 -- start start 2024-04-30T08:36:28.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.320+0000 7f157d035740 1 Processor -- start 2024-04-30T08:36:28.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.321+0000 7f157d035740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ae4e2a4d0 0x563ae51fe0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:28.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.321+0000 7f157d035740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ae4e326b0 0x563ae51fe5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:28.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.321+0000 7f157d035740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ae4e53390 0x563ae5205550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:28.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.321+0000 7f157d035740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x563ae51fc0b0 con 0x563ae4e53390 2024-04-30T08:36:28.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.321+0000 7f157d035740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x563ae5112990 con 0x563ae4e2a4d0 2024-04-30T08:36:28.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.321+0000 7f157d035740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563ae51fc230 con 0x563ae4e326b0 2024-04-30T08:36:28.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.321+0000 7f156fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ae4e53390 0x563ae5205550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:28.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.321+0000 7f156fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ae4e53390 0x563ae5205550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34452/0 (socket says 172.21.15.32:34452) 2024-04-30T08:36:28.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.321+0000 7f156fe00640 1 -- 172.21.15.32:0/114318053 learned_addr learned my addr 172.21.15.32:0/114318053 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:28.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.321+0000 7f156f400640 1 --2- 172.21.15.32:0/114318053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ae4e2a4d0 0x563ae51fe0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:28.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.321+0000 7f157d035740 10 client.0 _ll_register_callbacks cb 0x563ae5111110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:36:28.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.322+0000 7f156ea00640 1 --2- 172.21.15.32:0/114318053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ae4e326b0 0x563ae51fe5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:28.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.322+0000 7f156fe00640 1 -- 172.21.15.32:0/114318053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ae4e326b0 msgr2=0x563ae51fe5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:28.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.322+0000 7f156fe00640 1 --2- 172.21.15.32:0/114318053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ae4e326b0 0x563ae51fe5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.322+0000 7f156fe00640 1 -- 172.21.15.32:0/114318053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ae4e2a4d0 msgr2=0x563ae51fe0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:28.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.322+0000 7f156fe00640 1 --2- 172.21.15.32:0/114318053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ae4e2a4d0 0x563ae51fe0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.322+0000 7f156fe00640 1 -- 172.21.15.32:0/114318053 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ae4e6b3f0 con 0x563ae4e53390 2024-04-30T08:36:28.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.322+0000 7f156ea00640 1 --2- 172.21.15.32:0/114318053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ae4e326b0 0x563ae51fe5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-30T08:36:28.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.322+0000 7f156f400640 1 --2- 172.21.15.32:0/114318053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ae4e2a4d0 0x563ae51fe0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:36:28.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.322+0000 7f156fe00640 1 --2- 172.21.15.32:0/114318053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ae4e53390 0x563ae5205550 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f1568011ed0 tx=0x7f1568009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=391ac6d1357b6aef server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:28.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.322+0000 7f1558c00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:36:28.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.322+0000 7f1558c00640 1 -- 172.21.15.32:0/114318053 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1568004b20 con 0x563ae4e53390 2024-04-30T08:36:28.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.322+0000 7f1558c00640 1 -- 172.21.15.32:0/114318053 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f1568004cc0 con 0x563ae4e53390 2024-04-30T08:36:28.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.323+0000 7f157d035740 1 -- 172.21.15.32:0/114318053 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f155c040070 con 0x563ae4e53390 2024-04-30T08:36:28.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.323+0000 7f1558c00640 1 -- 172.21.15.32:0/114318053 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f156802f050 con 0x563ae4e53390 2024-04-30T08:36:28.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.323+0000 7f157d035740 10 client.15888 fetch_fsmap learned FSMap version 28 2024-04-30T08:36:28.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.323+0000 7f157d035740 1 -- 172.21.15.32:0/114318053 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563ae4e2b0f0 con 0x563ae4e53390 2024-04-30T08:36:28.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.323+0000 7f1558c00640 1 -- 172.21.15.32:0/114318053 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f155c040070 con 0x563ae4e53390 2024-04-30T08:36:28.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.324+0000 7f1558c00640 1 -- 172.21.15.32:0/114318053 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f1568020410 con 0x563ae4e53390 2024-04-30T08:36:28.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.324+0000 7f157d035740 10 client.15888 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:36:28.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.324+0000 7f157d035740 10 client.15888 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:36:28.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.324+0000 7f157d035740 20 client.15888 populate_metadata read hostname 'smithi032' 2024-04-30T08:36:28.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.324+0000 7f157d035740 1 --2- 172.21.15.32:0/114318053 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x563ae4e11f60 0x563ae520bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:28.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.324+0000 7f157d035740 4 client.15888 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:36:28.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.324+0000 7f157d035740 1 -- 172.21.15.32:0/114318053 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563ae516def0 con 0x563ae4e11f60 2024-04-30T08:36:28.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.324+0000 7f156f400640 1 --2- 172.21.15.32:0/114318053 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x563ae4e11f60 0x563ae520bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:28.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.325+0000 7f156f400640 1 --2- 172.21.15.32:0/114318053 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x563ae4e11f60 0x563ae520bb40 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a74ae5952f2454dc server_cookie=734c561a642231ac in_seq=0 out_seq=0 2024-04-30T08:36:28.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.325+0000 7f1558c00640 10 client.15888 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:28.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.326+0000 7f1558c00640 1 -- 172.21.15.32:0/114318053 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x563ae516def0 con 0x563ae4e11f60 2024-04-30T08:36:28.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.326+0000 7f1558c00640 10 client.15888 handle_command_reply: tid=0 2024-04-30T08:36:28.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.326+0000 7f157d035740 1 client.15888 shutdown 2024-04-30T08:36:28.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.326+0000 7f157d035740 1 -- 172.21.15.32:0/114318053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ae4e53390 msgr2=0x563ae5205550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:28.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.326+0000 7f157d035740 1 --2- 172.21.15.32:0/114318053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ae4e53390 0x563ae5205550 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f1568011ed0 tx=0x7f1568009520 comp rx=0 tx=0).stop 2024-04-30T08:36:28.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.326+0000 7f157d035740 1 -- 172.21.15.32:0/114318053 shutdown_connections 2024-04-30T08:36:28.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.326+0000 7f157d035740 1 --2- 172.21.15.32:0/114318053 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x563ae4e11f60 0x563ae520bb40 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.326+0000 7f157d035740 1 --2- 172.21.15.32:0/114318053 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ae4e53390 0x563ae5205550 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.326+0000 7f157d035740 1 --2- 172.21.15.32:0/114318053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ae4e326b0 0x563ae51fe5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.326+0000 7f157d035740 1 --2- 172.21.15.32:0/114318053 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ae4e2a4d0 0x563ae51fe0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.326+0000 7f157d035740 1 -- 172.21.15.32:0/114318053 >> 172.21.15.32:0/114318053 conn(0x563ae4e474e0 msgr2=0x563ae51eebb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:28.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.326+0000 7f1558c00640 0 client.15888 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:28.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.327+0000 7f157d035740 1 -- 172.21.15.32:0/114318053 shutdown_connections 2024-04-30T08:36:28.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.327+0000 7f157d035740 1 -- 172.21.15.32:0/114318053 wait complete. 2024-04-30T08:36:28.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.327+0000 7f157d035740 20 client.15888 trim_cache size 0 max 16384 2024-04-30T08:36:28.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.365+0000 7f157d035740 1 Processor -- start 2024-04-30T08:36:28.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.365+0000 7f157d035740 1 -- start start 2024-04-30T08:36:28.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.366+0000 7f157d035740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ae4e2a4d0 0x563ae51fc6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:28.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.366+0000 7f157d035740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ae4e326b0 0x563ae51fcc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:28.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.366+0000 7f157d035740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ae4e53390 0x563ae51ef500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:28.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.366+0000 7f157d035740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f15701aed00 con 0x563ae4e53390 2024-04-30T08:36:28.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.366+0000 7f157d035740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f15701b0840 con 0x563ae4e2a4d0 2024-04-30T08:36:28.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.366+0000 7f157d035740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f15701af3d0 con 0x563ae4e326b0 2024-04-30T08:36:28.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.366+0000 7f156ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ae4e326b0 0x563ae51fcc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:28.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.366+0000 7f156ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ae4e326b0 0x563ae51fcc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:40252/0 (socket says 172.21.15.32:40252) 2024-04-30T08:36:28.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.366+0000 7f156ea00640 1 -- 172.21.15.32:0/3203502879 learned_addr learned my addr 172.21.15.32:0/3203502879 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:28.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.366+0000 7f156f400640 1 --2- 172.21.15.32:0/3203502879 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ae4e2a4d0 0x563ae51fc6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:28.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.366+0000 7f156fe00640 1 --2- 172.21.15.32:0/3203502879 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ae4e53390 0x563ae51ef500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:28.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.367+0000 7f156ea00640 1 -- 172.21.15.32:0/3203502879 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ae4e2a4d0 msgr2=0x563ae51fc6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:28.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.367+0000 7f156ea00640 1 --2- 172.21.15.32:0/3203502879 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ae4e2a4d0 0x563ae51fc6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.367+0000 7f156ea00640 1 -- 172.21.15.32:0/3203502879 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ae4e53390 msgr2=0x563ae51ef500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:28.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.367+0000 7f156ea00640 1 --2- 172.21.15.32:0/3203502879 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ae4e53390 0x563ae51ef500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.367+0000 7f156ea00640 1 -- 172.21.15.32:0/3203502879 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563ae51e9e20 con 0x563ae4e326b0 2024-04-30T08:36:28.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.367+0000 7f156f400640 1 --2- 172.21.15.32:0/3203502879 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ae4e2a4d0 0x563ae51fc6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:36:28.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.367+0000 7f156ea00640 1 --2- 172.21.15.32:0/3203502879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ae4e326b0 0x563ae51fcc30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f155c009900 tx=0x7f155c031400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=616d9d1841145d1e server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:28.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.368+0000 7f155b400640 1 -- 172.21.15.32:0/3203502879 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155c035030 con 0x563ae4e326b0 2024-04-30T08:36:28.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.368+0000 7f155b400640 1 -- 172.21.15.32:0/3203502879 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f155c040040 con 0x563ae4e326b0 2024-04-30T08:36:28.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.368+0000 7f155b400640 1 -- 172.21.15.32:0/3203502879 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155c0319e0 con 0x563ae4e326b0 2024-04-30T08:36:28.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.369+0000 7f157d035740 1 -- 172.21.15.32:0/3203502879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ae4e326b0 msgr2=0x563ae51fcc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:28.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.369+0000 7f157d035740 1 --2- 172.21.15.32:0/3203502879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ae4e326b0 0x563ae51fcc30 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f155c009900 tx=0x7f155c031400 comp rx=0 tx=0).stop 2024-04-30T08:36:28.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.369+0000 7f157d035740 1 -- 172.21.15.32:0/3203502879 shutdown_connections 2024-04-30T08:36:28.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.369+0000 7f157d035740 1 --2- 172.21.15.32:0/3203502879 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ae4e53390 0x563ae51ef500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.369+0000 7f157d035740 1 --2- 172.21.15.32:0/3203502879 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ae4e326b0 0x563ae51fcc30 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.369+0000 7f157d035740 1 --2- 172.21.15.32:0/3203502879 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ae4e2a4d0 0x563ae51fc6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.369+0000 7f157d035740 1 -- 172.21.15.32:0/3203502879 >> 172.21.15.32:0/3203502879 conn(0x563ae4e474e0 msgr2=0x563ae51f8590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:28.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.369+0000 7f157d035740 1 -- 172.21.15.32:0/3203502879 shutdown_connections 2024-04-30T08:36:28.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.369+0000 7f157d035740 1 -- 172.21.15.32:0/3203502879 wait complete. 2024-04-30T08:36:28.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.370+0000 7f157d035740 1 -- start start 2024-04-30T08:36:28.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.370+0000 7f157d035740 1 Processor -- start 2024-04-30T08:36:28.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.371+0000 7f157d035740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ae4e2a4d0 0x563ae5201cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:28.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.371+0000 7f157d035740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ae4e326b0 0x563ae5202200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:28.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.371+0000 7f157d035740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ae4e53390 0x563ae5203130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:28.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.371+0000 7f156f400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ae4e2a4d0 0x563ae5201cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:28.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.371+0000 7f157d035740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f15701b0170 con 0x563ae4e53390 2024-04-30T08:36:28.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.371+0000 7f157d035740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f15701afaa0 con 0x563ae4e2a4d0 2024-04-30T08:36:28.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.371+0000 7f157d035740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f15701accc0 con 0x563ae4e326b0 2024-04-30T08:36:28.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.371+0000 7f156f400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ae4e2a4d0 0x563ae5201cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:33498/0 (socket says 172.21.15.32:33498) 2024-04-30T08:36:28.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.371+0000 7f156f400640 1 -- 172.21.15.32:0/2766944419 learned_addr learned my addr 172.21.15.32:0/2766944419 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:28.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.371+0000 7f156ea00640 1 --2- 172.21.15.32:0/2766944419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ae4e326b0 0x563ae5202200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:28.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.371+0000 7f157d035740 10 client.0 _ll_register_callbacks cb 0x7f1568009590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:36:28.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.372+0000 7f156fe00640 1 --2- 172.21.15.32:0/2766944419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ae4e53390 0x563ae5203130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:28.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.372+0000 7f156f400640 1 -- 172.21.15.32:0/2766944419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ae4e326b0 msgr2=0x563ae5202200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:28.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.372+0000 7f156f400640 1 --2- 172.21.15.32:0/2766944419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ae4e326b0 0x563ae5202200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.372+0000 7f156f400640 1 -- 172.21.15.32:0/2766944419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ae4e53390 msgr2=0x563ae5203130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:28.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.372+0000 7f156f400640 1 --2- 172.21.15.32:0/2766944419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ae4e53390 0x563ae5203130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.372+0000 7f156f400640 1 -- 172.21.15.32:0/2766944419 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563ae52036f0 con 0x563ae4e2a4d0 2024-04-30T08:36:28.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.372+0000 7f156ea00640 1 --2- 172.21.15.32:0/2766944419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ae4e326b0 0x563ae5202200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:36:28.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.372+0000 7f156f400640 1 --2- 172.21.15.32:0/2766944419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ae4e2a4d0 0x563ae5201cc0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f156000c6a0 tx=0x7f1560005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50a05640da7d8b17 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:28.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.372+0000 7f1558c00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:36:28.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.373+0000 7f1558c00640 1 -- 172.21.15.32:0/2766944419 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1560005680 con 0x563ae4e2a4d0 2024-04-30T08:36:28.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.373+0000 7f1558c00640 1 -- 172.21.15.32:0/2766944419 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f15600df1b0 con 0x563ae4e2a4d0 2024-04-30T08:36:28.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.373+0000 7f157d035740 1 -- 172.21.15.32:0/2766944419 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f155c040040 con 0x563ae4e2a4d0 2024-04-30T08:36:28.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.373+0000 7f1558c00640 1 -- 172.21.15.32:0/2766944419 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f15600df3b0 con 0x563ae4e2a4d0 2024-04-30T08:36:28.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.373+0000 7f157d035740 10 client.26024 fetch_fsmap learned FSMap version 28 2024-04-30T08:36:28.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.373+0000 7f157d035740 1 -- 172.21.15.32:0/2766944419 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x563ae52049f0 con 0x563ae4e2a4d0 2024-04-30T08:36:28.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.373+0000 7f1558c00640 1 -- 172.21.15.32:0/2766944419 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f155c040040 con 0x563ae4e2a4d0 2024-04-30T08:36:28.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.374+0000 7f1558c00640 1 -- 172.21.15.32:0/2766944419 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7f15600afec0 con 0x563ae4e2a4d0 2024-04-30T08:36:28.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.374+0000 7f157d035740 10 client.26024 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:36:28.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.374+0000 7f157d035740 10 client.26024 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:36:28.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.374+0000 7f157d035740 20 client.26024 populate_metadata read hostname 'smithi032' 2024-04-30T08:36:28.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.374+0000 7f157d035740 1 --2- 172.21.15.32:0/2766944419 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x563ae4de73a0 0x563ae5276200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:28.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.374+0000 7f157d035740 4 client.26024 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-30T08:36:28.377 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.374+0000 7f157d035740 1 -- 172.21.15.32:0/2766944419 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563ae5167960 con 0x563ae4de73a0 2024-04-30T08:36:28.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.374+0000 7f156ea00640 1 --2- 172.21.15.32:0/2766944419 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x563ae4de73a0 0x563ae5276200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:28.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.375+0000 7f156ea00640 1 --2- 172.21.15.32:0/2766944419 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x563ae4de73a0 0x563ae5276200 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f5f7fb76ff84090 server_cookie=cb6a2dfa223b9968 in_seq=0 out_seq=0 2024-04-30T08:36:28.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.375+0000 7f1558c00640 10 client.26024 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:28.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.375+0000 7f1558c00640 1 -- 172.21.15.32:0/2766944419 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x563ae5167960 con 0x563ae4de73a0 2024-04-30T08:36:28.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.375+0000 7f1558c00640 10 client.26024 handle_command_reply: tid=0 2024-04-30T08:36:28.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.375+0000 7f157d035740 1 client.26024 shutdown 2024-04-30T08:36:28.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.376+0000 7f157d035740 1 -- 172.21.15.32:0/2766944419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ae4e2a4d0 msgr2=0x563ae5201cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:28.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.376+0000 7f157d035740 1 --2- 172.21.15.32:0/2766944419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ae4e2a4d0 0x563ae5201cc0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f156000c6a0 tx=0x7f1560005f10 comp rx=0 tx=0).stop 2024-04-30T08:36:28.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.376+0000 7f157d035740 1 -- 172.21.15.32:0/2766944419 shutdown_connections 2024-04-30T08:36:28.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.376+0000 7f157d035740 1 --2- 172.21.15.32:0/2766944419 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x563ae4de73a0 0x563ae5276200 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.376+0000 7f157d035740 1 --2- 172.21.15.32:0/2766944419 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563ae4e53390 0x563ae5203130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.376+0000 7f157d035740 1 --2- 172.21.15.32:0/2766944419 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563ae4e326b0 0x563ae5202200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.376+0000 7f157d035740 1 --2- 172.21.15.32:0/2766944419 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563ae4e2a4d0 0x563ae5201cc0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.376+0000 7f157d035740 1 -- 172.21.15.32:0/2766944419 >> 172.21.15.32:0/2766944419 conn(0x563ae4e474e0 msgr2=0x563ae51fc5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:28.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.376+0000 7f1558c00640 0 client.26024 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:28.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.376+0000 7f157d035740 1 -- 172.21.15.32:0/2766944419 shutdown_connections 2024-04-30T08:36:28.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.376+0000 7f157d035740 1 -- 172.21.15.32:0/2766944419 wait complete. 2024-04-30T08:36:28.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.376+0000 7f157d035740 20 client.26024 trim_cache size 0 max 16384 2024-04-30T08:36:28.380 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-30T08:36:28.380 INFO:teuthology.orchestra.run.smithi032.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-30T08:36:28.380 INFO:teuthology.orchestra.run.smithi032.stdout: "scrubs": { 2024-04-30T08:36:28.380 INFO:teuthology.orchestra.run.smithi032.stdout: "a93545b2-02cb-49e3-a6cc-339a8df34046": { 2024-04-30T08:36:28.380 INFO:teuthology.orchestra.run.smithi032.stdout: "path": "/", 2024-04-30T08:36:28.380 INFO:teuthology.orchestra.run.smithi032.stdout: "tag": "a93545b2-02cb-49e3-a6cc-339a8df34046", 2024-04-30T08:36:28.380 INFO:teuthology.orchestra.run.smithi032.stdout: "options": "recursive,force" 2024-04-30T08:36:28.380 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:36:28.380 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:36:28.380 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-30T08:36:28.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.377+0000 7f1576000640 1 -- 172.21.15.32:0/3002531966 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f15540afc00 msgr2=0x7f15540b2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:28.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.377+0000 7f1576000640 1 --2- 172.21.15.32:0/3002531966 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f15540afc00 0x7f15540b2110 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f1568005fd0 tx=0x7f15680099f0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.377+0000 7f1576000640 1 -- 172.21.15.32:0/3002531966 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1570074fb0 msgr2=0x7f157019c500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:28.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.377+0000 7f1576000640 1 --2- 172.21.15.32:0/3002531966 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1570074fb0 0x7f157019c500 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f156000e9f0 tx=0x7f156000eeb0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.377+0000 7f1576000640 1 -- 172.21.15.32:0/3002531966 shutdown_connections 2024-04-30T08:36:28.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.378+0000 7f1576000640 1 --2- 172.21.15.32:0/3002531966 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f15540afc00 0x7f15540b2110 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.378+0000 7f1576000640 1 --2- 172.21.15.32:0/3002531966 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f157010bb50 0x7f15701a39d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.378+0000 7f1576000640 1 --2- 172.21.15.32:0/3002531966 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1570075900 0x7f157019ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.378+0000 7f1576000640 1 --2- 172.21.15.32:0/3002531966 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1570074fb0 0x7f157019c500 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:28.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.378+0000 7f1576000640 1 -- 172.21.15.32:0/3002531966 >> 172.21.15.32:0/3002531966 conn(0x7f15700fed70 msgr2=0x7f15700ffbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:28.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.378+0000 7f1576000640 1 -- 172.21.15.32:0/3002531966 shutdown_connections 2024-04-30T08:36:28.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:28.378+0000 7f1576000640 1 -- 172.21.15.32:0/3002531966 wait complete. 2024-04-30T08:36:28.418 INFO:tasks.cephfs.filesystem:scrub status for tag:a93545b2-02cb-49e3-a6cc-339a8df34046 - {'path': '/', 'tag': 'a93545b2-02cb-49e3-a6cc-339a8df34046', 'options': 'recursive,force'} 2024-04-30T08:36:29.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:28 smithi032 ceph-mon[28331]: pgmap v464: 129 pgs: 129 active+clean; 674 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-30T08:36:29.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:28 smithi032 ceph-mon[28331]: scrub queued for path: / 2024-04-30T08:36:29.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:28 smithi032 ceph-mon[28331]: scrub summary: idle+waiting paths [/] 2024-04-30T08:36:29.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:28 smithi032 ceph-mon[28331]: scrub summary: active paths [/] 2024-04-30T08:36:29.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:28 smithi032 ceph-mon[28331]: scrub summary: idle+waiting paths [/] 2024-04-30T08:36:29.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:28 smithi071 ceph-mon[33385]: pgmap v464: 129 pgs: 129 active+clean; 674 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-30T08:36:29.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:28 smithi071 ceph-mon[33385]: scrub queued for path: / 2024-04-30T08:36:29.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:28 smithi071 ceph-mon[33385]: scrub summary: idle+waiting paths [/] 2024-04-30T08:36:29.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:28 smithi071 ceph-mon[33385]: scrub summary: active paths [/] 2024-04-30T08:36:29.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:28 smithi071 ceph-mon[33385]: scrub summary: idle+waiting paths [/] 2024-04-30T08:36:29.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:28 smithi189 ceph-mon[33038]: pgmap v464: 129 pgs: 129 active+clean; 674 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-30T08:36:29.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:28 smithi189 ceph-mon[33038]: scrub queued for path: / 2024-04-30T08:36:29.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:28 smithi189 ceph-mon[33038]: scrub summary: idle+waiting paths [/] 2024-04-30T08:36:29.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:28 smithi189 ceph-mon[33038]: scrub summary: active paths [/] 2024-04-30T08:36:29.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:28 smithi189 ceph-mon[33038]: scrub summary: idle+waiting paths [/] 2024-04-30T08:36:30.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:29 smithi032 ceph-mon[28331]: scrub summary: idle 2024-04-30T08:36:30.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:29 smithi071 ceph-mon[33385]: scrub summary: idle 2024-04-30T08:36:30.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:29 smithi189 ceph-mon[33038]: scrub summary: idle 2024-04-30T08:36:31.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:30 smithi032 ceph-mon[28331]: pgmap v465: 129 pgs: 129 active+clean; 674 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 341 B/s wr, 12 op/s 2024-04-30T08:36:31.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:30 smithi071 ceph-mon[33385]: pgmap v465: 129 pgs: 129 active+clean; 674 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 341 B/s wr, 12 op/s 2024-04-30T08:36:31.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:30 smithi189 ceph-mon[33038]: pgmap v465: 129 pgs: 129 active+clean; 674 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 341 B/s wr, 12 op/s 2024-04-30T08:36:32.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:31 smithi032 ceph-mon[28331]: mgrmap e20: x(active, since 13m), standbys: z, y 2024-04-30T08:36:32.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:31 smithi071 ceph-mon[33385]: mgrmap e20: x(active, since 13m), standbys: z, y 2024-04-30T08:36:32.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:31 smithi189 ceph-mon[33038]: mgrmap e20: x(active, since 13m), standbys: z, y 2024-04-30T08:36:33.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:32 smithi032 ceph-mon[28331]: pgmap v466: 129 pgs: 129 active+clean; 674 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-30T08:36:33.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:32 smithi071 ceph-mon[33385]: pgmap v466: 129 pgs: 129 active+clean; 674 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-30T08:36:33.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:32 smithi189 ceph-mon[33038]: pgmap v466: 129 pgs: 129 active+clean; 674 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 597 B/s wr, 11 op/s 2024-04-30T08:36:35.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:34 smithi032 ceph-mon[28331]: pgmap v467: 129 pgs: 129 active+clean; 675 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 767 B/s wr, 11 op/s 2024-04-30T08:36:35.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:34 smithi071 ceph-mon[33385]: pgmap v467: 129 pgs: 129 active+clean; 675 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 767 B/s wr, 11 op/s 2024-04-30T08:36:35.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:34 smithi189 ceph-mon[33038]: pgmap v467: 129 pgs: 129 active+clean; 675 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 767 B/s wr, 11 op/s 2024-04-30T08:36:37.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:36 smithi032 ceph-mon[28331]: pgmap v468: 129 pgs: 129 active+clean; 675 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 938 B/s wr, 13 op/s 2024-04-30T08:36:37.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:36 smithi071 ceph-mon[33385]: pgmap v468: 129 pgs: 129 active+clean; 675 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 938 B/s wr, 13 op/s 2024-04-30T08:36:37.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:36 smithi189 ceph-mon[33038]: pgmap v468: 129 pgs: 129 active+clean; 675 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 7.2 KiB/s rd, 938 B/s wr, 13 op/s 2024-04-30T08:36:38.432 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.431+0000 7fb72d000640 1 -- 172.21.15.32:0/610954081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7281039d0 msgr2=0x7fb7281046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:38.433 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.431+0000 7fb72d000640 1 --2- 172.21.15.32:0/610954081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7281039d0 0x7fb7281046d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fb718009900 tx=0x7fb71802a400 comp rx=0 tx=0).stop 2024-04-30T08:36:38.433 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.431+0000 7fb72d000640 1 -- 172.21.15.32:0/610954081 shutdown_connections 2024-04-30T08:36:38.433 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.431+0000 7fb72d000640 1 --2- 172.21.15.32:0/610954081 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb728109910 0x7fb72810bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:38.433 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.431+0000 7fb72d000640 1 --2- 172.21.15.32:0/610954081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb728104c10 0x7fb728105020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:38.433 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.431+0000 7fb72d000640 1 --2- 172.21.15.32:0/610954081 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb7281039d0 0x7fb7281046d0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:38.433 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.431+0000 7fb72d000640 1 -- 172.21.15.32:0/610954081 >> 172.21.15.32:0/610954081 conn(0x7fb7280febc0 msgr2=0x7fb728101070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:38.433 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.431+0000 7fb72d000640 1 -- 172.21.15.32:0/610954081 shutdown_connections 2024-04-30T08:36:38.433 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.431+0000 7fb72d000640 1 -- 172.21.15.32:0/610954081 wait complete. 2024-04-30T08:36:38.434 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.433+0000 7fb72d000640 1 Processor -- start 2024-04-30T08:36:38.434 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.433+0000 7fb72d000640 1 -- start start 2024-04-30T08:36:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.433+0000 7fb72d000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7281039d0 0x7fb72819a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.433+0000 7fb72d000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb728104c10 0x7fb72819a610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.433+0000 7fb72d000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb728109910 0x7fb7281a15a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.434+0000 7fb72d000640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb72810f500 con 0x7fb728109910 2024-04-30T08:36:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.434+0000 7fb72d000640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb72810f380 con 0x7fb7281039d0 2024-04-30T08:36:38.435 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.434+0000 7fb72d000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb72810f680 con 0x7fb728104c10 2024-04-30T08:36:38.436 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.434+0000 7fb727e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb728109910 0x7fb7281a15a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:38.436 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.434+0000 7fb727400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7281039d0 0x7fb72819a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:38.436 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.434+0000 7fb727400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7281039d0 0x7fb72819a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:56316/0 (socket says 172.21.15.32:56316) 2024-04-30T08:36:38.436 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.434+0000 7fb727400640 1 -- 172.21.15.32:0/1378471726 learned_addr learned my addr 172.21.15.32:0/1378471726 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:38.436 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.434+0000 7fb726a00640 1 --2- 172.21.15.32:0/1378471726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb728104c10 0x7fb72819a610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:38.436 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.435+0000 7fb727e00640 1 -- 172.21.15.32:0/1378471726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb728104c10 msgr2=0x7fb72819a610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:38.437 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.435+0000 7fb727e00640 1 --2- 172.21.15.32:0/1378471726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb728104c10 0x7fb72819a610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:38.437 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.435+0000 7fb727e00640 1 -- 172.21.15.32:0/1378471726 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7281039d0 msgr2=0x7fb72819a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:38.437 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.435+0000 7fb727e00640 1 --2- 172.21.15.32:0/1378471726 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7281039d0 0x7fb72819a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:38.437 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.435+0000 7fb727e00640 1 -- 172.21.15.32:0/1378471726 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7281a1c10 con 0x7fb728109910 2024-04-30T08:36:38.437 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.435+0000 7fb727e00640 1 --2- 172.21.15.32:0/1378471726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb728109910 0x7fb7281a15a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb720009d10 tx=0x7fb720007590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:38.437 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.435+0000 7fb71fe00640 1 -- 172.21.15.32:0/1378471726 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb720009e60 con 0x7fb728109910 2024-04-30T08:36:38.437 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.436+0000 7fb71fe00640 1 -- 172.21.15.32:0/1378471726 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb7200100d0 con 0x7fb728109910 2024-04-30T08:36:38.438 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.436+0000 7fb72d000640 1 -- 172.21.15.32:0/1378471726 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb728074e50 con 0x7fb728109910 2024-04-30T08:36:38.438 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.436+0000 7fb71fe00640 1 -- 172.21.15.32:0/1378471726 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb72000ecb0 con 0x7fb728109910 2024-04-30T08:36:38.438 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.436+0000 7fb72d000640 1 -- 172.21.15.32:0/1378471726 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7280753a0 con 0x7fb728109910 2024-04-30T08:36:38.442 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.441+0000 7fb7355f4740 1 Processor -- start 2024-04-30T08:36:38.443 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.442+0000 7fb71fe00640 1 -- 172.21.15.32:0/1378471726 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7fb720004a40 con 0x7fb728109910 2024-04-30T08:36:38.443 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.442+0000 7fb7355f4740 1 -- start start 2024-04-30T08:36:38.444 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.442+0000 7fb7355f4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e1e2f41b0 0x561e1e6962e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:38.444 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.442+0000 7fb7355f4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1e2cb2f0 0x561e1e696860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:38.444 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.443+0000 7fb7355f4740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561e1e2d34d0 0x561e1e69d760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:38.444 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.443+0000 7fb7355f4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561e1e5fa550 con 0x561e1e2cb2f0 2024-04-30T08:36:38.444 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.443+0000 7fb7355f4740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x561e1e59fbc0 con 0x561e1e2d34d0 2024-04-30T08:36:38.444 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.443+0000 7fb7355f4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561e1e5f6510 con 0x561e1e2f41b0 2024-04-30T08:36:38.445 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.443+0000 7fb727400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1e2cb2f0 0x561e1e696860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:38.445 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.443+0000 7fb727400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1e2cb2f0 0x561e1e696860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51178/0 (socket says 172.21.15.32:51178) 2024-04-30T08:36:38.445 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.443+0000 7fb727400640 1 -- 172.21.15.32:0/2230654660 learned_addr learned my addr 172.21.15.32:0/2230654660 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:38.445 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.444+0000 7fb726a00640 1 --2- 172.21.15.32:0/2230654660 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561e1e2d34d0 0x561e1e69d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:38.445 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.444+0000 7fb726a00640 1 --2- 172.21.15.32:0/2230654660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e1e2f41b0 0x561e1e6962e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:38.446 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.444+0000 7fb727400640 1 -- 172.21.15.32:0/2230654660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e1e2f41b0 msgr2=0x561e1e6962e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:38.446 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.444+0000 7fb727400640 1 --2- 172.21.15.32:0/2230654660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e1e2f41b0 0x561e1e6962e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:38.446 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.444+0000 7fb727400640 1 -- 172.21.15.32:0/2230654660 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561e1e2d34d0 msgr2=0x561e1e69d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:38.446 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.444+0000 7fb727400640 1 --2- 172.21.15.32:0/2230654660 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561e1e2d34d0 0x561e1e69d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:38.446 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.444+0000 7fb727400640 1 -- 172.21.15.32:0/2230654660 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e1e30c210 con 0x561e1e2cb2f0 2024-04-30T08:36:38.447 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.444+0000 7fb71fe00640 1 --2- 172.21.15.32:0/1378471726 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb70c0afbb0 0x7fb70c0b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:38.447 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.444+0000 7fb726a00640 1 --2- 172.21.15.32:0/2230654660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e1e2f41b0 0x561e1e6962e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:36:38.447 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.445+0000 7fb71fe00640 1 -- 172.21.15.32:0/1378471726 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fb7200e2560 con 0x7fb728109910 2024-04-30T08:36:38.447 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.445+0000 7fb727400640 1 --2- 172.21.15.32:0/2230654660 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1e2cb2f0 0x561e1e696860 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb718005710 tx=0x7fb718004310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=845ae02cc2fde5ab server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:38.447 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.445+0000 7fb726a00640 1 --2- 172.21.15.32:0/1378471726 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb70c0afbb0 0x7fb70c0b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:38.447 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.445+0000 7fb71cc00640 1 -- 172.21.15.32:0/2230654660 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb718040070 con 0x561e1e2cb2f0 2024-04-30T08:36:38.447 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.446+0000 7fb71cc00640 1 -- 172.21.15.32:0/2230654660 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb718003aa0 con 0x561e1e2cb2f0 2024-04-30T08:36:38.447 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.446+0000 7fb71cc00640 1 -- 172.21.15.32:0/2230654660 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb718035040 con 0x561e1e2cb2f0 2024-04-30T08:36:38.448 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.447+0000 7fb7355f4740 1 -- 172.21.15.32:0/2230654660 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1e2cb2f0 msgr2=0x561e1e696860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:38.448 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.447+0000 7fb726a00640 1 --2- 172.21.15.32:0/1378471726 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb70c0afbb0 0x7fb70c0b20c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x561e1e6974b0 tx=0x7fb7140046c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:38.449 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.447+0000 7fb7355f4740 1 --2- 172.21.15.32:0/2230654660 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1e2cb2f0 0x561e1e696860 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb718005710 tx=0x7fb718004310 comp rx=0 tx=0).stop 2024-04-30T08:36:38.449 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.447+0000 7fb7355f4740 1 -- 172.21.15.32:0/2230654660 shutdown_connections 2024-04-30T08:36:38.449 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.447+0000 7fb7355f4740 1 --2- 172.21.15.32:0/2230654660 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561e1e2d34d0 0x561e1e69d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:38.449 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.447+0000 7fb7355f4740 1 --2- 172.21.15.32:0/2230654660 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1e2cb2f0 0x561e1e696860 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:38.449 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.447+0000 7fb7355f4740 1 --2- 172.21.15.32:0/2230654660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e1e2f41b0 0x561e1e6962e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:38.450 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.447+0000 7fb7355f4740 1 -- 172.21.15.32:0/2230654660 >> 172.21.15.32:0/2230654660 conn(0x561e1e2e8300 msgr2=0x561e1e691c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:38.450 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.448+0000 7fb7355f4740 1 -- 172.21.15.32:0/2230654660 shutdown_connections 2024-04-30T08:36:38.450 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.448+0000 7fb7355f4740 1 -- 172.21.15.32:0/2230654660 wait complete. 2024-04-30T08:36:38.450 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.449+0000 7fb7355f4740 1 -- start start 2024-04-30T08:36:38.450 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.449+0000 7fb7355f4740 1 Processor -- start 2024-04-30T08:36:38.451 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.450+0000 7fb7355f4740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561e1e2cb2f0 0x561e1e69fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:38.451 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.450+0000 7fb7355f4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1e2d34d0 0x561e1e6a0290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:38.451 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.450+0000 7fb7355f4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e1e2f41b0 0x561e1e6a71f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:38.451 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.450+0000 7fb7355f4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x561e1e69e150 con 0x561e1e2d34d0 2024-04-30T08:36:38.451 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.450+0000 7fb7355f4740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x561e1e5b3660 con 0x561e1e2cb2f0 2024-04-30T08:36:38.452 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.450+0000 7fb7355f4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x561e1e69e2d0 con 0x561e1e2f41b0 2024-04-30T08:36:38.452 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.450+0000 7fb7355f4740 10 client.0 _ll_register_callbacks cb 0x561e1e57d400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:36:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.450+0000 7fb727400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561e1e2cb2f0 0x561e1e69fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.450+0000 7fb727400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561e1e2cb2f0 0x561e1e69fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:56332/0 (socket says 172.21.15.32:56332) 2024-04-30T08:36:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.450+0000 7fb727400640 1 -- 172.21.15.32:0/3738924655 learned_addr learned my addr 172.21.15.32:0/3738924655 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.450+0000 7fb726a00640 1 --2- 172.21.15.32:0/3738924655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1e2d34d0 0x561e1e6a0290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.450+0000 7fb727e00640 1 --2- 172.21.15.32:0/3738924655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e1e2f41b0 0x561e1e6a71f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.451+0000 7fb727400640 1 -- 172.21.15.32:0/3738924655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e1e2f41b0 msgr2=0x561e1e6a71f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.451+0000 7fb727400640 1 --2- 172.21.15.32:0/3738924655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e1e2f41b0 0x561e1e6a71f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.451+0000 7fb727400640 1 -- 172.21.15.32:0/3738924655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1e2d34d0 msgr2=0x561e1e6a0290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.451+0000 7fb727400640 1 --2- 172.21.15.32:0/3738924655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1e2d34d0 0x561e1e6a0290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:38.453 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.451+0000 7fb727400640 1 -- 172.21.15.32:0/3738924655 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e1e3d8620 con 0x561e1e2cb2f0 2024-04-30T08:36:38.454 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.451+0000 7fb727400640 1 --2- 172.21.15.32:0/3738924655 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561e1e2cb2f0 0x561e1e69fd50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb718005ab0 tx=0x7fb718000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a802fd019857a076 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:38.454 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.451+0000 7fb711600640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:36:38.454 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.452+0000 7fb711600640 1 -- 172.21.15.32:0/3738924655 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb718035040 con 0x561e1e2cb2f0 2024-04-30T08:36:38.454 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.452+0000 7fb711600640 1 -- 172.21.15.32:0/3738924655 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fb718003800 con 0x561e1e2cb2f0 2024-04-30T08:36:38.454 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.452+0000 7fb7355f4740 1 -- 172.21.15.32:0/3738924655 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb718003aa0 con 0x561e1e2cb2f0 2024-04-30T08:36:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.452+0000 7fb711600640 1 -- 172.21.15.32:0/3738924655 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb718058050 con 0x561e1e2cb2f0 2024-04-30T08:36:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.453+0000 7fb7355f4740 10 client.26036 fetch_fsmap learned FSMap version 28 2024-04-30T08:36:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.453+0000 7fb7355f4740 1 -- 172.21.15.32:0/3738924655 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561e1e6aa120 con 0x561e1e2cb2f0 2024-04-30T08:36:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.453+0000 7fb711600640 1 -- 172.21.15.32:0/3738924655 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb718003aa0 con 0x561e1e2cb2f0 2024-04-30T08:36:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.454+0000 7fb711600640 1 -- 172.21.15.32:0/3738924655 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb7180457d0 con 0x561e1e2cb2f0 2024-04-30T08:36:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.454+0000 7fb7355f4740 10 client.26036 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:36:38.455 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.454+0000 7fb7355f4740 10 client.26036 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-30T08:36:38.456 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.454+0000 7fb7355f4740 20 client.26036 populate_metadata read hostname 'smithi032' 2024-04-30T08:36:38.456 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.454+0000 7fb7355f4740 1 --2- 172.21.15.32:0/3738924655 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x561e1e2b2d80 0x561e1e6ad480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:38.456 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.454+0000 7fb7355f4740 4 client.26036 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:36:38.456 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.454+0000 7fb7355f4740 1 -- 172.21.15.32:0/3738924655 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561e1e608160 con 0x561e1e2b2d80 2024-04-30T08:36:38.456 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.454+0000 7fb726a00640 1 --2- 172.21.15.32:0/3738924655 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x561e1e2b2d80 0x561e1e6ad480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:38.456 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.455+0000 7fb726a00640 1 --2- 172.21.15.32:0/3738924655 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x561e1e2b2d80 0x561e1e6ad480 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c120f5abb161ae1 server_cookie=7268acaa229ea4bd in_seq=0 out_seq=0 2024-04-30T08:36:38.456 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.455+0000 7fb711600640 10 client.26036 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:38.457 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.456+0000 7fb711600640 1 -- 172.21.15.32:0/3738924655 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x561e1e608160 con 0x561e1e2b2d80 2024-04-30T08:36:38.457 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.456+0000 7fb711600640 10 client.26036 handle_command_reply: tid=0 2024-04-30T08:36:38.457 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.456+0000 7fb7355f4740 1 client.26036 shutdown 2024-04-30T08:36:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.456+0000 7fb7355f4740 1 -- 172.21.15.32:0/3738924655 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561e1e2cb2f0 msgr2=0x561e1e69fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.456+0000 7fb7355f4740 1 --2- 172.21.15.32:0/3738924655 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561e1e2cb2f0 0x561e1e69fd50 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fb718005ab0 tx=0x7fb718000bc0 comp rx=0 tx=0).stop 2024-04-30T08:36:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.457+0000 7fb7355f4740 1 -- 172.21.15.32:0/3738924655 shutdown_connections 2024-04-30T08:36:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.457+0000 7fb7355f4740 1 --2- 172.21.15.32:0/3738924655 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x561e1e2b2d80 0x561e1e6ad480 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.457+0000 7fb7355f4740 1 --2- 172.21.15.32:0/3738924655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e1e2f41b0 0x561e1e6a71f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.457+0000 7fb7355f4740 1 --2- 172.21.15.32:0/3738924655 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1e2d34d0 0x561e1e6a0290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.457+0000 7fb7355f4740 1 --2- 172.21.15.32:0/3738924655 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561e1e2cb2f0 0x561e1e69fd50 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.457+0000 7fb7355f4740 1 -- 172.21.15.32:0/3738924655 >> 172.21.15.32:0/3738924655 conn(0x561e1e2e8300 msgr2=0x561e1e694b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.457+0000 7fb711600640 0 client.26036 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.457+0000 7fb7355f4740 1 -- 172.21.15.32:0/3738924655 shutdown_connections 2024-04-30T08:36:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.457+0000 7fb7355f4740 1 -- 172.21.15.32:0/3738924655 wait complete. 2024-04-30T08:36:38.459 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.457+0000 7fb7355f4740 20 client.26036 trim_cache size 0 max 16384 2024-04-30T08:36:38.490 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.489+0000 7fb7355f4740 1 Processor -- start 2024-04-30T08:36:38.491 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.489+0000 7fb7355f4740 1 -- start start 2024-04-30T08:36:38.491 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.490+0000 7fb7355f4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1e2cb2f0 0x561e1e691e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:38.491 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.490+0000 7fb7355f4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e1e2d34d0 0x561e1e6a7110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:38.491 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.490+0000 7fb7355f4740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561e1e2f41b0 0x561e1e6a7650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:38.491 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.490+0000 7fb7355f4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb7281aa770 con 0x561e1e2cb2f0 2024-04-30T08:36:38.491 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.490+0000 7fb7355f4740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb7281ac2b0 con 0x561e1e2f41b0 2024-04-30T08:36:38.492 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.490+0000 7fb7355f4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb7281aae40 con 0x561e1e2d34d0 2024-04-30T08:36:38.492 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.490+0000 7fb727400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1e2cb2f0 0x561e1e691e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:38.493 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.490+0000 7fb727e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561e1e2f41b0 0x561e1e6a7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:38.493 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.490+0000 7fb727400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1e2cb2f0 0x561e1e691e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51202/0 (socket says 172.21.15.32:51202) 2024-04-30T08:36:38.493 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.490+0000 7fb727400640 1 -- 172.21.15.32:0/103455519 learned_addr learned my addr 172.21.15.32:0/103455519 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:38.493 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.490+0000 7fb726a00640 1 --2- 172.21.15.32:0/103455519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e1e2d34d0 0x561e1e6a7110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:38.493 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.491+0000 7fb726a00640 1 -- 172.21.15.32:0/103455519 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561e1e2f41b0 msgr2=0x561e1e6a7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:38.493 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.491+0000 7fb726a00640 1 --2- 172.21.15.32:0/103455519 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561e1e2f41b0 0x561e1e6a7650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:38.493 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.491+0000 7fb726a00640 1 -- 172.21.15.32:0/103455519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1e2cb2f0 msgr2=0x561e1e691e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:38.493 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.491+0000 7fb726a00640 1 --2- 172.21.15.32:0/103455519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1e2cb2f0 0x561e1e691e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:38.493 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.491+0000 7fb726a00640 1 -- 172.21.15.32:0/103455519 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561e1e68d480 con 0x561e1e2d34d0 2024-04-30T08:36:38.493 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.491+0000 7fb727400640 1 --2- 172.21.15.32:0/103455519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1e2cb2f0 0x561e1e691e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:36:38.493 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.491+0000 7fb727e00640 1 --2- 172.21.15.32:0/103455519 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561e1e2f41b0 0x561e1e6a7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:36:38.494 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.491+0000 7fb726a00640 1 --2- 172.21.15.32:0/103455519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e1e2d34d0 0x561e1e6a7110 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x561e1e6adaa0 tx=0x7fb714012040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36d05d677807b1f1 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:38.494 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.492+0000 7fb713e00640 1 -- 172.21.15.32:0/103455519 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb714019040 con 0x561e1e2d34d0 2024-04-30T08:36:38.494 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.492+0000 7fb713e00640 1 -- 172.21.15.32:0/103455519 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb714009950 con 0x561e1e2d34d0 2024-04-30T08:36:38.494 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.492+0000 7fb713e00640 1 -- 172.21.15.32:0/103455519 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb714004a00 con 0x561e1e2d34d0 2024-04-30T08:36:38.495 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.493+0000 7fb7355f4740 1 -- 172.21.15.32:0/103455519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e1e2d34d0 msgr2=0x561e1e6a7110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:38.495 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.493+0000 7fb7355f4740 1 --2- 172.21.15.32:0/103455519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e1e2d34d0 0x561e1e6a7110 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x561e1e6adaa0 tx=0x7fb714012040 comp rx=0 tx=0).stop 2024-04-30T08:36:38.495 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.493+0000 7fb7355f4740 1 -- 172.21.15.32:0/103455519 shutdown_connections 2024-04-30T08:36:38.495 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.493+0000 7fb7355f4740 1 --2- 172.21.15.32:0/103455519 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561e1e2f41b0 0x561e1e6a7650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:38.495 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.493+0000 7fb7355f4740 1 --2- 172.21.15.32:0/103455519 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e1e2d34d0 0x561e1e6a7110 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:38.495 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.493+0000 7fb7355f4740 1 --2- 172.21.15.32:0/103455519 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1e2cb2f0 0x561e1e691e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:38.495 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.493+0000 7fb7355f4740 1 -- 172.21.15.32:0/103455519 >> 172.21.15.32:0/103455519 conn(0x561e1e2e8300 msgr2=0x561e1e69ff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:38.495 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.493+0000 7fb7355f4740 1 -- 172.21.15.32:0/103455519 shutdown_connections 2024-04-30T08:36:38.495 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.493+0000 7fb7355f4740 1 -- 172.21.15.32:0/103455519 wait complete. 2024-04-30T08:36:38.496 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.494+0000 7fb7355f4740 1 -- start start 2024-04-30T08:36:38.497 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.494+0000 7fb7355f4740 1 Processor -- start 2024-04-30T08:36:38.497 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.495+0000 7fb7355f4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1e2cb2f0 0x561e1e6a5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:38.497 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.495+0000 7fb7355f4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e1e2d34d0 0x561e1e6a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:38.497 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.495+0000 7fb7355f4740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561e1e2f41b0 0x561e1e6a65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:38.497 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.495+0000 7fb7355f4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb7281abbe0 con 0x561e1e2cb2f0 2024-04-30T08:36:38.497 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.495+0000 7fb7355f4740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb7281ab510 con 0x561e1e2f41b0 2024-04-30T08:36:38.497 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.495+0000 7fb7355f4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb7281a8730 con 0x561e1e2d34d0 2024-04-30T08:36:38.497 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.495+0000 7fb7355f4740 10 client.0 _ll_register_callbacks cb 0x7fb714009dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:36:38.498 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.496+0000 7fb727400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1e2cb2f0 0x561e1e6a5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:38.498 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.496+0000 7fb727400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1e2cb2f0 0x561e1e6a5190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:51218/0 (socket says 172.21.15.32:51218) 2024-04-30T08:36:38.498 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.496+0000 7fb727400640 1 -- 172.21.15.32:0/2171630083 learned_addr learned my addr 172.21.15.32:0/2171630083 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:38.498 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.496+0000 7fb726a00640 1 --2- 172.21.15.32:0/2171630083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e1e2d34d0 0x561e1e6a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:38.498 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.496+0000 7fb727e00640 1 --2- 172.21.15.32:0/2171630083 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561e1e2f41b0 0x561e1e6a65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:38.498 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.496+0000 7fb726a00640 1 -- 172.21.15.32:0/2171630083 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561e1e2f41b0 msgr2=0x561e1e6a65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:38.498 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.496+0000 7fb726a00640 1 --2- 172.21.15.32:0/2171630083 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561e1e2f41b0 0x561e1e6a65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:38.499 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.496+0000 7fb726a00640 1 -- 172.21.15.32:0/2171630083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1e2cb2f0 msgr2=0x561e1e6a5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:38.499 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.496+0000 7fb726a00640 1 --2- 172.21.15.32:0/2171630083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1e2cb2f0 0x561e1e6a5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:38.499 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.496+0000 7fb726a00640 1 -- 172.21.15.32:0/2171630083 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561e1e6a6b90 con 0x561e1e2d34d0 2024-04-30T08:36:38.499 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.497+0000 7fb726a00640 1 --2- 172.21.15.32:0/2171630083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e1e2d34d0 0x561e1e6a56d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb714010e90 tx=0x7fb714005a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84b5edfb7560d0df server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:38.499 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.497+0000 7fb711600640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-30T08:36:38.500 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.497+0000 7fb711600640 1 -- 172.21.15.32:0/2171630083 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb714021a80 con 0x561e1e2d34d0 2024-04-30T08:36:38.500 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.497+0000 7fb711600640 1 -- 172.21.15.32:0/2171630083 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fb714021c20 con 0x561e1e2d34d0 2024-04-30T08:36:38.500 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.497+0000 7fb7355f4740 1 -- 172.21.15.32:0/2171630083 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb714009950 con 0x561e1e2d34d0 2024-04-30T08:36:38.500 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.498+0000 7fb711600640 1 -- 172.21.15.32:0/2171630083 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fb7140300d0 con 0x561e1e2d34d0 2024-04-30T08:36:38.500 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.498+0000 7fb7355f4740 10 client.26044 fetch_fsmap learned FSMap version 28 2024-04-30T08:36:38.501 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.498+0000 7fb7355f4740 1 -- 172.21.15.32:0/2171630083 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x561e1e71df20 con 0x561e1e2d34d0 2024-04-30T08:36:38.501 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.498+0000 7fb711600640 1 -- 172.21.15.32:0/2171630083 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb71403e070 con 0x561e1e2d34d0 2024-04-30T08:36:38.501 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.498+0000 7fb711600640 1 -- 172.21.15.32:0/2171630083 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7fb714039620 con 0x561e1e2d34d0 2024-04-30T08:36:38.501 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.499+0000 7fb7355f4740 10 client.26044 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:36:38.501 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.499+0000 7fb7355f4740 10 client.26044 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-30T08:36:38.501 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.499+0000 7fb7355f4740 20 client.26044 populate_metadata read hostname 'smithi032' 2024-04-30T08:36:38.501 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.499+0000 7fb7355f4740 1 --2- 172.21.15.32:0/2171630083 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x561e1e2881c0 0x561e1e720880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:38.502 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.499+0000 7fb7355f4740 4 client.26044 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-30T08:36:38.502 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.499+0000 7fb7355f4740 1 -- 172.21.15.32:0/2171630083 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fb714009950 con 0x561e1e2881c0 2024-04-30T08:36:38.502 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.500+0000 7fb727400640 1 --2- 172.21.15.32:0/2171630083 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x561e1e2881c0 0x561e1e720880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:38.502 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.500+0000 7fb727400640 1 --2- 172.21.15.32:0/2171630083 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x561e1e2881c0 0x561e1e720880 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc95f09ebd7cc030 server_cookie=848071db87668f68 in_seq=0 out_seq=0 2024-04-30T08:36:38.502 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.500+0000 7fb711600640 10 client.26044 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:38.510 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.506+0000 7fb711600640 1 -- 172.21.15.32:0/2171630083 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fb714009950 con 0x561e1e2881c0 2024-04-30T08:36:38.510 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.506+0000 7fb711600640 10 client.26044 handle_command_reply: tid=0 2024-04-30T08:36:38.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.506+0000 7fb7355f4740 1 client.26044 shutdown 2024-04-30T08:36:38.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.506+0000 7fb7355f4740 1 -- 172.21.15.32:0/2171630083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e1e2d34d0 msgr2=0x561e1e6a56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:38.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.506+0000 7fb7355f4740 1 --2- 172.21.15.32:0/2171630083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e1e2d34d0 0x561e1e6a56d0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb714010e90 tx=0x7fb714005a90 comp rx=0 tx=0).stop 2024-04-30T08:36:38.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.506+0000 7fb7355f4740 1 -- 172.21.15.32:0/2171630083 shutdown_connections 2024-04-30T08:36:38.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.506+0000 7fb7355f4740 1 --2- 172.21.15.32:0/2171630083 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x561e1e2881c0 0x561e1e720880 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:38.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.507+0000 7fb7355f4740 1 --2- 172.21.15.32:0/2171630083 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x561e1e2f41b0 0x561e1e6a65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:38.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.507+0000 7fb7355f4740 1 --2- 172.21.15.32:0/2171630083 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561e1e2d34d0 0x561e1e6a56d0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:38.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.507+0000 7fb7355f4740 1 --2- 172.21.15.32:0/2171630083 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x561e1e2cb2f0 0x561e1e6a5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:38.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.507+0000 7fb7355f4740 1 -- 172.21.15.32:0/2171630083 >> 172.21.15.32:0/2171630083 conn(0x561e1e2e8300 msgr2=0x561e1e690230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:38.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.507+0000 7fb711600640 0 client.26044 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:38.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.507+0000 7fb7355f4740 1 -- 172.21.15.32:0/2171630083 shutdown_connections 2024-04-30T08:36:38.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.507+0000 7fb7355f4740 1 -- 172.21.15.32:0/2171630083 wait complete. 2024-04-30T08:36:38.511 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.507+0000 7fb7355f4740 20 client.26044 trim_cache size 0 max 16384 2024-04-30T08:36:38.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-30T08:36:38.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-30T08:36:38.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-30T08:36:38.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-30T08:36:38.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.508+0000 7fb72d000640 1 -- 172.21.15.32:0/1378471726 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb70c0afbb0 msgr2=0x7fb70c0b20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:38.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.508+0000 7fb72d000640 1 --2- 172.21.15.32:0/1378471726 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb70c0afbb0 0x7fb70c0b20c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x561e1e6974b0 tx=0x7fb7140046c0 comp rx=0 tx=0).stop 2024-04-30T08:36:38.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.508+0000 7fb72d000640 1 -- 172.21.15.32:0/1378471726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb728109910 msgr2=0x7fb7281a15a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:38.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.508+0000 7fb72d000640 1 --2- 172.21.15.32:0/1378471726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb728109910 0x7fb7281a15a0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fb720009d10 tx=0x7fb720007590 comp rx=0 tx=0).stop 2024-04-30T08:36:38.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.508+0000 7fb72d000640 1 -- 172.21.15.32:0/1378471726 shutdown_connections 2024-04-30T08:36:38.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.508+0000 7fb72d000640 1 --2- 172.21.15.32:0/1378471726 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb70c0afbb0 0x7fb70c0b20c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:38.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.508+0000 7fb72d000640 1 --2- 172.21.15.32:0/1378471726 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb728109910 0x7fb7281a15a0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:38.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.508+0000 7fb72d000640 1 --2- 172.21.15.32:0/1378471726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb728104c10 0x7fb72819a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:38.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.508+0000 7fb72d000640 1 --2- 172.21.15.32:0/1378471726 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb7281039d0 0x7fb72819a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:38.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.508+0000 7fb72d000640 1 -- 172.21.15.32:0/1378471726 >> 172.21.15.32:0/1378471726 conn(0x7fb7280febc0 msgr2=0x7fb7281058f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:38.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.508+0000 7fb72d000640 1 -- 172.21.15.32:0/1378471726 shutdown_connections 2024-04-30T08:36:38.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:38.508+0000 7fb72d000640 1 -- 172.21.15.32:0/1378471726 wait complete. 2024-04-30T08:36:39.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:38 smithi032 ceph-mon[28331]: pgmap v469: 129 pgs: 129 active+clean; 675 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 938 B/s wr, 11 op/s 2024-04-30T08:36:39.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:38 smithi071 ceph-mon[33385]: pgmap v469: 129 pgs: 129 active+clean; 675 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 938 B/s wr, 11 op/s 2024-04-30T08:36:39.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:38 smithi189 ceph-mon[33038]: pgmap v469: 129 pgs: 129 active+clean; 675 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 938 B/s wr, 11 op/s 2024-04-30T08:36:41.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:40 smithi071 ceph-mon[33385]: pgmap v470: 129 pgs: 129 active+clean; 676 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-04-30T08:36:41.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:40 smithi189 ceph-mon[33038]: pgmap v470: 129 pgs: 129 active+clean; 676 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-04-30T08:36:41.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:40 smithi032 ceph-mon[28331]: pgmap v470: 129 pgs: 129 active+clean; 676 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.9 KiB/s rd, 1.2 KiB/s wr, 14 op/s 2024-04-30T08:36:43.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:42 smithi071 ceph-mon[33385]: pgmap v471: 129 pgs: 129 active+clean; 676 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 853 B/s wr, 11 op/s 2024-04-30T08:36:43.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:42 smithi189 ceph-mon[33038]: pgmap v471: 129 pgs: 129 active+clean; 676 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 853 B/s wr, 11 op/s 2024-04-30T08:36:43.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:42 smithi032 ceph-mon[28331]: pgmap v471: 129 pgs: 129 active+clean; 676 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 853 B/s wr, 11 op/s 2024-04-30T08:36:45.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:44 smithi071 ceph-mon[33385]: pgmap v472: 129 pgs: 129 active+clean; 676 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-30T08:36:45.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:44 smithi189 ceph-mon[33038]: pgmap v472: 129 pgs: 129 active+clean; 676 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-30T08:36:45.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:44 smithi032 ceph-mon[28331]: pgmap v472: 129 pgs: 129 active+clean; 676 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 597 B/s wr, 10 op/s 2024-04-30T08:36:47.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:46 smithi071 ceph-mon[33385]: pgmap v473: 129 pgs: 129 active+clean; 676 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-30T08:36:47.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:46 smithi189 ceph-mon[33038]: pgmap v473: 129 pgs: 129 active+clean; 676 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-30T08:36:47.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:46 smithi032 ceph-mon[28331]: pgmap v473: 129 pgs: 129 active+clean; 676 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-30T08:36:49.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:48 smithi071 ceph-mon[33385]: pgmap v474: 129 pgs: 129 active+clean; 676 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-30T08:36:49.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:48 smithi189 ceph-mon[33038]: pgmap v474: 129 pgs: 129 active+clean; 676 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-30T08:36:49.358 INFO:tasks.workunit.client.0.smithi032.stderr:Updating files: 59% (7298/12271) Updating files: 60% (7363/12271) Updating files: 61% (7486/12271) Updating files: 62% (7609/12271) Updating files: 63% (7731/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 97% (12023/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-04-30T08:36:49.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:48 smithi032 ceph-mon[28331]: pgmap v474: 129 pgs: 129 active+clean; 676 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-04-30T08:36:50.092 INFO:tasks.workunit.client.0.smithi032.stderr:Note: switching to '8feaf7d9864e980041b042fd5d964ac18adfd20f'. 2024-04-30T08:36:50.093 INFO:tasks.workunit.client.0.smithi032.stderr: 2024-04-30T08:36:50.093 INFO:tasks.workunit.client.0.smithi032.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-30T08:36:50.093 INFO:tasks.workunit.client.0.smithi032.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-30T08:36:50.093 INFO:tasks.workunit.client.0.smithi032.stderr:state without impacting any branches by switching back to a branch. 2024-04-30T08:36:50.093 INFO:tasks.workunit.client.0.smithi032.stderr: 2024-04-30T08:36:50.093 INFO:tasks.workunit.client.0.smithi032.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-30T08:36:50.093 INFO:tasks.workunit.client.0.smithi032.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-30T08:36:50.093 INFO:tasks.workunit.client.0.smithi032.stderr: 2024-04-30T08:36:50.093 INFO:tasks.workunit.client.0.smithi032.stderr: git switch -c 2024-04-30T08:36:50.093 INFO:tasks.workunit.client.0.smithi032.stderr: 2024-04-30T08:36:50.093 INFO:tasks.workunit.client.0.smithi032.stderr:Or undo this operation with: 2024-04-30T08:36:50.093 INFO:tasks.workunit.client.0.smithi032.stderr: 2024-04-30T08:36:50.093 INFO:tasks.workunit.client.0.smithi032.stderr: git switch - 2024-04-30T08:36:50.094 INFO:tasks.workunit.client.0.smithi032.stderr: 2024-04-30T08:36:50.094 INFO:tasks.workunit.client.0.smithi032.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-30T08:36:50.094 INFO:tasks.workunit.client.0.smithi032.stderr: 2024-04-30T08:36:50.094 INFO:tasks.workunit.client.0.smithi032.stderr:HEAD is now at 8feaf7d9864 Merge PR #56810 into wip-pdonnell-testing-20240429.210911-debug 2024-04-30T08:36:50.107 DEBUG:teuthology.orchestra.run.smithi032:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-30T08:36:50.171 INFO:tasks.workunit.client.0.smithi032.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-30T08:36:50.179 INFO:tasks.workunit.client.0.smithi032.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-30T08:36:50.179 INFO:tasks.workunit.client.0.smithi032.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-30T08:36:51.084 INFO:tasks.workunit.client.0.smithi032.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-30T08:36:51.210 INFO:tasks.workunit.client.0.smithi032.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-30T08:36:51.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:50 smithi071 ceph-mon[33385]: pgmap v475: 129 pgs: 129 active+clean; 676 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-30T08:36:51.299 INFO:tasks.workunit.client.0.smithi032.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-30T08:36:51.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:50 smithi189 ceph-mon[33038]: pgmap v475: 129 pgs: 129 active+clean; 676 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-30T08:36:51.306 INFO:tasks.workunit.client.0.smithi032.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-30T08:36:51.306 INFO:tasks.workunit.client.0.smithi032.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-30T08:36:51.396 INFO:tasks.workunit.client.0.smithi032.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-30T08:36:51.407 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-30T08:36:51.408 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-30T08:36:51.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:50 smithi032 ceph-mon[28331]: pgmap v475: 129 pgs: 129 active+clean; 676 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-04-30T08:36:51.460 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2024-04-30T08:36:51.461 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2024-04-30T08:36:51.461 DEBUG:teuthology.orchestra.run.smithi032:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=8feaf7d9864e980041b042fd5d964ac18adfd20f TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-04-30T08:36:51.587 INFO:tasks.workunit.client.0.smithi032.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-04-30T08:36:51.587 INFO:tasks.workunit.client.0.smithi032.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/suites 2024-04-30T08:36:51.587 INFO:tasks.workunit.client.0.smithi032.stderr:+ trap -- retry EXIT 2024-04-30T08:36:51.587 INFO:tasks.workunit.client.0.smithi032.stderr:+ rm -rf ffsb 2024-04-30T08:36:51.590 INFO:tasks.workunit.client.0.smithi032.stderr:+ timeout 1800 git clone https://git.ceph.com/ffsb.git --depth 1 2024-04-30T08:36:51.599 INFO:tasks.workunit.client.0.smithi032.stderr:Cloning into 'ffsb'... 2024-04-30T08:36:51.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.744+0000 7ff574400640 1 -- 172.21.15.32:0/3300901322 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff56c103bb0 msgr2=0x7ff56c1048b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:51.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.744+0000 7ff574400640 1 --2- 172.21.15.32:0/3300901322 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff56c103bb0 0x7ff56c1048b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7ff560009900 tx=0x7ff56002a3f0 comp rx=0 tx=0).stop 2024-04-30T08:36:51.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.744+0000 7ff574400640 1 -- 172.21.15.32:0/3300901322 shutdown_connections 2024-04-30T08:36:51.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.744+0000 7ff574400640 1 --2- 172.21.15.32:0/3300901322 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff56c109af0 0x7ff56c10bf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:51.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.744+0000 7ff574400640 1 --2- 172.21.15.32:0/3300901322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff56c104df0 0x7ff56c105200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:51.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.744+0000 7ff574400640 1 --2- 172.21.15.32:0/3300901322 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff56c103bb0 0x7ff56c1048b0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:51.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.744+0000 7ff574400640 1 -- 172.21.15.32:0/3300901322 >> 172.21.15.32:0/3300901322 conn(0x7ff56c0fed80 msgr2=0x7ff56c101250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:51.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.744+0000 7ff574400640 1 -- 172.21.15.32:0/3300901322 shutdown_connections 2024-04-30T08:36:51.746 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.744+0000 7ff574400640 1 -- 172.21.15.32:0/3300901322 wait complete. 2024-04-30T08:36:51.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.745+0000 7ff574400640 1 Processor -- start 2024-04-30T08:36:51.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.745+0000 7ff574400640 1 -- start start 2024-04-30T08:36:51.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.746+0000 7ff574400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff56c103bb0 0x7ff56c19c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:51.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.746+0000 7ff574400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff56c104df0 0x7ff56c19ca50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:51.748 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.746+0000 7ff574400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff56c109af0 0x7ff56c1a39e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:51.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.746+0000 7ff574400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff56c10f700 con 0x7ff56c103bb0 2024-04-30T08:36:51.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.746+0000 7ff574400640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff56c10f580 con 0x7ff56c104df0 2024-04-30T08:36:51.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.746+0000 7ff574400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff56c10f880 con 0x7ff56c109af0 2024-04-30T08:36:51.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.746+0000 7ff571a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff56c103bb0 0x7ff56c19c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:51.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.746+0000 7ff571a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff56c103bb0 0x7ff56c19c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48230/0 (socket says 172.21.15.32:48230) 2024-04-30T08:36:51.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.746+0000 7ff571a00640 1 -- 172.21.15.32:0/1453857509 learned_addr learned my addr 172.21.15.32:0/1453857509 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:51.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.746+0000 7ff571000640 1 --2- 172.21.15.32:0/1453857509 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff56c104df0 0x7ff56c19ca50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:51.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.746+0000 7ff572400640 1 --2- 172.21.15.32:0/1453857509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff56c109af0 0x7ff56c1a39e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:51.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.747+0000 7ff571a00640 1 -- 172.21.15.32:0/1453857509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff56c109af0 msgr2=0x7ff56c1a39e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:51.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.747+0000 7ff571a00640 1 --2- 172.21.15.32:0/1453857509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff56c109af0 0x7ff56c1a39e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:51.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.747+0000 7ff571a00640 1 -- 172.21.15.32:0/1453857509 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff56c104df0 msgr2=0x7ff56c19ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:51.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.747+0000 7ff571a00640 1 --2- 172.21.15.32:0/1453857509 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff56c104df0 0x7ff56c19ca50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:51.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.747+0000 7ff571a00640 1 -- 172.21.15.32:0/1453857509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff56c1a4050 con 0x7ff56c103bb0 2024-04-30T08:36:51.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.747+0000 7ff571000640 1 --2- 172.21.15.32:0/1453857509 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff56c104df0 0x7ff56c19ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:36:51.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.747+0000 7ff571a00640 1 --2- 172.21.15.32:0/1453857509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff56c103bb0 0x7ff56c19c510 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff560031440 tx=0x7ff560003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:51.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.747+0000 7ff56a000640 1 -- 172.21.15.32:0/1453857509 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff56003f070 con 0x7ff56c103bb0 2024-04-30T08:36:51.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.747+0000 7ff56a000640 1 -- 172.21.15.32:0/1453857509 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff560031c50 con 0x7ff56c103bb0 2024-04-30T08:36:51.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.747+0000 7ff574400640 1 -- 172.21.15.32:0/1453857509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff56c1a42e0 con 0x7ff56c103bb0 2024-04-30T08:36:51.751 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.747+0000 7ff56a000640 1 -- 172.21.15.32:0/1453857509 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff560035030 con 0x7ff56c103bb0 2024-04-30T08:36:51.751 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.748+0000 7ff574400640 1 -- 172.21.15.32:0/1453857509 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff56c1a4820 con 0x7ff56c103bb0 2024-04-30T08:36:51.753 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.752+0000 7ff57b375740 1 Processor -- start 2024-04-30T08:36:51.753 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.752+0000 7ff57b375740 1 -- start start 2024-04-30T08:36:51.755 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.753+0000 7ff57b375740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a651331b0 0x558a654d50e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:51.755 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.753+0000 7ff57b375740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a6510a2f0 0x558a654d5660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:51.755 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.753+0000 7ff57b375740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a651124d0 0x558a654dc560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:51.755 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.753+0000 7ff57b375740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558a65439550 con 0x558a6510a2f0 2024-04-30T08:36:51.755 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.753+0000 7ff57b375740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x558a653debc0 con 0x558a651331b0 2024-04-30T08:36:51.755 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.753+0000 7ff57b375740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558a65435510 con 0x558a651124d0 2024-04-30T08:36:51.755 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.753+0000 7ff56a000640 1 -- 172.21.15.32:0/1453857509 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7ff560042070 con 0x7ff56c103bb0 2024-04-30T08:36:51.755 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.753+0000 7ff572400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a651331b0 0x558a654d50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:51.755 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.753+0000 7ff572400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a651331b0 0x558a654d50e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:44280/0 (socket says 172.21.15.32:44280) 2024-04-30T08:36:51.755 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.753+0000 7ff572400640 1 -- 172.21.15.32:0/3758677791 learned_addr learned my addr 172.21.15.32:0/3758677791 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:51.755 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.753+0000 7ff571000640 1 --2- 172.21.15.32:0/3758677791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a651124d0 0x558a654dc560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:51.755 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.753+0000 7ff572400640 1 -- 172.21.15.32:0/3758677791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a651124d0 msgr2=0x558a654dc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:51.755 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.753+0000 7ff572400640 1 --2- 172.21.15.32:0/3758677791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a651124d0 0x558a654dc560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:51.756 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.753+0000 7ff571a00640 1 --2- 172.21.15.32:0/3758677791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a6510a2f0 0x558a654d5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:51.756 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.753+0000 7ff572400640 1 -- 172.21.15.32:0/3758677791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a6510a2f0 msgr2=0x558a654d5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:51.756 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.753+0000 7ff572400640 1 --2- 172.21.15.32:0/3758677791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a6510a2f0 0x558a654d5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:51.756 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.754+0000 7ff572400640 1 -- 172.21.15.32:0/3758677791 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a6514b210 con 0x558a651331b0 2024-04-30T08:36:51.756 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.754+0000 7ff571a00640 1 --2- 172.21.15.32:0/3758677791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a6510a2f0 0x558a654d5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-30T08:36:51.757 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.754+0000 7ff572400640 1 --2- 172.21.15.32:0/3758677791 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a651331b0 0x558a654d50e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff564009fd0 tx=0x7ff5640118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cae2f00cc67329b6 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:51.758 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.754+0000 7ff55ea00640 1 -- 172.21.15.32:0/3758677791 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff564004d20 con 0x558a651331b0 2024-04-30T08:36:51.758 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.754+0000 7ff55ea00640 1 -- 172.21.15.32:0/3758677791 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff564004050 con 0x558a651331b0 2024-04-30T08:36:51.758 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.754+0000 7ff55ea00640 1 -- 172.21.15.32:0/3758677791 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff564005280 con 0x558a651331b0 2024-04-30T08:36:51.758 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.755+0000 7ff56a000640 1 --2- 172.21.15.32:0/1453857509 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff5540afb60 0x7ff5540b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:51.758 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.755+0000 7ff56a000640 1 -- 172.21.15.32:0/1453857509 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7ff5600ff730 con 0x7ff56c103bb0 2024-04-30T08:36:51.758 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.755+0000 7ff571a00640 1 --2- 172.21.15.32:0/1453857509 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff5540afb60 0x7ff5540b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:51.758 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.755+0000 7ff57b375740 1 -- 172.21.15.32:0/3758677791 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a651331b0 msgr2=0x558a654d50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:51.758 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.755+0000 7ff57b375740 1 --2- 172.21.15.32:0/3758677791 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a651331b0 0x558a654d50e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff564009fd0 tx=0x7ff5640118d0 comp rx=0 tx=0).stop 2024-04-30T08:36:51.758 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.755+0000 7ff57b375740 1 -- 172.21.15.32:0/3758677791 shutdown_connections 2024-04-30T08:36:51.758 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.755+0000 7ff57b375740 1 --2- 172.21.15.32:0/3758677791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a651124d0 0x558a654dc560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:51.758 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.755+0000 7ff57b375740 1 --2- 172.21.15.32:0/3758677791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a6510a2f0 0x558a654d5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:51.758 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.755+0000 7ff57b375740 1 --2- 172.21.15.32:0/3758677791 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a651331b0 0x558a654d50e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:51.758 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.755+0000 7ff57b375740 1 -- 172.21.15.32:0/3758677791 >> 172.21.15.32:0/3758677791 conn(0x558a65127300 msgr2=0x558a654d0a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:51.759 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.756+0000 7ff57b375740 1 -- 172.21.15.32:0/3758677791 shutdown_connections 2024-04-30T08:36:51.759 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.756+0000 7ff57b375740 1 -- 172.21.15.32:0/3758677791 wait complete. 2024-04-30T08:36:51.759 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.756+0000 7ff571a00640 1 --2- 172.21.15.32:0/1453857509 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff5540afb60 0x7ff5540b2070 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff5600c7880 tx=0x7ff56003b800 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:51.759 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.756+0000 7ff57b375740 1 -- start start 2024-04-30T08:36:51.761 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.757+0000 7ff57b375740 1 Processor -- start 2024-04-30T08:36:51.761 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.757+0000 7ff57b375740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a6510a2f0 0x558a654debd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:51.762 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.757+0000 7ff57b375740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a651124d0 0x558a654df110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:51.762 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.757+0000 7ff57b375740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a651331b0 0x558a654e6070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:51.762 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.757+0000 7ff57b375740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558a654dcfb0 con 0x558a651331b0 2024-04-30T08:36:51.762 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.757+0000 7ff57b375740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x558a653f2660 con 0x558a651124d0 2024-04-30T08:36:51.762 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.757+0000 7ff57b375740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558a654dd130 con 0x558a6510a2f0 2024-04-30T08:36:51.762 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.757+0000 7ff57b375740 10 client.0 _ll_register_callbacks cb 0x558a653bc400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:36:51.762 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.758+0000 7ff571000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a651124d0 0x558a654df110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:51.762 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.758+0000 7ff571000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a651124d0 0x558a654df110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:44288/0 (socket says 172.21.15.32:44288) 2024-04-30T08:36:51.762 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.758+0000 7ff571000640 1 -- 172.21.15.32:0/4212913941 learned_addr learned my addr 172.21.15.32:0/4212913941 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:51.762 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.758+0000 7ff572400640 1 --2- 172.21.15.32:0/4212913941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a651331b0 0x558a654e6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:51.762 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.758+0000 7ff572400640 1 --2- 172.21.15.32:0/4212913941 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a6510a2f0 0x558a654debd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:51.762 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.758+0000 7ff571000640 1 -- 172.21.15.32:0/4212913941 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a6510a2f0 msgr2=0x558a654debd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:51.763 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.758+0000 7ff571000640 1 --2- 172.21.15.32:0/4212913941 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a6510a2f0 0x558a654debd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:51.763 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.758+0000 7ff571000640 1 -- 172.21.15.32:0/4212913941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a651331b0 msgr2=0x558a654e6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:51.763 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.758+0000 7ff571000640 1 --2- 172.21.15.32:0/4212913941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a651331b0 0x558a654e6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:51.763 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.758+0000 7ff571000640 1 -- 172.21.15.32:0/4212913941 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a65217620 con 0x558a651124d0 2024-04-30T08:36:51.763 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.758+0000 7ff572400640 1 --2- 172.21.15.32:0/4212913941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a651331b0 0x558a654e6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:36:51.763 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.758+0000 7ff571000640 1 --2- 172.21.15.32:0/4212913941 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a651124d0 0x558a654df110 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff558004310 tx=0x7ff5580039c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f48ffdd960897a5e server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:51.763 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.758+0000 7ff553400640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:36:51.763 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.759+0000 7ff553400640 1 -- 172.21.15.32:0/4212913941 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff558005220 con 0x558a651124d0 2024-04-30T08:36:51.763 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.759+0000 7ff57b375740 1 -- 172.21.15.32:0/4212913941 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff564004050 con 0x558a651124d0 2024-04-30T08:36:51.763 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.759+0000 7ff553400640 1 -- 172.21.15.32:0/4212913941 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7ff5580053c0 con 0x558a651124d0 2024-04-30T08:36:51.763 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.759+0000 7ff553400640 1 -- 172.21.15.32:0/4212913941 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff558023050 con 0x558a651124d0 2024-04-30T08:36:51.763 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.759+0000 7ff57b375740 10 client.26054 fetch_fsmap learned FSMap version 28 2024-04-30T08:36:51.763 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.759+0000 7ff57b375740 1 -- 172.21.15.32:0/4212913941 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558a654e9030 con 0x558a651124d0 2024-04-30T08:36:51.763 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.759+0000 7ff553400640 1 -- 172.21.15.32:0/4212913941 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff564004050 con 0x558a651124d0 2024-04-30T08:36:51.764 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.760+0000 7ff553400640 1 -- 172.21.15.32:0/4212913941 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff558019030 con 0x558a651124d0 2024-04-30T08:36:51.764 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.760+0000 7ff57b375740 10 client.26054 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:36:51.764 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.760+0000 7ff57b375740 10 client.26054 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-30T08:36:51.764 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.760+0000 7ff57b375740 20 client.26054 populate_metadata read hostname 'smithi032' 2024-04-30T08:36:51.764 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.760+0000 7ff57b375740 1 --2- 172.21.15.32:0/4212913941 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x558a650f1d80 0x558a654ec700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:51.764 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.760+0000 7ff57b375740 4 client.26054 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:36:51.764 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.760+0000 7ff57b375740 1 -- 172.21.15.32:0/4212913941 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558a65447160 con 0x558a650f1d80 2024-04-30T08:36:51.764 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.760+0000 7ff572400640 1 --2- 172.21.15.32:0/4212913941 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x558a650f1d80 0x558a654ec700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:51.765 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.761+0000 7ff572400640 1 --2- 172.21.15.32:0/4212913941 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x558a650f1d80 0x558a654ec700 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3979174e93eea671 server_cookie=7e9d1fe700c38d6 in_seq=0 out_seq=0 2024-04-30T08:36:51.765 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.761+0000 7ff553400640 10 client.26054 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:51.765 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.763+0000 7ff553400640 1 -- 172.21.15.32:0/4212913941 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x558a65447160 con 0x558a650f1d80 2024-04-30T08:36:51.765 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.763+0000 7ff553400640 10 client.26054 handle_command_reply: tid=0 2024-04-30T08:36:51.765 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.763+0000 7ff57b375740 1 client.26054 shutdown 2024-04-30T08:36:51.765 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.764+0000 7ff57b375740 1 -- 172.21.15.32:0/4212913941 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a651124d0 msgr2=0x558a654df110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:51.765 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.764+0000 7ff57b375740 1 --2- 172.21.15.32:0/4212913941 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a651124d0 0x558a654df110 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff558004310 tx=0x7ff5580039c0 comp rx=0 tx=0).stop 2024-04-30T08:36:51.765 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.764+0000 7ff57b375740 1 -- 172.21.15.32:0/4212913941 shutdown_connections 2024-04-30T08:36:51.766 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.764+0000 7ff57b375740 1 --2- 172.21.15.32:0/4212913941 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x558a650f1d80 0x558a654ec700 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:51.766 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.764+0000 7ff57b375740 1 --2- 172.21.15.32:0/4212913941 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a651331b0 0x558a654e6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:51.766 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.764+0000 7ff57b375740 1 --2- 172.21.15.32:0/4212913941 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a651124d0 0x558a654df110 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:51.766 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.764+0000 7ff57b375740 1 --2- 172.21.15.32:0/4212913941 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a6510a2f0 0x558a654debd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:51.766 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.764+0000 7ff57b375740 1 -- 172.21.15.32:0/4212913941 >> 172.21.15.32:0/4212913941 conn(0x558a65127300 msgr2=0x558a654cfa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:51.766 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.764+0000 7ff553400640 0 client.26054 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:51.766 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.764+0000 7ff57b375740 1 -- 172.21.15.32:0/4212913941 shutdown_connections 2024-04-30T08:36:51.767 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.764+0000 7ff57b375740 1 -- 172.21.15.32:0/4212913941 wait complete. 2024-04-30T08:36:51.767 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.764+0000 7ff57b375740 20 client.26054 trim_cache size 0 max 16384 2024-04-30T08:36:51.783 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.781+0000 7ff57b375740 1 Processor -- start 2024-04-30T08:36:51.783 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.781+0000 7ff57b375740 1 -- start start 2024-04-30T08:36:51.783 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.781+0000 7ff57b375740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a6510a2f0 0x558a654d3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:51.783 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.781+0000 7ff57b375740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a651124d0 0x558a654d44c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:51.783 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.781+0000 7ff57b375740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a651331b0 0x558a654f1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:51.783 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.781+0000 7ff57b375740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff56c1aedb0 con 0x558a651124d0 2024-04-30T08:36:51.783 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.781+0000 7ff57b375740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff56c1b08f0 con 0x558a651331b0 2024-04-30T08:36:51.784 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.781+0000 7ff57b375740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff56c1af480 con 0x558a6510a2f0 2024-04-30T08:36:51.785 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.782+0000 7ff571000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a651124d0 0x558a654d44c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:51.785 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.782+0000 7ff572400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a6510a2f0 0x558a654d3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:51.785 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.782+0000 7ff571000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a651124d0 0x558a654d44c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48260/0 (socket says 172.21.15.32:48260) 2024-04-30T08:36:51.785 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.782+0000 7ff571000640 1 -- 172.21.15.32:0/2584459557 learned_addr learned my addr 172.21.15.32:0/2584459557 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:51.785 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.782+0000 7ff572400640 1 --2- 172.21.15.32:0/2584459557 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a651331b0 0x558a654f1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:51.785 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.782+0000 7ff571000640 1 -- 172.21.15.32:0/2584459557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a6510a2f0 msgr2=0x558a654d3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:51.785 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.782+0000 7ff571000640 1 --2- 172.21.15.32:0/2584459557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a6510a2f0 0x558a654d3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:51.785 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.782+0000 7ff571000640 1 -- 172.21.15.32:0/2584459557 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a651331b0 msgr2=0x558a654f1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:51.786 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.782+0000 7ff571000640 1 --2- 172.21.15.32:0/2584459557 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a651331b0 0x558a654f1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:51.786 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.782+0000 7ff571000640 1 -- 172.21.15.32:0/2584459557 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558a654cc260 con 0x558a651124d0 2024-04-30T08:36:51.786 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.783+0000 7ff571000640 1 --2- 172.21.15.32:0/2584459557 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a651124d0 0x558a654d44c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff558000c00 tx=0x7ff558005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df4c1581fb03331d server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:51.786 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.783+0000 7ff55e000640 1 -- 172.21.15.32:0/2584459557 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5580116d0 con 0x558a651124d0 2024-04-30T08:36:51.786 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.783+0000 7ff55e000640 1 -- 172.21.15.32:0/2584459557 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff558011b90 con 0x558a651124d0 2024-04-30T08:36:51.786 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.783+0000 7ff55e000640 1 -- 172.21.15.32:0/2584459557 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff558010440 con 0x558a651124d0 2024-04-30T08:36:51.789 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.784+0000 7ff57b375740 1 -- 172.21.15.32:0/2584459557 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a651124d0 msgr2=0x558a654d44c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:51.789 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.784+0000 7ff57b375740 1 --2- 172.21.15.32:0/2584459557 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a651124d0 0x558a654d44c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7ff558000c00 tx=0x7ff558005970 comp rx=0 tx=0).stop 2024-04-30T08:36:51.789 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.784+0000 7ff57b375740 1 -- 172.21.15.32:0/2584459557 shutdown_connections 2024-04-30T08:36:51.789 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.784+0000 7ff57b375740 1 --2- 172.21.15.32:0/2584459557 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a651331b0 0x558a654f1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:51.790 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.784+0000 7ff57b375740 1 --2- 172.21.15.32:0/2584459557 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a651124d0 0x558a654d44c0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:51.790 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.784+0000 7ff57b375740 1 --2- 172.21.15.32:0/2584459557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a6510a2f0 0x558a654d3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:51.790 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.784+0000 7ff57b375740 1 -- 172.21.15.32:0/2584459557 >> 172.21.15.32:0/2584459557 conn(0x558a65127300 msgr2=0x558a654d3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:51.790 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.784+0000 7ff57b375740 1 -- 172.21.15.32:0/2584459557 shutdown_connections 2024-04-30T08:36:51.790 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.784+0000 7ff57b375740 1 -- 172.21.15.32:0/2584459557 wait complete. 2024-04-30T08:36:51.790 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.785+0000 7ff57b375740 1 -- start start 2024-04-30T08:36:51.790 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.785+0000 7ff57b375740 1 Processor -- start 2024-04-30T08:36:51.790 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.785+0000 7ff57b375740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a6510a2f0 0x558a654e5d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:51.790 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.785+0000 7ff57b375740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a651124d0 0x558a654e62b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:51.790 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.785+0000 7ff57b375740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a651331b0 0x558a6555b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:51.790 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.785+0000 7ff57b375740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff56c1b0220 con 0x558a651331b0 2024-04-30T08:36:51.791 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.785+0000 7ff57b375740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff56c1afb50 con 0x558a651124d0 2024-04-30T08:36:51.791 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.785+0000 7ff57b375740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff56c1acd70 con 0x558a6510a2f0 2024-04-30T08:36:51.791 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.785+0000 7ff57b375740 10 client.0 _ll_register_callbacks cb 0x7ff55800bb20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:36:51.791 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.786+0000 7ff572400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a651331b0 0x558a6555b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:51.791 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.786+0000 7ff571000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a651124d0 0x558a654e62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:51.791 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.786+0000 7ff571000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a651124d0 0x558a654e62b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:44306/0 (socket says 172.21.15.32:44306) 2024-04-30T08:36:51.791 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.786+0000 7ff571000640 1 -- 172.21.15.32:0/3674268548 learned_addr learned my addr 172.21.15.32:0/3674268548 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:51.791 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.786+0000 7ff572400640 1 --2- 172.21.15.32:0/3674268548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a6510a2f0 0x558a654e5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:51.791 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.786+0000 7ff571000640 1 -- 172.21.15.32:0/3674268548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a6510a2f0 msgr2=0x558a654e5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:51.791 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.786+0000 7ff571000640 1 --2- 172.21.15.32:0/3674268548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a6510a2f0 0x558a654e5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:51.791 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.786+0000 7ff571000640 1 -- 172.21.15.32:0/3674268548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a651331b0 msgr2=0x558a6555b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:51.791 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.786+0000 7ff571000640 1 --2- 172.21.15.32:0/3674268548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a651331b0 0x558a6555b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:51.791 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.786+0000 7ff571000640 1 -- 172.21.15.32:0/3674268548 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558a6555bcd0 con 0x558a651124d0 2024-04-30T08:36:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.787+0000 7ff571000640 1 --2- 172.21.15.32:0/3674268548 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a651124d0 0x558a654e62b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff558004080 tx=0x7ff5580119a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=366b729ab6b52ced server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.787+0000 7ff553400640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:36:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.787+0000 7ff553400640 1 -- 172.21.15.32:0/3674268548 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff558035070 con 0x558a651124d0 2024-04-30T08:36:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.787+0000 7ff57b375740 1 -- 172.21.15.32:0/3674268548 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff558011b90 con 0x558a651124d0 2024-04-30T08:36:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.787+0000 7ff553400640 1 -- 172.21.15.32:0/3674268548 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7ff558027430 con 0x558a651124d0 2024-04-30T08:36:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.787+0000 7ff553400640 1 -- 172.21.15.32:0/3674268548 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7ff55803e050 con 0x558a651124d0 2024-04-30T08:36:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.787+0000 7ff57b375740 10 client.26060 fetch_fsmap learned FSMap version 28 2024-04-30T08:36:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.787+0000 7ff57b375740 1 -- 172.21.15.32:0/3674268548 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x558a6555cf60 con 0x558a651124d0 2024-04-30T08:36:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.787+0000 7ff553400640 1 -- 172.21.15.32:0/3674268548 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff558011b90 con 0x558a651124d0 2024-04-30T08:36:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.788+0000 7ff553400640 1 -- 172.21.15.32:0/3674268548 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 28) ==== 8362+0+0 (secure 0 0 0) 0x7ff558038020 con 0x558a651124d0 2024-04-30T08:36:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.788+0000 7ff57b375740 10 client.26060 fetch_fsmap finished waiting for FSMap version 28 2024-04-30T08:36:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.788+0000 7ff57b375740 10 client.26060 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-30T08:36:51.792 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.788+0000 7ff57b375740 20 client.26060 populate_metadata read hostname 'smithi032' 2024-04-30T08:36:51.793 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.788+0000 7ff57b375740 1 --2- 172.21.15.32:0/3674268548 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x558a650c71c0 0x558a6555f890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:51.793 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.788+0000 7ff57b375740 4 client.26060 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-30T08:36:51.793 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.788+0000 7ff57b375740 1 -- 172.21.15.32:0/3674268548 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558a65440fd0 con 0x558a650c71c0 2024-04-30T08:36:51.793 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.788+0000 7ff572400640 1 --2- 172.21.15.32:0/3674268548 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x558a650c71c0 0x558a6555f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:51.793 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.789+0000 7ff572400640 1 --2- 172.21.15.32:0/3674268548 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x558a650c71c0 0x558a6555f890 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c307a3b549015db server_cookie=d18e0d994408e250 in_seq=0 out_seq=0 2024-04-30T08:36:51.793 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:51.789+0000 7ff553400640 10 client.26060 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:52.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:52.714+0000 7ff553400640 1 -- 172.21.15.32:0/3674268548 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558a65440fd0 con 0x558a650c71c0 2024-04-30T08:36:52.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:52.714+0000 7ff553400640 10 client.26060 handle_command_reply: tid=0 2024-04-30T08:36:52.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:52.714+0000 7ff57b375740 1 client.26060 shutdown 2024-04-30T08:36:52.716 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:52.715+0000 7ff57b375740 1 -- 172.21.15.32:0/3674268548 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a651124d0 msgr2=0x558a654e62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:52.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:52.715+0000 7ff57b375740 1 --2- 172.21.15.32:0/3674268548 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a651124d0 0x558a654e62b0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7ff558004080 tx=0x7ff5580119a0 comp rx=0 tx=0).stop 2024-04-30T08:36:52.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:52.715+0000 7ff57b375740 1 -- 172.21.15.32:0/3674268548 shutdown_connections 2024-04-30T08:36:52.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:52.715+0000 7ff57b375740 1 --2- 172.21.15.32:0/3674268548 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x558a650c71c0 0x558a6555f890 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:52.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:52.715+0000 7ff57b375740 1 --2- 172.21.15.32:0/3674268548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558a651331b0 0x558a6555b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:52.717 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:52.715+0000 7ff57b375740 1 --2- 172.21.15.32:0/3674268548 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558a651124d0 0x558a654e62b0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:52.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:52.715+0000 7ff57b375740 1 --2- 172.21.15.32:0/3674268548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558a6510a2f0 0x558a654e5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:52.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:52.715+0000 7ff553400640 0 client.26060 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:52.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:52.715+0000 7ff57b375740 1 -- 172.21.15.32:0/3674268548 >> 172.21.15.32:0/3674268548 conn(0x558a65127300 msgr2=0x558a654d42a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:52.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:52.716+0000 7ff57b375740 1 -- 172.21.15.32:0/3674268548 shutdown_connections 2024-04-30T08:36:52.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:52.716+0000 7ff57b375740 1 -- 172.21.15.32:0/3674268548 wait complete. 2024-04-30T08:36:52.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:52.716+0000 7ff57b375740 20 client.26060 trim_cache size 0 max 16384 2024-04-30T08:36:52.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-30T08:36:52.718 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-30T08:36:52.719 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-30T08:36:52.719 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-30T08:36:52.719 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:52.718+0000 7ff574400640 1 -- 172.21.15.32:0/1453857509 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff5540afb60 msgr2=0x7ff5540b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:52.720 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:52.718+0000 7ff574400640 1 --2- 172.21.15.32:0/1453857509 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff5540afb60 0x7ff5540b2070 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7ff5600c7880 tx=0x7ff56003b800 comp rx=0 tx=0).stop 2024-04-30T08:36:52.720 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:52.718+0000 7ff574400640 1 -- 172.21.15.32:0/1453857509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff56c103bb0 msgr2=0x7ff56c19c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:52.720 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:52.718+0000 7ff574400640 1 --2- 172.21.15.32:0/1453857509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff56c103bb0 0x7ff56c19c510 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7ff560031440 tx=0x7ff560003c10 comp rx=0 tx=0).stop 2024-04-30T08:36:52.720 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:52.718+0000 7ff574400640 1 -- 172.21.15.32:0/1453857509 shutdown_connections 2024-04-30T08:36:52.720 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:52.718+0000 7ff574400640 1 --2- 172.21.15.32:0/1453857509 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff5540afb60 0x7ff5540b2070 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:52.720 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:52.718+0000 7ff574400640 1 --2- 172.21.15.32:0/1453857509 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff56c109af0 0x7ff56c1a39e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:52.720 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:52.718+0000 7ff574400640 1 --2- 172.21.15.32:0/1453857509 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff56c104df0 0x7ff56c19ca50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:52.720 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:52.718+0000 7ff574400640 1 --2- 172.21.15.32:0/1453857509 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff56c103bb0 0x7ff56c19c510 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:52.720 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:52.718+0000 7ff574400640 1 -- 172.21.15.32:0/1453857509 >> 172.21.15.32:0/1453857509 conn(0x7ff56c0fed80 msgr2=0x7ff56c0ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:52.720 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:52.718+0000 7ff574400640 1 -- 172.21.15.32:0/1453857509 shutdown_connections 2024-04-30T08:36:52.720 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:36:52.718+0000 7ff574400640 1 -- 172.21.15.32:0/1453857509 wait complete. 2024-04-30T08:36:52.926 INFO:tasks.workunit.client.0.smithi032.stderr:+ trap - EXIT 2024-04-30T08:36:52.927 INFO:tasks.workunit.client.0.smithi032.stderr:+ cd ffsb 2024-04-30T08:36:52.927 INFO:tasks.workunit.client.0.smithi032.stderr:+ ./configure 2024-04-30T08:36:53.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:52 smithi071 ceph-mon[33385]: pgmap v476: 129 pgs: 129 active+clean; 676 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-30T08:36:53.298 INFO:tasks.workunit.client.0.smithi032.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-04-30T08:36:53.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:52 smithi189 ceph-mon[33038]: pgmap v476: 129 pgs: 129 active+clean; 676 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-30T08:36:53.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:52 smithi032 ceph-mon[28331]: pgmap v476: 129 pgs: 129 active+clean; 676 KiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-30T08:36:54.320 INFO:tasks.workunit.client.0.smithi032.stdout:checking whether build environment is sane... yes 2024-04-30T08:36:54.346 INFO:tasks.workunit.client.0.smithi032.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-04-30T08:36:54.346 INFO:tasks.workunit.client.0.smithi032.stdout:checking for gawk... gawk 2024-04-30T08:36:54.378 INFO:tasks.workunit.client.0.smithi032.stdout:checking whether make sets $(MAKE)... yes 2024-04-30T08:36:54.469 INFO:tasks.workunit.client.0.smithi032.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-04-30T08:36:54.469 INFO:tasks.workunit.client.0.smithi032.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-04-30T08:36:54.492 INFO:tasks.workunit.client.0.smithi032.stdout:checking for style of include used by make... GNU 2024-04-30T08:36:54.500 INFO:tasks.workunit.client.0.smithi032.stdout:checking for gcc... gcc 2024-04-30T08:36:54.634 INFO:tasks.workunit.client.0.smithi032.stdout:checking for C compiler default output file name... a.out 2024-04-30T08:36:54.642 INFO:tasks.workunit.client.0.smithi032.stdout:checking whether the C compiler works... yes 2024-04-30T08:36:54.647 INFO:tasks.workunit.client.0.smithi032.stdout:checking whether we are cross compiling... no 2024-04-30T08:36:54.734 INFO:tasks.workunit.client.0.smithi032.stdout:checking for suffix of executables... 2024-04-30T08:36:54.816 INFO:tasks.workunit.client.0.smithi032.stdout:checking for suffix of object files... o 2024-04-30T08:36:54.901 INFO:tasks.workunit.client.0.smithi032.stdout:checking whether we are using the GNU C compiler... yes 2024-04-30T08:36:54.985 INFO:tasks.workunit.client.0.smithi032.stdout:checking whether gcc accepts -g... yes 2024-04-30T08:36:55.122 INFO:tasks.workunit.client.0.smithi032.stdout:checking for gcc option to accept ISO C89... none needed 2024-04-30T08:36:55.244 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:54 smithi032 ceph-mon[28331]: pgmap v477: 129 pgs: 129 active+clean; 676 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 3.7 KiB/s wr, 12 op/s 2024-04-30T08:36:55.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:54 smithi071 ceph-mon[33385]: pgmap v477: 129 pgs: 129 active+clean; 676 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 3.7 KiB/s wr, 12 op/s 2024-04-30T08:36:55.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:54 smithi189 ceph-mon[33038]: pgmap v477: 129 pgs: 129 active+clean; 676 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 3.7 KiB/s wr, 12 op/s 2024-04-30T08:36:55.325 INFO:tasks.workunit.client.0.smithi032.stdout:checking dependency style of gcc... gcc3 2024-04-30T08:36:55.478 INFO:tasks.workunit.client.0.smithi032.stdout:checking how to run the C preprocessor... gcc -E 2024-04-30T08:36:55.610 INFO:tasks.workunit.client.0.smithi032.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-04-30T08:36:55.614 INFO:tasks.workunit.client.0.smithi032.stdout:checking for egrep... /usr/bin/grep -E 2024-04-30T08:36:55.646 INFO:tasks.workunit.client.0.smithi032.stdout:checking for AIX... no 2024-04-30T08:36:55.674 INFO:tasks.workunit.client.0.smithi032.stdout:checking for gcc... (cached) gcc 2024-04-30T08:36:55.710 INFO:tasks.workunit.client.0.smithi032.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-04-30T08:36:55.712 INFO:tasks.workunit.client.0.smithi032.stdout:checking whether gcc accepts -g... (cached) yes 2024-04-30T08:36:55.712 INFO:tasks.workunit.client.0.smithi032.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-04-30T08:36:55.713 INFO:tasks.workunit.client.0.smithi032.stdout:checking dependency style of gcc... (cached) gcc3 2024-04-30T08:36:55.840 INFO:tasks.workunit.client.0.smithi032.stdout:checking for main in -lm... yes 2024-04-30T08:36:55.968 INFO:tasks.workunit.client.0.smithi032.stdout:checking for main in -lpthread... yes 2024-04-30T08:36:56.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:55 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:36:56.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:55 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:36:56.365 INFO:tasks.workunit.client.0.smithi032.stdout:checking for ANSI C header files... yes 2024-04-30T08:36:56.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:55 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:36:56.481 INFO:tasks.workunit.client.0.smithi032.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-04-30T08:36:56.595 INFO:tasks.workunit.client.0.smithi032.stdout:checking for sys/types.h... yes 2024-04-30T08:36:56.721 INFO:tasks.workunit.client.0.smithi032.stdout:checking for sys/stat.h... yes 2024-04-30T08:36:56.848 INFO:tasks.workunit.client.0.smithi032.stdout:checking for stdlib.h... yes 2024-04-30T08:36:56.973 INFO:tasks.workunit.client.0.smithi032.stdout:checking for string.h... yes 2024-04-30T08:36:57.180 INFO:tasks.workunit.client.0.smithi032.stdout:checking for memory.h... yes 2024-04-30T08:36:57.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:56 smithi071 ceph-mon[33385]: pgmap v478: 129 pgs: 129 active+clean; 2.0 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 128 KiB/s wr, 27 op/s 2024-04-30T08:36:57.282 INFO:tasks.workunit.client.0.smithi032.stdout:checking for strings.h... yes 2024-04-30T08:36:57.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:56 smithi189 ceph-mon[33038]: pgmap v478: 129 pgs: 129 active+clean; 2.0 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 128 KiB/s wr, 27 op/s 2024-04-30T08:36:57.343 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:56 smithi032 ceph-mon[28331]: pgmap v478: 129 pgs: 129 active+clean; 2.0 MiB data, 370 MiB used, 1.0 TiB / 1.0 TiB avail; 123 KiB/s rd, 128 KiB/s wr, 27 op/s 2024-04-30T08:36:57.389 INFO:tasks.workunit.client.0.smithi032.stdout:checking for inttypes.h... yes 2024-04-30T08:36:57.514 INFO:tasks.workunit.client.0.smithi032.stdout:checking for stdint.h... yes 2024-04-30T08:36:57.616 INFO:tasks.workunit.client.0.smithi032.stdout:checking for unistd.h... yes 2024-04-30T08:36:57.742 INFO:tasks.workunit.client.0.smithi032.stdout:checking pthread.h usability... yes 2024-04-30T08:36:57.806 INFO:tasks.workunit.client.0.smithi032.stdout:checking pthread.h presence... yes 2024-04-30T08:36:57.809 INFO:tasks.workunit.client.0.smithi032.stdout:checking for pthread.h... yes 2024-04-30T08:36:57.940 INFO:tasks.workunit.client.0.smithi032.stdout:checking fcntl.h usability... yes 2024-04-30T08:36:58.003 INFO:tasks.workunit.client.0.smithi032.stdout:checking fcntl.h presence... yes 2024-04-30T08:36:58.005 INFO:tasks.workunit.client.0.smithi032.stdout:checking for fcntl.h... yes 2024-04-30T08:36:58.133 INFO:tasks.workunit.client.0.smithi032.stdout:checking limits.h usability... yes 2024-04-30T08:36:58.188 INFO:tasks.workunit.client.0.smithi032.stdout:checking limits.h presence... yes 2024-04-30T08:36:58.190 INFO:tasks.workunit.client.0.smithi032.stdout:checking for limits.h... yes 2024-04-30T08:36:58.225 INFO:tasks.workunit.client.0.smithi032.stdout:checking for stdint.h... (cached) yes 2024-04-30T08:36:58.356 INFO:tasks.workunit.client.0.smithi032.stdout:checking sys/time.h usability... yes 2024-04-30T08:36:58.415 INFO:tasks.workunit.client.0.smithi032.stdout:checking sys/time.h presence... yes 2024-04-30T08:36:58.416 INFO:tasks.workunit.client.0.smithi032.stdout:checking for sys/time.h... yes 2024-04-30T08:36:58.419 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-30T08:36:58.419 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-30T08:36:58.452 INFO:tasks.workunit.client.0.smithi032.stdout:checking for unistd.h... (cached) yes 2024-04-30T08:36:58.552 INFO:tasks.workunit.client.0.smithi032.stdout:checking sys/vfs.h usability... yes 2024-04-30T08:36:58.602 INFO:tasks.workunit.client.0.smithi032.stdout:checking sys/vfs.h presence... yes 2024-04-30T08:36:58.604 INFO:tasks.workunit.client.0.smithi032.stdout:checking for sys/vfs.h... yes 2024-04-30T08:36:58.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.697+0000 7fdeeda00640 1 -- 172.21.15.32:0/1221870889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdee8102bb0 msgr2=0x7fdee8102fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:58.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.697+0000 7fdeeda00640 1 --2- 172.21.15.32:0/1221870889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdee8102bb0 0x7fdee8102fc0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fded4009860 tx=0x7fded402a340 comp rx=0 tx=0).stop 2024-04-30T08:36:58.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.697+0000 7fdeeda00640 1 -- 172.21.15.32:0/1221870889 shutdown_connections 2024-04-30T08:36:58.698 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.697+0000 7fdeeda00640 1 --2- 172.21.15.32:0/1221870889 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdee8103500 0x7fdee8103910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:58.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.697+0000 7fdeeda00640 1 --2- 172.21.15.32:0/1221870889 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdee8102bb0 0x7fdee8102fc0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:58.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.697+0000 7fdeeda00640 1 --2- 172.21.15.32:0/1221870889 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdee8102260 0x7fdee8102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:58.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.697+0000 7fdeeda00640 1 -- 172.21.15.32:0/1221870889 >> 172.21.15.32:0/1221870889 conn(0x7fdee80fda70 msgr2=0x7fdee80fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:58.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.697+0000 7fdeeda00640 1 -- 172.21.15.32:0/1221870889 shutdown_connections 2024-04-30T08:36:58.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.697+0000 7fdeeda00640 1 -- 172.21.15.32:0/1221870889 wait complete. 2024-04-30T08:36:58.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.698+0000 7fdeeda00640 1 Processor -- start 2024-04-30T08:36:58.701 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.699+0000 7fdeeda00640 1 -- start start 2024-04-30T08:36:58.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.699+0000 7fdeeda00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdee8102260 0x7fdee819c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:58.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.699+0000 7fdeeda00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdee8102bb0 0x7fdee819c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:58.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.699+0000 7fdeeda00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdee8103500 0x7fdee81a3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:58.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.699+0000 7fdeeda00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdee81116c0 con 0x7fdee8102bb0 2024-04-30T08:36:58.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.699+0000 7fdeeda00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fdee8111540 con 0x7fdee8103500 2024-04-30T08:36:58.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.699+0000 7fdeeda00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fdee8111840 con 0x7fdee8102260 2024-04-30T08:36:58.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.699+0000 7fdee7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdee8102260 0x7fdee819c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:58.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.699+0000 7fdee7400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdee8102bb0 0x7fdee819c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:58.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.699+0000 7fdee7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdee8102260 0x7fdee819c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:36716/0 (socket says 172.21.15.32:36716) 2024-04-30T08:36:58.702 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.699+0000 7fdee7e00640 1 -- 172.21.15.32:0/2116452668 learned_addr learned my addr 172.21.15.32:0/2116452668 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:58.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.699+0000 7fdeed000640 1 --2- 172.21.15.32:0/2116452668 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdee8103500 0x7fdee81a3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:58.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.700+0000 7fdee7e00640 1 -- 172.21.15.32:0/2116452668 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdee8103500 msgr2=0x7fdee81a3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:58.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.700+0000 7fdee7e00640 1 --2- 172.21.15.32:0/2116452668 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdee8103500 0x7fdee81a3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:58.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.700+0000 7fdee7e00640 1 -- 172.21.15.32:0/2116452668 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdee8102bb0 msgr2=0x7fdee819c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:58.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.700+0000 7fdee7e00640 1 --2- 172.21.15.32:0/2116452668 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdee8102bb0 0x7fdee819c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:58.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.700+0000 7fdee7e00640 1 -- 172.21.15.32:0/2116452668 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdee81a3e70 con 0x7fdee8102260 2024-04-30T08:36:58.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.700+0000 7fdee7e00640 1 --2- 172.21.15.32:0/2116452668 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdee8102260 0x7fdee819c330 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fded800e9a0 tx=0x7fded800ee60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:58.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.701+0000 7fdee4c00640 1 -- 172.21.15.32:0/2116452668 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded800cd80 con 0x7fdee8102260 2024-04-30T08:36:58.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.701+0000 7fdee4c00640 1 -- 172.21.15.32:0/2116452668 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fded80035f0 con 0x7fdee8102260 2024-04-30T08:36:58.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.701+0000 7fdee4c00640 1 -- 172.21.15.32:0/2116452668 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded8010690 con 0x7fdee8102260 2024-04-30T08:36:58.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.701+0000 7fdeeda00640 1 -- 172.21.15.32:0/2116452668 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdee81a4160 con 0x7fdee8102260 2024-04-30T08:36:58.703 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.701+0000 7fdeeda00640 1 -- 172.21.15.32:0/2116452668 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdee81a4660 con 0x7fdee8102260 2024-04-30T08:36:58.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.704+0000 7fdef604d740 1 Processor -- start 2024-04-30T08:36:58.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.704+0000 7fdef604d740 1 -- start start 2024-04-30T08:36:58.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.704+0000 7fdef604d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba76cd390 0x55fba7a6e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:58.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.704+0000 7fdef604d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba76a44d0 0x55fba7a6e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:58.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.704+0000 7fdef604d740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fba76ac6b0 0x55fba7a758c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:58.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.704+0000 7fdef604d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55fba79d3d50 con 0x55fba76a44d0 2024-04-30T08:36:58.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.704+0000 7fdef604d740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55fba7978ef0 con 0x55fba76ac6b0 2024-04-30T08:36:58.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.704+0000 7fdef604d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55fba79cfd10 con 0x55fba76cd390 2024-04-30T08:36:58.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.705+0000 7fdeed000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba76cd390 0x55fba7a6e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:58.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.705+0000 7fdeed000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba76cd390 0x55fba7a6e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:36726/0 (socket says 172.21.15.32:36726) 2024-04-30T08:36:58.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.705+0000 7fdeed000640 1 -- 172.21.15.32:0/1135377908 learned_addr learned my addr 172.21.15.32:0/1135377908 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:58.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.705+0000 7fdee7400640 1 --2- 172.21.15.32:0/1135377908 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fba76ac6b0 0x55fba7a758c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:58.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.705+0000 7fdee4c00640 1 -- 172.21.15.32:0/2116452668 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7fded8004000 con 0x7fdee8102260 2024-04-30T08:36:58.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.705+0000 7fdeed000640 1 -- 172.21.15.32:0/1135377908 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fba76ac6b0 msgr2=0x55fba7a758c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:58.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.705+0000 7fdeed000640 1 --2- 172.21.15.32:0/1135377908 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fba76ac6b0 0x55fba7a758c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:58.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.705+0000 7fdeed000640 1 -- 172.21.15.32:0/1135377908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba76a44d0 msgr2=0x55fba7a6e9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-30T08:36:58.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.706+0000 7fdeed000640 1 --2- 172.21.15.32:0/1135377908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba76a44d0 0x55fba7a6e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:58.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.706+0000 7fdeed000640 1 -- 172.21.15.32:0/1135377908 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fba77af7d0 con 0x55fba76cd390 2024-04-30T08:36:58.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.706+0000 7fdeed000640 1 --2- 172.21.15.32:0/1135377908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba76cd390 0x55fba7a6e440 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fdee0009f90 tx=0x7fdee00118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8741d0e7c724035 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:58.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.706+0000 7fdedd600640 1 -- 172.21.15.32:0/1135377908 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee0004d20 con 0x55fba76cd390 2024-04-30T08:36:58.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.706+0000 7fdedd600640 1 -- 172.21.15.32:0/1135377908 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdee0004050 con 0x55fba76cd390 2024-04-30T08:36:58.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.706+0000 7fdedd600640 1 -- 172.21.15.32:0/1135377908 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee0005220 con 0x55fba76cd390 2024-04-30T08:36:58.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.707+0000 7fdee4c00640 1 --2- 172.21.15.32:0/2116452668 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fdecc0afae0 0x7fdecc0b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:58.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.707+0000 7fdef604d740 1 -- 172.21.15.32:0/1135377908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba76cd390 msgr2=0x55fba7a6e440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:58.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.707+0000 7fdef604d740 1 --2- 172.21.15.32:0/1135377908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba76cd390 0x55fba7a6e440 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fdee0009f90 tx=0x7fdee00118d0 comp rx=0 tx=0).stop 2024-04-30T08:36:58.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.707+0000 7fdee7e00640 1 --2- 172.21.15.32:0/2116452668 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fdecc0afae0 0x7fdecc0b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:58.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.707+0000 7fdee4c00640 1 -- 172.21.15.32:0/2116452668 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fded80df290 con 0x7fdee8102260 2024-04-30T08:36:58.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.707+0000 7fdef604d740 1 -- 172.21.15.32:0/1135377908 shutdown_connections 2024-04-30T08:36:58.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.707+0000 7fdef604d740 1 --2- 172.21.15.32:0/1135377908 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fba76ac6b0 0x55fba7a758c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:58.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.707+0000 7fdef604d740 1 --2- 172.21.15.32:0/1135377908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba76a44d0 0x55fba7a6e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:58.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.707+0000 7fdef604d740 1 --2- 172.21.15.32:0/1135377908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba76cd390 0x55fba7a6e440 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:58.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.707+0000 7fdef604d740 1 -- 172.21.15.32:0/1135377908 >> 172.21.15.32:0/1135377908 conn(0x55fba76c14e0 msgr2=0x55fba7a69dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:58.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.708+0000 7fdef604d740 1 -- 172.21.15.32:0/1135377908 shutdown_connections 2024-04-30T08:36:58.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.708+0000 7fdef604d740 1 -- 172.21.15.32:0/1135377908 wait complete. 2024-04-30T08:36:58.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.708+0000 7fdee7e00640 1 --2- 172.21.15.32:0/2116452668 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fdecc0afae0 0x7fdecc0b1ff0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fded800efd0 tx=0x7fded8007850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:58.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.708+0000 7fdef604d740 1 -- start start 2024-04-30T08:36:58.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.708+0000 7fdef604d740 1 Processor -- start 2024-04-30T08:36:58.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.709+0000 7fdef604d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba76a44d0 0x55fba7a782e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:58.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.709+0000 7fdef604d740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fba76ac6b0 0x55fba7a78820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:58.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.709+0000 7fdef604d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba76cd390 0x55fba7a7f780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:58.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.709+0000 7fdef604d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55fba7a76310 con 0x55fba76cd390 2024-04-30T08:36:58.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.709+0000 7fdef604d740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55fba798c990 con 0x55fba76ac6b0 2024-04-30T08:36:58.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.709+0000 7fdef604d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55fba7a76490 con 0x55fba76a44d0 2024-04-30T08:36:58.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.709+0000 7fdef604d740 10 client.0 _ll_register_callbacks cb 0x55fba798b110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:36:58.712 INFO:tasks.workunit.client.0.smithi032.stdout:checking sys/limits.h usability... no 2024-04-30T08:36:58.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.709+0000 7fdee7400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fba76ac6b0 0x55fba7a78820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:58.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.709+0000 7fdee7400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fba76ac6b0 0x55fba7a78820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:35492/0 (socket says 172.21.15.32:35492) 2024-04-30T08:36:58.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.709+0000 7fdee7400640 1 -- 172.21.15.32:0/1859004249 learned_addr learned my addr 172.21.15.32:0/1859004249 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:58.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.709+0000 7fdeed000640 1 --2- 172.21.15.32:0/1859004249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba76a44d0 0x55fba7a782e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:58.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.709+0000 7fdeed000640 1 --2- 172.21.15.32:0/1859004249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba76cd390 0x55fba7a7f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:58.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.710+0000 7fdee7400640 1 -- 172.21.15.32:0/1859004249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba76a44d0 msgr2=0x55fba7a782e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:58.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.710+0000 7fdee7400640 1 --2- 172.21.15.32:0/1859004249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba76a44d0 0x55fba7a782e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:58.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.710+0000 7fdee7400640 1 -- 172.21.15.32:0/1859004249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba76cd390 msgr2=0x55fba7a7f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:58.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.710+0000 7fdee7400640 1 --2- 172.21.15.32:0/1859004249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba76cd390 0x55fba7a7f780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:58.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.710+0000 7fdee7400640 1 -- 172.21.15.32:0/1859004249 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fba76e53f0 con 0x55fba76ac6b0 2024-04-30T08:36:58.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.710+0000 7fdeed000640 1 --2- 172.21.15.32:0/1859004249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba76cd390 0x55fba7a7f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:36:58.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.710+0000 7fdee7400640 1 --2- 172.21.15.32:0/1859004249 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fba76ac6b0 0x55fba7a78820 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fded402a930 tx=0x7fded4005e10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e460c080067ce165 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:58.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.710+0000 7fded2000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:36:58.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.710+0000 7fded2000640 1 -- 172.21.15.32:0/1859004249 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded4037040 con 0x55fba76ac6b0 2024-04-30T08:36:58.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.710+0000 7fded2000640 1 -- 172.21.15.32:0/1859004249 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fded4004240 con 0x55fba76ac6b0 2024-04-30T08:36:58.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.710+0000 7fdef604d740 1 -- 172.21.15.32:0/1859004249 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdee0004050 con 0x55fba76ac6b0 2024-04-30T08:36:58.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.711+0000 7fded2000640 1 -- 172.21.15.32:0/1859004249 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fded404d050 con 0x55fba76ac6b0 2024-04-30T08:36:58.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.711+0000 7fdef604d740 10 client.26072 fetch_fsmap learned FSMap version 29 2024-04-30T08:36:58.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.711+0000 7fdef604d740 1 -- 172.21.15.32:0/1859004249 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fba76a50f0 con 0x55fba76ac6b0 2024-04-30T08:36:58.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.711+0000 7fded2000640 1 -- 172.21.15.32:0/1859004249 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdee0004050 con 0x55fba76ac6b0 2024-04-30T08:36:58.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.711+0000 7fded2000640 1 -- 172.21.15.32:0/1859004249 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7fded403d4e0 con 0x55fba76ac6b0 2024-04-30T08:36:58.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.711+0000 7fdef604d740 10 client.26072 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:36:58.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.711+0000 7fdef604d740 10 client.26072 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:36:58.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.711+0000 7fdef604d740 20 client.26072 populate_metadata read hostname 'smithi032' 2024-04-30T08:36:58.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.711+0000 7fdef604d740 1 --2- 172.21.15.32:0/1859004249 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55fba768bf60 0x55fba7a85ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:58.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.711+0000 7fdef604d740 4 client.26072 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:36:58.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.711+0000 7fdef604d740 1 -- 172.21.15.32:0/1859004249 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fba79e7ef0 con 0x55fba768bf60 2024-04-30T08:36:58.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.712+0000 7fdeed000640 1 --2- 172.21.15.32:0/1859004249 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55fba768bf60 0x55fba7a85ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:58.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.712+0000 7fdeed000640 1 --2- 172.21.15.32:0/1859004249 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55fba768bf60 0x55fba7a85ce0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba9d5cff78166cf7 server_cookie=9d34c79f736abf67 in_seq=0 out_seq=0 2024-04-30T08:36:58.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.712+0000 7fded2000640 10 client.26072 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:58.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.713+0000 7fded2000640 1 -- 172.21.15.32:0/1859004249 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55fba79e7ef0 con 0x55fba768bf60 2024-04-30T08:36:58.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.713+0000 7fded2000640 10 client.26072 handle_command_reply: tid=0 2024-04-30T08:36:58.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.713+0000 7fdef604d740 1 client.26072 shutdown 2024-04-30T08:36:58.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.713+0000 7fdef604d740 1 -- 172.21.15.32:0/1859004249 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fba76ac6b0 msgr2=0x55fba7a78820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:58.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.713+0000 7fdef604d740 1 --2- 172.21.15.32:0/1859004249 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fba76ac6b0 0x55fba7a78820 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fded402a930 tx=0x7fded4005e10 comp rx=0 tx=0).stop 2024-04-30T08:36:58.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.714+0000 7fdef604d740 1 -- 172.21.15.32:0/1859004249 shutdown_connections 2024-04-30T08:36:58.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.714+0000 7fdef604d740 1 --2- 172.21.15.32:0/1859004249 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55fba768bf60 0x55fba7a85ce0 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:58.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.714+0000 7fdef604d740 1 --2- 172.21.15.32:0/1859004249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba76cd390 0x55fba7a7f780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:58.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.714+0000 7fdef604d740 1 --2- 172.21.15.32:0/1859004249 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fba76ac6b0 0x55fba7a78820 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:58.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.714+0000 7fdef604d740 1 --2- 172.21.15.32:0/1859004249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba76a44d0 0x55fba7a782e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:58.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.714+0000 7fded2000640 0 client.26072 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:58.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.714+0000 7fdef604d740 1 -- 172.21.15.32:0/1859004249 >> 172.21.15.32:0/1859004249 conn(0x55fba76c14e0 msgr2=0x55fba7a68e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:58.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.714+0000 7fdef604d740 1 -- 172.21.15.32:0/1859004249 shutdown_connections 2024-04-30T08:36:58.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.714+0000 7fdef604d740 1 -- 172.21.15.32:0/1859004249 wait complete. 2024-04-30T08:36:58.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.714+0000 7fdef604d740 20 client.26072 trim_cache size 0 max 16384 2024-04-30T08:36:58.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.735+0000 7fdef604d740 1 Processor -- start 2024-04-30T08:36:58.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.736+0000 7fdef604d740 1 -- start start 2024-04-30T08:36:58.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.736+0000 7fdef604d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba76a44d0 0x55fba7a69750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:58.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.736+0000 7fdef604d740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fba76ac6b0 0x55fba7a7f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:58.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.736+0000 7fdef604d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba76cd390 0x55fba7a69c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:58.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.736+0000 7fdef604d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdee81aec40 con 0x55fba76cd390 2024-04-30T08:36:58.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.736+0000 7fdef604d740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fdee81b0810 con 0x55fba76ac6b0 2024-04-30T08:36:58.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.736+0000 7fdef604d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fdee81af310 con 0x55fba76a44d0 2024-04-30T08:36:58.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.736+0000 7fdee7400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fba76ac6b0 0x55fba7a7f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:58.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.736+0000 7fdeed000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba76cd390 0x55fba7a69c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:58.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.736+0000 7fdee7400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fba76ac6b0 0x55fba7a7f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:35508/0 (socket says 172.21.15.32:35508) 2024-04-30T08:36:58.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.736+0000 7fdee7400640 1 -- 172.21.15.32:0/35154603 learned_addr learned my addr 172.21.15.32:0/35154603 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:58.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.736+0000 7fdeed000640 1 --2- 172.21.15.32:0/35154603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba76a44d0 0x55fba7a69750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:58.738 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.737+0000 7fdeed000640 1 -- 172.21.15.32:0/35154603 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fba76ac6b0 msgr2=0x55fba7a7f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:58.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.737+0000 7fdeed000640 1 --2- 172.21.15.32:0/35154603 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fba76ac6b0 0x55fba7a7f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:58.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.737+0000 7fdeed000640 1 -- 172.21.15.32:0/35154603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba76cd390 msgr2=0x55fba7a69c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:58.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.737+0000 7fdeed000640 1 --2- 172.21.15.32:0/35154603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba76cd390 0x55fba7a69c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:58.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.737+0000 7fdeed000640 1 -- 172.21.15.32:0/35154603 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fba7a73160 con 0x55fba76a44d0 2024-04-30T08:36:58.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.737+0000 7fdee7400640 1 --2- 172.21.15.32:0/35154603 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fba76ac6b0 0x55fba7a7f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:36:58.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.737+0000 7fdeed000640 1 --2- 172.21.15.32:0/35154603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba76a44d0 0x55fba7a69750 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fdee001b040 tx=0x7fdee0009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7159c28cb281dea0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:58.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.738+0000 7fdedcc00640 1 -- 172.21.15.32:0/35154603 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee0004cb0 con 0x55fba76a44d0 2024-04-30T08:36:58.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.738+0000 7fdedcc00640 1 -- 172.21.15.32:0/35154603 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdee0004e50 con 0x55fba76a44d0 2024-04-30T08:36:58.739 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.738+0000 7fdedcc00640 1 -- 172.21.15.32:0/35154603 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdee002e9f0 con 0x55fba76a44d0 2024-04-30T08:36:58.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.739+0000 7fdef604d740 1 -- 172.21.15.32:0/35154603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba76a44d0 msgr2=0x55fba7a69750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:58.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.739+0000 7fdef604d740 1 --2- 172.21.15.32:0/35154603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba76a44d0 0x55fba7a69750 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fdee001b040 tx=0x7fdee0009500 comp rx=0 tx=0).stop 2024-04-30T08:36:58.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.739+0000 7fdef604d740 1 -- 172.21.15.32:0/35154603 shutdown_connections 2024-04-30T08:36:58.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.739+0000 7fdef604d740 1 --2- 172.21.15.32:0/35154603 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba76cd390 0x55fba7a69c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:58.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.739+0000 7fdef604d740 1 --2- 172.21.15.32:0/35154603 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fba76ac6b0 0x55fba7a7f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:58.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.739+0000 7fdef604d740 1 --2- 172.21.15.32:0/35154603 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba76a44d0 0x55fba7a69750 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:58.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.739+0000 7fdef604d740 1 -- 172.21.15.32:0/35154603 >> 172.21.15.32:0/35154603 conn(0x55fba76c14e0 msgr2=0x55fba7a76920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:58.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.739+0000 7fdef604d740 1 -- 172.21.15.32:0/35154603 shutdown_connections 2024-04-30T08:36:58.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.739+0000 7fdef604d740 1 -- 172.21.15.32:0/35154603 wait complete. 2024-04-30T08:36:58.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.739+0000 7fdef604d740 1 -- start start 2024-04-30T08:36:58.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.740+0000 7fdef604d740 1 Processor -- start 2024-04-30T08:36:58.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.740+0000 7fdef604d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba76a44d0 0x55fba7a7cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:58.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.740+0000 7fdef604d740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fba76ac6b0 0x55fba7a7d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:58.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.740+0000 7fdef604d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba76cd390 0x55fba7a7e330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:58.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.740+0000 7fdef604d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdee81b00b0 con 0x55fba76a44d0 2024-04-30T08:36:58.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.740+0000 7fdef604d740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fdee81af9e0 con 0x55fba76ac6b0 2024-04-30T08:36:58.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.740+0000 7fdef604d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fdee81acc00 con 0x55fba76cd390 2024-04-30T08:36:58.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.740+0000 7fdef604d740 10 client.0 _ll_register_callbacks cb 0x7fdee00093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:36:58.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.741+0000 7fdee7400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fba76ac6b0 0x55fba7a7d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:58.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.741+0000 7fdeed000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba76a44d0 0x55fba7a7cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:58.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.741+0000 7fdee7400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fba76ac6b0 0x55fba7a7d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:35520/0 (socket says 172.21.15.32:35520) 2024-04-30T08:36:58.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.741+0000 7fdee7400640 1 -- 172.21.15.32:0/651942319 learned_addr learned my addr 172.21.15.32:0/651942319 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:58.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.741+0000 7fdee7400640 1 -- 172.21.15.32:0/651942319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba76cd390 msgr2=0x55fba7a7e330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:58.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.741+0000 7fdee7400640 1 --2- 172.21.15.32:0/651942319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba76cd390 0x55fba7a7e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:58.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.741+0000 7fdee7400640 1 -- 172.21.15.32:0/651942319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba76a44d0 msgr2=0x55fba7a7cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:58.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.741+0000 7fdee7400640 1 --2- 172.21.15.32:0/651942319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba76a44d0 0x55fba7a7cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:58.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.741+0000 7fdee7400640 1 -- 172.21.15.32:0/651942319 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fba7a7e8f0 con 0x55fba76ac6b0 2024-04-30T08:36:58.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.741+0000 7fdee7400640 1 --2- 172.21.15.32:0/651942319 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fba76ac6b0 0x55fba7a7d450 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fded403d950 tx=0x7fded403dd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb864ac46d17fe70 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:58.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.741+0000 7fded2000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:36:58.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.741+0000 7fded2000640 1 -- 172.21.15.32:0/651942319 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fded4037040 con 0x55fba76ac6b0 2024-04-30T08:36:58.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.742+0000 7fdef604d740 1 -- 172.21.15.32:0/651942319 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdee0004e50 con 0x55fba76ac6b0 2024-04-30T08:36:58.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.742+0000 7fded2000640 1 -- 172.21.15.32:0/651942319 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fded4005040 con 0x55fba76ac6b0 2024-04-30T08:36:58.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.742+0000 7fded2000640 1 -- 172.21.15.32:0/651942319 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fded40308d0 con 0x55fba76ac6b0 2024-04-30T08:36:58.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.742+0000 7fdef604d740 10 client.26078 fetch_fsmap learned FSMap version 29 2024-04-30T08:36:58.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.742+0000 7fdef604d740 1 -- 172.21.15.32:0/651942319 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55fba7af68a0 con 0x55fba76ac6b0 2024-04-30T08:36:58.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.742+0000 7fded2000640 1 -- 172.21.15.32:0/651942319 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdee0004e50 con 0x55fba76ac6b0 2024-04-30T08:36:58.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.742+0000 7fded2000640 1 -- 172.21.15.32:0/651942319 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7fded4004b80 con 0x55fba76ac6b0 2024-04-30T08:36:58.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.742+0000 7fdef604d740 10 client.26078 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:36:58.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.742+0000 7fdef604d740 10 client.26078 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:36:58.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.742+0000 7fdef604d740 20 client.26078 populate_metadata read hostname 'smithi032' 2024-04-30T08:36:58.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.742+0000 7fdef604d740 1 --2- 172.21.15.32:0/651942319 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55fba76613a0 0x55fba7af9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:58.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.742+0000 7fdef604d740 4 client.26078 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-30T08:36:58.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.742+0000 7fdef604d740 1 -- 172.21.15.32:0/651942319 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55fba79e1960 con 0x55fba76613a0 2024-04-30T08:36:58.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.743+0000 7fdeed000640 1 --2- 172.21.15.32:0/651942319 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55fba76613a0 0x55fba7af9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:58.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.743+0000 7fded2000640 10 client.26078 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:58.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.743+0000 7fdeed000640 1 --2- 172.21.15.32:0/651942319 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55fba76613a0 0x55fba7af9270 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9f97a340ba94eb2 server_cookie=842ce8a0b007b304 in_seq=0 out_seq=0 2024-04-30T08:36:58.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.743+0000 7fded2000640 1 -- 172.21.15.32:0/651942319 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55fba79e1960 con 0x55fba76613a0 2024-04-30T08:36:58.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.743+0000 7fded2000640 10 client.26078 handle_command_reply: tid=0 2024-04-30T08:36:58.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.743+0000 7fdef604d740 1 client.26078 shutdown 2024-04-30T08:36:58.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.744+0000 7fdef604d740 1 -- 172.21.15.32:0/651942319 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fba76ac6b0 msgr2=0x55fba7a7d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:58.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.744+0000 7fdef604d740 1 --2- 172.21.15.32:0/651942319 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fba76ac6b0 0x55fba7a7d450 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fded403d950 tx=0x7fded403dd70 comp rx=0 tx=0).stop 2024-04-30T08:36:58.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.744+0000 7fdef604d740 1 -- 172.21.15.32:0/651942319 shutdown_connections 2024-04-30T08:36:58.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.744+0000 7fdef604d740 1 --2- 172.21.15.32:0/651942319 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55fba76613a0 0x55fba7af9270 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:58.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.744+0000 7fdef604d740 1 --2- 172.21.15.32:0/651942319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fba76cd390 0x55fba7a7e330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:58.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.744+0000 7fdef604d740 1 --2- 172.21.15.32:0/651942319 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55fba76ac6b0 0x55fba7a7d450 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:58.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.744+0000 7fdef604d740 1 --2- 172.21.15.32:0/651942319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55fba76a44d0 0x55fba7a7cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:58.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.744+0000 7fded2000640 0 client.26078 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:36:58.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.744+0000 7fdef604d740 1 -- 172.21.15.32:0/651942319 >> 172.21.15.32:0/651942319 conn(0x55fba76c14e0 msgr2=0x55fba7a69750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:58.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.744+0000 7fdef604d740 1 -- 172.21.15.32:0/651942319 shutdown_connections 2024-04-30T08:36:58.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.744+0000 7fdef604d740 1 -- 172.21.15.32:0/651942319 wait complete. 2024-04-30T08:36:58.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.744+0000 7fdef604d740 20 client.26078 trim_cache size 0 max 16384 2024-04-30T08:36:58.748 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-30T08:36:58.749 INFO:teuthology.orchestra.run.smithi032.stdout: "status": "no active scrubs running", 2024-04-30T08:36:58.749 INFO:teuthology.orchestra.run.smithi032.stdout: "scrubs": {} 2024-04-30T08:36:58.749 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-30T08:36:58.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.745+0000 7fdeeda00640 1 -- 172.21.15.32:0/2116452668 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fdecc0afae0 msgr2=0x7fdecc0b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:58.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.745+0000 7fdeeda00640 1 --2- 172.21.15.32:0/2116452668 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fdecc0afae0 0x7fdecc0b1ff0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fded800efd0 tx=0x7fded8007850 comp rx=0 tx=0).stop 2024-04-30T08:36:58.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.745+0000 7fdeeda00640 1 -- 172.21.15.32:0/2116452668 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdee8102260 msgr2=0x7fdee819c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:58.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.745+0000 7fdeeda00640 1 --2- 172.21.15.32:0/2116452668 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdee8102260 0x7fdee819c330 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fded800e9a0 tx=0x7fded800ee60 comp rx=0 tx=0).stop 2024-04-30T08:36:58.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.745+0000 7fdeeda00640 1 -- 172.21.15.32:0/2116452668 shutdown_connections 2024-04-30T08:36:58.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.745+0000 7fdeeda00640 1 --2- 172.21.15.32:0/2116452668 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fdecc0afae0 0x7fdecc0b1ff0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:58.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.745+0000 7fdeeda00640 1 --2- 172.21.15.32:0/2116452668 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdee8103500 0x7fdee81a3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:58.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.745+0000 7fdeeda00640 1 --2- 172.21.15.32:0/2116452668 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdee8102bb0 0x7fdee819c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:58.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.745+0000 7fdeeda00640 1 --2- 172.21.15.32:0/2116452668 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdee8102260 0x7fdee819c330 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:58.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.745+0000 7fdeeda00640 1 -- 172.21.15.32:0/2116452668 >> 172.21.15.32:0/2116452668 conn(0x7fdee80fda70 msgr2=0x7fdee810a470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:58.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.745+0000 7fdeeda00640 1 -- 172.21.15.32:0/2116452668 shutdown_connections 2024-04-30T08:36:58.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:58.745+0000 7fdeeda00640 1 -- 172.21.15.32:0/2116452668 wait complete. 2024-04-30T08:36:58.755 INFO:tasks.workunit.client.0.smithi032.stdout:checking sys/limits.h presence... no 2024-04-30T08:36:58.756 INFO:tasks.workunit.client.0.smithi032.stdout:checking for sys/limits.h... no 2024-04-30T08:36:58.771 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-30T08:36:58.772 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-30T08:36:58.829 INFO:tasks.workunit.client.0.smithi032.stdout:checking for an ANSI C-conforming const... yes 2024-04-30T08:36:58.907 INFO:tasks.workunit.client.0.smithi032.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-04-30T08:36:58.910 INFO:tasks.workunit.client.0.smithi032.stdout:checking for function prototypes... yes 2024-04-30T08:36:58.978 INFO:tasks.workunit.client.0.smithi032.stdout:checking whether setvbuf arguments are reversed... no 2024-04-30T08:36:59.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.094+0000 7fea62a00640 1 -- 172.21.15.32:0/1282323717 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fea5c106fe0 msgr2=0x7fea5c1073f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:59.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.094+0000 7fea62a00640 1 --2- 172.21.15.32:0/1282323717 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fea5c106fe0 0x7fea5c1073f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fea48009900 tx=0x7fea4802a410 comp rx=0 tx=0).stop 2024-04-30T08:36:59.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.094+0000 7fea62a00640 1 -- 172.21.15.32:0/1282323717 shutdown_connections 2024-04-30T08:36:59.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.094+0000 7fea62a00640 1 --2- 172.21.15.32:0/1282323717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea5c10bce0 0x7fea5c10e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.094+0000 7fea62a00640 1 --2- 172.21.15.32:0/1282323717 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fea5c106fe0 0x7fea5c1073f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.094+0000 7fea62a00640 1 --2- 172.21.15.32:0/1282323717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea5c105da0 0x7fea5c106aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.095 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.094+0000 7fea62a00640 1 -- 172.21.15.32:0/1282323717 >> 172.21.15.32:0/1282323717 conn(0x7fea5c100f90 msgr2=0x7fea5c103440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:59.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.094+0000 7fea62a00640 1 -- 172.21.15.32:0/1282323717 shutdown_connections 2024-04-30T08:36:59.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.094+0000 7fea62a00640 1 -- 172.21.15.32:0/1282323717 wait complete. 2024-04-30T08:36:59.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.095+0000 7fea62a00640 1 Processor -- start 2024-04-30T08:36:59.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.096+0000 7fea62a00640 1 -- start start 2024-04-30T08:36:59.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.096+0000 7fea62a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea5c105da0 0x7fea5c19c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:59.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.096+0000 7fea62a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea5c106fe0 0x7fea5c19ca20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:59.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.096+0000 7fea62a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fea5c10bce0 0x7fea5c1a39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:59.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.096+0000 7fea62a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fea5c1118d0 con 0x7fea5c105da0 2024-04-30T08:36:59.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.096+0000 7fea62a00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fea5c111750 con 0x7fea5c10bce0 2024-04-30T08:36:59.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.096+0000 7fea62a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fea5c111a50 con 0x7fea5c106fe0 2024-04-30T08:36:59.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.096+0000 7fea5be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea5c105da0 0x7fea5c19c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:59.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.096+0000 7fea5b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea5c106fe0 0x7fea5c19ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:59.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.096+0000 7fea60a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fea5c10bce0 0x7fea5c1a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:59.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.096+0000 7fea5be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea5c105da0 0x7fea5c19c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56344/0 (socket says 172.21.15.32:56344) 2024-04-30T08:36:59.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.096+0000 7fea5be00640 1 -- 172.21.15.32:0/3683907221 learned_addr learned my addr 172.21.15.32:0/3683907221 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:59.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.097+0000 7fea60a00640 1 -- 172.21.15.32:0/3683907221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea5c106fe0 msgr2=0x7fea5c19ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:59.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.097+0000 7fea60a00640 1 --2- 172.21.15.32:0/3683907221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea5c106fe0 0x7fea5c19ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.097+0000 7fea60a00640 1 -- 172.21.15.32:0/3683907221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea5c105da0 msgr2=0x7fea5c19c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:59.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.097+0000 7fea60a00640 1 --2- 172.21.15.32:0/3683907221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea5c105da0 0x7fea5c19c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.097+0000 7fea60a00640 1 -- 172.21.15.32:0/3683907221 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea5c1a4020 con 0x7fea5c10bce0 2024-04-30T08:36:59.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.097+0000 7fea60a00640 1 --2- 172.21.15.32:0/3683907221 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fea5c10bce0 0x7fea5c1a39b0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fea54009fd0 tx=0x7fea5400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:59.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.097+0000 7fea58c00640 1 -- 172.21.15.32:0/3683907221 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea5400ce20 con 0x7fea5c10bce0 2024-04-30T08:36:59.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.097+0000 7fea58c00640 1 -- 172.21.15.32:0/3683907221 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fea54005020 con 0x7fea5c10bce0 2024-04-30T08:36:59.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.097+0000 7fea58c00640 1 -- 172.21.15.32:0/3683907221 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea54010690 con 0x7fea5c10bce0 2024-04-30T08:36:59.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.097+0000 7fea62a00640 1 -- 172.21.15.32:0/3683907221 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea5c1a4310 con 0x7fea5c10bce0 2024-04-30T08:36:59.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.098+0000 7fea62a00640 1 -- 172.21.15.32:0/3683907221 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea5c1a4810 con 0x7fea5c10bce0 2024-04-30T08:36:59.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.099+0000 7fea62a00640 1 -- 172.21.15.32:0/3683907221 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea5c10aaf0 con 0x7fea5c10bce0 2024-04-30T08:36:59.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.101+0000 7fea58c00640 1 -- 172.21.15.32:0/3683907221 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7fea54010830 con 0x7fea5c10bce0 2024-04-30T08:36:59.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.103+0000 7fea58c00640 1 --2- 172.21.15.32:0/3683907221 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fea400afaf0 0x7fea400b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:59.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.103+0000 7fea58c00640 1 -- 172.21.15.32:0/3683907221 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fea540df680 con 0x7fea5c10bce0 2024-04-30T08:36:59.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.103+0000 7fea5be00640 1 --2- 172.21.15.32:0/3683907221 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fea400afaf0 0x7fea400b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:59.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.104+0000 7fea5be00640 1 --2- 172.21.15.32:0/3683907221 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fea400afaf0 0x7fea400b2000 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fea4c009c30 tx=0x7fea4c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:59.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.107+0000 7fea58c00640 1 -- 172.21.15.32:0/3683907221 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fea540a7e50 con 0x7fea5c10bce0 2024-04-30T08:36:59.134 INFO:tasks.workunit.client.0.smithi032.stdout:checking for system... yes 2024-04-30T08:36:59.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:36:58 smithi032 ceph-mon[28331]: pgmap v479: 129 pgs: 129 active+clean; 2.0 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 133 KiB/s wr, 30 op/s 2024-04-30T08:36:59.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:36:58 smithi071 ceph-mon[33385]: pgmap v479: 129 pgs: 129 active+clean; 2.0 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 133 KiB/s wr, 30 op/s 2024-04-30T08:36:59.273 INFO:tasks.workunit.client.0.smithi032.stdout:checking for gettimeofday... yes 2024-04-30T08:36:59.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:36:58 smithi189 ceph-mon[33038]: pgmap v479: 129 pgs: 129 active+clean; 2.0 MiB data, 371 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 133 KiB/s wr, 30 op/s 2024-04-30T08:36:59.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.361+0000 7fea62a00640 1 -- 172.21.15.32:0/3683907221 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fea5c02dbd0 con 0x7fea5c10bce0 2024-04-30T08:36:59.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.362+0000 7fea58c00640 1 -- 172.21.15.32:0/3683907221 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11932 (secure 0 0 0) 0x7fea540a7750 con 0x7fea5c10bce0 2024-04-30T08:36:59.363 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:36:59.363 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2235842676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2235842676},{"type":"v1","addr":"172.21.15.32:6835","nonce":2235842676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/3577009355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":3577009355},{"type":"v1","addr":"172.21.15.32:6841","nonce":3577009355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-30T08:31:52.236668+0000","modified":"2024-04-30T08:36:54.799807+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24541,"mds_3":24533,"mds_4":24517},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/463203038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":463203038},{"type":"v1","addr":"172.21.15.189:6833","nonce":463203038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6833/3635143591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3635143591},{"type":"v1","addr":"172.21.15.71:6833","nonce":3635143591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6837/2569449994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2569449994},{"type":"v1","addr":"172.21.15.32:6837","nonce":2569449994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/278686565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":278686565},{"type":"v1","addr":"172.21.15.189:6835","nonce":278686565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6835/1558303056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1558303056},{"type":"v1","addr":"172.21.15.71:6835","nonce":1558303056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.189:6837/686520873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":686520873},{"type":"v1","addr":"172.21.15.189:6837","nonce":686520873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.71:6837/2324039932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2324039932},{"type":"v1","addr":"172.21.15.71:6837","nonce":2324039932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6839/352970107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":352970107},{"type":"v1","addr":"172.21.15.189:6839","nonce":352970107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.32:6839/4160386524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4160386524},{"type":"v1","addr":"172.21.15.32:6839","nonce":4160386524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.71:6839/3517409993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3517409993},{"type":"v1","addr":"172.21.15.71:6839","nonce":3517409993}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24541,24590,24560,24517,24533]},"id":1}]} 2024-04-30T08:36:59.363 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-04-30T08:36:59.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.365+0000 7fea62a00640 1 -- 172.21.15.32:0/3683907221 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fea400afaf0 msgr2=0x7fea400b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:59.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.365+0000 7fea62a00640 1 --2- 172.21.15.32:0/3683907221 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fea400afaf0 0x7fea400b2000 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fea4c009c30 tx=0x7fea4c005f90 comp rx=0 tx=0).stop 2024-04-30T08:36:59.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.365+0000 7fea62a00640 1 -- 172.21.15.32:0/3683907221 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fea5c10bce0 msgr2=0x7fea5c1a39b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:59.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.365+0000 7fea62a00640 1 --2- 172.21.15.32:0/3683907221 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fea5c10bce0 0x7fea5c1a39b0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fea54009fd0 tx=0x7fea5400ef10 comp rx=0 tx=0).stop 2024-04-30T08:36:59.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.365+0000 7fea62a00640 1 -- 172.21.15.32:0/3683907221 shutdown_connections 2024-04-30T08:36:59.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.365+0000 7fea62a00640 1 --2- 172.21.15.32:0/3683907221 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fea400afaf0 0x7fea400b2000 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.365+0000 7fea62a00640 1 --2- 172.21.15.32:0/3683907221 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fea5c10bce0 0x7fea5c1a39b0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.365+0000 7fea62a00640 1 --2- 172.21.15.32:0/3683907221 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fea5c106fe0 0x7fea5c19ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.365+0000 7fea62a00640 1 --2- 172.21.15.32:0/3683907221 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fea5c105da0 0x7fea5c19c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.365+0000 7fea62a00640 1 -- 172.21.15.32:0/3683907221 >> 172.21.15.32:0/3683907221 conn(0x7fea5c100f90 msgr2=0x7fea5c107cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:59.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.365+0000 7fea62a00640 1 -- 172.21.15.32:0/3683907221 shutdown_connections 2024-04-30T08:36:59.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.365+0000 7fea62a00640 1 -- 172.21.15.32:0/3683907221 wait complete. 2024-04-30T08:36:59.404 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-30T08:36:59.435 INFO:tasks.workunit.client.0.smithi032.stdout:checking for mkdir... yes 2024-04-30T08:36:59.565 INFO:tasks.workunit.client.0.smithi032.stdout:checking for strerror... yes 2024-04-30T08:36:59.690 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.689+0000 7fad33a00640 1 -- 172.21.15.32:0/1235032785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad2c106fc0 msgr2=0x7fad2c1073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:59.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.689+0000 7fad33a00640 1 --2- 172.21.15.32:0/1235032785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad2c106fc0 0x7fad2c1073d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fad18009900 tx=0x7fad1802a3f0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.690+0000 7fad33a00640 1 -- 172.21.15.32:0/1235032785 shutdown_connections 2024-04-30T08:36:59.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.690+0000 7fad33a00640 1 --2- 172.21.15.32:0/1235032785 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad2c10bcc0 0x7fad2c10e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.690+0000 7fad33a00640 1 --2- 172.21.15.32:0/1235032785 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad2c106fc0 0x7fad2c1073d0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.690+0000 7fad33a00640 1 --2- 172.21.15.32:0/1235032785 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad2c105d80 0x7fad2c106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.690+0000 7fad33a00640 1 -- 172.21.15.32:0/1235032785 >> 172.21.15.32:0/1235032785 conn(0x7fad2c100f70 msgr2=0x7fad2c103420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:59.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.690+0000 7fad33a00640 1 -- 172.21.15.32:0/1235032785 shutdown_connections 2024-04-30T08:36:59.691 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.690+0000 7fad33a00640 1 -- 172.21.15.32:0/1235032785 wait complete. 2024-04-30T08:36:59.692 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.691+0000 7fad33a00640 1 Processor -- start 2024-04-30T08:36:59.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.691+0000 7fad33a00640 1 -- start start 2024-04-30T08:36:59.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.691+0000 7fad33a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad2c105d80 0x7fad2c19c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:59.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.691+0000 7fad33a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad2c106fc0 0x7fad2c19c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:59.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.691+0000 7fad33a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad2c10bcc0 0x7fad2c1a3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:59.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.692+0000 7fad33a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fad2c1118b0 con 0x7fad2c10bcc0 2024-04-30T08:36:59.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.692+0000 7fad33a00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fad2c111730 con 0x7fad2c106fc0 2024-04-30T08:36:59.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.692+0000 7fad33a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fad2c111a30 con 0x7fad2c105d80 2024-04-30T08:36:59.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.692+0000 7fad2be00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad2c106fc0 0x7fad2c19c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:59.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.692+0000 7fad31a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad2c10bcc0 0x7fad2c1a3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:59.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.692+0000 7fad2be00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad2c106fc0 0x7fad2c19c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:35562/0 (socket says 172.21.15.32:35562) 2024-04-30T08:36:59.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.692+0000 7fad2be00640 1 -- 172.21.15.32:0/3598643325 learned_addr learned my addr 172.21.15.32:0/3598643325 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:59.693 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.692+0000 7fad31000640 1 --2- 172.21.15.32:0/3598643325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad2c105d80 0x7fad2c19c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:59.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.692+0000 7fad2be00640 1 -- 172.21.15.32:0/3598643325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad2c105d80 msgr2=0x7fad2c19c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:59.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.692+0000 7fad2be00640 1 --2- 172.21.15.32:0/3598643325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad2c105d80 0x7fad2c19c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.692+0000 7fad2be00640 1 -- 172.21.15.32:0/3598643325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad2c10bcc0 msgr2=0x7fad2c1a3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:59.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.692+0000 7fad2be00640 1 --2- 172.21.15.32:0/3598643325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad2c10bcc0 0x7fad2c1a3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.694 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.692+0000 7fad2be00640 1 -- 172.21.15.32:0/3598643325 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad2c1a3fa0 con 0x7fad2c106fc0 2024-04-30T08:36:59.695 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.693+0000 7fad2be00640 1 --2- 172.21.15.32:0/3598643325 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad2c106fc0 0x7fad2c19c9a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fad18006010 tx=0x7fad18031a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:59.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.693+0000 7fad29600640 1 -- 172.21.15.32:0/3598643325 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad1803f030 con 0x7fad2c106fc0 2024-04-30T08:36:59.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.693+0000 7fad29600640 1 -- 172.21.15.32:0/3598643325 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad18003aa0 con 0x7fad2c106fc0 2024-04-30T08:36:59.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.693+0000 7fad33a00640 1 -- 172.21.15.32:0/3598643325 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad2c1a4230 con 0x7fad2c106fc0 2024-04-30T08:36:59.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.693+0000 7fad29600640 1 -- 172.21.15.32:0/3598643325 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad18035070 con 0x7fad2c106fc0 2024-04-30T08:36:59.696 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.693+0000 7fad33a00640 1 -- 172.21.15.32:0/3598643325 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad2c1a4720 con 0x7fad2c106fc0 2024-04-30T08:36:59.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.697+0000 7fad3aa3d740 1 Processor -- start 2024-04-30T08:36:59.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.697+0000 7fad29600640 1 -- 172.21.15.32:0/3598643325 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7fad18042030 con 0x7fad2c106fc0 2024-04-30T08:36:59.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.697+0000 7fad3aa3d740 1 -- start start 2024-04-30T08:36:59.697 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.697+0000 7fad3aa3d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d40aa7c390 0x55d40ae1d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:59.698 INFO:tasks.workunit.client.0.smithi032.stdout:checking for open64... yes 2024-04-30T08:36:59.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.697+0000 7fad3aa3d740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d40aa534d0 0x55d40ae1daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:59.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.697+0000 7fad3aa3d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40aa5b6b0 0x55d40ae249f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:59.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.697+0000 7fad3aa3d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d40ad82d50 con 0x55d40aa5b6b0 2024-04-30T08:36:59.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.697+0000 7fad3aa3d740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55d40ad280c0 con 0x55d40aa534d0 2024-04-30T08:36:59.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.697+0000 7fad3aa3d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d40ad7ed10 con 0x55d40aa7c390 2024-04-30T08:36:59.699 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.697+0000 7fad31000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d40aa534d0 0x55d40ae1daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:59.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.698+0000 7fad31a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d40aa7c390 0x55d40ae1d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:59.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.698+0000 7fad31000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d40aa534d0 0x55d40ae1daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:35574/0 (socket says 172.21.15.32:35574) 2024-04-30T08:36:59.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.698+0000 7fad31000640 1 -- 172.21.15.32:0/3615683548 learned_addr learned my addr 172.21.15.32:0/3615683548 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:59.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.698+0000 7fad2be00640 1 --2- 172.21.15.32:0/3615683548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40aa5b6b0 0x55d40ae249f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:59.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.698+0000 7fad31000640 1 -- 172.21.15.32:0/3615683548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d40aa7c390 msgr2=0x55d40ae1d570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:59.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.698+0000 7fad31000640 1 --2- 172.21.15.32:0/3615683548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d40aa7c390 0x55d40ae1d570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.698+0000 7fad31000640 1 -- 172.21.15.32:0/3615683548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40aa5b6b0 msgr2=0x55d40ae249f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:59.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.698+0000 7fad31000640 1 --2- 172.21.15.32:0/3615683548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40aa5b6b0 0x55d40ae249f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.698+0000 7fad31000640 1 -- 172.21.15.32:0/3615683548 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d40aa943f0 con 0x55d40aa534d0 2024-04-30T08:36:59.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.698+0000 7fad31000640 1 --2- 172.21.15.32:0/3615683548 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d40aa534d0 0x55d40ae1daf0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fad2000bc70 tx=0x7fad200107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=45f295cddac8c43f server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:59.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.699+0000 7fad1e000640 1 -- 172.21.15.32:0/3615683548 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad20004220 con 0x55d40aa534d0 2024-04-30T08:36:59.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.699+0000 7fad1e000640 1 -- 172.21.15.32:0/3615683548 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad2000be20 con 0x55d40aa534d0 2024-04-30T08:36:59.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.699+0000 7fad1e000640 1 -- 172.21.15.32:0/3615683548 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad20004c90 con 0x55d40aa534d0 2024-04-30T08:36:59.700 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.699+0000 7fad29600640 1 --2- 172.21.15.32:0/3598643325 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fad140afaf0 0x7fad140b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:59.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.699+0000 7fad29600640 1 -- 172.21.15.32:0/3598643325 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fad180ff560 con 0x7fad2c106fc0 2024-04-30T08:36:59.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.699+0000 7fad31a00640 1 --2- 172.21.15.32:0/3598643325 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fad140afaf0 0x7fad140b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:59.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.700+0000 7fad3aa3d740 1 -- 172.21.15.32:0/3615683548 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d40aa534d0 msgr2=0x55d40ae1daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:59.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.700+0000 7fad3aa3d740 1 --2- 172.21.15.32:0/3615683548 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d40aa534d0 0x55d40ae1daf0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fad2000bc70 tx=0x7fad200107a0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.700+0000 7fad3aa3d740 1 -- 172.21.15.32:0/3615683548 shutdown_connections 2024-04-30T08:36:59.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.700+0000 7fad3aa3d740 1 --2- 172.21.15.32:0/3615683548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40aa5b6b0 0x55d40ae249f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.700+0000 7fad3aa3d740 1 --2- 172.21.15.32:0/3615683548 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d40aa534d0 0x55d40ae1daf0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.700+0000 7fad3aa3d740 1 --2- 172.21.15.32:0/3615683548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d40aa7c390 0x55d40ae1d570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.700+0000 7fad3aa3d740 1 -- 172.21.15.32:0/3615683548 >> 172.21.15.32:0/3615683548 conn(0x55d40aa704e0 msgr2=0x55d40ae18f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:59.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.700+0000 7fad3aa3d740 1 -- 172.21.15.32:0/3615683548 shutdown_connections 2024-04-30T08:36:59.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.700+0000 7fad31a00640 1 --2- 172.21.15.32:0/3598643325 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fad140afaf0 0x7fad140b2000 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x55d40ae1e740 tx=0x7fad24009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:59.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.700+0000 7fad3aa3d740 1 -- 172.21.15.32:0/3615683548 wait complete. 2024-04-30T08:36:59.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.701+0000 7fad3aa3d740 1 -- start start 2024-04-30T08:36:59.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.701+0000 7fad3aa3d740 1 Processor -- start 2024-04-30T08:36:59.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.701+0000 7fad3aa3d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d40aa534d0 0x55d40ae27480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:59.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.701+0000 7fad3aa3d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40aa5b6b0 0x55d40ae279c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:59.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.701+0000 7fad3aa3d740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d40aa7c390 0x55d40ae2e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:59.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.701+0000 7fad3aa3d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d40ae25440 con 0x55d40aa5b6b0 2024-04-30T08:36:59.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.701+0000 7fad3aa3d740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55d40ad3bb50 con 0x55d40aa7c390 2024-04-30T08:36:59.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.701+0000 7fad3aa3d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d40ae255c0 con 0x55d40aa534d0 2024-04-30T08:36:59.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.701+0000 7fad3aa3d740 10 client.0 _ll_register_callbacks cb 0x55d40ad05420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:36:59.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.702+0000 7fad31000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d40aa534d0 0x55d40ae27480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:59.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.702+0000 7fad2be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40aa5b6b0 0x55d40ae279c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:59.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.702+0000 7fad31000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d40aa534d0 0x55d40ae27480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:36820/0 (socket says 172.21.15.32:36820) 2024-04-30T08:36:59.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.702+0000 7fad31000640 1 -- 172.21.15.32:0/4193141429 learned_addr learned my addr 172.21.15.32:0/4193141429 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:59.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.702+0000 7fad31a00640 1 --2- 172.21.15.32:0/4193141429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d40aa7c390 0x55d40ae2e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:59.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.702+0000 7fad31000640 1 -- 172.21.15.32:0/4193141429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d40aa7c390 msgr2=0x55d40ae2e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:59.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.702+0000 7fad31000640 1 --2- 172.21.15.32:0/4193141429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d40aa7c390 0x55d40ae2e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.702+0000 7fad31000640 1 -- 172.21.15.32:0/4193141429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40aa5b6b0 msgr2=0x55d40ae279c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:59.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.702+0000 7fad31000640 1 --2- 172.21.15.32:0/4193141429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40aa5b6b0 0x55d40ae279c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.702+0000 7fad31000640 1 -- 172.21.15.32:0/4193141429 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d40ab5e7d0 con 0x55d40aa534d0 2024-04-30T08:36:59.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.702+0000 7fad31a00640 1 --2- 172.21.15.32:0/4193141429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d40aa7c390 0x55d40ae2e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:36:59.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.702+0000 7fad2be00640 1 --2- 172.21.15.32:0/4193141429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40aa5b6b0 0x55d40ae279c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:36:59.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.703+0000 7fad31000640 1 --2- 172.21.15.32:0/4193141429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d40aa534d0 0x55d40ae27480 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fad20004770 tx=0x7fad2001d680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd458cc94d289e15 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:59.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.703+0000 7fad12a00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-30T08:36:59.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.703+0000 7fad12a00640 1 -- 172.21.15.32:0/4193141429 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad2002b060 con 0x55d40aa534d0 2024-04-30T08:36:59.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.703+0000 7fad3aa3d740 1 -- 172.21.15.32:0/4193141429 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad2000be20 con 0x55d40aa534d0 2024-04-30T08:36:59.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.703+0000 7fad12a00640 1 -- 172.21.15.32:0/4193141429 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fad200055f0 con 0x55d40aa534d0 2024-04-30T08:36:59.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.703+0000 7fad12a00640 1 -- 172.21.15.32:0/4193141429 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fad20032050 con 0x55d40aa534d0 2024-04-30T08:36:59.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.703+0000 7fad3aa3d740 10 client.26095 fetch_fsmap learned FSMap version 29 2024-04-30T08:36:59.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.703+0000 7fad3aa3d740 1 -- 172.21.15.32:0/4193141429 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d40ae318a0 con 0x55d40aa534d0 2024-04-30T08:36:59.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.704+0000 7fad12a00640 1 -- 172.21.15.32:0/4193141429 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad2000be20 con 0x55d40aa534d0 2024-04-30T08:36:59.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.704+0000 7fad12a00640 1 -- 172.21.15.32:0/4193141429 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7fad20011880 con 0x55d40aa534d0 2024-04-30T08:36:59.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.704+0000 7fad3aa3d740 10 client.26095 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:36:59.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.704+0000 7fad3aa3d740 10 client.26095 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-30T08:36:59.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.704+0000 7fad3aa3d740 20 client.26095 populate_metadata read hostname 'smithi032' 2024-04-30T08:36:59.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.704+0000 7fad3aa3d740 1 --2- 172.21.15.32:0/4193141429 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x55d40aa3af60 0x55d40ae34c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:59.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.704+0000 7fad3aa3d740 4 client.26095 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:36:59.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.704+0000 7fad3aa3d740 1 -- 172.21.15.32:0/4193141429 --> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d40ad90960 con 0x55d40aa3af60 2024-04-30T08:36:59.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.704+0000 7fad2be00640 1 --2- 172.21.15.32:0/4193141429 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x55d40aa3af60 0x55d40ae34c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:59.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.705+0000 7fad2be00640 1 --2- 172.21.15.32:0/4193141429 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x55d40aa3af60 0x55d40ae34c60 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=afcfb28e1a6b0893 server_cookie=fdf4c4d9a9804ba6 in_seq=0 out_seq=0 2024-04-30T08:36:59.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.705+0000 7fad12a00640 10 client.26095 ms_handle_connect on v2:172.21.15.189:6836/686520873 2024-04-30T08:36:59.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.706+0000 7fad12a00640 1 -- 172.21.15.32:0/4193141429 <== mds.4 v2:172.21.15.189:6836/686520873 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d40ad90960 con 0x55d40aa3af60 2024-04-30T08:36:59.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.706+0000 7fad12a00640 10 client.26095 handle_command_reply: tid=0 2024-04-30T08:36:59.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.706+0000 7fad3aa3d740 1 client.26095 shutdown 2024-04-30T08:36:59.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.706+0000 7fad3aa3d740 1 -- 172.21.15.32:0/4193141429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d40aa534d0 msgr2=0x55d40ae27480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:59.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.706+0000 7fad3aa3d740 1 --2- 172.21.15.32:0/4193141429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d40aa534d0 0x55d40ae27480 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fad20004770 tx=0x7fad2001d680 comp rx=0 tx=0).stop 2024-04-30T08:36:59.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.706+0000 7fad3aa3d740 1 -- 172.21.15.32:0/4193141429 shutdown_connections 2024-04-30T08:36:59.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.706+0000 7fad3aa3d740 1 --2- 172.21.15.32:0/4193141429 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x55d40aa3af60 0x55d40ae34c60 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.706+0000 7fad3aa3d740 1 --2- 172.21.15.32:0/4193141429 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d40aa7c390 0x55d40ae2e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.706+0000 7fad3aa3d740 1 --2- 172.21.15.32:0/4193141429 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40aa5b6b0 0x55d40ae279c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.706+0000 7fad3aa3d740 1 --2- 172.21.15.32:0/4193141429 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d40aa534d0 0x55d40ae27480 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.706+0000 7fad12a00640 0 client.26095 ms_handle_reset on v2:172.21.15.189:6836/686520873 2024-04-30T08:36:59.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.706+0000 7fad3aa3d740 1 -- 172.21.15.32:0/4193141429 >> 172.21.15.32:0/4193141429 conn(0x55d40aa704e0 msgr2=0x55d40ae17f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:59.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.706+0000 7fad3aa3d740 1 -- 172.21.15.32:0/4193141429 shutdown_connections 2024-04-30T08:36:59.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.706+0000 7fad3aa3d740 1 -- 172.21.15.32:0/4193141429 wait complete. 2024-04-30T08:36:59.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.706+0000 7fad3aa3d740 20 client.26095 trim_cache size 0 max 16384 2024-04-30T08:36:59.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.722+0000 7fad3aa3d740 1 Processor -- start 2024-04-30T08:36:59.722 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.722+0000 7fad3aa3d740 1 -- start start 2024-04-30T08:36:59.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.722+0000 7fad3aa3d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40aa534d0 0x55d40ae18fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:59.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.722+0000 7fad3aa3d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d40aa5b6b0 0x55d40ae2e840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:59.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.722+0000 7fad3aa3d740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d40aa7c390 0x55d40ae2ed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:59.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.722+0000 7fad3aa3d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fad2c1aed30 con 0x55d40aa534d0 2024-04-30T08:36:59.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.722+0000 7fad3aa3d740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fad2c1b0900 con 0x55d40aa7c390 2024-04-30T08:36:59.723 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.722+0000 7fad3aa3d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fad2c1af400 con 0x55d40aa5b6b0 2024-04-30T08:36:59.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.723+0000 7fad31000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40aa534d0 0x55d40ae18fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:59.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.723+0000 7fad31a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d40aa7c390 0x55d40ae2ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:59.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.723+0000 7fad2be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d40aa5b6b0 0x55d40ae2e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:59.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.723+0000 7fad2be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d40aa5b6b0 0x55d40ae2e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:36832/0 (socket says 172.21.15.32:36832) 2024-04-30T08:36:59.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.723+0000 7fad31a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d40aa7c390 0x55d40ae2ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:35592/0 (socket says 172.21.15.32:35592) 2024-04-30T08:36:59.724 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.723+0000 7fad2be00640 1 -- 172.21.15.32:0/337645644 learned_addr learned my addr 172.21.15.32:0/337645644 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:59.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.723+0000 7fad2be00640 1 -- 172.21.15.32:0/337645644 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d40aa7c390 msgr2=0x55d40ae2ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:59.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.723+0000 7fad2be00640 1 --2- 172.21.15.32:0/337645644 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d40aa7c390 0x55d40ae2ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.723+0000 7fad2be00640 1 -- 172.21.15.32:0/337645644 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40aa534d0 msgr2=0x55d40ae18fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:59.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.723+0000 7fad2be00640 1 --2- 172.21.15.32:0/337645644 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40aa534d0 0x55d40ae18fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.723+0000 7fad2be00640 1 -- 172.21.15.32:0/337645644 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d40ae1d120 con 0x55d40aa5b6b0 2024-04-30T08:36:59.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.723+0000 7fad2be00640 1 --2- 172.21.15.32:0/337645644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d40aa5b6b0 0x55d40ae2e840 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fad180c7420 tx=0x7fad180b0040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=463eaf5ae1db4e4f server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:59.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.724+0000 7fad1d600640 1 -- 172.21.15.32:0/337645644 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad18049040 con 0x55d40aa5b6b0 2024-04-30T08:36:59.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.724+0000 7fad1d600640 1 -- 172.21.15.32:0/337645644 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad180c7230 con 0x55d40aa5b6b0 2024-04-30T08:36:59.725 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.724+0000 7fad1d600640 1 -- 172.21.15.32:0/337645644 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad18102cd0 con 0x55d40aa5b6b0 2024-04-30T08:36:59.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.725+0000 7fad3aa3d740 1 -- 172.21.15.32:0/337645644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d40aa5b6b0 msgr2=0x55d40ae2e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:59.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.725+0000 7fad3aa3d740 1 --2- 172.21.15.32:0/337645644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d40aa5b6b0 0x55d40ae2e840 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fad180c7420 tx=0x7fad180b0040 comp rx=0 tx=0).stop 2024-04-30T08:36:59.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.725+0000 7fad3aa3d740 1 -- 172.21.15.32:0/337645644 shutdown_connections 2024-04-30T08:36:59.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.725+0000 7fad3aa3d740 1 --2- 172.21.15.32:0/337645644 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d40aa7c390 0x55d40ae2ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.725+0000 7fad3aa3d740 1 --2- 172.21.15.32:0/337645644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d40aa5b6b0 0x55d40ae2e840 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.725+0000 7fad3aa3d740 1 --2- 172.21.15.32:0/337645644 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40aa534d0 0x55d40ae18fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.725+0000 7fad3aa3d740 1 -- 172.21.15.32:0/337645644 >> 172.21.15.32:0/337645644 conn(0x55d40aa704e0 msgr2=0x55d40ae18890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:59.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.725+0000 7fad3aa3d740 1 -- 172.21.15.32:0/337645644 shutdown_connections 2024-04-30T08:36:59.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.725+0000 7fad3aa3d740 1 -- 172.21.15.32:0/337645644 wait complete. 2024-04-30T08:36:59.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.725+0000 7fad3aa3d740 1 -- start start 2024-04-30T08:36:59.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.725+0000 7fad3aa3d740 1 Processor -- start 2024-04-30T08:36:59.729 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.726+0000 7fad3aa3d740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d40aa534d0 0x55d40ae2f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:59.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.726+0000 7fad3aa3d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40aa5b6b0 0x55d40ae2bc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:59.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.726+0000 7fad3aa3d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d40aa7c390 0x55d40ae2c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:59.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.726+0000 7fad3aa3d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fad2c1b01a0 con 0x55d40aa5b6b0 2024-04-30T08:36:59.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.726+0000 7fad3aa3d740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fad2c1afad0 con 0x55d40aa534d0 2024-04-30T08:36:59.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.726+0000 7fad3aa3d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fad2c1accf0 con 0x55d40aa7c390 2024-04-30T08:36:59.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.726+0000 7fad3aa3d740 10 client.0 _ll_register_callbacks cb 0x7fad24009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:36:59.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.726+0000 7fad2be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40aa5b6b0 0x55d40ae2bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:59.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.726+0000 7fad31000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d40aa534d0 0x55d40ae2f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:59.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.726+0000 7fad31a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d40aa7c390 0x55d40ae2c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:59.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.726+0000 7fad31000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d40aa534d0 0x55d40ae2f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:35608/0 (socket says 172.21.15.32:35608) 2024-04-30T08:36:59.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.726+0000 7fad31000640 1 -- 172.21.15.32:0/3262868298 learned_addr learned my addr 172.21.15.32:0/3262868298 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:36:59.730 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.727+0000 7fad31000640 1 -- 172.21.15.32:0/3262868298 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d40aa7c390 msgr2=0x55d40ae2c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:59.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.727+0000 7fad31000640 1 --2- 172.21.15.32:0/3262868298 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d40aa7c390 0x55d40ae2c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.727+0000 7fad31000640 1 -- 172.21.15.32:0/3262868298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40aa5b6b0 msgr2=0x55d40ae2bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:36:59.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.727+0000 7fad31000640 1 --2- 172.21.15.32:0/3262868298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40aa5b6b0 0x55d40ae2bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.727+0000 7fad31000640 1 -- 172.21.15.32:0/3262868298 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d40ae9aee0 con 0x55d40aa534d0 2024-04-30T08:36:59.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.727+0000 7fad31000640 1 --2- 172.21.15.32:0/3262868298 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d40aa534d0 0x55d40ae2f0f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fad20013010 tx=0x7fad20004640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4a4a77bd95058dd server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:36:59.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.727+0000 7fad12a00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:36:59.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.727+0000 7fad12a00640 1 -- 172.21.15.32:0/3262868298 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad2003b060 con 0x55d40aa534d0 2024-04-30T08:36:59.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.727+0000 7fad3aa3d740 1 -- 172.21.15.32:0/3262868298 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fad180c7230 con 0x55d40aa534d0 2024-04-30T08:36:59.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.727+0000 7fad12a00640 1 -- 172.21.15.32:0/3262868298 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fad2000b690 con 0x55d40aa534d0 2024-04-30T08:36:59.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.728+0000 7fad12a00640 1 -- 172.21.15.32:0/3262868298 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fad2002b040 con 0x55d40aa534d0 2024-04-30T08:36:59.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.728+0000 7fad3aa3d740 10 client.26102 fetch_fsmap learned FSMap version 29 2024-04-30T08:36:59.731 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.728+0000 7fad3aa3d740 1 -- 172.21.15.32:0/3262868298 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d40ae9c100 con 0x55d40aa534d0 2024-04-30T08:36:59.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.728+0000 7fad12a00640 1 -- 172.21.15.32:0/3262868298 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fad180c7230 con 0x55d40aa534d0 2024-04-30T08:36:59.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.728+0000 7fad12a00640 1 -- 172.21.15.32:0/3262868298 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7fad2001d860 con 0x55d40aa534d0 2024-04-30T08:36:59.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.728+0000 7fad3aa3d740 10 client.26102 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:36:59.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.728+0000 7fad3aa3d740 10 client.26102 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-30T08:36:59.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.728+0000 7fad3aa3d740 20 client.26102 populate_metadata read hostname 'smithi032' 2024-04-30T08:36:59.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.728+0000 7fad3aa3d740 1 --2- 172.21.15.32:0/3262868298 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x55d40aa103a0 0x55d40ae9eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:36:59.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.728+0000 7fad3aa3d740 4 client.26102 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-30T08:36:59.732 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.728+0000 7fad3aa3d740 1 -- 172.21.15.32:0/3262868298 --> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d40ad8a7d0 con 0x55d40aa103a0 2024-04-30T08:36:59.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.729+0000 7fad2be00640 1 --2- 172.21.15.32:0/3262868298 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x55d40aa103a0 0x55d40ae9eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:36:59.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.729+0000 7fad2be00640 1 --2- 172.21.15.32:0/3262868298 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x55d40aa103a0 0x55d40ae9eac0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=86ca8db69d13b00f server_cookie=c31bea59af00e2fe in_seq=0 out_seq=0 2024-04-30T08:36:59.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.729+0000 7fad12a00640 10 client.26102 ms_handle_connect on v2:172.21.15.189:6836/686520873 2024-04-30T08:36:59.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.729+0000 7fad12a00640 1 -- 172.21.15.32:0/3262868298 <== mds.4 v2:172.21.15.189:6836/686520873 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d40ad8a7d0 con 0x55d40aa103a0 2024-04-30T08:36:59.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.729+0000 7fad12a00640 10 client.26102 handle_command_reply: tid=0 2024-04-30T08:36:59.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.730+0000 7fad3aa3d740 1 client.26102 shutdown 2024-04-30T08:36:59.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.730+0000 7fad3aa3d740 1 -- 172.21.15.32:0/3262868298 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d40aa534d0 msgr2=0x55d40ae2f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:59.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.730+0000 7fad3aa3d740 1 --2- 172.21.15.32:0/3262868298 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d40aa534d0 0x55d40ae2f0f0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fad20013010 tx=0x7fad20004640 comp rx=0 tx=0).stop 2024-04-30T08:36:59.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.730+0000 7fad3aa3d740 1 -- 172.21.15.32:0/3262868298 shutdown_connections 2024-04-30T08:36:59.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.730+0000 7fad3aa3d740 1 --2- 172.21.15.32:0/3262868298 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x55d40aa103a0 0x55d40ae9eac0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.730+0000 7fad3aa3d740 1 --2- 172.21.15.32:0/3262868298 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d40aa7c390 0x55d40ae2c230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.730+0000 7fad3aa3d740 1 --2- 172.21.15.32:0/3262868298 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d40aa5b6b0 0x55d40ae2bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.730+0000 7fad12a00640 0 client.26102 ms_handle_reset on v2:172.21.15.189:6836/686520873 2024-04-30T08:36:59.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.730+0000 7fad3aa3d740 1 --2- 172.21.15.32:0/3262868298 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d40aa534d0 0x55d40ae2f0f0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.730+0000 7fad3aa3d740 1 -- 172.21.15.32:0/3262868298 >> 172.21.15.32:0/3262868298 conn(0x55d40aa704e0 msgr2=0x55d40ae17510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:59.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.730+0000 7fad3aa3d740 1 -- 172.21.15.32:0/3262868298 shutdown_connections 2024-04-30T08:36:59.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.730+0000 7fad3aa3d740 1 -- 172.21.15.32:0/3262868298 wait complete. 2024-04-30T08:36:59.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.730+0000 7fad3aa3d740 20 client.26102 trim_cache size 0 max 16384 2024-04-30T08:36:59.735 INFO:teuthology.orchestra.run.smithi032.stdout:[] 2024-04-30T08:36:59.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.731+0000 7fad33a00640 1 -- 172.21.15.32:0/3598643325 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fad140afaf0 msgr2=0x7fad140b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:59.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.731+0000 7fad33a00640 1 --2- 172.21.15.32:0/3598643325 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fad140afaf0 0x7fad140b2000 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x55d40ae1e740 tx=0x7fad24009500 comp rx=0 tx=0).stop 2024-04-30T08:36:59.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.731+0000 7fad33a00640 1 -- 172.21.15.32:0/3598643325 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad2c106fc0 msgr2=0x7fad2c19c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:36:59.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.731+0000 7fad33a00640 1 --2- 172.21.15.32:0/3598643325 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad2c106fc0 0x7fad2c19c9a0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fad18006010 tx=0x7fad18031a50 comp rx=0 tx=0).stop 2024-04-30T08:36:59.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.731+0000 7fad33a00640 1 -- 172.21.15.32:0/3598643325 shutdown_connections 2024-04-30T08:36:59.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.731+0000 7fad33a00640 1 --2- 172.21.15.32:0/3598643325 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fad140afaf0 0x7fad140b2000 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.731+0000 7fad33a00640 1 --2- 172.21.15.32:0/3598643325 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fad2c10bcc0 0x7fad2c1a3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.731+0000 7fad33a00640 1 --2- 172.21.15.32:0/3598643325 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fad2c106fc0 0x7fad2c19c9a0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.731+0000 7fad33a00640 1 --2- 172.21.15.32:0/3598643325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fad2c105d80 0x7fad2c19c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:36:59.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.731+0000 7fad33a00640 1 -- 172.21.15.32:0/3598643325 >> 172.21.15.32:0/3598643325 conn(0x7fad2c100f70 msgr2=0x7fad2c107ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:36:59.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.731+0000 7fad33a00640 1 -- 172.21.15.32:0/3598643325 shutdown_connections 2024-04-30T08:36:59.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:36:59.731+0000 7fad33a00640 1 -- 172.21.15.32:0/3598643325 wait complete. 2024-04-30T08:36:59.760 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-30T08:36:59.830 INFO:tasks.workunit.client.0.smithi032.stdout:checking for stat64... yes 2024-04-30T08:36:59.959 INFO:tasks.workunit.client.0.smithi032.stdout:checking for fseeko64... yes 2024-04-30T08:37:00.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.072+0000 7f84ff400640 1 -- 172.21.15.32:0/3040967120 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84f8102a70 msgr2=0x7f84f8103770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:00.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.072+0000 7f84ff400640 1 --2- 172.21.15.32:0/3040967120 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84f8102a70 0x7f84f8103770 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f84e8009900 tx=0x7f84e802a3f0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.072+0000 7f84ff400640 1 -- 172.21.15.32:0/3040967120 shutdown_connections 2024-04-30T08:37:00.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.072+0000 7f84ff400640 1 --2- 172.21.15.32:0/3040967120 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84f81089b0 0x7f84f810add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.072+0000 7f84ff400640 1 --2- 172.21.15.32:0/3040967120 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84f8103cb0 0x7f84f81040c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.072+0000 7f84ff400640 1 --2- 172.21.15.32:0/3040967120 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84f8102a70 0x7f84f8103770 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.072+0000 7f84ff400640 1 -- 172.21.15.32:0/3040967120 >> 172.21.15.32:0/3040967120 conn(0x7f84f80fdc80 msgr2=0x7f84f8100110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:00.073 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.072+0000 7f84ff400640 1 -- 172.21.15.32:0/3040967120 shutdown_connections 2024-04-30T08:37:00.074 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.072+0000 7f84ff400640 1 -- 172.21.15.32:0/3040967120 wait complete. 2024-04-30T08:37:00.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.073+0000 7f84ff400640 1 Processor -- start 2024-04-30T08:37:00.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.073+0000 7f84ff400640 1 -- start start 2024-04-30T08:37:00.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.073+0000 7f84ff400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84f8102a70 0x7f84f819c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.074+0000 7f84ff400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84f8103cb0 0x7f84f819ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.074+0000 7f84ff400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84f81089b0 0x7f84f81a3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.074+0000 7f84ff400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f84f806c010 con 0x7f84f8103cb0 2024-04-30T08:37:00.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.074+0000 7f84ff400640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f84f806be90 con 0x7f84f81089b0 2024-04-30T08:37:00.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.074+0000 7f84ff400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f84f806c190 con 0x7f84f8102a70 2024-04-30T08:37:00.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.074+0000 7f84f7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84f8103cb0 0x7f84f819ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.074+0000 7f84f7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84f8103cb0 0x7f84f819ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56420/0 (socket says 172.21.15.32:56420) 2024-04-30T08:37:00.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.074+0000 7f84fca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84f8102a70 0x7f84f819c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.076 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.074+0000 7f84f7e00640 1 -- 172.21.15.32:0/1962000689 learned_addr learned my addr 172.21.15.32:0/1962000689 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:00.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.074+0000 7f84fd400640 1 --2- 172.21.15.32:0/1962000689 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84f81089b0 0x7f84f81a3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.074+0000 7f84f7e00640 1 -- 172.21.15.32:0/1962000689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84f8102a70 msgr2=0x7f84f819c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:00.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.074+0000 7f84f7e00640 1 --2- 172.21.15.32:0/1962000689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84f8102a70 0x7f84f819c550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.075+0000 7f84f7e00640 1 -- 172.21.15.32:0/1962000689 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84f81089b0 msgr2=0x7f84f81a3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:00.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.075+0000 7f84f7e00640 1 --2- 172.21.15.32:0/1962000689 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84f81089b0 0x7f84f81a3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.075+0000 7f84f7e00640 1 -- 172.21.15.32:0/1962000689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84f81a4090 con 0x7f84f8103cb0 2024-04-30T08:37:00.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.075+0000 7f84fd400640 1 --2- 172.21.15.32:0/1962000689 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84f81089b0 0x7f84f81a3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:37:00.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.075+0000 7f84fca00640 1 --2- 172.21.15.32:0/1962000689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84f8102a70 0x7f84f819c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:37:00.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.075+0000 7f84f7e00640 1 --2- 172.21.15.32:0/1962000689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84f8103cb0 0x7f84f819ca90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f84ec00e9f0 tx=0x7f84ec00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:00.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.075+0000 7f84f5600640 1 -- 172.21.15.32:0/1962000689 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84ec00cd80 con 0x7f84f8103cb0 2024-04-30T08:37:00.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.075+0000 7f84ff400640 1 -- 172.21.15.32:0/1962000689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84f81a4380 con 0x7f84f8103cb0 2024-04-30T08:37:00.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.075+0000 7f84ff400640 1 -- 172.21.15.32:0/1962000689 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84f81a48d0 con 0x7f84f8103cb0 2024-04-30T08:37:00.077 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.075+0000 7f84f5600640 1 -- 172.21.15.32:0/1962000689 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84ec0054d0 con 0x7f84f8103cb0 2024-04-30T08:37:00.078 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.075+0000 7f84f5600640 1 -- 172.21.15.32:0/1962000689 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84ec010690 con 0x7f84f8103cb0 2024-04-30T08:37:00.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.079+0000 7f8506565740 1 Processor -- start 2024-04-30T08:37:00.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.079+0000 7f8506565740 1 -- start start 2024-04-30T08:37:00.080 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.079+0000 7f8506565740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557704e1a390 0x5577051bb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.079+0000 7f8506565740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557704df14d0 0x5577051bbaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.079+0000 7f8506565740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557704df96b0 0x5577051c29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.079+0000 7f8506565740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x557705120d50 con 0x557704df14d0 2024-04-30T08:37:00.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.079+0000 7f8506565740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x5577050c60c0 con 0x557704e1a390 2024-04-30T08:37:00.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.079+0000 7f8506565740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55770511cd10 con 0x557704df96b0 2024-04-30T08:37:00.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.079+0000 7f84f5600640 1 -- 172.21.15.32:0/1962000689 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f84ec010830 con 0x7f84f8103cb0 2024-04-30T08:37:00.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.080+0000 7f84fd400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557704e1a390 0x5577051bb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.080+0000 7f84fca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557704df14d0 0x5577051bbaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.080+0000 7f84fd400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557704e1a390 0x5577051bb570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:35624/0 (socket says 172.21.15.32:35624) 2024-04-30T08:37:00.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.080+0000 7f84fd400640 1 -- 172.21.15.32:0/3296428964 learned_addr learned my addr 172.21.15.32:0/3296428964 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:00.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.080+0000 7f84f7e00640 1 --2- 172.21.15.32:0/3296428964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557704df96b0 0x5577051c29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.080+0000 7f84fd400640 1 -- 172.21.15.32:0/3296428964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557704df96b0 msgr2=0x5577051c29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:00.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.080+0000 7f84fd400640 1 --2- 172.21.15.32:0/3296428964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557704df96b0 0x5577051c29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.080+0000 7f84fd400640 1 -- 172.21.15.32:0/3296428964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557704df14d0 msgr2=0x5577051bbaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:00.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.080+0000 7f84fd400640 1 --2- 172.21.15.32:0/3296428964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557704df14d0 0x5577051bbaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.082 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.080+0000 7f84fd400640 1 -- 172.21.15.32:0/3296428964 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557704e323f0 con 0x557704e1a390 2024-04-30T08:37:00.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.081+0000 7f84fd400640 1 --2- 172.21.15.32:0/3296428964 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557704e1a390 0x5577051bb570 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f84f00117e0 tx=0x7f84f0011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6011721e32720175 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:00.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.081+0000 7f84e5c00640 1 -- 172.21.15.32:0/3296428964 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f0011310 con 0x557704e1a390 2024-04-30T08:37:00.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.081+0000 7f84e5c00640 1 -- 172.21.15.32:0/3296428964 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84f0003e20 con 0x557704e1a390 2024-04-30T08:37:00.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.081+0000 7f84e5c00640 1 -- 172.21.15.32:0/3296428964 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f00044d0 con 0x557704e1a390 2024-04-30T08:37:00.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.081+0000 7f84f5600640 1 --2- 172.21.15.32:0/1962000689 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f84e00afaf0 0x7f84e00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.082+0000 7f84f5600640 1 -- 172.21.15.32:0/1962000689 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f84ec0df420 con 0x7f84f8103cb0 2024-04-30T08:37:00.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.082+0000 7f84fca00640 1 --2- 172.21.15.32:0/1962000689 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f84e00afaf0 0x7f84e00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.082+0000 7f8506565740 1 -- 172.21.15.32:0/3296428964 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557704e1a390 msgr2=0x5577051bb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:00.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.082+0000 7f8506565740 1 --2- 172.21.15.32:0/3296428964 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557704e1a390 0x5577051bb570 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f84f00117e0 tx=0x7f84f0011ca0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.082+0000 7f8506565740 1 -- 172.21.15.32:0/3296428964 shutdown_connections 2024-04-30T08:37:00.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.082+0000 7f8506565740 1 --2- 172.21.15.32:0/3296428964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557704df96b0 0x5577051c29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.082+0000 7f8506565740 1 --2- 172.21.15.32:0/3296428964 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557704df14d0 0x5577051bbaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.082+0000 7f8506565740 1 --2- 172.21.15.32:0/3296428964 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557704e1a390 0x5577051bb570 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.084 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.082+0000 7f8506565740 1 -- 172.21.15.32:0/3296428964 >> 172.21.15.32:0/3296428964 conn(0x557704e0e4e0 msgr2=0x5577051b6f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:00.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.082+0000 7f8506565740 1 -- 172.21.15.32:0/3296428964 shutdown_connections 2024-04-30T08:37:00.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.082+0000 7f8506565740 1 -- 172.21.15.32:0/3296428964 wait complete. 2024-04-30T08:37:00.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.083+0000 7f84fca00640 1 --2- 172.21.15.32:0/1962000689 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f84e00afaf0 0x7f84e00b2000 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f84e802a9e0 tx=0x7f84e80314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:00.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.083+0000 7f8506565740 1 -- start start 2024-04-30T08:37:00.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.083+0000 7f8506565740 1 Processor -- start 2024-04-30T08:37:00.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.084+0000 7f8506565740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557704df14d0 0x5577051c5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.084+0000 7f8506565740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557704df96b0 0x5577051c59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.084+0000 7f8506565740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557704e1a390 0x5577051cc900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.084+0000 7f8506565740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5577051c3440 con 0x557704e1a390 2024-04-30T08:37:00.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.084+0000 7f8506565740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x5577050d9b50 con 0x557704df14d0 2024-04-30T08:37:00.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.084+0000 7f8506565740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5577051c35c0 con 0x557704df96b0 2024-04-30T08:37:00.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.084+0000 7f8506565740 10 client.0 _ll_register_callbacks cb 0x5577050a3420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:00.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.084+0000 7f84fd400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557704e1a390 0x5577051cc900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.084+0000 7f84fca00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557704df14d0 0x5577051c5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.084+0000 7f84fd400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557704e1a390 0x5577051cc900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56442/0 (socket says 172.21.15.32:56442) 2024-04-30T08:37:00.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.084+0000 7f84fd400640 1 -- 172.21.15.32:0/315728054 learned_addr learned my addr 172.21.15.32:0/315728054 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:00.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.084+0000 7f84f7e00640 1 --2- 172.21.15.32:0/315728054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557704df96b0 0x5577051c59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.085+0000 7f84fd400640 1 -- 172.21.15.32:0/315728054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557704df96b0 msgr2=0x5577051c59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:00.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.085+0000 7f84fd400640 1 --2- 172.21.15.32:0/315728054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557704df96b0 0x5577051c59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.085+0000 7f84fd400640 1 -- 172.21.15.32:0/315728054 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557704df14d0 msgr2=0x5577051c5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:00.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.085+0000 7f84fd400640 1 --2- 172.21.15.32:0/315728054 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557704df14d0 0x5577051c5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.085+0000 7f84fd400640 1 -- 172.21.15.32:0/315728054 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557704efc7d0 con 0x557704e1a390 2024-04-30T08:37:00.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.085+0000 7f84f7e00640 1 --2- 172.21.15.32:0/315728054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557704df96b0 0x5577051c59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:37:00.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.085+0000 7f84fd400640 1 --2- 172.21.15.32:0/315728054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557704e1a390 0x5577051cc900 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f84f00113c0 tx=0x7f84f00113f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f89ccc5919e86de0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:00.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.085+0000 7f84da000640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:37:00.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.085+0000 7f84da000640 1 -- 172.21.15.32:0/315728054 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f002a060 con 0x557704e1a390 2024-04-30T08:37:00.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.085+0000 7f84da000640 1 -- 172.21.15.32:0/315728054 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f84f000f9a0 con 0x557704e1a390 2024-04-30T08:37:00.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.086+0000 7f8506565740 1 -- 172.21.15.32:0/315728054 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84f0003e20 con 0x557704e1a390 2024-04-30T08:37:00.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.086+0000 7f84da000640 1 -- 172.21.15.32:0/315728054 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f84f0031050 con 0x557704e1a390 2024-04-30T08:37:00.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.086+0000 7f8506565740 10 client.15978 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:00.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.086+0000 7f8506565740 1 -- 172.21.15.32:0/315728054 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5577051cf830 con 0x557704e1a390 2024-04-30T08:37:00.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.086+0000 7f84da000640 1 -- 172.21.15.32:0/315728054 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84f0003e20 con 0x557704e1a390 2024-04-30T08:37:00.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.086+0000 7f84da000640 1 -- 172.21.15.32:0/315728054 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7f84f0018020 con 0x557704e1a390 2024-04-30T08:37:00.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.087+0000 7f8506565740 10 client.15978 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:00.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.087+0000 7f8506565740 10 client.15978 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-30T08:37:00.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.087+0000 7f8506565740 20 client.15978 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:00.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.087+0000 7f8506565740 1 --2- 172.21.15.32:0/315728054 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x557704dd8f60 0x5577051d2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.087+0000 7f8506565740 4 client.15978 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:37:00.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.087+0000 7f8506565740 1 -- 172.21.15.32:0/315728054 --> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55770512e960 con 0x557704dd8f60 2024-04-30T08:37:00.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.087+0000 7f84fca00640 1 --2- 172.21.15.32:0/315728054 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x557704dd8f60 0x5577051d2ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.088+0000 7f84fca00640 1 --2- 172.21.15.32:0/315728054 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x557704dd8f60 0x5577051d2ca0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=2bbe352cfa32162d server_cookie=3c1caf090ddcf48f in_seq=0 out_seq=0 2024-04-30T08:37:00.091 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.088+0000 7f84da000640 10 client.15978 ms_handle_connect on v2:172.21.15.71:6836/2324039932 2024-04-30T08:37:00.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.088+0000 7f84da000640 1 -- 172.21.15.32:0/315728054 <== mds.3 v2:172.21.15.71:6836/2324039932 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55770512e960 con 0x557704dd8f60 2024-04-30T08:37:00.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.088+0000 7f84da000640 10 client.15978 handle_command_reply: tid=0 2024-04-30T08:37:00.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.088+0000 7f8506565740 1 client.15978 shutdown 2024-04-30T08:37:00.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.089+0000 7f8506565740 1 -- 172.21.15.32:0/315728054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557704e1a390 msgr2=0x5577051cc900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:00.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.089+0000 7f8506565740 1 --2- 172.21.15.32:0/315728054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557704e1a390 0x5577051cc900 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f84f00113c0 tx=0x7f84f00113f0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.089+0000 7f8506565740 1 -- 172.21.15.32:0/315728054 shutdown_connections 2024-04-30T08:37:00.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.089+0000 7f8506565740 1 --2- 172.21.15.32:0/315728054 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x557704dd8f60 0x5577051d2ca0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.089+0000 7f8506565740 1 --2- 172.21.15.32:0/315728054 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557704e1a390 0x5577051cc900 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.089+0000 7f8506565740 1 --2- 172.21.15.32:0/315728054 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557704df96b0 0x5577051c59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.089+0000 7f8506565740 1 --2- 172.21.15.32:0/315728054 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557704df14d0 0x5577051c5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.089+0000 7f84da000640 0 client.15978 ms_handle_reset on v2:172.21.15.71:6836/2324039932 2024-04-30T08:37:00.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.089+0000 7f8506565740 1 -- 172.21.15.32:0/315728054 >> 172.21.15.32:0/315728054 conn(0x557704e0e4e0 msgr2=0x5577051b5f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:00.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.089+0000 7f8506565740 1 -- 172.21.15.32:0/315728054 shutdown_connections 2024-04-30T08:37:00.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.089+0000 7f8506565740 1 -- 172.21.15.32:0/315728054 wait complete. 2024-04-30T08:37:00.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.089+0000 7f8506565740 20 client.15978 trim_cache size 0 max 16384 2024-04-30T08:37:00.096 INFO:tasks.workunit.client.0.smithi032.stdout:checking for lrand48_r... yes 2024-04-30T08:37:00.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.105+0000 7f8506565740 1 Processor -- start 2024-04-30T08:37:00.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.105+0000 7f8506565740 1 -- start start 2024-04-30T08:37:00.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.105+0000 7f8506565740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557704df14d0 0x5577051c5b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.105+0000 7f8506565740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557704df96b0 0x5577051cc820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.105+0000 7f8506565740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557704e1a390 0x5577051d7490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.105+0000 7f8506565740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f84f81aef00 con 0x557704df14d0 2024-04-30T08:37:00.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.105+0000 7f8506565740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f84f81b0ad0 con 0x557704df96b0 2024-04-30T08:37:00.106 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.105+0000 7f8506565740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f84f81af5d0 con 0x557704e1a390 2024-04-30T08:37:00.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.105+0000 7f84f7e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557704df96b0 0x5577051cc820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.106+0000 7f84fca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557704df14d0 0x5577051c5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.106+0000 7f84f7e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557704df96b0 0x5577051cc820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:35638/0 (socket says 172.21.15.32:35638) 2024-04-30T08:37:00.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.106+0000 7f84f7e00640 1 -- 172.21.15.32:0/3372264330 learned_addr learned my addr 172.21.15.32:0/3372264330 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:00.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.106+0000 7f84fd400640 1 --2- 172.21.15.32:0/3372264330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557704e1a390 0x5577051d7490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.106+0000 7f84f7e00640 1 -- 172.21.15.32:0/3372264330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557704e1a390 msgr2=0x5577051d7490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:00.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.106+0000 7f84f7e00640 1 --2- 172.21.15.32:0/3372264330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557704e1a390 0x5577051d7490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.106+0000 7f84f7e00640 1 -- 172.21.15.32:0/3372264330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557704df14d0 msgr2=0x5577051c5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:00.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.106+0000 7f84f7e00640 1 --2- 172.21.15.32:0/3372264330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557704df14d0 0x5577051c5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.106+0000 7f84f7e00640 1 -- 172.21.15.32:0/3372264330 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577051c0310 con 0x557704df96b0 2024-04-30T08:37:00.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.106+0000 7f84f7e00640 1 --2- 172.21.15.32:0/3372264330 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557704df96b0 0x5577051cc820 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f84ec0a47a0 tx=0x7f84ec014000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eadf342d9c3c20b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:00.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.107+0000 7f84e5200640 1 -- 172.21.15.32:0/3372264330 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84ec090070 con 0x557704df96b0 2024-04-30T08:37:00.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.107+0000 7f84e5200640 1 -- 172.21.15.32:0/3372264330 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84ec0a7040 con 0x557704df96b0 2024-04-30T08:37:00.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.107+0000 7f84e5200640 1 -- 172.21.15.32:0/3372264330 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84ec0b0390 con 0x557704df96b0 2024-04-30T08:37:00.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.108+0000 7f8506565740 1 -- 172.21.15.32:0/3372264330 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557704df96b0 msgr2=0x5577051cc820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:00.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.108+0000 7f8506565740 1 --2- 172.21.15.32:0/3372264330 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557704df96b0 0x5577051cc820 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f84ec0a47a0 tx=0x7f84ec014000 comp rx=0 tx=0).stop 2024-04-30T08:37:00.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.108+0000 7f8506565740 1 -- 172.21.15.32:0/3372264330 shutdown_connections 2024-04-30T08:37:00.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.108+0000 7f8506565740 1 --2- 172.21.15.32:0/3372264330 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557704e1a390 0x5577051d7490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.108+0000 7f8506565740 1 --2- 172.21.15.32:0/3372264330 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557704df96b0 0x5577051cc820 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.108+0000 7f8506565740 1 --2- 172.21.15.32:0/3372264330 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557704df14d0 0x5577051c5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.108+0000 7f8506565740 1 -- 172.21.15.32:0/3372264330 >> 172.21.15.32:0/3372264330 conn(0x557704e0e4e0 msgr2=0x5577051c5460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:00.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.108+0000 7f8506565740 1 -- 172.21.15.32:0/3372264330 shutdown_connections 2024-04-30T08:37:00.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.108+0000 7f8506565740 1 -- 172.21.15.32:0/3372264330 wait complete. 2024-04-30T08:37:00.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.108+0000 7f8506565740 1 -- start start 2024-04-30T08:37:00.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.108+0000 7f8506565740 1 Processor -- start 2024-04-30T08:37:00.113 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.109+0000 7f8506565740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557704df14d0 0x5577051cb0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.109+0000 7f8506565740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557704df96b0 0x5577051cb630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.109+0000 7f8506565740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557704e1a390 0x5577051cbb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.109+0000 7f8506565740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f84f81b0370 con 0x557704e1a390 2024-04-30T08:37:00.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.109+0000 7f8506565740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f84f81afca0 con 0x557704df96b0 2024-04-30T08:37:00.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.109+0000 7f8506565740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f84f81acec0 con 0x557704df14d0 2024-04-30T08:37:00.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.109+0000 7f8506565740 10 client.0 _ll_register_callbacks cb 0x7f84ec005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:00.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.109+0000 7f84f7e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557704df96b0 0x5577051cb630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.109+0000 7f84fca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557704df14d0 0x5577051cb0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.109+0000 7f84fca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557704df14d0 0x5577051cb0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:36896/0 (socket says 172.21.15.32:36896) 2024-04-30T08:37:00.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.109+0000 7f84fca00640 1 -- 172.21.15.32:0/659772700 learned_addr learned my addr 172.21.15.32:0/659772700 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:00.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.109+0000 7f84f7e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557704df96b0 0x5577051cb630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:35648/0 (socket says 172.21.15.32:35648) 2024-04-30T08:37:00.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.109+0000 7f84fd400640 1 --2- 172.21.15.32:0/659772700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557704e1a390 0x5577051cbb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.110+0000 7f84f7e00640 1 -- 172.21.15.32:0/659772700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557704df14d0 msgr2=0x5577051cb0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:00.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.110+0000 7f84f7e00640 1 --2- 172.21.15.32:0/659772700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557704df14d0 0x5577051cb0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.110+0000 7f84f7e00640 1 -- 172.21.15.32:0/659772700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557704e1a390 msgr2=0x5577051cbb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:00.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.110+0000 7f84f7e00640 1 --2- 172.21.15.32:0/659772700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557704e1a390 0x5577051cbb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.110+0000 7f84f7e00640 1 -- 172.21.15.32:0/659772700 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557705239090 con 0x557704df96b0 2024-04-30T08:37:00.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.110+0000 7f84fca00640 1 --2- 172.21.15.32:0/659772700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557704df14d0 0x5577051cb0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:37:00.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.110+0000 7f84fd400640 1 --2- 172.21.15.32:0/659772700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557704e1a390 0x5577051cbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:37:00.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.110+0000 7f84f7e00640 1 --2- 172.21.15.32:0/659772700 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557704df96b0 0x5577051cb630 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f84ec0a47a0 tx=0x7f84ec0a75a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91739d28b4d987f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:00.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.110+0000 7f84da000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:37:00.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.110+0000 7f84da000640 1 -- 172.21.15.32:0/659772700 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84ec0a7c30 con 0x557704df96b0 2024-04-30T08:37:00.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.110+0000 7f84da000640 1 -- 172.21.15.32:0/659772700 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f84ec0aff00 con 0x557704df96b0 2024-04-30T08:37:00.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.110+0000 7f8506565740 1 -- 172.21.15.32:0/659772700 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84ec0a7040 con 0x557704df96b0 2024-04-30T08:37:00.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.110+0000 7f84da000640 1 -- 172.21.15.32:0/659772700 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f84ec0b0390 con 0x557704df96b0 2024-04-30T08:37:00.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.110+0000 7f8506565740 10 client.26120 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:00.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.110+0000 7f8506565740 1 -- 172.21.15.32:0/659772700 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55770523a310 con 0x557704df96b0 2024-04-30T08:37:00.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.111+0000 7f84da000640 1 -- 172.21.15.32:0/659772700 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f84ec0a7040 con 0x557704df96b0 2024-04-30T08:37:00.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.111+0000 7f84da000640 1 -- 172.21.15.32:0/659772700 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7f84ec0c8270 con 0x557704df96b0 2024-04-30T08:37:00.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.111+0000 7f8506565740 10 client.26120 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:00.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.111+0000 7f8506565740 10 client.26120 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-30T08:37:00.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.111+0000 7f8506565740 20 client.26120 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:00.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.111+0000 7f8506565740 1 --2- 172.21.15.32:0/659772700 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x557704dae3a0 0x55770523cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.111+0000 7f8506565740 4 client.26120 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-30T08:37:00.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.111+0000 7f8506565740 1 -- 172.21.15.32:0/659772700 --> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5577051287d0 con 0x557704dae3a0 2024-04-30T08:37:00.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.111+0000 7f84fca00640 1 --2- 172.21.15.32:0/659772700 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x557704dae3a0 0x55770523cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.112+0000 7f84fca00640 1 --2- 172.21.15.32:0/659772700 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x557704dae3a0 0x55770523cd20 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4dd1a3b9456254e5 server_cookie=477cdcee0f26f6b9 in_seq=0 out_seq=0 2024-04-30T08:37:00.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.112+0000 7f84da000640 10 client.26120 ms_handle_connect on v2:172.21.15.71:6836/2324039932 2024-04-30T08:37:00.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.112+0000 7f84da000640 1 -- 172.21.15.32:0/659772700 <== mds.3 v2:172.21.15.71:6836/2324039932 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5577051287d0 con 0x557704dae3a0 2024-04-30T08:37:00.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.112+0000 7f84da000640 10 client.26120 handle_command_reply: tid=0 2024-04-30T08:37:00.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.112+0000 7f8506565740 1 client.26120 shutdown 2024-04-30T08:37:00.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.112+0000 7f8506565740 1 -- 172.21.15.32:0/659772700 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557704df96b0 msgr2=0x5577051cb630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:00.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.112+0000 7f8506565740 1 --2- 172.21.15.32:0/659772700 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557704df96b0 0x5577051cb630 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f84ec0a47a0 tx=0x7f84ec0a75a0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.112+0000 7f8506565740 1 -- 172.21.15.32:0/659772700 shutdown_connections 2024-04-30T08:37:00.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.112+0000 7f8506565740 1 --2- 172.21.15.32:0/659772700 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x557704dae3a0 0x55770523cd20 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.112+0000 7f8506565740 1 --2- 172.21.15.32:0/659772700 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557704e1a390 0x5577051cbb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.112+0000 7f8506565740 1 --2- 172.21.15.32:0/659772700 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557704df96b0 0x5577051cb630 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.112+0000 7f8506565740 1 --2- 172.21.15.32:0/659772700 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557704df14d0 0x5577051cb0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.112+0000 7f84da000640 0 client.26120 ms_handle_reset on v2:172.21.15.71:6836/2324039932 2024-04-30T08:37:00.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.112+0000 7f8506565740 1 -- 172.21.15.32:0/659772700 >> 172.21.15.32:0/659772700 conn(0x557704e0e4e0 msgr2=0x5577051b9da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:00.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.113+0000 7f8506565740 1 -- 172.21.15.32:0/659772700 shutdown_connections 2024-04-30T08:37:00.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.113+0000 7f8506565740 1 -- 172.21.15.32:0/659772700 wait complete. 2024-04-30T08:37:00.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.113+0000 7f8506565740 20 client.26120 trim_cache size 0 max 16384 2024-04-30T08:37:00.118 INFO:teuthology.orchestra.run.smithi032.stdout:[] 2024-04-30T08:37:00.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.113+0000 7f84ff400640 1 -- 172.21.15.32:0/1962000689 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f84e00afaf0 msgr2=0x7f84e00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:00.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.113+0000 7f84ff400640 1 --2- 172.21.15.32:0/1962000689 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f84e00afaf0 0x7f84e00b2000 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f84e802a9e0 tx=0x7f84e80314a0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.113+0000 7f84ff400640 1 -- 172.21.15.32:0/1962000689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84f8103cb0 msgr2=0x7f84f819ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:00.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.113+0000 7f84ff400640 1 --2- 172.21.15.32:0/1962000689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84f8103cb0 0x7f84f819ca90 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f84ec00e9f0 tx=0x7f84ec00edf0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.114+0000 7f84ff400640 1 -- 172.21.15.32:0/1962000689 shutdown_connections 2024-04-30T08:37:00.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.114+0000 7f84ff400640 1 --2- 172.21.15.32:0/1962000689 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f84e00afaf0 0x7f84e00b2000 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.114+0000 7f84ff400640 1 --2- 172.21.15.32:0/1962000689 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f84f81089b0 0x7f84f81a3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.114+0000 7f84ff400640 1 --2- 172.21.15.32:0/1962000689 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f84f8103cb0 0x7f84f819ca90 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.114+0000 7f84ff400640 1 --2- 172.21.15.32:0/1962000689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f84f8102a70 0x7f84f819c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.114+0000 7f84ff400640 1 -- 172.21.15.32:0/1962000689 >> 172.21.15.32:0/1962000689 conn(0x7f84f80fdc80 msgr2=0x7f84f80fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:00.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.114+0000 7f84ff400640 1 -- 172.21.15.32:0/1962000689 shutdown_connections 2024-04-30T08:37:00.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.114+0000 7f84ff400640 1 -- 172.21.15.32:0/1962000689 wait complete. 2024-04-30T08:37:00.136 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:00 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3683907221' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:37:00.138 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-30T08:37:00.227 INFO:tasks.workunit.client.0.smithi032.stdout:checking for srand48_r... yes 2024-04-30T08:37:00.275 INFO:tasks.workunit.client.0.smithi032.stdout:configure: creating ./config.status 2024-04-30T08:37:00.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:00 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3683907221' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:37:00.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.457+0000 7f668ca00640 1 -- 172.21.15.32:0/4206578983 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6688103990 msgr2=0x7f6688104690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:00.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.457+0000 7f668ca00640 1 --2- 172.21.15.32:0/4206578983 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6688103990 0x7f6688104690 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f6678009900 tx=0x7f667802a3f0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.457+0000 7f668ca00640 1 -- 172.21.15.32:0/4206578983 shutdown_connections 2024-04-30T08:37:00.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.457+0000 7f668ca00640 1 --2- 172.21.15.32:0/4206578983 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66881098d0 0x7f668810bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.457+0000 7f668ca00640 1 --2- 172.21.15.32:0/4206578983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6688104bd0 0x7f6688104fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.457+0000 7f668ca00640 1 --2- 172.21.15.32:0/4206578983 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6688103990 0x7f6688104690 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.457+0000 7f668ca00640 1 -- 172.21.15.32:0/4206578983 >> 172.21.15.32:0/4206578983 conn(0x7f66880feba0 msgr2=0x7f6688101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:00.458 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.457+0000 7f668ca00640 1 -- 172.21.15.32:0/4206578983 shutdown_connections 2024-04-30T08:37:00.459 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.457+0000 7f668ca00640 1 -- 172.21.15.32:0/4206578983 wait complete. 2024-04-30T08:37:00.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.458+0000 7f668ca00640 1 Processor -- start 2024-04-30T08:37:00.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.458+0000 7f668ca00640 1 -- start start 2024-04-30T08:37:00.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.459+0000 7f668ca00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6688103990 0x7f668819c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.460 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.459+0000 7f668ca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6688104bd0 0x7f668819c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.459+0000 7f668ca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66881098d0 0x7f66881a3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.459+0000 7f668ca00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f668810f470 con 0x7f66881098d0 2024-04-30T08:37:00.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.459+0000 7f668ca00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f668810f2f0 con 0x7f6688103990 2024-04-30T08:37:00.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.459+0000 7f668ca00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f668810f5f0 con 0x7f6688104bd0 2024-04-30T08:37:00.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.459+0000 7f6686800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6688103990 0x7f668819c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.459+0000 7f6687200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66881098d0 0x7f66881a3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.459+0000 7f6686800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6688103990 0x7f668819c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:35664/0 (socket says 172.21.15.32:35664) 2024-04-30T08:37:00.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.459+0000 7f6686800640 1 -- 172.21.15.32:0/499326410 learned_addr learned my addr 172.21.15.32:0/499326410 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:00.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.459+0000 7f667fe00640 1 --2- 172.21.15.32:0/499326410 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6688104bd0 0x7f668819c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.460+0000 7f6686800640 1 -- 172.21.15.32:0/499326410 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6688104bd0 msgr2=0x7f668819c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:00.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.460+0000 7f6686800640 1 --2- 172.21.15.32:0/499326410 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6688104bd0 0x7f668819c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.460+0000 7f6686800640 1 -- 172.21.15.32:0/499326410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66881098d0 msgr2=0x7f66881a3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:00.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.460+0000 7f6686800640 1 --2- 172.21.15.32:0/499326410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66881098d0 0x7f66881a3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.460+0000 7f6686800640 1 -- 172.21.15.32:0/499326410 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66881a3e00 con 0x7f6688103990 2024-04-30T08:37:00.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.460+0000 7f6687200640 1 --2- 172.21.15.32:0/499326410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66881098d0 0x7f66881a3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:37:00.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.460+0000 7f6686800640 1 --2- 172.21.15.32:0/499326410 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6688103990 0x7f668819c2c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f6678006010 tx=0x7f6678004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:00.462 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.460+0000 7f667f400640 1 -- 172.21.15.32:0/499326410 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f667803f030 con 0x7f6688103990 2024-04-30T08:37:00.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.460+0000 7f667f400640 1 -- 172.21.15.32:0/499326410 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6678003aa0 con 0x7f6688103990 2024-04-30T08:37:00.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.460+0000 7f667f400640 1 -- 172.21.15.32:0/499326410 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6678035070 con 0x7f6688103990 2024-04-30T08:37:00.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.461+0000 7f668ca00640 1 -- 172.21.15.32:0/499326410 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66881a4090 con 0x7f6688103990 2024-04-30T08:37:00.463 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.461+0000 7f668ca00640 1 -- 172.21.15.32:0/499326410 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66881a4530 con 0x7f6688103990 2024-04-30T08:37:00.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.464+0000 7f6694406740 1 Processor -- start 2024-04-30T08:37:00.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.464+0000 7f6694406740 1 -- start start 2024-04-30T08:37:00.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.464+0000 7f6694406740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f1b984390 0x555f1bd25730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.465 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.464+0000 7f6694406740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f1b95b4d0 0x555f1bd25cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.464+0000 7f6694406740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555f1b9636b0 0x555f1bd2cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.464+0000 7f6694406740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555f1bc8ad50 con 0x555f1b95b4d0 2024-04-30T08:37:00.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.464+0000 7f6694406740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x555f1bc300c0 con 0x555f1b9636b0 2024-04-30T08:37:00.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.464+0000 7f6694406740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555f1bc86d10 con 0x555f1b984390 2024-04-30T08:37:00.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.465+0000 7f667f400640 1 -- 172.21.15.32:0/499326410 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f6678042030 con 0x7f6688103990 2024-04-30T08:37:00.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.465+0000 7f6687200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f1b984390 0x555f1bd25730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.465+0000 7f667fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555f1b9636b0 0x555f1bd2cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.465+0000 7f6687200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f1b984390 0x555f1bd25730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:36934/0 (socket says 172.21.15.32:36934) 2024-04-30T08:37:00.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.465+0000 7f6687200640 1 -- 172.21.15.32:0/2630515075 learned_addr learned my addr 172.21.15.32:0/2630515075 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:00.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.465+0000 7f6686800640 1 --2- 172.21.15.32:0/2630515075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f1b95b4d0 0x555f1bd25cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.465+0000 7f667fe00640 1 -- 172.21.15.32:0/2630515075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f1b984390 msgr2=0x555f1bd25730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:00.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.465+0000 7f667fe00640 1 --2- 172.21.15.32:0/2630515075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f1b984390 0x555f1bd25730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.465+0000 7f667fe00640 1 -- 172.21.15.32:0/2630515075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f1b95b4d0 msgr2=0x555f1bd25cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:00.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.465+0000 7f667fe00640 1 --2- 172.21.15.32:0/2630515075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f1b95b4d0 0x555f1bd25cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.465+0000 7f667fe00640 1 -- 172.21.15.32:0/2630515075 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f1b99c3f0 con 0x555f1b9636b0 2024-04-30T08:37:00.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.465+0000 7f6687200640 1 --2- 172.21.15.32:0/2630515075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f1b984390 0x555f1bd25730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:37:00.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.465+0000 7f6686800640 1 --2- 172.21.15.32:0/2630515075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f1b95b4d0 0x555f1bd25cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:37:00.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.466+0000 7f667fe00640 1 --2- 172.21.15.32:0/2630515075 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555f1b9636b0 0x555f1bd2cbb0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6670009bf0 tx=0x7f6670011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f59abca02d4475f6 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:00.469 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.466+0000 7f6677400640 1 -- 172.21.15.32:0/2630515075 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6670004050 con 0x555f1b9636b0 2024-04-30T08:37:00.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.466+0000 7f6677400640 1 -- 172.21.15.32:0/2630515075 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66700054d0 con 0x555f1b9636b0 2024-04-30T08:37:00.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.466+0000 7f6677400640 1 -- 172.21.15.32:0/2630515075 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6670004d10 con 0x555f1b9636b0 2024-04-30T08:37:00.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.467+0000 7f667f400640 1 --2- 172.21.15.32:0/499326410 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f666c0afb60 0x7f666c0b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.467+0000 7f667f400640 1 -- 172.21.15.32:0/499326410 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f66780ff580 con 0x7f6688103990 2024-04-30T08:37:00.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.467+0000 7f6694406740 1 -- 172.21.15.32:0/2630515075 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555f1b9636b0 msgr2=0x555f1bd2cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:00.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.467+0000 7f6694406740 1 --2- 172.21.15.32:0/2630515075 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555f1b9636b0 0x555f1bd2cbb0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f6670009bf0 tx=0x7f6670011810 comp rx=0 tx=0).stop 2024-04-30T08:37:00.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.467+0000 7f6687200640 1 --2- 172.21.15.32:0/499326410 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f666c0afb60 0x7f666c0b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.467+0000 7f6694406740 1 -- 172.21.15.32:0/2630515075 shutdown_connections 2024-04-30T08:37:00.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.467+0000 7f6694406740 1 --2- 172.21.15.32:0/2630515075 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555f1b9636b0 0x555f1bd2cbb0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.467+0000 7f6694406740 1 --2- 172.21.15.32:0/2630515075 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f1b95b4d0 0x555f1bd25cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.467+0000 7f6694406740 1 --2- 172.21.15.32:0/2630515075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f1b984390 0x555f1bd25730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.470 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.467+0000 7f6694406740 1 -- 172.21.15.32:0/2630515075 >> 172.21.15.32:0/2630515075 conn(0x555f1b9784e0 msgr2=0x555f1bd210c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:00.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.467+0000 7f6694406740 1 -- 172.21.15.32:0/2630515075 shutdown_connections 2024-04-30T08:37:00.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.467+0000 7f6694406740 1 -- 172.21.15.32:0/2630515075 wait complete. 2024-04-30T08:37:00.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.468+0000 7f6687200640 1 --2- 172.21.15.32:0/499326410 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f666c0afb60 0x7f666c0b2070 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6680010510 tx=0x7f6680009600 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:00.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.468+0000 7f6694406740 1 -- start start 2024-04-30T08:37:00.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.468+0000 7f6694406740 1 Processor -- start 2024-04-30T08:37:00.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.469+0000 7f6694406740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f1b95b4d0 0x555f1bd2f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.469+0000 7f6694406740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f1b9636b0 0x555f1bd2fb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.469+0000 7f6694406740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555f1b984390 0x555f1bd36af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.469+0000 7f6694406740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x555f1bd2d600 con 0x555f1b9636b0 2024-04-30T08:37:00.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.469+0000 7f6694406740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x555f1bc43b50 con 0x555f1b984390 2024-04-30T08:37:00.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.469+0000 7f6694406740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x555f1bd2d780 con 0x555f1b95b4d0 2024-04-30T08:37:00.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.469+0000 7f6694406740 10 client.0 _ll_register_callbacks cb 0x555f1bc0d420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:00.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.469+0000 7f667fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f1b9636b0 0x555f1bd2fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.469+0000 7f6686800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f1b95b4d0 0x555f1bd2f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.469+0000 7f667fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f1b9636b0 0x555f1bd2fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56494/0 (socket says 172.21.15.32:56494) 2024-04-30T08:37:00.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.469+0000 7f667fe00640 1 -- 172.21.15.32:0/3258701282 learned_addr learned my addr 172.21.15.32:0/3258701282 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:00.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.469+0000 7f6687200640 1 --2- 172.21.15.32:0/3258701282 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555f1b984390 0x555f1bd36af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.470+0000 7f6686800640 1 -- 172.21.15.32:0/3258701282 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555f1b984390 msgr2=0x555f1bd36af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:00.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.470+0000 7f6686800640 1 --2- 172.21.15.32:0/3258701282 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555f1b984390 0x555f1bd36af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.470+0000 7f6686800640 1 -- 172.21.15.32:0/3258701282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f1b9636b0 msgr2=0x555f1bd2fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:00.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.470+0000 7f6686800640 1 --2- 172.21.15.32:0/3258701282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f1b9636b0 0x555f1bd2fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.470+0000 7f6686800640 1 -- 172.21.15.32:0/3258701282 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f1ba667d0 con 0x555f1b95b4d0 2024-04-30T08:37:00.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.470+0000 7f6686800640 1 --2- 172.21.15.32:0/3258701282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f1b95b4d0 0x555f1bd2f650 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x555f1bd26b10 tx=0x7f6678003f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85413bd6667c2216 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:00.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.470+0000 7f666be00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-30T08:37:00.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.470+0000 7f666be00640 1 -- 172.21.15.32:0/3258701282 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6678049070 con 0x555f1b95b4d0 2024-04-30T08:37:00.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.470+0000 7f666be00640 1 -- 172.21.15.32:0/3258701282 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f6678003c50 con 0x555f1b95b4d0 2024-04-30T08:37:00.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.470+0000 7f6694406740 1 -- 172.21.15.32:0/3258701282 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66700054d0 con 0x555f1b95b4d0 2024-04-30T08:37:00.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.471+0000 7f666be00640 1 -- 172.21.15.32:0/3258701282 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f66780c7050 con 0x555f1b95b4d0 2024-04-30T08:37:00.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.471+0000 7f6694406740 10 client.26131 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:00.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.471+0000 7f6694406740 1 -- 172.21.15.32:0/3258701282 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555f1bd2d670 con 0x555f1b95b4d0 2024-04-30T08:37:00.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.471+0000 7f666be00640 1 -- 172.21.15.32:0/3258701282 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66700054d0 con 0x555f1b95b4d0 2024-04-30T08:37:00.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.471+0000 7f666be00640 1 -- 172.21.15.32:0/3258701282 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7f66780ff230 con 0x555f1b95b4d0 2024-04-30T08:37:00.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.471+0000 7f6694406740 10 client.26131 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:00.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.471+0000 7f6694406740 10 client.26131 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-30T08:37:00.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.471+0000 7f6694406740 20 client.26131 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:00.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.471+0000 7f6694406740 1 --2- 172.21.15.32:0/3258701282 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x555f1b942f60 0x555f1bd3d1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.471+0000 7f6694406740 4 client.26131 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:37:00.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.471+0000 7f6694406740 1 -- 172.21.15.32:0/3258701282 --> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555f1bc98960 con 0x555f1b942f60 2024-04-30T08:37:00.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.472+0000 7f667fe00640 1 --2- 172.21.15.32:0/3258701282 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x555f1b942f60 0x555f1bd3d1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.472+0000 7f667fe00640 1 --2- 172.21.15.32:0/3258701282 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x555f1b942f60 0x555f1bd3d1e0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f465e7cccb16b516 server_cookie=67b51ae2a71d5f98 in_seq=0 out_seq=0 2024-04-30T08:37:00.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.472+0000 7f666be00640 10 client.26131 ms_handle_connect on v2:172.21.15.189:6838/352970107 2024-04-30T08:37:00.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.473+0000 7f666be00640 1 -- 172.21.15.32:0/3258701282 <== mds.2 v2:172.21.15.189:6838/352970107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x555f1bc98960 con 0x555f1b942f60 2024-04-30T08:37:00.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.473+0000 7f666be00640 10 client.26131 handle_command_reply: tid=0 2024-04-30T08:37:00.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.473+0000 7f6694406740 1 client.26131 shutdown 2024-04-30T08:37:00.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.473+0000 7f6694406740 1 -- 172.21.15.32:0/3258701282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f1b95b4d0 msgr2=0x555f1bd2f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:00.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.473+0000 7f6694406740 1 --2- 172.21.15.32:0/3258701282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f1b95b4d0 0x555f1bd2f650 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x555f1bd26b10 tx=0x7f6678003f90 comp rx=0 tx=0).stop 2024-04-30T08:37:00.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.474+0000 7f6694406740 1 -- 172.21.15.32:0/3258701282 shutdown_connections 2024-04-30T08:37:00.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.474+0000 7f6694406740 1 --2- 172.21.15.32:0/3258701282 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x555f1b942f60 0x555f1bd3d1e0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.474+0000 7f6694406740 1 --2- 172.21.15.32:0/3258701282 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555f1b984390 0x555f1bd36af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.474+0000 7f6694406740 1 --2- 172.21.15.32:0/3258701282 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f1b9636b0 0x555f1bd2fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.474+0000 7f6694406740 1 --2- 172.21.15.32:0/3258701282 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f1b95b4d0 0x555f1bd2f650 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.474+0000 7f666be00640 0 client.26131 ms_handle_reset on v2:172.21.15.189:6838/352970107 2024-04-30T08:37:00.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.474+0000 7f6694406740 1 -- 172.21.15.32:0/3258701282 >> 172.21.15.32:0/3258701282 conn(0x555f1b9784e0 msgr2=0x555f1bd20100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:00.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.474+0000 7f6694406740 1 -- 172.21.15.32:0/3258701282 shutdown_connections 2024-04-30T08:37:00.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.474+0000 7f6694406740 1 -- 172.21.15.32:0/3258701282 wait complete. 2024-04-30T08:37:00.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.474+0000 7f6694406740 20 client.26131 trim_cache size 0 max 16384 2024-04-30T08:37:00.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.490+0000 7f6694406740 1 Processor -- start 2024-04-30T08:37:00.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.490+0000 7f6694406740 1 -- start start 2024-04-30T08:37:00.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.490+0000 7f6694406740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555f1b95b4d0 0x555f1bd2f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.490+0000 7f6694406740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f1b9636b0 0x555f1bd36a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.490+0000 7f6694406740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f1b984390 0x555f1bd36fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.490+0000 7f6694406740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f66881aebc0 con 0x555f1b984390 2024-04-30T08:37:00.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.490+0000 7f6694406740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f66881b0790 con 0x555f1b95b4d0 2024-04-30T08:37:00.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.490+0000 7f6694406740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f66881af290 con 0x555f1b9636b0 2024-04-30T08:37:00.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.490+0000 7f6686800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555f1b95b4d0 0x555f1bd2f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.490+0000 7f6686800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555f1b95b4d0 0x555f1bd2f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:35682/0 (socket says 172.21.15.32:35682) 2024-04-30T08:37:00.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.490+0000 7f6686800640 1 -- 172.21.15.32:0/3841629210 learned_addr learned my addr 172.21.15.32:0/3841629210 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:00.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.490+0000 7f6687200640 1 --2- 172.21.15.32:0/3841629210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f1b984390 0x555f1bd36fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.490+0000 7f667fe00640 1 --2- 172.21.15.32:0/3841629210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f1b9636b0 0x555f1bd36a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.491+0000 7f6686800640 1 -- 172.21.15.32:0/3841629210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f1b9636b0 msgr2=0x555f1bd36a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:00.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.491+0000 7f6686800640 1 --2- 172.21.15.32:0/3841629210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f1b9636b0 0x555f1bd36a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.491+0000 7f6686800640 1 -- 172.21.15.32:0/3841629210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f1b984390 msgr2=0x555f1bd36fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:00.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.491+0000 7f6686800640 1 --2- 172.21.15.32:0/3841629210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f1b984390 0x555f1bd36fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.491+0000 7f6686800640 1 -- 172.21.15.32:0/3841629210 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555f1bd1c8e0 con 0x555f1b95b4d0 2024-04-30T08:37:00.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.491+0000 7f6686800640 1 --2- 172.21.15.32:0/3841629210 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555f1b95b4d0 0x555f1bd2f930 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f667803d040 tx=0x7f6678009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=763460d278b175d7 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:00.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.491+0000 7f6676a00640 1 -- 172.21.15.32:0/3841629210 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6678049070 con 0x555f1b95b4d0 2024-04-30T08:37:00.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.491+0000 7f6676a00640 1 -- 172.21.15.32:0/3841629210 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66780c7210 con 0x555f1b95b4d0 2024-04-30T08:37:00.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.491+0000 7f6676a00640 1 -- 172.21.15.32:0/3841629210 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66780c6e70 con 0x555f1b95b4d0 2024-04-30T08:37:00.494 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.492+0000 7f6694406740 1 -- 172.21.15.32:0/3841629210 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555f1b95b4d0 msgr2=0x555f1bd2f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:00.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.492+0000 7f6694406740 1 --2- 172.21.15.32:0/3841629210 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555f1b95b4d0 0x555f1bd2f930 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f667803d040 tx=0x7f6678009560 comp rx=0 tx=0).stop 2024-04-30T08:37:00.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.492+0000 7f6694406740 1 -- 172.21.15.32:0/3841629210 shutdown_connections 2024-04-30T08:37:00.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.492+0000 7f6694406740 1 --2- 172.21.15.32:0/3841629210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f1b984390 0x555f1bd36fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.492+0000 7f6694406740 1 --2- 172.21.15.32:0/3841629210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f1b9636b0 0x555f1bd36a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.492+0000 7f6694406740 1 --2- 172.21.15.32:0/3841629210 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555f1b95b4d0 0x555f1bd2f930 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.492+0000 7f6694406740 1 -- 172.21.15.32:0/3841629210 >> 172.21.15.32:0/3841629210 conn(0x555f1b9784e0 msgr2=0x555f1bd2f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:00.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.492+0000 7f6694406740 1 -- 172.21.15.32:0/3841629210 shutdown_connections 2024-04-30T08:37:00.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.492+0000 7f6694406740 1 -- 172.21.15.32:0/3841629210 wait complete. 2024-04-30T08:37:00.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.493+0000 7f6694406740 1 -- start start 2024-04-30T08:37:00.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.493+0000 7f6694406740 1 Processor -- start 2024-04-30T08:37:00.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.494+0000 7f6694406740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f1b95b4d0 0x555f1bd343c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.494+0000 7f6694406740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f1b9636b0 0x555f1bd34900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.494+0000 7f6694406740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555f1b984390 0x555f1bd35830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.494+0000 7f6694406740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f66881b0030 con 0x555f1b95b4d0 2024-04-30T08:37:00.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.494+0000 7f6694406740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f66881af960 con 0x555f1b984390 2024-04-30T08:37:00.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.494+0000 7f6694406740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f66881acb80 con 0x555f1b9636b0 2024-04-30T08:37:00.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.494+0000 7f6694406740 10 client.0 _ll_register_callbacks cb 0x7f6678005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:00.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.494+0000 7f6686800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f1b95b4d0 0x555f1bd343c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.494+0000 7f6687200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555f1b984390 0x555f1bd35830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.494+0000 7f667fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f1b9636b0 0x555f1bd34900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.494+0000 7f6687200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555f1b984390 0x555f1bd35830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:35692/0 (socket says 172.21.15.32:35692) 2024-04-30T08:37:00.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.494+0000 7f6687200640 1 -- 172.21.15.32:0/1951283308 learned_addr learned my addr 172.21.15.32:0/1951283308 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:00.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.495+0000 7f667fe00640 1 -- 172.21.15.32:0/1951283308 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555f1b984390 msgr2=0x555f1bd35830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:00.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.495+0000 7f667fe00640 1 --2- 172.21.15.32:0/1951283308 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555f1b984390 0x555f1bd35830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.495+0000 7f667fe00640 1 -- 172.21.15.32:0/1951283308 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f1b95b4d0 msgr2=0x555f1bd343c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:00.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.495+0000 7f667fe00640 1 --2- 172.21.15.32:0/1951283308 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f1b95b4d0 0x555f1bd343c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.495+0000 7f667fe00640 1 -- 172.21.15.32:0/1951283308 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555f1bd35df0 con 0x555f1b9636b0 2024-04-30T08:37:00.499 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.495+0000 7f6687200640 1 --2- 172.21.15.32:0/1951283308 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555f1b984390 0x555f1bd35830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:37:00.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.495+0000 7f667fe00640 1 --2- 172.21.15.32:0/1951283308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f1b9636b0 0x555f1bd34900 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x555f1bd3d980 tx=0x7f66700097a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d962d8a26b8d29a server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:00.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.495+0000 7f666be00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-30T08:37:00.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.495+0000 7f666be00640 1 -- 172.21.15.32:0/1951283308 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f667002e3b0 con 0x555f1b9636b0 2024-04-30T08:37:00.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.495+0000 7f666be00640 1 -- 172.21.15.32:0/1951283308 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f667002e550 con 0x555f1b9636b0 2024-04-30T08:37:00.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.495+0000 7f6694406740 1 -- 172.21.15.32:0/1951283308 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f66780c7210 con 0x555f1b9636b0 2024-04-30T08:37:00.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.496+0000 7f666be00640 1 -- 172.21.15.32:0/1951283308 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f667002e750 con 0x555f1b9636b0 2024-04-30T08:37:00.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.496+0000 7f6694406740 10 client.26137 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:00.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.496+0000 7f6694406740 1 -- 172.21.15.32:0/1951283308 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x555f1bd37070 con 0x555f1b9636b0 2024-04-30T08:37:00.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.496+0000 7f666be00640 1 -- 172.21.15.32:0/1951283308 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f66780c7210 con 0x555f1b9636b0 2024-04-30T08:37:00.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.496+0000 7f666be00640 1 -- 172.21.15.32:0/1951283308 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7f6670019430 con 0x555f1b9636b0 2024-04-30T08:37:00.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.496+0000 7f6694406740 10 client.26137 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:00.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.496+0000 7f6694406740 10 client.26137 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-30T08:37:00.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.496+0000 7f6694406740 20 client.26137 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:00.500 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.496+0000 7f6694406740 1 --2- 172.21.15.32:0/1951283308 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x555f1b9183a0 0x555f1bdafdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.496+0000 7f6694406740 4 client.26137 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-30T08:37:00.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.496+0000 7f6694406740 1 -- 172.21.15.32:0/1951283308 --> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x555f1bc927d0 con 0x555f1b9183a0 2024-04-30T08:37:00.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.496+0000 7f6686800640 1 --2- 172.21.15.32:0/1951283308 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x555f1b9183a0 0x555f1bdafdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.497+0000 7f6686800640 1 --2- 172.21.15.32:0/1951283308 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x555f1b9183a0 0x555f1bdafdb0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=dc27444e3b515c81 server_cookie=3ea54d3f3250966a in_seq=0 out_seq=0 2024-04-30T08:37:00.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.497+0000 7f666be00640 10 client.26137 ms_handle_connect on v2:172.21.15.189:6838/352970107 2024-04-30T08:37:00.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.497+0000 7f666be00640 1 -- 172.21.15.32:0/1951283308 <== mds.2 v2:172.21.15.189:6838/352970107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x555f1bc927d0 con 0x555f1b9183a0 2024-04-30T08:37:00.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.497+0000 7f666be00640 10 client.26137 handle_command_reply: tid=0 2024-04-30T08:37:00.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.497+0000 7f6694406740 1 client.26137 shutdown 2024-04-30T08:37:00.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.497+0000 7f6694406740 1 -- 172.21.15.32:0/1951283308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f1b9636b0 msgr2=0x555f1bd34900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:00.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.497+0000 7f6694406740 1 --2- 172.21.15.32:0/1951283308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f1b9636b0 0x555f1bd34900 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x555f1bd3d980 tx=0x7f66700097a0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.498+0000 7f6694406740 1 -- 172.21.15.32:0/1951283308 shutdown_connections 2024-04-30T08:37:00.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.498+0000 7f6694406740 1 --2- 172.21.15.32:0/1951283308 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x555f1b9183a0 0x555f1bdafdb0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.498+0000 7f6694406740 1 --2- 172.21.15.32:0/1951283308 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x555f1b984390 0x555f1bd35830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.501 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.498+0000 7f6694406740 1 --2- 172.21.15.32:0/1951283308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f1b9636b0 0x555f1bd34900 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.498+0000 7f666be00640 0 client.26137 ms_handle_reset on v2:172.21.15.189:6838/352970107 2024-04-30T08:37:00.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.498+0000 7f6694406740 1 --2- 172.21.15.32:0/1951283308 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x555f1b95b4d0 0x555f1bd343c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.498+0000 7f6694406740 1 -- 172.21.15.32:0/1951283308 >> 172.21.15.32:0/1951283308 conn(0x555f1b9784e0 msgr2=0x555f1bd1f6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:00.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.498+0000 7f6694406740 1 -- 172.21.15.32:0/1951283308 shutdown_connections 2024-04-30T08:37:00.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.498+0000 7f6694406740 1 -- 172.21.15.32:0/1951283308 wait complete. 2024-04-30T08:37:00.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.498+0000 7f6694406740 20 client.26137 trim_cache size 0 max 16384 2024-04-30T08:37:00.502 INFO:teuthology.orchestra.run.smithi032.stdout:[] 2024-04-30T08:37:00.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.499+0000 7f668ca00640 1 -- 172.21.15.32:0/499326410 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f666c0afb60 msgr2=0x7f666c0b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:00.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.499+0000 7f668ca00640 1 --2- 172.21.15.32:0/499326410 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f666c0afb60 0x7f666c0b2070 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f6680010510 tx=0x7f6680009600 comp rx=0 tx=0).stop 2024-04-30T08:37:00.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.499+0000 7f668ca00640 1 -- 172.21.15.32:0/499326410 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6688103990 msgr2=0x7f668819c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:00.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.499+0000 7f668ca00640 1 --2- 172.21.15.32:0/499326410 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6688103990 0x7f668819c2c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f6678006010 tx=0x7f6678004270 comp rx=0 tx=0).stop 2024-04-30T08:37:00.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.499+0000 7f668ca00640 1 -- 172.21.15.32:0/499326410 shutdown_connections 2024-04-30T08:37:00.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.499+0000 7f668ca00640 1 --2- 172.21.15.32:0/499326410 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f666c0afb60 0x7f666c0b2070 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.499+0000 7f668ca00640 1 --2- 172.21.15.32:0/499326410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f66881098d0 0x7f66881a3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.499+0000 7f668ca00640 1 --2- 172.21.15.32:0/499326410 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6688104bd0 0x7f668819c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.499+0000 7f668ca00640 1 --2- 172.21.15.32:0/499326410 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6688103990 0x7f668819c2c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.499+0000 7f668ca00640 1 -- 172.21.15.32:0/499326410 >> 172.21.15.32:0/499326410 conn(0x7f66880feba0 msgr2=0x7f6688100fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:00.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.499+0000 7f668ca00640 1 -- 172.21.15.32:0/499326410 shutdown_connections 2024-04-30T08:37:00.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.499+0000 7f668ca00640 1 -- 172.21.15.32:0/499326410 wait complete. 2024-04-30T08:37:00.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:00 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3683907221' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:37:00.524 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-30T08:37:00.540 INFO:tasks.workunit.client.0.smithi032.stdout:config.status: creating Makefile 2024-04-30T08:37:00.577 INFO:tasks.workunit.client.0.smithi032.stdout:config.status: creating config.h 2024-04-30T08:37:00.620 INFO:tasks.workunit.client.0.smithi032.stdout:config.status: executing depfiles commands 2024-04-30T08:37:00.750 INFO:tasks.workunit.client.0.smithi032.stdout:config.status: executing default commands 2024-04-30T08:37:00.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.809+0000 7fab3f200640 1 -- 172.21.15.32:0/435807750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab4010b900 msgr2=0x7fab4010dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:00.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.809+0000 7fab3f200640 1 --2- 172.21.15.32:0/435807750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab4010b900 0x7fab4010dd40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fab38009be0 tx=0x7fab3802a6c0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.809+0000 7fab3f200640 1 -- 172.21.15.32:0/435807750 shutdown_connections 2024-04-30T08:37:00.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.809+0000 7fab3f200640 1 --2- 172.21.15.32:0/435807750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab4010b900 0x7fab4010dd40 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.809+0000 7fab3f200640 1 --2- 172.21.15.32:0/435807750 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab40075960 0x7fab4010b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.809+0000 7fab3f200640 1 --2- 172.21.15.32:0/435807750 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab40075010 0x7fab40075420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.809+0000 7fab3f200640 1 -- 172.21.15.32:0/435807750 >> 172.21.15.32:0/435807750 conn(0x7fab400feba0 msgr2=0x7fab40101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:00.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.809+0000 7fab3f200640 1 -- 172.21.15.32:0/435807750 shutdown_connections 2024-04-30T08:37:00.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.809+0000 7fab3f200640 1 -- 172.21.15.32:0/435807750 wait complete. 2024-04-30T08:37:00.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.810+0000 7fab3f200640 1 Processor -- start 2024-04-30T08:37:00.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.811+0000 7fab3f200640 1 -- start start 2024-04-30T08:37:00.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.811+0000 7fab3f200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab40075010 0x7fab4019c280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.811+0000 7fab3f200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab40075960 0x7fab4019c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.811+0000 7fab3f200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab4010b900 0x7fab401a3750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.811+0000 7fab3f200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fab401116a0 con 0x7fab40075960 2024-04-30T08:37:00.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.811+0000 7fab3f200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fab40111520 con 0x7fab4010b900 2024-04-30T08:37:00.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.811+0000 7fab3f200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fab40111820 con 0x7fab40075010 2024-04-30T08:37:00.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.811+0000 7fab3de00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab40075010 0x7fab4019c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.811+0000 7fab3e800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab4010b900 0x7fab401a3750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.811+0000 7fab3de00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab40075010 0x7fab4019c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:36988/0 (socket says 172.21.15.32:36988) 2024-04-30T08:37:00.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.811+0000 7fab3d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab40075960 0x7fab4019c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.811+0000 7fab3de00640 1 -- 172.21.15.32:0/90589339 learned_addr learned my addr 172.21.15.32:0/90589339 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:00.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.812+0000 7fab3de00640 1 -- 172.21.15.32:0/90589339 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab4010b900 msgr2=0x7fab401a3750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:00.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.812+0000 7fab3de00640 1 --2- 172.21.15.32:0/90589339 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab4010b900 0x7fab401a3750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.812+0000 7fab3de00640 1 -- 172.21.15.32:0/90589339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab40075960 msgr2=0x7fab4019c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:00.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.812+0000 7fab3de00640 1 --2- 172.21.15.32:0/90589339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab40075960 0x7fab4019c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.812+0000 7fab3de00640 1 -- 172.21.15.32:0/90589339 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab401a3dc0 con 0x7fab40075010 2024-04-30T08:37:00.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.812+0000 7fab3de00640 1 --2- 172.21.15.32:0/90589339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab40075010 0x7fab4019c280 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fab3000e9f0 tx=0x7fab3000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:00.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.813+0000 7fab36a00640 1 -- 172.21.15.32:0/90589339 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab3000ccc0 con 0x7fab40075010 2024-04-30T08:37:00.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.813+0000 7fab36a00640 1 -- 172.21.15.32:0/90589339 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab300108e0 con 0x7fab40075010 2024-04-30T08:37:00.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.813+0000 7fab3f200640 1 -- 172.21.15.32:0/90589339 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab401a40b0 con 0x7fab40075010 2024-04-30T08:37:00.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.813+0000 7fab36a00640 1 -- 172.21.15.32:0/90589339 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab30018880 con 0x7fab40075010 2024-04-30T08:37:00.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.813+0000 7fab3f200640 1 -- 172.21.15.32:0/90589339 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab401a4650 con 0x7fab40075010 2024-04-30T08:37:00.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.817+0000 7fab4bffd740 1 Processor -- start 2024-04-30T08:37:00.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.818+0000 7fab4bffd740 1 -- start start 2024-04-30T08:37:00.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.818+0000 7fab4bffd740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563cc8887390 0x563cc8c28730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.818+0000 7fab4bffd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc885e4d0 0x563cc8c28cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.818+0000 7fab4bffd740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cc88666b0 0x563cc8c2fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.818+0000 7fab4bffd740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x563cc8b8dd50 con 0x563cc88666b0 2024-04-30T08:37:00.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.818+0000 7fab4bffd740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x563cc8b330c0 con 0x563cc8887390 2024-04-30T08:37:00.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.818+0000 7fab4bffd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563cc8b89d10 con 0x563cc885e4d0 2024-04-30T08:37:00.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.818+0000 7fab3e800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563cc8887390 0x563cc8c28730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.818+0000 7fab3e800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563cc8887390 0x563cc8c28730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:35722/0 (socket says 172.21.15.32:35722) 2024-04-30T08:37:00.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.818+0000 7fab3e800640 1 -- 172.21.15.32:0/2433642527 learned_addr learned my addr 172.21.15.32:0/2433642527 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:00.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.818+0000 7fab3d400640 1 --2- 172.21.15.32:0/2433642527 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cc88666b0 0x563cc8c2fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.819+0000 7fab3e800640 1 -- 172.21.15.32:0/2433642527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc885e4d0 msgr2=0x563cc8c28cb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-30T08:37:00.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.819+0000 7fab3e800640 1 --2- 172.21.15.32:0/2433642527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc885e4d0 0x563cc8c28cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.819+0000 7fab3e800640 1 -- 172.21.15.32:0/2433642527 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cc88666b0 msgr2=0x563cc8c2fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:00.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.819+0000 7fab3e800640 1 --2- 172.21.15.32:0/2433642527 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cc88666b0 0x563cc8c2fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.819+0000 7fab3e800640 1 -- 172.21.15.32:0/2433642527 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563cc889f3f0 con 0x563cc8887390 2024-04-30T08:37:00.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.819+0000 7fab36a00640 1 -- 172.21.15.32:0/90589339 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7fab30010430 con 0x7fab40075010 2024-04-30T08:37:00.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.819+0000 7fab3e800640 1 --2- 172.21.15.32:0/2433642527 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563cc8887390 0x563cc8c28730 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fab40075ff0 tx=0x7fab38005c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=670ad1ef528fae06 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:00.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.820+0000 7fab2b400640 1 -- 172.21.15.32:0/2433642527 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab38041030 con 0x563cc8887390 2024-04-30T08:37:00.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.820+0000 7fab2b400640 1 -- 172.21.15.32:0/2433642527 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab38030560 con 0x563cc8887390 2024-04-30T08:37:00.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.820+0000 7fab2b400640 1 -- 172.21.15.32:0/2433642527 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab38036040 con 0x563cc8887390 2024-04-30T08:37:00.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.820+0000 7fab4bffd740 1 -- 172.21.15.32:0/2433642527 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563cc8887390 msgr2=0x563cc8c28730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:00.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.820+0000 7fab4bffd740 1 --2- 172.21.15.32:0/2433642527 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563cc8887390 0x563cc8c28730 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fab40075ff0 tx=0x7fab38005c80 comp rx=0 tx=0).stop 2024-04-30T08:37:00.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.821+0000 7fab4bffd740 1 -- 172.21.15.32:0/2433642527 shutdown_connections 2024-04-30T08:37:00.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.821+0000 7fab4bffd740 1 --2- 172.21.15.32:0/2433642527 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cc88666b0 0x563cc8c2fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.821+0000 7fab4bffd740 1 --2- 172.21.15.32:0/2433642527 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc885e4d0 0x563cc8c28cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.821+0000 7fab4bffd740 1 --2- 172.21.15.32:0/2433642527 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563cc8887390 0x563cc8c28730 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.821+0000 7fab4bffd740 1 -- 172.21.15.32:0/2433642527 >> 172.21.15.32:0/2433642527 conn(0x563cc887b4e0 msgr2=0x563cc8c240c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:00.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.821+0000 7fab36a00640 1 --2- 172.21.15.32:0/90589339 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fab240afb60 0x7fab240b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.821+0000 7fab4bffd740 1 -- 172.21.15.32:0/2433642527 shutdown_connections 2024-04-30T08:37:00.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.821+0000 7fab4bffd740 1 -- 172.21.15.32:0/2433642527 wait complete. 2024-04-30T08:37:00.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.821+0000 7fab36a00640 1 -- 172.21.15.32:0/90589339 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fab300df1d0 con 0x7fab40075010 2024-04-30T08:37:00.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.821+0000 7fab3d400640 1 --2- 172.21.15.32:0/90589339 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fab240afb60 0x7fab240b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.822+0000 7fab4bffd740 1 -- start start 2024-04-30T08:37:00.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.822+0000 7fab4bffd740 1 Processor -- start 2024-04-30T08:37:00.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.822+0000 7fab3d400640 1 --2- 172.21.15.32:0/90589339 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fab240afb60 0x7fab240b2070 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x563cc8c24ef0 tx=0x7fab2c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:00.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.822+0000 7fab4bffd740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cc885e4d0 0x563cc8c32650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.822+0000 7fab4bffd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc88666b0 0x563cc8c32b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.822+0000 7fab4bffd740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563cc8887390 0x563cc8c39af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.822+0000 7fab4bffd740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x563cc8c305a0 con 0x563cc885e4d0 2024-04-30T08:37:00.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.822+0000 7fab4bffd740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x563cc8b46b50 con 0x563cc8887390 2024-04-30T08:37:00.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.822+0000 7fab4bffd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x563cc8c30720 con 0x563cc88666b0 2024-04-30T08:37:00.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.822+0000 7fab4bffd740 10 client.0 _ll_register_callbacks cb 0x563cc8b10420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:00.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.823+0000 7fab3de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cc885e4d0 0x563cc8c32650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.823+0000 7fab3e800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563cc8887390 0x563cc8c39af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.823+0000 7fab3de00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cc885e4d0 0x563cc8c32650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56544/0 (socket says 172.21.15.32:56544) 2024-04-30T08:37:00.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.823+0000 7fab3de00640 1 -- 172.21.15.32:0/3027331992 learned_addr learned my addr 172.21.15.32:0/3027331992 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:00.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.823+0000 7fab3d400640 1 --2- 172.21.15.32:0/3027331992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc88666b0 0x563cc8c32b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.823+0000 7fab3de00640 1 -- 172.21.15.32:0/3027331992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc88666b0 msgr2=0x563cc8c32b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:00.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.823+0000 7fab3de00640 1 --2- 172.21.15.32:0/3027331992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc88666b0 0x563cc8c32b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.823+0000 7fab3de00640 1 -- 172.21.15.32:0/3027331992 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563cc8887390 msgr2=0x563cc8c39af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:00.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.823+0000 7fab3de00640 1 --2- 172.21.15.32:0/3027331992 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563cc8887390 0x563cc8c39af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.823+0000 7fab3de00640 1 -- 172.21.15.32:0/3027331992 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563cc89697d0 con 0x563cc885e4d0 2024-04-30T08:37:00.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.823+0000 7fab3de00640 1 --2- 172.21.15.32:0/3027331992 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cc885e4d0 0x563cc8c32650 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fab30009230 tx=0x7fab300107e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef77155aa85d436d server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:00.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.823+0000 7fab1fe00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:37:00.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.824+0000 7fab1fe00640 1 -- 172.21.15.32:0/3027331992 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab300ab650 con 0x563cc885e4d0 2024-04-30T08:37:00.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.824+0000 7fab1fe00640 1 -- 172.21.15.32:0/3027331992 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fab300ab120 con 0x563cc885e4d0 2024-04-30T08:37:00.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.824+0000 7fab4bffd740 1 -- 172.21.15.32:0/3027331992 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab38030560 con 0x563cc885e4d0 2024-04-30T08:37:00.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.824+0000 7fab1fe00640 1 -- 172.21.15.32:0/3027331992 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fab300b0100 con 0x563cc885e4d0 2024-04-30T08:37:00.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.824+0000 7fab4bffd740 10 client.16014 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:00.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.824+0000 7fab4bffd740 1 -- 172.21.15.32:0/3027331992 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563cc8c3ca90 con 0x563cc885e4d0 2024-04-30T08:37:00.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.824+0000 7fab1fe00640 1 -- 172.21.15.32:0/3027331992 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab38030560 con 0x563cc885e4d0 2024-04-30T08:37:00.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.825+0000 7fab1fe00640 1 -- 172.21.15.32:0/3027331992 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7fab300a2d50 con 0x563cc885e4d0 2024-04-30T08:37:00.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.825+0000 7fab4bffd740 10 client.16014 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:00.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.825+0000 7fab4bffd740 10 client.16014 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-30T08:37:00.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.825+0000 7fab4bffd740 20 client.16014 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:00.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.825+0000 7fab4bffd740 1 --2- 172.21.15.32:0/3027331992 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x563cc8845f60 0x563cc8c400e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.825+0000 7fab4bffd740 4 client.16014 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:37:00.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.825+0000 7fab4bffd740 1 -- 172.21.15.32:0/3027331992 --> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563cc8b9b960 con 0x563cc8845f60 2024-04-30T08:37:00.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.825+0000 7fab3d400640 1 --2- 172.21.15.32:0/3027331992 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x563cc8845f60 0x563cc8c400e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.829 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.826+0000 7fab3d400640 1 --2- 172.21.15.32:0/3027331992 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x563cc8845f60 0x563cc8c400e0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b78d61a6e7d978be server_cookie=1206bf16a9bc1425 in_seq=0 out_seq=0 2024-04-30T08:37:00.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.826+0000 7fab1fe00640 10 client.16014 ms_handle_connect on v2:172.21.15.32:6838/4160386524 2024-04-30T08:37:00.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.827+0000 7fab1fe00640 1 -- 172.21.15.32:0/3027331992 <== mds.1 v2:172.21.15.32:6838/4160386524 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x563cc8b9b960 con 0x563cc8845f60 2024-04-30T08:37:00.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.827+0000 7fab1fe00640 10 client.16014 handle_command_reply: tid=0 2024-04-30T08:37:00.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.827+0000 7fab4bffd740 1 client.16014 shutdown 2024-04-30T08:37:00.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.827+0000 7fab4bffd740 1 -- 172.21.15.32:0/3027331992 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cc885e4d0 msgr2=0x563cc8c32650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:00.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.827+0000 7fab4bffd740 1 --2- 172.21.15.32:0/3027331992 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cc885e4d0 0x563cc8c32650 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fab30009230 tx=0x7fab300107e0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.827+0000 7fab4bffd740 1 -- 172.21.15.32:0/3027331992 shutdown_connections 2024-04-30T08:37:00.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.827+0000 7fab4bffd740 1 --2- 172.21.15.32:0/3027331992 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x563cc8845f60 0x563cc8c400e0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.827+0000 7fab4bffd740 1 --2- 172.21.15.32:0/3027331992 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563cc8887390 0x563cc8c39af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.827+0000 7fab4bffd740 1 --2- 172.21.15.32:0/3027331992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc88666b0 0x563cc8c32b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.827+0000 7fab4bffd740 1 --2- 172.21.15.32:0/3027331992 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cc885e4d0 0x563cc8c32650 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.827+0000 7fab1fe00640 0 client.16014 ms_handle_reset on v2:172.21.15.32:6838/4160386524 2024-04-30T08:37:00.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.827+0000 7fab4bffd740 1 -- 172.21.15.32:0/3027331992 >> 172.21.15.32:0/3027331992 conn(0x563cc887b4e0 msgr2=0x563cc8c23100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:00.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.827+0000 7fab4bffd740 1 -- 172.21.15.32:0/3027331992 shutdown_connections 2024-04-30T08:37:00.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.828+0000 7fab4bffd740 1 -- 172.21.15.32:0/3027331992 wait complete. 2024-04-30T08:37:00.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.828+0000 7fab4bffd740 20 client.16014 trim_cache size 0 max 16384 2024-04-30T08:37:00.831 INFO:tasks.workunit.client.0.smithi032.stderr:+ make 2024-04-30T08:37:00.840 INFO:tasks.workunit.client.0.smithi032.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run aclocal-1.10 2024-04-30T08:37:00.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.843+0000 7fab4bffd740 1 Processor -- start 2024-04-30T08:37:00.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.843+0000 7fab4bffd740 1 -- start start 2024-04-30T08:37:00.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.844+0000 7fab4bffd740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563cc887b4e0 0x563cc8c31380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.844+0000 7fab4bffd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc885e4d0 0x563cc8c32220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.844+0000 7fab4bffd740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cc88666b0 0x563cc8c32760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.844+0000 7fab4bffd740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fab401aeab0 con 0x563cc88666b0 2024-04-30T08:37:00.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.844+0000 7fab4bffd740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fab401b0680 con 0x563cc887b4e0 2024-04-30T08:37:00.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.844+0000 7fab4bffd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fab401af180 con 0x563cc885e4d0 2024-04-30T08:37:00.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.844+0000 7fab3de00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563cc887b4e0 0x563cc8c31380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.844+0000 7fab3e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cc88666b0 0x563cc8c32760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.844+0000 7fab3de00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563cc887b4e0 0x563cc8c31380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:35744/0 (socket says 172.21.15.32:35744) 2024-04-30T08:37:00.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.844+0000 7fab3de00640 1 -- 172.21.15.32:0/2398323919 learned_addr learned my addr 172.21.15.32:0/2398323919 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:00.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.844+0000 7fab3d400640 1 --2- 172.21.15.32:0/2398323919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc885e4d0 0x563cc8c32220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.845+0000 7fab3de00640 1 -- 172.21.15.32:0/2398323919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc885e4d0 msgr2=0x563cc8c32220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:00.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.845+0000 7fab3de00640 1 --2- 172.21.15.32:0/2398323919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc885e4d0 0x563cc8c32220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.845+0000 7fab3de00640 1 -- 172.21.15.32:0/2398323919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cc88666b0 msgr2=0x563cc8c32760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:00.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.845+0000 7fab3de00640 1 --2- 172.21.15.32:0/2398323919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cc88666b0 0x563cc8c32760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.845+0000 7fab3de00640 1 -- 172.21.15.32:0/2398323919 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563cc8c336c0 con 0x563cc887b4e0 2024-04-30T08:37:00.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.845+0000 7fab3e800640 1 --2- 172.21.15.32:0/2398323919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cc88666b0 0x563cc8c32760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:37:00.847 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: aclocal-1.10: command not found 2024-04-30T08:37:00.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.845+0000 7fab3de00640 1 --2- 172.21.15.32:0/2398323919 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563cc887b4e0 0x563cc8c31380 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fab300b02c0 tx=0x7fab3000cef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=71eb528bd9453d91 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:00.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.845+0000 7fab2aa00640 1 -- 172.21.15.32:0/2398323919 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab300a6140 con 0x563cc887b4e0 2024-04-30T08:37:00.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.845+0000 7fab2aa00640 1 -- 172.21.15.32:0/2398323919 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab300a2cc0 con 0x563cc887b4e0 2024-04-30T08:37:00.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.845+0000 7fab2aa00640 1 -- 172.21.15.32:0/2398323919 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab300c1e20 con 0x563cc887b4e0 2024-04-30T08:37:00.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.846+0000 7fab4bffd740 1 -- 172.21.15.32:0/2398323919 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563cc887b4e0 msgr2=0x563cc8c31380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:00.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.846+0000 7fab4bffd740 1 --2- 172.21.15.32:0/2398323919 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563cc887b4e0 0x563cc8c31380 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fab300b02c0 tx=0x7fab3000cef0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.846+0000 7fab4bffd740 1 -- 172.21.15.32:0/2398323919 shutdown_connections 2024-04-30T08:37:00.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.846+0000 7fab4bffd740 1 --2- 172.21.15.32:0/2398323919 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cc88666b0 0x563cc8c32760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.846+0000 7fab4bffd740 1 --2- 172.21.15.32:0/2398323919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc885e4d0 0x563cc8c32220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.846+0000 7fab4bffd740 1 --2- 172.21.15.32:0/2398323919 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563cc887b4e0 0x563cc8c31380 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.846+0000 7fab4bffd740 1 -- 172.21.15.32:0/2398323919 >> 172.21.15.32:0/2398323919 conn(0x563cc8c2c370 msgr2=0x563cc8c30c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:00.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.846+0000 7fab4bffd740 1 -- 172.21.15.32:0/2398323919 shutdown_connections 2024-04-30T08:37:00.849 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.846+0000 7fab4bffd740 1 -- 172.21.15.32:0/2398323919 wait complete. 2024-04-30T08:37:00.849 INFO:tasks.workunit.client.0.smithi032.stderr:WARNING: `aclocal-1.10' is missing on your system. You should only need it if 2024-04-30T08:37:00.849 INFO:tasks.workunit.client.0.smithi032.stderr: you modified `acinclude.m4' or `configure.in'. You might want 2024-04-30T08:37:00.849 INFO:tasks.workunit.client.0.smithi032.stderr: to install the `Automake' and `Perl' packages. Grab them from 2024-04-30T08:37:00.849 INFO:tasks.workunit.client.0.smithi032.stderr: any GNU archive site. 2024-04-30T08:37:00.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.847+0000 7fab4bffd740 1 -- start start 2024-04-30T08:37:00.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.847+0000 7fab4bffd740 1 Processor -- start 2024-04-30T08:37:00.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.847+0000 7fab4bffd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc885e4d0 0x563cc8c367e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.847+0000 7fab4bffd740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cc88666b0 0x563cc8c36d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.847+0000 7fab4bffd740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563cc887b4e0 0x563cc8c37c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.848+0000 7fab4bffd740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fab401aff20 con 0x563cc88666b0 2024-04-30T08:37:00.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.848+0000 7fab4bffd740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fab401af850 con 0x563cc887b4e0 2024-04-30T08:37:00.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.848+0000 7fab4bffd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fab401aca70 con 0x563cc885e4d0 2024-04-30T08:37:00.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.848+0000 7fab4bffd740 10 client.0 _ll_register_callbacks cb 0x7fab300a6d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:00.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.848+0000 7fab3d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cc88666b0 0x563cc8c36d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.848+0000 7fab3e800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563cc887b4e0 0x563cc8c37c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.848+0000 7fab3d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cc88666b0 0x563cc8c36d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56556/0 (socket says 172.21.15.32:56556) 2024-04-30T08:37:00.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.848+0000 7fab3d400640 1 -- 172.21.15.32:0/1618447921 learned_addr learned my addr 172.21.15.32:0/1618447921 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:00.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.848+0000 7fab3de00640 1 --2- 172.21.15.32:0/1618447921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc885e4d0 0x563cc8c367e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.848+0000 7fab3e800640 1 -- 172.21.15.32:0/1618447921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc885e4d0 msgr2=0x563cc8c367e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:00.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.848+0000 7fab3e800640 1 --2- 172.21.15.32:0/1618447921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc885e4d0 0x563cc8c367e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.848+0000 7fab3e800640 1 -- 172.21.15.32:0/1618447921 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cc88666b0 msgr2=0x563cc8c36d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:00.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.848+0000 7fab3e800640 1 --2- 172.21.15.32:0/1618447921 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cc88666b0 0x563cc8c36d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.848+0000 7fab3e800640 1 -- 172.21.15.32:0/1618447921 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563cc8c381c0 con 0x563cc887b4e0 2024-04-30T08:37:00.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.848+0000 7fab3d400640 1 --2- 172.21.15.32:0/1618447921 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cc88666b0 0x563cc8c36d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:37:00.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.849+0000 7fab3e800640 1 --2- 172.21.15.32:0/1618447921 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563cc887b4e0 0x563cc8c37c00 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fab38006010 tx=0x7fab38030740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=294be67c46e2c180 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:00.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.849+0000 7fab1fe00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:37:00.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.849+0000 7fab1fe00640 1 -- 172.21.15.32:0/1618447921 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab38036070 con 0x563cc887b4e0 2024-04-30T08:37:00.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.849+0000 7fab1fe00640 1 -- 172.21.15.32:0/1618447921 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fab38030b30 con 0x563cc887b4e0 2024-04-30T08:37:00.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.849+0000 7fab4bffd740 1 -- 172.21.15.32:0/1618447921 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab300a2cc0 con 0x563cc887b4e0 2024-04-30T08:37:00.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.849+0000 7fab1fe00640 1 -- 172.21.15.32:0/1618447921 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fab38056050 con 0x563cc887b4e0 2024-04-30T08:37:00.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.849+0000 7fab4bffd740 10 client.26156 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:00.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.849+0000 7fab4bffd740 1 -- 172.21.15.32:0/1618447921 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563cc8c39440 con 0x563cc887b4e0 2024-04-30T08:37:00.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.850+0000 7fab1fe00640 1 -- 172.21.15.32:0/1618447921 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fab300a2cc0 con 0x563cc887b4e0 2024-04-30T08:37:00.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.850+0000 7fab1fe00640 1 -- 172.21.15.32:0/1618447921 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7fab380043b0 con 0x563cc887b4e0 2024-04-30T08:37:00.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.850+0000 7fab4bffd740 10 client.26156 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:00.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.850+0000 7fab4bffd740 10 client.26156 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-30T08:37:00.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.850+0000 7fab4bffd740 20 client.26156 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:00.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.850+0000 7fab4bffd740 1 --2- 172.21.15.32:0/1618447921 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x563cc8887390 0x563cc8caa780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:00.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.850+0000 7fab4bffd740 4 client.26156 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-30T08:37:00.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.850+0000 7fab4bffd740 1 -- 172.21.15.32:0/1618447921 --> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563cc8b957d0 con 0x563cc8887390 2024-04-30T08:37:00.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.850+0000 7fab3de00640 1 --2- 172.21.15.32:0/1618447921 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x563cc8887390 0x563cc8caa780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:00.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.851+0000 7fab3de00640 1 --2- 172.21.15.32:0/1618447921 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x563cc8887390 0x563cc8caa780 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=93315f3cd0042129 server_cookie=d72b5bc662828814 in_seq=0 out_seq=0 2024-04-30T08:37:00.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.851+0000 7fab1fe00640 10 client.26156 ms_handle_connect on v2:172.21.15.32:6838/4160386524 2024-04-30T08:37:00.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.851+0000 7fab1fe00640 1 -- 172.21.15.32:0/1618447921 <== mds.1 v2:172.21.15.32:6838/4160386524 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563cc8b957d0 con 0x563cc8887390 2024-04-30T08:37:00.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.851+0000 7fab1fe00640 10 client.26156 handle_command_reply: tid=0 2024-04-30T08:37:00.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.851+0000 7fab4bffd740 1 client.26156 shutdown 2024-04-30T08:37:00.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.852+0000 7fab4bffd740 1 -- 172.21.15.32:0/1618447921 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563cc887b4e0 msgr2=0x563cc8c37c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:00.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.852+0000 7fab4bffd740 1 --2- 172.21.15.32:0/1618447921 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563cc887b4e0 0x563cc8c37c00 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fab38006010 tx=0x7fab38030740 comp rx=0 tx=0).stop 2024-04-30T08:37:00.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.852+0000 7fab4bffd740 1 -- 172.21.15.32:0/1618447921 shutdown_connections 2024-04-30T08:37:00.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.852+0000 7fab4bffd740 1 --2- 172.21.15.32:0/1618447921 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x563cc8887390 0x563cc8caa780 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.852+0000 7fab4bffd740 1 --2- 172.21.15.32:0/1618447921 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563cc887b4e0 0x563cc8c37c00 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.852+0000 7fab4bffd740 1 --2- 172.21.15.32:0/1618447921 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563cc88666b0 0x563cc8c36d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.852+0000 7fab4bffd740 1 --2- 172.21.15.32:0/1618447921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563cc885e4d0 0x563cc8c367e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.852+0000 7fab4bffd740 1 -- 172.21.15.32:0/1618447921 >> 172.21.15.32:0/1618447921 conn(0x563cc8c2c370 msgr2=0x563cc8c32190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:00.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.852+0000 7fab1fe00640 0 client.26156 ms_handle_reset on v2:172.21.15.32:6838/4160386524 2024-04-30T08:37:00.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.852+0000 7fab4bffd740 1 -- 172.21.15.32:0/1618447921 shutdown_connections 2024-04-30T08:37:00.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.852+0000 7fab4bffd740 1 -- 172.21.15.32:0/1618447921 wait complete. 2024-04-30T08:37:00.856 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.852+0000 7fab4bffd740 20 client.26156 trim_cache size 0 max 16384 2024-04-30T08:37:00.857 INFO:teuthology.orchestra.run.smithi032.stdout:[] 2024-04-30T08:37:00.857 INFO:tasks.workunit.client.0.smithi032.stdout: cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run automake-1.10 --gnu 2024-04-30T08:37:00.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.853+0000 7fab3f200640 1 -- 172.21.15.32:0/90589339 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fab240afb60 msgr2=0x7fab240b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:00.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.853+0000 7fab3f200640 1 --2- 172.21.15.32:0/90589339 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fab240afb60 0x7fab240b2070 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x563cc8c24ef0 tx=0x7fab2c005f90 comp rx=0 tx=0).stop 2024-04-30T08:37:00.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.853+0000 7fab3f200640 1 -- 172.21.15.32:0/90589339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab40075010 msgr2=0x7fab4019c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:00.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.853+0000 7fab3f200640 1 --2- 172.21.15.32:0/90589339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab40075010 0x7fab4019c280 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fab3000e9f0 tx=0x7fab3000eeb0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.853+0000 7fab3f200640 1 -- 172.21.15.32:0/90589339 shutdown_connections 2024-04-30T08:37:00.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.853+0000 7fab3f200640 1 --2- 172.21.15.32:0/90589339 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fab240afb60 0x7fab240b2070 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.853+0000 7fab3f200640 1 --2- 172.21.15.32:0/90589339 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fab4010b900 0x7fab401a3750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.853+0000 7fab3f200640 1 --2- 172.21.15.32:0/90589339 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fab40075960 0x7fab4019c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.853+0000 7fab3f200640 1 --2- 172.21.15.32:0/90589339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab40075010 0x7fab4019c280 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:00.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.853+0000 7fab3f200640 1 -- 172.21.15.32:0/90589339 >> 172.21.15.32:0/90589339 conn(0x7fab400feba0 msgr2=0x7fab400ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:00.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.853+0000 7fab3f200640 1 -- 172.21.15.32:0/90589339 shutdown_connections 2024-04-30T08:37:00.859 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:00.853+0000 7fab3f200640 1 -- 172.21.15.32:0/90589339 wait complete. 2024-04-30T08:37:00.859 INFO:tasks.workunit.client.0.smithi032.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: automake-1.10: command not found 2024-04-30T08:37:00.859 INFO:tasks.workunit.client.0.smithi032.stderr:WARNING: `automake-1.10' is missing on your system. You should only need it if 2024-04-30T08:37:00.859 INFO:tasks.workunit.client.0.smithi032.stderr: you modified `Makefile.am', `acinclude.m4' or `configure.in'. 2024-04-30T08:37:00.859 INFO:tasks.workunit.client.0.smithi032.stderr: You might want to install the `Automake' and `Perl' packages. 2024-04-30T08:37:00.859 INFO:tasks.workunit.client.0.smithi032.stderr: Grab them from any GNU archive site. 2024-04-30T08:37:00.881 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-30T08:37:00.944 INFO:tasks.workunit.client.0.smithi032.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoconf 2024-04-30T08:37:01.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:01 smithi032 ceph-mon[28331]: pgmap v480: 129 pgs: 129 active+clean; 4.0 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 314 KiB/s wr, 50 op/s 2024-04-30T08:37:01.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.207+0000 7fdac8a00640 1 -- 172.21.15.32:0/2222437722 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdac41040f0 msgr2=0x7fdac4106510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:01.208 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.207+0000 7fdac8a00640 1 --2- 172.21.15.32:0/2222437722 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdac41040f0 0x7fdac4106510 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fdab4009860 tx=0x7fdab402a340 comp rx=0 tx=0).stop 2024-04-30T08:37:01.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.208+0000 7fdac8a00640 1 -- 172.21.15.32:0/2222437722 shutdown_connections 2024-04-30T08:37:01.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.208+0000 7fdac8a00640 1 --2- 172.21.15.32:0/2222437722 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdac4109610 0x7fdac410bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:01.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.208+0000 7fdac8a00640 1 --2- 172.21.15.32:0/2222437722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdac4106ae0 0x7fdac4108f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:01.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.208+0000 7fdac8a00640 1 --2- 172.21.15.32:0/2222437722 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdac41040f0 0x7fdac4106510 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:01.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.208+0000 7fdac8a00640 1 -- 172.21.15.32:0/2222437722 >> 172.21.15.32:0/2222437722 conn(0x7fdac40fda70 msgr2=0x7fdac40fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:01.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.208+0000 7fdac8a00640 1 -- 172.21.15.32:0/2222437722 shutdown_connections 2024-04-30T08:37:01.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.208+0000 7fdac8a00640 1 -- 172.21.15.32:0/2222437722 wait complete. 2024-04-30T08:37:01.209 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.209+0000 7fdac8a00640 1 Processor -- start 2024-04-30T08:37:01.210 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.209+0000 7fdac8a00640 1 -- start start 2024-04-30T08:37:01.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.210+0000 7fdac8a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdac41040f0 0x7fdac4103960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:01.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.210+0000 7fdac8a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdac4106ae0 0x7fdac4102000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:01.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.210+0000 7fdac8a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdac4109610 0x7fdac4102540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:01.211 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.210+0000 7fdac8a00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdac410f470 con 0x7fdac4106ae0 2024-04-30T08:37:01.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.210+0000 7fdac8a00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fdac410f2f0 con 0x7fdac41040f0 2024-04-30T08:37:01.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.210+0000 7fdac8a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fdac410f5f0 con 0x7fdac4109610 2024-04-30T08:37:01.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.210+0000 7fdac2800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdac41040f0 0x7fdac4103960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:01.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.210+0000 7fdac2800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdac41040f0 0x7fdac4103960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:35788/0 (socket says 172.21.15.32:35788) 2024-04-30T08:37:01.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.210+0000 7fdac2800640 1 -- 172.21.15.32:0/306794170 learned_addr learned my addr 172.21.15.32:0/306794170 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:01.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.210+0000 7fdac3200640 1 --2- 172.21.15.32:0/306794170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdac4109610 0x7fdac4102540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:01.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.210+0000 7fdac1e00640 1 --2- 172.21.15.32:0/306794170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdac4106ae0 0x7fdac4102000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:01.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.210+0000 7fdac2800640 1 -- 172.21.15.32:0/306794170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdac4109610 msgr2=0x7fdac4102540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:01.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.210+0000 7fdac2800640 1 --2- 172.21.15.32:0/306794170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdac4109610 0x7fdac4102540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:01.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.211+0000 7fdac2800640 1 -- 172.21.15.32:0/306794170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdac4106ae0 msgr2=0x7fdac4102000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:01.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.211+0000 7fdac2800640 1 --2- 172.21.15.32:0/306794170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdac4106ae0 0x7fdac4102000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:01.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.211+0000 7fdac2800640 1 -- 172.21.15.32:0/306794170 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdac4102c70 con 0x7fdac41040f0 2024-04-30T08:37:01.212 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.211+0000 7fdac3200640 1 --2- 172.21.15.32:0/306794170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdac4109610 0x7fdac4102540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:37:01.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.211+0000 7fdac1e00640 1 --2- 172.21.15.32:0/306794170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdac4106ae0 0x7fdac4102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:37:01.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.211+0000 7fdac2800640 1 --2- 172.21.15.32:0/306794170 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdac41040f0 0x7fdac4103960 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fdab4009780 tx=0x7fdab4032d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:01.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.211+0000 7fdabb400640 1 -- 172.21.15.32:0/306794170 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab4037040 con 0x7fdac41040f0 2024-04-30T08:37:01.213 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.211+0000 7fdabb400640 1 -- 172.21.15.32:0/306794170 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdab4030c10 con 0x7fdac41040f0 2024-04-30T08:37:01.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.211+0000 7fdac8a00640 1 -- 172.21.15.32:0/306794170 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdac41a4190 con 0x7fdac41040f0 2024-04-30T08:37:01.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.211+0000 7fdabb400640 1 -- 172.21.15.32:0/306794170 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab4003850 con 0x7fdac41040f0 2024-04-30T08:37:01.214 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.211+0000 7fdac8a00640 1 -- 172.21.15.32:0/306794170 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdac41a4590 con 0x7fdac41040f0 2024-04-30T08:37:01.215 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.215+0000 7fdad0428740 1 Processor -- start 2024-04-30T08:37:01.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.215+0000 7fdad0428740 1 -- start start 2024-04-30T08:37:01.216 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.215+0000 7fdabb400640 1 -- 172.21.15.32:0/306794170 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7fdab4043020 con 0x7fdac41040f0 2024-04-30T08:37:01.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.215+0000 7fdad0428740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564cdfde5390 0x564ce0186730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:01.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.215+0000 7fdad0428740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564cdfdbc4d0 0x564ce0186cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:01.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.215+0000 7fdad0428740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cdfdc46b0 0x564ce018dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:01.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.216+0000 7fdad0428740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564ce00ebd50 con 0x564cdfde5390 2024-04-30T08:37:01.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.216+0000 7fdad0428740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x564ce00910c0 con 0x564cdfdbc4d0 2024-04-30T08:37:01.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.216+0000 7fdad0428740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564ce00e7d10 con 0x564cdfdc46b0 2024-04-30T08:37:01.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.216+0000 7fdac3200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564cdfde5390 0x564ce0186730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:01.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.216+0000 7fdac2800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564cdfdbc4d0 0x564ce0186cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:01.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.216+0000 7fdac3200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564cdfde5390 0x564ce0186730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56588/0 (socket says 172.21.15.32:56588) 2024-04-30T08:37:01.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.216+0000 7fdac3200640 1 -- 172.21.15.32:0/4269258164 learned_addr learned my addr 172.21.15.32:0/4269258164 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:01.217 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.216+0000 7fdac1e00640 1 --2- 172.21.15.32:0/4269258164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cdfdc46b0 0x564ce018dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:01.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.216+0000 7fdac2800640 1 -- 172.21.15.32:0/4269258164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cdfdc46b0 msgr2=0x564ce018dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:01.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.216+0000 7fdac2800640 1 --2- 172.21.15.32:0/4269258164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cdfdc46b0 0x564ce018dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:01.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.216+0000 7fdac2800640 1 -- 172.21.15.32:0/4269258164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564cdfde5390 msgr2=0x564ce0186730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:01.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.216+0000 7fdac2800640 1 --2- 172.21.15.32:0/4269258164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564cdfde5390 0x564ce0186730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:01.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.216+0000 7fdac2800640 1 -- 172.21.15.32:0/4269258164 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564cdfdfd3f0 con 0x564cdfdbc4d0 2024-04-30T08:37:01.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.217+0000 7fdac2800640 1 --2- 172.21.15.32:0/4269258164 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564cdfdbc4d0 0x564ce0186cb0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fdab40fbad0 tx=0x7fdab41064d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fac59ee71d7438c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:01.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.217+0000 7fdaaf400640 1 -- 172.21.15.32:0/4269258164 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab4100110 con 0x564cdfdbc4d0 2024-04-30T08:37:01.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.217+0000 7fdaaf400640 1 -- 172.21.15.32:0/4269258164 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdab410b020 con 0x564cdfdbc4d0 2024-04-30T08:37:01.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.217+0000 7fdaaf400640 1 -- 172.21.15.32:0/4269258164 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab410d530 con 0x564cdfdbc4d0 2024-04-30T08:37:01.219 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.218+0000 7fdabb400640 1 --2- 172.21.15.32:0/306794170 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fdaa80afb60 0x7fdaa80b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:01.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.218+0000 7fdabb400640 1 -- 172.21.15.32:0/306794170 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fdab40fe440 con 0x7fdac41040f0 2024-04-30T08:37:01.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.218+0000 7fdad0428740 1 -- 172.21.15.32:0/4269258164 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564cdfdbc4d0 msgr2=0x564ce0186cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:01.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.218+0000 7fdad0428740 1 --2- 172.21.15.32:0/4269258164 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564cdfdbc4d0 0x564ce0186cb0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fdab40fbad0 tx=0x7fdab41064d0 comp rx=0 tx=0).stop 2024-04-30T08:37:01.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.218+0000 7fdac3200640 1 --2- 172.21.15.32:0/306794170 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fdaa80afb60 0x7fdaa80b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:01.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.218+0000 7fdad0428740 1 -- 172.21.15.32:0/4269258164 shutdown_connections 2024-04-30T08:37:01.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.218+0000 7fdad0428740 1 --2- 172.21.15.32:0/4269258164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cdfdc46b0 0x564ce018dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:01.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.218+0000 7fdad0428740 1 --2- 172.21.15.32:0/4269258164 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564cdfdbc4d0 0x564ce0186cb0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:01.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.218+0000 7fdad0428740 1 --2- 172.21.15.32:0/4269258164 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564cdfde5390 0x564ce0186730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:01.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.218+0000 7fdad0428740 1 -- 172.21.15.32:0/4269258164 >> 172.21.15.32:0/4269258164 conn(0x564cdfdd94e0 msgr2=0x564ce01820c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:01.220 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.219+0000 7fdad0428740 1 -- 172.21.15.32:0/4269258164 shutdown_connections 2024-04-30T08:37:01.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.219+0000 7fdad0428740 1 -- 172.21.15.32:0/4269258164 wait complete. 2024-04-30T08:37:01.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.219+0000 7fdac3200640 1 --2- 172.21.15.32:0/306794170 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fdaa80afb60 0x7fdaa80b2070 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x564ce0187900 tx=0x7fdabc011040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:01.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.219+0000 7fdad0428740 1 -- start start 2024-04-30T08:37:01.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.220+0000 7fdad0428740 1 Processor -- start 2024-04-30T08:37:01.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.220+0000 7fdad0428740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564cdfdbc4d0 0x564ce0190370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:01.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.220+0000 7fdad0428740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564cdfdc46b0 0x564ce01908b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:01.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.220+0000 7fdad0428740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cdfde5390 0x564ce0197810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:01.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.220+0000 7fdad0428740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x564ce018e3c0 con 0x564cdfdc46b0 2024-04-30T08:37:01.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.220+0000 7fdad0428740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x564ce00a4b50 con 0x564cdfdbc4d0 2024-04-30T08:37:01.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.220+0000 7fdad0428740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x564ce018e540 con 0x564cdfde5390 2024-04-30T08:37:01.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.220+0000 7fdad0428740 10 client.0 _ll_register_callbacks cb 0x564ce006e420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:01.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.220+0000 7fdac1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564cdfdc46b0 0x564ce01908b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:01.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.221+0000 7fdac2800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564cdfdbc4d0 0x564ce0190370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:01.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.221+0000 7fdac3200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cdfde5390 0x564ce0197810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:01.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.221+0000 7fdac2800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564cdfdbc4d0 0x564ce0190370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:35802/0 (socket says 172.21.15.32:35802) 2024-04-30T08:37:01.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.221+0000 7fdac2800640 1 -- 172.21.15.32:0/2596127725 learned_addr learned my addr 172.21.15.32:0/2596127725 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:01.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.221+0000 7fdac2800640 1 -- 172.21.15.32:0/2596127725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cdfde5390 msgr2=0x564ce0197810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:01.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.221+0000 7fdac2800640 1 --2- 172.21.15.32:0/2596127725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cdfde5390 0x564ce0197810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:01.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.221+0000 7fdac2800640 1 -- 172.21.15.32:0/2596127725 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564cdfdc46b0 msgr2=0x564ce01908b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:01.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.221+0000 7fdac2800640 1 --2- 172.21.15.32:0/2596127725 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564cdfdc46b0 0x564ce01908b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:01.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.221+0000 7fdac2800640 1 -- 172.21.15.32:0/2596127725 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564cdfec77d0 con 0x564cdfdbc4d0 2024-04-30T08:37:01.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.221+0000 7fdac2800640 1 --2- 172.21.15.32:0/2596127725 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564cdfdbc4d0 0x564ce0190370 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fdab40c62a0 tx=0x7fdab40fe6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1030a3cf91e2dabd server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:01.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.221+0000 7fdaa3e00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:37:01.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.221+0000 7fdaa3e00640 1 -- 172.21.15.32:0/2596127725 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab40c2460 con 0x564cdfdbc4d0 2024-04-30T08:37:01.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.222+0000 7fdaa3e00640 1 -- 172.21.15.32:0/2596127725 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fdab40fe830 con 0x564cdfdbc4d0 2024-04-30T08:37:01.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.222+0000 7fdad0428740 1 -- 172.21.15.32:0/2596127725 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdab410b020 con 0x564cdfdbc4d0 2024-04-30T08:37:01.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.222+0000 7fdaa3e00640 1 -- 172.21.15.32:0/2596127725 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fdab40c5d10 con 0x564cdfdbc4d0 2024-04-30T08:37:01.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.222+0000 7fdad0428740 10 client.26168 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:01.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.222+0000 7fdad0428740 1 -- 172.21.15.32:0/2596127725 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564ce018e3c0 con 0x564cdfdbc4d0 2024-04-30T08:37:01.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.222+0000 7fdaa3e00640 1 -- 172.21.15.32:0/2596127725 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdab410b020 con 0x564cdfdbc4d0 2024-04-30T08:37:01.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.222+0000 7fdaa3e00640 1 -- 172.21.15.32:0/2596127725 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7fdab40fe0f0 con 0x564cdfdbc4d0 2024-04-30T08:37:01.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.223+0000 7fdad0428740 10 client.26168 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:01.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.223+0000 7fdad0428740 10 client.26168 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:37:01.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.223+0000 7fdad0428740 20 client.26168 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:01.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.223+0000 7fdad0428740 1 --2- 172.21.15.32:0/2596127725 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x564cdfda3f60 0x564ce019dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:01.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.223+0000 7fdad0428740 4 client.26168 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:37:01.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.223+0000 7fdad0428740 1 -- 172.21.15.32:0/2596127725 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ce00f9960 con 0x564cdfda3f60 2024-04-30T08:37:01.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.223+0000 7fdac1e00640 1 --2- 172.21.15.32:0/2596127725 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x564cdfda3f60 0x564ce019dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:01.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.223+0000 7fdac1e00640 1 --2- 172.21.15.32:0/2596127725 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x564cdfda3f60 0x564ce019dc10 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd89753e20bdaf2d server_cookie=a77abc2bb7c7600e in_seq=0 out_seq=0 2024-04-30T08:37:01.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.223+0000 7fdaa3e00640 10 client.26168 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:01.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.224+0000 7fdaa3e00640 1 -- 172.21.15.32:0/2596127725 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x564ce00f9960 con 0x564cdfda3f60 2024-04-30T08:37:01.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.224+0000 7fdaa3e00640 10 client.26168 handle_command_reply: tid=0 2024-04-30T08:37:01.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.224+0000 7fdad0428740 1 client.26168 shutdown 2024-04-30T08:37:01.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.224+0000 7fdad0428740 1 -- 172.21.15.32:0/2596127725 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564cdfdbc4d0 msgr2=0x564ce0190370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:01.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.224+0000 7fdad0428740 1 --2- 172.21.15.32:0/2596127725 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564cdfdbc4d0 0x564ce0190370 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fdab40c62a0 tx=0x7fdab40fe6c0 comp rx=0 tx=0).stop 2024-04-30T08:37:01.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.225+0000 7fdad0428740 1 -- 172.21.15.32:0/2596127725 shutdown_connections 2024-04-30T08:37:01.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.225+0000 7fdad0428740 1 --2- 172.21.15.32:0/2596127725 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x564cdfda3f60 0x564ce019dc10 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:01.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.225+0000 7fdad0428740 1 --2- 172.21.15.32:0/2596127725 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cdfde5390 0x564ce0197810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:01.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.225+0000 7fdad0428740 1 --2- 172.21.15.32:0/2596127725 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564cdfdc46b0 0x564ce01908b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:01.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.225+0000 7fdaa3e00640 0 client.26168 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:01.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.225+0000 7fdad0428740 1 --2- 172.21.15.32:0/2596127725 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564cdfdbc4d0 0x564ce0190370 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:01.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.225+0000 7fdad0428740 1 -- 172.21.15.32:0/2596127725 >> 172.21.15.32:0/2596127725 conn(0x564cdfdd94e0 msgr2=0x564ce0181100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:01.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.225+0000 7fdad0428740 1 -- 172.21.15.32:0/2596127725 shutdown_connections 2024-04-30T08:37:01.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.225+0000 7fdad0428740 1 -- 172.21.15.32:0/2596127725 wait complete. 2024-04-30T08:37:01.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.225+0000 7fdad0428740 20 client.26168 trim_cache size 0 max 16384 2024-04-30T08:37:01.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.240+0000 7fdad0428740 1 Processor -- start 2024-04-30T08:37:01.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.240+0000 7fdad0428740 1 -- start start 2024-04-30T08:37:01.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.241+0000 7fdad0428740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564cdfdbc4d0 0x564ce018ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:01.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.241+0000 7fdad0428740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564cdfdc46b0 0x564ce0184f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:01.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.241+0000 7fdad0428740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cdfde5390 0x564ce01854a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:01.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.241+0000 7fdad0428740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdac41aeae0 con 0x564cdfdc46b0 2024-04-30T08:37:01.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.241+0000 7fdad0428740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fdac41b06b0 con 0x564cdfdbc4d0 2024-04-30T08:37:01.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.241+0000 7fdad0428740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fdac41af1b0 con 0x564cdfde5390 2024-04-30T08:37:01.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.241+0000 7fdac2800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564cdfdbc4d0 0x564ce018ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:01.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.241+0000 7fdac1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564cdfdc46b0 0x564ce0184f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:01.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.241+0000 7fdac2800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564cdfdbc4d0 0x564ce018ef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:35810/0 (socket says 172.21.15.32:35810) 2024-04-30T08:37:01.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.241+0000 7fdac1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564cdfdc46b0 0x564ce0184f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56600/0 (socket says 172.21.15.32:56600) 2024-04-30T08:37:01.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.241+0000 7fdac2800640 1 -- 172.21.15.32:0/1137326908 learned_addr learned my addr 172.21.15.32:0/1137326908 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:01.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.241+0000 7fdac3200640 1 --2- 172.21.15.32:0/1137326908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cdfde5390 0x564ce01854a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:01.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.242+0000 7fdac2800640 1 -- 172.21.15.32:0/1137326908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cdfde5390 msgr2=0x564ce01854a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:01.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.242+0000 7fdac2800640 1 --2- 172.21.15.32:0/1137326908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cdfde5390 0x564ce01854a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:01.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.242+0000 7fdac2800640 1 -- 172.21.15.32:0/1137326908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564cdfdc46b0 msgr2=0x564ce0184f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:01.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.242+0000 7fdac2800640 1 --2- 172.21.15.32:0/1137326908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564cdfdc46b0 0x564ce0184f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:01.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.242+0000 7fdac2800640 1 -- 172.21.15.32:0/1137326908 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ce01862e0 con 0x564cdfdbc4d0 2024-04-30T08:37:01.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.242+0000 7fdac3200640 1 --2- 172.21.15.32:0/1137326908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cdfde5390 0x564ce01854a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:37:01.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.242+0000 7fdac2800640 1 --2- 172.21.15.32:0/1137326908 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564cdfdbc4d0 0x564ce018ef80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fdab40c3810 tx=0x7fdab40045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=51927cb60a2a88e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:01.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.242+0000 7fdaaea00640 1 -- 172.21.15.32:0/1137326908 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab40c5ef0 con 0x564cdfdbc4d0 2024-04-30T08:37:01.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.242+0000 7fdaaea00640 1 -- 172.21.15.32:0/1137326908 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdab4101c50 con 0x564cdfdbc4d0 2024-04-30T08:37:01.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.242+0000 7fdaaea00640 1 -- 172.21.15.32:0/1137326908 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab40c6370 con 0x564cdfdbc4d0 2024-04-30T08:37:01.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.243+0000 7fdad0428740 1 -- 172.21.15.32:0/1137326908 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564cdfdbc4d0 msgr2=0x564ce018ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:01.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.243+0000 7fdad0428740 1 --2- 172.21.15.32:0/1137326908 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564cdfdbc4d0 0x564ce018ef80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fdab40c3810 tx=0x7fdab40045e0 comp rx=0 tx=0).stop 2024-04-30T08:37:01.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.243+0000 7fdad0428740 1 -- 172.21.15.32:0/1137326908 shutdown_connections 2024-04-30T08:37:01.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.243+0000 7fdad0428740 1 --2- 172.21.15.32:0/1137326908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cdfde5390 0x564ce01854a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:01.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.243+0000 7fdad0428740 1 --2- 172.21.15.32:0/1137326908 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564cdfdc46b0 0x564ce0184f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:01.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.243+0000 7fdad0428740 1 --2- 172.21.15.32:0/1137326908 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564cdfdbc4d0 0x564ce018ef80 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:01.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.243+0000 7fdad0428740 1 -- 172.21.15.32:0/1137326908 >> 172.21.15.32:0/1137326908 conn(0x564cdfdd94e0 msgr2=0x564ce018e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:01.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.243+0000 7fdad0428740 1 -- 172.21.15.32:0/1137326908 shutdown_connections 2024-04-30T08:37:01.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.243+0000 7fdad0428740 1 -- 172.21.15.32:0/1137326908 wait complete. 2024-04-30T08:37:01.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.244+0000 7fdad0428740 1 -- start start 2024-04-30T08:37:01.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.244+0000 7fdad0428740 1 Processor -- start 2024-04-30T08:37:01.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.244+0000 7fdad0428740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564cdfdbc4d0 0x564ce0196eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:01.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.244+0000 7fdad0428740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564cdfdc46b0 0x564ce0193a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:01.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.244+0000 7fdad0428740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cdfde5390 0x564ce01940d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:01.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.244+0000 7fdad0428740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fdac41aff50 con 0x564cdfdc46b0 2024-04-30T08:37:01.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.244+0000 7fdad0428740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fdac41af880 con 0x564cdfdbc4d0 2024-04-30T08:37:01.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.244+0000 7fdad0428740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fdac41acaa0 con 0x564cdfde5390 2024-04-30T08:37:01.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.245+0000 7fdad0428740 10 client.0 _ll_register_callbacks cb 0x7fdabc009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:01.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.245+0000 7fdac1e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564cdfdc46b0 0x564ce0193a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:01.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.245+0000 7fdac2800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564cdfdbc4d0 0x564ce0196eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:01.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.245+0000 7fdac2800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564cdfdbc4d0 0x564ce0196eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:35814/0 (socket says 172.21.15.32:35814) 2024-04-30T08:37:01.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.245+0000 7fdac2800640 1 -- 172.21.15.32:0/1035112830 learned_addr learned my addr 172.21.15.32:0/1035112830 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:01.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.245+0000 7fdac3200640 1 --2- 172.21.15.32:0/1035112830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cdfde5390 0x564ce01940d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:01.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.245+0000 7fdac2800640 1 -- 172.21.15.32:0/1035112830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cdfde5390 msgr2=0x564ce01940d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:01.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.245+0000 7fdac2800640 1 --2- 172.21.15.32:0/1035112830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cdfde5390 0x564ce01940d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:01.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.245+0000 7fdac2800640 1 -- 172.21.15.32:0/1035112830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564cdfdc46b0 msgr2=0x564ce0193a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:01.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.245+0000 7fdac2800640 1 --2- 172.21.15.32:0/1035112830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564cdfdc46b0 0x564ce0193a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:01.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.245+0000 7fdac2800640 1 -- 172.21.15.32:0/1035112830 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ce0203fe0 con 0x564cdfdbc4d0 2024-04-30T08:37:01.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.245+0000 7fdac2800640 1 --2- 172.21.15.32:0/1035112830 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564cdfdbc4d0 0x564ce0196eb0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fdab40ca950 tx=0x7fdab40cf110 comp rx=0 tx=0).ready entity=mon.2 client_cookie=56d289505521426d server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:01.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.245+0000 7fdaa3e00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:37:01.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.246+0000 7fdaa3e00640 1 -- 172.21.15.32:0/1035112830 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdab40c6370 con 0x564cdfdbc4d0 2024-04-30T08:37:01.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.246+0000 7fdaa3e00640 1 -- 172.21.15.32:0/1035112830 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fdab40c2b90 con 0x564cdfdbc4d0 2024-04-30T08:37:01.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.246+0000 7fdad0428740 1 -- 172.21.15.32:0/1035112830 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdab4101c50 con 0x564cdfdbc4d0 2024-04-30T08:37:01.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.246+0000 7fdaa3e00640 1 -- 172.21.15.32:0/1035112830 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fdab40c5ef0 con 0x564cdfdbc4d0 2024-04-30T08:37:01.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.246+0000 7fdad0428740 10 client.26174 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:01.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.246+0000 7fdad0428740 1 -- 172.21.15.32:0/1035112830 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x564ce0203fe0 con 0x564cdfdbc4d0 2024-04-30T08:37:01.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.246+0000 7fdaa3e00640 1 -- 172.21.15.32:0/1035112830 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fdab40e0080 con 0x564cdfdbc4d0 2024-04-30T08:37:01.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.246+0000 7fdaa3e00640 1 -- 172.21.15.32:0/1035112830 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7fdab40fe0f0 con 0x564cdfdbc4d0 2024-04-30T08:37:01.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.246+0000 7fdad0428740 10 client.26174 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:01.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.246+0000 7fdad0428740 10 client.26174 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:37:01.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.246+0000 7fdad0428740 20 client.26174 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:01.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.246+0000 7fdad0428740 1 --2- 172.21.15.32:0/1035112830 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x564cdfd793a0 0x564ce0207860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:01.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.246+0000 7fdad0428740 4 client.26174 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-30T08:37:01.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.246+0000 7fdad0428740 1 -- 172.21.15.32:0/1035112830 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564ce00f37d0 con 0x564cdfd793a0 2024-04-30T08:37:01.250 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.247+0000 7fdac1e00640 1 --2- 172.21.15.32:0/1035112830 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x564cdfd793a0 0x564ce0207860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:01.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.247+0000 7fdac1e00640 1 --2- 172.21.15.32:0/1035112830 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x564cdfd793a0 0x564ce0207860 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd60d99bcd5c8a8a server_cookie=41fa4f07a8ccf8f1 in_seq=0 out_seq=0 2024-04-30T08:37:01.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.247+0000 7fdaa3e00640 10 client.26174 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:01.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.247+0000 7fdaa3e00640 1 -- 172.21.15.32:0/1035112830 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564ce00f37d0 con 0x564cdfd793a0 2024-04-30T08:37:01.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.247+0000 7fdaa3e00640 10 client.26174 handle_command_reply: tid=0 2024-04-30T08:37:01.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.248+0000 7fdad0428740 1 client.26174 shutdown 2024-04-30T08:37:01.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.248+0000 7fdad0428740 1 -- 172.21.15.32:0/1035112830 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564cdfdbc4d0 msgr2=0x564ce0196eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:01.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.248+0000 7fdad0428740 1 --2- 172.21.15.32:0/1035112830 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564cdfdbc4d0 0x564ce0196eb0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fdab40ca950 tx=0x7fdab40cf110 comp rx=0 tx=0).stop 2024-04-30T08:37:01.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.248+0000 7fdad0428740 1 -- 172.21.15.32:0/1035112830 shutdown_connections 2024-04-30T08:37:01.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.248+0000 7fdad0428740 1 --2- 172.21.15.32:0/1035112830 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x564cdfd793a0 0x564ce0207860 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:01.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.248+0000 7fdad0428740 1 --2- 172.21.15.32:0/1035112830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564cdfde5390 0x564ce01940d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:01.252 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.248+0000 7fdad0428740 1 --2- 172.21.15.32:0/1035112830 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x564cdfdc46b0 0x564ce0193a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:01.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.248+0000 7fdad0428740 1 --2- 172.21.15.32:0/1035112830 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x564cdfdbc4d0 0x564ce0196eb0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:01.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.248+0000 7fdaa3e00640 0 client.26174 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:01.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.248+0000 7fdad0428740 1 -- 172.21.15.32:0/1035112830 >> 172.21.15.32:0/1035112830 conn(0x564cdfdd94e0 msgr2=0x564ce0184f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:01.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.248+0000 7fdad0428740 1 -- 172.21.15.32:0/1035112830 shutdown_connections 2024-04-30T08:37:01.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.248+0000 7fdad0428740 1 -- 172.21.15.32:0/1035112830 wait complete. 2024-04-30T08:37:01.253 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.248+0000 7fdad0428740 20 client.26174 trim_cache size 0 max 16384 2024-04-30T08:37:01.253 INFO:teuthology.orchestra.run.smithi032.stdout:[] 2024-04-30T08:37:01.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.249+0000 7fdac8a00640 1 -- 172.21.15.32:0/306794170 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fdaa80afb60 msgr2=0x7fdaa80b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:01.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.249+0000 7fdac8a00640 1 --2- 172.21.15.32:0/306794170 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fdaa80afb60 0x7fdaa80b2070 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x564ce0187900 tx=0x7fdabc011040 comp rx=0 tx=0).stop 2024-04-30T08:37:01.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.249+0000 7fdac8a00640 1 -- 172.21.15.32:0/306794170 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdac41040f0 msgr2=0x7fdac4103960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:01.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.249+0000 7fdac8a00640 1 --2- 172.21.15.32:0/306794170 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdac41040f0 0x7fdac4103960 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fdab4009780 tx=0x7fdab4032d50 comp rx=0 tx=0).stop 2024-04-30T08:37:01.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.249+0000 7fdac8a00640 1 -- 172.21.15.32:0/306794170 shutdown_connections 2024-04-30T08:37:01.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.249+0000 7fdac8a00640 1 --2- 172.21.15.32:0/306794170 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fdaa80afb60 0x7fdaa80b2070 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:01.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.249+0000 7fdac8a00640 1 --2- 172.21.15.32:0/306794170 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdac4109610 0x7fdac4102540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:01.254 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.249+0000 7fdac8a00640 1 --2- 172.21.15.32:0/306794170 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fdac4106ae0 0x7fdac4102000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:01.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.249+0000 7fdac8a00640 1 --2- 172.21.15.32:0/306794170 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fdac41040f0 0x7fdac4103960 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:01.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.249+0000 7fdac8a00640 1 -- 172.21.15.32:0/306794170 >> 172.21.15.32:0/306794170 conn(0x7fdac40fda70 msgr2=0x7fdac4106130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:01.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.249+0000 7fdac8a00640 1 -- 172.21.15.32:0/306794170 shutdown_connections 2024-04-30T08:37:01.255 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:01.249+0000 7fdac8a00640 1 -- 172.21.15.32:0/306794170 wait complete. 2024-04-30T08:37:01.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:01 smithi189 ceph-mon[33038]: pgmap v480: 129 pgs: 129 active+clean; 4.0 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 314 KiB/s wr, 50 op/s 2024-04-30T08:37:01.408 INFO:tasks.workunit.client.0.smithi032.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-04-30T08:37:01.408 INFO:tasks.workunit.client.0.smithi032.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-04-30T08:37:01.409 INFO:tasks.workunit.client.0.smithi032.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-04-30T08:37:01.409 INFO:tasks.workunit.client.0.smithi032.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-04-30T08:37:01.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:01 smithi071 ceph-mon[33385]: pgmap v480: 129 pgs: 129 active+clean; 4.0 MiB data, 377 MiB used, 1.0 TiB / 1.0 TiB avail; 241 KiB/s rd, 314 KiB/s wr, 50 op/s 2024-04-30T08:37:01.621 INFO:tasks.workunit.client.0.smithi032.stdout:/bin/sh ./config.status --recheck 2024-04-30T08:37:01.791 INFO:tasks.workunit.client.0.smithi032.stdout:running CONFIG_SHELL=/bin/sh /bin/sh ./configure --no-create --no-recursion 2024-04-30T08:37:02.071 INFO:tasks.workunit.client.0.smithi032.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-04-30T08:37:02.279 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-30T08:37:02.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-30T08:37:02.279 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-30T08:37:02.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.584+0000 7f485be00640 1 -- 172.21.15.32:0/4288669762 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f485c106df0 msgr2=0x7f485c107200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:02.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.584+0000 7f485be00640 1 --2- 172.21.15.32:0/4288669762 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f485c106df0 0x7f485c107200 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f4848009860 tx=0x7f484802a340 comp rx=0 tx=0).stop 2024-04-30T08:37:02.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.584+0000 7f485be00640 1 -- 172.21.15.32:0/4288669762 shutdown_connections 2024-04-30T08:37:02.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.584+0000 7f485be00640 1 --2- 172.21.15.32:0/4288669762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f485c10baf0 0x7f485c10df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.584+0000 7f485be00640 1 --2- 172.21.15.32:0/4288669762 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f485c106df0 0x7f485c107200 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.584+0000 7f485be00640 1 --2- 172.21.15.32:0/4288669762 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f485c105bb0 0x7f485c1068b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.584+0000 7f485be00640 1 -- 172.21.15.32:0/4288669762 >> 172.21.15.32:0/4288669762 conn(0x7f485c100da0 msgr2=0x7f485c103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:02.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.584+0000 7f485be00640 1 -- 172.21.15.32:0/4288669762 shutdown_connections 2024-04-30T08:37:02.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.584+0000 7f485be00640 1 -- 172.21.15.32:0/4288669762 wait complete. 2024-04-30T08:37:02.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.586+0000 7f485be00640 1 Processor -- start 2024-04-30T08:37:02.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.586+0000 7f485be00640 1 -- start start 2024-04-30T08:37:02.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.586+0000 7f485be00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f485c105bb0 0x7f485c19c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.586+0000 7f485be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f485c106df0 0x7f485c19c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.587+0000 7f485be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f485c10baf0 0x7f485c1a3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.587+0000 7f485be00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f485c1116e0 con 0x7f485c106df0 2024-04-30T08:37:02.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.587+0000 7f485be00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f485c111560 con 0x7f485c105bb0 2024-04-30T08:37:02.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.587+0000 7f485be00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f485c111860 con 0x7f485c10baf0 2024-04-30T08:37:02.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.587+0000 7f485a000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f485c106df0 0x7f485c19c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:02.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.587+0000 7f485aa00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f485c105bb0 0x7f485c19c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:02.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.587+0000 7f485a000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f485c106df0 0x7f485c19c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56638/0 (socket says 172.21.15.32:56638) 2024-04-30T08:37:02.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.587+0000 7f485a000640 1 -- 172.21.15.32:0/2172625039 learned_addr learned my addr 172.21.15.32:0/2172625039 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:02.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.587+0000 7f485b400640 1 --2- 172.21.15.32:0/2172625039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f485c10baf0 0x7f485c1a3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:02.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.587+0000 7f485aa00640 1 -- 172.21.15.32:0/2172625039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f485c10baf0 msgr2=0x7f485c1a3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:02.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.587+0000 7f485aa00640 1 --2- 172.21.15.32:0/2172625039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f485c10baf0 0x7f485c1a3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.587+0000 7f485aa00640 1 -- 172.21.15.32:0/2172625039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f485c106df0 msgr2=0x7f485c19c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:02.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.587+0000 7f485aa00640 1 --2- 172.21.15.32:0/2172625039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f485c106df0 0x7f485c19c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.587+0000 7f485aa00640 1 -- 172.21.15.32:0/2172625039 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f485c1a3e00 con 0x7f485c105bb0 2024-04-30T08:37:02.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.588+0000 7f485aa00640 1 --2- 172.21.15.32:0/2172625039 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f485c105bb0 0x7f485c19c2c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f485c105480 tx=0x7f484c00ee50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:02.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.588+0000 7f4853400640 1 -- 172.21.15.32:0/2172625039 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f484c00cc40 con 0x7f485c105bb0 2024-04-30T08:37:02.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.588+0000 7f4853400640 1 -- 172.21.15.32:0/2172625039 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f484c00cde0 con 0x7f485c105bb0 2024-04-30T08:37:02.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.588+0000 7f485be00640 1 -- 172.21.15.32:0/2172625039 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f485c1a40f0 con 0x7f485c105bb0 2024-04-30T08:37:02.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.588+0000 7f4853400640 1 -- 172.21.15.32:0/2172625039 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f484c010430 con 0x7f485c105bb0 2024-04-30T08:37:02.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.588+0000 7f485be00640 1 -- 172.21.15.32:0/2172625039 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f485c1a4640 con 0x7f485c105bb0 2024-04-30T08:37:02.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.592+0000 7f4868a20740 1 Processor -- start 2024-04-30T08:37:02.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.592+0000 7f4868a20740 1 -- start start 2024-04-30T08:37:02.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.592+0000 7f4853400640 1 -- 172.21.15.32:0/2172625039 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f484c0105d0 con 0x7f485c105bb0 2024-04-30T08:37:02.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.592+0000 7f4868a20740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590ccc28410 0x5590ccfca3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.592+0000 7f4868a20740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5590ccbff550 0x5590ccfca950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.592+0000 7f4868a20740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5590ccc07730 0x5590ccfd1850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.592+0000 7f4868a20740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5590ccf2eb50 con 0x5590ccc07730 2024-04-30T08:37:02.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.592+0000 7f4868a20740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x5590cced4080 con 0x5590ccbff550 2024-04-30T08:37:02.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.592+0000 7f4868a20740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5590ccf2ab10 con 0x5590ccc28410 2024-04-30T08:37:02.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.593+0000 7f485a000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5590ccc07730 0x5590ccfd1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:02.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.593+0000 7f485aa00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5590ccbff550 0x5590ccfca950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:02.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.593+0000 7f485b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590ccc28410 0x5590ccfca3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:02.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.593+0000 7f485aa00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5590ccbff550 0x5590ccfca950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:35852/0 (socket says 172.21.15.32:35852) 2024-04-30T08:37:02.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.593+0000 7f485a000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5590ccc07730 0x5590ccfd1850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56652/0 (socket says 172.21.15.32:56652) 2024-04-30T08:37:02.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.593+0000 7f485aa00640 1 -- 172.21.15.32:0/1907582472 learned_addr learned my addr 172.21.15.32:0/1907582472 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:02.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.593+0000 7f485b400640 1 -- 172.21.15.32:0/1907582472 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5590ccbff550 msgr2=0x5590ccfca950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:02.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.593+0000 7f485b400640 1 --2- 172.21.15.32:0/1907582472 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5590ccbff550 0x5590ccfca950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.595 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.593+0000 7f485b400640 1 -- 172.21.15.32:0/1907582472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5590ccc07730 msgr2=0x5590ccfd1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:02.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.593+0000 7f485b400640 1 --2- 172.21.15.32:0/1907582472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5590ccc07730 0x5590ccfd1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.593+0000 7f485b400640 1 -- 172.21.15.32:0/1907582472 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590ccef40c0 con 0x5590ccc28410 2024-04-30T08:37:02.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.593+0000 7f485a000640 1 --2- 172.21.15.32:0/1907582472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5590ccc07730 0x5590ccfd1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:37:02.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.594+0000 7f485b400640 1 --2- 172.21.15.32:0/1907582472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590ccc28410 0x5590ccfca3d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f485400e450 tx=0x7f48540095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a96b9841080eb8d server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:02.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.594+0000 7f4847e00640 1 -- 172.21.15.32:0/1907582472 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4854004b20 con 0x5590ccc28410 2024-04-30T08:37:02.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.594+0000 7f4847e00640 1 -- 172.21.15.32:0/1907582472 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4854004cc0 con 0x5590ccc28410 2024-04-30T08:37:02.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.594+0000 7f4847e00640 1 -- 172.21.15.32:0/1907582472 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48540055a0 con 0x5590ccc28410 2024-04-30T08:37:02.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.594+0000 7f4853400640 1 --2- 172.21.15.32:0/2172625039 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f48400afae0 0x7f48400b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.594+0000 7f4853400640 1 -- 172.21.15.32:0/2172625039 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f484c0df3e0 con 0x7f485c105bb0 2024-04-30T08:37:02.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.595+0000 7f485aa00640 1 --2- 172.21.15.32:0/2172625039 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f48400afae0 0x7f48400b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:02.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.595+0000 7f4868a20740 1 -- 172.21.15.32:0/1907582472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590ccc28410 msgr2=0x5590ccfca3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:02.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.595+0000 7f4868a20740 1 --2- 172.21.15.32:0/1907582472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590ccc28410 0x5590ccfca3d0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f485400e450 tx=0x7f48540095c0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.595+0000 7f4868a20740 1 -- 172.21.15.32:0/1907582472 shutdown_connections 2024-04-30T08:37:02.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.595+0000 7f4868a20740 1 --2- 172.21.15.32:0/1907582472 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5590ccc07730 0x5590ccfd1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.595+0000 7f4868a20740 1 --2- 172.21.15.32:0/1907582472 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5590ccbff550 0x5590ccfca950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.595+0000 7f4868a20740 1 --2- 172.21.15.32:0/1907582472 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590ccc28410 0x5590ccfca3d0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.595+0000 7f4868a20740 1 -- 172.21.15.32:0/1907582472 >> 172.21.15.32:0/1907582472 conn(0x5590ccc1c560 msgr2=0x5590ccfc5d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:02.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.595+0000 7f485aa00640 1 --2- 172.21.15.32:0/2172625039 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f48400afae0 0x7f48400b1ff0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f484c003970 tx=0x7f484c0e2ed0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:02.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.595+0000 7f4868a20740 1 -- 172.21.15.32:0/1907582472 shutdown_connections 2024-04-30T08:37:02.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.595+0000 7f4868a20740 1 -- 172.21.15.32:0/1907582472 wait complete. 2024-04-30T08:37:02.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.596+0000 7f4868a20740 1 -- start start 2024-04-30T08:37:02.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.596+0000 7f4868a20740 1 Processor -- start 2024-04-30T08:37:02.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.597+0000 7f4868a20740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590ccbff550 0x5590ccfd3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.597+0000 7f4868a20740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5590ccc07730 0x5590ccfd4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.597+0000 7f4868a20740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5590ccc28410 0x5590ccfdb3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.597+0000 7f4868a20740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5590ccfd22a0 con 0x5590ccc07730 2024-04-30T08:37:02.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.597+0000 7f4868a20740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x5590ccee7b20 con 0x5590ccc28410 2024-04-30T08:37:02.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.597+0000 7f4868a20740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5590ccfd2420 con 0x5590ccbff550 2024-04-30T08:37:02.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.597+0000 7f4868a20740 10 client.0 _ll_register_callbacks cb 0x5590ccf25a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:02.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.597+0000 7f485b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590ccbff550 0x5590ccfd3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:02.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.597+0000 7f485a000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5590ccc07730 0x5590ccfd4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:02.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.597+0000 7f485b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590ccbff550 0x5590ccfd3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:37086/0 (socket says 172.21.15.32:37086) 2024-04-30T08:37:02.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.597+0000 7f485b400640 1 -- 172.21.15.32:0/219777748 learned_addr learned my addr 172.21.15.32:0/219777748 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:02.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.597+0000 7f485b400640 1 --2- 172.21.15.32:0/219777748 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5590ccc28410 0x5590ccfdb3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:02.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.598+0000 7f485a000640 1 -- 172.21.15.32:0/219777748 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590ccbff550 msgr2=0x5590ccfd3f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:02.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.598+0000 7f485a000640 1 --2- 172.21.15.32:0/219777748 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590ccbff550 0x5590ccfd3f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.598+0000 7f485a000640 1 -- 172.21.15.32:0/219777748 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5590ccc28410 msgr2=0x5590ccfdb3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:02.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.598+0000 7f485a000640 1 --2- 172.21.15.32:0/219777748 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5590ccc28410 0x5590ccfdb3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.598+0000 7f485a000640 1 -- 172.21.15.32:0/219777748 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590ccd087e0 con 0x5590ccc07730 2024-04-30T08:37:02.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.598+0000 7f485a000640 1 --2- 172.21.15.32:0/219777748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5590ccc07730 0x5590ccfd4470 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4848000c00 tx=0x7f4848005d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e83731b65e6840f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:02.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.598+0000 7f4844c00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:37:02.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.598+0000 7f4844c00640 1 -- 172.21.15.32:0/219777748 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4848037040 con 0x5590ccc07730 2024-04-30T08:37:02.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.598+0000 7f4844c00640 1 -- 172.21.15.32:0/219777748 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f4848004040 con 0x5590ccc07730 2024-04-30T08:37:02.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.598+0000 7f4868a20740 1 -- 172.21.15.32:0/219777748 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4854004cc0 con 0x5590ccc07730 2024-04-30T08:37:02.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.599+0000 7f4844c00640 1 -- 172.21.15.32:0/219777748 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f484804c050 con 0x5590ccc07730 2024-04-30T08:37:02.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.599+0000 7f4868a20740 10 client.16050 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:02.602 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.599+0000 7f4868a20740 1 -- 172.21.15.32:0/219777748 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5590ccd68f70 con 0x5590ccc07730 2024-04-30T08:37:02.603 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.599+0000 7f4844c00640 1 -- 172.21.15.32:0/219777748 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4854004cc0 con 0x5590ccc07730 2024-04-30T08:37:02.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.599+0000 7f4844c00640 1 -- 172.21.15.32:0/219777748 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7f484803d480 con 0x5590ccc07730 2024-04-30T08:37:02.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.599+0000 7f4868a20740 10 client.16050 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:02.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.599+0000 7f4868a20740 10 client.16050 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:37:02.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.599+0000 7f4868a20740 20 client.16050 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:02.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.599+0000 7f4868a20740 1 --2- 172.21.15.32:0/219777748 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x5590ccbe71f0 0x5590ccfe1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.599+0000 7f4868a20740 4 client.16050 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:37:02.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.599+0000 7f4868a20740 1 -- 172.21.15.32:0/219777748 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5590ccf3c760 con 0x5590ccbe71f0 2024-04-30T08:37:02.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.600+0000 7f485b400640 1 --2- 172.21.15.32:0/219777748 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x5590ccbe71f0 0x5590ccfe1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:02.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.600+0000 7f485b400640 1 --2- 172.21.15.32:0/219777748 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x5590ccbe71f0 0x5590ccfe1910 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=848b29b906960dfa server_cookie=1bec49497bce2a0a in_seq=0 out_seq=0 2024-04-30T08:37:02.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.600+0000 7f4844c00640 10 client.16050 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:02.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.601+0000 7f4844c00640 1 -- 172.21.15.32:0/219777748 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5590ccf3c760 con 0x5590ccbe71f0 2024-04-30T08:37:02.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.601+0000 7f4844c00640 10 client.16050 handle_command_reply: tid=0 2024-04-30T08:37:02.604 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.601+0000 7f4868a20740 1 client.16050 shutdown 2024-04-30T08:37:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.602+0000 7f4868a20740 1 -- 172.21.15.32:0/219777748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5590ccc07730 msgr2=0x5590ccfd4470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.602+0000 7f4868a20740 1 --2- 172.21.15.32:0/219777748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5590ccc07730 0x5590ccfd4470 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f4848000c00 tx=0x7f4848005d10 comp rx=0 tx=0).stop 2024-04-30T08:37:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.602+0000 7f4868a20740 1 -- 172.21.15.32:0/219777748 shutdown_connections 2024-04-30T08:37:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.602+0000 7f4868a20740 1 --2- 172.21.15.32:0/219777748 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x5590ccbe71f0 0x5590ccfe1910 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.602+0000 7f4868a20740 1 --2- 172.21.15.32:0/219777748 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5590ccc28410 0x5590ccfdb3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.602+0000 7f4868a20740 1 --2- 172.21.15.32:0/219777748 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5590ccc07730 0x5590ccfd4470 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.602+0000 7f4868a20740 1 --2- 172.21.15.32:0/219777748 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590ccbff550 0x5590ccfd3f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.602+0000 7f4844c00640 0 client.16050 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.602+0000 7f4868a20740 1 -- 172.21.15.32:0/219777748 >> 172.21.15.32:0/219777748 conn(0x5590ccc1c560 msgr2=0x5590ccfc4d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.602+0000 7f4868a20740 1 -- 172.21.15.32:0/219777748 shutdown_connections 2024-04-30T08:37:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.602+0000 7f4868a20740 1 -- 172.21.15.32:0/219777748 wait complete. 2024-04-30T08:37:02.605 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.602+0000 7f4868a20740 20 client.16050 trim_cache size 0 max 16384 2024-04-30T08:37:02.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.622+0000 7f4868a20740 1 Processor -- start 2024-04-30T08:37:02.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.623+0000 7f4868a20740 1 -- start start 2024-04-30T08:37:02.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.623+0000 7f4868a20740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5590ccbff550 0x5590ccfcd980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.623 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.623+0000 7f4868a20740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590ccc07730 0x5590ccfdb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.623+0000 7f4868a20740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5590ccc28410 0x5590ccfcdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.623+0000 7f4868a20740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f485c1aeb90 con 0x5590ccbff550 2024-04-30T08:37:02.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.623+0000 7f4868a20740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f485c1b0760 con 0x5590ccc28410 2024-04-30T08:37:02.624 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.623+0000 7f4868a20740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f485c1af260 con 0x5590ccc07730 2024-04-30T08:37:02.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.623+0000 7f485b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5590ccbff550 0x5590ccfcd980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:02.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.623+0000 7f485b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5590ccbff550 0x5590ccfcd980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56662/0 (socket says 172.21.15.32:56662) 2024-04-30T08:37:02.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.623+0000 7f485b400640 1 -- 172.21.15.32:0/571165442 learned_addr learned my addr 172.21.15.32:0/571165442 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:02.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.623+0000 7f485a000640 1 --2- 172.21.15.32:0/571165442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590ccc07730 0x5590ccfdb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:02.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.624+0000 7f485b400640 1 --2- 172.21.15.32:0/571165442 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5590ccc28410 0x5590ccfcdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:02.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.624+0000 7f485b400640 1 -- 172.21.15.32:0/571165442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590ccc07730 msgr2=0x5590ccfdb2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:02.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.624+0000 7f485b400640 1 --2- 172.21.15.32:0/571165442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590ccc07730 0x5590ccfdb2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.624+0000 7f485b400640 1 -- 172.21.15.32:0/571165442 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5590ccc28410 msgr2=0x5590ccfcdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:02.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.624+0000 7f485b400640 1 --2- 172.21.15.32:0/571165442 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5590ccc28410 0x5590ccfcdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.624+0000 7f485b400640 1 -- 172.21.15.32:0/571165442 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5590ccc00170 con 0x5590ccbff550 2024-04-30T08:37:02.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.624+0000 7f485b400640 1 --2- 172.21.15.32:0/571165442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5590ccbff550 0x5590ccfcd980 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4854005060 tx=0x7f4854005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83b3aa73022a70e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:02.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.624+0000 7f4847400640 1 -- 172.21.15.32:0/571165442 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4854038070 con 0x5590ccbff550 2024-04-30T08:37:02.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.625+0000 7f4847400640 1 -- 172.21.15.32:0/571165442 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4854004c60 con 0x5590ccbff550 2024-04-30T08:37:02.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.625+0000 7f4847400640 1 -- 172.21.15.32:0/571165442 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f485400d7d0 con 0x5590ccbff550 2024-04-30T08:37:02.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.625+0000 7f4868a20740 1 -- 172.21.15.32:0/571165442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5590ccbff550 msgr2=0x5590ccfcd980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:02.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.626+0000 7f4868a20740 1 --2- 172.21.15.32:0/571165442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5590ccbff550 0x5590ccfcd980 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4854005060 tx=0x7f4854005bd0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.626+0000 7f4868a20740 1 -- 172.21.15.32:0/571165442 shutdown_connections 2024-04-30T08:37:02.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.626+0000 7f4868a20740 1 --2- 172.21.15.32:0/571165442 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5590ccc28410 0x5590ccfcdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.626+0000 7f4868a20740 1 --2- 172.21.15.32:0/571165442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590ccc07730 0x5590ccfdb2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.626+0000 7f4868a20740 1 --2- 172.21.15.32:0/571165442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5590ccbff550 0x5590ccfcd980 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.626+0000 7f4868a20740 1 -- 172.21.15.32:0/571165442 >> 172.21.15.32:0/571165442 conn(0x5590ccc1c560 msgr2=0x5590ccfc4d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:02.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.626+0000 7f4868a20740 1 -- 172.21.15.32:0/571165442 shutdown_connections 2024-04-30T08:37:02.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.626+0000 7f4868a20740 1 -- 172.21.15.32:0/571165442 wait complete. 2024-04-30T08:37:02.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.626+0000 7f4868a20740 1 -- start start 2024-04-30T08:37:02.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.626+0000 7f4868a20740 1 Processor -- start 2024-04-30T08:37:02.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.627+0000 7f4868a20740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5590ccbff550 0x5590ccfd9000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.627+0000 7f4868a20740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590ccc07730 0x5590ccfd9540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.627+0000 7f4868a20740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5590ccc28410 0x5590ccfda400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.627+0000 7f4868a20740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f485c1b0000 con 0x5590ccc28410 2024-04-30T08:37:02.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.627+0000 7f4868a20740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f485c1af930 con 0x5590ccbff550 2024-04-30T08:37:02.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.627+0000 7f4868a20740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f485c1acb50 con 0x5590ccc07730 2024-04-30T08:37:02.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.627+0000 7f4868a20740 10 client.0 _ll_register_callbacks cb 0x7f4854009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:02.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.627+0000 7f485b400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5590ccbff550 0x5590ccfd9000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:02.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.627+0000 7f485a000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590ccc07730 0x5590ccfd9540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:02.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.627+0000 7f485b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5590ccc28410 0x5590ccfda400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:02.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.627+0000 7f485a000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590ccc07730 0x5590ccfd9540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:37096/0 (socket says 172.21.15.32:37096) 2024-04-30T08:37:02.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.627+0000 7f485a000640 1 -- 172.21.15.32:0/2208967543 learned_addr learned my addr 172.21.15.32:0/2208967543 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:02.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.628+0000 7f485a000640 1 -- 172.21.15.32:0/2208967543 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5590ccbff550 msgr2=0x5590ccfd9000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:02.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.628+0000 7f485a000640 1 --2- 172.21.15.32:0/2208967543 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5590ccbff550 0x5590ccfd9000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.628+0000 7f485a000640 1 -- 172.21.15.32:0/2208967543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5590ccc28410 msgr2=0x5590ccfda400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:02.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.628+0000 7f485a000640 1 --2- 172.21.15.32:0/2208967543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5590ccc28410 0x5590ccfda400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.628+0000 7f485a000640 1 -- 172.21.15.32:0/2208967543 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5590ccfda9c0 con 0x5590ccc07730 2024-04-30T08:37:02.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.628+0000 7f485b400640 1 --2- 172.21.15.32:0/2208967543 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5590ccbff550 0x5590ccfd9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:37:02.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.628+0000 7f485a000640 1 --2- 172.21.15.32:0/2208967543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590ccc07730 0x5590ccfd9540 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f484802c010 tx=0x7f48480058c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f84c347c44e6b251 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:02.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.628+0000 7f4844c00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-30T08:37:02.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.628+0000 7f4844c00640 1 -- 172.21.15.32:0/2208967543 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4848054030 con 0x5590ccc07730 2024-04-30T08:37:02.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.628+0000 7f4844c00640 1 -- 172.21.15.32:0/2208967543 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f484803db10 con 0x5590ccc07730 2024-04-30T08:37:02.631 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.628+0000 7f4868a20740 1 -- 172.21.15.32:0/2208967543 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4854004c60 con 0x5590ccc07730 2024-04-30T08:37:02.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.628+0000 7f4844c00640 1 -- 172.21.15.32:0/2208967543 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f484805e050 con 0x5590ccc07730 2024-04-30T08:37:02.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.629+0000 7f4868a20740 10 client.26188 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:02.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.629+0000 7f4868a20740 1 -- 172.21.15.32:0/2208967543 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5590cd04a140 con 0x5590ccc07730 2024-04-30T08:37:02.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.629+0000 7f4844c00640 1 -- 172.21.15.32:0/2208967543 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4854004c60 con 0x5590ccc07730 2024-04-30T08:37:02.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.629+0000 7f4844c00640 1 -- 172.21.15.32:0/2208967543 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7f48480047e0 con 0x5590ccc07730 2024-04-30T08:37:02.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.629+0000 7f4868a20740 10 client.26188 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:02.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.629+0000 7f4868a20740 10 client.26188 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:37:02.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.629+0000 7f4868a20740 20 client.26188 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:02.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.629+0000 7f4868a20740 1 --2- 172.21.15.32:0/2208967543 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x5590ccbbc630 0x5590cd04cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.629+0000 7f4868a20740 4 client.26188 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-30T08:37:02.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.629+0000 7f4868a20740 1 -- 172.21.15.32:0/2208967543 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5590ccf365d0 con 0x5590ccbbc630 2024-04-30T08:37:02.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.629+0000 7f485b400640 1 --2- 172.21.15.32:0/2208967543 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x5590ccbbc630 0x5590cd04cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:02.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.630+0000 7f4844c00640 10 client.26188 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:02.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.630+0000 7f485b400640 1 --2- 172.21.15.32:0/2208967543 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x5590ccbbc630 0x5590cd04cbf0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f523705e7ec2a467 server_cookie=146b7cfee808f54f in_seq=0 out_seq=0 2024-04-30T08:37:02.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.631+0000 7f4844c00640 1 -- 172.21.15.32:0/2208967543 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5590ccf365d0 con 0x5590ccbbc630 2024-04-30T08:37:02.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.631+0000 7f4844c00640 10 client.26188 handle_command_reply: tid=0 2024-04-30T08:37:02.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.631+0000 7f4868a20740 1 client.26188 shutdown 2024-04-30T08:37:02.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.631+0000 7f4868a20740 1 -- 172.21.15.32:0/2208967543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590ccc07730 msgr2=0x5590ccfd9540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:02.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.631+0000 7f4868a20740 1 --2- 172.21.15.32:0/2208967543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590ccc07730 0x5590ccfd9540 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f484802c010 tx=0x7f48480058c0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.631+0000 7f4868a20740 1 -- 172.21.15.32:0/2208967543 shutdown_connections 2024-04-30T08:37:02.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.631+0000 7f4868a20740 1 --2- 172.21.15.32:0/2208967543 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x5590ccbbc630 0x5590cd04cbf0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.631+0000 7f4868a20740 1 --2- 172.21.15.32:0/2208967543 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5590ccc28410 0x5590ccfda400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.631+0000 7f4868a20740 1 --2- 172.21.15.32:0/2208967543 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590ccc07730 0x5590ccfd9540 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.631+0000 7f4868a20740 1 --2- 172.21.15.32:0/2208967543 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5590ccbff550 0x5590ccfd9000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.631+0000 7f4868a20740 1 -- 172.21.15.32:0/2208967543 >> 172.21.15.32:0/2208967543 conn(0x5590ccc1c560 msgr2=0x5590ccfc4d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:02.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.631+0000 7f4844c00640 0 client.26188 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:02.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.632+0000 7f4868a20740 1 -- 172.21.15.32:0/2208967543 shutdown_connections 2024-04-30T08:37:02.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.632+0000 7f4868a20740 1 -- 172.21.15.32:0/2208967543 wait complete. 2024-04-30T08:37:02.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.632+0000 7f4868a20740 20 client.26188 trim_cache size 0 max 16384 2024-04-30T08:37:02.635 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-30T08:37:02.635 INFO:teuthology.orchestra.run.smithi032.stdout: "return_code": 0, 2024-04-30T08:37:02.635 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_tag": "b09a0368-b6b9-4e13-b89f-87f44fecab4a", 2024-04-30T08:37:02.635 INFO:teuthology.orchestra.run.smithi032.stdout: "mode": "asynchronous" 2024-04-30T08:37:02.635 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-30T08:37:02.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.633+0000 7f485be00640 1 -- 172.21.15.32:0/2172625039 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f48400afae0 msgr2=0x7f48400b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:02.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.633+0000 7f485be00640 1 --2- 172.21.15.32:0/2172625039 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f48400afae0 0x7f48400b1ff0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f484c003970 tx=0x7f484c0e2ed0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.633+0000 7f485be00640 1 -- 172.21.15.32:0/2172625039 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f485c105bb0 msgr2=0x7f485c19c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:02.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.633+0000 7f485be00640 1 --2- 172.21.15.32:0/2172625039 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f485c105bb0 0x7f485c19c2c0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f485c105480 tx=0x7f484c00ee50 comp rx=0 tx=0).stop 2024-04-30T08:37:02.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.633+0000 7f485be00640 1 -- 172.21.15.32:0/2172625039 shutdown_connections 2024-04-30T08:37:02.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.633+0000 7f485be00640 1 --2- 172.21.15.32:0/2172625039 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f48400afae0 0x7f48400b1ff0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.633+0000 7f485be00640 1 --2- 172.21.15.32:0/2172625039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f485c10baf0 0x7f485c1a3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.633+0000 7f485be00640 1 --2- 172.21.15.32:0/2172625039 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f485c106df0 0x7f485c19c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.633+0000 7f485be00640 1 --2- 172.21.15.32:0/2172625039 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f485c105bb0 0x7f485c19c2c0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.633+0000 7f485be00640 1 -- 172.21.15.32:0/2172625039 >> 172.21.15.32:0/2172625039 conn(0x7f485c100da0 msgr2=0x7f485c107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:02.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.633+0000 7f485be00640 1 -- 172.21.15.32:0/2172625039 shutdown_connections 2024-04-30T08:37:02.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.633+0000 7f485be00640 1 -- 172.21.15.32:0/2172625039 wait complete. 2024-04-30T08:37:02.663 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-30T08:37:02.663 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-30T08:37:02.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.948+0000 7fa59c200640 1 -- 172.21.15.32:0/4118208781 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa594076a70 msgr2=0x7fa59410fd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:02.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.948+0000 7fa59c200640 1 --2- 172.21.15.32:0/4118208781 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa594076a70 0x7fa59410fd90 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa580009860 tx=0x7fa58002a340 comp rx=0 tx=0).stop 2024-04-30T08:37:02.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.948+0000 7fa59c200640 1 -- 172.21.15.32:0/4118208781 shutdown_connections 2024-04-30T08:37:02.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.948+0000 7fa59c200640 1 --2- 172.21.15.32:0/4118208781 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa5941102d0 0x7fa5941126f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.948+0000 7fa59c200640 1 --2- 172.21.15.32:0/4118208781 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa594076a70 0x7fa59410fd90 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.948+0000 7fa59c200640 1 --2- 172.21.15.32:0/4118208781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa594076120 0x7fa594076530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.948+0000 7fa59c200640 1 -- 172.21.15.32:0/4118208781 >> 172.21.15.32:0/4118208781 conn(0x7fa5940fed60 msgr2=0x7fa594101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:02.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.948+0000 7fa59c200640 1 -- 172.21.15.32:0/4118208781 shutdown_connections 2024-04-30T08:37:02.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.948+0000 7fa59c200640 1 -- 172.21.15.32:0/4118208781 wait complete. 2024-04-30T08:37:02.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.949+0000 7fa59c200640 1 Processor -- start 2024-04-30T08:37:02.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.950+0000 7fa59c200640 1 -- start start 2024-04-30T08:37:02.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.950+0000 7fa59c200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa594076120 0x7fa5941a0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.950+0000 7fa59c200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa594076a70 0x7fa5941a0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.950+0000 7fa59c200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5941102d0 0x7fa5941a7d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.950+0000 7fa59c200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa594115cb0 con 0x7fa5941102d0 2024-04-30T08:37:02.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.950+0000 7fa59c200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa594115b30 con 0x7fa594076a70 2024-04-30T08:37:02.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.950+0000 7fa59c200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa594115e30 con 0x7fa594076120 2024-04-30T08:37:02.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.950+0000 7fa599800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa594076120 0x7fa5941a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:02.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.950+0000 7fa599800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa594076120 0x7fa5941a0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:37114/0 (socket says 172.21.15.32:37114) 2024-04-30T08:37:02.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.950+0000 7fa599800640 1 -- 172.21.15.32:0/1598232236 learned_addr learned my addr 172.21.15.32:0/1598232236 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:02.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.951+0000 7fa598e00640 1 --2- 172.21.15.32:0/1598232236 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa594076a70 0x7fa5941a0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:02.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.951+0000 7fa59a200640 1 --2- 172.21.15.32:0/1598232236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5941102d0 0x7fa5941a7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:02.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.951+0000 7fa599800640 1 -- 172.21.15.32:0/1598232236 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa594076a70 msgr2=0x7fa5941a0db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:02.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.951+0000 7fa599800640 1 --2- 172.21.15.32:0/1598232236 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa594076a70 0x7fa5941a0db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.951+0000 7fa599800640 1 -- 172.21.15.32:0/1598232236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5941102d0 msgr2=0x7fa5941a7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:02.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.951+0000 7fa599800640 1 --2- 172.21.15.32:0/1598232236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5941102d0 0x7fa5941a7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.951+0000 7fa599800640 1 -- 172.21.15.32:0/1598232236 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5941a83b0 con 0x7fa594076120 2024-04-30T08:37:02.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.951+0000 7fa599800640 1 --2- 172.21.15.32:0/1598232236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa594076120 0x7fa5941a0870 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa58400c3f0 tx=0x7fa584009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:02.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.951+0000 7fa592000640 1 -- 172.21.15.32:0/1598232236 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa584003970 con 0x7fa594076120 2024-04-30T08:37:02.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.952+0000 7fa59c200640 1 -- 172.21.15.32:0/1598232236 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5941a86a0 con 0x7fa594076120 2024-04-30T08:37:02.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.952+0000 7fa592000640 1 -- 172.21.15.32:0/1598232236 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa584004020 con 0x7fa594076120 2024-04-30T08:37:02.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.952+0000 7fa592000640 1 -- 172.21.15.32:0/1598232236 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa58400ba90 con 0x7fa594076120 2024-04-30T08:37:02.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.952+0000 7fa59c200640 1 -- 172.21.15.32:0/1598232236 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5941a8bf0 con 0x7fa594076120 2024-04-30T08:37:02.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.956+0000 7fa5a3221740 1 Processor -- start 2024-04-30T08:37:02.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.956+0000 7fa5a3221740 1 -- start start 2024-04-30T08:37:02.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.957+0000 7fa5a3221740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620a057390 0x55620a3f8240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.957+0000 7fa5a3221740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55620a02e4d0 0x55620a3f87c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.957+0000 7fa5a3221740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55620a0366b0 0x55620a3ff6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.957+0000 7fa5a3221740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55620a35dd50 con 0x55620a057390 2024-04-30T08:37:02.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.957+0000 7fa5a3221740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55620a302ef0 con 0x55620a0366b0 2024-04-30T08:37:02.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.957+0000 7fa5a3221740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55620a359d10 con 0x55620a02e4d0 2024-04-30T08:37:02.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.957+0000 7fa59a200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620a057390 0x55620a3f8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:02.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.957+0000 7fa592000640 1 -- 172.21.15.32:0/1598232236 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7fa58401d020 con 0x7fa594076120 2024-04-30T08:37:02.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.957+0000 7fa59a200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620a057390 0x55620a3f8240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56708/0 (socket says 172.21.15.32:56708) 2024-04-30T08:37:02.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.957+0000 7fa59a200640 1 -- 172.21.15.32:0/789933790 learned_addr learned my addr 172.21.15.32:0/789933790 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:02.958 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.957+0000 7fa598e00640 1 --2- 172.21.15.32:0/789933790 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55620a0366b0 0x55620a3ff6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:02.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.958+0000 7fa598e00640 1 -- 172.21.15.32:0/789933790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55620a02e4d0 msgr2=0x55620a3f87c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:02.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.958+0000 7fa598e00640 1 --2- 172.21.15.32:0/789933790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55620a02e4d0 0x55620a3f87c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.958+0000 7fa598e00640 1 -- 172.21.15.32:0/789933790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620a057390 msgr2=0x55620a3f8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:02.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.958+0000 7fa598e00640 1 --2- 172.21.15.32:0/789933790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620a057390 0x55620a3f8240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.958+0000 7fa598e00640 1 -- 172.21.15.32:0/789933790 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55620a1397d0 con 0x55620a0366b0 2024-04-30T08:37:02.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.958+0000 7fa598e00640 1 --2- 172.21.15.32:0/789933790 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55620a0366b0 0x55620a3ff6c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa5941a1910 tx=0x7fa580005cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5356aae970844014 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:02.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.958+0000 7fa58aa00640 1 -- 172.21.15.32:0/789933790 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa580041070 con 0x55620a0366b0 2024-04-30T08:37:02.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.958+0000 7fa58aa00640 1 -- 172.21.15.32:0/789933790 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa580030e10 con 0x55620a0366b0 2024-04-30T08:37:02.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.958+0000 7fa58aa00640 1 -- 172.21.15.32:0/789933790 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa580037040 con 0x55620a0366b0 2024-04-30T08:37:02.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.959+0000 7fa592000640 1 --2- 172.21.15.32:0/1598232236 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa5780afaf0 0x7fa5780b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.959+0000 7fa592000640 1 -- 172.21.15.32:0/1598232236 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fa5840def30 con 0x7fa594076120 2024-04-30T08:37:02.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.959+0000 7fa59a200640 1 --2- 172.21.15.32:0/1598232236 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa5780afaf0 0x7fa5780b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:02.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.959+0000 7fa5a3221740 1 -- 172.21.15.32:0/789933790 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55620a0366b0 msgr2=0x55620a3ff6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:02.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.959+0000 7fa5a3221740 1 --2- 172.21.15.32:0/789933790 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55620a0366b0 0x55620a3ff6c0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fa5941a1910 tx=0x7fa580005cf0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.959+0000 7fa5a3221740 1 -- 172.21.15.32:0/789933790 shutdown_connections 2024-04-30T08:37:02.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.959+0000 7fa5a3221740 1 --2- 172.21.15.32:0/789933790 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55620a0366b0 0x55620a3ff6c0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.961 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.959+0000 7fa5a3221740 1 --2- 172.21.15.32:0/789933790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55620a02e4d0 0x55620a3f87c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.959+0000 7fa5a3221740 1 --2- 172.21.15.32:0/789933790 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620a057390 0x55620a3f8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.959+0000 7fa5a3221740 1 -- 172.21.15.32:0/789933790 >> 172.21.15.32:0/789933790 conn(0x55620a04b4e0 msgr2=0x55620a3f3bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:02.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.960+0000 7fa5a3221740 1 -- 172.21.15.32:0/789933790 shutdown_connections 2024-04-30T08:37:02.962 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.960+0000 7fa5a3221740 1 -- 172.21.15.32:0/789933790 wait complete. 2024-04-30T08:37:02.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.960+0000 7fa59a200640 1 --2- 172.21.15.32:0/1598232236 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa5780afaf0 0x7fa5780b2000 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x55620a3f9410 tx=0x7fa58c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:02.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.960+0000 7fa5a3221740 1 -- start start 2024-04-30T08:37:02.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.961+0000 7fa5a3221740 1 Processor -- start 2024-04-30T08:37:02.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.961+0000 7fa5a3221740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620a02e4d0 0x55620a4021a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.961+0000 7fa5a3221740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55620a0366b0 0x55620a4026e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.961+0000 7fa5a3221740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55620a057390 0x55620a409640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.961+0000 7fa5a3221740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55620a4000b0 con 0x55620a02e4d0 2024-04-30T08:37:02.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.961+0000 7fa5a3221740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55620a316990 con 0x55620a0366b0 2024-04-30T08:37:02.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.961+0000 7fa5a3221740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55620a400230 con 0x55620a057390 2024-04-30T08:37:02.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.961+0000 7fa5a3221740 10 client.0 _ll_register_callbacks cb 0x55620a315110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:02.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.961+0000 7fa599800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620a02e4d0 0x55620a4021a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:02.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.961+0000 7fa599800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620a02e4d0 0x55620a4021a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56714/0 (socket says 172.21.15.32:56714) 2024-04-30T08:37:02.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.961+0000 7fa599800640 1 -- 172.21.15.32:0/3089204436 learned_addr learned my addr 172.21.15.32:0/3089204436 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:02.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.961+0000 7fa598e00640 1 --2- 172.21.15.32:0/3089204436 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55620a0366b0 0x55620a4026e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:02.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.961+0000 7fa59a200640 1 --2- 172.21.15.32:0/3089204436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55620a057390 0x55620a409640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:02.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.962+0000 7fa599800640 1 -- 172.21.15.32:0/3089204436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55620a057390 msgr2=0x55620a409640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:02.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.962+0000 7fa599800640 1 --2- 172.21.15.32:0/3089204436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55620a057390 0x55620a409640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.962+0000 7fa599800640 1 -- 172.21.15.32:0/3089204436 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55620a0366b0 msgr2=0x55620a4026e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:02.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.962+0000 7fa599800640 1 --2- 172.21.15.32:0/3089204436 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55620a0366b0 0x55620a4026e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.962+0000 7fa599800640 1 -- 172.21.15.32:0/3089204436 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55620a06f3f0 con 0x55620a02e4d0 2024-04-30T08:37:02.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.962+0000 7fa599800640 1 --2- 172.21.15.32:0/3089204436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620a02e4d0 0x55620a4021a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa5840a6fd0 tx=0x7fa5840a7740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b72384679d080576 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:02.966 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.962+0000 7fa57f400640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:37:02.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.962+0000 7fa57f400640 1 -- 172.21.15.32:0/3089204436 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5840afea0 con 0x55620a02e4d0 2024-04-30T08:37:02.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.962+0000 7fa57f400640 1 -- 172.21.15.32:0/3089204436 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fa5840041d0 con 0x55620a02e4d0 2024-04-30T08:37:02.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.962+0000 7fa5a3221740 1 -- 172.21.15.32:0/3089204436 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa580030e10 con 0x55620a02e4d0 2024-04-30T08:37:02.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.963+0000 7fa57f400640 1 -- 172.21.15.32:0/3089204436 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa5840098a0 con 0x55620a02e4d0 2024-04-30T08:37:02.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.963+0000 7fa5a3221740 10 client.16068 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:02.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.963+0000 7fa5a3221740 1 -- 172.21.15.32:0/3089204436 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55620a02f0f0 con 0x55620a02e4d0 2024-04-30T08:37:02.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.963+0000 7fa57f400640 1 -- 172.21.15.32:0/3089204436 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa580030e10 con 0x55620a02e4d0 2024-04-30T08:37:02.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.963+0000 7fa57f400640 1 -- 172.21.15.32:0/3089204436 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7fa5840c9040 con 0x55620a02e4d0 2024-04-30T08:37:02.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.963+0000 7fa5a3221740 10 client.16068 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:02.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.963+0000 7fa5a3221740 10 client.16068 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:37:02.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.963+0000 7fa5a3221740 20 client.16068 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:02.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.963+0000 7fa5a3221740 1 --2- 172.21.15.32:0/3089204436 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55620a015f60 0x55620a40faa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.967 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.963+0000 7fa5a3221740 4 client.16068 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:37:02.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.963+0000 7fa5a3221740 1 -- 172.21.15.32:0/3089204436 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55620a371ef0 con 0x55620a015f60 2024-04-30T08:37:02.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.964+0000 7fa598e00640 1 --2- 172.21.15.32:0/3089204436 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55620a015f60 0x55620a40faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:02.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.964+0000 7fa598e00640 1 --2- 172.21.15.32:0/3089204436 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55620a015f60 0x55620a40faa0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=363cb78d3f7b3256 server_cookie=12b0992dc56feb2 in_seq=0 out_seq=0 2024-04-30T08:37:02.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.964+0000 7fa57f400640 10 client.16068 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:02.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.965+0000 7fa57f400640 1 -- 172.21.15.32:0/3089204436 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55620a371ef0 con 0x55620a015f60 2024-04-30T08:37:02.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.965+0000 7fa57f400640 10 client.16068 handle_command_reply: tid=0 2024-04-30T08:37:02.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.965+0000 7fa5a3221740 1 client.16068 shutdown 2024-04-30T08:37:02.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.965+0000 7fa5a3221740 1 -- 172.21.15.32:0/3089204436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620a02e4d0 msgr2=0x55620a4021a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:02.969 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.965+0000 7fa5a3221740 1 --2- 172.21.15.32:0/3089204436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620a02e4d0 0x55620a4021a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa5840a6fd0 tx=0x7fa5840a7740 comp rx=0 tx=0).stop 2024-04-30T08:37:02.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.965+0000 7fa5a3221740 1 -- 172.21.15.32:0/3089204436 shutdown_connections 2024-04-30T08:37:02.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.965+0000 7fa5a3221740 1 --2- 172.21.15.32:0/3089204436 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55620a015f60 0x55620a40faa0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.965+0000 7fa5a3221740 1 --2- 172.21.15.32:0/3089204436 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55620a057390 0x55620a409640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.965+0000 7fa57f400640 0 client.16068 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:02.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.965+0000 7fa5a3221740 1 --2- 172.21.15.32:0/3089204436 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55620a0366b0 0x55620a4026e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.965+0000 7fa5a3221740 1 --2- 172.21.15.32:0/3089204436 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620a02e4d0 0x55620a4021a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.965+0000 7fa5a3221740 1 -- 172.21.15.32:0/3089204436 >> 172.21.15.32:0/3089204436 conn(0x55620a04b4e0 msgr2=0x55620a3f2bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:02.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.966+0000 7fa5a3221740 1 -- 172.21.15.32:0/3089204436 shutdown_connections 2024-04-30T08:37:02.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.966+0000 7fa5a3221740 1 -- 172.21.15.32:0/3089204436 wait complete. 2024-04-30T08:37:02.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.966+0000 7fa5a3221740 20 client.16068 trim_cache size 0 max 16384 2024-04-30T08:37:02.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.986+0000 7fa5a3221740 1 Processor -- start 2024-04-30T08:37:02.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.986+0000 7fa5a3221740 1 -- start start 2024-04-30T08:37:02.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.986+0000 7fa5a3221740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55620a02e4d0 0x55620a3fcce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.986+0000 7fa5a3221740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620a0366b0 0x55620a3fb920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.986+0000 7fa5a3221740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55620a057390 0x55620a402480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.986+0000 7fa5a3221740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa5941b30b0 con 0x55620a0366b0 2024-04-30T08:37:02.987 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.986+0000 7fa5a3221740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa5941b4c80 con 0x55620a057390 2024-04-30T08:37:02.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.986+0000 7fa5a3221740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa5941b3780 con 0x55620a02e4d0 2024-04-30T08:37:02.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.987+0000 7fa59a200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55620a057390 0x55620a402480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:02.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.987+0000 7fa59a200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55620a057390 0x55620a402480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:35918/0 (socket says 172.21.15.32:35918) 2024-04-30T08:37:02.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.987+0000 7fa598e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620a0366b0 0x55620a3fb920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:02.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.987+0000 7fa59a200640 1 -- 172.21.15.32:0/3902923486 learned_addr learned my addr 172.21.15.32:0/3902923486 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:02.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.987+0000 7fa599800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55620a02e4d0 0x55620a3fcce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:02.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.987+0000 7fa59a200640 1 -- 172.21.15.32:0/3902923486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55620a02e4d0 msgr2=0x55620a3fcce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:02.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.987+0000 7fa59a200640 1 --2- 172.21.15.32:0/3902923486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55620a02e4d0 0x55620a3fcce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.987+0000 7fa59a200640 1 -- 172.21.15.32:0/3902923486 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620a0366b0 msgr2=0x55620a3fb920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:02.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.987+0000 7fa59a200640 1 --2- 172.21.15.32:0/3902923486 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620a0366b0 0x55620a3fb920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.987+0000 7fa59a200640 1 -- 172.21.15.32:0/3902923486 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55620a3ee8a0 con 0x55620a057390 2024-04-30T08:37:02.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.987+0000 7fa599800640 1 --2- 172.21.15.32:0/3902923486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55620a02e4d0 0x55620a3fcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:37:02.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.987+0000 7fa598e00640 1 --2- 172.21.15.32:0/3902923486 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620a0366b0 0x55620a3fb920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:37:02.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.988+0000 7fa59a200640 1 --2- 172.21.15.32:0/3902923486 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55620a057390 0x55620a402480 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa58c01f5b0 tx=0x7fa58c01a000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3a6b3e5a43e914e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:02.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.988+0000 7fa58a000640 1 -- 172.21.15.32:0/3902923486 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa58c011580 con 0x55620a057390 2024-04-30T08:37:02.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.988+0000 7fa58a000640 1 -- 172.21.15.32:0/3902923486 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa58c011bd0 con 0x55620a057390 2024-04-30T08:37:02.990 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.988+0000 7fa58a000640 1 -- 172.21.15.32:0/3902923486 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa58c01dd20 con 0x55620a057390 2024-04-30T08:37:02.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.989+0000 7fa5a3221740 1 -- 172.21.15.32:0/3902923486 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55620a057390 msgr2=0x55620a402480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:02.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.989+0000 7fa5a3221740 1 --2- 172.21.15.32:0/3902923486 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55620a057390 0x55620a402480 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa58c01f5b0 tx=0x7fa58c01a000 comp rx=0 tx=0).stop 2024-04-30T08:37:02.993 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.989+0000 7fa5a3221740 1 -- 172.21.15.32:0/3902923486 shutdown_connections 2024-04-30T08:37:02.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.989+0000 7fa5a3221740 1 --2- 172.21.15.32:0/3902923486 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55620a057390 0x55620a402480 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.989+0000 7fa5a3221740 1 --2- 172.21.15.32:0/3902923486 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620a0366b0 0x55620a3fb920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.989+0000 7fa5a3221740 1 --2- 172.21.15.32:0/3902923486 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55620a02e4d0 0x55620a3fcce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.989+0000 7fa5a3221740 1 -- 172.21.15.32:0/3902923486 >> 172.21.15.32:0/3902923486 conn(0x55620a04b4e0 msgr2=0x55620a401d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:02.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.989+0000 7fa5a3221740 1 -- 172.21.15.32:0/3902923486 shutdown_connections 2024-04-30T08:37:02.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.989+0000 7fa5a3221740 1 -- 172.21.15.32:0/3902923486 wait complete. 2024-04-30T08:37:02.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.990+0000 7fa5a3221740 1 -- start start 2024-04-30T08:37:02.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.990+0000 7fa5a3221740 1 Processor -- start 2024-04-30T08:37:02.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.990+0000 7fa5a3221740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55620a02e4d0 0x55620a405dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.990+0000 7fa5a3221740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55620a0366b0 0x55620a406310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.990+0000 7fa5a3221740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620a057390 0x55620a475e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.994 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.990+0000 7fa5a3221740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa5941b4520 con 0x55620a057390 2024-04-30T08:37:02.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.990+0000 7fa5a3221740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa5941b3e50 con 0x55620a0366b0 2024-04-30T08:37:02.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.990+0000 7fa5a3221740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa5941b1070 con 0x55620a02e4d0 2024-04-30T08:37:02.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.990+0000 7fa5a3221740 10 client.0 _ll_register_callbacks cb 0x7fa58400cef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:02.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.991+0000 7fa59a200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620a057390 0x55620a475e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:02.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.991+0000 7fa59a200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620a057390 0x55620a475e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56734/0 (socket says 172.21.15.32:56734) 2024-04-30T08:37:02.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.991+0000 7fa59a200640 1 -- 172.21.15.32:0/3929752734 learned_addr learned my addr 172.21.15.32:0/3929752734 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:02.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.991+0000 7fa598e00640 1 --2- 172.21.15.32:0/3929752734 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55620a0366b0 0x55620a406310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:02.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.991+0000 7fa599800640 1 --2- 172.21.15.32:0/3929752734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55620a02e4d0 0x55620a405dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:02.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.991+0000 7fa59a200640 1 -- 172.21.15.32:0/3929752734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55620a02e4d0 msgr2=0x55620a405dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:02.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.991+0000 7fa59a200640 1 --2- 172.21.15.32:0/3929752734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55620a02e4d0 0x55620a405dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.991+0000 7fa59a200640 1 -- 172.21.15.32:0/3929752734 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55620a0366b0 msgr2=0x55620a406310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:02.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.991+0000 7fa59a200640 1 --2- 172.21.15.32:0/3929752734 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55620a0366b0 0x55620a406310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.991+0000 7fa59a200640 1 -- 172.21.15.32:0/3929752734 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55620a4763b0 con 0x55620a057390 2024-04-30T08:37:02.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.991+0000 7fa59a200640 1 --2- 172.21.15.32:0/3929752734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620a057390 0x55620a475e40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fa58c009e30 tx=0x7fa58c009e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbd238f95deff910 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:02.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.991+0000 7fa57f400640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:37:02.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.992+0000 7fa57f400640 1 -- 172.21.15.32:0/3929752734 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa58c01dc20 con 0x55620a057390 2024-04-30T08:37:02.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.992+0000 7fa57f400640 1 -- 172.21.15.32:0/3929752734 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fa58c01ddc0 con 0x55620a057390 2024-04-30T08:37:02.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.992+0000 7fa5a3221740 1 -- 172.21.15.32:0/3929752734 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa58c011bd0 con 0x55620a057390 2024-04-30T08:37:02.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.992+0000 7fa57f400640 1 -- 172.21.15.32:0/3929752734 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fa58c011580 con 0x55620a057390 2024-04-30T08:37:02.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.992+0000 7fa5a3221740 10 client.16074 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:02.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.992+0000 7fa5a3221740 1 -- 172.21.15.32:0/3929752734 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55620a4776c0 con 0x55620a057390 2024-04-30T08:37:02.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.992+0000 7fa57f400640 1 -- 172.21.15.32:0/3929752734 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa58c011bd0 con 0x55620a057390 2024-04-30T08:37:02.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.993+0000 7fa57f400640 1 -- 172.21.15.32:0/3929752734 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7fa58c031020 con 0x55620a057390 2024-04-30T08:37:02.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.993+0000 7fa5a3221740 10 client.16074 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:02.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.993+0000 7fa5a3221740 10 client.16074 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:37:02.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.993+0000 7fa5a3221740 20 client.16074 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:02.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.993+0000 7fa5a3221740 1 --2- 172.21.15.32:0/3929752734 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x556209feb3a0 0x55620a47a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:02.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.993+0000 7fa5a3221740 4 client.16074 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-30T08:37:02.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.993+0000 7fa5a3221740 1 -- 172.21.15.32:0/3929752734 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55620a36b960 con 0x556209feb3a0 2024-04-30T08:37:02.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.993+0000 7fa599800640 1 --2- 172.21.15.32:0/3929752734 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x556209feb3a0 0x55620a47a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:02.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.994+0000 7fa599800640 1 --2- 172.21.15.32:0/3929752734 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x556209feb3a0 0x55620a47a0c0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8eaf60ef7d4ef1a2 server_cookie=20e0d5e38e71e349 in_seq=0 out_seq=0 2024-04-30T08:37:02.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.994+0000 7fa57f400640 10 client.16074 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:02.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.994+0000 7fa57f400640 1 -- 172.21.15.32:0/3929752734 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55620a36b960 con 0x556209feb3a0 2024-04-30T08:37:02.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.994+0000 7fa57f400640 10 client.16074 handle_command_reply: tid=0 2024-04-30T08:37:02.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.994+0000 7fa5a3221740 1 client.16074 shutdown 2024-04-30T08:37:02.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.994+0000 7fa5a3221740 1 -- 172.21.15.32:0/3929752734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620a057390 msgr2=0x55620a475e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:02.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.994+0000 7fa5a3221740 1 --2- 172.21.15.32:0/3929752734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620a057390 0x55620a475e40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fa58c009e30 tx=0x7fa58c009e60 comp rx=0 tx=0).stop 2024-04-30T08:37:02.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.994+0000 7fa5a3221740 1 -- 172.21.15.32:0/3929752734 shutdown_connections 2024-04-30T08:37:02.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.994+0000 7fa5a3221740 1 --2- 172.21.15.32:0/3929752734 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x556209feb3a0 0x55620a47a0c0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.994+0000 7fa5a3221740 1 --2- 172.21.15.32:0/3929752734 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55620a057390 0x55620a475e40 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.994+0000 7fa5a3221740 1 --2- 172.21.15.32:0/3929752734 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55620a0366b0 0x55620a406310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.994+0000 7fa5a3221740 1 --2- 172.21.15.32:0/3929752734 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55620a02e4d0 0x55620a405dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:02.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.994+0000 7fa57f400640 0 client.16074 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:02.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.994+0000 7fa5a3221740 1 -- 172.21.15.32:0/3929752734 >> 172.21.15.32:0/3929752734 conn(0x55620a04b4e0 msgr2=0x55620a4007c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:02.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.994+0000 7fa5a3221740 1 -- 172.21.15.32:0/3929752734 shutdown_connections 2024-04-30T08:37:02.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.994+0000 7fa5a3221740 1 -- 172.21.15.32:0/3929752734 wait complete. 2024-04-30T08:37:02.999 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.994+0000 7fa5a3221740 20 client.16074 trim_cache size 0 max 16384 2024-04-30T08:37:03.000 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-30T08:37:03.000 INFO:teuthology.orchestra.run.smithi032.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-30T08:37:03.000 INFO:teuthology.orchestra.run.smithi032.stdout: "scrubs": { 2024-04-30T08:37:03.000 INFO:teuthology.orchestra.run.smithi032.stdout: "b09a0368-b6b9-4e13-b89f-87f44fecab4a": { 2024-04-30T08:37:03.000 INFO:teuthology.orchestra.run.smithi032.stdout: "path": "/", 2024-04-30T08:37:03.000 INFO:teuthology.orchestra.run.smithi032.stdout: "tag": "b09a0368-b6b9-4e13-b89f-87f44fecab4a", 2024-04-30T08:37:03.000 INFO:teuthology.orchestra.run.smithi032.stdout: "options": "recursive,force" 2024-04-30T08:37:03.000 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:37:03.000 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:37:03.000 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-30T08:37:03.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.995+0000 7fa59c200640 1 -- 172.21.15.32:0/1598232236 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa5780afaf0 msgr2=0x7fa5780b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:03.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.995+0000 7fa59c200640 1 --2- 172.21.15.32:0/1598232236 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa5780afaf0 0x7fa5780b2000 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x55620a3f9410 tx=0x7fa58c005f90 comp rx=0 tx=0).stop 2024-04-30T08:37:03.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.995+0000 7fa59c200640 1 -- 172.21.15.32:0/1598232236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa594076120 msgr2=0x7fa5941a0870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:03.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.995+0000 7fa59c200640 1 --2- 172.21.15.32:0/1598232236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa594076120 0x7fa5941a0870 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fa58400c3f0 tx=0x7fa584009f90 comp rx=0 tx=0).stop 2024-04-30T08:37:03.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.995+0000 7fa59c200640 1 -- 172.21.15.32:0/1598232236 shutdown_connections 2024-04-30T08:37:03.001 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.995+0000 7fa59c200640 1 --2- 172.21.15.32:0/1598232236 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa5780afaf0 0x7fa5780b2000 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:03.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.995+0000 7fa59c200640 1 --2- 172.21.15.32:0/1598232236 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa5941102d0 0x7fa5941a7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:03.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.995+0000 7fa59c200640 1 --2- 172.21.15.32:0/1598232236 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa594076a70 0x7fa5941a0db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:03.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.995+0000 7fa59c200640 1 --2- 172.21.15.32:0/1598232236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa594076120 0x7fa5941a0870 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:03.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.995+0000 7fa59c200640 1 -- 172.21.15.32:0/1598232236 >> 172.21.15.32:0/1598232236 conn(0x7fa5940fed60 msgr2=0x7fa594103020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:03.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.995+0000 7fa59c200640 1 -- 172.21.15.32:0/1598232236 shutdown_connections 2024-04-30T08:37:03.002 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:02.995+0000 7fa59c200640 1 -- 172.21.15.32:0/1598232236 wait complete. 2024-04-30T08:37:03.023 INFO:tasks.cephfs.filesystem:scrub status for tag:b09a0368-b6b9-4e13-b89f-87f44fecab4a - {'path': '/', 'tag': 'b09a0368-b6b9-4e13-b89f-87f44fecab4a', 'options': 'recursive,force'} 2024-04-30T08:37:03.091 INFO:tasks.workunit.client.0.smithi032.stdout:checking whether build environment is sane... yes 2024-04-30T08:37:03.108 INFO:tasks.workunit.client.0.smithi032.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-04-30T08:37:03.109 INFO:tasks.workunit.client.0.smithi032.stdout:checking for gawk... gawk 2024-04-30T08:37:03.142 INFO:tasks.workunit.client.0.smithi032.stdout:checking whether make sets $(MAKE)... yes 2024-04-30T08:37:03.249 INFO:tasks.workunit.client.0.smithi032.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-04-30T08:37:03.250 INFO:tasks.workunit.client.0.smithi032.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-04-30T08:37:03.273 INFO:tasks.workunit.client.0.smithi032.stdout:checking for style of include used by make... GNU 2024-04-30T08:37:03.283 INFO:tasks.workunit.client.0.smithi032.stdout:checking for gcc... gcc 2024-04-30T08:37:03.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:03 smithi189 ceph-mon[33038]: pgmap v481: 129 pgs: 129 active+clean; 4.3 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 354 KiB/s wr, 60 op/s 2024-04-30T08:37:03.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:03 smithi189 ceph-mon[33038]: scrub queued for path: / 2024-04-30T08:37:03.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:03 smithi189 ceph-mon[33038]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:03.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:03 smithi189 ceph-mon[33038]: scrub summary: active paths [/] 2024-04-30T08:37:03.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:03 smithi189 ceph-mon[33038]: scrub summary: active paths [/] 2024-04-30T08:37:03.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:03 smithi189 ceph-mon[33038]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:03.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:03 smithi189 ceph-mon[33038]: scrub summary: active paths [/] 2024-04-30T08:37:03.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:03 smithi189 ceph-mon[33038]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:03.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:03 smithi189 ceph-mon[33038]: scrub summary: active paths [/] 2024-04-30T08:37:03.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:03 smithi189 ceph-mon[33038]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:03.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:03 smithi189 ceph-mon[33038]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:03.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:03 smithi032 ceph-mon[28331]: pgmap v481: 129 pgs: 129 active+clean; 4.3 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 354 KiB/s wr, 60 op/s 2024-04-30T08:37:03.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:03 smithi032 ceph-mon[28331]: scrub queued for path: / 2024-04-30T08:37:03.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:03 smithi032 ceph-mon[28331]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:03.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:03 smithi032 ceph-mon[28331]: scrub summary: active paths [/] 2024-04-30T08:37:03.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:03 smithi032 ceph-mon[28331]: scrub summary: active paths [/] 2024-04-30T08:37:03.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:03 smithi032 ceph-mon[28331]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:03.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:03 smithi032 ceph-mon[28331]: scrub summary: active paths [/] 2024-04-30T08:37:03.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:03 smithi032 ceph-mon[28331]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:03.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:03 smithi032 ceph-mon[28331]: scrub summary: active paths [/] 2024-04-30T08:37:03.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:03 smithi032 ceph-mon[28331]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:03.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:03 smithi032 ceph-mon[28331]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:03.486 INFO:tasks.workunit.client.0.smithi032.stdout:checking whether the C compiler works... yes 2024-04-30T08:37:03.487 INFO:tasks.workunit.client.0.smithi032.stdout:checking for C compiler default output file name... a.out 2024-04-30T08:37:03.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:03 smithi071 ceph-mon[33385]: pgmap v481: 129 pgs: 129 active+clean; 4.3 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 242 KiB/s rd, 354 KiB/s wr, 60 op/s 2024-04-30T08:37:03.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:03 smithi071 ceph-mon[33385]: scrub queued for path: / 2024-04-30T08:37:03.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:03 smithi071 ceph-mon[33385]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:03.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:03 smithi071 ceph-mon[33385]: scrub summary: active paths [/] 2024-04-30T08:37:03.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:03 smithi071 ceph-mon[33385]: scrub summary: active paths [/] 2024-04-30T08:37:03.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:03 smithi071 ceph-mon[33385]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:03.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:03 smithi071 ceph-mon[33385]: scrub summary: active paths [/] 2024-04-30T08:37:03.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:03 smithi071 ceph-mon[33385]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:03.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:03 smithi071 ceph-mon[33385]: scrub summary: active paths [/] 2024-04-30T08:37:03.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:03 smithi071 ceph-mon[33385]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:03.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:03 smithi071 ceph-mon[33385]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:03.577 INFO:tasks.workunit.client.0.smithi032.stdout:checking for suffix of executables... 2024-04-30T08:37:03.669 INFO:tasks.workunit.client.0.smithi032.stdout:checking whether we are cross compiling... no 2024-04-30T08:37:03.748 INFO:tasks.workunit.client.0.smithi032.stdout:checking for suffix of object files... o 2024-04-30T08:37:03.806 INFO:tasks.workunit.client.0.smithi032.stdout:checking whether we are using the GNU C compiler... yes 2024-04-30T08:37:03.869 INFO:tasks.workunit.client.0.smithi032.stdout:checking whether gcc accepts -g... yes 2024-04-30T08:37:03.952 INFO:tasks.workunit.client.0.smithi032.stdout:checking for gcc option to accept ISO C89... none needed 2024-04-30T08:37:04.160 INFO:tasks.workunit.client.0.smithi032.stdout:checking dependency style of gcc... gcc3 2024-04-30T08:37:04.267 INFO:tasks.workunit.client.0.smithi032.stdout:checking how to run the C preprocessor... gcc -E 2024-04-30T08:37:04.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:04 smithi189 ceph-mon[33038]: scrub summary: active paths [/] 2024-04-30T08:37:04.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:04 smithi189 ceph-mon[33038]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:04.375 INFO:tasks.workunit.client.0.smithi032.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-04-30T08:37:04.380 INFO:tasks.workunit.client.0.smithi032.stdout:checking for egrep... /usr/bin/grep -E 2024-04-30T08:37:04.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:04 smithi032 ceph-mon[28331]: scrub summary: active paths [/] 2024-04-30T08:37:04.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:04 smithi032 ceph-mon[28331]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:04.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:04 smithi071 ceph-mon[33385]: scrub summary: active paths [/] 2024-04-30T08:37:04.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:04 smithi071 ceph-mon[33385]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:04.658 INFO:tasks.workunit.client.0.smithi032.stdout:checking for ANSI C header files... yes 2024-04-30T08:37:04.746 INFO:tasks.workunit.client.0.smithi032.stdout:checking for sys/types.h... yes 2024-04-30T08:37:04.838 INFO:tasks.workunit.client.0.smithi032.stdout:checking for sys/stat.h... yes 2024-04-30T08:37:04.928 INFO:tasks.workunit.client.0.smithi032.stdout:checking for stdlib.h... yes 2024-04-30T08:37:05.011 INFO:tasks.workunit.client.0.smithi032.stdout:checking for string.h... yes 2024-04-30T08:37:05.101 INFO:tasks.workunit.client.0.smithi032.stdout:checking for memory.h... yes 2024-04-30T08:37:05.194 INFO:tasks.workunit.client.0.smithi032.stdout:checking for strings.h... yes 2024-04-30T08:37:05.288 INFO:tasks.workunit.client.0.smithi032.stdout:checking for inttypes.h... yes 2024-04-30T08:37:05.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:05 smithi189 ceph-mon[33038]: pgmap v482: 129 pgs: 129 active+clean; 4.1 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 355 KiB/s wr, 64 op/s 2024-04-30T08:37:05.384 INFO:tasks.workunit.client.0.smithi032.stdout:checking for stdint.h... yes 2024-04-30T08:37:05.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:05 smithi032 ceph-mon[28331]: pgmap v482: 129 pgs: 129 active+clean; 4.1 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 355 KiB/s wr, 64 op/s 2024-04-30T08:37:05.483 INFO:tasks.workunit.client.0.smithi032.stdout:checking for unistd.h... yes 2024-04-30T08:37:05.506 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:05 smithi071 ceph-mon[33385]: pgmap v482: 129 pgs: 129 active+clean; 4.1 MiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 244 KiB/s rd, 355 KiB/s wr, 64 op/s 2024-04-30T08:37:05.571 INFO:tasks.workunit.client.0.smithi032.stdout:checking minix/config.h usability... no 2024-04-30T08:37:05.627 INFO:tasks.workunit.client.0.smithi032.stdout:checking minix/config.h presence... no 2024-04-30T08:37:05.627 INFO:tasks.workunit.client.0.smithi032.stdout:checking for minix/config.h... no 2024-04-30T08:37:05.709 INFO:tasks.workunit.client.0.smithi032.stdout:checking whether it is safe to define __EXTENSIONS__... yes 2024-04-30T08:37:05.710 INFO:tasks.workunit.client.0.smithi032.stdout:checking for gcc... (cached) gcc 2024-04-30T08:37:05.800 INFO:tasks.workunit.client.0.smithi032.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-04-30T08:37:05.800 INFO:tasks.workunit.client.0.smithi032.stdout:checking whether gcc accepts -g... (cached) yes 2024-04-30T08:37:05.801 INFO:tasks.workunit.client.0.smithi032.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-04-30T08:37:05.802 INFO:tasks.workunit.client.0.smithi032.stdout:checking dependency style of gcc... (cached) gcc3 2024-04-30T08:37:05.905 INFO:tasks.workunit.client.0.smithi032.stdout:checking for main in -lm... yes 2024-04-30T08:37:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.961+0000 7f8cab600640 1 -- 172.21.15.32:0/2989530371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ca4103c20 msgr2=0x7f8ca4104920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.961+0000 7f8cab600640 1 --2- 172.21.15.32:0/2989530371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ca4103c20 0x7f8ca4104920 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f8c98009900 tx=0x7f8c9802a410 comp rx=0 tx=0).stop 2024-04-30T08:37:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.961+0000 7f8cab600640 1 -- 172.21.15.32:0/2989530371 shutdown_connections 2024-04-30T08:37:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.961+0000 7f8cab600640 1 --2- 172.21.15.32:0/2989530371 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ca4109b60 0x7f8ca410bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.961+0000 7f8cab600640 1 --2- 172.21.15.32:0/2989530371 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ca4104e60 0x7f8ca4105270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.961+0000 7f8cab600640 1 --2- 172.21.15.32:0/2989530371 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ca4103c20 0x7f8ca4104920 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.961+0000 7f8cab600640 1 -- 172.21.15.32:0/2989530371 >> 172.21.15.32:0/2989530371 conn(0x7f8ca40fedf0 msgr2=0x7f8ca41012c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.961+0000 7f8cab600640 1 -- 172.21.15.32:0/2989530371 shutdown_connections 2024-04-30T08:37:05.962 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.961+0000 7f8cab600640 1 -- 172.21.15.32:0/2989530371 wait complete. 2024-04-30T08:37:05.964 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.962+0000 7f8cab600640 1 Processor -- start 2024-04-30T08:37:05.964 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.962+0000 7f8cab600640 1 -- start start 2024-04-30T08:37:05.964 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.963+0000 7f8cab600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ca4103c20 0x7f8ca40779e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:05.964 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.963+0000 7f8cab600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ca4104e60 0x7f8ca4076080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:05.964 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.963+0000 7f8cab600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ca4109b60 0x7f8ca40765c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:05.964 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.963+0000 7f8cab600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca410f690 con 0x7f8ca4109b60 2024-04-30T08:37:05.964 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.963+0000 7f8cab600640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca410f510 con 0x7f8ca4104e60 2024-04-30T08:37:05.964 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.963+0000 7f8cab600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca410f810 con 0x7f8ca4103c20 2024-04-30T08:37:05.965 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.963+0000 7f8ca3e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ca4104e60 0x7f8ca4076080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:05.965 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.963+0000 7f8ca8c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ca4103c20 0x7f8ca40779e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:05.965 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.963+0000 7f8ca3e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ca4104e60 0x7f8ca4076080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:35960/0 (socket says 172.21.15.32:35960) 2024-04-30T08:37:05.965 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.963+0000 7f8ca3e00640 1 -- 172.21.15.32:0/1985297634 learned_addr learned my addr 172.21.15.32:0/1985297634 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:05.965 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.963+0000 7f8ca8c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ca4103c20 0x7f8ca40779e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:37170/0 (socket says 172.21.15.32:37170) 2024-04-30T08:37:05.965 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.963+0000 7f8ca9600640 1 --2- 172.21.15.32:0/1985297634 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ca4109b60 0x7f8ca40765c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:05.965 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.964+0000 7f8ca3e00640 1 -- 172.21.15.32:0/1985297634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ca4103c20 msgr2=0x7f8ca40779e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:05.965 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.964+0000 7f8ca3e00640 1 --2- 172.21.15.32:0/1985297634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ca4103c20 0x7f8ca40779e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:05.965 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.964+0000 7f8ca3e00640 1 -- 172.21.15.32:0/1985297634 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ca4109b60 msgr2=0x7f8ca40765c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:05.965 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.964+0000 7f8ca3e00640 1 --2- 172.21.15.32:0/1985297634 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ca4109b60 0x7f8ca40765c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:05.965 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.964+0000 7f8ca3e00640 1 -- 172.21.15.32:0/1985297634 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ca4076cf0 con 0x7f8ca4104e60 2024-04-30T08:37:05.966 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.964+0000 7f8ca3e00640 1 --2- 172.21.15.32:0/1985297634 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ca4104e60 0x7f8ca4076080 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8c9000e9f0 tx=0x7f8c9000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:05.966 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.964+0000 7f8ca1600640 1 -- 172.21.15.32:0/1985297634 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c9000cd80 con 0x7f8ca4104e60 2024-04-30T08:37:05.966 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.964+0000 7f8ca1600640 1 -- 172.21.15.32:0/1985297634 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c900108e0 con 0x7f8ca4104e60 2024-04-30T08:37:05.966 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.964+0000 7f8ca1600640 1 -- 172.21.15.32:0/1985297634 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c90018980 con 0x7f8ca4104e60 2024-04-30T08:37:05.966 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.964+0000 7f8cab600640 1 -- 172.21.15.32:0/1985297634 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ca41a4390 con 0x7f8ca4104e60 2024-04-30T08:37:05.966 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.965+0000 7f8cab600640 1 -- 172.21.15.32:0/1985297634 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ca41a4800 con 0x7f8ca4104e60 2024-04-30T08:37:05.969 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.968+0000 7f8cb276e740 1 Processor -- start 2024-04-30T08:37:05.969 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.968+0000 7f8cb276e740 1 -- start start 2024-04-30T08:37:05.969 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.968+0000 7f8ca1600640 1 -- 172.21.15.32:0/1985297634 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f8c90010430 con 0x7f8ca4104e60 2024-04-30T08:37:05.969 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.968+0000 7f8cb276e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557599d161b0 0x55759a0b80e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:05.969 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.968+0000 7f8cb276e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557599ced2f0 0x55759a0b8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:05.969 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.968+0000 7f8cb276e740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557599cf54d0 0x55759a0bf560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:05.969 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.968+0000 7f8cb276e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55759a01c550 con 0x557599ced2f0 2024-04-30T08:37:05.970 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.968+0000 7f8cb276e740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x557599fc1bc0 con 0x557599cf54d0 2024-04-30T08:37:05.970 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.968+0000 7f8cb276e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55759a018510 con 0x557599d161b0 2024-04-30T08:37:05.971 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.969+0000 7f8ca8c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557599ced2f0 0x55759a0b8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:05.971 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.969+0000 7f8ca3e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557599cf54d0 0x55759a0bf560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:05.971 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.969+0000 7f8ca8c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557599ced2f0 0x55759a0b8660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:56754/0 (socket says 172.21.15.32:56754) 2024-04-30T08:37:05.971 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.969+0000 7f8ca8c00640 1 -- 172.21.15.32:0/131072141 learned_addr learned my addr 172.21.15.32:0/131072141 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:05.972 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.969+0000 7f8ca9600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557599d161b0 0x55759a0b80e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:05.972 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.969+0000 7f8ca8c00640 1 -- 172.21.15.32:0/131072141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557599d161b0 msgr2=0x55759a0b80e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:05.972 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.969+0000 7f8ca8c00640 1 --2- 172.21.15.32:0/131072141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557599d161b0 0x55759a0b80e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:05.972 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.969+0000 7f8ca8c00640 1 -- 172.21.15.32:0/131072141 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557599cf54d0 msgr2=0x55759a0bf560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:05.972 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.969+0000 7f8ca8c00640 1 --2- 172.21.15.32:0/131072141 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557599cf54d0 0x55759a0bf560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:05.972 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.969+0000 7f8ca8c00640 1 -- 172.21.15.32:0/131072141 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557599d2e210 con 0x557599ced2f0 2024-04-30T08:37:05.972 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.969+0000 7f8ca9600640 1 --2- 172.21.15.32:0/131072141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557599d161b0 0x55759a0b80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:37:05.972 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.970+0000 7f8ca8c00640 1 --2- 172.21.15.32:0/131072141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557599ced2f0 0x55759a0b8660 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8c98009230 tx=0x7f8c98009620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d03eb15c46e50056 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:05.972 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.970+0000 7f8ca1600640 1 --2- 172.21.15.32:0/1985297634 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8c8c0afae0 0x7f8c8c0b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:05.972 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.970+0000 7f8c95c00640 1 -- 172.21.15.32:0/131072141 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c98035030 con 0x557599ced2f0 2024-04-30T08:37:05.972 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.970+0000 7f8ca1600640 1 -- 172.21.15.32:0/1985297634 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f8c900df330 con 0x7f8ca4104e60 2024-04-30T08:37:05.972 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.970+0000 7f8c95c00640 1 -- 172.21.15.32:0/131072141 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c98040070 con 0x557599ced2f0 2024-04-30T08:37:05.972 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.970+0000 7f8c95c00640 1 -- 172.21.15.32:0/131072141 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c98003950 con 0x557599ced2f0 2024-04-30T08:37:05.973 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.970+0000 7f8ca9600640 1 --2- 172.21.15.32:0/1985297634 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8c8c0afae0 0x7f8c8c0b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:05.974 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.972+0000 7f8cb276e740 1 -- 172.21.15.32:0/131072141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557599ced2f0 msgr2=0x55759a0b8660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:05.974 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.972+0000 7f8cb276e740 1 --2- 172.21.15.32:0/131072141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557599ced2f0 0x55759a0b8660 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f8c98009230 tx=0x7f8c98009620 comp rx=0 tx=0).stop 2024-04-30T08:37:05.974 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.972+0000 7f8ca9600640 1 --2- 172.21.15.32:0/1985297634 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8c8c0afae0 0x7f8c8c0b1ff0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x55759a0b92b0 tx=0x7f8c9c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:05.974 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.972+0000 7f8cb276e740 1 -- 172.21.15.32:0/131072141 shutdown_connections 2024-04-30T08:37:05.974 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.972+0000 7f8cb276e740 1 --2- 172.21.15.32:0/131072141 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557599cf54d0 0x55759a0bf560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:05.974 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.972+0000 7f8cb276e740 1 --2- 172.21.15.32:0/131072141 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557599ced2f0 0x55759a0b8660 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:05.974 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.972+0000 7f8cb276e740 1 --2- 172.21.15.32:0/131072141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557599d161b0 0x55759a0b80e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:05.974 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.972+0000 7f8cb276e740 1 -- 172.21.15.32:0/131072141 >> 172.21.15.32:0/131072141 conn(0x557599d0a300 msgr2=0x55759a0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:05.974 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.972+0000 7f8cb276e740 1 -- 172.21.15.32:0/131072141 shutdown_connections 2024-04-30T08:37:05.974 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.972+0000 7f8cb276e740 1 -- 172.21.15.32:0/131072141 wait complete. 2024-04-30T08:37:05.974 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.973+0000 7f8cb276e740 1 -- start start 2024-04-30T08:37:05.974 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.973+0000 7f8cb276e740 1 Processor -- start 2024-04-30T08:37:05.978 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.974+0000 7f8cb276e740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557599ced2f0 0x55759a0c1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:05.978 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.974+0000 7f8cb276e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557599cf54d0 0x55759a0c20c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:05.978 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.974+0000 7f8cb276e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557599d161b0 0x55759a0c8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:05.978 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.974+0000 7f8cb276e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55759a0bff50 con 0x557599cf54d0 2024-04-30T08:37:05.978 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.974+0000 7f8cb276e740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x557599fd5660 con 0x557599ced2f0 2024-04-30T08:37:05.978 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.974+0000 7f8cb276e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55759a0c00d0 con 0x557599d161b0 2024-04-30T08:37:05.978 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.974+0000 7f8ca8c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557599ced2f0 0x55759a0c1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:05.978 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.974+0000 7f8cb276e740 10 client.0 _ll_register_callbacks cb 0x557599f9f400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:05.978 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.974+0000 7f8ca8c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557599ced2f0 0x55759a0c1b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:35986/0 (socket says 172.21.15.32:35986) 2024-04-30T08:37:05.978 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.974+0000 7f8ca8c00640 1 -- 172.21.15.32:0/1139562038 learned_addr learned my addr 172.21.15.32:0/1139562038 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:05.978 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.974+0000 7f8ca3e00640 1 --2- 172.21.15.32:0/1139562038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557599cf54d0 0x55759a0c20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:05.979 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.975+0000 7f8ca9600640 1 --2- 172.21.15.32:0/1139562038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557599d161b0 0x55759a0c8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:05.979 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.975+0000 7f8ca8c00640 1 -- 172.21.15.32:0/1139562038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557599d161b0 msgr2=0x55759a0c8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:05.979 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.975+0000 7f8ca8c00640 1 --2- 172.21.15.32:0/1139562038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557599d161b0 0x55759a0c8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:05.979 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.975+0000 7f8ca8c00640 1 -- 172.21.15.32:0/1139562038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557599cf54d0 msgr2=0x55759a0c20c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:05.979 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.975+0000 7f8ca8c00640 1 --2- 172.21.15.32:0/1139562038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557599cf54d0 0x55759a0c20c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:05.979 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.975+0000 7f8ca8c00640 1 -- 172.21.15.32:0/1139562038 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557599dfa620 con 0x557599ced2f0 2024-04-30T08:37:05.979 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.975+0000 7f8ca8c00640 1 --2- 172.21.15.32:0/1139562038 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557599ced2f0 0x55759a0c1b80 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f8c98031500 tx=0x7f8c9802a410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d53c008957a1ef76 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:05.979 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.975+0000 7f8c8a000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:37:05.979 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.976+0000 7f8c8a000640 1 -- 172.21.15.32:0/1139562038 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c98035070 con 0x557599ced2f0 2024-04-30T08:37:05.979 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.976+0000 7f8cb276e740 1 -- 172.21.15.32:0/1139562038 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8c98040070 con 0x557599ced2f0 2024-04-30T08:37:05.979 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.976+0000 7f8c8a000640 1 -- 172.21.15.32:0/1139562038 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f8c980036d0 con 0x557599ced2f0 2024-04-30T08:37:05.979 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.976+0000 7f8c8a000640 1 -- 172.21.15.32:0/1139562038 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8c980038d0 con 0x557599ced2f0 2024-04-30T08:37:05.979 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.976+0000 7f8cb276e740 10 client.26222 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:05.979 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.976+0000 7f8cb276e740 1 -- 172.21.15.32:0/1139562038 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55759a0cbee0 con 0x557599ced2f0 2024-04-30T08:37:05.980 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.976+0000 7f8c8a000640 1 -- 172.21.15.32:0/1139562038 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c98040070 con 0x557599ced2f0 2024-04-30T08:37:05.980 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.976+0000 7f8c8a000640 1 -- 172.21.15.32:0/1139562038 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7f8c98045940 con 0x557599ced2f0 2024-04-30T08:37:05.980 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.977+0000 7f8cb276e740 10 client.26222 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:05.980 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.977+0000 7f8cb276e740 10 client.26222 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-30T08:37:05.980 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.977+0000 7f8cb276e740 20 client.26222 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:05.980 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.977+0000 7f8cb276e740 1 --2- 172.21.15.32:0/1139562038 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x557599cd4d80 0x55759a0cf240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:05.980 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.977+0000 7f8cb276e740 4 client.26222 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:37:05.980 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.977+0000 7f8cb276e740 1 -- 172.21.15.32:0/1139562038 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55759a02a160 con 0x557599cd4d80 2024-04-30T08:37:05.980 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.977+0000 7f8ca3e00640 1 --2- 172.21.15.32:0/1139562038 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x557599cd4d80 0x55759a0cf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:05.981 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.978+0000 7f8ca3e00640 1 --2- 172.21.15.32:0/1139562038 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x557599cd4d80 0x55759a0cf240 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4846ab5f0e207732 server_cookie=fe5baf4471348e58 in_seq=0 out_seq=0 2024-04-30T08:37:05.981 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.978+0000 7f8c8a000640 10 client.26222 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:05.981 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.978+0000 7f8c8a000640 1 -- 172.21.15.32:0/1139562038 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55759a02a160 con 0x557599cd4d80 2024-04-30T08:37:05.981 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.978+0000 7f8c8a000640 10 client.26222 handle_command_reply: tid=0 2024-04-30T08:37:05.981 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.978+0000 7f8cb276e740 1 client.26222 shutdown 2024-04-30T08:37:05.981 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.979+0000 7f8cb276e740 1 -- 172.21.15.32:0/1139562038 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557599ced2f0 msgr2=0x55759a0c1b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:05.982 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.979+0000 7f8cb276e740 1 --2- 172.21.15.32:0/1139562038 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557599ced2f0 0x55759a0c1b80 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f8c98031500 tx=0x7f8c9802a410 comp rx=0 tx=0).stop 2024-04-30T08:37:05.982 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.979+0000 7f8cb276e740 1 -- 172.21.15.32:0/1139562038 shutdown_connections 2024-04-30T08:37:05.982 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.979+0000 7f8cb276e740 1 --2- 172.21.15.32:0/1139562038 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x557599cd4d80 0x55759a0cf240 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:05.982 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.979+0000 7f8cb276e740 1 --2- 172.21.15.32:0/1139562038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557599d161b0 0x55759a0c8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:05.982 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.979+0000 7f8cb276e740 1 --2- 172.21.15.32:0/1139562038 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557599cf54d0 0x55759a0c20c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:05.982 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.979+0000 7f8cb276e740 1 --2- 172.21.15.32:0/1139562038 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557599ced2f0 0x55759a0c1b80 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:05.982 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.979+0000 7f8c8a000640 0 client.26222 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:05.982 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.979+0000 7f8cb276e740 1 -- 172.21.15.32:0/1139562038 >> 172.21.15.32:0/1139562038 conn(0x557599d0a300 msgr2=0x55759a0b6980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:05.982 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.979+0000 7f8cb276e740 1 -- 172.21.15.32:0/1139562038 shutdown_connections 2024-04-30T08:37:05.982 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.979+0000 7f8cb276e740 1 -- 172.21.15.32:0/1139562038 wait complete. 2024-04-30T08:37:05.982 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.979+0000 7f8cb276e740 20 client.26222 trim_cache size 0 max 16384 2024-04-30T08:37:05.997 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.997+0000 7f8cb276e740 1 Processor -- start 2024-04-30T08:37:05.997 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.997+0000 7f8cb276e740 1 -- start start 2024-04-30T08:37:05.998 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.997+0000 7f8cb276e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557599ced2f0 0x55759a0c1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:05.998 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.997+0000 7f8cb276e740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557599cf54d0 0x55759a0c8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:05.998 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.997+0000 7f8cb276e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557599d161b0 0x55759a0c9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:05.998 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.997+0000 7f8cb276e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca41aedf0 con 0x557599d161b0 2024-04-30T08:37:05.998 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.997+0000 7f8cb276e740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca41b09c0 con 0x557599cf54d0 2024-04-30T08:37:05.999 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.997+0000 7f8cb276e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca41af4c0 con 0x557599ced2f0 2024-04-30T08:37:05.999 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.997+0000 7f8ca3e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557599cf54d0 0x55759a0c8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:05.999 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.998+0000 7f8ca3e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557599cf54d0 0x55759a0c8ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:36000/0 (socket says 172.21.15.32:36000) 2024-04-30T08:37:05.999 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.998+0000 7f8ca3e00640 1 -- 172.21.15.32:0/801028698 learned_addr learned my addr 172.21.15.32:0/801028698 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:05.999 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.998+0000 7f8ca9600640 1 --2- 172.21.15.32:0/801028698 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557599d161b0 0x55759a0c9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:05.999 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.998+0000 7f8ca8c00640 1 --2- 172.21.15.32:0/801028698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557599ced2f0 0x55759a0c1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:05.999 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.998+0000 7f8ca3e00640 1 -- 172.21.15.32:0/801028698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557599ced2f0 msgr2=0x55759a0c1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:05.999 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.998+0000 7f8ca3e00640 1 --2- 172.21.15.32:0/801028698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557599ced2f0 0x55759a0c1e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:05.999 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.998+0000 7f8ca3e00640 1 -- 172.21.15.32:0/801028698 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557599d161b0 msgr2=0x55759a0c9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:05.999 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.998+0000 7f8ca3e00640 1 --2- 172.21.15.32:0/801028698 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557599d161b0 0x55759a0c9410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:05.999 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.998+0000 7f8ca3e00640 1 -- 172.21.15.32:0/801028698 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55759a0bca90 con 0x557599cf54d0 2024-04-30T08:37:06.000 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.998+0000 7f8ca3e00640 1 --2- 172.21.15.32:0/801028698 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557599cf54d0 0x55759a0c8ed0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8c900a7440 tx=0x7f8c900e2d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d53fd217ccf86e2b server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:06.000 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.998+0000 7f8c95200640 1 -- 172.21.15.32:0/801028698 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c900df170 con 0x557599cf54d0 2024-04-30T08:37:06.000 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.999+0000 7f8c95200640 1 -- 172.21.15.32:0/801028698 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8c90010720 con 0x557599cf54d0 2024-04-30T08:37:06.001 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.999+0000 7f8c95200640 1 -- 172.21.15.32:0/801028698 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c90010a90 con 0x557599cf54d0 2024-04-30T08:37:06.001 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.999+0000 7f8cb276e740 1 -- 172.21.15.32:0/801028698 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557599cf54d0 msgr2=0x55759a0c8ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:06.001 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:05.999+0000 7f8cb276e740 1 --2- 172.21.15.32:0/801028698 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557599cf54d0 0x55759a0c8ed0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f8c900a7440 tx=0x7f8c900e2d70 comp rx=0 tx=0).stop 2024-04-30T08:37:06.001 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.000+0000 7f8cb276e740 1 -- 172.21.15.32:0/801028698 shutdown_connections 2024-04-30T08:37:06.001 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.000+0000 7f8cb276e740 1 --2- 172.21.15.32:0/801028698 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557599d161b0 0x55759a0c9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:06.001 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.000+0000 7f8cb276e740 1 --2- 172.21.15.32:0/801028698 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557599cf54d0 0x55759a0c8ed0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:06.001 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.000+0000 7f8cb276e740 1 --2- 172.21.15.32:0/801028698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557599ced2f0 0x55759a0c1e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:06.001 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.000+0000 7f8cb276e740 1 -- 172.21.15.32:0/801028698 >> 172.21.15.32:0/801028698 conn(0x557599d0a300 msgr2=0x55759a0c1750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:06.001 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.000+0000 7f8cb276e740 1 -- 172.21.15.32:0/801028698 shutdown_connections 2024-04-30T08:37:06.001 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.000+0000 7f8cb276e740 1 -- 172.21.15.32:0/801028698 wait complete. 2024-04-30T08:37:06.001 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.000+0000 7f8cb276e740 1 -- start start 2024-04-30T08:37:06.002 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.000+0000 7f8cb276e740 1 Processor -- start 2024-04-30T08:37:06.002 INFO:tasks.workunit.client.0.smithi032.stdout:checking for main in -lpthread... yes 2024-04-30T08:37:06.004 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.001+0000 7f8cb276e740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557599ced2f0 0x55759a0c94b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:06.004 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.001+0000 7f8cb276e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557599cf54d0 0x55759a0c6030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:06.004 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.001+0000 7f8cb276e740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557599d161b0 0x55759a0c6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:06.004 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.001+0000 7f8cb276e740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca41b0260 con 0x557599ced2f0 2024-04-30T08:37:06.004 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.001+0000 7f8cb276e740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca41afb90 con 0x557599d161b0 2024-04-30T08:37:06.004 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.001+0000 7f8cb276e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f8ca41acdb0 con 0x557599cf54d0 2024-04-30T08:37:06.004 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.001+0000 7f8cb276e740 10 client.0 _ll_register_callbacks cb 0x7f8c900e2c40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:06.005 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.001+0000 7f8ca9600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557599d161b0 0x55759a0c6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:06.005 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.001+0000 7f8ca8c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557599ced2f0 0x55759a0c94b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:06.005 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.001+0000 7f8ca9600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557599d161b0 0x55759a0c6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:36008/0 (socket says 172.21.15.32:36008) 2024-04-30T08:37:06.005 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.001+0000 7f8ca9600640 1 -- 172.21.15.32:0/371390130 learned_addr learned my addr 172.21.15.32:0/371390130 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:06.005 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.001+0000 7f8ca3e00640 1 --2- 172.21.15.32:0/371390130 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557599cf54d0 0x55759a0c6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:06.005 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.002+0000 7f8ca9600640 1 -- 172.21.15.32:0/371390130 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557599cf54d0 msgr2=0x55759a0c6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:06.005 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.002+0000 7f8ca9600640 1 --2- 172.21.15.32:0/371390130 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557599cf54d0 0x55759a0c6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:06.005 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.002+0000 7f8ca9600640 1 -- 172.21.15.32:0/371390130 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557599ced2f0 msgr2=0x55759a0c94b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:06.005 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.002+0000 7f8ca9600640 1 --2- 172.21.15.32:0/371390130 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557599ced2f0 0x55759a0c94b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:06.005 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.002+0000 7f8ca9600640 1 -- 172.21.15.32:0/371390130 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55759a0c6b60 con 0x557599d161b0 2024-04-30T08:37:06.006 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.002+0000 7f8ca9600640 1 --2- 172.21.15.32:0/371390130 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557599d161b0 0x55759a0c6570 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c003f50 tx=0x7f8c9c004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d97f28b1f098490a server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:06.006 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.002+0000 7f8c8a000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:37:06.006 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.002+0000 7f8c8a000640 1 -- 172.21.15.32:0/371390130 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c9c00fe50 con 0x557599d161b0 2024-04-30T08:37:06.006 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.003+0000 7f8cb276e740 1 -- 172.21.15.32:0/371390130 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55759a023fd0 con 0x557599d161b0 2024-04-30T08:37:06.006 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.003+0000 7f8c8a000640 1 -- 172.21.15.32:0/371390130 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f8c9c004250 con 0x557599d161b0 2024-04-30T08:37:06.006 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.003+0000 7f8c8a000640 1 -- 172.21.15.32:0/371390130 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f8c9c037050 con 0x557599d161b0 2024-04-30T08:37:06.006 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.003+0000 7f8cb276e740 10 client.26228 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:06.006 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.003+0000 7f8cb276e740 1 -- 172.21.15.32:0/371390130 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55759a136710 con 0x557599d161b0 2024-04-30T08:37:06.006 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.003+0000 7f8c8a000640 1 -- 172.21.15.32:0/371390130 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x55759a023fd0 con 0x557599d161b0 2024-04-30T08:37:06.007 INFO:tasks.workunit.client.0.smithi032.stdout:checking for ANSI C header files... (cached) yes 2024-04-30T08:37:06.007 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.003+0000 7f8c8a000640 1 -- 172.21.15.32:0/371390130 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7f8c9c01ec70 con 0x557599d161b0 2024-04-30T08:37:06.008 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.004+0000 7f8cb276e740 10 client.26228 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:06.008 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.004+0000 7f8cb276e740 10 client.26228 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-30T08:37:06.008 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.004+0000 7f8cb276e740 20 client.26228 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:06.008 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.004+0000 7f8cb276e740 1 --2- 172.21.15.32:0/371390130 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x557599caa1c0 0x55759a1391f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:06.008 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.004+0000 7f8cb276e740 4 client.26228 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-30T08:37:06.008 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.004+0000 7f8cb276e740 1 -- 172.21.15.32:0/371390130 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557599f9f4b0 con 0x557599caa1c0 2024-04-30T08:37:06.008 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.004+0000 7f8ca8c00640 1 --2- 172.21.15.32:0/371390130 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x557599caa1c0 0x55759a1391f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:06.008 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.004+0000 7f8c8a000640 10 client.26228 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:06.009 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.004+0000 7f8ca8c00640 1 --2- 172.21.15.32:0/371390130 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x557599caa1c0 0x55759a1391f0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89eb3dccbea1dba0 server_cookie=3950fc1d026d7c2 in_seq=0 out_seq=0 2024-04-30T08:37:06.080 INFO:tasks.workunit.client.0.smithi032.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-04-30T08:37:06.258 INFO:tasks.workunit.client.0.smithi032.stdout:checking pthread.h usability... yes 2024-04-30T08:37:06.295 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.293+0000 7f8c8a000640 1 -- 172.21.15.32:0/371390130 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557599f9f4b0 con 0x557599caa1c0 2024-04-30T08:37:06.295 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.293+0000 7f8c8a000640 10 client.26228 handle_command_reply: tid=0 2024-04-30T08:37:06.295 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.293+0000 7f8cb276e740 1 client.26228 shutdown 2024-04-30T08:37:06.295 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.294+0000 7f8cb276e740 1 -- 172.21.15.32:0/371390130 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557599d161b0 msgr2=0x55759a0c6570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:06.295 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.294+0000 7f8cb276e740 1 --2- 172.21.15.32:0/371390130 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557599d161b0 0x55759a0c6570 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f8c9c003f50 tx=0x7f8c9c004ab0 comp rx=0 tx=0).stop 2024-04-30T08:37:06.295 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.294+0000 7f8cb276e740 1 -- 172.21.15.32:0/371390130 shutdown_connections 2024-04-30T08:37:06.295 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.294+0000 7f8cb276e740 1 --2- 172.21.15.32:0/371390130 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x557599caa1c0 0x55759a1391f0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.294+0000 7f8cb276e740 1 --2- 172.21.15.32:0/371390130 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557599d161b0 0x55759a0c6570 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.294+0000 7f8cb276e740 1 --2- 172.21.15.32:0/371390130 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557599cf54d0 0x55759a0c6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.294+0000 7f8c8a000640 0 client.26228 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.294+0000 7f8cb276e740 1 --2- 172.21.15.32:0/371390130 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557599ced2f0 0x55759a0c94b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.294+0000 7f8cb276e740 1 -- 172.21.15.32:0/371390130 >> 172.21.15.32:0/371390130 conn(0x557599d0a300 msgr2=0x55759a0b6870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.294+0000 7f8cb276e740 1 -- 172.21.15.32:0/371390130 shutdown_connections 2024-04-30T08:37:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.294+0000 7f8cb276e740 1 -- 172.21.15.32:0/371390130 wait complete. 2024-04-30T08:37:06.296 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.294+0000 7f8cb276e740 20 client.26228 trim_cache size 0 max 16384 2024-04-30T08:37:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-30T08:37:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-30T08:37:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-30T08:37:06.297 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-30T08:37:06.298 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.295+0000 7f8cab600640 1 -- 172.21.15.32:0/1985297634 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8c8c0afae0 msgr2=0x7f8c8c0b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:06.298 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.295+0000 7f8cab600640 1 --2- 172.21.15.32:0/1985297634 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8c8c0afae0 0x7f8c8c0b1ff0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x55759a0b92b0 tx=0x7f8c9c009500 comp rx=0 tx=0).stop 2024-04-30T08:37:06.298 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.295+0000 7f8cab600640 1 -- 172.21.15.32:0/1985297634 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ca4104e60 msgr2=0x7f8ca4076080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:06.298 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.295+0000 7f8cab600640 1 --2- 172.21.15.32:0/1985297634 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ca4104e60 0x7f8ca4076080 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8c9000e9f0 tx=0x7f8c9000eeb0 comp rx=0 tx=0).stop 2024-04-30T08:37:06.298 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.295+0000 7f8cab600640 1 -- 172.21.15.32:0/1985297634 shutdown_connections 2024-04-30T08:37:06.298 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.295+0000 7f8cab600640 1 --2- 172.21.15.32:0/1985297634 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f8c8c0afae0 0x7f8c8c0b1ff0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:06.298 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.295+0000 7f8cab600640 1 --2- 172.21.15.32:0/1985297634 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f8ca4109b60 0x7f8ca40765c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:06.298 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.295+0000 7f8cab600640 1 --2- 172.21.15.32:0/1985297634 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8ca4104e60 0x7f8ca4076080 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:06.298 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.295+0000 7f8cab600640 1 --2- 172.21.15.32:0/1985297634 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8ca4103c20 0x7f8ca40779e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:06.298 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.295+0000 7f8cab600640 1 -- 172.21.15.32:0/1985297634 >> 172.21.15.32:0/1985297634 conn(0x7f8ca40fedf0 msgr2=0x7f8ca4105b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:06.299 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.295+0000 7f8cab600640 1 -- 172.21.15.32:0/1985297634 shutdown_connections 2024-04-30T08:37:06.299 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:06.295+0000 7f8cab600640 1 -- 172.21.15.32:0/1985297634 wait complete. 2024-04-30T08:37:06.299 INFO:tasks.workunit.client.0.smithi032.stdout:checking pthread.h presence... yes 2024-04-30T08:37:06.299 INFO:tasks.workunit.client.0.smithi032.stdout:checking for pthread.h... yes 2024-04-30T08:37:06.388 INFO:tasks.workunit.client.0.smithi032.stdout:checking fcntl.h usability... yes 2024-04-30T08:37:06.420 INFO:tasks.workunit.client.0.smithi032.stdout:checking fcntl.h presence... yes 2024-04-30T08:37:06.420 INFO:tasks.workunit.client.0.smithi032.stdout:checking for fcntl.h... yes 2024-04-30T08:37:06.527 INFO:tasks.workunit.client.0.smithi032.stdout:checking limits.h usability... yes 2024-04-30T08:37:06.568 INFO:tasks.workunit.client.0.smithi032.stdout:checking limits.h presence... yes 2024-04-30T08:37:06.568 INFO:tasks.workunit.client.0.smithi032.stdout:checking for limits.h... yes 2024-04-30T08:37:06.604 INFO:tasks.workunit.client.0.smithi032.stdout:checking for stdint.h... (cached) yes 2024-04-30T08:37:06.691 INFO:tasks.workunit.client.0.smithi032.stdout:checking sys/time.h usability... yes 2024-04-30T08:37:06.731 INFO:tasks.workunit.client.0.smithi032.stdout:checking sys/time.h presence... yes 2024-04-30T08:37:06.732 INFO:tasks.workunit.client.0.smithi032.stdout:checking for sys/time.h... yes 2024-04-30T08:37:06.769 INFO:tasks.workunit.client.0.smithi032.stdout:checking for unistd.h... (cached) yes 2024-04-30T08:37:06.865 INFO:tasks.workunit.client.0.smithi032.stdout:checking sys/vfs.h usability... yes 2024-04-30T08:37:06.905 INFO:tasks.workunit.client.0.smithi032.stdout:checking sys/vfs.h presence... yes 2024-04-30T08:37:06.905 INFO:tasks.workunit.client.0.smithi032.stdout:checking for sys/vfs.h... yes 2024-04-30T08:37:07.040 INFO:tasks.workunit.client.0.smithi032.stdout:checking sys/limits.h usability... no 2024-04-30T08:37:07.106 INFO:tasks.workunit.client.0.smithi032.stdout:checking sys/limits.h presence... no 2024-04-30T08:37:07.106 INFO:tasks.workunit.client.0.smithi032.stdout:checking for sys/limits.h... no 2024-04-30T08:37:07.170 INFO:tasks.workunit.client.0.smithi032.stdout:checking for an ANSI C-conforming const... yes 2024-04-30T08:37:07.243 INFO:tasks.workunit.client.0.smithi032.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-04-30T08:37:07.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:07 smithi189 ceph-mon[33038]: pgmap v483: 129 pgs: 129 active+clean; 5.0 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 427 KiB/s wr, 69 op/s 2024-04-30T08:37:07.433 INFO:tasks.workunit.client.0.smithi032.stdout:checking for system... yes 2024-04-30T08:37:07.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:07 smithi032 ceph-mon[28331]: pgmap v483: 129 pgs: 129 active+clean; 5.0 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 427 KiB/s wr, 69 op/s 2024-04-30T08:37:07.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:07 smithi071 ceph-mon[33385]: pgmap v483: 129 pgs: 129 active+clean; 5.0 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 315 KiB/s rd, 427 KiB/s wr, 69 op/s 2024-04-30T08:37:07.565 INFO:tasks.workunit.client.0.smithi032.stdout:checking for gettimeofday... yes 2024-04-30T08:37:07.677 INFO:tasks.workunit.client.0.smithi032.stdout:checking for mkdir... yes 2024-04-30T08:37:07.803 INFO:tasks.workunit.client.0.smithi032.stdout:checking for strerror... yes 2024-04-30T08:37:07.941 INFO:tasks.workunit.client.0.smithi032.stdout:checking for open64... yes 2024-04-30T08:37:08.060 INFO:tasks.workunit.client.0.smithi032.stdout:checking for stat64... yes 2024-04-30T08:37:08.190 INFO:tasks.workunit.client.0.smithi032.stdout:checking for fseeko64... yes 2024-04-30T08:37:08.320 INFO:tasks.workunit.client.0.smithi032.stdout:checking for lrand48_r... yes 2024-04-30T08:37:08.458 INFO:tasks.workunit.client.0.smithi032.stdout:checking for srand48_r... yes 2024-04-30T08:37:08.539 INFO:tasks.workunit.client.0.smithi032.stdout:configure: creating ./config.status 2024-04-30T08:37:08.674 INFO:tasks.workunit.client.0.smithi032.stdout: /bin/sh ./config.status 2024-04-30T08:37:08.790 INFO:tasks.workunit.client.0.smithi032.stdout:config.status: creating Makefile 2024-04-30T08:37:08.829 INFO:tasks.workunit.client.0.smithi032.stdout:config.status: creating config.h 2024-04-30T08:37:08.882 INFO:tasks.workunit.client.0.smithi032.stdout:config.status: executing depfiles commands 2024-04-30T08:37:08.929 INFO:tasks.workunit.client.0.smithi032.stdout:config.status: executing default commands 2024-04-30T08:37:08.980 INFO:tasks.workunit.client.0.smithi032.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoheader 2024-04-30T08:37:09.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:09 smithi189 ceph-mon[33038]: pgmap v484: 129 pgs: 129 active+clean; 5.0 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 304 KiB/s wr, 71 op/s 2024-04-30T08:37:09.300 INFO:tasks.workunit.client.0.smithi032.stdout:rm -f stamp-h1 2024-04-30T08:37:09.304 INFO:tasks.workunit.client.0.smithi032.stdout:touch config.h.in 2024-04-30T08:37:09.309 INFO:tasks.workunit.client.0.smithi032.stdout:cd . && /bin/sh ./config.status config.h 2024-04-30T08:37:09.424 INFO:tasks.workunit.client.0.smithi032.stdout:config.status: creating config.h 2024-04-30T08:37:09.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:09 smithi032 ceph-mon[28331]: pgmap v484: 129 pgs: 129 active+clean; 5.0 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 304 KiB/s wr, 71 op/s 2024-04-30T08:37:09.477 INFO:tasks.workunit.client.0.smithi032.stdout:make all-am 2024-04-30T08:37:09.487 INFO:tasks.workunit.client.0.smithi032.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-04-30T08:37:09.487 INFO:tasks.workunit.client.0.smithi032.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2024-04-30T08:37:09.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:09 smithi071 ceph-mon[33385]: pgmap v484: 129 pgs: 129 active+clean; 5.0 MiB data, 380 MiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 304 KiB/s wr, 71 op/s 2024-04-30T08:37:09.786 INFO:tasks.workunit.client.0.smithi032.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2024-04-30T08:37:09.791 INFO:tasks.workunit.client.0.smithi032.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2024-04-30T08:37:09.942 INFO:tasks.workunit.client.0.smithi032.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2024-04-30T08:37:09.948 INFO:tasks.workunit.client.0.smithi032.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2024-04-30T08:37:10.083 INFO:tasks.workunit.client.0.smithi032.stdout:mv -f .deps/main.Tpo .deps/main.Po 2024-04-30T08:37:10.090 INFO:tasks.workunit.client.0.smithi032.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2024-04-30T08:37:10.230 INFO:tasks.workunit.client.0.smithi032.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2024-04-30T08:37:10.235 INFO:tasks.workunit.client.0.smithi032.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2024-04-30T08:37:10.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:10 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:37:10.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:10 smithi189 ceph-mon[33038]: scrub summary: idle 2024-04-30T08:37:10.422 INFO:tasks.workunit.client.0.smithi032.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2024-04-30T08:37:10.428 INFO:tasks.workunit.client.0.smithi032.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2024-04-30T08:37:10.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:10 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:37:10.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:10 smithi032 ceph-mon[28331]: scrub summary: idle 2024-04-30T08:37:10.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:10 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:37:10.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:10 smithi071 ceph-mon[33385]: scrub summary: idle 2024-04-30T08:37:10.513 INFO:tasks.workunit.client.0.smithi032.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2024-04-30T08:37:10.518 INFO:tasks.workunit.client.0.smithi032.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2024-04-30T08:37:10.602 INFO:tasks.workunit.client.0.smithi032.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2024-04-30T08:37:10.608 INFO:tasks.workunit.client.0.smithi032.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2024-04-30T08:37:10.690 INFO:tasks.workunit.client.0.smithi032.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2024-04-30T08:37:10.694 INFO:tasks.workunit.client.0.smithi032.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2024-04-30T08:37:10.964 INFO:tasks.workunit.client.0.smithi032.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2024-04-30T08:37:10.971 INFO:tasks.workunit.client.0.smithi032.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2024-04-30T08:37:11.184 INFO:tasks.workunit.client.0.smithi032.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2024-04-30T08:37:11.190 INFO:tasks.workunit.client.0.smithi032.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2024-04-30T08:37:11.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:11 smithi189 ceph-mon[33038]: scrub summary: idle 2024-04-30T08:37:11.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:11 smithi189 ceph-mon[33038]: scrub summary: idle 2024-04-30T08:37:11.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:11 smithi189 ceph-mon[33038]: scrub summary: idle 2024-04-30T08:37:11.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:11 smithi189 ceph-mon[33038]: pgmap v485: 129 pgs: 129 active+clean; 2.5 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 485 KiB/s wr, 154 op/s 2024-04-30T08:37:11.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:11 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:37:11.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:11 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:37:11.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:11 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:37:11.411 INFO:tasks.workunit.client.0.smithi032.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2024-04-30T08:37:11.419 INFO:tasks.workunit.client.0.smithi032.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2024-04-30T08:37:11.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:11 smithi032 ceph-mon[28331]: scrub summary: idle 2024-04-30T08:37:11.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:11 smithi032 ceph-mon[28331]: scrub summary: idle 2024-04-30T08:37:11.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:11 smithi032 ceph-mon[28331]: scrub summary: idle 2024-04-30T08:37:11.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:11 smithi032 ceph-mon[28331]: pgmap v485: 129 pgs: 129 active+clean; 2.5 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 485 KiB/s wr, 154 op/s 2024-04-30T08:37:11.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:11 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:37:11.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:11 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:37:11.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:11 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:37:11.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:11 smithi071 ceph-mon[33385]: scrub summary: idle 2024-04-30T08:37:11.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:11 smithi071 ceph-mon[33385]: scrub summary: idle 2024-04-30T08:37:11.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:11 smithi071 ceph-mon[33385]: scrub summary: idle 2024-04-30T08:37:11.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:11 smithi071 ceph-mon[33385]: pgmap v485: 129 pgs: 129 active+clean; 2.5 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 274 KiB/s rd, 485 KiB/s wr, 154 op/s 2024-04-30T08:37:11.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:11 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:37:11.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:11 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:37:11.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:11 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' 2024-04-30T08:37:11.533 INFO:tasks.workunit.client.0.smithi032.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2024-04-30T08:37:11.540 INFO:tasks.workunit.client.0.smithi032.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2024-04-30T08:37:11.657 INFO:tasks.workunit.client.0.smithi032.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2024-04-30T08:37:11.663 INFO:tasks.workunit.client.0.smithi032.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2024-04-30T08:37:11.827 INFO:tasks.workunit.client.0.smithi032.stdout:mv -f .deps/util.Tpo .deps/util.Po 2024-04-30T08:37:11.833 INFO:tasks.workunit.client.0.smithi032.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2024-04-30T08:37:11.892 INFO:tasks.workunit.client.0.smithi032.stderr:In file included from parser.h:22, 2024-04-30T08:37:11.914 INFO:tasks.workunit.client.0.smithi032.stderr: from parser.c:25: 2024-04-30T08:37:11.914 INFO:tasks.workunit.client.0.smithi032.stderr:parser.c: In function ‘init_filesys’: 2024-04-30T08:37:11.914 INFO:tasks.workunit.client.0.smithi032.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-30T08:37:11.914 INFO:tasks.workunit.client.0.smithi032.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-30T08:37:11.914 INFO:tasks.workunit.client.0.smithi032.stderr: | ^ 2024-04-30T08:37:11.914 INFO:tasks.workunit.client.0.smithi032.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-30T08:37:11.915 INFO:tasks.workunit.client.0.smithi032.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-30T08:37:11.915 INFO:tasks.workunit.client.0.smithi032.stderr: | ^~~~~~~~ 2024-04-30T08:37:11.915 INFO:tasks.workunit.client.0.smithi032.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-30T08:37:11.915 INFO:tasks.workunit.client.0.smithi032.stderr: 36 | container_of(ptr, type, member) 2024-04-30T08:37:11.915 INFO:tasks.workunit.client.0.smithi032.stderr: | ^~~~~~~~~~~~ 2024-04-30T08:37:11.915 INFO:tasks.workunit.client.0.smithi032.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-04-30T08:37:11.915 INFO:tasks.workunit.client.0.smithi032.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-30T08:37:11.915 INFO:tasks.workunit.client.0.smithi032.stderr: | ^~~~~~~~~~ 2024-04-30T08:37:11.915 INFO:tasks.workunit.client.0.smithi032.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-04-30T08:37:11.916 INFO:tasks.workunit.client.0.smithi032.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-04-30T08:37:11.916 INFO:tasks.workunit.client.0.smithi032.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-30T08:37:11.916 INFO:tasks.workunit.client.0.smithi032.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-30T08:37:11.916 INFO:tasks.workunit.client.0.smithi032.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-30T08:37:11.916 INFO:tasks.workunit.client.0.smithi032.stderr: | ^ 2024-04-30T08:37:11.916 INFO:tasks.workunit.client.0.smithi032.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-30T08:37:11.916 INFO:tasks.workunit.client.0.smithi032.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-30T08:37:11.916 INFO:tasks.workunit.client.0.smithi032.stderr: | ^~~~~~~~ 2024-04-30T08:37:11.916 INFO:tasks.workunit.client.0.smithi032.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-30T08:37:11.916 INFO:tasks.workunit.client.0.smithi032.stderr: 36 | container_of(ptr, type, member) 2024-04-30T08:37:11.917 INFO:tasks.workunit.client.0.smithi032.stderr: | ^~~~~~~~~~~~ 2024-04-30T08:37:11.917 INFO:tasks.workunit.client.0.smithi032.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-04-30T08:37:11.917 INFO:tasks.workunit.client.0.smithi032.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-30T08:37:11.917 INFO:tasks.workunit.client.0.smithi032.stderr: | ^~~~~~~~~~ 2024-04-30T08:37:11.917 INFO:tasks.workunit.client.0.smithi032.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-04-30T08:37:11.917 INFO:tasks.workunit.client.0.smithi032.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-04-30T08:37:11.917 INFO:tasks.workunit.client.0.smithi032.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-30T08:37:11.917 INFO:tasks.workunit.client.0.smithi032.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-30T08:37:11.917 INFO:tasks.workunit.client.0.smithi032.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-30T08:37:11.917 INFO:tasks.workunit.client.0.smithi032.stderr: | ^ 2024-04-30T08:37:11.917 INFO:tasks.workunit.client.0.smithi032.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-30T08:37:11.917 INFO:tasks.workunit.client.0.smithi032.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-30T08:37:11.918 INFO:tasks.workunit.client.0.smithi032.stderr: | ^~~~~~~~ 2024-04-30T08:37:11.918 INFO:tasks.workunit.client.0.smithi032.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-30T08:37:11.918 INFO:tasks.workunit.client.0.smithi032.stderr: 36 | container_of(ptr, type, member) 2024-04-30T08:37:11.918 INFO:tasks.workunit.client.0.smithi032.stderr: | ^~~~~~~~~~~~ 2024-04-30T08:37:11.918 INFO:tasks.workunit.client.0.smithi032.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-04-30T08:37:11.918 INFO:tasks.workunit.client.0.smithi032.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-30T08:37:11.918 INFO:tasks.workunit.client.0.smithi032.stderr: | ^~~~~~~~~~ 2024-04-30T08:37:11.918 INFO:tasks.workunit.client.0.smithi032.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-04-30T08:37:11.918 INFO:tasks.workunit.client.0.smithi032.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-04-30T08:37:11.918 INFO:tasks.workunit.client.0.smithi032.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-30T08:37:11.918 INFO:tasks.workunit.client.0.smithi032.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-30T08:37:11.918 INFO:tasks.workunit.client.0.smithi032.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-30T08:37:11.919 INFO:tasks.workunit.client.0.smithi032.stderr: | ^ 2024-04-30T08:37:11.919 INFO:tasks.workunit.client.0.smithi032.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-30T08:37:11.919 INFO:tasks.workunit.client.0.smithi032.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-30T08:37:11.919 INFO:tasks.workunit.client.0.smithi032.stderr: | ^~~~~~~~ 2024-04-30T08:37:11.919 INFO:tasks.workunit.client.0.smithi032.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-30T08:37:11.919 INFO:tasks.workunit.client.0.smithi032.stderr: 36 | container_of(ptr, type, member) 2024-04-30T08:37:11.919 INFO:tasks.workunit.client.0.smithi032.stderr: | ^~~~~~~~~~~~ 2024-04-30T08:37:11.919 INFO:tasks.workunit.client.0.smithi032.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-04-30T08:37:11.919 INFO:tasks.workunit.client.0.smithi032.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-30T08:37:11.919 INFO:tasks.workunit.client.0.smithi032.stderr: | ^~~~~~~~~~ 2024-04-30T08:37:11.919 INFO:tasks.workunit.client.0.smithi032.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-04-30T08:37:11.919 INFO:tasks.workunit.client.0.smithi032.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-04-30T08:37:11.920 INFO:tasks.workunit.client.0.smithi032.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-30T08:37:11.920 INFO:tasks.workunit.client.0.smithi032.stderr:parser.c: In function ‘init_tg_stats’: 2024-04-30T08:37:11.920 INFO:tasks.workunit.client.0.smithi032.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-30T08:37:11.920 INFO:tasks.workunit.client.0.smithi032.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-30T08:37:11.920 INFO:tasks.workunit.client.0.smithi032.stderr: | ^ 2024-04-30T08:37:11.920 INFO:tasks.workunit.client.0.smithi032.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-30T08:37:11.920 INFO:tasks.workunit.client.0.smithi032.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-30T08:37:11.920 INFO:tasks.workunit.client.0.smithi032.stderr: | ^~~~~~~~ 2024-04-30T08:37:11.920 INFO:tasks.workunit.client.0.smithi032.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-30T08:37:11.920 INFO:tasks.workunit.client.0.smithi032.stderr: 36 | container_of(ptr, type, member) 2024-04-30T08:37:11.920 INFO:tasks.workunit.client.0.smithi032.stderr: | ^~~~~~~~~~~~ 2024-04-30T08:37:11.920 INFO:tasks.workunit.client.0.smithi032.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-04-30T08:37:11.921 INFO:tasks.workunit.client.0.smithi032.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-30T08:37:11.921 INFO:tasks.workunit.client.0.smithi032.stderr: | ^~~~~~~~~~ 2024-04-30T08:37:11.921 INFO:tasks.workunit.client.0.smithi032.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-04-30T08:37:11.921 INFO:tasks.workunit.client.0.smithi032.stderr: 971 | list_for_each_entry(tmp_list, 2024-04-30T08:37:11.921 INFO:tasks.workunit.client.0.smithi032.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-30T08:37:11.921 INFO:tasks.workunit.client.0.smithi032.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-30T08:37:11.921 INFO:tasks.workunit.client.0.smithi032.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-30T08:37:11.921 INFO:tasks.workunit.client.0.smithi032.stderr: | ^ 2024-04-30T08:37:11.921 INFO:tasks.workunit.client.0.smithi032.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-30T08:37:11.921 INFO:tasks.workunit.client.0.smithi032.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-30T08:37:11.921 INFO:tasks.workunit.client.0.smithi032.stderr: | ^~~~~~~~ 2024-04-30T08:37:11.921 INFO:tasks.workunit.client.0.smithi032.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-30T08:37:11.921 INFO:tasks.workunit.client.0.smithi032.stderr: 36 | container_of(ptr, type, member) 2024-04-30T08:37:11.922 INFO:tasks.workunit.client.0.smithi032.stderr: | ^~~~~~~~~~~~ 2024-04-30T08:37:11.922 INFO:tasks.workunit.client.0.smithi032.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-04-30T08:37:11.922 INFO:tasks.workunit.client.0.smithi032.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-30T08:37:11.922 INFO:tasks.workunit.client.0.smithi032.stderr: | ^~~~~~~~~~ 2024-04-30T08:37:11.922 INFO:tasks.workunit.client.0.smithi032.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-04-30T08:37:11.922 INFO:tasks.workunit.client.0.smithi032.stderr: 971 | list_for_each_entry(tmp_list, 2024-04-30T08:37:11.922 INFO:tasks.workunit.client.0.smithi032.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-30T08:37:11.922 INFO:tasks.workunit.client.0.smithi032.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-30T08:37:11.922 INFO:tasks.workunit.client.0.smithi032.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-30T08:37:11.922 INFO:tasks.workunit.client.0.smithi032.stderr: | ^ 2024-04-30T08:37:11.923 INFO:tasks.workunit.client.0.smithi032.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-30T08:37:11.923 INFO:tasks.workunit.client.0.smithi032.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-30T08:37:11.923 INFO:tasks.workunit.client.0.smithi032.stderr: | ^~~~~~~~ 2024-04-30T08:37:11.923 INFO:tasks.workunit.client.0.smithi032.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-30T08:37:11.923 INFO:tasks.workunit.client.0.smithi032.stderr: 36 | container_of(ptr, type, member) 2024-04-30T08:37:11.923 INFO:tasks.workunit.client.0.smithi032.stderr: | ^~~~~~~~~~~~ 2024-04-30T08:37:11.923 INFO:tasks.workunit.client.0.smithi032.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-04-30T08:37:11.923 INFO:tasks.workunit.client.0.smithi032.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-30T08:37:11.923 INFO:tasks.workunit.client.0.smithi032.stderr: | ^~~~~~~~~~ 2024-04-30T08:37:11.923 INFO:tasks.workunit.client.0.smithi032.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-04-30T08:37:11.923 INFO:tasks.workunit.client.0.smithi032.stderr: 980 | list_for_each_entry(tmp_list, 2024-04-30T08:37:11.923 INFO:tasks.workunit.client.0.smithi032.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-30T08:37:11.924 INFO:tasks.workunit.client.0.smithi032.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-30T08:37:11.924 INFO:tasks.workunit.client.0.smithi032.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-30T08:37:11.924 INFO:tasks.workunit.client.0.smithi032.stderr: | ^ 2024-04-30T08:37:11.924 INFO:tasks.workunit.client.0.smithi032.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-30T08:37:11.924 INFO:tasks.workunit.client.0.smithi032.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-30T08:37:11.924 INFO:tasks.workunit.client.0.smithi032.stderr: | ^~~~~~~~ 2024-04-30T08:37:11.924 INFO:tasks.workunit.client.0.smithi032.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-30T08:37:11.924 INFO:tasks.workunit.client.0.smithi032.stderr: 36 | container_of(ptr, type, member) 2024-04-30T08:37:11.924 INFO:tasks.workunit.client.0.smithi032.stderr: | ^~~~~~~~~~~~ 2024-04-30T08:37:11.924 INFO:tasks.workunit.client.0.smithi032.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-04-30T08:37:11.924 INFO:tasks.workunit.client.0.smithi032.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-30T08:37:11.924 INFO:tasks.workunit.client.0.smithi032.stderr: | ^~~~~~~~~~ 2024-04-30T08:37:11.925 INFO:tasks.workunit.client.0.smithi032.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-04-30T08:37:11.925 INFO:tasks.workunit.client.0.smithi032.stderr: 980 | list_for_each_entry(tmp_list, 2024-04-30T08:37:11.925 INFO:tasks.workunit.client.0.smithi032.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-30T08:37:11.960 INFO:tasks.workunit.client.0.smithi032.stderr:parser.c: In function ‘get_optstr’: 2024-04-30T08:37:11.960 INFO:tasks.workunit.client.0.smithi032.stderr:parser.c:206:23: warning: ‘strnlen’ specified bound 4095 exceeds source size 1024 [-Wstringop-overread] 2024-04-30T08:37:11.960 INFO:tasks.workunit.client.0.smithi032.stderr: 206 | len = strnlen(temp, 4095) + 1; 2024-04-30T08:37:11.960 INFO:tasks.workunit.client.0.smithi032.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-30T08:37:11.960 INFO:tasks.workunit.client.0.smithi032.stderr:parser.c:200:14: note: source object declared here 2024-04-30T08:37:11.960 INFO:tasks.workunit.client.0.smithi032.stderr: 200 | char temp[BUFSIZE]; 2024-04-30T08:37:11.960 INFO:tasks.workunit.client.0.smithi032.stderr: | ^~~~ 2024-04-30T08:37:11.967 INFO:tasks.workunit.client.0.smithi032.stderr:parser.c: In function ‘size64_convert’: 2024-04-30T08:37:11.967 INFO:tasks.workunit.client.0.smithi032.stderr:parser.c:119:9: warning: ‘memcpy’ reading 3 bytes from a region of size 2 [-Wstringop-overread] 2024-04-30T08:37:11.968 INFO:tasks.workunit.client.0.smithi032.stderr: 119 | memcpy(unit, "\0", 3); 2024-04-30T08:37:11.968 INFO:tasks.workunit.client.0.smithi032.stderr: | ^~~~~~~~~~~~~~~~~~~~~ 2024-04-30T08:37:12.335 INFO:tasks.workunit.client.0.smithi032.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2024-04-30T08:37:12.340 INFO:tasks.workunit.client.0.smithi032.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2024-04-30T08:37:12.439 INFO:tasks.workunit.client.0.smithi032.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2024-04-30T08:37:12.445 INFO:tasks.workunit.client.0.smithi032.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2024-04-30T08:37:12.588 INFO:tasks.workunit.client.0.smithi032.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2024-04-30T08:37:12.595 INFO:tasks.workunit.client.0.smithi032.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2024-04-30T08:37:12.661 INFO:tasks.workunit.client.0.smithi032.stdout:mv -f .deps/list.Tpo .deps/list.Po 2024-04-30T08:37:12.670 INFO:tasks.workunit.client.0.smithi032.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2024-04-30T08:37:12.745 INFO:tasks.workunit.client.0.smithi032.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-04-30T08:37:12.748 INFO:tasks.workunit.client.0.smithi032.stderr:+ cd .. 2024-04-30T08:37:12.748 INFO:tasks.workunit.client.0.smithi032.stderr:+ mkdir tmp 2024-04-30T08:37:12.753 INFO:tasks.workunit.client.0.smithi032.stderr:+ cd tmp 2024-04-30T08:37:12.754 INFO:tasks.workunit.client.0.smithi032.stderr:+ for f in $mydir/*.ffsb 2024-04-30T08:37:12.754 INFO:tasks.workunit.client.0.smithi032.stderr:+ ../ffsb/ffsb /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/random_write.32.ffsb 2024-04-30T08:37:12.756 INFO:tasks.workunit.client.0.smithi032.stdout:FFSB version 6.0-RC2 started 2024-04-30T08:37:12.756 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-04-30T08:37:12.764 INFO:tasks.workunit.client.0.smithi032.stdout:benchmark time = 300 2024-04-30T08:37:12.764 INFO:tasks.workunit.client.0.smithi032.stdout:ThreadGroup 0 2024-04-30T08:37:12.764 INFO:tasks.workunit.client.0.smithi032.stdout:================ 2024-04-30T08:37:12.764 INFO:tasks.workunit.client.0.smithi032.stdout: num_threads = 32 2024-04-30T08:37:12.764 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-04-30T08:37:12.764 INFO:tasks.workunit.client.0.smithi032.stdout: read_random = off 2024-04-30T08:37:12.765 INFO:tasks.workunit.client.0.smithi032.stdout: read_size = 0 (0B) 2024-04-30T08:37:12.765 INFO:tasks.workunit.client.0.smithi032.stdout: read_blocksize = 0 (0B) 2024-04-30T08:37:12.765 INFO:tasks.workunit.client.0.smithi032.stdout: read_skip = off 2024-04-30T08:37:12.765 INFO:tasks.workunit.client.0.smithi032.stdout: read_skipsize = 0 (0B) 2024-04-30T08:37:12.765 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-04-30T08:37:12.765 INFO:tasks.workunit.client.0.smithi032.stdout: write_random = on 2024-04-30T08:37:12.765 INFO:tasks.workunit.client.0.smithi032.stdout: write_size = 5242880 (5MB) 2024-04-30T08:37:12.765 INFO:tasks.workunit.client.0.smithi032.stdout: fsync_file = 0 2024-04-30T08:37:12.765 INFO:tasks.workunit.client.0.smithi032.stdout: write_blocksize = 4096 (4KB) 2024-04-30T08:37:12.765 INFO:tasks.workunit.client.0.smithi032.stdout: wait time = 0 2024-04-30T08:37:12.765 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-04-30T08:37:12.765 INFO:tasks.workunit.client.0.smithi032.stdout: op weights 2024-04-30T08:37:12.765 INFO:tasks.workunit.client.0.smithi032.stdout: read = 0 (0.00%) 2024-04-30T08:37:12.766 INFO:tasks.workunit.client.0.smithi032.stdout: readall = 0 (0.00%) 2024-04-30T08:37:12.766 INFO:tasks.workunit.client.0.smithi032.stdout: write = 1 (100.00%) 2024-04-30T08:37:12.766 INFO:tasks.workunit.client.0.smithi032.stdout: create = 0 (0.00%) 2024-04-30T08:37:12.766 INFO:tasks.workunit.client.0.smithi032.stdout: append = 0 (0.00%) 2024-04-30T08:37:12.766 INFO:tasks.workunit.client.0.smithi032.stdout: delete = 0 (0.00%) 2024-04-30T08:37:12.766 INFO:tasks.workunit.client.0.smithi032.stdout: metaop = 0 (0.00%) 2024-04-30T08:37:12.766 INFO:tasks.workunit.client.0.smithi032.stdout: createdir = 0 (0.00%) 2024-04-30T08:37:12.766 INFO:tasks.workunit.client.0.smithi032.stdout: stat = 0 (0.00%) 2024-04-30T08:37:12.766 INFO:tasks.workunit.client.0.smithi032.stdout: writeall = 0 (0.00%) 2024-04-30T08:37:12.766 INFO:tasks.workunit.client.0.smithi032.stdout: writeall_fsync = 0 (0.00%) 2024-04-30T08:37:12.766 INFO:tasks.workunit.client.0.smithi032.stdout: open_close = 0 (0.00%) 2024-04-30T08:37:12.766 INFO:tasks.workunit.client.0.smithi032.stdout: write_fsync = 0 (0.00%) 2024-04-30T08:37:12.767 INFO:tasks.workunit.client.0.smithi032.stdout: create_fsync = 0 (0.00%) 2024-04-30T08:37:12.767 INFO:tasks.workunit.client.0.smithi032.stdout: append_fsync = 0 (0.00%) 2024-04-30T08:37:12.767 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-04-30T08:37:12.767 INFO:tasks.workunit.client.0.smithi032.stdout:FileSystem . 2024-04-30T08:37:12.767 INFO:tasks.workunit.client.0.smithi032.stdout:========== 2024-04-30T08:37:12.767 INFO:tasks.workunit.client.0.smithi032.stdout: num_dirs = 0 2024-04-30T08:37:12.767 INFO:tasks.workunit.client.0.smithi032.stdout: starting files = 128 2024-04-30T08:37:12.767 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-04-30T08:37:12.767 INFO:tasks.workunit.client.0.smithi032.stdout: min file size = 104857600 (100MB) 2024-04-30T08:37:12.767 INFO:tasks.workunit.client.0.smithi032.stdout: max file size = 104857600 (100MB) 2024-04-30T08:37:12.768 INFO:tasks.workunit.client.0.smithi032.stdout: directio = off 2024-04-30T08:37:12.768 INFO:tasks.workunit.client.0.smithi032.stdout: alignedio = on 2024-04-30T08:37:12.768 INFO:tasks.workunit.client.0.smithi032.stdout: bufferedio = off 2024-04-30T08:37:12.768 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-04-30T08:37:12.768 INFO:tasks.workunit.client.0.smithi032.stdout: aging is off 2024-04-30T08:37:12.769 INFO:tasks.workunit.client.0.smithi032.stdout: current utilization = 0.04% 2024-04-30T08:37:12.769 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-04-30T08:37:12.769 INFO:tasks.workunit.client.0.smithi032.stdout:checking existing fs: . 2024-04-30T08:37:12.769 INFO:tasks.workunit.client.0.smithi032.stdout:recreating new fileset 2024-04-30T08:37:12.769 INFO:tasks.workunit.client.0.smithi032.stderr:opendir: No such file or directory 2024-04-30T08:37:13.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:13 smithi189 ceph-mon[33038]: pgmap v486: 129 pgs: 129 active+clean; 2.7 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 347 KiB/s wr, 194 op/s 2024-04-30T08:37:13.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:13 smithi032 ceph-mon[28331]: pgmap v486: 129 pgs: 129 active+clean; 2.7 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 347 KiB/s wr, 194 op/s 2024-04-30T08:37:13.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:13 smithi071 ceph-mon[33385]: pgmap v486: 129 pgs: 129 active+clean; 2.7 MiB data, 396 MiB used, 1.0 TiB / 1.0 TiB avail; 158 KiB/s rd, 347 KiB/s wr, 194 op/s 2024-04-30T08:37:15.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:15 smithi032 ceph-mon[28331]: pgmap v487: 129 pgs: 129 active+clean; 3.2 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 358 KiB/s wr, 190 op/s 2024-04-30T08:37:15.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:15 smithi071 ceph-mon[33385]: pgmap v487: 129 pgs: 129 active+clean; 3.2 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 358 KiB/s wr, 190 op/s 2024-04-30T08:37:15.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:15 smithi189 ceph-mon[33038]: pgmap v487: 129 pgs: 129 active+clean; 3.2 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 358 KiB/s wr, 190 op/s 2024-04-30T08:37:17.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:17 smithi032 ceph-mon[28331]: pgmap v488: 129 pgs: 129 active+clean; 3.2 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 358 KiB/s wr, 188 op/s 2024-04-30T08:37:17.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:17 smithi071 ceph-mon[33385]: pgmap v488: 129 pgs: 129 active+clean; 3.2 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 358 KiB/s wr, 188 op/s 2024-04-30T08:37:17.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:17 smithi189 ceph-mon[33038]: pgmap v488: 129 pgs: 129 active+clean; 3.2 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 358 KiB/s wr, 188 op/s 2024-04-30T08:37:19.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:19 smithi032 ceph-mon[28331]: pgmap v489: 129 pgs: 129 active+clean; 3.2 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 283 KiB/s wr, 181 op/s 2024-04-30T08:37:19.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:19 smithi071 ceph-mon[33385]: pgmap v489: 129 pgs: 129 active+clean; 3.2 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 283 KiB/s wr, 181 op/s 2024-04-30T08:37:19.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.510+0000 7f494b200640 1 -- 172.21.15.32:0/4062467435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f494c104c20 msgr2=0x7f494c105030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:19.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.510+0000 7f494b200640 1 --2- 172.21.15.32:0/4062467435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f494c104c20 0x7f494c105030 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f4938009900 tx=0x7f493802a410 comp rx=0 tx=0).stop 2024-04-30T08:37:19.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.510+0000 7f494b200640 1 -- 172.21.15.32:0/4062467435 shutdown_connections 2024-04-30T08:37:19.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.510+0000 7f494b200640 1 --2- 172.21.15.32:0/4062467435 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f494c109920 0x7f494c10bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:19.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.510+0000 7f494b200640 1 --2- 172.21.15.32:0/4062467435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f494c104c20 0x7f494c105030 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:19.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.510+0000 7f494b200640 1 --2- 172.21.15.32:0/4062467435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f494c1039e0 0x7f494c1046e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:19.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.510+0000 7f494b200640 1 -- 172.21.15.32:0/4062467435 >> 172.21.15.32:0/4062467435 conn(0x7f494c0febd0 msgr2=0x7f494c101080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:19.512 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.510+0000 7f494b200640 1 -- 172.21.15.32:0/4062467435 shutdown_connections 2024-04-30T08:37:19.513 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.510+0000 7f494b200640 1 -- 172.21.15.32:0/4062467435 wait complete. 2024-04-30T08:37:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.511+0000 7f494b200640 1 Processor -- start 2024-04-30T08:37:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.511+0000 7f494b200640 1 -- start start 2024-04-30T08:37:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.512+0000 7f494b200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f494c1039e0 0x7f494c0768d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.512+0000 7f494b200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f494c104c20 0x7f494c074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.512+0000 7f494b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f494c109920 0x7f494c0754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.512+0000 7f494b200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f494c10f510 con 0x7f494c109920 2024-04-30T08:37:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.512+0000 7f494b200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f494c10f390 con 0x7f494c1039e0 2024-04-30T08:37:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.512+0000 7f494b200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f494c10f690 con 0x7f494c104c20 2024-04-30T08:37:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.512+0000 7f4949e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f494c1039e0 0x7f494c0768d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:19.514 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.512+0000 7f4949e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f494c1039e0 0x7f494c0768d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:44656/0 (socket says 172.21.15.32:44656) 2024-04-30T08:37:19.515 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.512+0000 7f4949e00640 1 -- 172.21.15.32:0/3428185497 learned_addr learned my addr 172.21.15.32:0/3428185497 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:19.515 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.512+0000 7f4949400640 1 --2- 172.21.15.32:0/3428185497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f494c104c20 0x7f494c074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:19.515 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.512+0000 7f494a800640 1 --2- 172.21.15.32:0/3428185497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f494c109920 0x7f494c0754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:19.515 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.512+0000 7f4949e00640 1 -- 172.21.15.32:0/3428185497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f494c104c20 msgr2=0x7f494c074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:19.515 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.512+0000 7f4949e00640 1 --2- 172.21.15.32:0/3428185497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f494c104c20 0x7f494c074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:19.515 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.512+0000 7f4949e00640 1 -- 172.21.15.32:0/3428185497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f494c109920 msgr2=0x7f494c0754b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:19.515 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.512+0000 7f4949e00640 1 --2- 172.21.15.32:0/3428185497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f494c109920 0x7f494c0754b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:19.515 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.512+0000 7f4949e00640 1 -- 172.21.15.32:0/3428185497 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f494c075be0 con 0x7f494c1039e0 2024-04-30T08:37:19.516 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.513+0000 7f4949e00640 1 --2- 172.21.15.32:0/3428185497 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f494c1039e0 0x7f494c0768d0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f493c005fd0 tx=0x7f493c00dde0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:19.516 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.513+0000 7f4942a00640 1 -- 172.21.15.32:0/3428185497 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f493c017070 con 0x7f494c1039e0 2024-04-30T08:37:19.516 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.513+0000 7f494b200640 1 -- 172.21.15.32:0/3428185497 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f494c1a4170 con 0x7f494c1039e0 2024-04-30T08:37:19.517 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.513+0000 7f4942a00640 1 -- 172.21.15.32:0/3428185497 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f493c009c30 con 0x7f494c1039e0 2024-04-30T08:37:19.517 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.513+0000 7f4942a00640 1 -- 172.21.15.32:0/3428185497 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f493c003d10 con 0x7f494c1039e0 2024-04-30T08:37:19.517 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.513+0000 7f494b200640 1 -- 172.21.15.32:0/3428185497 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f494c1a4660 con 0x7f494c1039e0 2024-04-30T08:37:19.518 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.516+0000 7f49581a4740 1 Processor -- start 2024-04-30T08:37:19.518 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.516+0000 7f49581a4740 1 -- start start 2024-04-30T08:37:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.517+0000 7f49581a4740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562e912881b0 0x562e9162a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.517+0000 7f49581a4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e9125f2f0 0x562e9162a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.517+0000 7f49581a4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562e912674d0 0x562e91631760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.517+0000 7f49581a4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x562e9158e550 con 0x562e912674d0 2024-04-30T08:37:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.517+0000 7f49581a4740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x562e91533bc0 con 0x562e912881b0 2024-04-30T08:37:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.517+0000 7f49581a4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562e9158a510 con 0x562e9125f2f0 2024-04-30T08:37:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.517+0000 7f494a800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562e912881b0 0x562e9162a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.517+0000 7f4949400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562e912674d0 0x562e91631760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.517+0000 7f494a800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562e912881b0 0x562e9162a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:44666/0 (socket says 172.21.15.32:44666) 2024-04-30T08:37:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.517+0000 7f494a800640 1 -- 172.21.15.32:0/3932043507 learned_addr learned my addr 172.21.15.32:0/3932043507 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.518+0000 7f494a800640 1 -- 172.21.15.32:0/3932043507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e9125f2f0 msgr2=0x562e9162a860 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-30T08:37:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.518+0000 7f494a800640 1 --2- 172.21.15.32:0/3932043507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e9125f2f0 0x562e9162a860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.518+0000 7f494a800640 1 -- 172.21.15.32:0/3932043507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562e912674d0 msgr2=0x562e91631760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.518+0000 7f494a800640 1 --2- 172.21.15.32:0/3932043507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562e912674d0 0x562e91631760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.518+0000 7f494a800640 1 -- 172.21.15.32:0/3932043507 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e912a0210 con 0x562e912881b0 2024-04-30T08:37:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.518+0000 7f494a800640 1 --2- 172.21.15.32:0/3932043507 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562e912881b0 0x562e9162a2e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4944009f50 tx=0x7f49440118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6673a8e24cdc28a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.518+0000 7f4937400640 1 -- 172.21.15.32:0/3932043507 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4944004b20 con 0x562e912881b0 2024-04-30T08:37:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.518+0000 7f4937400640 1 -- 172.21.15.32:0/3932043507 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4944004cc0 con 0x562e912881b0 2024-04-30T08:37:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.518+0000 7f4937400640 1 -- 172.21.15.32:0/3932043507 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49440053e0 con 0x562e912881b0 2024-04-30T08:37:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.519+0000 7f49581a4740 1 -- 172.21.15.32:0/3932043507 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562e912881b0 msgr2=0x562e9162a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.519+0000 7f49581a4740 1 --2- 172.21.15.32:0/3932043507 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562e912881b0 0x562e9162a2e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4944009f50 tx=0x7f49440118d0 comp rx=0 tx=0).stop 2024-04-30T08:37:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.519+0000 7f4942a00640 1 -- 172.21.15.32:0/3428185497 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f493c003eb0 con 0x7f494c1039e0 2024-04-30T08:37:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.519+0000 7f49581a4740 1 -- 172.21.15.32:0/3932043507 shutdown_connections 2024-04-30T08:37:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.519+0000 7f49581a4740 1 --2- 172.21.15.32:0/3932043507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562e912674d0 0x562e91631760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.519+0000 7f49581a4740 1 --2- 172.21.15.32:0/3932043507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e9125f2f0 0x562e9162a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.519+0000 7f49581a4740 1 --2- 172.21.15.32:0/3932043507 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562e912881b0 0x562e9162a2e0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.519+0000 7f49581a4740 1 -- 172.21.15.32:0/3932043507 >> 172.21.15.32:0/3932043507 conn(0x562e9127c300 msgr2=0x562e91625c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.519+0000 7f49581a4740 1 -- 172.21.15.32:0/3932043507 shutdown_connections 2024-04-30T08:37:19.524 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.519+0000 7f49581a4740 1 -- 172.21.15.32:0/3932043507 wait complete. 2024-04-30T08:37:19.524 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.520+0000 7f49581a4740 1 -- start start 2024-04-30T08:37:19.524 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.520+0000 7f49581a4740 1 Processor -- start 2024-04-30T08:37:19.527 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.520+0000 7f49581a4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562e9125f2f0 0x562e91633dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:19.528 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.521+0000 7f49581a4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e912674d0 0x562e91634310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:19.528 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.521+0000 7f49581a4740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562e912881b0 0x562e9163b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:19.528 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.521+0000 7f49581a4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x562e916321b0 con 0x562e9125f2f0 2024-04-30T08:37:19.528 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.521+0000 7f49581a4740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x562e91547660 con 0x562e912881b0 2024-04-30T08:37:19.528 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.521+0000 7f49581a4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x562e91632330 con 0x562e912674d0 2024-04-30T08:37:19.528 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.521+0000 7f49581a4740 10 client.0 _ll_register_callbacks cb 0x562e91511400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:19.528 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.521+0000 7f494a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562e9125f2f0 0x562e91633dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:19.528 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.521+0000 7f4942a00640 1 --2- 172.21.15.32:0/3428185497 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f49300af9f0 0x7f49300b1f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:19.528 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.521+0000 7f494a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562e9125f2f0 0x562e91633dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59932/0 (socket says 172.21.15.32:59932) 2024-04-30T08:37:19.528 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.521+0000 7f494a800640 1 -- 172.21.15.32:0/2033196312 learned_addr learned my addr 172.21.15.32:0/2033196312 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:19.529 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.521+0000 7f4949e00640 1 --2- 172.21.15.32:0/2033196312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e912674d0 0x562e91634310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:19.529 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.521+0000 7f4949400640 1 --2- 172.21.15.32:0/2033196312 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562e912881b0 0x562e9163b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:19.529 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.521+0000 7f494a800640 1 --2- 172.21.15.32:0/3428185497 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f49300af9f0 0x7f49300b1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:19.529 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.521+0000 7f4942a00640 1 -- 172.21.15.32:0/3428185497 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f493c0dec40 con 0x7f494c1039e0 2024-04-30T08:37:19.529 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.521+0000 7f4949e00640 1 -- 172.21.15.32:0/2033196312 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562e912881b0 msgr2=0x562e9163b270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:19.529 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.522+0000 7f4949e00640 1 --2- 172.21.15.32:0/2033196312 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562e912881b0 0x562e9163b270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:19.529 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.522+0000 7f4949e00640 1 -- 172.21.15.32:0/2033196312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562e9125f2f0 msgr2=0x562e91633dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:19.529 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.522+0000 7f4949e00640 1 --2- 172.21.15.32:0/2033196312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562e9125f2f0 0x562e91633dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:19.529 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.522+0000 7f4949e00640 1 -- 172.21.15.32:0/2033196312 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e9136c620 con 0x562e912674d0 2024-04-30T08:37:19.529 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.522+0000 7f4949400640 1 --2- 172.21.15.32:0/2033196312 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562e912881b0 0x562e9163b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:37:19.529 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.522+0000 7f4949e00640 1 --2- 172.21.15.32:0/2033196312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e912674d0 0x562e91634310 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f493c0dc340 tx=0x7f493c0a64c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7277c1185e6f183e server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:19.530 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.522+0000 7f492be00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-30T08:37:19.530 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.522+0000 7f492be00640 1 -- 172.21.15.32:0/2033196312 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f493c009de0 con 0x562e912674d0 2024-04-30T08:37:19.530 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.522+0000 7f492be00640 1 -- 172.21.15.32:0/2033196312 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f493c0a2fe0 con 0x562e912674d0 2024-04-30T08:37:19.530 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.522+0000 7f49581a4740 1 -- 172.21.15.32:0/2033196312 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4944004cc0 con 0x562e912674d0 2024-04-30T08:37:19.530 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.522+0000 7f494a800640 1 --2- 172.21.15.32:0/3428185497 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f49300af9f0 0x7f49300b1f00 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4944009d60 tx=0x7f4944009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:19.530 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.522+0000 7f492be00640 1 -- 172.21.15.32:0/2033196312 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f493c0a6d50 con 0x562e912674d0 2024-04-30T08:37:19.530 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.523+0000 7f49581a4740 10 client.26230 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:19.530 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.523+0000 7f49581a4740 1 -- 172.21.15.32:0/2033196312 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562e9163e210 con 0x562e912674d0 2024-04-30T08:37:19.530 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.523+0000 7f492be00640 1 -- 172.21.15.32:0/2033196312 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4944004cc0 con 0x562e912674d0 2024-04-30T08:37:19.530 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.523+0000 7f492be00640 1 -- 172.21.15.32:0/2033196312 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7f493c0c0050 con 0x562e912674d0 2024-04-30T08:37:19.530 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.523+0000 7f49581a4740 10 client.26230 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:19.531 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.523+0000 7f49581a4740 10 client.26230 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-30T08:37:19.531 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.523+0000 7f49581a4740 20 client.26230 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:19.531 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.523+0000 7f49581a4740 1 --2- 172.21.15.32:0/2033196312 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x562e91246d80 0x562e916418a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:19.531 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.523+0000 7f49581a4740 4 client.26230 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:37:19.531 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.523+0000 7f49581a4740 1 -- 172.21.15.32:0/2033196312 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e9159c160 con 0x562e91246d80 2024-04-30T08:37:19.531 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.523+0000 7f4949400640 1 --2- 172.21.15.32:0/2033196312 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x562e91246d80 0x562e916418a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:19.531 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.524+0000 7f4949400640 1 --2- 172.21.15.32:0/2033196312 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x562e91246d80 0x562e916418a0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b626a92e6fcb74b server_cookie=f653a32f00b20764 in_seq=0 out_seq=0 2024-04-30T08:37:19.531 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.524+0000 7f492be00640 10 client.26230 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:19.531 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.525+0000 7f492be00640 1 -- 172.21.15.32:0/2033196312 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x562e9159c160 con 0x562e91246d80 2024-04-30T08:37:19.531 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.525+0000 7f492be00640 10 client.26230 handle_command_reply: tid=0 2024-04-30T08:37:19.531 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.525+0000 7f49581a4740 1 client.26230 shutdown 2024-04-30T08:37:19.531 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.525+0000 7f49581a4740 1 -- 172.21.15.32:0/2033196312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e912674d0 msgr2=0x562e91634310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:19.531 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.525+0000 7f49581a4740 1 --2- 172.21.15.32:0/2033196312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e912674d0 0x562e91634310 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f493c0dc340 tx=0x7f493c0a64c0 comp rx=0 tx=0).stop 2024-04-30T08:37:19.531 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.525+0000 7f49581a4740 1 -- 172.21.15.32:0/2033196312 shutdown_connections 2024-04-30T08:37:19.532 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.525+0000 7f49581a4740 1 --2- 172.21.15.32:0/2033196312 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x562e91246d80 0x562e916418a0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:19.532 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.525+0000 7f49581a4740 1 --2- 172.21.15.32:0/2033196312 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562e912881b0 0x562e9163b270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:19.532 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.525+0000 7f49581a4740 1 --2- 172.21.15.32:0/2033196312 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e912674d0 0x562e91634310 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:19.532 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.525+0000 7f49581a4740 1 --2- 172.21.15.32:0/2033196312 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562e9125f2f0 0x562e91633dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:19.532 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.525+0000 7f492be00640 0 client.26230 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:19.532 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.525+0000 7f49581a4740 1 -- 172.21.15.32:0/2033196312 >> 172.21.15.32:0/2033196312 conn(0x562e9127c300 msgr2=0x562e91624c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:19.532 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.525+0000 7f49581a4740 1 -- 172.21.15.32:0/2033196312 shutdown_connections 2024-04-30T08:37:19.532 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.525+0000 7f49581a4740 1 -- 172.21.15.32:0/2033196312 wait complete. 2024-04-30T08:37:19.532 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.525+0000 7f49581a4740 20 client.26230 trim_cache size 0 max 16384 2024-04-30T08:37:19.544 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.542+0000 7f49581a4740 1 Processor -- start 2024-04-30T08:37:19.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.542+0000 7f49581a4740 1 -- start start 2024-04-30T08:37:19.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.542+0000 7f49581a4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562e9125f2f0 0x562e91633060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:19.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.542+0000 7f49581a4740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562e912674d0 0x562e9163b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:19.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.542+0000 7f49581a4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e912881b0 0x562e916335a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:19.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.542+0000 7f49581a4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f494c1aebf0 con 0x562e9125f2f0 2024-04-30T08:37:19.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.542+0000 7f49581a4740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f494c1b07c0 con 0x562e912674d0 2024-04-30T08:37:19.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.542+0000 7f49581a4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f494c1af2c0 con 0x562e912881b0 2024-04-30T08:37:19.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.543+0000 7f4949e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562e9125f2f0 0x562e91633060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:19.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.543+0000 7f4949400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562e912674d0 0x562e9163b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:19.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.543+0000 7f4949400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562e912674d0 0x562e9163b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:44682/0 (socket says 172.21.15.32:44682) 2024-04-30T08:37:19.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.543+0000 7f4949400640 1 -- 172.21.15.32:0/3570297163 learned_addr learned my addr 172.21.15.32:0/3570297163 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:19.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.543+0000 7f494a800640 1 --2- 172.21.15.32:0/3570297163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e912881b0 0x562e916335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:19.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.543+0000 7f4949400640 1 -- 172.21.15.32:0/3570297163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e912881b0 msgr2=0x562e916335a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:19.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.543+0000 7f4949400640 1 --2- 172.21.15.32:0/3570297163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e912881b0 0x562e916335a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:19.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.543+0000 7f4949400640 1 -- 172.21.15.32:0/3570297163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562e9125f2f0 msgr2=0x562e91633060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:19.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.543+0000 7f4949400640 1 --2- 172.21.15.32:0/3570297163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562e9125f2f0 0x562e91633060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:19.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.543+0000 7f4949400640 1 -- 172.21.15.32:0/3570297163 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e9163e040 con 0x562e912674d0 2024-04-30T08:37:19.546 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.543+0000 7f4949400640 1 --2- 172.21.15.32:0/3570297163 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562e912674d0 0x562e9163b190 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f4938031500 tx=0x7f4938031870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec9d25ad4fb8717f server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:19.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.544+0000 7f4936a00640 1 -- 172.21.15.32:0/3570297163 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4938035040 con 0x562e912674d0 2024-04-30T08:37:19.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.544+0000 7f4936a00640 1 -- 172.21.15.32:0/3570297163 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4938046870 con 0x562e912674d0 2024-04-30T08:37:19.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.544+0000 7f4936a00640 1 -- 172.21.15.32:0/3570297163 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f493803cba0 con 0x562e912674d0 2024-04-30T08:37:19.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.545+0000 7f49581a4740 1 -- 172.21.15.32:0/3570297163 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562e912674d0 msgr2=0x562e9163b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:19.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.545+0000 7f49581a4740 1 --2- 172.21.15.32:0/3570297163 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562e912674d0 0x562e9163b190 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f4938031500 tx=0x7f4938031870 comp rx=0 tx=0).stop 2024-04-30T08:37:19.549 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.545+0000 7f49581a4740 1 -- 172.21.15.32:0/3570297163 shutdown_connections 2024-04-30T08:37:19.550 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.545+0000 7f49581a4740 1 --2- 172.21.15.32:0/3570297163 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e912881b0 0x562e916335a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:19.550 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.545+0000 7f49581a4740 1 --2- 172.21.15.32:0/3570297163 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562e912674d0 0x562e9163b190 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:19.550 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.545+0000 7f49581a4740 1 --2- 172.21.15.32:0/3570297163 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562e9125f2f0 0x562e91633060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:19.550 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.545+0000 7f49581a4740 1 -- 172.21.15.32:0/3570297163 >> 172.21.15.32:0/3570297163 conn(0x562e9127c300 msgr2=0x562e9162e1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:19.550 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.545+0000 7f49581a4740 1 -- 172.21.15.32:0/3570297163 shutdown_connections 2024-04-30T08:37:19.550 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.545+0000 7f49581a4740 1 -- 172.21.15.32:0/3570297163 wait complete. 2024-04-30T08:37:19.550 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.545+0000 7f49581a4740 1 -- start start 2024-04-30T08:37:19.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.545+0000 7f49581a4740 1 Processor -- start 2024-04-30T08:37:19.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.546+0000 7f49581a4740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562e9125f2f0 0x562e9163af70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:19.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.546+0000 7f49581a4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e912674d0 0x562e91637af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:19.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.546+0000 7f49581a4740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562e912881b0 0x562e91638030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:19.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.546+0000 7f49581a4740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f494c1b0060 con 0x562e9125f2f0 2024-04-30T08:37:19.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.546+0000 7f49581a4740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f494c1af990 con 0x562e912881b0 2024-04-30T08:37:19.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.546+0000 7f49581a4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f494c1acbb0 con 0x562e912674d0 2024-04-30T08:37:19.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.546+0000 7f49581a4740 10 client.0 _ll_register_callbacks cb 0x7f493c0a9040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:19.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.546+0000 7f4949e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562e9125f2f0 0x562e9163af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:19.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.546+0000 7f4949e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562e9125f2f0 0x562e9163af70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:59952/0 (socket says 172.21.15.32:59952) 2024-04-30T08:37:19.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.546+0000 7f4949e00640 1 -- 172.21.15.32:0/1389413616 learned_addr learned my addr 172.21.15.32:0/1389413616 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:19.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.546+0000 7f494a800640 1 --2- 172.21.15.32:0/1389413616 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562e912881b0 0x562e91638030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:19.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.546+0000 7f4949400640 1 --2- 172.21.15.32:0/1389413616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e912674d0 0x562e91637af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:19.551 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.547+0000 7f4949e00640 1 -- 172.21.15.32:0/1389413616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e912674d0 msgr2=0x562e91637af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:19.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.547+0000 7f4949e00640 1 --2- 172.21.15.32:0/1389413616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e912674d0 0x562e91637af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:19.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.547+0000 7f4949e00640 1 -- 172.21.15.32:0/1389413616 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562e912881b0 msgr2=0x562e91638030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:19.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.547+0000 7f4949e00640 1 --2- 172.21.15.32:0/1389413616 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562e912881b0 0x562e91638030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:19.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.547+0000 7f4949e00640 1 -- 172.21.15.32:0/1389413616 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e916a7c20 con 0x562e9125f2f0 2024-04-30T08:37:19.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.547+0000 7f4949e00640 1 --2- 172.21.15.32:0/1389413616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562e9125f2f0 0x562e9163af70 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f493c0a6a50 tx=0x7f493c0a29c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e2e6e677bed4e37 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:19.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.547+0000 7f492be00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:37:19.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.547+0000 7f492be00640 1 -- 172.21.15.32:0/1389413616 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f493c004480 con 0x562e9125f2f0 2024-04-30T08:37:19.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.547+0000 7f492be00640 1 -- 172.21.15.32:0/1389413616 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f493c0a3340 con 0x562e9125f2f0 2024-04-30T08:37:19.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.547+0000 7f49581a4740 1 -- 172.21.15.32:0/1389413616 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4938046870 con 0x562e9125f2f0 2024-04-30T08:37:19.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.548+0000 7f492be00640 1 -- 172.21.15.32:0/1389413616 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f493c089050 con 0x562e9125f2f0 2024-04-30T08:37:19.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.548+0000 7f49581a4740 10 client.16110 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:19.552 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.548+0000 7f49581a4740 1 -- 172.21.15.32:0/1389413616 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x562e916a8d10 con 0x562e9125f2f0 2024-04-30T08:37:19.553 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:19 smithi189 ceph-mon[33038]: pgmap v489: 129 pgs: 129 active+clean; 3.2 MiB data, 402 MiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 283 KiB/s wr, 181 op/s 2024-04-30T08:37:19.553 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.548+0000 7f492be00640 1 -- 172.21.15.32:0/1389413616 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4938046870 con 0x562e9125f2f0 2024-04-30T08:37:19.554 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.548+0000 7f492be00640 1 -- 172.21.15.32:0/1389413616 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7f493c0c0040 con 0x562e9125f2f0 2024-04-30T08:37:19.554 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.548+0000 7f49581a4740 10 client.16110 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:19.554 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.548+0000 7f49581a4740 10 client.16110 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-30T08:37:19.554 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.548+0000 7f49581a4740 20 client.16110 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:19.554 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.548+0000 7f49581a4740 1 --2- 172.21.15.32:0/1389413616 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x562e9121c1c0 0x562e916ab470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:19.554 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.548+0000 7f49581a4740 4 client.16110 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-30T08:37:19.554 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.548+0000 7f49581a4740 1 -- 172.21.15.32:0/1389413616 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562e91595fd0 con 0x562e9121c1c0 2024-04-30T08:37:19.554 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.548+0000 7f4949400640 1 --2- 172.21.15.32:0/1389413616 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x562e9121c1c0 0x562e916ab470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:19.554 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.549+0000 7f4949400640 1 --2- 172.21.15.32:0/1389413616 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x562e9121c1c0 0x562e916ab470 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5feab8d0d67058f server_cookie=d3345017d21ed812 in_seq=0 out_seq=0 2024-04-30T08:37:19.554 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.549+0000 7f492be00640 10 client.16110 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.605+0000 7f492be00640 1 -- 172.21.15.32:0/1389413616 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562e91595fd0 con 0x562e9121c1c0 2024-04-30T08:37:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.605+0000 7f492be00640 10 client.16110 handle_command_reply: tid=0 2024-04-30T08:37:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.605+0000 7f49581a4740 1 client.16110 shutdown 2024-04-30T08:37:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.605+0000 7f49581a4740 1 -- 172.21.15.32:0/1389413616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562e9125f2f0 msgr2=0x562e9163af70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.605+0000 7f49581a4740 1 --2- 172.21.15.32:0/1389413616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562e9125f2f0 0x562e9163af70 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f493c0a6a50 tx=0x7f493c0a29c0 comp rx=0 tx=0).stop 2024-04-30T08:37:19.607 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.605+0000 7f49581a4740 1 -- 172.21.15.32:0/1389413616 shutdown_connections 2024-04-30T08:37:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.605+0000 7f49581a4740 1 --2- 172.21.15.32:0/1389413616 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x562e9121c1c0 0x562e916ab470 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.605+0000 7f49581a4740 1 --2- 172.21.15.32:0/1389413616 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x562e912881b0 0x562e91638030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.605+0000 7f49581a4740 1 --2- 172.21.15.32:0/1389413616 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562e912674d0 0x562e91637af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.605+0000 7f492be00640 0 client.16110 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.605+0000 7f49581a4740 1 --2- 172.21.15.32:0/1389413616 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x562e9125f2f0 0x562e9163af70 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.605+0000 7f49581a4740 1 -- 172.21.15.32:0/1389413616 >> 172.21.15.32:0/1389413616 conn(0x562e9127c300 msgr2=0x562e91633060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.605+0000 7f49581a4740 1 -- 172.21.15.32:0/1389413616 shutdown_connections 2024-04-30T08:37:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.605+0000 7f49581a4740 1 -- 172.21.15.32:0/1389413616 wait complete. 2024-04-30T08:37:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.606+0000 7f49581a4740 20 client.16110 trim_cache size 0 max 16384 2024-04-30T08:37:19.608 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-30T08:37:19.609 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-30T08:37:19.609 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-30T08:37:19.609 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-30T08:37:19.609 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.606+0000 7f494b200640 1 -- 172.21.15.32:0/3428185497 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f49300af9f0 msgr2=0x7f49300b1f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:19.610 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.606+0000 7f494b200640 1 --2- 172.21.15.32:0/3428185497 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f49300af9f0 0x7f49300b1f00 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f4944009d60 tx=0x7f4944009500 comp rx=0 tx=0).stop 2024-04-30T08:37:19.610 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.606+0000 7f494b200640 1 -- 172.21.15.32:0/3428185497 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f494c1039e0 msgr2=0x7f494c0768d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:19.610 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.606+0000 7f494b200640 1 --2- 172.21.15.32:0/3428185497 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f494c1039e0 0x7f494c0768d0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f493c005fd0 tx=0x7f493c00dde0 comp rx=0 tx=0).stop 2024-04-30T08:37:19.610 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.606+0000 7f494b200640 1 -- 172.21.15.32:0/3428185497 shutdown_connections 2024-04-30T08:37:19.610 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.606+0000 7f494b200640 1 --2- 172.21.15.32:0/3428185497 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f49300af9f0 0x7f49300b1f00 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:19.610 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.607+0000 7f494b200640 1 --2- 172.21.15.32:0/3428185497 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f494c109920 0x7f494c0754b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:19.610 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.607+0000 7f494b200640 1 --2- 172.21.15.32:0/3428185497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f494c104c20 0x7f494c074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:19.611 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.607+0000 7f494b200640 1 --2- 172.21.15.32:0/3428185497 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f494c1039e0 0x7f494c0768d0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:19.611 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.607+0000 7f494b200640 1 -- 172.21.15.32:0/3428185497 >> 172.21.15.32:0/3428185497 conn(0x7f494c0febd0 msgr2=0x7f494c105900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:19.611 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.607+0000 7f494b200640 1 -- 172.21.15.32:0/3428185497 shutdown_connections 2024-04-30T08:37:19.611 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:19.607+0000 7f494b200640 1 -- 172.21.15.32:0/3428185497 wait complete. 2024-04-30T08:37:21.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:21 smithi032 ceph-mon[28331]: pgmap v490: 129 pgs: 129 active+clean; 56 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 4.7 MiB/s wr, 186 op/s 2024-04-30T08:37:21.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:21 smithi071 ceph-mon[33385]: pgmap v490: 129 pgs: 129 active+clean; 56 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 4.7 MiB/s wr, 186 op/s 2024-04-30T08:37:21.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:21 smithi189 ceph-mon[33038]: pgmap v490: 129 pgs: 129 active+clean; 56 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 4.7 MiB/s wr, 186 op/s 2024-04-30T08:37:23.445 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:23 smithi032 ceph-mon[28331]: pgmap v491: 129 pgs: 129 active+clean; 217 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 18 MiB/s wr, 120 op/s 2024-04-30T08:37:23.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:23 smithi071 ceph-mon[33385]: pgmap v491: 129 pgs: 129 active+clean; 217 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 18 MiB/s wr, 120 op/s 2024-04-30T08:37:23.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:23 smithi189 ceph-mon[33038]: pgmap v491: 129 pgs: 129 active+clean; 217 MiB data, 429 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 18 MiB/s wr, 120 op/s 2024-04-30T08:37:25.446 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:25 smithi032 ceph-mon[28331]: pgmap v492: 129 pgs: 129 active+clean; 409 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 34 MiB/s wr, 82 op/s 2024-04-30T08:37:25.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:25 smithi071 ceph-mon[33385]: pgmap v492: 129 pgs: 129 active+clean; 409 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 34 MiB/s wr, 82 op/s 2024-04-30T08:37:25.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:25 smithi189 ceph-mon[33038]: pgmap v492: 129 pgs: 129 active+clean; 409 MiB data, 455 MiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 34 MiB/s wr, 82 op/s 2024-04-30T08:37:27.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:27 smithi032 ceph-mon[28331]: pgmap v493: 129 pgs: 129 active+clean; 829 MiB data, 501 MiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 69 MiB/s wr, 88 op/s 2024-04-30T08:37:27.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:27 smithi071 ceph-mon[33385]: pgmap v493: 129 pgs: 129 active+clean; 829 MiB data, 501 MiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 69 MiB/s wr, 88 op/s 2024-04-30T08:37:27.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:27 smithi189 ceph-mon[33038]: pgmap v493: 129 pgs: 129 active+clean; 829 MiB data, 501 MiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 69 MiB/s wr, 88 op/s 2024-04-30T08:37:29.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:29 smithi032 ceph-mon[28331]: pgmap v494: 129 pgs: 129 active+clean; 1.1 GiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 MiB/s wr, 92 op/s 2024-04-30T08:37:29.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:29 smithi071 ceph-mon[33385]: pgmap v494: 129 pgs: 129 active+clean; 1.1 GiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 MiB/s wr, 92 op/s 2024-04-30T08:37:29.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:29 smithi189 ceph-mon[33038]: pgmap v494: 129 pgs: 129 active+clean; 1.1 GiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 95 KiB/s rd, 92 MiB/s wr, 92 op/s 2024-04-30T08:37:31.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:31 smithi032 ceph-mon[28331]: pgmap v495: 129 pgs: 129 active+clean; 1.7 GiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 145 MiB/s wr, 116 op/s 2024-04-30T08:37:31.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:31 smithi071 ceph-mon[33385]: pgmap v495: 129 pgs: 129 active+clean; 1.7 GiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 145 MiB/s wr, 116 op/s 2024-04-30T08:37:31.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:31 smithi189 ceph-mon[33038]: pgmap v495: 129 pgs: 129 active+clean; 1.7 GiB data, 627 MiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 145 MiB/s wr, 116 op/s 2024-04-30T08:37:32.935 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.933+0000 7fac17400640 1 -- 172.21.15.32:0/4185743664 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac10109af0 msgr2=0x7fac1010bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:32.935 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.933+0000 7fac17400640 1 --2- 172.21.15.32:0/4185743664 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac10109af0 0x7fac1010bf10 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fac08009ca0 tx=0x7fac0802a8c0 comp rx=0 tx=0).stop 2024-04-30T08:37:32.935 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.933+0000 7fac17400640 1 -- 172.21.15.32:0/4185743664 shutdown_connections 2024-04-30T08:37:32.935 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.933+0000 7fac17400640 1 --2- 172.21.15.32:0/4185743664 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac10109af0 0x7fac1010bf10 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:32.935 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.933+0000 7fac17400640 1 --2- 172.21.15.32:0/4185743664 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac10104df0 0x7fac10105200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:32.935 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.933+0000 7fac17400640 1 --2- 172.21.15.32:0/4185743664 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac10103bb0 0x7fac101048b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:32.935 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.933+0000 7fac17400640 1 -- 172.21.15.32:0/4185743664 >> 172.21.15.32:0/4185743664 conn(0x7fac100fed80 msgr2=0x7fac10101250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:32.935 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.933+0000 7fac17400640 1 -- 172.21.15.32:0/4185743664 shutdown_connections 2024-04-30T08:37:32.935 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.933+0000 7fac17400640 1 -- 172.21.15.32:0/4185743664 wait complete. 2024-04-30T08:37:32.937 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.934+0000 7fac17400640 1 Processor -- start 2024-04-30T08:37:32.937 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.934+0000 7fac17400640 1 -- start start 2024-04-30T08:37:32.937 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.934+0000 7fac17400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac10103bb0 0x7fac1019a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:32.937 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.934+0000 7fac17400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac10104df0 0x7fac1019a760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:32.937 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.934+0000 7fac17400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac10109af0 0x7fac101a16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:32.937 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.934+0000 7fac17400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fac1010f710 con 0x7fac10109af0 2024-04-30T08:37:32.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.934+0000 7fac17400640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fac1010f590 con 0x7fac10104df0 2024-04-30T08:37:32.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.934+0000 7fac17400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fac1010f890 con 0x7fac10103bb0 2024-04-30T08:37:32.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.935+0000 7fac15400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac10109af0 0x7fac101a16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:32.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.935+0000 7fac15400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac10109af0 0x7fac101a16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48074/0 (socket says 172.21.15.32:48074) 2024-04-30T08:37:32.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.935+0000 7fac15400640 1 -- 172.21.15.32:0/3807174435 learned_addr learned my addr 172.21.15.32:0/3807174435 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:32.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.935+0000 7fac0fe00640 1 --2- 172.21.15.32:0/3807174435 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac10104df0 0x7fac1019a760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:32.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.935+0000 7fac14a00640 1 --2- 172.21.15.32:0/3807174435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac10103bb0 0x7fac1019a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:32.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.935+0000 7fac15400640 1 -- 172.21.15.32:0/3807174435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac10103bb0 msgr2=0x7fac1019a220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:32.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.935+0000 7fac15400640 1 --2- 172.21.15.32:0/3807174435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac10103bb0 0x7fac1019a220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:32.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.935+0000 7fac15400640 1 -- 172.21.15.32:0/3807174435 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac10104df0 msgr2=0x7fac1019a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:32.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.935+0000 7fac15400640 1 --2- 172.21.15.32:0/3807174435 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac10104df0 0x7fac1019a760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:32.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.935+0000 7fac15400640 1 -- 172.21.15.32:0/3807174435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac101a1d60 con 0x7fac10109af0 2024-04-30T08:37:32.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.935+0000 7fac0fe00640 1 --2- 172.21.15.32:0/3807174435 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac10104df0 0x7fac1019a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:37:32.939 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.936+0000 7fac15400640 1 --2- 172.21.15.32:0/3807174435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac10109af0 0x7fac101a16f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fac08009430 tx=0x7fac080317c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:32.939 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.936+0000 7fac0d600640 1 -- 172.21.15.32:0/3807174435 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac08003bf0 con 0x7fac10109af0 2024-04-30T08:37:32.940 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.936+0000 7fac0d600640 1 -- 172.21.15.32:0/3807174435 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac08003d90 con 0x7fac10109af0 2024-04-30T08:37:32.940 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.936+0000 7fac17400640 1 -- 172.21.15.32:0/3807174435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac101a1ff0 con 0x7fac10109af0 2024-04-30T08:37:32.940 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.936+0000 7fac0d600640 1 -- 172.21.15.32:0/3807174435 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac0802fc80 con 0x7fac10109af0 2024-04-30T08:37:32.940 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.936+0000 7fac17400640 1 -- 172.21.15.32:0/3807174435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac101a24b0 con 0x7fac10109af0 2024-04-30T08:37:32.942 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.940+0000 7fac1e484740 1 Processor -- start 2024-04-30T08:37:32.943 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.942+0000 7fac0d600640 1 -- 172.21.15.32:0/3807174435 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7fac08004710 con 0x7fac10109af0 2024-04-30T08:37:32.943 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.942+0000 7fac1e484740 1 -- start start 2024-04-30T08:37:32.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.942+0000 7fac1e484740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5589604661b0 0x5589608080e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:32.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.942+0000 7fac1e484740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55896043d2f0 0x558960808660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:32.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.942+0000 7fac1e484740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589604454d0 0x55896080f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:32.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.942+0000 7fac1e484740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55896076c550 con 0x5589604454d0 2024-04-30T08:37:32.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.942+0000 7fac1e484740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x558960711bc0 con 0x5589604661b0 2024-04-30T08:37:32.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.942+0000 7fac1e484740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558960768510 con 0x55896043d2f0 2024-04-30T08:37:32.947 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.942+0000 7fac0fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5589604661b0 0x5589608080e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:32.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.942+0000 7fac0fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5589604661b0 0x5589608080e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:51020/0 (socket says 172.21.15.32:51020) 2024-04-30T08:37:32.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.942+0000 7fac0fe00640 1 -- 172.21.15.32:0/680199901 learned_addr learned my addr 172.21.15.32:0/680199901 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:32.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.943+0000 7fac14a00640 1 --2- 172.21.15.32:0/680199901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55896043d2f0 0x558960808660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:32.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.943+0000 7fac0fe00640 1 --2- 172.21.15.32:0/680199901 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589604454d0 0x55896080f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:32.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.943+0000 7fac14a00640 1 -- 172.21.15.32:0/680199901 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5589604661b0 msgr2=0x5589608080e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:32.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.943+0000 7fac14a00640 1 --2- 172.21.15.32:0/680199901 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5589604661b0 0x5589608080e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:32.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.943+0000 7fac14a00640 1 -- 172.21.15.32:0/680199901 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589604454d0 msgr2=0x55896080f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:32.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.943+0000 7fac14a00640 1 --2- 172.21.15.32:0/680199901 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589604454d0 0x55896080f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:32.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.943+0000 7fac14a00640 1 -- 172.21.15.32:0/680199901 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55896047e210 con 0x55896043d2f0 2024-04-30T08:37:32.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.943+0000 7fac14a00640 1 --2- 172.21.15.32:0/680199901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55896043d2f0 0x558960808660 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fac101047b0 tx=0x7fac0400bbe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ab851a6c3101a458 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:32.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.943+0000 7fac0d600640 1 --2- 172.21.15.32:0/3807174435 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fabf80afb60 0x7fabf80b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:32.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.943+0000 7fac01c00640 1 -- 172.21.15.32:0/680199901 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac04003da0 con 0x55896043d2f0 2024-04-30T08:37:32.948 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.943+0000 7fac01c00640 1 -- 172.21.15.32:0/680199901 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac04015070 con 0x55896043d2f0 2024-04-30T08:37:32.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.943+0000 7fac01c00640 1 -- 172.21.15.32:0/680199901 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac040116a0 con 0x55896043d2f0 2024-04-30T08:37:32.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.943+0000 7fac0d600640 1 -- 172.21.15.32:0/3807174435 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fac081005d0 con 0x7fac10109af0 2024-04-30T08:37:32.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.944+0000 7fac0fe00640 1 --2- 172.21.15.32:0/3807174435 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fabf80afb60 0x7fabf80b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:32.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.945+0000 7fac1e484740 1 -- 172.21.15.32:0/680199901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55896043d2f0 msgr2=0x558960808660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:32.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.945+0000 7fac1e484740 1 --2- 172.21.15.32:0/680199901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55896043d2f0 0x558960808660 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fac101047b0 tx=0x7fac0400bbe0 comp rx=0 tx=0).stop 2024-04-30T08:37:32.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.945+0000 7fac0fe00640 1 --2- 172.21.15.32:0/3807174435 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fabf80afb60 0x7fabf80b2070 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x5589608092b0 tx=0x7fabfc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:32.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.945+0000 7fac1e484740 1 -- 172.21.15.32:0/680199901 shutdown_connections 2024-04-30T08:37:32.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.945+0000 7fac1e484740 1 --2- 172.21.15.32:0/680199901 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589604454d0 0x55896080f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:32.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.945+0000 7fac1e484740 1 --2- 172.21.15.32:0/680199901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55896043d2f0 0x558960808660 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:32.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.945+0000 7fac1e484740 1 --2- 172.21.15.32:0/680199901 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5589604661b0 0x5589608080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:32.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.945+0000 7fac1e484740 1 -- 172.21.15.32:0/680199901 >> 172.21.15.32:0/680199901 conn(0x55896045a300 msgr2=0x558960803a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:32.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.945+0000 7fac1e484740 1 -- 172.21.15.32:0/680199901 shutdown_connections 2024-04-30T08:37:32.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.945+0000 7fac1e484740 1 -- 172.21.15.32:0/680199901 wait complete. 2024-04-30T08:37:32.949 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.945+0000 7fac1e484740 1 -- start start 2024-04-30T08:37:32.950 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.946+0000 7fac1e484740 1 Processor -- start 2024-04-30T08:37:32.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.946+0000 7fac1e484740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55896043d2f0 0x558960815fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:32.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.946+0000 7fac1e484740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589604454d0 0x558960816520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:32.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.946+0000 7fac1e484740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5589604661b0 0x55896081d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:32.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.946+0000 7fac1e484740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55896080ffb0 con 0x55896043d2f0 2024-04-30T08:37:32.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.946+0000 7fac1e484740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x558960725660 con 0x5589604661b0 2024-04-30T08:37:32.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.946+0000 7fac1e484740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558960810130 con 0x5589604454d0 2024-04-30T08:37:32.954 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.946+0000 7fac1e484740 10 client.0 _ll_register_callbacks cb 0x5589606ef400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:32.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.946+0000 7fac14a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55896043d2f0 0x558960815fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:32.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.947+0000 7fac14a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55896043d2f0 0x558960815fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48096/0 (socket says 172.21.15.32:48096) 2024-04-30T08:37:32.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.947+0000 7fac14a00640 1 -- 172.21.15.32:0/3808906272 learned_addr learned my addr 172.21.15.32:0/3808906272 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:32.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.947+0000 7fac0fe00640 1 --2- 172.21.15.32:0/3808906272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589604454d0 0x558960816520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:32.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.947+0000 7fac15400640 1 --2- 172.21.15.32:0/3808906272 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5589604661b0 0x55896081d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:32.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.947+0000 7fac0fe00640 1 -- 172.21.15.32:0/3808906272 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5589604661b0 msgr2=0x55896081d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:32.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.947+0000 7fac0fe00640 1 --2- 172.21.15.32:0/3808906272 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5589604661b0 0x55896081d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:32.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.947+0000 7fac0fe00640 1 -- 172.21.15.32:0/3808906272 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55896043d2f0 msgr2=0x558960815fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:32.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.947+0000 7fac0fe00640 1 --2- 172.21.15.32:0/3808906272 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55896043d2f0 0x558960815fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:32.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.947+0000 7fac0fe00640 1 -- 172.21.15.32:0/3808906272 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55896054a620 con 0x5589604454d0 2024-04-30T08:37:32.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.947+0000 7fac15400640 1 --2- 172.21.15.32:0/3808906272 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5589604661b0 0x55896081d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:37:32.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.947+0000 7fac0fe00640 1 --2- 172.21.15.32:0/3808906272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589604454d0 0x558960816520 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fabfc004180 tx=0x7fabfc004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18acd28491af02c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:32.955 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.947+0000 7fabf6000640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-30T08:37:32.956 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.947+0000 7fabf6000640 1 -- 172.21.15.32:0/3808906272 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabfc003d80 con 0x5589604454d0 2024-04-30T08:37:32.956 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.947+0000 7fac1e484740 1 -- 172.21.15.32:0/3808906272 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fac04015070 con 0x5589604454d0 2024-04-30T08:37:32.956 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.948+0000 7fabf6000640 1 -- 172.21.15.32:0/3808906272 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fabfc005020 con 0x5589604454d0 2024-04-30T08:37:32.956 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.948+0000 7fabf6000640 1 -- 172.21.15.32:0/3808906272 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fabfc005220 con 0x5589604454d0 2024-04-30T08:37:32.956 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.948+0000 7fac1e484740 10 client.26248 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:32.956 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.948+0000 7fac1e484740 1 -- 172.21.15.32:0/3808906272 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558960820460 con 0x5589604454d0 2024-04-30T08:37:32.956 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.948+0000 7fabf6000640 1 -- 172.21.15.32:0/3808906272 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fac04015070 con 0x5589604454d0 2024-04-30T08:37:32.956 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.948+0000 7fabf6000640 1 -- 172.21.15.32:0/3808906272 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7fabfc02a070 con 0x5589604454d0 2024-04-30T08:37:32.956 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.948+0000 7fac1e484740 10 client.26248 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:32.956 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.948+0000 7fac1e484740 10 client.26248 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-30T08:37:32.956 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.948+0000 7fac1e484740 20 client.26248 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.948+0000 7fac1e484740 1 --2- 172.21.15.32:0/3808906272 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x558960424d80 0x558960823cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.948+0000 7fac1e484740 4 client.26248 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:37:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.948+0000 7fac1e484740 1 -- 172.21.15.32:0/3808906272 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55896077a160 con 0x558960424d80 2024-04-30T08:37:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.949+0000 7fac14a00640 1 --2- 172.21.15.32:0/3808906272 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x558960424d80 0x558960823cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.949+0000 7fac14a00640 1 --2- 172.21.15.32:0/3808906272 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x558960424d80 0x558960823cf0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dcae54a9ffbae9fe server_cookie=717cb2a79d791792 in_seq=0 out_seq=0 2024-04-30T08:37:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.949+0000 7fabf6000640 10 client.26248 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.950+0000 7fabf6000640 1 -- 172.21.15.32:0/3808906272 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55896077a160 con 0x558960424d80 2024-04-30T08:37:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.950+0000 7fabf6000640 10 client.26248 handle_command_reply: tid=0 2024-04-30T08:37:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.950+0000 7fac1e484740 1 client.26248 shutdown 2024-04-30T08:37:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.950+0000 7fac1e484740 1 -- 172.21.15.32:0/3808906272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589604454d0 msgr2=0x558960816520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.950+0000 7fac1e484740 1 --2- 172.21.15.32:0/3808906272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589604454d0 0x558960816520 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fabfc004180 tx=0x7fabfc004730 comp rx=0 tx=0).stop 2024-04-30T08:37:32.957 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.950+0000 7fac1e484740 1 -- 172.21.15.32:0/3808906272 shutdown_connections 2024-04-30T08:37:32.958 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.950+0000 7fac1e484740 1 --2- 172.21.15.32:0/3808906272 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x558960424d80 0x558960823cf0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:32.958 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.950+0000 7fac1e484740 1 --2- 172.21.15.32:0/3808906272 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5589604661b0 0x55896081d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:32.958 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.950+0000 7fac1e484740 1 --2- 172.21.15.32:0/3808906272 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589604454d0 0x558960816520 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:32.958 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.950+0000 7fabf6000640 0 client.26248 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:32.958 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.950+0000 7fac1e484740 1 --2- 172.21.15.32:0/3808906272 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55896043d2f0 0x558960815fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:32.958 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.950+0000 7fac1e484740 1 -- 172.21.15.32:0/3808906272 >> 172.21.15.32:0/3808906272 conn(0x55896045a300 msgr2=0x558960806980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:32.958 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.951+0000 7fac1e484740 1 -- 172.21.15.32:0/3808906272 shutdown_connections 2024-04-30T08:37:32.958 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.951+0000 7fac1e484740 1 -- 172.21.15.32:0/3808906272 wait complete. 2024-04-30T08:37:32.958 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.951+0000 7fac1e484740 20 client.26248 trim_cache size 0 max 16384 2024-04-30T08:37:32.975 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.974+0000 7fac1e484740 1 Processor -- start 2024-04-30T08:37:32.978 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.976+0000 7fac1e484740 1 -- start start 2024-04-30T08:37:32.978 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.977+0000 7fac1e484740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55896043d2f0 0x55896080bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:32.979 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.977+0000 7fac1e484740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589604454d0 0x55896080c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:32.979 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.977+0000 7fac1e484740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5589604661b0 0x55896081d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:32.979 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.977+0000 7fac1e484740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fac101aa970 con 0x55896043d2f0 2024-04-30T08:37:32.979 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.977+0000 7fac1e484740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fac101ac540 con 0x5589604661b0 2024-04-30T08:37:32.979 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.977+0000 7fac1e484740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fac101ab040 con 0x5589604454d0 2024-04-30T08:37:32.979 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.977+0000 7fac14a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55896043d2f0 0x55896080bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:32.979 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.978+0000 7fac0fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589604454d0 0x55896080c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:32.979 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.978+0000 7fac0fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589604454d0 0x55896080c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:41748/0 (socket says 172.21.15.32:41748) 2024-04-30T08:37:32.980 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.978+0000 7fac0fe00640 1 -- 172.21.15.32:0/3861130507 learned_addr learned my addr 172.21.15.32:0/3861130507 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:32.980 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.978+0000 7fac0fe00640 1 -- 172.21.15.32:0/3861130507 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5589604661b0 msgr2=0x55896081d330 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-30T08:37:32.980 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.978+0000 7fac0fe00640 1 --2- 172.21.15.32:0/3861130507 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5589604661b0 0x55896081d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:32.980 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.978+0000 7fac0fe00640 1 -- 172.21.15.32:0/3861130507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55896043d2f0 msgr2=0x55896080bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:32.980 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.978+0000 7fac0fe00640 1 --2- 172.21.15.32:0/3861130507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55896043d2f0 0x55896080bfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:32.981 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.978+0000 7fac0fe00640 1 -- 172.21.15.32:0/3861130507 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5589607ff260 con 0x5589604454d0 2024-04-30T08:37:32.981 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.979+0000 7fac0fe00640 1 --2- 172.21.15.32:0/3861130507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589604454d0 0x55896080c520 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fabfc000c00 tx=0x7fabfc009cc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=de98d17e0b63bee3 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:32.981 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.979+0000 7fac01200640 1 -- 172.21.15.32:0/3861130507 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabfc0053c0 con 0x5589604454d0 2024-04-30T08:37:32.981 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.979+0000 7fac01200640 1 -- 172.21.15.32:0/3861130507 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabfc003d80 con 0x5589604454d0 2024-04-30T08:37:32.981 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.979+0000 7fac01200640 1 -- 172.21.15.32:0/3861130507 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabfc0227b0 con 0x5589604454d0 2024-04-30T08:37:32.983 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.981+0000 7fac1e484740 1 -- 172.21.15.32:0/3861130507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589604454d0 msgr2=0x55896080c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:32.983 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.982+0000 7fac1e484740 1 --2- 172.21.15.32:0/3861130507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589604454d0 0x55896080c520 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fabfc000c00 tx=0x7fabfc009cc0 comp rx=0 tx=0).stop 2024-04-30T08:37:32.984 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.982+0000 7fac1e484740 1 -- 172.21.15.32:0/3861130507 shutdown_connections 2024-04-30T08:37:32.984 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.982+0000 7fac1e484740 1 --2- 172.21.15.32:0/3861130507 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5589604661b0 0x55896081d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:32.984 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.982+0000 7fac1e484740 1 --2- 172.21.15.32:0/3861130507 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5589604454d0 0x55896080c520 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:32.984 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.982+0000 7fac1e484740 1 --2- 172.21.15.32:0/3861130507 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55896043d2f0 0x55896080bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:32.984 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.982+0000 7fac1e484740 1 -- 172.21.15.32:0/3861130507 >> 172.21.15.32:0/3861130507 conn(0x55896045a300 msgr2=0x558960808060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:32.985 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.983+0000 7fac1e484740 1 -- 172.21.15.32:0/3861130507 shutdown_connections 2024-04-30T08:37:32.985 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.983+0000 7fac1e484740 1 -- 172.21.15.32:0/3861130507 wait complete. 2024-04-30T08:37:32.986 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.985+0000 7fac1e484740 1 -- start start 2024-04-30T08:37:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.987+0000 7fac1e484740 1 Processor -- start 2024-04-30T08:37:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.987+0000 7fac1e484740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55896043d2f0 0x558960814fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.988+0000 7fac1e484740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5589604454d0 0x558960815520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:32.989 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.988+0000 7fac14a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55896043d2f0 0x558960814fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:32.993 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.988+0000 7fac14a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55896043d2f0 0x558960814fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:41752/0 (socket says 172.21.15.32:41752) 2024-04-30T08:37:32.993 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.988+0000 7fac14a00640 1 -- 172.21.15.32:0/1051343018 learned_addr learned my addr 172.21.15.32:0/1051343018 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:32.993 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.988+0000 7fac1e484740 1 --2- 172.21.15.32:0/1051343018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589604661b0 0x558960811b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:32.993 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.988+0000 7fac1e484740 1 -- 172.21.15.32:0/1051343018 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fac101abde0 con 0x5589604661b0 2024-04-30T08:37:32.993 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.988+0000 7fac1e484740 1 -- 172.21.15.32:0/1051343018 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fac101ab710 con 0x5589604454d0 2024-04-30T08:37:32.993 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.988+0000 7fac1e484740 1 -- 172.21.15.32:0/1051343018 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fac101a8930 con 0x55896043d2f0 2024-04-30T08:37:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.988+0000 7fac1e484740 10 client.0 _ll_register_callbacks cb 0x7fabfc00fe00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.989+0000 7fac14a00640 1 -- 172.21.15.32:0/1051343018 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5589604454d0 msgr2=0x558960815520 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-30T08:37:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.989+0000 7fac14a00640 1 --2- 172.21.15.32:0/1051343018 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5589604454d0 0x558960815520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:32.994 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.989+0000 7fac14a00640 1 -- 172.21.15.32:0/1051343018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589604661b0 msgr2=0x558960811b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-30T08:37:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.991+0000 7fac15400640 1 --2- 172.21.15.32:0/1051343018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589604661b0 0x558960811b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.991+0000 7fac14a00640 1 --2- 172.21.15.32:0/1051343018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589604661b0 0x558960811b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.991+0000 7fac14a00640 1 -- 172.21.15.32:0/1051343018 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558960812280 con 0x55896043d2f0 2024-04-30T08:37:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.991+0000 7fac15400640 1 --2- 172.21.15.32:0/1051343018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589604661b0 0x558960811b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-30T08:37:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.992+0000 7fac14a00640 1 --2- 172.21.15.32:0/1051343018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55896043d2f0 0x558960814fe0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fac04006010 tx=0x7fac04004040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2475e15e5c66e7de server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.992+0000 7fabf6000640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-30T08:37:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.992+0000 7fabf6000640 1 -- 172.21.15.32:0/1051343018 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac04030020 con 0x55896043d2f0 2024-04-30T08:37:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.992+0000 7fabf6000640 1 -- 172.21.15.32:0/1051343018 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fac04005240 con 0x55896043d2f0 2024-04-30T08:37:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.993+0000 7fac1e484740 1 -- 172.21.15.32:0/1051343018 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fabfc003d80 con 0x55896043d2f0 2024-04-30T08:37:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.993+0000 7fabf6000640 1 -- 172.21.15.32:0/1051343018 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fac04005850 con 0x55896043d2f0 2024-04-30T08:37:32.995 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.993+0000 7fac1e484740 10 client.26254 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:32.996 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.993+0000 7fac1e484740 1 -- 172.21.15.32:0/1051343018 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55896081cef0 con 0x55896043d2f0 2024-04-30T08:37:32.996 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.994+0000 7fabf6000640 1 -- 172.21.15.32:0/1051343018 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fabfc003d80 con 0x55896043d2f0 2024-04-30T08:37:32.996 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.994+0000 7fabf6000640 1 -- 172.21.15.32:0/1051343018 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7fac0401a6e0 con 0x55896043d2f0 2024-04-30T08:37:32.996 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.994+0000 7fac1e484740 10 client.26254 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:32.996 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.994+0000 7fac1e484740 10 client.26254 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-30T08:37:32.996 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.994+0000 7fac1e484740 20 client.26254 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:32.996 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.994+0000 7fac1e484740 1 --2- 172.21.15.32:0/1051343018 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x5589603fa1c0 0x55896081d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:32.997 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.994+0000 7fac1e484740 4 client.26254 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-30T08:37:32.997 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.994+0000 7fac1e484740 1 -- 172.21.15.32:0/1051343018 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558960773fd0 con 0x5589603fa1c0 2024-04-30T08:37:32.997 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.996+0000 7fac0fe00640 1 --2- 172.21.15.32:0/1051343018 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x5589603fa1c0 0x55896081d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:33.000 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.998+0000 7fac0fe00640 1 --2- 172.21.15.32:0/1051343018 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x5589603fa1c0 0x55896081d620 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9807aa69dea58de server_cookie=8804099ce2187572 in_seq=0 out_seq=0 2024-04-30T08:37:33.000 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:32.998+0000 7fabf6000640 10 client.26254 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:33.025 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-30T08:37:33.025 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-30T08:37:33.132 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:33.129+0000 7fabf6000640 1 -- 172.21.15.32:0/1051343018 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558960773fd0 con 0x5589603fa1c0 2024-04-30T08:37:33.132 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:33.129+0000 7fabf6000640 10 client.26254 handle_command_reply: tid=0 2024-04-30T08:37:33.132 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:33.129+0000 7fac1e484740 1 client.26254 shutdown 2024-04-30T08:37:33.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:33.130+0000 7fac1e484740 1 -- 172.21.15.32:0/1051343018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55896043d2f0 msgr2=0x558960814fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:33.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:33.130+0000 7fac1e484740 1 --2- 172.21.15.32:0/1051343018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55896043d2f0 0x558960814fe0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fac04006010 tx=0x7fac04004040 comp rx=0 tx=0).stop 2024-04-30T08:37:33.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:33.131+0000 7fac1e484740 1 -- 172.21.15.32:0/1051343018 shutdown_connections 2024-04-30T08:37:33.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:33.131+0000 7fac1e484740 1 --2- 172.21.15.32:0/1051343018 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x5589603fa1c0 0x55896081d620 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:33.131+0000 7fac1e484740 1 --2- 172.21.15.32:0/1051343018 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5589604661b0 0x558960811b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:33.131+0000 7fac1e484740 1 --2- 172.21.15.32:0/1051343018 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5589604454d0 0x558960815520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:33.131+0000 7fac1e484740 1 --2- 172.21.15.32:0/1051343018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55896043d2f0 0x558960814fe0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:33.131+0000 7fabf6000640 0 client.26254 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:33.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:33.131+0000 7fac1e484740 1 -- 172.21.15.32:0/1051343018 >> 172.21.15.32:0/1051343018 conn(0x55896045a300 msgr2=0x558960815f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:33.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:33.132+0000 7fac1e484740 1 -- 172.21.15.32:0/1051343018 shutdown_connections 2024-04-30T08:37:33.135 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:33.132+0000 7fac1e484740 1 -- 172.21.15.32:0/1051343018 wait complete. 2024-04-30T08:37:33.135 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:33.133+0000 7fac1e484740 20 client.26254 trim_cache size 0 max 16384 2024-04-30T08:37:33.136 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-30T08:37:33.136 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-30T08:37:33.136 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-30T08:37:33.136 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-30T08:37:33.145 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:33.144+0000 7fac17400640 1 -- 172.21.15.32:0/3807174435 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fabf80afb60 msgr2=0x7fabf80b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:33.146 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:33.144+0000 7fac17400640 1 --2- 172.21.15.32:0/3807174435 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fabf80afb60 0x7fabf80b2070 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x5589608092b0 tx=0x7fabfc005f90 comp rx=0 tx=0).stop 2024-04-30T08:37:33.146 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:33.144+0000 7fac17400640 1 -- 172.21.15.32:0/3807174435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac10109af0 msgr2=0x7fac101a16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:33.146 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:33.144+0000 7fac17400640 1 --2- 172.21.15.32:0/3807174435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac10109af0 0x7fac101a16f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fac08009430 tx=0x7fac080317c0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.146 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:33.145+0000 7fac17400640 1 -- 172.21.15.32:0/3807174435 shutdown_connections 2024-04-30T08:37:33.147 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:33.145+0000 7fac17400640 1 --2- 172.21.15.32:0/3807174435 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fabf80afb60 0x7fabf80b2070 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.147 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:33.145+0000 7fac17400640 1 --2- 172.21.15.32:0/3807174435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fac10109af0 0x7fac101a16f0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.147 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:33.145+0000 7fac17400640 1 --2- 172.21.15.32:0/3807174435 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fac10104df0 0x7fac1019a760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.148 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:33.146+0000 7fac17400640 1 --2- 172.21.15.32:0/3807174435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fac10103bb0 0x7fac1019a220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.148 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:33.146+0000 7fac17400640 1 -- 172.21.15.32:0/3807174435 >> 172.21.15.32:0/3807174435 conn(0x7fac100fed80 msgr2=0x7fac10105ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:33.152 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:33.150+0000 7fac17400640 1 -- 172.21.15.32:0/3807174435 shutdown_connections 2024-04-30T08:37:33.153 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:33.151+0000 7fac17400640 1 -- 172.21.15.32:0/3807174435 wait complete. 2024-04-30T08:37:33.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.431+0000 7f14a7800640 1 -- 172.21.15.32:0/689467835 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14a0103d10 msgr2=0x7f14a0104120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:33.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.431+0000 7f14a7800640 1 --2- 172.21.15.32:0/689467835 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14a0103d10 0x7f14a0104120 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f148c009900 tx=0x7f148c02a3f0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.434+0000 7f14a7800640 1 -- 172.21.15.32:0/689467835 shutdown_connections 2024-04-30T08:37:33.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.434+0000 7f14a7800640 1 --2- 172.21.15.32:0/689467835 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14a0108a10 0x7f14a010ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.434+0000 7f14a7800640 1 --2- 172.21.15.32:0/689467835 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14a0103d10 0x7f14a0104120 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.434+0000 7f14a7800640 1 --2- 172.21.15.32:0/689467835 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14a0102ad0 0x7f14a01037d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.434+0000 7f14a7800640 1 -- 172.21.15.32:0/689467835 >> 172.21.15.32:0/689467835 conn(0x7f14a00fdcc0 msgr2=0x7f14a0100170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:33.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.434+0000 7f14a7800640 1 -- 172.21.15.32:0/689467835 shutdown_connections 2024-04-30T08:37:33.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.434+0000 7f14a7800640 1 -- 172.21.15.32:0/689467835 wait complete. 2024-04-30T08:37:33.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.435+0000 7f14a7800640 1 Processor -- start 2024-04-30T08:37:33.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.436+0000 7f14a7800640 1 -- start start 2024-04-30T08:37:33.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.436+0000 7f14a7800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14a0102ad0 0x7f14a0198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:33.438 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.436+0000 7f14a7800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14a0103d10 0x7f14a0198680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:33.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.436+0000 7f14a7800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14a0108a10 0x7f14a01a7dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:33.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.436+0000 7f14a7800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f14a010f700 con 0x7f14a0108a10 2024-04-30T08:37:33.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.436+0000 7f14a7800640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f14a010f580 con 0x7f14a0103d10 2024-04-30T08:37:33.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.436+0000 7f14a7800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f14a010f880 con 0x7f14a0102ad0 2024-04-30T08:37:33.439 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.437+0000 7f14a4e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14a0102ad0 0x7f14a0198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:33.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.437+0000 7f149fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14a0103d10 0x7f14a0198680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:33.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.437+0000 7f14a4e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14a0102ad0 0x7f14a0198140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:41764/0 (socket says 172.21.15.32:41764) 2024-04-30T08:37:33.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.437+0000 7f14a4e00640 1 -- 172.21.15.32:0/3978603633 learned_addr learned my addr 172.21.15.32:0/3978603633 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:33.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.437+0000 7f14a5800640 1 --2- 172.21.15.32:0/3978603633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14a0108a10 0x7f14a01a7dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:33.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.438+0000 7f14a4e00640 1 -- 172.21.15.32:0/3978603633 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14a0103d10 msgr2=0x7f14a0198680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:33.440 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.438+0000 7f14a4e00640 1 --2- 172.21.15.32:0/3978603633 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14a0103d10 0x7f14a0198680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.438+0000 7f14a4e00640 1 -- 172.21.15.32:0/3978603633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14a0108a10 msgr2=0x7f14a01a7dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:33.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.439+0000 7f14a4e00640 1 --2- 172.21.15.32:0/3978603633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14a0108a10 0x7f14a01a7dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.439+0000 7f14a4e00640 1 -- 172.21.15.32:0/3978603633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14a01a8440 con 0x7f14a0102ad0 2024-04-30T08:37:33.441 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.439+0000 7f14a5800640 1 --2- 172.21.15.32:0/3978603633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14a0108a10 0x7f14a01a7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:37:33.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:33 smithi032 ceph-mon[28331]: pgmap v496: 129 pgs: 129 active+clean; 2.2 GiB data, 703 MiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 184 MiB/s wr, 106 op/s 2024-04-30T08:37:33.442 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.440+0000 7f14a4e00640 1 --2- 172.21.15.32:0/3978603633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14a0102ad0 0x7f14a0198140 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f149400e9f0 tx=0x7f149400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:33.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.451+0000 7f149d600640 1 -- 172.21.15.32:0/3978603633 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f149400ccc0 con 0x7f14a0102ad0 2024-04-30T08:37:33.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.451+0000 7f149d600640 1 -- 172.21.15.32:0/3978603633 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14940035f0 con 0x7f14a0102ad0 2024-04-30T08:37:33.453 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.452+0000 7f149d600640 1 -- 172.21.15.32:0/3978603633 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f149400ce60 con 0x7f14a0102ad0 2024-04-30T08:37:33.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.452+0000 7f14a7800640 1 -- 172.21.15.32:0/3978603633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14a01a8730 con 0x7f14a0102ad0 2024-04-30T08:37:33.454 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.452+0000 7f14a7800640 1 -- 172.21.15.32:0/3978603633 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14a01a8cb0 con 0x7f14a0102ad0 2024-04-30T08:37:33.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.469+0000 7f149d600640 1 -- 172.21.15.32:0/3978603633 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f1494010690 con 0x7f14a0102ad0 2024-04-30T08:37:33.475 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.474+0000 7f149d600640 1 --2- 172.21.15.32:0/3978603633 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f14880afc40 0x7f14880b2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:33.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.475+0000 7f149d600640 1 -- 172.21.15.32:0/3978603633 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f14940e01d0 con 0x7f14a0102ad0 2024-04-30T08:37:33.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.476+0000 7f149fe00640 1 --2- 172.21.15.32:0/3978603633 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f14880afc40 0x7f14880b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:33.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.477+0000 7f149fe00640 1 --2- 172.21.15.32:0/3978603633 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f14880afc40 0x7f14880b2150 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f148c009230 tx=0x7f148c031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:33.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.478+0000 7f14ae870740 1 Processor -- start 2024-04-30T08:37:33.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.478+0000 7f14ae870740 1 -- start start 2024-04-30T08:37:33.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.479+0000 7f14ae870740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55592d020390 0x55592d3c1240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:33.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.479+0000 7f14ae870740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55592cff74d0 0x55592d3c17c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:33.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.479+0000 7f14ae870740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55592cfff6b0 0x55592d3c86c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:33.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.479+0000 7f14ae870740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55592d326d50 con 0x55592d020390 2024-04-30T08:37:33.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.479+0000 7f14ae870740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55592d2cbef0 con 0x55592cff74d0 2024-04-30T08:37:33.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.479+0000 7f14ae870740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55592d322d10 con 0x55592cfff6b0 2024-04-30T08:37:33.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.480+0000 7f14a4e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55592d020390 0x55592d3c1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:33.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.480+0000 7f149fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55592cff74d0 0x55592d3c17c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:33.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.480+0000 7f14a4e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55592d020390 0x55592d3c1240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48138/0 (socket says 172.21.15.32:48138) 2024-04-30T08:37:33.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.480+0000 7f149fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55592cff74d0 0x55592d3c17c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:51044/0 (socket says 172.21.15.32:51044) 2024-04-30T08:37:33.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.480+0000 7f14a4e00640 1 -- 172.21.15.32:0/1736168295 learned_addr learned my addr 172.21.15.32:0/1736168295 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:33.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.480+0000 7f14a5800640 1 --2- 172.21.15.32:0/1736168295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55592cfff6b0 0x55592d3c86c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:33.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.480+0000 7f14a4e00640 1 -- 172.21.15.32:0/1736168295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55592cfff6b0 msgr2=0x55592d3c86c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:33.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.480+0000 7f14a4e00640 1 --2- 172.21.15.32:0/1736168295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55592cfff6b0 0x55592d3c86c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.480+0000 7f14a4e00640 1 -- 172.21.15.32:0/1736168295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55592cff74d0 msgr2=0x55592d3c17c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:33.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.480+0000 7f14a4e00640 1 --2- 172.21.15.32:0/1736168295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55592cff74d0 0x55592d3c17c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.480+0000 7f14a4e00640 1 -- 172.21.15.32:0/1736168295 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55592d1027d0 con 0x55592d020390 2024-04-30T08:37:33.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.481+0000 7f14a4e00640 1 --2- 172.21.15.32:0/1736168295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55592d020390 0x55592d3c1240 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f14940dd870 tx=0x7f1494003a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d4f4b690c8328b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:33.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.482+0000 7f1492000640 1 -- 172.21.15.32:0/1736168295 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14940ac690 con 0x55592d020390 2024-04-30T08:37:33.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.482+0000 7f1492000640 1 -- 172.21.15.32:0/1736168295 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14940b1140 con 0x55592d020390 2024-04-30T08:37:33.485 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.482+0000 7f1492000640 1 -- 172.21.15.32:0/1736168295 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14940b0f10 con 0x55592d020390 2024-04-30T08:37:33.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.484+0000 7f14ae870740 1 -- 172.21.15.32:0/1736168295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55592d020390 msgr2=0x55592d3c1240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:33.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.484+0000 7f14ae870740 1 --2- 172.21.15.32:0/1736168295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55592d020390 0x55592d3c1240 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f14940dd870 tx=0x7f1494003a10 comp rx=0 tx=0).stop 2024-04-30T08:37:33.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.489+0000 7f14ae870740 1 -- 172.21.15.32:0/1736168295 shutdown_connections 2024-04-30T08:37:33.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.489+0000 7f14ae870740 1 --2- 172.21.15.32:0/1736168295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55592cfff6b0 0x55592d3c86c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.489+0000 7f14ae870740 1 --2- 172.21.15.32:0/1736168295 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55592cff74d0 0x55592d3c17c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.489+0000 7f14ae870740 1 --2- 172.21.15.32:0/1736168295 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55592d020390 0x55592d3c1240 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.489+0000 7f14ae870740 1 -- 172.21.15.32:0/1736168295 >> 172.21.15.32:0/1736168295 conn(0x55592d0144e0 msgr2=0x55592d3bcbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:33.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.490+0000 7f14ae870740 1 -- 172.21.15.32:0/1736168295 shutdown_connections 2024-04-30T08:37:33.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.491+0000 7f14ae870740 1 -- 172.21.15.32:0/1736168295 wait complete. 2024-04-30T08:37:33.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.492+0000 7f14ae870740 1 -- start start 2024-04-30T08:37:33.493 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.492+0000 7f14ae870740 1 Processor -- start 2024-04-30T08:37:33.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.494+0000 7f14ae870740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55592cff74d0 0x55592d3cb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:33.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.494+0000 7f14ae870740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55592cfff6b0 0x55592d3cb710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:33.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.494+0000 7f14ae870740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55592d020390 0x55592d3d2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:33.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.494+0000 7f14ae870740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55592d3c9110 con 0x55592cff74d0 2024-04-30T08:37:33.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.494+0000 7f14ae870740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55592d2df990 con 0x55592cfff6b0 2024-04-30T08:37:33.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.494+0000 7f14ae870740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55592d3c9290 con 0x55592d020390 2024-04-30T08:37:33.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.494+0000 7f14ae870740 10 client.0 _ll_register_callbacks cb 0x55592d2de110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:33.496 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.495+0000 7f149fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55592cfff6b0 0x55592d3cb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:33.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.495+0000 7f149fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55592cfff6b0 0x55592d3cb710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:51050/0 (socket says 172.21.15.32:51050) 2024-04-30T08:37:33.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.495+0000 7f149fe00640 1 -- 172.21.15.32:0/3567959323 learned_addr learned my addr 172.21.15.32:0/3567959323 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:33.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.496+0000 7f149fe00640 1 -- 172.21.15.32:0/3567959323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55592d020390 msgr2=0x55592d3d2670 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-30T08:37:33.497 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.496+0000 7f149fe00640 1 --2- 172.21.15.32:0/3567959323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55592d020390 0x55592d3d2670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.498 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.496+0000 7f149fe00640 1 -- 172.21.15.32:0/3567959323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55592cff74d0 msgr2=0x55592d3cb1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-30T08:37:33.502 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.499+0000 7f149fe00640 1 --2- 172.21.15.32:0/3567959323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55592cff74d0 0x55592d3cb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.499+0000 7f149fe00640 1 -- 172.21.15.32:0/3567959323 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55592d0383f0 con 0x55592cfff6b0 2024-04-30T08:37:33.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:33 smithi071 ceph-mon[33385]: pgmap v496: 129 pgs: 129 active+clean; 2.2 GiB data, 703 MiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 184 MiB/s wr, 106 op/s 2024-04-30T08:37:33.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.502+0000 7f1486a00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:37:33.504 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.503+0000 7f149fe00640 1 --2- 172.21.15.32:0/3567959323 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55592cfff6b0 0x55592d3cb710 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f148c009900 tx=0x7f148c009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f98a61d3f238c47 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:33.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.503+0000 7f1486a00640 1 -- 172.21.15.32:0/3567959323 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f148c035050 con 0x55592cfff6b0 2024-04-30T08:37:33.505 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.504+0000 7f1486a00640 1 -- 172.21.15.32:0/3567959323 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f148c003d40 con 0x55592cfff6b0 2024-04-30T08:37:33.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.504+0000 7f14ae870740 1 -- 172.21.15.32:0/3567959323 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f14940b1140 con 0x55592cfff6b0 2024-04-30T08:37:33.506 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.505+0000 7f1486a00640 1 -- 172.21.15.32:0/3567959323 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f148c053470 con 0x55592cfff6b0 2024-04-30T08:37:33.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.505+0000 7f14ae870740 10 client.26270 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:33.507 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.506+0000 7f14ae870740 1 -- 172.21.15.32:0/3567959323 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55592cff80f0 con 0x55592cfff6b0 2024-04-30T08:37:33.509 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.508+0000 7f1486a00640 1 -- 172.21.15.32:0/3567959323 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f14940b1140 con 0x55592cfff6b0 2024-04-30T08:37:33.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.509+0000 7f1486a00640 1 -- 172.21.15.32:0/3567959323 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7f148c04a540 con 0x55592cfff6b0 2024-04-30T08:37:33.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.509+0000 7f14ae870740 10 client.26270 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:33.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.509+0000 7f14ae870740 10 client.26270 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:37:33.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.509+0000 7f14ae870740 20 client.26270 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:33.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.509+0000 7f14ae870740 1 --2- 172.21.15.32:0/3567959323 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55592cfdef60 0x55592d3d8c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:33.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.510+0000 7f14ae870740 4 client.26270 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:37:33.511 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.510+0000 7f14ae870740 1 -- 172.21.15.32:0/3567959323 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55592d33aef0 con 0x55592cfdef60 2024-04-30T08:37:33.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.510+0000 7f14a4e00640 1 --2- 172.21.15.32:0/3567959323 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55592cfdef60 0x55592d3d8c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:33.512 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.511+0000 7f1486a00640 10 client.26270 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:33.513 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.512+0000 7f14a4e00640 1 --2- 172.21.15.32:0/3567959323 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55592cfdef60 0x55592d3d8c60 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f814943cbdd1e13d server_cookie=ba65b3b1ec5cd3d7 in_seq=0 out_seq=0 2024-04-30T08:37:33.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.513+0000 7f1486a00640 1 -- 172.21.15.32:0/3567959323 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55592d33aef0 con 0x55592cfdef60 2024-04-30T08:37:33.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.513+0000 7f1486a00640 10 client.26270 handle_command_reply: tid=0 2024-04-30T08:37:33.515 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.513+0000 7f14ae870740 1 client.26270 shutdown 2024-04-30T08:37:33.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.514+0000 7f14ae870740 1 -- 172.21.15.32:0/3567959323 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55592cfff6b0 msgr2=0x55592d3cb710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:33.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.514+0000 7f14ae870740 1 --2- 172.21.15.32:0/3567959323 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55592cfff6b0 0x55592d3cb710 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f148c009900 tx=0x7f148c009260 comp rx=0 tx=0).stop 2024-04-30T08:37:33.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.514+0000 7f14ae870740 1 -- 172.21.15.32:0/3567959323 shutdown_connections 2024-04-30T08:37:33.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.514+0000 7f14ae870740 1 --2- 172.21.15.32:0/3567959323 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55592cfdef60 0x55592d3d8c60 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.514+0000 7f14ae870740 1 --2- 172.21.15.32:0/3567959323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55592d020390 0x55592d3d2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.514+0000 7f14ae870740 1 --2- 172.21.15.32:0/3567959323 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55592cfff6b0 0x55592d3cb710 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.514+0000 7f14ae870740 1 --2- 172.21.15.32:0/3567959323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55592cff74d0 0x55592d3cb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.516 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.514+0000 7f1486a00640 0 client.26270 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:33.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.514+0000 7f14ae870740 1 -- 172.21.15.32:0/3567959323 >> 172.21.15.32:0/3567959323 conn(0x55592d0144e0 msgr2=0x55592d3bbbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:33.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.514+0000 7f14ae870740 1 -- 172.21.15.32:0/3567959323 shutdown_connections 2024-04-30T08:37:33.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.514+0000 7f14ae870740 1 -- 172.21.15.32:0/3567959323 wait complete. 2024-04-30T08:37:33.517 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.515+0000 7f14ae870740 20 client.26270 trim_cache size 0 max 16384 2024-04-30T08:37:33.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:33 smithi189 ceph-mon[33038]: pgmap v496: 129 pgs: 129 active+clean; 2.2 GiB data, 703 MiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 184 MiB/s wr, 106 op/s 2024-04-30T08:37:33.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.553+0000 7f14ae870740 1 Processor -- start 2024-04-30T08:37:33.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.553+0000 7f14ae870740 1 -- start start 2024-04-30T08:37:33.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.554+0000 7f14ae870740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55592cff74d0 0x55592d3c5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:33.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.554+0000 7f14ae870740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55592cfff6b0 0x55592d3c04b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:33.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.554+0000 7f14ae870740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55592d020390 0x55592d3de460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:33.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.554+0000 7f14ae870740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f14a01b3330 con 0x55592cff74d0 2024-04-30T08:37:33.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.554+0000 7f14ae870740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f14a01b4f00 con 0x55592cfff6b0 2024-04-30T08:37:33.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.554+0000 7f14ae870740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f14a01b3a00 con 0x55592d020390 2024-04-30T08:37:33.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.555+0000 7f14a4e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55592cff74d0 0x55592d3c5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:33.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.555+0000 7f14a5800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55592d020390 0x55592d3de460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:33.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.555+0000 7f14a4e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55592cff74d0 0x55592d3c5cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48154/0 (socket says 172.21.15.32:48154) 2024-04-30T08:37:33.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.555+0000 7f14a4e00640 1 -- 172.21.15.32:0/3317755670 learned_addr learned my addr 172.21.15.32:0/3317755670 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:33.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.555+0000 7f149fe00640 1 --2- 172.21.15.32:0/3317755670 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55592cfff6b0 0x55592d3c04b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:33.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.555+0000 7f149fe00640 1 -- 172.21.15.32:0/3317755670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55592d020390 msgr2=0x55592d3de460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:33.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.555+0000 7f149fe00640 1 --2- 172.21.15.32:0/3317755670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55592d020390 0x55592d3de460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.557 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.555+0000 7f149fe00640 1 -- 172.21.15.32:0/3317755670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55592cff74d0 msgr2=0x55592d3c5cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:33.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.556+0000 7f149fe00640 1 --2- 172.21.15.32:0/3317755670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55592cff74d0 0x55592d3c5cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.556+0000 7f149fe00640 1 -- 172.21.15.32:0/3317755670 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55592d3d6900 con 0x55592cfff6b0 2024-04-30T08:37:33.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.556+0000 7f14a4e00640 1 --2- 172.21.15.32:0/3317755670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55592cff74d0 0x55592d3c5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:37:33.558 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.557+0000 7f149fe00640 1 --2- 172.21.15.32:0/3317755670 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55592cfff6b0 0x55592d3c04b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f148c04a910 tx=0x7f148c031850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5071d40e006bcf29 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:33.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.557+0000 7f1491600640 1 -- 172.21.15.32:0/3317755670 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f148c035070 con 0x55592cfff6b0 2024-04-30T08:37:33.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.558+0000 7f1491600640 1 -- 172.21.15.32:0/3317755670 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f148c04e070 con 0x55592cfff6b0 2024-04-30T08:37:33.559 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.558+0000 7f1491600640 1 -- 172.21.15.32:0/3317755670 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f148c053870 con 0x55592cfff6b0 2024-04-30T08:37:33.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.559+0000 7f14ae870740 1 -- 172.21.15.32:0/3317755670 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55592cfff6b0 msgr2=0x55592d3c04b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:33.560 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.559+0000 7f14ae870740 1 --2- 172.21.15.32:0/3317755670 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55592cfff6b0 0x55592d3c04b0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f148c04a910 tx=0x7f148c031850 comp rx=0 tx=0).stop 2024-04-30T08:37:33.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.559+0000 7f14ae870740 1 -- 172.21.15.32:0/3317755670 shutdown_connections 2024-04-30T08:37:33.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.559+0000 7f14ae870740 1 --2- 172.21.15.32:0/3317755670 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55592d020390 0x55592d3de460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.559+0000 7f14ae870740 1 --2- 172.21.15.32:0/3317755670 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55592cfff6b0 0x55592d3c04b0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.559+0000 7f14ae870740 1 --2- 172.21.15.32:0/3317755670 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55592cff74d0 0x55592d3c5cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.559+0000 7f14ae870740 1 -- 172.21.15.32:0/3317755670 >> 172.21.15.32:0/3317755670 conn(0x55592d0144e0 msgr2=0x55592d3c97f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:33.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.560+0000 7f14ae870740 1 -- 172.21.15.32:0/3317755670 shutdown_connections 2024-04-30T08:37:33.561 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.560+0000 7f14ae870740 1 -- 172.21.15.32:0/3317755670 wait complete. 2024-04-30T08:37:33.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.560+0000 7f14ae870740 1 -- start start 2024-04-30T08:37:33.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.561+0000 7f14ae870740 1 Processor -- start 2024-04-30T08:37:33.562 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.561+0000 7f14ae870740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55592cff74d0 0x55592d3d1ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:33.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.561+0000 7f14ae870740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55592cfff6b0 0x55592d3d23e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:33.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.561+0000 7f14ae870740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55592d020390 0x55592d450010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:33.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.561+0000 7f14ae870740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f14a01b47a0 con 0x55592d020390 2024-04-30T08:37:33.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.561+0000 7f14ae870740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f14a01b40d0 con 0x55592cff74d0 2024-04-30T08:37:33.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.561+0000 7f14ae870740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f14a01b12f0 con 0x55592cfff6b0 2024-04-30T08:37:33.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.561+0000 7f14ae870740 10 client.0 _ll_register_callbacks cb 0x7f148c031640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:33.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.562+0000 7f149fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55592cfff6b0 0x55592d3d23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:33.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.562+0000 7f149fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55592cfff6b0 0x55592d3d23e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:41782/0 (socket says 172.21.15.32:41782) 2024-04-30T08:37:33.563 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.562+0000 7f149fe00640 1 -- 172.21.15.32:0/1466924162 learned_addr learned my addr 172.21.15.32:0/1466924162 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:33.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.562+0000 7f149fe00640 1 -- 172.21.15.32:0/1466924162 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55592cff74d0 msgr2=0x55592d3d1ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:33.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.562+0000 7f149fe00640 1 --2- 172.21.15.32:0/1466924162 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55592cff74d0 0x55592d3d1ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.562+0000 7f149fe00640 1 -- 172.21.15.32:0/1466924162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55592d020390 msgr2=0x55592d450010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:33.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.562+0000 7f149fe00640 1 --2- 172.21.15.32:0/1466924162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55592d020390 0x55592d450010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.562+0000 7f149fe00640 1 -- 172.21.15.32:0/1466924162 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55592d450580 con 0x55592cfff6b0 2024-04-30T08:37:33.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.562+0000 7f149fe00640 1 --2- 172.21.15.32:0/1466924162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55592cfff6b0 0x55592d3d23e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f148c031b80 tx=0x7f148c0314c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a51644741d7efd1 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:33.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.562+0000 7f1486a00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-30T08:37:33.564 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.563+0000 7f1486a00640 1 -- 172.21.15.32:0/1466924162 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f148c035040 con 0x55592cfff6b0 2024-04-30T08:37:33.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.563+0000 7f14ae870740 1 -- 172.21.15.32:0/1466924162 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f148c04e070 con 0x55592cfff6b0 2024-04-30T08:37:33.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.564+0000 7f1486a00640 1 -- 172.21.15.32:0/1466924162 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f148c053610 con 0x55592cfff6b0 2024-04-30T08:37:33.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.564+0000 7f1486a00640 1 -- 172.21.15.32:0/1466924162 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f148c078050 con 0x55592cfff6b0 2024-04-30T08:37:33.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.564+0000 7f14ae870740 10 client.26269 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:33.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.564+0000 7f14ae870740 1 -- 172.21.15.32:0/1466924162 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55592d4517f0 con 0x55592cfff6b0 2024-04-30T08:37:33.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.564+0000 7f1486a00640 1 -- 172.21.15.32:0/1466924162 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f148c04e070 con 0x55592cfff6b0 2024-04-30T08:37:33.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.564+0000 7f1486a00640 1 -- 172.21.15.32:0/1466924162 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7f148c065890 con 0x55592cfff6b0 2024-04-30T08:37:33.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.564+0000 7f14ae870740 10 client.26269 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:33.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.564+0000 7f14ae870740 10 client.26269 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:37:33.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.564+0000 7f14ae870740 20 client.26269 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:33.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.565+0000 7f14ae870740 1 --2- 172.21.15.32:0/1466924162 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55592cfb43a0 0x55592d454190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:33.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.565+0000 7f14ae870740 4 client.26269 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-30T08:37:33.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.565+0000 7f14ae870740 1 -- 172.21.15.32:0/1466924162 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55592d334960 con 0x55592cfb43a0 2024-04-30T08:37:33.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.565+0000 7f14a4e00640 1 --2- 172.21.15.32:0/1466924162 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55592cfb43a0 0x55592d454190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:33.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.566+0000 7f14a4e00640 1 --2- 172.21.15.32:0/1466924162 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55592cfb43a0 0x55592d454190 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c40b45f7701339a0 server_cookie=3ed171f755dd5bc7 in_seq=0 out_seq=0 2024-04-30T08:37:33.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.566+0000 7f1486a00640 10 client.26269 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:33.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.566+0000 7f1486a00640 1 -- 172.21.15.32:0/1466924162 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55592d334960 con 0x55592cfb43a0 2024-04-30T08:37:33.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.566+0000 7f1486a00640 10 client.26269 handle_command_reply: tid=0 2024-04-30T08:37:33.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.566+0000 7f14ae870740 1 client.26269 shutdown 2024-04-30T08:37:33.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.568+0000 7f14ae870740 1 -- 172.21.15.32:0/1466924162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55592cfff6b0 msgr2=0x55592d3d23e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:33.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.568+0000 7f14ae870740 1 --2- 172.21.15.32:0/1466924162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55592cfff6b0 0x55592d3d23e0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f148c031b80 tx=0x7f148c0314c0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.568+0000 7f14ae870740 1 -- 172.21.15.32:0/1466924162 shutdown_connections 2024-04-30T08:37:33.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.568+0000 7f14ae870740 1 --2- 172.21.15.32:0/1466924162 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55592cfb43a0 0x55592d454190 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.568+0000 7f14ae870740 1 --2- 172.21.15.32:0/1466924162 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55592d020390 0x55592d450010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.568+0000 7f14ae870740 1 --2- 172.21.15.32:0/1466924162 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55592cfff6b0 0x55592d3d23e0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.568+0000 7f14ae870740 1 --2- 172.21.15.32:0/1466924162 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55592cff74d0 0x55592d3d1ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.568+0000 7f14ae870740 1 -- 172.21.15.32:0/1466924162 >> 172.21.15.32:0/1466924162 conn(0x55592d0144e0 msgr2=0x55592d3c0210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:33.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.569+0000 7f1486a00640 0 client.26269 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:33.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.569+0000 7f14ae870740 1 -- 172.21.15.32:0/1466924162 shutdown_connections 2024-04-30T08:37:33.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.569+0000 7f14ae870740 1 -- 172.21.15.32:0/1466924162 wait complete. 2024-04-30T08:37:33.571 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.569+0000 7f14ae870740 20 client.26269 trim_cache size 0 max 16384 2024-04-30T08:37:33.571 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-30T08:37:33.571 INFO:teuthology.orchestra.run.smithi032.stdout: "status": "no active scrubs running", 2024-04-30T08:37:33.572 INFO:teuthology.orchestra.run.smithi032.stdout: "scrubs": {} 2024-04-30T08:37:33.572 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-30T08:37:33.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.570+0000 7f14a7800640 1 -- 172.21.15.32:0/3978603633 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f14880afc40 msgr2=0x7f14880b2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:33.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.570+0000 7f14a7800640 1 --2- 172.21.15.32:0/3978603633 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f14880afc40 0x7f14880b2150 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f148c009230 tx=0x7f148c031400 comp rx=0 tx=0).stop 2024-04-30T08:37:33.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.570+0000 7f14a7800640 1 -- 172.21.15.32:0/3978603633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14a0102ad0 msgr2=0x7f14a0198140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:33.572 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.570+0000 7f14a7800640 1 --2- 172.21.15.32:0/3978603633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14a0102ad0 0x7f14a0198140 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f149400e9f0 tx=0x7f149400eeb0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.577+0000 7f14a7800640 1 -- 172.21.15.32:0/3978603633 shutdown_connections 2024-04-30T08:37:33.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.577+0000 7f14a7800640 1 --2- 172.21.15.32:0/3978603633 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f14880afc40 0x7f14880b2150 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.577+0000 7f14a7800640 1 --2- 172.21.15.32:0/3978603633 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f14a0108a10 0x7f14a01a7dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.577+0000 7f14a7800640 1 --2- 172.21.15.32:0/3978603633 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f14a0103d10 0x7f14a0198680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.577+0000 7f14a7800640 1 --2- 172.21.15.32:0/3978603633 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f14a0102ad0 0x7f14a0198140 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:33.578 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.577+0000 7f14a7800640 1 -- 172.21.15.32:0/3978603633 >> 172.21.15.32:0/3978603633 conn(0x7f14a00fdcc0 msgr2=0x7f14a00fe290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:33.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.585+0000 7f14a7800640 1 -- 172.21.15.32:0/3978603633 shutdown_connections 2024-04-30T08:37:33.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:33.585+0000 7f14a7800640 1 -- 172.21.15.32:0/3978603633 wait complete. 2024-04-30T08:37:33.650 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-30T08:37:33.650 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-30T08:37:34.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.105+0000 7fa279e00640 1 -- 172.21.15.32:0/2252723174 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa274109a90 msgr2=0x7fa27410beb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:34.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.105+0000 7fa279e00640 1 --2- 172.21.15.32:0/2252723174 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa274109a90 0x7fa27410beb0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fa26c009ca0 tx=0x7fa26c02a8b0 comp rx=0 tx=0).stop 2024-04-30T08:37:34.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.105+0000 7fa279e00640 1 -- 172.21.15.32:0/2252723174 shutdown_connections 2024-04-30T08:37:34.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.105+0000 7fa279e00640 1 --2- 172.21.15.32:0/2252723174 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa274109a90 0x7fa27410beb0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:34.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.105+0000 7fa279e00640 1 --2- 172.21.15.32:0/2252723174 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa274104d90 0x7fa2741051a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:34.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.105+0000 7fa279e00640 1 --2- 172.21.15.32:0/2252723174 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa274103b50 0x7fa274104850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:34.107 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.105+0000 7fa279e00640 1 -- 172.21.15.32:0/2252723174 >> 172.21.15.32:0/2252723174 conn(0x7fa2740fed40 msgr2=0x7fa2741011f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:34.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.105+0000 7fa279e00640 1 -- 172.21.15.32:0/2252723174 shutdown_connections 2024-04-30T08:37:34.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.105+0000 7fa279e00640 1 -- 172.21.15.32:0/2252723174 wait complete. 2024-04-30T08:37:34.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.106+0000 7fa279e00640 1 Processor -- start 2024-04-30T08:37:34.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.106+0000 7fa279e00640 1 -- start start 2024-04-30T08:37:34.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.107+0000 7fa279e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa274103b50 0x7fa2740768d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:34.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.107+0000 7fa279e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa274104d90 0x7fa274074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:34.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.107+0000 7fa279e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa274109a90 0x7fa2740754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:34.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.107+0000 7fa279e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa27410f6d0 con 0x7fa274103b50 2024-04-30T08:37:34.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.107+0000 7fa279e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa27410f550 con 0x7fa274104d90 2024-04-30T08:37:34.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.107+0000 7fa279e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa27410f850 con 0x7fa274109a90 2024-04-30T08:37:34.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.107+0000 7fa272000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa274104d90 0x7fa274074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:34.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.107+0000 7fa272a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa274103b50 0x7fa2740768d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:34.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.107+0000 7fa272000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa274104d90 0x7fa274074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:51082/0 (socket says 172.21.15.32:51082) 2024-04-30T08:37:34.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.107+0000 7fa273400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa274109a90 0x7fa2740754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:34.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.107+0000 7fa273400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa274109a90 0x7fa2740754b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:41798/0 (socket says 172.21.15.32:41798) 2024-04-30T08:37:34.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.107+0000 7fa273400640 1 -- 172.21.15.32:0/847889596 learned_addr learned my addr 172.21.15.32:0/847889596 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:34.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.107+0000 7fa272000640 1 -- 172.21.15.32:0/847889596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa274109a90 msgr2=0x7fa2740754b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:34.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.107+0000 7fa272000640 1 --2- 172.21.15.32:0/847889596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa274109a90 0x7fa2740754b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:34.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.107+0000 7fa272000640 1 -- 172.21.15.32:0/847889596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa274103b50 msgr2=0x7fa2740768d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:34.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.107+0000 7fa272000640 1 --2- 172.21.15.32:0/847889596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa274103b50 0x7fa2740768d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:34.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.107+0000 7fa272000640 1 -- 172.21.15.32:0/847889596 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa274075be0 con 0x7fa274104d90 2024-04-30T08:37:34.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.107+0000 7fa273400640 1 --2- 172.21.15.32:0/847889596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa274109a90 0x7fa2740754b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:37:34.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.109+0000 7fa272000640 1 --2- 172.21.15.32:0/847889596 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa274104d90 0x7fa274074f70 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa26000d720 tx=0x7fa26000dbe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:34.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.109+0000 7fa278a00640 1 -- 172.21.15.32:0/847889596 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa260017070 con 0x7fa274104d90 2024-04-30T08:37:34.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.109+0000 7fa278a00640 1 -- 172.21.15.32:0/847889596 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa26000de30 con 0x7fa274104d90 2024-04-30T08:37:34.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.109+0000 7fa278a00640 1 -- 172.21.15.32:0/847889596 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa260004d00 con 0x7fa274104d90 2024-04-30T08:37:34.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.109+0000 7fa279e00640 1 -- 172.21.15.32:0/847889596 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2741a42e0 con 0x7fa274104d90 2024-04-30T08:37:34.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.109+0000 7fa279e00640 1 -- 172.21.15.32:0/847889596 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2741a4750 con 0x7fa274104d90 2024-04-30T08:37:34.112 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.110+0000 7fa279e00640 1 -- 172.21.15.32:0/847889596 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa274069390 con 0x7fa274104d90 2024-04-30T08:37:34.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.113+0000 7fa278a00640 1 -- 172.21.15.32:0/847889596 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7fa260004ea0 con 0x7fa274104d90 2024-04-30T08:37:34.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.115+0000 7fa278a00640 1 --2- 172.21.15.32:0/847889596 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa2580afb70 0x7fa2580b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:34.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.115+0000 7fa278a00640 1 -- 172.21.15.32:0/847889596 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fa2600dec00 con 0x7fa274104d90 2024-04-30T08:37:34.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.115+0000 7fa272a00640 1 --2- 172.21.15.32:0/847889596 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa2580afb70 0x7fa2580b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:34.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.116+0000 7fa278a00640 1 -- 172.21.15.32:0/847889596 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa2600a6e00 con 0x7fa274104d90 2024-04-30T08:37:34.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.117+0000 7fa272a00640 1 --2- 172.21.15.32:0/847889596 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa2580afb70 0x7fa2580b2080 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa274104750 tx=0x7fa264005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:34.613 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.612+0000 7fa279e00640 1 -- 172.21.15.32:0/847889596 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa27402df40 con 0x7fa274104d90 2024-04-30T08:37:34.622 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:37:34.622 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2235842676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2235842676},{"type":"v1","addr":"172.21.15.32:6835","nonce":2235842676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/3577009355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":3577009355},{"type":"v1","addr":"172.21.15.32:6841","nonce":3577009355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-30T08:31:52.236668+0000","modified":"2024-04-30T08:36:54.799807+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24541,"mds_3":24533,"mds_4":24517},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/463203038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":463203038},{"type":"v1","addr":"172.21.15.189:6833","nonce":463203038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6833/3635143591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3635143591},{"type":"v1","addr":"172.21.15.71:6833","nonce":3635143591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6837/2569449994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2569449994},{"type":"v1","addr":"172.21.15.32:6837","nonce":2569449994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/278686565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":278686565},{"type":"v1","addr":"172.21.15.189:6835","nonce":278686565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6835/1558303056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1558303056},{"type":"v1","addr":"172.21.15.71:6835","nonce":1558303056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.189:6837/686520873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":686520873},{"type":"v1","addr":"172.21.15.189:6837","nonce":686520873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.71:6837/2324039932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2324039932},{"type":"v1","addr":"172.21.15.71:6837","nonce":2324039932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6839/352970107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":352970107},{"type":"v1","addr":"172.21.15.189:6839","nonce":352970107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.32:6839/4160386524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4160386524},{"type":"v1","addr":"172.21.15.32:6839","nonce":4160386524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.71:6839/3517409993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3517409993},{"type":"v1","addr":"172.21.15.71:6839","nonce":3517409993}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24541,24590,24560,24517,24533]},"id":1}]} 2024-04-30T08:37:34.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.619+0000 7fa278a00640 1 -- 172.21.15.32:0/847889596 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11932 (secure 0 0 0) 0x7fa2600ab100 con 0x7fa274104d90 2024-04-30T08:37:34.622 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-04-30T08:37:34.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.624+0000 7fa279e00640 1 -- 172.21.15.32:0/847889596 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa2580afb70 msgr2=0x7fa2580b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:34.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.624+0000 7fa279e00640 1 --2- 172.21.15.32:0/847889596 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa2580afb70 0x7fa2580b2080 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fa274104750 tx=0x7fa264005f90 comp rx=0 tx=0).stop 2024-04-30T08:37:34.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.624+0000 7fa279e00640 1 -- 172.21.15.32:0/847889596 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa274104d90 msgr2=0x7fa274074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:34.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.624+0000 7fa279e00640 1 --2- 172.21.15.32:0/847889596 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa274104d90 0x7fa274074f70 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa26000d720 tx=0x7fa26000dbe0 comp rx=0 tx=0).stop 2024-04-30T08:37:34.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.625+0000 7fa279e00640 1 -- 172.21.15.32:0/847889596 shutdown_connections 2024-04-30T08:37:34.626 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.625+0000 7fa279e00640 1 --2- 172.21.15.32:0/847889596 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa2580afb70 0x7fa2580b2080 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:34.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.625+0000 7fa279e00640 1 --2- 172.21.15.32:0/847889596 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa274109a90 0x7fa2740754b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:34.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.625+0000 7fa279e00640 1 --2- 172.21.15.32:0/847889596 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa274104d90 0x7fa274074f70 secure :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fa26000d720 tx=0x7fa26000dbe0 comp rx=0 tx=0).stop 2024-04-30T08:37:34.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.625+0000 7fa279e00640 1 --2- 172.21.15.32:0/847889596 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa274103b50 0x7fa2740768d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:34.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.625+0000 7fa279e00640 1 -- 172.21.15.32:0/847889596 >> 172.21.15.32:0/847889596 conn(0x7fa2740fed40 msgr2=0x7fa274105a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:34.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.627+0000 7fa279e00640 1 -- 172.21.15.32:0/847889596 shutdown_connections 2024-04-30T08:37:34.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:34.628+0000 7fa279e00640 1 -- 172.21.15.32:0/847889596 wait complete. 2024-04-30T08:37:34.664 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-30T08:37:35.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.086+0000 7f1761200640 1 -- 172.21.15.32:0/1976515035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f175c102260 msgr2=0x7f175c104680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:35.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.086+0000 7f1761200640 1 --2- 172.21.15.32:0/1976515035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f175c102260 0x7f175c104680 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f174c009900 tx=0x7f174c02a420 comp rx=0 tx=0).stop 2024-04-30T08:37:35.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.086+0000 7f1761200640 1 -- 172.21.15.32:0/1976515035 shutdown_connections 2024-04-30T08:37:35.088 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.086+0000 7f1761200640 1 --2- 172.21.15.32:0/1976515035 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f175c107520 0x7f175c107930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.086+0000 7f1761200640 1 --2- 172.21.15.32:0/1976515035 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f175c104bc0 0x7f175c106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.086+0000 7f1761200640 1 --2- 172.21.15.32:0/1976515035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f175c102260 0x7f175c104680 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.086+0000 7f1761200640 1 -- 172.21.15.32:0/1976515035 >> 172.21.15.32:0/1976515035 conn(0x7f175c0fda70 msgr2=0x7f175c0fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:35.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.087+0000 7f1761200640 1 -- 172.21.15.32:0/1976515035 shutdown_connections 2024-04-30T08:37:35.089 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.087+0000 7f1761200640 1 -- 172.21.15.32:0/1976515035 wait complete. 2024-04-30T08:37:35.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.088+0000 7f1761200640 1 Processor -- start 2024-04-30T08:37:35.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.088+0000 7f1761200640 1 -- start start 2024-04-30T08:37:35.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.088+0000 7f1761200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f175c102260 0x7f175c1a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.088+0000 7f1761200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f175c104bc0 0x7f175c1a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.089+0000 7f175b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f175c102260 0x7f175c1a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:35.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.089+0000 7f175b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f175c102260 0x7f175c1a06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:41816/0 (socket says 172.21.15.32:41816) 2024-04-30T08:37:35.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.089+0000 7f175aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f175c104bc0 0x7f175c1a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:35.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.089+0000 7f175aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f175c104bc0 0x7f175c1a0c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48202/0 (socket says 172.21.15.32:48202) 2024-04-30T08:37:35.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.089+0000 7f175b400640 1 -- 172.21.15.32:0/1144273417 learned_addr learned my addr 172.21.15.32:0/1144273417 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:35.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.089+0000 7f1761200640 1 --2- 172.21.15.32:0/1144273417 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f175c107520 0x7f175c1a7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.089+0000 7f1761200640 1 -- 172.21.15.32:0/1144273417 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f175c115af0 con 0x7f175c104bc0 2024-04-30T08:37:35.092 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.089+0000 7f1761200640 1 -- 172.21.15.32:0/1144273417 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f175c115970 con 0x7f175c107520 2024-04-30T08:37:35.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.089+0000 7f1761200640 1 -- 172.21.15.32:0/1144273417 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f175c115c70 con 0x7f175c102260 2024-04-30T08:37:35.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.090+0000 7f175be00640 1 --2- 172.21.15.32:0/1144273417 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f175c107520 0x7f175c1a7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:35.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.090+0000 7f175b400640 1 -- 172.21.15.32:0/1144273417 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f175c107520 msgr2=0x7f175c1a7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:35.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.090+0000 7f175b400640 1 --2- 172.21.15.32:0/1144273417 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f175c107520 0x7f175c1a7ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.090+0000 7f175b400640 1 -- 172.21.15.32:0/1144273417 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f175c104bc0 msgr2=0x7f175c1a0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:35.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.090+0000 7f175b400640 1 --2- 172.21.15.32:0/1144273417 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f175c104bc0 0x7f175c1a0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.090+0000 7f175b400640 1 -- 172.21.15.32:0/1144273417 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f175c1a8210 con 0x7f175c102260 2024-04-30T08:37:35.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.090+0000 7f175be00640 1 --2- 172.21.15.32:0/1144273417 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f175c107520 0x7f175c1a7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:37:35.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.090+0000 7f175b400640 1 --2- 172.21.15.32:0/1144273417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f175c102260 0x7f175c1a06d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f174c031470 tx=0x7f174c003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:35.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.091+0000 7f1753e00640 1 -- 172.21.15.32:0/1144273417 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f174c03f070 con 0x7f175c102260 2024-04-30T08:37:35.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.091+0000 7f1753e00640 1 -- 172.21.15.32:0/1144273417 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f174c031bd0 con 0x7f175c102260 2024-04-30T08:37:35.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.091+0000 7f1761200640 1 -- 172.21.15.32:0/1144273417 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f175c1a84a0 con 0x7f175c102260 2024-04-30T08:37:35.093 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.091+0000 7f1753e00640 1 -- 172.21.15.32:0/1144273417 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f174c035030 con 0x7f175c102260 2024-04-30T08:37:35.094 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.091+0000 7f1761200640 1 -- 172.21.15.32:0/1144273417 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f175c1a8990 con 0x7f175c102260 2024-04-30T08:37:35.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.094+0000 7f176980a740 1 Processor -- start 2024-04-30T08:37:35.096 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.095+0000 7f176980a740 1 -- start start 2024-04-30T08:37:35.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.095+0000 7f176980a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5558ceef8390 0x5558cf299730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.095+0000 7f176980a740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5558ceecf4d0 0x5558cf299cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.095+0000 7f176980a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558ceed76b0 0x5558cf2a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.095+0000 7f176980a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5558cf1fed50 con 0x5558ceef8390 2024-04-30T08:37:35.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.095+0000 7f176980a740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x5558cf1a40c0 con 0x5558ceecf4d0 2024-04-30T08:37:35.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.095+0000 7f176980a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5558cf1fad10 con 0x5558ceed76b0 2024-04-30T08:37:35.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.095+0000 7f175be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5558ceef8390 0x5558cf299730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:35.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.095+0000 7f175be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5558ceef8390 0x5558cf299730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48208/0 (socket says 172.21.15.32:48208) 2024-04-30T08:37:35.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.095+0000 7f175be00640 1 -- 172.21.15.32:0/613478435 learned_addr learned my addr 172.21.15.32:0/613478435 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:35.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.096+0000 7f175be00640 1 -- 172.21.15.32:0/613478435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558ceed76b0 msgr2=0x5558cf2a0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:35.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.096+0000 7f175be00640 1 --2- 172.21.15.32:0/613478435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558ceed76b0 0x5558cf2a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.096+0000 7f175be00640 1 -- 172.21.15.32:0/613478435 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5558ceecf4d0 msgr2=0x5558cf299cb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-30T08:37:35.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.096+0000 7f175be00640 1 --2- 172.21.15.32:0/613478435 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5558ceecf4d0 0x5558cf299cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.098 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.096+0000 7f175be00640 1 -- 172.21.15.32:0/613478435 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558cef103f0 con 0x5558ceef8390 2024-04-30T08:37:35.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.096+0000 7f175be00640 1 --2- 172.21.15.32:0/613478435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5558ceef8390 0x5558cf299730 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f175400f360 tx=0x7f175400f820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95477dd3bc103ff3 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:35.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.096+0000 7f1753e00640 1 -- 172.21.15.32:0/1144273417 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f174c042070 con 0x7f175c102260 2024-04-30T08:37:35.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.096+0000 7f1750c00640 1 -- 172.21.15.32:0/613478435 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1754015070 con 0x5558ceef8390 2024-04-30T08:37:35.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.097+0000 7f1750c00640 1 -- 172.21.15.32:0/613478435 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1754004d20 con 0x5558ceef8390 2024-04-30T08:37:35.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.097+0000 7f1750c00640 1 -- 172.21.15.32:0/613478435 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1754005440 con 0x5558ceef8390 2024-04-30T08:37:35.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.098+0000 7f176980a740 1 -- 172.21.15.32:0/613478435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5558ceef8390 msgr2=0x5558cf299730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:35.099 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.098+0000 7f176980a740 1 --2- 172.21.15.32:0/613478435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5558ceef8390 0x5558cf299730 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f175400f360 tx=0x7f175400f820 comp rx=0 tx=0).stop 2024-04-30T08:37:35.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.098+0000 7f176980a740 1 -- 172.21.15.32:0/613478435 shutdown_connections 2024-04-30T08:37:35.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.098+0000 7f176980a740 1 --2- 172.21.15.32:0/613478435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558ceed76b0 0x5558cf2a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.098+0000 7f176980a740 1 --2- 172.21.15.32:0/613478435 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5558ceecf4d0 0x5558cf299cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.100 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.098+0000 7f176980a740 1 --2- 172.21.15.32:0/613478435 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5558ceef8390 0x5558cf299730 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.098+0000 7f176980a740 1 -- 172.21.15.32:0/613478435 >> 172.21.15.32:0/613478435 conn(0x5558ceeec4e0 msgr2=0x5558cf2950c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:35.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.099+0000 7f176980a740 1 -- 172.21.15.32:0/613478435 shutdown_connections 2024-04-30T08:37:35.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.099+0000 7f176980a740 1 -- 172.21.15.32:0/613478435 wait complete. 2024-04-30T08:37:35.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.099+0000 7f1753e00640 1 --2- 172.21.15.32:0/1144273417 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f17400afa10 0x7f17400b1f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.099+0000 7f1753e00640 1 -- 172.21.15.32:0/1144273417 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f174c0ff540 con 0x7f175c102260 2024-04-30T08:37:35.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.100+0000 7f176980a740 1 -- start start 2024-04-30T08:37:35.101 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.100+0000 7f176980a740 1 Processor -- start 2024-04-30T08:37:35.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.101+0000 7f175be00640 1 --2- 172.21.15.32:0/1144273417 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f17400afa10 0x7f17400b1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:35.102 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.101+0000 7f176980a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558ceecf4d0 0x5558cf2a3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.101+0000 7f176980a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5558ceed76b0 0x5558cf2a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.103 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.101+0000 7f176980a740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5558ceef8390 0x5558cf2aab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.101+0000 7f176980a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5558cf2a1650 con 0x5558ceed76b0 2024-04-30T08:37:35.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.101+0000 7f176980a740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x5558cf1b7b50 con 0x5558ceef8390 2024-04-30T08:37:35.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.101+0000 7f176980a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5558cf2a17d0 con 0x5558ceecf4d0 2024-04-30T08:37:35.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.101+0000 7f176980a740 10 client.0 _ll_register_callbacks cb 0x5558cf181420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:35.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.101+0000 7f175be00640 1 --2- 172.21.15.32:0/1144273417 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f17400afa10 0x7f17400b1f20 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1754009dc0 tx=0x7f1754005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:35.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.102+0000 7f175b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558ceecf4d0 0x5558cf2a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:35.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.102+0000 7f175b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558ceecf4d0 0x5558cf2a3690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:41834/0 (socket says 172.21.15.32:41834) 2024-04-30T08:37:35.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.102+0000 7f175b400640 1 -- 172.21.15.32:0/2156605776 learned_addr learned my addr 172.21.15.32:0/2156605776 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:35.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.102+0000 7f175be00640 1 --2- 172.21.15.32:0/2156605776 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5558ceef8390 0x5558cf2aab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:35.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.102+0000 7f175aa00640 1 --2- 172.21.15.32:0/2156605776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5558ceed76b0 0x5558cf2a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:35.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.102+0000 7f175b400640 1 -- 172.21.15.32:0/2156605776 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5558ceef8390 msgr2=0x5558cf2aab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:35.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.102+0000 7f175b400640 1 --2- 172.21.15.32:0/2156605776 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5558ceef8390 0x5558cf2aab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.102+0000 7f175b400640 1 -- 172.21.15.32:0/2156605776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5558ceed76b0 msgr2=0x5558cf2a3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:35.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.102+0000 7f175b400640 1 --2- 172.21.15.32:0/2156605776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5558ceed76b0 0x5558cf2a3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.102+0000 7f175b400640 1 -- 172.21.15.32:0/2156605776 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558cefda7d0 con 0x5558ceecf4d0 2024-04-30T08:37:35.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.102+0000 7f175b400640 1 --2- 172.21.15.32:0/2156605776 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558ceecf4d0 0x5558cf2a3690 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f174c0c4880 tx=0x7f174c0c7120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e085fe9d780c7b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:35.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.102+0000 7f1745600640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-30T08:37:35.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.103+0000 7f1745600640 1 -- 172.21.15.32:0/2156605776 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f174c049070 con 0x5558ceecf4d0 2024-04-30T08:37:35.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.103+0000 7f1745600640 1 -- 172.21.15.32:0/2156605776 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f174c031d80 con 0x5558ceecf4d0 2024-04-30T08:37:35.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.103+0000 7f176980a740 1 -- 172.21.15.32:0/2156605776 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1754004d20 con 0x5558ceecf4d0 2024-04-30T08:37:35.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.103+0000 7f1745600640 1 -- 172.21.15.32:0/2156605776 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f174c0cb9c0 con 0x5558ceecf4d0 2024-04-30T08:37:35.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.103+0000 7f176980a740 10 client.26284 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:35.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.103+0000 7f176980a740 1 -- 172.21.15.32:0/2156605776 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5558cf2a1770 con 0x5558ceecf4d0 2024-04-30T08:37:35.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.103+0000 7f1745600640 1 -- 172.21.15.32:0/2156605776 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1754004d20 con 0x5558ceecf4d0 2024-04-30T08:37:35.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.103+0000 7f1745600640 1 -- 172.21.15.32:0/2156605776 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7f174c0cffe0 con 0x5558ceecf4d0 2024-04-30T08:37:35.108 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.104+0000 7f176980a740 10 client.26284 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:35.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.104+0000 7f176980a740 10 client.26284 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-30T08:37:35.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.104+0000 7f176980a740 20 client.26284 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:35.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.104+0000 7f176980a740 1 --2- 172.21.15.32:0/2156605776 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x5558ceeb6f60 0x5558cf2b1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.104+0000 7f176980a740 4 client.26284 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:37:35.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.104+0000 7f176980a740 1 -- 172.21.15.32:0/2156605776 --> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5558cf20c960 con 0x5558ceeb6f60 2024-04-30T08:37:35.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.104+0000 7f175aa00640 1 --2- 172.21.15.32:0/2156605776 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x5558ceeb6f60 0x5558cf2b1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:35.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.105+0000 7f175aa00640 1 --2- 172.21.15.32:0/2156605776 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x5558ceeb6f60 0x5558cf2b1190 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a06c72ac5e09db76 server_cookie=f11b913c4d4dba71 in_seq=0 out_seq=0 2024-04-30T08:37:35.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.105+0000 7f1745600640 10 client.26284 ms_handle_connect on v2:172.21.15.189:6836/686520873 2024-04-30T08:37:35.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.105+0000 7f1745600640 1 -- 172.21.15.32:0/2156605776 <== mds.4 v2:172.21.15.189:6836/686520873 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5558cf20c960 con 0x5558ceeb6f60 2024-04-30T08:37:35.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.105+0000 7f1745600640 10 client.26284 handle_command_reply: tid=0 2024-04-30T08:37:35.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.105+0000 7f176980a740 1 client.26284 shutdown 2024-04-30T08:37:35.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.106+0000 7f176980a740 1 -- 172.21.15.32:0/2156605776 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558ceecf4d0 msgr2=0x5558cf2a3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:35.109 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.106+0000 7f176980a740 1 --2- 172.21.15.32:0/2156605776 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558ceecf4d0 0x5558cf2a3690 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f174c0c4880 tx=0x7f174c0c7120 comp rx=0 tx=0).stop 2024-04-30T08:37:35.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.106+0000 7f176980a740 1 -- 172.21.15.32:0/2156605776 shutdown_connections 2024-04-30T08:37:35.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.106+0000 7f176980a740 1 --2- 172.21.15.32:0/2156605776 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x5558ceeb6f60 0x5558cf2b1190 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.106+0000 7f176980a740 1 --2- 172.21.15.32:0/2156605776 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5558ceef8390 0x5558cf2aab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.106+0000 7f176980a740 1 --2- 172.21.15.32:0/2156605776 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5558ceed76b0 0x5558cf2a3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.106+0000 7f176980a740 1 --2- 172.21.15.32:0/2156605776 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558ceecf4d0 0x5558cf2a3690 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.106+0000 7f1745600640 0 client.26284 ms_handle_reset on v2:172.21.15.189:6836/686520873 2024-04-30T08:37:35.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.106+0000 7f176980a740 1 -- 172.21.15.32:0/2156605776 >> 172.21.15.32:0/2156605776 conn(0x5558ceeec4e0 msgr2=0x5558cf294100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:35.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.106+0000 7f176980a740 1 -- 172.21.15.32:0/2156605776 shutdown_connections 2024-04-30T08:37:35.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.106+0000 7f176980a740 1 -- 172.21.15.32:0/2156605776 wait complete. 2024-04-30T08:37:35.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.106+0000 7f176980a740 20 client.26284 trim_cache size 0 max 16384 2024-04-30T08:37:35.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.124+0000 7f176980a740 1 Processor -- start 2024-04-30T08:37:35.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.124+0000 7f176980a740 1 -- start start 2024-04-30T08:37:35.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.124+0000 7f176980a740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5558ceecf4d0 0x5558cf295230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.124+0000 7f176980a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5558ceed76b0 0x5558cf2aaae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.124+0000 7f176980a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558ceef8390 0x5558cf2936d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.124+0000 7f176980a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f175c1b2ee0 con 0x5558ceed76b0 2024-04-30T08:37:35.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.125+0000 7f176980a740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f175c1b4ab0 con 0x5558ceecf4d0 2024-04-30T08:37:35.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.125+0000 7f176980a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f175c1b35b0 con 0x5558ceef8390 2024-04-30T08:37:35.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.125+0000 7f175aa00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5558ceed76b0 0x5558cf2aaae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:35.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.125+0000 7f175b400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5558ceecf4d0 0x5558cf295230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:35.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.125+0000 7f175b400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5558ceecf4d0 0x5558cf295230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:51122/0 (socket says 172.21.15.32:51122) 2024-04-30T08:37:35.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.125+0000 7f175b400640 1 -- 172.21.15.32:0/16978256 learned_addr learned my addr 172.21.15.32:0/16978256 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:35.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.125+0000 7f175be00640 1 --2- 172.21.15.32:0/16978256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558ceef8390 0x5558cf2936d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:35.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.125+0000 7f175b400640 1 -- 172.21.15.32:0/16978256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558ceef8390 msgr2=0x5558cf2936d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:35.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.125+0000 7f175b400640 1 --2- 172.21.15.32:0/16978256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558ceef8390 0x5558cf2936d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.125+0000 7f175b400640 1 -- 172.21.15.32:0/16978256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5558ceed76b0 msgr2=0x5558cf2aaae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:35.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.125+0000 7f175b400640 1 --2- 172.21.15.32:0/16978256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5558ceed76b0 0x5558cf2aaae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.128 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.125+0000 7f175b400640 1 -- 172.21.15.32:0/16978256 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558cf29e4f0 con 0x5558ceecf4d0 2024-04-30T08:37:35.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.126+0000 7f175b400640 1 --2- 172.21.15.32:0/16978256 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5558ceecf4d0 0x5558cf295230 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f174c0c2f10 tx=0x7f174c03b890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94f9a5abcff34c13 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:35.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.126+0000 7f1747e00640 1 -- 172.21.15.32:0/16978256 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f174c049070 con 0x5558ceecf4d0 2024-04-30T08:37:35.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.126+0000 7f1747e00640 1 -- 172.21.15.32:0/16978256 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f174c031d80 con 0x5558ceecf4d0 2024-04-30T08:37:35.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.126+0000 7f1747e00640 1 -- 172.21.15.32:0/16978256 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f174c0cffe0 con 0x5558ceecf4d0 2024-04-30T08:37:35.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.127+0000 7f176980a740 1 -- 172.21.15.32:0/16978256 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5558ceecf4d0 msgr2=0x5558cf295230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:35.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.127+0000 7f176980a740 1 --2- 172.21.15.32:0/16978256 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5558ceecf4d0 0x5558cf295230 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f174c0c2f10 tx=0x7f174c03b890 comp rx=0 tx=0).stop 2024-04-30T08:37:35.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.128+0000 7f176980a740 1 -- 172.21.15.32:0/16978256 shutdown_connections 2024-04-30T08:37:35.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.128+0000 7f176980a740 1 --2- 172.21.15.32:0/16978256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558ceef8390 0x5558cf2936d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.128+0000 7f176980a740 1 --2- 172.21.15.32:0/16978256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5558ceed76b0 0x5558cf2aaae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.128+0000 7f176980a740 1 --2- 172.21.15.32:0/16978256 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5558ceecf4d0 0x5558cf295230 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.128+0000 7f176980a740 1 -- 172.21.15.32:0/16978256 >> 172.21.15.32:0/16978256 conn(0x5558ceeec4e0 msgr2=0x5558cf294b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:35.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.128+0000 7f176980a740 1 -- 172.21.15.32:0/16978256 shutdown_connections 2024-04-30T08:37:35.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.128+0000 7f176980a740 1 -- 172.21.15.32:0/16978256 wait complete. 2024-04-30T08:37:35.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.129+0000 7f176980a740 1 -- start start 2024-04-30T08:37:35.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.129+0000 7f176980a740 1 Processor -- start 2024-04-30T08:37:35.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.129+0000 7f176980a740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5558ceecf4d0 0x5558cf2a7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.129+0000 7f176980a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558ceed76b0 0x5558cf2a8270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.129+0000 7f176980a740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5558ceef8390 0x5558cf2a91a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.129+0000 7f176980a740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f175c1b4350 con 0x5558ceef8390 2024-04-30T08:37:35.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.129+0000 7f176980a740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f175c1b3c80 con 0x5558ceecf4d0 2024-04-30T08:37:35.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.130+0000 7f176980a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f175c1b0ea0 con 0x5558ceed76b0 2024-04-30T08:37:35.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.130+0000 7f176980a740 10 client.0 _ll_register_callbacks cb 0x7f174c005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:35.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.130+0000 7f175b400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5558ceecf4d0 0x5558cf2a7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:35.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.130+0000 7f175b400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5558ceecf4d0 0x5558cf2a7d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:51130/0 (socket says 172.21.15.32:51130) 2024-04-30T08:37:35.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.130+0000 7f175b400640 1 -- 172.21.15.32:0/185661277 learned_addr learned my addr 172.21.15.32:0/185661277 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:35.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.130+0000 7f175aa00640 1 --2- 172.21.15.32:0/185661277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558ceed76b0 0x5558cf2a8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:35.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.130+0000 7f175be00640 1 --2- 172.21.15.32:0/185661277 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5558ceef8390 0x5558cf2a91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:35.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.130+0000 7f175aa00640 1 -- 172.21.15.32:0/185661277 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5558ceecf4d0 msgr2=0x5558cf2a7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:35.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.130+0000 7f175aa00640 1 --2- 172.21.15.32:0/185661277 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5558ceecf4d0 0x5558cf2a7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.130+0000 7f175aa00640 1 -- 172.21.15.32:0/185661277 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5558ceef8390 msgr2=0x5558cf2a91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:35.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.130+0000 7f175aa00640 1 --2- 172.21.15.32:0/185661277 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5558ceef8390 0x5558cf2a91a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.130+0000 7f175aa00640 1 -- 172.21.15.32:0/185661277 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558cf2a9760 con 0x5558ceed76b0 2024-04-30T08:37:35.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.130+0000 7f175b400640 1 --2- 172.21.15.32:0/185661277 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5558ceecf4d0 0x5558cf2a7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:37:35.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.131+0000 7f175aa00640 1 --2- 172.21.15.32:0/185661277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558ceed76b0 0x5558cf2a8270 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x5558cf2b18a0 tx=0x7f1748010800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f79fa37e44f5ae7 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:35.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.131+0000 7f1745600640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-30T08:37:35.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.131+0000 7f1745600640 1 -- 172.21.15.32:0/185661277 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1748004460 con 0x5558ceed76b0 2024-04-30T08:37:35.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.131+0000 7f1745600640 1 -- 172.21.15.32:0/185661277 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f1748005020 con 0x5558ceed76b0 2024-04-30T08:37:35.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.131+0000 7f176980a740 1 -- 172.21.15.32:0/185661277 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f174c031d80 con 0x5558ceed76b0 2024-04-30T08:37:35.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.132+0000 7f1745600640 1 -- 172.21.15.32:0/185661277 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f174802d050 con 0x5558ceed76b0 2024-04-30T08:37:35.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.134+0000 7f176980a740 10 client.26290 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:35.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.134+0000 7f176980a740 1 -- 172.21.15.32:0/185661277 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5558cf2aaab0 con 0x5558ceed76b0 2024-04-30T08:37:35.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.134+0000 7f1745600640 1 -- 172.21.15.32:0/185661277 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f174c031d80 con 0x5558ceed76b0 2024-04-30T08:37:35.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.134+0000 7f1745600640 1 -- 172.21.15.32:0/185661277 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7f174801d380 con 0x5558ceed76b0 2024-04-30T08:37:35.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.135+0000 7f176980a740 10 client.26290 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:35.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.135+0000 7f176980a740 10 client.26290 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-30T08:37:35.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.135+0000 7f176980a740 20 client.26290 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:35.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.135+0000 7f176980a740 1 --2- 172.21.15.32:0/185661277 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x5558cee8c3a0 0x5558cf32c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.135+0000 7f176980a740 4 client.26290 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-30T08:37:35.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.135+0000 7f176980a740 1 -- 172.21.15.32:0/185661277 --> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5558cf2067d0 con 0x5558cee8c3a0 2024-04-30T08:37:35.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.136+0000 7f175b400640 1 --2- 172.21.15.32:0/185661277 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x5558cee8c3a0 0x5558cf32c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:35.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.137+0000 7f1745600640 10 client.26290 ms_handle_connect on v2:172.21.15.189:6836/686520873 2024-04-30T08:37:35.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.137+0000 7f175b400640 1 --2- 172.21.15.32:0/185661277 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x5558cee8c3a0 0x5558cf32c940 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=2edc7aaede83b6f4 server_cookie=f25320dd15dc6195 in_seq=0 out_seq=0 2024-04-30T08:37:35.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.137+0000 7f1745600640 1 -- 172.21.15.32:0/185661277 <== mds.4 v2:172.21.15.189:6836/686520873 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5558cf2067d0 con 0x5558cee8c3a0 2024-04-30T08:37:35.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.137+0000 7f1745600640 10 client.26290 handle_command_reply: tid=0 2024-04-30T08:37:35.138 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.137+0000 7f176980a740 1 client.26290 shutdown 2024-04-30T08:37:35.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.137+0000 7f176980a740 1 -- 172.21.15.32:0/185661277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558ceed76b0 msgr2=0x5558cf2a8270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:35.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.137+0000 7f176980a740 1 --2- 172.21.15.32:0/185661277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558ceed76b0 0x5558cf2a8270 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x5558cf2b18a0 tx=0x7f1748010800 comp rx=0 tx=0).stop 2024-04-30T08:37:35.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.137+0000 7f176980a740 1 -- 172.21.15.32:0/185661277 shutdown_connections 2024-04-30T08:37:35.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.137+0000 7f176980a740 1 --2- 172.21.15.32:0/185661277 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x5558cee8c3a0 0x5558cf32c940 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.137+0000 7f176980a740 1 --2- 172.21.15.32:0/185661277 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5558ceef8390 0x5558cf2a91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.137+0000 7f176980a740 1 --2- 172.21.15.32:0/185661277 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5558ceed76b0 0x5558cf2a8270 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.137+0000 7f176980a740 1 --2- 172.21.15.32:0/185661277 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5558ceecf4d0 0x5558cf2a7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.137+0000 7f176980a740 1 -- 172.21.15.32:0/185661277 >> 172.21.15.32:0/185661277 conn(0x5558ceeec4e0 msgr2=0x5558cf2936d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:35.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.137+0000 7f1745600640 0 client.26290 ms_handle_reset on v2:172.21.15.189:6836/686520873 2024-04-30T08:37:35.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.138+0000 7f176980a740 1 -- 172.21.15.32:0/185661277 shutdown_connections 2024-04-30T08:37:35.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.139+0000 7f176980a740 1 -- 172.21.15.32:0/185661277 wait complete. 2024-04-30T08:37:35.141 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.139+0000 7f176980a740 20 client.26290 trim_cache size 0 max 16384 2024-04-30T08:37:35.143 INFO:teuthology.orchestra.run.smithi032.stdout:[] 2024-04-30T08:37:35.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.143+0000 7f1761200640 1 -- 172.21.15.32:0/1144273417 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f17400afa10 msgr2=0x7f17400b1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:35.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.143+0000 7f1761200640 1 --2- 172.21.15.32:0/1144273417 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f17400afa10 0x7f17400b1f20 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f1754009dc0 tx=0x7f1754005fd0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.144+0000 7f1761200640 1 -- 172.21.15.32:0/1144273417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f175c102260 msgr2=0x7f175c1a06d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:35.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.144+0000 7f1761200640 1 --2- 172.21.15.32:0/1144273417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f175c102260 0x7f175c1a06d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f174c031470 tx=0x7f174c003c10 comp rx=0 tx=0).stop 2024-04-30T08:37:35.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.144+0000 7f1761200640 1 -- 172.21.15.32:0/1144273417 shutdown_connections 2024-04-30T08:37:35.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.144+0000 7f1761200640 1 --2- 172.21.15.32:0/1144273417 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f17400afa10 0x7f17400b1f20 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.146 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.144+0000 7f1761200640 1 --2- 172.21.15.32:0/1144273417 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f175c107520 0x7f175c1a7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.144+0000 7f1761200640 1 --2- 172.21.15.32:0/1144273417 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f175c104bc0 0x7f175c1a0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.144+0000 7f1761200640 1 --2- 172.21.15.32:0/1144273417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f175c102260 0x7f175c1a06d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.144+0000 7f1761200640 1 -- 172.21.15.32:0/1144273417 >> 172.21.15.32:0/1144273417 conn(0x7f175c0fda70 msgr2=0x7f175c1030a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:35.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.144+0000 7f1761200640 1 -- 172.21.15.32:0/1144273417 shutdown_connections 2024-04-30T08:37:35.147 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.145+0000 7f1761200640 1 -- 172.21.15.32:0/1144273417 wait complete. 2024-04-30T08:37:35.215 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-30T08:37:35.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:35 smithi032 ceph-mon[28331]: pgmap v497: 129 pgs: 129 active+clean; 2.6 GiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 201 MiB/s wr, 103 op/s 2024-04-30T08:37:35.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:35 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/847889596' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:37:35.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:35 smithi071 ceph-mon[33385]: pgmap v497: 129 pgs: 129 active+clean; 2.6 GiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 201 MiB/s wr, 103 op/s 2024-04-30T08:37:35.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:35 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/847889596' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:37:35.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:35 smithi189 ceph-mon[33038]: pgmap v497: 129 pgs: 129 active+clean; 2.6 GiB data, 744 MiB used, 1.0 TiB / 1.0 TiB avail; 97 KiB/s rd, 201 MiB/s wr, 103 op/s 2024-04-30T08:37:35.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:35 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/847889596' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:37:35.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.618+0000 7f94aea00640 1 -- 172.21.15.32:0/2652815833 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f94a8106dd0 msgr2=0x7f94a81091f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:35.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.618+0000 7f94aea00640 1 --2- 172.21.15.32:0/2652815833 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f94a8106dd0 0x7f94a81091f0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f949c009900 tx=0x7f949c02a440 comp rx=0 tx=0).stop 2024-04-30T08:37:35.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.618+0000 7f94aea00640 1 -- 172.21.15.32:0/2652815833 shutdown_connections 2024-04-30T08:37:35.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.618+0000 7f94aea00640 1 --2- 172.21.15.32:0/2652815833 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94a8109730 0x7f94a810bb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.619+0000 7f94aea00640 1 --2- 172.21.15.32:0/2652815833 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f94a8106dd0 0x7f94a81091f0 secure :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f949c009900 tx=0x7f949c02a440 comp rx=0 tx=0).stop 2024-04-30T08:37:35.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.619+0000 7f94aea00640 1 --2- 172.21.15.32:0/2652815833 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94a8104470 0x7f94a8106890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.619+0000 7f94aea00640 1 -- 172.21.15.32:0/2652815833 >> 172.21.15.32:0/2652815833 conn(0x7f94a80fda70 msgr2=0x7f94a80fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:35.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.620+0000 7f94aea00640 1 -- 172.21.15.32:0/2652815833 shutdown_connections 2024-04-30T08:37:35.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.621+0000 7f94aea00640 1 -- 172.21.15.32:0/2652815833 wait complete. 2024-04-30T08:37:35.627 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.622+0000 7f94aea00640 1 Processor -- start 2024-04-30T08:37:35.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.628+0000 7f94aea00640 1 -- start start 2024-04-30T08:37:35.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.629+0000 7f94aea00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f94a8104470 0x7f94a81132f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.632 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.629+0000 7f94aea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94a8109730 0x7f94a81138b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.629+0000 7f94aea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94a81187b0 0x7f94a8118bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.629+0000 7f94aea00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f94a810d350 con 0x7f94a8109730 2024-04-30T08:37:35.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.629+0000 7f94aea00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f94a810d1d0 con 0x7f94a8104470 2024-04-30T08:37:35.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.629+0000 7f94aea00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f94a810d4d0 con 0x7f94a81187b0 2024-04-30T08:37:35.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.629+0000 7f94ad600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f94a8104470 0x7f94a81132f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:35.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.629+0000 7f94ad600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f94a8104470 0x7f94a81132f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:51144/0 (socket says 172.21.15.32:51144) 2024-04-30T08:37:35.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.629+0000 7f94ad600640 1 -- 172.21.15.32:0/4254813212 learned_addr learned my addr 172.21.15.32:0/4254813212 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:35.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.630+0000 7f94ae000640 1 --2- 172.21.15.32:0/4254813212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94a81187b0 0x7f94a8118bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:35.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.630+0000 7f94ad600640 1 -- 172.21.15.32:0/4254813212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94a81187b0 msgr2=0x7f94a8118bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:35.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.630+0000 7f94ad600640 1 --2- 172.21.15.32:0/4254813212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94a81187b0 0x7f94a8118bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.630+0000 7f94ad600640 1 -- 172.21.15.32:0/4254813212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94a8109730 msgr2=0x7f94a81138b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:35.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.630+0000 7f94ad600640 1 --2- 172.21.15.32:0/4254813212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94a8109730 0x7f94a81138b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.633 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.630+0000 7f94ad600640 1 -- 172.21.15.32:0/4254813212 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94a81192f0 con 0x7f94a8104470 2024-04-30T08:37:35.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.631+0000 7f94ad600640 1 --2- 172.21.15.32:0/4254813212 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f94a8104470 0x7f94a81132f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f949400b470 tx=0x7f949400b930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:35.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.631+0000 7f94a6000640 1 -- 172.21.15.32:0/4254813212 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9494015020 con 0x7f94a8104470 2024-04-30T08:37:35.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.631+0000 7f94a6000640 1 -- 172.21.15.32:0/4254813212 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f9494011d40 con 0x7f94a8104470 2024-04-30T08:37:35.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.631+0000 7f94aea00640 1 -- 172.21.15.32:0/4254813212 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94a81b13b0 con 0x7f94a8104470 2024-04-30T08:37:35.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.631+0000 7f94a6000640 1 -- 172.21.15.32:0/4254813212 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f949400f340 con 0x7f94a8104470 2024-04-30T08:37:35.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.631+0000 7f94aea00640 1 -- 172.21.15.32:0/4254813212 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94a81b1940 con 0x7f94a8104470 2024-04-30T08:37:35.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.637+0000 7f94a6000640 1 -- 172.21.15.32:0/4254813212 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f9494018020 con 0x7f94a8104470 2024-04-30T08:37:35.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.643+0000 7f94a6000640 1 --2- 172.21.15.32:0/4254813212 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f94900afbb0 0x7f94900b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.643+0000 7f94a6000640 1 -- 172.21.15.32:0/4254813212 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f94940dea80 con 0x7f94a8104470 2024-04-30T08:37:35.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.643+0000 7f94acc00640 1 --2- 172.21.15.32:0/4254813212 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f94900afbb0 0x7f94900b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:35.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.644+0000 7f94acc00640 1 --2- 172.21.15.32:0/4254813212 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f94900afbb0 0x7f94900b20c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f949c031500 tx=0x7f949c031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:35.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.645+0000 7f94b6fd7740 1 Processor -- start 2024-04-30T08:37:35.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.647+0000 7f94b6fd7740 1 -- start start 2024-04-30T08:37:35.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.647+0000 7f94b6fd7740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560aa6d0c390 0x560aa70ad730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.647+0000 7f94b6fd7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560aa6ce34d0 0x560aa70adcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.647+0000 7f94b6fd7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560aa6ceb6b0 0x560aa70b4bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.647+0000 7f94b6fd7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560aa7012d50 con 0x560aa6ceb6b0 2024-04-30T08:37:35.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.647+0000 7f94b6fd7740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x560aa6fb80c0 con 0x560aa6d0c390 2024-04-30T08:37:35.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.647+0000 7f94b6fd7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560aa700ed10 con 0x560aa6ce34d0 2024-04-30T08:37:35.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.648+0000 7f94ae000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560aa6ceb6b0 0x560aa70b4bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:35.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.648+0000 7f94ae000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560aa6ceb6b0 0x560aa70b4bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48262/0 (socket says 172.21.15.32:48262) 2024-04-30T08:37:35.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.648+0000 7f94ae000640 1 -- 172.21.15.32:0/3144265980 learned_addr learned my addr 172.21.15.32:0/3144265980 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:35.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.648+0000 7f94acc00640 1 --2- 172.21.15.32:0/3144265980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560aa6ce34d0 0x560aa70adcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:35.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.648+0000 7f94ae000640 1 -- 172.21.15.32:0/3144265980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560aa6ce34d0 msgr2=0x560aa70adcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:35.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.648+0000 7f94ae000640 1 --2- 172.21.15.32:0/3144265980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560aa6ce34d0 0x560aa70adcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.648+0000 7f94ae000640 1 -- 172.21.15.32:0/3144265980 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560aa6d0c390 msgr2=0x560aa70ad730 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-30T08:37:35.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.648+0000 7f94ae000640 1 --2- 172.21.15.32:0/3144265980 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560aa6d0c390 0x560aa70ad730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.648+0000 7f94ae000640 1 -- 172.21.15.32:0/3144265980 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560aa6d243f0 con 0x560aa6ceb6b0 2024-04-30T08:37:35.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.649+0000 7f94ae000640 1 --2- 172.21.15.32:0/3144265980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560aa6ceb6b0 0x560aa70b4bb0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f94a0009ea0 tx=0x7f94a0009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f348e1369e53c8f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:35.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.649+0000 7f949a600640 1 -- 172.21.15.32:0/3144265980 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a0011640 con 0x560aa6ceb6b0 2024-04-30T08:37:35.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.649+0000 7f949a600640 1 -- 172.21.15.32:0/3144265980 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94a0011c90 con 0x560aa6ceb6b0 2024-04-30T08:37:35.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.650+0000 7f949a600640 1 -- 172.21.15.32:0/3144265980 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94a001bca0 con 0x560aa6ceb6b0 2024-04-30T08:37:35.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.651+0000 7f94b6fd7740 1 -- 172.21.15.32:0/3144265980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560aa6ceb6b0 msgr2=0x560aa70b4bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:35.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.651+0000 7f94b6fd7740 1 --2- 172.21.15.32:0/3144265980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560aa6ceb6b0 0x560aa70b4bb0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f94a0009ea0 tx=0x7f94a0009f90 comp rx=0 tx=0).stop 2024-04-30T08:37:35.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.651+0000 7f94b6fd7740 1 -- 172.21.15.32:0/3144265980 shutdown_connections 2024-04-30T08:37:35.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.651+0000 7f94b6fd7740 1 --2- 172.21.15.32:0/3144265980 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560aa6ceb6b0 0x560aa70b4bb0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.651+0000 7f94b6fd7740 1 --2- 172.21.15.32:0/3144265980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560aa6ce34d0 0x560aa70adcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.651+0000 7f94b6fd7740 1 --2- 172.21.15.32:0/3144265980 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560aa6d0c390 0x560aa70ad730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.651+0000 7f94b6fd7740 1 -- 172.21.15.32:0/3144265980 >> 172.21.15.32:0/3144265980 conn(0x560aa6d004e0 msgr2=0x560aa70a90c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:35.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.652+0000 7f94b6fd7740 1 -- 172.21.15.32:0/3144265980 shutdown_connections 2024-04-30T08:37:35.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.652+0000 7f94b6fd7740 1 -- 172.21.15.32:0/3144265980 wait complete. 2024-04-30T08:37:35.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.652+0000 7f94b6fd7740 1 -- start start 2024-04-30T08:37:35.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.653+0000 7f94b6fd7740 1 Processor -- start 2024-04-30T08:37:35.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.653+0000 7f94b6fd7740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560aa6ce34d0 0x560aa70b77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.653+0000 7f94b6fd7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560aa6ceb6b0 0x560aa70b7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.654+0000 7f94b6fd7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560aa6ccaf60 0x560aa70bec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.654+0000 7f94b6fd7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560aa70b5600 con 0x560aa6ccaf60 2024-04-30T08:37:35.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.654+0000 7f94b6fd7740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x560aa6fcbb50 con 0x560aa6ce34d0 2024-04-30T08:37:35.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.654+0000 7f94b6fd7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560aa70b5780 con 0x560aa6ceb6b0 2024-04-30T08:37:35.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.654+0000 7f94b6fd7740 10 client.0 _ll_register_callbacks cb 0x560aa6f95420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:35.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.654+0000 7f94ad600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560aa6ce34d0 0x560aa70b77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:35.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.654+0000 7f94ad600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560aa6ce34d0 0x560aa70b77c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:51164/0 (socket says 172.21.15.32:51164) 2024-04-30T08:37:35.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.654+0000 7f94ad600640 1 -- 172.21.15.32:0/754088824 learned_addr learned my addr 172.21.15.32:0/754088824 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:35.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.654+0000 7f94ae000640 1 --2- 172.21.15.32:0/754088824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560aa6ccaf60 0x560aa70bec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:35.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.654+0000 7f94ad600640 1 -- 172.21.15.32:0/754088824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560aa6ceb6b0 msgr2=0x560aa70b7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:35.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.654+0000 7f94ad600640 1 --2- 172.21.15.32:0/754088824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560aa6ceb6b0 0x560aa70b7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.655+0000 7f94ad600640 1 -- 172.21.15.32:0/754088824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560aa6ccaf60 msgr2=0x560aa70bec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:35.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.655+0000 7f94ad600640 1 --2- 172.21.15.32:0/754088824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560aa6ccaf60 0x560aa70bec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.655+0000 7f94ad600640 1 -- 172.21.15.32:0/754088824 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560aa6dee7d0 con 0x560aa6ce34d0 2024-04-30T08:37:35.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.655+0000 7f94ad600640 1 --2- 172.21.15.32:0/754088824 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560aa6ce34d0 0x560aa70b77c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f949400be30 tx=0x7f94940a7290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ecdd2405a17820c server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:35.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.655+0000 7f948ea00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:37:35.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.655+0000 7f948ea00640 1 -- 172.21.15.32:0/754088824 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94940aaf40 con 0x560aa6ce34d0 2024-04-30T08:37:35.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.655+0000 7f948ea00640 1 -- 172.21.15.32:0/754088824 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f9494011890 con 0x560aa6ce34d0 2024-04-30T08:37:35.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.655+0000 7f94b6fd7740 1 -- 172.21.15.32:0/754088824 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94a0011c90 con 0x560aa6ce34d0 2024-04-30T08:37:35.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.655+0000 7f948ea00640 1 -- 172.21.15.32:0/754088824 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f94940c6050 con 0x560aa6ce34d0 2024-04-30T08:37:35.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.656+0000 7f94b6fd7740 10 client.26303 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:35.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.656+0000 7f94b6fd7740 1 -- 172.21.15.32:0/754088824 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560aa70c1c00 con 0x560aa6ce34d0 2024-04-30T08:37:35.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.656+0000 7f948ea00640 1 -- 172.21.15.32:0/754088824 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94a0011c90 con 0x560aa6ce34d0 2024-04-30T08:37:35.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.657+0000 7f948ea00640 1 -- 172.21.15.32:0/754088824 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7f94940a26e0 con 0x560aa6ce34d0 2024-04-30T08:37:35.660 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.658+0000 7f94b6fd7740 10 client.26303 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:35.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.658+0000 7f94b6fd7740 10 client.26303 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-30T08:37:35.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.658+0000 7f94b6fd7740 20 client.26303 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:35.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.658+0000 7f94b6fd7740 1 --2- 172.21.15.32:0/754088824 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x560aa6ca03a0 0x560aa70c53b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.658+0000 7f94b6fd7740 4 client.26303 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:37:35.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.658+0000 7f94b6fd7740 1 -- 172.21.15.32:0/754088824 --> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560aa7020960 con 0x560aa6ca03a0 2024-04-30T08:37:35.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.659+0000 7f94acc00640 1 --2- 172.21.15.32:0/754088824 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x560aa6ca03a0 0x560aa70c53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:35.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.663+0000 7f948ea00640 10 client.26303 ms_handle_connect on v2:172.21.15.71:6836/2324039932 2024-04-30T08:37:35.664 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.663+0000 7f94acc00640 1 --2- 172.21.15.32:0/754088824 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x560aa6ca03a0 0x560aa70c53b0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f79ba58bb8b84e97 server_cookie=62dcc22e0b2f58f3 in_seq=0 out_seq=0 2024-04-30T08:37:35.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.664+0000 7f948ea00640 1 -- 172.21.15.32:0/754088824 <== mds.3 v2:172.21.15.71:6836/2324039932 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560aa7020960 con 0x560aa6ca03a0 2024-04-30T08:37:35.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.664+0000 7f948ea00640 10 client.26303 handle_command_reply: tid=0 2024-04-30T08:37:35.665 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.664+0000 7f94b6fd7740 1 client.26303 shutdown 2024-04-30T08:37:35.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.665+0000 7f94b6fd7740 1 -- 172.21.15.32:0/754088824 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560aa6ce34d0 msgr2=0x560aa70b77c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:35.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.665+0000 7f94b6fd7740 1 --2- 172.21.15.32:0/754088824 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560aa6ce34d0 0x560aa70b77c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f949400be30 tx=0x7f94940a7290 comp rx=0 tx=0).stop 2024-04-30T08:37:35.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.665+0000 7f94b6fd7740 1 -- 172.21.15.32:0/754088824 shutdown_connections 2024-04-30T08:37:35.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.665+0000 7f94b6fd7740 1 --2- 172.21.15.32:0/754088824 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x560aa6ca03a0 0x560aa70c53b0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.665+0000 7f948ea00640 0 client.26303 ms_handle_reset on v2:172.21.15.71:6836/2324039932 2024-04-30T08:37:35.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.665+0000 7f94b6fd7740 1 --2- 172.21.15.32:0/754088824 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560aa6ccaf60 0x560aa70bec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.665+0000 7f94b6fd7740 1 --2- 172.21.15.32:0/754088824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560aa6ceb6b0 0x560aa70b7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.665+0000 7f94b6fd7740 1 --2- 172.21.15.32:0/754088824 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560aa6ce34d0 0x560aa70b77c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.665+0000 7f94b6fd7740 1 -- 172.21.15.32:0/754088824 >> 172.21.15.32:0/754088824 conn(0x560aa6d004e0 msgr2=0x560aa70a8100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:35.668 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.666+0000 7f94b6fd7740 1 -- 172.21.15.32:0/754088824 shutdown_connections 2024-04-30T08:37:35.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.667+0000 7f94b6fd7740 1 -- 172.21.15.32:0/754088824 wait complete. 2024-04-30T08:37:35.669 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.667+0000 7f94b6fd7740 20 client.26303 trim_cache size 0 max 16384 2024-04-30T08:37:35.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.730+0000 7f94b6fd7740 1 Processor -- start 2024-04-30T08:37:35.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.730+0000 7f94b6fd7740 1 -- start start 2024-04-30T08:37:35.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.731+0000 7f94b6fd7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560aa6ca03a0 0x560aa70a8a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.731+0000 7f94b6fd7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560aa6ccaf60 0x560aa70a8f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.731+0000 7f94b6fd7740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560aa6ce34d0 0x560aa70b7390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.731+0000 7f94b6fd7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f94a81bbed0 con 0x560aa6ccaf60 2024-04-30T08:37:35.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.731+0000 7f94b6fd7740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f94a81bdaa0 con 0x560aa6ce34d0 2024-04-30T08:37:35.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.731+0000 7f94b6fd7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f94a81bc5a0 con 0x560aa6ca03a0 2024-04-30T08:37:35.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.732+0000 7f94ae000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560aa6ce34d0 0x560aa70b7390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:35.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.732+0000 7f94ae000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560aa6ce34d0 0x560aa70b7390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:51176/0 (socket says 172.21.15.32:51176) 2024-04-30T08:37:35.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.732+0000 7f94ae000640 1 -- 172.21.15.32:0/2719015996 learned_addr learned my addr 172.21.15.32:0/2719015996 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:35.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.732+0000 7f94ad600640 1 --2- 172.21.15.32:0/2719015996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560aa6ca03a0 0x560aa70a8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:35.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.733+0000 7f94ad600640 1 -- 172.21.15.32:0/2719015996 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560aa6ce34d0 msgr2=0x560aa70b7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:35.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.733+0000 7f94ad600640 1 --2- 172.21.15.32:0/2719015996 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560aa6ce34d0 0x560aa70b7390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.733+0000 7f94ad600640 1 -- 172.21.15.32:0/2719015996 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560aa6ccaf60 msgr2=0x560aa70a8f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-30T08:37:35.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.733+0000 7f94ad600640 1 --2- 172.21.15.32:0/2719015996 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560aa6ccaf60 0x560aa70a8f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.733+0000 7f94ad600640 1 -- 172.21.15.32:0/2719015996 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560aa70b8620 con 0x560aa6ca03a0 2024-04-30T08:37:35.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.734+0000 7f94ad600640 1 --2- 172.21.15.32:0/2719015996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560aa6ca03a0 0x560aa70a8a50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9494000c00 tx=0x7f94940a6b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8fc2879b99a22474 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:35.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.734+0000 7f9499c00640 1 -- 172.21.15.32:0/2719015996 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94940a2a50 con 0x560aa6ca03a0 2024-04-30T08:37:35.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.734+0000 7f9499c00640 1 -- 172.21.15.32:0/2719015996 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94940aaf40 con 0x560aa6ca03a0 2024-04-30T08:37:35.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.734+0000 7f9499c00640 1 -- 172.21.15.32:0/2719015996 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94940a5df0 con 0x560aa6ca03a0 2024-04-30T08:37:35.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.738+0000 7f94b6fd7740 1 -- 172.21.15.32:0/2719015996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560aa6ca03a0 msgr2=0x560aa70a8a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:35.742 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.738+0000 7f94b6fd7740 1 --2- 172.21.15.32:0/2719015996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560aa6ca03a0 0x560aa70a8a50 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f9494000c00 tx=0x7f94940a6b90 comp rx=0 tx=0).stop 2024-04-30T08:37:35.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.739+0000 7f94b6fd7740 1 -- 172.21.15.32:0/2719015996 shutdown_connections 2024-04-30T08:37:35.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.739+0000 7f94b6fd7740 1 --2- 172.21.15.32:0/2719015996 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560aa6ce34d0 0x560aa70b7390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.739+0000 7f94b6fd7740 1 --2- 172.21.15.32:0/2719015996 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560aa6ccaf60 0x560aa70a8f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.739+0000 7f94b6fd7740 1 --2- 172.21.15.32:0/2719015996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560aa6ca03a0 0x560aa70a8a50 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.739+0000 7f94b6fd7740 1 -- 172.21.15.32:0/2719015996 >> 172.21.15.32:0/2719015996 conn(0x560aa6d004e0 msgr2=0x560aa70b5e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:35.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.740+0000 7f94b6fd7740 1 -- 172.21.15.32:0/2719015996 shutdown_connections 2024-04-30T08:37:35.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.740+0000 7f94b6fd7740 1 -- 172.21.15.32:0/2719015996 wait complete. 2024-04-30T08:37:35.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.741+0000 7f94b6fd7740 1 -- start start 2024-04-30T08:37:35.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.741+0000 7f94b6fd7740 1 Processor -- start 2024-04-30T08:37:35.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.744+0000 7f94b6fd7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560aa6ca03a0 0x560aa70bbd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.745+0000 7f94b6fd7740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560aa6ccaf60 0x560aa70bc280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.745+0000 7f94b6fd7740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560aa6ce34d0 0x560aa70bd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.745+0000 7f94ad600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560aa6ca03a0 0x560aa70bbd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:35.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.745+0000 7f94b6fd7740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f94a81bd340 con 0x560aa6ce34d0 2024-04-30T08:37:35.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.745+0000 7f94b6fd7740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f94a81bcc70 con 0x560aa6ccaf60 2024-04-30T08:37:35.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.745+0000 7f94b6fd7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f94a81b9e90 con 0x560aa6ca03a0 2024-04-30T08:37:35.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.745+0000 7f94b6fd7740 10 client.0 _ll_register_callbacks cb 0x7f94940a2410 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:35.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.745+0000 7f94ad600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560aa6ca03a0 0x560aa70bbd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:41920/0 (socket says 172.21.15.32:41920) 2024-04-30T08:37:35.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.745+0000 7f94ad600640 1 -- 172.21.15.32:0/450276714 learned_addr learned my addr 172.21.15.32:0/450276714 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:35.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.746+0000 7f94ad600640 1 -- 172.21.15.32:0/450276714 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560aa6ccaf60 msgr2=0x560aa70bc280 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-30T08:37:35.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.746+0000 7f94ad600640 1 --2- 172.21.15.32:0/450276714 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560aa6ccaf60 0x560aa70bc280 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.746+0000 7f94ad600640 1 -- 172.21.15.32:0/450276714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560aa6ce34d0 msgr2=0x560aa70bd1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:35.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.746+0000 7f94ad600640 1 --2- 172.21.15.32:0/450276714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560aa6ce34d0 0x560aa70bd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.747+0000 7f94ad600640 1 -- 172.21.15.32:0/450276714 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560aa70b2670 con 0x560aa6ca03a0 2024-04-30T08:37:35.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.748+0000 7f94ad600640 1 --2- 172.21.15.32:0/450276714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560aa6ca03a0 0x560aa70bbd40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f94940a5880 tx=0x7f94940a7290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68ab9ab611c78e38 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:35.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.748+0000 7f948ea00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-30T08:37:35.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.749+0000 7f948ea00640 1 -- 172.21.15.32:0/450276714 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94940a2a50 con 0x560aa6ca03a0 2024-04-30T08:37:35.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.749+0000 7f948ea00640 1 -- 172.21.15.32:0/450276714 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f94940a3180 con 0x560aa6ca03a0 2024-04-30T08:37:35.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.749+0000 7f94b6fd7740 1 -- 172.21.15.32:0/450276714 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94940aaf40 con 0x560aa6ca03a0 2024-04-30T08:37:35.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.750+0000 7f948ea00640 1 -- 172.21.15.32:0/450276714 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f94940a5a90 con 0x560aa6ca03a0 2024-04-30T08:37:35.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.750+0000 7f94b6fd7740 10 client.26299 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:35.752 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.750+0000 7f94b6fd7740 1 -- 172.21.15.32:0/450276714 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560aa712d6f0 con 0x560aa6ca03a0 2024-04-30T08:37:35.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.750+0000 7f948ea00640 1 -- 172.21.15.32:0/450276714 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f94940aaf40 con 0x560aa6ca03a0 2024-04-30T08:37:35.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.750+0000 7f948ea00640 1 -- 172.21.15.32:0/450276714 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7f94940c9320 con 0x560aa6ca03a0 2024-04-30T08:37:35.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.751+0000 7f94b6fd7740 10 client.26299 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:35.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.751+0000 7f94b6fd7740 10 client.26299 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-30T08:37:35.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.751+0000 7f94b6fd7740 20 client.26299 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:35.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.751+0000 7f94b6fd7740 1 --2- 172.21.15.32:0/450276714 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x560aa6ceb6b0 0x560aa712fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:35.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.751+0000 7f94b6fd7740 4 client.26299 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-30T08:37:35.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.751+0000 7f94b6fd7740 1 -- 172.21.15.32:0/450276714 --> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560aa701a7d0 con 0x560aa6ceb6b0 2024-04-30T08:37:35.753 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.752+0000 7f94acc00640 1 --2- 172.21.15.32:0/450276714 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x560aa6ceb6b0 0x560aa712fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:35.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.755+0000 7f94acc00640 1 --2- 172.21.15.32:0/450276714 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x560aa6ceb6b0 0x560aa712fcd0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9b2972e60fc9532f server_cookie=4df758941bd5b40 in_seq=0 out_seq=0 2024-04-30T08:37:35.756 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.755+0000 7f948ea00640 10 client.26299 ms_handle_connect on v2:172.21.15.71:6836/2324039932 2024-04-30T08:37:35.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.755+0000 7f948ea00640 1 -- 172.21.15.32:0/450276714 <== mds.3 v2:172.21.15.71:6836/2324039932 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560aa701a7d0 con 0x560aa6ceb6b0 2024-04-30T08:37:35.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.755+0000 7f948ea00640 10 client.26299 handle_command_reply: tid=0 2024-04-30T08:37:35.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.755+0000 7f94b6fd7740 1 client.26299 shutdown 2024-04-30T08:37:35.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.761+0000 7f94b6fd7740 1 -- 172.21.15.32:0/450276714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560aa6ca03a0 msgr2=0x560aa70bbd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:35.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.761+0000 7f94b6fd7740 1 --2- 172.21.15.32:0/450276714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560aa6ca03a0 0x560aa70bbd40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f94940a5880 tx=0x7f94940a7290 comp rx=0 tx=0).stop 2024-04-30T08:37:35.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.761+0000 7f94b6fd7740 1 -- 172.21.15.32:0/450276714 shutdown_connections 2024-04-30T08:37:35.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.761+0000 7f94b6fd7740 1 --2- 172.21.15.32:0/450276714 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x560aa6ceb6b0 0x560aa712fcd0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.761+0000 7f94b6fd7740 1 --2- 172.21.15.32:0/450276714 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x560aa6ce34d0 0x560aa70bd1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.761+0000 7f94b6fd7740 1 --2- 172.21.15.32:0/450276714 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x560aa6ccaf60 0x560aa70bc280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.761+0000 7f94b6fd7740 1 --2- 172.21.15.32:0/450276714 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560aa6ca03a0 0x560aa70bbd40 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.761+0000 7f94b6fd7740 1 -- 172.21.15.32:0/450276714 >> 172.21.15.32:0/450276714 conn(0x560aa6d004e0 msgr2=0x560aa70a8c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:35.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.761+0000 7f948ea00640 0 client.26299 ms_handle_reset on v2:172.21.15.71:6836/2324039932 2024-04-30T08:37:35.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.762+0000 7f94b6fd7740 1 -- 172.21.15.32:0/450276714 shutdown_connections 2024-04-30T08:37:35.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.762+0000 7f94b6fd7740 1 -- 172.21.15.32:0/450276714 wait complete. 2024-04-30T08:37:35.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.762+0000 7f94b6fd7740 20 client.26299 trim_cache size 0 max 16384 2024-04-30T08:37:35.766 INFO:teuthology.orchestra.run.smithi032.stdout:[] 2024-04-30T08:37:35.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.765+0000 7f94aea00640 1 -- 172.21.15.32:0/4254813212 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f94900afbb0 msgr2=0x7f94900b20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:35.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.765+0000 7f94aea00640 1 --2- 172.21.15.32:0/4254813212 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f94900afbb0 0x7f94900b20c0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f949c031500 tx=0x7f949c031400 comp rx=0 tx=0).stop 2024-04-30T08:37:35.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.765+0000 7f94aea00640 1 -- 172.21.15.32:0/4254813212 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f94a8104470 msgr2=0x7f94a81132f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:35.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.765+0000 7f94aea00640 1 --2- 172.21.15.32:0/4254813212 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f94a8104470 0x7f94a81132f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f949400b470 tx=0x7f949400b930 comp rx=0 tx=0).stop 2024-04-30T08:37:35.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.766+0000 7f94aea00640 1 -- 172.21.15.32:0/4254813212 shutdown_connections 2024-04-30T08:37:35.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.766+0000 7f94aea00640 1 --2- 172.21.15.32:0/4254813212 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f94900afbb0 0x7f94900b20c0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.766+0000 7f94aea00640 1 --2- 172.21.15.32:0/4254813212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94a81187b0 0x7f94a8118bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.766+0000 7f94aea00640 1 --2- 172.21.15.32:0/4254813212 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f94a8109730 0x7f94a81138b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.766+0000 7f94aea00640 1 --2- 172.21.15.32:0/4254813212 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f94a8104470 0x7f94a81132f0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:35.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.766+0000 7f94aea00640 1 -- 172.21.15.32:0/4254813212 >> 172.21.15.32:0/4254813212 conn(0x7f94a80fda70 msgr2=0x7f94a8108e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:35.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.768+0000 7f94aea00640 1 -- 172.21.15.32:0/4254813212 shutdown_connections 2024-04-30T08:37:35.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:35.768+0000 7f94aea00640 1 -- 172.21.15.32:0/4254813212 wait complete. 2024-04-30T08:37:35.827 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-30T08:37:36.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.221+0000 7ff690600640 1 -- 172.21.15.32:0/542015257 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff68810b390 msgr2=0x7ff68810b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:36.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.221+0000 7ff690600640 1 --2- 172.21.15.32:0/542015257 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff68810b390 0x7ff68810b7a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7ff674009900 tx=0x7ff67402a3f0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.221+0000 7ff690600640 1 -- 172.21.15.32:0/542015257 shutdown_connections 2024-04-30T08:37:36.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.221+0000 7ff690600640 1 --2- 172.21.15.32:0/542015257 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6881100b0 0x7ff6881104c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.221+0000 7ff690600640 1 --2- 172.21.15.32:0/542015257 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff68810b390 0x7ff68810b7a0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.221+0000 7ff690600640 1 --2- 172.21.15.32:0/542015257 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff68810a4a0 0x7ff68810ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.221+0000 7ff690600640 1 -- 172.21.15.32:0/542015257 >> 172.21.15.32:0/542015257 conn(0x7ff688070560 msgr2=0x7ff688072a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:36.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.221+0000 7ff690600640 1 -- 172.21.15.32:0/542015257 shutdown_connections 2024-04-30T08:37:36.223 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.221+0000 7ff690600640 1 -- 172.21.15.32:0/542015257 wait complete. 2024-04-30T08:37:36.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.222+0000 7ff690600640 1 Processor -- start 2024-04-30T08:37:36.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.222+0000 7ff690600640 1 -- start start 2024-04-30T08:37:36.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.223+0000 7ff690600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff68810a4a0 0x7ff6881a9100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.223+0000 7ff690600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff68810b390 0x7ff6881a9640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.224 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.223+0000 7ff690600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6881100b0 0x7ff6881b05d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.223+0000 7ff690600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff68811e560 con 0x7ff6881100b0 2024-04-30T08:37:36.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.223+0000 7ff690600640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff68811e3e0 con 0x7ff68810b390 2024-04-30T08:37:36.225 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.223+0000 7ff690600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff68811e6e0 con 0x7ff68810a4a0 2024-04-30T08:37:36.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.223+0000 7ff68dc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff68810a4a0 0x7ff6881a9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:36.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.223+0000 7ff68dc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff68810a4a0 0x7ff6881a9100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:41944/0 (socket says 172.21.15.32:41944) 2024-04-30T08:37:36.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.223+0000 7ff68dc00640 1 -- 172.21.15.32:0/2409653027 learned_addr learned my addr 172.21.15.32:0/2409653027 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:36.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.223+0000 7ff68e600640 1 --2- 172.21.15.32:0/2409653027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6881100b0 0x7ff6881b05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:36.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.224+0000 7ff68d200640 1 --2- 172.21.15.32:0/2409653027 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff68810b390 0x7ff6881a9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:36.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.224+0000 7ff68e600640 1 -- 172.21.15.32:0/2409653027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff68810a4a0 msgr2=0x7ff6881a9100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:36.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.224+0000 7ff68e600640 1 --2- 172.21.15.32:0/2409653027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff68810a4a0 0x7ff6881a9100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.224+0000 7ff68e600640 1 -- 172.21.15.32:0/2409653027 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff68810b390 msgr2=0x7ff6881a9640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:36.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.224+0000 7ff68e600640 1 --2- 172.21.15.32:0/2409653027 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff68810b390 0x7ff6881a9640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.226 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.224+0000 7ff68e600640 1 -- 172.21.15.32:0/2409653027 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6881b0c40 con 0x7ff6881100b0 2024-04-30T08:37:36.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.224+0000 7ff68d200640 1 --2- 172.21.15.32:0/2409653027 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff68810b390 0x7ff6881a9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-30T08:37:36.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.224+0000 7ff68e600640 1 --2- 172.21.15.32:0/2409653027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6881100b0 0x7ff6881b05d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff680009fd0 tx=0x7ff68000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:36.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.225+0000 7ff686000640 1 -- 172.21.15.32:0/2409653027 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff68000ce20 con 0x7ff6881100b0 2024-04-30T08:37:36.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.225+0000 7ff690600640 1 -- 172.21.15.32:0/2409653027 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6881b0f30 con 0x7ff6881100b0 2024-04-30T08:37:36.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.225+0000 7ff686000640 1 -- 172.21.15.32:0/2409653027 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff680005020 con 0x7ff6881100b0 2024-04-30T08:37:36.227 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.225+0000 7ff690600640 1 -- 172.21.15.32:0/2409653027 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6881b1390 con 0x7ff6881100b0 2024-04-30T08:37:36.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.226+0000 7ff686000640 1 -- 172.21.15.32:0/2409653027 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff680004d20 con 0x7ff6881100b0 2024-04-30T08:37:36.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.230+0000 7ff69761d740 1 Processor -- start 2024-04-30T08:37:36.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.231+0000 7ff686000640 1 -- 172.21.15.32:0/2409653027 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7ff6800054d0 con 0x7ff6881100b0 2024-04-30T08:37:36.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.232+0000 7ff69761d740 1 -- start start 2024-04-30T08:37:36.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.233+0000 7ff69761d740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b52054390 0x557b523f5570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.233+0000 7ff69761d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b5202b4d0 0x557b523f5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.233+0000 7ff69761d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557b520336b0 0x557b523fc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.233+0000 7ff69761d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x557b5235ad50 con 0x557b520336b0 2024-04-30T08:37:36.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.233+0000 7ff69761d740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x557b523000c0 con 0x557b52054390 2024-04-30T08:37:36.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.233+0000 7ff69761d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x557b52356d10 con 0x557b5202b4d0 2024-04-30T08:37:36.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.234+0000 7ff68d200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b52054390 0x557b523f5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:36.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.234+0000 7ff68d200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b52054390 0x557b523f5570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:51206/0 (socket says 172.21.15.32:51206) 2024-04-30T08:37:36.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.234+0000 7ff68dc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b5202b4d0 0x557b523f5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:36.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.234+0000 7ff68dc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b5202b4d0 0x557b523f5af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:41946/0 (socket says 172.21.15.32:41946) 2024-04-30T08:37:36.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.234+0000 7ff68dc00640 1 -- 172.21.15.32:0/565738113 learned_addr learned my addr 172.21.15.32:0/565738113 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:36.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.234+0000 7ff68d200640 1 --2- 172.21.15.32:0/565738113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557b520336b0 0x557b523fc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:36.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.235+0000 7ff68dc00640 1 -- 172.21.15.32:0/565738113 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b52054390 msgr2=0x557b523f5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:36.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.235+0000 7ff68dc00640 1 --2- 172.21.15.32:0/565738113 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b52054390 0x557b523f5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.235+0000 7ff68dc00640 1 -- 172.21.15.32:0/565738113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557b520336b0 msgr2=0x557b523fc9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:36.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.235+0000 7ff68dc00640 1 --2- 172.21.15.32:0/565738113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557b520336b0 0x557b523fc9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.235+0000 7ff68dc00640 1 -- 172.21.15.32:0/565738113 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b5206c3f0 con 0x557b5202b4d0 2024-04-30T08:37:36.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.236+0000 7ff68dc00640 1 --2- 172.21.15.32:0/565738113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b5202b4d0 0x557b523f5af0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff67c009db0 tx=0x7ff67c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3ce832846cfa71e server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:36.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.236+0000 7ff67aa00640 1 -- 172.21.15.32:0/565738113 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff67c01e070 con 0x557b5202b4d0 2024-04-30T08:37:36.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.236+0000 7ff67aa00640 1 -- 172.21.15.32:0/565738113 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff67c019ae0 con 0x557b5202b4d0 2024-04-30T08:37:36.238 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.236+0000 7ff67aa00640 1 -- 172.21.15.32:0/565738113 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff67c00ea10 con 0x557b5202b4d0 2024-04-30T08:37:36.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.238+0000 7ff69761d740 1 -- 172.21.15.32:0/565738113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b5202b4d0 msgr2=0x557b523f5af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:36.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.238+0000 7ff69761d740 1 --2- 172.21.15.32:0/565738113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b5202b4d0 0x557b523f5af0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7ff67c009db0 tx=0x7ff67c011810 comp rx=0 tx=0).stop 2024-04-30T08:37:36.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.238+0000 7ff69761d740 1 -- 172.21.15.32:0/565738113 shutdown_connections 2024-04-30T08:37:36.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.238+0000 7ff69761d740 1 --2- 172.21.15.32:0/565738113 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557b520336b0 0x557b523fc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.238+0000 7ff69761d740 1 --2- 172.21.15.32:0/565738113 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b5202b4d0 0x557b523f5af0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.238+0000 7ff69761d740 1 --2- 172.21.15.32:0/565738113 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b52054390 0x557b523f5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.238+0000 7ff69761d740 1 -- 172.21.15.32:0/565738113 >> 172.21.15.32:0/565738113 conn(0x557b520484e0 msgr2=0x557b523f0f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:36.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.238+0000 7ff686000640 1 --2- 172.21.15.32:0/2409653027 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff6700afaf0 0x7ff6700b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.238+0000 7ff69761d740 1 -- 172.21.15.32:0/565738113 shutdown_connections 2024-04-30T08:37:36.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.239+0000 7ff69761d740 1 -- 172.21.15.32:0/565738113 wait complete. 2024-04-30T08:37:36.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.239+0000 7ff68dc00640 1 --2- 172.21.15.32:0/2409653027 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff6700afaf0 0x7ff6700b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:36.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.240+0000 7ff69761d740 1 -- start start 2024-04-30T08:37:36.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.240+0000 7ff686000640 1 -- 172.21.15.32:0/2409653027 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7ff6800df5a0 con 0x7ff6881100b0 2024-04-30T08:37:36.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.240+0000 7ff69761d740 1 Processor -- start 2024-04-30T08:37:36.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.240+0000 7ff68dc00640 1 --2- 172.21.15.32:0/2409653027 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff6700afaf0 0x7ff6700b2000 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff67c009b70 tx=0x7ff67c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:36.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.241+0000 7ff69761d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557b5202b4d0 0x557b52410480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.241+0000 7ff69761d740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b520336b0 0x557b524109c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.241+0000 7ff69761d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b52054390 0x557b52417920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.241+0000 7ff69761d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x557b5240e450 con 0x557b5202b4d0 2024-04-30T08:37:36.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.241+0000 7ff69761d740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x557b52313b50 con 0x557b520336b0 2024-04-30T08:37:36.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.241+0000 7ff69761d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x557b5240e5d0 con 0x557b52054390 2024-04-30T08:37:36.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.241+0000 7ff69761d740 10 client.0 _ll_register_callbacks cb 0x557b522dd420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:36.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.241+0000 7ff68dc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557b5202b4d0 0x557b52410480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:36.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.241+0000 7ff68dc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557b5202b4d0 0x557b52410480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:48318/0 (socket says 172.21.15.32:48318) 2024-04-30T08:37:36.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.241+0000 7ff68dc00640 1 -- 172.21.15.32:0/801759819 learned_addr learned my addr 172.21.15.32:0/801759819 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:36.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.241+0000 7ff68e600640 1 --2- 172.21.15.32:0/801759819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b52054390 0x557b52417920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:36.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.241+0000 7ff68d200640 1 --2- 172.21.15.32:0/801759819 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b520336b0 0x557b524109c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:36.244 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.242+0000 7ff68e600640 1 -- 172.21.15.32:0/801759819 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b520336b0 msgr2=0x557b524109c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:36.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.242+0000 7ff68e600640 1 --2- 172.21.15.32:0/801759819 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b520336b0 0x557b524109c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.242+0000 7ff68e600640 1 -- 172.21.15.32:0/801759819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557b5202b4d0 msgr2=0x557b52410480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:36.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.242+0000 7ff68e600640 1 --2- 172.21.15.32:0/801759819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557b5202b4d0 0x557b52410480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.242+0000 7ff68e600640 1 -- 172.21.15.32:0/801759819 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b521367d0 con 0x557b52054390 2024-04-30T08:37:36.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.242+0000 7ff68e600640 1 --2- 172.21.15.32:0/801759819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b52054390 0x557b52417920 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff6800a7640 tx=0x7ff6800a7cd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5758e744be0c8924 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:36.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.242+0000 7ff66f400640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-30T08:37:36.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.242+0000 7ff66f400640 1 -- 172.21.15.32:0/801759819 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6800051d0 con 0x557b52054390 2024-04-30T08:37:36.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.242+0000 7ff66f400640 1 -- 172.21.15.32:0/801759819 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7ff680090070 con 0x557b52054390 2024-04-30T08:37:36.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.242+0000 7ff69761d740 1 -- 172.21.15.32:0/801759819 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff67c019ae0 con 0x557b52054390 2024-04-30T08:37:36.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.243+0000 7ff66f400640 1 -- 172.21.15.32:0/801759819 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff6800a71c0 con 0x557b52054390 2024-04-30T08:37:36.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.243+0000 7ff69761d740 10 client.26311 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:36.245 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.243+0000 7ff69761d740 1 -- 172.21.15.32:0/801759819 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557b5241a970 con 0x557b52054390 2024-04-30T08:37:36.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.243+0000 7ff66f400640 1 -- 172.21.15.32:0/801759819 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff67c019ae0 con 0x557b52054390 2024-04-30T08:37:36.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.243+0000 7ff66f400640 1 -- 172.21.15.32:0/801759819 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7ff680020020 con 0x557b52054390 2024-04-30T08:37:36.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.243+0000 7ff69761d740 10 client.26311 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:36.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.243+0000 7ff69761d740 10 client.26311 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-30T08:37:36.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.243+0000 7ff69761d740 20 client.26311 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:36.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.243+0000 7ff69761d740 1 --2- 172.21.15.32:0/801759819 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x557b52012f60 0x557b5241e010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.243+0000 7ff69761d740 4 client.26311 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:37:36.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.243+0000 7ff69761d740 1 -- 172.21.15.32:0/801759819 --> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557b52368960 con 0x557b52012f60 2024-04-30T08:37:36.247 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.243+0000 7ff68dc00640 1 --2- 172.21.15.32:0/801759819 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x557b52012f60 0x557b5241e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:36.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.244+0000 7ff68dc00640 1 --2- 172.21.15.32:0/801759819 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x557b52012f60 0x557b5241e010 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=d6cf557db0b19d41 server_cookie=42773087b44aa871 in_seq=0 out_seq=0 2024-04-30T08:37:36.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.244+0000 7ff66f400640 10 client.26311 ms_handle_connect on v2:172.21.15.189:6838/352970107 2024-04-30T08:37:36.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.245+0000 7ff66f400640 1 -- 172.21.15.32:0/801759819 <== mds.2 v2:172.21.15.189:6838/352970107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x557b52368960 con 0x557b52012f60 2024-04-30T08:37:36.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.245+0000 7ff66f400640 10 client.26311 handle_command_reply: tid=0 2024-04-30T08:37:36.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.245+0000 7ff69761d740 1 client.26311 shutdown 2024-04-30T08:37:36.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.245+0000 7ff69761d740 1 -- 172.21.15.32:0/801759819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b52054390 msgr2=0x557b52417920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:36.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.245+0000 7ff69761d740 1 --2- 172.21.15.32:0/801759819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b52054390 0x557b52417920 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff6800a7640 tx=0x7ff6800a7cd0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.245+0000 7ff69761d740 1 -- 172.21.15.32:0/801759819 shutdown_connections 2024-04-30T08:37:36.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.245+0000 7ff69761d740 1 --2- 172.21.15.32:0/801759819 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x557b52012f60 0x557b5241e010 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.245+0000 7ff69761d740 1 --2- 172.21.15.32:0/801759819 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b52054390 0x557b52417920 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.245+0000 7ff69761d740 1 --2- 172.21.15.32:0/801759819 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b520336b0 0x557b524109c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.245+0000 7ff69761d740 1 --2- 172.21.15.32:0/801759819 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557b5202b4d0 0x557b52410480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.245+0000 7ff69761d740 1 -- 172.21.15.32:0/801759819 >> 172.21.15.32:0/801759819 conn(0x557b520484e0 msgr2=0x557b523eff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:36.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.245+0000 7ff66f400640 0 client.26311 ms_handle_reset on v2:172.21.15.189:6838/352970107 2024-04-30T08:37:36.248 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.245+0000 7ff69761d740 1 -- 172.21.15.32:0/801759819 shutdown_connections 2024-04-30T08:37:36.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.245+0000 7ff69761d740 1 -- 172.21.15.32:0/801759819 wait complete. 2024-04-30T08:37:36.249 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.246+0000 7ff69761d740 20 client.26311 trim_cache size 0 max 16384 2024-04-30T08:37:36.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.277+0000 7ff69761d740 1 Processor -- start 2024-04-30T08:37:36.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.277+0000 7ff69761d740 1 -- start start 2024-04-30T08:37:36.278 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.277+0000 7ff69761d740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b5202b4d0 0x557b523f0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.277+0000 7ff69761d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b520336b0 0x557b52417840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.277+0000 7ff69761d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557b52054390 0x557b523f0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.277+0000 7ff69761d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff6881b3170 con 0x557b52054390 2024-04-30T08:37:36.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.277+0000 7ff69761d740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff6881b4d40 con 0x557b5202b4d0 2024-04-30T08:37:36.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.277+0000 7ff69761d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff6881b3840 con 0x557b520336b0 2024-04-30T08:37:36.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.278+0000 7ff68d200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b520336b0 0x557b52417840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:36.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.278+0000 7ff68dc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b5202b4d0 0x557b523f0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:36.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.278+0000 7ff68d200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b520336b0 0x557b52417840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:41962/0 (socket says 172.21.15.32:41962) 2024-04-30T08:37:36.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.278+0000 7ff68d200640 1 -- 172.21.15.32:0/1723298305 learned_addr learned my addr 172.21.15.32:0/1723298305 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:36.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.279+0000 7ff68d200640 1 -- 172.21.15.32:0/1723298305 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b5202b4d0 msgr2=0x557b523f0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:36.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.279+0000 7ff68d200640 1 --2- 172.21.15.32:0/1723298305 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b5202b4d0 0x557b523f0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.279+0000 7ff68d200640 1 -- 172.21.15.32:0/1723298305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557b52054390 msgr2=0x557b523f0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:36.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.279+0000 7ff68d200640 1 --2- 172.21.15.32:0/1723298305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557b52054390 0x557b523f0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.279+0000 7ff68d200640 1 -- 172.21.15.32:0/1723298305 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557b523fa310 con 0x557b520336b0 2024-04-30T08:37:36.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.280+0000 7ff68d200640 1 --2- 172.21.15.32:0/1723298305 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b520336b0 0x557b52417840 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff674009900 tx=0x7ff67402a3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c55221a190f6b813 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:36.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.281+0000 7ff67a000640 1 -- 172.21.15.32:0/1723298305 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff674035040 con 0x557b520336b0 2024-04-30T08:37:36.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.281+0000 7ff67a000640 1 -- 172.21.15.32:0/1723298305 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6740046c0 con 0x557b520336b0 2024-04-30T08:37:36.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.281+0000 7ff67a000640 1 -- 172.21.15.32:0/1723298305 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6740038e0 con 0x557b520336b0 2024-04-30T08:37:36.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.282+0000 7ff69761d740 1 -- 172.21.15.32:0/1723298305 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b520336b0 msgr2=0x557b52417840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:36.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.282+0000 7ff69761d740 1 --2- 172.21.15.32:0/1723298305 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b520336b0 0x557b52417840 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff674009900 tx=0x7ff67402a3f0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.282+0000 7ff69761d740 1 -- 172.21.15.32:0/1723298305 shutdown_connections 2024-04-30T08:37:36.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.282+0000 7ff69761d740 1 --2- 172.21.15.32:0/1723298305 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557b52054390 0x557b523f0dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.282+0000 7ff69761d740 1 --2- 172.21.15.32:0/1723298305 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b520336b0 0x557b52417840 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.282+0000 7ff69761d740 1 --2- 172.21.15.32:0/1723298305 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b5202b4d0 0x557b523f0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.284 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.282+0000 7ff69761d740 1 -- 172.21.15.32:0/1723298305 >> 172.21.15.32:0/1723298305 conn(0x557b520484e0 msgr2=0x557b5240eac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:36.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.283+0000 7ff69761d740 1 -- 172.21.15.32:0/1723298305 shutdown_connections 2024-04-30T08:37:36.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.283+0000 7ff69761d740 1 -- 172.21.15.32:0/1723298305 wait complete. 2024-04-30T08:37:36.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.283+0000 7ff69761d740 1 -- start start 2024-04-30T08:37:36.285 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.284+0000 7ff69761d740 1 Processor -- start 2024-04-30T08:37:36.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.284+0000 7ff69761d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b5202b4d0 0x557b52415860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.284+0000 7ff69761d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557b520336b0 0x557b52415da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.284+0000 7ff69761d740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b52054390 0x557b52416cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.284+0000 7ff69761d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff6881b45e0 con 0x557b520336b0 2024-04-30T08:37:36.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.284+0000 7ff69761d740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff6881b3f10 con 0x557b52054390 2024-04-30T08:37:36.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.284+0000 7ff69761d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff688119e90 con 0x557b5202b4d0 2024-04-30T08:37:36.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.285+0000 7ff69761d740 10 client.0 _ll_register_callbacks cb 0x7ff680004ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:36.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.285+0000 7ff68e600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b52054390 0x557b52416cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:36.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.285+0000 7ff68e600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b52054390 0x557b52416cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:51234/0 (socket says 172.21.15.32:51234) 2024-04-30T08:37:36.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.285+0000 7ff68e600640 1 -- 172.21.15.32:0/3026244985 learned_addr learned my addr 172.21.15.32:0/3026244985 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:36.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.285+0000 7ff68e600640 1 -- 172.21.15.32:0/3026244985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b5202b4d0 msgr2=0x557b52415860 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-30T08:37:36.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.285+0000 7ff68e600640 1 --2- 172.21.15.32:0/3026244985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b5202b4d0 0x557b52415860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.285+0000 7ff68e600640 1 -- 172.21.15.32:0/3026244985 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557b520336b0 msgr2=0x557b52415da0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-30T08:37:36.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.286+0000 7ff68d200640 1 --2- 172.21.15.32:0/3026244985 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557b520336b0 0x557b52415da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:36.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.286+0000 7ff68e600640 1 --2- 172.21.15.32:0/3026244985 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557b520336b0 0x557b52415da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.286+0000 7ff68e600640 1 -- 172.21.15.32:0/3026244985 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557b52417290 con 0x557b52054390 2024-04-30T08:37:36.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.286+0000 7ff68d200640 1 --2- 172.21.15.32:0/3026244985 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557b520336b0 0x557b52415da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-30T08:37:36.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.287+0000 7ff66f400640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:37:36.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.287+0000 7ff68e600640 1 --2- 172.21.15.32:0/3026244985 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b52054390 0x557b52416cd0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff6800a73b0 tx=0x7ff6800a32d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=257b4036677cdb5a server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:36.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.287+0000 7ff66f400640 1 -- 172.21.15.32:0/3026244985 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6800a7480 con 0x557b52054390 2024-04-30T08:37:36.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.287+0000 7ff66f400640 1 -- 172.21.15.32:0/3026244985 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7ff68008eb50 con 0x557b52054390 2024-04-30T08:37:36.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.288+0000 7ff69761d740 1 -- 172.21.15.32:0/3026244985 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6740046c0 con 0x557b52054390 2024-04-30T08:37:36.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.288+0000 7ff66f400640 1 -- 172.21.15.32:0/3026244985 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff6800051d0 con 0x557b52054390 2024-04-30T08:37:36.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.288+0000 7ff69761d740 10 client.26321 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:36.290 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.289+0000 7ff69761d740 1 -- 172.21.15.32:0/3026244985 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557b52486b50 con 0x557b52054390 2024-04-30T08:37:36.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.289+0000 7ff66f400640 1 -- 172.21.15.32:0/3026244985 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff6740046c0 con 0x557b52054390 2024-04-30T08:37:36.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.289+0000 7ff66f400640 1 -- 172.21.15.32:0/3026244985 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7ff680020070 con 0x557b52054390 2024-04-30T08:37:36.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.290+0000 7ff69761d740 10 client.26321 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:36.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.290+0000 7ff69761d740 10 client.26321 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-30T08:37:36.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.290+0000 7ff69761d740 20 client.26321 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:36.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.290+0000 7ff69761d740 1 --2- 172.21.15.32:0/3026244985 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x557b51fe83a0 0x557b52489400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.290+0000 7ff69761d740 4 client.26321 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-30T08:37:36.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.290+0000 7ff69761d740 1 -- 172.21.15.32:0/3026244985 --> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557b523627d0 con 0x557b51fe83a0 2024-04-30T08:37:36.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.290+0000 7ff68dc00640 1 --2- 172.21.15.32:0/3026244985 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x557b51fe83a0 0x557b52489400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:36.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.291+0000 7ff68dc00640 1 --2- 172.21.15.32:0/3026244985 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x557b51fe83a0 0x557b52489400 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=f66c6c50b6c9565 server_cookie=2027cb1c2870ae60 in_seq=0 out_seq=0 2024-04-30T08:37:36.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.291+0000 7ff66f400640 10 client.26321 ms_handle_connect on v2:172.21.15.189:6838/352970107 2024-04-30T08:37:36.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.291+0000 7ff66f400640 1 -- 172.21.15.32:0/3026244985 <== mds.2 v2:172.21.15.189:6838/352970107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557b523627d0 con 0x557b51fe83a0 2024-04-30T08:37:36.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.291+0000 7ff66f400640 10 client.26321 handle_command_reply: tid=0 2024-04-30T08:37:36.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.291+0000 7ff69761d740 1 client.26321 shutdown 2024-04-30T08:37:36.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.292+0000 7ff69761d740 1 -- 172.21.15.32:0/3026244985 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b52054390 msgr2=0x557b52416cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:36.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.292+0000 7ff69761d740 1 --2- 172.21.15.32:0/3026244985 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b52054390 0x557b52416cd0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7ff6800a73b0 tx=0x7ff6800a32d0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.292+0000 7ff69761d740 1 -- 172.21.15.32:0/3026244985 shutdown_connections 2024-04-30T08:37:36.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.292+0000 7ff69761d740 1 --2- 172.21.15.32:0/3026244985 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x557b51fe83a0 0x557b52489400 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.292+0000 7ff69761d740 1 --2- 172.21.15.32:0/3026244985 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557b52054390 0x557b52416cd0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.292+0000 7ff69761d740 1 --2- 172.21.15.32:0/3026244985 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557b520336b0 0x557b52415da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.292+0000 7ff69761d740 1 --2- 172.21.15.32:0/3026244985 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557b5202b4d0 0x557b52415860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.292+0000 7ff69761d740 1 -- 172.21.15.32:0/3026244985 >> 172.21.15.32:0/3026244985 conn(0x557b520484e0 msgr2=0x557b523f0890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:36.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.292+0000 7ff66f400640 0 client.26321 ms_handle_reset on v2:172.21.15.189:6838/352970107 2024-04-30T08:37:36.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.292+0000 7ff69761d740 1 -- 172.21.15.32:0/3026244985 shutdown_connections 2024-04-30T08:37:36.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.292+0000 7ff69761d740 1 -- 172.21.15.32:0/3026244985 wait complete. 2024-04-30T08:37:36.295 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.292+0000 7ff69761d740 20 client.26321 trim_cache size 0 max 16384 2024-04-30T08:37:36.295 INFO:teuthology.orchestra.run.smithi032.stdout:[] 2024-04-30T08:37:36.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.293+0000 7ff690600640 1 -- 172.21.15.32:0/2409653027 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff6700afaf0 msgr2=0x7ff6700b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:36.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.293+0000 7ff690600640 1 --2- 172.21.15.32:0/2409653027 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff6700afaf0 0x7ff6700b2000 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff67c009b70 tx=0x7ff67c000bc0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.293+0000 7ff690600640 1 -- 172.21.15.32:0/2409653027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6881100b0 msgr2=0x7ff6881b05d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:36.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.293+0000 7ff690600640 1 --2- 172.21.15.32:0/2409653027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6881100b0 0x7ff6881b05d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7ff680009fd0 tx=0x7ff68000ef10 comp rx=0 tx=0).stop 2024-04-30T08:37:36.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.294+0000 7ff690600640 1 -- 172.21.15.32:0/2409653027 shutdown_connections 2024-04-30T08:37:36.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.294+0000 7ff690600640 1 --2- 172.21.15.32:0/2409653027 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff6700afaf0 0x7ff6700b2000 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.294+0000 7ff690600640 1 --2- 172.21.15.32:0/2409653027 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff6881100b0 0x7ff6881b05d0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.294+0000 7ff690600640 1 --2- 172.21.15.32:0/2409653027 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff68810b390 0x7ff6881a9640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.294+0000 7ff690600640 1 --2- 172.21.15.32:0/2409653027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff68810a4a0 0x7ff6881a9100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.296 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.294+0000 7ff690600640 1 -- 172.21.15.32:0/2409653027 >> 172.21.15.32:0/2409653027 conn(0x7ff688070560 msgr2=0x7ff68810c090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:36.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.295+0000 7ff690600640 1 -- 172.21.15.32:0/2409653027 shutdown_connections 2024-04-30T08:37:36.297 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.295+0000 7ff690600640 1 -- 172.21.15.32:0/2409653027 wait complete. 2024-04-30T08:37:36.358 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-30T08:37:36.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.758+0000 7f3809e00640 1 -- 172.21.15.32:0/3268172652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3804075960 msgr2=0x7f380410fd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:36.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.758+0000 7f3809e00640 1 --2- 172.21.15.32:0/3268172652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3804075960 0x7f380410fd80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f37f0009860 tx=0x7f37f002a360 comp rx=0 tx=0).stop 2024-04-30T08:37:36.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.760+0000 7f3809e00640 1 -- 172.21.15.32:0/3268172652 shutdown_connections 2024-04-30T08:37:36.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.760+0000 7f3809e00640 1 --2- 172.21.15.32:0/3268172652 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f38041102c0 0x7f38041126e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.760+0000 7f3809e00640 1 --2- 172.21.15.32:0/3268172652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3804075960 0x7f380410fd80 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.761+0000 7f3809e00640 1 --2- 172.21.15.32:0/3268172652 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f3804075010 0x7f3804075420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.761+0000 7f3809e00640 1 -- 172.21.15.32:0/3268172652 >> 172.21.15.32:0/3268172652 conn(0x7f38040fed50 msgr2=0x7f3804101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:36.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.761+0000 7f3809e00640 1 -- 172.21.15.32:0/3268172652 shutdown_connections 2024-04-30T08:37:36.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.761+0000 7f3809e00640 1 -- 172.21.15.32:0/3268172652 wait complete. 2024-04-30T08:37:36.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.763+0000 7f3809e00640 1 Processor -- start 2024-04-30T08:37:36.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.763+0000 7f3809e00640 1 -- start start 2024-04-30T08:37:36.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.764+0000 7f3809e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3804075010 0x7f380410cae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.764+0000 7f3809e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3804075960 0x7f380410d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.764+0000 7f3809e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f38041102c0 0x7f380410df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.764+0000 7f3802000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3804075960 0x7f380410d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:36.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.764+0000 7f3802000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3804075960 0x7f380410d020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:41986/0 (socket says 172.21.15.32:41986) 2024-04-30T08:37:36.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.764+0000 7f3802000640 1 -- 172.21.15.32:0/712112208 learned_addr learned my addr 172.21.15.32:0/712112208 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:36.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.764+0000 7f3809e00640 1 -- 172.21.15.32:0/712112208 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f3804115d00 con 0x7f3804075010 2024-04-30T08:37:36.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.764+0000 7f3809e00640 1 -- 172.21.15.32:0/712112208 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f3804115b80 con 0x7f38041102c0 2024-04-30T08:37:36.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.764+0000 7f3809e00640 1 -- 172.21.15.32:0/712112208 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f3804115e80 con 0x7f3804075960 2024-04-30T08:37:36.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.764+0000 7f3803400640 1 --2- 172.21.15.32:0/712112208 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f38041102c0 0x7f380410df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:36.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.765+0000 7f3803400640 1 -- 172.21.15.32:0/712112208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3804075960 msgr2=0x7f380410d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:36.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.765+0000 7f3802a00640 1 --2- 172.21.15.32:0/712112208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3804075010 0x7f380410cae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:36.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.765+0000 7f3803400640 1 --2- 172.21.15.32:0/712112208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3804075960 0x7f380410d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.765+0000 7f3803400640 1 -- 172.21.15.32:0/712112208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3804075010 msgr2=0x7f380410cae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:36.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.765+0000 7f3803400640 1 --2- 172.21.15.32:0/712112208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3804075010 0x7f380410cae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.765+0000 7f3803400640 1 -- 172.21.15.32:0/712112208 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f380410e5f0 con 0x7f38041102c0 2024-04-30T08:37:36.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.766+0000 7f3803400640 1 --2- 172.21.15.32:0/712112208 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f38041102c0 0x7f380410df80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f37fc005fd0 tx=0x7f37fc00c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:36.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.766+0000 7f3802a00640 1 --2- 172.21.15.32:0/712112208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3804075010 0x7f380410cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-30T08:37:36.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.766+0000 7f3808a00640 1 -- 172.21.15.32:0/712112208 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37fc003970 con 0x7f38041102c0 2024-04-30T08:37:36.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.766+0000 7f3809e00640 1 -- 172.21.15.32:0/712112208 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f380410e8e0 con 0x7f38041102c0 2024-04-30T08:37:36.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.766+0000 7f3809e00640 1 -- 172.21.15.32:0/712112208 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f380410ee30 con 0x7f38041102c0 2024-04-30T08:37:36.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.766+0000 7f3808a00640 1 -- 172.21.15.32:0/712112208 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37fc004020 con 0x7f38041102c0 2024-04-30T08:37:36.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.766+0000 7f3808a00640 1 -- 172.21.15.32:0/712112208 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37fc00b900 con 0x7f38041102c0 2024-04-30T08:37:36.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.770+0000 7f3810e28740 1 Processor -- start 2024-04-30T08:37:36.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.770+0000 7f3808a00640 1 -- 172.21.15.32:0/712112208 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f37fc01d020 con 0x7f38041102c0 2024-04-30T08:37:36.772 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.771+0000 7f3810e28740 1 -- start start 2024-04-30T08:37:36.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.771+0000 7f3810e28740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612a3ddd390 0x5612a417e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.771+0000 7f3810e28740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a3db44d0 0x5612a417eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.771+0000 7f3810e28740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5612a3dbc6b0 0x5612a41859f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.771+0000 7f3810e28740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5612a40e3d50 con 0x5612a3ddd390 2024-04-30T08:37:36.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.771+0000 7f3810e28740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x5612a40890c0 con 0x5612a3dbc6b0 2024-04-30T08:37:36.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.771+0000 7f3810e28740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5612a40dfd10 con 0x5612a3db44d0 2024-04-30T08:37:36.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.772+0000 7f3802000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5612a3dbc6b0 0x5612a41859f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:36.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.772+0000 7f3802000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5612a3dbc6b0 0x5612a41859f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:51266/0 (socket says 172.21.15.32:51266) 2024-04-30T08:37:36.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.772+0000 7f3802000640 1 -- 172.21.15.32:0/1878965496 learned_addr learned my addr 172.21.15.32:0/1878965496 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:36.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.772+0000 7f3802000640 1 --2- 172.21.15.32:0/1878965496 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612a3ddd390 0x5612a417e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:36.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.772+0000 7f3802a00640 1 --2- 172.21.15.32:0/1878965496 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a3db44d0 0x5612a417eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:36.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.772+0000 7f3802000640 1 -- 172.21.15.32:0/1878965496 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a3db44d0 msgr2=0x5612a417eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:36.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.773+0000 7f3802000640 1 --2- 172.21.15.32:0/1878965496 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a3db44d0 0x5612a417eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.773+0000 7f3802000640 1 -- 172.21.15.32:0/1878965496 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612a3ddd390 msgr2=0x5612a417e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:36.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.773+0000 7f3802000640 1 --2- 172.21.15.32:0/1878965496 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612a3ddd390 0x5612a417e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.773+0000 7f3802000640 1 -- 172.21.15.32:0/1878965496 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612a3df53f0 con 0x5612a3dbc6b0 2024-04-30T08:37:36.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.773+0000 7f3802000640 1 --2- 172.21.15.32:0/1878965496 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5612a3dbc6b0 0x5612a41859f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f380410db80 tx=0x7f37f0009570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d59785dcc7503477 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:36.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.773+0000 7f3808a00640 1 --2- 172.21.15.32:0/712112208 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f37e80afb70 0x7f37e80b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.773+0000 7f37efe00640 1 -- 172.21.15.32:0/1878965496 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f0044020 con 0x5612a3dbc6b0 2024-04-30T08:37:36.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.773+0000 7f3808a00640 1 -- 172.21.15.32:0/712112208 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f37fc0def10 con 0x7f38041102c0 2024-04-30T08:37:36.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.773+0000 7f37efe00640 1 -- 172.21.15.32:0/1878965496 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37f0038040 con 0x5612a3dbc6b0 2024-04-30T08:37:36.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.773+0000 7f37efe00640 1 -- 172.21.15.32:0/1878965496 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f0037040 con 0x5612a3dbc6b0 2024-04-30T08:37:36.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.774+0000 7f3802a00640 1 --2- 172.21.15.32:0/1878965496 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a3db44d0 0x5612a417eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-30T08:37:36.776 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.774+0000 7f3810e28740 1 -- 172.21.15.32:0/1878965496 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5612a3dbc6b0 msgr2=0x5612a41859f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:36.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.774+0000 7f3810e28740 1 --2- 172.21.15.32:0/1878965496 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5612a3dbc6b0 0x5612a41859f0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f380410db80 tx=0x7f37f0009570 comp rx=0 tx=0).stop 2024-04-30T08:37:36.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.774+0000 7f3802a00640 1 --2- 172.21.15.32:0/712112208 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f37e80afb70 0x7f37e80b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:36.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.775+0000 7f3810e28740 1 -- 172.21.15.32:0/1878965496 shutdown_connections 2024-04-30T08:37:36.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.775+0000 7f3810e28740 1 --2- 172.21.15.32:0/1878965496 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5612a3dbc6b0 0x5612a41859f0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.775+0000 7f3810e28740 1 --2- 172.21.15.32:0/1878965496 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a3db44d0 0x5612a417eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.775+0000 7f3810e28740 1 --2- 172.21.15.32:0/1878965496 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612a3ddd390 0x5612a417e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.775+0000 7f3810e28740 1 -- 172.21.15.32:0/1878965496 >> 172.21.15.32:0/1878965496 conn(0x5612a3dd14e0 msgr2=0x5612a4179f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:36.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.775+0000 7f3802a00640 1 --2- 172.21.15.32:0/712112208 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f37e80afb70 0x7f37e80b2080 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f37f4006fd0 tx=0x7f37f4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:36.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.776+0000 7f3810e28740 1 -- 172.21.15.32:0/1878965496 shutdown_connections 2024-04-30T08:37:36.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.776+0000 7f3810e28740 1 -- 172.21.15.32:0/1878965496 wait complete. 2024-04-30T08:37:36.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.776+0000 7f3810e28740 1 -- start start 2024-04-30T08:37:36.778 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.776+0000 7f3810e28740 1 Processor -- start 2024-04-30T08:37:36.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.778+0000 7f3810e28740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a3db44d0 0x5612a4188490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.778+0000 7f3810e28740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5612a3dbc6b0 0x5612a41889d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.778+0000 7f3810e28740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612a3ddd390 0x5612a418f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.778+0000 7f3802a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a3db44d0 0x5612a4188490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:36.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.779+0000 7f3802000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5612a3dbc6b0 0x5612a41889d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:36.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.779+0000 7f3802a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a3db44d0 0x5612a4188490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:41990/0 (socket says 172.21.15.32:41990) 2024-04-30T08:37:36.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.779+0000 7f3802000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5612a3dbc6b0 0x5612a41889d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:51278/0 (socket says 172.21.15.32:51278) 2024-04-30T08:37:36.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.779+0000 7f3802a00640 1 -- 172.21.15.32:0/3574165676 learned_addr learned my addr 172.21.15.32:0/3574165676 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:36.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.779+0000 7f3810e28740 1 -- 172.21.15.32:0/3574165676 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5612a41863e0 con 0x5612a3ddd390 2024-04-30T08:37:36.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.779+0000 7f3810e28740 1 -- 172.21.15.32:0/3574165676 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x5612a409cb50 con 0x5612a3dbc6b0 2024-04-30T08:37:36.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.779+0000 7f3810e28740 1 -- 172.21.15.32:0/3574165676 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5612a4186560 con 0x5612a3db44d0 2024-04-30T08:37:36.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.779+0000 7f3810e28740 10 client.0 _ll_register_callbacks cb 0x5612a4066420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:36.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.779+0000 7f3802000640 1 -- 172.21.15.32:0/3574165676 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a3db44d0 msgr2=0x5612a4188490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:36.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.779+0000 7f3802000640 1 --2- 172.21.15.32:0/3574165676 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a3db44d0 0x5612a4188490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.779+0000 7f3802000640 1 -- 172.21.15.32:0/3574165676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612a3ddd390 msgr2=0x5612a418f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:36.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.779+0000 7f3802000640 1 --2- 172.21.15.32:0/3574165676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612a3ddd390 0x5612a418f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.779+0000 7f3802000640 1 -- 172.21.15.32:0/3574165676 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612a3ebf7d0 con 0x5612a3dbc6b0 2024-04-30T08:37:36.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.779+0000 7f3802a00640 1 --2- 172.21.15.32:0/3574165676 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a3db44d0 0x5612a4188490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:37:36.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.780+0000 7f3802000640 1 --2- 172.21.15.32:0/3574165676 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5612a3dbc6b0 0x5612a41889d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f37f0000c00 tx=0x7f37f0032670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b073a202ad56de68 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:36.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.780+0000 7f37ecc00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:37:36.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.780+0000 7f37ecc00640 1 -- 172.21.15.32:0/3574165676 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f0044030 con 0x5612a3dbc6b0 2024-04-30T08:37:36.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.780+0000 7f37ecc00640 1 -- 172.21.15.32:0/3574165676 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f37f0032c70 con 0x5612a3dbc6b0 2024-04-30T08:37:36.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.780+0000 7f3810e28740 1 -- 172.21.15.32:0/3574165676 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37f0038040 con 0x5612a3dbc6b0 2024-04-30T08:37:36.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.780+0000 7f37ecc00640 1 -- 172.21.15.32:0/3574165676 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f37f00564c0 con 0x5612a3dbc6b0 2024-04-30T08:37:36.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.780+0000 7f3810e28740 10 client.26330 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:36.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.780+0000 7f3810e28740 1 -- 172.21.15.32:0/3574165676 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5612a41927b0 con 0x5612a3dbc6b0 2024-04-30T08:37:36.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.781+0000 7f37ecc00640 1 -- 172.21.15.32:0/3574165676 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37f0038040 con 0x5612a3dbc6b0 2024-04-30T08:37:36.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.781+0000 7f37ecc00640 1 -- 172.21.15.32:0/3574165676 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7f37f0049c80 con 0x5612a3dbc6b0 2024-04-30T08:37:36.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.781+0000 7f3810e28740 10 client.26330 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:36.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.781+0000 7f3810e28740 10 client.26330 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-30T08:37:36.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.781+0000 7f3810e28740 20 client.26330 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:36.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.781+0000 7f3810e28740 1 --2- 172.21.15.32:0/3574165676 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x5612a3d9bf60 0x5612a4195b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.781+0000 7f3810e28740 4 client.26330 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:37:36.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.781+0000 7f3810e28740 1 -- 172.21.15.32:0/3574165676 --> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5612a40f1960 con 0x5612a3d9bf60 2024-04-30T08:37:36.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.783+0000 7f3802a00640 1 --2- 172.21.15.32:0/3574165676 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x5612a3d9bf60 0x5612a4195b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:36.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.785+0000 7f3802a00640 1 --2- 172.21.15.32:0/3574165676 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x5612a3d9bf60 0x5612a4195b10 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=75343bfb9bcdd05 server_cookie=d67b50df1a0ae9f4 in_seq=0 out_seq=0 2024-04-30T08:37:36.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.785+0000 7f37ecc00640 10 client.26330 ms_handle_connect on v2:172.21.15.32:6838/4160386524 2024-04-30T08:37:36.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.786+0000 7f37ecc00640 1 -- 172.21.15.32:0/3574165676 <== mds.1 v2:172.21.15.32:6838/4160386524 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5612a40f1960 con 0x5612a3d9bf60 2024-04-30T08:37:36.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.786+0000 7f37ecc00640 10 client.26330 handle_command_reply: tid=0 2024-04-30T08:37:36.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.786+0000 7f3810e28740 1 client.26330 shutdown 2024-04-30T08:37:36.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.787+0000 7f3810e28740 1 -- 172.21.15.32:0/3574165676 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5612a3dbc6b0 msgr2=0x5612a41889d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:36.788 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.787+0000 7f3810e28740 1 --2- 172.21.15.32:0/3574165676 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5612a3dbc6b0 0x5612a41889d0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f37f0000c00 tx=0x7f37f0032670 comp rx=0 tx=0).stop 2024-04-30T08:37:36.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.787+0000 7f3810e28740 1 -- 172.21.15.32:0/3574165676 shutdown_connections 2024-04-30T08:37:36.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.787+0000 7f3810e28740 1 --2- 172.21.15.32:0/3574165676 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x5612a3d9bf60 0x5612a4195b10 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.787+0000 7f3810e28740 1 --2- 172.21.15.32:0/3574165676 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612a3ddd390 0x5612a418f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.787+0000 7f3810e28740 1 --2- 172.21.15.32:0/3574165676 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5612a3dbc6b0 0x5612a41889d0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.787+0000 7f3810e28740 1 --2- 172.21.15.32:0/3574165676 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a3db44d0 0x5612a4188490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.787+0000 7f3810e28740 1 -- 172.21.15.32:0/3574165676 >> 172.21.15.32:0/3574165676 conn(0x5612a3dd14e0 msgr2=0x5612a4178f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:36.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.787+0000 7f37ecc00640 0 client.26330 ms_handle_reset on v2:172.21.15.32:6838/4160386524 2024-04-30T08:37:36.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.790+0000 7f3810e28740 1 -- 172.21.15.32:0/3574165676 shutdown_connections 2024-04-30T08:37:36.791 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.790+0000 7f3810e28740 1 -- 172.21.15.32:0/3574165676 wait complete. 2024-04-30T08:37:36.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.790+0000 7f3810e28740 20 client.26330 trim_cache size 0 max 16384 2024-04-30T08:37:36.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.809+0000 7f3810e28740 1 Processor -- start 2024-04-30T08:37:36.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.809+0000 7f3810e28740 1 -- start start 2024-04-30T08:37:36.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.810+0000 7f3810e28740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a3db44d0 0x5612a417a070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.810+0000 7f3810e28740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5612a3dbc6b0 0x5612a4186ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.810+0000 7f3810e28740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612a3ddd390 0x5612a4186ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.810+0000 7f3810e28740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f38041bb980 con 0x5612a3ddd390 2024-04-30T08:37:36.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.810+0000 7f3810e28740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f38041bd550 con 0x5612a3dbc6b0 2024-04-30T08:37:36.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.810+0000 7f3810e28740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f38041bc050 con 0x5612a3db44d0 2024-04-30T08:37:36.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.810+0000 7f3802a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a3db44d0 0x5612a417a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:36.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.810+0000 7f3802000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5612a3dbc6b0 0x5612a4186ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:36.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.810+0000 7f3802000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5612a3dbc6b0 0x5612a4186ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:51284/0 (socket says 172.21.15.32:51284) 2024-04-30T08:37:36.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.810+0000 7f3802000640 1 -- 172.21.15.32:0/3990730946 learned_addr learned my addr 172.21.15.32:0/3990730946 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:36.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.811+0000 7f3803400640 1 --2- 172.21.15.32:0/3990730946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612a3ddd390 0x5612a4186ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:36.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.811+0000 7f3802a00640 1 -- 172.21.15.32:0/3990730946 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5612a3dbc6b0 msgr2=0x5612a4186ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:36.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.811+0000 7f3802a00640 1 --2- 172.21.15.32:0/3990730946 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5612a3dbc6b0 0x5612a4186ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.811+0000 7f3802a00640 1 -- 172.21.15.32:0/3990730946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612a3ddd390 msgr2=0x5612a4186ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:36.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.811+0000 7f3802a00640 1 --2- 172.21.15.32:0/3990730946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612a3ddd390 0x5612a4186ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.811+0000 7f3802a00640 1 -- 172.21.15.32:0/3990730946 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612a4183360 con 0x5612a3db44d0 2024-04-30T08:37:36.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.812+0000 7f3802a00640 1 --2- 172.21.15.32:0/3990730946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a3db44d0 0x5612a417a070 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x5612a41960f0 tx=0x7f37f4010000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d1b80587ca0998ec server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:36.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.812+0000 7f37ef400640 1 -- 172.21.15.32:0/3990730946 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f402b030 con 0x5612a3db44d0 2024-04-30T08:37:36.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.812+0000 7f37ef400640 1 -- 172.21.15.32:0/3990730946 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37f401a040 con 0x5612a3db44d0 2024-04-30T08:37:36.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.812+0000 7f37ef400640 1 -- 172.21.15.32:0/3990730946 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37f4019040 con 0x5612a3db44d0 2024-04-30T08:37:36.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.814+0000 7f3810e28740 1 -- 172.21.15.32:0/3990730946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a3db44d0 msgr2=0x5612a417a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:36.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.814+0000 7f3810e28740 1 --2- 172.21.15.32:0/3990730946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a3db44d0 0x5612a417a070 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x5612a41960f0 tx=0x7f37f4010000 comp rx=0 tx=0).stop 2024-04-30T08:37:36.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.814+0000 7f3810e28740 1 -- 172.21.15.32:0/3990730946 shutdown_connections 2024-04-30T08:37:36.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.814+0000 7f3810e28740 1 --2- 172.21.15.32:0/3990730946 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612a3ddd390 0x5612a4186ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.814+0000 7f3810e28740 1 --2- 172.21.15.32:0/3990730946 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5612a3dbc6b0 0x5612a4186ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.815 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.814+0000 7f3810e28740 1 --2- 172.21.15.32:0/3990730946 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a3db44d0 0x5612a417a070 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.814+0000 7f3810e28740 1 -- 172.21.15.32:0/3990730946 >> 172.21.15.32:0/3990730946 conn(0x5612a3dd14e0 msgr2=0x5612a4179960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:36.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.814+0000 7f3810e28740 1 -- 172.21.15.32:0/3990730946 shutdown_connections 2024-04-30T08:37:36.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.815+0000 7f3810e28740 1 -- 172.21.15.32:0/3990730946 wait complete. 2024-04-30T08:37:36.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.815+0000 7f3810e28740 1 -- start start 2024-04-30T08:37:36.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.816+0000 7f3810e28740 1 Processor -- start 2024-04-30T08:37:36.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.816+0000 7f3810e28740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612a3db44d0 0x5612a418be30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.816+0000 7f3810e28740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5612a3dbc6b0 0x5612a418c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.816+0000 7f3810e28740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a3ddd390 0x5612a41fbd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.816+0000 7f3810e28740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f38041bcdf0 con 0x5612a3db44d0 2024-04-30T08:37:36.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.816+0000 7f3810e28740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f38041bc720 con 0x5612a3dbc6b0 2024-04-30T08:37:36.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.816+0000 7f3810e28740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f38041b9940 con 0x5612a3ddd390 2024-04-30T08:37:36.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.816+0000 7f3810e28740 10 client.0 _ll_register_callbacks cb 0x7f37f00096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:36.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.817+0000 7f3803400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a3ddd390 0x5612a41fbd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:36.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.817+0000 7f3803400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a3ddd390 0x5612a41fbd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:52270/0 (socket says 172.21.15.32:52270) 2024-04-30T08:37:36.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.817+0000 7f3803400640 1 -- 172.21.15.32:0/2006525414 learned_addr learned my addr 172.21.15.32:0/2006525414 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:36.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.817+0000 7f3802a00640 1 --2- 172.21.15.32:0/2006525414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612a3db44d0 0x5612a418be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:36.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.817+0000 7f3803400640 1 -- 172.21.15.32:0/2006525414 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5612a3dbc6b0 msgr2=0x5612a418c370 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-30T08:37:36.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.817+0000 7f3802000640 1 --2- 172.21.15.32:0/2006525414 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5612a3dbc6b0 0x5612a418c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:36.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.817+0000 7f3803400640 1 --2- 172.21.15.32:0/2006525414 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5612a3dbc6b0 0x5612a418c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.817+0000 7f3803400640 1 -- 172.21.15.32:0/2006525414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612a3db44d0 msgr2=0x5612a418be30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:36.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.817+0000 7f3803400640 1 --2- 172.21.15.32:0/2006525414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612a3db44d0 0x5612a418be30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.817+0000 7f3803400640 1 -- 172.21.15.32:0/2006525414 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5612a418c9e0 con 0x5612a3ddd390 2024-04-30T08:37:36.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.817+0000 7f3802a00640 1 --2- 172.21.15.32:0/2006525414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612a3db44d0 0x5612a418be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:37:36.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.817+0000 7f3803400640 1 --2- 172.21.15.32:0/2006525414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a3ddd390 0x5612a41fbd50 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f37fc0a6d20 tx=0x7f37fc0a6b30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d3c80109105b5153 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:36.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.817+0000 7f37ecc00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-30T08:37:36.820 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.818+0000 7f37ecc00640 1 -- 172.21.15.32:0/2006525414 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37fc0a7020 con 0x5612a3ddd390 2024-04-30T08:37:36.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.818+0000 7f37ecc00640 1 -- 172.21.15.32:0/2006525414 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f37fc0ab0d0 con 0x5612a3ddd390 2024-04-30T08:37:36.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.818+0000 7f3810e28740 1 -- 172.21.15.32:0/2006525414 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f37f401a040 con 0x5612a3ddd390 2024-04-30T08:37:36.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.818+0000 7f37ecc00640 1 -- 172.21.15.32:0/2006525414 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f37fc0ab3d0 con 0x5612a3ddd390 2024-04-30T08:37:36.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.818+0000 7f3810e28740 10 client.26329 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:36.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.818+0000 7f3810e28740 1 -- 172.21.15.32:0/2006525414 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5612a420e420 con 0x5612a3ddd390 2024-04-30T08:37:36.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.819+0000 7f37ecc00640 1 -- 172.21.15.32:0/2006525414 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f37f401a040 con 0x5612a3ddd390 2024-04-30T08:37:36.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.819+0000 7f37ecc00640 1 -- 172.21.15.32:0/2006525414 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7f37fc00b040 con 0x5612a3ddd390 2024-04-30T08:37:36.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.820+0000 7f3810e28740 10 client.26329 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:36.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.820+0000 7f3810e28740 10 client.26329 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-30T08:37:36.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.820+0000 7f3810e28740 20 client.26329 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:36.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.820+0000 7f3810e28740 1 --2- 172.21.15.32:0/2006525414 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x5612a3d713a0 0x5612a4210dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:36.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.820+0000 7f3810e28740 4 client.26329 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-30T08:37:36.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.820+0000 7f3810e28740 1 -- 172.21.15.32:0/2006525414 --> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5612a40eb7d0 con 0x5612a3d713a0 2024-04-30T08:37:36.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.820+0000 7f3802a00640 1 --2- 172.21.15.32:0/2006525414 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x5612a3d713a0 0x5612a4210dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:36.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.821+0000 7f3802a00640 1 --2- 172.21.15.32:0/2006525414 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x5612a3d713a0 0x5612a4210dc0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=def964f392c37fbd server_cookie=5ded03dc6f7e5a2e in_seq=0 out_seq=0 2024-04-30T08:37:36.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.821+0000 7f37ecc00640 10 client.26329 ms_handle_connect on v2:172.21.15.32:6838/4160386524 2024-04-30T08:37:36.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.821+0000 7f37ecc00640 1 -- 172.21.15.32:0/2006525414 <== mds.1 v2:172.21.15.32:6838/4160386524 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5612a40eb7d0 con 0x5612a3d713a0 2024-04-30T08:37:36.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.821+0000 7f37ecc00640 10 client.26329 handle_command_reply: tid=0 2024-04-30T08:37:36.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.822+0000 7f3810e28740 1 client.26329 shutdown 2024-04-30T08:37:36.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.822+0000 7f3810e28740 1 -- 172.21.15.32:0/2006525414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a3ddd390 msgr2=0x5612a41fbd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:36.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.822+0000 7f3810e28740 1 --2- 172.21.15.32:0/2006525414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a3ddd390 0x5612a41fbd50 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f37fc0a6d20 tx=0x7f37fc0a6b30 comp rx=0 tx=0).stop 2024-04-30T08:37:36.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.822+0000 7f3810e28740 1 -- 172.21.15.32:0/2006525414 shutdown_connections 2024-04-30T08:37:36.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.822+0000 7f3810e28740 1 --2- 172.21.15.32:0/2006525414 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x5612a3d713a0 0x5612a4210dc0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.822+0000 7f3810e28740 1 --2- 172.21.15.32:0/2006525414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5612a3ddd390 0x5612a41fbd50 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.822+0000 7f3810e28740 1 --2- 172.21.15.32:0/2006525414 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5612a3dbc6b0 0x5612a418c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.822+0000 7f3810e28740 1 --2- 172.21.15.32:0/2006525414 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5612a3db44d0 0x5612a418be30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.822+0000 7f3810e28740 1 -- 172.21.15.32:0/2006525414 >> 172.21.15.32:0/2006525414 conn(0x5612a3dd14e0 msgr2=0x5612a4186ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:36.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.822+0000 7f37ecc00640 0 client.26329 ms_handle_reset on v2:172.21.15.32:6838/4160386524 2024-04-30T08:37:36.825 INFO:teuthology.orchestra.run.smithi032.stdout:[] 2024-04-30T08:37:36.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.822+0000 7f3810e28740 1 -- 172.21.15.32:0/2006525414 shutdown_connections 2024-04-30T08:37:36.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.822+0000 7f3810e28740 1 -- 172.21.15.32:0/2006525414 wait complete. 2024-04-30T08:37:36.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.822+0000 7f3810e28740 20 client.26329 trim_cache size 0 max 16384 2024-04-30T08:37:36.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.824+0000 7f3809e00640 1 -- 172.21.15.32:0/712112208 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f37e80afb70 msgr2=0x7f37e80b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:36.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.824+0000 7f3809e00640 1 --2- 172.21.15.32:0/712112208 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f37e80afb70 0x7f37e80b2080 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f37f4006fd0 tx=0x7f37f4008040 comp rx=0 tx=0).stop 2024-04-30T08:37:36.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.824+0000 7f3809e00640 1 -- 172.21.15.32:0/712112208 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f38041102c0 msgr2=0x7f380410df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:36.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.824+0000 7f3809e00640 1 --2- 172.21.15.32:0/712112208 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f38041102c0 0x7f380410df80 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f37fc005fd0 tx=0x7f37fc00c800 comp rx=0 tx=0).stop 2024-04-30T08:37:36.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.824+0000 7f3809e00640 1 -- 172.21.15.32:0/712112208 shutdown_connections 2024-04-30T08:37:36.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.824+0000 7f3809e00640 1 --2- 172.21.15.32:0/712112208 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f37e80afb70 0x7f37e80b2080 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.824+0000 7f3809e00640 1 --2- 172.21.15.32:0/712112208 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f38041102c0 0x7f380410df80 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.824+0000 7f3809e00640 1 --2- 172.21.15.32:0/712112208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3804075960 0x7f380410d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.824+0000 7f3809e00640 1 --2- 172.21.15.32:0/712112208 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f3804075010 0x7f380410cae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:36.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.824+0000 7f3809e00640 1 -- 172.21.15.32:0/712112208 >> 172.21.15.32:0/712112208 conn(0x7f38040fed50 msgr2=0x7f3804103120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:36.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.824+0000 7f3809e00640 1 -- 172.21.15.32:0/712112208 shutdown_connections 2024-04-30T08:37:36.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:36.824+0000 7f3809e00640 1 -- 172.21.15.32:0/712112208 wait complete. 2024-04-30T08:37:36.897 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-30T08:37:37.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.285+0000 7ff46e600640 1 -- 172.21.15.32:0/1169973894 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff468103ca0 msgr2=0x7ff4681040b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:37.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.285+0000 7ff46e600640 1 --2- 172.21.15.32:0/1169973894 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff468103ca0 0x7ff4681040b0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff454009900 tx=0x7ff45402a410 comp rx=0 tx=0).stop 2024-04-30T08:37:37.287 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.285+0000 7ff46e600640 1 -- 172.21.15.32:0/1169973894 shutdown_connections 2024-04-30T08:37:37.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.285+0000 7ff46e600640 1 --2- 172.21.15.32:0/1169973894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4681089a0 0x7ff46810adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:37.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.285+0000 7ff46e600640 1 --2- 172.21.15.32:0/1169973894 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff468103ca0 0x7ff4681040b0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:37.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.285+0000 7ff46e600640 1 --2- 172.21.15.32:0/1169973894 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff468102a60 0x7ff468103760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:37.288 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.285+0000 7ff46e600640 1 -- 172.21.15.32:0/1169973894 >> 172.21.15.32:0/1169973894 conn(0x7ff4680fdc50 msgr2=0x7ff468100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:37.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.285+0000 7ff46e600640 1 -- 172.21.15.32:0/1169973894 shutdown_connections 2024-04-30T08:37:37.289 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.286+0000 7ff46e600640 1 -- 172.21.15.32:0/1169973894 wait complete. 2024-04-30T08:37:37.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.289+0000 7ff46e600640 1 Processor -- start 2024-04-30T08:37:37.291 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.289+0000 7ff46e600640 1 -- start start 2024-04-30T08:37:37.292 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.290+0000 7ff46e600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff468102a60 0x7ff468071e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:37.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.290+0000 7ff46e600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff468103ca0 0x7ff468072390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:37.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.290+0000 7ff46e600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4681089a0 0x7ff4680732f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:37.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.290+0000 7ff46e600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff46810d540 con 0x7ff468103ca0 2024-04-30T08:37:37.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.290+0000 7ff46e600640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff46810d3c0 con 0x7ff468102a60 2024-04-30T08:37:37.293 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.290+0000 7ff46e600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff46810d6c0 con 0x7ff4681089a0 2024-04-30T08:37:37.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.291+0000 7ff467400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff468102a60 0x7ff468071e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:37.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.291+0000 7ff467400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff468102a60 0x7ff468071e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:59360/0 (socket says 172.21.15.32:59360) 2024-04-30T08:37:37.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.291+0000 7ff467400640 1 -- 172.21.15.32:0/3344003872 learned_addr learned my addr 172.21.15.32:0/3344003872 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:37.294 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.291+0000 7ff466a00640 1 --2- 172.21.15.32:0/3344003872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff468103ca0 0x7ff468072390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:37.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.292+0000 7ff467e00640 1 --2- 172.21.15.32:0/3344003872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4681089a0 0x7ff4680732f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:37.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.292+0000 7ff466a00640 1 -- 172.21.15.32:0/3344003872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4681089a0 msgr2=0x7ff4680732f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:37.299 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.292+0000 7ff466a00640 1 --2- 172.21.15.32:0/3344003872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4681089a0 0x7ff4680732f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:37.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.292+0000 7ff466a00640 1 -- 172.21.15.32:0/3344003872 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff468102a60 msgr2=0x7ff468071e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:37.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.292+0000 7ff466a00640 1 --2- 172.21.15.32:0/3344003872 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff468102a60 0x7ff468071e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:37.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.292+0000 7ff466a00640 1 -- 172.21.15.32:0/3344003872 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff468073960 con 0x7ff468103ca0 2024-04-30T08:37:37.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.292+0000 7ff467400640 1 --2- 172.21.15.32:0/3344003872 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff468102a60 0x7ff468071e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:37:37.300 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.293+0000 7ff466a00640 1 --2- 172.21.15.32:0/3344003872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff468103ca0 0x7ff468072390 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff45402aa00 tx=0x7ff454003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:37.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.294+0000 7ff45fe00640 1 -- 172.21.15.32:0/3344003872 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff45403f070 con 0x7ff468103ca0 2024-04-30T08:37:37.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.294+0000 7ff45fe00640 1 -- 172.21.15.32:0/3344003872 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff454031c50 con 0x7ff468103ca0 2024-04-30T08:37:37.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.294+0000 7ff45fe00640 1 -- 172.21.15.32:0/3344003872 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff454035030 con 0x7ff468103ca0 2024-04-30T08:37:37.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.295+0000 7ff46e600640 1 -- 172.21.15.32:0/3344003872 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff468073bf0 con 0x7ff468103ca0 2024-04-30T08:37:37.301 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.296+0000 7ff46e600640 1 -- 172.21.15.32:0/3344003872 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4680740e0 con 0x7ff468103ca0 2024-04-30T08:37:37.306 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.304+0000 7ff45fe00640 1 -- 172.21.15.32:0/3344003872 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7ff454042070 con 0x7ff468103ca0 2024-04-30T08:37:37.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.305+0000 7ff475587740 1 Processor -- start 2024-04-30T08:37:37.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.305+0000 7ff475587740 1 -- start start 2024-04-30T08:37:37.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.306+0000 7ff475587740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c391e390 0x5567c3cbf570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:37.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.306+0000 7ff475587740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567c38f54d0 0x5567c3cbfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:37.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.306+0000 7ff475587740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567c38fd6b0 0x5567c3cc69f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:37.307 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.306+0000 7ff475587740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5567c3c24d50 con 0x5567c38f54d0 2024-04-30T08:37:37.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.306+0000 7ff475587740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x5567c3bca0c0 con 0x5567c38fd6b0 2024-04-30T08:37:37.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.306+0000 7ff475587740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5567c3c20d10 con 0x5567c391e390 2024-04-30T08:37:37.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.306+0000 7ff467e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c391e390 0x5567c3cbf570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:37.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.306+0000 7ff467e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c391e390 0x5567c3cbf570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:52288/0 (socket says 172.21.15.32:52288) 2024-04-30T08:37:37.308 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.306+0000 7ff467e00640 1 -- 172.21.15.32:0/1934505690 learned_addr learned my addr 172.21.15.32:0/1934505690 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:37.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.307+0000 7ff467400640 1 --2- 172.21.15.32:0/1934505690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567c38f54d0 0x5567c3cbfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:37.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.307+0000 7ff467e00640 1 -- 172.21.15.32:0/1934505690 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567c38fd6b0 msgr2=0x5567c3cc69f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-30T08:37:37.309 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.308+0000 7ff467e00640 1 --2- 172.21.15.32:0/1934505690 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567c38fd6b0 0x5567c3cc69f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:37.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.308+0000 7ff467e00640 1 -- 172.21.15.32:0/1934505690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567c38f54d0 msgr2=0x5567c3cbfaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:37.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.308+0000 7ff467e00640 1 --2- 172.21.15.32:0/1934505690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567c38f54d0 0x5567c3cbfaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:37.310 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.308+0000 7ff467e00640 1 -- 172.21.15.32:0/1934505690 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567c39363f0 con 0x5567c391e390 2024-04-30T08:37:37.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.309+0000 7ff467e00640 1 --2- 172.21.15.32:0/1934505690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c391e390 0x5567c3cbf570 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff460009fc0 tx=0x7ff460010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1a63e5c53f74d850 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:37.311 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.309+0000 7ff45fe00640 1 --2- 172.21.15.32:0/3344003872 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff44c0b40c0 0x7ff44c0b65d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:37.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.310+0000 7ff467400640 1 --2- 172.21.15.32:0/3344003872 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff44c0b40c0 0x7ff44c0b65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:37.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.310+0000 7ff45cc00640 1 -- 172.21.15.32:0/1934505690 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff460003b70 con 0x5567c391e390 2024-04-30T08:37:37.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.310+0000 7ff45fe00640 1 -- 172.21.15.32:0/3344003872 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7ff4540ffda0 con 0x7ff468103ca0 2024-04-30T08:37:37.312 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.310+0000 7ff45cc00640 1 -- 172.21.15.32:0/1934505690 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff460003d10 con 0x5567c391e390 2024-04-30T08:37:37.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.311+0000 7ff45cc00640 1 -- 172.21.15.32:0/1934505690 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff460011a40 con 0x5567c391e390 2024-04-30T08:37:37.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.311+0000 7ff475587740 1 -- 172.21.15.32:0/1934505690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c391e390 msgr2=0x5567c3cbf570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:37.313 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.311+0000 7ff475587740 1 --2- 172.21.15.32:0/1934505690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c391e390 0x5567c3cbf570 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff460009fc0 tx=0x7ff460010860 comp rx=0 tx=0).stop 2024-04-30T08:37:37.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.312+0000 7ff467400640 1 --2- 172.21.15.32:0/3344003872 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff44c0b40c0 0x7ff44c0b65d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x5567c3cc0950 tx=0x7ff458009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:37.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.312+0000 7ff475587740 1 -- 172.21.15.32:0/1934505690 shutdown_connections 2024-04-30T08:37:37.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.312+0000 7ff475587740 1 --2- 172.21.15.32:0/1934505690 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567c38fd6b0 0x5567c3cc69f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:37.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.312+0000 7ff475587740 1 --2- 172.21.15.32:0/1934505690 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567c38f54d0 0x5567c3cbfaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:37.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.312+0000 7ff475587740 1 --2- 172.21.15.32:0/1934505690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c391e390 0x5567c3cbf570 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:37.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.312+0000 7ff475587740 1 -- 172.21.15.32:0/1934505690 >> 172.21.15.32:0/1934505690 conn(0x5567c39124e0 msgr2=0x5567c3cbaf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:37.314 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.312+0000 7ff475587740 1 -- 172.21.15.32:0/1934505690 shutdown_connections 2024-04-30T08:37:37.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.313+0000 7ff475587740 1 -- 172.21.15.32:0/1934505690 wait complete. 2024-04-30T08:37:37.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.314+0000 7ff475587740 1 -- start start 2024-04-30T08:37:37.315 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.314+0000 7ff475587740 1 Processor -- start 2024-04-30T08:37:37.316 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.315+0000 7ff475587740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567c38f54d0 0x5567c3cd1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:37.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.315+0000 7ff475587740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c38fd6b0 0x5567c3cd2320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:37.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.315+0000 7ff475587740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567c391e390 0x5567c3cd9280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:37.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.315+0000 7ff475587740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5567c3cc7490 con 0x5567c38f54d0 2024-04-30T08:37:37.317 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.315+0000 7ff475587740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x5567c3bddb50 con 0x5567c391e390 2024-04-30T08:37:37.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.315+0000 7ff475587740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5567c3cc7610 con 0x5567c38fd6b0 2024-04-30T08:37:37.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.316+0000 7ff475587740 10 client.0 _ll_register_callbacks cb 0x5567c3ba7420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:37.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.316+0000 7ff467400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567c38f54d0 0x5567c3cd1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:37.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.316+0000 7ff467400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567c38f54d0 0x5567c3cd1de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34516/0 (socket says 172.21.15.32:34516) 2024-04-30T08:37:37.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.316+0000 7ff467400640 1 -- 172.21.15.32:0/2196486674 learned_addr learned my addr 172.21.15.32:0/2196486674 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:37.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.316+0000 7ff466a00640 1 --2- 172.21.15.32:0/2196486674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c38fd6b0 0x5567c3cd2320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:37.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.316+0000 7ff467400640 1 -- 172.21.15.32:0/2196486674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c38fd6b0 msgr2=0x5567c3cd2320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:37.318 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.316+0000 7ff467e00640 1 --2- 172.21.15.32:0/2196486674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567c391e390 0x5567c3cd9280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:37.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.316+0000 7ff467400640 1 --2- 172.21.15.32:0/2196486674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c38fd6b0 0x5567c3cd2320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:37.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.316+0000 7ff467400640 1 -- 172.21.15.32:0/2196486674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567c391e390 msgr2=0x5567c3cd9280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:37.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.316+0000 7ff467400640 1 --2- 172.21.15.32:0/2196486674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567c391e390 0x5567c3cd9280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:37.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.316+0000 7ff467400640 1 -- 172.21.15.32:0/2196486674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567c3a007d0 con 0x5567c38f54d0 2024-04-30T08:37:37.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.318+0000 7ff467e00640 1 --2- 172.21.15.32:0/2196486674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567c391e390 0x5567c3cd9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-30T08:37:37.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.319+0000 7ff451600640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:37:37.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.319+0000 7ff467400640 1 --2- 172.21.15.32:0/2196486674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567c38f54d0 0x5567c3cd1de0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff458009a50 tx=0x7ff4580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46b4ad22e4d5ce3 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:37.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.320+0000 7ff451600640 1 -- 172.21.15.32:0/2196486674 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff458026020 con 0x5567c38f54d0 2024-04-30T08:37:37.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.320+0000 7ff451600640 1 -- 172.21.15.32:0/2196486674 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7ff4580138a0 con 0x5567c38f54d0 2024-04-30T08:37:37.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.320+0000 7ff475587740 1 -- 172.21.15.32:0/2196486674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff460003d10 con 0x5567c38f54d0 2024-04-30T08:37:37.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.321+0000 7ff451600640 1 -- 172.21.15.32:0/2196486674 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff45801e040 con 0x5567c38f54d0 2024-04-30T08:37:37.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.321+0000 7ff475587740 10 client.16194 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:37.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.321+0000 7ff475587740 1 -- 172.21.15.32:0/2196486674 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5567c3cdc280 con 0x5567c38f54d0 2024-04-30T08:37:37.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.321+0000 7ff451600640 1 -- 172.21.15.32:0/2196486674 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff460003d10 con 0x5567c38f54d0 2024-04-30T08:37:37.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.322+0000 7ff451600640 1 -- 172.21.15.32:0/2196486674 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7ff458021030 con 0x5567c38f54d0 2024-04-30T08:37:37.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.322+0000 7ff475587740 10 client.16194 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:37.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.322+0000 7ff475587740 10 client.16194 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:37:37.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.322+0000 7ff475587740 20 client.16194 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:37.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.322+0000 7ff475587740 1 --2- 172.21.15.32:0/2196486674 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x5567c38dcf60 0x5567c3cdf9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:37.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.322+0000 7ff475587740 4 client.16194 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:37:37.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.322+0000 7ff475587740 1 -- 172.21.15.32:0/2196486674 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5567c3c32960 con 0x5567c38dcf60 2024-04-30T08:37:37.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.323+0000 7ff466a00640 1 --2- 172.21.15.32:0/2196486674 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x5567c38dcf60 0x5567c3cdf9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:37.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.323+0000 7ff466a00640 1 --2- 172.21.15.32:0/2196486674 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x5567c38dcf60 0x5567c3cdf9f0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=47c69c6ae1754827 server_cookie=d0374b5d7b900798 in_seq=0 out_seq=0 2024-04-30T08:37:37.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.323+0000 7ff451600640 10 client.16194 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:37.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.325+0000 7ff451600640 1 -- 172.21.15.32:0/2196486674 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5567c3c32960 con 0x5567c38dcf60 2024-04-30T08:37:37.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.325+0000 7ff451600640 10 client.16194 handle_command_reply: tid=0 2024-04-30T08:37:37.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.325+0000 7ff475587740 1 client.16194 shutdown 2024-04-30T08:37:37.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.325+0000 7ff475587740 1 -- 172.21.15.32:0/2196486674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567c38f54d0 msgr2=0x5567c3cd1de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:37.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.325+0000 7ff475587740 1 --2- 172.21.15.32:0/2196486674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567c38f54d0 0x5567c3cd1de0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7ff458009a50 tx=0x7ff4580046c0 comp rx=0 tx=0).stop 2024-04-30T08:37:37.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.325+0000 7ff475587740 1 -- 172.21.15.32:0/2196486674 shutdown_connections 2024-04-30T08:37:37.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.325+0000 7ff475587740 1 --2- 172.21.15.32:0/2196486674 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x5567c38dcf60 0x5567c3cdf9f0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:37.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.326+0000 7ff475587740 1 --2- 172.21.15.32:0/2196486674 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567c391e390 0x5567c3cd9280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:37.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.326+0000 7ff475587740 1 --2- 172.21.15.32:0/2196486674 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c38fd6b0 0x5567c3cd2320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:37.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.326+0000 7ff475587740 1 --2- 172.21.15.32:0/2196486674 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567c38f54d0 0x5567c3cd1de0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:37.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.326+0000 7ff475587740 1 -- 172.21.15.32:0/2196486674 >> 172.21.15.32:0/2196486674 conn(0x5567c39124e0 msgr2=0x5567c3cb9f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:37.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.326+0000 7ff451600640 0 client.16194 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:37.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.326+0000 7ff475587740 1 -- 172.21.15.32:0/2196486674 shutdown_connections 2024-04-30T08:37:37.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.326+0000 7ff475587740 1 -- 172.21.15.32:0/2196486674 wait complete. 2024-04-30T08:37:37.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.326+0000 7ff475587740 20 client.16194 trim_cache size 0 max 16384 2024-04-30T08:37:37.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.361+0000 7ff475587740 1 Processor -- start 2024-04-30T08:37:37.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.361+0000 7ff475587740 1 -- start start 2024-04-30T08:37:37.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.362+0000 7ff475587740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567c38f54d0 0x5567c3cd1410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:37.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.362+0000 7ff475587740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567c38fd6b0 0x5567c3cd9210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:37.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.362+0000 7ff475587740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c391e390 0x5567c3cbe4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:37.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.362+0000 7ff475587740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff4681b31f0 con 0x5567c38f54d0 2024-04-30T08:37:37.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.362+0000 7ff475587740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff4681b4dc0 con 0x5567c38fd6b0 2024-04-30T08:37:37.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.362+0000 7ff475587740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff4681b38c0 con 0x5567c391e390 2024-04-30T08:37:37.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.363+0000 7ff467400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567c38f54d0 0x5567c3cd1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:37.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.363+0000 7ff466a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567c38fd6b0 0x5567c3cd9210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:37.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.363+0000 7ff467400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567c38f54d0 0x5567c3cd1410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34518/0 (socket says 172.21.15.32:34518) 2024-04-30T08:37:37.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.363+0000 7ff467400640 1 -- 172.21.15.32:0/3111303711 learned_addr learned my addr 172.21.15.32:0/3111303711 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:37.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.364+0000 7ff466a00640 1 -- 172.21.15.32:0/3111303711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c391e390 msgr2=0x5567c3cbe4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:37.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.364+0000 7ff466a00640 1 --2- 172.21.15.32:0/3111303711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c391e390 0x5567c3cbe4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:37.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.364+0000 7ff466a00640 1 -- 172.21.15.32:0/3111303711 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567c38f54d0 msgr2=0x5567c3cd1410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:37.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.364+0000 7ff466a00640 1 --2- 172.21.15.32:0/3111303711 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567c38f54d0 0x5567c3cd1410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:37.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.364+0000 7ff466a00640 1 -- 172.21.15.32:0/3111303711 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5567c3cc43d0 con 0x5567c38fd6b0 2024-04-30T08:37:37.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.364+0000 7ff467400640 1 --2- 172.21.15.32:0/3111303711 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567c38f54d0 0x5567c3cd1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:37:37.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.365+0000 7ff466a00640 1 --2- 172.21.15.32:0/3111303711 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567c38fd6b0 0x5567c3cd9210 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7ff454000c00 tx=0x7ff454005e60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aec9e92858afdfb0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:37.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.366+0000 7ff453e00640 1 -- 172.21.15.32:0/3111303711 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4540e6020 con 0x5567c38fd6b0 2024-04-30T08:37:37.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.366+0000 7ff453e00640 1 -- 172.21.15.32:0/3111303711 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff4540cc220 con 0x5567c38fd6b0 2024-04-30T08:37:37.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.366+0000 7ff453e00640 1 -- 172.21.15.32:0/3111303711 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4540c80d0 con 0x5567c38fd6b0 2024-04-30T08:37:37.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.367+0000 7ff475587740 1 -- 172.21.15.32:0/3111303711 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567c38fd6b0 msgr2=0x5567c3cd9210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:37.368 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.367+0000 7ff475587740 1 --2- 172.21.15.32:0/3111303711 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567c38fd6b0 0x5567c3cd9210 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7ff454000c00 tx=0x7ff454005e60 comp rx=0 tx=0).stop 2024-04-30T08:37:37.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.368+0000 7ff475587740 1 -- 172.21.15.32:0/3111303711 shutdown_connections 2024-04-30T08:37:37.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.368+0000 7ff475587740 1 --2- 172.21.15.32:0/3111303711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c391e390 0x5567c3cbe4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:37.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.368+0000 7ff475587740 1 --2- 172.21.15.32:0/3111303711 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567c38fd6b0 0x5567c3cd9210 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:37.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.368+0000 7ff475587740 1 --2- 172.21.15.32:0/3111303711 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567c38f54d0 0x5567c3cd1410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:37.369 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.368+0000 7ff475587740 1 -- 172.21.15.32:0/3111303711 >> 172.21.15.32:0/3111303711 conn(0x5567c39124e0 msgr2=0x5567c3cbdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:37.370 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.368+0000 7ff475587740 1 -- 172.21.15.32:0/3111303711 shutdown_connections 2024-04-30T08:37:37.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.369+0000 7ff475587740 1 -- 172.21.15.32:0/3111303711 wait complete. 2024-04-30T08:37:37.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.369+0000 7ff475587740 1 -- start start 2024-04-30T08:37:37.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.370+0000 7ff475587740 1 Processor -- start 2024-04-30T08:37:37.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.371+0000 7ff475587740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c38f54d0 0x5567c3cd7c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:37.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.371+0000 7ff475587740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567c38fd6b0 0x5567c3cd8140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:37.372 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.371+0000 7ff475587740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567c391e390 0x5567c3cd8680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:37.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.371+0000 7ff467400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c38f54d0 0x5567c3cd7c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:37.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.371+0000 7ff467400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c38f54d0 0x5567c3cd7c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:52318/0 (socket says 172.21.15.32:52318) 2024-04-30T08:37:37.373 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.371+0000 7ff475587740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ff4681b4660 con 0x5567c38fd6b0 2024-04-30T08:37:37.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.371+0000 7ff475587740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ff4681b3f90 con 0x5567c391e390 2024-04-30T08:37:37.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.371+0000 7ff475587740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ff4681b11b0 con 0x5567c38f54d0 2024-04-30T08:37:37.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.371+0000 7ff466a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567c38fd6b0 0x5567c3cd8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:37.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.371+0000 7ff475587740 10 client.0 _ll_register_callbacks cb 0x7ff458009ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:37.374 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.372+0000 7ff467e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567c391e390 0x5567c3cd8680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:37.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.372+0000 7ff466a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567c38fd6b0 0x5567c3cd8140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34528/0 (socket says 172.21.15.32:34528) 2024-04-30T08:37:37.375 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.372+0000 7ff466a00640 1 -- 172.21.15.32:0/1567537610 learned_addr learned my addr 172.21.15.32:0/1567537610 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:37.378 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.372+0000 7ff467e00640 1 -- 172.21.15.32:0/1567537610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c38f54d0 msgr2=0x5567c3cd7c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:37.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.372+0000 7ff467e00640 1 --2- 172.21.15.32:0/1567537610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c38f54d0 0x5567c3cd7c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:37.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.372+0000 7ff467e00640 1 -- 172.21.15.32:0/1567537610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567c38fd6b0 msgr2=0x5567c3cd8140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:37.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.372+0000 7ff467e00640 1 --2- 172.21.15.32:0/1567537610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567c38fd6b0 0x5567c3cd8140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:37.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.372+0000 7ff467e00640 1 -- 172.21.15.32:0/1567537610 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5567c3d45e40 con 0x5567c391e390 2024-04-30T08:37:37.379 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.373+0000 7ff466a00640 1 --2- 172.21.15.32:0/1567537610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567c38fd6b0 0x5567c3cd8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:37:37.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.373+0000 7ff467e00640 1 --2- 172.21.15.32:0/1567537610 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567c391e390 0x5567c3cd8680 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x5567c3bac5b0 tx=0x7ff460009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c02162b71f2d8377 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:37.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.373+0000 7ff451600640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:37:37.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.374+0000 7ff451600640 1 -- 172.21.15.32:0/1567537610 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff460003e50 con 0x5567c391e390 2024-04-30T08:37:37.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.374+0000 7ff451600640 1 -- 172.21.15.32:0/1567537610 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7ff460011500 con 0x5567c391e390 2024-04-30T08:37:37.380 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.374+0000 7ff475587740 1 -- 172.21.15.32:0/1567537610 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4540cc220 con 0x5567c391e390 2024-04-30T08:37:37.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.374+0000 7ff451600640 1 -- 172.21.15.32:0/1567537610 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ff460011740 con 0x5567c391e390 2024-04-30T08:37:37.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.374+0000 7ff475587740 10 client.26345 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:37.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.374+0000 7ff475587740 1 -- 172.21.15.32:0/1567537610 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5567c3d470f0 con 0x5567c391e390 2024-04-30T08:37:37.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.375+0000 7ff451600640 1 -- 172.21.15.32:0/1567537610 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ff4540cc220 con 0x5567c391e390 2024-04-30T08:37:37.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.375+0000 7ff451600640 1 -- 172.21.15.32:0/1567537610 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7ff460005640 con 0x5567c391e390 2024-04-30T08:37:37.381 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.375+0000 7ff475587740 10 client.26345 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:37.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.375+0000 7ff475587740 10 client.26345 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:37:37.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.375+0000 7ff475587740 20 client.26345 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:37.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.375+0000 7ff475587740 1 --2- 172.21.15.32:0/1567537610 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x5567c38b23a0 0x5567c3d49940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:37.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.375+0000 7ff475587740 4 client.26345 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-30T08:37:37.382 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.375+0000 7ff475587740 1 -- 172.21.15.32:0/1567537610 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5567c3c2c7d0 con 0x5567c38b23a0 2024-04-30T08:37:37.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.376+0000 7ff467400640 1 --2- 172.21.15.32:0/1567537610 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x5567c38b23a0 0x5567c3d49940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:37.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.377+0000 7ff451600640 10 client.26345 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:37.383 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.377+0000 7ff467400640 1 --2- 172.21.15.32:0/1567537610 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x5567c38b23a0 0x5567c3d49940 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5aa1362bb1539048 server_cookie=5642d5402272c394 in_seq=0 out_seq=0 2024-04-30T08:37:37.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.377+0000 7ff451600640 1 -- 172.21.15.32:0/1567537610 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5567c3c2c7d0 con 0x5567c38b23a0 2024-04-30T08:37:37.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.378+0000 7ff451600640 10 client.26345 handle_command_reply: tid=0 2024-04-30T08:37:37.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.378+0000 7ff475587740 1 client.26345 shutdown 2024-04-30T08:37:37.385 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.378+0000 7ff475587740 1 -- 172.21.15.32:0/1567537610 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567c391e390 msgr2=0x5567c3cd8680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:37.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.378+0000 7ff475587740 1 --2- 172.21.15.32:0/1567537610 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567c391e390 0x5567c3cd8680 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x5567c3bac5b0 tx=0x7ff460009240 comp rx=0 tx=0).stop 2024-04-30T08:37:37.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.378+0000 7ff475587740 1 -- 172.21.15.32:0/1567537610 shutdown_connections 2024-04-30T08:37:37.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.378+0000 7ff475587740 1 --2- 172.21.15.32:0/1567537610 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x5567c38b23a0 0x5567c3d49940 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:37.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.378+0000 7ff475587740 1 --2- 172.21.15.32:0/1567537610 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5567c391e390 0x5567c3cd8680 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:37.386 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.378+0000 7ff475587740 1 --2- 172.21.15.32:0/1567537610 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5567c38fd6b0 0x5567c3cd8140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:37.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.378+0000 7ff475587740 1 --2- 172.21.15.32:0/1567537610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5567c38f54d0 0x5567c3cd7c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:37.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.378+0000 7ff451600640 0 client.26345 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:37.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.378+0000 7ff475587740 1 -- 172.21.15.32:0/1567537610 >> 172.21.15.32:0/1567537610 conn(0x5567c39124e0 msgr2=0x5567c3cbe7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:37.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.379+0000 7ff475587740 1 -- 172.21.15.32:0/1567537610 shutdown_connections 2024-04-30T08:37:37.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.379+0000 7ff475587740 1 -- 172.21.15.32:0/1567537610 wait complete. 2024-04-30T08:37:37.387 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.379+0000 7ff475587740 20 client.26345 trim_cache size 0 max 16384 2024-04-30T08:37:37.388 INFO:teuthology.orchestra.run.smithi032.stdout:[] 2024-04-30T08:37:37.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.380+0000 7ff46e600640 1 -- 172.21.15.32:0/3344003872 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff44c0b40c0 msgr2=0x7ff44c0b65d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:37.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.380+0000 7ff46e600640 1 --2- 172.21.15.32:0/3344003872 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff44c0b40c0 0x7ff44c0b65d0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x5567c3cc0950 tx=0x7ff458009190 comp rx=0 tx=0).stop 2024-04-30T08:37:37.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.380+0000 7ff46e600640 1 -- 172.21.15.32:0/3344003872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff468103ca0 msgr2=0x7ff468072390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:37.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.381+0000 7ff46e600640 1 --2- 172.21.15.32:0/3344003872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff468103ca0 0x7ff468072390 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff45402aa00 tx=0x7ff454003c10 comp rx=0 tx=0).stop 2024-04-30T08:37:37.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.381+0000 7ff46e600640 1 -- 172.21.15.32:0/3344003872 shutdown_connections 2024-04-30T08:37:37.390 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.381+0000 7ff46e600640 1 --2- 172.21.15.32:0/3344003872 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ff44c0b40c0 0x7ff44c0b65d0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:37.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.381+0000 7ff46e600640 1 --2- 172.21.15.32:0/3344003872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4681089a0 0x7ff4680732f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:37.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.381+0000 7ff46e600640 1 --2- 172.21.15.32:0/3344003872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ff468103ca0 0x7ff468072390 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:37.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.381+0000 7ff46e600640 1 --2- 172.21.15.32:0/3344003872 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ff468102a60 0x7ff468071e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:37.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.381+0000 7ff46e600640 1 -- 172.21.15.32:0/3344003872 >> 172.21.15.32:0/3344003872 conn(0x7ff4680fdc50 msgr2=0x7ff468104980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:37.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.382+0000 7ff46e600640 1 -- 172.21.15.32:0/3344003872 shutdown_connections 2024-04-30T08:37:37.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:37.382+0000 7ff46e600640 1 -- 172.21.15.32:0/3344003872 wait complete. 2024-04-30T08:37:37.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:37 smithi032 ceph-mon[28331]: pgmap v498: 129 pgs: 129 active+clean; 3.1 GiB data, 814 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 233 MiB/s wr, 100 op/s 2024-04-30T08:37:37.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:37 smithi071 ceph-mon[33385]: pgmap v498: 129 pgs: 129 active+clean; 3.1 GiB data, 814 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 233 MiB/s wr, 100 op/s 2024-04-30T08:37:37.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:37 smithi189 ceph-mon[33038]: pgmap v498: 129 pgs: 129 active+clean; 3.1 GiB data, 814 MiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 233 MiB/s wr, 100 op/s 2024-04-30T08:37:38.444 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-30T08:37:38.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-30T08:37:38.444 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-30T08:37:38.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.912+0000 7f74f2600640 1 -- 172.21.15.32:0/1128391304 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74ec106df0 msgr2=0x7f74ec107200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:38.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.912+0000 7f74f2600640 1 --2- 172.21.15.32:0/1128391304 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74ec106df0 0x7f74ec107200 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f74d8009900 tx=0x7f74d802a420 comp rx=0 tx=0).stop 2024-04-30T08:37:38.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.912+0000 7f74f2600640 1 -- 172.21.15.32:0/1128391304 shutdown_connections 2024-04-30T08:37:38.914 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.912+0000 7f74f2600640 1 --2- 172.21.15.32:0/1128391304 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74ec10baf0 0x7f74ec10df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:38.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.912+0000 7f74f2600640 1 --2- 172.21.15.32:0/1128391304 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74ec106df0 0x7f74ec107200 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:38.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.912+0000 7f74f2600640 1 --2- 172.21.15.32:0/1128391304 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74ec105bb0 0x7f74ec1068b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:38.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.912+0000 7f74f2600640 1 -- 172.21.15.32:0/1128391304 >> 172.21.15.32:0/1128391304 conn(0x7f74ec100da0 msgr2=0x7f74ec103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:38.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.912+0000 7f74f2600640 1 -- 172.21.15.32:0/1128391304 shutdown_connections 2024-04-30T08:37:38.915 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.913+0000 7f74f2600640 1 -- 172.21.15.32:0/1128391304 wait complete. 2024-04-30T08:37:38.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.915+0000 7f74f2600640 1 Processor -- start 2024-04-30T08:37:38.916 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.915+0000 7f74f2600640 1 -- start start 2024-04-30T08:37:38.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.916+0000 7f74f2600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74ec105bb0 0x7f74ec19c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:38.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.916+0000 7f74f2600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74ec106df0 0x7f74ec19c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:38.917 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.916+0000 7f74f2600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74ec10baf0 0x7f74ec1a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:38.918 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.916+0000 7f74f2600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f74ec111730 con 0x7f74ec106df0 2024-04-30T08:37:38.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.916+0000 7f74f2600640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f74ec1115b0 con 0x7f74ec105bb0 2024-04-30T08:37:38.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.916+0000 7f74f2600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f74ec1118b0 con 0x7f74ec10baf0 2024-04-30T08:37:38.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.916+0000 7f74f1200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74ec105bb0 0x7f74ec19c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:38.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.916+0000 7f74f1200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74ec105bb0 0x7f74ec19c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:59406/0 (socket says 172.21.15.32:59406) 2024-04-30T08:37:38.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.916+0000 7f74f1200640 1 -- 172.21.15.32:0/520805967 learned_addr learned my addr 172.21.15.32:0/520805967 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:38.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.916+0000 7f74ebe00640 1 --2- 172.21.15.32:0/520805967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74ec106df0 0x7f74ec19c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:38.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.917+0000 7f74f1c00640 1 --2- 172.21.15.32:0/520805967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74ec10baf0 0x7f74ec1a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:38.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.917+0000 7f74f1200640 1 -- 172.21.15.32:0/520805967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74ec10baf0 msgr2=0x7f74ec1a37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:38.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.917+0000 7f74f1200640 1 --2- 172.21.15.32:0/520805967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74ec10baf0 0x7f74ec1a37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:38.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.917+0000 7f74f1200640 1 -- 172.21.15.32:0/520805967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74ec106df0 msgr2=0x7f74ec19c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:38.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.917+0000 7f74f1200640 1 --2- 172.21.15.32:0/520805967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74ec106df0 0x7f74ec19c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:38.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.917+0000 7f74f1200640 1 -- 172.21.15.32:0/520805967 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74ec1a3e50 con 0x7f74ec105bb0 2024-04-30T08:37:38.919 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.917+0000 7f74f1c00640 1 --2- 172.21.15.32:0/520805967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74ec10baf0 0x7f74ec1a37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-30T08:37:38.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.917+0000 7f74ebe00640 1 --2- 172.21.15.32:0/520805967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74ec106df0 0x7f74ec19c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:37:38.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.918+0000 7f74f1200640 1 --2- 172.21.15.32:0/520805967 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74ec105bb0 0x7f74ec19c310 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f74e0006fd0 tx=0x7f74e000b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:38.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.918+0000 7f74e9600640 1 -- 172.21.15.32:0/520805967 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e0011830 con 0x7f74ec105bb0 2024-04-30T08:37:38.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.918+0000 7f74f2600640 1 -- 172.21.15.32:0/520805967 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74ec1a4140 con 0x7f74ec105bb0 2024-04-30T08:37:38.920 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.918+0000 7f74f2600640 1 -- 172.21.15.32:0/520805967 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74ec1a46e0 con 0x7f74ec105bb0 2024-04-30T08:37:38.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.918+0000 7f74e9600640 1 -- 172.21.15.32:0/520805967 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74e0016070 con 0x7f74ec105bb0 2024-04-30T08:37:38.921 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.918+0000 7f74e9600640 1 -- 172.21.15.32:0/520805967 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e0010e00 con 0x7f74ec105bb0 2024-04-30T08:37:38.926 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.925+0000 7f74fac02740 1 Processor -- start 2024-04-30T08:37:38.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.926+0000 7f74fac02740 1 -- start start 2024-04-30T08:37:38.927 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.926+0000 7f74e9600640 1 -- 172.21.15.32:0/520805967 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f74e001b480 con 0x7f74ec105bb0 2024-04-30T08:37:38.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.927+0000 7f74fac02740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56358df98410 0x56358e342be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:38.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.927+0000 7f74fac02740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56358df6f550 0x56358e343160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:38.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.927+0000 7f74fac02740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358df77730 0x56358e34a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:38.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.927+0000 7f74fac02740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56358e29eb50 con 0x56358df98410 2024-04-30T08:37:38.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.927+0000 7f74fac02740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x56358e244080 con 0x56358df6f550 2024-04-30T08:37:38.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.927+0000 7f74fac02740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56358e29ab10 con 0x56358df77730 2024-04-30T08:37:38.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.927+0000 7f74f1200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56358df6f550 0x56358e343160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:38.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.927+0000 7f74f1c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56358df98410 0x56358e342be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:38.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.927+0000 7f74f1c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56358df98410 0x56358e342be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34554/0 (socket says 172.21.15.32:34554) 2024-04-30T08:37:38.929 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.927+0000 7f74f1c00640 1 -- 172.21.15.32:0/401251872 learned_addr learned my addr 172.21.15.32:0/401251872 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:38.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.928+0000 7f74f1c00640 1 -- 172.21.15.32:0/401251872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358df77730 msgr2=0x56358e34a060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-30T08:37:38.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.928+0000 7f74f1c00640 1 --2- 172.21.15.32:0/401251872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358df77730 0x56358e34a060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:38.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.928+0000 7f74f1c00640 1 -- 172.21.15.32:0/401251872 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56358df6f550 msgr2=0x56358e343160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:38.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.928+0000 7f74f1c00640 1 --2- 172.21.15.32:0/401251872 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56358df6f550 0x56358e343160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:38.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.928+0000 7f74f1c00640 1 -- 172.21.15.32:0/401251872 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56358e2640c0 con 0x56358df98410 2024-04-30T08:37:38.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.928+0000 7f74f1200640 1 --2- 172.21.15.32:0/401251872 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56358df6f550 0x56358e343160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:37:38.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.928+0000 7f74e9600640 1 --2- 172.21.15.32:0/520805967 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f74d40afbe0 0x7f74d40b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:38.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.929+0000 7f74e9600640 1 -- 172.21.15.32:0/520805967 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f74e00de580 con 0x7f74ec105bb0 2024-04-30T08:37:38.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.929+0000 7f74f1200640 1 --2- 172.21.15.32:0/520805967 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f74d40afbe0 0x7f74d40b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:38.931 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.930+0000 7f74f1c00640 1 --2- 172.21.15.32:0/401251872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56358df98410 0x56358e342be0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f74e4009f90 tx=0x7f74e400e780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc3cac1bf59f241 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:38.932 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.930+0000 7f74f1200640 1 --2- 172.21.15.32:0/520805967 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f74d40afbe0 0x7f74d40b20f0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f74e00a7340 tx=0x7f74e00e1f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:38.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.931+0000 7f74de000640 1 -- 172.21.15.32:0/401251872 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e4016070 con 0x56358df98410 2024-04-30T08:37:38.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.933+0000 7f74de000640 1 -- 172.21.15.32:0/401251872 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74e4004d20 con 0x56358df98410 2024-04-30T08:37:38.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.933+0000 7f74de000640 1 -- 172.21.15.32:0/401251872 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e40117f0 con 0x56358df98410 2024-04-30T08:37:38.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.938+0000 7f74fac02740 1 -- 172.21.15.32:0/401251872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56358df98410 msgr2=0x56358e342be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:38.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.938+0000 7f74fac02740 1 --2- 172.21.15.32:0/401251872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56358df98410 0x56358e342be0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f74e4009f90 tx=0x7f74e400e780 comp rx=0 tx=0).stop 2024-04-30T08:37:38.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.939+0000 7f74fac02740 1 -- 172.21.15.32:0/401251872 shutdown_connections 2024-04-30T08:37:38.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.939+0000 7f74fac02740 1 --2- 172.21.15.32:0/401251872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358df77730 0x56358e34a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:38.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.939+0000 7f74fac02740 1 --2- 172.21.15.32:0/401251872 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56358df6f550 0x56358e343160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:38.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.939+0000 7f74fac02740 1 --2- 172.21.15.32:0/401251872 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56358df98410 0x56358e342be0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:38.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.939+0000 7f74fac02740 1 -- 172.21.15.32:0/401251872 >> 172.21.15.32:0/401251872 conn(0x56358df8c560 msgr2=0x56358e33e570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:38.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.940+0000 7f74fac02740 1 -- 172.21.15.32:0/401251872 shutdown_connections 2024-04-30T08:37:38.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.940+0000 7f74fac02740 1 -- 172.21.15.32:0/401251872 wait complete. 2024-04-30T08:37:38.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.945+0000 7f74fac02740 1 -- start start 2024-04-30T08:37:38.949 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.948+0000 7f74fac02740 1 Processor -- start 2024-04-30T08:37:38.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.955+0000 7f74fac02740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56358df6f550 0x56358e34c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:38.963 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.957+0000 7f74fac02740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358df77730 0x56358e34ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:38.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.960+0000 7f74ebe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358df77730 0x56358e34ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:38.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.960+0000 7f74ebe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358df77730 0x56358e34ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:52346/0 (socket says 172.21.15.32:52346) 2024-04-30T08:37:38.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.963+0000 7f74fac02740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56358df98410 0x56358e353c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:38.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.963+0000 7f74fac02740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56358e34aab0 con 0x56358df98410 2024-04-30T08:37:38.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.963+0000 7f74fac02740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x56358e257b20 con 0x56358df6f550 2024-04-30T08:37:38.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.963+0000 7f74fac02740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56358e34ac30 con 0x56358df77730 2024-04-30T08:37:38.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.964+0000 7f74fac02740 10 client.0 _ll_register_callbacks cb 0x56358e295a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:38.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.957+0000 7f74f1c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56358df6f550 0x56358e34c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:38.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.966+0000 7f74f1c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56358df6f550 0x56358e34c770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:59428/0 (socket says 172.21.15.32:59428) 2024-04-30T08:37:38.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.966+0000 7f74f1c00640 1 -- 172.21.15.32:0/431753828 learned_addr learned my addr 172.21.15.32:0/431753828 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:38.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.968+0000 7f74ebe00640 1 -- 172.21.15.32:0/431753828 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56358df6f550 msgr2=0x56358e34c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:38.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.968+0000 7f74ebe00640 1 --2- 172.21.15.32:0/431753828 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56358df6f550 0x56358e34c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:38.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.968+0000 7f74ebe00640 1 -- 172.21.15.32:0/431753828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56358df98410 msgr2=0x56358e353c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:38.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.968+0000 7f74ebe00640 1 --2- 172.21.15.32:0/431753828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56358df98410 0x56358e353c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:38.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.969+0000 7f74ebe00640 1 -- 172.21.15.32:0/431753828 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56358e0787e0 con 0x56358df77730 2024-04-30T08:37:38.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.969+0000 7f74f1c00640 1 --2- 172.21.15.32:0/431753828 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56358df6f550 0x56358e34c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:37:38.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.971+0000 7f74ebe00640 1 --2- 172.21.15.32:0/431753828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358df77730 0x56358e34ccb0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f74d8005ab0 tx=0x7f74d8031840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65ae7847abfd0bb6 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:38.972 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.971+0000 7f74d2a00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-30T08:37:38.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.971+0000 7f74d2a00640 1 -- 172.21.15.32:0/431753828 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d8035070 con 0x56358df77730 2024-04-30T08:37:38.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.972+0000 7f74d2a00640 1 -- 172.21.15.32:0/431753828 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f74d8004240 con 0x56358df77730 2024-04-30T08:37:38.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.972+0000 7f74fac02740 1 -- 172.21.15.32:0/431753828 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74e4004d20 con 0x56358df77730 2024-04-30T08:37:38.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.973+0000 7f74d2a00640 1 -- 172.21.15.32:0/431753828 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f74d8004440 con 0x56358df77730 2024-04-30T08:37:38.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.973+0000 7f74fac02740 10 client.26350 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:38.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.973+0000 7f74fac02740 1 -- 172.21.15.32:0/431753828 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56358e0d8f70 con 0x56358df77730 2024-04-30T08:37:38.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.974+0000 7f74d2a00640 1 -- 172.21.15.32:0/431753828 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74d8003aa0 con 0x56358df77730 2024-04-30T08:37:38.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.974+0000 7f74d2a00640 1 -- 172.21.15.32:0/431753828 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7f74d803c880 con 0x56358df77730 2024-04-30T08:37:38.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.974+0000 7f74fac02740 10 client.26350 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:38.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.974+0000 7f74fac02740 10 client.26350 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:37:38.976 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.975+0000 7f74fac02740 20 client.26350 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:38.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.975+0000 7f74fac02740 1 --2- 172.21.15.32:0/431753828 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x56358df571f0 0x56358e35a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:38.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.975+0000 7f74fac02740 4 client.26350 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:37:38.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.975+0000 7f74fac02740 1 -- 172.21.15.32:0/431753828 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f74e4004d20 con 0x56358df571f0 2024-04-30T08:37:38.977 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.975+0000 7f74f1c00640 1 --2- 172.21.15.32:0/431753828 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x56358df571f0 0x56358e35a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:38.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.977+0000 7f74f1c00640 1 --2- 172.21.15.32:0/431753828 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x56358df571f0 0x56358e35a230 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65c8adb61ce80942 server_cookie=394b04eb7334aca4 in_seq=0 out_seq=0 2024-04-30T08:37:38.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.977+0000 7f74d2a00640 10 client.26350 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:38.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.978+0000 7f74d2a00640 1 -- 172.21.15.32:0/431753828 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7f74e4004d20 con 0x56358df571f0 2024-04-30T08:37:38.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.978+0000 7f74d2a00640 10 client.26350 handle_command_reply: tid=0 2024-04-30T08:37:38.979 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.978+0000 7f74fac02740 1 client.26350 shutdown 2024-04-30T08:37:38.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.978+0000 7f74fac02740 1 -- 172.21.15.32:0/431753828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358df77730 msgr2=0x56358e34ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:38.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.978+0000 7f74fac02740 1 --2- 172.21.15.32:0/431753828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358df77730 0x56358e34ccb0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f74d8005ab0 tx=0x7f74d8031840 comp rx=0 tx=0).stop 2024-04-30T08:37:38.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.979+0000 7f74fac02740 1 -- 172.21.15.32:0/431753828 shutdown_connections 2024-04-30T08:37:38.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.979+0000 7f74fac02740 1 --2- 172.21.15.32:0/431753828 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x56358df571f0 0x56358e35a230 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:38.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.979+0000 7f74fac02740 1 --2- 172.21.15.32:0/431753828 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56358df98410 0x56358e353c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:38.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.979+0000 7f74fac02740 1 --2- 172.21.15.32:0/431753828 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358df77730 0x56358e34ccb0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:38.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.979+0000 7f74fac02740 1 --2- 172.21.15.32:0/431753828 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56358df6f550 0x56358e34c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:38.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.979+0000 7f74fac02740 1 -- 172.21.15.32:0/431753828 >> 172.21.15.32:0/431753828 conn(0x56358df8c560 msgr2=0x56358e334d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:38.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.979+0000 7f74d2a00640 0 client.26350 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:38.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.980+0000 7f74fac02740 1 -- 172.21.15.32:0/431753828 shutdown_connections 2024-04-30T08:37:38.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.980+0000 7f74fac02740 1 -- 172.21.15.32:0/431753828 wait complete. 2024-04-30T08:37:38.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:38.980+0000 7f74fac02740 20 client.26350 trim_cache size 0 max 16384 2024-04-30T08:37:39.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.021+0000 7f74fac02740 1 Processor -- start 2024-04-30T08:37:39.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.021+0000 7f74fac02740 1 -- start start 2024-04-30T08:37:39.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.021+0000 7f74fac02740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56358df6f550 0x56358e33e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:39.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.021+0000 7f74fac02740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358df98410 0x56358e353bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:39.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.021+0000 7f74fac02740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56358df2c630 0x56358e354110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:39.023 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.021+0000 7f74fac02740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f74cc00fcf0 con 0x56358df2c630 2024-04-30T08:37:39.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.022+0000 7f74fac02740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f74cc0118c0 con 0x56358df6f550 2024-04-30T08:37:39.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.022+0000 7f74fac02740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f74cc0103c0 con 0x56358df98410 2024-04-30T08:37:39.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.022+0000 7f74f1c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56358df6f550 0x56358e33e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:39.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.022+0000 7f74ebe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358df98410 0x56358e353bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:39.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.022+0000 7f74ebe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358df98410 0x56358e353bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:52348/0 (socket says 172.21.15.32:52348) 2024-04-30T08:37:39.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.022+0000 7f74ebe00640 1 -- 172.21.15.32:0/1201031003 learned_addr learned my addr 172.21.15.32:0/1201031003 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:39.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.023+0000 7f74f1c00640 1 --2- 172.21.15.32:0/1201031003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56358df2c630 0x56358e354110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:39.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.023+0000 7f74ebe00640 1 -- 172.21.15.32:0/1201031003 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56358df6f550 msgr2=0x56358e33e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:39.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.023+0000 7f74ebe00640 1 --2- 172.21.15.32:0/1201031003 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56358df6f550 0x56358e33e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.023+0000 7f74ebe00640 1 -- 172.21.15.32:0/1201031003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56358df2c630 msgr2=0x56358e354110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:39.025 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.023+0000 7f74ebe00640 1 --2- 172.21.15.32:0/1201031003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56358df2c630 0x56358e354110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.023+0000 7f74ebe00640 1 -- 172.21.15.32:0/1201031003 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56358df70170 con 0x56358df98410 2024-04-30T08:37:39.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.024+0000 7f74ebe00640 1 --2- 172.21.15.32:0/1201031003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358df98410 0x56358e353bd0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f74d8031500 tx=0x7f74d8001e70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c08f1e4b65e278af server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:39.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.024+0000 7f74dd600640 1 -- 172.21.15.32:0/1201031003 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d8035040 con 0x56358df98410 2024-04-30T08:37:39.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.025+0000 7f74dd600640 1 -- 172.21.15.32:0/1201031003 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74d8058070 con 0x56358df98410 2024-04-30T08:37:39.026 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.025+0000 7f74dd600640 1 -- 172.21.15.32:0/1201031003 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74d803bcf0 con 0x56358df98410 2024-04-30T08:37:39.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.027+0000 7f74fac02740 1 -- 172.21.15.32:0/1201031003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358df98410 msgr2=0x56358e353bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:39.028 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.027+0000 7f74fac02740 1 --2- 172.21.15.32:0/1201031003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358df98410 0x56358e353bd0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f74d8031500 tx=0x7f74d8001e70 comp rx=0 tx=0).stop 2024-04-30T08:37:39.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.028+0000 7f74fac02740 1 -- 172.21.15.32:0/1201031003 shutdown_connections 2024-04-30T08:37:39.029 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.028+0000 7f74fac02740 1 --2- 172.21.15.32:0/1201031003 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56358df2c630 0x56358e354110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.028+0000 7f74fac02740 1 --2- 172.21.15.32:0/1201031003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358df98410 0x56358e353bd0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.028+0000 7f74fac02740 1 --2- 172.21.15.32:0/1201031003 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56358df6f550 0x56358e33e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.028+0000 7f74fac02740 1 -- 172.21.15.32:0/1201031003 >> 172.21.15.32:0/1201031003 conn(0x56358df8c560 msgr2=0x56358e341da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:39.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.029+0000 7f74fac02740 1 -- 172.21.15.32:0/1201031003 shutdown_connections 2024-04-30T08:37:39.030 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.029+0000 7f74fac02740 1 -- 172.21.15.32:0/1201031003 wait complete. 2024-04-30T08:37:39.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.032+0000 7f74fac02740 1 -- start start 2024-04-30T08:37:39.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.033+0000 7f74fac02740 1 Processor -- start 2024-04-30T08:37:39.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.033+0000 7f74fac02740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56358df2c630 0x56358e351800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:39.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.033+0000 7f74fac02740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56358df6f550 0x56358e351d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:39.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.034+0000 7f74fac02740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358df98410 0x56358e352c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:39.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.034+0000 7f74fac02740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f74cc011160 con 0x56358df6f550 2024-04-30T08:37:39.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.034+0000 7f74fac02740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f74cc010a90 con 0x56358df2c630 2024-04-30T08:37:39.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.034+0000 7f74fac02740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f74cc00dcb0 con 0x56358df98410 2024-04-30T08:37:39.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.034+0000 7f74fac02740 10 client.0 _ll_register_callbacks cb 0x7f74d8009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:39.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.034+0000 7f74f1c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358df98410 0x56358e352c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:39.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.034+0000 7f74f1c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358df98410 0x56358e352c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:52350/0 (socket says 172.21.15.32:52350) 2024-04-30T08:37:39.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.034+0000 7f74f1c00640 1 -- 172.21.15.32:0/1404366455 learned_addr learned my addr 172.21.15.32:0/1404366455 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:39.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.034+0000 7f74f1c00640 1 --2- 172.21.15.32:0/1404366455 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56358df2c630 0x56358e351800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:39.037 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.035+0000 7f74ebe00640 1 --2- 172.21.15.32:0/1404366455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56358df6f550 0x56358e351d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:39.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.036+0000 7f74f1c00640 1 -- 172.21.15.32:0/1404366455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358df98410 msgr2=0x56358e352c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:39.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.036+0000 7f74f1c00640 1 --2- 172.21.15.32:0/1404366455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358df98410 0x56358e352c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.036+0000 7f74f1c00640 1 -- 172.21.15.32:0/1404366455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56358df6f550 msgr2=0x56358e351d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:39.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.036+0000 7f74f1c00640 1 --2- 172.21.15.32:0/1404366455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56358df6f550 0x56358e351d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.036+0000 7f74f1c00640 1 -- 172.21.15.32:0/1404366455 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56358e353230 con 0x56358df2c630 2024-04-30T08:37:39.038 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.036+0000 7f74ebe00640 1 --2- 172.21.15.32:0/1404366455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56358df6f550 0x56358e351d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:37:39.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.037+0000 7f74f1c00640 1 --2- 172.21.15.32:0/1404366455 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56358df2c630 0x56358e351800 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f74e400ec80 tx=0x7f74e4005c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1b00c09d31010e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:39.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.037+0000 7f74d2a00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:37:39.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.037+0000 7f74d2a00640 1 -- 172.21.15.32:0/1404366455 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74e4016040 con 0x56358df2c630 2024-04-30T08:37:39.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.037+0000 7f74d2a00640 1 -- 172.21.15.32:0/1404366455 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f74e4011610 con 0x56358df2c630 2024-04-30T08:37:39.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.038+0000 7f74fac02740 1 -- 172.21.15.32:0/1404366455 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74d8058070 con 0x56358df2c630 2024-04-30T08:37:39.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.038+0000 7f74d2a00640 1 -- 172.21.15.32:0/1404366455 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f74e4011810 con 0x56358df2c630 2024-04-30T08:37:39.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.038+0000 7f74fac02740 10 client.26363 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:39.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.038+0000 7f74fac02740 1 -- 172.21.15.32:0/1404366455 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56358e3d3580 con 0x56358df2c630 2024-04-30T08:37:39.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.039+0000 7f74d2a00640 1 -- 172.21.15.32:0/1404366455 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f74d8058070 con 0x56358df2c630 2024-04-30T08:37:39.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.039+0000 7f74d2a00640 1 -- 172.21.15.32:0/1404366455 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7f74e400f040 con 0x56358df2c630 2024-04-30T08:37:39.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.039+0000 7f74fac02740 10 client.26363 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:39.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.039+0000 7f74fac02740 10 client.26363 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:37:39.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.039+0000 7f74fac02740 20 client.26363 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:39.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.039+0000 7f74fac02740 1 --2- 172.21.15.32:0/1404366455 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x56358e3d3d80 0x56358e3d61a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:39.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.039+0000 7f74fac02740 4 client.26363 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-30T08:37:39.041 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.039+0000 7f74fac02740 1 -- 172.21.15.32:0/1404366455 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x56358e2ac760 con 0x56358e3d3d80 2024-04-30T08:37:39.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.040+0000 7f74ebe00640 1 --2- 172.21.15.32:0/1404366455 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x56358e3d3d80 0x56358e3d61a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:39.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.041+0000 7f74ebe00640 1 --2- 172.21.15.32:0/1404366455 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x56358e3d3d80 0x56358e3d61a0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e5e192c6e177a5a server_cookie=98400cc4490ea604 in_seq=0 out_seq=0 2024-04-30T08:37:39.042 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.041+0000 7f74d2a00640 10 client.26363 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:39.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.042+0000 7f74d2a00640 1 -- 172.21.15.32:0/1404366455 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x56358e2ac760 con 0x56358e3d3d80 2024-04-30T08:37:39.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.042+0000 7f74d2a00640 10 client.26363 handle_command_reply: tid=0 2024-04-30T08:37:39.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.042+0000 7f74fac02740 1 client.26363 shutdown 2024-04-30T08:37:39.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.042+0000 7f74fac02740 1 -- 172.21.15.32:0/1404366455 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56358df2c630 msgr2=0x56358e351800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:39.044 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.042+0000 7f74fac02740 1 --2- 172.21.15.32:0/1404366455 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56358df2c630 0x56358e351800 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f74e400ec80 tx=0x7f74e4005c30 comp rx=0 tx=0).stop 2024-04-30T08:37:39.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.043+0000 7f74fac02740 1 -- 172.21.15.32:0/1404366455 shutdown_connections 2024-04-30T08:37:39.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.043+0000 7f74fac02740 1 --2- 172.21.15.32:0/1404366455 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x56358e3d3d80 0x56358e3d61a0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.045 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.043+0000 7f74fac02740 1 --2- 172.21.15.32:0/1404366455 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56358df98410 0x56358e352c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.043+0000 7f74d2a00640 0 client.26363 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:39.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.043+0000 7f74fac02740 1 --2- 172.21.15.32:0/1404366455 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56358df6f550 0x56358e351d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.043+0000 7f74fac02740 1 --2- 172.21.15.32:0/1404366455 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56358df2c630 0x56358e351800 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.043+0000 7f74fac02740 1 -- 172.21.15.32:0/1404366455 >> 172.21.15.32:0/1404366455 conn(0x56358df8c560 msgr2=0x56358e334d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:39.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.043+0000 7f74fac02740 1 -- 172.21.15.32:0/1404366455 shutdown_connections 2024-04-30T08:37:39.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.043+0000 7f74fac02740 1 -- 172.21.15.32:0/1404366455 wait complete. 2024-04-30T08:37:39.046 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.044+0000 7f74fac02740 20 client.26363 trim_cache size 0 max 16384 2024-04-30T08:37:39.046 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-30T08:37:39.046 INFO:teuthology.orchestra.run.smithi032.stdout: "return_code": 0, 2024-04-30T08:37:39.047 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_tag": "4dd6d866-1d6b-4d41-b94f-7569508c10c8", 2024-04-30T08:37:39.047 INFO:teuthology.orchestra.run.smithi032.stdout: "mode": "asynchronous" 2024-04-30T08:37:39.047 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-30T08:37:39.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.045+0000 7f74f2600640 1 -- 172.21.15.32:0/520805967 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f74d40afbe0 msgr2=0x7f74d40b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:39.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.045+0000 7f74f2600640 1 --2- 172.21.15.32:0/520805967 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f74d40afbe0 0x7f74d40b20f0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f74e00a7340 tx=0x7f74e00e1f40 comp rx=0 tx=0).stop 2024-04-30T08:37:39.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.045+0000 7f74f2600640 1 -- 172.21.15.32:0/520805967 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74ec105bb0 msgr2=0x7f74ec19c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:39.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.045+0000 7f74f2600640 1 --2- 172.21.15.32:0/520805967 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74ec105bb0 0x7f74ec19c310 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f74e0006fd0 tx=0x7f74e000b8e0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.046+0000 7f74f2600640 1 -- 172.21.15.32:0/520805967 shutdown_connections 2024-04-30T08:37:39.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.046+0000 7f74f2600640 1 --2- 172.21.15.32:0/520805967 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f74d40afbe0 0x7f74d40b20f0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.046+0000 7f74f2600640 1 --2- 172.21.15.32:0/520805967 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74ec10baf0 0x7f74ec1a37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.046+0000 7f74f2600640 1 --2- 172.21.15.32:0/520805967 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f74ec106df0 0x7f74ec19c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.046+0000 7f74f2600640 1 --2- 172.21.15.32:0/520805967 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f74ec105bb0 0x7f74ec19c310 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.046+0000 7f74f2600640 1 -- 172.21.15.32:0/520805967 >> 172.21.15.32:0/520805967 conn(0x7f74ec100da0 msgr2=0x7f74ec107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:39.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.046+0000 7f74f2600640 1 -- 172.21.15.32:0/520805967 shutdown_connections 2024-04-30T08:37:39.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.046+0000 7f74f2600640 1 -- 172.21.15.32:0/520805967 wait complete. 2024-04-30T08:37:39.092 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-30T08:37:39.092 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-30T08:37:39.452 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:39 smithi032 ceph-mon[28331]: pgmap v499: 129 pgs: 129 active+clean; 3.4 GiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 221 MiB/s wr, 96 op/s 2024-04-30T08:37:39.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:39 smithi071 ceph-mon[33385]: pgmap v499: 129 pgs: 129 active+clean; 3.4 GiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 221 MiB/s wr, 96 op/s 2024-04-30T08:37:39.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:39 smithi189 ceph-mon[33038]: pgmap v499: 129 pgs: 129 active+clean; 3.4 GiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 221 MiB/s wr, 96 op/s 2024-04-30T08:37:39.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.563+0000 7fbe8b200640 1 -- 172.21.15.32:0/1844287359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe8c107520 msgr2=0x7fbe8c107930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:39.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.563+0000 7fbe8b200640 1 --2- 172.21.15.32:0/1844287359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe8c107520 0x7fbe8c107930 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fbe84009be0 tx=0x7fbe8402a6c0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.563+0000 7fbe8b200640 1 -- 172.21.15.32:0/1844287359 shutdown_connections 2024-04-30T08:37:39.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.563+0000 7fbe8b200640 1 --2- 172.21.15.32:0/1844287359 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe8c107520 0x7fbe8c107930 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.563+0000 7fbe8b200640 1 --2- 172.21.15.32:0/1844287359 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbe8c104bc0 0x7fbe8c106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.563+0000 7fbe8b200640 1 --2- 172.21.15.32:0/1844287359 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe8c102260 0x7fbe8c104680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.565 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.563+0000 7fbe8b200640 1 -- 172.21.15.32:0/1844287359 >> 172.21.15.32:0/1844287359 conn(0x7fbe8c0fda70 msgr2=0x7fbe8c0fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:39.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.563+0000 7fbe8b200640 1 -- 172.21.15.32:0/1844287359 shutdown_connections 2024-04-30T08:37:39.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.563+0000 7fbe8b200640 1 -- 172.21.15.32:0/1844287359 wait complete. 2024-04-30T08:37:39.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.565+0000 7fbe8b200640 1 Processor -- start 2024-04-30T08:37:39.566 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.565+0000 7fbe8b200640 1 -- start start 2024-04-30T08:37:39.567 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.565+0000 7fbe8b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe8c102260 0x7fbe8c1a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:39.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.565+0000 7fbe8b200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbe8c104bc0 0x7fbe8c1a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:39.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.565+0000 7fbe8b200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe8c107520 0x7fbe8c1a7b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:39.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.565+0000 7fbe8b200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbe8c115aa0 con 0x7fbe8c102260 2024-04-30T08:37:39.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.565+0000 7fbe8b200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fbe8c115920 con 0x7fbe8c104bc0 2024-04-30T08:37:39.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.565+0000 7fbe8b200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbe8c115c20 con 0x7fbe8c107520 2024-04-30T08:37:39.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.566+0000 7fbe89400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbe8c104bc0 0x7fbe8c1a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:39.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.566+0000 7fbe89400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbe8c104bc0 0x7fbe8c1a0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:59478/0 (socket says 172.21.15.32:59478) 2024-04-30T08:37:39.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.566+0000 7fbe89400640 1 -- 172.21.15.32:0/3877847317 learned_addr learned my addr 172.21.15.32:0/3877847317 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:39.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.566+0000 7fbe89e00640 1 --2- 172.21.15.32:0/3877847317 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe8c102260 0x7fbe8c1a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:39.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.566+0000 7fbe8a800640 1 --2- 172.21.15.32:0/3877847317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe8c107520 0x7fbe8c1a7b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:39.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.566+0000 7fbe89400640 1 -- 172.21.15.32:0/3877847317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe8c107520 msgr2=0x7fbe8c1a7b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:39.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.566+0000 7fbe89400640 1 --2- 172.21.15.32:0/3877847317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe8c107520 0x7fbe8c1a7b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.568 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.566+0000 7fbe89400640 1 -- 172.21.15.32:0/3877847317 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe8c102260 msgr2=0x7fbe8c1a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:39.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.566+0000 7fbe89400640 1 --2- 172.21.15.32:0/3877847317 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe8c102260 0x7fbe8c1a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.566+0000 7fbe89400640 1 -- 172.21.15.32:0/3877847317 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe8c1a81c0 con 0x7fbe8c104bc0 2024-04-30T08:37:39.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.566+0000 7fbe8a800640 1 --2- 172.21.15.32:0/3877847317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe8c107520 0x7fbe8c1a7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-30T08:37:39.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.566+0000 7fbe89e00640 1 --2- 172.21.15.32:0/3877847317 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe8c102260 0x7fbe8c1a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:37:39.569 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.566+0000 7fbe89400640 1 --2- 172.21.15.32:0/3877847317 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbe8c104bc0 0x7fbe8c1a0bc0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fbe7400e9c0 tx=0x7fbe7400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:39.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.567+0000 7fbe82a00640 1 -- 172.21.15.32:0/3877847317 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe7400ccc0 con 0x7fbe8c104bc0 2024-04-30T08:37:39.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.567+0000 7fbe82a00640 1 -- 172.21.15.32:0/3877847317 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe740054d0 con 0x7fbe8c104bc0 2024-04-30T08:37:39.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.567+0000 7fbe8b200640 1 -- 172.21.15.32:0/3877847317 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe8c1a84b0 con 0x7fbe8c104bc0 2024-04-30T08:37:39.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.567+0000 7fbe82a00640 1 -- 172.21.15.32:0/3877847317 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe74010430 con 0x7fbe8c104bc0 2024-04-30T08:37:39.570 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.567+0000 7fbe8b200640 1 -- 172.21.15.32:0/3877847317 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe8c1a8a50 con 0x7fbe8c104bc0 2024-04-30T08:37:39.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.571+0000 7fbe97c35740 1 Processor -- start 2024-04-30T08:37:39.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.571+0000 7fbe97c35740 1 -- start start 2024-04-30T08:37:39.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.572+0000 7fbe97c35740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563359f07390 0x56335a2b9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:39.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.572+0000 7fbe97c35740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563359ede4d0 0x56335a2b99d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:39.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.572+0000 7fbe97c35740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563359ee66b0 0x56335a2c08d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:39.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.572+0000 7fbe97c35740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56335a20dd50 con 0x563359f07390 2024-04-30T08:37:39.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.572+0000 7fbe97c35740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x56335a1b2ef0 con 0x563359ee66b0 2024-04-30T08:37:39.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.572+0000 7fbe97c35740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56335a209d10 con 0x563359ede4d0 2024-04-30T08:37:39.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.572+0000 7fbe8a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563359f07390 0x56335a2b9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:39.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.572+0000 7fbe8a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563359f07390 0x56335a2b9450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34616/0 (socket says 172.21.15.32:34616) 2024-04-30T08:37:39.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.572+0000 7fbe8a800640 1 -- 172.21.15.32:0/1506033210 learned_addr learned my addr 172.21.15.32:0/1506033210 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:39.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.573+0000 7fbe82a00640 1 -- 172.21.15.32:0/3877847317 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7fbe74004f80 con 0x7fbe8c104bc0 2024-04-30T08:37:39.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.573+0000 7fbe89400640 1 --2- 172.21.15.32:0/1506033210 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563359ee66b0 0x56335a2c08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:39.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.573+0000 7fbe8a800640 1 -- 172.21.15.32:0/1506033210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563359ede4d0 msgr2=0x56335a2b99d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-30T08:37:39.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.573+0000 7fbe8a800640 1 --2- 172.21.15.32:0/1506033210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563359ede4d0 0x56335a2b99d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.573+0000 7fbe8a800640 1 -- 172.21.15.32:0/1506033210 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563359ee66b0 msgr2=0x56335a2c08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:39.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.573+0000 7fbe8a800640 1 --2- 172.21.15.32:0/1506033210 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563359ee66b0 0x56335a2c08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.573+0000 7fbe8a800640 1 -- 172.21.15.32:0/1506033210 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563359fe97d0 con 0x563359f07390 2024-04-30T08:37:39.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.573+0000 7fbe89400640 1 --2- 172.21.15.32:0/1506033210 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563359ee66b0 0x56335a2c08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:37:39.581 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.574+0000 7fbe8a800640 1 --2- 172.21.15.32:0/1506033210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563359f07390 0x56335a2b9450 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fbe84034040 tx=0x7fbe84030420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6eed595b85b6c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:39.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.575+0000 7fbe82a00640 1 --2- 172.21.15.32:0/3877847317 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbe700afbe0 0x7fbe700b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:39.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.575+0000 7fbe82a00640 1 -- 172.21.15.32:0/3877847317 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fbe740df4d0 con 0x7fbe8c104bc0 2024-04-30T08:37:39.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.575+0000 7fbe7b000640 1 -- 172.21.15.32:0/1506033210 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe84062070 con 0x563359f07390 2024-04-30T08:37:39.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.575+0000 7fbe7b000640 1 -- 172.21.15.32:0/1506033210 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe84058070 con 0x563359f07390 2024-04-30T08:37:39.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.575+0000 7fbe7b000640 1 -- 172.21.15.32:0/1506033210 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe840044a0 con 0x563359f07390 2024-04-30T08:37:39.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.576+0000 7fbe97c35740 1 -- 172.21.15.32:0/1506033210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563359f07390 msgr2=0x56335a2b9450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:39.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.576+0000 7fbe97c35740 1 --2- 172.21.15.32:0/1506033210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563359f07390 0x56335a2b9450 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fbe84034040 tx=0x7fbe84030420 comp rx=0 tx=0).stop 2024-04-30T08:37:39.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.576+0000 7fbe97c35740 1 -- 172.21.15.32:0/1506033210 shutdown_connections 2024-04-30T08:37:39.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.576+0000 7fbe97c35740 1 --2- 172.21.15.32:0/1506033210 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563359ee66b0 0x56335a2c08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.576+0000 7fbe97c35740 1 --2- 172.21.15.32:0/1506033210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563359ede4d0 0x56335a2b99d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.576+0000 7fbe97c35740 1 --2- 172.21.15.32:0/1506033210 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563359f07390 0x56335a2b9450 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.582 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.576+0000 7fbe97c35740 1 -- 172.21.15.32:0/1506033210 >> 172.21.15.32:0/1506033210 conn(0x563359efb4e0 msgr2=0x56335a2a3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:39.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.576+0000 7fbe97c35740 1 -- 172.21.15.32:0/1506033210 shutdown_connections 2024-04-30T08:37:39.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.577+0000 7fbe97c35740 1 -- 172.21.15.32:0/1506033210 wait complete. 2024-04-30T08:37:39.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.577+0000 7fbe89e00640 1 --2- 172.21.15.32:0/3877847317 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbe700afbe0 0x7fbe700b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:39.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.577+0000 7fbe97c35740 1 -- start start 2024-04-30T08:37:39.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.578+0000 7fbe97c35740 1 Processor -- start 2024-04-30T08:37:39.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.578+0000 7fbe97c35740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563359ede4d0 0x56335a2aa420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:39.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.578+0000 7fbe97c35740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563359ee66b0 0x56335a2aa960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:39.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.578+0000 7fbe97c35740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563359f07390 0x56335a2b1870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:39.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.578+0000 7fbe97c35740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x56335a2a8210 con 0x563359ede4d0 2024-04-30T08:37:39.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.578+0000 7fbe97c35740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x56335a1c6990 con 0x563359ee66b0 2024-04-30T08:37:39.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.578+0000 7fbe97c35740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x56335a2a8390 con 0x563359f07390 2024-04-30T08:37:39.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.579+0000 7fbe97c35740 10 client.0 _ll_register_callbacks cb 0x56335a1c5110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:39.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.579+0000 7fbe8a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563359f07390 0x56335a2b1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:39.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.579+0000 7fbe8a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563359f07390 0x56335a2b1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:52380/0 (socket says 172.21.15.32:52380) 2024-04-30T08:37:39.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.579+0000 7fbe8a800640 1 -- 172.21.15.32:0/2839868702 learned_addr learned my addr 172.21.15.32:0/2839868702 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:39.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.580+0000 7fbe89e00640 1 --2- 172.21.15.32:0/2839868702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563359ede4d0 0x56335a2aa420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:39.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.581+0000 7fbe89e00640 1 --2- 172.21.15.32:0/3877847317 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbe700afbe0 0x7fbe700b20f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c0058c0 tx=0x7fbe7c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:39.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.582+0000 7fbe89400640 1 --2- 172.21.15.32:0/2839868702 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563359ee66b0 0x56335a2aa960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:39.584 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.582+0000 7fbe8a800640 1 -- 172.21.15.32:0/2839868702 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563359ee66b0 msgr2=0x56335a2aa960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:39.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.582+0000 7fbe8a800640 1 --2- 172.21.15.32:0/2839868702 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563359ee66b0 0x56335a2aa960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.582+0000 7fbe8a800640 1 -- 172.21.15.32:0/2839868702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563359ede4d0 msgr2=0x56335a2aa420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:39.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.582+0000 7fbe8a800640 1 --2- 172.21.15.32:0/2839868702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563359ede4d0 0x56335a2aa420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.585 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.582+0000 7fbe8a800640 1 -- 172.21.15.32:0/2839868702 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563359f1f3f0 con 0x563359f07390 2024-04-30T08:37:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.582+0000 7fbe8a800640 1 --2- 172.21.15.32:0/2839868702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563359f07390 0x56335a2b1870 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fbe84009880 tx=0x7fbe84003c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=125a7031158dd5a4 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.582+0000 7fbe6f400640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-30T08:37:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.582+0000 7fbe6f400640 1 -- 172.21.15.32:0/2839868702 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe84062050 con 0x563359f07390 2024-04-30T08:37:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.583+0000 7fbe6f400640 1 -- 172.21.15.32:0/2839868702 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fbe84004220 con 0x563359f07390 2024-04-30T08:37:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.583+0000 7fbe97c35740 1 -- 172.21.15.32:0/2839868702 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe84058070 con 0x563359f07390 2024-04-30T08:37:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.583+0000 7fbe6f400640 1 -- 172.21.15.32:0/2839868702 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbe84036040 con 0x563359f07390 2024-04-30T08:37:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.583+0000 7fbe97c35740 10 client.26362 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.583+0000 7fbe97c35740 1 -- 172.21.15.32:0/2839868702 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x563359edf0f0 con 0x563359f07390 2024-04-30T08:37:39.586 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.584+0000 7fbe6f400640 1 -- 172.21.15.32:0/2839868702 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe8403f070 con 0x563359f07390 2024-04-30T08:37:39.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.584+0000 7fbe6f400640 1 -- 172.21.15.32:0/2839868702 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7fbe84039070 con 0x563359f07390 2024-04-30T08:37:39.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.584+0000 7fbe97c35740 10 client.26362 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:39.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.584+0000 7fbe97c35740 10 client.26362 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:37:39.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.584+0000 7fbe97c35740 20 client.26362 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:39.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.584+0000 7fbe97c35740 1 --2- 172.21.15.32:0/2839868702 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x563359ec5f60 0x56335a2b7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:39.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.584+0000 7fbe97c35740 4 client.26362 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:37:39.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.584+0000 7fbe97c35740 1 -- 172.21.15.32:0/2839868702 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fbe84058070 con 0x563359ec5f60 2024-04-30T08:37:39.587 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.586+0000 7fbe89e00640 1 --2- 172.21.15.32:0/2839868702 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x563359ec5f60 0x56335a2b7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:39.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.589+0000 7fbe6f400640 10 client.26362 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:39.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.589+0000 7fbe89e00640 1 --2- 172.21.15.32:0/2839868702 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x563359ec5f60 0x56335a2b7ef0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=29618fbb62c12c72 server_cookie=ace3b8440120ce30 in_seq=0 out_seq=0 2024-04-30T08:37:39.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.589+0000 7fbe6f400640 1 -- 172.21.15.32:0/2839868702 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x7fbe84058070 con 0x563359ec5f60 2024-04-30T08:37:39.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.589+0000 7fbe6f400640 10 client.26362 handle_command_reply: tid=0 2024-04-30T08:37:39.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.590+0000 7fbe97c35740 1 client.26362 shutdown 2024-04-30T08:37:39.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.590+0000 7fbe97c35740 1 -- 172.21.15.32:0/2839868702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563359f07390 msgr2=0x56335a2b1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:39.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.590+0000 7fbe97c35740 1 --2- 172.21.15.32:0/2839868702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563359f07390 0x56335a2b1870 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fbe84009880 tx=0x7fbe84003c20 comp rx=0 tx=0).stop 2024-04-30T08:37:39.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.590+0000 7fbe97c35740 1 -- 172.21.15.32:0/2839868702 shutdown_connections 2024-04-30T08:37:39.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.590+0000 7fbe97c35740 1 --2- 172.21.15.32:0/2839868702 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x563359ec5f60 0x56335a2b7ef0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.590+0000 7fbe97c35740 1 --2- 172.21.15.32:0/2839868702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563359f07390 0x56335a2b1870 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.590+0000 7fbe97c35740 1 --2- 172.21.15.32:0/2839868702 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563359ee66b0 0x56335a2aa960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.590+0000 7fbe97c35740 1 --2- 172.21.15.32:0/2839868702 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563359ede4d0 0x56335a2aa420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.590+0000 7fbe97c35740 1 -- 172.21.15.32:0/2839868702 >> 172.21.15.32:0/2839868702 conn(0x563359efb4e0 msgr2=0x56335a2a3dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:39.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.590+0000 7fbe6f400640 0 client.26362 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:39.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.590+0000 7fbe97c35740 1 -- 172.21.15.32:0/2839868702 shutdown_connections 2024-04-30T08:37:39.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.591+0000 7fbe97c35740 1 -- 172.21.15.32:0/2839868702 wait complete. 2024-04-30T08:37:39.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.591+0000 7fbe97c35740 20 client.26362 trim_cache size 0 max 16384 2024-04-30T08:37:39.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.634+0000 7fbe97c35740 1 Processor -- start 2024-04-30T08:37:39.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.634+0000 7fbe97c35740 1 -- start start 2024-04-30T08:37:39.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.635+0000 7fbe97c35740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563359ede4d0 0x56335a2a90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:39.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.635+0000 7fbe97c35740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563359ee66b0 0x56335a2bdba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:39.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.635+0000 7fbe97c35740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563359f07390 0x56335a2aab30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:39.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.635+0000 7fbe97c35740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbe8c06e040 con 0x563359ede4d0 2024-04-30T08:37:39.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.635+0000 7fbe97c35740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fbe8c06fc10 con 0x563359f07390 2024-04-30T08:37:39.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.635+0000 7fbe97c35740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbe8c06e710 con 0x563359ee66b0 2024-04-30T08:37:39.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.635+0000 7fbe89400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563359ee66b0 0x56335a2bdba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:39.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.635+0000 7fbe89400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563359ee66b0 0x56335a2bdba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:52392/0 (socket says 172.21.15.32:52392) 2024-04-30T08:37:39.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.635+0000 7fbe89400640 1 -- 172.21.15.32:0/118305226 learned_addr learned my addr 172.21.15.32:0/118305226 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:39.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.635+0000 7fbe8a800640 1 --2- 172.21.15.32:0/118305226 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563359f07390 0x56335a2aab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:39.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.635+0000 7fbe89e00640 1 --2- 172.21.15.32:0/118305226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563359ede4d0 0x56335a2a90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:39.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.636+0000 7fbe89400640 1 -- 172.21.15.32:0/118305226 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563359f07390 msgr2=0x56335a2aab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:39.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.636+0000 7fbe89400640 1 --2- 172.21.15.32:0/118305226 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563359f07390 0x56335a2aab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.636+0000 7fbe89400640 1 -- 172.21.15.32:0/118305226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563359ede4d0 msgr2=0x56335a2a90b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:39.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.636+0000 7fbe89400640 1 --2- 172.21.15.32:0/118305226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563359ede4d0 0x56335a2a90b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.636+0000 7fbe89400640 1 -- 172.21.15.32:0/118305226 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56335a2a84c0 con 0x563359ee66b0 2024-04-30T08:37:39.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.636+0000 7fbe89e00640 1 --2- 172.21.15.32:0/118305226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563359ede4d0 0x56335a2a90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:37:39.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.636+0000 7fbe89400640 1 --2- 172.21.15.32:0/118305226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563359ee66b0 0x56335a2bdba0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fbe740a72a0 tx=0x7fbe74004a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93af5992eb888086 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:39.639 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.637+0000 7fbe7a600640 1 -- 172.21.15.32:0/118305226 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe740a6ef0 con 0x563359ee66b0 2024-04-30T08:37:39.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.637+0000 7fbe7a600640 1 -- 172.21.15.32:0/118305226 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe740a70b0 con 0x563359ee66b0 2024-04-30T08:37:39.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.637+0000 7fbe7a600640 1 -- 172.21.15.32:0/118305226 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe740a6d10 con 0x563359ee66b0 2024-04-30T08:37:39.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.638+0000 7fbe97c35740 1 -- 172.21.15.32:0/118305226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563359ee66b0 msgr2=0x56335a2bdba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:39.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.638+0000 7fbe97c35740 1 --2- 172.21.15.32:0/118305226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563359ee66b0 0x56335a2bdba0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fbe740a72a0 tx=0x7fbe74004a20 comp rx=0 tx=0).stop 2024-04-30T08:37:39.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.638+0000 7fbe97c35740 1 -- 172.21.15.32:0/118305226 shutdown_connections 2024-04-30T08:37:39.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.638+0000 7fbe97c35740 1 --2- 172.21.15.32:0/118305226 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563359f07390 0x56335a2aab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.638+0000 7fbe97c35740 1 --2- 172.21.15.32:0/118305226 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563359ee66b0 0x56335a2bdba0 secure :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fbe740a72a0 tx=0x7fbe74004a20 comp rx=0 tx=0).stop 2024-04-30T08:37:39.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.638+0000 7fbe97c35740 1 --2- 172.21.15.32:0/118305226 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563359ede4d0 0x56335a2a90b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.640 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.638+0000 7fbe97c35740 1 -- 172.21.15.32:0/118305226 >> 172.21.15.32:0/118305226 conn(0x563359efb4e0 msgr2=0x56335a2aa420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:39.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.638+0000 7fbe97c35740 1 -- 172.21.15.32:0/118305226 shutdown_connections 2024-04-30T08:37:39.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.638+0000 7fbe97c35740 1 -- 172.21.15.32:0/118305226 wait complete. 2024-04-30T08:37:39.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.639+0000 7fbe97c35740 1 -- start start 2024-04-30T08:37:39.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.639+0000 7fbe97c35740 1 Processor -- start 2024-04-30T08:37:39.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.640+0000 7fbe97c35740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563359ede4d0 0x56335a2b1840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:39.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.640+0000 7fbe97c35740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563359f07390 0x56335a2b1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:39.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.640+0000 7fbe97c35740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563359e9b3a0 0x56335a2ae3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:39.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.640+0000 7fbe97c35740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fbe8c06f4b0 con 0x563359f07390 2024-04-30T08:37:39.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.640+0000 7fbe97c35740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fbe8c06ede0 con 0x563359e9b3a0 2024-04-30T08:37:39.642 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.640+0000 7fbe97c35740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fbe8c1105f0 con 0x563359ede4d0 2024-04-30T08:37:39.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.640+0000 7fbe97c35740 10 client.0 _ll_register_callbacks cb 0x7fbe740e2d70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:39.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.640+0000 7fbe89e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563359ede4d0 0x56335a2b1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:39.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.640+0000 7fbe89e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563359ede4d0 0x56335a2b1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:52408/0 (socket says 172.21.15.32:52408) 2024-04-30T08:37:39.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.640+0000 7fbe89e00640 1 -- 172.21.15.32:0/1783725073 learned_addr learned my addr 172.21.15.32:0/1783725073 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:39.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.640+0000 7fbe8a800640 1 --2- 172.21.15.32:0/1783725073 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563359e9b3a0 0x56335a2ae3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:39.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.640+0000 7fbe89400640 1 --2- 172.21.15.32:0/1783725073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563359f07390 0x56335a2b1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:39.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.640+0000 7fbe89e00640 1 -- 172.21.15.32:0/1783725073 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563359e9b3a0 msgr2=0x56335a2ae3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:39.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.640+0000 7fbe89e00640 1 --2- 172.21.15.32:0/1783725073 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563359e9b3a0 0x56335a2ae3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.640+0000 7fbe89e00640 1 -- 172.21.15.32:0/1783725073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563359f07390 msgr2=0x56335a2b1d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:39.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.640+0000 7fbe89e00640 1 --2- 172.21.15.32:0/1783725073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563359f07390 0x56335a2b1d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.640+0000 7fbe89e00640 1 -- 172.21.15.32:0/1783725073 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56335a2aea00 con 0x563359ede4d0 2024-04-30T08:37:39.643 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.640+0000 7fbe89400640 1 --2- 172.21.15.32:0/1783725073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563359f07390 0x56335a2b1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-30T08:37:39.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.641+0000 7fbe89e00640 1 --2- 172.21.15.32:0/1783725073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563359ede4d0 0x56335a2b1840 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x56335a2ba830 tx=0x7fbe7c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee195efb4ef3ed91 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:39.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.641+0000 7fbe6f400640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-30T08:37:39.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.641+0000 7fbe6f400640 1 -- 172.21.15.32:0/1783725073 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe7c028740 con 0x563359ede4d0 2024-04-30T08:37:39.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.641+0000 7fbe6f400640 1 -- 172.21.15.32:0/1783725073 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fbe7c0288e0 con 0x563359ede4d0 2024-04-30T08:37:39.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.641+0000 7fbe97c35740 1 -- 172.21.15.32:0/1783725073 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe740a70b0 con 0x563359ede4d0 2024-04-30T08:37:39.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.642+0000 7fbe6f400640 1 -- 172.21.15.32:0/1783725073 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fbe7c038050 con 0x563359ede4d0 2024-04-30T08:37:39.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.642+0000 7fbe97c35740 10 client.26368 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:39.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.642+0000 7fbe97c35740 1 -- 172.21.15.32:0/1783725073 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x56335a37c8b0 con 0x563359ede4d0 2024-04-30T08:37:39.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.642+0000 7fbe6f400640 1 -- 172.21.15.32:0/1783725073 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fbe740a70b0 con 0x563359ede4d0 2024-04-30T08:37:39.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.642+0000 7fbe6f400640 1 -- 172.21.15.32:0/1783725073 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7fbe7c01e420 con 0x563359ede4d0 2024-04-30T08:37:39.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.642+0000 7fbe97c35740 10 client.26368 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:39.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.642+0000 7fbe97c35740 10 client.26368 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:37:39.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.642+0000 7fbe97c35740 20 client.26368 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:39.645 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.642+0000 7fbe97c35740 1 --2- 172.21.15.32:0/1783725073 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x56335a37d350 0x56335a37f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:39.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.642+0000 7fbe97c35740 4 client.26368 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-30T08:37:39.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.642+0000 7fbe97c35740 1 -- 172.21.15.32:0/1783725073 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56335a221ef0 con 0x56335a37d350 2024-04-30T08:37:39.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.643+0000 7fbe89400640 1 --2- 172.21.15.32:0/1783725073 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x56335a37d350 0x56335a37f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:39.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.644+0000 7fbe6f400640 10 client.26368 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:39.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.644+0000 7fbe89400640 1 --2- 172.21.15.32:0/1783725073 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x56335a37d350 0x56335a37f770 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa00fddf26353b08 server_cookie=41e59b368acd0b72 in_seq=0 out_seq=0 2024-04-30T08:37:39.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.644+0000 7fbe6f400640 1 -- 172.21.15.32:0/1783725073 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56335a221ef0 con 0x56335a37d350 2024-04-30T08:37:39.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.644+0000 7fbe6f400640 10 client.26368 handle_command_reply: tid=0 2024-04-30T08:37:39.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.644+0000 7fbe97c35740 1 client.26368 shutdown 2024-04-30T08:37:39.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.645+0000 7fbe97c35740 1 -- 172.21.15.32:0/1783725073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563359ede4d0 msgr2=0x56335a2b1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:39.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.645+0000 7fbe97c35740 1 --2- 172.21.15.32:0/1783725073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563359ede4d0 0x56335a2b1840 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x56335a2ba830 tx=0x7fbe7c004730 comp rx=0 tx=0).stop 2024-04-30T08:37:39.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.645+0000 7fbe97c35740 1 -- 172.21.15.32:0/1783725073 shutdown_connections 2024-04-30T08:37:39.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.645+0000 7fbe97c35740 1 --2- 172.21.15.32:0/1783725073 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x56335a37d350 0x56335a37f770 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.645+0000 7fbe97c35740 1 --2- 172.21.15.32:0/1783725073 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x563359e9b3a0 0x56335a2ae3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.645+0000 7fbe97c35740 1 --2- 172.21.15.32:0/1783725073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x563359f07390 0x56335a2b1d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.645+0000 7fbe97c35740 1 --2- 172.21.15.32:0/1783725073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563359ede4d0 0x56335a2b1840 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.645+0000 7fbe6f400640 0 client.26368 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:39.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.645+0000 7fbe97c35740 1 -- 172.21.15.32:0/1783725073 >> 172.21.15.32:0/1783725073 conn(0x563359efb4e0 msgr2=0x56335a2a7360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:39.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.645+0000 7fbe97c35740 1 -- 172.21.15.32:0/1783725073 shutdown_connections 2024-04-30T08:37:39.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.645+0000 7fbe97c35740 1 -- 172.21.15.32:0/1783725073 wait complete. 2024-04-30T08:37:39.648 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-30T08:37:39.648 INFO:teuthology.orchestra.run.smithi032.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-30T08:37:39.648 INFO:teuthology.orchestra.run.smithi032.stdout: "scrubs": { 2024-04-30T08:37:39.649 INFO:teuthology.orchestra.run.smithi032.stdout: "4dd6d866-1d6b-4d41-b94f-7569508c10c8": { 2024-04-30T08:37:39.649 INFO:teuthology.orchestra.run.smithi032.stdout: "path": "/", 2024-04-30T08:37:39.649 INFO:teuthology.orchestra.run.smithi032.stdout: "tag": "4dd6d866-1d6b-4d41-b94f-7569508c10c8", 2024-04-30T08:37:39.649 INFO:teuthology.orchestra.run.smithi032.stdout: "options": "recursive,force" 2024-04-30T08:37:39.649 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:37:39.649 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:37:39.649 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-30T08:37:39.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.645+0000 7fbe97c35740 20 client.26368 trim_cache size 0 max 16384 2024-04-30T08:37:39.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.646+0000 7fbe8b200640 1 -- 172.21.15.32:0/3877847317 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbe700afbe0 msgr2=0x7fbe700b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:39.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.646+0000 7fbe8b200640 1 --2- 172.21.15.32:0/3877847317 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbe700afbe0 0x7fbe700b20f0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fbe7c0058c0 tx=0x7fbe7c005850 comp rx=0 tx=0).stop 2024-04-30T08:37:39.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.646+0000 7fbe8b200640 1 -- 172.21.15.32:0/3877847317 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbe8c104bc0 msgr2=0x7fbe8c1a0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:39.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.646+0000 7fbe8b200640 1 --2- 172.21.15.32:0/3877847317 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbe8c104bc0 0x7fbe8c1a0bc0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fbe7400e9c0 tx=0x7fbe7400ee80 comp rx=0 tx=0).stop 2024-04-30T08:37:39.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.647+0000 7fbe8b200640 1 -- 172.21.15.32:0/3877847317 shutdown_connections 2024-04-30T08:37:39.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.647+0000 7fbe8b200640 1 --2- 172.21.15.32:0/3877847317 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fbe700afbe0 0x7fbe700b20f0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.647+0000 7fbe8b200640 1 --2- 172.21.15.32:0/3877847317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe8c107520 0x7fbe8c1a7b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.647+0000 7fbe8b200640 1 --2- 172.21.15.32:0/3877847317 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fbe8c104bc0 0x7fbe8c1a0bc0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.647+0000 7fbe8b200640 1 --2- 172.21.15.32:0/3877847317 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fbe8c102260 0x7fbe8c1a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:39.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.647+0000 7fbe8b200640 1 -- 172.21.15.32:0/3877847317 >> 172.21.15.32:0/3877847317 conn(0x7fbe8c0fda70 msgr2=0x7fbe8c1030c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:39.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.647+0000 7fbe8b200640 1 -- 172.21.15.32:0/3877847317 shutdown_connections 2024-04-30T08:37:39.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:37:39.647+0000 7fbe8b200640 1 -- 172.21.15.32:0/3877847317 wait complete. 2024-04-30T08:37:39.700 INFO:tasks.cephfs.filesystem:scrub status for tag:4dd6d866-1d6b-4d41-b94f-7569508c10c8 - {'path': '/', 'tag': '4dd6d866-1d6b-4d41-b94f-7569508c10c8', 'options': 'recursive,force'} 2024-04-30T08:37:40.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:40 smithi071 ceph-mon[33385]: scrub queued for path: / 2024-04-30T08:37:40.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:40 smithi071 ceph-mon[33385]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:40.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:40 smithi071 ceph-mon[33385]: scrub summary: active paths [/] 2024-04-30T08:37:40.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:40 smithi071 ceph-mon[33385]: scrub summary: active paths [/] 2024-04-30T08:37:40.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:40 smithi071 ceph-mon[33385]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:40.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:40 smithi071 ceph-mon[33385]: scrub summary: active paths [/] 2024-04-30T08:37:40.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:40 smithi071 ceph-mon[33385]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:40.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:40 smithi071 ceph-mon[33385]: scrub summary: active paths [/] 2024-04-30T08:37:40.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:40 smithi071 ceph-mon[33385]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:40.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:40 smithi071 ceph-mon[33385]: scrub summary: active paths [/] 2024-04-30T08:37:40.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:40 smithi071 ceph-mon[33385]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:40.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:40 smithi071 ceph-mon[33385]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:40.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:40 smithi189 ceph-mon[33038]: scrub queued for path: / 2024-04-30T08:37:40.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:40 smithi189 ceph-mon[33038]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:40.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:40 smithi189 ceph-mon[33038]: scrub summary: active paths [/] 2024-04-30T08:37:40.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:40 smithi189 ceph-mon[33038]: scrub summary: active paths [/] 2024-04-30T08:37:40.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:40 smithi189 ceph-mon[33038]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:40.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:40 smithi189 ceph-mon[33038]: scrub summary: active paths [/] 2024-04-30T08:37:40.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:40 smithi189 ceph-mon[33038]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:40.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:40 smithi189 ceph-mon[33038]: scrub summary: active paths [/] 2024-04-30T08:37:40.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:40 smithi189 ceph-mon[33038]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:40.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:40 smithi189 ceph-mon[33038]: scrub summary: active paths [/] 2024-04-30T08:37:40.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:40 smithi189 ceph-mon[33038]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:40.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:40 smithi189 ceph-mon[33038]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:40.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:40 smithi032 ceph-mon[28331]: scrub queued for path: / 2024-04-30T08:37:40.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:40 smithi032 ceph-mon[28331]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:40.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:40 smithi032 ceph-mon[28331]: scrub summary: active paths [/] 2024-04-30T08:37:40.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:40 smithi032 ceph-mon[28331]: scrub summary: active paths [/] 2024-04-30T08:37:40.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:40 smithi032 ceph-mon[28331]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:40.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:40 smithi032 ceph-mon[28331]: scrub summary: active paths [/] 2024-04-30T08:37:40.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:40 smithi032 ceph-mon[28331]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:40.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:40 smithi032 ceph-mon[28331]: scrub summary: active paths [/] 2024-04-30T08:37:40.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:40 smithi032 ceph-mon[28331]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:40.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:40 smithi032 ceph-mon[28331]: scrub summary: active paths [/] 2024-04-30T08:37:40.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:40 smithi032 ceph-mon[28331]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:40.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:40 smithi032 ceph-mon[28331]: scrub summary: idle+waiting paths [/] 2024-04-30T08:37:41.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:41 smithi071 ceph-mon[33385]: pgmap v500: 129 pgs: 129 active+clean; 4.1 GiB data, 934 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 259 MiB/s wr, 118 op/s 2024-04-30T08:37:41.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:41 smithi189 ceph-mon[33038]: pgmap v500: 129 pgs: 129 active+clean; 4.1 GiB data, 934 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 259 MiB/s wr, 118 op/s 2024-04-30T08:37:41.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:41 smithi032 ceph-mon[28331]: pgmap v500: 129 pgs: 129 active+clean; 4.1 GiB data, 934 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 259 MiB/s wr, 118 op/s 2024-04-30T08:37:43.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:43 smithi071 ceph-mon[33385]: pgmap v501: 129 pgs: 129 active+clean; 4.6 GiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 245 MiB/s wr, 117 op/s 2024-04-30T08:37:43.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:43 smithi189 ceph-mon[33038]: pgmap v501: 129 pgs: 129 active+clean; 4.6 GiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 245 MiB/s wr, 117 op/s 2024-04-30T08:37:43.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:43 smithi032 ceph-mon[28331]: pgmap v501: 129 pgs: 129 active+clean; 4.6 GiB data, 1014 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 245 MiB/s wr, 117 op/s 2024-04-30T08:37:45.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:45 smithi071 ceph-mon[33385]: pgmap v502: 129 pgs: 129 active+clean; 5.0 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 233 MiB/s wr, 135 op/s 2024-04-30T08:37:45.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:45 smithi071 ceph-mon[33385]: scrub summary: idle 2024-04-30T08:37:45.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:45 smithi071 ceph-mon[33385]: scrub summary: idle 2024-04-30T08:37:45.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:45 smithi189 ceph-mon[33038]: pgmap v502: 129 pgs: 129 active+clean; 5.0 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 233 MiB/s wr, 135 op/s 2024-04-30T08:37:45.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:45 smithi189 ceph-mon[33038]: scrub summary: idle 2024-04-30T08:37:45.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:45 smithi189 ceph-mon[33038]: scrub summary: idle 2024-04-30T08:37:45.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:45 smithi032 ceph-mon[28331]: pgmap v502: 129 pgs: 129 active+clean; 5.0 GiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 105 KiB/s rd, 233 MiB/s wr, 135 op/s 2024-04-30T08:37:45.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:45 smithi032 ceph-mon[28331]: scrub summary: idle 2024-04-30T08:37:45.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:45 smithi032 ceph-mon[28331]: scrub summary: idle 2024-04-30T08:37:46.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:46 smithi071 ceph-mon[33385]: scrub summary: idle 2024-04-30T08:37:46.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:46 smithi071 ceph-mon[33385]: scrub summary: idle 2024-04-30T08:37:46.532 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.531+0000 7f4d84c00640 1 -- 172.21.15.32:0/919653252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d80102480 msgr2=0x7f4d801048a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:46.532 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.531+0000 7f4d84c00640 1 --2- 172.21.15.32:0/919653252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d80102480 0x7f4d801048a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f4d78009ca0 tx=0x7f4d7802a8d0 comp rx=0 tx=0).stop 2024-04-30T08:37:46.535 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.534+0000 7f4d84c00640 1 -- 172.21.15.32:0/919653252 shutdown_connections 2024-04-30T08:37:46.535 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.534+0000 7f4d84c00640 1 --2- 172.21.15.32:0/919653252 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d80107740 0x7f4d80107b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:46.535 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.534+0000 7f4d84c00640 1 --2- 172.21.15.32:0/919653252 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d80104de0 0x7f4d80107200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:46.535 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.534+0000 7f4d84c00640 1 --2- 172.21.15.32:0/919653252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d80102480 0x7f4d801048a0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:46.535 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.534+0000 7f4d84c00640 1 -- 172.21.15.32:0/919653252 >> 172.21.15.32:0/919653252 conn(0x7f4d800fdc90 msgr2=0x7f4d80100140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:46.538 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.537+0000 7f4d84c00640 1 -- 172.21.15.32:0/919653252 shutdown_connections 2024-04-30T08:37:46.538 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.537+0000 7f4d84c00640 1 -- 172.21.15.32:0/919653252 wait complete. 2024-04-30T08:37:46.540 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.539+0000 7f4d84c00640 1 Processor -- start 2024-04-30T08:37:46.542 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.539+0000 7f4d84c00640 1 -- start start 2024-04-30T08:37:46.542 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.540+0000 7f4d84c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d80102480 0x7f4d801a0970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:46.542 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.540+0000 7f4d84c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d80104de0 0x7f4d801a0eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:46.542 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.540+0000 7f4d84c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d80107740 0x7f4d801a7e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:46.543 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.540+0000 7f4d84c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4d80115d00 con 0x7f4d80104de0 2024-04-30T08:37:46.543 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.541+0000 7f4d84c00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4d80115b80 con 0x7f4d80107740 2024-04-30T08:37:46.543 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.541+0000 7f4d84c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4d80115e80 con 0x7f4d80102480 2024-04-30T08:37:46.543 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.540+0000 7f4d7d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d80104de0 0x7f4d801a0eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:46.543 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.541+0000 7f4d7e800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d80107740 0x7f4d801a7e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:46.543 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.540+0000 7f4d7de00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d80102480 0x7f4d801a0970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:46.543 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.541+0000 7f4d7e800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d80107740 0x7f4d801a7e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:59556/0 (socket says 172.21.15.32:59556) 2024-04-30T08:37:46.543 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.541+0000 7f4d7e800640 1 -- 172.21.15.32:0/422313112 learned_addr learned my addr 172.21.15.32:0/422313112 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:46.543 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.541+0000 7f4d7d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d80104de0 0x7f4d801a0eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34672/0 (socket says 172.21.15.32:34672) 2024-04-30T08:37:46.544 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.541+0000 7f4d7e800640 1 -- 172.21.15.32:0/422313112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d80102480 msgr2=0x7f4d801a0970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:46.544 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.541+0000 7f4d7e800640 1 --2- 172.21.15.32:0/422313112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d80102480 0x7f4d801a0970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:46.544 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.541+0000 7f4d7e800640 1 -- 172.21.15.32:0/422313112 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d80104de0 msgr2=0x7f4d801a0eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:46.544 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.541+0000 7f4d7e800640 1 --2- 172.21.15.32:0/422313112 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d80104de0 0x7f4d801a0eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:46.544 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.541+0000 7f4d7e800640 1 -- 172.21.15.32:0/422313112 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d80108700 con 0x7f4d80107740 2024-04-30T08:37:46.544 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.542+0000 7f4d7de00640 1 --2- 172.21.15.32:0/422313112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d80102480 0x7f4d801a0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:37:46.544 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.542+0000 7f4d7d400640 1 --2- 172.21.15.32:0/422313112 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d80104de0 0x7f4d801a0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:37:46.544 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.542+0000 7f4d7e800640 1 --2- 172.21.15.32:0/422313112 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d80107740 0x7f4d801a7e40 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c00c970 tx=0x7f4d6c00ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:46.544 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.542+0000 7f4d76a00640 1 -- 172.21.15.32:0/422313112 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d6c004230 con 0x7f4d80107740 2024-04-30T08:37:46.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.542+0000 7f4d76a00640 1 -- 172.21.15.32:0/422313112 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d6c0043d0 con 0x7f4d80107740 2024-04-30T08:37:46.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.543+0000 7f4d76a00640 1 -- 172.21.15.32:0/422313112 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d6c00f450 con 0x7f4d80107740 2024-04-30T08:37:46.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.543+0000 7f4d84c00640 1 -- 172.21.15.32:0/422313112 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d801089f0 con 0x7f4d80107740 2024-04-30T08:37:46.545 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.543+0000 7f4d84c00640 1 -- 172.21.15.32:0/422313112 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d80108f70 con 0x7f4d80107740 2024-04-30T08:37:46.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:46 smithi189 ceph-mon[33038]: scrub summary: idle 2024-04-30T08:37:46.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:46 smithi189 ceph-mon[33038]: scrub summary: idle 2024-04-30T08:37:46.555 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.554+0000 7f4d76a00640 1 -- 172.21.15.32:0/422313112 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f4d6c0079b0 con 0x7f4d80107740 2024-04-30T08:37:46.558 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.557+0000 7f4d8bc29740 1 Processor -- start 2024-04-30T08:37:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.557+0000 7f4d8bc29740 1 -- start start 2024-04-30T08:37:46.559 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.558+0000 7f4d8bc29740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558110ac61b0 0x558110e680e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:46.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.558+0000 7f4d8bc29740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558110a9d2f0 0x558110e68660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:46.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.558+0000 7f4d8bc29740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558110aa54d0 0x558110e6f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:46.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.558+0000 7f4d8bc29740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558110dcc550 con 0x558110ac61b0 2024-04-30T08:37:46.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.558+0000 7f4d8bc29740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x558110d71bc0 con 0x558110aa54d0 2024-04-30T08:37:46.560 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.558+0000 7f4d8bc29740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558110dc8510 con 0x558110a9d2f0 2024-04-30T08:37:46.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.559+0000 7f4d76a00640 1 --2- 172.21.15.32:0/422313112 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4d640afcb0 0x7f4d640b21c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:46.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.559+0000 7f4d76a00640 1 -- 172.21.15.32:0/422313112 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f4d6c0df0e0 con 0x7f4d80107740 2024-04-30T08:37:46.561 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.559+0000 7f4d7e800640 1 --2- 172.21.15.32:0/422313112 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4d640afcb0 0x7f4d640b21c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:46.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.560+0000 7f4d7de00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558110a9d2f0 0x558110e68660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:46.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.560+0000 7f4d7de00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558110a9d2f0 0x558110e68660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:52440/0 (socket says 172.21.15.32:52440) 2024-04-30T08:37:46.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.560+0000 7f4d7de00640 1 -- 172.21.15.32:0/2459853137 learned_addr learned my addr 172.21.15.32:0/2459853137 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:46.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.560+0000 7f4d7d400640 1 --2- 172.21.15.32:0/2459853137 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558110aa54d0 0x558110e6f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:46.562 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.560+0000 7f4d7d400640 1 --2- 172.21.15.32:0/2459853137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558110ac61b0 0x558110e680e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:46.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.560+0000 7f4d7de00640 1 -- 172.21.15.32:0/2459853137 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558110aa54d0 msgr2=0x558110e6f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:46.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.560+0000 7f4d7de00640 1 --2- 172.21.15.32:0/2459853137 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558110aa54d0 0x558110e6f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:46.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.560+0000 7f4d7de00640 1 -- 172.21.15.32:0/2459853137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558110ac61b0 msgr2=0x558110e680e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:46.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.560+0000 7f4d7de00640 1 --2- 172.21.15.32:0/2459853137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558110ac61b0 0x558110e680e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:46.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.560+0000 7f4d7de00640 1 -- 172.21.15.32:0/2459853137 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558110ade210 con 0x558110a9d2f0 2024-04-30T08:37:46.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.561+0000 7f4d7de00640 1 --2- 172.21.15.32:0/2459853137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558110a9d2f0 0x558110e68660 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4d78035000 tx=0x7f4d78009920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=529ff42c3c30b5b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:46.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.561+0000 7f4d6b000640 1 -- 172.21.15.32:0/2459853137 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d78049070 con 0x558110a9d2f0 2024-04-30T08:37:46.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.561+0000 7f4d6b000640 1 -- 172.21.15.32:0/2459853137 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d78036040 con 0x558110a9d2f0 2024-04-30T08:37:46.563 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.561+0000 7f4d6b000640 1 -- 172.21.15.32:0/2459853137 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d78044670 con 0x558110a9d2f0 2024-04-30T08:37:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.565+0000 7f4d7e800640 1 --2- 172.21.15.32:0/422313112 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4d640afcb0 0x7f4d640b21c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c0a4420 tx=0x7f4d6c005df0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.565+0000 7f4d8bc29740 1 -- 172.21.15.32:0/2459853137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558110a9d2f0 msgr2=0x558110e68660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:46.567 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.565+0000 7f4d8bc29740 1 --2- 172.21.15.32:0/2459853137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558110a9d2f0 0x558110e68660 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f4d78035000 tx=0x7f4d78009920 comp rx=0 tx=0).stop 2024-04-30T08:37:46.568 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.566+0000 7f4d8bc29740 1 -- 172.21.15.32:0/2459853137 shutdown_connections 2024-04-30T08:37:46.568 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.566+0000 7f4d8bc29740 1 --2- 172.21.15.32:0/2459853137 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558110aa54d0 0x558110e6f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:46.568 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.566+0000 7f4d8bc29740 1 --2- 172.21.15.32:0/2459853137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558110a9d2f0 0x558110e68660 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:46.568 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.566+0000 7f4d8bc29740 1 --2- 172.21.15.32:0/2459853137 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558110ac61b0 0x558110e680e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:46.568 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.566+0000 7f4d8bc29740 1 -- 172.21.15.32:0/2459853137 >> 172.21.15.32:0/2459853137 conn(0x558110aba300 msgr2=0x558110e63a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:46.568 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.567+0000 7f4d8bc29740 1 -- 172.21.15.32:0/2459853137 shutdown_connections 2024-04-30T08:37:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.567+0000 7f4d8bc29740 1 -- 172.21.15.32:0/2459853137 wait complete. 2024-04-30T08:37:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.568+0000 7f4d8bc29740 1 -- start start 2024-04-30T08:37:46.570 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.569+0000 7f4d8bc29740 1 Processor -- start 2024-04-30T08:37:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.570+0000 7f4d8bc29740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558110a9d2f0 0x558110e71a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.570+0000 7f4d8bc29740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558110aa54d0 0x558110e71f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.570+0000 7f4d8bc29740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558110ac61b0 0x558110e78ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.570+0000 7f4d8bc29740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x558110e6ff50 con 0x558110aa54d0 2024-04-30T08:37:46.572 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.570+0000 7f4d8bc29740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x558110d85660 con 0x558110a9d2f0 2024-04-30T08:37:46.573 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.570+0000 7f4d8bc29740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x558110e700d0 con 0x558110ac61b0 2024-04-30T08:37:46.573 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.570+0000 7f4d8bc29740 10 client.0 _ll_register_callbacks cb 0x558110d4f400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:46.575 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.573+0000 7f4d7d400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558110a9d2f0 0x558110e71a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:46.575 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.573+0000 7f4d7d400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558110a9d2f0 0x558110e71a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:59574/0 (socket says 172.21.15.32:59574) 2024-04-30T08:37:46.575 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.573+0000 7f4d7d400640 1 -- 172.21.15.32:0/3868194390 learned_addr learned my addr 172.21.15.32:0/3868194390 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:46.575 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.573+0000 7f4d7d400640 1 --2- 172.21.15.32:0/3868194390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558110ac61b0 0x558110e78ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:46.575 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.573+0000 7f4d7de00640 1 --2- 172.21.15.32:0/3868194390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558110aa54d0 0x558110e71f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:46.576 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.574+0000 7f4d7d400640 1 -- 172.21.15.32:0/3868194390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558110ac61b0 msgr2=0x558110e78ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:46.576 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.574+0000 7f4d7d400640 1 --2- 172.21.15.32:0/3868194390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558110ac61b0 0x558110e78ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:46.576 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.574+0000 7f4d7d400640 1 -- 172.21.15.32:0/3868194390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558110aa54d0 msgr2=0x558110e71f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:46.576 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.574+0000 7f4d7d400640 1 --2- 172.21.15.32:0/3868194390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558110aa54d0 0x558110e71f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:46.577 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.574+0000 7f4d7d400640 1 -- 172.21.15.32:0/3868194390 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558110baa620 con 0x558110a9d2f0 2024-04-30T08:37:46.577 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.574+0000 7f4d7d400640 1 --2- 172.21.15.32:0/3868194390 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558110a9d2f0 0x558110e71a50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4d70019bc0 tx=0x7f4d700046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2803c96101258562 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:46.577 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.574+0000 7f4d5f400640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:37:46.577 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.575+0000 7f4d5f400640 1 -- 172.21.15.32:0/3868194390 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d70014df0 con 0x558110a9d2f0 2024-04-30T08:37:46.577 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.575+0000 7f4d5f400640 1 -- 172.21.15.32:0/3868194390 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f4d70016400 con 0x558110a9d2f0 2024-04-30T08:37:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.575+0000 7f4d8bc29740 1 -- 172.21.15.32:0/3868194390 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d78036040 con 0x558110a9d2f0 2024-04-30T08:37:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.576+0000 7f4d5f400640 1 -- 172.21.15.32:0/3868194390 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4d700059c0 con 0x558110a9d2f0 2024-04-30T08:37:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.576+0000 7f4d8bc29740 10 client.26387 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.576+0000 7f4d8bc29740 1 -- 172.21.15.32:0/3868194390 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558110e8cf00 con 0x558110a9d2f0 2024-04-30T08:37:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.576+0000 7f4d5f400640 1 -- 172.21.15.32:0/3868194390 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d78036040 con 0x558110a9d2f0 2024-04-30T08:37:46.578 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.576+0000 7f4d5f400640 1 -- 172.21.15.32:0/3868194390 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7f4d70015040 con 0x558110a9d2f0 2024-04-30T08:37:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.577+0000 7f4d8bc29740 10 client.26387 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.577+0000 7f4d8bc29740 10 client.26387 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-30T08:37:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.577+0000 7f4d8bc29740 20 client.26387 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.577+0000 7f4d8bc29740 1 --2- 172.21.15.32:0/3868194390 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x558110a84d80 0x558110e906c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.577+0000 7f4d8bc29740 4 client.26387 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:37:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.577+0000 7f4d8bc29740 1 -- 172.21.15.32:0/3868194390 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558110dda160 con 0x558110a84d80 2024-04-30T08:37:46.579 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.577+0000 7f4d7de00640 1 --2- 172.21.15.32:0/3868194390 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x558110a84d80 0x558110e906c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.578+0000 7f4d7de00640 1 --2- 172.21.15.32:0/3868194390 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x558110a84d80 0x558110e906c0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d5140cc4da72967 server_cookie=b718cc8a1dd886e4 in_seq=0 out_seq=0 2024-04-30T08:37:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.578+0000 7f4d5f400640 10 client.26387 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.579+0000 7f4d5f400640 1 -- 172.21.15.32:0/3868194390 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x558110dda160 con 0x558110a84d80 2024-04-30T08:37:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.579+0000 7f4d5f400640 10 client.26387 handle_command_reply: tid=0 2024-04-30T08:37:46.580 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.579+0000 7f4d8bc29740 1 client.26387 shutdown 2024-04-30T08:37:46.581 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.580+0000 7f4d8bc29740 1 -- 172.21.15.32:0/3868194390 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558110a9d2f0 msgr2=0x558110e71a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:46.583 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.581+0000 7f4d8bc29740 1 --2- 172.21.15.32:0/3868194390 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558110a9d2f0 0x558110e71a50 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4d70019bc0 tx=0x7f4d700046c0 comp rx=0 tx=0).stop 2024-04-30T08:37:46.583 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.581+0000 7f4d8bc29740 1 -- 172.21.15.32:0/3868194390 shutdown_connections 2024-04-30T08:37:46.583 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.581+0000 7f4d8bc29740 1 --2- 172.21.15.32:0/3868194390 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x558110a84d80 0x558110e906c0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:46.583 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.581+0000 7f4d8bc29740 1 --2- 172.21.15.32:0/3868194390 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558110ac61b0 0x558110e78ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:46.583 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.581+0000 7f4d8bc29740 1 --2- 172.21.15.32:0/3868194390 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558110aa54d0 0x558110e71f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:46.583 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.581+0000 7f4d8bc29740 1 --2- 172.21.15.32:0/3868194390 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558110a9d2f0 0x558110e71a50 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:46.584 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.581+0000 7f4d8bc29740 1 -- 172.21.15.32:0/3868194390 >> 172.21.15.32:0/3868194390 conn(0x558110aba300 msgr2=0x558110e66980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:46.584 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.581+0000 7f4d5f400640 0 client.26387 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:46.584 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.581+0000 7f4d8bc29740 1 -- 172.21.15.32:0/3868194390 shutdown_connections 2024-04-30T08:37:46.584 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.582+0000 7f4d8bc29740 1 -- 172.21.15.32:0/3868194390 wait complete. 2024-04-30T08:37:46.584 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.582+0000 7f4d8bc29740 20 client.26387 trim_cache size 0 max 16384 2024-04-30T08:37:46.613 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.611+0000 7f4d8bc29740 1 Processor -- start 2024-04-30T08:37:46.613 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.611+0000 7f4d8bc29740 1 -- start start 2024-04-30T08:37:46.613 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.611+0000 7f4d8bc29740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558110a9d2f0 0x558110e66870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:46.613 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.611+0000 7f4d8bc29740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558110aa54d0 0x558110e66db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:46.613 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.611+0000 7f4d8bc29740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558110ac61b0 0x558110e672f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:46.613 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.611+0000 7f4d8bc29740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4d8006ebd0 con 0x558110ac61b0 2024-04-30T08:37:46.613 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.611+0000 7f4d8bc29740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4d800707a0 con 0x558110aa54d0 2024-04-30T08:37:46.613 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.611+0000 7f4d8bc29740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4d8006f2a0 con 0x558110a9d2f0 2024-04-30T08:37:46.614 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.611+0000 7f4d7d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558110ac61b0 0x558110e672f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:46.615 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.611+0000 7f4d7d400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558110ac61b0 0x558110e672f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:34702/0 (socket says 172.21.15.32:34702) 2024-04-30T08:37:46.615 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.611+0000 7f4d7d400640 1 -- 172.21.15.32:0/171224807 learned_addr learned my addr 172.21.15.32:0/171224807 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:46.615 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.612+0000 7f4d7d400640 1 --2- 172.21.15.32:0/171224807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558110a9d2f0 0x558110e66870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:46.615 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.612+0000 7f4d7de00640 1 --2- 172.21.15.32:0/171224807 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558110aa54d0 0x558110e66db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:46.615 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.612+0000 7f4d7d400640 1 -- 172.21.15.32:0/171224807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558110a9d2f0 msgr2=0x558110e66870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:46.615 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.612+0000 7f4d7d400640 1 --2- 172.21.15.32:0/171224807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558110a9d2f0 0x558110e66870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:46.615 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.612+0000 7f4d7d400640 1 -- 172.21.15.32:0/171224807 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558110aa54d0 msgr2=0x558110e66db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:46.615 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.612+0000 7f4d7d400640 1 --2- 172.21.15.32:0/171224807 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558110aa54d0 0x558110e66db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:46.615 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.612+0000 7f4d7d400640 1 -- 172.21.15.32:0/171224807 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558110e72ac0 con 0x558110ac61b0 2024-04-30T08:37:46.615 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.612+0000 7f4d7de00640 1 --2- 172.21.15.32:0/171224807 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558110aa54d0 0x558110e66db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:37:46.615 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.612+0000 7f4d7d400640 1 --2- 172.21.15.32:0/171224807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558110ac61b0 0x558110e672f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4d7000af30 tx=0x7f4d7000a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a2b43e9b442bc23 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:46.615 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.613+0000 7f4d6a600640 1 -- 172.21.15.32:0/171224807 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d70016400 con 0x558110ac61b0 2024-04-30T08:37:46.616 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.613+0000 7f4d6a600640 1 -- 172.21.15.32:0/171224807 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d70016a50 con 0x558110ac61b0 2024-04-30T08:37:46.616 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.613+0000 7f4d6a600640 1 -- 172.21.15.32:0/171224807 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d7002cd20 con 0x558110ac61b0 2024-04-30T08:37:46.619 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.614+0000 7f4d8bc29740 1 -- 172.21.15.32:0/171224807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558110ac61b0 msgr2=0x558110e672f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:46.619 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.614+0000 7f4d8bc29740 1 --2- 172.21.15.32:0/171224807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558110ac61b0 0x558110e672f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f4d7000af30 tx=0x7f4d7000a040 comp rx=0 tx=0).stop 2024-04-30T08:37:46.619 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.614+0000 7f4d8bc29740 1 -- 172.21.15.32:0/171224807 shutdown_connections 2024-04-30T08:37:46.619 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.614+0000 7f4d8bc29740 1 --2- 172.21.15.32:0/171224807 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558110ac61b0 0x558110e672f0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:46.619 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.614+0000 7f4d8bc29740 1 --2- 172.21.15.32:0/171224807 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558110aa54d0 0x558110e66db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:46.619 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.614+0000 7f4d8bc29740 1 --2- 172.21.15.32:0/171224807 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558110a9d2f0 0x558110e66870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:46.619 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.614+0000 7f4d8bc29740 1 -- 172.21.15.32:0/171224807 >> 172.21.15.32:0/171224807 conn(0x558110aba300 msgr2=0x558110e6bf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:46.619 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.614+0000 7f4d8bc29740 1 -- 172.21.15.32:0/171224807 shutdown_connections 2024-04-30T08:37:46.619 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.614+0000 7f4d8bc29740 1 -- 172.21.15.32:0/171224807 wait complete. 2024-04-30T08:37:46.619 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.614+0000 7f4d8bc29740 1 -- start start 2024-04-30T08:37:46.619 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.615+0000 7f4d8bc29740 1 Processor -- start 2024-04-30T08:37:46.620 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.615+0000 7f4d8bc29740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558110a9d2f0 0x558110e75570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:46.620 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.615+0000 7f4d8bc29740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558110aa54d0 0x558110e75ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:46.620 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.615+0000 7f4d8bc29740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558110ac61b0 0x558110ef6a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:46.620 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.615+0000 7f4d8bc29740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4d80070040 con 0x558110aa54d0 2024-04-30T08:37:46.620 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.615+0000 7f4d8bc29740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4d8006f970 con 0x558110a9d2f0 2024-04-30T08:37:46.620 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.615+0000 7f4d8bc29740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4d80110ad0 con 0x558110ac61b0 2024-04-30T08:37:46.620 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.615+0000 7f4d8bc29740 10 client.0 _ll_register_callbacks cb 0x7f4d700197d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:37:46.620 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.615+0000 7f4d7d400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558110a9d2f0 0x558110e75570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:46.620 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.615+0000 7f4d7d400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558110a9d2f0 0x558110e75570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:59592/0 (socket says 172.21.15.32:59592) 2024-04-30T08:37:46.620 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.615+0000 7f4d7d400640 1 -- 172.21.15.32:0/3566045821 learned_addr learned my addr 172.21.15.32:0/3566045821 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:37:46.620 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.616+0000 7f4d7de00640 1 --2- 172.21.15.32:0/3566045821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558110aa54d0 0x558110e75ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:46.620 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.616+0000 7f4d7d400640 1 --2- 172.21.15.32:0/3566045821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558110ac61b0 0x558110ef6a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.616+0000 7f4d7d400640 1 -- 172.21.15.32:0/3566045821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558110ac61b0 msgr2=0x558110ef6a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.616+0000 7f4d7d400640 1 --2- 172.21.15.32:0/3566045821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558110ac61b0 0x558110ef6a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.616+0000 7f4d7d400640 1 -- 172.21.15.32:0/3566045821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558110aa54d0 msgr2=0x558110e75ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:37:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.616+0000 7f4d7d400640 1 --2- 172.21.15.32:0/3566045821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558110aa54d0 0x558110e75ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.616+0000 7f4d7d400640 1 -- 172.21.15.32:0/3566045821 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558110ef6fa0 con 0x558110a9d2f0 2024-04-30T08:37:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.616+0000 7f4d7d400640 1 --2- 172.21.15.32:0/3566045821 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558110a9d2f0 0x558110e75570 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4d700198e0 tx=0x7f4d70000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cdc9dce9c879ea1 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:37:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.616+0000 7f4d5f400640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:37:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.616+0000 7f4d5f400640 1 -- 172.21.15.32:0/3566045821 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d70016400 con 0x558110a9d2f0 2024-04-30T08:37:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.616+0000 7f4d5f400640 1 -- 172.21.15.32:0/3566045821 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f4d700165a0 con 0x558110a9d2f0 2024-04-30T08:37:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.616+0000 7f4d8bc29740 1 -- 172.21.15.32:0/3566045821 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d70016a50 con 0x558110a9d2f0 2024-04-30T08:37:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.617+0000 7f4d5f400640 1 -- 172.21.15.32:0/3566045821 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f4d700167a0 con 0x558110a9d2f0 2024-04-30T08:37:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.617+0000 7f4d8bc29740 10 client.26393 fetch_fsmap learned FSMap version 29 2024-04-30T08:37:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.617+0000 7f4d8bc29740 1 -- 172.21.15.32:0/3566045821 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x558110ef8170 con 0x558110a9d2f0 2024-04-30T08:37:46.621 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.617+0000 7f4d5f400640 1 -- 172.21.15.32:0/3566045821 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4d70016a50 con 0x558110a9d2f0 2024-04-30T08:37:46.622 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.617+0000 7f4d5f400640 1 -- 172.21.15.32:0/3566045821 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7f4d7003b060 con 0x558110a9d2f0 2024-04-30T08:37:46.622 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.617+0000 7f4d8bc29740 10 client.26393 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:37:46.622 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.617+0000 7f4d8bc29740 10 client.26393 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-30T08:37:46.622 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.617+0000 7f4d8bc29740 20 client.26393 populate_metadata read hostname 'smithi032' 2024-04-30T08:37:46.622 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.617+0000 7f4d8bc29740 1 --2- 172.21.15.32:0/3566045821 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x558110a5a1c0 0x558110efabc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:37:46.622 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.617+0000 7f4d8bc29740 4 client.26393 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-30T08:37:46.622 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.617+0000 7f4d8bc29740 1 -- 172.21.15.32:0/3566045821 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558110dd3fd0 con 0x558110a5a1c0 2024-04-30T08:37:46.622 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.618+0000 7f4d7de00640 1 --2- 172.21.15.32:0/3566045821 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x558110a5a1c0 0x558110efabc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:37:46.622 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.618+0000 7f4d7de00640 1 --2- 172.21.15.32:0/3566045821 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x558110a5a1c0 0x558110efabc0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ac7a53e67451cb0 server_cookie=758bacc273c9388a in_seq=0 out_seq=0 2024-04-30T08:37:46.623 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.618+0000 7f4d5f400640 10 client.26393 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:46.663 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.660+0000 7f4d5f400640 1 -- 172.21.15.32:0/3566045821 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558110dd3fd0 con 0x558110a5a1c0 2024-04-30T08:37:46.663 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.660+0000 7f4d5f400640 10 client.26393 handle_command_reply: tid=0 2024-04-30T08:37:46.663 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.660+0000 7f4d8bc29740 1 client.26393 shutdown 2024-04-30T08:37:46.663 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.661+0000 7f4d8bc29740 1 -- 172.21.15.32:0/3566045821 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558110a9d2f0 msgr2=0x558110e75570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:46.663 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.661+0000 7f4d8bc29740 1 --2- 172.21.15.32:0/3566045821 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558110a9d2f0 0x558110e75570 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4d700198e0 tx=0x7f4d70000bc0 comp rx=0 tx=0).stop 2024-04-30T08:37:46.663 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.661+0000 7f4d8bc29740 1 -- 172.21.15.32:0/3566045821 shutdown_connections 2024-04-30T08:37:46.663 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.661+0000 7f4d8bc29740 1 --2- 172.21.15.32:0/3566045821 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x558110a5a1c0 0x558110efabc0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:46.663 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.661+0000 7f4d8bc29740 1 --2- 172.21.15.32:0/3566045821 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558110ac61b0 0x558110ef6a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:46.663 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.661+0000 7f4d8bc29740 1 --2- 172.21.15.32:0/3566045821 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x558110aa54d0 0x558110e75ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:46.664 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.661+0000 7f4d8bc29740 1 --2- 172.21.15.32:0/3566045821 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x558110a9d2f0 0x558110e75570 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:46.664 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.661+0000 7f4d8bc29740 1 -- 172.21.15.32:0/3566045821 >> 172.21.15.32:0/3566045821 conn(0x558110aba300 msgr2=0x558110e66870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:46.664 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.661+0000 7f4d5f400640 0 client.26393 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:37:46.664 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.661+0000 7f4d8bc29740 1 -- 172.21.15.32:0/3566045821 shutdown_connections 2024-04-30T08:37:46.664 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.661+0000 7f4d8bc29740 1 -- 172.21.15.32:0/3566045821 wait complete. 2024-04-30T08:37:46.664 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.661+0000 7f4d8bc29740 20 client.26393 trim_cache size 0 max 16384 2024-04-30T08:37:46.664 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-30T08:37:46.664 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-30T08:37:46.664 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-30T08:37:46.664 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-30T08:37:46.665 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.662+0000 7f4d84c00640 1 -- 172.21.15.32:0/422313112 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4d640afcb0 msgr2=0x7f4d640b21c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:46.665 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.662+0000 7f4d84c00640 1 --2- 172.21.15.32:0/422313112 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4d640afcb0 0x7f4d640b21c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c0a4420 tx=0x7f4d6c005df0 comp rx=0 tx=0).stop 2024-04-30T08:37:46.665 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.662+0000 7f4d84c00640 1 -- 172.21.15.32:0/422313112 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d80107740 msgr2=0x7f4d801a7e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:37:46.665 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.662+0000 7f4d84c00640 1 --2- 172.21.15.32:0/422313112 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d80107740 0x7f4d801a7e40 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f4d6c00c970 tx=0x7f4d6c00ce30 comp rx=0 tx=0).stop 2024-04-30T08:37:46.665 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.662+0000 7f4d84c00640 1 -- 172.21.15.32:0/422313112 shutdown_connections 2024-04-30T08:37:46.665 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.662+0000 7f4d84c00640 1 --2- 172.21.15.32:0/422313112 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4d640afcb0 0x7f4d640b21c0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:46.666 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.662+0000 7f4d84c00640 1 --2- 172.21.15.32:0/422313112 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4d80107740 0x7f4d801a7e40 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:46.666 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.662+0000 7f4d84c00640 1 --2- 172.21.15.32:0/422313112 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4d80104de0 0x7f4d801a0eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:46.666 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.662+0000 7f4d84c00640 1 --2- 172.21.15.32:0/422313112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4d80102480 0x7f4d801a0970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:37:46.666 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.662+0000 7f4d84c00640 1 -- 172.21.15.32:0/422313112 >> 172.21.15.32:0/422313112 conn(0x7f4d800fdc90 msgr2=0x7f4d801033c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:37:46.666 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.663+0000 7f4d84c00640 1 -- 172.21.15.32:0/422313112 shutdown_connections 2024-04-30T08:37:46.666 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:37:46.663+0000 7f4d84c00640 1 -- 172.21.15.32:0/422313112 wait complete. 2024-04-30T08:37:46.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:46 smithi032 ceph-mon[28331]: scrub summary: idle 2024-04-30T08:37:46.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:46 smithi032 ceph-mon[28331]: scrub summary: idle 2024-04-30T08:37:47.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:47 smithi071 ceph-mon[33385]: pgmap v503: 129 pgs: 129 active+clean; 5.3 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 234 MiB/s wr, 125 op/s 2024-04-30T08:37:47.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:47 smithi189 ceph-mon[33038]: pgmap v503: 129 pgs: 129 active+clean; 5.3 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 234 MiB/s wr, 125 op/s 2024-04-30T08:37:47.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:47 smithi032 ceph-mon[28331]: pgmap v503: 129 pgs: 129 active+clean; 5.3 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 234 MiB/s wr, 125 op/s 2024-04-30T08:37:49.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:49 smithi071 ceph-mon[33385]: pgmap v504: 129 pgs: 129 active+clean; 5.6 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 215 MiB/s wr, 114 op/s 2024-04-30T08:37:49.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:49 smithi189 ceph-mon[33038]: pgmap v504: 129 pgs: 129 active+clean; 5.6 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 215 MiB/s wr, 114 op/s 2024-04-30T08:37:49.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:49 smithi032 ceph-mon[28331]: pgmap v504: 129 pgs: 129 active+clean; 5.6 GiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 215 MiB/s wr, 114 op/s 2024-04-30T08:37:51.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:51 smithi071 ceph-mon[33385]: pgmap v505: 129 pgs: 129 active+clean; 6.3 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 244 MiB/s wr, 134 op/s 2024-04-30T08:37:51.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:51 smithi189 ceph-mon[33038]: pgmap v505: 129 pgs: 129 active+clean; 6.3 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 244 MiB/s wr, 134 op/s 2024-04-30T08:37:51.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:51 smithi032 ceph-mon[28331]: pgmap v505: 129 pgs: 129 active+clean; 6.3 GiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 244 MiB/s wr, 134 op/s 2024-04-30T08:37:53.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:53 smithi189 ceph-mon[33038]: pgmap v506: 129 pgs: 129 active+clean; 6.9 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 236 MiB/s wr, 125 op/s 2024-04-30T08:37:53.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:53 smithi032 ceph-mon[28331]: pgmap v506: 129 pgs: 129 active+clean; 6.9 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 236 MiB/s wr, 125 op/s 2024-04-30T08:37:53.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:53 smithi071 ceph-mon[33385]: pgmap v506: 129 pgs: 129 active+clean; 6.9 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 236 MiB/s wr, 125 op/s 2024-04-30T08:37:55.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:55 smithi189 ceph-mon[33038]: pgmap v507: 129 pgs: 129 active+clean; 7.4 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 238 MiB/s wr, 121 op/s 2024-04-30T08:37:55.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:55 smithi032 ceph-mon[28331]: pgmap v507: 129 pgs: 129 active+clean; 7.4 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 238 MiB/s wr, 121 op/s 2024-04-30T08:37:55.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:55 smithi071 ceph-mon[33385]: pgmap v507: 129 pgs: 129 active+clean; 7.4 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 238 MiB/s wr, 121 op/s 2024-04-30T08:37:57.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:57 smithi189 ceph-mon[33038]: pgmap v508: 129 pgs: 129 active+clean; 7.8 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 242 MiB/s wr, 102 op/s 2024-04-30T08:37:57.554 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:57 smithi071 ceph-mon[33385]: pgmap v508: 129 pgs: 129 active+clean; 7.8 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 242 MiB/s wr, 102 op/s 2024-04-30T08:37:57.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:57 smithi032 ceph-mon[28331]: pgmap v508: 129 pgs: 129 active+clean; 7.8 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 242 MiB/s wr, 102 op/s 2024-04-30T08:37:59.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:37:59 smithi189 ceph-mon[33038]: pgmap v509: 129 pgs: 129 active+clean; 7.9 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 224 MiB/s wr, 98 op/s 2024-04-30T08:37:59.694 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:37:59 smithi032 ceph-mon[28331]: pgmap v509: 129 pgs: 129 active+clean; 7.9 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 224 MiB/s wr, 98 op/s 2024-04-30T08:37:59.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:37:59 smithi071 ceph-mon[33385]: pgmap v509: 129 pgs: 129 active+clean; 7.9 GiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 224 MiB/s wr, 98 op/s 2024-04-30T08:38:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.050+0000 7f43ad600640 1 -- 172.21.15.32:0/1289425638 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43a810a4a0 msgr2=0x7f43a810ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.050+0000 7f43ad600640 1 --2- 172.21.15.32:0/1289425638 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43a810a4a0 0x7f43a810ae90 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f4398009900 tx=0x7f439802a410 comp rx=0 tx=0).stop 2024-04-30T08:38:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.050+0000 7f43ad600640 1 -- 172.21.15.32:0/1289425638 shutdown_connections 2024-04-30T08:38:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.050+0000 7f43ad600640 1 --2- 172.21.15.32:0/1289425638 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43a81100f0 0x7f43a8110500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.050+0000 7f43ad600640 1 --2- 172.21.15.32:0/1289425638 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43a810b3d0 0x7f43a810b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.050+0000 7f43ad600640 1 --2- 172.21.15.32:0/1289425638 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43a810a4a0 0x7f43a810ae90 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:00.051 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.050+0000 7f43ad600640 1 -- 172.21.15.32:0/1289425638 >> 172.21.15.32:0/1289425638 conn(0x7f43a80705c0 msgr2=0x7f43a8072a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.051+0000 7f43ad600640 1 -- 172.21.15.32:0/1289425638 shutdown_connections 2024-04-30T08:38:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.051+0000 7f43ad600640 1 -- 172.21.15.32:0/1289425638 wait complete. 2024-04-30T08:38:00.052 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.052+0000 7f43ad600640 1 Processor -- start 2024-04-30T08:38:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.052+0000 7f43ad600640 1 -- start start 2024-04-30T08:38:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.053+0000 7f43ad600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43a810a4a0 0x7f43a81170f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.053+0000 7f43ad600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43a810b3d0 0x7f43a8117630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.053+0000 7f43ad600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43a81100f0 0x7f43a8118590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.053+0000 7f43ad600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f43a811e5a0 con 0x7f43a810a4a0 2024-04-30T08:38:00.053 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.053+0000 7f43ad600640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f43a811e420 con 0x7f43a810b3d0 2024-04-30T08:38:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.053+0000 7f43ad600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f43a811e720 con 0x7f43a81100f0 2024-04-30T08:38:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.053+0000 7f439fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43a810b3d0 0x7f43a8117630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.053+0000 7f439fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43a810b3d0 0x7f43a8117630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:60232/0 (socket says 172.21.15.32:60232) 2024-04-30T08:38:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.053+0000 7f439fe00640 1 -- 172.21.15.32:0/244586256 learned_addr learned my addr 172.21.15.32:0/244586256 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:00.054 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.053+0000 7f43a7400640 1 --2- 172.21.15.32:0/244586256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43a81100f0 0x7f43a8118590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.054+0000 7f43a6a00640 1 --2- 172.21.15.32:0/244586256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43a810a4a0 0x7f43a81170f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:00.055 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.054+0000 7f43a7400640 1 -- 172.21.15.32:0/244586256 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43a810b3d0 msgr2=0x7f43a8117630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.054+0000 7f43a7400640 1 --2- 172.21.15.32:0/244586256 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43a810b3d0 0x7f43a8117630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.054+0000 7f43a7400640 1 -- 172.21.15.32:0/244586256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43a810a4a0 msgr2=0x7f43a81170f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.054+0000 7f43a7400640 1 --2- 172.21.15.32:0/244586256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43a810a4a0 0x7f43a81170f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.054+0000 7f43a7400640 1 -- 172.21.15.32:0/244586256 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43a8118c00 con 0x7f43a81100f0 2024-04-30T08:38:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.054+0000 7f43a6a00640 1 --2- 172.21.15.32:0/244586256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43a810a4a0 0x7f43a81170f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-30T08:38:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.054+0000 7f43a7400640 1 --2- 172.21.15.32:0/244586256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43a81100f0 0x7f43a8118590 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f43a000da10 tx=0x7f43a000ded0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.055+0000 7f439f400640 1 -- 172.21.15.32:0/244586256 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a000bd70 con 0x7f43a81100f0 2024-04-30T08:38:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.055+0000 7f439f400640 1 -- 172.21.15.32:0/244586256 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f43a0003e20 con 0x7f43a81100f0 2024-04-30T08:38:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.055+0000 7f439f400640 1 -- 172.21.15.32:0/244586256 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a00106c0 con 0x7f43a81100f0 2024-04-30T08:38:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.055+0000 7f43ad600640 1 -- 172.21.15.32:0/244586256 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43a8118ef0 con 0x7f43a81100f0 2024-04-30T08:38:00.056 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.055+0000 7f43ad600640 1 -- 172.21.15.32:0/244586256 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43a81193f0 con 0x7f43a81100f0 2024-04-30T08:38:00.060 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.060+0000 7f43b45f3740 1 Processor -- start 2024-04-30T08:38:00.062 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.061+0000 7f439f400640 1 -- 172.21.15.32:0/244586256 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f43a0004270 con 0x7f43a81100f0 2024-04-30T08:38:00.062 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.061+0000 7f43b45f3740 1 -- start start 2024-04-30T08:38:00.062 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.061+0000 7f43b45f3740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8da8ec1b0 0x55f8dac8e0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:00.062 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.062+0000 7f43b45f3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8da8c32f0 0x55f8dac8e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:00.062 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.062+0000 7f43b45f3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8da8cb4d0 0x55f8dac95560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:00.062 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.062+0000 7f43b45f3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f8dabf2550 con 0x55f8da8cb4d0 2024-04-30T08:38:00.063 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.062+0000 7f43b45f3740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55f8dab97bc0 con 0x55f8da8ec1b0 2024-04-30T08:38:00.063 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.062+0000 7f43b45f3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f8dabee510 con 0x55f8da8c32f0 2024-04-30T08:38:00.063 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.062+0000 7f43a6a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8da8c32f0 0x55f8dac8e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:00.063 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.062+0000 7f439fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8da8ec1b0 0x55f8dac8e0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:00.063 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.062+0000 7f43a6a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8da8c32f0 0x55f8dac8e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:37060/0 (socket says 172.21.15.32:37060) 2024-04-30T08:38:00.064 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.062+0000 7f43a6a00640 1 -- 172.21.15.32:0/1468288350 learned_addr learned my addr 172.21.15.32:0/1468288350 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:00.064 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.063+0000 7f439fe00640 1 --2- 172.21.15.32:0/1468288350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8da8cb4d0 0x55f8dac95560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:00.064 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.063+0000 7f43a6a00640 1 -- 172.21.15.32:0/1468288350 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8da8ec1b0 msgr2=0x55f8dac8e0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:00.064 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.063+0000 7f43a6a00640 1 --2- 172.21.15.32:0/1468288350 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8da8ec1b0 0x55f8dac8e0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:00.064 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.063+0000 7f43a6a00640 1 -- 172.21.15.32:0/1468288350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8da8cb4d0 msgr2=0x55f8dac95560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:00.064 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.063+0000 7f43a6a00640 1 --2- 172.21.15.32:0/1468288350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8da8cb4d0 0x55f8dac95560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:00.064 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.063+0000 7f43a6a00640 1 -- 172.21.15.32:0/1468288350 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8da904210 con 0x55f8da8c32f0 2024-04-30T08:38:00.065 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.063+0000 7f439f400640 1 --2- 172.21.15.32:0/244586256 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f438c0afc30 0x7f438c0b2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:00.065 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.063+0000 7f439fe00640 1 --2- 172.21.15.32:0/1468288350 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8da8ec1b0 0x55f8dac8e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:38:00.065 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.064+0000 7f439f400640 1 -- 172.21.15.32:0/244586256 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f43a00df3f0 con 0x7f43a81100f0 2024-04-30T08:38:00.065 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.064+0000 7f439fe00640 1 --2- 172.21.15.32:0/244586256 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f438c0afc30 0x7f438c0b2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:00.066 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.064+0000 7f43a6a00640 1 --2- 172.21.15.32:0/1468288350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8da8c32f0 0x55f8dac8e660 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4398005ab0 tx=0x7f43980040c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c950fd8ca32b8f1a server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:00.066 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.065+0000 7f4397400640 1 -- 172.21.15.32:0/1468288350 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f439803f070 con 0x55f8da8c32f0 2024-04-30T08:38:00.066 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.065+0000 7f4397400640 1 -- 172.21.15.32:0/1468288350 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4398004430 con 0x55f8da8c32f0 2024-04-30T08:38:00.066 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.065+0000 7f4397400640 1 -- 172.21.15.32:0/1468288350 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4398035030 con 0x55f8da8c32f0 2024-04-30T08:38:00.067 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.066+0000 7f43b45f3740 1 -- 172.21.15.32:0/1468288350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8da8c32f0 msgr2=0x55f8dac8e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:00.067 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.066+0000 7f43b45f3740 1 --2- 172.21.15.32:0/1468288350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8da8c32f0 0x55f8dac8e660 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f4398005ab0 tx=0x7f43980040c0 comp rx=0 tx=0).stop 2024-04-30T08:38:00.067 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.066+0000 7f439fe00640 1 --2- 172.21.15.32:0/244586256 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f438c0afc30 0x7f438c0b2140 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x55f8dac8f2b0 tx=0x7f4390005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:00.067 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.066+0000 7f43b45f3740 1 -- 172.21.15.32:0/1468288350 shutdown_connections 2024-04-30T08:38:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.066+0000 7f43b45f3740 1 --2- 172.21.15.32:0/1468288350 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8da8cb4d0 0x55f8dac95560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.066+0000 7f43b45f3740 1 --2- 172.21.15.32:0/1468288350 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8da8c32f0 0x55f8dac8e660 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.066+0000 7f43b45f3740 1 --2- 172.21.15.32:0/1468288350 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8da8ec1b0 0x55f8dac8e0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.066+0000 7f43b45f3740 1 -- 172.21.15.32:0/1468288350 >> 172.21.15.32:0/1468288350 conn(0x55f8da8e0300 msgr2=0x55f8dac89a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.066+0000 7f43b45f3740 1 -- 172.21.15.32:0/1468288350 shutdown_connections 2024-04-30T08:38:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.066+0000 7f43b45f3740 1 -- 172.21.15.32:0/1468288350 wait complete. 2024-04-30T08:38:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.067+0000 7f43b45f3740 1 -- start start 2024-04-30T08:38:00.068 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.067+0000 7f43b45f3740 1 Processor -- start 2024-04-30T08:38:00.069 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.067+0000 7f43b45f3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8da8c32f0 0x55f8dac97b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.067+0000 7f43b45f3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8da8cb4d0 0x55f8dac98050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.067+0000 7f43b45f3740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8da8ec1b0 0x55f8dac9efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.067+0000 7f43b45f3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55f8dac95f50 con 0x55f8da8cb4d0 2024-04-30T08:38:00.070 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.067+0000 7f43b45f3740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55f8dabab660 con 0x55f8da8ec1b0 2024-04-30T08:38:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.067+0000 7f43b45f3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55f8dac960d0 con 0x55f8da8c32f0 2024-04-30T08:38:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.068+0000 7f43b45f3740 10 client.0 _ll_register_callbacks cb 0x55f8dab75400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:38:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.068+0000 7f43a6a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8da8c32f0 0x55f8dac97b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.068+0000 7f43a7400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8da8ec1b0 0x55f8dac9efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.068+0000 7f43a7400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8da8ec1b0 0x55f8dac9efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:60258/0 (socket says 172.21.15.32:60258) 2024-04-30T08:38:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.068+0000 7f43a7400640 1 -- 172.21.15.32:0/2621885323 learned_addr learned my addr 172.21.15.32:0/2621885323 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.069+0000 7f43a7400640 1 -- 172.21.15.32:0/2621885323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8da8c32f0 msgr2=0x55f8dac97b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.069+0000 7f43a7400640 1 --2- 172.21.15.32:0/2621885323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8da8c32f0 0x55f8dac97b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.069+0000 7f43a7400640 1 -- 172.21.15.32:0/2621885323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8da8cb4d0 msgr2=0x55f8dac98050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.069+0000 7f43a7400640 1 --2- 172.21.15.32:0/2621885323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8da8cb4d0 0x55f8dac98050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:00.071 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.069+0000 7f43a7400640 1 -- 172.21.15.32:0/2621885323 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8da9d0620 con 0x55f8da8ec1b0 2024-04-30T08:38:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.069+0000 7f43a7400640 1 --2- 172.21.15.32:0/2621885323 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8da8ec1b0 0x55f8dac9efb0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f43a0004af0 tx=0x7f43a00040c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=484d0b57d3145e8d server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:00.072 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.069+0000 7f438be00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:38:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.070+0000 7f438be00640 1 -- 172.21.15.32:0/2621885323 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a00c1060 con 0x55f8da8ec1b0 2024-04-30T08:38:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.070+0000 7f438be00640 1 -- 172.21.15.32:0/2621885323 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f43a00a3750 con 0x55f8da8ec1b0 2024-04-30T08:38:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.070+0000 7f43b45f3740 1 -- 172.21.15.32:0/2621885323 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4398004430 con 0x55f8da8ec1b0 2024-04-30T08:38:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.071+0000 7f438be00640 1 -- 172.21.15.32:0/2621885323 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f43a00c6050 con 0x55f8da8ec1b0 2024-04-30T08:38:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.071+0000 7f43b45f3740 10 client.26405 fetch_fsmap learned FSMap version 29 2024-04-30T08:38:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.071+0000 7f43b45f3740 1 -- 172.21.15.32:0/2621885323 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f8daca1fb0 con 0x55f8da8ec1b0 2024-04-30T08:38:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.071+0000 7f438be00640 1 -- 172.21.15.32:0/2621885323 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4398004430 con 0x55f8da8ec1b0 2024-04-30T08:38:00.074 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.071+0000 7f438be00640 1 -- 172.21.15.32:0/2621885323 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7f43a0024060 con 0x55f8da8ec1b0 2024-04-30T08:38:00.075 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.073+0000 7f43b45f3740 10 client.26405 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:38:00.075 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.073+0000 7f43b45f3740 10 client.26405 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-30T08:38:00.075 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.073+0000 7f43b45f3740 20 client.26405 populate_metadata read hostname 'smithi032' 2024-04-30T08:38:00.075 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.073+0000 7f43b45f3740 1 --2- 172.21.15.32:0/2621885323 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55f8da8aad80 0x55f8daca57b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:00.075 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.073+0000 7f43b45f3740 4 client.26405 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:38:00.075 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.073+0000 7f43b45f3740 1 -- 172.21.15.32:0/2621885323 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f8dac00160 con 0x55f8da8aad80 2024-04-30T08:38:00.075 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.073+0000 7f43a6a00640 1 --2- 172.21.15.32:0/2621885323 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55f8da8aad80 0x55f8daca57b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:00.076 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.074+0000 7f438be00640 10 client.26405 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:38:00.076 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.075+0000 7f43a6a00640 1 --2- 172.21.15.32:0/2621885323 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55f8da8aad80 0x55f8daca57b0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d417d8d202db8cbf server_cookie=b51e3cf7d3f34ea9 in_seq=0 out_seq=0 2024-04-30T08:38:00.078 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.076+0000 7f438be00640 1 -- 172.21.15.32:0/2621885323 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55f8dac00160 con 0x55f8da8aad80 2024-04-30T08:38:00.078 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.076+0000 7f438be00640 10 client.26405 handle_command_reply: tid=0 2024-04-30T08:38:00.078 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.076+0000 7f43b45f3740 1 client.26405 shutdown 2024-04-30T08:38:00.080 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.079+0000 7f43b45f3740 1 -- 172.21.15.32:0/2621885323 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8da8ec1b0 msgr2=0x55f8dac9efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:00.080 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.079+0000 7f43b45f3740 1 --2- 172.21.15.32:0/2621885323 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8da8ec1b0 0x55f8dac9efb0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f43a0004af0 tx=0x7f43a00040c0 comp rx=0 tx=0).stop 2024-04-30T08:38:00.080 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.079+0000 7f43b45f3740 1 -- 172.21.15.32:0/2621885323 shutdown_connections 2024-04-30T08:38:00.080 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.079+0000 7f43b45f3740 1 --2- 172.21.15.32:0/2621885323 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55f8da8aad80 0x55f8daca57b0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:00.080 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.079+0000 7f43b45f3740 1 --2- 172.21.15.32:0/2621885323 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8da8ec1b0 0x55f8dac9efb0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:00.080 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.079+0000 7f43b45f3740 1 --2- 172.21.15.32:0/2621885323 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8da8cb4d0 0x55f8dac98050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:00.080 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.079+0000 7f43b45f3740 1 --2- 172.21.15.32:0/2621885323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8da8c32f0 0x55f8dac97b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:00.080 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.079+0000 7f43b45f3740 1 -- 172.21.15.32:0/2621885323 >> 172.21.15.32:0/2621885323 conn(0x55f8da8e0300 msgr2=0x55f8dac8c980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:00.080 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.079+0000 7f438be00640 0 client.26405 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:38:00.081 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.080+0000 7f43b45f3740 1 -- 172.21.15.32:0/2621885323 shutdown_connections 2024-04-30T08:38:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.080+0000 7f43b45f3740 1 -- 172.21.15.32:0/2621885323 wait complete. 2024-04-30T08:38:00.082 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.080+0000 7f43b45f3740 20 client.26405 trim_cache size 0 max 16384 2024-04-30T08:38:00.124 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.123+0000 7f43b45f3740 1 Processor -- start 2024-04-30T08:38:00.124 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.123+0000 7f43b45f3740 1 -- start start 2024-04-30T08:38:00.124 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.123+0000 7f43b45f3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8da8c32f0 0x55f8dac87fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:00.124 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.123+0000 7f43b45f3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8da8cb4d0 0x55f8dac88520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:00.124 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.123+0000 7f43b45f3740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8da8ec1b0 0x55f8dac88a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:00.124 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.123+0000 7f43b45f3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f43a81c4120 con 0x55f8da8c32f0 2024-04-30T08:38:00.124 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.123+0000 7f43b45f3740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f43a81c5cf0 con 0x55f8da8ec1b0 2024-04-30T08:38:00.124 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.123+0000 7f43b45f3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f43a81c47f0 con 0x55f8da8cb4d0 2024-04-30T08:38:00.125 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.124+0000 7f43a6a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8da8c32f0 0x55f8dac87fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:00.125 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.124+0000 7f439fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8da8cb4d0 0x55f8dac88520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:00.125 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.124+0000 7f439fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8da8cb4d0 0x55f8dac88520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:37082/0 (socket says 172.21.15.32:37082) 2024-04-30T08:38:00.125 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.124+0000 7f439fe00640 1 -- 172.21.15.32:0/2622984791 learned_addr learned my addr 172.21.15.32:0/2622984791 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:00.126 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.124+0000 7f43a7400640 1 --2- 172.21.15.32:0/2622984791 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8da8ec1b0 0x55f8dac88a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:00.126 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.124+0000 7f439fe00640 1 -- 172.21.15.32:0/2622984791 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8da8ec1b0 msgr2=0x55f8dac88a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:00.126 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.124+0000 7f439fe00640 1 --2- 172.21.15.32:0/2622984791 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8da8ec1b0 0x55f8dac88a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:00.126 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.124+0000 7f439fe00640 1 -- 172.21.15.32:0/2622984791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8da8c32f0 msgr2=0x55f8dac87fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:00.126 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.124+0000 7f439fe00640 1 --2- 172.21.15.32:0/2622984791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8da8c32f0 0x55f8dac87fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:00.126 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.124+0000 7f439fe00640 1 -- 172.21.15.32:0/2622984791 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8dac85260 con 0x55f8da8cb4d0 2024-04-30T08:38:00.126 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.124+0000 7f43a6a00640 1 --2- 172.21.15.32:0/2622984791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8da8c32f0 0x55f8dac87fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:38:00.126 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.125+0000 7f439fe00640 1 --2- 172.21.15.32:0/2622984791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8da8cb4d0 0x55f8dac88520 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4390012940 tx=0x7f4390009720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91dbe979320cc922 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:00.126 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.125+0000 7f4396a00640 1 -- 172.21.15.32:0/2622984791 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f439002e070 con 0x55f8da8cb4d0 2024-04-30T08:38:00.126 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.125+0000 7f4396a00640 1 -- 172.21.15.32:0/2622984791 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4390021dc0 con 0x55f8da8cb4d0 2024-04-30T08:38:00.126 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.125+0000 7f4396a00640 1 -- 172.21.15.32:0/2622984791 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f439001fbe0 con 0x55f8da8cb4d0 2024-04-30T08:38:00.127 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.126+0000 7f43b45f3740 1 -- 172.21.15.32:0/2622984791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8da8cb4d0 msgr2=0x55f8dac88520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:00.127 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.126+0000 7f43b45f3740 1 --2- 172.21.15.32:0/2622984791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8da8cb4d0 0x55f8dac88520 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4390012940 tx=0x7f4390009720 comp rx=0 tx=0).stop 2024-04-30T08:38:00.127 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.126+0000 7f43b45f3740 1 -- 172.21.15.32:0/2622984791 shutdown_connections 2024-04-30T08:38:00.127 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.126+0000 7f43b45f3740 1 --2- 172.21.15.32:0/2622984791 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8da8ec1b0 0x55f8dac88a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:00.127 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.126+0000 7f43b45f3740 1 --2- 172.21.15.32:0/2622984791 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8da8cb4d0 0x55f8dac88520 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:00.128 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.126+0000 7f43b45f3740 1 --2- 172.21.15.32:0/2622984791 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8da8c32f0 0x55f8dac87fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:00.128 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.126+0000 7f43b45f3740 1 -- 172.21.15.32:0/2622984791 >> 172.21.15.32:0/2622984791 conn(0x55f8da8e0300 msgr2=0x55f8dac8e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:00.128 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.126+0000 7f43b45f3740 1 -- 172.21.15.32:0/2622984791 shutdown_connections 2024-04-30T08:38:00.128 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.126+0000 7f43b45f3740 1 -- 172.21.15.32:0/2622984791 wait complete. 2024-04-30T08:38:00.128 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.127+0000 7f43b45f3740 1 -- start start 2024-04-30T08:38:00.129 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.127+0000 7f43b45f3740 1 Processor -- start 2024-04-30T08:38:00.129 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.128+0000 7f43b45f3740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8da8c32f0 0x55f8dac9d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:00.130 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.128+0000 7f43b45f3740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8da8cb4d0 0x55f8dac9d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:00.130 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.128+0000 7f43b45f3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8da8ec1b0 0x55f8dac9e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:00.130 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.128+0000 7f43b45f3740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f43a81c5590 con 0x55f8da8cb4d0 2024-04-30T08:38:00.130 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.128+0000 7f43b45f3740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f43a81c4ec0 con 0x55f8da8c32f0 2024-04-30T08:38:00.130 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.128+0000 7f43b45f3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f43a81c20e0 con 0x55f8da8ec1b0 2024-04-30T08:38:00.130 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.128+0000 7f43b45f3740 10 client.0 _ll_register_callbacks cb 0x7f43a0004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:38:00.130 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.128+0000 7f43a6a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8da8c32f0 0x55f8dac9d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:00.130 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.128+0000 7f43a7400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8da8ec1b0 0x55f8dac9e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:00.130 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.128+0000 7f43a6a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8da8c32f0 0x55f8dac9d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:60274/0 (socket says 172.21.15.32:60274) 2024-04-30T08:38:00.130 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.128+0000 7f43a6a00640 1 -- 172.21.15.32:0/2348929658 learned_addr learned my addr 172.21.15.32:0/2348929658 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:00.130 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.128+0000 7f439fe00640 1 --2- 172.21.15.32:0/2348929658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8da8cb4d0 0x55f8dac9d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:00.130 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.129+0000 7f43a7400640 1 -- 172.21.15.32:0/2348929658 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8da8c32f0 msgr2=0x55f8dac9d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:00.131 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.129+0000 7f43a7400640 1 --2- 172.21.15.32:0/2348929658 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8da8c32f0 0x55f8dac9d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:00.131 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.129+0000 7f43a7400640 1 -- 172.21.15.32:0/2348929658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8da8cb4d0 msgr2=0x55f8dac9d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:00.131 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.129+0000 7f43a7400640 1 --2- 172.21.15.32:0/2348929658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8da8cb4d0 0x55f8dac9d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:00.131 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.129+0000 7f43a7400640 1 -- 172.21.15.32:0/2348929658 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8dac9eb00 con 0x55f8da8ec1b0 2024-04-30T08:38:00.131 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.129+0000 7f439fe00640 1 --2- 172.21.15.32:0/2348929658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8da8cb4d0 0x55f8dac9d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:38:00.131 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.129+0000 7f43a7400640 1 --2- 172.21.15.32:0/2348929658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8da8ec1b0 0x55f8dac9e540 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f43a00045c0 tx=0x7f43a00c8360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9951077dc8440d3b server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:00.131 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.129+0000 7f438be00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-30T08:38:00.132 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.129+0000 7f438be00640 1 -- 172.21.15.32:0/2348929658 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43a00b4030 con 0x55f8da8ec1b0 2024-04-30T08:38:00.132 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.129+0000 7f43b45f3740 1 -- 172.21.15.32:0/2348929658 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4390021dc0 con 0x55f8da8ec1b0 2024-04-30T08:38:00.132 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.129+0000 7f438be00640 1 -- 172.21.15.32:0/2348929658 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f43a00a3ab0 con 0x55f8da8ec1b0 2024-04-30T08:38:00.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.130+0000 7f438be00640 1 -- 172.21.15.32:0/2348929658 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f43a00a6c30 con 0x55f8da8ec1b0 2024-04-30T08:38:00.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.130+0000 7f43b45f3740 10 client.26404 fetch_fsmap learned FSMap version 29 2024-04-30T08:38:00.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.130+0000 7f43b45f3740 1 -- 172.21.15.32:0/2348929658 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55f8dad0dcc0 con 0x55f8da8ec1b0 2024-04-30T08:38:00.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.130+0000 7f438be00640 1 -- 172.21.15.32:0/2348929658 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4390021dc0 con 0x55f8da8ec1b0 2024-04-30T08:38:00.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.130+0000 7f438be00640 1 -- 172.21.15.32:0/2348929658 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7f43a00a6720 con 0x55f8da8ec1b0 2024-04-30T08:38:00.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.130+0000 7f43b45f3740 10 client.26404 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:38:00.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.130+0000 7f43b45f3740 10 client.26404 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-30T08:38:00.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.130+0000 7f43b45f3740 20 client.26404 populate_metadata read hostname 'smithi032' 2024-04-30T08:38:00.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.130+0000 7f43b45f3740 1 --2- 172.21.15.32:0/2348929658 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55f8da8801c0 0x55f8dad105c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:00.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.130+0000 7f43b45f3740 4 client.26404 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-30T08:38:00.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.130+0000 7f43b45f3740 1 -- 172.21.15.32:0/2348929658 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f8dabf9fd0 con 0x55f8da8801c0 2024-04-30T08:38:00.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.131+0000 7f43a6a00640 1 --2- 172.21.15.32:0/2348929658 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55f8da8801c0 0x55f8dad105c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:00.133 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.131+0000 7f43a6a00640 1 --2- 172.21.15.32:0/2348929658 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55f8da8801c0 0x55f8dad105c0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=135328b923e41b7 server_cookie=f899952173b91152 in_seq=0 out_seq=0 2024-04-30T08:38:00.134 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.131+0000 7f438be00640 10 client.26404 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:38:00.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.194+0000 7f438be00640 1 -- 172.21.15.32:0/2348929658 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f8dabf9fd0 con 0x55f8da8801c0 2024-04-30T08:38:00.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.194+0000 7f438be00640 10 client.26404 handle_command_reply: tid=0 2024-04-30T08:38:00.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.194+0000 7f43b45f3740 1 client.26404 shutdown 2024-04-30T08:38:00.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.194+0000 7f43b45f3740 1 -- 172.21.15.32:0/2348929658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8da8ec1b0 msgr2=0x55f8dac9e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:00.195 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.194+0000 7f43b45f3740 1 --2- 172.21.15.32:0/2348929658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8da8ec1b0 0x55f8dac9e540 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f43a00045c0 tx=0x7f43a00c8360 comp rx=0 tx=0).stop 2024-04-30T08:38:00.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.195+0000 7f43b45f3740 1 -- 172.21.15.32:0/2348929658 shutdown_connections 2024-04-30T08:38:00.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.195+0000 7f43b45f3740 1 --2- 172.21.15.32:0/2348929658 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55f8da8801c0 0x55f8dad105c0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:00.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.195+0000 7f43b45f3740 1 --2- 172.21.15.32:0/2348929658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f8da8ec1b0 0x55f8dac9e540 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:00.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.195+0000 7f43b45f3740 1 --2- 172.21.15.32:0/2348929658 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55f8da8cb4d0 0x55f8dac9d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:00.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.195+0000 7f43b45f3740 1 --2- 172.21.15.32:0/2348929658 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55f8da8c32f0 0x55f8dac9d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:00.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.195+0000 7f438be00640 0 client.26404 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:38:00.196 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.195+0000 7f43b45f3740 1 -- 172.21.15.32:0/2348929658 >> 172.21.15.32:0/2348929658 conn(0x55f8da8e0300 msgr2=0x55f8dac87fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:00.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.196+0000 7f43b45f3740 1 -- 172.21.15.32:0/2348929658 shutdown_connections 2024-04-30T08:38:00.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.196+0000 7f43b45f3740 1 -- 172.21.15.32:0/2348929658 wait complete. 2024-04-30T08:38:00.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.196+0000 7f43b45f3740 20 client.26404 trim_cache size 0 max 16384 2024-04-30T08:38:00.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-30T08:38:00.197 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-30T08:38:00.198 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-30T08:38:00.198 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-30T08:38:00.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.198+0000 7f43ad600640 1 -- 172.21.15.32:0/244586256 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f438c0afc30 msgr2=0x7f438c0b2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:00.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.198+0000 7f43ad600640 1 --2- 172.21.15.32:0/244586256 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f438c0afc30 0x7f438c0b2140 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x55f8dac8f2b0 tx=0x7f4390005f90 comp rx=0 tx=0).stop 2024-04-30T08:38:00.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.198+0000 7f43ad600640 1 -- 172.21.15.32:0/244586256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43a81100f0 msgr2=0x7f43a8118590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:00.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.198+0000 7f43ad600640 1 --2- 172.21.15.32:0/244586256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43a81100f0 0x7f43a8118590 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f43a000da10 tx=0x7f43a000ded0 comp rx=0 tx=0).stop 2024-04-30T08:38:00.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.198+0000 7f43ad600640 1 -- 172.21.15.32:0/244586256 shutdown_connections 2024-04-30T08:38:00.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.198+0000 7f43ad600640 1 --2- 172.21.15.32:0/244586256 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f438c0afc30 0x7f438c0b2140 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:00.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.198+0000 7f43ad600640 1 --2- 172.21.15.32:0/244586256 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f43a81100f0 0x7f43a8118590 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:00.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.198+0000 7f43ad600640 1 --2- 172.21.15.32:0/244586256 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f43a810b3d0 0x7f43a8117630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:00.199 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.198+0000 7f43ad600640 1 --2- 172.21.15.32:0/244586256 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f43a810a4a0 0x7f43a81170f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:00.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.198+0000 7f43ad600640 1 -- 172.21.15.32:0/244586256 >> 172.21.15.32:0/244586256 conn(0x7f43a80705c0 msgr2=0x7f43a810c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:00.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.198+0000 7f43ad600640 1 -- 172.21.15.32:0/244586256 shutdown_connections 2024-04-30T08:38:00.200 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:00.198+0000 7f43ad600640 1 -- 172.21.15.32:0/244586256 wait complete. 2024-04-30T08:38:01.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:01 smithi189 ceph-mon[33038]: pgmap v510: 129 pgs: 129 active+clean; 8.6 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 254 MiB/s wr, 117 op/s 2024-04-30T08:38:01.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:01 smithi032 ceph-mon[28331]: pgmap v510: 129 pgs: 129 active+clean; 8.6 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 254 MiB/s wr, 117 op/s 2024-04-30T08:38:01.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:01 smithi071 ceph-mon[33385]: pgmap v510: 129 pgs: 129 active+clean; 8.6 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 254 MiB/s wr, 117 op/s 2024-04-30T08:38:02.218 INFO:tasks.workunit.client.0.smithi032.stdout:fs setup took 49 secs 2024-04-30T08:38:03.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:03 smithi189 ceph-mon[33038]: pgmap v511: 129 pgs: 129 active+clean; 9.2 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 247 MiB/s wr, 104 op/s 2024-04-30T08:38:03.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:03 smithi032 ceph-mon[28331]: pgmap v511: 129 pgs: 129 active+clean; 9.2 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 247 MiB/s wr, 104 op/s 2024-04-30T08:38:03.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:03 smithi071 ceph-mon[33385]: pgmap v511: 129 pgs: 129 active+clean; 9.2 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 247 MiB/s wr, 104 op/s 2024-04-30T08:38:05.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:05 smithi189 ceph-mon[33038]: pgmap v512: 129 pgs: 129 active+clean; 9.6 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 231 MiB/s wr, 106 op/s 2024-04-30T08:38:05.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:05 smithi032 ceph-mon[28331]: pgmap v512: 129 pgs: 129 active+clean; 9.6 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 231 MiB/s wr, 106 op/s 2024-04-30T08:38:05.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:05 smithi071 ceph-mon[33385]: pgmap v512: 129 pgs: 129 active+clean; 9.6 GiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 231 MiB/s wr, 106 op/s 2024-04-30T08:38:07.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:07 smithi189 ceph-mon[33038]: pgmap v513: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 235 MiB/s wr, 101 op/s 2024-04-30T08:38:07.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:07 smithi032 ceph-mon[28331]: pgmap v513: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 235 MiB/s wr, 101 op/s 2024-04-30T08:38:07.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:07 smithi071 ceph-mon[33385]: pgmap v513: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 235 MiB/s wr, 101 op/s 2024-04-30T08:38:09.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:09 smithi032 ceph-mon[28331]: pgmap v514: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 225 MiB/s wr, 98 op/s 2024-04-30T08:38:09.702 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-30T08:38:09.702 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-30T08:38:09.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:09 smithi071 ceph-mon[33385]: pgmap v514: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 225 MiB/s wr, 98 op/s 2024-04-30T08:38:09.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:09 smithi189 ceph-mon[33038]: pgmap v514: 129 pgs: 129 active+clean; 10 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 225 MiB/s wr, 98 op/s 2024-04-30T08:38:10.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.256+0000 7f5946200640 1 -- 172.21.15.32:0/2954006319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5940105d70 msgr2=0x7f5940106a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:10.258 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.256+0000 7f5946200640 1 --2- 172.21.15.32:0/2954006319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5940105d70 0x7f5940106a70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5930009900 tx=0x7f593002a3f0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.257+0000 7f5946200640 1 -- 172.21.15.32:0/2954006319 shutdown_connections 2024-04-30T08:38:10.259 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.257+0000 7f5946200640 1 --2- 172.21.15.32:0/2954006319 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f594010bcb0 0x7f594010e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.257+0000 7f5946200640 1 --2- 172.21.15.32:0/2954006319 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5940106fb0 0x7f59401073c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.257+0000 7f5946200640 1 --2- 172.21.15.32:0/2954006319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5940105d70 0x7f5940106a70 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.258+0000 7f5946200640 1 -- 172.21.15.32:0/2954006319 >> 172.21.15.32:0/2954006319 conn(0x7f5940100fa0 msgr2=0x7f5940103410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:10.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.258+0000 7f5946200640 1 -- 172.21.15.32:0/2954006319 shutdown_connections 2024-04-30T08:38:10.260 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.258+0000 7f5946200640 1 -- 172.21.15.32:0/2954006319 wait complete. 2024-04-30T08:38:10.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.259+0000 7f5946200640 1 Processor -- start 2024-04-30T08:38:10.261 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.260+0000 7f5946200640 1 -- start start 2024-04-30T08:38:10.262 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.260+0000 7f5946200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5940105d70 0x7f594019c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:10.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.260+0000 7f5946200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5940106fb0 0x7f594019ca00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:10.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.260+0000 7f5946200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f594010bcb0 0x7f59401a3990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:10.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.260+0000 7f5946200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f59401118c0 con 0x7f5940106fb0 2024-04-30T08:38:10.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.260+0000 7f5946200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5940111740 con 0x7f5940105d70 2024-04-30T08:38:10.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.260+0000 7f5946200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5940111a40 con 0x7f594010bcb0 2024-04-30T08:38:10.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.261+0000 7f593f400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5940105d70 0x7f594019c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:10.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.261+0000 7f593f400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5940105d70 0x7f594019c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:36370/0 (socket says 172.21.15.32:36370) 2024-04-30T08:38:10.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.261+0000 7f593f400640 1 -- 172.21.15.32:0/4129500378 learned_addr learned my addr 172.21.15.32:0/4129500378 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:10.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.261+0000 7f593fe00640 1 --2- 172.21.15.32:0/4129500378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f594010bcb0 0x7f59401a3990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:10.263 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.261+0000 7f593ea00640 1 --2- 172.21.15.32:0/4129500378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5940106fb0 0x7f594019ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:10.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.262+0000 7f593fe00640 1 -- 172.21.15.32:0/4129500378 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5940105d70 msgr2=0x7f594019c4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:10.264 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.262+0000 7f593fe00640 1 --2- 172.21.15.32:0/4129500378 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5940105d70 0x7f594019c4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.262+0000 7f593fe00640 1 -- 172.21.15.32:0/4129500378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5940106fb0 msgr2=0x7f594019ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:10.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.262+0000 7f593fe00640 1 --2- 172.21.15.32:0/4129500378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5940106fb0 0x7f594019ca00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.262+0000 7f593fe00640 1 -- 172.21.15.32:0/4129500378 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59401a4000 con 0x7f594010bcb0 2024-04-30T08:38:10.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.262+0000 7f593f400640 1 --2- 172.21.15.32:0/4129500378 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5940105d70 0x7f594019c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:38:10.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.262+0000 7f593fe00640 1 --2- 172.21.15.32:0/4129500378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f594010bcb0 0x7f59401a3990 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5938009fd0 tx=0x7f593800ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:10.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.263+0000 7f5937e00640 1 -- 172.21.15.32:0/4129500378 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f593800ce20 con 0x7f594010bcb0 2024-04-30T08:38:10.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.263+0000 7f5937e00640 1 -- 172.21.15.32:0/4129500378 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5938005020 con 0x7f594010bcb0 2024-04-30T08:38:10.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.263+0000 7f5937e00640 1 -- 172.21.15.32:0/4129500378 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5938010690 con 0x7f594010bcb0 2024-04-30T08:38:10.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.263+0000 7f5946200640 1 -- 172.21.15.32:0/4129500378 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59400781d0 con 0x7f594010bcb0 2024-04-30T08:38:10.265 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.263+0000 7f5946200640 1 -- 172.21.15.32:0/4129500378 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f59400786d0 con 0x7f594010bcb0 2024-04-30T08:38:10.268 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.267+0000 7f594d271740 1 Processor -- start 2024-04-30T08:38:10.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.268+0000 7f5937e00640 1 -- 172.21.15.32:0/4129500378 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f59380054d0 con 0x7f594010bcb0 2024-04-30T08:38:10.269 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.268+0000 7f594d271740 1 -- start start 2024-04-30T08:38:10.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.268+0000 7f594d271740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557924a8b390 0x557924e2c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:10.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.268+0000 7f594d271740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557924a624d0 0x557924e2c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:10.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.268+0000 7f594d271740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557924a6a6b0 0x557924e336c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:10.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.268+0000 7f594d271740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x557924d91d50 con 0x557924a624d0 2024-04-30T08:38:10.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.268+0000 7f594d271740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x557924d36ef0 con 0x557924a6a6b0 2024-04-30T08:38:10.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.268+0000 7f594d271740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x557924d8dd10 con 0x557924a8b390 2024-04-30T08:38:10.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.269+0000 7f593ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557924a8b390 0x557924e2c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:10.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.269+0000 7f593ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557924a8b390 0x557924e2c240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:32786/0 (socket says 172.21.15.32:32786) 2024-04-30T08:38:10.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.269+0000 7f593f400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557924a624d0 0x557924e2c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:10.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.269+0000 7f593ea00640 1 -- 172.21.15.32:0/1788249117 learned_addr learned my addr 172.21.15.32:0/1788249117 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:10.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.269+0000 7f593ea00640 1 --2- 172.21.15.32:0/1788249117 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557924a6a6b0 0x557924e336c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:10.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.269+0000 7f593ea00640 1 -- 172.21.15.32:0/1788249117 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557924a6a6b0 msgr2=0x557924e336c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:10.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.269+0000 7f593ea00640 1 --2- 172.21.15.32:0/1788249117 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557924a6a6b0 0x557924e336c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.269+0000 7f593ea00640 1 -- 172.21.15.32:0/1788249117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557924a624d0 msgr2=0x557924e2c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:10.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.269+0000 7f593ea00640 1 --2- 172.21.15.32:0/1788249117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557924a624d0 0x557924e2c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.269+0000 7f593ea00640 1 -- 172.21.15.32:0/1788249117 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557924b6d7d0 con 0x557924a8b390 2024-04-30T08:38:10.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.269+0000 7f593f400640 1 --2- 172.21.15.32:0/1788249117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557924a624d0 0x557924e2c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:38:10.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.270+0000 7f593ea00640 1 --2- 172.21.15.32:0/1788249117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557924a8b390 0x557924e2c240 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f592c009460 tx=0x7f592c0095f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=857b002f86a8b0b1 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:10.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.270+0000 7f5937e00640 1 --2- 172.21.15.32:0/4129500378 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f59240afc40 0x7f59240b2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:10.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.270+0000 7f592be00640 1 -- 172.21.15.32:0/1788249117 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f592c0150f0 con 0x557924a8b390 2024-04-30T08:38:10.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.270+0000 7f592be00640 1 -- 172.21.15.32:0/1788249117 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f592c00e6f0 con 0x557924a8b390 2024-04-30T08:38:10.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.270+0000 7f592be00640 1 -- 172.21.15.32:0/1788249117 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f592c003810 con 0x557924a8b390 2024-04-30T08:38:10.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.270+0000 7f5937e00640 1 -- 172.21.15.32:0/4129500378 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f59380df660 con 0x7f594010bcb0 2024-04-30T08:38:10.273 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.270+0000 7f593f400640 1 --2- 172.21.15.32:0/4129500378 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f59240afc40 0x7f59240b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:10.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.271+0000 7f594d271740 1 -- 172.21.15.32:0/1788249117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557924a8b390 msgr2=0x557924e2c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:10.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.271+0000 7f594d271740 1 --2- 172.21.15.32:0/1788249117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557924a8b390 0x557924e2c240 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f592c009460 tx=0x7f592c0095f0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.271+0000 7f594d271740 1 -- 172.21.15.32:0/1788249117 shutdown_connections 2024-04-30T08:38:10.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.271+0000 7f594d271740 1 --2- 172.21.15.32:0/1788249117 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557924a6a6b0 0x557924e336c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.271+0000 7f594d271740 1 --2- 172.21.15.32:0/1788249117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557924a624d0 0x557924e2c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.271+0000 7f594d271740 1 --2- 172.21.15.32:0/1788249117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557924a8b390 0x557924e2c240 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.271+0000 7f594d271740 1 -- 172.21.15.32:0/1788249117 >> 172.21.15.32:0/1788249117 conn(0x557924a7f4e0 msgr2=0x557924e27bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:10.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.271+0000 7f594d271740 1 -- 172.21.15.32:0/1788249117 shutdown_connections 2024-04-30T08:38:10.275 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.271+0000 7f594d271740 1 -- 172.21.15.32:0/1788249117 wait complete. 2024-04-30T08:38:10.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.271+0000 7f593f400640 1 --2- 172.21.15.32:0/4129500378 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f59240afc40 0x7f59240b2150 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f5930000c00 tx=0x7f59300314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:10.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.272+0000 7f594d271740 1 -- start start 2024-04-30T08:38:10.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.272+0000 7f594d271740 1 Processor -- start 2024-04-30T08:38:10.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.273+0000 7f594d271740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557924a624d0 0x557924e36090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:10.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.273+0000 7f594d271740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557924a6a6b0 0x557924e365d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:10.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.273+0000 7f594d271740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557924a8b390 0x557924e3d530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:10.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.273+0000 7f594d271740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x557924e340b0 con 0x557924a8b390 2024-04-30T08:38:10.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.273+0000 7f594d271740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x557924d4a990 con 0x557924a6a6b0 2024-04-30T08:38:10.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.273+0000 7f594d271740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x557924e34230 con 0x557924a624d0 2024-04-30T08:38:10.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.273+0000 7f594d271740 10 client.0 _ll_register_callbacks cb 0x557924d49110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:38:10.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.273+0000 7f593f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557924a624d0 0x557924e36090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:10.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.273+0000 7f593ea00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557924a6a6b0 0x557924e365d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:10.276 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.273+0000 7f593fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557924a8b390 0x557924e3d530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:10.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.273+0000 7f593f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557924a624d0 0x557924e36090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:32800/0 (socket says 172.21.15.32:32800) 2024-04-30T08:38:10.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.273+0000 7f593ea00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557924a6a6b0 0x557924e365d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:36390/0 (socket says 172.21.15.32:36390) 2024-04-30T08:38:10.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.273+0000 7f593f400640 1 -- 172.21.15.32:0/1989711077 learned_addr learned my addr 172.21.15.32:0/1989711077 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:10.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.274+0000 7f593fe00640 1 -- 172.21.15.32:0/1989711077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557924a624d0 msgr2=0x557924e36090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:10.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.274+0000 7f593fe00640 1 --2- 172.21.15.32:0/1989711077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557924a624d0 0x557924e36090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.274+0000 7f593fe00640 1 -- 172.21.15.32:0/1989711077 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557924a6a6b0 msgr2=0x557924e365d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:10.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.274+0000 7f593fe00640 1 --2- 172.21.15.32:0/1989711077 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557924a6a6b0 0x557924e365d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.274+0000 7f593fe00640 1 -- 172.21.15.32:0/1989711077 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557924aa33f0 con 0x557924a8b390 2024-04-30T08:38:10.277 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.274+0000 7f593f400640 1 --2- 172.21.15.32:0/1989711077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557924a624d0 0x557924e36090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:38:10.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.274+0000 7f593fe00640 1 --2- 172.21.15.32:0/1989711077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557924a8b390 0x557924e3d530 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f59380a7430 tx=0x7f59380a7ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a71d7f8a66cd7625 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:10.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.274+0000 7f5928c00640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:38:10.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.274+0000 7f5928c00640 1 -- 172.21.15.32:0/1989711077 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59380b0590 con 0x557924a8b390 2024-04-30T08:38:10.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.274+0000 7f5928c00640 1 -- 172.21.15.32:0/1989711077 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f59380a3d20 con 0x557924a8b390 2024-04-30T08:38:10.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.274+0000 7f594d271740 1 -- 172.21.15.32:0/1989711077 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f592c00e6f0 con 0x557924a8b390 2024-04-30T08:38:10.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.275+0000 7f5928c00640 1 -- 172.21.15.32:0/1989711077 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f59380df350 con 0x557924a8b390 2024-04-30T08:38:10.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.275+0000 7f594d271740 10 client.16263 fetch_fsmap learned FSMap version 29 2024-04-30T08:38:10.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.275+0000 7f594d271740 1 -- 172.21.15.32:0/1989711077 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557924a630f0 con 0x557924a8b390 2024-04-30T08:38:10.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.275+0000 7f5928c00640 1 -- 172.21.15.32:0/1989711077 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f592c00e6f0 con 0x557924a8b390 2024-04-30T08:38:10.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.275+0000 7f5928c00640 1 -- 172.21.15.32:0/1989711077 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7f59380a30a0 con 0x557924a8b390 2024-04-30T08:38:10.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.276+0000 7f594d271740 10 client.16263 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:38:10.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.276+0000 7f594d271740 10 client.16263 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:38:10.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.276+0000 7f594d271740 20 client.16263 populate_metadata read hostname 'smithi032' 2024-04-30T08:38:10.279 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.276+0000 7f594d271740 1 --2- 172.21.15.32:0/1989711077 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x557924a49f60 0x557924e43ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:10.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.276+0000 7f594d271740 4 client.16263 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:38:10.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.276+0000 7f594d271740 1 -- 172.21.15.32:0/1989711077 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557924da5ef0 con 0x557924a49f60 2024-04-30T08:38:10.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.276+0000 7f593f400640 1 --2- 172.21.15.32:0/1989711077 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x557924a49f60 0x557924e43ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:10.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.278+0000 7f5928c00640 10 client.16263 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:38:10.280 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.278+0000 7f593f400640 1 --2- 172.21.15.32:0/1989711077 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x557924a49f60 0x557924e43ab0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59074511f4738f1f server_cookie=651e8d31dd16d4c6 in_seq=0 out_seq=0 2024-04-30T08:38:10.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.278+0000 7f5928c00640 1 -- 172.21.15.32:0/1989711077 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x557924da5ef0 con 0x557924a49f60 2024-04-30T08:38:10.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.278+0000 7f5928c00640 10 client.16263 handle_command_reply: tid=0 2024-04-30T08:38:10.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.279+0000 7f594d271740 1 client.16263 shutdown 2024-04-30T08:38:10.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.279+0000 7f594d271740 1 -- 172.21.15.32:0/1989711077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557924a8b390 msgr2=0x557924e3d530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:10.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.279+0000 7f594d271740 1 --2- 172.21.15.32:0/1989711077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557924a8b390 0x557924e3d530 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f59380a7430 tx=0x7f59380a7ef0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.279+0000 7f594d271740 1 -- 172.21.15.32:0/1989711077 shutdown_connections 2024-04-30T08:38:10.281 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.279+0000 7f594d271740 1 --2- 172.21.15.32:0/1989711077 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x557924a49f60 0x557924e43ab0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.279+0000 7f594d271740 1 --2- 172.21.15.32:0/1989711077 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557924a8b390 0x557924e3d530 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.279+0000 7f594d271740 1 --2- 172.21.15.32:0/1989711077 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557924a6a6b0 0x557924e365d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.279+0000 7f594d271740 1 --2- 172.21.15.32:0/1989711077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557924a624d0 0x557924e36090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.279+0000 7f594d271740 1 -- 172.21.15.32:0/1989711077 >> 172.21.15.32:0/1989711077 conn(0x557924a7f4e0 msgr2=0x557924e26bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:10.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.280+0000 7f5928c00640 0 client.16263 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:38:10.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.280+0000 7f594d271740 1 -- 172.21.15.32:0/1989711077 shutdown_connections 2024-04-30T08:38:10.282 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.281+0000 7f594d271740 1 -- 172.21.15.32:0/1989711077 wait complete. 2024-04-30T08:38:10.283 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.282+0000 7f594d271740 20 client.16263 trim_cache size 0 max 16384 2024-04-30T08:38:10.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.318+0000 7f594d271740 1 Processor -- start 2024-04-30T08:38:10.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.319+0000 7f594d271740 1 -- start start 2024-04-30T08:38:10.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.319+0000 7f594d271740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557924a624d0 0x557924e345d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:10.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.319+0000 7f594d271740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557924a6a6b0 0x557924e3d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:10.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.319+0000 7f594d271740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557924a8b390 0x557924e34b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:10.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.319+0000 7f594d271740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f59401b3160 con 0x557924a6a6b0 2024-04-30T08:38:10.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.319+0000 7f594d271740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f59401b4d30 con 0x557924a624d0 2024-04-30T08:38:10.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.319+0000 7f594d271740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f59401b3830 con 0x557924a8b390 2024-04-30T08:38:10.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.320+0000 7f593ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557924a6a6b0 0x557924e3d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:10.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.321+0000 7f593fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557924a8b390 0x557924e34b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:10.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.321+0000 7f593fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557924a8b390 0x557924e34b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:32804/0 (socket says 172.21.15.32:32804) 2024-04-30T08:38:10.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.321+0000 7f593fe00640 1 -- 172.21.15.32:0/166498136 learned_addr learned my addr 172.21.15.32:0/166498136 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:10.323 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.322+0000 7f593f400640 1 --2- 172.21.15.32:0/166498136 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557924a624d0 0x557924e345d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:10.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.322+0000 7f593ea00640 1 -- 172.21.15.32:0/166498136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557924a8b390 msgr2=0x557924e34b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:10.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.322+0000 7f593ea00640 1 --2- 172.21.15.32:0/166498136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557924a8b390 0x557924e34b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.322+0000 7f593ea00640 1 -- 172.21.15.32:0/166498136 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557924a624d0 msgr2=0x557924e345d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:10.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.322+0000 7f593ea00640 1 --2- 172.21.15.32:0/166498136 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557924a624d0 0x557924e345d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.322+0000 7f593ea00640 1 -- 172.21.15.32:0/166498136 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557924a4a190 con 0x557924a6a6b0 2024-04-30T08:38:10.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.323+0000 7f593ea00640 1 --2- 172.21.15.32:0/166498136 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557924a6a6b0 0x557924e3d450 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f592c015830 tx=0x7f592c009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f14fa740039fe229 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:10.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.323+0000 7f592b400640 1 -- 172.21.15.32:0/166498136 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f592c025030 con 0x557924a6a6b0 2024-04-30T08:38:10.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.324+0000 7f592b400640 1 -- 172.21.15.32:0/166498136 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f592c0054f0 con 0x557924a6a6b0 2024-04-30T08:38:10.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.324+0000 7f592b400640 1 -- 172.21.15.32:0/166498136 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f592c00eac0 con 0x557924a6a6b0 2024-04-30T08:38:10.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.325+0000 7f594d271740 1 -- 172.21.15.32:0/166498136 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557924a6a6b0 msgr2=0x557924e3d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:10.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.326+0000 7f594d271740 1 --2- 172.21.15.32:0/166498136 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557924a6a6b0 0x557924e3d450 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f592c015830 tx=0x7f592c009fd0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.327+0000 7f594d271740 1 -- 172.21.15.32:0/166498136 shutdown_connections 2024-04-30T08:38:10.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.330+0000 7f594d271740 1 --2- 172.21.15.32:0/166498136 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557924a8b390 0x557924e34b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.330+0000 7f594d271740 1 --2- 172.21.15.32:0/166498136 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557924a6a6b0 0x557924e3d450 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.330+0000 7f594d271740 1 --2- 172.21.15.32:0/166498136 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557924a624d0 0x557924e345d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.330+0000 7f594d271740 1 -- 172.21.15.32:0/166498136 >> 172.21.15.32:0/166498136 conn(0x557924a7f4e0 msgr2=0x557924e30560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:10.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.331+0000 7f594d271740 1 -- 172.21.15.32:0/166498136 shutdown_connections 2024-04-30T08:38:10.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.331+0000 7f594d271740 1 -- 172.21.15.32:0/166498136 wait complete. 2024-04-30T08:38:10.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.331+0000 7f594d271740 1 -- start start 2024-04-30T08:38:10.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.332+0000 7f594d271740 1 Processor -- start 2024-04-30T08:38:10.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.333+0000 7f594d271740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557924a624d0 0x557924e3d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:10.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.333+0000 7f594d271740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557924a6a6b0 0x557924e39f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:10.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.333+0000 7f594d271740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557924a8b390 0x557924e3a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:10.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.333+0000 7f594d271740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f59401b45d0 con 0x557924a6a6b0 2024-04-30T08:38:10.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.333+0000 7f594d271740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f59401b3f00 con 0x557924a8b390 2024-04-30T08:38:10.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.333+0000 7f594d271740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f59401b1140 con 0x557924a624d0 2024-04-30T08:38:10.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.333+0000 7f594d271740 10 client.0 _ll_register_callbacks cb 0x7f59380a35f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:38:10.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.333+0000 7f593fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557924a8b390 0x557924e3a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:10.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.333+0000 7f593fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557924a8b390 0x557924e3a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:36402/0 (socket says 172.21.15.32:36402) 2024-04-30T08:38:10.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.333+0000 7f593fe00640 1 -- 172.21.15.32:0/351157669 learned_addr learned my addr 172.21.15.32:0/351157669 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:10.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.333+0000 7f593f400640 1 --2- 172.21.15.32:0/351157669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557924a624d0 0x557924e3d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:10.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.334+0000 7f593ea00640 1 --2- 172.21.15.32:0/351157669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557924a6a6b0 0x557924e39f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:10.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.334+0000 7f593fe00640 1 -- 172.21.15.32:0/351157669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557924a624d0 msgr2=0x557924e3d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:10.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.334+0000 7f593fe00640 1 --2- 172.21.15.32:0/351157669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557924a624d0 0x557924e3d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.334+0000 7f593fe00640 1 -- 172.21.15.32:0/351157669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557924a6a6b0 msgr2=0x557924e39f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:10.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.334+0000 7f593fe00640 1 --2- 172.21.15.32:0/351157669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557924a6a6b0 0x557924e39f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.334+0000 7f593fe00640 1 -- 172.21.15.32:0/351157669 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557924ea9e80 con 0x557924a8b390 2024-04-30T08:38:10.336 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.334+0000 7f593ea00640 1 --2- 172.21.15.32:0/351157669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557924a6a6b0 0x557924e39f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:38:10.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.335+0000 7f593fe00640 1 --2- 172.21.15.32:0/351157669 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557924a8b390 0x557924e3a590 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f59380a49a0 tx=0x7f59380053e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bec5d33e604daea server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:10.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.335+0000 7f5928c00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:38:10.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.335+0000 7f5928c00640 1 -- 172.21.15.32:0/351157669 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f59380c6020 con 0x557924a8b390 2024-04-30T08:38:10.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.335+0000 7f5928c00640 1 -- 172.21.15.32:0/351157669 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f59380b0310 con 0x557924a8b390 2024-04-30T08:38:10.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.335+0000 7f594d271740 1 -- 172.21.15.32:0/351157669 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f592c0054f0 con 0x557924a8b390 2024-04-30T08:38:10.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.336+0000 7f5928c00640 1 -- 172.21.15.32:0/351157669 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f59380a7500 con 0x557924a8b390 2024-04-30T08:38:10.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.336+0000 7f594d271740 10 client.26426 fetch_fsmap learned FSMap version 29 2024-04-30T08:38:10.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.336+0000 7f594d271740 1 -- 172.21.15.32:0/351157669 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557924eab160 con 0x557924a8b390 2024-04-30T08:38:10.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.336+0000 7f5928c00640 1 -- 172.21.15.32:0/351157669 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f592c0054f0 con 0x557924a8b390 2024-04-30T08:38:10.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.336+0000 7f5928c00640 1 -- 172.21.15.32:0/351157669 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7f59380b7020 con 0x557924a8b390 2024-04-30T08:38:10.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.337+0000 7f594d271740 10 client.26426 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:38:10.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.337+0000 7f594d271740 10 client.26426 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:38:10.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.337+0000 7f594d271740 20 client.26426 populate_metadata read hostname 'smithi032' 2024-04-30T08:38:10.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.337+0000 7f594d271740 1 --2- 172.21.15.32:0/351157669 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x557924a1f3a0 0x557924eadc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:10.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.337+0000 7f594d271740 4 client.26426 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-30T08:38:10.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.337+0000 7f594d271740 1 -- 172.21.15.32:0/351157669 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "scrub status"}) -- 0x557924d9f960 con 0x557924a1f3a0 2024-04-30T08:38:10.339 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.337+0000 7f593f400640 1 --2- 172.21.15.32:0/351157669 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x557924a1f3a0 0x557924eadc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:10.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.338+0000 7f5928c00640 10 client.26426 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:38:10.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.339+0000 7f593f400640 1 --2- 172.21.15.32:0/351157669 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x557924a1f3a0 0x557924eadc10 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c44aed8515371148 server_cookie=7d476212bdd3db3e in_seq=0 out_seq=0 2024-04-30T08:38:10.340 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.339+0000 7f5928c00640 1 -- 172.21.15.32:0/351157669 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x557924d9f960 con 0x557924a1f3a0 2024-04-30T08:38:10.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.339+0000 7f5928c00640 10 client.26426 handle_command_reply: tid=0 2024-04-30T08:38:10.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.339+0000 7f594d271740 1 client.26426 shutdown 2024-04-30T08:38:10.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.340+0000 7f594d271740 1 -- 172.21.15.32:0/351157669 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557924a8b390 msgr2=0x557924e3a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:10.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.340+0000 7f594d271740 1 --2- 172.21.15.32:0/351157669 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557924a8b390 0x557924e3a590 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f59380a49a0 tx=0x7f59380053e0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.340+0000 7f594d271740 1 -- 172.21.15.32:0/351157669 shutdown_connections 2024-04-30T08:38:10.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.340+0000 7f594d271740 1 --2- 172.21.15.32:0/351157669 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x557924a1f3a0 0x557924eadc10 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.340+0000 7f5928c00640 0 client.26426 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:38:10.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.340+0000 7f594d271740 1 --2- 172.21.15.32:0/351157669 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x557924a8b390 0x557924e3a590 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.340+0000 7f594d271740 1 --2- 172.21.15.32:0/351157669 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557924a6a6b0 0x557924e39f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.340+0000 7f594d271740 1 --2- 172.21.15.32:0/351157669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557924a624d0 0x557924e3d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.342 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.340+0000 7f594d271740 1 -- 172.21.15.32:0/351157669 >> 172.21.15.32:0/351157669 conn(0x557924a7f4e0 msgr2=0x557924e345d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:10.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.341+0000 7f594d271740 1 -- 172.21.15.32:0/351157669 shutdown_connections 2024-04-30T08:38:10.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.341+0000 7f594d271740 1 -- 172.21.15.32:0/351157669 wait complete. 2024-04-30T08:38:10.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.341+0000 7f594d271740 20 client.26426 trim_cache size 0 max 16384 2024-04-30T08:38:10.344 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-30T08:38:10.345 INFO:teuthology.orchestra.run.smithi032.stdout: "status": "no active scrubs running", 2024-04-30T08:38:10.345 INFO:teuthology.orchestra.run.smithi032.stdout: "scrubs": {} 2024-04-30T08:38:10.345 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-30T08:38:10.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.343+0000 7f5946200640 1 -- 172.21.15.32:0/4129500378 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f59240afc40 msgr2=0x7f59240b2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:10.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.343+0000 7f5946200640 1 --2- 172.21.15.32:0/4129500378 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f59240afc40 0x7f59240b2150 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f5930000c00 tx=0x7f59300314a0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.343+0000 7f5946200640 1 -- 172.21.15.32:0/4129500378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f594010bcb0 msgr2=0x7f59401a3990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:10.345 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.343+0000 7f5946200640 1 --2- 172.21.15.32:0/4129500378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f594010bcb0 0x7f59401a3990 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f5938009fd0 tx=0x7f593800ef10 comp rx=0 tx=0).stop 2024-04-30T08:38:10.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.343+0000 7f5946200640 1 -- 172.21.15.32:0/4129500378 shutdown_connections 2024-04-30T08:38:10.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.343+0000 7f5946200640 1 --2- 172.21.15.32:0/4129500378 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f59240afc40 0x7f59240b2150 secure :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f5930000c00 tx=0x7f59300314a0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.343+0000 7f5946200640 1 --2- 172.21.15.32:0/4129500378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f594010bcb0 0x7f59401a3990 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.343+0000 7f5946200640 1 --2- 172.21.15.32:0/4129500378 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5940106fb0 0x7f594019ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.343+0000 7f5946200640 1 --2- 172.21.15.32:0/4129500378 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5940105d70 0x7f594019c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.343+0000 7f5946200640 1 -- 172.21.15.32:0/4129500378 >> 172.21.15.32:0/4129500378 conn(0x7f5940100fa0 msgr2=0x7f5940107c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:10.346 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.344+0000 7f5946200640 1 -- 172.21.15.32:0/4129500378 shutdown_connections 2024-04-30T08:38:10.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.345+0000 7f5946200640 1 -- 172.21.15.32:0/4129500378 wait complete. 2024-04-30T08:38:10.401 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-30T08:38:10.401 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-30T08:38:10.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.756+0000 7f1cc8800640 1 -- 172.21.15.32:0/829434644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1cc010a580 msgr2=0x7f1cc010b280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:10.758 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.756+0000 7f1cc8800640 1 --2- 172.21.15.32:0/829434644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1cc010a580 0x7f1cc010b280 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f1cac0074e0 tx=0x7f1cac02b060 comp rx=0 tx=0).stop 2024-04-30T08:38:10.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.756+0000 7f1cc8800640 1 -- 172.21.15.32:0/829434644 shutdown_connections 2024-04-30T08:38:10.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.756+0000 7f1cc8800640 1 --2- 172.21.15.32:0/829434644 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1cc01104c0 0x7f1cc01128e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.756+0000 7f1cc8800640 1 --2- 172.21.15.32:0/829434644 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cc010b7c0 0x7f1cc010bbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.756+0000 7f1cc8800640 1 --2- 172.21.15.32:0/829434644 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1cc010a580 0x7f1cc010b280 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.756+0000 7f1cc8800640 1 -- 172.21.15.32:0/829434644 >> 172.21.15.32:0/829434644 conn(0x7f1cc00709a0 msgr2=0x7f1cc0072e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:10.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.756+0000 7f1cc8800640 1 -- 172.21.15.32:0/829434644 shutdown_connections 2024-04-30T08:38:10.759 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.756+0000 7f1cc8800640 1 -- 172.21.15.32:0/829434644 wait complete. 2024-04-30T08:38:10.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.758+0000 7f1cc8800640 1 Processor -- start 2024-04-30T08:38:10.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.758+0000 7f1cc8800640 1 -- start start 2024-04-30T08:38:10.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.758+0000 7f1cc8800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cc010a580 0x7f1cc01a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:10.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.758+0000 7f1cc8800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1cc010b7c0 0x7f1cc01a5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:10.760 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.758+0000 7f1cc8800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1cc01104c0 0x7f1cc01ac180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:10.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.758+0000 7f1cc8800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc0115cb0 con 0x7f1cc010a580 2024-04-30T08:38:10.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.758+0000 7f1cc8800640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc0115b30 con 0x7f1cc010b7c0 2024-04-30T08:38:10.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.758+0000 7f1cc8800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1cc0115e30 con 0x7f1cc01104c0 2024-04-30T08:38:10.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.759+0000 7f1cc5400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1cc010b7c0 0x7f1cc01a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:10.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.759+0000 7f1cc5400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1cc010b7c0 0x7f1cc01a5200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:36420/0 (socket says 172.21.15.32:36420) 2024-04-30T08:38:10.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.759+0000 7f1cc5400640 1 -- 172.21.15.32:0/3953234308 learned_addr learned my addr 172.21.15.32:0/3953234308 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:10.761 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.759+0000 7f1cc5e00640 1 --2- 172.21.15.32:0/3953234308 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cc010a580 0x7f1cc01a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:10.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.759+0000 7f1cc6800640 1 --2- 172.21.15.32:0/3953234308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1cc01104c0 0x7f1cc01ac180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:10.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.759+0000 7f1cc5400640 1 -- 172.21.15.32:0/3953234308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1cc01104c0 msgr2=0x7f1cc01ac180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:10.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.759+0000 7f1cc5400640 1 --2- 172.21.15.32:0/3953234308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1cc01104c0 0x7f1cc01ac180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.759+0000 7f1cc5400640 1 -- 172.21.15.32:0/3953234308 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cc010a580 msgr2=0x7f1cc01a4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:10.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.759+0000 7f1cc5400640 1 --2- 172.21.15.32:0/3953234308 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cc010a580 0x7f1cc01a4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:10.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.759+0000 7f1cc5400640 1 -- 172.21.15.32:0/3953234308 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cc01ac7f0 con 0x7f1cc010b7c0 2024-04-30T08:38:10.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.759+0000 7f1cc5400640 1 --2- 172.21.15.32:0/3953234308 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1cc010b7c0 0x7f1cc01a5200 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1cb400e9f0 tx=0x7f1cb400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:10.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.760+0000 7f1cbaa00640 1 -- 172.21.15.32:0/3953234308 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb400cd80 con 0x7f1cc010b7c0 2024-04-30T08:38:10.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.760+0000 7f1cc8800640 1 -- 172.21.15.32:0/3953234308 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cc01acae0 con 0x7f1cc010b7c0 2024-04-30T08:38:10.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.760+0000 7f1cbaa00640 1 -- 172.21.15.32:0/3953234308 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1cb40108e0 con 0x7f1cc010b7c0 2024-04-30T08:38:10.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.760+0000 7f1cbaa00640 1 -- 172.21.15.32:0/3953234308 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1cb4018960 con 0x7f1cc010b7c0 2024-04-30T08:38:10.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.760+0000 7f1cc8800640 1 -- 172.21.15.32:0/3953234308 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cc01ad050 con 0x7f1cc010b7c0 2024-04-30T08:38:10.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.761+0000 7f1cc8800640 1 -- 172.21.15.32:0/3953234308 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cc010f2d0 con 0x7f1cc010b7c0 2024-04-30T08:38:10.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.766+0000 7f1cbaa00640 1 -- 172.21.15.32:0/3953234308 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f1cb4010430 con 0x7f1cc010b7c0 2024-04-30T08:38:10.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.768+0000 7f1cbaa00640 1 --2- 172.21.15.32:0/3953234308 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f1ca80afb20 0x7f1ca80b2030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:10.769 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.768+0000 7f1cbaa00640 1 -- 172.21.15.32:0/3953234308 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f1cb40df870 con 0x7f1cc010b7c0 2024-04-30T08:38:10.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.768+0000 7f1cc5e00640 1 --2- 172.21.15.32:0/3953234308 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f1ca80afb20 0x7f1ca80b2030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:10.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.769+0000 7f1cbaa00640 1 -- 172.21.15.32:0/3953234308 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1cb40a8040 con 0x7f1cc010b7c0 2024-04-30T08:38:10.770 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:10.769+0000 7f1cc5e00640 1 --2- 172.21.15.32:0/3953234308 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f1ca80afb20 0x7f1ca80b2030 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f1cac006010 tx=0x7f1cac02d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:11.266 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.264+0000 7f1cc8800640 1 -- 172.21.15.32:0/3953234308 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1cc002dbd0 con 0x7f1cc010b7c0 2024-04-30T08:38:11.267 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.266+0000 7f1cbaa00640 1 -- 172.21.15.32:0/3953234308 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) ==== 94+0+11932 (secure 0 0 0) 0x7f1cb40b00c0 con 0x7f1cc010b7c0 2024-04-30T08:38:11.268 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:38:11.268 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":29,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2235842676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2235842676},{"type":"v1","addr":"172.21.15.32:6835","nonce":2235842676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/3577009355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":3577009355},{"type":"v1","addr":"172.21.15.32:6841","nonce":3577009355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":29,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-30T08:31:52.236668+0000","modified":"2024-04-30T08:36:54.799807+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24541,"mds_3":24533,"mds_4":24517},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/463203038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":463203038},{"type":"v1","addr":"172.21.15.189:6833","nonce":463203038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6833/3635143591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3635143591},{"type":"v1","addr":"172.21.15.71:6833","nonce":3635143591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6837/2569449994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2569449994},{"type":"v1","addr":"172.21.15.32:6837","nonce":2569449994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/278686565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":278686565},{"type":"v1","addr":"172.21.15.189:6835","nonce":278686565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6835/1558303056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1558303056},{"type":"v1","addr":"172.21.15.71:6835","nonce":1558303056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.189:6837/686520873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":686520873},{"type":"v1","addr":"172.21.15.189:6837","nonce":686520873}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.71:6837/2324039932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2324039932},{"type":"v1","addr":"172.21.15.71:6837","nonce":2324039932}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6839/352970107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":352970107},{"type":"v1","addr":"172.21.15.189:6839","nonce":352970107}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.32:6839/4160386524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4160386524},{"type":"v1","addr":"172.21.15.32:6839","nonce":4160386524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":4,"addr":"172.21.15.71:6839/3517409993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3517409993},{"type":"v1","addr":"172.21.15.71:6839","nonce":3517409993}]},"join_fscid":-1,"export_targets":[2,3,4],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24541,24590,24560,24517,24533]},"id":1}]} 2024-04-30T08:38:11.268 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 29 2024-04-30T08:38:11.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.269+0000 7f1cc8800640 1 -- 172.21.15.32:0/3953234308 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f1ca80afb20 msgr2=0x7f1ca80b2030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:11.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.269+0000 7f1cc8800640 1 --2- 172.21.15.32:0/3953234308 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f1ca80afb20 0x7f1ca80b2030 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f1cac006010 tx=0x7f1cac02d040 comp rx=0 tx=0).stop 2024-04-30T08:38:11.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.269+0000 7f1cc8800640 1 -- 172.21.15.32:0/3953234308 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1cc010b7c0 msgr2=0x7f1cc01a5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:11.270 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.269+0000 7f1cc8800640 1 --2- 172.21.15.32:0/3953234308 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1cc010b7c0 0x7f1cc01a5200 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f1cb400e9f0 tx=0x7f1cb400eeb0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.269+0000 7f1cc8800640 1 -- 172.21.15.32:0/3953234308 shutdown_connections 2024-04-30T08:38:11.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.269+0000 7f1cc8800640 1 --2- 172.21.15.32:0/3953234308 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f1ca80afb20 0x7f1ca80b2030 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.269+0000 7f1cc8800640 1 --2- 172.21.15.32:0/3953234308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1cc01104c0 0x7f1cc01ac180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.269+0000 7f1cc8800640 1 --2- 172.21.15.32:0/3953234308 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1cc010b7c0 0x7f1cc01a5200 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.269+0000 7f1cc8800640 1 --2- 172.21.15.32:0/3953234308 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1cc010a580 0x7f1cc01a4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.269+0000 7f1cc8800640 1 -- 172.21.15.32:0/3953234308 >> 172.21.15.32:0/3953234308 conn(0x7f1cc00709a0 msgr2=0x7f1cc010c4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:11.271 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.269+0000 7f1cc8800640 1 -- 172.21.15.32:0/3953234308 shutdown_connections 2024-04-30T08:38:11.272 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.269+0000 7f1cc8800640 1 -- 172.21.15.32:0/3953234308 wait complete. 2024-04-30T08:38:11.319 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-04-30T08:38:11.602 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:11 smithi189 ceph-mon[33038]: pgmap v515: 129 pgs: 129 active+clean; 11 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 286 MiB/s wr, 114 op/s 2024-04-30T08:38:11.602 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:11 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:38:11.602 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:11 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3953234308' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:38:11.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:11 smithi032 ceph-mon[28331]: pgmap v515: 129 pgs: 129 active+clean; 11 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 286 MiB/s wr, 114 op/s 2024-04-30T08:38:11.694 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:11 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:38:11.694 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:11 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3953234308' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:38:11.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:11 smithi071 ceph-mon[33385]: pgmap v515: 129 pgs: 129 active+clean; 11 GiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 286 MiB/s wr, 114 op/s 2024-04-30T08:38:11.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:11 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:38:11.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:11 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3953234308' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:38:11.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.760+0000 7f111e600640 1 -- 172.21.15.32:0/2527490243 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1118102880 msgr2=0x7f1118103580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:11.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.760+0000 7f111e600640 1 --2- 172.21.15.32:0/2527490243 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1118102880 0x7f1118103580 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f110c009900 tx=0x7f110c02a3f0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.761+0000 7f111e600640 1 -- 172.21.15.32:0/2527490243 shutdown_connections 2024-04-30T08:38:11.762 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.761+0000 7f111e600640 1 --2- 172.21.15.32:0/2527490243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11181087c0 0x7f111810abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.761+0000 7f111e600640 1 --2- 172.21.15.32:0/2527490243 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1118103ac0 0x7f1118103ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.761+0000 7f111e600640 1 --2- 172.21.15.32:0/2527490243 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1118102880 0x7f1118103580 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.761+0000 7f111e600640 1 -- 172.21.15.32:0/2527490243 >> 172.21.15.32:0/2527490243 conn(0x7f11180fda70 msgr2=0x7f11180fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:11.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.761+0000 7f111e600640 1 -- 172.21.15.32:0/2527490243 shutdown_connections 2024-04-30T08:38:11.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.761+0000 7f111e600640 1 -- 172.21.15.32:0/2527490243 wait complete. 2024-04-30T08:38:11.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.762+0000 7f111e600640 1 Processor -- start 2024-04-30T08:38:11.763 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.762+0000 7f111e600640 1 -- start start 2024-04-30T08:38:11.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.763+0000 7f111e600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1118102880 0x7f111819c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:11.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.763+0000 7f111e600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1118103ac0 0x7f111819c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:11.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.763+0000 7f111e600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11181087c0 0x7f11181a3850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:11.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.763+0000 7f111e600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f111810e430 con 0x7f1118102880 2024-04-30T08:38:11.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.763+0000 7f111e600640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f111810e2b0 con 0x7f1118103ac0 2024-04-30T08:38:11.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.763+0000 7f111e600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f111810e5b0 con 0x7f11181087c0 2024-04-30T08:38:11.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.764+0000 7f1117e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1118103ac0 0x7f111819c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:11.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.764+0000 7f1117e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1118103ac0 0x7f111819c8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:36440/0 (socket says 172.21.15.32:36440) 2024-04-30T08:38:11.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.764+0000 7f1117e00640 1 -- 172.21.15.32:0/1875152438 learned_addr learned my addr 172.21.15.32:0/1875152438 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:11.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.764+0000 7f111dc00640 1 --2- 172.21.15.32:0/1875152438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11181087c0 0x7f11181a3850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:11.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.764+0000 7f111d200640 1 --2- 172.21.15.32:0/1875152438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1118102880 0x7f111819c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:11.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.765+0000 7f111d200640 1 -- 172.21.15.32:0/1875152438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11181087c0 msgr2=0x7f11181a3850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:11.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.765+0000 7f111d200640 1 --2- 172.21.15.32:0/1875152438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11181087c0 0x7f11181a3850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.765+0000 7f111d200640 1 -- 172.21.15.32:0/1875152438 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1118103ac0 msgr2=0x7f111819c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:11.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.765+0000 7f111d200640 1 --2- 172.21.15.32:0/1875152438 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1118103ac0 0x7f111819c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.765+0000 7f111d200640 1 -- 172.21.15.32:0/1875152438 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11181a3e10 con 0x7f1118102880 2024-04-30T08:38:11.767 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.765+0000 7f111d200640 1 --2- 172.21.15.32:0/1875152438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1118102880 0x7f111819c380 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f110c009230 tx=0x7f110c003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:11.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.766+0000 7f1115600640 1 -- 172.21.15.32:0/1875152438 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f110c03f070 con 0x7f1118102880 2024-04-30T08:38:11.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.766+0000 7f1115600640 1 -- 172.21.15.32:0/1875152438 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f110c03b460 con 0x7f1118102880 2024-04-30T08:38:11.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.766+0000 7f1115600640 1 -- 172.21.15.32:0/1875152438 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f110c035030 con 0x7f1118102880 2024-04-30T08:38:11.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.766+0000 7f111e600640 1 -- 172.21.15.32:0/1875152438 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11181a40a0 con 0x7f1118102880 2024-04-30T08:38:11.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.766+0000 7f111e600640 1 -- 172.21.15.32:0/1875152438 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11181a4590 con 0x7f1118102880 2024-04-30T08:38:11.777 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.775+0000 7f1126ba2740 1 Processor -- start 2024-04-30T08:38:11.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.777+0000 7f1126ba2740 1 -- start start 2024-04-30T08:38:11.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.777+0000 7f1126ba2740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56062054f390 0x5606208f8f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:11.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.778+0000 7f1126ba2740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5606205264d0 0x5606208f94c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:11.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.778+0000 7f1126ba2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56062052e6b0 0x5606209003c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:11.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.778+0000 7f1126ba2740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x560620855d50 con 0x5606205264d0 2024-04-30T08:38:11.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.778+0000 7f1126ba2740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x5606207fb0c0 con 0x56062054f390 2024-04-30T08:38:11.780 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.778+0000 7f1126ba2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x560620851d10 con 0x56062052e6b0 2024-04-30T08:38:11.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.778+0000 7f111dc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56062054f390 0x5606208f8f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:11.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.779+0000 7f1117e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56062052e6b0 0x5606209003c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:11.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.779+0000 7f1117e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56062052e6b0 0x5606209003c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:32868/0 (socket says 172.21.15.32:32868) 2024-04-30T08:38:11.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.779+0000 7f1117e00640 1 -- 172.21.15.32:0/4244579340 learned_addr learned my addr 172.21.15.32:0/4244579340 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:11.783 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.781+0000 7f1117e00640 1 -- 172.21.15.32:0/4244579340 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56062054f390 msgr2=0x5606208f8f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:11.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.782+0000 7f1117e00640 1 --2- 172.21.15.32:0/4244579340 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56062054f390 0x5606208f8f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.783+0000 7f1117e00640 1 -- 172.21.15.32:0/4244579340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5606205264d0 msgr2=0x5606208f94c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-30T08:38:11.784 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.783+0000 7f1115600640 1 -- 172.21.15.32:0/1875152438 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f110c042070 con 0x7f1118102880 2024-04-30T08:38:11.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.784+0000 7f111d200640 1 --2- 172.21.15.32:0/4244579340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5606205264d0 0x5606208f94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:11.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.784+0000 7f1117e00640 1 --2- 172.21.15.32:0/4244579340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5606205264d0 0x5606208f94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.786 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.785+0000 7f1117e00640 1 -- 172.21.15.32:0/4244579340 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606205673f0 con 0x56062052e6b0 2024-04-30T08:38:11.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.785+0000 7f111dc00640 1 --2- 172.21.15.32:0/4244579340 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56062054f390 0x5606208f8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:38:11.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.785+0000 7f111d200640 1 --2- 172.21.15.32:0/4244579340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5606205264d0 0x5606208f94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-30T08:38:11.787 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.785+0000 7f1117e00640 1 --2- 172.21.15.32:0/4244579340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56062052e6b0 0x5606209003c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f110400ad30 tx=0x7f110400f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2bce0bd5199e1acd server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:11.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.787+0000 7f110a000640 1 -- 172.21.15.32:0/4244579340 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f110401d070 con 0x56062052e6b0 2024-04-30T08:38:11.789 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.787+0000 7f110a000640 1 -- 172.21.15.32:0/4244579340 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f110400ae20 con 0x56062052e6b0 2024-04-30T08:38:11.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.787+0000 7f110a000640 1 -- 172.21.15.32:0/4244579340 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1104018ba0 con 0x56062052e6b0 2024-04-30T08:38:11.790 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.787+0000 7f1115600640 1 --2- 172.21.15.32:0/1875152438 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f11000afbe0 0x7f11000b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:11.792 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.789+0000 7f111dc00640 1 --2- 172.21.15.32:0/1875152438 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f11000afbe0 0x7f11000b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:11.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.789+0000 7f1115600640 1 -- 172.21.15.32:0/1875152438 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f110c0ff800 con 0x7f1118102880 2024-04-30T08:38:11.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.790+0000 7f111dc00640 1 --2- 172.21.15.32:0/1875152438 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f11000afbe0 0x7f11000b20f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1110009fd0 tx=0x7f111000d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:11.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.790+0000 7f1126ba2740 1 -- 172.21.15.32:0/4244579340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56062052e6b0 msgr2=0x5606209003c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:11.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.790+0000 7f1126ba2740 1 --2- 172.21.15.32:0/4244579340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56062052e6b0 0x5606209003c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f110400ad30 tx=0x7f110400f790 comp rx=0 tx=0).stop 2024-04-30T08:38:11.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.791+0000 7f1126ba2740 1 -- 172.21.15.32:0/4244579340 shutdown_connections 2024-04-30T08:38:11.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.791+0000 7f1126ba2740 1 --2- 172.21.15.32:0/4244579340 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56062052e6b0 0x5606209003c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.791+0000 7f1126ba2740 1 --2- 172.21.15.32:0/4244579340 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5606205264d0 0x5606208f94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.791+0000 7f1126ba2740 1 --2- 172.21.15.32:0/4244579340 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56062054f390 0x5606208f8f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.791+0000 7f1126ba2740 1 -- 172.21.15.32:0/4244579340 >> 172.21.15.32:0/4244579340 conn(0x5606205434e0 msgr2=0x5606208f48d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:11.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.791+0000 7f1126ba2740 1 -- 172.21.15.32:0/4244579340 shutdown_connections 2024-04-30T08:38:11.793 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.791+0000 7f1126ba2740 1 -- 172.21.15.32:0/4244579340 wait complete. 2024-04-30T08:38:11.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.792+0000 7f1126ba2740 1 -- start start 2024-04-30T08:38:11.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.792+0000 7f1126ba2740 1 Processor -- start 2024-04-30T08:38:11.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.793+0000 7f1126ba2740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5606205264d0 0x5606208edcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:11.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.793+0000 7f1126ba2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56062052e6b0 0x5606208ee230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:11.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.793+0000 7f1126ba2740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56062054f390 0x5606208f3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:11.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.793+0000 7f1126ba2740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5606208ebd70 con 0x5606205264d0 2024-04-30T08:38:11.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.793+0000 7f1126ba2740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x56062080eb50 con 0x56062054f390 2024-04-30T08:38:11.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.793+0000 7f1126ba2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5606208ebef0 con 0x56062052e6b0 2024-04-30T08:38:11.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.793+0000 7f1126ba2740 10 client.0 _ll_register_callbacks cb 0x5606207d8420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:38:11.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.793+0000 7f111d200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5606205264d0 0x5606208edcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:11.795 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.793+0000 7f111d200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5606205264d0 0x5606208edcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49104/0 (socket says 172.21.15.32:49104) 2024-04-30T08:38:11.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.793+0000 7f111d200640 1 -- 172.21.15.32:0/3111707363 learned_addr learned my addr 172.21.15.32:0/3111707363 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:11.796 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.793+0000 7f111dc00640 1 --2- 172.21.15.32:0/3111707363 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56062054f390 0x5606208f3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:11.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.794+0000 7f1117e00640 1 --2- 172.21.15.32:0/3111707363 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56062052e6b0 0x5606208ee230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:11.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.794+0000 7f111dc00640 1 -- 172.21.15.32:0/3111707363 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56062052e6b0 msgr2=0x5606208ee230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:11.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.794+0000 7f111dc00640 1 --2- 172.21.15.32:0/3111707363 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56062052e6b0 0x5606208ee230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.794+0000 7f111dc00640 1 -- 172.21.15.32:0/3111707363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5606205264d0 msgr2=0x5606208edcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:11.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.794+0000 7f111dc00640 1 --2- 172.21.15.32:0/3111707363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5606205264d0 0x5606208edcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.794+0000 7f111dc00640 1 -- 172.21.15.32:0/3111707363 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5606206317d0 con 0x56062054f390 2024-04-30T08:38:11.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.794+0000 7f111d200640 1 --2- 172.21.15.32:0/3111707363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5606205264d0 0x5606208edcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:38:11.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.794+0000 7f111dc00640 1 --2- 172.21.15.32:0/3111707363 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56062054f390 0x5606208f3180 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1110018e40 tx=0x7f1110004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3578e6fa534ffe1 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:11.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.795+0000 7f10fea00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:38:11.797 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.795+0000 7f10fea00640 1 -- 172.21.15.32:0/3111707363 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f111001e040 con 0x56062054f390 2024-04-30T08:38:11.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.795+0000 7f10fea00640 1 -- 172.21.15.32:0/3111707363 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f11100223f0 con 0x56062054f390 2024-04-30T08:38:11.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.795+0000 7f1126ba2740 1 -- 172.21.15.32:0/3111707363 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f110400ae20 con 0x56062054f390 2024-04-30T08:38:11.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.795+0000 7f10fea00640 1 -- 172.21.15.32:0/3111707363 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f1110030050 con 0x56062054f390 2024-04-30T08:38:11.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.795+0000 7f1126ba2740 10 client.26444 fetch_fsmap learned FSMap version 29 2024-04-30T08:38:11.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.795+0000 7f1126ba2740 1 -- 172.21.15.32:0/3111707363 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560620915890 con 0x56062054f390 2024-04-30T08:38:11.798 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.796+0000 7f10fea00640 1 -- 172.21.15.32:0/3111707363 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f110400ae20 con 0x56062054f390 2024-04-30T08:38:11.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.796+0000 7f10fea00640 1 -- 172.21.15.32:0/3111707363 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7f1110021030 con 0x56062054f390 2024-04-30T08:38:11.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.796+0000 7f1126ba2740 10 client.26444 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:38:11.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.796+0000 7f1126ba2740 10 client.26444 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-30T08:38:11.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.796+0000 7f1126ba2740 20 client.26444 populate_metadata read hostname 'smithi032' 2024-04-30T08:38:11.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.796+0000 7f1126ba2740 1 --2- 172.21.15.32:0/3111707363 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x56062050df60 0x560620919010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:11.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.796+0000 7f1126ba2740 4 client.26444 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:38:11.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.796+0000 7f1126ba2740 1 -- 172.21.15.32:0/3111707363 --> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560620863960 con 0x56062050df60 2024-04-30T08:38:11.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.796+0000 7f111d200640 1 --2- 172.21.15.32:0/3111707363 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x56062050df60 0x560620919010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:11.799 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.797+0000 7f10fea00640 10 client.26444 ms_handle_connect on v2:172.21.15.189:6836/686520873 2024-04-30T08:38:11.801 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.800+0000 7f111d200640 1 --2- 172.21.15.32:0/3111707363 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x56062050df60 0x560620919010 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5e3275600ca7b69e server_cookie=f12d653271ebe19f in_seq=0 out_seq=0 2024-04-30T08:38:11.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.801+0000 7f10fea00640 1 -- 172.21.15.32:0/3111707363 <== mds.4 v2:172.21.15.189:6836/686520873 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x560620863960 con 0x56062050df60 2024-04-30T08:38:11.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.801+0000 7f10fea00640 10 client.26444 handle_command_reply: tid=0 2024-04-30T08:38:11.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.801+0000 7f1126ba2740 1 client.26444 shutdown 2024-04-30T08:38:11.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.802+0000 7f1126ba2740 1 -- 172.21.15.32:0/3111707363 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56062054f390 msgr2=0x5606208f3180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:11.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.802+0000 7f1126ba2740 1 --2- 172.21.15.32:0/3111707363 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56062054f390 0x5606208f3180 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1110018e40 tx=0x7f1110004a40 comp rx=0 tx=0).stop 2024-04-30T08:38:11.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.802+0000 7f1126ba2740 1 -- 172.21.15.32:0/3111707363 shutdown_connections 2024-04-30T08:38:11.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.802+0000 7f1126ba2740 1 --2- 172.21.15.32:0/3111707363 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x56062050df60 0x560620919010 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.802+0000 7f1126ba2740 1 --2- 172.21.15.32:0/3111707363 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56062054f390 0x5606208f3180 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.802+0000 7f1126ba2740 1 --2- 172.21.15.32:0/3111707363 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56062052e6b0 0x5606208ee230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.802+0000 7f1126ba2740 1 --2- 172.21.15.32:0/3111707363 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5606205264d0 0x5606208edcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.802+0000 7f1126ba2740 1 -- 172.21.15.32:0/3111707363 >> 172.21.15.32:0/3111707363 conn(0x5606205434e0 msgr2=0x5606208eb100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:11.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.802+0000 7f10fea00640 0 client.26444 ms_handle_reset on v2:172.21.15.189:6836/686520873 2024-04-30T08:38:11.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.802+0000 7f1126ba2740 1 -- 172.21.15.32:0/3111707363 shutdown_connections 2024-04-30T08:38:11.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.802+0000 7f1126ba2740 1 -- 172.21.15.32:0/3111707363 wait complete. 2024-04-30T08:38:11.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.802+0000 7f1126ba2740 20 client.26444 trim_cache size 0 max 16384 2024-04-30T08:38:11.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.820+0000 7f1126ba2740 1 Processor -- start 2024-04-30T08:38:11.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.821+0000 7f1126ba2740 1 -- start start 2024-04-30T08:38:11.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.821+0000 7f1126ba2740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5606205264d0 0x5606208ec9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:11.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.822+0000 7f1126ba2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56062052e6b0 0x5606208f3e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:11.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.822+0000 7f1126ba2740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56062054f390 0x5606208f43d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:11.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.823+0000 7f111d200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5606205264d0 0x5606208ec9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:11.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.824+0000 7f1126ba2740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f11181aa7e0 con 0x56062054f390 2024-04-30T08:38:11.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.824+0000 7f1126ba2740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f11181ac3b0 con 0x5606205264d0 2024-04-30T08:38:11.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.824+0000 7f1126ba2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f11181aaeb0 con 0x56062052e6b0 2024-04-30T08:38:11.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.824+0000 7f1117e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56062052e6b0 0x5606208f3e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:11.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.824+0000 7f111d200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5606205264d0 0x5606208ec9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:36466/0 (socket says 172.21.15.32:36466) 2024-04-30T08:38:11.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.824+0000 7f111d200640 1 -- 172.21.15.32:0/659352566 learned_addr learned my addr 172.21.15.32:0/659352566 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:11.826 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.824+0000 7f111dc00640 1 --2- 172.21.15.32:0/659352566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56062054f390 0x5606208f43d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:11.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.825+0000 7f111d200640 1 -- 172.21.15.32:0/659352566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56062052e6b0 msgr2=0x5606208f3e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:11.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.825+0000 7f111d200640 1 --2- 172.21.15.32:0/659352566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56062052e6b0 0x5606208f3e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.825+0000 7f111d200640 1 -- 172.21.15.32:0/659352566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56062054f390 msgr2=0x5606208f43d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:11.827 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.825+0000 7f111d200640 1 --2- 172.21.15.32:0/659352566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56062054f390 0x5606208f43d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.825+0000 7f111d200640 1 -- 172.21.15.32:0/659352566 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5606208e78e0 con 0x5606205264d0 2024-04-30T08:38:11.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.825+0000 7f111dc00640 1 --2- 172.21.15.32:0/659352566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56062054f390 0x5606208f43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:38:11.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.825+0000 7f1117e00640 1 --2- 172.21.15.32:0/659352566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56062052e6b0 0x5606208f3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:38:11.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.826+0000 7f111d200640 1 --2- 172.21.15.32:0/659352566 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5606205264d0 0x5606208ec9a0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f110c0b0010 tx=0x7f110c0fcdc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=543fe2b78b7ccc8f server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:11.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.826+0000 7f1109600640 1 -- 172.21.15.32:0/659352566 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f110c049040 con 0x5606205264d0 2024-04-30T08:38:11.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.826+0000 7f1109600640 1 -- 172.21.15.32:0/659352566 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f110c0c7490 con 0x5606205264d0 2024-04-30T08:38:11.828 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.826+0000 7f1109600640 1 -- 172.21.15.32:0/659352566 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f110c03b610 con 0x5606205264d0 2024-04-30T08:38:11.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.828+0000 7f1126ba2740 1 -- 172.21.15.32:0/659352566 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5606205264d0 msgr2=0x5606208ec9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:11.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.828+0000 7f1126ba2740 1 --2- 172.21.15.32:0/659352566 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5606205264d0 0x5606208ec9a0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f110c0b0010 tx=0x7f110c0fcdc0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.828+0000 7f1126ba2740 1 -- 172.21.15.32:0/659352566 shutdown_connections 2024-04-30T08:38:11.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.828+0000 7f1126ba2740 1 --2- 172.21.15.32:0/659352566 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56062054f390 0x5606208f43d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.829+0000 7f1126ba2740 1 --2- 172.21.15.32:0/659352566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56062052e6b0 0x5606208f3e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.829+0000 7f1126ba2740 1 --2- 172.21.15.32:0/659352566 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5606205264d0 0x5606208ec9a0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.829+0000 7f1126ba2740 1 -- 172.21.15.32:0/659352566 >> 172.21.15.32:0/659352566 conn(0x5606205434e0 msgr2=0x5606208ec290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:11.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.829+0000 7f1126ba2740 1 -- 172.21.15.32:0/659352566 shutdown_connections 2024-04-30T08:38:11.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.829+0000 7f1126ba2740 1 -- 172.21.15.32:0/659352566 wait complete. 2024-04-30T08:38:11.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.829+0000 7f1126ba2740 1 -- start start 2024-04-30T08:38:11.832 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.830+0000 7f1126ba2740 1 Processor -- start 2024-04-30T08:38:11.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.831+0000 7f1126ba2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606205264d0 0x56062092a6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:11.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.831+0000 7f1126ba2740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56062052e6b0 0x560620927240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:11.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.831+0000 7f1126ba2740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56062054f390 0x5606209277a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:11.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.831+0000 7f1126ba2740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f11181abc50 con 0x56062052e6b0 2024-04-30T08:38:11.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.831+0000 7f1126ba2740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f11181ab580 con 0x56062054f390 2024-04-30T08:38:11.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.831+0000 7f1126ba2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f11181a87a0 con 0x5606205264d0 2024-04-30T08:38:11.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.831+0000 7f1126ba2740 10 client.0 _ll_register_callbacks cb 0x7f1110009ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:38:11.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.832+0000 7f111dc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56062054f390 0x5606209277a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:11.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.832+0000 7f111dc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56062054f390 0x5606209277a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:36474/0 (socket says 172.21.15.32:36474) 2024-04-30T08:38:11.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.832+0000 7f111dc00640 1 -- 172.21.15.32:0/1634006072 learned_addr learned my addr 172.21.15.32:0/1634006072 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:11.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.833+0000 7f1117e00640 1 --2- 172.21.15.32:0/1634006072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56062052e6b0 0x560620927240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:11.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.833+0000 7f111d200640 1 --2- 172.21.15.32:0/1634006072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606205264d0 0x56062092a6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:11.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.833+0000 7f111d200640 1 -- 172.21.15.32:0/1634006072 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56062054f390 msgr2=0x5606209277a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:11.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.833+0000 7f111d200640 1 --2- 172.21.15.32:0/1634006072 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56062054f390 0x5606209277a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.833+0000 7f111d200640 1 -- 172.21.15.32:0/1634006072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56062052e6b0 msgr2=0x560620927240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:11.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.833+0000 7f111d200640 1 --2- 172.21.15.32:0/1634006072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56062052e6b0 0x560620927240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.833+0000 7f111d200640 1 -- 172.21.15.32:0/1634006072 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56062097efe0 con 0x5606205264d0 2024-04-30T08:38:11.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.834+0000 7f111d200640 1 --2- 172.21.15.32:0/1634006072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606205264d0 0x56062092a6c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x5606207dadd0 tx=0x7f110c02a3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=707ca6bdcd5d288d server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:11.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.834+0000 7f10fea00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-30T08:38:11.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.834+0000 7f10fea00640 1 -- 172.21.15.32:0/1634006072 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f110c049070 con 0x5606205264d0 2024-04-30T08:38:11.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.834+0000 7f10fea00640 1 -- 172.21.15.32:0/1634006072 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f110c0ea0b0 con 0x5606205264d0 2024-04-30T08:38:11.836 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.834+0000 7f1126ba2740 1 -- 172.21.15.32:0/1634006072 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f110c0c7490 con 0x5606205264d0 2024-04-30T08:38:11.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.835+0000 7f10fea00640 1 -- 172.21.15.32:0/1634006072 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f110c0cbc80 con 0x5606205264d0 2024-04-30T08:38:11.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.835+0000 7f1126ba2740 10 client.26440 fetch_fsmap learned FSMap version 29 2024-04-30T08:38:11.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.835+0000 7f1126ba2740 1 -- 172.21.15.32:0/1634006072 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x560620980210 con 0x5606205264d0 2024-04-30T08:38:11.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.836+0000 7f10fea00640 1 -- 172.21.15.32:0/1634006072 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f110c0c7490 con 0x5606205264d0 2024-04-30T08:38:11.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.836+0000 7f10fea00640 1 -- 172.21.15.32:0/1634006072 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7f110c0d0330 con 0x5606205264d0 2024-04-30T08:38:11.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.836+0000 7f1126ba2740 10 client.26440 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:38:11.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.836+0000 7f1126ba2740 10 client.26440 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.c 2024-04-30T08:38:11.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.836+0000 7f1126ba2740 20 client.26440 populate_metadata read hostname 'smithi032' 2024-04-30T08:38:11.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.836+0000 7f1126ba2740 1 --2- 172.21.15.32:0/1634006072 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x5606204e33a0 0x560620982940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:11.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.836+0000 7f1126ba2740 4 client.26440 mds_command: new command op to 24517 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-30T08:38:11.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.836+0000 7f1126ba2740 1 -- 172.21.15.32:0/1634006072 --> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56062085d7d0 con 0x5606204e33a0 2024-04-30T08:38:11.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.837+0000 7f1117e00640 1 --2- 172.21.15.32:0/1634006072 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x5606204e33a0 0x560620982940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:11.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.841+0000 7f1117e00640 1 --2- 172.21.15.32:0/1634006072 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x5606204e33a0 0x560620982940 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=d051653dde5bf4a server_cookie=40c08cf5bfcba87b in_seq=0 out_seq=0 2024-04-30T08:38:11.842 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.841+0000 7f10fea00640 10 client.26440 ms_handle_connect on v2:172.21.15.189:6836/686520873 2024-04-30T08:38:11.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.841+0000 7f10fea00640 1 -- 172.21.15.32:0/1634006072 <== mds.4 v2:172.21.15.189:6836/686520873 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56062085d7d0 con 0x5606204e33a0 2024-04-30T08:38:11.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.842+0000 7f10fea00640 10 client.26440 handle_command_reply: tid=0 2024-04-30T08:38:11.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.842+0000 7f1126ba2740 1 client.26440 shutdown 2024-04-30T08:38:11.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.842+0000 7f1126ba2740 1 -- 172.21.15.32:0/1634006072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606205264d0 msgr2=0x56062092a6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:11.843 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.842+0000 7f1126ba2740 1 --2- 172.21.15.32:0/1634006072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606205264d0 0x56062092a6c0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x5606207dadd0 tx=0x7f110c02a3f0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.842+0000 7f1126ba2740 1 -- 172.21.15.32:0/1634006072 shutdown_connections 2024-04-30T08:38:11.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.842+0000 7f1126ba2740 1 --2- 172.21.15.32:0/1634006072 >> [v2:172.21.15.189:6836/686520873,v1:172.21.15.189:6837/686520873] conn(0x5606204e33a0 0x560620982940 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.844 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.842+0000 7f1126ba2740 1 --2- 172.21.15.32:0/1634006072 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x56062054f390 0x5606209277a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.842+0000 7f10fea00640 0 client.26440 ms_handle_reset on v2:172.21.15.189:6836/686520873 2024-04-30T08:38:11.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.842+0000 7f1126ba2740 1 --2- 172.21.15.32:0/1634006072 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x56062052e6b0 0x560620927240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.842+0000 7f1126ba2740 1 --2- 172.21.15.32:0/1634006072 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606205264d0 0x56062092a6c0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.842+0000 7f1126ba2740 1 -- 172.21.15.32:0/1634006072 >> 172.21.15.32:0/1634006072 conn(0x5606205434e0 msgr2=0x5606208fc0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:11.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.842+0000 7f1126ba2740 1 -- 172.21.15.32:0/1634006072 shutdown_connections 2024-04-30T08:38:11.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.842+0000 7f1126ba2740 1 -- 172.21.15.32:0/1634006072 wait complete. 2024-04-30T08:38:11.845 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.843+0000 7f1126ba2740 20 client.26440 trim_cache size 0 max 16384 2024-04-30T08:38:11.845 INFO:teuthology.orchestra.run.smithi032.stdout:[] 2024-04-30T08:38:11.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.845+0000 7f111e600640 1 -- 172.21.15.32:0/1875152438 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f11000afbe0 msgr2=0x7f11000b20f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:11.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.845+0000 7f111e600640 1 --2- 172.21.15.32:0/1875152438 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f11000afbe0 0x7f11000b20f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f1110009fd0 tx=0x7f111000d040 comp rx=0 tx=0).stop 2024-04-30T08:38:11.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.845+0000 7f111e600640 1 -- 172.21.15.32:0/1875152438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1118102880 msgr2=0x7f111819c380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:11.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.845+0000 7f111e600640 1 --2- 172.21.15.32:0/1875152438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1118102880 0x7f111819c380 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f110c009230 tx=0x7f110c003d50 comp rx=0 tx=0).stop 2024-04-30T08:38:11.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.845+0000 7f111e600640 1 -- 172.21.15.32:0/1875152438 shutdown_connections 2024-04-30T08:38:11.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.845+0000 7f111e600640 1 --2- 172.21.15.32:0/1875152438 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f11000afbe0 0x7f11000b20f0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.845+0000 7f111e600640 1 --2- 172.21.15.32:0/1875152438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11181087c0 0x7f11181a3850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.845+0000 7f111e600640 1 --2- 172.21.15.32:0/1875152438 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1118103ac0 0x7f111819c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:11.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.845+0000 7f111e600640 1 --2- 172.21.15.32:0/1875152438 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1118102880 0x7f111819c380 secure :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f110c009230 tx=0x7f110c003d50 comp rx=0 tx=0).stop 2024-04-30T08:38:11.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.845+0000 7f111e600640 1 -- 172.21.15.32:0/1875152438 >> 172.21.15.32:0/1875152438 conn(0x7f11180fda70 msgr2=0x7f11180ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:11.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.846+0000 7f111e600640 1 -- 172.21.15.32:0/1875152438 shutdown_connections 2024-04-30T08:38:11.848 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:11.846+0000 7f111e600640 1 -- 172.21.15.32:0/1875152438 wait complete. 2024-04-30T08:38:11.886 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-04-30T08:38:12.184 INFO:tasks.workunit.client.0.smithi032.stdout:Syncing()...9 sec 2024-04-30T08:38:12.186 INFO:tasks.workunit.client.0.smithi032.stdout:Starting Actual Benchmark At: Tue Apr 30 08:38:12 2024 2024-04-30T08:38:12.186 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-04-30T08:38:12.666 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.663+0000 7ffb04400640 1 -- 172.21.15.32:0/3060234031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffafc10b390 msgr2=0x7ffafc10b7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:12.667 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.663+0000 7ffb04400640 1 --2- 172.21.15.32:0/3060234031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffafc10b390 0x7ffafc10b7a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7ffaf401acb0 tx=0x7ffaf403b920 comp rx=0 tx=0).stop 2024-04-30T08:38:12.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.674+0000 7ffb04400640 1 -- 172.21.15.32:0/3060234031 shutdown_connections 2024-04-30T08:38:12.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.674+0000 7ffb04400640 1 --2- 172.21.15.32:0/3060234031 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffafc1100b0 0x7ffafc1104c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:12.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.674+0000 7ffb04400640 1 --2- 172.21.15.32:0/3060234031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffafc10b390 0x7ffafc10b7a0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:12.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.674+0000 7ffb04400640 1 --2- 172.21.15.32:0/3060234031 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffafc10a4a0 0x7ffafc10ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:12.679 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.674+0000 7ffb04400640 1 -- 172.21.15.32:0/3060234031 >> 172.21.15.32:0/3060234031 conn(0x7ffafc070560 msgr2=0x7ffafc072a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:12.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.676+0000 7ffb04400640 1 -- 172.21.15.32:0/3060234031 shutdown_connections 2024-04-30T08:38:12.687 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.686+0000 7ffb04400640 1 -- 172.21.15.32:0/3060234031 wait complete. 2024-04-30T08:38:12.689 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.687+0000 7ffb04400640 1 Processor -- start 2024-04-30T08:38:12.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.697+0000 7ffb04400640 1 -- start start 2024-04-30T08:38:12.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.697+0000 7ffb04400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffafc10a4a0 0x7ffafc1a0980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:12.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.697+0000 7ffb04400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffafc10b390 0x7ffafc1a0ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:12.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.697+0000 7ffb04400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffafc1100b0 0x7ffafc1a7e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:12.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.697+0000 7ffb04400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ffafc113da0 con 0x7ffafc10b390 2024-04-30T08:38:12.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.697+0000 7ffb04400640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ffafc113c20 con 0x7ffafc10a4a0 2024-04-30T08:38:12.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.697+0000 7ffb04400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ffafc113f20 con 0x7ffafc1100b0 2024-04-30T08:38:12.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.699+0000 7ffb01000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffafc10b390 0x7ffafc1a0ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:12.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.699+0000 7ffb01000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffafc10b390 0x7ffafc1a0ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49154/0 (socket says 172.21.15.32:49154) 2024-04-30T08:38:12.704 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.699+0000 7ffb01000640 1 -- 172.21.15.32:0/1019517126 learned_addr learned my addr 172.21.15.32:0/1019517126 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:12.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.701+0000 7ffb02400640 1 --2- 172.21.15.32:0/1019517126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffafc1100b0 0x7ffafc1a7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:12.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.702+0000 7ffb01a00640 1 --2- 172.21.15.32:0/1019517126 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffafc10a4a0 0x7ffafc1a0980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:12.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.702+0000 7ffb01000640 1 -- 172.21.15.32:0/1019517126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffafc1100b0 msgr2=0x7ffafc1a7e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:12.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.702+0000 7ffb01000640 1 --2- 172.21.15.32:0/1019517126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffafc1100b0 0x7ffafc1a7e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:12.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.702+0000 7ffb01000640 1 -- 172.21.15.32:0/1019517126 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffafc10a4a0 msgr2=0x7ffafc1a0980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:12.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.702+0000 7ffb01000640 1 --2- 172.21.15.32:0/1019517126 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffafc10a4a0 0x7ffafc1a0980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:12.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.702+0000 7ffb01000640 1 -- 172.21.15.32:0/1019517126 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffafc1a8410 con 0x7ffafc10b390 2024-04-30T08:38:12.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.703+0000 7ffb01000640 1 --2- 172.21.15.32:0/1019517126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffafc10b390 0x7ffafc1a0ec0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ffaf403bf10 tx=0x7ffaf4040b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:12.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.712+0000 7ffafa000640 1 -- 172.21.15.32:0/1019517126 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf4045040 con 0x7ffafc10b390 2024-04-30T08:38:12.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.712+0000 7ffafa000640 1 -- 172.21.15.32:0/1019517126 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffaf4040eb0 con 0x7ffafc10b390 2024-04-30T08:38:12.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.712+0000 7ffafa000640 1 -- 172.21.15.32:0/1019517126 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf404b8c0 con 0x7ffafc10b390 2024-04-30T08:38:12.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.713+0000 7ffb04400640 1 -- 172.21.15.32:0/1019517126 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffafc1a86a0 con 0x7ffafc10b390 2024-04-30T08:38:12.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.713+0000 7ffb04400640 1 -- 172.21.15.32:0/1019517126 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffafc1a8b10 con 0x7ffafc10b390 2024-04-30T08:38:12.727 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.724+0000 7ffb0b56d740 1 Processor -- start 2024-04-30T08:38:12.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.728+0000 7ffafa000640 1 -- 172.21.15.32:0/1019517126 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7ffaf4048020 con 0x7ffafc10b390 2024-04-30T08:38:12.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.730+0000 7ffafa000640 1 --2- 172.21.15.32:0/1019517126 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ffae40afd00 0x7ffae40b2210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:12.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.734+0000 7ffb02400640 1 --2- 172.21.15.32:0/1019517126 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ffae40afd00 0x7ffae40b2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:12.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.734+0000 7ffafa000640 1 -- 172.21.15.32:0/1019517126 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7ffaf41117a0 con 0x7ffafc10b390 2024-04-30T08:38:12.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.736+0000 7ffb02400640 1 --2- 172.21.15.32:0/1019517126 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ffae40afd00 0x7ffae40b2210 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ffae800ab40 tx=0x7ffae800a250 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:12.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.739+0000 7ffb0b56d740 1 -- start start 2024-04-30T08:38:12.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.739+0000 7ffb0b56d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557786834390 0x557786bd5570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:12.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.739+0000 7ffb0b56d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55778680b4d0 0x557786bd5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:12.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.739+0000 7ffb0b56d740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5577868136b0 0x557786bdc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:12.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.739+0000 7ffb0b56d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x557786b3ad50 con 0x557786834390 2024-04-30T08:38:12.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.739+0000 7ffb0b56d740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x557786ae00c0 con 0x5577868136b0 2024-04-30T08:38:12.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.739+0000 7ffb0b56d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x557786b36d10 con 0x55778680b4d0 2024-04-30T08:38:12.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.743+0000 7ffb01a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557786834390 0x557786bd5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:12.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.743+0000 7ffb01a00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557786834390 0x557786bd5570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49158/0 (socket says 172.21.15.32:49158) 2024-04-30T08:38:12.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.743+0000 7ffb01a00640 1 -- 172.21.15.32:0/2020819717 learned_addr learned my addr 172.21.15.32:0/2020819717 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:12.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.743+0000 7ffb01a00640 1 -- 172.21.15.32:0/2020819717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55778680b4d0 msgr2=0x557786bd5af0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-30T08:38:12.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.743+0000 7ffb01a00640 1 --2- 172.21.15.32:0/2020819717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55778680b4d0 0x557786bd5af0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:12.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.743+0000 7ffb01a00640 1 -- 172.21.15.32:0/2020819717 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5577868136b0 msgr2=0x557786bdc9f0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-30T08:38:12.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.743+0000 7ffb01a00640 1 --2- 172.21.15.32:0/2020819717 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5577868136b0 0x557786bdc9f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:12.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.743+0000 7ffb01a00640 1 -- 172.21.15.32:0/2020819717 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55778684c3f0 con 0x557786834390 2024-04-30T08:38:12.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.744+0000 7ffb01a00640 1 --2- 172.21.15.32:0/2020819717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557786834390 0x557786bd5570 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ffaf000b7f0 tx=0x7ffaf000bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ad4bf809f7fc20f server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:12.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.746+0000 7ffaeea00640 1 -- 172.21.15.32:0/2020819717 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf000d950 con 0x557786834390 2024-04-30T08:38:12.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.746+0000 7ffaeea00640 1 -- 172.21.15.32:0/2020819717 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffaf0018070 con 0x557786834390 2024-04-30T08:38:12.750 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.746+0000 7ffaeea00640 1 -- 172.21.15.32:0/2020819717 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf00123e0 con 0x557786834390 2024-04-30T08:38:12.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.747+0000 7ffb0b56d740 1 -- 172.21.15.32:0/2020819717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557786834390 msgr2=0x557786bd5570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:12.751 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.747+0000 7ffb0b56d740 1 --2- 172.21.15.32:0/2020819717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557786834390 0x557786bd5570 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ffaf000b7f0 tx=0x7ffaf000bcb0 comp rx=0 tx=0).stop 2024-04-30T08:38:12.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.756+0000 7ffb0b56d740 1 -- 172.21.15.32:0/2020819717 shutdown_connections 2024-04-30T08:38:12.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.756+0000 7ffb0b56d740 1 --2- 172.21.15.32:0/2020819717 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5577868136b0 0x557786bdc9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:12.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.756+0000 7ffb0b56d740 1 --2- 172.21.15.32:0/2020819717 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55778680b4d0 0x557786bd5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:12.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.756+0000 7ffb0b56d740 1 --2- 172.21.15.32:0/2020819717 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557786834390 0x557786bd5570 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:12.757 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.756+0000 7ffb0b56d740 1 -- 172.21.15.32:0/2020819717 >> 172.21.15.32:0/2020819717 conn(0x5577868284e0 msgr2=0x557786bd0f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:12.764 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.759+0000 7ffb0b56d740 1 -- 172.21.15.32:0/2020819717 shutdown_connections 2024-04-30T08:38:12.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.759+0000 7ffb0b56d740 1 -- 172.21.15.32:0/2020819717 wait complete. 2024-04-30T08:38:12.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.760+0000 7ffb0b56d740 1 -- start start 2024-04-30T08:38:12.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.760+0000 7ffb0b56d740 1 Processor -- start 2024-04-30T08:38:12.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.761+0000 7ffb0b56d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55778680b4d0 0x557786bdf310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:12.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.761+0000 7ffb0b56d740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5577868136b0 0x557786bdf850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:12.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.761+0000 7ffb0b56d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557786834390 0x557786be67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:12.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.761+0000 7ffb0b56d740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x557786bdd440 con 0x557786834390 2024-04-30T08:38:12.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.761+0000 7ffb0b56d740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x557786af3b50 con 0x5577868136b0 2024-04-30T08:38:12.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.761+0000 7ffb0b56d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x557786bdd5c0 con 0x55778680b4d0 2024-04-30T08:38:12.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.761+0000 7ffb01a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55778680b4d0 0x557786bdf310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:12.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.762+0000 7ffb0b56d740 10 client.0 _ll_register_callbacks cb 0x557786abd420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:38:12.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.762+0000 7ffb01a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55778680b4d0 0x557786bdf310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:32936/0 (socket says 172.21.15.32:32936) 2024-04-30T08:38:12.765 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.762+0000 7ffb01a00640 1 -- 172.21.15.32:0/2351591015 learned_addr learned my addr 172.21.15.32:0/2351591015 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:12.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.762+0000 7ffb01a00640 1 -- 172.21.15.32:0/2351591015 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5577868136b0 msgr2=0x557786bdf850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:12.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.762+0000 7ffb01a00640 1 --2- 172.21.15.32:0/2351591015 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5577868136b0 0x557786bdf850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:12.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.762+0000 7ffb01a00640 1 -- 172.21.15.32:0/2351591015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557786834390 msgr2=0x557786be67b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-30T08:38:12.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.762+0000 7ffb02400640 1 --2- 172.21.15.32:0/2351591015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557786834390 0x557786be67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:12.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.762+0000 7ffb01a00640 1 --2- 172.21.15.32:0/2351591015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557786834390 0x557786be67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:12.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.762+0000 7ffb01a00640 1 -- 172.21.15.32:0/2351591015 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577869167d0 con 0x55778680b4d0 2024-04-30T08:38:12.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.763+0000 7ffb02400640 1 --2- 172.21.15.32:0/2351591015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557786834390 0x557786be67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-30T08:38:12.766 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.763+0000 7ffb01a00640 1 --2- 172.21.15.32:0/2351591015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55778680b4d0 0x557786bdf310 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ffaf0017000 tx=0x7ffaf00136b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=46409373da20088b server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:12.768 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.766+0000 7ffae3400640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-30T08:38:12.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.766+0000 7ffae3400640 1 -- 172.21.15.32:0/2351591015 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf0022020 con 0x55778680b4d0 2024-04-30T08:38:12.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.766+0000 7ffae3400640 1 -- 172.21.15.32:0/2351591015 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7ffaf000dd60 con 0x55778680b4d0 2024-04-30T08:38:12.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.770+0000 7ffb0b56d740 1 -- 172.21.15.32:0/2351591015 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffaf0018070 con 0x55778680b4d0 2024-04-30T08:38:12.771 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.770+0000 7ffae3400640 1 -- 172.21.15.32:0/2351591015 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ffaf001c420 con 0x55778680b4d0 2024-04-30T08:38:12.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.770+0000 7ffb0b56d740 10 client.26449 fetch_fsmap learned FSMap version 29 2024-04-30T08:38:12.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.770+0000 7ffb0b56d740 1 -- 172.21.15.32:0/2351591015 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x557786c0b760 con 0x55778680b4d0 2024-04-30T08:38:12.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.771+0000 7ffae3400640 1 -- 172.21.15.32:0/2351591015 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffaf0018070 con 0x55778680b4d0 2024-04-30T08:38:12.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.771+0000 7ffae3400640 1 -- 172.21.15.32:0/2351591015 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7ffaf002f020 con 0x55778680b4d0 2024-04-30T08:38:12.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.771+0000 7ffb0b56d740 10 client.26449 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:38:12.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.771+0000 7ffb0b56d740 10 client.26449 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-30T08:38:12.773 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.771+0000 7ffb0b56d740 20 client.26449 populate_metadata read hostname 'smithi032' 2024-04-30T08:38:12.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.771+0000 7ffb0b56d740 1 --2- 172.21.15.32:0/2351591015 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x5577867f2f60 0x557786c0eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:12.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.771+0000 7ffb0b56d740 4 client.26449 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:38:12.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.771+0000 7ffb0b56d740 1 -- 172.21.15.32:0/2351591015 --> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557786b48960 con 0x5577867f2f60 2024-04-30T08:38:12.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.772+0000 7ffb01000640 1 --2- 172.21.15.32:0/2351591015 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x5577867f2f60 0x557786c0eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:12.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.772+0000 7ffb01000640 1 --2- 172.21.15.32:0/2351591015 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x5577867f2f60 0x557786c0eb80 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=4ce6a87442e16302 server_cookie=5d5c19c01dce5982 in_seq=0 out_seq=0 2024-04-30T08:38:12.774 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.772+0000 7ffae3400640 10 client.26449 ms_handle_connect on v2:172.21.15.71:6836/2324039932 2024-04-30T08:38:12.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.773+0000 7ffae3400640 1 -- 172.21.15.32:0/2351591015 <== mds.3 v2:172.21.15.71:6836/2324039932 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x557786b48960 con 0x5577867f2f60 2024-04-30T08:38:12.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.773+0000 7ffae3400640 10 client.26449 handle_command_reply: tid=0 2024-04-30T08:38:12.775 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.773+0000 7ffb0b56d740 1 client.26449 shutdown 2024-04-30T08:38:12.781 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.780+0000 7ffb0b56d740 1 -- 172.21.15.32:0/2351591015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55778680b4d0 msgr2=0x557786bdf310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:12.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.780+0000 7ffb0b56d740 1 --2- 172.21.15.32:0/2351591015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55778680b4d0 0x557786bdf310 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ffaf0017000 tx=0x7ffaf00136b0 comp rx=0 tx=0).stop 2024-04-30T08:38:12.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.780+0000 7ffb0b56d740 1 -- 172.21.15.32:0/2351591015 shutdown_connections 2024-04-30T08:38:12.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.780+0000 7ffb0b56d740 1 --2- 172.21.15.32:0/2351591015 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x5577867f2f60 0x557786c0eb80 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:12.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.780+0000 7ffb0b56d740 1 --2- 172.21.15.32:0/2351591015 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557786834390 0x557786be67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:12.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.780+0000 7ffb0b56d740 1 --2- 172.21.15.32:0/2351591015 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5577868136b0 0x557786bdf850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:12.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.780+0000 7ffb0b56d740 1 --2- 172.21.15.32:0/2351591015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55778680b4d0 0x557786bdf310 secure :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ffaf0017000 tx=0x7ffaf00136b0 comp rx=0 tx=0).stop 2024-04-30T08:38:12.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.780+0000 7ffb0b56d740 1 -- 172.21.15.32:0/2351591015 >> 172.21.15.32:0/2351591015 conn(0x5577868284e0 msgr2=0x557786bcff40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:12.782 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.780+0000 7ffae3400640 0 client.26449 ms_handle_reset on v2:172.21.15.71:6836/2324039932 2024-04-30T08:38:12.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.782+0000 7ffb0b56d740 1 -- 172.21.15.32:0/2351591015 shutdown_connections 2024-04-30T08:38:12.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.783+0000 7ffb0b56d740 1 -- 172.21.15.32:0/2351591015 wait complete. 2024-04-30T08:38:12.785 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.784+0000 7ffb0b56d740 20 client.26449 trim_cache size 0 max 16384 2024-04-30T08:38:12.938 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.935+0000 7ffb0b56d740 1 Processor -- start 2024-04-30T08:38:12.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.937+0000 7ffb0b56d740 1 -- start start 2024-04-30T08:38:12.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.937+0000 7ffb0b56d740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5577868136b0 0x557786bd0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:12.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.937+0000 7ffb0b56d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557786834390 0x557786bcfc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:12.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.938+0000 7ffb01000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557786834390 0x557786bcfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:12.940 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.938+0000 7ffb01000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557786834390 0x557786bcfc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49170/0 (socket says 172.21.15.32:49170) 2024-04-30T08:38:12.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.938+0000 7ffb01a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5577868136b0 0x557786bd0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:12.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.938+0000 7ffb01000640 1 -- 172.21.15.32:0/629651334 learned_addr learned my addr 172.21.15.32:0/629651334 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:12.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.938+0000 7ffb0b56d740 1 --2- 172.21.15.32:0/629651334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577867c83a0 0x557786be6740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:12.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.939+0000 7ffb0b56d740 1 -- 172.21.15.32:0/629651334 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ffafc1b3120 con 0x557786834390 2024-04-30T08:38:12.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.939+0000 7ffb0b56d740 1 -- 172.21.15.32:0/629651334 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ffafc1b4cf0 con 0x5577868136b0 2024-04-30T08:38:12.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.939+0000 7ffb0b56d740 1 -- 172.21.15.32:0/629651334 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ffafc1b37f0 con 0x5577867c83a0 2024-04-30T08:38:12.941 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.939+0000 7ffb02400640 1 --2- 172.21.15.32:0/629651334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577867c83a0 0x557786be6740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:12.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:12 smithi032 ceph-mon[28331]: pgmap v516: 129 pgs: 129 active+clean; 12 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 277 MiB/s wr, 100 op/s 2024-04-30T08:38:12.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.939+0000 7ffb02400640 1 -- 172.21.15.32:0/629651334 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5577868136b0 msgr2=0x557786bd0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:12.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.940+0000 7ffb02400640 1 --2- 172.21.15.32:0/629651334 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5577868136b0 0x557786bd0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:12.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.940+0000 7ffb02400640 1 -- 172.21.15.32:0/629651334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557786834390 msgr2=0x557786bcfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:12.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.940+0000 7ffb02400640 1 --2- 172.21.15.32:0/629651334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557786834390 0x557786bcfc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:12.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.940+0000 7ffb02400640 1 -- 172.21.15.32:0/629651334 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557786bd5120 con 0x5577867c83a0 2024-04-30T08:38:12.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.940+0000 7ffb01000640 1 --2- 172.21.15.32:0/629651334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557786834390 0x557786bcfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:38:12.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.941+0000 7ffb02400640 1 --2- 172.21.15.32:0/629651334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577867c83a0 0x557786be6740 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ffae800f790 tx=0x7ffae8012000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=37c527e48268320e server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:12.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.941+0000 7ffaee000640 1 -- 172.21.15.32:0/629651334 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae8016070 con 0x5577867c83a0 2024-04-30T08:38:12.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.941+0000 7ffaee000640 1 -- 172.21.15.32:0/629651334 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7ffae8010e00 con 0x5577867c83a0 2024-04-30T08:38:12.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.941+0000 7ffaee000640 1 -- 172.21.15.32:0/629651334 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffae801a8d0 con 0x5577867c83a0 2024-04-30T08:38:12.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.947+0000 7ffb0b56d740 1 -- 172.21.15.32:0/629651334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577867c83a0 msgr2=0x557786be6740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:12.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.947+0000 7ffb0b56d740 1 --2- 172.21.15.32:0/629651334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577867c83a0 0x557786be6740 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ffae800f790 tx=0x7ffae8012000 comp rx=0 tx=0).stop 2024-04-30T08:38:12.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.950+0000 7ffb0b56d740 1 -- 172.21.15.32:0/629651334 shutdown_connections 2024-04-30T08:38:12.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.950+0000 7ffb0b56d740 1 --2- 172.21.15.32:0/629651334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5577867c83a0 0x557786be6740 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:12.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.950+0000 7ffb0b56d740 1 --2- 172.21.15.32:0/629651334 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x557786834390 0x557786bcfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:12.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.950+0000 7ffb0b56d740 1 --2- 172.21.15.32:0/629651334 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5577868136b0 0x557786bd0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:12.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.950+0000 7ffb0b56d740 1 -- 172.21.15.32:0/629651334 >> 172.21.15.32:0/629651334 conn(0x5577868284e0 msgr2=0x557786bcf510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:12.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.951+0000 7ffb0b56d740 1 -- 172.21.15.32:0/629651334 shutdown_connections 2024-04-30T08:38:12.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.951+0000 7ffb0b56d740 1 -- 172.21.15.32:0/629651334 wait complete. 2024-04-30T08:38:12.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.951+0000 7ffb0b56d740 1 -- start start 2024-04-30T08:38:12.953 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.952+0000 7ffb0b56d740 1 Processor -- start 2024-04-30T08:38:12.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.954+0000 7ffb0b56d740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5577867c83a0 0x557786be6e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:12.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.954+0000 7ffb0b56d740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5577868136b0 0x557786be39d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:12.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.955+0000 7ffb01a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5577867c83a0 0x557786be6e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:12.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.955+0000 7ffb01a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5577867c83a0 0x557786be6e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:36528/0 (socket says 172.21.15.32:36528) 2024-04-30T08:38:12.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.955+0000 7ffb01000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5577868136b0 0x557786be39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:12.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.955+0000 7ffb01000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5577868136b0 0x557786be39d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49182/0 (socket says 172.21.15.32:49182) 2024-04-30T08:38:12.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.955+0000 7ffb01a00640 1 -- 172.21.15.32:0/1211117540 learned_addr learned my addr 172.21.15.32:0/1211117540 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:12.957 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.955+0000 7ffb0b56d740 1 --2- 172.21.15.32:0/1211117540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557786834390 0x557786be3f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:12.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.956+0000 7ffb0b56d740 1 -- 172.21.15.32:0/1211117540 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7ffafc1b4590 con 0x5577868136b0 2024-04-30T08:38:12.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.956+0000 7ffb0b56d740 1 -- 172.21.15.32:0/1211117540 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7ffafc1b3ec0 con 0x5577867c83a0 2024-04-30T08:38:12.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.956+0000 7ffb0b56d740 1 -- 172.21.15.32:0/1211117540 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7ffafc1b10e0 con 0x557786834390 2024-04-30T08:38:12.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.956+0000 7ffb02400640 1 --2- 172.21.15.32:0/1211117540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557786834390 0x557786be3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:12.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.957+0000 7ffb01000640 1 -- 172.21.15.32:0/1211117540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557786834390 msgr2=0x557786be3f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:12.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.957+0000 7ffb01000640 1 --2- 172.21.15.32:0/1211117540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557786834390 0x557786be3f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:12.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.957+0000 7ffb01000640 1 -- 172.21.15.32:0/1211117540 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5577867c83a0 msgr2=0x557786be6e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:12.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.957+0000 7ffb01000640 1 --2- 172.21.15.32:0/1211117540 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5577867c83a0 0x557786be6e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:12.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.957+0000 7ffb01000640 1 -- 172.21.15.32:0/1211117540 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557786c74e60 con 0x5577868136b0 2024-04-30T08:38:12.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.957+0000 7ffb01000640 1 --2- 172.21.15.32:0/1211117540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5577868136b0 0x557786be39d0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7ffaf40d6a50 tx=0x7ffaf4056f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb42ab439eeccf77 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:12.964 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.958+0000 7ffb0b56d740 10 client.0 _ll_register_callbacks cb 0x7ffaf000b670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:38:12.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.958+0000 7ffb0b56d740 1 -- 172.21.15.32:0/1211117540 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffae8010e00 con 0x5577868136b0 2024-04-30T08:38:12.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.961+0000 7ffae3400640 10 client.16299 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:38:12.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.961+0000 7ffae3400640 1 -- 172.21.15.32:0/1211117540 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffaf40d9f70 con 0x5577868136b0 2024-04-30T08:38:12.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.961+0000 7ffae3400640 1 -- 172.21.15.32:0/1211117540 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7ffaf40d5a70 con 0x5577868136b0 2024-04-30T08:38:12.965 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.963+0000 7ffae3400640 1 -- 172.21.15.32:0/1211117540 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7ffaf40d9870 con 0x5577868136b0 2024-04-30T08:38:12.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.966+0000 7ffb0b56d740 10 client.16299 fetch_fsmap learned FSMap version 29 2024-04-30T08:38:12.968 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.966+0000 7ffb0b56d740 1 -- 172.21.15.32:0/1211117540 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({fsmap=29}) -- 0x557786c75e70 con 0x5577868136b0 2024-04-30T08:38:12.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.968+0000 7ffae3400640 1 -- 172.21.15.32:0/1211117540 <== mon.0 v2:172.21.15.32:3300/0 4 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7ffaf405e070 con 0x5577868136b0 2024-04-30T08:38:12.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.968+0000 7ffb0b56d740 10 client.16299 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:38:12.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.968+0000 7ffb0b56d740 10 client.16299 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.h 2024-04-30T08:38:12.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.968+0000 7ffb0b56d740 20 client.16299 populate_metadata read hostname 'smithi032' 2024-04-30T08:38:12.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.968+0000 7ffb0b56d740 1 --2- 172.21.15.32:0/1211117540 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x557786c76730 0x557786c78be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:12.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.968+0000 7ffb0b56d740 4 client.16299 mds_command: new command op to 24533 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-30T08:38:12.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.968+0000 7ffb0b56d740 1 -- 172.21.15.32:0/1211117540 --> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557786b427d0 con 0x557786c76730 2024-04-30T08:38:12.970 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.968+0000 7ffb01a00640 1 --2- 172.21.15.32:0/1211117540 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x557786c76730 0x557786c78be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:12.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.969+0000 7ffae3400640 10 client.16299 ms_handle_connect on v2:172.21.15.71:6836/2324039932 2024-04-30T08:38:12.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.969+0000 7ffb01a00640 1 --2- 172.21.15.32:0/1211117540 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x557786c76730 0x557786c78be0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=dde3ae5ba60ceeb0 server_cookie=4926cf0c812c90bf in_seq=0 out_seq=0 2024-04-30T08:38:12.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.970+0000 7ffae3400640 1 -- 172.21.15.32:0/1211117540 <== mds.3 v2:172.21.15.71:6836/2324039932 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557786b427d0 con 0x557786c76730 2024-04-30T08:38:12.971 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.970+0000 7ffae3400640 10 client.16299 handle_command_reply: tid=0 2024-04-30T08:38:12.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.970+0000 7ffb0b56d740 1 client.16299 shutdown 2024-04-30T08:38:12.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.970+0000 7ffb0b56d740 1 -- 172.21.15.32:0/1211117540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5577868136b0 msgr2=0x557786be39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:12.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.970+0000 7ffb0b56d740 1 --2- 172.21.15.32:0/1211117540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5577868136b0 0x557786be39d0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7ffaf40d6a50 tx=0x7ffaf4056f60 comp rx=0 tx=0).stop 2024-04-30T08:38:12.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.971+0000 7ffb0b56d740 1 -- 172.21.15.32:0/1211117540 shutdown_connections 2024-04-30T08:38:12.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.971+0000 7ffb0b56d740 1 --2- 172.21.15.32:0/1211117540 >> [v2:172.21.15.71:6836/2324039932,v1:172.21.15.71:6837/2324039932] conn(0x557786c76730 0x557786c78be0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:12.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.971+0000 7ffae3400640 0 client.16299 ms_handle_reset on v2:172.21.15.71:6836/2324039932 2024-04-30T08:38:12.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.971+0000 7ffb0b56d740 1 --2- 172.21.15.32:0/1211117540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557786834390 0x557786be3f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:12.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.971+0000 7ffb0b56d740 1 --2- 172.21.15.32:0/1211117540 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x5577868136b0 0x557786be39d0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:12.973 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.971+0000 7ffb0b56d740 1 --2- 172.21.15.32:0/1211117540 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x5577867c83a0 0x557786be6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:12.974 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.971+0000 7ffb0b56d740 1 -- 172.21.15.32:0/1211117540 >> 172.21.15.32:0/1211117540 conn(0x5577868284e0 msgr2=0x557786bde120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:12.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.973+0000 7ffb0b56d740 1 -- 172.21.15.32:0/1211117540 shutdown_connections 2024-04-30T08:38:12.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.973+0000 7ffb0b56d740 1 -- 172.21.15.32:0/1211117540 wait complete. 2024-04-30T08:38:12.975 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.973+0000 7ffb0b56d740 20 client.16299 trim_cache size 0 max 16384 2024-04-30T08:38:12.981 INFO:teuthology.orchestra.run.smithi032.stdout:[] 2024-04-30T08:38:12.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.982+0000 7ffb04400640 1 -- 172.21.15.32:0/1019517126 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ffae40afd00 msgr2=0x7ffae40b2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:12.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.982+0000 7ffb04400640 1 --2- 172.21.15.32:0/1019517126 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ffae40afd00 0x7ffae40b2210 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ffae800ab40 tx=0x7ffae800a250 comp rx=0 tx=0).stop 2024-04-30T08:38:12.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.982+0000 7ffb04400640 1 -- 172.21.15.32:0/1019517126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffafc10b390 msgr2=0x7ffafc1a0ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:12.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.982+0000 7ffb04400640 1 --2- 172.21.15.32:0/1019517126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffafc10b390 0x7ffafc1a0ec0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ffaf403bf10 tx=0x7ffaf4040b30 comp rx=0 tx=0).stop 2024-04-30T08:38:12.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.983+0000 7ffb04400640 1 -- 172.21.15.32:0/1019517126 shutdown_connections 2024-04-30T08:38:12.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.983+0000 7ffb04400640 1 --2- 172.21.15.32:0/1019517126 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7ffae40afd00 0x7ffae40b2210 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:12.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.983+0000 7ffb04400640 1 --2- 172.21.15.32:0/1019517126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffafc1100b0 0x7ffafc1a7e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:12.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.983+0000 7ffb04400640 1 --2- 172.21.15.32:0/1019517126 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7ffafc10b390 0x7ffafc1a0ec0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:12.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.983+0000 7ffb04400640 1 --2- 172.21.15.32:0/1019517126 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7ffafc10a4a0 0x7ffafc1a0980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:12.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.983+0000 7ffb04400640 1 -- 172.21.15.32:0/1019517126 >> 172.21.15.32:0/1019517126 conn(0x7ffafc070560 msgr2=0x7ffafc10e0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:12.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.987+0000 7ffb04400640 1 -- 172.21.15.32:0/1019517126 shutdown_connections 2024-04-30T08:38:12.989 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:12.987+0000 7ffb04400640 1 -- 172.21.15.32:0/1019517126 wait complete. 2024-04-30T08:38:13.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:12 smithi071 ceph-mon[33385]: pgmap v516: 129 pgs: 129 active+clean; 12 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 277 MiB/s wr, 100 op/s 2024-04-30T08:38:13.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:12 smithi189 ceph-mon[33038]: pgmap v516: 129 pgs: 129 active+clean; 12 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 28 KiB/s rd, 277 MiB/s wr, 100 op/s 2024-04-30T08:38:13.056 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-30T08:38:13.637 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:13 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-30T08:38:13.637 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:13 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:38:13.637 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:13 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:38:14.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:13 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-30T08:38:14.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:13 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:38:14.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:13 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:38:14.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:13 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-30T08:38:14.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:13 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-30T08:38:14.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:13 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-30T08:38:14.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.812+0000 7f639dc00640 1 -- 172.21.15.32:0/2607268244 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f639810a3c0 msgr2=0x7f639810acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:14.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.812+0000 7f639dc00640 1 --2- 172.21.15.32:0/2607268244 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f639810a3c0 0x7f639810acd0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6388009900 tx=0x7f638802a420 comp rx=0 tx=0).stop 2024-04-30T08:38:14.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.816+0000 7f639dc00640 1 -- 172.21.15.32:0/2607268244 shutdown_connections 2024-04-30T08:38:14.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.816+0000 7f639dc00640 1 --2- 172.21.15.32:0/2607268244 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f639810ff10 0x7f6398110320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:14.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.816+0000 7f639dc00640 1 --2- 172.21.15.32:0/2607268244 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f639810b210 0x7f639810b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:14.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.816+0000 7f639dc00640 1 --2- 172.21.15.32:0/2607268244 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f639810a3c0 0x7f639810acd0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:14.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.816+0000 7f639dc00640 1 -- 172.21.15.32:0/2607268244 >> 172.21.15.32:0/2607268244 conn(0x7f63980703a0 msgr2=0x7f6398072850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:14.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.817+0000 7f639dc00640 1 -- 172.21.15.32:0/2607268244 shutdown_connections 2024-04-30T08:38:14.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.817+0000 7f639dc00640 1 -- 172.21.15.32:0/2607268244 wait complete. 2024-04-30T08:38:14.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.831+0000 7f639dc00640 1 Processor -- start 2024-04-30T08:38:14.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.833+0000 7f639dc00640 1 -- start start 2024-04-30T08:38:14.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.833+0000 7f639dc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f639810a3c0 0x7f6398119f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:14.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.833+0000 7f639dc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f639810b210 0x7f639811a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:14.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.833+0000 7f639dc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f639810ff10 0x7f639811b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:14.840 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.836+0000 7f639dc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f6398112090 con 0x7f639810ff10 2024-04-30T08:38:14.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.836+0000 7f639dc00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f6398111f10 con 0x7f639810a3c0 2024-04-30T08:38:14.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.836+0000 7f639dc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f6398112210 con 0x7f639810b210 2024-04-30T08:38:14.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.836+0000 7f639d200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f639810ff10 0x7f639811b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:14.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.836+0000 7f639d200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f639810ff10 0x7f639811b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49196/0 (socket says 172.21.15.32:49196) 2024-04-30T08:38:14.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.836+0000 7f639d200640 1 -- 172.21.15.32:0/533805471 learned_addr learned my addr 172.21.15.32:0/533805471 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:14.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.840+0000 7f639d200640 1 -- 172.21.15.32:0/533805471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f639810b210 msgr2=0x7f639811a490 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-30T08:38:14.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.840+0000 7f639d200640 1 --2- 172.21.15.32:0/533805471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f639810b210 0x7f639811a490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:14.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.840+0000 7f639d200640 1 -- 172.21.15.32:0/533805471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f639810a3c0 msgr2=0x7f6398119f50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-30T08:38:14.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.840+0000 7f639d200640 1 --2- 172.21.15.32:0/533805471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f639810a3c0 0x7f6398119f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:14.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.840+0000 7f639d200640 1 -- 172.21.15.32:0/533805471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f639811ba60 con 0x7f639810ff10 2024-04-30T08:38:14.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.842+0000 7f639d200640 1 --2- 172.21.15.32:0/533805471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f639810ff10 0x7f639811b3f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f639000cad0 tx=0x7f639000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:14.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.852+0000 7f6394c00640 1 -- 172.21.15.32:0/533805471 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f639000ae20 con 0x7f639810ff10 2024-04-30T08:38:14.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.852+0000 7f6394c00640 1 -- 172.21.15.32:0/533805471 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f6390005020 con 0x7f639810ff10 2024-04-30T08:38:14.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.852+0000 7f6394c00640 1 -- 172.21.15.32:0/533805471 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f639000f6b0 con 0x7f639810ff10 2024-04-30T08:38:14.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.879+0000 7f639dc00640 1 -- 172.21.15.32:0/533805471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f639811bd50 con 0x7f639810ff10 2024-04-30T08:38:14.886 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.879+0000 7f639dc00640 1 -- 172.21.15.32:0/533805471 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f639811c220 con 0x7f639810ff10 2024-04-30T08:38:14.891 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.890+0000 7f63a6230740 1 Processor -- start 2024-04-30T08:38:14.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.890+0000 7f63a6230740 1 -- start start 2024-04-30T08:38:14.892 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.891+0000 7f63a6230740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d734bee390 0x55d734f8f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:14.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.891+0000 7f63a6230740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d734bc54d0 0x55d734f8fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:14.893 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.891+0000 7f63a6230740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d734bcd6b0 0x55d734f96bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:14.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.892+0000 7f6397e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d734bc54d0 0x55d734f8fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:14.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.892+0000 7f6397e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d734bc54d0 0x55d734f8fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:36554/0 (socket says 172.21.15.32:36554) 2024-04-30T08:38:14.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.892+0000 7f6397400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d734bee390 0x55d734f8f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:14.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.892+0000 7f6397400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d734bee390 0x55d734f8f730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:32978/0 (socket says 172.21.15.32:32978) 2024-04-30T08:38:14.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.893+0000 7f63a6230740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d734ef4d50 con 0x55d734bcd6b0 2024-04-30T08:38:14.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.893+0000 7f63a6230740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55d734e9a0c0 con 0x55d734bc54d0 2024-04-30T08:38:14.894 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.893+0000 7f63a6230740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d734ef0d10 con 0x55d734bee390 2024-04-30T08:38:14.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.893+0000 7f6397e00640 1 -- 172.21.15.32:0/841909865 learned_addr learned my addr 172.21.15.32:0/841909865 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:14.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.893+0000 7f6397400640 1 -- 172.21.15.32:0/841909865 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d734bc54d0 msgr2=0x55d734f8fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:14.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.893+0000 7f6397400640 1 --2- 172.21.15.32:0/841909865 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d734bc54d0 0x55d734f8fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:14.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.893+0000 7f6397400640 1 -- 172.21.15.32:0/841909865 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d734bcd6b0 msgr2=0x55d734f96bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:14.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.893+0000 7f6397400640 1 --2- 172.21.15.32:0/841909865 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d734bcd6b0 0x55d734f96bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:14.895 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.893+0000 7f6397400640 1 -- 172.21.15.32:0/841909865 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d734c063f0 con 0x55d734bee390 2024-04-30T08:38:14.896 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.894+0000 7f6397400640 1 --2- 172.21.15.32:0/841909865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d734bee390 0x55d734f8f730 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6384009bd0 tx=0x7f63840107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8a86c9d9047cf39 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:14.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.898+0000 7f638d600640 1 -- 172.21.15.32:0/841909865 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6384005020 con 0x55d734bee390 2024-04-30T08:38:14.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.898+0000 7f638d600640 1 -- 172.21.15.32:0/841909865 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63840047a0 con 0x55d734bee390 2024-04-30T08:38:14.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.898+0000 7f638d600640 1 -- 172.21.15.32:0/841909865 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63840196b0 con 0x55d734bee390 2024-04-30T08:38:14.906 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.903+0000 7f6394c00640 1 -- 172.21.15.32:0/533805471 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f63900051d0 con 0x7f639810ff10 2024-04-30T08:38:14.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.905+0000 7f6394c00640 1 --2- 172.21.15.32:0/533805471 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f637c0afc70 0x7f637c0b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:14.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.906+0000 7f63a6230740 1 -- 172.21.15.32:0/841909865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d734bee390 msgr2=0x55d734f8f730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:14.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.906+0000 7f63a6230740 1 --2- 172.21.15.32:0/841909865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d734bee390 0x55d734f8f730 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f6384009bd0 tx=0x7f63840107a0 comp rx=0 tx=0).stop 2024-04-30T08:38:14.909 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.907+0000 7f6394c00640 1 -- 172.21.15.32:0/533805471 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f63900a3a90 con 0x7f639810ff10 2024-04-30T08:38:14.924 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.919+0000 7f6397e00640 1 --2- 172.21.15.32:0/533805471 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f637c0afc70 0x7f637c0b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:14.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.928+0000 7f63a6230740 1 -- 172.21.15.32:0/841909865 shutdown_connections 2024-04-30T08:38:14.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.928+0000 7f63a6230740 1 --2- 172.21.15.32:0/841909865 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d734bcd6b0 0x55d734f96bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:14.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.928+0000 7f63a6230740 1 --2- 172.21.15.32:0/841909865 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d734bc54d0 0x55d734f8fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:14.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.928+0000 7f63a6230740 1 --2- 172.21.15.32:0/841909865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d734bee390 0x55d734f8f730 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:14.936 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.928+0000 7f63a6230740 1 -- 172.21.15.32:0/841909865 >> 172.21.15.32:0/841909865 conn(0x55d734be24e0 msgr2=0x55d734f8b0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:14.980 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.978+0000 7f63a6230740 1 -- 172.21.15.32:0/841909865 shutdown_connections 2024-04-30T08:38:14.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.987+0000 7f6397e00640 1 --2- 172.21.15.32:0/533805471 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f637c0afc70 0x7f637c0b2180 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f6388004c10 tx=0x7f6388005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:14.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.995+0000 7f63a6230740 1 -- 172.21.15.32:0/841909865 wait complete. 2024-04-30T08:38:14.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.996+0000 7f63a6230740 1 -- start start 2024-04-30T08:38:14.998 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:14.996+0000 7f63a6230740 1 Processor -- start 2024-04-30T08:38:15.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.004+0000 7f63a6230740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d734bc54d0 0x55d734f99740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:15.006 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.004+0000 7f63a6230740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d734bcd6b0 0x55d734f99c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:15.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.008+0000 7f63a6230740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d734bee390 0x55d734fa0be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:15.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.008+0000 7f63a6230740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55d734f97600 con 0x55d734bee390 2024-04-30T08:38:15.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.008+0000 7f63a6230740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55d734eadb50 con 0x55d734bc54d0 2024-04-30T08:38:15.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.008+0000 7f63a6230740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55d734f97780 con 0x55d734bcd6b0 2024-04-30T08:38:15.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.008+0000 7f63a6230740 10 client.0 _ll_register_callbacks cb 0x55d734e77420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:38:15.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.011+0000 7f6397e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d734bc54d0 0x55d734f99740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:15.014 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.011+0000 7f6397e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d734bc54d0 0x55d734f99740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:36562/0 (socket says 172.21.15.32:36562) 2024-04-30T08:38:15.015 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.011+0000 7f6397e00640 1 -- 172.21.15.32:0/2628578873 learned_addr learned my addr 172.21.15.32:0/2628578873 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:15.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.014+0000 7f6397400640 1 --2- 172.21.15.32:0/2628578873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d734bcd6b0 0x55d734f99c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:15.016 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.015+0000 7f639d200640 1 --2- 172.21.15.32:0/2628578873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d734bee390 0x55d734fa0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:15.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.016+0000 7f6397e00640 1 -- 172.21.15.32:0/2628578873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d734bcd6b0 msgr2=0x55d734f99c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:15.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.016+0000 7f6397e00640 1 --2- 172.21.15.32:0/2628578873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d734bcd6b0 0x55d734f99c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.016+0000 7f6397e00640 1 -- 172.21.15.32:0/2628578873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d734bee390 msgr2=0x55d734fa0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:15.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.016+0000 7f6397e00640 1 --2- 172.21.15.32:0/2628578873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d734bee390 0x55d734fa0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.021 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.016+0000 7f6397e00640 1 -- 172.21.15.32:0/2628578873 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d734cd07d0 con 0x55d734bc54d0 2024-04-30T08:38:15.024 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.022+0000 7f6397e00640 1 --2- 172.21.15.32:0/2628578873 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d734bc54d0 0x55d734f99740 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f6388009230 tx=0x7f6388004b10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f503b1c3b722484 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:15.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.028+0000 7f6382000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:38:15.035 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.028+0000 7f6382000640 1 -- 172.21.15.32:0/2628578873 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638804e070 con 0x55d734bc54d0 2024-04-30T08:38:15.036 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.028+0000 7f6382000640 1 -- 172.21.15.32:0/2628578873 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f638804a860 con 0x55d734bc54d0 2024-04-30T08:38:15.061 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.043+0000 7f63a6230740 1 -- 172.21.15.32:0/2628578873 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63840047a0 con 0x55d734bc54d0 2024-04-30T08:38:15.062 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.058+0000 7f6382000640 1 -- 172.21.15.32:0/2628578873 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6388058050 con 0x55d734bc54d0 2024-04-30T08:38:15.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.071+0000 7f63a6230740 10 client.26465 fetch_fsmap learned FSMap version 29 2024-04-30T08:38:15.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.071+0000 7f63a6230740 1 -- 172.21.15.32:0/2628578873 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d734fa3b80 con 0x55d734bc54d0 2024-04-30T08:38:15.079 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.076+0000 7f6382000640 1 -- 172.21.15.32:0/2628578873 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63840047a0 con 0x55d734bc54d0 2024-04-30T08:38:15.085 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.083+0000 7f6382000640 1 -- 172.21.15.32:0/2628578873 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7f638804aa60 con 0x55d734bc54d0 2024-04-30T08:38:15.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.086+0000 7f63a6230740 10 client.26465 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:38:15.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.086+0000 7f63a6230740 10 client.26465 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-30T08:38:15.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.086+0000 7f63a6230740 20 client.26465 populate_metadata read hostname 'smithi032' 2024-04-30T08:38:15.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.086+0000 7f63a6230740 1 --2- 172.21.15.32:0/2628578873 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x55d734bacf60 0x55d734fa7770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:15.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.086+0000 7f63a6230740 4 client.26465 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:38:15.090 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.086+0000 7f63a6230740 1 -- 172.21.15.32:0/2628578873 --> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d734f02960 con 0x55d734bacf60 2024-04-30T08:38:15.097 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.096+0000 7f6397400640 1 --2- 172.21.15.32:0/2628578873 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x55d734bacf60 0x55d734fa7770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:15.104 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.103+0000 7f6382000640 10 client.26465 ms_handle_connect on v2:172.21.15.189:6838/352970107 2024-04-30T08:38:15.105 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.103+0000 7f6397400640 1 --2- 172.21.15.32:0/2628578873 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x55d734bacf60 0x55d734fa7770 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=65bd60455103154 server_cookie=e839a4341fb3d35f in_seq=0 out_seq=0 2024-04-30T08:38:15.110 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.109+0000 7f6382000640 1 -- 172.21.15.32:0/2628578873 <== mds.2 v2:172.21.15.189:6838/352970107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55d734f02960 con 0x55d734bacf60 2024-04-30T08:38:15.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.109+0000 7f6382000640 10 client.26465 handle_command_reply: tid=0 2024-04-30T08:38:15.111 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.109+0000 7f63a6230740 1 client.26465 shutdown 2024-04-30T08:38:15.114 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.112+0000 7f63a6230740 1 -- 172.21.15.32:0/2628578873 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d734bc54d0 msgr2=0x55d734f99740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:15.115 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.114+0000 7f63a6230740 1 --2- 172.21.15.32:0/2628578873 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d734bc54d0 0x55d734f99740 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f6388009230 tx=0x7f6388004b10 comp rx=0 tx=0).stop 2024-04-30T08:38:15.116 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.115+0000 7f63a6230740 1 -- 172.21.15.32:0/2628578873 shutdown_connections 2024-04-30T08:38:15.117 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.116+0000 7f63a6230740 1 --2- 172.21.15.32:0/2628578873 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x55d734bacf60 0x55d734fa7770 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.118 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.116+0000 7f6382000640 0 client.26465 ms_handle_reset on v2:172.21.15.189:6838/352970107 2024-04-30T08:38:15.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.117+0000 7f63a6230740 1 --2- 172.21.15.32:0/2628578873 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d734bee390 0x55d734fa0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.119 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.117+0000 7f63a6230740 1 --2- 172.21.15.32:0/2628578873 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d734bcd6b0 0x55d734f99c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.117+0000 7f63a6230740 1 --2- 172.21.15.32:0/2628578873 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d734bc54d0 0x55d734f99740 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.120 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.117+0000 7f63a6230740 1 -- 172.21.15.32:0/2628578873 >> 172.21.15.32:0/2628578873 conn(0x55d734be24e0 msgr2=0x55d734f8a100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:15.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.121+0000 7f63a6230740 1 -- 172.21.15.32:0/2628578873 shutdown_connections 2024-04-30T08:38:15.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.123+0000 7f63a6230740 1 -- 172.21.15.32:0/2628578873 wait complete. 2024-04-30T08:38:15.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.123+0000 7f63a6230740 20 client.26465 trim_cache size 0 max 16384 2024-04-30T08:38:15.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.353+0000 7f63a6230740 1 Processor -- start 2024-04-30T08:38:15.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.355+0000 7f63a6230740 1 -- start start 2024-04-30T08:38:15.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.356+0000 7f63a6230740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d734bc54d0 0x55d734f8b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:15.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.356+0000 7f63a6230740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d734bcd6b0 0x55d734fa0b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:15.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.356+0000 7f63a6230740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d734bee390 0x55d734fa10b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:15.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.356+0000 7f63a6230740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f63981c3f90 con 0x55d734bcd6b0 2024-04-30T08:38:15.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.356+0000 7f63a6230740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f63981c5b60 con 0x55d734bee390 2024-04-30T08:38:15.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.356+0000 7f63a6230740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f63981c4660 con 0x55d734bc54d0 2024-04-30T08:38:15.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.358+0000 7f639d200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d734bee390 0x55d734fa10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:15.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.358+0000 7f639d200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d734bee390 0x55d734fa10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:36574/0 (socket says 172.21.15.32:36574) 2024-04-30T08:38:15.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.358+0000 7f639d200640 1 -- 172.21.15.32:0/2011255410 learned_addr learned my addr 172.21.15.32:0/2011255410 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:15.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.360+0000 7f6397e00640 1 --2- 172.21.15.32:0/2011255410 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d734bc54d0 0x55d734f8b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:15.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.360+0000 7f639d200640 1 -- 172.21.15.32:0/2011255410 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d734bc54d0 msgr2=0x55d734f8b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:15.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.360+0000 7f639d200640 1 --2- 172.21.15.32:0/2011255410 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d734bc54d0 0x55d734f8b160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.360+0000 7f639d200640 1 -- 172.21.15.32:0/2011255410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d734bcd6b0 msgr2=0x55d734fa0b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:15.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.360+0000 7f639d200640 1 --2- 172.21.15.32:0/2011255410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d734bcd6b0 0x55d734fa0b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.360+0000 7f639d200640 1 -- 172.21.15.32:0/2011255410 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d734f97b50 con 0x55d734bee390 2024-04-30T08:38:15.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.363+0000 7f639d200640 1 --2- 172.21.15.32:0/2011255410 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d734bee390 0x55d734fa10b0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f63900a7430 tx=0x7f63900aeb60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94a7f342ea09fe65 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:15.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.370+0000 7f638cc00640 1 -- 172.21.15.32:0/2011255410 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63900a3c90 con 0x55d734bee390 2024-04-30T08:38:15.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.370+0000 7f638cc00640 1 -- 172.21.15.32:0/2011255410 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63900ab850 con 0x55d734bee390 2024-04-30T08:38:15.371 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.370+0000 7f638cc00640 1 -- 172.21.15.32:0/2011255410 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63900a3e30 con 0x55d734bee390 2024-04-30T08:38:15.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.371+0000 7f63a6230740 1 -- 172.21.15.32:0/2011255410 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d734bee390 msgr2=0x55d734fa10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:15.376 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.371+0000 7f63a6230740 1 --2- 172.21.15.32:0/2011255410 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d734bee390 0x55d734fa10b0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f63900a7430 tx=0x7f63900aeb60 comp rx=0 tx=0).stop 2024-04-30T08:38:15.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.390+0000 7f63a6230740 1 -- 172.21.15.32:0/2011255410 shutdown_connections 2024-04-30T08:38:15.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.390+0000 7f63a6230740 1 --2- 172.21.15.32:0/2011255410 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d734bee390 0x55d734fa10b0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.390+0000 7f63a6230740 1 --2- 172.21.15.32:0/2011255410 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d734bcd6b0 0x55d734fa0b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.390+0000 7f63a6230740 1 --2- 172.21.15.32:0/2011255410 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d734bc54d0 0x55d734f8b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.390+0000 7f63a6230740 1 -- 172.21.15.32:0/2011255410 >> 172.21.15.32:0/2011255410 conn(0x55d734be24e0 msgr2=0x55d734f8aa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:15.395 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.394+0000 7f63a6230740 1 -- 172.21.15.32:0/2011255410 shutdown_connections 2024-04-30T08:38:15.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.402+0000 7f63a6230740 1 -- 172.21.15.32:0/2011255410 wait complete. 2024-04-30T08:38:15.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.402+0000 7f63a6230740 1 -- start start 2024-04-30T08:38:15.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.403+0000 7f63a6230740 1 Processor -- start 2024-04-30T08:38:15.405 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.403+0000 7f63a6230740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d734bc54d0 0x55d734f9e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:15.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.407+0000 7f63a6230740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d734bcd6b0 0x55d734f9e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:15.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.410+0000 7f6397e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d734bc54d0 0x55d734f9e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:15.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.410+0000 7f6397e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d734bc54d0 0x55d734f9e390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49228/0 (socket says 172.21.15.32:49228) 2024-04-30T08:38:15.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.411+0000 7f63a6230740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d734bee390 0x55d734f9f800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:15.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.411+0000 7f63a6230740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f63981c5400 con 0x55d734bc54d0 2024-04-30T08:38:15.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.411+0000 7f63a6230740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f63981c4d30 con 0x55d734bee390 2024-04-30T08:38:15.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.411+0000 7f63a6230740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f63981c1f50 con 0x55d734bcd6b0 2024-04-30T08:38:15.413 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.411+0000 7f63a6230740 10 client.0 _ll_register_callbacks cb 0x7f6388041070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:38:15.414 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.413+0000 7f6397e00640 1 -- 172.21.15.32:0/1766815743 learned_addr learned my addr 172.21.15.32:0/1766815743 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:15.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.415+0000 7f639d200640 1 --2- 172.21.15.32:0/1766815743 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d734bee390 0x55d734f9f800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:15.417 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.415+0000 7f6397400640 1 --2- 172.21.15.32:0/1766815743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d734bcd6b0 0x55d734f9e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:15.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.419+0000 7f6397400640 1 -- 172.21.15.32:0/1766815743 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d734bee390 msgr2=0x55d734f9f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:15.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.419+0000 7f6397400640 1 --2- 172.21.15.32:0/1766815743 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d734bee390 0x55d734f9f800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.420 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.419+0000 7f6397400640 1 -- 172.21.15.32:0/1766815743 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d734bc54d0 msgr2=0x55d734f9e390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:15.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.421+0000 7f6397400640 1 --2- 172.21.15.32:0/1766815743 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d734bc54d0 0x55d734f9e390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.422 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.421+0000 7f6397400640 1 -- 172.21.15.32:0/1766815743 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d734f945c0 con 0x55d734bcd6b0 2024-04-30T08:38:15.423 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.421+0000 7f6397e00640 1 --2- 172.21.15.32:0/1766815743 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d734bc54d0 0x55d734f9e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:38:15.426 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.423+0000 7f639d200640 1 --2- 172.21.15.32:0/1766815743 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d734bee390 0x55d734f9f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-30T08:38:15.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.423+0000 7f6397400640 1 --2- 172.21.15.32:0/1766815743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d734bcd6b0 0x55d734f9e8d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x55d734f9a7b0 tx=0x7f6384009690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42fc158bdbe5e821 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:15.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.423+0000 7f6382000640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-30T08:38:15.428 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.425+0000 7f6382000640 1 -- 172.21.15.32:0/1766815743 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6384009e00 con 0x55d734bcd6b0 2024-04-30T08:38:15.429 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.426+0000 7f6382000640 1 -- 172.21.15.32:0/1766815743 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f6384004030 con 0x55d734bcd6b0 2024-04-30T08:38:15.437 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.432+0000 7f63a6230740 1 -- 172.21.15.32:0/1766815743 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f63900ab850 con 0x55d734bcd6b0 2024-04-30T08:38:15.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.442+0000 7f6382000640 1 -- 172.21.15.32:0/1766815743 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f6384004230 con 0x55d734bcd6b0 2024-04-30T08:38:15.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.442+0000 7f63a6230740 10 client.26467 fetch_fsmap learned FSMap version 29 2024-04-30T08:38:15.444 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.442+0000 7f63a6230740 1 -- 172.21.15.32:0/1766815743 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55d734f97810 con 0x55d734bcd6b0 2024-04-30T08:38:15.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.444+0000 7f6382000640 1 -- 172.21.15.32:0/1766815743 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f63900ab850 con 0x55d734bcd6b0 2024-04-30T08:38:15.445 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.444+0000 7f6382000640 1 -- 172.21.15.32:0/1766815743 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7f63840280f0 con 0x55d734bcd6b0 2024-04-30T08:38:15.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.445+0000 7f63a6230740 10 client.26467 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:38:15.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.445+0000 7f63a6230740 10 client.26467 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-30T08:38:15.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.445+0000 7f63a6230740 20 client.26467 populate_metadata read hostname 'smithi032' 2024-04-30T08:38:15.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.445+0000 7f63a6230740 1 --2- 172.21.15.32:0/1766815743 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x55d734b823a0 0x55d735011b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:15.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.445+0000 7f63a6230740 4 client.26467 mds_command: new command op to 24541 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-30T08:38:15.447 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.445+0000 7f63a6230740 1 -- 172.21.15.32:0/1766815743 --> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d734efc7d0 con 0x55d734b823a0 2024-04-30T08:38:15.449 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.448+0000 7f6397e00640 1 --2- 172.21.15.32:0/1766815743 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x55d734b823a0 0x55d735011b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:15.451 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.450+0000 7f6397e00640 1 --2- 172.21.15.32:0/1766815743 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x55d734b823a0 0x55d735011b30 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1a1cabc738311a9b server_cookie=f7c5cde2578c3b43 in_seq=0 out_seq=0 2024-04-30T08:38:15.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.450+0000 7f6382000640 10 client.26467 ms_handle_connect on v2:172.21.15.189:6838/352970107 2024-04-30T08:38:15.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.450+0000 7f6382000640 1 -- 172.21.15.32:0/1766815743 <== mds.2 v2:172.21.15.189:6838/352970107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d734efc7d0 con 0x55d734b823a0 2024-04-30T08:38:15.452 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.450+0000 7f6382000640 10 client.26467 handle_command_reply: tid=0 2024-04-30T08:38:15.461 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.460+0000 7f63a6230740 1 client.26467 shutdown 2024-04-30T08:38:15.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.464+0000 7f63a6230740 1 -- 172.21.15.32:0/1766815743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d734bcd6b0 msgr2=0x55d734f9e8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:15.466 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.464+0000 7f63a6230740 1 --2- 172.21.15.32:0/1766815743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d734bcd6b0 0x55d734f9e8d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x55d734f9a7b0 tx=0x7f6384009690 comp rx=0 tx=0).stop 2024-04-30T08:38:15.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.472+0000 7f63a6230740 1 -- 172.21.15.32:0/1766815743 shutdown_connections 2024-04-30T08:38:15.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.472+0000 7f63a6230740 1 --2- 172.21.15.32:0/1766815743 >> [v2:172.21.15.189:6838/352970107,v1:172.21.15.189:6839/352970107] conn(0x55d734b823a0 0x55d735011b30 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.472+0000 7f63a6230740 1 --2- 172.21.15.32:0/1766815743 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55d734bee390 0x55d734f9f800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.472+0000 7f63a6230740 1 --2- 172.21.15.32:0/1766815743 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d734bcd6b0 0x55d734f9e8d0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.472+0000 7f63a6230740 1 --2- 172.21.15.32:0/1766815743 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55d734bc54d0 0x55d734f9e390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.472+0000 7f63a6230740 1 -- 172.21.15.32:0/1766815743 >> 172.21.15.32:0/1766815743 conn(0x55d734be24e0 msgr2=0x55d734f8e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:15.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.476+0000 7f6382000640 0 client.26467 ms_handle_reset on v2:172.21.15.189:6838/352970107 2024-04-30T08:38:15.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.481+0000 7f63a6230740 1 -- 172.21.15.32:0/1766815743 shutdown_connections 2024-04-30T08:38:15.487 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.486+0000 7f63a6230740 1 -- 172.21.15.32:0/1766815743 wait complete. 2024-04-30T08:38:15.503 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.498+0000 7f63a6230740 20 client.26467 trim_cache size 0 max 16384 2024-04-30T08:38:15.511 INFO:teuthology.orchestra.run.smithi032.stdout:[] 2024-04-30T08:38:15.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.514+0000 7f639dc00640 1 -- 172.21.15.32:0/533805471 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f637c0afc70 msgr2=0x7f637c0b2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:15.522 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.514+0000 7f639dc00640 1 --2- 172.21.15.32:0/533805471 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f637c0afc70 0x7f637c0b2180 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f6388004c10 tx=0x7f6388005fd0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.517+0000 7f639dc00640 1 -- 172.21.15.32:0/533805471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f639810ff10 msgr2=0x7f639811b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:15.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.517+0000 7f639dc00640 1 --2- 172.21.15.32:0/533805471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f639810ff10 0x7f639811b3f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f639000cad0 tx=0x7f639000cf90 comp rx=0 tx=0).stop 2024-04-30T08:38:15.526 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.525+0000 7f639dc00640 1 -- 172.21.15.32:0/533805471 shutdown_connections 2024-04-30T08:38:15.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.526+0000 7f639dc00640 1 --2- 172.21.15.32:0/533805471 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f637c0afc70 0x7f637c0b2180 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.526+0000 7f639dc00640 1 --2- 172.21.15.32:0/533805471 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f639810ff10 0x7f639811b3f0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.527+0000 7f639dc00640 1 --2- 172.21.15.32:0/533805471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f639810b210 0x7f639811a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.529 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.527+0000 7f639dc00640 1 --2- 172.21.15.32:0/533805471 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f639810a3c0 0x7f6398119f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.530 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.529+0000 7f639dc00640 1 -- 172.21.15.32:0/533805471 >> 172.21.15.32:0/533805471 conn(0x7f63980703a0 msgr2=0x7f639810df00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:15.540 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.537+0000 7f639dc00640 1 -- 172.21.15.32:0/533805471 shutdown_connections 2024-04-30T08:38:15.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:15.543+0000 7f639dc00640 1 -- 172.21.15.32:0/533805471 wait complete. 2024-04-30T08:38:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.721+0000 7f5869c00640 1 -- 172.21.15.32:0/3676213276 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f586410a3c0 msgr2=0x7f586410acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.721+0000 7f5869c00640 1 --2- 172.21.15.32:0/3676213276 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f586410a3c0 0x7f586410acd0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f5850009860 tx=0x7f585002a380 comp rx=0 tx=0).stop 2024-04-30T08:38:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.733+0000 7f5869c00640 1 -- 172.21.15.32:0/3676213276 shutdown_connections 2024-04-30T08:38:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.733+0000 7f5869c00640 1 --2- 172.21.15.32:0/3676213276 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f586410ff30 0x7f5864110340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.733+0000 7f5869c00640 1 --2- 172.21.15.32:0/3676213276 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f586410b210 0x7f586410b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.733+0000 7f5869c00640 1 --2- 172.21.15.32:0/3676213276 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f586410a3c0 0x7f586410acd0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.733+0000 7f5869c00640 1 -- 172.21.15.32:0/3676213276 >> 172.21.15.32:0/3676213276 conn(0x7f58640703b0 msgr2=0x7f5864072860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.735+0000 7f5869c00640 1 -- 172.21.15.32:0/3676213276 shutdown_connections 2024-04-30T08:38:15.738 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.736+0000 7f5869c00640 1 -- 172.21.15.32:0/3676213276 wait complete. 2024-04-30T08:38:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.737+0000 7f5869c00640 1 Processor -- start 2024-04-30T08:38:15.749 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.742+0000 7f5869c00640 1 -- start start 2024-04-30T08:38:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.742+0000 7f5869c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f586410a3c0 0x7f58641a8f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.742+0000 7f5869c00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f586410b210 0x7f58641a9470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.744+0000 7f5869c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f586410ff30 0x7f58641b0400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.744+0000 7f5869c00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f586411e2d0 con 0x7f586410b210 2024-04-30T08:38:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.745+0000 7f5869c00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f586411e150 con 0x7f586410a3c0 2024-04-30T08:38:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.745+0000 7f5869c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f586411e450 con 0x7f586410ff30 2024-04-30T08:38:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.745+0000 7f5863e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f586410a3c0 0x7f58641a8f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.745+0000 7f5863e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f586410a3c0 0x7f58641a8f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:36596/0 (socket says 172.21.15.32:36596) 2024-04-30T08:38:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.745+0000 7f5863e00640 1 -- 172.21.15.32:0/1198253074 learned_addr learned my addr 172.21.15.32:0/1198253074 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.745+0000 7f5863400640 1 --2- 172.21.15.32:0/1198253074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f586410b210 0x7f58641a9470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.748+0000 7f5869200640 1 --2- 172.21.15.32:0/1198253074 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f586410ff30 0x7f58641b0400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:15.752 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-30T08:38:15.753 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.748+0000 7f5863e00640 1 -- 172.21.15.32:0/1198253074 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f586410ff30 msgr2=0x7f58641b0400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:15.753 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.748+0000 7f5863e00640 1 --2- 172.21.15.32:0/1198253074 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f586410ff30 0x7f58641b0400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.753 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.748+0000 7f5863e00640 1 -- 172.21.15.32:0/1198253074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f586410b210 msgr2=0x7f58641a9470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:15.754 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.748+0000 7f5863e00640 1 --2- 172.21.15.32:0/1198253074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f586410b210 0x7f58641a9470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.754 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.749+0000 7f5863e00640 1 -- 172.21.15.32:0/1198253074 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5864111d10 con 0x7f586410a3c0 2024-04-30T08:38:15.754 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.749+0000 7f5863e00640 1 --2- 172.21.15.32:0/1198253074 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f586410a3c0 0x7f58641a8f30 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f5850000c00 tx=0x7f5850030d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:15.754 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.750+0000 7f5860c00640 1 -- 172.21.15.32:0/1198253074 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5850037050 con 0x7f586410a3c0 2024-04-30T08:38:15.754 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.750+0000 7f5860c00640 1 -- 172.21.15.32:0/1198253074 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5850043070 con 0x7f586410a3c0 2024-04-30T08:38:15.754 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.750+0000 7f5860c00640 1 -- 172.21.15.32:0/1198253074 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5850003c20 con 0x7f586410a3c0 2024-04-30T08:38:15.784 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.752+0000 7f5869c00640 1 -- 172.21.15.32:0/1198253074 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5864111fa0 con 0x7f586410a3c0 2024-04-30T08:38:15.784 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.752+0000 7f5869c00640 1 -- 172.21.15.32:0/1198253074 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5864112440 con 0x7f586410a3c0 2024-04-30T08:38:15.784 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.769+0000 7f5860c00640 1 -- 172.21.15.32:0/1198253074 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f5850004040 con 0x7f586410a3c0 2024-04-30T08:38:15.784 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.775+0000 7f58721ea740 1 Processor -- start 2024-04-30T08:38:15.784 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.780+0000 7f5860c00640 1 --2- 172.21.15.32:0/1198253074 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f58480afc70 0x7f58480b2180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:15.784 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.781+0000 7f5869200640 1 --2- 172.21.15.32:0/1198253074 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f58480afc70 0x7f58480b2180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:15.792 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.788+0000 7f5860c00640 1 -- 172.21.15.32:0/1198253074 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f58500c37b0 con 0x7f586410a3c0 2024-04-30T08:38:15.793 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.790+0000 7f5869200640 1 --2- 172.21.15.32:0/1198253074 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f58480afc70 0x7f58480b2180 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f585c005fd0 tx=0x7f585c0095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:15.793 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.790+0000 7f58721ea740 1 -- start start 2024-04-30T08:38:15.793 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.790+0000 7f58721ea740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556140fc21b0 0x5561413642d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:15.793 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.790+0000 7f58721ea740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556140f992f0 0x556141364850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:15.793 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.791+0000 7f58721ea740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556140fa14d0 0x55614136b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:15.793 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.791+0000 7f58721ea740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x5561412c8550 con 0x556140f992f0 2024-04-30T08:38:15.793 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.791+0000 7f58721ea740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55614126dbc0 con 0x556140fa14d0 2024-04-30T08:38:15.793 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.791+0000 7f58721ea740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x5561412c4510 con 0x556140fc21b0 2024-04-30T08:38:15.793 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.791+0000 7f5869200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556140fa14d0 0x55614136b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:15.793 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.791+0000 7f5869200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556140fa14d0 0x55614136b750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:36600/0 (socket says 172.21.15.32:36600) 2024-04-30T08:38:15.793 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.791+0000 7f5869200640 1 -- 172.21.15.32:0/3384660958 learned_addr learned my addr 172.21.15.32:0/3384660958 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:15.794 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.791+0000 7f5863400640 1 --2- 172.21.15.32:0/3384660958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556140f992f0 0x556141364850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:15.794 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.791+0000 7f5869200640 1 -- 172.21.15.32:0/3384660958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556140fc21b0 msgr2=0x5561413642d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-30T08:38:15.794 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.792+0000 7f5869200640 1 --2- 172.21.15.32:0/3384660958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556140fc21b0 0x5561413642d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.794 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.792+0000 7f5869200640 1 -- 172.21.15.32:0/3384660958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556140f992f0 msgr2=0x556141364850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:15.794 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.792+0000 7f5869200640 1 --2- 172.21.15.32:0/3384660958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556140f992f0 0x556141364850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.794 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.792+0000 7f5869200640 1 -- 172.21.15.32:0/3384660958 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556140fda210 con 0x556140fa14d0 2024-04-30T08:38:15.797 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.792+0000 7f5869200640 1 --2- 172.21.15.32:0/3384660958 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556140fa14d0 0x55614136b750 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f585c00e4b0 tx=0x7f585c013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d6f1e5e3eab3dca server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:15.797 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.792+0000 7f5859600640 1 -- 172.21.15.32:0/3384660958 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f585c00ede0 con 0x556140fa14d0 2024-04-30T08:38:15.797 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.793+0000 7f5859600640 1 -- 172.21.15.32:0/3384660958 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f585c003e40 con 0x556140fa14d0 2024-04-30T08:38:15.797 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.793+0000 7f5859600640 1 -- 172.21.15.32:0/3384660958 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f585c01b670 con 0x556140fa14d0 2024-04-30T08:38:15.812 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.805+0000 7f58721ea740 1 -- 172.21.15.32:0/3384660958 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556140fa14d0 msgr2=0x55614136b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:15.812 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.805+0000 7f58721ea740 1 --2- 172.21.15.32:0/3384660958 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556140fa14d0 0x55614136b750 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f585c00e4b0 tx=0x7f585c013000 comp rx=0 tx=0).stop 2024-04-30T08:38:15.812 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.805+0000 7f58721ea740 1 -- 172.21.15.32:0/3384660958 shutdown_connections 2024-04-30T08:38:15.812 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.805+0000 7f58721ea740 1 --2- 172.21.15.32:0/3384660958 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556140fa14d0 0x55614136b750 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.812 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.805+0000 7f58721ea740 1 --2- 172.21.15.32:0/3384660958 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556140f992f0 0x556141364850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.812 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.805+0000 7f58721ea740 1 --2- 172.21.15.32:0/3384660958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556140fc21b0 0x5561413642d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.812 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.805+0000 7f58721ea740 1 -- 172.21.15.32:0/3384660958 >> 172.21.15.32:0/3384660958 conn(0x556140fb6300 msgr2=0x55614135fc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:15.812 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.809+0000 7f58721ea740 1 -- 172.21.15.32:0/3384660958 shutdown_connections 2024-04-30T08:38:15.812 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.809+0000 7f58721ea740 1 -- 172.21.15.32:0/3384660958 wait complete. 2024-04-30T08:38:15.813 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.812+0000 7f58721ea740 1 -- start start 2024-04-30T08:38:15.814 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.813+0000 7f58721ea740 1 Processor -- start 2024-04-30T08:38:15.818 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.817+0000 7f58721ea740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556140f992f0 0x55614136dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:15.818 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.817+0000 7f58721ea740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556140fa14d0 0x55614136e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:15.820 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.817+0000 7f5863e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556140f992f0 0x55614136dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:15.820 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.817+0000 7f5863e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556140f992f0 0x55614136dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:36616/0 (socket says 172.21.15.32:36616) 2024-04-30T08:38:15.821 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.818+0000 7f5863400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556140fa14d0 0x55614136e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.818+0000 7f5863400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556140fa14d0 0x55614136e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49254/0 (socket says 172.21.15.32:49254) 2024-04-30T08:38:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.819+0000 7f5863e00640 1 -- 172.21.15.32:0/1338474935 learned_addr learned my addr 172.21.15.32:0/1338474935 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.819+0000 7f58721ea740 1 --2- 172.21.15.32:0/1338474935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556140fc21b0 0x556141375380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.819+0000 7f58721ea740 1 -- 172.21.15.32:0/1338474935 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55614136c1a0 con 0x556140fa14d0 2024-04-30T08:38:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.819+0000 7f58721ea740 1 -- 172.21.15.32:0/1338474935 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x556141281660 con 0x556140f992f0 2024-04-30T08:38:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.819+0000 7f58721ea740 1 -- 172.21.15.32:0/1338474935 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55614136c320 con 0x556140fc21b0 2024-04-30T08:38:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.819+0000 7f58721ea740 10 client.0 _ll_register_callbacks cb 0x55614124b400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:38:15.822 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.819+0000 7f5869200640 1 --2- 172.21.15.32:0/1338474935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556140fc21b0 0x556141375380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.821+0000 7f5863e00640 1 -- 172.21.15.32:0/1338474935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556140fc21b0 msgr2=0x556141375380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.821+0000 7f5863e00640 1 --2- 172.21.15.32:0/1338474935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556140fc21b0 0x556141375380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.821+0000 7f5863e00640 1 -- 172.21.15.32:0/1338474935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556140fa14d0 msgr2=0x55614136e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.821+0000 7f5863e00640 1 --2- 172.21.15.32:0/1338474935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556140fa14d0 0x55614136e420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.823 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.821+0000 7f5863e00640 1 -- 172.21.15.32:0/1338474935 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561410a6620 con 0x556140f992f0 2024-04-30T08:38:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.822+0000 7f5869200640 1 --2- 172.21.15.32:0/1338474935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556140fc21b0 0x556141375380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:38:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.822+0000 7f5863e00640 1 --2- 172.21.15.32:0/1338474935 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556140f992f0 0x55614136dee0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f58500c71e0 tx=0x7f58500c6c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=62d59e4d61c2d6e1 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.822+0000 7f584e000640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:38:15.824 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.823+0000 7f5863400640 1 --2- 172.21.15.32:0/1338474935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556140fa14d0 0x55614136e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:38:15.827 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.825+0000 7f584e000640 1 -- 172.21.15.32:0/1338474935 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58500e0070 con 0x556140f992f0 2024-04-30T08:38:15.827 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.825+0000 7f584e000640 1 -- 172.21.15.32:0/1338474935 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f58500cb170 con 0x556140f992f0 2024-04-30T08:38:15.827 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.826+0000 7f58721ea740 1 -- 172.21.15.32:0/1338474935 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5561412d6160 con 0x556140f992f0 2024-04-30T08:38:15.828 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.827+0000 7f584e000640 1 -- 172.21.15.32:0/1338474935 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f58500c6d60 con 0x556140f992f0 2024-04-30T08:38:15.841 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.833+0000 7f58721ea740 10 client.26480 fetch_fsmap learned FSMap version 29 2024-04-30T08:38:15.842 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.833+0000 7f58721ea740 1 -- 172.21.15.32:0/1338474935 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x5561413782b0 con 0x556140f992f0 2024-04-30T08:38:15.842 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.834+0000 7f584e000640 1 -- 172.21.15.32:0/1338474935 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5561412d6160 con 0x556140f992f0 2024-04-30T08:38:15.842 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.834+0000 7f584e000640 1 -- 172.21.15.32:0/1338474935 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7f58500cfc20 con 0x556140f992f0 2024-04-30T08:38:15.842 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.834+0000 7f58721ea740 10 client.26480 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:38:15.842 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.834+0000 7f58721ea740 10 client.26480 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-30T08:38:15.842 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.834+0000 7f58721ea740 20 client.26480 populate_metadata read hostname 'smithi032' 2024-04-30T08:38:15.842 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.834+0000 7f58721ea740 1 --2- 172.21.15.32:0/1338474935 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x556140f80d80 0x55614137bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:15.842 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.834+0000 7f58721ea740 4 client.26480 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:38:15.842 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.834+0000 7f58721ea740 1 -- 172.21.15.32:0/1338474935 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5561412cffd0 con 0x556140f80d80 2024-04-30T08:38:15.842 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.839+0000 7f5863400640 1 --2- 172.21.15.32:0/1338474935 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x556140f80d80 0x55614137bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:15.842 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.840+0000 7f5863400640 1 --2- 172.21.15.32:0/1338474935 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x556140f80d80 0x55614137bce0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8dc751a77a68ade4 server_cookie=33706503622eb593 in_seq=0 out_seq=0 2024-04-30T08:38:15.842 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.840+0000 7f584e000640 10 client.26480 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:38:15.849 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.843+0000 7f584e000640 1 -- 172.21.15.32:0/1338474935 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x5561412cffd0 con 0x556140f80d80 2024-04-30T08:38:15.849 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.843+0000 7f584e000640 10 client.26480 handle_command_reply: tid=0 2024-04-30T08:38:15.849 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.843+0000 7f58721ea740 1 client.26480 shutdown 2024-04-30T08:38:15.850 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.844+0000 7f58721ea740 1 -- 172.21.15.32:0/1338474935 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556140f992f0 msgr2=0x55614136dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:15.850 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.844+0000 7f58721ea740 1 --2- 172.21.15.32:0/1338474935 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556140f992f0 0x55614136dee0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f58500c71e0 tx=0x7f58500c6c60 comp rx=0 tx=0).stop 2024-04-30T08:38:15.850 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.844+0000 7f58721ea740 1 -- 172.21.15.32:0/1338474935 shutdown_connections 2024-04-30T08:38:15.850 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.847+0000 7f58721ea740 1 --2- 172.21.15.32:0/1338474935 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x556140f80d80 0x55614137bce0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.851 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.849+0000 7f58721ea740 1 --2- 172.21.15.32:0/1338474935 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556140fc21b0 0x556141375380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.851 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.849+0000 7f584e000640 0 client.26480 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:38:15.851 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.849+0000 7f58721ea740 1 --2- 172.21.15.32:0/1338474935 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556140fa14d0 0x55614136e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.851 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.849+0000 7f58721ea740 1 --2- 172.21.15.32:0/1338474935 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556140f992f0 0x55614136dee0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.851 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.849+0000 7f58721ea740 1 -- 172.21.15.32:0/1338474935 >> 172.21.15.32:0/1338474935 conn(0x556140fb6300 msgr2=0x55614135ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:15.851 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.850+0000 7f58721ea740 1 -- 172.21.15.32:0/1338474935 shutdown_connections 2024-04-30T08:38:15.851 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.850+0000 7f58721ea740 1 -- 172.21.15.32:0/1338474935 wait complete. 2024-04-30T08:38:15.852 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.851+0000 7f58721ea740 20 client.26480 trim_cache size 0 max 16384 2024-04-30T08:38:15.910 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.905+0000 7f58721ea740 1 Processor -- start 2024-04-30T08:38:15.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.916+0000 7f58721ea740 1 -- start start 2024-04-30T08:38:15.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.916+0000 7f58721ea740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556140f80d80 0x55614135fc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:15.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.916+0000 7f58721ea740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556140f992f0 0x556141375310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:15.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.916+0000 7f58721ea740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556140fa14d0 0x556141375850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:15.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.916+0000 7f58721ea740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f58641c4060 con 0x556140fa14d0 2024-04-30T08:38:15.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.916+0000 7f58721ea740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f58641c5c30 con 0x556140f992f0 2024-04-30T08:38:15.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.916+0000 7f58721ea740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f58641c4730 con 0x556140f80d80 2024-04-30T08:38:15.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.917+0000 7f5863e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556140f80d80 0x55614135fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:15.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.917+0000 7f5863e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556140f80d80 0x55614135fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:33056/0 (socket says 172.21.15.32:33056) 2024-04-30T08:38:15.919 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.917+0000 7f5863e00640 1 -- 172.21.15.32:0/1721308180 learned_addr learned my addr 172.21.15.32:0/1721308180 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.919+0000 7f5863e00640 1 -- 172.21.15.32:0/1721308180 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556140f992f0 msgr2=0x556141375310 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-30T08:38:15.920 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.919+0000 7f5863e00640 1 --2- 172.21.15.32:0/1721308180 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556140f992f0 0x556141375310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.919+0000 7f5863e00640 1 -- 172.21.15.32:0/1721308180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556140fa14d0 msgr2=0x556141375850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:15.921 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.919+0000 7f5863e00640 1 --2- 172.21.15.32:0/1721308180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556140fa14d0 0x556141375850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.920+0000 7f5863e00640 1 -- 172.21.15.32:0/1721308180 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55614135b450 con 0x556140f80d80 2024-04-30T08:38:15.923 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.921+0000 7f5863e00640 1 --2- 172.21.15.32:0/1721308180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556140f80d80 0x55614135fc90 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f5850003fd0 tx=0x7f58500045f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1d3466ae41b1598d server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:15.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.923+0000 7f5858c00640 1 -- 172.21.15.32:0/1721308180 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58500e0070 con 0x556140f80d80 2024-04-30T08:38:15.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.923+0000 7f5858c00640 1 -- 172.21.15.32:0/1721308180 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5850038040 con 0x556140f80d80 2024-04-30T08:38:15.926 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.923+0000 7f5858c00640 1 -- 172.21.15.32:0/1721308180 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58500c7950 con 0x556140f80d80 2024-04-30T08:38:15.927 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.926+0000 7f58721ea740 1 -- 172.21.15.32:0/1721308180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556140f80d80 msgr2=0x55614135fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:15.928 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.926+0000 7f58721ea740 1 --2- 172.21.15.32:0/1721308180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556140f80d80 0x55614135fc90 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f5850003fd0 tx=0x7f58500045f0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.929 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.928+0000 7f58721ea740 1 -- 172.21.15.32:0/1721308180 shutdown_connections 2024-04-30T08:38:15.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.928+0000 7f58721ea740 1 --2- 172.21.15.32:0/1721308180 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556140fa14d0 0x556141375850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.928+0000 7f58721ea740 1 --2- 172.21.15.32:0/1721308180 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556140f992f0 0x556141375310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.928+0000 7f58721ea740 1 --2- 172.21.15.32:0/1721308180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556140f80d80 0x55614135fc90 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.928+0000 7f58721ea740 1 -- 172.21.15.32:0/1721308180 >> 172.21.15.32:0/1721308180 conn(0x556140fb6300 msgr2=0x556141363490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:15.930 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.929+0000 7f58721ea740 1 -- 172.21.15.32:0/1721308180 shutdown_connections 2024-04-30T08:38:15.934 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.929+0000 7f58721ea740 1 -- 172.21.15.32:0/1721308180 wait complete. 2024-04-30T08:38:15.934 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.929+0000 7f58721ea740 1 -- start start 2024-04-30T08:38:15.934 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.930+0000 7f58721ea740 1 Processor -- start 2024-04-30T08:38:15.935 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.933+0000 7f58721ea740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556140f80d80 0x556141372bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:15.935 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.933+0000 7f58721ea740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556140f992f0 0x556141373110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:15.935 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.933+0000 7f58721ea740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556140fa14d0 0x556141373ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:15.935 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.934+0000 7f58721ea740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f58641c54d0 con 0x556140f992f0 2024-04-30T08:38:15.935 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.934+0000 7f58721ea740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f58641c4e00 con 0x556140fa14d0 2024-04-30T08:38:15.935 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.934+0000 7f58721ea740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f586411a1b0 con 0x556140f80d80 2024-04-30T08:38:15.936 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.934+0000 7f58721ea740 10 client.0 _ll_register_callbacks cb 0x7f5850005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:38:15.936 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.935+0000 7f5863400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556140f992f0 0x556141373110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:15.936 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.935+0000 7f5863400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556140f992f0 0x556141373110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:49270/0 (socket says 172.21.15.32:49270) 2024-04-30T08:38:15.937 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.935+0000 7f5863400640 1 -- 172.21.15.32:0/2984557802 learned_addr learned my addr 172.21.15.32:0/2984557802 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:15.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.936+0000 7f5863400640 1 -- 172.21.15.32:0/2984557802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556140f80d80 msgr2=0x556141372bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:15.938 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.937+0000 7f5863e00640 1 --2- 172.21.15.32:0/2984557802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556140f80d80 0x556141372bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:15.939 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.937+0000 7f5869200640 1 --2- 172.21.15.32:0/2984557802 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556140fa14d0 0x556141373ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:15.939 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.937+0000 7f5863400640 1 --2- 172.21.15.32:0/2984557802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556140f80d80 0x556141372bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.939 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.937+0000 7f5863400640 1 -- 172.21.15.32:0/2984557802 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556140fa14d0 msgr2=0x556141373ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:15.939 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.937+0000 7f5863400640 1 --2- 172.21.15.32:0/2984557802 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556140fa14d0 0x556141373ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:15.939 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.937+0000 7f5863400640 1 -- 172.21.15.32:0/2984557802 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55614137c210 con 0x556140f992f0 2024-04-30T08:38:15.939 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.937+0000 7f5863e00640 1 --2- 172.21.15.32:0/2984557802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556140f80d80 0x556141372bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-30T08:38:15.942 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.940+0000 7f5863400640 1 --2- 172.21.15.32:0/2984557802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556140f992f0 0x556141373110 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x5561413609f0 tx=0x7f5854013b10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c509d1eafb60fb9 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:15.951 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.941+0000 7f584e000640 10 client.0 ms_handle_connect on v2:172.21.15.32:3300/0 2024-04-30T08:38:15.951 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.941+0000 7f584e000640 1 -- 172.21.15.32:0/2984557802 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f585401e070 con 0x556140f992f0 2024-04-30T08:38:15.951 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.943+0000 7f584e000640 1 -- 172.21.15.32:0/2984557802 <== mon.0 v2:172.21.15.32:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f585401a3e0 con 0x556140f992f0 2024-04-30T08:38:15.951 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.943+0000 7f58721ea740 1 -- 172.21.15.32:0/2984557802 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5850038040 con 0x556140f992f0 2024-04-30T08:38:15.951 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.946+0000 7f584e000640 1 -- 172.21.15.32:0/2984557802 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f5854028050 con 0x556140f992f0 2024-04-30T08:38:15.951 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.950+0000 7f58721ea740 10 client.16323 fetch_fsmap learned FSMap version 29 2024-04-30T08:38:15.952 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.950+0000 7f58721ea740 1 -- 172.21.15.32:0/2984557802 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x556141375770 con 0x556140f992f0 2024-04-30T08:38:15.969 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.968+0000 7f584e000640 1 -- 172.21.15.32:0/2984557802 <== mon.0 v2:172.21.15.32:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x5561412cffd0 con 0x556140f992f0 2024-04-30T08:38:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.969+0000 7f584e000640 1 -- 172.21.15.32:0/2984557802 <== mon.0 v2:172.21.15.32:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7f585400f040 con 0x556140f992f0 2024-04-30T08:38:15.973 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.969+0000 7f58721ea740 10 client.16323 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:38:15.974 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.969+0000 7f58721ea740 10 client.16323 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-04-30T08:38:15.974 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.971+0000 7f58721ea740 20 client.16323 populate_metadata read hostname 'smithi032' 2024-04-30T08:38:15.974 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.971+0000 7f58721ea740 1 --2- 172.21.15.32:0/2984557802 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x556140fc21b0 0x5561413e64a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:15.974 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.971+0000 7f58721ea740 4 client.16323 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-04-30T08:38:15.974 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.971+0000 7f58721ea740 1 -- 172.21.15.32:0/2984557802 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f5850038040 con 0x556140fc21b0 2024-04-30T08:38:15.974 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.971+0000 7f5863e00640 1 --2- 172.21.15.32:0/2984557802 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x556140fc21b0 0x5561413e64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:15.979 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.975+0000 7f584e000640 10 client.16323 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:38:15.980 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:38:15.975+0000 7f5863e00640 1 --2- 172.21.15.32:0/2984557802 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x556140fc21b0 0x5561413e64a0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d392d3ae4ac6e574 server_cookie=ab3a42b5cefce5ca in_seq=0 out_seq=0 2024-04-30T08:38:16.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.576+0000 7f86ba600640 1 -- 172.21.15.32:0/3431378568 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86b41098f0 msgr2=0x7f86b410bd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:16.577 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.576+0000 7f86ba600640 1 --2- 172.21.15.32:0/3431378568 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86b41098f0 0x7f86b410bd10 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f86ac009ca0 tx=0x7f86ac02a8b0 comp rx=0 tx=0).stop 2024-04-30T08:38:16.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.581+0000 7f86ba600640 1 -- 172.21.15.32:0/3431378568 shutdown_connections 2024-04-30T08:38:16.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.581+0000 7f86ba600640 1 --2- 172.21.15.32:0/3431378568 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86b41098f0 0x7f86b410bd10 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:16.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.581+0000 7f86ba600640 1 --2- 172.21.15.32:0/3431378568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86b4104bf0 0x7f86b4105000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:16.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.581+0000 7f86ba600640 1 --2- 172.21.15.32:0/3431378568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86b41039b0 0x7f86b41046b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:16.583 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.581+0000 7f86ba600640 1 -- 172.21.15.32:0/3431378568 >> 172.21.15.32:0/3431378568 conn(0x7f86b40feba0 msgr2=0x7f86b4101050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:16.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.584+0000 7f86ba600640 1 -- 172.21.15.32:0/3431378568 shutdown_connections 2024-04-30T08:38:16.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.585+0000 7f86ba600640 1 -- 172.21.15.32:0/3431378568 wait complete. 2024-04-30T08:38:16.588 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.586+0000 7f86ba600640 1 Processor -- start 2024-04-30T08:38:16.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.587+0000 7f86ba600640 1 -- start start 2024-04-30T08:38:16.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.587+0000 7f86ba600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86b41039b0 0x7f86b4197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:16.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.587+0000 7f86ba600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86b4104bf0 0x7f86b41983c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:16.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.587+0000 7f86ba600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86b41098f0 0x7f86b419f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:16.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.587+0000 7f86ba600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f86b410f4c0 con 0x7f86b4104bf0 2024-04-30T08:38:16.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.587+0000 7f86ba600640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f86b410f340 con 0x7f86b41098f0 2024-04-30T08:38:16.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.587+0000 7f86ba600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f86b410f640 con 0x7f86b41039b0 2024-04-30T08:38:16.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.588+0000 7f86b9c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86b41098f0 0x7f86b419f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:16.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.588+0000 7f86b9c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86b41098f0 0x7f86b419f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:36654/0 (socket says 172.21.15.32:36654) 2024-04-30T08:38:16.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.588+0000 7f86b9c00640 1 -- 172.21.15.32:0/386452073 learned_addr learned my addr 172.21.15.32:0/386452073 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:16.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.589+0000 7f86b9c00640 1 -- 172.21.15.32:0/386452073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86b41039b0 msgr2=0x7f86b4197e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:16.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.589+0000 7f86b9c00640 1 --2- 172.21.15.32:0/386452073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86b41039b0 0x7f86b4197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:16.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.589+0000 7f86b9c00640 1 -- 172.21.15.32:0/386452073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86b4104bf0 msgr2=0x7f86b41983c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:16.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.589+0000 7f86b9c00640 1 --2- 172.21.15.32:0/386452073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86b4104bf0 0x7f86b41983c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:16.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.589+0000 7f86b9c00640 1 -- 172.21.15.32:0/386452073 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86b419f9c0 con 0x7f86b41098f0 2024-04-30T08:38:16.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.590+0000 7f86b9c00640 1 --2- 172.21.15.32:0/386452073 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86b41098f0 0x7f86b419f350 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f86ac004af0 tx=0x7f86ac003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:16.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.591+0000 7f86b1600640 1 -- 172.21.15.32:0/386452073 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ac004240 con 0x7f86b41098f0 2024-04-30T08:38:16.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.591+0000 7f86b1600640 1 -- 172.21.15.32:0/386452073 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86ac040070 con 0x7f86b41098f0 2024-04-30T08:38:16.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.591+0000 7f86b1600640 1 -- 172.21.15.32:0/386452073 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86ac004240 con 0x7f86b41098f0 2024-04-30T08:38:16.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.595+0000 7f86ba600640 1 -- 172.21.15.32:0/386452073 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86b419fc50 con 0x7f86b41098f0 2024-04-30T08:38:16.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.595+0000 7f86ba600640 1 -- 172.21.15.32:0/386452073 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86b41a0120 con 0x7f86b41098f0 2024-04-30T08:38:16.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.608+0000 7f86c2c81740 1 Processor -- start 2024-04-30T08:38:16.611 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.609+0000 7f86b1600640 1 -- 172.21.15.32:0/386452073 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f86ac0043e0 con 0x7f86b41098f0 2024-04-30T08:38:16.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.615+0000 7f86c2c81740 1 -- start start 2024-04-30T08:38:16.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.615+0000 7f86c2c81740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559d9934f390 0x559d996f0730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:16.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.615+0000 7f86c2c81740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d993264d0 0x559d996f0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:16.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.615+0000 7f86c2c81740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d9932e6b0 0x559d996f7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:16.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.615+0000 7f86c2c81740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x559d99655d50 con 0x559d9932e6b0 2024-04-30T08:38:16.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.615+0000 7f86c2c81740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x559d995fb0c0 con 0x559d9934f390 2024-04-30T08:38:16.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.615+0000 7f86c2c81740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559d99651d10 con 0x559d993264d0 2024-04-30T08:38:16.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.616+0000 7f86b9200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d993264d0 0x559d996f0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:16.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.617+0000 7f86b1600640 1 --2- 172.21.15.32:0/386452073 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f869c0aff30 0x7f869c0b2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:16.619 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.617+0000 7f86b9200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d993264d0 0x559d996f0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:33096/0 (socket says 172.21.15.32:33096) 2024-04-30T08:38:16.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.617+0000 7f86b9200640 1 -- 172.21.15.32:0/2966741205 learned_addr learned my addr 172.21.15.32:0/2966741205 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:16.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.617+0000 7f86b3e00640 1 --2- 172.21.15.32:0/2966741205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d9932e6b0 0x559d996f7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:16.620 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.617+0000 7f86b9c00640 1 --2- 172.21.15.32:0/386452073 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f869c0aff30 0x7f869c0b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:16.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.618+0000 7f86b1600640 1 -- 172.21.15.32:0/386452073 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f86ac004040 con 0x7f86b41098f0 2024-04-30T08:38:16.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.618+0000 7f86b9200640 1 -- 172.21.15.32:0/2966741205 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559d9934f390 msgr2=0x559d996f0730 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-30T08:38:16.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.618+0000 7f86b9200640 1 --2- 172.21.15.32:0/2966741205 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559d9934f390 0x559d996f0730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:16.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.618+0000 7f86b9200640 1 -- 172.21.15.32:0/2966741205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d9932e6b0 msgr2=0x559d996f7bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:16.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.618+0000 7f86b9200640 1 --2- 172.21.15.32:0/2966741205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d9932e6b0 0x559d996f7bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:16.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.619+0000 7f86b9200640 1 -- 172.21.15.32:0/2966741205 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d993673f0 con 0x559d993264d0 2024-04-30T08:38:16.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.619+0000 7f86b9c00640 1 --2- 172.21.15.32:0/386452073 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f869c0aff30 0x7f869c0b2440 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f86ac0c76c0 tx=0x7f86ac105240 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:16.621 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.620+0000 7f86b9200640 1 --2- 172.21.15.32:0/2966741205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d993264d0 0x559d996f0cb0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f86a8009bc0 tx=0x7f86a800e720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a12b86cddcadbf8 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:16.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.620+0000 7f86a6000640 1 -- 172.21.15.32:0/2966741205 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a8005020 con 0x559d993264d0 2024-04-30T08:38:16.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.620+0000 7f86a6000640 1 -- 172.21.15.32:0/2966741205 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86a80047a0 con 0x559d993264d0 2024-04-30T08:38:16.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.621+0000 7f86a6000640 1 -- 172.21.15.32:0/2966741205 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a8010690 con 0x559d993264d0 2024-04-30T08:38:16.622 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.621+0000 7f86b3e00640 1 --2- 172.21.15.32:0/2966741205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d9932e6b0 0x559d996f7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:38:16.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.624+0000 7f86c2c81740 1 -- 172.21.15.32:0/2966741205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d993264d0 msgr2=0x559d996f0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:16.625 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.624+0000 7f86c2c81740 1 --2- 172.21.15.32:0/2966741205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d993264d0 0x559d996f0cb0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f86a8009bc0 tx=0x7f86a800e720 comp rx=0 tx=0).stop 2024-04-30T08:38:16.628 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.626+0000 7f86c2c81740 1 -- 172.21.15.32:0/2966741205 shutdown_connections 2024-04-30T08:38:16.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.627+0000 7f86c2c81740 1 --2- 172.21.15.32:0/2966741205 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d9932e6b0 0x559d996f7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:16.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.627+0000 7f86c2c81740 1 --2- 172.21.15.32:0/2966741205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d993264d0 0x559d996f0cb0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:16.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.627+0000 7f86c2c81740 1 --2- 172.21.15.32:0/2966741205 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559d9934f390 0x559d996f0730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:16.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.627+0000 7f86c2c81740 1 -- 172.21.15.32:0/2966741205 >> 172.21.15.32:0/2966741205 conn(0x559d993434e0 msgr2=0x559d996ec0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:16.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.627+0000 7f86c2c81740 1 -- 172.21.15.32:0/2966741205 shutdown_connections 2024-04-30T08:38:16.629 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.627+0000 7f86c2c81740 1 -- 172.21.15.32:0/2966741205 wait complete. 2024-04-30T08:38:16.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.628+0000 7f86c2c81740 1 -- start start 2024-04-30T08:38:16.630 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.628+0000 7f86c2c81740 1 Processor -- start 2024-04-30T08:38:16.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.632+0000 7f86c2c81740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d993264d0 0x559d996fa720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:16.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.632+0000 7f86c2c81740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559d9932e6b0 0x559d996fac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:16.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.632+0000 7f86c2c81740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d9934f390 0x559d99701b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:16.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.632+0000 7f86c2c81740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x559d996f8600 con 0x559d993264d0 2024-04-30T08:38:16.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.632+0000 7f86c2c81740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x559d9960eb50 con 0x559d9932e6b0 2024-04-30T08:38:16.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.632+0000 7f86c2c81740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x559d996f8780 con 0x559d9934f390 2024-04-30T08:38:16.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.632+0000 7f86c2c81740 10 client.0 _ll_register_callbacks cb 0x559d995d8420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:38:16.634 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.632+0000 7f86b9200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559d9932e6b0 0x559d996fac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:16.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.633+0000 7f86b9200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559d9932e6b0 0x559d996fac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:36672/0 (socket says 172.21.15.32:36672) 2024-04-30T08:38:16.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.633+0000 7f86b3e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d9934f390 0x559d99701b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:16.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.633+0000 7f86b3e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d9934f390 0x559d99701b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:33112/0 (socket says 172.21.15.32:33112) 2024-04-30T08:38:16.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.633+0000 7f86b3e00640 1 -- 172.21.15.32:0/1587476368 learned_addr learned my addr 172.21.15.32:0/1587476368 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:16.635 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.634+0000 7f86b3e00640 1 -- 172.21.15.32:0/1587476368 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559d9932e6b0 msgr2=0x559d996fac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:16.636 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.634+0000 7f86b3e00640 1 --2- 172.21.15.32:0/1587476368 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559d9932e6b0 0x559d996fac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:16.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.634+0000 7f86b3e00640 1 -- 172.21.15.32:0/1587476368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d993264d0 msgr2=0x559d996fa720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:16.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.634+0000 7f86b3e00640 1 --2- 172.21.15.32:0/1587476368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d993264d0 0x559d996fa720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:16.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.634+0000 7f86b3e00640 1 -- 172.21.15.32:0/1587476368 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d994317d0 con 0x559d9934f390 2024-04-30T08:38:16.637 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.635+0000 7f86b9200640 1 --2- 172.21.15.32:0/1587476368 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559d9932e6b0 0x559d996fac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:38:16.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.636+0000 7f86b3e00640 1 --2- 172.21.15.32:0/1587476368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d9934f390 0x559d99701b50 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f86a000bf10 tx=0x7f86a00127c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7dd567a493bb2e04 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:16.638 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.636+0000 7f869aa00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-04-30T08:38:16.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.637+0000 7f869aa00640 1 -- 172.21.15.32:0/1587476368 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a0004460 con 0x559d9934f390 2024-04-30T08:38:16.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.637+0000 7f869aa00640 1 -- 172.21.15.32:0/1587476368 <== mon.1 v2:172.21.15.189:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f86a0004260 con 0x559d9934f390 2024-04-30T08:38:16.641 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.640+0000 7f86c2c81740 1 -- 172.21.15.32:0/1587476368 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86a80047a0 con 0x559d9934f390 2024-04-30T08:38:16.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.642+0000 7f869aa00640 1 -- 172.21.15.32:0/1587476368 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f86a0004460 con 0x559d9934f390 2024-04-30T08:38:16.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.642+0000 7f86c2c81740 10 client.26485 fetch_fsmap learned FSMap version 29 2024-04-30T08:38:16.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.643+0000 7f86c2c81740 1 -- 172.21.15.32:0/1587476368 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559d99704b50 con 0x559d9934f390 2024-04-30T08:38:16.644 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.643+0000 7f869aa00640 1 -- 172.21.15.32:0/1587476368 <== mon.1 v2:172.21.15.189:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86a80047a0 con 0x559d9934f390 2024-04-30T08:38:16.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.645+0000 7f869aa00640 1 -- 172.21.15.32:0/1587476368 <== mon.1 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7f86a0019030 con 0x559d9934f390 2024-04-30T08:38:16.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.645+0000 7f86c2c81740 10 client.26485 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:38:16.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.645+0000 7f86c2c81740 10 client.26485 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-30T08:38:16.646 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.645+0000 7f86c2c81740 20 client.26485 populate_metadata read hostname 'smithi032' 2024-04-30T08:38:16.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.645+0000 7f86c2c81740 1 --2- 172.21.15.32:0/1587476368 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x559d9930df60 0x559d99708140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:16.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.645+0000 7f86c2c81740 4 client.26485 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:38:16.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.645+0000 7f86c2c81740 1 -- 172.21.15.32:0/1587476368 --> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559d99663960 con 0x559d9930df60 2024-04-30T08:38:16.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.646+0000 7f86b9200640 1 --2- 172.21.15.32:0/1587476368 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x559d9930df60 0x559d99708140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:16.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.649+0000 7f86b9200640 1 --2- 172.21.15.32:0/1587476368 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x559d9930df60 0x559d99708140 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2f06abfa81ef9972 server_cookie=5787d0304ccc5fe7 in_seq=0 out_seq=0 2024-04-30T08:38:16.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.649+0000 7f869aa00640 10 client.26485 ms_handle_connect on v2:172.21.15.32:6838/4160386524 2024-04-30T08:38:16.654 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.653+0000 7f869aa00640 1 -- 172.21.15.32:0/1587476368 <== mds.1 v2:172.21.15.32:6838/4160386524 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x559d99663960 con 0x559d9930df60 2024-04-30T08:38:16.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.653+0000 7f869aa00640 10 client.26485 handle_command_reply: tid=0 2024-04-30T08:38:16.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.653+0000 7f86c2c81740 1 client.26485 shutdown 2024-04-30T08:38:16.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.656+0000 7f86c2c81740 1 -- 172.21.15.32:0/1587476368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d9934f390 msgr2=0x559d99701b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:16.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.656+0000 7f86c2c81740 1 --2- 172.21.15.32:0/1587476368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d9934f390 0x559d99701b50 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f86a000bf10 tx=0x7f86a00127c0 comp rx=0 tx=0).stop 2024-04-30T08:38:16.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.656+0000 7f86c2c81740 1 -- 172.21.15.32:0/1587476368 shutdown_connections 2024-04-30T08:38:16.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.656+0000 7f86c2c81740 1 --2- 172.21.15.32:0/1587476368 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x559d9930df60 0x559d99708140 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:16.661 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.657+0000 7f86c2c81740 1 --2- 172.21.15.32:0/1587476368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d9934f390 0x559d99701b50 secure :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f86a000bf10 tx=0x7f86a00127c0 comp rx=0 tx=0).stop 2024-04-30T08:38:16.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.657+0000 7f86c2c81740 1 --2- 172.21.15.32:0/1587476368 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559d9932e6b0 0x559d996fac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:16.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.657+0000 7f86c2c81740 1 --2- 172.21.15.32:0/1587476368 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d993264d0 0x559d996fa720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:16.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.657+0000 7f86c2c81740 1 -- 172.21.15.32:0/1587476368 >> 172.21.15.32:0/1587476368 conn(0x559d993434e0 msgr2=0x559d996eb100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:16.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.657+0000 7f869aa00640 0 client.26485 ms_handle_reset on v2:172.21.15.32:6838/4160386524 2024-04-30T08:38:16.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.657+0000 7f86c2c81740 1 -- 172.21.15.32:0/1587476368 shutdown_connections 2024-04-30T08:38:16.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.657+0000 7f86c2c81740 1 -- 172.21.15.32:0/1587476368 wait complete. 2024-04-30T08:38:16.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.657+0000 7f86c2c81740 20 client.26485 trim_cache size 0 max 16384 2024-04-30T08:38:16.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.714+0000 7f86c2c81740 1 Processor -- start 2024-04-30T08:38:16.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.714+0000 7f86c2c81740 1 -- start start 2024-04-30T08:38:16.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.715+0000 7f86c2c81740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d993264d0 0x559d996fa160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:16.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.716+0000 7f86c2c81740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559d9932e6b0 0x559d99701a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:16.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.716+0000 7f86c2c81740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d992e33a0 0x559d996ebb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:16.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.716+0000 7f86c2c81740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f86b41c3f30 con 0x559d993264d0 2024-04-30T08:38:16.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.716+0000 7f86c2c81740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f86b41c5b00 con 0x559d9932e6b0 2024-04-30T08:38:16.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.716+0000 7f86c2c81740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f86b41c4600 con 0x559d992e33a0 2024-04-30T08:38:16.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.717+0000 7f86b9200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559d9932e6b0 0x559d99701a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:16.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.717+0000 7f86b9200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559d9932e6b0 0x559d99701a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:36676/0 (socket says 172.21.15.32:36676) 2024-04-30T08:38:16.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.717+0000 7f86b9200640 1 -- 172.21.15.32:0/4232539172 learned_addr learned my addr 172.21.15.32:0/4232539172 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:16.719 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.717+0000 7f86b3e00640 1 --2- 172.21.15.32:0/4232539172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d992e33a0 0x559d996ebb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:16.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.717+0000 7f86b3e00640 1 --2- 172.21.15.32:0/4232539172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d993264d0 0x559d996fa160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:16.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.718+0000 7f86b9200640 1 -- 172.21.15.32:0/4232539172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d992e33a0 msgr2=0x559d996ebb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:16.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.718+0000 7f86b9200640 1 --2- 172.21.15.32:0/4232539172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d992e33a0 0x559d996ebb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:16.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.718+0000 7f86b9200640 1 -- 172.21.15.32:0/4232539172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d993264d0 msgr2=0x559d996fa160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:16.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.719+0000 7f86b9200640 1 --2- 172.21.15.32:0/4232539172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d993264d0 0x559d996fa160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:16.720 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.719+0000 7f86b9200640 1 -- 172.21.15.32:0/4232539172 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559d996f5550 con 0x559d9932e6b0 2024-04-30T08:38:16.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.719+0000 7f86b3e00640 1 --2- 172.21.15.32:0/4232539172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d993264d0 0x559d996fa160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-30T08:38:16.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.719+0000 7f86b9200640 1 --2- 172.21.15.32:0/4232539172 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559d9932e6b0 0x559d99701a70 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f86a8009230 tx=0x7f86a8009e30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=99909c2ce3ba9512 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:16.721 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.720+0000 7f86a5600640 1 -- 172.21.15.32:0/4232539172 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a801d040 con 0x559d9932e6b0 2024-04-30T08:38:16.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.722+0000 7f86a5600640 1 -- 172.21.15.32:0/4232539172 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86a80097c0 con 0x559d9932e6b0 2024-04-30T08:38:16.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.723+0000 7f86a5600640 1 -- 172.21.15.32:0/4232539172 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a8018940 con 0x559d9932e6b0 2024-04-30T08:38:16.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.724+0000 7f86c2c81740 1 -- 172.21.15.32:0/4232539172 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559d9932e6b0 msgr2=0x559d99701a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:16.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.724+0000 7f86c2c81740 1 --2- 172.21.15.32:0/4232539172 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559d9932e6b0 0x559d99701a70 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f86a8009230 tx=0x7f86a8009e30 comp rx=0 tx=0).stop 2024-04-30T08:38:16.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.724+0000 7f86c2c81740 1 -- 172.21.15.32:0/4232539172 shutdown_connections 2024-04-30T08:38:16.733 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.724+0000 7f86c2c81740 1 --2- 172.21.15.32:0/4232539172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d992e33a0 0x559d996ebb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:16.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.724+0000 7f86c2c81740 1 --2- 172.21.15.32:0/4232539172 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559d9932e6b0 0x559d99701a70 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:16.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.724+0000 7f86c2c81740 1 --2- 172.21.15.32:0/4232539172 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d993264d0 0x559d996fa160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:16.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.724+0000 7f86c2c81740 1 -- 172.21.15.32:0/4232539172 >> 172.21.15.32:0/4232539172 conn(0x559d993434e0 msgr2=0x559d996f9a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:16.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.725+0000 7f86c2c81740 1 -- 172.21.15.32:0/4232539172 shutdown_connections 2024-04-30T08:38:16.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.725+0000 7f86c2c81740 1 -- 172.21.15.32:0/4232539172 wait complete. 2024-04-30T08:38:16.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.726+0000 7f86c2c81740 1 -- start start 2024-04-30T08:38:16.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.726+0000 7f86c2c81740 1 Processor -- start 2024-04-30T08:38:16.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.728+0000 7f86c2c81740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d992e33a0 0x559d996fec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:16.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.728+0000 7f86c2c81740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559d993264d0 0x559d996ff1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:16.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.729+0000 7f86c2c81740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d9932e6b0 0x559d99700100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:16.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.729+0000 7f86c2c81740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f86b41c53a0 con 0x559d992e33a0 2024-04-30T08:38:16.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.729+0000 7f86c2c81740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f86b41c4cd0 con 0x559d993264d0 2024-04-30T08:38:16.734 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.729+0000 7f86c2c81740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f86b4073560 con 0x559d9932e6b0 2024-04-30T08:38:16.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.729+0000 7f86c2c81740 10 client.0 _ll_register_callbacks cb 0x7f86a000b6d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:38:16.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.729+0000 7f86b9200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559d993264d0 0x559d996ff1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:16.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.729+0000 7f86b9200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559d993264d0 0x559d996ff1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:36678/0 (socket says 172.21.15.32:36678) 2024-04-30T08:38:16.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.729+0000 7f86b9200640 1 -- 172.21.15.32:0/3736058635 learned_addr learned my addr 172.21.15.32:0/3736058635 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:16.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.730+0000 7f86b3e00640 1 --2- 172.21.15.32:0/3736058635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d9932e6b0 0x559d99700100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:16.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.730+0000 7f86b3e00640 1 --2- 172.21.15.32:0/3736058635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d992e33a0 0x559d996fec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:16.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.730+0000 7f86b9200640 1 -- 172.21.15.32:0/3736058635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d9932e6b0 msgr2=0x559d99700100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:16.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.730+0000 7f86b9200640 1 --2- 172.21.15.32:0/3736058635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d9932e6b0 0x559d99700100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:16.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.730+0000 7f86b9200640 1 -- 172.21.15.32:0/3736058635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d992e33a0 msgr2=0x559d996fec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:16.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.730+0000 7f86b9200640 1 --2- 172.21.15.32:0/3736058635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d992e33a0 0x559d996fec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:16.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.730+0000 7f86b9200640 1 -- 172.21.15.32:0/3736058635 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559d997006c0 con 0x559d993264d0 2024-04-30T08:38:16.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.731+0000 7f86b9200640 1 --2- 172.21.15.32:0/3736058635 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559d993264d0 0x559d996ff1d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f86a8005e00 tx=0x7f86a80093b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd0af4370026301f server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:16.735 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.731+0000 7f869aa00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:38:16.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.733+0000 7f869aa00640 1 -- 172.21.15.32:0/3736058635 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86a8038020 con 0x559d993264d0 2024-04-30T08:38:16.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.733+0000 7f869aa00640 1 -- 172.21.15.32:0/3736058635 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f86a80186c0 con 0x559d993264d0 2024-04-30T08:38:16.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.733+0000 7f86c2c81740 1 -- 172.21.15.32:0/3736058635 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86a80097c0 con 0x559d993264d0 2024-04-30T08:38:16.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.734+0000 7f869aa00640 1 -- 172.21.15.32:0/3736058635 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7f86a80188c0 con 0x559d993264d0 2024-04-30T08:38:16.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.734+0000 7f86c2c81740 10 client.26495 fetch_fsmap learned FSMap version 29 2024-04-30T08:38:16.736 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.734+0000 7f86c2c81740 1 -- 172.21.15.32:0/3736058635 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x559d997018b0 con 0x559d993264d0 2024-04-30T08:38:16.737 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.735+0000 7f869aa00640 1 -- 172.21.15.32:0/3736058635 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f86a80097c0 con 0x559d993264d0 2024-04-30T08:38:16.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.738+0000 7f869aa00640 1 -- 172.21.15.32:0/3736058635 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7f86a8024430 con 0x559d993264d0 2024-04-30T08:38:16.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.738+0000 7f86c2c81740 10 client.26495 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:38:16.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.738+0000 7f86c2c81740 10 client.26495 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-04-30T08:38:16.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.739+0000 7f86c2c81740 20 client.26495 populate_metadata read hostname 'smithi032' 2024-04-30T08:38:16.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.739+0000 7f86c2c81740 1 --2- 172.21.15.32:0/3736058635 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x559d99770650 0x559d99772a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:16.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.739+0000 7f86c2c81740 4 client.26495 mds_command: new command op to 24560 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-30T08:38:16.740 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.739+0000 7f86c2c81740 1 -- 172.21.15.32:0/3736058635 --> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559d9965d7d0 con 0x559d99770650 2024-04-30T08:38:16.741 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.739+0000 7f86b3e00640 1 --2- 172.21.15.32:0/3736058635 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x559d99770650 0x559d99772a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:16.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.741+0000 7f86b3e00640 1 --2- 172.21.15.32:0/3736058635 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x559d99770650 0x559d99772a70 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=2da3c6f300268d5b server_cookie=f83e5f4baccf40be in_seq=0 out_seq=0 2024-04-30T08:38:16.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.741+0000 7f869aa00640 10 client.26495 ms_handle_connect on v2:172.21.15.32:6838/4160386524 2024-04-30T08:38:16.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.741+0000 7f869aa00640 1 -- 172.21.15.32:0/3736058635 <== mds.1 v2:172.21.15.32:6838/4160386524 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559d9965d7d0 con 0x559d99770650 2024-04-30T08:38:16.743 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.741+0000 7f869aa00640 10 client.26495 handle_command_reply: tid=0 2024-04-30T08:38:16.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.741+0000 7f86c2c81740 1 client.26495 shutdown 2024-04-30T08:38:16.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.742+0000 7f86c2c81740 1 -- 172.21.15.32:0/3736058635 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559d993264d0 msgr2=0x559d996ff1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:16.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.742+0000 7f86c2c81740 1 --2- 172.21.15.32:0/3736058635 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559d993264d0 0x559d996ff1d0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f86a8005e00 tx=0x7f86a80093b0 comp rx=0 tx=0).stop 2024-04-30T08:38:16.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.743+0000 7f86c2c81740 1 -- 172.21.15.32:0/3736058635 shutdown_connections 2024-04-30T08:38:16.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.743+0000 7f86c2c81740 1 --2- 172.21.15.32:0/3736058635 >> [v2:172.21.15.32:6838/4160386524,v1:172.21.15.32:6839/4160386524] conn(0x559d99770650 0x559d99772a70 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:16.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.743+0000 7f86c2c81740 1 --2- 172.21.15.32:0/3736058635 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d9932e6b0 0x559d99700100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:16.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.743+0000 7f86c2c81740 1 --2- 172.21.15.32:0/3736058635 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x559d993264d0 0x559d996ff1d0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:16.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.743+0000 7f86c2c81740 1 --2- 172.21.15.32:0/3736058635 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x559d992e33a0 0x559d996fec90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:16.744 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.743+0000 7f86c2c81740 1 -- 172.21.15.32:0/3736058635 >> 172.21.15.32:0/3736058635 conn(0x559d993434e0 msgr2=0x559d996eba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:16.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.743+0000 7f869aa00640 0 client.26495 ms_handle_reset on v2:172.21.15.32:6838/4160386524 2024-04-30T08:38:16.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.743+0000 7f86c2c81740 1 -- 172.21.15.32:0/3736058635 shutdown_connections 2024-04-30T08:38:16.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.743+0000 7f86c2c81740 1 -- 172.21.15.32:0/3736058635 wait complete. 2024-04-30T08:38:16.745 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.744+0000 7f86c2c81740 20 client.26495 trim_cache size 0 max 16384 2024-04-30T08:38:16.746 INFO:teuthology.orchestra.run.smithi032.stdout:[] 2024-04-30T08:38:16.746 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.745+0000 7f86ba600640 1 -- 172.21.15.32:0/386452073 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f869c0aff30 msgr2=0x7f869c0b2440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:16.747 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.745+0000 7f86ba600640 1 --2- 172.21.15.32:0/386452073 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f869c0aff30 0x7f869c0b2440 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f86ac0c76c0 tx=0x7f86ac105240 comp rx=0 tx=0).stop 2024-04-30T08:38:16.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.746+0000 7f86ba600640 1 -- 172.21.15.32:0/386452073 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86b41098f0 msgr2=0x7f86b419f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:16.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.746+0000 7f86ba600640 1 --2- 172.21.15.32:0/386452073 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86b41098f0 0x7f86b419f350 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f86ac004af0 tx=0x7f86ac003ba0 comp rx=0 tx=0).stop 2024-04-30T08:38:16.748 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.746+0000 7f86ba600640 1 -- 172.21.15.32:0/386452073 shutdown_connections 2024-04-30T08:38:16.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.746+0000 7f86ba600640 1 --2- 172.21.15.32:0/386452073 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f869c0aff30 0x7f869c0b2440 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:16.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.746+0000 7f86ba600640 1 --2- 172.21.15.32:0/386452073 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f86b41098f0 0x7f86b419f350 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:16.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.746+0000 7f86ba600640 1 --2- 172.21.15.32:0/386452073 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f86b4104bf0 0x7f86b41983c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:16.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.746+0000 7f86ba600640 1 --2- 172.21.15.32:0/386452073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f86b41039b0 0x7f86b4197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:16.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.746+0000 7f86ba600640 1 -- 172.21.15.32:0/386452073 >> 172.21.15.32:0/386452073 conn(0x7f86b40feba0 msgr2=0x7f86b40fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:16.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.746+0000 7f86ba600640 1 -- 172.21.15.32:0/386452073 shutdown_connections 2024-04-30T08:38:16.749 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:16.746+0000 7f86ba600640 1 -- 172.21.15.32:0/386452073 wait complete. 2024-04-30T08:38:16.807 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-30T08:38:17.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:16 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:38:17.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:16 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:38:17.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:16 smithi032 ceph-mon[28331]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:38:17.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:16 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:38:17.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:16 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:38:17.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:16 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:38:17.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:16 smithi032 ceph-mon[28331]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:38:17.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:16 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:38:17.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:16 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:38:17.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:16 smithi071 ceph-mon[33385]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:38:17.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:16 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:38:17.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:16 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:38:17.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:16 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:38:17.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:16 smithi071 ceph-mon[33385]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:38:17.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:16 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:38:17.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:16 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:38:17.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:16 smithi189 ceph-mon[33038]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:38:17.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:16 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:38:17.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:16 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:38:17.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:16 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:38:17.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:16 smithi189 ceph-mon[33038]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:38:17.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.318+0000 7fecca800640 1 -- 172.21.15.32:0/3087843568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecc4107520 msgr2=0x7fecc4107930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:17.319 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.318+0000 7fecca800640 1 --2- 172.21.15.32:0/3087843568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecc4107520 0x7fecc4107930 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fecbc009be0 tx=0x7fecbc02a6d0 comp rx=0 tx=0).stop 2024-04-30T08:38:17.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.318+0000 7fecca800640 1 -- 172.21.15.32:0/3087843568 shutdown_connections 2024-04-30T08:38:17.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.318+0000 7fecca800640 1 --2- 172.21.15.32:0/3087843568 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecc4107520 0x7fecc4107930 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:17.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.319+0000 7fecca800640 1 --2- 172.21.15.32:0/3087843568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecc4104bc0 0x7fecc4106fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:17.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.319+0000 7fecca800640 1 --2- 172.21.15.32:0/3087843568 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fecc4102260 0x7fecc4104680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:17.320 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.319+0000 7fecca800640 1 -- 172.21.15.32:0/3087843568 >> 172.21.15.32:0/3087843568 conn(0x7fecc40fda70 msgr2=0x7fecc40fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:17.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.319+0000 7fecca800640 1 -- 172.21.15.32:0/3087843568 shutdown_connections 2024-04-30T08:38:17.321 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.319+0000 7fecca800640 1 -- 172.21.15.32:0/3087843568 wait complete. 2024-04-30T08:38:17.322 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.321+0000 7fecca800640 1 Processor -- start 2024-04-30T08:38:17.324 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.322+0000 7fecca800640 1 -- start start 2024-04-30T08:38:17.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.324+0000 7fecca800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecc4102260 0x7fecc41a06d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:17.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.324+0000 7fecca800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fecc4104bc0 0x7fecc41a0c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:17.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.324+0000 7fecca800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecc4107520 0x7fecc41a7ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:17.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.324+0000 7fecca800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fecc4115aa0 con 0x7fecc4107520 2024-04-30T08:38:17.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.324+0000 7fecca800640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fecc4115920 con 0x7fecc4104bc0 2024-04-30T08:38:17.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.324+0000 7fecca800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fecc4115c20 con 0x7fecc4102260 2024-04-30T08:38:17.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.325+0000 7fecc9400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecc4102260 0x7fecc41a06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:17.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.325+0000 7fecc8a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fecc4104bc0 0x7fecc41a0c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:17.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.325+0000 7fecc9400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecc4102260 0x7fecc41a06d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:45234/0 (socket says 172.21.15.32:45234) 2024-04-30T08:38:17.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.325+0000 7fecc9400640 1 -- 172.21.15.32:0/3859594487 learned_addr learned my addr 172.21.15.32:0/3859594487 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:17.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.326+0000 7fecc9e00640 1 --2- 172.21.15.32:0/3859594487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecc4107520 0x7fecc41a7ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:17.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.327+0000 7fecc9400640 1 -- 172.21.15.32:0/3859594487 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fecc4104bc0 msgr2=0x7fecc41a0c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:17.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.327+0000 7fecc9400640 1 --2- 172.21.15.32:0/3859594487 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fecc4104bc0 0x7fecc41a0c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:17.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.327+0000 7fecc9400640 1 -- 172.21.15.32:0/3859594487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecc4107520 msgr2=0x7fecc41a7ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:17.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.327+0000 7fecc9400640 1 --2- 172.21.15.32:0/3859594487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecc4107520 0x7fecc41a7ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:17.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.327+0000 7fecc9400640 1 -- 172.21.15.32:0/3859594487 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecc41a8210 con 0x7fecc4102260 2024-04-30T08:38:17.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.328+0000 7fecc8a00640 1 --2- 172.21.15.32:0/3859594487 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fecc4104bc0 0x7fecc41a0c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:38:17.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.328+0000 7fecc9e00640 1 --2- 172.21.15.32:0/3859594487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fecc4107520 0x7fecc41a7ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:38:17.330 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.328+0000 7fecc9400640 1 --2- 172.21.15.32:0/3859594487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecc4102260 0x7fecc41a06d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fecb4005fd0 tx=0x7fecb400dd20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:17.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.330+0000 7fecc2000640 1 -- 172.21.15.32:0/3859594487 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb4017070 con 0x7fecc4102260 2024-04-30T08:38:17.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.330+0000 7fecc2000640 1 -- 172.21.15.32:0/3859594487 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecb400de50 con 0x7fecc4102260 2024-04-30T08:38:17.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.330+0000 7fecc2000640 1 -- 172.21.15.32:0/3859594487 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb4003c60 con 0x7fecc4102260 2024-04-30T08:38:17.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.330+0000 7fecca800640 1 -- 172.21.15.32:0/3859594487 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecc41a8500 con 0x7fecc4102260 2024-04-30T08:38:17.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.330+0000 7fecca800640 1 -- 172.21.15.32:0/3859594487 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecc41a8a50 con 0x7fecc4102260 2024-04-30T08:38:17.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.341+0000 7fecc2000640 1 -- 172.21.15.32:0/3859594487 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7fecb4010460 con 0x7fecc4102260 2024-04-30T08:38:17.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.345+0000 7fecc2000640 1 --2- 172.21.15.32:0/3859594487 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fecac0afcf0 0x7fecac0b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:17.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.345+0000 7fecd2dbc740 1 Processor -- start 2024-04-30T08:38:17.347 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.346+0000 7fecd2dbc740 1 -- start start 2024-04-30T08:38:17.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.347+0000 7fecd2dbc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b005231390 0x55b0055d2730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:17.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.346+0000 7fecc2000640 1 -- 172.21.15.32:0/3859594487 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fecb40dea10 con 0x7fecc4102260 2024-04-30T08:38:17.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.347+0000 7fecd2dbc740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b0052084d0 0x55b0055d2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:17.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.347+0000 7fecd2dbc740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b0052106b0 0x55b0055d9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:17.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.347+0000 7fecc8a00640 1 --2- 172.21.15.32:0/3859594487 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fecac0afcf0 0x7fecac0b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:17.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.348+0000 7fecc9400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b005231390 0x55b0055d2730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:17.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.348+0000 7fecc9400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b005231390 0x55b0055d2730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:45242/0 (socket says 172.21.15.32:45242) 2024-04-30T08:38:17.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.348+0000 7fecc9400640 1 -- 172.21.15.32:0/2735131982 learned_addr learned my addr 172.21.15.32:0/2735131982 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:17.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.348+0000 7fecc9e00640 1 --2- 172.21.15.32:0/2735131982 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b0052106b0 0x55b0055d9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:17.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.348+0000 7fecc8a00640 1 --2- 172.21.15.32:0/2735131982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b0052084d0 0x55b0055d2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:17.350 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.349+0000 7fecd2dbc740 1 -- 172.21.15.32:0/2735131982 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b005537d50 con 0x55b0052084d0 2024-04-30T08:38:17.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.349+0000 7fecd2dbc740 1 -- 172.21.15.32:0/2735131982 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55b0054dd0c0 con 0x55b0052106b0 2024-04-30T08:38:17.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.349+0000 7fecd2dbc740 1 -- 172.21.15.32:0/2735131982 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b005533d10 con 0x55b005231390 2024-04-30T08:38:17.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.350+0000 7fecc9400640 1 -- 172.21.15.32:0/2735131982 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b0052106b0 msgr2=0x55b0055d9bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:17.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.350+0000 7fecc9400640 1 --2- 172.21.15.32:0/2735131982 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b0052106b0 0x55b0055d9bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:17.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.350+0000 7fecc9400640 1 -- 172.21.15.32:0/2735131982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b0052084d0 msgr2=0x55b0055d2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:17.351 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.350+0000 7fecc9400640 1 --2- 172.21.15.32:0/2735131982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b0052084d0 0x55b0055d2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:17.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.350+0000 7fecc9400640 1 -- 172.21.15.32:0/2735131982 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b0052493f0 con 0x55b005231390 2024-04-30T08:38:17.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.350+0000 7fecc9e00640 1 --2- 172.21.15.32:0/2735131982 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b0052106b0 0x55b0055d9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:38:17.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.350+0000 7fecc8a00640 1 --2- 172.21.15.32:0/2735131982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b0052084d0 0x55b0055d2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:38:17.352 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.351+0000 7fecc9400640 1 --2- 172.21.15.32:0/2735131982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b005231390 0x55b0055d2730 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fecb4009610 tx=0x7fecb40a7500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=834a5e8f81e51632 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:17.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.351+0000 7fecc8a00640 1 --2- 172.21.15.32:0/3859594487 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fecac0afcf0 0x7fecac0b2200 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fecc41a1770 tx=0x7fecb8009b90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:17.354 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.353+0000 7fecb2600640 1 -- 172.21.15.32:0/2735131982 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb40abda0 con 0x55b005231390 2024-04-30T08:38:17.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.353+0000 7fecb2600640 1 -- 172.21.15.32:0/2735131982 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecb40afe00 con 0x55b005231390 2024-04-30T08:38:17.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.353+0000 7fecb2600640 1 -- 172.21.15.32:0/2735131982 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb40abda0 con 0x55b005231390 2024-04-30T08:38:17.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.355+0000 7fecd2dbc740 1 -- 172.21.15.32:0/2735131982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b005231390 msgr2=0x55b0055d2730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:38:17.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.355+0000 7fecd2dbc740 1 --2- 172.21.15.32:0/2735131982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b005231390 0x55b0055d2730 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fecb4009610 tx=0x7fecb40a7500 comp rx=0 tx=0).stop 2024-04-30T08:38:17.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.355+0000 7fecd2dbc740 1 -- 172.21.15.32:0/2735131982 shutdown_connections 2024-04-30T08:38:17.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.355+0000 7fecd2dbc740 1 --2- 172.21.15.32:0/2735131982 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b0052106b0 0x55b0055d9bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:17.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.356+0000 7fecd2dbc740 1 --2- 172.21.15.32:0/2735131982 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b0052084d0 0x55b0055d2cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:17.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.356+0000 7fecd2dbc740 1 --2- 172.21.15.32:0/2735131982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b005231390 0x55b0055d2730 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:17.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.356+0000 7fecd2dbc740 1 -- 172.21.15.32:0/2735131982 >> 172.21.15.32:0/2735131982 conn(0x55b0052254e0 msgr2=0x55b0055ce0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:38:17.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.357+0000 7fecd2dbc740 1 -- 172.21.15.32:0/2735131982 shutdown_connections 2024-04-30T08:38:17.358 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.357+0000 7fecd2dbc740 1 -- 172.21.15.32:0/2735131982 wait complete. 2024-04-30T08:38:17.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.358+0000 7fecd2dbc740 1 -- start start 2024-04-30T08:38:17.359 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.358+0000 7fecd2dbc740 1 Processor -- start 2024-04-30T08:38:17.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.360+0000 7fecd2dbc740 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b0052084d0 0x55b0055dc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:17.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.360+0000 7fecd2dbc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0052106b0 0x55b0055dcaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:17.361 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.360+0000 7fecd2dbc740 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b005231390 0x55b0055e3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:17.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.360+0000 7fecd2dbc740 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x55b0055da540 con 0x55b0052084d0 2024-04-30T08:38:17.362 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.361+0000 7fecc9e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b005231390 0x55b0055e3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:17.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.361+0000 7fecc9e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b005231390 0x55b0055e3a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:33138/0 (socket says 172.21.15.32:33138) 2024-04-30T08:38:17.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.361+0000 7fecc9e00640 1 -- 172.21.15.32:0/1460586475 learned_addr learned my addr 172.21.15.32:0/1460586475 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:38:17.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.361+0000 7fecd2dbc740 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x55b0054f0b50 con 0x55b005231390 2024-04-30T08:38:17.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.361+0000 7fecd2dbc740 1 -- 172.21.15.32:0/1460586475 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x55b0055da6c0 con 0x55b0052106b0 2024-04-30T08:38:17.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.361+0000 7fecd2dbc740 10 client.0 _ll_register_callbacks cb 0x55b0054ba420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-30T08:38:17.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.361+0000 7fecc8a00640 1 --2- 172.21.15.32:0/1460586475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0052106b0 0x55b0055dcaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:17.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.361+0000 7fecc9400640 1 --2- 172.21.15.32:0/1460586475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b0052084d0 0x55b0055dc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:17.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.361+0000 7fecc9e00640 1 -- 172.21.15.32:0/1460586475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0052106b0 msgr2=0x55b0055dcaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:17.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.361+0000 7fecc9e00640 1 --2- 172.21.15.32:0/1460586475 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b0052106b0 0x55b0055dcaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:17.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.361+0000 7fecc9e00640 1 -- 172.21.15.32:0/1460586475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b0052084d0 msgr2=0x55b0055dc5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-30T08:38:17.363 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.361+0000 7fecc9e00640 1 --2- 172.21.15.32:0/1460586475 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x55b0052084d0 0x55b0055dc5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:38:17.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.361+0000 7fecc9e00640 1 -- 172.21.15.32:0/1460586475 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b0053137d0 con 0x55b005231390 2024-04-30T08:38:17.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.362+0000 7fecc9e00640 1 --2- 172.21.15.32:0/1460586475 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x55b005231390 0x55b0055e3a50 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fecbc0309d0 tx=0x7fecbc009910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ef34deaab33d6449 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:38:17.364 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.362+0000 7feca6a00640 10 client.0 ms_handle_connect on v2:172.21.15.71:3300/0 2024-04-30T08:38:17.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.362+0000 7feca6a00640 1 -- 172.21.15.32:0/1460586475 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecbc042070 con 0x55b005231390 2024-04-30T08:38:17.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.362+0000 7fecd2dbc740 1 -- 172.21.15.32:0/1460586475 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecb40afe00 con 0x55b005231390 2024-04-30T08:38:17.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.362+0000 7feca6a00640 1 -- 172.21.15.32:0/1460586475 <== mon.2 v2:172.21.15.71:3300/0 2 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fecbc003b70 con 0x55b005231390 2024-04-30T08:38:17.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.363+0000 7feca6a00640 1 -- 172.21.15.32:0/1460586475 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) ==== 24+0+0 (secure 0 0 0) 0x7fecbc04e050 con 0x55b005231390 2024-04-30T08:38:17.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.363+0000 7fecd2dbc740 10 client.26507 fetch_fsmap learned FSMap version 29 2024-04-30T08:38:17.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.363+0000 7fecd2dbc740 1 -- 172.21.15.32:0/1460586475 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) -- 0x55b0055e6a30 con 0x55b005231390 2024-04-30T08:38:17.365 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.363+0000 7feca6a00640 1 -- 172.21.15.32:0/1460586475 <== mon.2 v2:172.21.15.71:3300/0 4 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fecb40afe00 con 0x55b005231390 2024-04-30T08:38:17.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.363+0000 7feca6a00640 1 -- 172.21.15.32:0/1460586475 <== mon.2 v2:172.21.15.71:3300/0 5 ==== fsmap(e 29) ==== 8374+0+0 (secure 0 0 0) 0x7fecbc03c040 con 0x55b005231390 2024-04-30T08:38:17.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.363+0000 7fecd2dbc740 10 client.26507 fetch_fsmap finished waiting for FSMap version 29 2024-04-30T08:38:17.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.363+0000 7fecd2dbc740 10 client.26507 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-04-30T08:38:17.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.364+0000 7fecd2dbc740 20 client.26507 populate_metadata read hostname 'smithi032' 2024-04-30T08:38:17.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.364+0000 7fecd2dbc740 1 --2- 172.21.15.32:0/1460586475 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55b0051eff60 0x55b0055ea1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:38:17.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.364+0000 7fecd2dbc740 4 client.26507 mds_command: new command op to 24590 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-30T08:38:17.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.364+0000 7fecd2dbc740 1 -- 172.21.15.32:0/1460586475 --> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b005545960 con 0x55b0051eff60 2024-04-30T08:38:17.366 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.364+0000 7fecc9400640 1 --2- 172.21.15.32:0/1460586475 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55b0051eff60 0x55b0055ea1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:38:17.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.365+0000 7fecc9400640 1 --2- 172.21.15.32:0/1460586475 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x55b0051eff60 0x55b0055ea1b0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1fde1b26de19309d server_cookie=1ee4d82388c147c0 in_seq=0 out_seq=0 2024-04-30T08:38:17.367 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:38:17.366+0000 7feca6a00640 10 client.26507 ms_handle_connect on v2:172.21.15.71:6838/3517409993 2024-04-30T08:38:18.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:17 smithi071 ceph-mon[33385]: pgmap v517: 129 pgs: 129 active+clean; 12 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 262 MiB/s wr, 144 op/s 2024-04-30T08:38:18.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:17 smithi071 ceph-mon[33385]: pgmap v518: 129 pgs: 129 active+clean; 12 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 243 MiB/s wr, 527 op/s 2024-04-30T08:38:18.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:17 smithi071 ceph-mon[33385]: pgmap v519: 129 pgs: 129 active+clean; 13 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 210 MiB/s wr, 1.12k op/s 2024-04-30T08:38:18.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:17 smithi189 ceph-mon[33038]: pgmap v517: 129 pgs: 129 active+clean; 12 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 262 MiB/s wr, 144 op/s 2024-04-30T08:38:18.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:17 smithi189 ceph-mon[33038]: pgmap v518: 129 pgs: 129 active+clean; 12 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 243 MiB/s wr, 527 op/s 2024-04-30T08:38:18.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:17 smithi189 ceph-mon[33038]: pgmap v519: 129 pgs: 129 active+clean; 13 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 210 MiB/s wr, 1.12k op/s 2024-04-30T08:38:18.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:17 smithi032 ceph-mon[28331]: pgmap v517: 129 pgs: 129 active+clean; 12 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 262 MiB/s wr, 144 op/s 2024-04-30T08:38:18.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:17 smithi032 ceph-mon[28331]: pgmap v518: 129 pgs: 129 active+clean; 12 GiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 243 MiB/s wr, 527 op/s 2024-04-30T08:38:18.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:17 smithi032 ceph-mon[28331]: pgmap v519: 129 pgs: 129 active+clean; 13 GiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 210 MiB/s wr, 1.12k op/s 2024-04-30T08:38:20.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:19 smithi071 ceph-mon[33385]: pgmap v520: 129 pgs: 129 active+clean; 13 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 202 MiB/s wr, 3.79k op/s 2024-04-30T08:38:20.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:19 smithi189 ceph-mon[33038]: pgmap v520: 129 pgs: 129 active+clean; 13 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 202 MiB/s wr, 3.79k op/s 2024-04-30T08:38:20.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:19 smithi032 ceph-mon[28331]: pgmap v520: 129 pgs: 129 active+clean; 13 GiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 202 MiB/s wr, 3.79k op/s 2024-04-30T08:38:22.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:21 smithi071 ceph-mon[33385]: pgmap v521: 129 pgs: 129 active+clean; 13 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 139 MiB/s wr, 5.28k op/s 2024-04-30T08:38:22.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:21 smithi189 ceph-mon[33038]: pgmap v521: 129 pgs: 129 active+clean; 13 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 139 MiB/s wr, 5.28k op/s 2024-04-30T08:38:22.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:21 smithi032 ceph-mon[28331]: pgmap v521: 129 pgs: 129 active+clean; 13 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 43 KiB/s rd, 139 MiB/s wr, 5.28k op/s 2024-04-30T08:38:24.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:23 smithi071 ceph-mon[33385]: pgmap v522: 129 pgs: 129 active+clean; 13 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 108 MiB/s wr, 7.89k op/s 2024-04-30T08:38:24.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:23 smithi189 ceph-mon[33038]: pgmap v522: 129 pgs: 129 active+clean; 13 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 108 MiB/s wr, 7.89k op/s 2024-04-30T08:38:24.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:23 smithi032 ceph-mon[28331]: pgmap v522: 129 pgs: 129 active+clean; 13 GiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 108 MiB/s wr, 7.89k op/s 2024-04-30T08:38:26.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:25 smithi071 ceph-mon[33385]: pgmap v523: 129 pgs: 129 active+clean; 13 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 86 MiB/s wr, 9.03k op/s 2024-04-30T08:38:26.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:25 smithi189 ceph-mon[33038]: pgmap v523: 129 pgs: 129 active+clean; 13 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 86 MiB/s wr, 9.03k op/s 2024-04-30T08:38:26.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:25 smithi032 ceph-mon[28331]: pgmap v523: 129 pgs: 129 active+clean; 13 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 86 MiB/s wr, 9.03k op/s 2024-04-30T08:38:28.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:27 smithi071 ceph-mon[33385]: pgmap v524: 129 pgs: 129 active+clean; 13 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 83 MiB/s wr, 10.69k op/s 2024-04-30T08:38:28.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:27 smithi189 ceph-mon[33038]: pgmap v524: 129 pgs: 129 active+clean; 13 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 83 MiB/s wr, 10.69k op/s 2024-04-30T08:38:28.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:27 smithi032 ceph-mon[28331]: pgmap v524: 129 pgs: 129 active+clean; 13 GiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 83 MiB/s wr, 10.69k op/s 2024-04-30T08:38:30.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:30 smithi071 ceph-mon[33385]: pgmap v525: 129 pgs: 129 active+clean; 13 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 86 MiB/s wr, 12.07k op/s 2024-04-30T08:38:30.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:30 smithi189 ceph-mon[33038]: pgmap v525: 129 pgs: 129 active+clean; 13 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 86 MiB/s wr, 12.07k op/s 2024-04-30T08:38:30.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:30 smithi032 ceph-mon[28331]: pgmap v525: 129 pgs: 129 active+clean; 13 GiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 86 MiB/s wr, 12.07k op/s 2024-04-30T08:38:32.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:31 smithi071 ceph-mon[33385]: pgmap v526: 129 pgs: 129 active+clean; 13 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 89 MiB/s wr, 12.02k op/s 2024-04-30T08:38:32.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:31 smithi189 ceph-mon[33038]: pgmap v526: 129 pgs: 129 active+clean; 13 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 89 MiB/s wr, 12.02k op/s 2024-04-30T08:38:32.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:31 smithi032 ceph-mon[28331]: pgmap v526: 129 pgs: 129 active+clean; 13 GiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 89 MiB/s wr, 12.02k op/s 2024-04-30T08:38:34.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:33 smithi071 ceph-mon[33385]: pgmap v527: 129 pgs: 129 active+clean; 13 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 97 MiB/s wr, 12.85k op/s 2024-04-30T08:38:34.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:33 smithi189 ceph-mon[33038]: pgmap v527: 129 pgs: 129 active+clean; 13 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 97 MiB/s wr, 12.85k op/s 2024-04-30T08:38:34.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:33 smithi032 ceph-mon[28331]: pgmap v527: 129 pgs: 129 active+clean; 13 GiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 97 MiB/s wr, 12.85k op/s 2024-04-30T08:38:36.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:35 smithi071 ceph-mon[33385]: pgmap v528: 129 pgs: 129 active+clean; 13 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 89 MiB/s wr, 11.51k op/s 2024-04-30T08:38:36.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:35 smithi189 ceph-mon[33038]: pgmap v528: 129 pgs: 129 active+clean; 13 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 89 MiB/s wr, 11.51k op/s 2024-04-30T08:38:36.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:35 smithi032 ceph-mon[28331]: pgmap v528: 129 pgs: 129 active+clean; 13 GiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 89 MiB/s wr, 11.51k op/s 2024-04-30T08:38:39.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:38 smithi071 ceph-mon[33385]: pgmap v529: 129 pgs: 129 active+clean; 13 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 93 MiB/s wr, 11.93k op/s 2024-04-30T08:38:39.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:38 smithi189 ceph-mon[33038]: pgmap v529: 129 pgs: 129 active+clean; 13 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 93 MiB/s wr, 11.93k op/s 2024-04-30T08:38:39.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:38 smithi032 ceph-mon[28331]: pgmap v529: 129 pgs: 129 active+clean; 13 GiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 93 MiB/s wr, 11.93k op/s 2024-04-30T08:38:40.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:39 smithi071 ceph-mon[33385]: pgmap v530: 129 pgs: 129 active+clean; 13 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 93 MiB/s wr, 11.83k op/s 2024-04-30T08:38:40.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:39 smithi189 ceph-mon[33038]: pgmap v530: 129 pgs: 129 active+clean; 13 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 93 MiB/s wr, 11.83k op/s 2024-04-30T08:38:40.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:39 smithi032 ceph-mon[28331]: pgmap v530: 129 pgs: 129 active+clean; 13 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 93 MiB/s wr, 11.83k op/s 2024-04-30T08:38:43.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:42 smithi071 ceph-mon[33385]: pgmap v531: 129 pgs: 129 active+clean; 13 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 90 MiB/s wr, 11.39k op/s 2024-04-30T08:38:43.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:42 smithi189 ceph-mon[33038]: pgmap v531: 129 pgs: 129 active+clean; 13 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 90 MiB/s wr, 11.39k op/s 2024-04-30T08:38:43.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:42 smithi032 ceph-mon[28331]: pgmap v531: 129 pgs: 129 active+clean; 13 GiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 90 MiB/s wr, 11.39k op/s 2024-04-30T08:38:45.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:44 smithi071 ceph-mon[33385]: pgmap v532: 129 pgs: 129 active+clean; 13 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 92 MiB/s wr, 11.64k op/s 2024-04-30T08:38:45.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:44 smithi189 ceph-mon[33038]: pgmap v532: 129 pgs: 129 active+clean; 13 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 92 MiB/s wr, 11.64k op/s 2024-04-30T08:38:45.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:44 smithi032 ceph-mon[28331]: pgmap v532: 129 pgs: 129 active+clean; 13 GiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 92 MiB/s wr, 11.64k op/s 2024-04-30T08:38:46.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:46 smithi189 ceph-mon[33038]: pgmap v533: 129 pgs: 129 active+clean; 13 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 81 MiB/s wr, 10.25k op/s 2024-04-30T08:38:46.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:46 smithi032 ceph-mon[28331]: pgmap v533: 129 pgs: 129 active+clean; 13 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 81 MiB/s wr, 10.25k op/s 2024-04-30T08:38:46.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:46 smithi071 ceph-mon[33385]: pgmap v533: 129 pgs: 129 active+clean; 13 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 81 MiB/s wr, 10.25k op/s 2024-04-30T08:38:49.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:48 smithi071 ceph-mon[33385]: pgmap v534: 129 pgs: 129 active+clean; 13 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 83 MiB/s wr, 10.53k op/s 2024-04-30T08:38:49.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:48 smithi189 ceph-mon[33038]: pgmap v534: 129 pgs: 129 active+clean; 13 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 83 MiB/s wr, 10.53k op/s 2024-04-30T08:38:49.448 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:48 smithi032 ceph-mon[28331]: pgmap v534: 129 pgs: 129 active+clean; 13 GiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 83 MiB/s wr, 10.53k op/s 2024-04-30T08:38:50.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:50 smithi189 ceph-mon[33038]: pgmap v535: 129 pgs: 129 active+clean; 13 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 88 MiB/s wr, 11.12k op/s 2024-04-30T08:38:50.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:50 smithi032 ceph-mon[28331]: pgmap v535: 129 pgs: 129 active+clean; 13 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 88 MiB/s wr, 11.12k op/s 2024-04-30T08:38:50.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:50 smithi071 ceph-mon[33385]: pgmap v535: 129 pgs: 129 active+clean; 13 GiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 88 MiB/s wr, 11.12k op/s 2024-04-30T08:38:53.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:52 smithi071 ceph-mon[33385]: pgmap v536: 129 pgs: 129 active+clean; 13 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 84 MiB/s wr, 10.46k op/s 2024-04-30T08:38:53.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:52 smithi189 ceph-mon[33038]: pgmap v536: 129 pgs: 129 active+clean; 13 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 84 MiB/s wr, 10.46k op/s 2024-04-30T08:38:53.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:52 smithi032 ceph-mon[28331]: pgmap v536: 129 pgs: 129 active+clean; 13 GiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 84 MiB/s wr, 10.46k op/s 2024-04-30T08:38:54.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:54 smithi189 ceph-mon[33038]: pgmap v537: 129 pgs: 129 active+clean; 13 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 94 MiB/s wr, 11.77k op/s 2024-04-30T08:38:54.444 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:54 smithi032 ceph-mon[28331]: pgmap v537: 129 pgs: 129 active+clean; 13 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 94 MiB/s wr, 11.77k op/s 2024-04-30T08:38:54.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:54 smithi071 ceph-mon[33385]: pgmap v537: 129 pgs: 129 active+clean; 13 GiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 94 MiB/s wr, 11.77k op/s 2024-04-30T08:38:57.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:56 smithi071 ceph-mon[33385]: pgmap v538: 129 pgs: 129 active+clean; 13 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 77 MiB/s wr, 9.69k op/s 2024-04-30T08:38:57.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:56 smithi189 ceph-mon[33038]: pgmap v538: 129 pgs: 129 active+clean; 13 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 77 MiB/s wr, 9.69k op/s 2024-04-30T08:38:57.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:56 smithi032 ceph-mon[28331]: pgmap v538: 129 pgs: 129 active+clean; 13 GiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 77 MiB/s wr, 9.69k op/s 2024-04-30T08:38:58.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:38:57 smithi071 ceph-mon[33385]: pgmap v539: 129 pgs: 129 active+clean; 13 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 78 MiB/s wr, 9.83k op/s 2024-04-30T08:38:58.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:38:57 smithi189 ceph-mon[33038]: pgmap v539: 129 pgs: 129 active+clean; 13 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 78 MiB/s wr, 9.83k op/s 2024-04-30T08:38:58.343 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:38:57 smithi032 ceph-mon[28331]: pgmap v539: 129 pgs: 129 active+clean; 13 GiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 78 MiB/s wr, 9.83k op/s 2024-04-30T08:39:01.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:00 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:39:01.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:00 smithi032 ceph-mon[28331]: pgmap v540: 129 pgs: 129 active+clean; 13 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 90 MiB/s wr, 11.12k op/s 2024-04-30T08:39:01.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:00 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:39:01.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:00 smithi071 ceph-mon[33385]: pgmap v540: 129 pgs: 129 active+clean; 13 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 90 MiB/s wr, 11.12k op/s 2024-04-30T08:39:01.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:00 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:39:01.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:00 smithi189 ceph-mon[33038]: pgmap v540: 129 pgs: 129 active+clean; 13 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 90 MiB/s wr, 11.12k op/s 2024-04-30T08:39:02.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:02 smithi189 ceph-mon[33038]: pgmap v541: 129 pgs: 129 active+clean; 13 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 85 MiB/s wr, 10.47k op/s 2024-04-30T08:39:02.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:02 smithi032 ceph-mon[28331]: pgmap v541: 129 pgs: 129 active+clean; 13 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 85 MiB/s wr, 10.47k op/s 2024-04-30T08:39:02.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:02 smithi071 ceph-mon[33385]: pgmap v541: 129 pgs: 129 active+clean; 13 GiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 85 MiB/s wr, 10.47k op/s 2024-04-30T08:39:04.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:04 smithi189 ceph-mon[33038]: pgmap v542: 129 pgs: 129 active+clean; 13 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 96 MiB/s wr, 11.81k op/s 2024-04-30T08:39:04.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:04 smithi032 ceph-mon[28331]: pgmap v542: 129 pgs: 129 active+clean; 13 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 96 MiB/s wr, 11.81k op/s 2024-04-30T08:39:04.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:04 smithi071 ceph-mon[33385]: pgmap v542: 129 pgs: 129 active+clean; 13 GiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 96 MiB/s wr, 11.81k op/s 2024-04-30T08:39:06.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:06 smithi189 ceph-mon[33038]: pgmap v543: 129 pgs: 129 active+clean; 13 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 77 MiB/s wr, 9.40k op/s 2024-04-30T08:39:06.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:06 smithi032 ceph-mon[28331]: pgmap v543: 129 pgs: 129 active+clean; 13 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 77 MiB/s wr, 9.40k op/s 2024-04-30T08:39:06.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:06 smithi071 ceph-mon[33385]: pgmap v543: 129 pgs: 129 active+clean; 13 GiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 77 MiB/s wr, 9.40k op/s 2024-04-30T08:39:09.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:08 smithi071 ceph-mon[33385]: pgmap v544: 129 pgs: 129 active+clean; 13 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 80 MiB/s wr, 9.72k op/s 2024-04-30T08:39:09.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:08 smithi189 ceph-mon[33038]: pgmap v544: 129 pgs: 129 active+clean; 13 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 80 MiB/s wr, 9.72k op/s 2024-04-30T08:39:09.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:08 smithi032 ceph-mon[28331]: pgmap v544: 129 pgs: 129 active+clean; 13 GiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 80 MiB/s wr, 9.72k op/s 2024-04-30T08:39:10.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:10 smithi071 ceph-mon[33385]: pgmap v545: 129 pgs: 129 active+clean; 13 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 92 MiB/s wr, 11.10k op/s 2024-04-30T08:39:10.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:10 smithi189 ceph-mon[33038]: pgmap v545: 129 pgs: 129 active+clean; 13 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 92 MiB/s wr, 11.10k op/s 2024-04-30T08:39:10.444 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:09 smithi032 ceph-mon[28331]: pgmap v545: 129 pgs: 129 active+clean; 13 GiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 92 MiB/s wr, 11.10k op/s 2024-04-30T08:39:11.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:11 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:39:11.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:11 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:39:11.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:11 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:39:12.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:12 smithi189 ceph-mon[33038]: pgmap v546: 129 pgs: 129 active+clean; 13 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 82 MiB/s wr, 9.78k op/s 2024-04-30T08:39:12.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:12 smithi032 ceph-mon[28331]: pgmap v546: 129 pgs: 129 active+clean; 13 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 82 MiB/s wr, 9.78k op/s 2024-04-30T08:39:12.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:12 smithi071 ceph-mon[33385]: pgmap v546: 129 pgs: 129 active+clean; 13 GiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 82 MiB/s wr, 9.78k op/s 2024-04-30T08:39:14.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:14 smithi189 ceph-mon[33038]: pgmap v547: 129 pgs: 129 active+clean; 13 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 91 MiB/s wr, 10.80k op/s 2024-04-30T08:39:14.445 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:14 smithi032 ceph-mon[28331]: pgmap v547: 129 pgs: 129 active+clean; 13 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 91 MiB/s wr, 10.80k op/s 2024-04-30T08:39:14.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:14 smithi071 ceph-mon[33385]: pgmap v547: 129 pgs: 129 active+clean; 13 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 91 MiB/s wr, 10.80k op/s 2024-04-30T08:39:16.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:16 smithi189 ceph-mon[33038]: pgmap v548: 129 pgs: 129 active+clean; 13 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 76 MiB/s wr, 8.84k op/s 2024-04-30T08:39:16.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:16 smithi032 ceph-mon[28331]: pgmap v548: 129 pgs: 129 active+clean; 13 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 76 MiB/s wr, 8.84k op/s 2024-04-30T08:39:16.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:16 smithi071 ceph-mon[33385]: pgmap v548: 129 pgs: 129 active+clean; 13 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 76 MiB/s wr, 8.84k op/s 2024-04-30T08:39:17.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:17 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:39:17.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:17 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:39:17.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:17 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:39:18.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:18 smithi032 ceph-mon[28331]: pgmap v549: 129 pgs: 129 active+clean; 13 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 82 MiB/s wr, 9.47k op/s 2024-04-30T08:39:18.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:18 smithi071 ceph-mon[33385]: pgmap v549: 129 pgs: 129 active+clean; 13 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 82 MiB/s wr, 9.47k op/s 2024-04-30T08:39:18.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:18 smithi189 ceph-mon[33038]: pgmap v549: 129 pgs: 129 active+clean; 13 GiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 82 MiB/s wr, 9.47k op/s 2024-04-30T08:39:19.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:19 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:39:19.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:19 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:39:19.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:19 smithi032 ceph-mon[28331]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:39:19.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:19 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:39:19.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:19 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:39:19.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:19 smithi071 ceph-mon[33385]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:39:19.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:19 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:39:19.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:19 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:39:19.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:19 smithi189 ceph-mon[33038]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:39:20.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:20 smithi032 ceph-mon[28331]: pgmap v550: 129 pgs: 129 active+clean; 13 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 94 MiB/s wr, 10.66k op/s 2024-04-30T08:39:20.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:20 smithi071 ceph-mon[33385]: pgmap v550: 129 pgs: 129 active+clean; 13 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 94 MiB/s wr, 10.66k op/s 2024-04-30T08:39:20.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:20 smithi189 ceph-mon[33038]: pgmap v550: 129 pgs: 129 active+clean; 13 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 94 MiB/s wr, 10.66k op/s 2024-04-30T08:39:22.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:22 smithi189 ceph-mon[33038]: pgmap v551: 129 pgs: 129 active+clean; 13 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 MiB/s wr, 9.53k op/s 2024-04-30T08:39:22.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:22 smithi032 ceph-mon[28331]: pgmap v551: 129 pgs: 129 active+clean; 13 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 MiB/s wr, 9.53k op/s 2024-04-30T08:39:22.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:22 smithi071 ceph-mon[33385]: pgmap v551: 129 pgs: 129 active+clean; 13 GiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 MiB/s wr, 9.53k op/s 2024-04-30T08:39:24.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:24 smithi189 ceph-mon[33038]: pgmap v552: 129 pgs: 129 active+clean; 13 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 88 MiB/s wr, 9.91k op/s 2024-04-30T08:39:24.445 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:24 smithi032 ceph-mon[28331]: pgmap v552: 129 pgs: 129 active+clean; 13 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 88 MiB/s wr, 9.91k op/s 2024-04-30T08:39:24.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:24 smithi071 ceph-mon[33385]: pgmap v552: 129 pgs: 129 active+clean; 13 GiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 88 MiB/s wr, 9.91k op/s 2024-04-30T08:39:26.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:26 smithi189 ceph-mon[33038]: pgmap v553: 129 pgs: 129 active+clean; 13 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 79 MiB/s wr, 8.91k op/s 2024-04-30T08:39:26.489 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:26 smithi032 ceph-mon[28331]: pgmap v553: 129 pgs: 129 active+clean; 13 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 79 MiB/s wr, 8.91k op/s 2024-04-30T08:39:26.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:26 smithi071 ceph-mon[33385]: pgmap v553: 129 pgs: 129 active+clean; 13 GiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 79 MiB/s wr, 8.91k op/s 2024-04-30T08:39:28.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:28 smithi189 ceph-mon[33038]: pgmap v554: 129 pgs: 129 active+clean; 13 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 80 MiB/s wr, 9.18k op/s 2024-04-30T08:39:28.304 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:28 smithi071 ceph-mon[33385]: pgmap v554: 129 pgs: 129 active+clean; 13 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 80 MiB/s wr, 9.18k op/s 2024-04-30T08:39:28.958 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:28 smithi032 ceph-mon[28331]: pgmap v554: 129 pgs: 129 active+clean; 13 GiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 80 MiB/s wr, 9.18k op/s 2024-04-30T08:39:31.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:31 smithi071 ceph-mon[33385]: pgmap v555: 129 pgs: 129 active+clean; 13 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 89 MiB/s wr, 10.48k op/s 2024-04-30T08:39:31.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:31 smithi189 ceph-mon[33038]: pgmap v555: 129 pgs: 129 active+clean; 13 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 89 MiB/s wr, 10.48k op/s 2024-04-30T08:39:31.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:31 smithi032 ceph-mon[28331]: pgmap v555: 129 pgs: 129 active+clean; 13 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 89 MiB/s wr, 10.48k op/s 2024-04-30T08:39:32.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:32 smithi189 ceph-mon[33038]: pgmap v556: 129 pgs: 129 active+clean; 13 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 77 MiB/s wr, 9.21k op/s 2024-04-30T08:39:32.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:32 smithi032 ceph-mon[28331]: pgmap v556: 129 pgs: 129 active+clean; 13 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 77 MiB/s wr, 9.21k op/s 2024-04-30T08:39:32.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:32 smithi071 ceph-mon[33385]: pgmap v556: 129 pgs: 129 active+clean; 13 GiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 77 MiB/s wr, 9.21k op/s 2024-04-30T08:39:35.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:35 smithi071 ceph-mon[33385]: pgmap v557: 129 pgs: 129 active+clean; 13 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 81 MiB/s wr, 9.98k op/s 2024-04-30T08:39:35.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:35 smithi189 ceph-mon[33038]: pgmap v557: 129 pgs: 129 active+clean; 13 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 81 MiB/s wr, 9.98k op/s 2024-04-30T08:39:35.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:34 smithi032 ceph-mon[28331]: pgmap v557: 129 pgs: 129 active+clean; 13 GiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 81 MiB/s wr, 9.98k op/s 2024-04-30T08:39:36.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:36 smithi032 ceph-mon[28331]: pgmap v558: 129 pgs: 129 active+clean; 13 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 75 MiB/s wr, 9.40k op/s 2024-04-30T08:39:36.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:36 smithi071 ceph-mon[33385]: pgmap v558: 129 pgs: 129 active+clean; 13 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 75 MiB/s wr, 9.40k op/s 2024-04-30T08:39:36.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:36 smithi189 ceph-mon[33038]: pgmap v558: 129 pgs: 129 active+clean; 13 GiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 75 MiB/s wr, 9.40k op/s 2024-04-30T08:39:39.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:39 smithi189 ceph-mon[33038]: pgmap v559: 129 pgs: 129 active+clean; 13 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 68 MiB/s wr, 8.65k op/s 2024-04-30T08:39:39.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:39 smithi032 ceph-mon[28331]: pgmap v559: 129 pgs: 129 active+clean; 13 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 68 MiB/s wr, 8.65k op/s 2024-04-30T08:39:39.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:39 smithi071 ceph-mon[33385]: pgmap v559: 129 pgs: 129 active+clean; 13 GiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 68 MiB/s wr, 8.65k op/s 2024-04-30T08:39:40.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:40 smithi189 ceph-mon[33038]: pgmap v560: 129 pgs: 129 active+clean; 13 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 80 MiB/s wr, 10.33k op/s 2024-04-30T08:39:40.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:40 smithi032 ceph-mon[28331]: pgmap v560: 129 pgs: 129 active+clean; 13 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 80 MiB/s wr, 10.33k op/s 2024-04-30T08:39:40.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:40 smithi071 ceph-mon[33385]: pgmap v560: 129 pgs: 129 active+clean; 13 GiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 80 MiB/s wr, 10.33k op/s 2024-04-30T08:39:43.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:43 smithi189 ceph-mon[33038]: pgmap v561: 129 pgs: 129 active+clean; 13 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 67 MiB/s wr, 8.81k op/s 2024-04-30T08:39:43.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:43 smithi032 ceph-mon[28331]: pgmap v561: 129 pgs: 129 active+clean; 13 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 67 MiB/s wr, 8.81k op/s 2024-04-30T08:39:43.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:43 smithi071 ceph-mon[33385]: pgmap v561: 129 pgs: 129 active+clean; 13 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 67 MiB/s wr, 8.81k op/s 2024-04-30T08:39:44.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:44 smithi189 ceph-mon[33038]: pgmap v562: 129 pgs: 129 active+clean; 13 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 72 MiB/s wr, 9.73k op/s 2024-04-30T08:39:44.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:44 smithi032 ceph-mon[28331]: pgmap v562: 129 pgs: 129 active+clean; 13 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 72 MiB/s wr, 9.73k op/s 2024-04-30T08:39:44.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:44 smithi071 ceph-mon[33385]: pgmap v562: 129 pgs: 129 active+clean; 13 GiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 72 MiB/s wr, 9.73k op/s 2024-04-30T08:39:47.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:47 smithi071 ceph-mon[33385]: pgmap v563: 129 pgs: 129 active+clean; 13 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 61 MiB/s wr, 8.46k op/s 2024-04-30T08:39:47.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:47 smithi189 ceph-mon[33038]: pgmap v563: 129 pgs: 129 active+clean; 13 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 61 MiB/s wr, 8.46k op/s 2024-04-30T08:39:47.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:47 smithi032 ceph-mon[28331]: pgmap v563: 129 pgs: 129 active+clean; 13 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 61 MiB/s wr, 8.46k op/s 2024-04-30T08:39:49.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:49 smithi189 ceph-mon[33038]: pgmap v564: 129 pgs: 129 active+clean; 13 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 56 MiB/s wr, 7.91k op/s 2024-04-30T08:39:49.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:49 smithi032 ceph-mon[28331]: pgmap v564: 129 pgs: 129 active+clean; 13 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 56 MiB/s wr, 7.91k op/s 2024-04-30T08:39:49.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:49 smithi071 ceph-mon[33385]: pgmap v564: 129 pgs: 129 active+clean; 13 GiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 56 MiB/s wr, 7.91k op/s 2024-04-30T08:39:51.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:51 smithi189 ceph-mon[33038]: pgmap v565: 129 pgs: 129 active+clean; 13 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 64 MiB/s wr, 9.14k op/s 2024-04-30T08:39:51.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:51 smithi032 ceph-mon[28331]: pgmap v565: 129 pgs: 129 active+clean; 13 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 64 MiB/s wr, 9.14k op/s 2024-04-30T08:39:51.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:51 smithi071 ceph-mon[33385]: pgmap v565: 129 pgs: 129 active+clean; 13 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 64 MiB/s wr, 9.14k op/s 2024-04-30T08:39:53.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:53 smithi189 ceph-mon[33038]: pgmap v566: 129 pgs: 129 active+clean; 13 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 51 MiB/s wr, 7.43k op/s 2024-04-30T08:39:53.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:53 smithi032 ceph-mon[28331]: pgmap v566: 129 pgs: 129 active+clean; 13 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 51 MiB/s wr, 7.43k op/s 2024-04-30T08:39:53.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:53 smithi071 ceph-mon[33385]: pgmap v566: 129 pgs: 129 active+clean; 13 GiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 51 MiB/s wr, 7.43k op/s 2024-04-30T08:39:54.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:54 smithi189 ceph-mon[33038]: pgmap v567: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 53 MiB/s wr, 7.98k op/s 2024-04-30T08:39:54.445 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:54 smithi032 ceph-mon[28331]: pgmap v567: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 53 MiB/s wr, 7.98k op/s 2024-04-30T08:39:54.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:54 smithi071 ceph-mon[33385]: pgmap v567: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 53 MiB/s wr, 7.98k op/s 2024-04-30T08:39:57.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:57 smithi189 ceph-mon[33038]: pgmap v568: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 49 MiB/s wr, 7.52k op/s 2024-04-30T08:39:57.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:57 smithi032 ceph-mon[28331]: pgmap v568: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 49 MiB/s wr, 7.52k op/s 2024-04-30T08:39:57.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:57 smithi071 ceph-mon[33385]: pgmap v568: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 49 MiB/s wr, 7.52k op/s 2024-04-30T08:39:59.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:39:59 smithi189 ceph-mon[33038]: pgmap v569: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 46 MiB/s wr, 7.21k op/s 2024-04-30T08:39:59.305 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:39:59 smithi071 ceph-mon[33385]: pgmap v569: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 46 MiB/s wr, 7.21k op/s 2024-04-30T08:39:59.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:39:59 smithi032 ceph-mon[28331]: pgmap v569: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 46 MiB/s wr, 7.21k op/s 2024-04-30T08:40:00.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:00 smithi189 ceph-mon[33038]: overall HEALTH_OK 2024-04-30T08:40:00.444 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:00 smithi032 ceph-mon[28331]: overall HEALTH_OK 2024-04-30T08:40:00.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:00 smithi071 ceph-mon[33385]: overall HEALTH_OK 2024-04-30T08:40:01.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:01 smithi189 ceph-mon[33038]: pgmap v570: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 53 MiB/s wr, 8.32k op/s 2024-04-30T08:40:01.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:01 smithi032 ceph-mon[28331]: pgmap v570: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 53 MiB/s wr, 8.32k op/s 2024-04-30T08:40:01.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:01 smithi071 ceph-mon[33385]: pgmap v570: 129 pgs: 129 active+clean; 13 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 53 MiB/s wr, 8.32k op/s 2024-04-30T08:40:02.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:02 smithi032 ceph-mon[28331]: pgmap v571: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 47 MiB/s wr, 7.36k op/s 2024-04-30T08:40:02.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:02 smithi071 ceph-mon[33385]: pgmap v571: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 47 MiB/s wr, 7.36k op/s 2024-04-30T08:40:02.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:02 smithi189 ceph-mon[33038]: pgmap v571: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 47 MiB/s wr, 7.36k op/s 2024-04-30T08:40:05.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:05 smithi189 ceph-mon[33038]: pgmap v572: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 51 MiB/s wr, 7.88k op/s 2024-04-30T08:40:05.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:05 smithi032 ceph-mon[28331]: pgmap v572: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 51 MiB/s wr, 7.88k op/s 2024-04-30T08:40:05.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:05 smithi071 ceph-mon[33385]: pgmap v572: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 51 MiB/s wr, 7.88k op/s 2024-04-30T08:40:07.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:07 smithi189 ceph-mon[33038]: pgmap v573: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 47 MiB/s wr, 7.21k op/s 2024-04-30T08:40:07.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:07 smithi032 ceph-mon[28331]: pgmap v573: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 47 MiB/s wr, 7.21k op/s 2024-04-30T08:40:07.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:07 smithi071 ceph-mon[33385]: pgmap v573: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 47 MiB/s wr, 7.21k op/s 2024-04-30T08:40:08.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:08 smithi189 ceph-mon[33038]: pgmap v574: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 44 MiB/s wr, 6.79k op/s 2024-04-30T08:40:08.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:08 smithi032 ceph-mon[28331]: pgmap v574: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 44 MiB/s wr, 6.79k op/s 2024-04-30T08:40:08.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:08 smithi071 ceph-mon[33385]: pgmap v574: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 44 MiB/s wr, 6.79k op/s 2024-04-30T08:40:11.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:11 smithi189 ceph-mon[33038]: pgmap v575: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 50 MiB/s wr, 7.77k op/s 2024-04-30T08:40:11.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:11 smithi032 ceph-mon[28331]: pgmap v575: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 50 MiB/s wr, 7.77k op/s 2024-04-30T08:40:11.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:11 smithi071 ceph-mon[33385]: pgmap v575: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 50 MiB/s wr, 7.77k op/s 2024-04-30T08:40:12.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:12 smithi032 ceph-mon[28331]: pgmap v576: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 48 MiB/s wr, 7.40k op/s 2024-04-30T08:40:12.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:12 smithi071 ceph-mon[33385]: pgmap v576: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 48 MiB/s wr, 7.40k op/s 2024-04-30T08:40:12.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:12 smithi189 ceph-mon[33038]: pgmap v576: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 48 MiB/s wr, 7.40k op/s 2024-04-30T08:40:15.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:15 smithi189 ceph-mon[33038]: pgmap v577: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 50 MiB/s wr, 7.76k op/s 2024-04-30T08:40:15.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:15 smithi032 ceph-mon[28331]: pgmap v577: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 50 MiB/s wr, 7.76k op/s 2024-04-30T08:40:15.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:15 smithi071 ceph-mon[33385]: pgmap v577: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 50 MiB/s wr, 7.76k op/s 2024-04-30T08:40:17.158 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-04-30T08:40:17.159 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_8feaf7d9864e980041b042fd5d964ac18adfd20f/qa/tasks/fwd_scrub.py", line 38, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_8feaf7d9864e980041b042fd5d964ac18adfd20f/qa/tasks/fwd_scrub.py", line 55, in do_scrub self._scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_8feaf7d9864e980041b042fd5d964ac18adfd20f/qa/tasks/fwd_scrub.py", line 79, in _scrub self._check_damage() File "/home/teuthworker/src/git.ceph.com_ceph-c_8feaf7d9864e980041b042fd5d964ac18adfd20f/qa/tasks/fwd_scrub.py", line 82, in _check_damage rdmg = self.fs.get_damage() File "/home/teuthworker/src/git.ceph.com_ceph-c_8feaf7d9864e980041b042fd5d964ac18adfd20f/qa/tasks/cephfs/filesystem.py", line 1833, in get_damage result[rank] = self.get_damage(rank=rank) File "/home/teuthworker/src/git.ceph.com_ceph-c_8feaf7d9864e980041b042fd5d964ac18adfd20f/qa/tasks/cephfs/filesystem.py", line 1836, in get_damage return self.rank_tell(['damage', 'ls'], rank=rank) File "/home/teuthworker/src/git.ceph.com_ceph-c_8feaf7d9864e980041b042fd5d964ac18adfd20f/qa/tasks/cephfs/filesystem.py", line 1372, in rank_tell return self.mds_tell(*args, mds_id=f"{self.id}:{rank}", **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_8feaf7d9864e980041b042fd5d964ac18adfd20f/qa/tasks/cephfs/filesystem.py", line 1356, in mds_tell out = self.get_ceph_cmd_stdout("tell", f"mds.{mds_id}", *args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_8feaf7d9864e980041b042fd5d964ac18adfd20f/qa/tasks/ceph_test_case.py", line 55, in get_ceph_cmd_stdout return self.run_ceph_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_8feaf7d9864e980041b042fd5d964ac18adfd20f/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd return self.mon_manager.run_cluster_cmd(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_8feaf7d9864e980041b042fd5d964ac18adfd20f/qa/tasks/ceph_manager.py", line 1647, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi032 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls' 2024-04-30T08:40:17.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:17 smithi032 ceph-mon[28331]: pgmap v578: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 48 MiB/s wr, 7.47k op/s 2024-04-30T08:40:17.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:17 smithi071 ceph-mon[33385]: pgmap v578: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 48 MiB/s wr, 7.47k op/s 2024-04-30T08:40:17.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:17 smithi189 ceph-mon[33038]: pgmap v578: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 48 MiB/s wr, 7.47k op/s 2024-04-30T08:40:18.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:18 smithi189 ceph-mon[33038]: pgmap v579: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 49 MiB/s wr, 7.65k op/s 2024-04-30T08:40:18.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:18 smithi032 ceph-mon[28331]: pgmap v579: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 49 MiB/s wr, 7.65k op/s 2024-04-30T08:40:18.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:18 smithi071 ceph-mon[33385]: pgmap v579: 129 pgs: 129 active+clean; 13 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 49 MiB/s wr, 7.65k op/s 2024-04-30T08:40:19.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:19 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:40:20.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:19 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:40:20.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:19 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:40:20.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:20 smithi032 ceph-mon[28331]: pgmap v580: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 54 MiB/s wr, 8.24k op/s 2024-04-30T08:40:21.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:20 smithi071 ceph-mon[33385]: pgmap v580: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 54 MiB/s wr, 8.24k op/s 2024-04-30T08:40:21.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:20 smithi189 ceph-mon[33038]: pgmap v580: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 54 MiB/s wr, 8.24k op/s 2024-04-30T08:40:23.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:23 smithi189 ceph-mon[33038]: pgmap v581: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 50 MiB/s wr, 7.57k op/s 2024-04-30T08:40:23.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:23 smithi032 ceph-mon[28331]: pgmap v581: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 50 MiB/s wr, 7.57k op/s 2024-04-30T08:40:23.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:23 smithi071 ceph-mon[33385]: pgmap v581: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 50 MiB/s wr, 7.57k op/s 2024-04-30T08:40:24.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:24 smithi032 ceph-mon[28331]: pgmap v582: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 46 MiB/s wr, 6.87k op/s 2024-04-30T08:40:24.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:24 smithi071 ceph-mon[33385]: pgmap v582: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 46 MiB/s wr, 6.87k op/s 2024-04-30T08:40:24.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:24 smithi189 ceph-mon[33038]: pgmap v582: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 46 MiB/s wr, 6.87k op/s 2024-04-30T08:40:27.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:27 smithi189 ceph-mon[33038]: pgmap v583: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 45 MiB/s wr, 6.91k op/s 2024-04-30T08:40:27.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:27 smithi032 ceph-mon[28331]: pgmap v583: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 45 MiB/s wr, 6.91k op/s 2024-04-30T08:40:27.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:27 smithi071 ceph-mon[33385]: pgmap v583: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 45 MiB/s wr, 6.91k op/s 2024-04-30T08:40:28.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:28 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:40:28.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:28 smithi189 ceph-mon[33038]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:40:28.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:28 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:40:28.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:28 smithi189 ceph-mon[33038]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:40:28.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:28 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:40:28.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:28 smithi189 ceph-mon[33038]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:40:28.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:28 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:40:28.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:28 smithi189 ceph-mon[33038]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:40:28.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:28 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:40:28.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:28 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:40:28.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:28 smithi189 ceph-mon[33038]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:40:28.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:28 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:40:28.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:28 smithi032 ceph-mon[28331]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:40:28.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:28 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:40:28.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:28 smithi032 ceph-mon[28331]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:40:28.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:28 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:40:28.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:28 smithi032 ceph-mon[28331]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:40:28.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:28 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:40:28.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:28 smithi032 ceph-mon[28331]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:40:28.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:28 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:40:28.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:28 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:40:28.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:28 smithi032 ceph-mon[28331]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:40:28.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:28 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:40:28.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:28 smithi071 ceph-mon[33385]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:40:28.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:28 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:40:28.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:28 smithi071 ceph-mon[33385]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:40:28.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:28 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:40:28.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:28 smithi071 ceph-mon[33385]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:40:28.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:28 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:40:28.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:28 smithi071 ceph-mon[33385]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:40:28.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:28 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:40:28.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:28 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:40:28.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:28 smithi071 ceph-mon[33385]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:40:29.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:29 smithi189 ceph-mon[33038]: Adjusting osd_memory_target on smithi032 to 177.5M 2024-04-30T08:40:29.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:29 smithi189 ceph-mon[33038]: Unable to set osd_memory_target on smithi032 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-30T08:40:29.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:29 smithi189 ceph-mon[33038]: pgmap v584: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 42 MiB/s wr, 6.54k op/s 2024-04-30T08:40:29.342 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:29 smithi032 ceph-mon[28331]: Adjusting osd_memory_target on smithi032 to 177.5M 2024-04-30T08:40:29.404 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:29 smithi071 ceph-mon[33385]: Adjusting osd_memory_target on smithi032 to 177.5M 2024-04-30T08:40:29.404 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:29 smithi071 ceph-mon[33385]: Unable to set osd_memory_target on smithi032 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-30T08:40:29.404 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:29 smithi071 ceph-mon[33385]: pgmap v584: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 42 MiB/s wr, 6.54k op/s 2024-04-30T08:40:29.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:29 smithi032 ceph-mon[28331]: Unable to set osd_memory_target on smithi032 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-30T08:40:29.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:29 smithi032 ceph-mon[28331]: pgmap v584: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 42 MiB/s wr, 6.54k op/s 2024-04-30T08:40:30.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:30 smithi032 ceph-mon[28331]: pgmap v585: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 48 MiB/s wr, 7.51k op/s 2024-04-30T08:40:30.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:30 smithi071 ceph-mon[33385]: pgmap v585: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 48 MiB/s wr, 7.51k op/s 2024-04-30T08:40:30.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:30 smithi189 ceph-mon[33038]: pgmap v585: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 48 MiB/s wr, 7.51k op/s 2024-04-30T08:40:33.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:33 smithi189 ceph-mon[33038]: pgmap v586: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 43 MiB/s wr, 7.04k op/s 2024-04-30T08:40:33.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:33 smithi032 ceph-mon[28331]: pgmap v586: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 43 MiB/s wr, 7.04k op/s 2024-04-30T08:40:33.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:33 smithi071 ceph-mon[33385]: pgmap v586: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 43 MiB/s wr, 7.04k op/s 2024-04-30T08:40:35.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:35 smithi189 ceph-mon[33038]: pgmap v587: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 42 MiB/s wr, 6.94k op/s 2024-04-30T08:40:35.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:35 smithi032 ceph-mon[28331]: pgmap v587: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 42 MiB/s wr, 6.94k op/s 2024-04-30T08:40:35.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:35 smithi071 ceph-mon[33385]: pgmap v587: 129 pgs: 129 active+clean; 13 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 42 MiB/s wr, 6.94k op/s 2024-04-30T08:40:36.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:36 smithi189 ceph-mon[33038]: pgmap v588: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 46 MiB/s wr, 7.81k op/s 2024-04-30T08:40:36.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:36 smithi032 ceph-mon[28331]: pgmap v588: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 46 MiB/s wr, 7.81k op/s 2024-04-30T08:40:36.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:36 smithi071 ceph-mon[33385]: pgmap v588: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 46 MiB/s wr, 7.81k op/s 2024-04-30T08:40:39.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:39 smithi189 ceph-mon[33038]: pgmap v589: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 39 MiB/s wr, 6.65k op/s 2024-04-30T08:40:39.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:39 smithi032 ceph-mon[28331]: pgmap v589: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 39 MiB/s wr, 6.65k op/s 2024-04-30T08:40:39.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:39 smithi071 ceph-mon[33385]: pgmap v589: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 39 MiB/s wr, 6.65k op/s 2024-04-30T08:40:40.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:40 smithi032 ceph-mon[28331]: pgmap v590: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 46 MiB/s wr, 7.74k op/s 2024-04-30T08:40:40.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:40 smithi071 ceph-mon[33385]: pgmap v590: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 46 MiB/s wr, 7.74k op/s 2024-04-30T08:40:40.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:40 smithi189 ceph-mon[33038]: pgmap v590: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 46 MiB/s wr, 7.74k op/s 2024-04-30T08:40:43.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:42 smithi071 ceph-mon[33385]: pgmap v591: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 43 MiB/s wr, 7.11k op/s 2024-04-30T08:40:43.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:42 smithi189 ceph-mon[33038]: pgmap v591: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 43 MiB/s wr, 7.11k op/s 2024-04-30T08:40:43.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:42 smithi032 ceph-mon[28331]: pgmap v591: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 43 MiB/s wr, 7.11k op/s 2024-04-30T08:40:45.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:45 smithi189 ceph-mon[33038]: pgmap v592: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 39 MiB/s wr, 6.34k op/s 2024-04-30T08:40:45.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:45 smithi032 ceph-mon[28331]: pgmap v592: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 39 MiB/s wr, 6.34k op/s 2024-04-30T08:40:45.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:45 smithi071 ceph-mon[33385]: pgmap v592: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 39 MiB/s wr, 6.34k op/s 2024-04-30T08:40:46.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:46 smithi189 ceph-mon[33038]: pgmap v593: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 49 MiB/s wr, 7.59k op/s 2024-04-30T08:40:46.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:46 smithi032 ceph-mon[28331]: pgmap v593: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 49 MiB/s wr, 7.59k op/s 2024-04-30T08:40:46.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:46 smithi071 ceph-mon[33385]: pgmap v593: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 49 MiB/s wr, 7.59k op/s 2024-04-30T08:40:49.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:49 smithi189 ceph-mon[33038]: pgmap v594: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 43 MiB/s wr, 6.52k op/s 2024-04-30T08:40:49.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:49 smithi032 ceph-mon[28331]: pgmap v594: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 43 MiB/s wr, 6.52k op/s 2024-04-30T08:40:49.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:49 smithi071 ceph-mon[33385]: pgmap v594: 129 pgs: 129 active+clean; 13 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 43 MiB/s wr, 6.52k op/s 2024-04-30T08:40:51.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:51 smithi189 ceph-mon[33038]: pgmap v595: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 54 MiB/s wr, 8.18k op/s 2024-04-30T08:40:51.445 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:51 smithi032 ceph-mon[28331]: pgmap v595: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 54 MiB/s wr, 8.18k op/s 2024-04-30T08:40:51.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:51 smithi071 ceph-mon[33385]: pgmap v595: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 54 MiB/s wr, 8.18k op/s 2024-04-30T08:40:52.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:52 smithi032 ceph-mon[28331]: pgmap v596: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 48 MiB/s wr, 7.26k op/s 2024-04-30T08:40:52.510 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:52 smithi071 ceph-mon[33385]: pgmap v596: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 48 MiB/s wr, 7.26k op/s 2024-04-30T08:40:52.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:52 smithi189 ceph-mon[33038]: pgmap v596: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 48 MiB/s wr, 7.26k op/s 2024-04-30T08:40:54.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:54 smithi032 ceph-mon[28331]: pgmap v597: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 44 MiB/s wr, 6.51k op/s 2024-04-30T08:40:54.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:54 smithi071 ceph-mon[33385]: pgmap v597: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 44 MiB/s wr, 6.51k op/s 2024-04-30T08:40:54.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:54 smithi189 ceph-mon[33038]: pgmap v597: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 44 MiB/s wr, 6.51k op/s 2024-04-30T08:40:57.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:57 smithi189 ceph-mon[33038]: pgmap v598: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 52 MiB/s wr, 7.83k op/s 2024-04-30T08:40:57.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:57 smithi032 ceph-mon[28331]: pgmap v598: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 52 MiB/s wr, 7.83k op/s 2024-04-30T08:40:57.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:57 smithi071 ceph-mon[33385]: pgmap v598: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 52 MiB/s wr, 7.83k op/s 2024-04-30T08:40:58.444 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:40:58 smithi032 ceph-mon[28331]: pgmap v599: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 42 MiB/s wr, 6.54k op/s 2024-04-30T08:40:58.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:40:58 smithi071 ceph-mon[33385]: pgmap v599: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 42 MiB/s wr, 6.54k op/s 2024-04-30T08:40:58.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:40:58 smithi189 ceph-mon[33038]: pgmap v599: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 42 MiB/s wr, 6.54k op/s 2024-04-30T08:41:00.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:00 smithi032 ceph-mon[28331]: pgmap v600: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 48 MiB/s wr, 7.51k op/s 2024-04-30T08:41:00.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:00 smithi071 ceph-mon[33385]: pgmap v600: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 48 MiB/s wr, 7.51k op/s 2024-04-30T08:41:00.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:00 smithi189 ceph-mon[33038]: pgmap v600: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 48 MiB/s wr, 7.51k op/s 2024-04-30T08:41:02.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:02 smithi032 ceph-mon[28331]: pgmap v601: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 42 MiB/s wr, 6.84k op/s 2024-04-30T08:41:02.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:02 smithi071 ceph-mon[33385]: pgmap v601: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 42 MiB/s wr, 6.84k op/s 2024-04-30T08:41:02.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:02 smithi189 ceph-mon[33038]: pgmap v601: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 42 MiB/s wr, 6.84k op/s 2024-04-30T08:41:04.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:04 smithi032 ceph-mon[28331]: pgmap v602: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 40 MiB/s wr, 6.54k op/s 2024-04-30T08:41:04.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:04 smithi071 ceph-mon[33385]: pgmap v602: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 40 MiB/s wr, 6.54k op/s 2024-04-30T08:41:04.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:04 smithi189 ceph-mon[33038]: pgmap v602: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 40 MiB/s wr, 6.54k op/s 2024-04-30T08:41:06.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:06 smithi032 ceph-mon[28331]: pgmap v603: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 45 MiB/s wr, 7.42k op/s 2024-04-30T08:41:06.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:06 smithi071 ceph-mon[33385]: pgmap v603: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 45 MiB/s wr, 7.42k op/s 2024-04-30T08:41:06.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:06 smithi189 ceph-mon[33038]: pgmap v603: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 45 MiB/s wr, 7.42k op/s 2024-04-30T08:41:08.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:08 smithi032 ceph-mon[28331]: pgmap v604: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 39 MiB/s wr, 6.44k op/s 2024-04-30T08:41:08.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:08 smithi071 ceph-mon[33385]: pgmap v604: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 39 MiB/s wr, 6.44k op/s 2024-04-30T08:41:08.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:08 smithi189 ceph-mon[33038]: pgmap v604: 129 pgs: 129 active+clean; 13 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 39 MiB/s wr, 6.44k op/s 2024-04-30T08:41:10.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:10 smithi032 ceph-mon[28331]: pgmap v605: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 47 MiB/s wr, 7.74k op/s 2024-04-30T08:41:10.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:10 smithi071 ceph-mon[33385]: pgmap v605: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 47 MiB/s wr, 7.74k op/s 2024-04-30T08:41:10.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:10 smithi189 ceph-mon[33038]: pgmap v605: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 47 MiB/s wr, 7.74k op/s 2024-04-30T08:41:12.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:12 smithi032 ceph-mon[28331]: pgmap v606: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 45 MiB/s wr, 7.35k op/s 2024-04-30T08:41:12.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:12 smithi071 ceph-mon[33385]: pgmap v606: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 45 MiB/s wr, 7.35k op/s 2024-04-30T08:41:12.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:12 smithi189 ceph-mon[33038]: pgmap v606: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 45 MiB/s wr, 7.35k op/s 2024-04-30T08:41:14.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:14 smithi032 ceph-mon[28331]: pgmap v607: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 41 MiB/s wr, 6.70k op/s 2024-04-30T08:41:14.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:14 smithi071 ceph-mon[33385]: pgmap v607: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 41 MiB/s wr, 6.70k op/s 2024-04-30T08:41:14.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:14 smithi189 ceph-mon[33038]: pgmap v607: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 41 MiB/s wr, 6.70k op/s 2024-04-30T08:41:15.828 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:41:15.826+0000 7f585b400640 1 -- 172.21.15.32:0/1198253074 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f58380034f0 con 0x7f586410a3c0 2024-04-30T08:41:15.828 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:41:15.827+0000 7f5860c00640 1 -- 172.21.15.32:0/1198253074 <== mon.2 v2:172.21.15.71:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x55614135b450 con 0x7f586410a3c0 2024-04-30T08:41:16.041 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:41:16.040+0000 7f584cc00640 1 -- 172.21.15.32:0/2984557802 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f58300034c0 con 0x556140f992f0 2024-04-30T08:41:16.041 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:41:16.041+0000 7f584e000640 1 -- 172.21.15.32:0/2984557802 <== mon.0 v2:172.21.15.32:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f58300034c0 con 0x556140f992f0 2024-04-30T08:41:16.351 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:16 smithi189 ceph-mon[33038]: pgmap v608: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 46 MiB/s wr, 7.53k op/s 2024-04-30T08:41:16.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:16 smithi032 ceph-mon[28331]: pgmap v608: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 46 MiB/s wr, 7.53k op/s 2024-04-30T08:41:16.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:16 smithi071 ceph-mon[33385]: pgmap v608: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 46 MiB/s wr, 7.53k op/s 2024-04-30T08:41:18.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:17 smithi189 ceph-mon[33038]: Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-04-30T08:41:18.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:17 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:41:18.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:17 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:18.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:17 smithi032 ceph-mon[28331]: Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-04-30T08:41:18.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:17 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:41:18.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:17 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:18.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:17 smithi071 ceph-mon[33385]: Health check failed: 1 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-04-30T08:41:18.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:17 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:41:18.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:17 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:19.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:18 smithi189 ceph-mon[33038]: pgmap v609: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 41 MiB/s wr, 6.91k op/s 2024-04-30T08:41:19.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:18 smithi032 ceph-mon[28331]: pgmap v609: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 41 MiB/s wr, 6.91k op/s 2024-04-30T08:41:19.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:18 smithi071 ceph-mon[33385]: pgmap v609: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 41 MiB/s wr, 6.91k op/s 2024-04-30T08:41:20.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:19 smithi189 ceph-mon[33038]: Health check failed: 0 slow ops, oldest one blocked for 181 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:41:20.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:19 smithi032 ceph-mon[28331]: Health check failed: 0 slow ops, oldest one blocked for 181 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:41:20.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:19 smithi071 ceph-mon[33385]: Health check failed: 0 slow ops, oldest one blocked for 181 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:41:20.693 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:20 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:20.320+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:21.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:20 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:41:21.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:20 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:21.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:20 smithi032 ceph-mon[28331]: pgmap v610: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 48 MiB/s wr, 7.88k op/s 2024-04-30T08:41:21.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:20 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:21.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:20 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:41:21.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:20 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:21.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:20 smithi071 ceph-mon[33385]: pgmap v610: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 48 MiB/s wr, 7.88k op/s 2024-04-30T08:41:21.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:20 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:21.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:20 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:41:21.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:20 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:21.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:20 smithi189 ceph-mon[33038]: pgmap v610: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 48 MiB/s wr, 7.88k op/s 2024-04-30T08:41:21.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:20 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:21.691 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:21 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:21.298+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:22.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:21 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:22.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:21 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:22.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:21 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:22.691 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:22 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:22.272+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:23.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:22 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:41:23.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:22 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:41:23.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:22 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:23.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:22 smithi032 ceph-mon[28331]: pgmap v611: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 39 MiB/s wr, 6.37k op/s 2024-04-30T08:41:23.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:22 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:23.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:22 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:41:23.258 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:22 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:41:23.258 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:22 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:23.258 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:22 smithi071 ceph-mon[33385]: pgmap v611: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 39 MiB/s wr, 6.37k op/s 2024-04-30T08:41:23.258 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:22 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:23.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:22 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:41:23.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:22 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:41:23.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:22 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:23.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:22 smithi189 ceph-mon[33038]: pgmap v611: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 39 MiB/s wr, 6.37k op/s 2024-04-30T08:41:23.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:22 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:23.694 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:23 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:23.277+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:24.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:23 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:24.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:23 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:24.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:23 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:24.691 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:24 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:24.287+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:25.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:24 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:41:25.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:24 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:25.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:24 smithi032 ceph-mon[28331]: pgmap v612: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 35 MiB/s wr, 5.77k op/s 2024-04-30T08:41:25.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:24 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:25.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:24 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:41:25.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:24 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:25.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:24 smithi071 ceph-mon[33385]: pgmap v612: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 35 MiB/s wr, 5.77k op/s 2024-04-30T08:41:25.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:24 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:25.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:24 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:41:25.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:24 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:25.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:24 smithi189 ceph-mon[33038]: pgmap v612: 129 pgs: 129 active+clean; 13 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 35 MiB/s wr, 5.77k op/s 2024-04-30T08:41:25.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:24 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:25.691 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:25 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:25.296+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:26.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:25 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 187 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:41:26.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:25 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:26.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:25 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 187 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:41:26.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:25 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:26.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:25 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 187 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:41:26.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:25 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:26.691 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:26 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:26.341+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:27.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:26 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:41:27.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:26 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:27.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:26 smithi032 ceph-mon[28331]: pgmap v613: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 42 MiB/s wr, 6.88k op/s 2024-04-30T08:41:27.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:26 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:27.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:26 smithi032 ceph-mon[28331]: Health check update: 3 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-04-30T08:41:27.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:26 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:41:27.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:26 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:27.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:26 smithi071 ceph-mon[33385]: pgmap v613: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 42 MiB/s wr, 6.88k op/s 2024-04-30T08:41:27.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:26 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:27.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:26 smithi071 ceph-mon[33385]: Health check update: 3 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-04-30T08:41:27.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:26 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:41:27.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:26 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:27.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:26 smithi189 ceph-mon[33038]: pgmap v613: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 42 MiB/s wr, 6.88k op/s 2024-04-30T08:41:27.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:26 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:27.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:26 smithi189 ceph-mon[33038]: Health check update: 3 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-04-30T08:41:27.691 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:27 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:27.356+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:28.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:27 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:28.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:27 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:41:28.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:27 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:28.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:27 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:41:28.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:27 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:28.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:27 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:41:28.945 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:28 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:28.329+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:29.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:28 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:41:29.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:28 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:29.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:28 smithi071 ceph-mon[33385]: pgmap v614: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 37 MiB/s wr, 6.04k op/s 2024-04-30T08:41:29.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:28 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:28 smithi071 ceph-mon[33385]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:28 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:28 smithi071 ceph-mon[33385]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:28 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.255 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:28 smithi071 ceph-mon[33385]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.255 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:28 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.255 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:28 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.255 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:28 smithi071 ceph-mon[33385]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.255 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:28 smithi071 ceph-mon[33385]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.255 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:28 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.255 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:28 smithi071 ceph-mon[33385]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.255 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:28 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.255 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:28 smithi071 ceph-mon[33385]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.255 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:28 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.255 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:28 smithi071 ceph-mon[33385]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.255 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:28 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:29.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:28 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:41:29.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:28 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:29.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:28 smithi189 ceph-mon[33038]: pgmap v614: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 37 MiB/s wr, 6.04k op/s 2024-04-30T08:41:29.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:28 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:28 smithi189 ceph-mon[33038]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:28 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:28 smithi189 ceph-mon[33038]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:28 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:28 smithi189 ceph-mon[33038]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.302 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:28 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.302 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:28 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.302 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:28 smithi189 ceph-mon[33038]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.302 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:28 smithi189 ceph-mon[33038]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.302 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:28 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.302 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:28 smithi189 ceph-mon[33038]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.302 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:28 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.302 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:28 smithi189 ceph-mon[33038]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.302 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:28 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.303 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:28 smithi189 ceph-mon[33038]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.303 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:28 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:29.345 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:28 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:41:29.345 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:28 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:29.345 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:28 smithi032 ceph-mon[28331]: pgmap v614: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 37 MiB/s wr, 6.04k op/s 2024-04-30T08:41:29.345 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:28 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.345 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:28 smithi032 ceph-mon[28331]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.345 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:28 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.345 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:28 smithi032 ceph-mon[28331]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.345 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:28 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.345 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:28 smithi032 ceph-mon[28331]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.345 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:28 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.345 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:28 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.346 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:28 smithi032 ceph-mon[28331]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.346 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:28 smithi032 ceph-mon[28331]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.346 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:28 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.346 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:28 smithi032 ceph-mon[28331]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.346 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:28 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.346 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:28 smithi032 ceph-mon[28331]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.346 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:28 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.346 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:28 smithi032 ceph-mon[28331]: from='mgr.14152 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-30T08:41:29.346 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:28 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:29.691 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:29 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:29.334+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:30.225 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:29 smithi071 ceph-mon[33385]: Adjusting osd_memory_target on smithi071 to 176.2M 2024-04-30T08:41:30.225 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:29 smithi071 ceph-mon[33385]: Unable to set osd_memory_target on smithi071 to 184770969: error parsing value: Value '184770969' is below minimum 939524096 2024-04-30T08:41:30.225 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:29 smithi071 ceph-mon[33385]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-04-30T08:41:30.225 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:29 smithi071 ceph-mon[33385]: Unable to set osd_memory_target on smithi189 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-30T08:41:30.225 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:29 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:30.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:29 smithi189 ceph-mon[33038]: Adjusting osd_memory_target on smithi071 to 176.2M 2024-04-30T08:41:30.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:29 smithi189 ceph-mon[33038]: Unable to set osd_memory_target on smithi071 to 184770969: error parsing value: Value '184770969' is below minimum 939524096 2024-04-30T08:41:30.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:29 smithi189 ceph-mon[33038]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-04-30T08:41:30.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:29 smithi189 ceph-mon[33038]: Unable to set osd_memory_target on smithi189 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-30T08:41:30.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:29 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:30.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:29 smithi032 ceph-mon[28331]: Adjusting osd_memory_target on smithi071 to 176.2M 2024-04-30T08:41:30.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:29 smithi032 ceph-mon[28331]: Unable to set osd_memory_target on smithi071 to 184770969: error parsing value: Value '184770969' is below minimum 939524096 2024-04-30T08:41:30.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:29 smithi032 ceph-mon[28331]: Adjusting osd_memory_target on smithi189 to 176.2M 2024-04-30T08:41:30.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:29 smithi032 ceph-mon[28331]: Unable to set osd_memory_target on smithi189 to 184787456: error parsing value: Value '184787456' is below minimum 939524096 2024-04-30T08:41:30.387 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:29 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:30.692 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:30 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:30.384+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:31.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:30 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:41:31.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:30 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:31.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:30 smithi071 ceph-mon[33385]: pgmap v615: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 44 MiB/s wr, 7.26k op/s 2024-04-30T08:41:31.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:30 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:31.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:30 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:41:31.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:30 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:31.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:30 smithi189 ceph-mon[33038]: pgmap v615: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 44 MiB/s wr, 7.26k op/s 2024-04-30T08:41:31.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:30 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:31.433 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:30 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:41:31.433 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:30 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:31.433 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:30 smithi032 ceph-mon[28331]: pgmap v615: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 44 MiB/s wr, 7.26k op/s 2024-04-30T08:41:31.433 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:30 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:31.691 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:31 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:31.426+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:32.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:31 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:41:32.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:31 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:41:32.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:31 smithi071 ceph-mon[33385]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:41:32.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:31 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:32.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:31 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 192 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:41:32.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:31 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:41:32.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:31 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:41:32.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:31 smithi189 ceph-mon[33038]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:41:32.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:31 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:32.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:31 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 192 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:41:32.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:31 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:41:32.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:31 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:41:32.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:31 smithi032 ceph-mon[28331]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:41:32.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:31 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:32.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:31 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 192 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:41:32.941 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:32 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:32.476+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:33.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:32 smithi071 ceph-mon[33385]: pgmap v616: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 35 MiB/s wr, 6.02k op/s 2024-04-30T08:41:33.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:32 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:33.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:32 smithi189 ceph-mon[33038]: pgmap v616: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 35 MiB/s wr, 6.02k op/s 2024-04-30T08:41:33.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:32 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:33.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:32 smithi032 ceph-mon[28331]: pgmap v616: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 35 MiB/s wr, 6.02k op/s 2024-04-30T08:41:33.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:32 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:33.942 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:33 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:33.528+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:34.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:34 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:34.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:34 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:34.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:34 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:35.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:35 smithi189 ceph-mon[33038]: pgmap v617: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 35 MiB/s wr, 6.10k op/s 2024-04-30T08:41:35.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:35 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:41:35.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:35 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:41:35.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:35 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:35.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:35 smithi032 ceph-mon[28331]: pgmap v617: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 35 MiB/s wr, 6.10k op/s 2024-04-30T08:41:35.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:35 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:41:35.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:35 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:41:35.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:35 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:35.442 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:34 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:34.950+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:35.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:35 smithi071 ceph-mon[33385]: pgmap v617: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 35 MiB/s wr, 6.10k op/s 2024-04-30T08:41:35.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:35 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:41:35.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:35 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:41:35.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:35 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:36.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:36 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:36.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:36 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:36.441 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:35 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:35.948+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:36.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:36 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:37.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:37 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:37.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:37 smithi032 ceph-mon[28331]: pgmap v618: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 41 MiB/s wr, 7.01k op/s 2024-04-30T08:41:37.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:37 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:41:37.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:37 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:37.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:37 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 198 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:41:37.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:37 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:37.192 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:36 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:36.919+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:37.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:37 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:37.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:37 smithi189 ceph-mon[33038]: pgmap v618: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 41 MiB/s wr, 7.01k op/s 2024-04-30T08:41:37.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:37 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:41:37.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:37 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:37.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:37 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 198 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:41:37.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:37 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:37.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:37 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:37.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:37 smithi071 ceph-mon[33385]: pgmap v618: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 41 MiB/s wr, 7.01k op/s 2024-04-30T08:41:37.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:37 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:41:37.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:37 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:37.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:37 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 198 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:41:37.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:37 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:38.441 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:37 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:37.963+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:39.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:39 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:39.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:39 smithi032 ceph-mon[28331]: pgmap v619: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 33 MiB/s wr, 5.62k op/s 2024-04-30T08:41:39.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:39 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:41:39.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:39 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:41:39.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:39 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:39.442 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:38 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:38.962+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:39.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:39 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:39.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:39 smithi071 ceph-mon[33385]: pgmap v619: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 33 MiB/s wr, 5.62k op/s 2024-04-30T08:41:39.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:39 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:41:39.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:39 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:41:39.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:39 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:39.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:39 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:39.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:39 smithi189 ceph-mon[33038]: pgmap v619: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 33 MiB/s wr, 5.62k op/s 2024-04-30T08:41:39.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:39 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:41:39.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:39 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:41:39.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:39 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:40.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:40 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:40.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:40 smithi032 ceph-mon[28331]: pgmap v620: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 42 MiB/s wr, 7.01k op/s 2024-04-30T08:41:40.441 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:40 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:40.001+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:40.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:40 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:40.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:40 smithi071 ceph-mon[33385]: pgmap v620: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 42 MiB/s wr, 7.01k op/s 2024-04-30T08:41:40.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:40 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:40.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:40 smithi189 ceph-mon[33038]: pgmap v620: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 42 MiB/s wr, 7.01k op/s 2024-04-30T08:41:41.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:41 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:41.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:41 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:41:41.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:41 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:41.442 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:40 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:40.961+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:41.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:41 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:41.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:41 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:41:41.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:41 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:41.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:41 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:41.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:41 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:41:41.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:41 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:42.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:42 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:42.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:42 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 204 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:41:42.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:42 smithi032 ceph-mon[28331]: pgmap v621: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 36 MiB/s wr, 5.74k op/s 2024-04-30T08:41:42.442 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:42 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:41.988+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:42.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:42 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:42.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:42 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 204 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:41:42.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:42 smithi071 ceph-mon[33385]: pgmap v621: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 36 MiB/s wr, 5.74k op/s 2024-04-30T08:41:42.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:42 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:42.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:42 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 204 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:41:42.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:42 smithi189 ceph-mon[33038]: pgmap v621: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 36 MiB/s wr, 5.74k op/s 2024-04-30T08:41:43.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:43 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:43.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:43 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:41:43.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:43 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:41:43.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:43 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:43.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:43 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:43.442 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:42 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:42.958+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:43.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:43 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:43.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:43 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:41:43.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:43 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:41:43.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:43 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:43.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:43 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:43.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:43 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:43.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:43 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:41:43.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:43 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:41:43.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:43 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:43.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:43 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:44.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:44 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:44.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:44 smithi032 ceph-mon[28331]: pgmap v622: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 36 MiB/s wr, 5.65k op/s 2024-04-30T08:41:44.442 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:43 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:43.956+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:44.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:44 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:44.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:44 smithi071 ceph-mon[33385]: pgmap v622: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 36 MiB/s wr, 5.65k op/s 2024-04-30T08:41:44.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:44 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:44.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:44 smithi189 ceph-mon[33038]: pgmap v622: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 36 MiB/s wr, 5.65k op/s 2024-04-30T08:41:45.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:45 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:41:45.194 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:45 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:45.194 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:45 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:45.194 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:44 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:44.908+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:45.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:45 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:41:45.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:45 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:45.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:45 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:45.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:45 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:41:45.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:45 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:45.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:45 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:46.147 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:45 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:45.867+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:46.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:46 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:46.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:46 smithi032 ceph-mon[28331]: pgmap v623: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 43 MiB/s wr, 6.67k op/s 2024-04-30T08:41:46.452 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:46 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:46.452 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:46 smithi189 ceph-mon[33038]: pgmap v623: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 43 MiB/s wr, 6.67k op/s 2024-04-30T08:41:46.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:46 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:46.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:46 smithi071 ceph-mon[33385]: pgmap v623: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 43 MiB/s wr, 6.67k op/s 2024-04-30T08:41:47.157 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:46 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:46.880+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:47.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:47 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:41:47.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:47 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:47.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:47 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 209 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:41:47.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:47 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:47.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:47 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:41:47.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:47 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:47.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:47 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 209 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:41:47.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:47 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:47.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:47 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:41:47.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:47 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:47.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:47 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 209 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:41:47.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:47 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:48.187 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:47 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:47.837+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:48.187 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:48 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:48.187 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:48 smithi032 ceph-mon[28331]: pgmap v624: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 38 MiB/s wr, 5.87k op/s 2024-04-30T08:41:48.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:48 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:48.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:48 smithi071 ceph-mon[33385]: pgmap v624: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 38 MiB/s wr, 5.87k op/s 2024-04-30T08:41:48.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:48 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:48.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:48 smithi189 ceph-mon[33038]: pgmap v624: 129 pgs: 129 active+clean; 13 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 38 MiB/s wr, 5.87k op/s 2024-04-30T08:41:49.188 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:49 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:41:49.189 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:48 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:48.809+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:49.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:49 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:49.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:49 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:49.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:49 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:41:49.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:49 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:49.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:49 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:49.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:49 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:41:49.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:49 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:49.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:49 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:50.192 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:49 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:49.825+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:50.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:50 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:50.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:50 smithi071 ceph-mon[33385]: pgmap v625: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 47 MiB/s wr, 7.18k op/s 2024-04-30T08:41:50.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:50 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:50.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:50 smithi189 ceph-mon[33038]: pgmap v625: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 47 MiB/s wr, 7.18k op/s 2024-04-30T08:41:50.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:50 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:50.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:50 smithi032 ceph-mon[28331]: pgmap v625: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 47 MiB/s wr, 7.18k op/s 2024-04-30T08:41:51.191 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:50 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:50.851+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:51.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:51 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:41:51.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:51 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:51.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:51 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:51.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:51 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:41:51.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:51 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:51.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:51 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:51.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:51 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:41:51.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:51 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:51.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:51 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:52.195 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:51 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:51.895+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:52.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:52 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 214 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:41:52.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:52 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:52.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:52 smithi071 ceph-mon[33385]: pgmap v626: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 41 MiB/s wr, 6.37k op/s 2024-04-30T08:41:52.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:52 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 214 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:41:52.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:52 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:52.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:52 smithi189 ceph-mon[33038]: pgmap v626: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 41 MiB/s wr, 6.37k op/s 2024-04-30T08:41:52.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:52 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 214 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:41:52.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:52 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:52.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:52 smithi032 ceph-mon[28331]: pgmap v626: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 41 MiB/s wr, 6.37k op/s 2024-04-30T08:41:53.243 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:52 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:52.938+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:53.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:53 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:53.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:53 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:53.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:53 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:54.268 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:53 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:53.947+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:54.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:54 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:54.555 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:54 smithi189 ceph-mon[33038]: pgmap v627: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 36 MiB/s wr, 5.74k op/s 2024-04-30T08:41:54.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:54 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:54.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:54 smithi032 ceph-mon[28331]: pgmap v627: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 36 MiB/s wr, 5.74k op/s 2024-04-30T08:41:54.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:54 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:54.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:54 smithi071 ceph-mon[33385]: pgmap v627: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 36 MiB/s wr, 5.74k op/s 2024-04-30T08:41:55.283 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:54 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:54.953+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:55.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:55 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:41:55.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:55 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:41:55.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:55 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:55.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:55 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:55.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:55 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:41:55.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:55 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:41:55.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:55 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:55.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:55 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:55.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:55 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:41:55.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:55 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:41:55.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:55 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:55.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:55 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:56.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:56 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:41:56.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:56 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:56.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:56 smithi189 ceph-mon[33038]: pgmap v628: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 46 MiB/s wr, 7.37k op/s 2024-04-30T08:41:56.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:56 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:41:56.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:56 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:56.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:56 smithi032 ceph-mon[28331]: pgmap v628: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 46 MiB/s wr, 7.37k op/s 2024-04-30T08:41:56.691 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:56 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:56.504+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:56.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:56 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:41:56.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:56 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:56.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:56 smithi071 ceph-mon[33385]: pgmap v628: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 46 MiB/s wr, 7.37k op/s 2024-04-30T08:41:57.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:57 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:57.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:57 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 219 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:41:57.691 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:57 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:57.473+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:57.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:57 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:57.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:57 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 219 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:41:57.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:57 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:57.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:57 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 219 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:41:58.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:58 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:41:58.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:58 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:58.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:58 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:58.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:58 smithi032 ceph-mon[28331]: pgmap v629: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 41 MiB/s wr, 6.56k op/s 2024-04-30T08:41:58.692 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:58 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:58.529+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:58.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:58 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:41:58.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:58 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:58.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:58 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:58.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:58 smithi071 ceph-mon[33385]: pgmap v629: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 41 MiB/s wr, 6.56k op/s 2024-04-30T08:41:58.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:58 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:41:58.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:58 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:58.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:58 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:58.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:58 smithi189 ceph-mon[33038]: pgmap v629: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 41 MiB/s wr, 6.56k op/s 2024-04-30T08:41:59.695 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:59 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:41:59.695 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:59 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:59.695 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:41:59 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:59.696 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:41:59 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:41:59.561+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:41:59.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:59 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:41:59.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:59 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:59.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:41:59 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:41:59.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:59 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:41:59.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:59 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:41:59.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:41:59 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:00.654 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:00 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:00.654 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:00 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:00.654 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:00 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:00.654 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:00 smithi071 ceph-mon[33385]: pgmap v630: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 44 MiB/s wr, 7.13k op/s 2024-04-30T08:42:00.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:00 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:00.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:00 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:00.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:00 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:00.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:00 smithi032 ceph-mon[28331]: pgmap v630: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 44 MiB/s wr, 7.13k op/s 2024-04-30T08:42:00.694 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:00 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:00.525+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:00.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:00 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:00.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:00 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:00.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:00 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:00.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:00 smithi189 ceph-mon[33038]: pgmap v630: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 44 MiB/s wr, 7.13k op/s 2024-04-30T08:42:01.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:01 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:01.692 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:01 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:01.492+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:01.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:01 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:01.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:01 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:02.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:02 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:02.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:02 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:02.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:02 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:02.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:02 smithi032 ceph-mon[28331]: pgmap v631: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 42 MiB/s wr, 6.76k op/s 2024-04-30T08:42:02.692 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:02 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:02.546+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:02.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:02 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:02.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:02 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:02.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:02 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:02.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:02 smithi071 ceph-mon[33385]: pgmap v631: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 42 MiB/s wr, 6.76k op/s 2024-04-30T08:42:02.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:02 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:02.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:02 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:02.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:02 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:02.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:02 smithi189 ceph-mon[33038]: pgmap v631: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 42 MiB/s wr, 6.76k op/s 2024-04-30T08:42:03.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:03 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 226 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:42:03.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:03 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:03.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:03 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:03.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:03 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:03.692 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:03 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:03.588+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:03.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:03 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 226 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:42:03.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:03 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:03.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:03 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:03.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:03 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:03.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:03 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 226 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:42:03.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:03 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:03.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:03 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:03.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:03 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:04.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:04 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:04.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:04 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:04.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:04 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:04.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:04 smithi032 ceph-mon[28331]: pgmap v632: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 34 MiB/s wr, 5.56k op/s 2024-04-30T08:42:04.692 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:04 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:04.576+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:04.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:04 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:04.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:04 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:04.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:04 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:04.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:04 smithi071 ceph-mon[33385]: pgmap v632: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 34 MiB/s wr, 5.56k op/s 2024-04-30T08:42:04.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:04 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:04.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:04 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:04.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:04 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:04.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:04 smithi189 ceph-mon[33038]: pgmap v632: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 34 MiB/s wr, 5.56k op/s 2024-04-30T08:42:05.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:05 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:05.693 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:05 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:05.619+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:05.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:05 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:05.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:05 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:06.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:06 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:06.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:06 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:06.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:06 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:06.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:06 smithi071 ceph-mon[33385]: pgmap v633: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 44 MiB/s wr, 7.15k op/s 2024-04-30T08:42:06.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:06 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:06.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:06 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:06.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:06 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:06.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:06 smithi189 ceph-mon[33038]: pgmap v633: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 44 MiB/s wr, 7.15k op/s 2024-04-30T08:42:06.941 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:06 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:06.582+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:06.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:06 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:06.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:06 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:06.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:06 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:06.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:06 smithi032 ceph-mon[28331]: pgmap v633: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 44 MiB/s wr, 7.15k op/s 2024-04-30T08:42:07.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:07 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:07.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:07 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:07.941 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:07 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:07.544+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:07.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:07 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:08.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:08 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:08.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:08 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:08.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:08 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:08.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:08 smithi071 ceph-mon[33385]: pgmap v634: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 38 MiB/s wr, 6.07k op/s 2024-04-30T08:42:08.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:08 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:08.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:08 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:08.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:08 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:08.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:08 smithi189 ceph-mon[33038]: pgmap v634: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 38 MiB/s wr, 6.07k op/s 2024-04-30T08:42:08.942 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:08 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:08.520+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:08.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:08 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:08.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:08 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:08.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:08 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:08.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:08 smithi032 ceph-mon[28331]: pgmap v634: 129 pgs: 129 active+clean; 13 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 38 MiB/s wr, 6.07k op/s 2024-04-30T08:42:09.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:09 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 231 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:42:09.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:09 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:09.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:09 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 231 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:42:09.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:09 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:09.941 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:09 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:09.557+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:09.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:09 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 231 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:42:09.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:09 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:10.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:10 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:10.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:10 smithi071 ceph-mon[33385]: pgmap v635: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 40 MiB/s wr, 6.51k op/s 2024-04-30T08:42:10.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:10 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:10.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:10 smithi189 ceph-mon[33038]: pgmap v635: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 40 MiB/s wr, 6.51k op/s 2024-04-30T08:42:10.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:10 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:10.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:10 smithi032 ceph-mon[28331]: pgmap v635: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 40 MiB/s wr, 6.51k op/s 2024-04-30T08:42:10.942 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:10 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:10.552+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:11.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:11 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:11.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:11 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:11.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:11 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:11.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:11 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:11.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:11 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:11.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:11 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:11.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:11 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:11.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:11 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:11.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:11 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:11.942 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:11 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:11.509+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:12.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:12 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:12.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:12 smithi032 ceph-mon[28331]: pgmap v636: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 40 MiB/s wr, 6.38k op/s 2024-04-30T08:42:12.942 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:12 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:12.524+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:13.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:12 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:13.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:12 smithi189 ceph-mon[33038]: pgmap v636: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 40 MiB/s wr, 6.38k op/s 2024-04-30T08:42:13.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:12 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:13.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:12 smithi071 ceph-mon[33385]: pgmap v636: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 40 MiB/s wr, 6.38k op/s 2024-04-30T08:42:13.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:13 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:13.942 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:13 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:13.547+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:14.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:13 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:14.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:13 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:14.941 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:14 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:14.594+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:14.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:14 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:14.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:14 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:14.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:14 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:14.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:14 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:14.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:14 smithi032 ceph-mon[28331]: pgmap v637: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 33 MiB/s wr, 5.29k op/s 2024-04-30T08:42:15.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:14 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:15.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:14 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:15.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:14 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:15.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:14 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:15.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:14 smithi071 ceph-mon[33385]: pgmap v637: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 33 MiB/s wr, 5.29k op/s 2024-04-30T08:42:15.303 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:14 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:15.303 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:14 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:15.303 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:14 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:15.303 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:14 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:15.303 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:14 smithi189 ceph-mon[33038]: pgmap v637: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 33 MiB/s wr, 5.29k op/s 2024-04-30T08:42:15.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:15 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:15.942 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:15 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:15.625+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:16.202 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:15 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:16.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:15 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:16.941 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:16 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:16.634+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:16.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:16 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:16.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:16 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:16.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:16 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:16.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:16 smithi032 ceph-mon[28331]: pgmap v638: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 40 MiB/s wr, 6.27k op/s 2024-04-30T08:42:16.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:16 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 236 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:42:17.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:16 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:17.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:16 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:17.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:16 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:17.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:16 smithi071 ceph-mon[33385]: pgmap v638: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 40 MiB/s wr, 6.27k op/s 2024-04-30T08:42:17.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:16 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 236 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:42:17.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:16 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:17.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:16 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:17.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:16 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:17.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:16 smithi189 ceph-mon[33038]: pgmap v638: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 40 MiB/s wr, 6.27k op/s 2024-04-30T08:42:17.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:16 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 236 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:42:18.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:17 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:18.191 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:17 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:17.744+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:18.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:17 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:18.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:17 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:19.191 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:18 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:18.752+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:19.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:18 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:19.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:18 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:19.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:18 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:19.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:18 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:19.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:18 smithi032 ceph-mon[28331]: pgmap v639: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 35 MiB/s wr, 5.52k op/s 2024-04-30T08:42:19.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:18 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:19.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:18 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:19.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:18 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:19.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:18 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:19.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:18 smithi071 ceph-mon[33385]: pgmap v639: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 35 MiB/s wr, 5.52k op/s 2024-04-30T08:42:19.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:18 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:19.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:18 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:19.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:18 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:19.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:18 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:19.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:18 smithi189 ceph-mon[33038]: pgmap v639: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 35 MiB/s wr, 5.52k op/s 2024-04-30T08:42:20.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:19 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:20.192 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:19 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:19.781+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:20.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:19 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:20.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:19 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:21.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:20 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:21.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:20 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:21.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:20 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:21.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:20 smithi032 ceph-mon[28331]: pgmap v640: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 34 MiB/s wr, 5.30k op/s 2024-04-30T08:42:21.192 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:20 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:20.808+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:21.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:20 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:21.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:20 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:21.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:20 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:21.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:20 smithi071 ceph-mon[33385]: pgmap v640: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 34 MiB/s wr, 5.30k op/s 2024-04-30T08:42:21.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:20 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:21.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:20 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:21.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:20 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:21.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:20 smithi189 ceph-mon[33038]: pgmap v640: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 34 MiB/s wr, 5.30k op/s 2024-04-30T08:42:22.194 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:21 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:22.194 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:21 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 241 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:42:22.194 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:21 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:21.853+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:22.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:21 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:22.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:21 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 241 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:42:22.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:21 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:22.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:21 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 241 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:42:23.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:22 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:23.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:22 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:23.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:22 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:23.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:22 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:23.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:22 smithi071 ceph-mon[33385]: pgmap v641: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 37 MiB/s wr, 5.81k op/s 2024-04-30T08:42:23.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:22 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:23.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:22 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:23.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:22 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:23.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:22 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:23.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:22 smithi189 ceph-mon[33038]: pgmap v641: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 37 MiB/s wr, 5.81k op/s 2024-04-30T08:42:23.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:22 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:23.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:22 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:23.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:22 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:23.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:22 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:23.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:22 smithi032 ceph-mon[28331]: pgmap v641: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 37 MiB/s wr, 5.81k op/s 2024-04-30T08:42:23.442 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:22 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:22.987+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:24.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:23 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:24.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:23 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:24.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:23 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:24.442 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:24 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:24.041+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:25.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:25 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:25.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:25 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:25.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:25 smithi071 ceph-mon[33385]: pgmap v642: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 33 MiB/s wr, 5.21k op/s 2024-04-30T08:42:25.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:25 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:25.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:25 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:25.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:25 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:25.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:25 smithi189 ceph-mon[33038]: pgmap v642: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 33 MiB/s wr, 5.21k op/s 2024-04-30T08:42:25.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:25 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:25.445 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:25 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:25.445 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:25 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:25.445 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:25 smithi032 ceph-mon[28331]: pgmap v642: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 33 MiB/s wr, 5.21k op/s 2024-04-30T08:42:25.445 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:25 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:25.445 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:25 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:25.001+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:26.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:26 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:26.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:26 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:26.441 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:26 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:26.049+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:26.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:26 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:27.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:27 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:27.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:27 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:27.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:27 smithi189 ceph-mon[33038]: pgmap v643: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 41 MiB/s wr, 6.45k op/s 2024-04-30T08:42:27.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:27 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:27.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:27 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 246 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:42:27.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:27 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:27.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:27 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:27.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:27 smithi032 ceph-mon[28331]: pgmap v643: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 41 MiB/s wr, 6.45k op/s 2024-04-30T08:42:27.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:27 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:27.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:27 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 246 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:42:27.442 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:27 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:27.046+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:27.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:27 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:27.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:27 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:27.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:27 smithi071 ceph-mon[33385]: pgmap v643: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 41 MiB/s wr, 6.45k op/s 2024-04-30T08:42:27.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:27 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:27.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:27 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 246 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:42:28.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:28 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:28.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:28 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:28.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:28 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:29.074 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:28 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:28.685+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:29.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:29 smithi032 ceph-mon[28331]: pgmap v644: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 40 MiB/s wr, 6.32k op/s 2024-04-30T08:42:29.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:29 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:29.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:29 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:29.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:29 smithi071 ceph-mon[33385]: pgmap v644: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 40 MiB/s wr, 6.32k op/s 2024-04-30T08:42:29.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:29 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:29.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:29 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:29.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:29 smithi189 ceph-mon[33038]: pgmap v644: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 40 MiB/s wr, 6.32k op/s 2024-04-30T08:42:29.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:29 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:29.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:29 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:30.096 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:29 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:29.723+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:30.404 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:30 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:30.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:30 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:30.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:30 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:31.101 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:31 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:31.101 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:30 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:30.766+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:31.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:31 smithi032 ceph-mon[28331]: pgmap v645: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 39 MiB/s wr, 6.19k op/s 2024-04-30T08:42:31.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:31 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:31.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:31 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:31.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:31 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:31.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:31 smithi071 ceph-mon[33385]: pgmap v645: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 39 MiB/s wr, 6.19k op/s 2024-04-30T08:42:31.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:31 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:31.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:31 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:31.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:31 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:31.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:31 smithi189 ceph-mon[33038]: pgmap v645: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 39 MiB/s wr, 6.19k op/s 2024-04-30T08:42:31.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:31 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:31.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:31 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:32.115 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:31 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:31.787+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:32.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:32 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:32.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:32 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:42:32.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:32 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 251 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:42:32.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:32 smithi032 ceph-mon[28331]: pgmap v646: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 43 MiB/s wr, 6.97k op/s 2024-04-30T08:42:32.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:32 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:32.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:32 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:42:32.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:32 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 251 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:42:32.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:32 smithi071 ceph-mon[33385]: pgmap v646: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 43 MiB/s wr, 6.97k op/s 2024-04-30T08:42:32.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:32 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:32.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:32 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:42:32.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:32 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 251 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:42:32.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:32 smithi189 ceph-mon[33038]: pgmap v646: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 43 MiB/s wr, 6.97k op/s 2024-04-30T08:42:33.116 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:32 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:32.747+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:33.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:33 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:33.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:33 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:33.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:33 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:34.142 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:33 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:33.812+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:34.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:34 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:34.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:34 smithi032 ceph-mon[28331]: pgmap v647: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 35 MiB/s wr, 5.66k op/s 2024-04-30T08:42:34.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:34 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:34.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:34 smithi071 ceph-mon[33385]: pgmap v647: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 35 MiB/s wr, 5.66k op/s 2024-04-30T08:42:34.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:34 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:34.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:34 smithi189 ceph-mon[33038]: pgmap v647: 129 pgs: 129 active+clean; 13 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 35 MiB/s wr, 5.66k op/s 2024-04-30T08:42:35.142 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:35 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:35.143 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:35 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:35.143 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:35 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:35.143 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:35 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:35.143 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:34 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:34.842+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:35.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:35 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:35.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:35 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:35.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:35 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:35.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:35 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:35.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:35 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:35.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:35 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:35.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:35 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:35.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:35 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:36.161 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:36 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:36.161 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:36 smithi032 ceph-mon[28331]: pgmap v648: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 42 MiB/s wr, 6.71k op/s 2024-04-30T08:42:36.161 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:35 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:35.879+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:36.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:36 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:36.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:36 smithi071 ceph-mon[33385]: pgmap v648: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 42 MiB/s wr, 6.71k op/s 2024-04-30T08:42:36.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:36 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:36.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:36 smithi189 ceph-mon[33038]: pgmap v648: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 42 MiB/s wr, 6.71k op/s 2024-04-30T08:42:37.172 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:36 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:36.876+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:37.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:37 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:37.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:37 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:37.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:37 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:37.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:37 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 257 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:42:37.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:37 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:37.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:37 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:37.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:37 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:37.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:37 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 257 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:42:37.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:37 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:37.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:37 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:37.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:37 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:37.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:37 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 257 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:42:38.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:38 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:38.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:38 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:42:38.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:38 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:42:38.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:38 smithi032 ceph-mon[28331]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:42:38.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:38 smithi032 ceph-mon[28331]: pgmap v649: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 34 MiB/s wr, 5.55k op/s 2024-04-30T08:42:38.171 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:37 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:37.876+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:38.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:38 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:38.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:38 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:42:38.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:38 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:42:38.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:38 smithi071 ceph-mon[33385]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:42:38.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:38 smithi071 ceph-mon[33385]: pgmap v649: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 34 MiB/s wr, 5.55k op/s 2024-04-30T08:42:38.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:38 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:38.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:38 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:42:38.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:38 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:42:38.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:38 smithi189 ceph-mon[33038]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:42:38.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:38 smithi189 ceph-mon[33038]: pgmap v649: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 34 MiB/s wr, 5.55k op/s 2024-04-30T08:42:39.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:39 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:39.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:39 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:39.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:39 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:39.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:39 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:39.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:39 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:39.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:39 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:39.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:39 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:39.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:39 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:39.573 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:39 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:39.573 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:39 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:39.573 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:39 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:39.573 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:39 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:39.573 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:39 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:39.562+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:40.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:40 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:40.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:40 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:40.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:40 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:40.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:40 smithi032 ceph-mon[28331]: pgmap v650: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 35 MiB/s wr, 5.62k op/s 2024-04-30T08:42:40.943 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:40 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:40.548+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:41.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:40 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:41.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:40 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:41.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:40 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:41.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:40 smithi071 ceph-mon[33385]: pgmap v650: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 35 MiB/s wr, 5.62k op/s 2024-04-30T08:42:41.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:40 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:41.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:40 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:41.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:40 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:41.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:40 smithi189 ceph-mon[33038]: pgmap v650: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 35 MiB/s wr, 5.62k op/s 2024-04-30T08:42:41.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:41 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:41.941 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:41 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:41.505+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:42.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:41 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:42.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:41 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:42.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:42 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:42.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:42 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:42.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:42 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:42.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:42 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 262 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:42:42.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:42 smithi032 ceph-mon[28331]: pgmap v651: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 39 MiB/s wr, 6.26k op/s 2024-04-30T08:42:42.942 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:42 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:42.544+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:43.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:42 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:43.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:42 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:43.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:42 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:43.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:42 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 262 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:42:43.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:42 smithi071 ceph-mon[33385]: pgmap v651: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 39 MiB/s wr, 6.26k op/s 2024-04-30T08:42:43.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:42 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:43.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:42 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:43.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:42 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:43.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:42 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 262 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:42:43.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:42 smithi189 ceph-mon[33038]: pgmap v651: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 39 MiB/s wr, 6.26k op/s 2024-04-30T08:42:43.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:43 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:43.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:43 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:43.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:43 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:43.942 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:43 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:43.586+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:44.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:43 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:44.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:43 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:44.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:43 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:44.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:43 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:44.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:43 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:44.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:43 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:45.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:44 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:45.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:44 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:45.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:44 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:45.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:44 smithi071 ceph-mon[33385]: pgmap v652: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 32 MiB/s wr, 5.11k op/s 2024-04-30T08:42:45.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:44 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:45.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:44 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:45.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:44 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:45.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:44 smithi189 ceph-mon[33038]: pgmap v652: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 32 MiB/s wr, 5.11k op/s 2024-04-30T08:42:45.117 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:44 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:45.118 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:44 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:45.118 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:44 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:45.118 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:44 smithi032 ceph-mon[28331]: pgmap v652: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 32 MiB/s wr, 5.11k op/s 2024-04-30T08:42:45.441 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:45 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:45.113+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:46.442 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:46 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:46.072+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:47.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:46 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:47.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:46 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:47.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:46 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:47.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:46 smithi071 ceph-mon[33385]: pgmap v653: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 42 MiB/s wr, 6.63k op/s 2024-04-30T08:42:47.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:46 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:47.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:46 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:47.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:46 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:47.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:46 smithi189 ceph-mon[33038]: pgmap v653: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 42 MiB/s wr, 6.63k op/s 2024-04-30T08:42:47.107 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:46 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:47.107 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:46 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:47.107 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:46 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:47.107 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:46 smithi032 ceph-mon[28331]: pgmap v653: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 42 MiB/s wr, 6.63k op/s 2024-04-30T08:42:47.441 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:47 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:47.097+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:48.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:47 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:48.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:47 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 268 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:42:48.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:47 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:48.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:47 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 268 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:42:48.152 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:47 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:48.153 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:47 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 268 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:42:48.441 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:48 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:48.145+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:49.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:48 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:49.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:48 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:49.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:48 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:49.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:48 smithi071 ceph-mon[33385]: pgmap v654: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 35 MiB/s wr, 5.54k op/s 2024-04-30T08:42:49.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:48 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:49.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:48 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:49.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:48 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:49.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:48 smithi189 ceph-mon[33038]: pgmap v654: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 35 MiB/s wr, 5.54k op/s 2024-04-30T08:42:49.189 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:48 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:49.189 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:48 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:49.189 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:48 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:49.189 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:48 smithi032 ceph-mon[28331]: pgmap v654: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 35 MiB/s wr, 5.54k op/s 2024-04-30T08:42:49.441 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:49 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:49.177+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:50.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:49 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:50.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:49 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:50.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:49 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:51.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:50 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:51.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:50 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:51.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:50 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:51.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:50 smithi189 ceph-mon[33038]: pgmap v655: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 36 MiB/s wr, 5.70k op/s 2024-04-30T08:42:51.117 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:50 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:51.118 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:50 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:51.118 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:50 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:51.118 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:50 smithi032 ceph-mon[28331]: pgmap v655: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 36 MiB/s wr, 5.70k op/s 2024-04-30T08:42:51.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:50 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:51.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:50 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:51.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:50 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:51.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:50 smithi071 ceph-mon[33385]: pgmap v655: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 36 MiB/s wr, 5.70k op/s 2024-04-30T08:42:51.441 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:51 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:51.088+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:52.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:51 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:52.090 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:51 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:52.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:51 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:52.442 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:52 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:52.081+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:53.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:52 smithi189 ceph-mon[33038]: pgmap v656: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 43 MiB/s wr, 6.74k op/s 2024-04-30T08:42:53.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:52 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:53.056 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:52 smithi032 ceph-mon[28331]: pgmap v656: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 43 MiB/s wr, 6.74k op/s 2024-04-30T08:42:53.056 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:52 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:53.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:52 smithi071 ceph-mon[33385]: pgmap v656: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 43 MiB/s wr, 6.74k op/s 2024-04-30T08:42:53.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:52 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:53.441 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:53 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:53.046+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:54.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:53 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 273 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:42:54.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:53 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:54.193 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:54 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:54.044+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:54.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:53 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 273 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:42:54.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:53 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:54.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:53 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 273 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:42:54.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:53 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:55.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:54 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:55.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:54 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:55.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:54 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:55.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:54 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:55.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:54 smithi032 ceph-mon[28331]: pgmap v657: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 35 MiB/s wr, 5.53k op/s 2024-04-30T08:42:55.192 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:55 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:54.998+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:55.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:54 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:55.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:54 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:55.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:54 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:55.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:54 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:55.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:54 smithi071 ceph-mon[33385]: pgmap v657: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 35 MiB/s wr, 5.53k op/s 2024-04-30T08:42:55.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:54 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:55.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:54 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:55.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:54 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:55.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:54 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:55.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:54 smithi189 ceph-mon[33038]: pgmap v657: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 35 MiB/s wr, 5.53k op/s 2024-04-30T08:42:56.193 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:55 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:55.952+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:56.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:55 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:56.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:55 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:56.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:55 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:57.195 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:56 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:57.196 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:56 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:57.196 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:56 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:57.196 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:56 smithi032 ceph-mon[28331]: pgmap v658: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 45 MiB/s wr, 7.25k op/s 2024-04-30T08:42:57.196 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:57 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:57.013+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:57.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:56 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:57.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:56 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:57.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:56 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:57.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:56 smithi071 ceph-mon[33385]: pgmap v658: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 45 MiB/s wr, 7.25k op/s 2024-04-30T08:42:57.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:56 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:42:57.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:56 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:57.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:56 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:57.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:56 smithi189 ceph-mon[33038]: pgmap v658: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 45 MiB/s wr, 7.25k op/s 2024-04-30T08:42:58.193 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:58 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:58.013+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:58.194 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:57 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:58.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:57 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:58.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:57 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:59.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:58 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:59.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:58 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:59.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:58 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:59.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:58 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:59.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:58 smithi032 ceph-mon[28331]: pgmap v659: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 35 MiB/s wr, 5.66k op/s 2024-04-30T08:42:59.192 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:58 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:58.985+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:42:59.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:58 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:59.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:58 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:59.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:58 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:59.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:58 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:59.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:58 smithi071 ceph-mon[33385]: pgmap v659: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 35 MiB/s wr, 5.66k op/s 2024-04-30T08:42:59.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:58 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:42:59.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:58 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:42:59.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:58 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:42:59.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:58 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:42:59.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:58 smithi189 ceph-mon[33038]: pgmap v659: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 35 MiB/s wr, 5.66k op/s 2024-04-30T08:43:00.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:42:59 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:00.192 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:42:59 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:42:59.992+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:00.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:42:59 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:00.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:42:59 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:01.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:00 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:01.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:00 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:01.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:00 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:01.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:00 smithi071 ceph-mon[33385]: pgmap v660: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 36 MiB/s wr, 5.76k op/s 2024-04-30T08:43:01.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:00 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:01.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:00 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:01.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:00 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:01.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:00 smithi189 ceph-mon[33038]: pgmap v660: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 36 MiB/s wr, 5.76k op/s 2024-04-30T08:43:01.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:00 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:01.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:00 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:01.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:00 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:01.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:00 smithi032 ceph-mon[28331]: pgmap v660: 129 pgs: 129 active+clean; 13 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 36 MiB/s wr, 5.76k op/s 2024-04-30T08:43:01.442 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:01 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:01.015+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:02.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:01 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:02.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:01 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 280 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:02.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:01 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:02.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:01 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 280 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:02.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:01 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:02.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:01 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 280 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:02.444 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:01 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:01.976+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:03.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:03 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:03.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:03 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:03.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:03 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:03.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:03 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:03.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:03 smithi189 ceph-mon[33038]: pgmap v661: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 45 MiB/s wr, 7.19k op/s 2024-04-30T08:43:03.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:03 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:03.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:03 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:03.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:03 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:03.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:03 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:03.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:03 smithi032 ceph-mon[28331]: pgmap v661: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 45 MiB/s wr, 7.19k op/s 2024-04-30T08:43:03.442 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:02 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:02.969+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:03.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:03 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:03.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:03 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:03.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:03 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:03.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:03 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:03.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:03 smithi071 ceph-mon[33385]: pgmap v661: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 45 MiB/s wr, 7.19k op/s 2024-04-30T08:43:04.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:04 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:04.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:04 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:04.441 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:03 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:03.992+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:04.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:04 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:05.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:05 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:05.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:05 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:05.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:05 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:05.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:05 smithi189 ceph-mon[33038]: pgmap v662: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 34 MiB/s wr, 5.44k op/s 2024-04-30T08:43:05.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:05 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:05.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:05 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:05.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:05 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:05.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:05 smithi032 ceph-mon[28331]: pgmap v662: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 34 MiB/s wr, 5.44k op/s 2024-04-30T08:43:05.443 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:05 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:05.008+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:05.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:05 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:05.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:05 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:05.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:05 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:05.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:05 smithi071 ceph-mon[33385]: pgmap v662: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 34 MiB/s wr, 5.44k op/s 2024-04-30T08:43:06.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:06 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:06.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:06 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:06.441 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:06 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:06.030+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:06.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:06 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:07.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:07 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:07.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:07 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:07.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:07 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:07.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:07 smithi032 ceph-mon[28331]: pgmap v663: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 41 MiB/s wr, 6.66k op/s 2024-04-30T08:43:07.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:07 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 285 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:07.442 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:07 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:07.098+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:07.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:07 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:07.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:07 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:07.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:07 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:07.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:07 smithi071 ceph-mon[33385]: pgmap v663: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 41 MiB/s wr, 6.66k op/s 2024-04-30T08:43:07.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:07 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 285 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:07.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:07 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:07.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:07 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:07.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:07 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:07.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:07 smithi189 ceph-mon[33038]: pgmap v663: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 41 MiB/s wr, 6.66k op/s 2024-04-30T08:43:07.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:07 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 285 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:08.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:08 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:08.441 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:08 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:08.095+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:08.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:08 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:08.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:08 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:09.442 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:09 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:09.051+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:09.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:09 smithi032 ceph-mon[28331]: pgmap v664: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 34 MiB/s wr, 5.38k op/s 2024-04-30T08:43:09.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:09 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:09.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:09 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:09.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:09 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:09.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:09 smithi071 ceph-mon[33385]: pgmap v664: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 34 MiB/s wr, 5.38k op/s 2024-04-30T08:43:09.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:09 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:09.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:09 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:09.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:09 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:09.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:09 smithi189 ceph-mon[33038]: pgmap v664: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 34 MiB/s wr, 5.38k op/s 2024-04-30T08:43:09.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:09 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:09.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:09 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:09.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:09 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:10.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:10 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:10.441 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:10 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:10.105+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:10.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:10 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:10.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:10 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:11.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:11 smithi032 ceph-mon[28331]: pgmap v665: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 37 MiB/s wr, 5.89k op/s 2024-04-30T08:43:11.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:11 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:11.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:11 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:11.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:11 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:11.442 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:11 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:11.111+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:11.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:11 smithi071 ceph-mon[33385]: pgmap v665: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 37 MiB/s wr, 5.89k op/s 2024-04-30T08:43:11.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:11 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:11.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:11 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:11.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:11 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:11.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:11 smithi189 ceph-mon[33038]: pgmap v665: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 37 MiB/s wr, 5.89k op/s 2024-04-30T08:43:11.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:11 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:11.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:11 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:11.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:11 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:12.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:12 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:12.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:12 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 290 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:12.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:12 smithi032 ceph-mon[28331]: pgmap v666: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 44 MiB/s wr, 7.07k op/s 2024-04-30T08:43:12.441 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:12 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:12.185+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:12.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:12 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:12.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:12 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 290 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:12.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:12 smithi071 ceph-mon[33385]: pgmap v666: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 44 MiB/s wr, 7.07k op/s 2024-04-30T08:43:12.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:12 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:12.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:12 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 290 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:12.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:12 smithi189 ceph-mon[33038]: pgmap v666: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 44 MiB/s wr, 7.07k op/s 2024-04-30T08:43:13.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:13 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:13.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:13 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:13.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:13 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:13.691 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:13 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:13.228+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:14.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:14 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:14.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:14 smithi071 ceph-mon[33385]: pgmap v667: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 32 MiB/s wr, 5.32k op/s 2024-04-30T08:43:14.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:14 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:14.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:14 smithi189 ceph-mon[33038]: pgmap v667: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 32 MiB/s wr, 5.32k op/s 2024-04-30T08:43:14.694 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:14 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:14.694 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:14 smithi032 ceph-mon[28331]: pgmap v667: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 32 MiB/s wr, 5.32k op/s 2024-04-30T08:43:14.695 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:14 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:14.193+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:15.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:15 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:15.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:15 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:15.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:15 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:15.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:15 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:15.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:15 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:15.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:15 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:15.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:15 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:15.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:15 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:15.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:15 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:15.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:15 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:15.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:15 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:15.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:15 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:15.692 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:15 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:15.197+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:16.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:16 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:16.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:16 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:16.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:16 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:16.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:16 smithi032 ceph-mon[28331]: pgmap v668: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 32 MiB/s wr, 5.41k op/s 2024-04-30T08:43:16.442 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:16 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:16.165+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:16.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:16 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:16.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:16 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:16.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:16 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:16.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:16 smithi189 ceph-mon[33038]: pgmap v668: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 32 MiB/s wr, 5.41k op/s 2024-04-30T08:43:16.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:16 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:16.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:16 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:16.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:16 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:16.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:16 smithi071 ceph-mon[33385]: pgmap v668: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 32 MiB/s wr, 5.41k op/s 2024-04-30T08:43:17.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:17 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:17.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:17 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 295 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:17.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:17 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:17.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:17 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 295 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:17.692 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:17 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:17.299+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:17.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:17 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:17.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:17 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 295 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:18.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:18 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:18.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:18 smithi032 ceph-mon[28331]: pgmap v669: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 31 MiB/s wr, 5.20k op/s 2024-04-30T08:43:18.691 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:18 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:18.335+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:18.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:18 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:18.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:18 smithi071 ceph-mon[33385]: pgmap v669: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 31 MiB/s wr, 5.20k op/s 2024-04-30T08:43:18.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:18 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:18.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:18 smithi189 ceph-mon[33038]: pgmap v669: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 31 MiB/s wr, 5.20k op/s 2024-04-30T08:43:19.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:19 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:19.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:19 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:19.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:19 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:19.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:19 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:19.692 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:19 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:19.377+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:19.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:19 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:19.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:19 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:19.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:19 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:19.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:19 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:19.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:19 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:19.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:19 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:19.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:19 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:19.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:19 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:20.694 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:20 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:20.694 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:20 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:20.694 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:20 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:20.694 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:20 smithi032 ceph-mon[28331]: pgmap v670: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 31 MiB/s wr, 5.35k op/s 2024-04-30T08:43:20.694 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:20 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:20.416+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:20.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:20 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:20.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:20 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:20.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:20 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:20.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:20 smithi071 ceph-mon[33385]: pgmap v670: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 31 MiB/s wr, 5.35k op/s 2024-04-30T08:43:20.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:20 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:20.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:20 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:20.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:20 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:20.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:20 smithi189 ceph-mon[33038]: pgmap v670: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 31 MiB/s wr, 5.35k op/s 2024-04-30T08:43:21.691 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:21 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:21.414+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:21.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:21 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:21.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:21 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:21.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:21 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:22.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:22 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:22.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:22 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:22.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:22 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:22.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:22 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 300 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:22.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:22 smithi032 ceph-mon[28331]: pgmap v671: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 37 MiB/s wr, 6.22k op/s 2024-04-30T08:43:22.692 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:22 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:22.430+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:22.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:22 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:22.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:22 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:22.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:22 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:22.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:22 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 300 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:22.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:22 smithi071 ceph-mon[33385]: pgmap v671: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 37 MiB/s wr, 6.22k op/s 2024-04-30T08:43:22.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:22 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:22.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:22 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:22.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:22 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:22.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:22 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 300 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:22.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:22 smithi189 ceph-mon[33038]: pgmap v671: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 37 MiB/s wr, 6.22k op/s 2024-04-30T08:43:23.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:23 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:23.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:23 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:23.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:23 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:23.692 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:23 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:23.430+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:23.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:23 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:23.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:23 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:23.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:23 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:23.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:23 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:23.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:23 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:23.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:23 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:24.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:24 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:24.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:24 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:24.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:24 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:24.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:24 smithi032 ceph-mon[28331]: pgmap v672: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 28 MiB/s wr, 4.53k op/s 2024-04-30T08:43:24.692 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:24 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:24.484+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:24.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:24 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:24.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:24 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:24.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:24 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:24.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:24 smithi071 ceph-mon[33385]: pgmap v672: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 28 MiB/s wr, 4.53k op/s 2024-04-30T08:43:24.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:24 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:24.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:24 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:24.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:24 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:24.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:24 smithi189 ceph-mon[33038]: pgmap v672: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 28 MiB/s wr, 4.53k op/s 2024-04-30T08:43:25.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:25 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:25.692 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:25 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:25.506+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:25.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:25 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:25.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:25 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:26.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:26 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:26.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:26 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:26.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:26 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:26.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:26 smithi032 ceph-mon[28331]: pgmap v673: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 29 MiB/s wr, 4.76k op/s 2024-04-30T08:43:26.692 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:26 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:26.523+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:26.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:26 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:26.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:26 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:26.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:26 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:26.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:26 smithi071 ceph-mon[33385]: pgmap v673: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 29 MiB/s wr, 4.76k op/s 2024-04-30T08:43:26.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:26 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:26.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:26 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:26.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:26 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:26.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:26 smithi189 ceph-mon[33038]: pgmap v673: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 29 MiB/s wr, 4.76k op/s 2024-04-30T08:43:27.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:27 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:27.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:27 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 306 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:27.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:27 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:27.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:27 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 306 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:27.881 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:27 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:27.882 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:27 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 306 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:28.191 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:27 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:27.877+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:28.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:28 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:28.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:28 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:28.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:28 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:28.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:28 smithi071 ceph-mon[33385]: pgmap v674: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 36 MiB/s wr, 5.82k op/s 2024-04-30T08:43:28.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:28 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:28.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:28 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:28.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:28 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:28.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:28 smithi189 ceph-mon[33038]: pgmap v674: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 36 MiB/s wr, 5.82k op/s 2024-04-30T08:43:28.891 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:28 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:28.891 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:28 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:28.891 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:28 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:28.891 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:28 smithi032 ceph-mon[28331]: pgmap v674: 129 pgs: 129 active+clean; 13 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 36 MiB/s wr, 5.82k op/s 2024-04-30T08:43:29.191 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:28 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:28.885+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:30.191 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:29 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:29.851+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:31.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:30 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:31.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:30 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:31.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:30 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:31.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:30 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:31.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:30 smithi189 ceph-mon[33038]: pgmap v675: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 33 MiB/s wr, 5.31k op/s 2024-04-30T08:43:31.192 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:30 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:30.851+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:31.194 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:30 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:31.194 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:30 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:31.194 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:30 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:31.194 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:30 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:31.194 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:30 smithi032 ceph-mon[28331]: pgmap v675: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 33 MiB/s wr, 5.31k op/s 2024-04-30T08:43:31.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:30 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:31.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:30 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:31.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:30 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:31.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:30 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:31.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:30 smithi071 ceph-mon[33385]: pgmap v675: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 33 MiB/s wr, 5.31k op/s 2024-04-30T08:43:32.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:31 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:32.192 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:31 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:31.866+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:32.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:31 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:32.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:31 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:33.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:32 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 311 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:33.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:32 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:33.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:32 smithi189 ceph-mon[33038]: pgmap v676: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 38 MiB/s wr, 6.14k op/s 2024-04-30T08:43:33.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:32 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 311 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:33.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:32 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:33.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:32 smithi032 ceph-mon[28331]: pgmap v676: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 38 MiB/s wr, 6.14k op/s 2024-04-30T08:43:33.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:32 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 311 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:33.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:32 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:33.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:32 smithi071 ceph-mon[33385]: pgmap v676: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 38 MiB/s wr, 6.14k op/s 2024-04-30T08:43:34.691 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:34 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:34.249+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:35.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:34 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:35.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:34 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:35.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:34 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:35.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:34 smithi032 ceph-mon[28331]: pgmap v677: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 33 MiB/s wr, 5.54k op/s 2024-04-30T08:43:35.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:34 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:35.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:34 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:35.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:34 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:35.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:34 smithi071 ceph-mon[33385]: pgmap v677: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 33 MiB/s wr, 5.54k op/s 2024-04-30T08:43:35.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:34 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:35.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:34 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:35.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:34 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:35.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:34 smithi189 ceph-mon[33038]: pgmap v677: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 33 MiB/s wr, 5.54k op/s 2024-04-30T08:43:35.691 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:35 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:35.241+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:36.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:35 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:36.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:35 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:36.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:35 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:36.691 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:36 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:36.293+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:37.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:36 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:37.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:36 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:37.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:36 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:37.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:36 smithi032 ceph-mon[28331]: pgmap v678: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 40 MiB/s wr, 6.77k op/s 2024-04-30T08:43:37.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:36 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 316 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:37.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:36 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:37.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:36 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:37.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:36 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:37.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:36 smithi071 ceph-mon[33385]: pgmap v678: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 40 MiB/s wr, 6.77k op/s 2024-04-30T08:43:37.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:36 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 316 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:37.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:36 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:37.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:36 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:37.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:36 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:37.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:36 smithi189 ceph-mon[33038]: pgmap v678: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 40 MiB/s wr, 6.77k op/s 2024-04-30T08:43:37.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:36 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 316 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:37.691 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:37 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:37.323+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:38.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:37 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:38.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:37 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:38.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:37 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:38.691 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:38 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:38.344+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:39.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:38 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:39.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:38 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:39.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:38 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:39.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:38 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:39.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:38 smithi032 ceph-mon[28331]: pgmap v679: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 47 MiB/s wr, 8.06k op/s 2024-04-30T08:43:39.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:38 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:43:39.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:38 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:39.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:38 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:39.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:38 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:39.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:38 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:39.258 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:38 smithi071 ceph-mon[33385]: pgmap v679: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 47 MiB/s wr, 8.06k op/s 2024-04-30T08:43:39.258 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:38 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:43:39.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:38 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:39.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:38 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:39.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:38 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:39.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:38 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:39.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:38 smithi189 ceph-mon[33038]: pgmap v679: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 47 MiB/s wr, 8.06k op/s 2024-04-30T08:43:39.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:38 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:43:40.161 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:39 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:40.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:39 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:40.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:39 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:40.446 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:40 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:40.142+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:41.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:40 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:41.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:40 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:41.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:40 smithi032 ceph-mon[28331]: pgmap v680: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 44 MiB/s wr, 7.68k op/s 2024-04-30T08:43:41.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:40 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:43:41.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:40 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:43:41.174 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:40 smithi032 ceph-mon[28331]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:43:41.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:40 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:41.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:40 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:41.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:40 smithi071 ceph-mon[33385]: pgmap v680: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 44 MiB/s wr, 7.68k op/s 2024-04-30T08:43:41.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:40 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:43:41.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:40 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:43:41.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:40 smithi071 ceph-mon[33385]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:43:41.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:40 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:41.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:40 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:41.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:40 smithi189 ceph-mon[33038]: pgmap v680: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 44 MiB/s wr, 7.68k op/s 2024-04-30T08:43:41.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:40 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:43:41.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:40 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:43:41.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:40 smithi189 ceph-mon[33038]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:43:41.442 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:41 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:41.160+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:42.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:41 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:42.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:41 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 322 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:42.192 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:42 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:42.164+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:42.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:41 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:42.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:41 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 322 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:42.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:41 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:42.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:41 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 322 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:43.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:42 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:43.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:42 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:43.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:42 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:43.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:42 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:43.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:42 smithi071 ceph-mon[33385]: pgmap v681: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 48 MiB/s wr, 8.38k op/s 2024-04-30T08:43:43.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:42 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:43.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:42 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:43.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:42 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:43.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:42 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:43.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:42 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:43.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:42 smithi189 ceph-mon[33038]: pgmap v681: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 48 MiB/s wr, 8.38k op/s 2024-04-30T08:43:43.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:42 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:43.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:42 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:43.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:42 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:43.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:42 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:43.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:42 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:43.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:42 smithi032 ceph-mon[28331]: pgmap v681: 129 pgs: 129 active+clean; 13 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 48 MiB/s wr, 8.38k op/s 2024-04-30T08:43:43.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:42 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:43.443 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:43 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:43.166+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:44.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:43 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:44.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:43 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:44.441 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:44 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:44.128+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:44.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:43 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:45.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:44 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:45.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:44 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:45.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:44 smithi071 ceph-mon[33385]: pgmap v682: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 41 MiB/s wr, 7.24k op/s 2024-04-30T08:43:45.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:44 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:45.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:44 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:45.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:44 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:45.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:44 smithi189 ceph-mon[33038]: pgmap v682: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 41 MiB/s wr, 7.24k op/s 2024-04-30T08:43:45.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:44 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:45.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:44 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:45.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:44 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:45.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:44 smithi032 ceph-mon[28331]: pgmap v682: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s rd, 41 MiB/s wr, 7.24k op/s 2024-04-30T08:43:45.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:44 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:46.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:46 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:46.193 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:45 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:45.811+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:46.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:46 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:46.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:46 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:47.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:47 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:47.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:47 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:47.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:47 smithi032 ceph-mon[28331]: pgmap v683: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 40 MiB/s wr, 7.33k op/s 2024-04-30T08:43:47.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:47 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:47.192 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:46 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:46.807+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:47.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:47 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:47.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:47 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:47.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:47 smithi189 ceph-mon[33038]: pgmap v683: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 40 MiB/s wr, 7.33k op/s 2024-04-30T08:43:47.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:47 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:47.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:47 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:47.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:47 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:47.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:47 smithi071 ceph-mon[33385]: pgmap v683: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 40 MiB/s wr, 7.33k op/s 2024-04-30T08:43:47.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:47 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:48.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:48 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 328 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:48.194 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:48 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:48.194 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:47 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:47.878+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:48.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:48 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 328 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:48.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:48 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:48.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:48 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 328 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:48.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:48 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:49.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:49 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:49.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:49 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:49.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:49 smithi032 ceph-mon[28331]: pgmap v684: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 42 MiB/s wr, 7.75k op/s 2024-04-30T08:43:49.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:49 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:49.192 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:48 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:48.900+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:49.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:49 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:49.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:49 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:49.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:49 smithi189 ceph-mon[33038]: pgmap v684: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 42 MiB/s wr, 7.75k op/s 2024-04-30T08:43:49.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:49 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:49.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:49 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:49.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:49 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:49.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:49 smithi071 ceph-mon[33385]: pgmap v684: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 KiB/s rd, 42 MiB/s wr, 7.75k op/s 2024-04-30T08:43:49.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:49 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:50.192 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:49 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:49.854+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:50.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:50 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:50.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:50 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:50.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:50 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:51.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:51 smithi032 ceph-mon[28331]: pgmap v685: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 36 MiB/s wr, 6.77k op/s 2024-04-30T08:43:51.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:51 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:51.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:51 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:51.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:51 smithi071 ceph-mon[33385]: pgmap v685: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 36 MiB/s wr, 6.77k op/s 2024-04-30T08:43:51.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:51 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:51.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:51 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:51.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:51 smithi189 ceph-mon[33038]: pgmap v685: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 36 MiB/s wr, 6.77k op/s 2024-04-30T08:43:51.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:51 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:51.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:51 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:52.442 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:52 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:52.186+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:52.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:52 smithi032 ceph-mon[28331]: pgmap v686: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 38 MiB/s wr, 7.12k op/s 2024-04-30T08:43:52.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:52 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 334 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:52.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:52 smithi071 ceph-mon[33385]: pgmap v686: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 38 MiB/s wr, 7.12k op/s 2024-04-30T08:43:52.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:52 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 334 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:52.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:52 smithi189 ceph-mon[33038]: pgmap v686: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 38 MiB/s wr, 7.12k op/s 2024-04-30T08:43:52.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:52 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 334 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:53.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:53 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:53.442 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:53 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:53.165+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:53.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:53 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:53.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:53 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:54.441 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:54 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:54.195+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:54.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:54 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:54.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:54 smithi032 ceph-mon[28331]: pgmap v687: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 34 MiB/s wr, 6.55k op/s 2024-04-30T08:43:54.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:54 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:54.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:54 smithi071 ceph-mon[33385]: pgmap v687: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 34 MiB/s wr, 6.55k op/s 2024-04-30T08:43:54.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:54 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:54.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:54 smithi189 ceph-mon[33038]: pgmap v687: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 34 MiB/s wr, 6.55k op/s 2024-04-30T08:43:55.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:55 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:55.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:55 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:55.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:55 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:55.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:55 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:55.443 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:55 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:55.147+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:55.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:55 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:55.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:55 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:55.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:55 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:55.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:55 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:55.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:55 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:55.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:55 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:55.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:55 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:55.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:55 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:56.441 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:56 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:56.162+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:56.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:56 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:56.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:56 smithi032 ceph-mon[28331]: pgmap v688: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 37 MiB/s wr, 7.21k op/s 2024-04-30T08:43:56.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:56 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:56.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:56 smithi071 ceph-mon[33385]: pgmap v688: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 37 MiB/s wr, 7.21k op/s 2024-04-30T08:43:56.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:56 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:56.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:56 smithi189 ceph-mon[33038]: pgmap v688: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 37 MiB/s wr, 7.21k op/s 2024-04-30T08:43:57.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:57 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:57.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:57 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:57.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:57 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:57.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:57 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:57.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:57 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:57.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:57 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:57.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:57 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:43:57.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:57 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:43:57.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:57 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:58.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:58 smithi071 ceph-mon[33385]: pgmap v689: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 39 MiB/s wr, 7.60k op/s 2024-04-30T08:43:58.524 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:58 smithi032 ceph-mon[28331]: pgmap v689: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 39 MiB/s wr, 7.60k op/s 2024-04-30T08:43:58.524 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:58 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:58.516+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:58.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:58 smithi189 ceph-mon[33038]: pgmap v689: 129 pgs: 129 active+clean; 13 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 39 MiB/s wr, 7.60k op/s 2024-04-30T08:43:59.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:59 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 340 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:59.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:59 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:59.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:59 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:59.502 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:43:59 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:59.503 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:43:59 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:43:59.496+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:43:59.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:59 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 340 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:59.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:59 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:59.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:59 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:59.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:43:59 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:43:59.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:59 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 340 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:43:59.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:59 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:43:59.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:59 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:43:59.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:43:59 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:00.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:00 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:00.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:00 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:44:00.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:00 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:00.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:00 smithi189 ceph-mon[33038]: pgmap v690: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 31 MiB/s wr, 6.12k op/s 2024-04-30T08:44:00.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:00 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:00.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:00 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:44:00.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:00 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:00.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:00 smithi032 ceph-mon[28331]: pgmap v690: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 31 MiB/s wr, 6.12k op/s 2024-04-30T08:44:00.692 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:00 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:00.469+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:00.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:00 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:00.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:00 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:44:00.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:00 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:00.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:00 smithi071 ceph-mon[33385]: pgmap v690: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 31 MiB/s wr, 6.12k op/s 2024-04-30T08:44:01.506 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:01 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:01.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:01 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:01.654 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:01 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:01.941 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:01 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:01.497+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:02.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:02 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:02.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:02 smithi189 ceph-mon[33038]: pgmap v691: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 33 MiB/s wr, 6.58k op/s 2024-04-30T08:44:02.556 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:02 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:02.556 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:02 smithi032 ceph-mon[28331]: pgmap v691: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 33 MiB/s wr, 6.58k op/s 2024-04-30T08:44:02.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:02 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:02.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:02 smithi071 ceph-mon[33385]: pgmap v691: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 33 MiB/s wr, 6.58k op/s 2024-04-30T08:44:02.941 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:02 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:02.547+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:03.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:03 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:03.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:03 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:44:03.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:03 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:44:03.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:03 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:03.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:03 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:03.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:03 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:44:03.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:03 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:44:03.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:03 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:03.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:03 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:03.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:03 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:44:03.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:03 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:44:03.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:03 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:04.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:04 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:04.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:04 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:44:04.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:04 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:04.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:04 smithi189 ceph-mon[33038]: pgmap v692: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 27 MiB/s wr, 5.54k op/s 2024-04-30T08:44:04.695 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:04 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:04.695 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:04 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:44:04.695 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:04 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:04.695 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:04 smithi032 ceph-mon[28331]: pgmap v692: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 27 MiB/s wr, 5.54k op/s 2024-04-30T08:44:04.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:04 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:04.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:04 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:44:04.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:04 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:04.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:04 smithi071 ceph-mon[33385]: pgmap v692: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 27 MiB/s wr, 5.54k op/s 2024-04-30T08:44:05.442 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:05 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:05.166+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:06.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:06 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:06.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:06 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:44:06.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:06 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:06.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:06 smithi032 ceph-mon[28331]: pgmap v693: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 27 MiB/s wr, 5.42k op/s 2024-04-30T08:44:06.444 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:06 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:06.173+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:06.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:06 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:06.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:06 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:44:06.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:06 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:06.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:06 smithi071 ceph-mon[33385]: pgmap v693: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 27 MiB/s wr, 5.42k op/s 2024-04-30T08:44:06.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:06 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:06.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:06 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:44:06.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:06 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:06.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:06 smithi189 ceph-mon[33038]: pgmap v693: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 27 MiB/s wr, 5.42k op/s 2024-04-30T08:44:07.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:07 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:07.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:07 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 347 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:44:07.442 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:07 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:07.153+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:07.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:07 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:07.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:07 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 347 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:44:07.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:07 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:07.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:07 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 347 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:44:08.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:08 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:08.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:08 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:44:08.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:08 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:08.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:08 smithi032 ceph-mon[28331]: pgmap v694: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 26 MiB/s wr, 5.32k op/s 2024-04-30T08:44:08.442 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:08 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:08.136+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:08.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:08 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:08.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:08 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:44:08.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:08 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:08.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:08 smithi071 ceph-mon[33385]: pgmap v694: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 26 MiB/s wr, 5.32k op/s 2024-04-30T08:44:08.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:08 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:08.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:08 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:44:08.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:08 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:08.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:08 smithi189 ceph-mon[33038]: pgmap v694: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 26 MiB/s wr, 5.32k op/s 2024-04-30T08:44:09.363 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:09 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:09.089+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:09.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:09 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:09.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:09 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:09.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:09 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:10.381 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:10 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:10.121+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:10.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:10 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:10.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:10 smithi032 ceph-mon[28331]: pgmap v695: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 21 MiB/s wr, 4.10k op/s 2024-04-30T08:44:10.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:10 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:10.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:10 smithi071 ceph-mon[33385]: pgmap v695: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 21 MiB/s wr, 4.10k op/s 2024-04-30T08:44:10.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:10 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:10.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:10 smithi189 ceph-mon[33038]: pgmap v695: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 21 MiB/s wr, 4.10k op/s 2024-04-30T08:44:11.383 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:11 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:11.086+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:11.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:11 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:11.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:11 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:44:11.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:11 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:11.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:11 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:11.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:11 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:44:11.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:11 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:11.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:11 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:11.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:11 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:44:11.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:11 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:12.387 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:12 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:12.093+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:12.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:12 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:12.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:12 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 354 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:44:12.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:12 smithi032 ceph-mon[28331]: pgmap v696: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 25 MiB/s wr, 5.06k op/s 2024-04-30T08:44:12.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:12 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:12.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:12 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 354 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:44:12.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:12 smithi071 ceph-mon[33385]: pgmap v696: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 25 MiB/s wr, 5.06k op/s 2024-04-30T08:44:12.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:12 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:12.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:12 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 354 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:44:12.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:12 smithi189 ceph-mon[33038]: pgmap v696: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 25 MiB/s wr, 5.06k op/s 2024-04-30T08:44:13.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:13 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:13.442 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:13 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:13.143+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:13.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:13 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:13.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:13 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:14.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:14 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:14.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:14 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:44:14.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:14 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:44:14.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:14 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:14.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:14 smithi032 ceph-mon[28331]: pgmap v697: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 20 MiB/s wr, 4.15k op/s 2024-04-30T08:44:14.445 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:14 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:14.172+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:14.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:14 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:14.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:14 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:44:14.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:14 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:44:14.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:14 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:14.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:14 smithi071 ceph-mon[33385]: pgmap v697: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 20 MiB/s wr, 4.15k op/s 2024-04-30T08:44:14.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:14 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:14.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:14 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:44:14.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:14 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:44:14.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:14 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:14.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:14 smithi189 ceph-mon[33038]: pgmap v697: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 20 MiB/s wr, 4.15k op/s 2024-04-30T08:44:15.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:15 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:15.693 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:15 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:15.215+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:15.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:15 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:15.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:15 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:44:15.886+0000 7f585b400640 1 -- 172.21.15.32:0/1198253074 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f5838005850 con 0x7f586410a3c0 2024-04-30T08:44:15.889 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:44:15.887+0000 7f5860c00640 1 -- 172.21.15.32:0/1198253074 <== mon.2 v2:172.21.15.71:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 580+0+0 (secure 0 0 0) 0x7f5838005850 con 0x7f586410a3c0 2024-04-30T08:44:16.112 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:44:16.110+0000 7f584cc00640 1 -- 172.21.15.32:0/2984557802 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- auth(proto 2 165 bytes epoch 0) -- 0x7f5830005850 con 0x556140f992f0 2024-04-30T08:44:16.112 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:44:16.111+0000 7f584e000640 1 -- 172.21.15.32:0/2984557802 <== mon.0 v2:172.21.15.32:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) ==== 409+0+0 (secure 0 0 0) 0x7f5830005850 con 0x556140f992f0 2024-04-30T08:44:16.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:16 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:16.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:16 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:44:16.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:16 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:16.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:16 smithi032 ceph-mon[28331]: pgmap v698: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 24 MiB/s wr, 4.92k op/s 2024-04-30T08:44:16.692 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:16 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:16.266+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:16.701 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:16 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:16.701 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:16 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:44:16.701 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:16 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:16.701 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:16 smithi189 ceph-mon[33038]: pgmap v698: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 24 MiB/s wr, 4.92k op/s 2024-04-30T08:44:16.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:16 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:16.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:16 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:44:16.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:16 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:16.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:16 smithi071 ceph-mon[33385]: pgmap v698: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 24 MiB/s wr, 4.92k op/s 2024-04-30T08:44:17.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:17 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:17.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:17 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 359 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:44:17.693 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:17 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:17.245+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:17.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:17 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:17.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:17 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 359 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:44:17.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:17 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:17.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:17 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 359 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:44:18.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:18 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:18.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:18 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:44:18.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:18 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:44:18.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:18 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:18.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:18 smithi032 ceph-mon[28331]: pgmap v699: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 21 MiB/s wr, 4.40k op/s 2024-04-30T08:44:18.692 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:18 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:18.245+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:18.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:18 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:18.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:18 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:44:18.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:18 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:44:18.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:18 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:18.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:18 smithi071 ceph-mon[33385]: pgmap v699: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 21 MiB/s wr, 4.40k op/s 2024-04-30T08:44:18.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:18 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:18.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:18 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:44:18.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:18 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:44:18.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:18 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:18.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:18 smithi189 ceph-mon[33038]: pgmap v699: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 21 MiB/s wr, 4.40k op/s 2024-04-30T08:44:19.691 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:19 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:19.228+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:20.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:19 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:20.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:19 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:20.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:19 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:20.691 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:20 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:20.218+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:21.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:20 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:21.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:20 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:44:21.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:20 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:21.193 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:20 smithi032 ceph-mon[28331]: pgmap v700: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 19 MiB/s wr, 3.90k op/s 2024-04-30T08:44:21.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:20 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:21.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:20 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:44:21.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:20 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:21.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:20 smithi071 ceph-mon[33385]: pgmap v700: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 19 MiB/s wr, 3.90k op/s 2024-04-30T08:44:21.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:20 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:21.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:20 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:44:21.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:20 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:21.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:20 smithi189 ceph-mon[33038]: pgmap v700: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 19 MiB/s wr, 3.90k op/s 2024-04-30T08:44:21.691 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:21 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:21.365+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:22.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:21 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:22.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:21 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:22.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:21 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:22.691 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:22 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:22.350+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:23.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:22 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:23.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:22 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:44:23.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:22 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:44:23.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:22 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:23.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:22 smithi032 ceph-mon[28331]: pgmap v701: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 20 MiB/s wr, 4.19k op/s 2024-04-30T08:44:23.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:22 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:23.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:22 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:44:23.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:22 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:44:23.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:22 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:23.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:22 smithi071 ceph-mon[33385]: pgmap v701: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 20 MiB/s wr, 4.19k op/s 2024-04-30T08:44:23.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:22 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:23.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:22 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:44:23.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:22 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:44:23.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:22 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:23.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:22 smithi189 ceph-mon[33038]: pgmap v701: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 20 MiB/s wr, 4.19k op/s 2024-04-30T08:44:23.691 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:23 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:23.353+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:24.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:23 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:24.192 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:23 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 364 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:44:24.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:23 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:24.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:23 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 364 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:44:24.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:23 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:24.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:23 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 364 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:44:24.691 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:24 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:24.341+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:25.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:24 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:25.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:24 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:44:25.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:24 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:25.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:24 smithi032 ceph-mon[28331]: pgmap v702: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 15 MiB/s wr, 3.12k op/s 2024-04-30T08:44:25.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:24 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:25.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:24 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:44:25.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:24 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:25.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:24 smithi071 ceph-mon[33385]: pgmap v702: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 15 MiB/s wr, 3.12k op/s 2024-04-30T08:44:25.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:24 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:25.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:24 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:44:25.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:24 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:25.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:24 smithi189 ceph-mon[33038]: pgmap v702: 129 pgs: 129 active+clean; 13 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 15 MiB/s wr, 3.12k op/s 2024-04-30T08:44:25.691 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:25 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:25.359+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:26.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:25 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:26.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:25 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:26.419 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:25 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:26.691 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:26 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:26.416+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:27.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:26 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:27.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:26 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:44:27.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:26 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:27.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:26 smithi071 ceph-mon[33385]: pgmap v703: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 19 MiB/s wr, 4.01k op/s 2024-04-30T08:44:27.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:26 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:27.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:26 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:44:27.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:26 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:27.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:26 smithi189 ceph-mon[33038]: pgmap v703: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 19 MiB/s wr, 4.01k op/s 2024-04-30T08:44:27.388 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:26 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:27.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:26 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:44:27.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:26 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:27.389 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:26 smithi032 ceph-mon[28331]: pgmap v703: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 19 MiB/s wr, 4.01k op/s 2024-04-30T08:44:27.389 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:27 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:27.386+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:28.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:27 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:28.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:27 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:28.370 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:27 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:28.691 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:28 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:28.368+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:29.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:28 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:29.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:28 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:44:29.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:28 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:29.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:28 smithi071 ceph-mon[33385]: pgmap v704: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 14 MiB/s wr, 2.91k op/s 2024-04-30T08:44:29.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:28 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:29.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:28 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:44:29.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:28 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:29.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:28 smithi189 ceph-mon[33038]: pgmap v704: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 14 MiB/s wr, 2.91k op/s 2024-04-30T08:44:29.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:28 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:29.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:28 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:44:29.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:28 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:29.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:28 smithi032 ceph-mon[28331]: pgmap v704: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 14 MiB/s wr, 2.91k op/s 2024-04-30T08:44:29.941 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:29 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:29.425+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:30.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:29 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:30.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:29 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:30.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:29 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:30.942 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:30 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:30.458+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:31.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:30 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:31.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:30 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:44:31.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:30 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:31.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:30 smithi071 ceph-mon[33385]: pgmap v705: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 13 MiB/s wr, 2.77k op/s 2024-04-30T08:44:31.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:30 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:31.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:30 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:44:31.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:30 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:31.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:30 smithi189 ceph-mon[33038]: pgmap v705: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 13 MiB/s wr, 2.77k op/s 2024-04-30T08:44:31.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:30 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:31.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:30 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:44:31.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:30 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:31.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:30 smithi032 ceph-mon[28331]: pgmap v705: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 13 MiB/s wr, 2.77k op/s 2024-04-30T08:44:31.941 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:31 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:31.445+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:32.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:31 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:32.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:31 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 369 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:44:32.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:31 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:32.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:31 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 369 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:44:32.406 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:31 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:32.407 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:31 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 369 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:44:32.407 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:32 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:32.404+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:33.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:33 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:33.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:33 smithi189 ceph-mon[33038]: pgmap v706: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 15 MiB/s wr, 2.93k op/s 2024-04-30T08:44:33.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:33 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:33.386 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:33 smithi032 ceph-mon[28331]: pgmap v706: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 15 MiB/s wr, 2.93k op/s 2024-04-30T08:44:33.387 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:33 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:33.382+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:33.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:33 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:33.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:33 smithi071 ceph-mon[33385]: pgmap v706: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 15 MiB/s wr, 2.93k op/s 2024-04-30T08:44:34.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:34 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:34.357 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:34 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:34.353+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:34.357 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:34 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:34.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:34 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:35.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:35 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:35.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:35 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:44:35.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:35 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:44:35.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:35 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:35.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:35 smithi189 ceph-mon[33038]: pgmap v707: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 11 MiB/s wr, 2.22k op/s 2024-04-30T08:44:35.335 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:35 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:35.335 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:35 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:44:35.335 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:35 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:44:35.335 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:35 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:35.335 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:35 smithi032 ceph-mon[28331]: pgmap v707: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 11 MiB/s wr, 2.22k op/s 2024-04-30T08:44:35.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:35 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:35.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:35 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:44:35.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:35 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:44:35.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:35 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:35.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:35 smithi071 ceph-mon[33385]: pgmap v707: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 11 MiB/s wr, 2.22k op/s 2024-04-30T08:44:35.691 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:35 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:35.333+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:36.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:36 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:36.333 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:36 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:36.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:36 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:36.691 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:36 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:36.332+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:37.298 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:37 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:37.298 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:37 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:44:37.298 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:37 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:37.298 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:37 smithi032 ceph-mon[28331]: pgmap v708: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 15 MiB/s wr, 2.87k op/s 2024-04-30T08:44:37.298 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:37 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 375 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:44:37.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:37 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:37.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:37 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:44:37.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:37 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:37.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:37 smithi189 ceph-mon[33038]: pgmap v708: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 15 MiB/s wr, 2.87k op/s 2024-04-30T08:44:37.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:37 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 375 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:44:37.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:37 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:37.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:37 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:44:37.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:37 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:37.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:37 smithi071 ceph-mon[33385]: pgmap v708: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 15 MiB/s wr, 2.87k op/s 2024-04-30T08:44:37.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:37 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 375 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:44:37.691 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:37 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:37.297+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:38.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:38 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:38.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:38 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:38.324 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:38 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:38.324 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:38 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:38.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:38 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:38.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:38 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:38.691 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:44:38 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:44:38.323+0000 7ff432400700 -1 osd.9 85 get_health_metrics reporting 1 slow ops, oldest is osd_op(mds.0.14:5083 2.1f 2.1ba6bc1f (undecoded) ondisk+write+known_if_redirected+full_force+supports_pool_eio e85) 2024-04-30T08:44:39.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:39 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:44:39.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:39 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:44:39.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:39 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:39.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:39 smithi032 ceph-mon[28331]: pgmap v709: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 8.9 MiB/s wr, 1.72k op/s 2024-04-30T08:44:39.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:39 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:44:39.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:39 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:44:39.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:39 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:39.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:39 smithi071 ceph-mon[33385]: pgmap v709: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 8.9 MiB/s wr, 1.72k op/s 2024-04-30T08:44:39.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:39 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:44:39.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:39 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:44:39.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:39 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:39.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:39 smithi189 ceph-mon[33038]: pgmap v709: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 8.9 MiB/s wr, 1.72k op/s 2024-04-30T08:44:40.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:44:40.370+0000 7f584e000640 1 -- 172.21.15.32:0/2984557802 <== mds.0 v2:172.21.15.71:6838/3517409993 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f5850038040 con 0x556140fc21b0 2024-04-30T08:44:40.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:44:40.370+0000 7f584e000640 10 client.16323 handle_command_reply: tid=0 2024-04-30T08:44:40.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:44:40.370+0000 7f58721ea740 1 client.16323 shutdown 2024-04-30T08:44:40.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:44:40.370+0000 7f58721ea740 1 -- 172.21.15.32:0/2984557802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556140f992f0 msgr2=0x556141373110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:44:40.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:44:40.370+0000 7f58721ea740 1 --2- 172.21.15.32:0/2984557802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556140f992f0 0x556141373110 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x5561413609f0 tx=0x7f5854013b10 comp rx=0 tx=0).stop 2024-04-30T08:44:40.371 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:44:40.370+0000 7f58721ea740 1 -- 172.21.15.32:0/2984557802 shutdown_connections 2024-04-30T08:44:40.372 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:44:40.370+0000 7f58721ea740 1 --2- 172.21.15.32:0/2984557802 >> [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] conn(0x556140fc21b0 0x5561413e64a0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:40.372 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:44:40.370+0000 7f58721ea740 1 --2- 172.21.15.32:0/2984557802 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x556140fa14d0 0x556141373ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:40.372 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:44:40.370+0000 7f58721ea740 1 --2- 172.21.15.32:0/2984557802 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x556140f992f0 0x556141373110 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:40.372 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:44:40.370+0000 7f58721ea740 1 --2- 172.21.15.32:0/2984557802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556140f80d80 0x556141372bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:40.372 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:44:40.370+0000 7f584e000640 0 client.16323 ms_handle_reset on v2:172.21.15.71:6838/3517409993 2024-04-30T08:44:40.372 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:44:40.370+0000 7f58721ea740 1 -- 172.21.15.32:0/2984557802 >> 172.21.15.32:0/2984557802 conn(0x556140fb6300 msgr2=0x55614135ec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:44:40.372 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:44:40.370+0000 7f58721ea740 1 -- 172.21.15.32:0/2984557802 shutdown_connections 2024-04-30T08:44:40.372 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:44:40.371+0000 7f58721ea740 1 -- 172.21.15.32:0/2984557802 wait complete. 2024-04-30T08:44:40.372 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:44:40.371+0000 7f58721ea740 20 client.16323 trim_cache size 0 max 16384 2024-04-30T08:44:40.372 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:{ 2024-04-30T08:44:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "message": "", 2024-04-30T08:44:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout: "return_code": 0 2024-04-30T08:44:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stdout:} 2024-04-30T08:44:40.373 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:44:40.371+0000 7f5869c00640 1 -- 172.21.15.32:0/1198253074 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f58480afc70 msgr2=0x7f58480b2180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:44:40.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:44:40.372+0000 7f5869c00640 1 --2- 172.21.15.32:0/1198253074 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f58480afc70 0x7f58480b2180 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f585c005fd0 tx=0x7f585c0095c0 comp rx=0 tx=0).stop 2024-04-30T08:44:40.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:44:40.372+0000 7f5869c00640 1 -- 172.21.15.32:0/1198253074 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f586410a3c0 msgr2=0x7f58641a8f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:44:40.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:44:40.372+0000 7f5869c00640 1 --2- 172.21.15.32:0/1198253074 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f586410a3c0 0x7f58641a8f30 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f5850000c00 tx=0x7f5850030d70 comp rx=0 tx=0).stop 2024-04-30T08:44:40.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:44:40.372+0000 7f5869c00640 1 -- 172.21.15.32:0/1198253074 shutdown_connections 2024-04-30T08:44:40.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:44:40.372+0000 7f5869c00640 1 --2- 172.21.15.32:0/1198253074 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f58480afc70 0x7f58480b2180 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:40.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:44:40.372+0000 7f5869c00640 1 --2- 172.21.15.32:0/1198253074 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f586410ff30 0x7f58641b0400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:40.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:44:40.372+0000 7f5869c00640 1 --2- 172.21.15.32:0/1198253074 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f586410b210 0x7f58641a9470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:40.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:44:40.372+0000 7f5869c00640 1 --2- 172.21.15.32:0/1198253074 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f586410a3c0 0x7f58641a8f30 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:40.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:44:40.372+0000 7f5869c00640 1 -- 172.21.15.32:0/1198253074 >> 172.21.15.32:0/1198253074 conn(0x7f58640703b0 msgr2=0x7f586410bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:44:40.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:44:40.372+0000 7f5869c00640 1 -- 172.21.15.32:0/1198253074 shutdown_connections 2024-04-30T08:44:40.374 INFO:teuthology.task.background_exec.ubuntu@smithi032.front.sepia.ceph.com.smithi032.stderr:2024-04-30T08:44:40.372+0000 7f5869c00640 1 -- 172.21.15.32:0/1198253074 wait complete. 2024-04-30T08:44:40.415 INFO:tasks.workunit.client.0.smithi032.stdout:Syncing()...76 sec 2024-04-30T08:44:40.415 INFO:tasks.workunit.client.0.smithi032.stdout:FFSB benchmark finished at: Tue Apr 30 08:44:40 2024 2024-04-30T08:44:40.415 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-04-30T08:44:40.415 INFO:tasks.workunit.client.0.smithi032.stdout:Results: 2024-04-30T08:44:40.415 INFO:tasks.workunit.client.0.smithi032.stdout:Benchmark took 388.23 sec 2024-04-30T08:44:40.415 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-04-30T08:44:40.415 INFO:tasks.workunit.client.0.smithi032.stdout:Total Results 2024-04-30T08:44:40.415 INFO:tasks.workunit.client.0.smithi032.stdout:=============== 2024-04-30T08:44:40.415 INFO:tasks.workunit.client.0.smithi032.stdout: Op Name Transactions Trans/sec % Trans % Op Weight Throughput 2024-04-30T08:44:40.415 INFO:tasks.workunit.client.0.smithi032.stdout: ======= ============ ========= ======= =========== ========== 2024-04-30T08:44:40.415 INFO:tasks.workunit.client.0.smithi032.stdout: write : 6320640 16280.65 100.000% 100.000% 63.6MB/sec 2024-04-30T08:44:40.415 INFO:tasks.workunit.client.0.smithi032.stdout:- 2024-04-30T08:44:40.415 INFO:tasks.workunit.client.0.smithi032.stdout:16280.65 Transactions per Second 2024-04-30T08:44:40.416 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-04-30T08:44:40.416 INFO:tasks.workunit.client.0.smithi032.stdout:Throughput Results 2024-04-30T08:44:40.416 INFO:tasks.workunit.client.0.smithi032.stdout:=================== 2024-04-30T08:44:40.416 INFO:tasks.workunit.client.0.smithi032.stdout:Write Throughput: 63.6MB/sec 2024-04-30T08:44:40.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:40 smithi032 ceph-mon[28331]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:40.481 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-04-30T08:44:40.481 INFO:tasks.workunit.client.0.smithi032.stdout:System Call Latency statistics in millisecs 2024-04-30T08:44:40.481 INFO:tasks.workunit.client.0.smithi032.stdout:===== 2024-04-30T08:44:40.481 INFO:tasks.workunit.client.0.smithi032.stdout: Min Avg Max Total Calls 2024-04-30T08:44:40.481 INFO:tasks.workunit.client.0.smithi032.stdout: ======== ======== ======== ============ 2024-04-30T08:44:40.481 INFO:tasks.workunit.client.0.smithi032.stdout:[ open] 0.018000 0.170730 39.096001 4938 2024-04-30T08:44:40.481 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-04-30T08:44:40.481 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-04-30T08:44:40.481 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-04-30T08:44:40.481 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-04-30T08:44:40.481 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-04-30T08:44:40.482 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-04-30T08:44:40.482 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2024-04-30T08:44:40.482 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[7] 20.000000 - 50.000000 : 4 2024-04-30T08:44:40.482 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[8] 10.000000 - 20.000000 : 9 2024-04-30T08:44:40.482 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[9] 5.000000 - 10.000000 : 19 2024-04-30T08:44:40.482 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[10] 2.000000 - 5.000000 : 49 2024-04-30T08:44:40.482 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[11] 1.000000 - 2.000000 : 19 2024-04-30T08:44:40.482 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[12] 0.500000 - 1.000000 : 25 2024-04-30T08:44:40.482 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[13] 0.200000 - 0.500000 : 55 2024-04-30T08:44:40.482 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[14] 0.100000 - 0.200000 : 346 2024-04-30T08:44:40.482 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[15] 0.050000 - 0.100000 : 569 2024-04-30T08:44:40.482 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[16] 0.020000 - 0.050000 : 3842 2024-04-30T08:44:40.482 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[17] 0.010000 - 0.020000 : 1 2024-04-30T08:44:40.483 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-04-30T08:44:40.483 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-04-30T08:44:40.483 INFO:tasks.workunit.client.0.smithi032.stdout:[ write] 0.003000 1.535953 27462.835938 6320640 2024-04-30T08:44:40.483 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[0] 5000.000000 - 10000.000000 : 316 2024-04-30T08:44:40.483 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[1] 2000.000000 - 5000.000000 : 229 2024-04-30T08:44:40.483 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[2] 1000.000000 - 2000.000000 : 124 2024-04-30T08:44:40.483 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[3] 500.000000 - 1000.000000 : 160 2024-04-30T08:44:40.483 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[4] 200.000000 - 500.000000 : 281 2024-04-30T08:44:40.483 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[5] 100.000000 - 200.000000 : 281 2024-04-30T08:44:40.483 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[6] 50.000000 - 100.000000 : 486 2024-04-30T08:44:40.483 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[7] 20.000000 - 50.000000 : 2231 2024-04-30T08:44:40.483 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[8] 10.000000 - 20.000000 : 2771 2024-04-30T08:44:40.483 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[9] 5.000000 - 10.000000 : 2563 2024-04-30T08:44:40.483 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[10] 2.000000 - 5.000000 : 4340 2024-04-30T08:44:40.484 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[11] 1.000000 - 2.000000 : 2492 2024-04-30T08:44:40.484 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[12] 0.500000 - 1.000000 : 2187 2024-04-30T08:44:40.484 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[13] 0.200000 - 0.500000 : 3811 2024-04-30T08:44:40.484 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[14] 0.100000 - 0.200000 : 8251 2024-04-30T08:44:40.484 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[15] 0.050000 - 0.100000 : 25745 2024-04-30T08:44:40.484 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[16] 0.020000 - 0.050000 : 210085 2024-04-30T08:44:40.484 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[17] 0.010000 - 0.020000 : 3135003 2024-04-30T08:44:40.484 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[18] 0.000000 - 0.010000 : 2918902 2024-04-30T08:44:40.484 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-04-30T08:44:40.484 INFO:tasks.workunit.client.0.smithi032.stdout:[ lseek] 0.000000 0.002748 66.509003 6320640 2024-04-30T08:44:40.485 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-04-30T08:44:40.485 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-04-30T08:44:40.485 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-04-30T08:44:40.485 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-04-30T08:44:40.485 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-04-30T08:44:40.485 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-04-30T08:44:40.485 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[6] 50.000000 - 100.000000 : 2 2024-04-30T08:44:40.485 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[7] 20.000000 - 50.000000 : 120 2024-04-30T08:44:40.485 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[8] 10.000000 - 20.000000 : 211 2024-04-30T08:44:40.485 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[9] 5.000000 - 10.000000 : 227 2024-04-30T08:44:40.485 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[10] 2.000000 - 5.000000 : 419 2024-04-30T08:44:40.485 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[11] 1.000000 - 2.000000 : 211 2024-04-30T08:44:40.485 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[12] 0.500000 - 1.000000 : 191 2024-04-30T08:44:40.485 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[13] 0.200000 - 0.500000 : 293 2024-04-30T08:44:40.486 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[14] 0.100000 - 0.200000 : 362 2024-04-30T08:44:40.486 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[15] 0.050000 - 0.100000 : 1217 2024-04-30T08:44:40.486 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[16] 0.020000 - 0.050000 : 5555 2024-04-30T08:44:40.486 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[17] 0.010000 - 0.020000 : 3722 2024-04-30T08:44:40.486 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[18] 0.000000 - 0.010000 : 6308110 2024-04-30T08:44:40.486 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-04-30T08:44:40.486 INFO:tasks.workunit.client.0.smithi032.stdout:[ close] 0.004000 0.015618 13.914000 4938 2024-04-30T08:44:40.486 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-04-30T08:44:40.486 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-04-30T08:44:40.486 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-04-30T08:44:40.486 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-04-30T08:44:40.487 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-04-30T08:44:40.487 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-04-30T08:44:40.487 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2024-04-30T08:44:40.487 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[7] 20.000000 - 50.000000 : 0 2024-04-30T08:44:40.487 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[8] 10.000000 - 20.000000 : 1 2024-04-30T08:44:40.487 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[9] 5.000000 - 10.000000 : 1 2024-04-30T08:44:40.487 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[10] 2.000000 - 5.000000 : 0 2024-04-30T08:44:40.487 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[11] 1.000000 - 2.000000 : 4 2024-04-30T08:44:40.487 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[12] 0.500000 - 1.000000 : 0 2024-04-30T08:44:40.487 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[13] 0.200000 - 0.500000 : 5 2024-04-30T08:44:40.487 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[14] 0.100000 - 0.200000 : 2 2024-04-30T08:44:40.487 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[15] 0.050000 - 0.100000 : 14 2024-04-30T08:44:40.487 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[16] 0.020000 - 0.050000 : 38 2024-04-30T08:44:40.487 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[17] 0.010000 - 0.020000 : 2852 2024-04-30T08:44:40.488 INFO:tasks.workunit.client.0.smithi032.stdout: msec_range[18] 0.000000 - 0.010000 : 2021 2024-04-30T08:44:40.488 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-04-30T08:44:40.488 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-04-30T08:44:40.488 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-04-30T08:44:40.488 INFO:tasks.workunit.client.0.smithi032.stdout: 2024-04-30T08:44:40.488 INFO:tasks.workunit.client.0.smithi032.stdout:1.2% User Time 2024-04-30T08:44:40.488 INFO:tasks.workunit.client.0.smithi032.stdout:18.8% System Time 2024-04-30T08:44:40.488 INFO:tasks.workunit.client.0.smithi032.stdout:20.0% CPU Utilization 2024-04-30T08:44:40.496 INFO:tasks.workunit.client.0.smithi032.stderr:+ cd .. 2024-04-30T08:44:40.497 INFO:tasks.workunit.client.0.smithi032.stderr:+ rm -r tmp ffsb 2024-04-30T08:44:40.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:40 smithi071 ceph-mon[33385]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:40.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:40 smithi189 ceph-mon[33038]: 1 slow requests (by type [ 'queued for pg' : 1 ] most affected pool [ 'cephfs_metadata' : 1 ]) 2024-04-30T08:44:41.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:41 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:44:41.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:41 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:41.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:41 smithi032 ceph-mon[28331]: pgmap v710: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8.9 MiB/s wr, 1.72k op/s 2024-04-30T08:44:41.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:41 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:44:41.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:41 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:44:41.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:41 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:41.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:41 smithi071 ceph-mon[33385]: pgmap v710: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8.9 MiB/s wr, 1.72k op/s 2024-04-30T08:44:41.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:41 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:44:41.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:41 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:44:41.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:41 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:41.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:41 smithi189 ceph-mon[33038]: pgmap v710: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 8.9 MiB/s wr, 1.72k op/s 2024-04-30T08:44:41.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:41 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:44:42.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:42 smithi032 ceph-mon[28331]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:44:42.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:42 smithi032 ceph-mon[28331]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:44:42.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:42 smithi032 ceph-mon[28331]: MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 382 secs 2024-04-30T08:44:42.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:42 smithi032 ceph-mon[28331]: Health check update: 1 slow ops, oldest one blocked for 380 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:44:42.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:42 smithi032 ceph-mon[28331]: Health check update: 2 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-04-30T08:44:42.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:42 smithi032 ceph-mon[28331]: pgmap v711: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 11 MiB/s wr, 2.18k op/s 2024-04-30T08:44:42.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:42 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:44:42.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:42 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:44:42.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:42 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:43.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:42 smithi071 ceph-mon[33385]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:44:43.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:42 smithi071 ceph-mon[33385]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:44:43.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:42 smithi071 ceph-mon[33385]: MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 382 secs 2024-04-30T08:44:43.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:42 smithi071 ceph-mon[33385]: Health check update: 1 slow ops, oldest one blocked for 380 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:44:43.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:42 smithi071 ceph-mon[33385]: Health check update: 2 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-04-30T08:44:43.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:42 smithi071 ceph-mon[33385]: pgmap v711: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 11 MiB/s wr, 2.18k op/s 2024-04-30T08:44:43.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:42 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:44:43.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:42 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:44:43.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:42 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:43.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:42 smithi189 ceph-mon[33038]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:44:43.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:42 smithi189 ceph-mon[33038]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:44:43.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:42 smithi189 ceph-mon[33038]: MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 382 secs 2024-04-30T08:44:43.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:42 smithi189 ceph-mon[33038]: Health check update: 1 slow ops, oldest one blocked for 380 sec, osd.9 has slow ops (SLOW_OPS) 2024-04-30T08:44:43.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:42 smithi189 ceph-mon[33038]: Health check update: 2 MDSs report slow metadata IOs (MDS_SLOW_METADATA_IO) 2024-04-30T08:44:43.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:42 smithi189 ceph-mon[33038]: pgmap v711: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 11 MiB/s wr, 2.18k op/s 2024-04-30T08:44:43.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:42 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:44:43.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:42 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6834/1558303056,v1:172.21.15.71:6835/1558303056] up:standby-replay 2024-04-30T08:44:43.051 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:42 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:43.852 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-04-30T08:44:43.853 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-04-30T08:44:44.104 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:43 smithi032 ceph-mon[28331]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:44:44.104 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:43 smithi032 ceph-mon[28331]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:44:44.105 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:43 smithi032 ceph-mon[28331]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:44:44.105 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:43 smithi032 ceph-mon[28331]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:44:44.105 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:43 smithi032 ceph-mon[28331]: Health check cleared: SLOW_OPS (was: 1 slow ops, oldest one blocked for 380 sec, osd.9 has slow ops) 2024-04-30T08:44:44.105 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:43 smithi032 ceph-mon[28331]: MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 382 secs 2024-04-30T08:44:44.145 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0... 2024-04-30T08:44:44.145 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-30T08:44:44.253 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:43 smithi071 ceph-mon[33385]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:44:44.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:43 smithi071 ceph-mon[33385]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:44:44.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:43 smithi071 ceph-mon[33385]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:44:44.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:43 smithi071 ceph-mon[33385]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:44:44.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:43 smithi071 ceph-mon[33385]: Health check cleared: SLOW_OPS (was: 1 slow ops, oldest one blocked for 380 sec, osd.9 has slow ops) 2024-04-30T08:44:44.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:43 smithi071 ceph-mon[33385]: MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 382 secs 2024-04-30T08:44:44.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:43 smithi189 ceph-mon[33038]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:44:44.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:43 smithi189 ceph-mon[33038]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:44:44.300 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:43 smithi189 ceph-mon[33038]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:44:44.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:43 smithi189 ceph-mon[33038]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:44:44.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:43 smithi189 ceph-mon[33038]: Health check cleared: SLOW_OPS (was: 1 slow ops, oldest one blocked for 380 sec, osd.9 has slow ops) 2024-04-30T08:44:44.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:43 smithi189 ceph-mon[33038]: MDS health message cleared (mds.?): 1 slow metadata IOs are blocked > 180 secs, oldest blocked for 382 secs 2024-04-30T08:44:45.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:45 smithi032 ceph-mon[28331]: pgmap v712: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 7.0 MiB/s wr, 1.39k op/s 2024-04-30T08:44:45.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:45 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:44:45.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:45 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:45.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:45 smithi071 ceph-mon[33385]: pgmap v712: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 7.0 MiB/s wr, 1.39k op/s 2024-04-30T08:44:45.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:45 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:44:45.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:45 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:45.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:45 smithi189 ceph-mon[33038]: pgmap v712: 129 pgs: 129 active+clean; 13 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 7.0 MiB/s wr, 1.39k op/s 2024-04-30T08:44:45.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:45 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/463203038,v1:172.21.15.189:6833/463203038] up:standby-replay 2024-04-30T08:44:45.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:45 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:46.171 DEBUG:teuthology.parallel:result is None 2024-04-30T08:44:46.172 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-04-30T08:44:46.294 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-04-30T08:44:46.294 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-04-30T08:44:46.304 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi032.front.sepia.ceph.com 2024-04-30T08:44:46.328 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-04-30T08:44:46.341 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-04-30T08:44:46.341 ERROR:teuthology.run_tasks:Manager failed: fwd_scrub Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_8feaf7d9864e980041b042fd5d964ac18adfd20f/qa/tasks/fwd_scrub.py", line 164, in task stop_all_fwd_scrubbers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/git.ceph.com_ceph-c_8feaf7d9864e980041b042fd5d964ac18adfd20f/qa/tasks/fwd_scrub.py", line 99, in stop_all_fwd_scrubbers raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}") RuntimeError: error during scrub thrashing: Command failed on smithi032 with status 124: 'sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls' 2024-04-30T08:44:46.342 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-30T08:44:46.357 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-30T08:44:46.445 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:46 smithi032 ceph-mon[28331]: MDS health message cleared (mds.?): 100+ slow metadata IOs are blocked > 180 secs, oldest blocked for 299 secs 2024-04-30T08:44:46.451 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:46 smithi189 ceph-mon[33038]: MDS health message cleared (mds.?): 100+ slow metadata IOs are blocked > 180 secs, oldest blocked for 299 secs 2024-04-30T08:44:46.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:46 smithi071 ceph-mon[33385]: MDS health message cleared (mds.?): 100+ slow metadata IOs are blocked > 180 secs, oldest blocked for 299 secs 2024-04-30T08:44:46.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.977+0000 7f0dddc00640 1 -- 172.21.15.32:0/2755934956 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0dd81040d0 msgr2=0x7f0dd81064f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:44:46.978 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.977+0000 7f0dddc00640 1 --2- 172.21.15.32:0/2755934956 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0dd81040d0 0x7f0dd81064f0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f0dd0009ca0 tx=0x7f0dd002a8c0 comp rx=0 tx=0).stop 2024-04-30T08:44:46.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.981+0000 7f0dddc00640 1 -- 172.21.15.32:0/2755934956 shutdown_connections 2024-04-30T08:44:46.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.981+0000 7f0dddc00640 1 --2- 172.21.15.32:0/2755934956 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dd8109630 0x7f0dd810bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:46.981 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.981+0000 7f0dddc00640 1 --2- 172.21.15.32:0/2755934956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dd8106ac0 0x7f0dd8108fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:46.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.981+0000 7f0dddc00640 1 --2- 172.21.15.32:0/2755934956 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0dd81040d0 0x7f0dd81064f0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:46.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.981+0000 7f0dddc00640 1 -- 172.21.15.32:0/2755934956 >> 172.21.15.32:0/2755934956 conn(0x7f0dd80fda70 msgr2=0x7f0dd80fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:44:46.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.981+0000 7f0dddc00640 1 -- 172.21.15.32:0/2755934956 shutdown_connections 2024-04-30T08:44:46.982 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.981+0000 7f0dddc00640 1 -- 172.21.15.32:0/2755934956 wait complete. 2024-04-30T08:44:46.983 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.983+0000 7f0dddc00640 1 Processor -- start 2024-04-30T08:44:46.984 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.983+0000 7f0dddc00640 1 -- start start 2024-04-30T08:44:46.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.983+0000 7f0dddc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dd81040d0 0x7f0dd819c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:44:46.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.983+0000 7f0dddc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0dd8106ac0 0x7f0dd819c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:44:46.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.983+0000 7f0dddc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dd8109630 0x7f0dd81a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:44:46.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.983+0000 7f0dddc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd810f4c0 con 0x7f0dd8109630 2024-04-30T08:44:46.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.984+0000 7f0dddc00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd810f340 con 0x7f0dd8106ac0 2024-04-30T08:44:46.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.984+0000 7f0dddc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0dd810f640 con 0x7f0dd81040d0 2024-04-30T08:44:46.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.984+0000 7f0dd7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dd81040d0 0x7f0dd819c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:44:46.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.984+0000 7f0dd7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dd81040d0 0x7f0dd819c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:43148/0 (socket says 172.21.15.32:43148) 2024-04-30T08:44:46.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.984+0000 7f0dd7e00640 1 -- 172.21.15.32:0/4006611954 learned_addr learned my addr 172.21.15.32:0/4006611954 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:44:46.985 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.984+0000 7f0ddd200640 1 --2- 172.21.15.32:0/4006611954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dd8109630 0x7f0dd81a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:44:46.986 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.984+0000 7f0dd7400640 1 --2- 172.21.15.32:0/4006611954 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0dd8106ac0 0x7f0dd819c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:44:46.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.985+0000 7f0dd7e00640 1 -- 172.21.15.32:0/4006611954 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0dd8106ac0 msgr2=0x7f0dd819c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:44:46.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.985+0000 7f0dd7e00640 1 --2- 172.21.15.32:0/4006611954 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0dd8106ac0 0x7f0dd819c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:46.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.985+0000 7f0dd7e00640 1 -- 172.21.15.32:0/4006611954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dd8109630 msgr2=0x7f0dd81a37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:44:46.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.985+0000 7f0dd7e00640 1 --2- 172.21.15.32:0/4006611954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dd8109630 0x7f0dd81a37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:46.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.985+0000 7f0dd7e00640 1 -- 172.21.15.32:0/4006611954 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0dd806c6c0 con 0x7f0dd81040d0 2024-04-30T08:44:46.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.985+0000 7f0ddd200640 1 --2- 172.21.15.32:0/4006611954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dd8109630 0x7f0dd81a37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:44:46.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.985+0000 7f0dd7e00640 1 --2- 172.21.15.32:0/4006611954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dd81040d0 0x7f0dd819c310 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f0dd0031b50 tx=0x7f0dd0031850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:44:46.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.987+0000 7f0dd4c00640 1 -- 172.21.15.32:0/4006611954 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd0004040 con 0x7f0dd81040d0 2024-04-30T08:44:46.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.987+0000 7f0dd4c00640 1 -- 172.21.15.32:0/4006611954 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0dd0004690 con 0x7f0dd81040d0 2024-04-30T08:44:46.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.987+0000 7f0dd4c00640 1 -- 172.21.15.32:0/4006611954 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0dd0004040 con 0x7f0dd81040d0 2024-04-30T08:44:46.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.987+0000 7f0dddc00640 1 -- 172.21.15.32:0/4006611954 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0dd8194ba0 con 0x7f0dd81040d0 2024-04-30T08:44:46.988 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.987+0000 7f0dddc00640 1 -- 172.21.15.32:0/4006611954 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0dd8195120 con 0x7f0dd81040d0 2024-04-30T08:44:46.991 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.991+0000 7f0dddc00640 1 -- 172.21.15.32:0/4006611954 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0dd8195440 con 0x7f0dd81040d0 2024-04-30T08:44:46.995 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.993+0000 7f0dd4c00640 1 -- 172.21.15.32:0/4006611954 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f0dd00041e0 con 0x7f0dd81040d0 2024-04-30T08:44:46.996 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.996+0000 7f0dd4c00640 1 --2- 172.21.15.32:0/4006611954 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f0dbc0afdd0 0x7f0dbc0b22e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:44:46.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.996+0000 7f0dd4c00640 1 -- 172.21.15.32:0/4006611954 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f0dd0109220 con 0x7f0dd81040d0 2024-04-30T08:44:46.997 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:46.997+0000 7f0dd7400640 1 --2- 172.21.15.32:0/4006611954 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f0dbc0afdd0 0x7f0dbc0b22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:44:47.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.000+0000 7f0dd7400640 1 --2- 172.21.15.32:0/4006611954 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f0dbc0afdd0 0x7f0dbc0b22e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0dd819d3b0 tx=0x7f0dc8009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:44:47.000 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.000+0000 7f0dd4c00640 1 -- 172.21.15.32:0/4006611954 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0dd00d19f0 con 0x7f0dd81040d0 2024-04-30T08:44:47.408 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.407+0000 7f0dddc00640 1 -- 172.21.15.32:0/4006611954 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0dd8195c60 con 0x7f0dd81040d0 2024-04-30T08:44:47.411 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.410+0000 7f0dd4c00640 1 -- 172.21.15.32:0/4006611954 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+24730 (secure 0 0 0) 0x7f0dd00d12f0 con 0x7f0dd81040d0 2024-04-30T08:44:47.412 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:44:47.412 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":85,"fsid":"98c0a670-06ca-11ef-bc93-c7b262605968","created":"2024-04-30T08:22:11.122875+0000","modified":"2024-04-30T08:33:31.872704+0000","last_up_change":"2024-04-30T08:30:50.350439+0000","last_in_change":"2024-04-30T08:30:30.210620+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-30T08:26:19.728331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-30T08:31:49.800542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-30T08:31:51.211320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-30T08:31:55.716625+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a7e060c8-1cbb-418f-9a03-06693f6b3f52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6803","nonce":2624700402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6805","nonce":2624700402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6809","nonce":2624700402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6807","nonce":2624700402}]},"public_addr":"172.21.15.32:6803/2624700402","cluster_addr":"172.21.15.32:6805/2624700402","heartbeat_back_addr":"172.21.15.32:6809/2624700402","heartbeat_front_addr":"172.21.15.32:6807/2624700402","state":["exists","up"]},{"osd":1,"uuid":"72ef2fb1-e5ca-48ef-aa3a-4796a64b922f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6801","nonce":444649711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6803","nonce":444649711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6807","nonce":444649711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6805","nonce":444649711}]},"public_addr":"172.21.15.71:6801/444649711","cluster_addr":"172.21.15.71:6803/444649711","heartbeat_back_addr":"172.21.15.71:6807/444649711","heartbeat_front_addr":"172.21.15.71:6805/444649711","state":["exists","up"]},{"osd":2,"uuid":"704e5e8a-935e-47c7-ae28-27119876ac09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6801","nonce":3255528671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6803","nonce":3255528671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6807","nonce":3255528671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6805","nonce":3255528671}]},"public_addr":"172.21.15.189:6801/3255528671","cluster_addr":"172.21.15.189:6803/3255528671","heartbeat_back_addr":"172.21.15.189:6807/3255528671","heartbeat_front_addr":"172.21.15.189:6805/3255528671","state":["exists","up"]},{"osd":3,"uuid":"575004f7-6559-4a62-863e-e33078b0eebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6811","nonce":1384702112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6813","nonce":1384702112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6817","nonce":1384702112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6815","nonce":1384702112}]},"public_addr":"172.21.15.32:6811/1384702112","cluster_addr":"172.21.15.32:6813/1384702112","heartbeat_back_addr":"172.21.15.32:6817/1384702112","heartbeat_front_addr":"172.21.15.32:6815/1384702112","state":["exists","up"]},{"osd":4,"uuid":"e1c1e894-074d-4c78-89ef-9e5a114268bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6809","nonce":4250534574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6811","nonce":4250534574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6815","nonce":4250534574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6813","nonce":4250534574}]},"public_addr":"172.21.15.71:6809/4250534574","cluster_addr":"172.21.15.71:6811/4250534574","heartbeat_back_addr":"172.21.15.71:6815/4250534574","heartbeat_front_addr":"172.21.15.71:6813/4250534574","state":["exists","up"]},{"osd":5,"uuid":"12c022d4-fac6-40c4-ae7a-691571bee5d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6809","nonce":4245346198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6811","nonce":4245346198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6815","nonce":4245346198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6813","nonce":4245346198}]},"public_addr":"172.21.15.189:6809/4245346198","cluster_addr":"172.21.15.189:6811/4245346198","heartbeat_back_addr":"172.21.15.189:6815/4245346198","heartbeat_front_addr":"172.21.15.189:6813/4245346198","state":["exists","up"]},{"osd":6,"uuid":"b413f8ee-3d4a-415a-a2ba-d68437b7330d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6819","nonce":1910389190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6821","nonce":1910389190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6825","nonce":1910389190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6823","nonce":1910389190}]},"public_addr":"172.21.15.32:6819/1910389190","cluster_addr":"172.21.15.32:6821/1910389190","heartbeat_back_addr":"172.21.15.32:6825/1910389190","heartbeat_front_addr":"172.21.15.32:6823/1910389190","state":["exists","up"]},{"osd":7,"uuid":"aac64fb3-fc13-4126-b842-7b06ad5ce238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6817","nonce":692607380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6819","nonce":692607380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6823","nonce":692607380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6821","nonce":692607380}]},"public_addr":"172.21.15.71:6817/692607380","cluster_addr":"172.21.15.71:6819/692607380","heartbeat_back_addr":"172.21.15.71:6823/692607380","heartbeat_front_addr":"172.21.15.71:6821/692607380","state":["exists","up"]},{"osd":8,"uuid":"cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6817","nonce":3973697069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6819","nonce":3973697069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6823","nonce":3973697069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6821","nonce":3973697069}]},"public_addr":"172.21.15.189:6817/3973697069","cluster_addr":"172.21.15.189:6819/3973697069","heartbeat_back_addr":"172.21.15.189:6823/3973697069","heartbeat_front_addr":"172.21.15.189:6821/3973697069","state":["exists","up"]},{"osd":9,"uuid":"ef4a1645-ae6e-45c3-82ad-161510e62aa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6827","nonce":46066685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6829","nonce":46066685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6833","nonce":46066685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6831","nonce":46066685}]},"public_addr":"172.21.15.32:6827/46066685","cluster_addr":"172.21.15.32:6829/46066685","heartbeat_back_addr":"172.21.15.32:6833/46066685","heartbeat_front_addr":"172.21.15.32:6831/46066685","state":["exists","up"]},{"osd":10,"uuid":"25b26a38-a644-409f-b941-59efdf04a3ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6825","nonce":1886290293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6827","nonce":1886290293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6831","nonce":1886290293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6829","nonce":1886290293}]},"public_addr":"172.21.15.71:6825/1886290293","cluster_addr":"172.21.15.71:6827/1886290293","heartbeat_back_addr":"172.21.15.71:6831/1886290293","heartbeat_front_addr":"172.21.15.71:6829/1886290293","state":["exists","up"]},{"osd":11,"uuid":"ab96b3ff-3688-455b-af64-60226c5bfbff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6825","nonce":850185001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6827","nonce":850185001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6831","nonce":850185001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6829","nonce":850185001}]},"public_addr":"172.21.15.189:6825/850185001","cluster_addr":"172.21.15.189:6827/850185001","heartbeat_back_addr":"172.21.15.189:6831/850185001","heartbeat_front_addr":"172.21.15.189:6829/850185001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:27.220308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:53.230783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:17.190642+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:52.603799+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:20.568671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:46.158363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:22.706933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:50.665173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:16.602410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:53.814698+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:22.435189+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:48.651949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.e","mappings":[{"from":4,"to":7}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.28","mappings":[{"from":3,"to":6},{"from":5,"to":8}]},{"pgid":"3.30","mappings":[{"from":1,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1266897222":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/2915689533":"2024-05-01T08:23:29.526664+0000","172.21.15.32:6801/4019351306":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/425836193":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/1134075722":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6801/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/21059253":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/3245550894":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/4019351306":"2024-05-01T08:23:29.526664+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-30T08:44:47.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.417+0000 7f0dddc00640 1 -- 172.21.15.32:0/4006611954 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f0dbc0afdd0 msgr2=0x7f0dbc0b22e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:44:47.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.417+0000 7f0dddc00640 1 --2- 172.21.15.32:0/4006611954 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f0dbc0afdd0 0x7f0dbc0b22e0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f0dd819d3b0 tx=0x7f0dc8009110 comp rx=0 tx=0).stop 2024-04-30T08:44:47.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.417+0000 7f0dddc00640 1 -- 172.21.15.32:0/4006611954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dd81040d0 msgr2=0x7f0dd819c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:44:47.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.417+0000 7f0dddc00640 1 --2- 172.21.15.32:0/4006611954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dd81040d0 0x7f0dd819c310 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f0dd0031b50 tx=0x7f0dd0031850 comp rx=0 tx=0).stop 2024-04-30T08:44:47.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.417+0000 7f0dddc00640 1 -- 172.21.15.32:0/4006611954 shutdown_connections 2024-04-30T08:44:47.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.417+0000 7f0dddc00640 1 --2- 172.21.15.32:0/4006611954 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f0dbc0afdd0 0x7f0dbc0b22e0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:47.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.417+0000 7f0dddc00640 1 --2- 172.21.15.32:0/4006611954 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0dd8109630 0x7f0dd81a37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:47.418 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.417+0000 7f0dddc00640 1 --2- 172.21.15.32:0/4006611954 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0dd8106ac0 0x7f0dd819c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:47.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.417+0000 7f0dddc00640 1 --2- 172.21.15.32:0/4006611954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0dd81040d0 0x7f0dd819c310 secure :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f0dd0031b50 tx=0x7f0dd0031850 comp rx=0 tx=0).stop 2024-04-30T08:44:47.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.417+0000 7f0dddc00640 1 -- 172.21.15.32:0/4006611954 >> 172.21.15.32:0/4006611954 conn(0x7f0dd80fda70 msgr2=0x7f0dd8101ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:44:47.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.418+0000 7f0dddc00640 1 -- 172.21.15.32:0/4006611954 shutdown_connections 2024-04-30T08:44:47.419 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.419+0000 7f0dddc00640 1 -- 172.21.15.32:0/4006611954 wait complete. 2024-04-30T08:44:47.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:47 smithi032 ceph-mon[28331]: pgmap v713: 129 pgs: 129 active+clean; 12 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 7.0 MiB/s wr, 1.41k op/s 2024-04-30T08:44:47.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:47 smithi032 ceph-mon[28331]: Health check cleared: MDS_SLOW_METADATA_IO (was: 1 MDSs report slow metadata IOs) 2024-04-30T08:44:47.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:47 smithi032 ceph-mon[28331]: Cluster is now healthy 2024-04-30T08:44:47.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:47 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:44:47.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:47 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:47.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:47 smithi032 ceph-mon[28331]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:44:47.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:47 smithi032 ceph-mon[28331]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:44:47.470 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-30T08:26:19.728331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-30T08:31:49.800542+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-30T08:31:51.211320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-30T08:31:55.716625+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-30T08:44:47.470 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-30T08:44:47.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:47 smithi071 ceph-mon[33385]: pgmap v713: 129 pgs: 129 active+clean; 12 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 7.0 MiB/s wr, 1.41k op/s 2024-04-30T08:44:47.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:47 smithi071 ceph-mon[33385]: Health check cleared: MDS_SLOW_METADATA_IO (was: 1 MDSs report slow metadata IOs) 2024-04-30T08:44:47.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:47 smithi071 ceph-mon[33385]: Cluster is now healthy 2024-04-30T08:44:47.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:47 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:44:47.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:47 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:47.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:47 smithi071 ceph-mon[33385]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:44:47.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:47 smithi071 ceph-mon[33385]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:44:47.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:47 smithi189 ceph-mon[33038]: pgmap v713: 129 pgs: 129 active+clean; 12 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 7.0 MiB/s wr, 1.41k op/s 2024-04-30T08:44:47.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:47 smithi189 ceph-mon[33038]: Health check cleared: MDS_SLOW_METADATA_IO (was: 1 MDSs report slow metadata IOs) 2024-04-30T08:44:47.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:47 smithi189 ceph-mon[33038]: Cluster is now healthy 2024-04-30T08:44:47.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:47 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6838/3517409993,v1:172.21.15.71:6839/3517409993] up:active 2024-04-30T08:44:47.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:47 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:47.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:47 smithi189 ceph-mon[33038]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:44:47.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:47 smithi189 ceph-mon[33038]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:44:47.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.940+0000 7f7ee6800640 1 -- 172.21.15.32:0/1045871766 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ee01042d0 msgr2=0x7f7ee01066f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:44:47.942 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.940+0000 7f7ee6800640 1 --2- 172.21.15.32:0/1045871766 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ee01042d0 0x7f7ee01066f0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8009be0 tx=0x7f7ed802a6c0 comp rx=0 tx=0).stop 2024-04-30T08:44:47.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.940+0000 7f7ee6800640 1 -- 172.21.15.32:0/1045871766 shutdown_connections 2024-04-30T08:44:47.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.940+0000 7f7ee6800640 1 --2- 172.21.15.32:0/1045871766 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ee01097f0 0x7f7ee010bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:47.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.940+0000 7f7ee6800640 1 --2- 172.21.15.32:0/1045871766 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ee0106cc0 0x7f7ee0109160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:47.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.940+0000 7f7ee6800640 1 --2- 172.21.15.32:0/1045871766 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ee01042d0 0x7f7ee01066f0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:47.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.940+0000 7f7ee6800640 1 -- 172.21.15.32:0/1045871766 >> 172.21.15.32:0/1045871766 conn(0x7f7ee00fdc50 msgr2=0x7f7ee0100100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:44:47.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.940+0000 7f7ee6800640 1 -- 172.21.15.32:0/1045871766 shutdown_connections 2024-04-30T08:44:47.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.940+0000 7f7ee6800640 1 -- 172.21.15.32:0/1045871766 wait complete. 2024-04-30T08:44:47.943 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.942+0000 7f7ee6800640 1 Processor -- start 2024-04-30T08:44:47.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.942+0000 7f7ee6800640 1 -- start start 2024-04-30T08:44:47.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.943+0000 7f7ee6800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ee01042d0 0x7f7ee00743e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:44:47.944 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.943+0000 7f7ee6800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ee0106cc0 0x7f7ee0070f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:44:47.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.943+0000 7f7ee6800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ee01097f0 0x7f7ee00714c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:44:47.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.943+0000 7f7ee6800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee006cea0 con 0x7f7ee01097f0 2024-04-30T08:44:47.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.943+0000 7f7ee6800640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee006cd20 con 0x7f7ee01042d0 2024-04-30T08:44:47.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.943+0000 7f7ee6800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee006d020 con 0x7f7ee0106cc0 2024-04-30T08:44:47.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.944+0000 7f7edea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ee0106cc0 0x7f7ee0070f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:44:47.945 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.944+0000 7f7edea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ee0106cc0 0x7f7ee0070f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:43156/0 (socket says 172.21.15.32:43156) 2024-04-30T08:44:47.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.944+0000 7f7edea00640 1 -- 172.21.15.32:0/883603309 learned_addr learned my addr 172.21.15.32:0/883603309 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:44:47.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.944+0000 7f7edfe00640 1 --2- 172.21.15.32:0/883603309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ee01097f0 0x7f7ee00714c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:44:47.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.945+0000 7f7edf400640 1 --2- 172.21.15.32:0/883603309 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ee01042d0 0x7f7ee00743e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:44:47.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.945+0000 7f7edf400640 1 -- 172.21.15.32:0/883603309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ee0106cc0 msgr2=0x7f7ee0070f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:44:47.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.945+0000 7f7edf400640 1 --2- 172.21.15.32:0/883603309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ee0106cc0 0x7f7ee0070f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:47.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.945+0000 7f7edf400640 1 -- 172.21.15.32:0/883603309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ee01097f0 msgr2=0x7f7ee00714c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:44:47.946 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.945+0000 7f7edf400640 1 --2- 172.21.15.32:0/883603309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ee01097f0 0x7f7ee00714c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:47.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.945+0000 7f7edf400640 1 -- 172.21.15.32:0/883603309 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ee0074a80 con 0x7f7ee01042d0 2024-04-30T08:44:47.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.945+0000 7f7edfe00640 1 --2- 172.21.15.32:0/883603309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ee01097f0 0x7f7ee00714c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:44:47.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.946+0000 7f7edf400640 1 --2- 172.21.15.32:0/883603309 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ee01042d0 0x7f7ee00743e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8009b30 tx=0x7f7ed8030d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:44:47.947 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.947+0000 7f7ed7e00640 1 -- 172.21.15.32:0/883603309 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8041020 con 0x7f7ee01042d0 2024-04-30T08:44:47.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.947+0000 7f7ed7e00640 1 -- 172.21.15.32:0/883603309 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f7ed80318f0 con 0x7f7ee01042d0 2024-04-30T08:44:47.950 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.947+0000 7f7ed7e00640 1 -- 172.21.15.32:0/883603309 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed8036040 con 0x7f7ee01042d0 2024-04-30T08:44:47.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.947+0000 7f7ee6800640 1 -- 172.21.15.32:0/883603309 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ee01b1320 con 0x7f7ee01042d0 2024-04-30T08:44:47.951 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.947+0000 7f7ee6800640 1 -- 172.21.15.32:0/883603309 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ee01b16d0 con 0x7f7ee01042d0 2024-04-30T08:44:47.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.952+0000 7f7ee6800640 1 -- 172.21.15.32:0/883603309 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ee006c320 con 0x7f7ee01042d0 2024-04-30T08:44:47.952 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.952+0000 7f7ed7e00640 1 -- 172.21.15.32:0/883603309 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f7ed8044020 con 0x7f7ee01042d0 2024-04-30T08:44:47.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.954+0000 7f7ed7e00640 1 --2- 172.21.15.32:0/883603309 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f7ec40afc40 0x7f7ec40b2150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:44:47.954 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.954+0000 7f7edea00640 1 --2- 172.21.15.32:0/883603309 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f7ec40afc40 0x7f7ec40b2150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:44:47.955 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.955+0000 7f7ed7e00640 1 -- 172.21.15.32:0/883603309 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f7ed80ffc10 con 0x7f7ee01042d0 2024-04-30T08:44:47.956 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.955+0000 7f7edea00640 1 --2- 172.21.15.32:0/883603309 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f7ec40afc40 0x7f7ec40b2150 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8005fd0 tx=0x7f7ec800b2c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:44:47.960 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:47.960+0000 7f7ed7e00640 1 -- 172.21.15.32:0/883603309 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7ed804b020 con 0x7f7ee01042d0 2024-04-30T08:44:48.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.348+0000 7f7ee6800640 1 -- 172.21.15.32:0/883603309 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7ee01b1b20 con 0x7f7ee01042d0 2024-04-30T08:44:48.349 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.349+0000 7f7ed7e00640 1 -- 172.21.15.32:0/883603309 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) ==== 93+0+10 (secure 0 0 0) 0x7f7ed80c8420 con 0x7f7ee01042d0 2024-04-30T08:44:48.350 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-30T08:44:48.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.355+0000 7f7ee6800640 1 -- 172.21.15.32:0/883603309 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f7ec40afc40 msgr2=0x7f7ec40b2150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:44:48.356 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.355+0000 7f7ee6800640 1 --2- 172.21.15.32:0/883603309 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f7ec40afc40 0x7f7ec40b2150 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f7ec8005fd0 tx=0x7f7ec800b2c0 comp rx=0 tx=0).stop 2024-04-30T08:44:48.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.355+0000 7f7ee6800640 1 -- 172.21.15.32:0/883603309 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ee01042d0 msgr2=0x7f7ee00743e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:44:48.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.355+0000 7f7ee6800640 1 --2- 172.21.15.32:0/883603309 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ee01042d0 0x7f7ee00743e0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8009b30 tx=0x7f7ed8030d10 comp rx=0 tx=0).stop 2024-04-30T08:44:48.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.355+0000 7f7ee6800640 1 -- 172.21.15.32:0/883603309 shutdown_connections 2024-04-30T08:44:48.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.355+0000 7f7ee6800640 1 --2- 172.21.15.32:0/883603309 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f7ec40afc40 0x7f7ec40b2150 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:48.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.355+0000 7f7ee6800640 1 --2- 172.21.15.32:0/883603309 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f7ee01097f0 0x7f7ee00714c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:48.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.355+0000 7f7ee6800640 1 --2- 172.21.15.32:0/883603309 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7ee0106cc0 0x7f7ee0070f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:48.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.356+0000 7f7ee6800640 1 --2- 172.21.15.32:0/883603309 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f7ee01042d0 0x7f7ee00743e0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:48.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.356+0000 7f7ee6800640 1 -- 172.21.15.32:0/883603309 >> 172.21.15.32:0/883603309 conn(0x7f7ee00fdc50 msgr2=0x7f7ee01022c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:44:48.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.356+0000 7f7ee6800640 1 -- 172.21.15.32:0/883603309 shutdown_connections 2024-04-30T08:44:48.357 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.356+0000 7f7ee6800640 1 -- 172.21.15.32:0/883603309 wait complete. 2024-04-30T08:44:48.401 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:48 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/4006611954' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:44:48.403 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-30T08:44:48.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:48 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/4006611954' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:44:48.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:48 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/4006611954' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:44:48.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.801+0000 7f3744400640 1 -- 172.21.15.32:0/514761076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f373c1102c0 msgr2=0x7f373c1126e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:44:48.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.801+0000 7f3744400640 1 --2- 172.21.15.32:0/514761076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f373c1102c0 0x7f373c1126e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f3734009be0 tx=0x7f373402a6c0 comp rx=0 tx=0).stop 2024-04-30T08:44:48.802 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.801+0000 7f3744400640 1 -- 172.21.15.32:0/514761076 shutdown_connections 2024-04-30T08:44:48.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.801+0000 7f3744400640 1 --2- 172.21.15.32:0/514761076 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f373c1102c0 0x7f373c1126e0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:48.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.801+0000 7f3744400640 1 --2- 172.21.15.32:0/514761076 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f373c075960 0x7f373c10fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:48.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.801+0000 7f3744400640 1 --2- 172.21.15.32:0/514761076 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f373c075010 0x7f373c075420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:48.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.801+0000 7f3744400640 1 -- 172.21.15.32:0/514761076 >> 172.21.15.32:0/514761076 conn(0x7f373c0fed50 msgr2=0x7f373c101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:44:48.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.801+0000 7f3744400640 1 -- 172.21.15.32:0/514761076 shutdown_connections 2024-04-30T08:44:48.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.801+0000 7f3744400640 1 -- 172.21.15.32:0/514761076 wait complete. 2024-04-30T08:44:48.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.803+0000 7f3744400640 1 Processor -- start 2024-04-30T08:44:48.803 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.803+0000 7f3744400640 1 -- start start 2024-04-30T08:44:48.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.803+0000 7f3744400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f373c075010 0x7f373c1a0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:44:48.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.803+0000 7f3744400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f373c075960 0x7f373c1a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:44:48.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.803+0000 7f3744400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f373c1102c0 0x7f373c1a7d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:44:48.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.803+0000 7f3744400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f373c115ca0 con 0x7f373c075010 2024-04-30T08:44:48.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.803+0000 7f3744400640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f373c115b20 con 0x7f373c075960 2024-04-30T08:44:48.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.803+0000 7f3744400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f373c115e20 con 0x7f373c1102c0 2024-04-30T08:44:48.804 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.804+0000 7f3741000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f373c075960 0x7f373c1a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:44:48.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.804+0000 7f3741000640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f373c075960 0x7f373c1a0da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:48170/0 (socket says 172.21.15.32:48170) 2024-04-30T08:44:48.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.804+0000 7f3741000640 1 -- 172.21.15.32:0/2519495127 learned_addr learned my addr 172.21.15.32:0/2519495127 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:44:48.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.804+0000 7f3742400640 1 --2- 172.21.15.32:0/2519495127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f373c1102c0 0x7f373c1a7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:44:48.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.804+0000 7f3742400640 1 -- 172.21.15.32:0/2519495127 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f373c075960 msgr2=0x7f373c1a0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:44:48.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.804+0000 7f3742400640 1 --2- 172.21.15.32:0/2519495127 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f373c075960 0x7f373c1a0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:48.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.804+0000 7f3742400640 1 -- 172.21.15.32:0/2519495127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f373c075010 msgr2=0x7f373c1a0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:44:48.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.804+0000 7f3742400640 1 --2- 172.21.15.32:0/2519495127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f373c075010 0x7f373c1a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:48.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.804+0000 7f3742400640 1 -- 172.21.15.32:0/2519495127 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f373c1a83a0 con 0x7f373c1102c0 2024-04-30T08:44:48.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.805+0000 7f3742400640 1 --2- 172.21.15.32:0/2519495127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f373c1102c0 0x7f373c1a7d30 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f3734034040 tx=0x7f3734030d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:44:48.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.805+0000 7f373a000640 1 -- 172.21.15.32:0/2519495127 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3734041020 con 0x7f373c1102c0 2024-04-30T08:44:48.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.805+0000 7f3744400640 1 -- 172.21.15.32:0/2519495127 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f373c1a8630 con 0x7f373c1102c0 2024-04-30T08:44:48.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.805+0000 7f373a000640 1 -- 172.21.15.32:0/2519495127 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f3734030ec0 con 0x7f373c1102c0 2024-04-30T08:44:48.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.805+0000 7f3744400640 1 -- 172.21.15.32:0/2519495127 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f373c1a8b70 con 0x7f373c1102c0 2024-04-30T08:44:48.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.806+0000 7f373a000640 1 -- 172.21.15.32:0/2519495127 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3734036040 con 0x7f373c1102c0 2024-04-30T08:44:48.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.807+0000 7f3744400640 1 -- 172.21.15.32:0/2519495127 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f371c0052c0 con 0x7f373c1102c0 2024-04-30T08:44:48.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.810+0000 7f373a000640 1 -- 172.21.15.32:0/2519495127 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f3734044020 con 0x7f373c1102c0 2024-04-30T08:44:48.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.811+0000 7f373a000640 1 --2- 172.21.15.32:0/2519495127 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f37240afbb0 0x7f37240b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:44:48.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.811+0000 7f373a000640 1 -- 172.21.15.32:0/2519495127 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f37340ffc20 con 0x7f373c1102c0 2024-04-30T08:44:48.812 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.812+0000 7f3741a00640 1 --2- 172.21.15.32:0/2519495127 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f37240afbb0 0x7f37240b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:44:48.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.816+0000 7f373a000640 1 -- 172.21.15.32:0/2519495127 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f373404b070 con 0x7f373c1102c0 2024-04-30T08:44:48.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:48.816+0000 7f3741a00640 1 --2- 172.21.15.32:0/2519495127 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f37240afbb0 0x7f37240b20c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f37300058c0 tx=0x7f3730005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:44:49.199 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.197+0000 7f3744400640 1 -- 172.21.15.32:0/2519495127 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f371c005880 con 0x7f373c1102c0 2024-04-30T08:44:49.199 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-30T08:44:49.200 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.198+0000 7f373a000640 1 -- 172.21.15.32:0/2519495127 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) ==== 104+0+11 (secure 0 0 0) 0x7f37340c83f0 con 0x7f373c1102c0 2024-04-30T08:44:49.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.201+0000 7f3744400640 1 -- 172.21.15.32:0/2519495127 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f37240afbb0 msgr2=0x7f37240b20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:44:49.201 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.201+0000 7f3744400640 1 --2- 172.21.15.32:0/2519495127 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f37240afbb0 0x7f37240b20c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f37300058c0 tx=0x7f3730005850 comp rx=0 tx=0).stop 2024-04-30T08:44:49.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.201+0000 7f3744400640 1 -- 172.21.15.32:0/2519495127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f373c1102c0 msgr2=0x7f373c1a7d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:44:49.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.201+0000 7f3744400640 1 --2- 172.21.15.32:0/2519495127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f373c1102c0 0x7f373c1a7d30 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f3734034040 tx=0x7f3734030d10 comp rx=0 tx=0).stop 2024-04-30T08:44:49.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.201+0000 7f3744400640 1 -- 172.21.15.32:0/2519495127 shutdown_connections 2024-04-30T08:44:49.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.201+0000 7f3744400640 1 --2- 172.21.15.32:0/2519495127 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f37240afbb0 0x7f37240b20c0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:49.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.201+0000 7f3744400640 1 --2- 172.21.15.32:0/2519495127 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f373c1102c0 0x7f373c1a7d30 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:49.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.201+0000 7f3744400640 1 --2- 172.21.15.32:0/2519495127 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f373c075960 0x7f373c1a0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:49.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.201+0000 7f3744400640 1 --2- 172.21.15.32:0/2519495127 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f373c075010 0x7f373c1a0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:49.202 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.201+0000 7f3744400640 1 -- 172.21.15.32:0/2519495127 >> 172.21.15.32:0/2519495127 conn(0x7f373c0fed50 msgr2=0x7f373c103120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:44:49.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.202+0000 7f3744400640 1 -- 172.21.15.32:0/2519495127 shutdown_connections 2024-04-30T08:44:49.203 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.202+0000 7f3744400640 1 -- 172.21.15.32:0/2519495127 wait complete. 2024-04-30T08:44:49.239 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-30T08:44:49.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:49 smithi032 ceph-mon[28331]: pgmap v714: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 623 op/s 2024-04-30T08:44:49.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:49 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:49.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:49 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:44:49.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:49 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:44:49.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:49 smithi032 ceph-mon[28331]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:44:49.443 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:49 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/883603309' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:44:49.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:49 smithi071 ceph-mon[33385]: pgmap v714: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 623 op/s 2024-04-30T08:44:49.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:49 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:49.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:49 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:44:49.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:49 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:44:49.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:49 smithi071 ceph-mon[33385]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:44:49.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:49 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/883603309' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:44:49.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:49 smithi189 ceph-mon[33038]: pgmap v714: 129 pgs: 129 active+clean; 11 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 623 op/s 2024-04-30T08:44:49.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:49 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:49.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:49 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:44:49.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:49 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:44:49.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:49 smithi189 ceph-mon[33038]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:44:49.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:49 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/883603309' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:44:49.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.646+0000 7f5ccdc00640 1 -- 172.21.15.32:0/59250458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5cc8104da0 msgr2=0x7f5cc81051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:44:49.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.646+0000 7f5ccdc00640 1 --2- 172.21.15.32:0/59250458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5cc8104da0 0x7f5cc81051b0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f5cc0009ca0 tx=0x7f5cc002a8c0 comp rx=0 tx=0).stop 2024-04-30T08:44:49.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.647+0000 7f5ccdc00640 1 -- 172.21.15.32:0/59250458 shutdown_connections 2024-04-30T08:44:49.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.647+0000 7f5ccdc00640 1 --2- 172.21.15.32:0/59250458 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cc8109aa0 0x7f5cc810bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:49.647 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.647+0000 7f5ccdc00640 1 --2- 172.21.15.32:0/59250458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5cc8104da0 0x7f5cc81051b0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:49.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.647+0000 7f5ccdc00640 1 --2- 172.21.15.32:0/59250458 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cc8103b60 0x7f5cc8104860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:49.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.647+0000 7f5ccdc00640 1 -- 172.21.15.32:0/59250458 >> 172.21.15.32:0/59250458 conn(0x7f5cc80fed50 msgr2=0x7f5cc8101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:44:49.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.647+0000 7f5ccdc00640 1 -- 172.21.15.32:0/59250458 shutdown_connections 2024-04-30T08:44:49.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.647+0000 7f5ccdc00640 1 -- 172.21.15.32:0/59250458 wait complete. 2024-04-30T08:44:49.650 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.649+0000 7f5ccdc00640 1 Processor -- start 2024-04-30T08:44:49.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.654+0000 7f5ccdc00640 1 -- start start 2024-04-30T08:44:49.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.654+0000 7f5ccdc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5cc8103b60 0x7f5cc819c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:44:49.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.654+0000 7f5ccdc00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cc8104da0 0x7f5cc819c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:44:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.654+0000 7f5ccdc00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cc8109aa0 0x7f5cc81a3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:44:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.654+0000 7f5ccdc00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc810f6b0 con 0x7f5cc8104da0 2024-04-30T08:44:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.654+0000 7f5ccdc00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc810f530 con 0x7f5cc8109aa0 2024-04-30T08:44:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.654+0000 7f5ccdc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc810f830 con 0x7f5cc8103b60 2024-04-30T08:44:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.655+0000 7f5cc7400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cc8109aa0 0x7f5cc81a3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:44:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.655+0000 7f5cc6000640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cc8104da0 0x7f5cc819c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:44:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.655+0000 7f5cc7400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cc8109aa0 0x7f5cc81a3950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:48196/0 (socket says 172.21.15.32:48196) 2024-04-30T08:44:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.655+0000 7f5cc7400640 1 -- 172.21.15.32:0/344090399 learned_addr learned my addr 172.21.15.32:0/344090399 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:44:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.655+0000 7f5cc7400640 1 -- 172.21.15.32:0/344090399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5cc8103b60 msgr2=0x7f5cc819c480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-30T08:44:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.655+0000 7f5cc7400640 1 --2- 172.21.15.32:0/344090399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5cc8103b60 0x7f5cc819c480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.655+0000 7f5cc7400640 1 -- 172.21.15.32:0/344090399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cc8104da0 msgr2=0x7f5cc819c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:44:49.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.655+0000 7f5cc7400640 1 --2- 172.21.15.32:0/344090399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cc8104da0 0x7f5cc819c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:49.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.655+0000 7f5cc7400640 1 -- 172.21.15.32:0/344090399 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cc81a3fc0 con 0x7f5cc8109aa0 2024-04-30T08:44:49.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.656+0000 7f5cc7400640 1 --2- 172.21.15.32:0/344090399 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cc8109aa0 0x7f5cc81a3950 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f5cb800e9b0 tx=0x7f5cb800ee70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:44:49.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.660+0000 7f5cbf400640 1 -- 172.21.15.32:0/344090399 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb800cd80 con 0x7f5cc8109aa0 2024-04-30T08:44:49.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.660+0000 7f5cbf400640 1 -- 172.21.15.32:0/344090399 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5cb80035f0 con 0x7f5cc8109aa0 2024-04-30T08:44:49.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.660+0000 7f5cbf400640 1 -- 172.21.15.32:0/344090399 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb8010670 con 0x7f5cc8109aa0 2024-04-30T08:44:49.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.661+0000 7f5ccdc00640 1 -- 172.21.15.32:0/344090399 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cc81a42b0 con 0x7f5cc8109aa0 2024-04-30T08:44:49.662 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.661+0000 7f5ccdc00640 1 -- 172.21.15.32:0/344090399 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cc81a47b0 con 0x7f5cc8109aa0 2024-04-30T08:44:49.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.663+0000 7f5ccdc00640 1 -- 172.21.15.32:0/344090399 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cc81a4a70 con 0x7f5cc8109aa0 2024-04-30T08:44:49.670 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.668+0000 7f5cbf400640 1 -- 172.21.15.32:0/344090399 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f5cb8004020 con 0x7f5cc8109aa0 2024-04-30T08:44:49.672 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.672+0000 7f5cbf400640 1 --2- 172.21.15.32:0/344090399 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f5cac0afcc0 0x7f5cac0b21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:44:49.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.672+0000 7f5cbf400640 1 -- 172.21.15.32:0/344090399 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f5cb80df2b0 con 0x7f5cc8109aa0 2024-04-30T08:44:49.674 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.674+0000 7f5cc6a00640 1 --2- 172.21.15.32:0/344090399 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f5cac0afcc0 0x7f5cac0b21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:44:49.678 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.677+0000 7f5cc6a00640 1 --2- 172.21.15.32:0/344090399 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f5cac0afcc0 0x7f5cac0b21d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8104760 tx=0x7f5cb4005ba0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:44:49.680 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:49.678+0000 7f5cbf400640 1 -- 172.21.15.32:0/344090399 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5cb80a7a80 con 0x7f5cc8109aa0 2024-04-30T08:44:50.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.041+0000 7f5ccdc00640 1 -- 172.21.15.32:0/344090399 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5cc81a4c40 con 0x7f5cc8109aa0 2024-04-30T08:44:50.043 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.042+0000 7f5cbf400640 1 -- 172.21.15.32:0/344090399 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) ==== 100+0+11 (secure 0 0 0) 0x7f5cb80a7380 con 0x7f5cc8109aa0 2024-04-30T08:44:50.043 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-04-30T08:44:50.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.046+0000 7f5ccdc00640 1 -- 172.21.15.32:0/344090399 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f5cac0afcc0 msgr2=0x7f5cac0b21d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:44:50.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.046+0000 7f5ccdc00640 1 --2- 172.21.15.32:0/344090399 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f5cac0afcc0 0x7f5cac0b21d0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8104760 tx=0x7f5cb4005ba0 comp rx=0 tx=0).stop 2024-04-30T08:44:50.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.046+0000 7f5ccdc00640 1 -- 172.21.15.32:0/344090399 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cc8109aa0 msgr2=0x7f5cc81a3950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:44:50.047 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.046+0000 7f5ccdc00640 1 --2- 172.21.15.32:0/344090399 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cc8109aa0 0x7f5cc81a3950 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f5cb800e9b0 tx=0x7f5cb800ee70 comp rx=0 tx=0).stop 2024-04-30T08:44:50.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.046+0000 7f5ccdc00640 1 -- 172.21.15.32:0/344090399 shutdown_connections 2024-04-30T08:44:50.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.046+0000 7f5ccdc00640 1 --2- 172.21.15.32:0/344090399 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f5cac0afcc0 0x7f5cac0b21d0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:50.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.046+0000 7f5ccdc00640 1 --2- 172.21.15.32:0/344090399 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5cc8109aa0 0x7f5cc81a3950 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:50.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.046+0000 7f5ccdc00640 1 --2- 172.21.15.32:0/344090399 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5cc8104da0 0x7f5cc819c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:50.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.046+0000 7f5ccdc00640 1 --2- 172.21.15.32:0/344090399 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5cc8103b60 0x7f5cc819c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:50.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.046+0000 7f5ccdc00640 1 -- 172.21.15.32:0/344090399 >> 172.21.15.32:0/344090399 conn(0x7f5cc80fed50 msgr2=0x7f5cc80ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:44:50.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.047+0000 7f5ccdc00640 1 -- 172.21.15.32:0/344090399 shutdown_connections 2024-04-30T08:44:50.048 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.047+0000 7f5ccdc00640 1 -- 172.21.15.32:0/344090399 wait complete. 2024-04-30T08:44:50.083 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-30T08:44:50.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:50 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2519495127' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-30T08:44:50.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:50 smithi032 ceph-mon[28331]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:50.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:50 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/344090399' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-30T08:44:50.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:50 smithi032 ceph-mon[28331]: pgmap v715: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 648 op/s 2024-04-30T08:44:50.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.470+0000 7f99ff800640 1 -- 172.21.15.32:0/1263786724 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99f8104da0 msgr2=0x7f99f81051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:44:50.471 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.470+0000 7f99ff800640 1 --2- 172.21.15.32:0/1263786724 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99f8104da0 0x7f99f81051b0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f99e4009860 tx=0x7f99e402a360 comp rx=0 tx=0).stop 2024-04-30T08:44:50.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.471+0000 7f99ff800640 1 -- 172.21.15.32:0/1263786724 shutdown_connections 2024-04-30T08:44:50.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.471+0000 7f99ff800640 1 --2- 172.21.15.32:0/1263786724 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99f8109aa0 0x7f99f810bec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:50.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.471+0000 7f99ff800640 1 --2- 172.21.15.32:0/1263786724 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99f8104da0 0x7f99f81051b0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:50.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.471+0000 7f99ff800640 1 --2- 172.21.15.32:0/1263786724 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99f8103b60 0x7f99f8104860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:50.472 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.471+0000 7f99ff800640 1 -- 172.21.15.32:0/1263786724 >> 172.21.15.32:0/1263786724 conn(0x7f99f80fed50 msgr2=0x7f99f8101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:44:50.473 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.473+0000 7f99ff800640 1 -- 172.21.15.32:0/1263786724 shutdown_connections 2024-04-30T08:44:50.474 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.473+0000 7f99ff800640 1 -- 172.21.15.32:0/1263786724 wait complete. 2024-04-30T08:44:50.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.476+0000 7f99ff800640 1 Processor -- start 2024-04-30T08:44:50.476 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.476+0000 7f99ff800640 1 -- start start 2024-04-30T08:44:50.477 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.477+0000 7f99ff800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99f8103b60 0x7f99f819c480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:44:50.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.477+0000 7f99ff800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99f8104da0 0x7f99f819c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:44:50.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.477+0000 7f99ff800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99f8109aa0 0x7f99f81a3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:44:50.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.477+0000 7f99ff800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f99f8075280 con 0x7f99f8109aa0 2024-04-30T08:44:50.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.477+0000 7f99ff800640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f99f8075100 con 0x7f99f8103b60 2024-04-30T08:44:50.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.477+0000 7f99ff800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f99f8075400 con 0x7f99f8104da0 2024-04-30T08:44:50.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.477+0000 7f99f7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99f8104da0 0x7f99f819c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:44:50.478 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.477+0000 7f99f7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99f8104da0 0x7f99f819c9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:43218/0 (socket says 172.21.15.32:43218) 2024-04-30T08:44:50.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.477+0000 7f99f7e00640 1 -- 172.21.15.32:0/2042163201 learned_addr learned my addr 172.21.15.32:0/2042163201 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:44:50.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.477+0000 7f99fce00640 1 --2- 172.21.15.32:0/2042163201 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99f8103b60 0x7f99f819c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:44:50.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.478+0000 7f99fce00640 1 -- 172.21.15.32:0/2042163201 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99f8104da0 msgr2=0x7f99f819c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:44:50.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.478+0000 7f99fce00640 1 --2- 172.21.15.32:0/2042163201 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99f8104da0 0x7f99f819c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:50.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.478+0000 7f99fce00640 1 -- 172.21.15.32:0/2042163201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99f8109aa0 msgr2=0x7f99f81a3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:44:50.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.478+0000 7f99fce00640 1 --2- 172.21.15.32:0/2042163201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99f8109aa0 0x7f99f81a3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:50.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.478+0000 7f99fce00640 1 -- 172.21.15.32:0/2042163201 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99f81a3fc0 con 0x7f99f8103b60 2024-04-30T08:44:50.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.478+0000 7f99fce00640 1 --2- 172.21.15.32:0/2042163201 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99f8103b60 0x7f99f819c480 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f99f8103430 tx=0x7f99ec00db30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:44:50.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.479+0000 7f99f5600640 1 -- 172.21.15.32:0/2042163201 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99ec017070 con 0x7f99f8103b60 2024-04-30T08:44:50.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.479+0000 7f99f5600640 1 -- 172.21.15.32:0/2042163201 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f99ec009cd0 con 0x7f99f8103b60 2024-04-30T08:44:50.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.479+0000 7f99f5600640 1 -- 172.21.15.32:0/2042163201 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99ec003b20 con 0x7f99f8103b60 2024-04-30T08:44:50.480 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.479+0000 7f99ff800640 1 -- 172.21.15.32:0/2042163201 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99f81a42b0 con 0x7f99f8103b60 2024-04-30T08:44:50.481 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.479+0000 7f99ff800640 1 -- 172.21.15.32:0/2042163201 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99f81a47b0 con 0x7f99f8103b60 2024-04-30T08:44:50.482 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.482+0000 7f99ff800640 1 -- 172.21.15.32:0/2042163201 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99f8107190 con 0x7f99f8103b60 2024-04-30T08:44:50.486 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.485+0000 7f99f5600640 1 -- 172.21.15.32:0/2042163201 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f99ec0042b0 con 0x7f99f8103b60 2024-04-30T08:44:50.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.487+0000 7f99f5600640 1 --2- 172.21.15.32:0/2042163201 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f99e00afaf0 0x7f99e00b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:44:50.488 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.488+0000 7f99f5600640 1 -- 172.21.15.32:0/2042163201 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f99ec0a2c80 con 0x7f99f8103b60 2024-04-30T08:44:50.490 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.489+0000 7f99f7e00640 1 --2- 172.21.15.32:0/2042163201 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f99e00afaf0 0x7f99e00b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:44:50.491 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.491+0000 7f99f7e00640 1 --2- 172.21.15.32:0/2042163201 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f99e00afaf0 0x7f99e00b2000 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f99e4009460 tx=0x7f99e40304f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:44:50.495 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.494+0000 7f99f5600640 1 -- 172.21.15.32:0/2042163201 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f99ec0a0e20 con 0x7f99f8103b60 2024-04-30T08:44:50.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:50 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2519495127' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-30T08:44:50.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:50 smithi071 ceph-mon[33385]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:50.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:50 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/344090399' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-30T08:44:50.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:50 smithi071 ceph-mon[33385]: pgmap v715: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 648 op/s 2024-04-30T08:44:50.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:50 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2519495127' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-30T08:44:50.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:50 smithi189 ceph-mon[33038]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-04-30T08:44:50.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:50 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/344090399' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-30T08:44:50.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:50 smithi189 ceph-mon[33038]: pgmap v715: 129 pgs: 129 active+clean; 10 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 648 op/s 2024-04-30T08:44:50.846 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.845+0000 7f99ff800640 1 -- 172.21.15.32:0/2042163201 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f99f8063820 con 0x7f99f8103b60 2024-04-30T08:44:50.847 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-30T08:44:50.847 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.846+0000 7f99f5600640 1 -- 172.21.15.32:0/2042163201 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v85) ==== 103+0+11 (secure 0 0 0) 0x7f99ec0a6d90 con 0x7f99f8103b60 2024-04-30T08:44:50.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.853+0000 7f99ff800640 1 -- 172.21.15.32:0/2042163201 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f99e00afaf0 msgr2=0x7f99e00b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:44:50.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.853+0000 7f99ff800640 1 --2- 172.21.15.32:0/2042163201 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f99e00afaf0 0x7f99e00b2000 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f99e4009460 tx=0x7f99e40304f0 comp rx=0 tx=0).stop 2024-04-30T08:44:50.853 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.853+0000 7f99ff800640 1 -- 172.21.15.32:0/2042163201 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99f8103b60 msgr2=0x7f99f819c480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:44:50.854 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.853+0000 7f99ff800640 1 --2- 172.21.15.32:0/2042163201 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99f8103b60 0x7f99f819c480 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f99f8103430 tx=0x7f99ec00db30 comp rx=0 tx=0).stop 2024-04-30T08:44:50.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.854+0000 7f99ff800640 1 -- 172.21.15.32:0/2042163201 shutdown_connections 2024-04-30T08:44:50.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.854+0000 7f99ff800640 1 --2- 172.21.15.32:0/2042163201 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f99e00afaf0 0x7f99e00b2000 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:50.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.854+0000 7f99ff800640 1 --2- 172.21.15.32:0/2042163201 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f99f8109aa0 0x7f99f81a3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:50.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.854+0000 7f99ff800640 1 --2- 172.21.15.32:0/2042163201 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f99f8104da0 0x7f99f819c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:50.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.854+0000 7f99ff800640 1 --2- 172.21.15.32:0/2042163201 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f99f8103b60 0x7f99f819c480 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:50.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.854+0000 7f99ff800640 1 -- 172.21.15.32:0/2042163201 >> 172.21.15.32:0/2042163201 conn(0x7f99f80fed50 msgr2=0x7f99f8101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:44:50.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.854+0000 7f99ff800640 1 -- 172.21.15.32:0/2042163201 shutdown_connections 2024-04-30T08:44:50.855 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:50.854+0000 7f99ff800640 1 -- 172.21.15.32:0/2042163201 wait complete. 2024-04-30T08:44:50.900 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-30T08:44:51.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.323+0000 7fa438600640 1 -- 172.21.15.32:0/4006818117 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa430112140 msgr2=0x7fa4301145a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:44:51.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.323+0000 7fa438600640 1 --2- 172.21.15.32:0/4006818117 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa430112140 0x7fa4301145a0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fa428009ca0 tx=0x7fa42802a8c0 comp rx=0 tx=0).stop 2024-04-30T08:44:51.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.324+0000 7fa438600640 1 -- 172.21.15.32:0/4006818117 shutdown_connections 2024-04-30T08:44:51.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.324+0000 7fa438600640 1 --2- 172.21.15.32:0/4006818117 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa430112140 0x7fa4301145a0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:51.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.324+0000 7fa438600640 1 --2- 172.21.15.32:0/4006818117 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa43010f7e0 0x7fa430111c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:51.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.324+0000 7fa438600640 1 --2- 172.21.15.32:0/4006818117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa43010a040 0x7fa43010a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:51.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.324+0000 7fa438600640 1 -- 172.21.15.32:0/4006818117 >> 172.21.15.32:0/4006818117 conn(0x7fa430070560 msgr2=0x7fa430072a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:44:51.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.324+0000 7fa438600640 1 -- 172.21.15.32:0/4006818117 shutdown_connections 2024-04-30T08:44:51.325 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.324+0000 7fa438600640 1 -- 172.21.15.32:0/4006818117 wait complete. 2024-04-30T08:44:51.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.325+0000 7fa438600640 1 Processor -- start 2024-04-30T08:44:51.326 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.326+0000 7fa438600640 1 -- start start 2024-04-30T08:44:51.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.326+0000 7fa438600640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa43010a040 0x7fa43010d040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:44:51.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.326+0000 7fa438600640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa43010f7e0 0x7fa43010b6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:44:51.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.326+0000 7fa438600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa430112140 0x7fa43010bc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:44:51.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.326+0000 7fa438600640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fa430117f70 con 0x7fa43010f7e0 2024-04-30T08:44:51.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.326+0000 7fa438600640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fa430117df0 con 0x7fa43010a040 2024-04-30T08:44:51.327 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.326+0000 7fa438600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fa4301180f0 con 0x7fa430112140 2024-04-30T08:44:51.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.326+0000 7fa436600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa430112140 0x7fa43010bc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:44:51.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.326+0000 7fa435c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa43010a040 0x7fa43010d040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:44:51.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.326+0000 7fa435c00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa43010a040 0x7fa43010d040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:48246/0 (socket says 172.21.15.32:48246) 2024-04-30T08:44:51.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.326+0000 7fa436600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa430112140 0x7fa43010bc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:43240/0 (socket says 172.21.15.32:43240) 2024-04-30T08:44:51.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.326+0000 7fa435c00640 1 -- 172.21.15.32:0/2063780753 learned_addr learned my addr 172.21.15.32:0/2063780753 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:44:51.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.327+0000 7fa435200640 1 --2- 172.21.15.32:0/2063780753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa43010f7e0 0x7fa43010b6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:44:51.328 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.327+0000 7fa435c00640 1 -- 172.21.15.32:0/2063780753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa430112140 msgr2=0x7fa43010bc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:44:51.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.327+0000 7fa435c00640 1 --2- 172.21.15.32:0/2063780753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa430112140 0x7fa43010bc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:51.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.327+0000 7fa435c00640 1 -- 172.21.15.32:0/2063780753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa43010f7e0 msgr2=0x7fa43010b6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:44:51.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.327+0000 7fa435c00640 1 --2- 172.21.15.32:0/2063780753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa43010f7e0 0x7fa43010b6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:51.329 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.327+0000 7fa435c00640 1 -- 172.21.15.32:0/2063780753 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4301a69d0 con 0x7fa43010a040 2024-04-30T08:44:51.331 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.330+0000 7fa435c00640 1 --2- 172.21.15.32:0/2063780753 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa43010a040 0x7fa43010d040 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fa42400b450 tx=0x7fa42400b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:44:51.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.331+0000 7fa42e000640 1 -- 172.21.15.32:0/2063780753 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa424015020 con 0x7fa43010a040 2024-04-30T08:44:51.332 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.331+0000 7fa42e000640 1 -- 172.21.15.32:0/2063780753 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fa424011d40 con 0x7fa43010a040 2024-04-30T08:44:51.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.331+0000 7fa42e000640 1 -- 172.21.15.32:0/2063780753 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa424010e00 con 0x7fa43010a040 2024-04-30T08:44:51.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.331+0000 7fa438600640 1 -- 172.21.15.32:0/2063780753 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4301a6cc0 con 0x7fa43010a040 2024-04-30T08:44:51.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.331+0000 7fa438600640 1 -- 172.21.15.32:0/2063780753 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4301a71e0 con 0x7fa43010a040 2024-04-30T08:44:51.335 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.334+0000 7fa438600640 1 -- 172.21.15.32:0/2063780753 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa4100052c0 con 0x7fa43010a040 2024-04-30T08:44:51.341 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.340+0000 7fa42e000640 1 -- 172.21.15.32:0/2063780753 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7fa424018020 con 0x7fa43010a040 2024-04-30T08:44:51.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.343+0000 7fa42e000640 1 --2- 172.21.15.32:0/2063780753 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa4180afbb0 0x7fa4180b20c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:44:51.343 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.343+0000 7fa42e000640 1 -- 172.21.15.32:0/2063780753 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fa4240de8a0 con 0x7fa43010a040 2024-04-30T08:44:51.344 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.343+0000 7fa435200640 1 --2- 172.21.15.32:0/2063780753 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa4180afbb0 0x7fa4180b20c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:44:51.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.349+0000 7fa42e000640 1 -- 172.21.15.32:0/2063780753 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa42401f020 con 0x7fa43010a040 2024-04-30T08:44:51.353 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.349+0000 7fa435200640 1 --2- 172.21.15.32:0/2063780753 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa4180afbb0 0x7fa4180b20c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fa43010c9e0 tx=0x7fa41c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:44:51.693 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:51 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2042163201' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-30T08:44:51.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:51 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2042163201' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-30T08:44:51.779 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.779+0000 7fa438600640 1 -- 172.21.15.32:0/2063780753 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7fa4100055b0 con 0x7fa43010a040 2024-04-30T08:44:51.794 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.793+0000 7fa42e000640 1 -- 172.21.15.32:0/2063780753 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v20) ==== 81+0+803127 (secure 0 0 0) 0x7fa42400d040 con 0x7fa43010a040 2024-04-30T08:44:51.801 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:44:51.802 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-30T08:44:51.802 INFO:teuthology.orchestra.run.smithi032.stdout: "epoch": 20, 2024-04-30T08:44:51.802 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.802 INFO:teuthology.orchestra.run.smithi032.stdout: "active_gid": 14152, 2024-04-30T08:44:51.802 INFO:teuthology.orchestra.run.smithi032.stdout: "active_name": "x", 2024-04-30T08:44:51.802 INFO:teuthology.orchestra.run.smithi032.stdout: "active_addrs": { 2024-04-30T08:44:51.802 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-30T08:44:51.802 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:51.802 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-30T08:44:51.802 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6800", 2024-04-30T08:44:51.802 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2930853529 2024-04-30T08:44:51.803 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.803 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:51.803 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v1", 2024-04-30T08:44:51.803 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:6801", 2024-04-30T08:44:51.803 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2930853529 2024-04-30T08:44:51.803 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:51.803 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-30T08:44:51.803 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.803 INFO:teuthology.orchestra.run.smithi032.stdout: "active_addr": "172.21.15.32:6801/2930853529", 2024-04-30T08:44:51.803 INFO:teuthology.orchestra.run.smithi032.stdout: "active_change": "2024-04-30T08:23:29.526909+0000", 2024-04-30T08:44:51.803 INFO:teuthology.orchestra.run.smithi032.stdout: "active_mgr_features": 4540701547738038271, 2024-04-30T08:44:51.803 INFO:teuthology.orchestra.run.smithi032.stdout: "available": true, 2024-04-30T08:44:51.804 INFO:teuthology.orchestra.run.smithi032.stdout: "standbys": [ 2024-04-30T08:44:51.804 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:51.804 INFO:teuthology.orchestra.run.smithi032.stdout: "gid": 24109, 2024-04-30T08:44:51.804 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "z", 2024-04-30T08:44:51.804 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_features": 4540701547738038271, 2024-04-30T08:44:51.804 INFO:teuthology.orchestra.run.smithi032.stdout: "available_modules": [ 2024-04-30T08:44:51.804 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:51.804 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "alerts", 2024-04-30T08:44:51.804 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:51.804 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:51.804 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:51.804 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2024-04-30T08:44:51.805 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2024-04-30T08:44:51.805 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:51.805 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.805 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.805 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2024-04-30T08:44:51.805 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.805 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.805 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.805 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "How frequently to reexamine health status", 2024-04-30T08:44:51.805 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.805 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.805 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.805 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.806 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:51.806 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:51.806 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.806 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.806 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.806 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:51.807 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.807 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.807 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:51.807 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:51.807 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:51.807 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:51.807 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:51.807 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:51.808 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:51.808 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:51.808 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.808 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.808 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.808 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.808 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:51.808 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:51.808 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.808 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.808 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.809 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:51.809 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.809 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.809 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.809 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.809 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.809 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.809 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.809 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:51.809 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:51.809 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.810 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.810 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.810 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:51.810 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.810 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.810 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:51.810 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:51.810 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:51.810 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:51.810 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:51.810 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:51.810 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:51.811 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:51.811 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.811 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.811 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.812 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.812 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:51.812 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:51.812 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.812 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.812 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.812 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:51.812 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.812 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.812 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.812 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.813 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.813 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.813 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.813 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.813 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_destination": { 2024-04-30T08:44:51.813 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_destination", 2024-04-30T08:44:51.813 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.813 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.813 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.813 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:51.813 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.813 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.813 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.813 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Email address to send alerts to", 2024-04-30T08:44:51.814 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.814 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.814 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.814 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_from_name": { 2024-04-30T08:44:51.814 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_from_name", 2024-04-30T08:44:51.814 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.814 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.814 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.814 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "Ceph", 2024-04-30T08:44:51.814 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.814 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.814 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.815 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Email From: name", 2024-04-30T08:44:51.815 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.815 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.815 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.815 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.815 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_host": { 2024-04-30T08:44:51.815 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_host", 2024-04-30T08:44:51.815 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.815 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.815 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.815 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:51.815 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.815 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.815 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.816 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "SMTP server", 2024-04-30T08:44:51.816 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.816 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.816 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.816 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.816 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_password": { 2024-04-30T08:44:51.816 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_password", 2024-04-30T08:44:51.816 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.816 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.816 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.816 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:51.816 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.816 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.816 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.816 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Password to authenticate with", 2024-04-30T08:44:51.817 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.817 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.817 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.817 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.817 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_port": { 2024-04-30T08:44:51.817 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_port", 2024-04-30T08:44:51.817 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:51.817 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.817 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.817 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "465", 2024-04-30T08:44:51.817 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.817 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.817 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.817 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "SMTP port", 2024-04-30T08:44:51.818 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.818 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.818 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.818 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.818 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_sender": { 2024-04-30T08:44:51.818 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_sender", 2024-04-30T08:44:51.818 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.818 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.818 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.818 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:51.818 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.818 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.818 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.818 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "SMTP envelope sender", 2024-04-30T08:44:51.819 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.819 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.819 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.819 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.819 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_ssl": { 2024-04-30T08:44:51.819 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_ssl", 2024-04-30T08:44:51.819 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.819 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.819 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.819 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:51.819 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.819 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.819 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.820 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-30T08:44:51.820 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.820 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.820 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.820 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.820 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_user": { 2024-04-30T08:44:51.820 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_user", 2024-04-30T08:44:51.820 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.820 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.820 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.820 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:51.820 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.820 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.821 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.821 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "User to authenticate as", 2024-04-30T08:44:51.821 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.821 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.821 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.821 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.821 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:51.821 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:51.821 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:51.821 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:51.821 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.821 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:51.821 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.821 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.822 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.822 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.822 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.822 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.822 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.822 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:51.822 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:51.822 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.822 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:51.822 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "balancer", 2024-04-30T08:44:51.822 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:51.822 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:51.822 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:51.822 INFO:teuthology.orchestra.run.smithi032.stdout: "active": { 2024-04-30T08:44:51.823 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "active", 2024-04-30T08:44:51.823 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.823 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.823 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.823 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:51.823 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.823 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.823 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.823 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "automatically balance PGs across cluster", 2024-04-30T08:44:51.823 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.823 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.823 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.823 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.824 INFO:teuthology.orchestra.run.smithi032.stdout: "begin_time": { 2024-04-30T08:44:51.824 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "begin_time", 2024-04-30T08:44:51.824 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.824 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.824 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.824 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0000", 2024-04-30T08:44:51.824 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.824 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.824 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.824 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "beginning time of day to automatically balance", 2024-04-30T08:44:51.824 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-30T08:44:51.824 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.824 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.824 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.825 INFO:teuthology.orchestra.run.smithi032.stdout: "begin_weekday": { 2024-04-30T08:44:51.825 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "begin_weekday", 2024-04-30T08:44:51.825 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2024-04-30T08:44:51.825 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.825 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.825 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:51.825 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "0", 2024-04-30T08:44:51.825 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "6", 2024-04-30T08:44:51.825 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.825 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-30T08:44:51.825 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-30T08:44:51.825 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.825 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.825 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.825 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_compat_max_iterations": { 2024-04-30T08:44:51.826 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crush_compat_max_iterations", 2024-04-30T08:44:51.826 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2024-04-30T08:44:51.826 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.826 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.826 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "25", 2024-04-30T08:44:51.826 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "1", 2024-04-30T08:44:51.826 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "250", 2024-04-30T08:44:51.826 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.826 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-30T08:44:51.826 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.826 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.826 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.826 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.827 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_compat_metrics": { 2024-04-30T08:44:51.827 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crush_compat_metrics", 2024-04-30T08:44:51.827 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.827 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.827 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.827 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "pgs,objects,bytes", 2024-04-30T08:44:51.827 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.827 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.827 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.827 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-30T08:44:51.827 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-30T08:44:51.827 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.827 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.827 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.828 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_compat_step": { 2024-04-30T08:44:51.828 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crush_compat_step", 2024-04-30T08:44:51.828 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-30T08:44:51.828 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.828 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.828 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0.5", 2024-04-30T08:44:51.828 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "0.001", 2024-04-30T08:44:51.828 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "0.999", 2024-04-30T08:44:51.828 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.828 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "aggressiveness of optimization", 2024-04-30T08:44:51.828 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-30T08:44:51.828 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.828 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.828 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.828 INFO:teuthology.orchestra.run.smithi032.stdout: "end_time": { 2024-04-30T08:44:51.829 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "end_time", 2024-04-30T08:44:51.829 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.829 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.829 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.829 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "2359", 2024-04-30T08:44:51.829 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.829 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.829 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.829 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "ending time of day to automatically balance", 2024-04-30T08:44:51.829 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-30T08:44:51.829 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.829 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.829 INFO:teuthology.orchestra.run.smithi032.stdout: "end_weekday": { 2024-04-30T08:44:51.830 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "end_weekday", 2024-04-30T08:44:51.830 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2024-04-30T08:44:51.830 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.830 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.830 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:51.830 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "0", 2024-04-30T08:44:51.830 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "6", 2024-04-30T08:44:51.830 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.830 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-30T08:44:51.830 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-30T08:44:51.830 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.830 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.830 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.830 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:51.831 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:51.831 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.831 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.831 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.831 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:51.831 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.831 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.831 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:51.831 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:51.831 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:51.831 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:51.831 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:51.831 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:51.832 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:51.832 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:51.832 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.832 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.832 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.832 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.832 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.832 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:51.832 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:51.832 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.832 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.832 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.832 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:51.832 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.833 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.833 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.833 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.833 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.833 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.833 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.833 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.833 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:51.833 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:51.833 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.833 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.833 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.833 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:51.833 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.834 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.834 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:51.834 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:51.834 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:51.834 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:51.834 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:51.834 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:51.834 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:51.834 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:51.834 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.834 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.834 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.834 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.835 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.835 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:51.835 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:51.835 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.835 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.835 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.835 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:51.835 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.835 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.835 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.835 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.835 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.835 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.835 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.836 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.836 INFO:teuthology.orchestra.run.smithi032.stdout: "min_score": { 2024-04-30T08:44:51.836 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "min_score", 2024-04-30T08:44:51.836 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-30T08:44:51.836 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.836 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.836 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:51.836 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.836 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.836 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.836 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-30T08:44:51.836 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.836 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.836 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.837 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.837 INFO:teuthology.orchestra.run.smithi032.stdout: "mode": { 2024-04-30T08:44:51.837 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mode", 2024-04-30T08:44:51.837 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.837 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.837 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.837 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "upmap", 2024-04-30T08:44:51.837 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.837 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.837 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:51.837 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-compat", 2024-04-30T08:44:51.837 INFO:teuthology.orchestra.run.smithi032.stdout: "none", 2024-04-30T08:44:51.837 INFO:teuthology.orchestra.run.smithi032.stdout: "read", 2024-04-30T08:44:51.837 INFO:teuthology.orchestra.run.smithi032.stdout: "upmap", 2024-04-30T08:44:51.838 INFO:teuthology.orchestra.run.smithi032.stdout: "upmap-read" 2024-04-30T08:44:51.838 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:51.838 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Balancer mode", 2024-04-30T08:44:51.838 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.838 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.838 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.838 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.838 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_ids": { 2024-04-30T08:44:51.838 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "pool_ids", 2024-04-30T08:44:51.838 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.838 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.838 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.838 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:51.838 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.839 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.839 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.839 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-30T08:44:51.839 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.839 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.839 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.839 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.839 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2024-04-30T08:44:51.839 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2024-04-30T08:44:51.839 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:51.839 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.839 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.840 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2024-04-30T08:44:51.840 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.840 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.840 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.840 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-30T08:44:51.840 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.840 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.840 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.840 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.840 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:51.840 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:51.840 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:51.840 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:51.840 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.841 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:51.841 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.841 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.841 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.841 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.841 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.841 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.841 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.841 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.841 INFO:teuthology.orchestra.run.smithi032.stdout: "upmap_max_deviation": { 2024-04-30T08:44:51.841 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "upmap_max_deviation", 2024-04-30T08:44:51.841 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:51.841 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.842 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.842 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5", 2024-04-30T08:44:51.842 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "1", 2024-04-30T08:44:51.842 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.842 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.842 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-30T08:44:51.842 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-30T08:44:51.842 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.842 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.842 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.842 INFO:teuthology.orchestra.run.smithi032.stdout: "upmap_max_optimizations": { 2024-04-30T08:44:51.842 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "upmap_max_optimizations", 2024-04-30T08:44:51.842 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2024-04-30T08:44:51.842 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.843 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.843 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-30T08:44:51.843 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.843 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.843 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.843 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-30T08:44:51.843 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.843 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.843 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.843 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:51.843 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:51.843 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.843 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:51.843 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cephadm", 2024-04-30T08:44:51.844 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:51.844 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:51.844 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:51.844 INFO:teuthology.orchestra.run.smithi032.stdout: "agent_down_multiplier": { 2024-04-30T08:44:51.844 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "agent_down_multiplier", 2024-04-30T08:44:51.844 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-30T08:44:51.844 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.844 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.844 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "3.0", 2024-04-30T08:44:51.844 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.844 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.844 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.844 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-30T08:44:51.844 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.845 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.845 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.845 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.845 INFO:teuthology.orchestra.run.smithi032.stdout: "agent_refresh_rate": { 2024-04-30T08:44:51.845 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "agent_refresh_rate", 2024-04-30T08:44:51.845 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:51.845 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.845 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.845 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "20", 2024-04-30T08:44:51.845 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.845 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.845 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.846 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-30T08:44:51.846 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.846 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.846 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.846 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.846 INFO:teuthology.orchestra.run.smithi032.stdout: "agent_starting_port": { 2024-04-30T08:44:51.846 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "agent_starting_port", 2024-04-30T08:44:51.846 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:51.846 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.846 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.846 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "4721", 2024-04-30T08:44:51.846 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.846 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.847 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.847 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-30T08:44:51.847 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.847 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.847 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.847 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.847 INFO:teuthology.orchestra.run.smithi032.stdout: "allow_ptrace": { 2024-04-30T08:44:51.847 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "allow_ptrace", 2024-04-30T08:44:51.847 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.847 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.847 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.847 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:51.847 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.847 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.848 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.848 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-30T08:44:51.848 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-30T08:44:51.848 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.848 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.848 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.848 INFO:teuthology.orchestra.run.smithi032.stdout: "autotune_interval": { 2024-04-30T08:44:51.848 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "autotune_interval", 2024-04-30T08:44:51.848 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:51.848 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.848 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.848 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2024-04-30T08:44:51.848 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.849 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.849 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.849 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-30T08:44:51.849 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.849 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.849 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.849 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.849 INFO:teuthology.orchestra.run.smithi032.stdout: "autotune_memory_target_ratio": { 2024-04-30T08:44:51.849 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "autotune_memory_target_ratio", 2024-04-30T08:44:51.849 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-30T08:44:51.849 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.849 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.849 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0.7", 2024-04-30T08:44:51.849 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.850 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.850 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.850 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-30T08:44:51.850 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.850 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.850 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.850 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.850 INFO:teuthology.orchestra.run.smithi032.stdout: "cephadm_log_destination": { 2024-04-30T08:44:51.850 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cephadm_log_destination", 2024-04-30T08:44:51.850 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.850 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.850 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.850 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:51.851 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.851 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.851 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:51.851 INFO:teuthology.orchestra.run.smithi032.stdout: "file", 2024-04-30T08:44:51.851 INFO:teuthology.orchestra.run.smithi032.stdout: "file,syslog", 2024-04-30T08:44:51.851 INFO:teuthology.orchestra.run.smithi032.stdout: "syslog" 2024-04-30T08:44:51.851 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:51.851 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-30T08:44:51.851 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.851 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.851 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.851 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.851 INFO:teuthology.orchestra.run.smithi032.stdout: "cgroups_split": { 2024-04-30T08:44:51.851 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cgroups_split", 2024-04-30T08:44:51.852 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.852 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.852 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.852 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:51.852 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.852 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.852 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.852 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-30T08:44:51.852 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.852 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.852 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.852 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.852 INFO:teuthology.orchestra.run.smithi032.stdout: "config_checks_enabled": { 2024-04-30T08:44:51.852 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "config_checks_enabled", 2024-04-30T08:44:51.853 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.853 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.853 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.853 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:51.853 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.853 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.853 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.853 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-30T08:44:51.853 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.853 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.853 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.853 INFO:teuthology.orchestra.run.smithi032.stdout: "config_dashboard": { 2024-04-30T08:44:51.853 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "config_dashboard", 2024-04-30T08:44:51.853 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.853 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.854 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.854 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:51.854 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.854 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.854 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.854 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-30T08:44:51.854 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.854 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.854 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.854 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.854 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_alertmanager": { 2024-04-30T08:44:51.854 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_alertmanager", 2024-04-30T08:44:51.854 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.854 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.855 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.855 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-30T08:44:51.855 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.855 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.855 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.855 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Prometheus container image", 2024-04-30T08:44:51.855 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.855 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.855 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.855 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.855 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_base": { 2024-04-30T08:44:51.855 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_base", 2024-04-30T08:44:51.855 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.855 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.855 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.856 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-30T08:44:51.856 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.856 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.856 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.856 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Container image name, without the tag", 2024-04-30T08:44:51.856 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.856 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.856 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.856 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.856 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_elasticsearch": { 2024-04-30T08:44:51.856 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_elasticsearch", 2024-04-30T08:44:51.856 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.856 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.856 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.857 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-30T08:44:51.857 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.857 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.857 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.857 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "elasticsearch container image", 2024-04-30T08:44:51.857 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.857 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.857 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.857 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.857 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_grafana": { 2024-04-30T08:44:51.857 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_grafana", 2024-04-30T08:44:51.857 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.857 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.858 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.858 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-30T08:44:51.858 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.858 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.858 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.858 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Prometheus container image", 2024-04-30T08:44:51.858 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.858 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.858 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.858 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.858 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_haproxy": { 2024-04-30T08:44:51.858 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_haproxy", 2024-04-30T08:44:51.858 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.859 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.859 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.859 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-30T08:44:51.859 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.859 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.859 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.859 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "HAproxy container image", 2024-04-30T08:44:51.859 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.859 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.859 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.859 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.859 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_jaeger_agent": { 2024-04-30T08:44:51.859 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_jaeger_agent", 2024-04-30T08:44:51.859 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.860 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.860 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.860 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-30T08:44:51.860 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.860 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.860 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.860 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Jaeger agent container image", 2024-04-30T08:44:51.860 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.860 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.860 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.860 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.860 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_jaeger_collector": { 2024-04-30T08:44:51.860 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_jaeger_collector", 2024-04-30T08:44:51.860 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.861 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.861 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.861 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-30T08:44:51.861 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.861 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.861 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.861 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Jaeger collector container image", 2024-04-30T08:44:51.861 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.861 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.861 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.861 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.861 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_jaeger_query": { 2024-04-30T08:44:51.861 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_jaeger_query", 2024-04-30T08:44:51.862 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.862 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.862 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.862 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-30T08:44:51.862 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.862 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.862 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.862 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Jaeger query container image", 2024-04-30T08:44:51.862 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.862 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.862 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.862 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.862 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_keepalived": { 2024-04-30T08:44:51.862 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_keepalived", 2024-04-30T08:44:51.863 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.863 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.863 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.863 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-30T08:44:51.863 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.863 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.863 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.863 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Keepalived container image", 2024-04-30T08:44:51.863 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.863 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.863 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.863 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.863 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_loki": { 2024-04-30T08:44:51.863 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_loki", 2024-04-30T08:44:51.864 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.864 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.864 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.864 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-30T08:44:51.864 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.864 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.864 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.864 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Loki container image", 2024-04-30T08:44:51.864 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.864 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.864 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.864 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.864 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_node_exporter": { 2024-04-30T08:44:51.864 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_node_exporter", 2024-04-30T08:44:51.865 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.865 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.865 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.865 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-30T08:44:51.865 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.865 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.865 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.865 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Prometheus container image", 2024-04-30T08:44:51.865 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.865 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.865 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.865 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.865 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_nvmeof": { 2024-04-30T08:44:51.865 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_nvmeof", 2024-04-30T08:44:51.866 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.866 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.866 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.866 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-30T08:44:51.866 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.866 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.866 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.866 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Nvme-of container image", 2024-04-30T08:44:51.866 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.866 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.866 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.866 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.866 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_prometheus": { 2024-04-30T08:44:51.866 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_prometheus", 2024-04-30T08:44:51.867 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.867 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.867 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.867 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-30T08:44:51.867 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.867 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.867 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.867 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Prometheus container image", 2024-04-30T08:44:51.867 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.867 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.867 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.867 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.867 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_promtail": { 2024-04-30T08:44:51.867 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_promtail", 2024-04-30T08:44:51.868 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.868 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.868 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.868 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-30T08:44:51.868 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.868 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.868 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.868 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Promtail container image", 2024-04-30T08:44:51.868 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.868 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.868 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.868 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.868 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_samba": { 2024-04-30T08:44:51.868 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_samba", 2024-04-30T08:44:51.869 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.869 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.869 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.869 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-30T08:44:51.869 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.869 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.869 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.869 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Samba/SMB container image", 2024-04-30T08:44:51.869 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.869 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.869 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.869 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.869 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_snmp_gateway": { 2024-04-30T08:44:51.870 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_snmp_gateway", 2024-04-30T08:44:51.870 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.870 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.870 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.870 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-30T08:44:51.870 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.870 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.870 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.870 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "SNMP Gateway container image", 2024-04-30T08:44:51.870 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.870 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.870 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.870 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.870 INFO:teuthology.orchestra.run.smithi032.stdout: "container_init": { 2024-04-30T08:44:51.871 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_init", 2024-04-30T08:44:51.871 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.871 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.871 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.871 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:51.871 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.871 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.871 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.871 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Run podman/docker with `--init`", 2024-04-30T08:44:51.871 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.871 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.871 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.871 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.872 INFO:teuthology.orchestra.run.smithi032.stdout: "daemon_cache_timeout": { 2024-04-30T08:44:51.872 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "daemon_cache_timeout", 2024-04-30T08:44:51.872 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:51.872 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.872 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.872 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2024-04-30T08:44:51.872 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.872 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.872 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.872 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-30T08:44:51.872 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.872 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.872 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.873 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.873 INFO:teuthology.orchestra.run.smithi032.stdout: "default_cephadm_command_timeout": { 2024-04-30T08:44:51.873 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "default_cephadm_command_timeout", 2024-04-30T08:44:51.873 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:51.873 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.873 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.873 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "900", 2024-04-30T08:44:51.873 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.873 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.873 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.873 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-30T08:44:51.873 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.873 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.873 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.873 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.874 INFO:teuthology.orchestra.run.smithi032.stdout: "default_registry": { 2024-04-30T08:44:51.874 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "default_registry", 2024-04-30T08:44:51.874 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.874 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.874 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.874 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "docker.io", 2024-04-30T08:44:51.874 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.874 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.874 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.874 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-30T08:44:51.874 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.874 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.874 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.875 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.875 INFO:teuthology.orchestra.run.smithi032.stdout: "device_cache_timeout": { 2024-04-30T08:44:51.875 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "device_cache_timeout", 2024-04-30T08:44:51.875 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:51.875 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.875 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.875 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "1800", 2024-04-30T08:44:51.875 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.875 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.875 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.875 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "seconds to cache device inventory", 2024-04-30T08:44:51.875 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.875 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.875 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.876 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.876 INFO:teuthology.orchestra.run.smithi032.stdout: "device_enhanced_scan": { 2024-04-30T08:44:51.876 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "device_enhanced_scan", 2024-04-30T08:44:51.876 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.876 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.876 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.876 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:51.876 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.876 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.876 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.876 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-30T08:44:51.876 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.876 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.876 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.876 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.877 INFO:teuthology.orchestra.run.smithi032.stdout: "facts_cache_timeout": { 2024-04-30T08:44:51.877 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "facts_cache_timeout", 2024-04-30T08:44:51.877 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:51.877 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.877 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.877 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2024-04-30T08:44:51.877 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.877 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.877 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.877 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "seconds to cache host facts data", 2024-04-30T08:44:51.877 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.877 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.877 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.877 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.878 INFO:teuthology.orchestra.run.smithi032.stdout: "grafana_dashboards_path": { 2024-04-30T08:44:51.878 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "grafana_dashboards_path", 2024-04-30T08:44:51.878 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.878 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.878 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.878 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-30T08:44:51.878 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.878 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.878 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.878 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-30T08:44:51.878 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.878 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.878 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.878 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.879 INFO:teuthology.orchestra.run.smithi032.stdout: "host_check_interval": { 2024-04-30T08:44:51.879 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "host_check_interval", 2024-04-30T08:44:51.879 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:51.879 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.879 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.879 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2024-04-30T08:44:51.879 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.879 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.879 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.879 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to perform a host check", 2024-04-30T08:44:51.879 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.879 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.879 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.880 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.880 INFO:teuthology.orchestra.run.smithi032.stdout: "hw_monitoring": { 2024-04-30T08:44:51.880 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "hw_monitoring", 2024-04-30T08:44:51.880 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.880 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.880 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.880 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:51.880 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.880 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.880 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.880 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-30T08:44:51.880 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.880 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.880 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.880 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.881 INFO:teuthology.orchestra.run.smithi032.stdout: "inventory_list_all": { 2024-04-30T08:44:51.881 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "inventory_list_all", 2024-04-30T08:44:51.881 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.881 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.881 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.881 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:51.881 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.881 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.881 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.881 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-30T08:44:51.881 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.881 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.881 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.882 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.882 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:51.882 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:51.882 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.882 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.882 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.882 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:51.882 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.882 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.882 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:51.882 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:51.882 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:51.882 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:51.883 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:51.883 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:51.883 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:51.883 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:51.883 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.883 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.883 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.883 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.883 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.883 INFO:teuthology.orchestra.run.smithi032.stdout: "log_refresh_metadata": { 2024-04-30T08:44:51.883 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_refresh_metadata", 2024-04-30T08:44:51.883 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.883 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.883 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.884 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:51.884 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.884 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.884 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.884 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-30T08:44:51.884 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.884 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.884 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.884 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.884 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:51.884 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:51.884 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.884 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.884 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.885 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:51.885 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.885 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.885 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.885 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-30T08:44:51.885 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.885 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.885 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.885 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.885 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:51.885 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:51.885 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.885 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.885 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.886 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:51.886 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.886 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.886 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:51.886 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:51.886 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:51.886 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:51.886 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:51.886 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:51.886 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:51.886 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:51.886 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.886 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.886 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.887 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.887 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.887 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:51.887 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:51.887 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.887 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.887 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.887 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:51.887 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.887 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.887 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.887 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.887 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.887 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.888 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.888 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.888 INFO:teuthology.orchestra.run.smithi032.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-30T08:44:51.888 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-30T08:44:51.888 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.888 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.888 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.888 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:51.888 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.888 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.888 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.888 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-30T08:44:51.888 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.888 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.889 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.889 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.889 INFO:teuthology.orchestra.run.smithi032.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-30T08:44:51.889 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-30T08:44:51.889 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.889 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.889 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.889 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "*", 2024-04-30T08:44:51.889 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.889 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.889 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.889 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-30T08:44:51.889 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.889 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.889 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.890 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.890 INFO:teuthology.orchestra.run.smithi032.stdout: "max_count_per_host": { 2024-04-30T08:44:51.890 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "max_count_per_host", 2024-04-30T08:44:51.890 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:51.890 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.890 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.890 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-30T08:44:51.890 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.890 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.890 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.890 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "max number of daemons per service per host", 2024-04-30T08:44:51.890 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.890 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.890 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.891 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.891 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd_draining_count": { 2024-04-30T08:44:51.891 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "max_osd_draining_count", 2024-04-30T08:44:51.891 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:51.891 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.891 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.891 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-30T08:44:51.891 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.891 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.891 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.891 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-30T08:44:51.891 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.891 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.892 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.892 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.892 INFO:teuthology.orchestra.run.smithi032.stdout: "migration_current": { 2024-04-30T08:44:51.892 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "migration_current", 2024-04-30T08:44:51.892 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:51.892 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.892 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.892 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:51.892 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.892 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.892 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.892 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "internal - do not modify", 2024-04-30T08:44:51.892 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.892 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.893 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.893 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.893 INFO:teuthology.orchestra.run.smithi032.stdout: "mode": { 2024-04-30T08:44:51.893 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mode", 2024-04-30T08:44:51.893 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.893 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.893 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.893 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "root", 2024-04-30T08:44:51.893 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.893 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.893 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:51.893 INFO:teuthology.orchestra.run.smithi032.stdout: "cephadm-package", 2024-04-30T08:44:51.893 INFO:teuthology.orchestra.run.smithi032.stdout: "root" 2024-04-30T08:44:51.893 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:51.894 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "mode for remote execution of cephadm", 2024-04-30T08:44:51.894 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.894 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.894 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.894 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.894 INFO:teuthology.orchestra.run.smithi032.stdout: "oob_default_addr": { 2024-04-30T08:44:51.894 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "oob_default_addr", 2024-04-30T08:44:51.894 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.894 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.894 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.894 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "169.254.1.1", 2024-04-30T08:44:51.894 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.894 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.895 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.895 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-30T08:44:51.895 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.895 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.895 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.895 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.895 INFO:teuthology.orchestra.run.smithi032.stdout: "prometheus_alerts_path": { 2024-04-30T08:44:51.895 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "prometheus_alerts_path", 2024-04-30T08:44:51.895 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.895 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.895 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.895 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-30T08:44:51.895 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.895 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.896 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.896 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-30T08:44:51.896 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.896 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.896 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.896 INFO:teuthology.orchestra.run.smithi032.stdout: "registry_insecure": { 2024-04-30T08:44:51.896 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "registry_insecure", 2024-04-30T08:44:51.896 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.896 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.896 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.896 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:51.896 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.896 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.897 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.897 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-30T08:44:51.897 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.897 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.897 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.897 INFO:teuthology.orchestra.run.smithi032.stdout: "registry_password": { 2024-04-30T08:44:51.897 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "registry_password", 2024-04-30T08:44:51.897 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.897 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.897 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.897 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:51.897 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.898 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.898 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.898 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-30T08:44:51.898 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.898 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.898 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.898 INFO:teuthology.orchestra.run.smithi032.stdout: "registry_url": { 2024-04-30T08:44:51.898 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "registry_url", 2024-04-30T08:44:51.898 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.898 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.898 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.898 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:51.899 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.899 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.899 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.899 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-30T08:44:51.899 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.899 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.899 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.899 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.899 INFO:teuthology.orchestra.run.smithi032.stdout: "registry_username": { 2024-04-30T08:44:51.899 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "registry_username", 2024-04-30T08:44:51.899 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.899 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.899 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.899 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:51.900 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.900 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.900 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.900 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-30T08:44:51.900 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.900 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.900 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.900 INFO:teuthology.orchestra.run.smithi032.stdout: "secure_monitoring_stack": { 2024-04-30T08:44:51.900 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "secure_monitoring_stack", 2024-04-30T08:44:51.900 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.900 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.900 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.900 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:51.901 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.901 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.901 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.901 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-30T08:44:51.901 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.901 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.901 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.901 INFO:teuthology.orchestra.run.smithi032.stdout: "service_discovery_port": { 2024-04-30T08:44:51.901 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "service_discovery_port", 2024-04-30T08:44:51.901 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:51.901 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.901 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.901 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8765", 2024-04-30T08:44:51.901 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.902 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.902 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.902 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "cephadm service discovery port", 2024-04-30T08:44:51.902 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.902 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.902 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.902 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:51.902 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:51.902 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:51.902 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:51.902 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.902 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:51.903 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.903 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.903 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.903 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.903 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.903 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.903 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.903 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.903 INFO:teuthology.orchestra.run.smithi032.stdout: "ssh_config_file": { 2024-04-30T08:44:51.903 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ssh_config_file", 2024-04-30T08:44:51.903 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.903 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.903 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.903 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:51.904 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.904 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.904 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.904 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-30T08:44:51.904 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.904 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.904 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.904 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.904 INFO:teuthology.orchestra.run.smithi032.stdout: "use_agent": { 2024-04-30T08:44:51.904 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "use_agent", 2024-04-30T08:44:51.904 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.904 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.904 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.904 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:51.905 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.905 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.905 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.905 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-30T08:44:51.905 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.905 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.905 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.905 INFO:teuthology.orchestra.run.smithi032.stdout: "use_repo_digest": { 2024-04-30T08:44:51.905 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "use_repo_digest", 2024-04-30T08:44:51.905 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.905 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.905 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.905 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:51.906 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.906 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.906 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.906 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-30T08:44:51.906 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.906 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.906 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.906 INFO:teuthology.orchestra.run.smithi032.stdout: "warn_on_failed_host_check": { 2024-04-30T08:44:51.906 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "warn_on_failed_host_check", 2024-04-30T08:44:51.906 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.906 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.906 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.907 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:51.907 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.907 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.907 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.907 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "raise a health warning if the host check fails", 2024-04-30T08:44:51.907 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.907 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.907 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.907 INFO:teuthology.orchestra.run.smithi032.stdout: "warn_on_stray_daemons": { 2024-04-30T08:44:51.907 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "warn_on_stray_daemons", 2024-04-30T08:44:51.907 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.908 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.908 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.908 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:51.908 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.908 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.908 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.908 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-30T08:44:51.908 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.908 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.908 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.908 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.908 INFO:teuthology.orchestra.run.smithi032.stdout: "warn_on_stray_hosts": { 2024-04-30T08:44:51.908 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "warn_on_stray_hosts", 2024-04-30T08:44:51.909 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.909 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.909 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.909 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:51.909 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.909 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.909 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.909 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-30T08:44:51.909 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.909 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.909 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.909 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:51.909 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:51.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.910 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:51.910 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crash", 2024-04-30T08:44:51.910 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:51.910 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:51.910 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:51.910 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:51.910 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:51.910 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.910 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.910 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.910 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:51.910 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.910 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.910 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:51.911 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:51.911 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:51.911 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:51.911 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:51.911 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:51.911 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:51.911 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:51.911 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.911 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.911 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.911 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.911 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.911 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:51.911 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:51.912 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.912 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.912 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.912 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:51.912 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.912 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.912 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.912 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.912 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.912 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.912 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.912 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.912 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:51.912 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:51.912 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.912 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.913 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.913 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:51.913 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.913 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.913 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:51.913 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:51.913 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:51.913 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:51.913 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:51.913 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:51.913 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:51.913 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:51.913 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.913 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.913 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.914 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.914 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.914 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:51.914 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:51.914 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.914 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.914 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.914 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:51.914 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.914 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.914 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.914 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.914 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.914 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.914 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.915 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.915 INFO:teuthology.orchestra.run.smithi032.stdout: "retain_interval": { 2024-04-30T08:44:51.915 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "retain_interval", 2024-04-30T08:44:51.915 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:51.915 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.915 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.915 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "31536000", 2024-04-30T08:44:51.915 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.915 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.915 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.915 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-30T08:44:51.915 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.915 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.915 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.916 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:51.916 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:51.916 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:51.916 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:51.916 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.916 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:51.916 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.916 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.916 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.916 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.916 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.916 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.916 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.917 INFO:teuthology.orchestra.run.smithi032.stdout: "warn_recent_interval": { 2024-04-30T08:44:51.917 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "warn_recent_interval", 2024-04-30T08:44:51.917 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:51.917 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.917 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.917 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "1209600", 2024-04-30T08:44:51.917 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.917 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.917 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.917 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-30T08:44:51.917 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.917 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.917 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.917 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:51.917 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:51.918 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.918 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:51.918 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "dashboard", 2024-04-30T08:44:51.918 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:51.918 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:51.918 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:51.918 INFO:teuthology.orchestra.run.smithi032.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-30T08:44:51.918 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-30T08:44:51.918 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:51.918 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.918 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.918 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-30T08:44:51.918 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.918 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.919 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.919 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.919 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.919 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.919 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.919 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.919 INFO:teuthology.orchestra.run.smithi032.stdout: "ALERTMANAGER_API_HOST": { 2024-04-30T08:44:51.919 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-30T08:44:51.919 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.919 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.919 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.919 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:51.919 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.919 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.919 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.920 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.920 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.920 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.920 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.920 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.920 INFO:teuthology.orchestra.run.smithi032.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-30T08:44:51.920 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-30T08:44:51.920 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.920 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.920 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.920 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:51.920 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.920 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.920 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.921 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.921 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.921 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.921 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.921 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.921 INFO:teuthology.orchestra.run.smithi032.stdout: "AUDIT_API_ENABLED": { 2024-04-30T08:44:51.921 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "AUDIT_API_ENABLED", 2024-04-30T08:44:51.921 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.921 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.921 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.921 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:51.921 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.921 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.921 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.922 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.922 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.922 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.922 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.922 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.922 INFO:teuthology.orchestra.run.smithi032.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-30T08:44:51.922 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-30T08:44:51.922 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.922 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.922 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.922 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:51.922 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.922 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.923 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.923 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.923 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.923 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.923 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.923 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.923 INFO:teuthology.orchestra.run.smithi032.stdout: "ENABLE_BROWSABLE_API": { 2024-04-30T08:44:51.923 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-30T08:44:51.923 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.923 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.923 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.923 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:51.923 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.923 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.923 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.924 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.924 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.924 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.924 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.924 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.924 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-30T08:44:51.924 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-30T08:44:51.924 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.924 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.924 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.924 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:51.924 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.924 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.924 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.925 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.925 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.925 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.925 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.925 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-30T08:44:51.925 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-30T08:44:51.925 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.925 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.925 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.925 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:51.925 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.925 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.925 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.926 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.926 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.926 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.926 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.926 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-30T08:44:51.926 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-30T08:44:51.926 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.926 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.926 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.926 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:51.926 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.926 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.926 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.927 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.927 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.927 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.927 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.927 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.927 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-30T08:44:51.927 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-30T08:44:51.927 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.927 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.927 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.927 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:51.927 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.927 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.927 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.928 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.929 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.929 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.929 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.929 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-30T08:44:51.929 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-30T08:44:51.929 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.930 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.930 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.930 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:51.930 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.930 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.930 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.930 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.930 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.930 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.930 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.930 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-30T08:44:51.930 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-30T08:44:51.931 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.931 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.931 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.931 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:51.931 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.931 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.931 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.931 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.931 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.931 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.931 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.931 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.931 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-30T08:44:51.931 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-30T08:44:51.932 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.932 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.932 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.932 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:51.932 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.932 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.932 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.932 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.932 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.932 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.932 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.932 INFO:teuthology.orchestra.run.smithi032.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-30T08:44:51.932 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-30T08:44:51.933 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.933 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.933 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.933 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:51.933 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.933 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.933 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.933 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.933 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.933 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.933 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.933 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.933 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_API_PASSWORD": { 2024-04-30T08:44:51.933 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-30T08:44:51.934 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.934 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.934 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.934 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "admin", 2024-04-30T08:44:51.934 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.934 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.934 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.934 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.934 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.934 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.934 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.934 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.934 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-30T08:44:51.934 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-30T08:44:51.935 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.935 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.935 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.935 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:51.935 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.935 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.935 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.935 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.935 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.935 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.935 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.935 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.935 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_API_URL": { 2024-04-30T08:44:51.936 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_API_URL", 2024-04-30T08:44:51.936 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.936 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.936 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.936 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:51.936 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.936 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.936 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.936 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.936 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.936 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.936 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.936 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_API_USERNAME": { 2024-04-30T08:44:51.937 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-30T08:44:51.937 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.937 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.937 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.937 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "admin", 2024-04-30T08:44:51.937 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.937 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.937 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.937 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.937 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.937 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.937 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.937 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-30T08:44:51.938 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-30T08:44:51.938 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.938 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.938 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.938 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:51.938 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.938 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.938 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.938 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.938 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.938 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.938 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.938 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.938 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-30T08:44:51.939 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-30T08:44:51.939 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.939 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.939 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.939 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:51.939 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.939 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.939 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.939 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.939 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.939 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.939 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.939 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.940 INFO:teuthology.orchestra.run.smithi032.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-30T08:44:51.940 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-30T08:44:51.940 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.940 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.940 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.940 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:51.940 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.940 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.940 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.940 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.940 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.940 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.940 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.941 INFO:teuthology.orchestra.run.smithi032.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-30T08:44:51.941 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-30T08:44:51.941 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.941 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.941 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.941 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:51.941 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.941 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.941 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.941 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.941 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.941 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.941 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.942 INFO:teuthology.orchestra.run.smithi032.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-30T08:44:51.942 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-30T08:44:51.942 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.942 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.942 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.942 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "[]", 2024-04-30T08:44:51.942 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.942 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.942 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.942 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.942 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.942 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.942 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.943 INFO:teuthology.orchestra.run.smithi032.stdout: "MULTICLUSTER_CONFIG": { 2024-04-30T08:44:51.943 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-30T08:44:51.943 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.943 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.943 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.943 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "{}", 2024-04-30T08:44:51.943 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.943 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.943 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.943 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.943 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.943 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.943 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.944 INFO:teuthology.orchestra.run.smithi032.stdout: "PROMETHEUS_API_HOST": { 2024-04-30T08:44:51.944 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-30T08:44:51.944 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.944 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.944 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.944 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:51.944 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.944 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.944 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.944 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.944 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.944 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.944 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.944 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.945 INFO:teuthology.orchestra.run.smithi032.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-30T08:44:51.945 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-30T08:44:51.945 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.945 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.945 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.945 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:51.945 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.945 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.945 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.945 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.945 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.945 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.945 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.946 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-30T08:44:51.946 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-30T08:44:51.946 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.946 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.946 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.946 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:51.946 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.946 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.946 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.946 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.946 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.946 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.946 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.947 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-30T08:44:51.947 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-30T08:44:51.947 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.947 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.947 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.947 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:51.947 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.947 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.947 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.947 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.947 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.947 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.947 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.947 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.948 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-30T08:44:51.948 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-30T08:44:51.948 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.948 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.948 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.948 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:51.948 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.948 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.948 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.948 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.948 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.948 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.948 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.949 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.949 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-30T08:44:51.949 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-30T08:44:51.949 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.949 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.949 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.949 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:51.949 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.949 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.949 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.949 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.949 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.949 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.950 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.950 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-30T08:44:51.950 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-30T08:44:51.950 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.950 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.950 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.950 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:51.950 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.950 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.950 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.950 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.950 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.950 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.951 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.951 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-30T08:44:51.951 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-30T08:44:51.951 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.951 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.951 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.951 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:51.951 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.951 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.951 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.951 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.951 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.952 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.952 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.952 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-30T08:44:51.952 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-30T08:44:51.952 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.952 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.952 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.952 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:51.952 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.952 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.952 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.952 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.953 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.953 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.953 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.953 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_ENABLED": { 2024-04-30T08:44:51.953 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-30T08:44:51.953 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.953 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.953 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.953 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:51.953 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.953 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.953 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.953 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.954 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.954 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.954 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.954 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-30T08:44:51.954 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-30T08:44:51.954 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.954 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.954 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.954 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-30T08:44:51.954 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.954 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.954 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.955 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.955 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.955 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.955 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.955 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.955 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-30T08:44:51.955 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-30T08:44:51.955 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:51.955 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.955 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.955 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-30T08:44:51.955 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.955 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.955 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.956 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.956 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.956 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.956 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.956 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-30T08:44:51.956 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-30T08:44:51.956 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:51.956 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.956 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.956 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8", 2024-04-30T08:44:51.956 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.956 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.956 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.957 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.957 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.957 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.957 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.957 INFO:teuthology.orchestra.run.smithi032.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-30T08:44:51.957 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-30T08:44:51.957 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:51.957 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.957 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.957 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "45", 2024-04-30T08:44:51.957 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.957 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.957 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.958 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.958 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.958 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.958 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.958 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.958 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_ACCESS_KEY": { 2024-04-30T08:44:51.958 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-30T08:44:51.958 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.958 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.958 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.958 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:51.958 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.958 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.959 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.959 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.959 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.959 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.959 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.959 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-30T08:44:51.959 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-30T08:44:51.959 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.959 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.959 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.959 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "admin", 2024-04-30T08:44:51.959 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.959 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.959 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.960 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.960 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.960 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.960 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.960 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_SECRET_KEY": { 2024-04-30T08:44:51.960 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-30T08:44:51.960 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.960 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.960 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.960 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:51.960 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.960 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.960 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.961 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.961 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.961 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.961 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.961 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_SSL_VERIFY": { 2024-04-30T08:44:51.961 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-30T08:44:51.961 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.961 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.961 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.961 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:51.962 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.962 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.962 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.962 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.962 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.962 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.962 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.962 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.962 INFO:teuthology.orchestra.run.smithi032.stdout: "UNSAFE_TLS_v1_2": { 2024-04-30T08:44:51.962 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-30T08:44:51.962 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.962 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.962 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.963 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:51.963 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.963 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.963 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.963 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.963 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.963 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.963 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.963 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.963 INFO:teuthology.orchestra.run.smithi032.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-30T08:44:51.963 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-30T08:44:51.964 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:51.964 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.964 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.964 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:51.964 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.964 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.964 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.964 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.964 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.964 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.964 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.964 INFO:teuthology.orchestra.run.smithi032.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-30T08:44:51.964 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-30T08:44:51.965 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:51.965 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.965 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.965 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-30T08:44:51.965 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.965 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.965 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.965 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.965 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.965 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.965 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.965 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.965 INFO:teuthology.orchestra.run.smithi032.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-30T08:44:51.965 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-30T08:44:51.966 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:51.966 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.966 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.966 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5", 2024-04-30T08:44:51.966 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.966 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.966 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.966 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.966 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.966 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.966 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.966 INFO:teuthology.orchestra.run.smithi032.stdout: "cross_origin_url": { 2024-04-30T08:44:51.967 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cross_origin_url", 2024-04-30T08:44:51.967 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.967 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.967 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.967 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:51.967 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.967 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.967 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.967 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.967 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.967 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.967 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.967 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.968 INFO:teuthology.orchestra.run.smithi032.stdout: "crt_file": { 2024-04-30T08:44:51.968 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crt_file", 2024-04-30T08:44:51.968 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.968 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.968 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.968 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:51.968 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.968 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.968 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.968 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.968 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.968 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.968 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.968 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.969 INFO:teuthology.orchestra.run.smithi032.stdout: "debug": { 2024-04-30T08:44:51.969 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "debug", 2024-04-30T08:44:51.969 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.969 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.969 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.969 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:51.969 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.969 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.969 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.969 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Enable/disable debug options", 2024-04-30T08:44:51.969 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.969 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.969 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.969 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.970 INFO:teuthology.orchestra.run.smithi032.stdout: "jwt_token_ttl": { 2024-04-30T08:44:51.970 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "jwt_token_ttl", 2024-04-30T08:44:51.970 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:51.970 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.970 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.970 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "28800", 2024-04-30T08:44:51.970 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.970 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.970 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.970 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.970 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.970 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.970 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.971 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.971 INFO:teuthology.orchestra.run.smithi032.stdout: "key_file": { 2024-04-30T08:44:51.971 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "key_file", 2024-04-30T08:44:51.971 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.971 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.971 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.971 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:51.971 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.971 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.971 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.971 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.971 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.971 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.971 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.971 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.972 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:51.972 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:51.972 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.972 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.972 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.972 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:51.972 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.972 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.972 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:51.972 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:51.972 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:51.972 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:51.972 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:51.972 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:51.972 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:51.973 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:51.973 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.973 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.973 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.973 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.973 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.973 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:51.973 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:51.973 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.973 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.973 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.973 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:51.973 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.974 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.974 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.974 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.974 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.974 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.974 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.974 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.974 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:51.974 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:51.974 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.974 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.974 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.974 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:51.974 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.975 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.975 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:51.975 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:51.975 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:51.975 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:51.975 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:51.975 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:51.975 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:51.975 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:51.975 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.975 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.975 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.975 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.975 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.976 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:51.976 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:51.976 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.976 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.976 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.976 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:51.976 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.976 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.976 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.976 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.976 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.976 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.976 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.976 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.977 INFO:teuthology.orchestra.run.smithi032.stdout: "motd": { 2024-04-30T08:44:51.977 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "motd", 2024-04-30T08:44:51.977 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.977 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.977 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.977 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:51.977 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.977 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.977 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.977 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "The message of the day", 2024-04-30T08:44:51.977 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.977 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.977 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.978 INFO:teuthology.orchestra.run.smithi032.stdout: "redirect_resolve_ip_addr": { 2024-04-30T08:44:51.978 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "redirect_resolve_ip_addr", 2024-04-30T08:44:51.978 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.978 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.978 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.978 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:51.978 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.978 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.978 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.978 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.978 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.978 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.978 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.979 INFO:teuthology.orchestra.run.smithi032.stdout: "server_addr": { 2024-04-30T08:44:51.979 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_addr", 2024-04-30T08:44:51.979 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.979 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.979 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.979 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "::", 2024-04-30T08:44:51.979 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.979 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.979 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.979 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.979 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.979 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.979 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.979 INFO:teuthology.orchestra.run.smithi032.stdout: "server_port": { 2024-04-30T08:44:51.980 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_port", 2024-04-30T08:44:51.980 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:51.980 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.980 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.980 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8080", 2024-04-30T08:44:51.980 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.980 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.980 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.980 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.980 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.980 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.980 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.980 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.981 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:51.981 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:51.981 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:51.981 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:51.981 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.981 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:51.981 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.981 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.981 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.981 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.981 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.981 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.981 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.981 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.982 INFO:teuthology.orchestra.run.smithi032.stdout: "ssl": { 2024-04-30T08:44:51.982 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ssl", 2024-04-30T08:44:51.982 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.982 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.982 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.982 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:51.982 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.982 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.982 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.982 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.982 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.982 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.982 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.983 INFO:teuthology.orchestra.run.smithi032.stdout: "ssl_server_port": { 2024-04-30T08:44:51.983 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ssl_server_port", 2024-04-30T08:44:51.983 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:51.983 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.983 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.983 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8443", 2024-04-30T08:44:51.983 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.983 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.983 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.983 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.983 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.983 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.983 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.984 INFO:teuthology.orchestra.run.smithi032.stdout: "standby_behaviour": { 2024-04-30T08:44:51.984 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "standby_behaviour", 2024-04-30T08:44:51.984 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.984 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.984 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.984 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "redirect", 2024-04-30T08:44:51.984 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.984 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.984 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:51.984 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:51.984 INFO:teuthology.orchestra.run.smithi032.stdout: "redirect" 2024-04-30T08:44:51.984 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:51.984 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.984 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.985 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.985 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.985 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.985 INFO:teuthology.orchestra.run.smithi032.stdout: "standby_error_status_code": { 2024-04-30T08:44:51.985 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "standby_error_status_code", 2024-04-30T08:44:51.985 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:51.985 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.985 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.985 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "500", 2024-04-30T08:44:51.985 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "400", 2024-04-30T08:44:51.985 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "599", 2024-04-30T08:44:51.985 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.985 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.985 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.986 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.986 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.986 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.986 INFO:teuthology.orchestra.run.smithi032.stdout: "url_prefix": { 2024-04-30T08:44:51.986 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "url_prefix", 2024-04-30T08:44:51.986 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.986 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.986 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:51.986 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:51.986 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.986 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.986 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.986 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.987 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.987 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.987 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.987 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:51.987 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:51.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.987 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:51.987 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "devicehealth", 2024-04-30T08:44:51.987 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:51.987 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:51.987 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:51.987 INFO:teuthology.orchestra.run.smithi032.stdout: "enable_monitoring": { 2024-04-30T08:44:51.987 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "enable_monitoring", 2024-04-30T08:44:51.987 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.988 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.988 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.988 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:51.988 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.988 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.988 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.988 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "monitor device health metrics", 2024-04-30T08:44:51.988 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.988 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.988 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.988 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.988 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:51.988 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:51.988 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.988 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.989 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.989 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:51.989 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.989 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.989 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:51.989 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:51.989 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:51.989 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:51.989 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:51.989 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:51.989 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:51.989 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:51.989 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.990 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.990 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.990 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.990 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.990 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:51.990 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:51.990 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.990 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.990 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.990 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:51.990 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.990 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.990 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.990 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.991 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.991 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.991 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.991 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.991 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:51.991 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:51.991 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.991 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.991 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.991 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:51.991 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.991 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.991 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:51.992 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:51.992 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:51.992 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:51.992 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:51.992 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:51.992 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:51.992 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:51.992 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.992 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.992 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.992 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.992 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.992 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:51.992 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:51.993 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.993 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.993 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.993 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:51.993 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.993 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.993 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.993 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:51.993 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.993 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.993 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.993 INFO:teuthology.orchestra.run.smithi032.stdout: "mark_out_threshold": { 2024-04-30T08:44:51.994 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mark_out_threshold", 2024-04-30T08:44:51.994 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:51.994 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.994 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.994 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "2419200", 2024-04-30T08:44:51.994 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.994 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.994 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.994 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-30T08:44:51.994 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.994 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.994 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.994 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.994 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": { 2024-04-30T08:44:51.995 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "pool_name", 2024-04-30T08:44:51.995 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:51.995 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.995 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.995 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "device_health_metrics", 2024-04-30T08:44:51.995 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.995 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.995 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.995 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-30T08:44:51.995 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.995 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.995 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.995 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.996 INFO:teuthology.orchestra.run.smithi032.stdout: "retention_period": { 2024-04-30T08:44:51.996 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "retention_period", 2024-04-30T08:44:51.996 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:51.996 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.996 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.996 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "15552000", 2024-04-30T08:44:51.996 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.996 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.996 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.996 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how long to retain device health metrics", 2024-04-30T08:44:51.996 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.996 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.996 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.997 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.997 INFO:teuthology.orchestra.run.smithi032.stdout: "scrape_frequency": { 2024-04-30T08:44:51.997 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "scrape_frequency", 2024-04-30T08:44:51.997 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:51.997 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.997 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.997 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "86400", 2024-04-30T08:44:51.997 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.997 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.997 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.997 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-30T08:44:51.997 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.997 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.997 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.998 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.998 INFO:teuthology.orchestra.run.smithi032.stdout: "self_heal": { 2024-04-30T08:44:51.998 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "self_heal", 2024-04-30T08:44:51.998 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:51.998 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.998 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.998 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:51.998 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.998 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.998 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.998 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-30T08:44:51.998 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.998 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:51.998 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:51.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:51.999 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2024-04-30T08:44:51.999 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2024-04-30T08:44:51.999 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:51.999 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:51.999 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:51.999 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2024-04-30T08:44:51.999 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:51.999 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:51.999 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:51.999 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to wake up and check device health", 2024-04-30T08:44:51.999 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:51.999 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.000 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.000 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.000 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.000 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.000 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.000 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.000 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.000 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.000 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.000 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.000 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.000 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.000 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.000 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.001 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.001 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.001 INFO:teuthology.orchestra.run.smithi032.stdout: "warn_threshold": { 2024-04-30T08:44:52.001 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "warn_threshold", 2024-04-30T08:44:52.001 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.001 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.001 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.001 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "7257600", 2024-04-30T08:44:52.001 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.001 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.001 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.001 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-30T08:44:52.001 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.001 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.002 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.002 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.002 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.002 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.002 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.002 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "diskprediction_local", 2024-04-30T08:44:52.002 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.002 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.002 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.002 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.002 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.002 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.002 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.003 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.003 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.003 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.003 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.003 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.003 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.003 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.003 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.003 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.003 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.003 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.003 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.003 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.004 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.004 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.004 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.004 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.004 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.004 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.004 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.004 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.004 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.004 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.004 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.004 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.004 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.004 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.005 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.005 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.005 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.005 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.005 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.005 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.005 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.005 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.005 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.005 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.005 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.005 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.005 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.006 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.006 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.006 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.006 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.006 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.006 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.006 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.006 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.006 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.006 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.006 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.006 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.006 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.006 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.007 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.007 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.007 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.007 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.007 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.007 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.007 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.007 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.007 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.007 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.007 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.007 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.007 INFO:teuthology.orchestra.run.smithi032.stdout: "predict_interval": { 2024-04-30T08:44:52.008 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "predict_interval", 2024-04-30T08:44:52.008 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.008 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.008 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.008 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "86400", 2024-04-30T08:44:52.008 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.008 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.008 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.008 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.008 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.008 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.008 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.009 INFO:teuthology.orchestra.run.smithi032.stdout: "predictor_model": { 2024-04-30T08:44:52.009 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "predictor_model", 2024-04-30T08:44:52.009 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.009 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.009 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.009 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "prophetstor", 2024-04-30T08:44:52.009 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.009 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.009 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.009 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.009 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.009 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.009 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.010 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2024-04-30T08:44:52.010 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2024-04-30T08:44:52.010 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.010 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.010 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.010 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2024-04-30T08:44:52.010 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.010 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.010 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.010 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.010 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.010 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.010 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.011 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.011 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.011 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.011 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.011 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.011 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.011 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.011 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.011 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.011 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.011 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.011 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.011 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.011 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.012 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.012 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.012 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.012 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.012 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "influx", 2024-04-30T08:44:52.012 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": false, 2024-04-30T08:44:52.012 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "influxdb python module not found", 2024-04-30T08:44:52.012 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.012 INFO:teuthology.orchestra.run.smithi032.stdout: "batch_size": { 2024-04-30T08:44:52.012 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "batch_size", 2024-04-30T08:44:52.012 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.012 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.012 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.012 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5000", 2024-04-30T08:44:52.013 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.013 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.013 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.013 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-30T08:44:52.013 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.013 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.013 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.013 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.013 INFO:teuthology.orchestra.run.smithi032.stdout: "database": { 2024-04-30T08:44:52.013 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "database", 2024-04-30T08:44:52.013 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.013 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.013 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.013 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "ceph", 2024-04-30T08:44:52.014 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.014 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.014 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.014 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-30T08:44:52.014 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.014 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.014 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.014 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.014 INFO:teuthology.orchestra.run.smithi032.stdout: "hostname": { 2024-04-30T08:44:52.014 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "hostname", 2024-04-30T08:44:52.014 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.014 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.014 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.015 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.015 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.015 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.015 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.015 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "InfluxDB server hostname", 2024-04-30T08:44:52.015 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.015 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.015 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.015 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.015 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2024-04-30T08:44:52.015 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2024-04-30T08:44:52.015 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.015 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.015 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.015 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "30", 2024-04-30T08:44:52.016 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "5", 2024-04-30T08:44:52.016 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.016 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.016 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-30T08:44:52.016 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.016 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.016 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.016 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.016 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.016 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.016 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.016 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.016 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.017 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.017 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.017 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.017 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.017 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.017 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.017 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.017 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.017 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.017 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.017 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.017 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.017 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.017 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.018 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.018 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.018 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.018 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.018 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.018 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.018 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.018 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.018 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.018 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.018 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.018 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.019 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.019 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.019 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.019 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.019 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.019 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.019 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.019 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.019 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.019 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.019 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.019 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.019 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.019 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.020 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.020 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.020 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.020 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.020 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.020 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.020 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.020 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.020 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.020 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.020 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.020 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.020 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.020 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.021 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.021 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.021 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.021 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.021 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.021 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.021 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.021 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.021 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.021 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.021 INFO:teuthology.orchestra.run.smithi032.stdout: "password": { 2024-04-30T08:44:52.021 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "password", 2024-04-30T08:44:52.021 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.021 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.022 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.022 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.022 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.022 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.022 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.022 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "password of InfluxDB server user", 2024-04-30T08:44:52.022 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.022 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.022 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.022 INFO:teuthology.orchestra.run.smithi032.stdout: "port": { 2024-04-30T08:44:52.022 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "port", 2024-04-30T08:44:52.022 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.023 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.023 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.023 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8086", 2024-04-30T08:44:52.023 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.023 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.023 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.023 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "InfluxDB server port", 2024-04-30T08:44:52.023 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.023 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.023 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.023 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.023 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.023 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.023 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.024 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.024 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.024 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.024 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.024 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.024 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.024 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.024 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.024 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.024 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.024 INFO:teuthology.orchestra.run.smithi032.stdout: "ssl": { 2024-04-30T08:44:52.024 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ssl", 2024-04-30T08:44:52.024 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.024 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.025 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.025 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "false", 2024-04-30T08:44:52.025 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.025 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.025 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.025 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-30T08:44:52.025 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.025 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.025 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.025 INFO:teuthology.orchestra.run.smithi032.stdout: "threads": { 2024-04-30T08:44:52.025 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "threads", 2024-04-30T08:44:52.025 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.025 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.026 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.026 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5", 2024-04-30T08:44:52.026 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "1", 2024-04-30T08:44:52.026 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "32", 2024-04-30T08:44:52.026 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.026 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-30T08:44:52.026 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.026 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.026 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.026 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.026 INFO:teuthology.orchestra.run.smithi032.stdout: "username": { 2024-04-30T08:44:52.026 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "username", 2024-04-30T08:44:52.026 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.027 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.027 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.027 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.027 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.027 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.027 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.027 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "username of InfluxDB server user", 2024-04-30T08:44:52.027 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.027 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.027 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.027 INFO:teuthology.orchestra.run.smithi032.stdout: "verify_ssl": { 2024-04-30T08:44:52.027 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "verify_ssl", 2024-04-30T08:44:52.027 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.028 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.028 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.028 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "true", 2024-04-30T08:44:52.028 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.028 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.028 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.028 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-30T08:44:52.028 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.028 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.028 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.028 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.028 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.028 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.028 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.029 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "insights", 2024-04-30T08:44:52.029 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.029 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.029 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.029 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.029 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.029 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.029 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.029 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.029 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.029 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.029 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.029 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.029 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.030 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.030 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.030 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.030 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.030 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.030 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.030 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.030 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.030 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.030 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.030 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.030 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.031 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.031 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.031 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.031 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.031 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.031 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.031 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.031 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.031 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.031 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.031 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.031 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.031 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.031 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.032 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.032 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.032 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.032 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.032 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.032 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.032 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.032 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.032 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.032 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.032 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.032 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.032 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.033 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.033 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.033 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.033 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.033 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.033 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.033 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.033 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.033 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.033 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.033 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.033 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.033 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.034 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.034 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.034 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.034 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.034 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.034 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.034 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.034 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.034 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.034 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.034 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.034 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.034 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.034 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.035 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.035 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.035 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.035 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.035 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.035 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.035 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.035 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.035 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "iostat", 2024-04-30T08:44:52.035 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.035 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.035 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.036 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.036 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.036 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.036 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.036 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.036 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.036 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.036 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.036 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.036 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.036 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.036 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.036 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.036 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.037 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.037 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.037 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.037 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.037 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.037 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.037 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.037 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.037 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.037 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.037 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.037 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.037 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.037 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.038 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.038 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.038 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.038 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.038 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.038 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.038 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.038 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.038 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.038 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.038 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.038 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.038 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.039 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.039 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.039 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.039 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.039 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.039 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.039 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.039 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.039 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.039 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.039 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.039 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.039 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.040 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.040 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.040 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.040 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.040 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.040 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.040 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.040 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.040 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.040 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.040 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.040 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.040 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.041 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.041 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.041 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.041 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.041 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.041 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.041 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.041 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.041 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.041 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.041 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.041 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.041 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.041 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.042 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.042 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.042 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "k8sevents", 2024-04-30T08:44:52.042 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.042 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.042 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.042 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_event_retention_days": { 2024-04-30T08:44:52.042 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ceph_event_retention_days", 2024-04-30T08:44:52.042 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.042 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.042 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.042 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "7", 2024-04-30T08:44:52.042 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.043 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.043 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.043 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-30T08:44:52.043 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.043 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.043 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.043 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.043 INFO:teuthology.orchestra.run.smithi032.stdout: "config_check_secs": { 2024-04-30T08:44:52.043 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "config_check_secs", 2024-04-30T08:44:52.043 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.043 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.043 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.043 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-30T08:44:52.043 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "10", 2024-04-30T08:44:52.043 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.044 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.044 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-30T08:44:52.044 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.044 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.044 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.044 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.044 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.044 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.044 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.044 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.044 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.044 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.045 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.045 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.045 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.045 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.045 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.045 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.045 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.045 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.045 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.045 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.045 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.045 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.045 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.045 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.046 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.046 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.046 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.046 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.046 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.046 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.046 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.046 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.046 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.046 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.046 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.046 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.046 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.047 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.047 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.047 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.047 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.047 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.047 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.047 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.047 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.047 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.047 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.047 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.047 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.047 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.047 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.048 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.048 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.048 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.048 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.048 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.048 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.048 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.048 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.048 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.048 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.048 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.048 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.048 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.048 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.049 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.049 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.049 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.049 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.049 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.049 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.049 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.049 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.049 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.049 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.049 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.049 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.049 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.050 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.050 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.050 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.050 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.050 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.050 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.050 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.050 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.050 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.050 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.050 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.050 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "localpool", 2024-04-30T08:44:52.050 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.051 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.051 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.051 INFO:teuthology.orchestra.run.smithi032.stdout: "failure_domain": { 2024-04-30T08:44:52.051 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "failure_domain", 2024-04-30T08:44:52.051 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.051 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.051 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.051 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "host", 2024-04-30T08:44:52.051 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.051 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.051 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.051 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "failure domain for any created local pool", 2024-04-30T08:44:52.051 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-30T08:44:52.051 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.052 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.052 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.052 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.052 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.052 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.052 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.052 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.052 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.052 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.052 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.052 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.052 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.052 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.052 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.053 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.053 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.053 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.053 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.053 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.053 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.053 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.053 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.053 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.053 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.053 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.053 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.053 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.053 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.054 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.054 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.054 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.054 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.054 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.054 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.054 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.054 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.054 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.054 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.054 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.054 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.054 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.054 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.055 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.055 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.055 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.055 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.055 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.055 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.055 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.055 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.055 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.055 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.055 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.055 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.055 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.055 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.056 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.056 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.056 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.056 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.056 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.056 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.056 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.056 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.056 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.056 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.056 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.056 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.056 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.056 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.056 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.057 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.057 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": { 2024-04-30T08:44:52.057 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "min_size", 2024-04-30T08:44:52.057 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.057 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.057 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.057 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.057 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.057 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.057 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.057 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "default min_size for any created local pool", 2024-04-30T08:44:52.057 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-30T08:44:52.057 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.057 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.058 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.058 INFO:teuthology.orchestra.run.smithi032.stdout: "num_rep": { 2024-04-30T08:44:52.058 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "num_rep", 2024-04-30T08:44:52.058 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.058 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.058 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.058 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "3", 2024-04-30T08:44:52.058 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.058 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.058 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.058 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "default replica count for any created local pool", 2024-04-30T08:44:52.058 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.058 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.059 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.059 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.059 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": { 2024-04-30T08:44:52.059 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "pg_num", 2024-04-30T08:44:52.059 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.059 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.059 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.059 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "128", 2024-04-30T08:44:52.059 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.059 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.059 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.059 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "default pg_num for any created local pool", 2024-04-30T08:44:52.059 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.059 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.060 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.060 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.060 INFO:teuthology.orchestra.run.smithi032.stdout: "prefix": { 2024-04-30T08:44:52.060 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "prefix", 2024-04-30T08:44:52.060 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.060 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.060 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.060 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.060 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.060 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.060 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.060 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "name prefix for any created local pool", 2024-04-30T08:44:52.060 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.061 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.061 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.061 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.061 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.061 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.061 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.061 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.061 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.061 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.061 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.061 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.061 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.061 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.061 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.062 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.062 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.062 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.062 INFO:teuthology.orchestra.run.smithi032.stdout: "subtree": { 2024-04-30T08:44:52.062 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "subtree", 2024-04-30T08:44:52.062 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.062 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.062 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.062 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "rack", 2024-04-30T08:44:52.062 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.062 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.062 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.062 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-30T08:44:52.063 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-30T08:44:52.063 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.063 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.063 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.063 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.063 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.063 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.063 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mds_autoscaler", 2024-04-30T08:44:52.063 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.063 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.063 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.063 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.063 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.064 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.064 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.064 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.064 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.064 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.064 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.064 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.064 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.064 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.064 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.064 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.064 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.064 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.064 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.065 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.065 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.065 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.065 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.065 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.065 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.065 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.065 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.065 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.065 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.065 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.065 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.065 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.066 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.066 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.066 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.066 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.066 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.066 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.066 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.066 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.066 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.066 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.066 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.066 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.066 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.067 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.067 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.067 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.067 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.067 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.067 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.067 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.067 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.067 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.067 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.067 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.067 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.067 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.068 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.068 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.068 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.068 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.068 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.068 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.068 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.068 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.068 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.068 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.068 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.068 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.068 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.068 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.069 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.069 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.069 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.069 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.069 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.069 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.069 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.069 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.069 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.069 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.069 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.069 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.069 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.069 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.069 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.070 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.070 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.070 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mirroring", 2024-04-30T08:44:52.070 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.070 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.070 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.070 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.070 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.070 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.070 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.070 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.070 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.070 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.070 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.071 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.071 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.071 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.071 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.071 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.071 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.071 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.071 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.071 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.071 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.071 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.071 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.071 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.071 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.072 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.072 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.072 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.072 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.072 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.072 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.072 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.072 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.072 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.072 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.072 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.072 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.072 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.073 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.073 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.073 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.073 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.073 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.073 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.073 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.073 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.073 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.073 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.073 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.073 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.073 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.073 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.074 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.074 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.074 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.074 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.074 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.074 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.074 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.074 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.074 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.074 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.074 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.074 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.074 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.074 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.075 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.075 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.075 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.075 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.075 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.075 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.075 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.075 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.075 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.075 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.075 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.075 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.075 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.076 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.076 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.076 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.076 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.076 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.076 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.076 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.076 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.076 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.076 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.076 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.076 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "nfs", 2024-04-30T08:44:52.076 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.076 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.076 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.077 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.077 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.077 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.077 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.077 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.077 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.077 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.077 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.077 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.077 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.077 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.077 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.077 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.078 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.078 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.078 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.078 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.078 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.078 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.078 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.078 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.078 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.078 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.078 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.078 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.078 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.078 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.079 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.079 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.079 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.079 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.079 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.079 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.079 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.079 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.079 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.079 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.079 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.079 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.079 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.080 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.080 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.080 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.080 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.080 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.080 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.080 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.080 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.080 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.080 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.080 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.080 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.080 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.080 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.081 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.081 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.081 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.081 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.081 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.081 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.081 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.081 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.081 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.081 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.081 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.081 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.081 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.081 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.082 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.082 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.082 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.082 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.082 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.082 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.082 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.082 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.082 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.082 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.082 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.082 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.083 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.083 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.083 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.083 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.083 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.083 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.083 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.083 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "orchestrator", 2024-04-30T08:44:52.083 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.083 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.083 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.083 INFO:teuthology.orchestra.run.smithi032.stdout: "fail_fs": { 2024-04-30T08:44:52.083 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "fail_fs", 2024-04-30T08:44:52.083 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.084 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.084 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.084 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.084 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.084 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.084 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.084 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-30T08:44:52.084 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.084 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.084 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.084 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.084 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.084 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.085 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.085 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.085 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.085 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.085 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.085 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.085 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.085 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.085 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.085 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.085 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.085 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.085 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.085 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.085 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.086 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.086 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.086 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.086 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.086 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.086 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.086 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.086 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.086 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.086 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.086 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.086 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.086 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.087 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.087 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.087 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.087 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.087 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.087 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.087 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.087 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.087 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.087 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.087 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.087 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.087 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.087 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.087 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.088 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.088 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.088 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.088 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.088 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.088 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.088 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.088 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.088 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.088 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.088 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.088 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.088 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.088 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.089 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.089 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.089 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.089 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.089 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.089 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.089 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.089 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.089 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.089 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.089 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.089 INFO:teuthology.orchestra.run.smithi032.stdout: "orchestrator": { 2024-04-30T08:44:52.089 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "orchestrator", 2024-04-30T08:44:52.089 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.090 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.090 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.090 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.090 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.090 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.090 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.090 INFO:teuthology.orchestra.run.smithi032.stdout: "cephadm", 2024-04-30T08:44:52.090 INFO:teuthology.orchestra.run.smithi032.stdout: "rook", 2024-04-30T08:44:52.090 INFO:teuthology.orchestra.run.smithi032.stdout: "test_orchestrator" 2024-04-30T08:44:52.090 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.090 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Orchestrator backend", 2024-04-30T08:44:52.090 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.090 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.090 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.091 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.091 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.091 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.091 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.091 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.091 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.091 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.091 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.091 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.091 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.091 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.091 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.091 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.092 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.092 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.092 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.092 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.092 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.092 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "osd_perf_query", 2024-04-30T08:44:52.092 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.092 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.092 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.092 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.092 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.092 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.092 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.092 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.093 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.093 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.093 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.093 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.093 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.093 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.093 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.093 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.093 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.093 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.093 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.093 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.093 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.093 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.094 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.094 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.094 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.094 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.094 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.094 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.094 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.094 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.094 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.094 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.094 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.094 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.094 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.094 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.095 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.095 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.095 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.095 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.095 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.095 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.095 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.095 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.095 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.095 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.095 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.095 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.095 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.096 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.096 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.096 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.096 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.096 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.096 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.096 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.096 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.096 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.096 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.096 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.096 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.096 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.096 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.097 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.097 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.097 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.097 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.097 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.097 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.097 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.097 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.097 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.097 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.097 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.097 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.097 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.098 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.098 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.098 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.098 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.098 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.098 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.098 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.098 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.098 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.098 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.098 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.098 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.098 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.099 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.099 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "osd_support", 2024-04-30T08:44:52.099 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.099 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.099 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.099 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.099 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.099 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.099 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.099 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.099 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.099 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.099 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.099 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.100 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.100 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.100 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.100 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.100 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.100 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.100 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.100 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.100 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.100 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.100 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.100 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.100 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.101 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.101 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.101 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.101 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.101 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.101 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.101 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.101 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.101 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.101 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.101 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.101 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.101 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.101 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.102 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.102 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.102 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.102 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.102 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.102 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.102 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.102 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.102 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.102 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.102 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.102 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.102 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.102 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.103 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.103 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.103 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.103 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.103 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.103 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.103 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.103 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.103 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.103 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.103 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.103 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.103 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.104 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.104 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.104 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.104 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.104 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.104 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.104 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.104 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.104 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.104 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.104 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.104 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.104 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.104 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.105 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.105 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.105 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.105 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.105 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.105 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.105 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.105 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.105 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.105 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.105 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "pg_autoscaler", 2024-04-30T08:44:52.105 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.105 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.105 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.106 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.106 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.106 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.106 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.106 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.106 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.106 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.106 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.106 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.106 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.106 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.106 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.106 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.107 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.107 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.107 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.107 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.107 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.107 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.107 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.107 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.107 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.107 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.107 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.107 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.107 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.108 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.108 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.108 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.108 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.108 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.108 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.108 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.108 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.108 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.108 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.108 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.108 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.108 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.108 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.109 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.109 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.109 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.109 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.109 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.109 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.109 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.109 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.109 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.109 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.109 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.109 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.109 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.109 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.110 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.110 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.110 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.110 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.110 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.110 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.110 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.110 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.110 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.110 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.110 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.110 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.110 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.110 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.111 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.111 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.111 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2024-04-30T08:44:52.111 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2024-04-30T08:44:52.111 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.111 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.111 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.111 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2024-04-30T08:44:52.111 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.111 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.111 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.111 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.111 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.111 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.112 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.112 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.112 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.112 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.112 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.112 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.112 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.112 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.112 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.112 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.112 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.112 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.112 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.112 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.113 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.113 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.113 INFO:teuthology.orchestra.run.smithi032.stdout: "threshold": { 2024-04-30T08:44:52.113 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "threshold", 2024-04-30T08:44:52.113 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-30T08:44:52.113 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.113 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.113 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "3.0", 2024-04-30T08:44:52.113 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "1.0", 2024-04-30T08:44:52.113 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.113 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.113 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "scaling threshold", 2024-04-30T08:44:52.113 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-30T08:44:52.113 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.114 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.114 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.114 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.114 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.114 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.114 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "progress", 2024-04-30T08:44:52.114 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.114 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.114 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.114 INFO:teuthology.orchestra.run.smithi032.stdout: "allow_pg_recovery_event": { 2024-04-30T08:44:52.114 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "allow_pg_recovery_event", 2024-04-30T08:44:52.114 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.114 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.115 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.115 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.115 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.115 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.115 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.115 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-30T08:44:52.115 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.115 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.115 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.115 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.115 INFO:teuthology.orchestra.run.smithi032.stdout: "enabled": { 2024-04-30T08:44:52.115 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "enabled", 2024-04-30T08:44:52.115 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.116 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.116 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.116 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.116 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.116 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.116 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.116 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.116 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.116 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.116 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.116 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.116 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.116 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.117 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.117 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.117 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.117 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.117 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.117 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.117 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.117 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.117 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.117 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.117 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.117 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.117 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.117 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.117 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.118 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.118 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.118 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.118 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.118 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.118 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.118 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.118 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.118 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.118 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.118 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.118 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.118 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.118 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.119 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.119 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.119 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.119 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.119 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.119 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.119 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.119 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.119 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.119 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.119 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.119 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.119 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.120 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.120 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.120 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.120 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.120 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.120 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.120 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.120 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.120 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.120 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.120 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.120 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.120 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.120 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.120 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.121 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.121 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.121 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.121 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.121 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.121 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.121 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.121 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.121 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.121 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.121 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.121 INFO:teuthology.orchestra.run.smithi032.stdout: "max_completed_events": { 2024-04-30T08:44:52.121 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "max_completed_events", 2024-04-30T08:44:52.121 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.122 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.122 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.122 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "50", 2024-04-30T08:44:52.122 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.122 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.122 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.122 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "number of past completed events to remember", 2024-04-30T08:44:52.122 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.122 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.122 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.122 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.122 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2024-04-30T08:44:52.122 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2024-04-30T08:44:52.123 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.123 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.123 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.123 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5", 2024-04-30T08:44:52.123 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.123 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.123 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.123 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how long the module is going to sleep", 2024-04-30T08:44:52.123 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.123 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.123 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.123 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.123 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.123 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.123 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.124 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.124 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.124 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.124 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.124 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.124 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.124 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.124 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.124 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.124 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.124 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.124 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.124 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.124 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.125 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "prometheus", 2024-04-30T08:44:52.125 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.125 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.125 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.125 INFO:teuthology.orchestra.run.smithi032.stdout: "cache": { 2024-04-30T08:44:52.125 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cache", 2024-04-30T08:44:52.125 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.125 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.125 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.125 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.125 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.125 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.125 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.125 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.125 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.126 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.126 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.126 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.126 INFO:teuthology.orchestra.run.smithi032.stdout: "exclude_perf_counters": { 2024-04-30T08:44:52.126 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "exclude_perf_counters", 2024-04-30T08:44:52.126 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.126 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.126 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.126 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.126 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.126 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.126 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.126 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-30T08:44:52.127 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-30T08:44:52.127 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.127 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.127 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.127 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.127 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.127 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.127 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.127 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.127 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.127 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.127 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.127 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.127 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.128 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.128 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.128 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.128 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.128 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.128 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.128 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.128 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.128 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.128 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.129 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.129 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.129 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.129 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.129 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.129 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.129 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.129 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.129 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.129 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.129 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.129 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.129 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.129 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.130 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.130 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.130 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.130 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.130 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.130 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.130 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.130 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.130 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.130 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.130 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.130 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.130 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.131 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.131 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.131 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.131 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.131 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.131 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.131 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.131 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.133 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.133 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.133 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.133 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.133 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.133 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.133 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.133 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.134 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.134 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.134 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.134 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.134 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.134 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.134 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.134 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_stats_pools": { 2024-04-30T08:44:52.134 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rbd_stats_pools", 2024-04-30T08:44:52.134 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.134 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.134 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.134 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.135 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.135 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.135 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.135 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.135 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.135 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.135 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.135 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.135 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-30T08:44:52.135 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-30T08:44:52.135 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.135 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.135 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.135 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "300", 2024-04-30T08:44:52.136 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.136 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.136 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.136 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.136 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.136 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.136 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.136 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.136 INFO:teuthology.orchestra.run.smithi032.stdout: "scrape_interval": { 2024-04-30T08:44:52.136 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "scrape_interval", 2024-04-30T08:44:52.136 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-30T08:44:52.136 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.136 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.136 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "15.0", 2024-04-30T08:44:52.137 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.137 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.137 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.137 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.137 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.137 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.137 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.137 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.137 INFO:teuthology.orchestra.run.smithi032.stdout: "server_addr": { 2024-04-30T08:44:52.137 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_addr", 2024-04-30T08:44:52.137 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.137 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.137 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.138 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "::", 2024-04-30T08:44:52.138 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.138 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.138 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.138 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-30T08:44:52.138 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.138 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.138 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.138 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.138 INFO:teuthology.orchestra.run.smithi032.stdout: "server_port": { 2024-04-30T08:44:52.138 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_port", 2024-04-30T08:44:52.138 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.138 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.138 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.139 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "9283", 2024-04-30T08:44:52.139 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.139 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.139 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.139 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-30T08:44:52.139 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.139 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.139 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.139 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.139 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.139 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.139 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.139 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.140 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.140 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.140 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.140 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.140 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.140 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.140 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.140 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.140 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.140 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.140 INFO:teuthology.orchestra.run.smithi032.stdout: "stale_cache_strategy": { 2024-04-30T08:44:52.140 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "stale_cache_strategy", 2024-04-30T08:44:52.140 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.140 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.141 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.141 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "log", 2024-04-30T08:44:52.141 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.141 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.141 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.141 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.141 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.141 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.141 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.141 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.141 INFO:teuthology.orchestra.run.smithi032.stdout: "standby_behaviour": { 2024-04-30T08:44:52.141 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "standby_behaviour", 2024-04-30T08:44:52.141 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.141 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.142 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.142 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "default", 2024-04-30T08:44:52.142 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.142 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.142 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.142 INFO:teuthology.orchestra.run.smithi032.stdout: "default", 2024-04-30T08:44:52.142 INFO:teuthology.orchestra.run.smithi032.stdout: "error" 2024-04-30T08:44:52.142 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.142 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.142 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.142 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.142 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.142 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.143 INFO:teuthology.orchestra.run.smithi032.stdout: "standby_error_status_code": { 2024-04-30T08:44:52.143 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "standby_error_status_code", 2024-04-30T08:44:52.143 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.143 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.143 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.143 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "500", 2024-04-30T08:44:52.143 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "400", 2024-04-30T08:44:52.143 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "599", 2024-04-30T08:44:52.143 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.143 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.143 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.143 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.143 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.143 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.144 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.144 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.144 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.144 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rbd_support", 2024-04-30T08:44:52.144 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.144 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.144 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.144 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.144 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.144 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.144 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.144 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.144 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.145 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.145 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.145 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.145 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.145 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.145 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.145 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.145 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.145 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.145 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.145 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.145 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.145 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.145 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.145 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.146 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.146 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.146 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.146 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.146 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.146 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.146 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.146 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.146 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.146 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.146 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.146 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.146 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.147 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.147 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.147 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.147 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.147 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.147 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.147 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.147 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.147 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.147 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.147 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.147 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.147 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.147 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.147 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.148 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.148 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.148 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.148 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.148 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.148 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.148 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.148 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.148 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.148 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.148 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.148 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.148 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.148 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.148 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.149 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.149 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.149 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.149 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.149 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.149 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.149 INFO:teuthology.orchestra.run.smithi032.stdout: "max_concurrent_snap_create": { 2024-04-30T08:44:52.149 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "max_concurrent_snap_create", 2024-04-30T08:44:52.149 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.149 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.149 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.149 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-30T08:44:52.149 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.149 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.150 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.150 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.150 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.150 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.150 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.150 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.150 INFO:teuthology.orchestra.run.smithi032.stdout: "mirror_snapshot_schedule": { 2024-04-30T08:44:52.150 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mirror_snapshot_schedule", 2024-04-30T08:44:52.150 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.150 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.150 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.150 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.150 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.150 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.151 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.151 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.151 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.151 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.151 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.151 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.151 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.151 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.151 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.151 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.151 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.151 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.151 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.152 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.152 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.152 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.152 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.152 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.152 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.152 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.152 INFO:teuthology.orchestra.run.smithi032.stdout: "trash_purge_schedule": { 2024-04-30T08:44:52.152 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "trash_purge_schedule", 2024-04-30T08:44:52.152 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.152 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.152 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.152 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.152 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.152 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.153 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.153 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.153 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.153 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.153 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.153 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.153 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.153 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.153 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.153 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "restful", 2024-04-30T08:44:52.153 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.153 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.153 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.153 INFO:teuthology.orchestra.run.smithi032.stdout: "enable_auth": { 2024-04-30T08:44:52.154 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "enable_auth", 2024-04-30T08:44:52.154 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.154 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.154 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.154 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.154 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.154 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.154 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.154 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.154 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.154 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.154 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.154 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.154 INFO:teuthology.orchestra.run.smithi032.stdout: "key_file": { 2024-04-30T08:44:52.155 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "key_file", 2024-04-30T08:44:52.155 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.155 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.155 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.155 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.155 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.155 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.155 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.155 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.155 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.155 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.155 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.155 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.155 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.156 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.156 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.156 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.156 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.156 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.156 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.156 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.156 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.156 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.156 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.156 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.156 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.156 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.157 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.157 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.157 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.157 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.157 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.157 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.157 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.157 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.157 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.157 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.157 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.157 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.157 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.157 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.158 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.158 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.158 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.158 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.158 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.158 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.158 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.158 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.158 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.158 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.158 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.158 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.158 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.159 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.159 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.159 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.159 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.159 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.159 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.159 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.159 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.159 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.159 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.159 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.159 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.159 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.159 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.160 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.160 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.160 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.160 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.160 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.160 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.160 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.160 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.160 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.160 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.160 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.160 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.160 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.160 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.161 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.161 INFO:teuthology.orchestra.run.smithi032.stdout: "server_addr": { 2024-04-30T08:44:52.161 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_addr", 2024-04-30T08:44:52.161 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.161 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.161 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.161 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.161 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.161 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.161 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.161 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.161 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.161 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.162 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.162 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.162 INFO:teuthology.orchestra.run.smithi032.stdout: "server_port": { 2024-04-30T08:44:52.162 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_port", 2024-04-30T08:44:52.162 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.162 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.162 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.162 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.162 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.162 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.162 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.162 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.162 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.163 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.163 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.163 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.163 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.163 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.163 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.163 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.163 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.163 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.163 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.163 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.163 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.163 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.164 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.164 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.164 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.164 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.164 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.164 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.164 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.164 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rgw", 2024-04-30T08:44:52.164 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.164 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.164 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.164 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.164 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.164 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.165 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.165 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.165 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.165 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.165 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.165 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.165 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.165 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.165 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.165 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.165 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.165 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.165 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.166 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.166 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.166 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.166 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.166 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.166 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.166 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.166 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.166 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.166 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.166 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.166 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.166 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.167 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.167 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.167 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.167 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.167 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.167 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.167 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.167 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.167 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.167 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.167 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.167 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.167 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.167 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.168 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.168 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.168 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.168 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.168 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.168 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.168 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.168 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.168 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.168 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.168 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.168 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.168 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.168 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.168 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.169 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.169 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.169 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.169 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.169 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.169 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.169 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.169 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.169 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.169 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.169 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.169 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.169 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.170 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.170 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.170 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.170 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.170 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.170 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.170 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.170 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.170 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.170 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.170 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.170 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.170 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.171 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.171 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.171 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.171 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rook", 2024-04-30T08:44:52.171 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.171 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.171 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.171 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.171 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.171 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.171 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.171 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.171 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.171 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.172 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.172 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.172 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.172 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.172 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.172 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.172 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.172 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.172 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.172 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.172 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.172 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.172 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.173 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.173 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.173 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.173 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.173 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.173 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.173 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.173 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.173 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.173 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.173 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.173 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.173 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.173 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.174 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.174 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.174 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.174 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.174 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.174 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.174 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.174 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.174 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.174 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.174 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.174 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.174 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.175 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.175 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.175 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.175 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.175 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.175 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.175 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.175 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.175 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.175 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.175 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.175 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.175 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.175 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.176 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.176 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.176 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.176 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.176 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.176 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.176 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.176 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.176 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.176 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.176 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.176 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.176 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.176 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.176 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.177 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.177 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.177 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.177 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.177 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.177 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.177 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.177 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.177 INFO:teuthology.orchestra.run.smithi032.stdout: "storage_class": { 2024-04-30T08:44:52.177 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "storage_class", 2024-04-30T08:44:52.177 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.177 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.177 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.177 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "local", 2024-04-30T08:44:52.178 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.178 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.178 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.178 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-30T08:44:52.178 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.178 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.178 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.178 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.178 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.178 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.178 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.178 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "selftest", 2024-04-30T08:44:52.178 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.178 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.179 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.179 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.179 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.179 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.179 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.179 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.179 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.179 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.179 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.179 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.179 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.179 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.179 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.179 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.180 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.180 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.180 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.180 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.180 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.180 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.180 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.180 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.180 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.180 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.180 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.180 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.180 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.180 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.180 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.181 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.181 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.181 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.181 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.181 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.181 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.181 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.181 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.181 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.181 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.181 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.181 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.181 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.181 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.182 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.182 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.182 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.182 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.182 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.182 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.182 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.182 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.182 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.182 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.182 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.182 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.182 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.182 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.183 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.184 INFO:teuthology.orchestra.run.smithi032.stdout: "roption1": { 2024-04-30T08:44:52.184 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "roption1", 2024-04-30T08:44:52.184 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.184 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.184 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.184 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.184 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.184 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.184 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.184 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.184 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.184 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.184 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.184 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.184 INFO:teuthology.orchestra.run.smithi032.stdout: "roption2": { 2024-04-30T08:44:52.185 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "roption2", 2024-04-30T08:44:52.185 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.185 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.185 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.185 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "xyz", 2024-04-30T08:44:52.185 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.185 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.185 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.185 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.185 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.185 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.185 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.185 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.185 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption1": { 2024-04-30T08:44:52.186 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption1", 2024-04-30T08:44:52.186 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.186 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.186 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.186 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.186 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.186 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.186 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.186 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.186 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.186 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.186 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.186 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.187 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption2": { 2024-04-30T08:44:52.187 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption2", 2024-04-30T08:44:52.187 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.187 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.187 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.187 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.187 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.187 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.187 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.187 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.187 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.187 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.187 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.187 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.188 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption3": { 2024-04-30T08:44:52.188 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption3", 2024-04-30T08:44:52.188 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-30T08:44:52.188 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.188 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.188 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.188 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.188 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.188 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.188 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.188 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.188 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.188 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.188 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.189 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption4": { 2024-04-30T08:44:52.189 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption4", 2024-04-30T08:44:52.189 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.189 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.189 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.189 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.189 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.189 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.189 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.189 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.189 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.189 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.189 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.189 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.190 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption5": { 2024-04-30T08:44:52.190 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption5", 2024-04-30T08:44:52.190 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.190 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.190 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.190 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.190 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.190 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.190 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.190 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.190 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.190 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.190 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.190 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.191 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption6": { 2024-04-30T08:44:52.191 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption6", 2024-04-30T08:44:52.191 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.191 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.191 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.191 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.191 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.191 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.191 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.191 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.191 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.191 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.191 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.192 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.192 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption7": { 2024-04-30T08:44:52.192 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption7", 2024-04-30T08:44:52.192 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.192 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.192 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.192 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.192 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "1", 2024-04-30T08:44:52.192 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "42", 2024-04-30T08:44:52.192 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.192 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.192 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.192 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.192 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.193 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.193 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.193 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.193 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.193 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.193 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.193 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.193 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.193 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.193 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.193 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.193 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.193 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.193 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.194 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.194 INFO:teuthology.orchestra.run.smithi032.stdout: "testkey": { 2024-04-30T08:44:52.194 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "testkey", 2024-04-30T08:44:52.194 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.194 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.194 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.194 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.194 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.194 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.194 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.194 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.194 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.194 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.194 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.195 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.195 INFO:teuthology.orchestra.run.smithi032.stdout: "testlkey": { 2024-04-30T08:44:52.195 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "testlkey", 2024-04-30T08:44:52.195 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.195 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.195 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.195 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.195 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.195 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.195 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.195 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.195 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.195 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.195 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.196 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.196 INFO:teuthology.orchestra.run.smithi032.stdout: "testnewline": { 2024-04-30T08:44:52.196 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "testnewline", 2024-04-30T08:44:52.196 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.196 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.196 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.196 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.196 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.196 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.196 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.196 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.196 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.196 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.196 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.197 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.197 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.197 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.197 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.197 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "snap_schedule", 2024-04-30T08:44:52.197 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.197 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.197 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.197 INFO:teuthology.orchestra.run.smithi032.stdout: "allow_m_granularity": { 2024-04-30T08:44:52.197 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "allow_m_granularity", 2024-04-30T08:44:52.197 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.197 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.197 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.197 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.198 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.198 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.198 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.198 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "allow minute scheduled snapshots", 2024-04-30T08:44:52.198 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.198 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.198 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.198 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.198 INFO:teuthology.orchestra.run.smithi032.stdout: "dump_on_update": { 2024-04-30T08:44:52.198 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "dump_on_update", 2024-04-30T08:44:52.198 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.198 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.198 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.199 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.199 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.199 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.199 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.199 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "dump database to debug log on update", 2024-04-30T08:44:52.199 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.199 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.199 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.199 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.199 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.199 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.199 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.199 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.199 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.200 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.200 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.200 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.200 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.200 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.200 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.200 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.200 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.200 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.200 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.200 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.200 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.200 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.201 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.201 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.201 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.201 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.201 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.201 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.201 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.201 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.201 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.201 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.201 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.201 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.201 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.202 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.202 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.202 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.202 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.202 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.202 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.202 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.202 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.202 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.202 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.202 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.202 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.202 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.202 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.203 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.203 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.203 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.203 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.203 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.203 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.203 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.203 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.203 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.203 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.203 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.203 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.203 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.203 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.204 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.204 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.204 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.204 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.204 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.204 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.204 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.204 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.204 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.204 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.204 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.204 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.204 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.205 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.205 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.205 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.205 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.205 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.205 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.205 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.205 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.205 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.205 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.205 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.205 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.205 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.205 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.206 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.206 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "stats", 2024-04-30T08:44:52.206 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.206 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.206 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.206 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.206 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.206 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.206 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.206 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.206 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.206 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.206 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.207 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.207 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.207 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.207 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.207 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.207 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.207 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.207 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.207 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.207 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.207 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.207 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.207 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.207 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.208 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.208 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.208 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.208 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.208 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.208 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.208 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.208 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.208 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.208 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.208 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.208 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.208 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.208 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.209 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.209 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.209 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.209 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.209 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.209 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.209 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.209 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.209 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.209 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.209 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.209 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.209 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.209 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.209 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.210 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.210 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.210 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.210 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.210 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.210 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.210 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.210 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.210 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.210 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.210 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.210 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.210 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.211 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.211 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.211 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.211 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.211 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.211 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.211 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.211 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.211 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.211 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.211 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.211 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.212 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.212 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.212 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.212 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.212 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.212 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.212 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.212 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.212 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.212 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.212 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.212 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "status", 2024-04-30T08:44:52.212 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.212 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.213 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.213 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.213 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.213 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.213 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.213 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.213 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.213 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.213 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.213 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.213 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.213 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.213 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.213 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.213 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.214 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.214 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.214 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.214 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.214 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.214 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.214 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.214 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.214 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.214 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.214 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.214 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.214 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.215 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.215 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.215 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.215 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.215 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.215 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.215 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.215 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.215 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.215 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.215 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.215 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.215 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.216 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.216 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.216 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.216 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.216 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.216 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.216 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.216 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.216 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.216 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.216 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.216 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.216 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.216 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.217 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.217 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.217 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.217 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.217 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.217 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.217 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.217 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.217 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.217 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.217 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.217 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.218 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.218 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.218 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.218 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.218 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.218 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.218 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.218 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.218 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.218 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.218 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.218 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.218 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.218 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.219 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.219 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.219 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.219 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.219 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.219 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.219 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "telegraf", 2024-04-30T08:44:52.219 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.219 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.219 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.219 INFO:teuthology.orchestra.run.smithi032.stdout: "address": { 2024-04-30T08:44:52.219 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "address", 2024-04-30T08:44:52.219 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.220 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.220 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.220 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-30T08:44:52.220 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.220 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.220 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.220 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.220 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.220 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.220 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.220 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.220 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2024-04-30T08:44:52.220 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2024-04-30T08:44:52.220 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.221 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.221 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.221 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "15", 2024-04-30T08:44:52.221 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.221 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.221 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.221 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.221 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.221 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.221 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.221 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.221 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.221 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.221 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.222 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.222 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.222 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.222 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.222 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.222 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.222 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.222 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.222 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.222 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.222 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.222 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.222 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.223 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.223 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.223 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.223 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.223 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.223 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.223 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.223 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.223 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.223 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.223 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.223 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.223 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.223 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.224 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.224 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.224 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.224 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.224 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.224 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.224 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.224 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.224 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.224 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.224 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.224 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.224 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.224 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.225 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.225 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.225 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.225 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.225 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.225 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.225 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.225 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.225 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.225 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.225 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.225 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.225 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.226 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.226 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.226 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.226 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.226 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.226 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.226 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.226 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.226 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.226 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.226 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.226 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.226 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.227 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.227 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.227 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.227 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.227 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.227 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.227 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.227 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.227 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.227 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.227 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.227 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.227 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.227 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.228 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.228 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "telemetry", 2024-04-30T08:44:52.228 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.228 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.228 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.228 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_basic": { 2024-04-30T08:44:52.228 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_basic", 2024-04-30T08:44:52.228 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.228 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.228 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.228 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.228 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.228 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.228 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.229 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-30T08:44:52.229 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.229 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.229 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.229 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_crash": { 2024-04-30T08:44:52.229 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_crash", 2024-04-30T08:44:52.229 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.229 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.229 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.229 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.229 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.229 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.229 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.230 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-30T08:44:52.230 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.230 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.230 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.230 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_device": { 2024-04-30T08:44:52.230 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_device", 2024-04-30T08:44:52.230 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.230 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.230 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.230 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.230 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.231 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.231 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.231 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-30T08:44:52.231 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.231 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.231 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.231 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.231 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_ident": { 2024-04-30T08:44:52.231 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_ident", 2024-04-30T08:44:52.231 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.231 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.231 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.231 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.231 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.232 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.232 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.232 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-30T08:44:52.232 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.232 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.232 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.232 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_perf": { 2024-04-30T08:44:52.232 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_perf", 2024-04-30T08:44:52.232 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.232 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.232 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.232 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.233 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.233 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.233 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.233 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-30T08:44:52.233 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.233 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.233 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.233 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.233 INFO:teuthology.orchestra.run.smithi032.stdout: "contact": { 2024-04-30T08:44:52.233 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "contact", 2024-04-30T08:44:52.233 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.233 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.233 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.233 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.234 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.234 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.234 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.234 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.234 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.234 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.234 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.234 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.234 INFO:teuthology.orchestra.run.smithi032.stdout: "description": { 2024-04-30T08:44:52.234 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "description", 2024-04-30T08:44:52.234 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.234 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.234 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.235 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.235 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.235 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.235 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.235 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.235 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.235 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.235 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.235 INFO:teuthology.orchestra.run.smithi032.stdout: "device_url": { 2024-04-30T08:44:52.235 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "device_url", 2024-04-30T08:44:52.235 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.235 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.235 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.235 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-30T08:44:52.236 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.236 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.236 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.236 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.236 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.236 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.236 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.236 INFO:teuthology.orchestra.run.smithi032.stdout: "enabled": { 2024-04-30T08:44:52.236 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "enabled", 2024-04-30T08:44:52.236 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.236 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.236 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.236 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.237 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.237 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.237 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.237 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.237 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.237 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.237 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.237 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2024-04-30T08:44:52.237 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2024-04-30T08:44:52.237 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.237 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.237 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.237 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "24", 2024-04-30T08:44:52.238 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "8", 2024-04-30T08:44:52.238 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.238 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.238 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.238 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.238 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.238 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.238 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.238 INFO:teuthology.orchestra.run.smithi032.stdout: "last_opt_revision": { 2024-04-30T08:44:52.238 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "last_opt_revision", 2024-04-30T08:44:52.238 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.238 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.238 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.239 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "1", 2024-04-30T08:44:52.239 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.239 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.239 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.239 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.239 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.239 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.239 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.239 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.239 INFO:teuthology.orchestra.run.smithi032.stdout: "leaderboard": { 2024-04-30T08:44:52.239 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "leaderboard", 2024-04-30T08:44:52.239 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.239 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.239 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.240 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.240 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.240 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.240 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.240 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.240 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.240 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.240 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.240 INFO:teuthology.orchestra.run.smithi032.stdout: "leaderboard_description": { 2024-04-30T08:44:52.240 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "leaderboard_description", 2024-04-30T08:44:52.240 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.240 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.240 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.241 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.241 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.241 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.241 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.241 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.241 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.241 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.241 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.241 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.241 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.241 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.241 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.241 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.241 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.242 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.242 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.242 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.242 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.242 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.242 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.242 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.242 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.242 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.242 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.242 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.242 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.242 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.243 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.243 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.243 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.243 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.243 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.243 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.243 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.243 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.243 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.243 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.243 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.243 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.243 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.244 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.244 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.244 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.244 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.244 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.244 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.244 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.244 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.244 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.244 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.244 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.244 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.244 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.244 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.245 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.245 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.245 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.245 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.245 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.245 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.245 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.245 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.245 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.245 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.245 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.245 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.245 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.245 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.246 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.246 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.246 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.246 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.246 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.246 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.246 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.246 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.246 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.246 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.246 INFO:teuthology.orchestra.run.smithi032.stdout: "organization": { 2024-04-30T08:44:52.246 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "organization", 2024-04-30T08:44:52.246 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.247 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.247 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.247 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.247 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.247 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.247 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.247 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.247 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.247 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.247 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.247 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.247 INFO:teuthology.orchestra.run.smithi032.stdout: "proxy": { 2024-04-30T08:44:52.247 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "proxy", 2024-04-30T08:44:52.248 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.248 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.248 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.248 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.248 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.248 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.248 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.248 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.248 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.248 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.248 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.248 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.248 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.248 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.249 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.249 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.249 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.249 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.249 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.249 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.249 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.249 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.249 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.249 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.249 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.249 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.249 INFO:teuthology.orchestra.run.smithi032.stdout: "url": { 2024-04-30T08:44:52.249 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "url", 2024-04-30T08:44:52.249 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.250 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.250 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.250 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-30T08:44:52.250 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.250 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.250 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.250 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.250 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.250 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.250 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.250 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.250 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.250 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.251 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.251 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "test_orchestrator", 2024-04-30T08:44:52.251 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.251 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.251 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.251 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.251 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.251 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.251 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.251 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.251 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.251 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.251 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.252 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.252 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.252 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.252 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.252 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.252 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.252 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.252 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.252 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.252 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.252 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.252 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.252 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.252 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.253 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.254 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.254 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.254 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.254 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.254 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.254 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.254 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.254 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.254 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.254 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.254 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.254 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.254 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.254 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.255 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.255 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.255 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.255 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.255 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.255 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.255 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.255 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.255 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.255 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.255 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.255 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.255 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.255 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.256 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.256 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.256 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.256 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.256 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.256 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.256 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.256 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.256 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.256 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.256 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.256 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.256 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.257 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.257 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.257 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.257 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.257 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.257 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.257 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.257 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.257 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.257 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "volumes", 2024-04-30T08:44:52.257 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.257 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.257 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.258 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.258 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.258 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.258 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.258 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.258 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.258 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.258 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.258 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.258 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.258 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.258 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.258 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.259 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.259 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.259 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.259 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.259 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.259 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.259 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.259 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.259 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.259 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.259 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.259 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.259 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.259 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.260 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.260 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.260 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.260 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.260 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.260 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.260 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.260 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.260 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.260 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.260 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.260 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.260 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.260 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.261 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.261 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.261 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.261 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.261 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.261 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.261 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.261 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.261 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.261 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.261 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.261 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.261 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.261 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.262 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.262 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.262 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.262 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.262 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.262 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.262 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.262 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.262 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.262 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.262 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.262 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.262 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.263 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.263 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.263 INFO:teuthology.orchestra.run.smithi032.stdout: "max_concurrent_clones": { 2024-04-30T08:44:52.263 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "max_concurrent_clones", 2024-04-30T08:44:52.263 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.263 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.263 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.263 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "4", 2024-04-30T08:44:52.263 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.263 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.263 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.263 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-30T08:44:52.263 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.263 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.264 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.264 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.264 INFO:teuthology.orchestra.run.smithi032.stdout: "periodic_async_work": { 2024-04-30T08:44:52.264 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "periodic_async_work", 2024-04-30T08:44:52.264 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.264 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.264 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.264 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.264 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.264 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.264 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.264 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Periodically check for async work", 2024-04-30T08:44:52.264 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.264 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.265 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.265 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.265 INFO:teuthology.orchestra.run.smithi032.stdout: "snapshot_clone_delay": { 2024-04-30T08:44:52.265 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "snapshot_clone_delay", 2024-04-30T08:44:52.265 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.265 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.265 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.265 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.265 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.265 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.265 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.265 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-30T08:44:52.265 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.265 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.266 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.266 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.266 INFO:teuthology.orchestra.run.smithi032.stdout: "snapshot_clone_no_wait": { 2024-04-30T08:44:52.266 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "snapshot_clone_no_wait", 2024-04-30T08:44:52.266 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.266 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.266 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.266 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.266 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.266 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.266 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.266 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-30T08:44:52.266 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.266 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.267 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.267 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.267 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.267 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.267 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.267 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.267 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.267 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.267 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.267 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.267 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.267 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.267 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.267 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.268 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.268 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.268 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.268 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.268 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.268 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "zabbix", 2024-04-30T08:44:52.268 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.268 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.268 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.268 INFO:teuthology.orchestra.run.smithi032.stdout: "discovery_interval": { 2024-04-30T08:44:52.268 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "discovery_interval", 2024-04-30T08:44:52.268 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2024-04-30T08:44:52.268 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.268 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.269 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "100", 2024-04-30T08:44:52.269 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.269 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.269 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.269 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.269 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.269 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.269 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.269 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.269 INFO:teuthology.orchestra.run.smithi032.stdout: "identifier": { 2024-04-30T08:44:52.269 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "identifier", 2024-04-30T08:44:52.269 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.269 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.269 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.270 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.270 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.270 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.270 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.270 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.270 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.270 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.270 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.270 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.270 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2024-04-30T08:44:52.270 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2024-04-30T08:44:52.270 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.270 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.271 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.271 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2024-04-30T08:44:52.271 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.271 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.271 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.271 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.271 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.271 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.271 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.271 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.271 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.271 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.271 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.271 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.272 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.272 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.272 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.272 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.272 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.272 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.272 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.272 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.272 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.272 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.272 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.272 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.272 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.273 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.273 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.273 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.273 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.273 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.273 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.273 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.273 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.273 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.273 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.273 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.273 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.273 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.273 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.273 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.274 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.274 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.274 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.274 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.274 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.274 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.274 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.274 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.274 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.274 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.274 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.274 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.274 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.275 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.275 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.275 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.275 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.275 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.275 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.275 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.275 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.275 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.275 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.275 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.275 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.275 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.276 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.276 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.276 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.276 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.276 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.276 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.276 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.276 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.276 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.276 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.276 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.276 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.276 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.276 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.277 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.277 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.277 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.277 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.277 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.277 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.277 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.277 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.277 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.277 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.277 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.277 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.277 INFO:teuthology.orchestra.run.smithi032.stdout: "zabbix_host": { 2024-04-30T08:44:52.277 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "zabbix_host", 2024-04-30T08:44:52.277 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.278 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.278 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.278 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.278 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.278 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.278 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.278 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.278 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.278 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.278 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.278 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.278 INFO:teuthology.orchestra.run.smithi032.stdout: "zabbix_port": { 2024-04-30T08:44:52.278 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "zabbix_port", 2024-04-30T08:44:52.278 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.279 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.279 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.279 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10051", 2024-04-30T08:44:52.279 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.279 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.279 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.279 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.279 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.279 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.279 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.279 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.279 INFO:teuthology.orchestra.run.smithi032.stdout: "zabbix_sender": { 2024-04-30T08:44:52.279 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "zabbix_sender", 2024-04-30T08:44:52.279 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.280 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.280 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.280 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-30T08:44:52.280 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.280 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.280 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.280 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.280 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.280 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.280 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.280 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.280 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.280 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.280 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-30T08:44:52.281 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.281 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.281 INFO:teuthology.orchestra.run.smithi032.stdout: "gid": 24116, 2024-04-30T08:44:52.281 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "y", 2024-04-30T08:44:52.281 INFO:teuthology.orchestra.run.smithi032.stdout: "mgr_features": 4540701547738038271, 2024-04-30T08:44:52.281 INFO:teuthology.orchestra.run.smithi032.stdout: "available_modules": [ 2024-04-30T08:44:52.281 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.281 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "alerts", 2024-04-30T08:44:52.281 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.281 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.281 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.281 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2024-04-30T08:44:52.281 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2024-04-30T08:44:52.281 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.282 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.282 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.282 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2024-04-30T08:44:52.282 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.282 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.282 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.282 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "How frequently to reexamine health status", 2024-04-30T08:44:52.282 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.282 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.282 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.282 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.282 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.282 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.282 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.283 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.283 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.283 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.283 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.283 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.283 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.283 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.283 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.283 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.283 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.283 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.283 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.283 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.283 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.284 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.284 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.284 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.284 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.284 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.284 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.284 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.284 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.284 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.284 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.284 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.284 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.284 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.284 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.285 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.285 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.285 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.285 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.285 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.285 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.285 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.285 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.285 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.285 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.285 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.285 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.285 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.286 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.286 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.286 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.286 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.286 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.286 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.286 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.286 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.286 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.286 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.286 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.286 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.286 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.287 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.287 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.287 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.287 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.287 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.287 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.287 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.287 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.287 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.287 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.287 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.287 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.287 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.288 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_destination": { 2024-04-30T08:44:52.288 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_destination", 2024-04-30T08:44:52.288 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.288 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.288 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.288 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.288 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.288 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.288 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.288 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Email address to send alerts to", 2024-04-30T08:44:52.288 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.288 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.288 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.289 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.289 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_from_name": { 2024-04-30T08:44:52.289 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_from_name", 2024-04-30T08:44:52.289 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.289 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.289 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.289 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "Ceph", 2024-04-30T08:44:52.289 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.289 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.289 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.289 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Email From: name", 2024-04-30T08:44:52.289 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.289 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.289 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.290 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.290 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_host": { 2024-04-30T08:44:52.290 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_host", 2024-04-30T08:44:52.290 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.290 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.290 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.290 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.290 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.290 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.290 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.290 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "SMTP server", 2024-04-30T08:44:52.290 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.290 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.291 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.291 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.291 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_password": { 2024-04-30T08:44:52.291 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_password", 2024-04-30T08:44:52.291 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.291 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.291 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.291 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.291 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.291 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.291 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.291 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Password to authenticate with", 2024-04-30T08:44:52.291 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.292 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.292 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.292 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.292 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_port": { 2024-04-30T08:44:52.292 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_port", 2024-04-30T08:44:52.292 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.292 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.292 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.292 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "465", 2024-04-30T08:44:52.292 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.292 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.292 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.292 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "SMTP port", 2024-04-30T08:44:52.292 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.293 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.293 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.293 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.293 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_sender": { 2024-04-30T08:44:52.293 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_sender", 2024-04-30T08:44:52.293 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.293 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.293 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.293 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.293 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.293 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.293 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.293 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "SMTP envelope sender", 2024-04-30T08:44:52.293 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.294 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.294 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.294 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.294 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_ssl": { 2024-04-30T08:44:52.294 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_ssl", 2024-04-30T08:44:52.294 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.294 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.294 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.294 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.294 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.294 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.294 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.294 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-30T08:44:52.294 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.295 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.295 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.295 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.295 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_user": { 2024-04-30T08:44:52.295 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_user", 2024-04-30T08:44:52.295 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.295 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.295 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.295 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.295 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.295 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.295 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.295 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "User to authenticate as", 2024-04-30T08:44:52.296 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.296 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.296 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.296 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.296 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.296 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.296 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.296 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.296 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.296 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.296 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.296 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.296 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.296 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.297 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.297 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.297 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.297 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.297 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.297 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.297 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.297 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "balancer", 2024-04-30T08:44:52.297 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.297 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.297 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.297 INFO:teuthology.orchestra.run.smithi032.stdout: "active": { 2024-04-30T08:44:52.297 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "active", 2024-04-30T08:44:52.297 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.297 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.298 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.298 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.298 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.298 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.298 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.298 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "automatically balance PGs across cluster", 2024-04-30T08:44:52.298 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.298 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.298 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.298 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.298 INFO:teuthology.orchestra.run.smithi032.stdout: "begin_time": { 2024-04-30T08:44:52.298 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "begin_time", 2024-04-30T08:44:52.299 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.299 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.299 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.299 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0000", 2024-04-30T08:44:52.299 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.299 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.299 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.299 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "beginning time of day to automatically balance", 2024-04-30T08:44:52.299 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-30T08:44:52.299 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.299 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.299 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.299 INFO:teuthology.orchestra.run.smithi032.stdout: "begin_weekday": { 2024-04-30T08:44:52.299 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "begin_weekday", 2024-04-30T08:44:52.300 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2024-04-30T08:44:52.300 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.300 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.300 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.300 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "0", 2024-04-30T08:44:52.300 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "6", 2024-04-30T08:44:52.300 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.300 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-30T08:44:52.300 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-30T08:44:52.300 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.300 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.300 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.300 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_compat_max_iterations": { 2024-04-30T08:44:52.300 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crush_compat_max_iterations", 2024-04-30T08:44:52.301 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2024-04-30T08:44:52.301 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.301 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.301 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "25", 2024-04-30T08:44:52.301 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "1", 2024-04-30T08:44:52.301 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "250", 2024-04-30T08:44:52.301 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.301 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-30T08:44:52.301 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.301 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.301 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.301 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.301 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_compat_metrics": { 2024-04-30T08:44:52.301 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crush_compat_metrics", 2024-04-30T08:44:52.302 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.302 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.302 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.302 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "pgs,objects,bytes", 2024-04-30T08:44:52.302 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.302 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.302 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.302 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-30T08:44:52.302 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-30T08:44:52.302 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.302 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.302 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.302 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_compat_step": { 2024-04-30T08:44:52.303 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crush_compat_step", 2024-04-30T08:44:52.303 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-30T08:44:52.303 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.303 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.303 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0.5", 2024-04-30T08:44:52.303 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "0.001", 2024-04-30T08:44:52.303 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "0.999", 2024-04-30T08:44:52.303 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.303 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "aggressiveness of optimization", 2024-04-30T08:44:52.303 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-30T08:44:52.303 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.303 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.303 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.304 INFO:teuthology.orchestra.run.smithi032.stdout: "end_time": { 2024-04-30T08:44:52.304 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "end_time", 2024-04-30T08:44:52.304 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.304 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.304 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.304 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "2359", 2024-04-30T08:44:52.304 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.304 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.304 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.304 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "ending time of day to automatically balance", 2024-04-30T08:44:52.304 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-30T08:44:52.304 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.304 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.304 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.305 INFO:teuthology.orchestra.run.smithi032.stdout: "end_weekday": { 2024-04-30T08:44:52.305 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "end_weekday", 2024-04-30T08:44:52.305 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2024-04-30T08:44:52.305 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.305 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.305 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.305 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "0", 2024-04-30T08:44:52.305 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "6", 2024-04-30T08:44:52.305 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.305 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-30T08:44:52.305 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-30T08:44:52.305 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.305 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.305 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.306 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.306 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.306 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.306 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.306 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.306 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.306 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.306 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.306 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.306 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.306 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.306 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.306 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.306 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.307 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.307 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.307 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.307 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.307 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.307 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.307 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.307 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.307 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.307 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.307 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.307 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.307 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.308 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.308 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.308 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.308 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.308 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.308 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.308 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.308 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.308 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.308 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.308 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.308 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.308 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.308 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.309 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.309 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.309 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.309 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.309 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.309 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.309 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.309 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.309 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.309 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.309 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.309 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.309 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.309 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.310 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.310 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.310 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.310 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.310 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.310 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.310 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.310 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.310 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.310 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.310 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.310 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.310 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.311 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.311 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.311 INFO:teuthology.orchestra.run.smithi032.stdout: "min_score": { 2024-04-30T08:44:52.311 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "min_score", 2024-04-30T08:44:52.311 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-30T08:44:52.311 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.311 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.311 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.311 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.311 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.311 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.311 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-30T08:44:52.311 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.312 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.312 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.312 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.312 INFO:teuthology.orchestra.run.smithi032.stdout: "mode": { 2024-04-30T08:44:52.312 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mode", 2024-04-30T08:44:52.312 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.312 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.312 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.312 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "upmap", 2024-04-30T08:44:52.312 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.312 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.312 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.312 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-compat", 2024-04-30T08:44:52.312 INFO:teuthology.orchestra.run.smithi032.stdout: "none", 2024-04-30T08:44:52.313 INFO:teuthology.orchestra.run.smithi032.stdout: "read", 2024-04-30T08:44:52.313 INFO:teuthology.orchestra.run.smithi032.stdout: "upmap", 2024-04-30T08:44:52.313 INFO:teuthology.orchestra.run.smithi032.stdout: "upmap-read" 2024-04-30T08:44:52.313 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.313 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Balancer mode", 2024-04-30T08:44:52.313 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.313 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.313 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.313 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.313 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_ids": { 2024-04-30T08:44:52.313 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "pool_ids", 2024-04-30T08:44:52.313 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.313 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.313 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.314 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.314 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.314 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.314 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.314 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-30T08:44:52.314 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.314 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.314 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.314 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.314 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2024-04-30T08:44:52.314 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2024-04-30T08:44:52.314 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.314 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.315 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.315 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2024-04-30T08:44:52.315 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.315 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.315 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.315 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-30T08:44:52.315 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.315 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.315 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.315 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.315 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.315 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.315 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.315 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.316 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.316 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.316 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.316 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.316 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.316 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.316 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.316 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.316 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.316 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.316 INFO:teuthology.orchestra.run.smithi032.stdout: "upmap_max_deviation": { 2024-04-30T08:44:52.316 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "upmap_max_deviation", 2024-04-30T08:44:52.316 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.316 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.317 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.317 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5", 2024-04-30T08:44:52.317 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "1", 2024-04-30T08:44:52.317 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.317 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.317 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-30T08:44:52.317 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-30T08:44:52.317 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.317 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.317 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.317 INFO:teuthology.orchestra.run.smithi032.stdout: "upmap_max_optimizations": { 2024-04-30T08:44:52.317 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "upmap_max_optimizations", 2024-04-30T08:44:52.317 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2024-04-30T08:44:52.317 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.318 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.318 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-30T08:44:52.318 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.318 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.318 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.318 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-30T08:44:52.318 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.318 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.318 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.318 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.318 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.318 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.318 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.319 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cephadm", 2024-04-30T08:44:52.319 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.319 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.319 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.319 INFO:teuthology.orchestra.run.smithi032.stdout: "agent_down_multiplier": { 2024-04-30T08:44:52.319 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "agent_down_multiplier", 2024-04-30T08:44:52.319 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-30T08:44:52.319 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.319 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.319 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "3.0", 2024-04-30T08:44:52.319 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.319 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.319 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.319 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-30T08:44:52.320 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.320 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.320 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.320 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.320 INFO:teuthology.orchestra.run.smithi032.stdout: "agent_refresh_rate": { 2024-04-30T08:44:52.320 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "agent_refresh_rate", 2024-04-30T08:44:52.320 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.320 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.320 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.320 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "20", 2024-04-30T08:44:52.320 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.320 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.320 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.320 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-30T08:44:52.321 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.321 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.321 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.321 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.321 INFO:teuthology.orchestra.run.smithi032.stdout: "agent_starting_port": { 2024-04-30T08:44:52.321 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "agent_starting_port", 2024-04-30T08:44:52.321 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.321 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.321 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.321 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "4721", 2024-04-30T08:44:52.321 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.321 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.321 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.321 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-30T08:44:52.322 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.322 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.322 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.322 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.322 INFO:teuthology.orchestra.run.smithi032.stdout: "allow_ptrace": { 2024-04-30T08:44:52.322 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "allow_ptrace", 2024-04-30T08:44:52.322 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.322 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.322 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.322 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.322 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.322 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.322 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.322 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-30T08:44:52.323 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-30T08:44:52.323 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.323 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.323 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.323 INFO:teuthology.orchestra.run.smithi032.stdout: "autotune_interval": { 2024-04-30T08:44:52.323 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "autotune_interval", 2024-04-30T08:44:52.323 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.323 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.323 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.323 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2024-04-30T08:44:52.323 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.323 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.323 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.323 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-30T08:44:52.324 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.324 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.324 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.324 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.324 INFO:teuthology.orchestra.run.smithi032.stdout: "autotune_memory_target_ratio": { 2024-04-30T08:44:52.324 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "autotune_memory_target_ratio", 2024-04-30T08:44:52.324 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-30T08:44:52.324 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.324 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.324 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0.7", 2024-04-30T08:44:52.324 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.324 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.324 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.325 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-30T08:44:52.325 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.325 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.325 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.325 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.325 INFO:teuthology.orchestra.run.smithi032.stdout: "cephadm_log_destination": { 2024-04-30T08:44:52.325 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cephadm_log_destination", 2024-04-30T08:44:52.325 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.325 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.325 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.325 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.325 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.325 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.326 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.326 INFO:teuthology.orchestra.run.smithi032.stdout: "file", 2024-04-30T08:44:52.326 INFO:teuthology.orchestra.run.smithi032.stdout: "file,syslog", 2024-04-30T08:44:52.326 INFO:teuthology.orchestra.run.smithi032.stdout: "syslog" 2024-04-30T08:44:52.326 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.326 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-30T08:44:52.326 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.326 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.326 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.326 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.326 INFO:teuthology.orchestra.run.smithi032.stdout: "cgroups_split": { 2024-04-30T08:44:52.326 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cgroups_split", 2024-04-30T08:44:52.326 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.326 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.327 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.327 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.327 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.327 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.327 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.327 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-30T08:44:52.327 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.327 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.327 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.327 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.327 INFO:teuthology.orchestra.run.smithi032.stdout: "config_checks_enabled": { 2024-04-30T08:44:52.327 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "config_checks_enabled", 2024-04-30T08:44:52.327 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.327 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.328 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.328 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.328 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.328 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.328 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.328 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-30T08:44:52.328 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.328 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.328 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.328 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.328 INFO:teuthology.orchestra.run.smithi032.stdout: "config_dashboard": { 2024-04-30T08:44:52.328 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "config_dashboard", 2024-04-30T08:44:52.328 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.329 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.329 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.329 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.329 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.329 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.329 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.329 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-30T08:44:52.329 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.329 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.329 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.329 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.329 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_alertmanager": { 2024-04-30T08:44:52.329 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_alertmanager", 2024-04-30T08:44:52.329 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.330 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.330 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.330 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-30T08:44:52.330 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.330 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.330 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.330 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Prometheus container image", 2024-04-30T08:44:52.330 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.330 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.330 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.330 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.330 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_base": { 2024-04-30T08:44:52.330 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_base", 2024-04-30T08:44:52.331 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.331 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.331 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.331 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-30T08:44:52.331 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.331 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.331 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.331 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Container image name, without the tag", 2024-04-30T08:44:52.331 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.331 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.331 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.331 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.331 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_elasticsearch": { 2024-04-30T08:44:52.331 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_elasticsearch", 2024-04-30T08:44:52.332 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.332 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.332 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.332 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-30T08:44:52.332 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.332 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.332 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.332 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "elasticsearch container image", 2024-04-30T08:44:52.332 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.332 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.332 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.332 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.332 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_grafana": { 2024-04-30T08:44:52.332 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_grafana", 2024-04-30T08:44:52.333 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.333 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.333 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.333 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-30T08:44:52.333 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.333 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.333 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.333 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Prometheus container image", 2024-04-30T08:44:52.333 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.333 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.333 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.333 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.333 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_haproxy": { 2024-04-30T08:44:52.334 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_haproxy", 2024-04-30T08:44:52.334 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.334 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.334 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.334 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-30T08:44:52.334 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.334 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.334 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.334 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "HAproxy container image", 2024-04-30T08:44:52.334 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.334 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.334 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.334 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.334 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_jaeger_agent": { 2024-04-30T08:44:52.335 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_jaeger_agent", 2024-04-30T08:44:52.335 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.335 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.335 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.335 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-30T08:44:52.335 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.335 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.335 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.335 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Jaeger agent container image", 2024-04-30T08:44:52.335 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.335 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.335 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.335 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.335 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_jaeger_collector": { 2024-04-30T08:44:52.336 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_jaeger_collector", 2024-04-30T08:44:52.336 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.336 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.336 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.336 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-30T08:44:52.336 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.336 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.336 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.336 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Jaeger collector container image", 2024-04-30T08:44:52.336 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.336 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.336 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.336 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.337 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_jaeger_query": { 2024-04-30T08:44:52.337 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_jaeger_query", 2024-04-30T08:44:52.337 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.337 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.337 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.337 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-30T08:44:52.337 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.337 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.337 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.337 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Jaeger query container image", 2024-04-30T08:44:52.337 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.337 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.337 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.337 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.338 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_keepalived": { 2024-04-30T08:44:52.338 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_keepalived", 2024-04-30T08:44:52.338 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.338 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.338 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.338 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-30T08:44:52.338 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.338 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.338 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.338 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Keepalived container image", 2024-04-30T08:44:52.338 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.338 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.338 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.339 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.339 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_loki": { 2024-04-30T08:44:52.339 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_loki", 2024-04-30T08:44:52.339 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.339 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.339 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.339 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-30T08:44:52.339 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.339 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.339 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.339 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Loki container image", 2024-04-30T08:44:52.339 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.339 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.339 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.340 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.340 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_node_exporter": { 2024-04-30T08:44:52.340 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_node_exporter", 2024-04-30T08:44:52.340 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.340 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.340 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.340 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-30T08:44:52.340 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.340 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.340 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.340 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Prometheus container image", 2024-04-30T08:44:52.340 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.340 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.340 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.341 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.341 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_nvmeof": { 2024-04-30T08:44:52.341 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_nvmeof", 2024-04-30T08:44:52.341 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.341 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.341 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.341 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-30T08:44:52.341 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.341 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.341 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.341 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Nvme-of container image", 2024-04-30T08:44:52.341 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.341 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.342 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.342 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.342 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_prometheus": { 2024-04-30T08:44:52.342 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_prometheus", 2024-04-30T08:44:52.342 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.342 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.342 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.342 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-30T08:44:52.342 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.342 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.342 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.342 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Prometheus container image", 2024-04-30T08:44:52.342 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.343 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.343 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.343 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.343 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_promtail": { 2024-04-30T08:44:52.343 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_promtail", 2024-04-30T08:44:52.343 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.343 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.343 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.343 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-30T08:44:52.343 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.343 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.343 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.343 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Promtail container image", 2024-04-30T08:44:52.344 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.344 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.344 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.344 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.344 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_samba": { 2024-04-30T08:44:52.344 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_samba", 2024-04-30T08:44:52.344 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.344 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.344 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.344 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-30T08:44:52.344 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.344 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.344 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.344 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Samba/SMB container image", 2024-04-30T08:44:52.345 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.345 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.345 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.345 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.345 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_snmp_gateway": { 2024-04-30T08:44:52.345 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_snmp_gateway", 2024-04-30T08:44:52.345 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.345 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.345 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.345 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-30T08:44:52.345 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.346 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.346 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.346 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "SNMP Gateway container image", 2024-04-30T08:44:52.346 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.346 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.346 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.346 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.346 INFO:teuthology.orchestra.run.smithi032.stdout: "container_init": { 2024-04-30T08:44:52.346 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_init", 2024-04-30T08:44:52.346 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.346 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.346 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.346 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.347 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.347 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.347 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.347 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Run podman/docker with `--init`", 2024-04-30T08:44:52.347 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.347 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.347 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.347 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.347 INFO:teuthology.orchestra.run.smithi032.stdout: "daemon_cache_timeout": { 2024-04-30T08:44:52.347 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "daemon_cache_timeout", 2024-04-30T08:44:52.347 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.347 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.347 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.347 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2024-04-30T08:44:52.348 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.348 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.348 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.348 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-30T08:44:52.348 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.348 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.348 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.348 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.348 INFO:teuthology.orchestra.run.smithi032.stdout: "default_cephadm_command_timeout": { 2024-04-30T08:44:52.348 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "default_cephadm_command_timeout", 2024-04-30T08:44:52.348 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.348 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.348 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.348 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "900", 2024-04-30T08:44:52.348 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.349 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.349 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.349 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-30T08:44:52.349 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.349 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.349 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.349 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.349 INFO:teuthology.orchestra.run.smithi032.stdout: "default_registry": { 2024-04-30T08:44:52.349 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "default_registry", 2024-04-30T08:44:52.349 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.349 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.349 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.349 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "docker.io", 2024-04-30T08:44:52.350 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.350 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.350 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.350 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-30T08:44:52.350 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.350 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.350 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.350 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.350 INFO:teuthology.orchestra.run.smithi032.stdout: "device_cache_timeout": { 2024-04-30T08:44:52.350 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "device_cache_timeout", 2024-04-30T08:44:52.350 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.350 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.350 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.350 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "1800", 2024-04-30T08:44:52.351 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.351 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.351 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.351 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "seconds to cache device inventory", 2024-04-30T08:44:52.351 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.351 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.351 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.351 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.351 INFO:teuthology.orchestra.run.smithi032.stdout: "device_enhanced_scan": { 2024-04-30T08:44:52.351 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "device_enhanced_scan", 2024-04-30T08:44:52.351 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.351 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.351 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.352 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.352 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.352 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.352 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.352 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-30T08:44:52.352 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.352 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.352 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.352 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.352 INFO:teuthology.orchestra.run.smithi032.stdout: "facts_cache_timeout": { 2024-04-30T08:44:52.352 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "facts_cache_timeout", 2024-04-30T08:44:52.352 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.352 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.352 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.353 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2024-04-30T08:44:52.353 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.353 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.353 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.353 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "seconds to cache host facts data", 2024-04-30T08:44:52.353 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.353 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.353 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.353 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.353 INFO:teuthology.orchestra.run.smithi032.stdout: "grafana_dashboards_path": { 2024-04-30T08:44:52.353 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "grafana_dashboards_path", 2024-04-30T08:44:52.353 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.353 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.354 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.354 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-30T08:44:52.354 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.354 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.354 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.354 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-30T08:44:52.354 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.354 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.354 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.354 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.354 INFO:teuthology.orchestra.run.smithi032.stdout: "host_check_interval": { 2024-04-30T08:44:52.354 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "host_check_interval", 2024-04-30T08:44:52.354 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.355 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.355 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.355 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2024-04-30T08:44:52.355 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.355 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.355 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.355 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to perform a host check", 2024-04-30T08:44:52.355 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.355 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.355 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.355 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.355 INFO:teuthology.orchestra.run.smithi032.stdout: "hw_monitoring": { 2024-04-30T08:44:52.355 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "hw_monitoring", 2024-04-30T08:44:52.355 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.355 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.356 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.356 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.356 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.356 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.356 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.356 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-30T08:44:52.356 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.356 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.356 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.356 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.356 INFO:teuthology.orchestra.run.smithi032.stdout: "inventory_list_all": { 2024-04-30T08:44:52.356 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "inventory_list_all", 2024-04-30T08:44:52.356 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.356 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.357 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.357 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.357 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.357 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.357 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.357 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-30T08:44:52.357 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.357 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.357 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.357 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.357 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.357 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.357 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.357 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.358 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.358 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.358 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.358 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.358 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.358 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.358 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.358 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.358 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.358 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.358 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.358 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.358 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.359 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.359 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.359 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.359 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.359 INFO:teuthology.orchestra.run.smithi032.stdout: "log_refresh_metadata": { 2024-04-30T08:44:52.359 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_refresh_metadata", 2024-04-30T08:44:52.359 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.359 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.359 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.359 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.359 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.359 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.359 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.359 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-30T08:44:52.360 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.360 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.360 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.360 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.360 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.360 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.360 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.360 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.360 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.360 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.360 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.360 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.360 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.360 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-30T08:44:52.360 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.361 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.361 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.361 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.361 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.361 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.361 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.361 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.361 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.361 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.361 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.361 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.361 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.361 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.362 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.362 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.362 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.362 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.362 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.362 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.362 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.362 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.362 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.362 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.362 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.362 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.362 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.362 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.363 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.363 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.363 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.363 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.363 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.363 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.363 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.363 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.363 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.363 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.363 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.363 INFO:teuthology.orchestra.run.smithi032.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-30T08:44:52.363 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-30T08:44:52.363 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.364 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.364 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.364 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.364 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.364 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.364 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.364 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-30T08:44:52.364 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.364 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.364 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.364 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.364 INFO:teuthology.orchestra.run.smithi032.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-30T08:44:52.364 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-30T08:44:52.364 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.365 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.365 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.365 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "*", 2024-04-30T08:44:52.365 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.365 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.365 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.365 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-30T08:44:52.365 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.365 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.365 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.365 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.365 INFO:teuthology.orchestra.run.smithi032.stdout: "max_count_per_host": { 2024-04-30T08:44:52.365 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "max_count_per_host", 2024-04-30T08:44:52.365 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.366 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.366 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.366 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-30T08:44:52.366 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.366 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.366 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.366 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "max number of daemons per service per host", 2024-04-30T08:44:52.366 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.366 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.366 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.366 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.366 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd_draining_count": { 2024-04-30T08:44:52.366 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "max_osd_draining_count", 2024-04-30T08:44:52.367 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.367 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.367 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.367 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-30T08:44:52.367 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.367 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.367 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.367 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-30T08:44:52.367 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.367 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.367 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.367 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.367 INFO:teuthology.orchestra.run.smithi032.stdout: "migration_current": { 2024-04-30T08:44:52.367 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "migration_current", 2024-04-30T08:44:52.368 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.368 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.368 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.368 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.368 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.368 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.368 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.368 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "internal - do not modify", 2024-04-30T08:44:52.368 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.368 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.368 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.368 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.368 INFO:teuthology.orchestra.run.smithi032.stdout: "mode": { 2024-04-30T08:44:52.368 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mode", 2024-04-30T08:44:52.369 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.369 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.369 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.369 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "root", 2024-04-30T08:44:52.369 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.369 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.369 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.369 INFO:teuthology.orchestra.run.smithi032.stdout: "cephadm-package", 2024-04-30T08:44:52.369 INFO:teuthology.orchestra.run.smithi032.stdout: "root" 2024-04-30T08:44:52.369 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.369 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "mode for remote execution of cephadm", 2024-04-30T08:44:52.369 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.369 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.369 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.370 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.370 INFO:teuthology.orchestra.run.smithi032.stdout: "oob_default_addr": { 2024-04-30T08:44:52.370 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "oob_default_addr", 2024-04-30T08:44:52.370 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.370 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.370 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.370 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "169.254.1.1", 2024-04-30T08:44:52.370 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.370 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.370 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.370 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-30T08:44:52.370 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.370 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.370 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.371 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.371 INFO:teuthology.orchestra.run.smithi032.stdout: "prometheus_alerts_path": { 2024-04-30T08:44:52.371 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "prometheus_alerts_path", 2024-04-30T08:44:52.371 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.371 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.371 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.371 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-30T08:44:52.371 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.371 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.371 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.371 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-30T08:44:52.371 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.371 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.371 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.372 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.372 INFO:teuthology.orchestra.run.smithi032.stdout: "registry_insecure": { 2024-04-30T08:44:52.372 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "registry_insecure", 2024-04-30T08:44:52.372 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.372 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.372 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.372 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.372 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.372 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.372 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.372 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-30T08:44:52.372 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.372 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.373 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.373 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.373 INFO:teuthology.orchestra.run.smithi032.stdout: "registry_password": { 2024-04-30T08:44:52.373 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "registry_password", 2024-04-30T08:44:52.373 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.373 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.373 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.373 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.373 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.373 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.373 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.373 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-30T08:44:52.373 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.373 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.374 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.374 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.374 INFO:teuthology.orchestra.run.smithi032.stdout: "registry_url": { 2024-04-30T08:44:52.374 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "registry_url", 2024-04-30T08:44:52.374 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.374 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.374 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.374 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.374 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.374 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.374 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.374 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-30T08:44:52.374 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.375 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.375 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.375 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.375 INFO:teuthology.orchestra.run.smithi032.stdout: "registry_username": { 2024-04-30T08:44:52.375 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "registry_username", 2024-04-30T08:44:52.375 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.375 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.375 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.375 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.375 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.375 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.375 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.375 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-30T08:44:52.375 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.376 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.376 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.376 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.376 INFO:teuthology.orchestra.run.smithi032.stdout: "secure_monitoring_stack": { 2024-04-30T08:44:52.376 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "secure_monitoring_stack", 2024-04-30T08:44:52.376 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.376 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.376 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.376 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.376 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.376 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.376 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.376 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-30T08:44:52.376 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.377 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.377 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.377 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.377 INFO:teuthology.orchestra.run.smithi032.stdout: "service_discovery_port": { 2024-04-30T08:44:52.377 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "service_discovery_port", 2024-04-30T08:44:52.377 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.377 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.377 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.377 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8765", 2024-04-30T08:44:52.377 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.377 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.377 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.377 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "cephadm service discovery port", 2024-04-30T08:44:52.377 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.378 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.378 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.378 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.378 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.378 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.378 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.378 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.378 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.378 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.378 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.378 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.378 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.378 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.379 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.379 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.379 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.379 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.379 INFO:teuthology.orchestra.run.smithi032.stdout: "ssh_config_file": { 2024-04-30T08:44:52.379 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ssh_config_file", 2024-04-30T08:44:52.379 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.379 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.379 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.379 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.379 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.379 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.379 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.380 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-30T08:44:52.380 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.380 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.380 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.380 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.380 INFO:teuthology.orchestra.run.smithi032.stdout: "use_agent": { 2024-04-30T08:44:52.380 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "use_agent", 2024-04-30T08:44:52.380 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.380 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.380 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.380 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.380 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.380 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.380 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.381 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-30T08:44:52.381 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.381 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.381 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.381 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.381 INFO:teuthology.orchestra.run.smithi032.stdout: "use_repo_digest": { 2024-04-30T08:44:52.381 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "use_repo_digest", 2024-04-30T08:44:52.381 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.381 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.381 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.381 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.381 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.381 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.381 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.382 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-30T08:44:52.382 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.382 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.382 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.382 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.382 INFO:teuthology.orchestra.run.smithi032.stdout: "warn_on_failed_host_check": { 2024-04-30T08:44:52.382 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "warn_on_failed_host_check", 2024-04-30T08:44:52.382 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.382 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.382 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.382 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.382 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.382 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.383 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.383 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "raise a health warning if the host check fails", 2024-04-30T08:44:52.383 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.383 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.383 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.383 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.383 INFO:teuthology.orchestra.run.smithi032.stdout: "warn_on_stray_daemons": { 2024-04-30T08:44:52.383 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "warn_on_stray_daemons", 2024-04-30T08:44:52.383 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.383 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.383 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.383 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.383 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.383 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.384 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.384 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-30T08:44:52.384 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.384 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.384 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.384 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.384 INFO:teuthology.orchestra.run.smithi032.stdout: "warn_on_stray_hosts": { 2024-04-30T08:44:52.384 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "warn_on_stray_hosts", 2024-04-30T08:44:52.384 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.384 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.384 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.384 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.384 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.384 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.384 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.385 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-30T08:44:52.385 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.385 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.385 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.385 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.385 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.385 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.385 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.385 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crash", 2024-04-30T08:44:52.385 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.385 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.385 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.385 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.385 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.386 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.386 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.386 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.386 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.386 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.386 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.386 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.386 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.386 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.386 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.386 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.386 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.386 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.386 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.387 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.387 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.387 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.387 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.387 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.387 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.387 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.387 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.387 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.387 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.387 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.387 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.387 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.388 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.388 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.388 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.388 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.388 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.388 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.388 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.388 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.388 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.388 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.388 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.388 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.388 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.388 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.389 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.389 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.389 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.389 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.389 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.389 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.389 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.389 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.389 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.389 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.389 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.389 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.389 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.389 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.390 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.390 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.390 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.390 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.390 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.390 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.390 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.390 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.390 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.390 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.390 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.390 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.390 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.390 INFO:teuthology.orchestra.run.smithi032.stdout: "retain_interval": { 2024-04-30T08:44:52.391 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "retain_interval", 2024-04-30T08:44:52.391 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.391 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.391 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.391 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "31536000", 2024-04-30T08:44:52.391 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.391 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.391 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.391 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-30T08:44:52.391 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.391 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.391 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.391 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.391 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.392 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.392 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.392 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.392 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.392 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.392 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.392 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.392 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.392 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.392 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.392 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.392 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.392 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.392 INFO:teuthology.orchestra.run.smithi032.stdout: "warn_recent_interval": { 2024-04-30T08:44:52.393 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "warn_recent_interval", 2024-04-30T08:44:52.393 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.393 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.393 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.393 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "1209600", 2024-04-30T08:44:52.393 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.393 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.393 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.393 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-30T08:44:52.393 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.393 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.393 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.393 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.394 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.394 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.394 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.394 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "dashboard", 2024-04-30T08:44:52.394 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.394 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.394 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.394 INFO:teuthology.orchestra.run.smithi032.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-30T08:44:52.394 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-30T08:44:52.394 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.394 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.394 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.394 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-30T08:44:52.394 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.395 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.395 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.395 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.395 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.395 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.395 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.395 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.395 INFO:teuthology.orchestra.run.smithi032.stdout: "ALERTMANAGER_API_HOST": { 2024-04-30T08:44:52.395 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-30T08:44:52.395 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.395 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.395 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.395 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.395 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.396 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.396 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.396 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.396 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.396 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.396 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.396 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.396 INFO:teuthology.orchestra.run.smithi032.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-30T08:44:52.396 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-30T08:44:52.396 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.396 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.396 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.396 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.396 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.396 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.397 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.397 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.397 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.397 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.397 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.397 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.397 INFO:teuthology.orchestra.run.smithi032.stdout: "AUDIT_API_ENABLED": { 2024-04-30T08:44:52.397 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "AUDIT_API_ENABLED", 2024-04-30T08:44:52.397 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.397 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.397 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.397 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.397 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.397 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.398 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.398 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.398 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.398 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.398 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.398 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.398 INFO:teuthology.orchestra.run.smithi032.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-30T08:44:52.398 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-30T08:44:52.398 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.398 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.398 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.398 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.398 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.399 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.399 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.399 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.399 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.399 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.399 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.399 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.399 INFO:teuthology.orchestra.run.smithi032.stdout: "ENABLE_BROWSABLE_API": { 2024-04-30T08:44:52.399 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-30T08:44:52.399 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.399 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.399 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.399 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.399 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.399 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.400 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.400 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.400 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.400 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.400 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.400 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.400 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-30T08:44:52.400 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-30T08:44:52.400 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.400 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.400 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.400 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.400 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.400 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.401 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.401 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.401 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.401 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.401 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.401 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.401 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-30T08:44:52.401 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-30T08:44:52.401 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.401 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.401 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.401 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.401 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.401 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.402 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.402 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.402 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.402 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.402 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.402 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.402 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-30T08:44:52.402 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-30T08:44:52.402 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.402 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.402 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.402 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.402 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.403 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.403 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.403 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.403 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.403 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.403 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.403 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.403 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-30T08:44:52.403 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-30T08:44:52.403 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.403 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.403 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.403 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.403 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.403 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.404 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.404 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.404 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.404 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.404 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.404 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.404 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-30T08:44:52.404 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-30T08:44:52.404 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.404 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.404 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.404 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.404 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.405 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.405 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.405 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.405 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.405 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.405 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.405 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.405 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-30T08:44:52.405 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-30T08:44:52.405 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.405 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.405 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.405 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.405 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.405 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.406 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.406 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.406 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.406 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.406 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.406 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.406 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-30T08:44:52.406 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-30T08:44:52.406 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.406 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.406 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.406 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.406 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.407 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.407 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.407 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.407 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.407 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.407 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.407 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.407 INFO:teuthology.orchestra.run.smithi032.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-30T08:44:52.407 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-30T08:44:52.407 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.407 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.407 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.407 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.407 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.407 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.408 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.408 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.408 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.408 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.408 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.408 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.408 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_API_PASSWORD": { 2024-04-30T08:44:52.408 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-30T08:44:52.408 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.408 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.408 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.408 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "admin", 2024-04-30T08:44:52.408 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.408 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.409 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.409 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.409 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.409 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.409 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.409 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.409 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-30T08:44:52.409 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-30T08:44:52.409 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.409 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.409 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.409 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.409 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.409 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.410 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.410 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.410 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.410 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.410 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.410 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_API_URL": { 2024-04-30T08:44:52.410 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_API_URL", 2024-04-30T08:44:52.410 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.410 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.410 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.410 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.410 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.410 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.411 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.411 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.411 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.411 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.411 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.411 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.411 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_API_USERNAME": { 2024-04-30T08:44:52.411 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-30T08:44:52.411 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.411 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.411 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.411 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "admin", 2024-04-30T08:44:52.411 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.411 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.412 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.412 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.412 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.412 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.412 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.412 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-30T08:44:52.412 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-30T08:44:52.412 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.412 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.412 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.412 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.412 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.412 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.413 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.413 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.413 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.413 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.413 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.413 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-30T08:44:52.413 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-30T08:44:52.413 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.413 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.413 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.413 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.413 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.413 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.414 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.414 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.414 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.414 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.414 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.414 INFO:teuthology.orchestra.run.smithi032.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-30T08:44:52.414 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-30T08:44:52.414 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.414 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.414 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.414 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.414 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.415 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.415 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.415 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.415 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.415 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.415 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.415 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.415 INFO:teuthology.orchestra.run.smithi032.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-30T08:44:52.415 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-30T08:44:52.415 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.415 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.415 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.415 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.415 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.416 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.416 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.416 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.416 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.416 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.416 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.416 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.416 INFO:teuthology.orchestra.run.smithi032.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-30T08:44:52.416 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-30T08:44:52.416 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.416 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.416 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.416 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "[]", 2024-04-30T08:44:52.416 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.417 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.417 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.417 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.417 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.417 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.417 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.417 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.417 INFO:teuthology.orchestra.run.smithi032.stdout: "MULTICLUSTER_CONFIG": { 2024-04-30T08:44:52.417 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-30T08:44:52.417 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.417 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.417 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.417 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "{}", 2024-04-30T08:44:52.417 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.418 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.418 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.418 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.418 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.418 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.418 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.418 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.418 INFO:teuthology.orchestra.run.smithi032.stdout: "PROMETHEUS_API_HOST": { 2024-04-30T08:44:52.418 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-30T08:44:52.418 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.418 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.418 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.418 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.419 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.419 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.419 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.419 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.419 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.419 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.419 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.419 INFO:teuthology.orchestra.run.smithi032.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-30T08:44:52.419 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-30T08:44:52.419 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.419 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.419 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.419 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.420 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.420 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.420 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.420 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.420 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.420 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.420 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.420 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.420 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-30T08:44:52.420 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-30T08:44:52.420 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.420 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.420 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.420 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.421 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.421 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.421 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.421 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.421 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.421 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.421 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.421 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.421 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-30T08:44:52.421 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-30T08:44:52.421 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.421 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.421 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.421 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.422 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.422 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.422 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.422 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.422 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.422 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.422 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.422 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.422 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-30T08:44:52.422 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-30T08:44:52.422 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.422 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.422 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.422 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.423 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.423 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.423 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.423 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.423 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.423 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.423 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.423 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.423 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-30T08:44:52.423 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-30T08:44:52.423 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.423 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.423 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.423 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.424 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.424 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.424 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.424 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.424 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.424 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.424 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.424 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.424 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-30T08:44:52.424 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-30T08:44:52.424 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.424 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.424 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.424 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.425 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.425 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.425 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.425 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.425 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.425 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.425 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.425 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.425 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-30T08:44:52.425 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-30T08:44:52.425 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.425 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.425 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.425 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.426 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.426 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.426 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.426 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.426 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.426 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.426 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.426 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.426 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-30T08:44:52.426 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-30T08:44:52.426 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.426 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.426 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.427 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.427 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.427 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.427 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.427 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.427 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.427 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.427 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.427 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.427 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_ENABLED": { 2024-04-30T08:44:52.427 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-30T08:44:52.427 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.427 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.427 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.428 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.428 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.428 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.428 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.428 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.428 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.428 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.428 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.428 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.428 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-30T08:44:52.428 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-30T08:44:52.428 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.428 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.428 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.429 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-30T08:44:52.429 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.429 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.429 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.429 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.429 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.429 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.429 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.429 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.429 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-30T08:44:52.429 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-30T08:44:52.429 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.429 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.429 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.430 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-30T08:44:52.430 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.430 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.430 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.430 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.430 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.430 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.430 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.430 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.430 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-30T08:44:52.430 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-30T08:44:52.430 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.430 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.431 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.431 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8", 2024-04-30T08:44:52.431 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.431 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.431 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.431 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.431 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.431 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.431 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.431 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.431 INFO:teuthology.orchestra.run.smithi032.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-30T08:44:52.431 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-30T08:44:52.431 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.431 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.432 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.432 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "45", 2024-04-30T08:44:52.432 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.432 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.432 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.432 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.432 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.432 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.432 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.432 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.432 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_ACCESS_KEY": { 2024-04-30T08:44:52.432 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-30T08:44:52.432 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.432 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.433 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.433 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.433 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.433 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.433 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.433 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.433 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.433 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.433 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.433 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.433 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-30T08:44:52.433 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-30T08:44:52.433 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.433 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.433 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.434 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "admin", 2024-04-30T08:44:52.434 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.434 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.434 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.434 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.434 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.434 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.434 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.434 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.434 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_SECRET_KEY": { 2024-04-30T08:44:52.434 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-30T08:44:52.434 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.434 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.435 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.435 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.435 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.435 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.435 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.435 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.435 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.435 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.435 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.435 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.435 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_SSL_VERIFY": { 2024-04-30T08:44:52.435 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-30T08:44:52.435 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.435 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.435 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.436 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.436 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.436 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.436 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.436 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.436 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.436 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.436 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.436 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.436 INFO:teuthology.orchestra.run.smithi032.stdout: "UNSAFE_TLS_v1_2": { 2024-04-30T08:44:52.436 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-30T08:44:52.436 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.436 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.436 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.436 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.437 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.437 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.437 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.437 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.437 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.437 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.437 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.437 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.437 INFO:teuthology.orchestra.run.smithi032.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-30T08:44:52.437 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-30T08:44:52.437 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.437 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.437 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.437 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.438 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.438 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.438 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.438 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.438 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.438 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.438 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.438 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.438 INFO:teuthology.orchestra.run.smithi032.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-30T08:44:52.438 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-30T08:44:52.438 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.438 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.438 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.438 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-30T08:44:52.439 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.439 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.439 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.439 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.439 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.439 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.439 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.439 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.439 INFO:teuthology.orchestra.run.smithi032.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-30T08:44:52.439 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-30T08:44:52.439 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.439 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.439 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.439 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5", 2024-04-30T08:44:52.440 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.440 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.440 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.440 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.440 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.440 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.440 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.440 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.440 INFO:teuthology.orchestra.run.smithi032.stdout: "cross_origin_url": { 2024-04-30T08:44:52.440 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cross_origin_url", 2024-04-30T08:44:52.440 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.440 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.440 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.440 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.441 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.441 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.441 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.441 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.441 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.441 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.441 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.441 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.441 INFO:teuthology.orchestra.run.smithi032.stdout: "crt_file": { 2024-04-30T08:44:52.441 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crt_file", 2024-04-30T08:44:52.441 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.441 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.441 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.441 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.442 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.442 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.442 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.442 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.442 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.442 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.442 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.442 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.442 INFO:teuthology.orchestra.run.smithi032.stdout: "debug": { 2024-04-30T08:44:52.442 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "debug", 2024-04-30T08:44:52.442 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.442 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.442 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.443 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.443 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.443 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.443 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.443 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Enable/disable debug options", 2024-04-30T08:44:52.443 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.443 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.443 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.443 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.443 INFO:teuthology.orchestra.run.smithi032.stdout: "jwt_token_ttl": { 2024-04-30T08:44:52.443 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "jwt_token_ttl", 2024-04-30T08:44:52.443 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.443 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.443 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.444 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "28800", 2024-04-30T08:44:52.444 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.444 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.444 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.444 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.444 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.444 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.444 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.444 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.444 INFO:teuthology.orchestra.run.smithi032.stdout: "key_file": { 2024-04-30T08:44:52.444 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "key_file", 2024-04-30T08:44:52.444 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.444 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.444 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.444 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.445 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.445 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.445 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.445 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.445 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.445 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.445 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.445 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.445 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.445 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.445 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.445 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.445 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.445 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.446 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.446 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.446 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.446 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.446 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.446 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.446 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.446 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.446 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.446 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.446 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.446 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.446 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.446 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.447 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.447 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.447 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.447 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.447 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.447 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.447 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.447 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.447 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.448 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:51 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2042163201' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-30T08:44:52.451 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.451 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.451 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.451 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.451 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.451 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.451 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.451 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.451 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.452 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.452 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.452 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.452 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.452 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.452 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.452 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.452 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.452 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.452 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.452 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.452 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.452 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.452 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.453 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.453 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.453 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.453 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.453 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.453 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.453 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.453 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.453 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.453 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.453 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.453 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.453 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.453 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.454 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.454 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.454 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.454 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.454 INFO:teuthology.orchestra.run.smithi032.stdout: "motd": { 2024-04-30T08:44:52.454 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "motd", 2024-04-30T08:44:52.454 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.454 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.454 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.454 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.454 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.454 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.454 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.454 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "The message of the day", 2024-04-30T08:44:52.455 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.455 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.455 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.455 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.455 INFO:teuthology.orchestra.run.smithi032.stdout: "redirect_resolve_ip_addr": { 2024-04-30T08:44:52.455 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "redirect_resolve_ip_addr", 2024-04-30T08:44:52.455 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.455 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.455 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.455 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.455 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.455 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.455 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.455 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.456 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.456 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.456 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.456 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.456 INFO:teuthology.orchestra.run.smithi032.stdout: "server_addr": { 2024-04-30T08:44:52.456 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_addr", 2024-04-30T08:44:52.456 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.456 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.456 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.456 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "::", 2024-04-30T08:44:52.456 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.456 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.456 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.457 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.457 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.457 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.457 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.457 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.457 INFO:teuthology.orchestra.run.smithi032.stdout: "server_port": { 2024-04-30T08:44:52.457 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_port", 2024-04-30T08:44:52.457 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.457 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.457 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.457 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8080", 2024-04-30T08:44:52.457 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.457 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.457 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.457 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.458 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.458 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.458 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.458 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.458 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.458 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.458 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.458 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.458 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.458 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.458 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.458 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.458 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.459 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.459 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.459 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.459 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.459 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.459 INFO:teuthology.orchestra.run.smithi032.stdout: "ssl": { 2024-04-30T08:44:52.459 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ssl", 2024-04-30T08:44:52.459 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.459 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.459 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.459 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.459 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.459 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.459 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.460 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.460 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.460 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.460 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.460 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.460 INFO:teuthology.orchestra.run.smithi032.stdout: "ssl_server_port": { 2024-04-30T08:44:52.460 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ssl_server_port", 2024-04-30T08:44:52.460 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.460 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.460 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.460 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8443", 2024-04-30T08:44:52.460 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.460 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.460 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.460 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.461 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.461 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.461 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.461 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.461 INFO:teuthology.orchestra.run.smithi032.stdout: "standby_behaviour": { 2024-04-30T08:44:52.461 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "standby_behaviour", 2024-04-30T08:44:52.461 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.461 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.461 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.461 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "redirect", 2024-04-30T08:44:52.461 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.461 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.461 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.461 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.462 INFO:teuthology.orchestra.run.smithi032.stdout: "redirect" 2024-04-30T08:44:52.462 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.462 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.462 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.462 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.462 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.462 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.462 INFO:teuthology.orchestra.run.smithi032.stdout: "standby_error_status_code": { 2024-04-30T08:44:52.462 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "standby_error_status_code", 2024-04-30T08:44:52.462 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.462 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.462 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.462 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "500", 2024-04-30T08:44:52.463 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "400", 2024-04-30T08:44:52.463 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "599", 2024-04-30T08:44:52.463 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.463 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.463 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.463 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.463 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.463 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.463 INFO:teuthology.orchestra.run.smithi032.stdout: "url_prefix": { 2024-04-30T08:44:52.463 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "url_prefix", 2024-04-30T08:44:52.463 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.463 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.463 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.463 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.464 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.464 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.464 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.464 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.464 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.464 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.464 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.464 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.464 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.464 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.464 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.464 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "devicehealth", 2024-04-30T08:44:52.464 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.464 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.465 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.465 INFO:teuthology.orchestra.run.smithi032.stdout: "enable_monitoring": { 2024-04-30T08:44:52.465 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "enable_monitoring", 2024-04-30T08:44:52.465 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.465 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.465 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.465 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.465 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.465 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.465 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.465 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "monitor device health metrics", 2024-04-30T08:44:52.465 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.465 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.465 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.466 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.466 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.466 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.466 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.466 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.466 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.466 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.466 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.466 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.466 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.466 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.466 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.466 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.467 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.467 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.467 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.467 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.467 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.467 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.467 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.467 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.467 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.467 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.467 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.467 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.467 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.467 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.468 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.468 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.468 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.468 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.468 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.468 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.468 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.468 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.468 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.468 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.468 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.468 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.468 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.468 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.469 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.469 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.469 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.469 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.469 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.469 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.469 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.469 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.469 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.469 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.469 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.469 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.469 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.469 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.470 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.470 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.470 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.470 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.470 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.470 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.470 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.470 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.470 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.470 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.470 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.470 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.470 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.471 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.471 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.471 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.471 INFO:teuthology.orchestra.run.smithi032.stdout: "mark_out_threshold": { 2024-04-30T08:44:52.471 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mark_out_threshold", 2024-04-30T08:44:52.471 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.471 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.471 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.471 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "2419200", 2024-04-30T08:44:52.471 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.471 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.471 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.471 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-30T08:44:52.471 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.472 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.472 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.472 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.472 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": { 2024-04-30T08:44:52.472 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "pool_name", 2024-04-30T08:44:52.472 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.472 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.472 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.472 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "device_health_metrics", 2024-04-30T08:44:52.472 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.472 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.472 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.472 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-30T08:44:52.472 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.473 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.473 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.473 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.473 INFO:teuthology.orchestra.run.smithi032.stdout: "retention_period": { 2024-04-30T08:44:52.473 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "retention_period", 2024-04-30T08:44:52.473 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.473 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.473 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.473 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "15552000", 2024-04-30T08:44:52.473 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.473 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.473 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.473 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how long to retain device health metrics", 2024-04-30T08:44:52.473 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.474 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.474 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.474 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.474 INFO:teuthology.orchestra.run.smithi032.stdout: "scrape_frequency": { 2024-04-30T08:44:52.474 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "scrape_frequency", 2024-04-30T08:44:52.474 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.474 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.474 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.474 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "86400", 2024-04-30T08:44:52.474 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.474 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.474 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.474 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-30T08:44:52.474 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.475 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.475 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.475 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.475 INFO:teuthology.orchestra.run.smithi032.stdout: "self_heal": { 2024-04-30T08:44:52.475 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "self_heal", 2024-04-30T08:44:52.475 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.475 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.475 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.475 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.475 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.475 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.475 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.475 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-30T08:44:52.475 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.476 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.476 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.476 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.476 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2024-04-30T08:44:52.476 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2024-04-30T08:44:52.476 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.476 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.476 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.476 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2024-04-30T08:44:52.476 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.476 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.476 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.476 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to wake up and check device health", 2024-04-30T08:44:52.476 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.476 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.477 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.477 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.477 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.477 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.477 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.477 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.477 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.477 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.477 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.477 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.477 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.477 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.477 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.477 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.478 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.478 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.478 INFO:teuthology.orchestra.run.smithi032.stdout: "warn_threshold": { 2024-04-30T08:44:52.478 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "warn_threshold", 2024-04-30T08:44:52.478 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.478 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.478 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.478 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "7257600", 2024-04-30T08:44:52.478 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.478 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.478 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.478 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-30T08:44:52.478 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.479 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.479 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.479 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.479 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.479 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.479 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.479 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "diskprediction_local", 2024-04-30T08:44:52.479 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.479 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.479 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.479 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.479 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.479 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.479 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.480 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.480 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.480 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.480 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.480 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.480 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.480 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.480 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.480 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.480 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.480 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.480 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.480 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.480 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.480 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.481 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.481 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.481 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.481 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.481 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.481 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.481 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.481 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.481 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.481 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.481 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.481 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.481 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.481 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.482 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.482 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.482 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.482 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.482 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.482 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.482 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.482 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.482 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.482 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.482 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.482 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.482 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.482 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.483 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.483 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.483 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.483 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.483 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.483 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.483 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.483 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.483 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.483 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.483 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.483 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.483 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.484 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.484 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.484 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.484 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.484 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.484 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.484 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.484 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.484 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.484 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.484 INFO:teuthology.orchestra.run.smithi032.stdout: "predict_interval": { 2024-04-30T08:44:52.484 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "predict_interval", 2024-04-30T08:44:52.484 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.484 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.485 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.485 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "86400", 2024-04-30T08:44:52.485 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.485 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.485 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.485 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.485 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.485 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.485 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.485 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.485 INFO:teuthology.orchestra.run.smithi032.stdout: "predictor_model": { 2024-04-30T08:44:52.485 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "predictor_model", 2024-04-30T08:44:52.485 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.485 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.485 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.486 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "prophetstor", 2024-04-30T08:44:52.486 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.486 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.486 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.486 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.486 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.486 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.486 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.486 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.486 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2024-04-30T08:44:52.486 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2024-04-30T08:44:52.486 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.486 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.487 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.487 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2024-04-30T08:44:52.487 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.487 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.487 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.487 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.487 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.487 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.487 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.487 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.487 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.487 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.487 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.487 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.487 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.488 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.488 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.488 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.488 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.488 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.488 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.488 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.488 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.488 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.488 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.488 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.488 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.488 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "influx", 2024-04-30T08:44:52.488 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": false, 2024-04-30T08:44:52.489 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "influxdb python module not found", 2024-04-30T08:44:52.489 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.489 INFO:teuthology.orchestra.run.smithi032.stdout: "batch_size": { 2024-04-30T08:44:52.489 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "batch_size", 2024-04-30T08:44:52.489 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.489 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.489 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.489 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5000", 2024-04-30T08:44:52.489 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.489 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.489 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.489 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-30T08:44:52.489 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.489 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.490 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.490 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.490 INFO:teuthology.orchestra.run.smithi032.stdout: "database": { 2024-04-30T08:44:52.490 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "database", 2024-04-30T08:44:52.490 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.490 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.490 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.490 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "ceph", 2024-04-30T08:44:52.490 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.490 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.490 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.490 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-30T08:44:52.490 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.490 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.491 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.491 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.491 INFO:teuthology.orchestra.run.smithi032.stdout: "hostname": { 2024-04-30T08:44:52.491 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "hostname", 2024-04-30T08:44:52.491 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.491 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.491 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.491 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.491 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.491 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.491 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.492 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "InfluxDB server hostname", 2024-04-30T08:44:52.492 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.492 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.492 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.492 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.492 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2024-04-30T08:44:52.492 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2024-04-30T08:44:52.492 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.492 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.492 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.492 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "30", 2024-04-30T08:44:52.492 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "5", 2024-04-30T08:44:52.492 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.493 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.493 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-30T08:44:52.493 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.493 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.493 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.493 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.493 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.493 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.493 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.493 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.493 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.493 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.493 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.493 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.494 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.494 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.494 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.494 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.494 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.494 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.494 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.494 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.494 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.494 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.494 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.494 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.494 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.494 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.495 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.495 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.495 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.495 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.495 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.495 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.495 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.495 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.495 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.495 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.495 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.495 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.495 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.495 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.496 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.496 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.496 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.496 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.496 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.496 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.496 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.496 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.496 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.496 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.496 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.496 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.496 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.496 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.497 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.497 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.497 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.497 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.497 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.497 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.497 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.497 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.497 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.497 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.497 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.497 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.497 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.498 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.498 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.498 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.498 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.498 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.498 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.498 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.498 INFO:teuthology.orchestra.run.smithi032.stdout: "password": { 2024-04-30T08:44:52.498 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "password", 2024-04-30T08:44:52.498 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.498 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.498 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.498 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.498 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.499 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.499 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.499 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "password of InfluxDB server user", 2024-04-30T08:44:52.499 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.499 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.499 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.499 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.499 INFO:teuthology.orchestra.run.smithi032.stdout: "port": { 2024-04-30T08:44:52.499 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "port", 2024-04-30T08:44:52.499 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.499 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.499 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.499 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8086", 2024-04-30T08:44:52.499 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.500 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.500 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.500 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "InfluxDB server port", 2024-04-30T08:44:52.500 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.500 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.500 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.500 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.500 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.500 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.500 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.500 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.500 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.500 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.500 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.501 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.501 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.501 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.501 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.501 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.501 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.501 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.501 INFO:teuthology.orchestra.run.smithi032.stdout: "ssl": { 2024-04-30T08:44:52.501 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ssl", 2024-04-30T08:44:52.501 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.501 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.501 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.501 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "false", 2024-04-30T08:44:52.501 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.502 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.502 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.502 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-30T08:44:52.502 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.502 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.502 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.502 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.502 INFO:teuthology.orchestra.run.smithi032.stdout: "threads": { 2024-04-30T08:44:52.502 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "threads", 2024-04-30T08:44:52.502 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.502 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.502 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.502 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5", 2024-04-30T08:44:52.502 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "1", 2024-04-30T08:44:52.503 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "32", 2024-04-30T08:44:52.503 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.503 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-30T08:44:52.503 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.503 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.503 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.503 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.503 INFO:teuthology.orchestra.run.smithi032.stdout: "username": { 2024-04-30T08:44:52.503 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "username", 2024-04-30T08:44:52.503 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.503 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.503 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.503 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.504 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.504 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.504 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.504 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "username of InfluxDB server user", 2024-04-30T08:44:52.504 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.504 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.504 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.504 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.504 INFO:teuthology.orchestra.run.smithi032.stdout: "verify_ssl": { 2024-04-30T08:44:52.504 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "verify_ssl", 2024-04-30T08:44:52.504 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.504 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.504 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.504 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "true", 2024-04-30T08:44:52.504 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.505 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.505 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.505 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-30T08:44:52.505 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.505 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.505 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.505 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.505 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.505 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.505 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.505 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "insights", 2024-04-30T08:44:52.505 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.506 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.506 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.506 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.506 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.506 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.506 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.506 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.506 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.506 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.506 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.506 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.506 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.506 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.506 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.507 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.507 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.507 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.507 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.507 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.507 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.507 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.507 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.507 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.507 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.507 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.507 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.507 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.507 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.508 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.508 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.508 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.508 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.508 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.508 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.508 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.508 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.508 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.508 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.508 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.508 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.508 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.508 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.509 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.509 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.509 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.509 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.509 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.509 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.509 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.509 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.509 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.509 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.509 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.509 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.509 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.509 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.510 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.510 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.510 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.510 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.510 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.510 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.510 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.510 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.510 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.510 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.510 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.510 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.510 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.511 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.511 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.511 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.511 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.511 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.511 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.511 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.511 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.511 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.511 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.511 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.511 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.511 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.511 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.511 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.512 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.512 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.512 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.512 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.512 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.512 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "iostat", 2024-04-30T08:44:52.512 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.512 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.512 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.512 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.512 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.512 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.512 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.512 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.513 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.513 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.513 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.513 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.513 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.513 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.513 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.513 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.513 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.513 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.513 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.513 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.513 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.514 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.514 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.514 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.514 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.514 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.514 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.514 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.514 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.514 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.514 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.514 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.514 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.514 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.514 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.515 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.515 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.515 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.515 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.515 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.515 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.515 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.515 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.515 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.515 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.515 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.515 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.515 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.515 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.516 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.516 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.516 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.516 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.516 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.516 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.516 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.516 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.516 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.516 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.516 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.516 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.516 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.516 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.517 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.517 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.517 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.517 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.517 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.517 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.517 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.517 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.517 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.517 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.517 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.517 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.517 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.517 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.518 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.518 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.518 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.518 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.518 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.518 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.518 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.518 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.518 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.518 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.518 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.518 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.518 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.519 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "k8sevents", 2024-04-30T08:44:52.519 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.519 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.519 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.519 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_event_retention_days": { 2024-04-30T08:44:52.519 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ceph_event_retention_days", 2024-04-30T08:44:52.519 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.519 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.519 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.519 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "7", 2024-04-30T08:44:52.519 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.519 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.519 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.520 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-30T08:44:52.520 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.520 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.520 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.520 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.520 INFO:teuthology.orchestra.run.smithi032.stdout: "config_check_secs": { 2024-04-30T08:44:52.520 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "config_check_secs", 2024-04-30T08:44:52.520 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.520 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.520 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.520 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-30T08:44:52.520 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "10", 2024-04-30T08:44:52.520 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.520 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.520 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-30T08:44:52.521 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.521 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.521 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.521 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.521 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.521 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.521 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.521 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.521 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.521 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.521 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.521 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.521 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.521 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.522 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.522 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.522 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.522 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.522 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.522 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.522 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.522 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.522 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.522 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.522 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.522 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.522 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.522 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.523 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.523 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.523 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.523 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.523 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.523 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.523 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.523 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.523 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.523 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.523 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.523 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.523 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.523 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.524 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.524 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.524 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.524 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.524 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.524 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.524 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.524 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.524 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.524 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.524 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.524 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.524 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.524 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.524 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.525 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.525 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.525 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.525 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.525 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.525 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.525 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.525 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.525 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.525 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.525 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.525 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.525 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.525 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.525 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.526 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.526 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.526 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.526 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.526 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.526 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.526 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.526 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.526 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.526 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.526 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.526 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.526 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.527 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.527 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.527 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.527 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.527 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.527 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.527 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "localpool", 2024-04-30T08:44:52.527 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.527 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.527 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.527 INFO:teuthology.orchestra.run.smithi032.stdout: "failure_domain": { 2024-04-30T08:44:52.527 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "failure_domain", 2024-04-30T08:44:52.527 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.527 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.528 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.528 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "host", 2024-04-30T08:44:52.528 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.528 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.528 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.528 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "failure domain for any created local pool", 2024-04-30T08:44:52.528 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-30T08:44:52.528 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.528 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.528 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.528 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.528 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.528 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.528 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.528 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.529 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.529 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.529 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.529 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.529 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.529 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.529 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.529 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.529 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.529 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.529 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.529 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.529 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.529 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.529 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.530 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.530 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.530 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.530 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.530 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.530 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.530 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.530 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.530 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.530 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.530 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.530 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.530 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.530 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.531 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.531 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.531 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.531 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.531 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.531 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.531 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.531 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.531 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.531 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.531 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.531 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.531 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.531 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.531 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.532 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.532 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.532 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.532 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.532 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.532 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.532 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.532 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.532 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.532 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.532 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.532 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.532 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.532 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.532 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.533 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.533 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.533 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.533 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.533 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.533 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.533 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": { 2024-04-30T08:44:52.533 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "min_size", 2024-04-30T08:44:52.533 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.533 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.533 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.533 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.533 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.533 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.533 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.534 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "default min_size for any created local pool", 2024-04-30T08:44:52.534 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-30T08:44:52.534 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.534 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.534 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.534 INFO:teuthology.orchestra.run.smithi032.stdout: "num_rep": { 2024-04-30T08:44:52.534 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "num_rep", 2024-04-30T08:44:52.534 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.534 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.534 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.534 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "3", 2024-04-30T08:44:52.534 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.534 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.535 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.535 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "default replica count for any created local pool", 2024-04-30T08:44:52.535 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.535 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.535 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.535 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.535 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": { 2024-04-30T08:44:52.535 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "pg_num", 2024-04-30T08:44:52.535 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.535 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.535 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.535 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "128", 2024-04-30T08:44:52.535 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.536 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.536 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.536 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "default pg_num for any created local pool", 2024-04-30T08:44:52.536 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.536 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.536 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.536 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.536 INFO:teuthology.orchestra.run.smithi032.stdout: "prefix": { 2024-04-30T08:44:52.536 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "prefix", 2024-04-30T08:44:52.536 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.536 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.536 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.536 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.537 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.537 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.537 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.537 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "name prefix for any created local pool", 2024-04-30T08:44:52.537 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.537 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.537 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.537 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.537 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.537 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.537 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.537 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.537 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.537 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.538 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.538 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.538 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.538 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.538 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.538 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.538 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.538 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.538 INFO:teuthology.orchestra.run.smithi032.stdout: "subtree": { 2024-04-30T08:44:52.538 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "subtree", 2024-04-30T08:44:52.538 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.538 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.538 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.539 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "rack", 2024-04-30T08:44:52.539 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.539 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.539 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.539 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-30T08:44:52.539 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-30T08:44:52.539 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.539 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.539 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.539 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.539 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.539 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.539 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mds_autoscaler", 2024-04-30T08:44:52.539 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.539 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.541 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.541 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.541 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.541 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.541 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.541 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.541 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.542 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.542 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.542 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.542 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.542 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.542 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.542 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.542 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.542 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.542 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.542 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.542 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.542 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.542 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.543 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.543 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.543 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.543 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.543 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.543 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.543 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.543 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.543 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.543 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.543 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.543 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.543 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.544 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.544 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.544 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.544 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.544 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.544 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.544 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.544 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.544 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.544 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.544 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.544 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.544 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.544 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.544 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.545 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.545 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.545 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.545 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.545 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.545 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.545 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.545 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.545 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.545 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.545 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.545 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.545 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.545 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.546 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.546 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.546 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.546 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.546 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.546 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.546 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.546 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.546 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.546 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.546 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.546 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.546 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.546 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.547 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.547 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.547 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.547 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.547 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.547 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.547 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.547 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.547 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.547 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.547 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.547 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mirroring", 2024-04-30T08:44:52.547 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.547 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.548 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.548 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.548 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.548 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.548 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.548 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.548 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.548 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.548 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.548 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.548 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.548 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.549 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.549 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.549 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.549 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.549 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.549 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.549 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.549 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.549 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.549 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.549 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.549 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.549 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.549 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.550 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.550 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.550 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.550 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.550 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.550 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.550 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.550 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.550 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.550 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.550 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.550 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.550 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.551 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.551 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.551 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.551 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.551 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.551 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.551 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.551 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.551 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.551 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.551 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.551 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.551 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.551 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.552 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.552 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.552 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.552 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.552 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.552 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.552 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.552 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.552 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.552 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.552 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.552 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.552 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.553 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.553 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.553 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.553 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.553 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.553 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.553 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.553 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.553 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.553 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.553 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.553 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.553 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.553 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.554 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.554 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.554 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.554 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.554 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.554 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.554 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.554 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.554 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "nfs", 2024-04-30T08:44:52.554 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.554 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.554 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.554 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.555 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.555 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.555 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.555 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.555 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.555 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.555 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.555 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.555 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.555 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.555 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.555 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.555 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.555 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.555 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.556 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.556 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.556 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.556 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.556 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.556 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.556 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.556 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.556 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.556 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.556 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.556 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.556 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.556 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.557 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.557 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.557 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.557 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.557 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.557 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.557 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.557 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.557 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.557 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.557 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.557 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.557 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.558 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.558 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.558 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.558 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.558 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.558 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.558 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.558 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.558 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.558 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.558 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.558 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.558 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.558 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.559 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.559 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.559 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.559 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.559 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.559 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.559 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.559 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.559 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.559 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.559 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.559 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.559 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.559 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.560 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.560 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.560 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.560 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.560 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.560 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.560 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.560 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.560 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.560 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.560 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.560 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.560 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.560 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.560 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.561 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.561 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "orchestrator", 2024-04-30T08:44:52.561 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.561 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.561 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.561 INFO:teuthology.orchestra.run.smithi032.stdout: "fail_fs": { 2024-04-30T08:44:52.561 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "fail_fs", 2024-04-30T08:44:52.561 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.561 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.561 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.561 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.561 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.561 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.562 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.562 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-30T08:44:52.562 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.562 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.562 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.562 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.562 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.562 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.562 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.562 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.562 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.562 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.562 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.563 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.563 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.563 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.563 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.563 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.563 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.563 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.563 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.563 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.563 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.563 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.563 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.563 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.563 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.564 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.564 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.564 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.564 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.564 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.564 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.564 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.564 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.564 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.564 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.564 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.564 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.564 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.564 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.565 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.565 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.565 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.565 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.565 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.565 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.565 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.565 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.565 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.565 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.565 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.565 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.565 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.566 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.566 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.566 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.566 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.566 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.566 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.566 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.566 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.566 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.566 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.566 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.566 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.566 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.567 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.567 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.567 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.567 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.567 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.567 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.567 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.567 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.567 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.567 INFO:teuthology.orchestra.run.smithi032.stdout: "orchestrator": { 2024-04-30T08:44:52.567 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "orchestrator", 2024-04-30T08:44:52.567 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.567 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.567 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.567 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.568 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.568 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.568 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.568 INFO:teuthology.orchestra.run.smithi032.stdout: "cephadm", 2024-04-30T08:44:52.568 INFO:teuthology.orchestra.run.smithi032.stdout: "rook", 2024-04-30T08:44:52.568 INFO:teuthology.orchestra.run.smithi032.stdout: "test_orchestrator" 2024-04-30T08:44:52.568 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.568 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Orchestrator backend", 2024-04-30T08:44:52.568 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.568 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.568 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.568 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.568 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.568 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.569 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.569 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.569 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.569 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.569 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.569 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.569 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.569 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.569 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.569 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.569 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.569 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.569 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.569 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.569 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.570 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "osd_perf_query", 2024-04-30T08:44:52.570 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.570 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.570 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.570 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.570 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.570 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.570 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.570 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.570 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.570 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.570 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.570 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.570 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.571 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.571 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.571 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.571 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.571 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.571 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.571 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.571 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.571 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.571 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.571 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.571 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.571 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.571 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.572 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.572 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.572 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.572 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.572 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.572 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.572 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.572 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.572 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.572 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.572 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.572 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.572 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.572 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.573 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.573 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.573 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.573 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.573 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.573 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.573 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.573 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.573 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.573 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.573 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.573 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.573 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.573 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.574 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.574 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.574 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.574 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.574 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.574 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.574 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.574 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.574 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.574 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.574 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.574 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.574 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.575 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.575 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.575 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.575 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.575 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.575 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.575 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.575 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.575 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.575 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.575 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.575 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.575 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.575 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.576 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.576 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.576 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.576 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.576 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.576 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.576 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.576 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.576 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "osd_support", 2024-04-30T08:44:52.576 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.576 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.576 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.576 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.576 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.577 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.577 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.577 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.577 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.577 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.577 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.577 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.577 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.577 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.577 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.577 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.577 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.577 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.577 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.578 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.578 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.578 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.578 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.578 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.578 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.578 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.578 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.578 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.578 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.578 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.578 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.578 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.579 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.579 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.579 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.579 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.579 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.579 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.579 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.579 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.579 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.579 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.579 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.579 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.579 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.579 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.579 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.580 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.580 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.580 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.580 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.580 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.580 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.580 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.580 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.580 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.580 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.580 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.580 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.580 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.581 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.581 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.581 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.581 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.581 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.581 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.581 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.581 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.581 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.581 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.581 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.581 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.581 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.581 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.581 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.582 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.582 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.582 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.582 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.582 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.582 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.582 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.582 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.582 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.582 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.582 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.582 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.582 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.583 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.583 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.583 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "pg_autoscaler", 2024-04-30T08:44:52.583 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.583 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.583 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.583 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.583 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.583 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.583 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.583 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.583 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.583 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.583 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.584 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.584 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.584 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.584 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.584 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.584 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.584 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.584 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.584 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.584 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.584 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.584 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.584 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.584 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.585 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.585 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.585 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.585 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.585 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.585 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.585 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.585 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.585 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.585 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.585 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.585 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.585 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.585 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.585 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.586 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.586 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.586 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.586 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.586 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.586 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.586 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.586 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.586 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.586 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.586 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.586 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.586 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.587 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.587 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.587 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.587 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.587 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.587 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.587 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.587 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.587 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.587 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.587 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.587 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.587 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.587 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.588 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.588 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.588 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.588 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.588 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.588 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.588 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2024-04-30T08:44:52.588 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2024-04-30T08:44:52.588 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.588 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.588 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.588 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2024-04-30T08:44:52.588 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.588 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.588 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.589 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.589 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.589 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.589 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.589 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.589 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.589 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.589 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.589 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.589 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.589 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.589 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.589 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.589 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.590 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.590 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.590 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.590 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.590 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.590 INFO:teuthology.orchestra.run.smithi032.stdout: "threshold": { 2024-04-30T08:44:52.590 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "threshold", 2024-04-30T08:44:52.590 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-30T08:44:52.590 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.590 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.590 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "3.0", 2024-04-30T08:44:52.590 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "1.0", 2024-04-30T08:44:52.590 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.591 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.591 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "scaling threshold", 2024-04-30T08:44:52.591 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-30T08:44:52.591 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.591 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.591 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.591 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.591 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.591 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.591 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "progress", 2024-04-30T08:44:52.591 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.591 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.591 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.592 INFO:teuthology.orchestra.run.smithi032.stdout: "allow_pg_recovery_event": { 2024-04-30T08:44:52.592 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "allow_pg_recovery_event", 2024-04-30T08:44:52.592 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.592 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.592 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.592 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.592 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.592 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.592 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.592 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-30T08:44:52.592 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.592 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.592 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.592 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.593 INFO:teuthology.orchestra.run.smithi032.stdout: "enabled": { 2024-04-30T08:44:52.593 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "enabled", 2024-04-30T08:44:52.593 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.593 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.593 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.593 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.593 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.593 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.593 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.593 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.593 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.593 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.593 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.594 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.594 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.594 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.594 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.594 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.594 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.594 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.594 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.594 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.594 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.594 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.594 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.594 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.594 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.595 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.595 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.595 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.595 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.595 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.595 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.595 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.595 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.595 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.595 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.595 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.595 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.595 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.595 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.596 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.596 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.596 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.596 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.596 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.596 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.596 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.596 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.596 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.596 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.596 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.596 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.596 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.596 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.597 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.597 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.597 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.597 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.597 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.597 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.597 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.597 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.597 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.597 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.597 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.597 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.597 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.597 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.598 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.598 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.598 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.598 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.598 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.598 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.598 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.598 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.598 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.598 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.598 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.598 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.598 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.599 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.599 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.599 INFO:teuthology.orchestra.run.smithi032.stdout: "max_completed_events": { 2024-04-30T08:44:52.599 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "max_completed_events", 2024-04-30T08:44:52.599 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.599 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.599 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.599 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "50", 2024-04-30T08:44:52.599 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.599 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.599 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.599 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "number of past completed events to remember", 2024-04-30T08:44:52.599 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.599 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.600 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.600 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.600 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2024-04-30T08:44:52.600 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2024-04-30T08:44:52.600 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.600 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.600 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.600 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5", 2024-04-30T08:44:52.600 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.600 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.600 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.600 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how long the module is going to sleep", 2024-04-30T08:44:52.600 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.600 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.601 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.601 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.601 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.601 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.601 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.601 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.601 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.601 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.601 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.601 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.601 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.601 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.601 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.601 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.602 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.602 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.602 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.602 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.602 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.602 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "prometheus", 2024-04-30T08:44:52.602 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.602 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.602 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.602 INFO:teuthology.orchestra.run.smithi032.stdout: "cache": { 2024-04-30T08:44:52.602 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cache", 2024-04-30T08:44:52.602 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.602 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.603 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.603 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.603 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.603 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.603 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.603 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.603 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.603 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.603 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.603 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.603 INFO:teuthology.orchestra.run.smithi032.stdout: "exclude_perf_counters": { 2024-04-30T08:44:52.603 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "exclude_perf_counters", 2024-04-30T08:44:52.603 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.603 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.604 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.604 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.604 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.604 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.604 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.604 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-30T08:44:52.604 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-30T08:44:52.604 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.604 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.604 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.604 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.604 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.604 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.605 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.605 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.605 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.605 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.605 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.605 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.605 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.605 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.605 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.605 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.605 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.605 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.605 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.605 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.606 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.606 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.606 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.606 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.606 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.606 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.606 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.606 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.606 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.606 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.606 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.606 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.606 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.607 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.607 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.607 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.607 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.607 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.607 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.607 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.607 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.607 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.607 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.607 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.607 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.607 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.608 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.608 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.608 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.608 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.608 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.608 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.608 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.608 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.608 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.608 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.608 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.608 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.608 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.609 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.609 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.609 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.609 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.609 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.609 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.609 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.609 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.609 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.609 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.609 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.609 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.609 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.609 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.609 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_stats_pools": { 2024-04-30T08:44:52.610 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rbd_stats_pools", 2024-04-30T08:44:52.610 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.610 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.610 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.610 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.610 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.610 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.610 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.610 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.610 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.610 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.610 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.610 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.611 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-30T08:44:52.611 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-30T08:44:52.611 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.611 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.611 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.611 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "300", 2024-04-30T08:44:52.611 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.611 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.611 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.611 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.611 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.611 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.611 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.611 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.612 INFO:teuthology.orchestra.run.smithi032.stdout: "scrape_interval": { 2024-04-30T08:44:52.612 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "scrape_interval", 2024-04-30T08:44:52.612 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-30T08:44:52.612 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.612 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.612 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "15.0", 2024-04-30T08:44:52.612 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.612 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.612 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.612 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.612 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.612 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.612 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.612 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.612 INFO:teuthology.orchestra.run.smithi032.stdout: "server_addr": { 2024-04-30T08:44:52.613 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_addr", 2024-04-30T08:44:52.613 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.613 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.613 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.613 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "::", 2024-04-30T08:44:52.613 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.613 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.613 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.613 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-30T08:44:52.613 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.613 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.613 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.613 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.613 INFO:teuthology.orchestra.run.smithi032.stdout: "server_port": { 2024-04-30T08:44:52.614 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_port", 2024-04-30T08:44:52.614 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.614 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.614 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.614 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "9283", 2024-04-30T08:44:52.614 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.614 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.614 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.614 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-30T08:44:52.614 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.614 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.614 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.614 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.615 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.615 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.615 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.615 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.615 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.615 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.615 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.615 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.615 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.615 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.615 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.615 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.615 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.615 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.615 INFO:teuthology.orchestra.run.smithi032.stdout: "stale_cache_strategy": { 2024-04-30T08:44:52.615 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "stale_cache_strategy", 2024-04-30T08:44:52.616 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.616 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.616 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.616 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "log", 2024-04-30T08:44:52.616 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.616 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.616 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.616 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.616 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.616 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.616 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.616 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.616 INFO:teuthology.orchestra.run.smithi032.stdout: "standby_behaviour": { 2024-04-30T08:44:52.616 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "standby_behaviour", 2024-04-30T08:44:52.617 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.617 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.617 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.617 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "default", 2024-04-30T08:44:52.617 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.617 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.617 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.617 INFO:teuthology.orchestra.run.smithi032.stdout: "default", 2024-04-30T08:44:52.617 INFO:teuthology.orchestra.run.smithi032.stdout: "error" 2024-04-30T08:44:52.617 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.617 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.617 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.617 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.618 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.618 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.618 INFO:teuthology.orchestra.run.smithi032.stdout: "standby_error_status_code": { 2024-04-30T08:44:52.618 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "standby_error_status_code", 2024-04-30T08:44:52.618 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.618 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.618 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.618 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "500", 2024-04-30T08:44:52.618 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "400", 2024-04-30T08:44:52.618 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "599", 2024-04-30T08:44:52.618 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.618 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.618 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.618 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.619 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.619 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.619 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.619 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.619 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.619 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rbd_support", 2024-04-30T08:44:52.619 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.619 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.619 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.619 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.619 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.619 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.619 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.620 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.620 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.620 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.620 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.620 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.620 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.620 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.620 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.620 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.620 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.620 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.620 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.620 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.620 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.621 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.621 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.621 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.621 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.621 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.621 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.621 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.621 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.621 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.621 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.621 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.621 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.621 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.621 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.622 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.622 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.622 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.622 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.622 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.622 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.622 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.622 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.622 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.622 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.622 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.622 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.622 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.622 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.623 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.623 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.623 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.623 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.623 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.623 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.623 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.623 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.623 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.623 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.623 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.623 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.623 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.623 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.624 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.624 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.624 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.624 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.624 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.624 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.624 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.624 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.624 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.624 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.624 INFO:teuthology.orchestra.run.smithi032.stdout: "max_concurrent_snap_create": { 2024-04-30T08:44:52.624 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "max_concurrent_snap_create", 2024-04-30T08:44:52.624 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.624 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.624 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.625 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-30T08:44:52.625 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.625 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.625 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.625 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.625 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.625 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.625 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.625 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.625 INFO:teuthology.orchestra.run.smithi032.stdout: "mirror_snapshot_schedule": { 2024-04-30T08:44:52.625 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mirror_snapshot_schedule", 2024-04-30T08:44:52.625 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.625 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.625 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.625 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.626 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.626 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.626 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.626 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.626 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.626 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.626 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.626 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.626 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.626 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.626 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.626 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.626 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.626 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.627 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.627 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.627 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.627 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.627 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.627 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.627 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.627 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.627 INFO:teuthology.orchestra.run.smithi032.stdout: "trash_purge_schedule": { 2024-04-30T08:44:52.627 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "trash_purge_schedule", 2024-04-30T08:44:52.627 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.627 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.627 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.627 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.627 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.628 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.628 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.628 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.628 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.628 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.628 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.628 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.628 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.628 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.628 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.628 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "restful", 2024-04-30T08:44:52.628 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.628 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.628 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.629 INFO:teuthology.orchestra.run.smithi032.stdout: "enable_auth": { 2024-04-30T08:44:52.629 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "enable_auth", 2024-04-30T08:44:52.629 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.629 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.629 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.629 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.629 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.629 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.629 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.629 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.629 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.629 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.629 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.629 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.630 INFO:teuthology.orchestra.run.smithi032.stdout: "key_file": { 2024-04-30T08:44:52.630 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "key_file", 2024-04-30T08:44:52.630 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.630 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.630 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.630 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.630 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.630 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.630 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.630 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.630 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.630 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.630 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.631 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.631 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.631 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.631 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.631 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.631 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.631 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.631 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.631 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.631 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.631 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.631 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.631 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.631 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.632 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.632 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.632 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.632 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.632 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.632 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.632 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.632 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.632 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.632 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.632 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.632 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.632 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.632 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.633 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.633 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.633 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.633 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.633 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.633 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.633 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.633 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.633 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.633 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.633 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.633 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.633 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.633 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.634 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.634 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.634 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.634 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.634 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.634 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.634 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.634 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.634 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.634 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.634 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.634 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.634 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.635 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.635 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.635 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.635 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.635 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.635 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.635 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.635 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.635 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.635 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.635 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.635 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.635 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.635 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.636 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.636 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.636 INFO:teuthology.orchestra.run.smithi032.stdout: "server_addr": { 2024-04-30T08:44:52.636 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_addr", 2024-04-30T08:44:52.636 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.636 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.636 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.636 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.636 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.636 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.636 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.636 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.636 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.636 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.637 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.637 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.637 INFO:teuthology.orchestra.run.smithi032.stdout: "server_port": { 2024-04-30T08:44:52.637 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_port", 2024-04-30T08:44:52.637 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.637 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.637 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.637 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.637 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.637 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.637 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.637 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.637 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.637 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.637 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.638 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.638 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.638 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.638 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.638 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.638 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.638 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.638 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.638 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.638 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.638 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.638 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.638 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.638 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.639 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.639 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.639 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.639 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.639 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rgw", 2024-04-30T08:44:52.639 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.639 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.639 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.639 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.639 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.639 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.639 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.639 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.639 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.640 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.640 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.640 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.640 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.640 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.640 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.640 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.640 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.640 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.640 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.640 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.640 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.640 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.640 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.641 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.641 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.641 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.641 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.641 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.641 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.641 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.641 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.641 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.641 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.641 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.641 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.641 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.642 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.642 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.642 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.642 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.642 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.642 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.642 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.642 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.642 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.642 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.642 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.642 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.642 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.642 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.643 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.643 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.643 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.643 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.643 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.643 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.643 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.643 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.643 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.643 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.643 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.643 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.643 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.644 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.644 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.644 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.644 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.644 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.644 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.644 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.644 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.644 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.644 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.644 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.644 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.644 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.644 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.644 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.645 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.645 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.645 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.645 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.645 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.645 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.645 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.645 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.645 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.645 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.645 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.645 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.645 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rook", 2024-04-30T08:44:52.646 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.646 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.646 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.646 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.646 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.646 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.646 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.646 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.646 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.646 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.646 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.646 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.646 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.647 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.647 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.647 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.647 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.647 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.647 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.647 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.647 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.647 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.647 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.647 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.647 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.647 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.647 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.648 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.648 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.648 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.648 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.648 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.648 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.648 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.648 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.648 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.648 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.648 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.648 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.648 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.649 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.649 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.649 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.649 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.649 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.649 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.649 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.649 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.649 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.649 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.649 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.649 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.649 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.649 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.650 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.650 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.650 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.650 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.650 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.650 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.650 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.650 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.650 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.650 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.650 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.650 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.650 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.650 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.651 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.651 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.651 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.651 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.651 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.651 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.651 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.651 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.651 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.651 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.651 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.651 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.651 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.651 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.652 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.652 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.652 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.652 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.652 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.652 INFO:teuthology.orchestra.run.smithi032.stdout: "storage_class": { 2024-04-30T08:44:52.652 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "storage_class", 2024-04-30T08:44:52.652 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.652 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.652 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.652 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "local", 2024-04-30T08:44:52.652 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.652 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.652 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.653 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-30T08:44:52.653 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.653 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.653 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.653 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.653 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.653 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.653 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.653 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "selftest", 2024-04-30T08:44:52.653 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.653 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.653 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.653 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.653 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.654 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.654 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.654 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.654 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.654 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.654 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.654 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.654 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.654 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.654 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.654 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.654 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.654 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.654 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.655 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.655 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.655 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.655 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.655 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.655 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.655 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.655 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.655 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.655 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.655 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.655 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.655 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.655 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.656 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.656 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.656 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.656 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.656 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.656 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.656 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.656 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.656 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.656 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.656 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.656 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.656 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.656 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.657 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.657 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.657 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.657 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.657 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.657 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.657 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.657 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.657 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.657 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.657 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.657 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.657 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.657 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.658 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.658 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.658 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.658 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.658 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.658 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.658 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.658 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.658 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.658 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.658 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.658 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.658 INFO:teuthology.orchestra.run.smithi032.stdout: "roption1": { 2024-04-30T08:44:52.659 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "roption1", 2024-04-30T08:44:52.659 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.659 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.659 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.659 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.659 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.659 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.659 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.659 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.659 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.659 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.659 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.659 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.660 INFO:teuthology.orchestra.run.smithi032.stdout: "roption2": { 2024-04-30T08:44:52.660 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "roption2", 2024-04-30T08:44:52.660 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.660 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.660 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.660 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "xyz", 2024-04-30T08:44:52.660 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.660 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.660 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.660 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.660 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.660 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.660 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.660 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.661 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption1": { 2024-04-30T08:44:52.661 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption1", 2024-04-30T08:44:52.661 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.661 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.661 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.661 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.661 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.661 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.661 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.661 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.661 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.661 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.661 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.662 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.662 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption2": { 2024-04-30T08:44:52.662 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption2", 2024-04-30T08:44:52.662 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.662 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.662 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.662 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.662 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.662 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.662 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.662 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.662 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.662 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.662 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.663 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.663 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption3": { 2024-04-30T08:44:52.663 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption3", 2024-04-30T08:44:52.663 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-30T08:44:52.663 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.663 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.663 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.663 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.663 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.663 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.663 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.663 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.663 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.664 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.664 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.664 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption4": { 2024-04-30T08:44:52.664 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption4", 2024-04-30T08:44:52.664 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.664 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.664 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.664 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.664 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.664 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.664 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.664 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.664 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.664 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.665 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.665 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.665 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption5": { 2024-04-30T08:44:52.665 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption5", 2024-04-30T08:44:52.665 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.665 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.665 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.665 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.665 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.665 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.665 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.665 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.665 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.665 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.666 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.666 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.666 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption6": { 2024-04-30T08:44:52.666 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption6", 2024-04-30T08:44:52.666 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.666 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.666 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.666 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.666 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.666 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.666 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.666 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.666 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.667 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.667 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.667 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.667 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption7": { 2024-04-30T08:44:52.667 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption7", 2024-04-30T08:44:52.667 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.667 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.667 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.667 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.667 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "1", 2024-04-30T08:44:52.667 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "42", 2024-04-30T08:44:52.667 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.667 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.667 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.668 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.668 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.668 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.668 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.668 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.668 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.668 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.668 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.668 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.668 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.668 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.668 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.668 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.668 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.669 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.669 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.669 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.669 INFO:teuthology.orchestra.run.smithi032.stdout: "testkey": { 2024-04-30T08:44:52.669 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "testkey", 2024-04-30T08:44:52.669 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.669 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.669 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.669 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.669 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.669 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.669 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.669 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.669 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.670 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.670 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.670 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.670 INFO:teuthology.orchestra.run.smithi032.stdout: "testlkey": { 2024-04-30T08:44:52.670 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "testlkey", 2024-04-30T08:44:52.670 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.670 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.670 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.670 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.670 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.670 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.670 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.670 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.670 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.671 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.671 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.671 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.671 INFO:teuthology.orchestra.run.smithi032.stdout: "testnewline": { 2024-04-30T08:44:52.671 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "testnewline", 2024-04-30T08:44:52.671 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.671 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.671 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.671 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.671 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.671 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.671 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.671 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.672 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.672 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.672 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.672 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.672 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.672 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.672 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.672 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "snap_schedule", 2024-04-30T08:44:52.672 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.672 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.672 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.672 INFO:teuthology.orchestra.run.smithi032.stdout: "allow_m_granularity": { 2024-04-30T08:44:52.672 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "allow_m_granularity", 2024-04-30T08:44:52.672 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.673 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.673 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.673 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.673 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.673 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.673 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.673 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "allow minute scheduled snapshots", 2024-04-30T08:44:52.673 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.673 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.673 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.673 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.673 INFO:teuthology.orchestra.run.smithi032.stdout: "dump_on_update": { 2024-04-30T08:44:52.673 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "dump_on_update", 2024-04-30T08:44:52.673 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.674 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.674 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.674 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.674 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.674 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.674 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.674 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "dump database to debug log on update", 2024-04-30T08:44:52.674 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.674 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.674 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.674 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.674 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.674 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.675 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.675 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.675 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.675 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.675 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.675 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.675 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.675 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.675 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.675 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.675 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.675 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.675 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.676 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.676 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.676 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.676 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.676 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.676 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.676 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.676 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.676 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.676 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.676 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.676 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.676 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.676 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.677 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.677 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.677 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.677 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.677 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.677 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.677 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.677 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.677 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.677 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.677 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.677 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.677 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.677 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.678 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.678 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.678 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.678 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.678 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.678 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.678 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.678 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.678 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.678 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.678 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.678 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.678 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.679 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.679 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.679 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.679 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.679 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.679 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.679 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.679 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.679 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.679 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.679 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.679 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.679 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.679 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.679 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.680 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.680 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.680 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.680 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.680 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.680 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.680 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.680 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.680 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.680 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.680 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.680 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.680 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.680 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.681 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.681 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.681 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "stats", 2024-04-30T08:44:52.681 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.681 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.681 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.681 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.681 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.681 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.681 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.681 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.681 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.681 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.681 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.682 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.682 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.682 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.682 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.682 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.682 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.682 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.682 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.682 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.682 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.682 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.682 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.682 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.683 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.683 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.683 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.683 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.683 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.683 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.683 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.683 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.683 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.683 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.683 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.683 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.683 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.683 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.684 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.684 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.684 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.684 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.684 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.684 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.684 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.684 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.684 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.684 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.684 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.684 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.684 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.684 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.685 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.685 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.685 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.685 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.685 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.685 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.685 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.685 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.685 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.685 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.685 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.685 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.685 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.685 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.686 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.686 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.686 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.686 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.686 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.686 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.686 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.686 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.686 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.686 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.686 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.686 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.686 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.686 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.687 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.687 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.687 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.687 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.687 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.687 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.687 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.687 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.687 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.687 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.687 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "status", 2024-04-30T08:44:52.687 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.687 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.687 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.688 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.688 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.688 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.688 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.688 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.688 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.688 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.688 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.688 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.688 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.688 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.688 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.688 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.689 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.689 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.689 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.689 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.689 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.689 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.689 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.689 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.689 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.689 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.689 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.689 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.689 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.690 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.690 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.690 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.690 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.690 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.690 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.690 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.690 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.690 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.690 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.690 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.690 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.690 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.691 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.691 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.691 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.691 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.691 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.691 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.691 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.691 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.691 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.691 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.691 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.691 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.691 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.691 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.692 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.692 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.692 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.692 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.692 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.692 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.692 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.692 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.692 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.692 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.692 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.692 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.692 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.692 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.692 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.692 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.693 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.693 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.693 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.693 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.693 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.693 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.693 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.693 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.693 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.693 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.693 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.693 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.693 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.693 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.694 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.694 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.694 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.694 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.694 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "telegraf", 2024-04-30T08:44:52.694 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.694 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.694 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.694 INFO:teuthology.orchestra.run.smithi032.stdout: "address": { 2024-04-30T08:44:52.694 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "address", 2024-04-30T08:44:52.694 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.694 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.694 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.694 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-30T08:44:52.695 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.695 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.695 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.695 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.695 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.695 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.695 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.695 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.695 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2024-04-30T08:44:52.695 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2024-04-30T08:44:52.695 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.695 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.695 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.695 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "15", 2024-04-30T08:44:52.695 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.696 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.696 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.696 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.696 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.696 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.696 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.696 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.696 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.696 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.696 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.696 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.696 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.696 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.696 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.696 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.697 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.697 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.697 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.697 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.697 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.697 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.697 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.697 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.697 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.697 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.697 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.697 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.697 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.697 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.697 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.698 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.698 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.698 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.698 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.698 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.698 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.698 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.698 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.698 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.698 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.698 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.698 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.698 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.698 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.699 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.699 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.699 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.699 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.699 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.699 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.699 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.699 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.699 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.699 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.699 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.699 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.699 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.699 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.699 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.700 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.700 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.700 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.700 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.700 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.700 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.700 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.700 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.700 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.700 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.700 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.700 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.700 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.701 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.701 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.701 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.701 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.701 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.701 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.701 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.701 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.701 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.701 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.701 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.701 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.701 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.701 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.702 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.702 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.702 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.702 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.702 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.702 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.702 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.702 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.702 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "telemetry", 2024-04-30T08:44:52.702 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.702 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.702 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.702 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_basic": { 2024-04-30T08:44:52.703 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_basic", 2024-04-30T08:44:52.703 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.703 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.703 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.703 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.703 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.703 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.703 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.703 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-30T08:44:52.703 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.703 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.703 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.703 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.703 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_crash": { 2024-04-30T08:44:52.703 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_crash", 2024-04-30T08:44:52.704 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.704 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.704 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.704 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.704 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.704 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.704 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.704 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-30T08:44:52.704 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.704 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.704 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.704 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.704 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_device": { 2024-04-30T08:44:52.705 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_device", 2024-04-30T08:44:52.705 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.705 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.705 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.705 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.705 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.705 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.705 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.705 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-30T08:44:52.705 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.705 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.705 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.705 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.705 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_ident": { 2024-04-30T08:44:52.706 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_ident", 2024-04-30T08:44:52.706 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.706 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.706 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.706 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.706 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.706 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.706 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.706 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-30T08:44:52.706 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.706 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.706 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.706 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.707 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_perf": { 2024-04-30T08:44:52.707 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_perf", 2024-04-30T08:44:52.707 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.707 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.707 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.707 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.707 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.707 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.707 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.707 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-30T08:44:52.707 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.707 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.707 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.707 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.708 INFO:teuthology.orchestra.run.smithi032.stdout: "contact": { 2024-04-30T08:44:52.708 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "contact", 2024-04-30T08:44:52.708 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.708 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.708 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.708 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.708 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.708 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.708 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.708 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.708 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.708 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.708 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.708 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.708 INFO:teuthology.orchestra.run.smithi032.stdout: "description": { 2024-04-30T08:44:52.709 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "description", 2024-04-30T08:44:52.709 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.709 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.709 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.709 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.709 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.709 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.709 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.709 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.709 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.709 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.709 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.709 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.709 INFO:teuthology.orchestra.run.smithi032.stdout: "device_url": { 2024-04-30T08:44:52.710 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "device_url", 2024-04-30T08:44:52.710 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.710 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.710 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.710 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-30T08:44:52.710 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.710 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.710 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.710 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.710 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.710 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.710 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.710 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.711 INFO:teuthology.orchestra.run.smithi032.stdout: "enabled": { 2024-04-30T08:44:52.711 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "enabled", 2024-04-30T08:44:52.711 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.711 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.711 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.711 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.711 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.711 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.711 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.711 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.711 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.711 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.711 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.711 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.712 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2024-04-30T08:44:52.712 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2024-04-30T08:44:52.712 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.712 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.712 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.712 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "24", 2024-04-30T08:44:52.712 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "8", 2024-04-30T08:44:52.712 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.712 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.712 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.712 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.712 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.712 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.712 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.712 INFO:teuthology.orchestra.run.smithi032.stdout: "last_opt_revision": { 2024-04-30T08:44:52.713 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "last_opt_revision", 2024-04-30T08:44:52.713 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.713 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.713 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.713 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "1", 2024-04-30T08:44:52.713 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.713 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.713 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.713 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.713 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.713 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.713 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.713 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.713 INFO:teuthology.orchestra.run.smithi032.stdout: "leaderboard": { 2024-04-30T08:44:52.714 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "leaderboard", 2024-04-30T08:44:52.714 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.714 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.714 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.714 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.714 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.714 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.714 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.714 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.714 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.714 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.714 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.714 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.714 INFO:teuthology.orchestra.run.smithi032.stdout: "leaderboard_description": { 2024-04-30T08:44:52.715 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "leaderboard_description", 2024-04-30T08:44:52.715 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.715 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.715 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.715 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.715 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.715 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.715 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.715 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.715 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.715 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.715 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.715 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.715 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.716 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.716 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.716 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.716 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.716 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.716 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.716 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.716 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.716 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.716 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.716 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.716 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.716 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.717 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.717 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.717 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.717 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.717 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.717 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.717 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.717 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.717 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.717 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.717 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.717 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.717 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.718 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.718 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.718 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.718 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.718 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.718 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.718 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.718 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.718 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.718 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.718 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.718 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.718 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.718 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.719 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.719 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.719 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.719 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.719 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.719 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.719 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.719 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.719 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.719 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.719 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.719 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.719 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.719 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.719 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.720 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.720 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.720 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.720 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.720 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.720 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.720 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.720 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.720 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.720 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.720 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.720 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.720 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.720 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.721 INFO:teuthology.orchestra.run.smithi032.stdout: "organization": { 2024-04-30T08:44:52.721 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "organization", 2024-04-30T08:44:52.721 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.721 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.721 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.721 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.721 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.721 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.721 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.721 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.721 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.721 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.721 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.721 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.721 INFO:teuthology.orchestra.run.smithi032.stdout: "proxy": { 2024-04-30T08:44:52.721 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "proxy", 2024-04-30T08:44:52.722 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.722 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.722 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.722 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.722 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.722 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.722 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.722 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.722 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.722 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.722 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.722 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.722 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.722 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.723 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.723 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.723 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.723 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.723 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.723 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.723 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.723 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.723 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.723 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.723 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.723 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.723 INFO:teuthology.orchestra.run.smithi032.stdout: "url": { 2024-04-30T08:44:52.723 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "url", 2024-04-30T08:44:52.723 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.724 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.724 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.724 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-30T08:44:52.724 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.724 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.724 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.724 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.724 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.724 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.724 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.724 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.724 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.724 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.724 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.725 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "test_orchestrator", 2024-04-30T08:44:52.725 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.725 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.725 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.725 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.725 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.725 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.725 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.725 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.725 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.725 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.725 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.726 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.726 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.726 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.726 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.726 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.726 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.726 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.726 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.726 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.726 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.726 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.726 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.726 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.726 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.727 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.727 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.727 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.727 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.727 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.727 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.727 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.727 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.727 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.727 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.727 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.727 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.727 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.727 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.727 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.728 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.728 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.728 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.728 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.728 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.728 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.728 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.728 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.728 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.728 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.728 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.728 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.728 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.729 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.729 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.729 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.729 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.729 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.729 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.729 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.729 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.729 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.729 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.729 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.729 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.729 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.729 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.730 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.730 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.730 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.730 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.730 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.730 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.730 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.730 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.730 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.730 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.730 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.730 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.730 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.731 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.731 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.731 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.731 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.731 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.731 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.731 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.731 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.731 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.731 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.731 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "volumes", 2024-04-30T08:44:52.731 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.731 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.731 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.732 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.732 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.732 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.732 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.732 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.732 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.732 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.732 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.732 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.732 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.732 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.732 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.732 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.732 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.733 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.733 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.733 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.733 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.733 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.733 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.733 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.733 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.733 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.733 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.733 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.733 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.733 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.733 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.734 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.734 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.734 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.734 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.734 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.734 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.734 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.734 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.734 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.734 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.734 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.734 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.734 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.734 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.735 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.735 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.735 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.735 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.735 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.735 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.735 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.735 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.735 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.735 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.735 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.735 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.735 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.735 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.735 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.736 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.736 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.736 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.736 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.736 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.736 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.736 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.736 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.736 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.736 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.736 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.736 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.736 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.736 INFO:teuthology.orchestra.run.smithi032.stdout: "max_concurrent_clones": { 2024-04-30T08:44:52.736 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "max_concurrent_clones", 2024-04-30T08:44:52.737 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.737 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.737 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.737 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "4", 2024-04-30T08:44:52.737 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.737 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.737 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.737 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-30T08:44:52.737 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.737 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.737 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.737 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.737 INFO:teuthology.orchestra.run.smithi032.stdout: "periodic_async_work": { 2024-04-30T08:44:52.737 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "periodic_async_work", 2024-04-30T08:44:52.737 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.738 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.738 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.738 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.738 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.738 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.738 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.738 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Periodically check for async work", 2024-04-30T08:44:52.738 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.738 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.738 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.738 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.738 INFO:teuthology.orchestra.run.smithi032.stdout: "snapshot_clone_delay": { 2024-04-30T08:44:52.738 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "snapshot_clone_delay", 2024-04-30T08:44:52.738 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.739 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.739 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.739 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.739 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.739 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.739 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.739 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-30T08:44:52.739 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.739 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.739 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.739 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.739 INFO:teuthology.orchestra.run.smithi032.stdout: "snapshot_clone_no_wait": { 2024-04-30T08:44:52.739 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "snapshot_clone_no_wait", 2024-04-30T08:44:52.739 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.740 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.740 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.740 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.740 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.740 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.740 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.740 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-30T08:44:52.740 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.740 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.740 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.740 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.740 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.740 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.740 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.741 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.741 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.741 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.741 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.741 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.741 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.741 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.741 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.741 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.741 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.741 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.741 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.741 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.741 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.742 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "zabbix", 2024-04-30T08:44:52.742 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.742 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.742 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.742 INFO:teuthology.orchestra.run.smithi032.stdout: "discovery_interval": { 2024-04-30T08:44:52.742 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "discovery_interval", 2024-04-30T08:44:52.742 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2024-04-30T08:44:52.742 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.742 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.742 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "100", 2024-04-30T08:44:52.742 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.742 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.742 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.743 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.743 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.743 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.743 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.743 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.743 INFO:teuthology.orchestra.run.smithi032.stdout: "identifier": { 2024-04-30T08:44:52.743 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "identifier", 2024-04-30T08:44:52.743 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.743 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.743 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.743 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.743 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.743 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.743 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.744 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.744 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.744 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.744 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.744 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.744 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2024-04-30T08:44:52.744 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2024-04-30T08:44:52.744 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.744 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.744 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.744 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2024-04-30T08:44:52.744 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.744 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.745 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.745 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.745 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.745 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.745 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.745 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.745 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.745 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.745 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.745 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.745 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.745 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.745 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.745 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.746 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.746 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.746 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.746 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.746 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.746 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.746 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.746 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.746 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.746 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.746 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.746 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.746 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.746 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.747 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.747 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.747 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.747 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.747 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.747 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.747 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.747 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.747 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.747 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.747 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.747 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.747 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.747 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.748 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.748 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.748 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.748 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.748 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.748 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.748 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.748 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.748 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.748 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.748 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.748 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.748 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.748 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.749 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.749 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.749 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.749 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.749 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.749 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.749 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.749 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.749 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.749 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.749 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.749 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.749 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.749 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.750 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.750 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.750 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.750 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.750 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.750 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.750 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.750 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.750 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.750 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.750 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.750 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.750 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.751 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.751 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.751 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.751 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.751 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.751 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.751 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.751 INFO:teuthology.orchestra.run.smithi032.stdout: "zabbix_host": { 2024-04-30T08:44:52.751 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "zabbix_host", 2024-04-30T08:44:52.751 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.751 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.751 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.751 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.751 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.752 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.752 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.752 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.752 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.752 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.752 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.752 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.752 INFO:teuthology.orchestra.run.smithi032.stdout: "zabbix_port": { 2024-04-30T08:44:52.752 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "zabbix_port", 2024-04-30T08:44:52.752 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.752 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.752 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.752 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10051", 2024-04-30T08:44:52.752 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.753 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.753 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.753 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.753 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.753 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.753 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.753 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.753 INFO:teuthology.orchestra.run.smithi032.stdout: "zabbix_sender": { 2024-04-30T08:44:52.753 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "zabbix_sender", 2024-04-30T08:44:52.753 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.753 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.753 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.753 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-30T08:44:52.753 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.754 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.754 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.754 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.754 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.754 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.754 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.754 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.754 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.754 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.754 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-30T08:44:52.754 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.754 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.754 INFO:teuthology.orchestra.run.smithi032.stdout: "modules": [ 2024-04-30T08:44:52.754 INFO:teuthology.orchestra.run.smithi032.stdout: "cephadm", 2024-04-30T08:44:52.755 INFO:teuthology.orchestra.run.smithi032.stdout: "dashboard", 2024-04-30T08:44:52.755 INFO:teuthology.orchestra.run.smithi032.stdout: "iostat", 2024-04-30T08:44:52.755 INFO:teuthology.orchestra.run.smithi032.stdout: "nfs", 2024-04-30T08:44:52.755 INFO:teuthology.orchestra.run.smithi032.stdout: "restful" 2024-04-30T08:44:52.755 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.755 INFO:teuthology.orchestra.run.smithi032.stdout: "available_modules": [ 2024-04-30T08:44:52.755 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.755 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "alerts", 2024-04-30T08:44:52.755 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.755 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.755 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.755 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2024-04-30T08:44:52.755 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2024-04-30T08:44:52.755 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.756 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.756 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.756 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2024-04-30T08:44:52.756 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.756 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.756 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.756 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "How frequently to reexamine health status", 2024-04-30T08:44:52.756 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.756 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.756 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.756 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.756 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.756 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.756 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.757 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.757 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.757 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.757 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.757 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.757 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.757 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.757 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.757 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.757 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.757 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.757 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.757 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.758 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.758 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.758 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.758 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.758 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.758 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.758 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.758 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.758 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.758 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.758 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.758 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.758 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.759 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.759 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.759 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.759 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.759 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.759 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.759 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.759 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.759 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.759 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.759 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.759 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.759 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.760 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.760 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.760 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.760 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.760 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.760 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.760 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.760 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.760 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.760 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.760 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.760 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.760 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.760 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.760 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.761 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.761 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.761 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.761 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.761 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.761 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.761 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.761 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.761 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.761 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.761 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.761 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.761 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.761 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_destination": { 2024-04-30T08:44:52.761 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_destination", 2024-04-30T08:44:52.762 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.762 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.762 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.762 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.762 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.762 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.762 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.762 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Email address to send alerts to", 2024-04-30T08:44:52.762 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.762 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.762 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.762 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.762 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_from_name": { 2024-04-30T08:44:52.763 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_from_name", 2024-04-30T08:44:52.763 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.763 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.763 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.763 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "Ceph", 2024-04-30T08:44:52.763 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.763 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.763 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.763 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Email From: name", 2024-04-30T08:44:52.763 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.763 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.763 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.763 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.763 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_host": { 2024-04-30T08:44:52.764 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_host", 2024-04-30T08:44:52.764 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.764 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.764 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.764 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.764 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.764 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.764 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.764 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "SMTP server", 2024-04-30T08:44:52.764 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.764 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.764 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.764 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.764 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_password": { 2024-04-30T08:44:52.764 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_password", 2024-04-30T08:44:52.765 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.765 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.765 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.765 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.765 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.765 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.765 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.765 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Password to authenticate with", 2024-04-30T08:44:52.765 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.765 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.765 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.765 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.765 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_port": { 2024-04-30T08:44:52.765 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_port", 2024-04-30T08:44:52.766 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.766 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.766 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.766 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "465", 2024-04-30T08:44:52.766 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.766 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.766 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.766 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "SMTP port", 2024-04-30T08:44:52.766 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.766 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.766 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.766 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.766 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_sender": { 2024-04-30T08:44:52.766 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_sender", 2024-04-30T08:44:52.767 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.767 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.767 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.767 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.767 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.767 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.767 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.767 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "SMTP envelope sender", 2024-04-30T08:44:52.767 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.767 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.767 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.767 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.767 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_ssl": { 2024-04-30T08:44:52.767 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_ssl", 2024-04-30T08:44:52.768 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.768 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.768 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.768 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.768 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.768 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.768 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.768 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-30T08:44:52.768 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.768 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.768 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.768 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.768 INFO:teuthology.orchestra.run.smithi032.stdout: "smtp_user": { 2024-04-30T08:44:52.768 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "smtp_user", 2024-04-30T08:44:52.769 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.769 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.769 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.769 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.769 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.769 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.769 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.769 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "User to authenticate as", 2024-04-30T08:44:52.769 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.769 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.769 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.769 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.769 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.769 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.770 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.770 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.770 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.770 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.770 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.770 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.770 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.770 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.770 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.770 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.770 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.770 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.770 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.770 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.771 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.771 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "balancer", 2024-04-30T08:44:52.771 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.771 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.771 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.771 INFO:teuthology.orchestra.run.smithi032.stdout: "active": { 2024-04-30T08:44:52.771 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "active", 2024-04-30T08:44:52.771 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.771 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.771 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.771 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.771 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.771 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.771 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.772 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "automatically balance PGs across cluster", 2024-04-30T08:44:52.772 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.772 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.772 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.772 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.772 INFO:teuthology.orchestra.run.smithi032.stdout: "begin_time": { 2024-04-30T08:44:52.772 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "begin_time", 2024-04-30T08:44:52.772 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.772 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.772 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.772 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0000", 2024-04-30T08:44:52.772 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.772 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.772 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.772 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "beginning time of day to automatically balance", 2024-04-30T08:44:52.773 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-30T08:44:52.773 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.773 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.773 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.773 INFO:teuthology.orchestra.run.smithi032.stdout: "begin_weekday": { 2024-04-30T08:44:52.773 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "begin_weekday", 2024-04-30T08:44:52.773 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2024-04-30T08:44:52.773 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.773 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.773 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.773 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "0", 2024-04-30T08:44:52.773 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "6", 2024-04-30T08:44:52.773 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.773 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-30T08:44:52.774 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-30T08:44:52.774 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.774 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.774 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.774 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_compat_max_iterations": { 2024-04-30T08:44:52.774 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crush_compat_max_iterations", 2024-04-30T08:44:52.774 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2024-04-30T08:44:52.774 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.774 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.774 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "25", 2024-04-30T08:44:52.774 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "1", 2024-04-30T08:44:52.774 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "250", 2024-04-30T08:44:52.774 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.775 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-30T08:44:52.775 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.775 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.775 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.775 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.775 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_compat_metrics": { 2024-04-30T08:44:52.775 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crush_compat_metrics", 2024-04-30T08:44:52.775 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.775 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.775 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.775 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "pgs,objects,bytes", 2024-04-30T08:44:52.775 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.775 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.775 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.775 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-30T08:44:52.776 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-30T08:44:52.776 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.776 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.776 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.776 INFO:teuthology.orchestra.run.smithi032.stdout: "crush_compat_step": { 2024-04-30T08:44:52.776 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crush_compat_step", 2024-04-30T08:44:52.776 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-30T08:44:52.776 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.776 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.776 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0.5", 2024-04-30T08:44:52.776 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "0.001", 2024-04-30T08:44:52.776 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "0.999", 2024-04-30T08:44:52.776 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.776 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "aggressiveness of optimization", 2024-04-30T08:44:52.777 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-30T08:44:52.777 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.777 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.777 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.777 INFO:teuthology.orchestra.run.smithi032.stdout: "end_time": { 2024-04-30T08:44:52.777 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "end_time", 2024-04-30T08:44:52.777 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.777 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.777 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.777 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "2359", 2024-04-30T08:44:52.777 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.777 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.777 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.777 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "ending time of day to automatically balance", 2024-04-30T08:44:52.777 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-30T08:44:52.778 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.778 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.778 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.778 INFO:teuthology.orchestra.run.smithi032.stdout: "end_weekday": { 2024-04-30T08:44:52.778 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "end_weekday", 2024-04-30T08:44:52.778 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2024-04-30T08:44:52.778 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.778 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.778 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.778 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "0", 2024-04-30T08:44:52.778 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "6", 2024-04-30T08:44:52.778 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.778 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-30T08:44:52.779 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-30T08:44:52.779 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.779 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.779 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.779 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.779 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.779 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.779 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.779 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.779 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.779 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.779 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.779 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.779 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.779 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.780 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.780 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.780 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.780 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.780 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.780 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.780 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.780 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.780 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.780 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.780 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.780 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.780 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.780 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.780 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.781 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.781 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.781 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.781 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.781 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.781 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.781 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.781 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.781 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.781 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.781 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.781 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.781 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.781 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.781 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.782 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.782 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.782 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.782 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.782 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.782 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.782 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.782 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.782 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.782 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.782 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.782 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.782 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.782 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.783 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.783 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.783 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.783 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.783 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.783 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.783 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.783 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.783 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.783 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.783 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.783 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.783 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.783 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.783 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.784 INFO:teuthology.orchestra.run.smithi032.stdout: "min_score": { 2024-04-30T08:44:52.784 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "min_score", 2024-04-30T08:44:52.784 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-30T08:44:52.784 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.784 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.784 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.784 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.784 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.784 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.784 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-30T08:44:52.784 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.784 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.784 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.784 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.784 INFO:teuthology.orchestra.run.smithi032.stdout: "mode": { 2024-04-30T08:44:52.785 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mode", 2024-04-30T08:44:52.785 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.785 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.785 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.785 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "upmap", 2024-04-30T08:44:52.785 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.785 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.785 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.785 INFO:teuthology.orchestra.run.smithi032.stdout: "crush-compat", 2024-04-30T08:44:52.785 INFO:teuthology.orchestra.run.smithi032.stdout: "none", 2024-04-30T08:44:52.785 INFO:teuthology.orchestra.run.smithi032.stdout: "read", 2024-04-30T08:44:52.785 INFO:teuthology.orchestra.run.smithi032.stdout: "upmap", 2024-04-30T08:44:52.785 INFO:teuthology.orchestra.run.smithi032.stdout: "upmap-read" 2024-04-30T08:44:52.785 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.785 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Balancer mode", 2024-04-30T08:44:52.786 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.786 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.786 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.786 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.786 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_ids": { 2024-04-30T08:44:52.786 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "pool_ids", 2024-04-30T08:44:52.786 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.786 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.786 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.786 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.786 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.786 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.786 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.786 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-30T08:44:52.787 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.787 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.787 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.787 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.787 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2024-04-30T08:44:52.787 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2024-04-30T08:44:52.787 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.787 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.787 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.787 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2024-04-30T08:44:52.787 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.787 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.787 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.787 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-30T08:44:52.787 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.788 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.788 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.788 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.788 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.788 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.788 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.788 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.788 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.788 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.788 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.788 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.788 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.788 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.788 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.789 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.789 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.789 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.789 INFO:teuthology.orchestra.run.smithi032.stdout: "upmap_max_deviation": { 2024-04-30T08:44:52.789 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "upmap_max_deviation", 2024-04-30T08:44:52.789 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.789 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.789 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.789 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5", 2024-04-30T08:44:52.789 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "1", 2024-04-30T08:44:52.789 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.789 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.789 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-30T08:44:52.789 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-30T08:44:52.790 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.790 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.790 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.790 INFO:teuthology.orchestra.run.smithi032.stdout: "upmap_max_optimizations": { 2024-04-30T08:44:52.790 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "upmap_max_optimizations", 2024-04-30T08:44:52.790 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2024-04-30T08:44:52.790 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.790 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.790 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-30T08:44:52.790 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.790 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.790 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.790 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-30T08:44:52.791 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.791 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.791 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.791 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.791 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.791 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.791 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.791 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cephadm", 2024-04-30T08:44:52.791 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.791 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.791 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.791 INFO:teuthology.orchestra.run.smithi032.stdout: "agent_down_multiplier": { 2024-04-30T08:44:52.791 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "agent_down_multiplier", 2024-04-30T08:44:52.791 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-30T08:44:52.792 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.792 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.792 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "3.0", 2024-04-30T08:44:52.792 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.792 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.792 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.792 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-30T08:44:52.792 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.792 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.792 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.792 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.792 INFO:teuthology.orchestra.run.smithi032.stdout: "agent_refresh_rate": { 2024-04-30T08:44:52.792 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "agent_refresh_rate", 2024-04-30T08:44:52.792 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.793 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.793 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.793 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "20", 2024-04-30T08:44:52.793 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.793 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.793 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.793 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-30T08:44:52.793 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.793 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.793 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.793 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.793 INFO:teuthology.orchestra.run.smithi032.stdout: "agent_starting_port": { 2024-04-30T08:44:52.793 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "agent_starting_port", 2024-04-30T08:44:52.793 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.793 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.794 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.794 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "4721", 2024-04-30T08:44:52.794 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.794 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.794 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.794 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-30T08:44:52.794 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.794 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.794 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.794 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.794 INFO:teuthology.orchestra.run.smithi032.stdout: "allow_ptrace": { 2024-04-30T08:44:52.794 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "allow_ptrace", 2024-04-30T08:44:52.794 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.795 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.795 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.795 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.795 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.795 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.795 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.795 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-30T08:44:52.795 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-30T08:44:52.795 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.795 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.795 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.795 INFO:teuthology.orchestra.run.smithi032.stdout: "autotune_interval": { 2024-04-30T08:44:52.795 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "autotune_interval", 2024-04-30T08:44:52.795 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.796 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.796 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.796 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2024-04-30T08:44:52.796 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.796 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.796 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.796 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-30T08:44:52.796 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.796 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.796 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.796 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.796 INFO:teuthology.orchestra.run.smithi032.stdout: "autotune_memory_target_ratio": { 2024-04-30T08:44:52.796 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "autotune_memory_target_ratio", 2024-04-30T08:44:52.796 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-30T08:44:52.796 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.797 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.797 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0.7", 2024-04-30T08:44:52.797 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.797 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.797 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.797 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-30T08:44:52.797 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.797 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.797 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.797 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.797 INFO:teuthology.orchestra.run.smithi032.stdout: "cephadm_log_destination": { 2024-04-30T08:44:52.797 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cephadm_log_destination", 2024-04-30T08:44:52.797 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.797 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.797 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.798 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.798 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.798 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.798 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.798 INFO:teuthology.orchestra.run.smithi032.stdout: "file", 2024-04-30T08:44:52.798 INFO:teuthology.orchestra.run.smithi032.stdout: "file,syslog", 2024-04-30T08:44:52.798 INFO:teuthology.orchestra.run.smithi032.stdout: "syslog" 2024-04-30T08:44:52.798 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.798 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-30T08:44:52.798 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.798 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.798 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.798 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.799 INFO:teuthology.orchestra.run.smithi032.stdout: "cgroups_split": { 2024-04-30T08:44:52.799 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cgroups_split", 2024-04-30T08:44:52.799 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.799 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.799 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.799 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.799 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.799 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.799 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.799 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-30T08:44:52.799 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.799 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.799 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.799 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.800 INFO:teuthology.orchestra.run.smithi032.stdout: "config_checks_enabled": { 2024-04-30T08:44:52.800 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "config_checks_enabled", 2024-04-30T08:44:52.800 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.800 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.800 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.800 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.800 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.800 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.800 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.800 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-30T08:44:52.800 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.800 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.800 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.800 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.801 INFO:teuthology.orchestra.run.smithi032.stdout: "config_dashboard": { 2024-04-30T08:44:52.801 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "config_dashboard", 2024-04-30T08:44:52.801 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.801 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.801 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.801 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.801 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.801 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.801 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.801 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-30T08:44:52.801 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.801 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.801 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.801 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.802 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_alertmanager": { 2024-04-30T08:44:52.802 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_alertmanager", 2024-04-30T08:44:52.802 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.802 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.802 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.802 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-30T08:44:52.802 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.802 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.802 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.802 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Prometheus container image", 2024-04-30T08:44:52.802 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.802 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.802 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.802 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.803 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_base": { 2024-04-30T08:44:52.803 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_base", 2024-04-30T08:44:52.803 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.803 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.803 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.803 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-30T08:44:52.803 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.803 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.803 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.803 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Container image name, without the tag", 2024-04-30T08:44:52.803 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.803 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.803 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.803 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.804 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_elasticsearch": { 2024-04-30T08:44:52.804 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_elasticsearch", 2024-04-30T08:44:52.804 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.804 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.804 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.804 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-30T08:44:52.804 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.804 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.804 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.804 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "elasticsearch container image", 2024-04-30T08:44:52.804 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.804 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.804 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.804 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.805 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_grafana": { 2024-04-30T08:44:52.805 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_grafana", 2024-04-30T08:44:52.805 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.805 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.805 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.805 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-30T08:44:52.805 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.805 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.805 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.805 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Prometheus container image", 2024-04-30T08:44:52.805 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.805 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.805 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.805 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.806 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_haproxy": { 2024-04-30T08:44:52.806 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_haproxy", 2024-04-30T08:44:52.806 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.806 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.806 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.806 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-30T08:44:52.806 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.806 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.806 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.806 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "HAproxy container image", 2024-04-30T08:44:52.806 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.806 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.806 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.806 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.807 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_jaeger_agent": { 2024-04-30T08:44:52.807 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_jaeger_agent", 2024-04-30T08:44:52.807 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.807 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.807 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.807 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-30T08:44:52.807 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.807 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.807 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.807 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Jaeger agent container image", 2024-04-30T08:44:52.807 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.807 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.807 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.808 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.808 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_jaeger_collector": { 2024-04-30T08:44:52.808 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_jaeger_collector", 2024-04-30T08:44:52.808 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.808 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.808 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.808 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-30T08:44:52.808 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.808 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.808 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.808 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Jaeger collector container image", 2024-04-30T08:44:52.808 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.808 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.809 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.809 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.809 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_jaeger_query": { 2024-04-30T08:44:52.809 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_jaeger_query", 2024-04-30T08:44:52.809 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.809 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.809 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.809 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-30T08:44:52.809 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.809 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.809 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.809 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Jaeger query container image", 2024-04-30T08:44:52.809 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.809 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.810 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.810 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.810 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_keepalived": { 2024-04-30T08:44:52.810 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_keepalived", 2024-04-30T08:44:52.810 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.810 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.810 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.810 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-30T08:44:52.810 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.810 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.810 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.810 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Keepalived container image", 2024-04-30T08:44:52.810 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.810 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.811 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.811 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.811 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_loki": { 2024-04-30T08:44:52.811 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_loki", 2024-04-30T08:44:52.811 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.811 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.811 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.811 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-30T08:44:52.811 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.811 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.811 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.811 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Loki container image", 2024-04-30T08:44:52.811 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.812 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.812 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.812 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.812 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_node_exporter": { 2024-04-30T08:44:52.812 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_node_exporter", 2024-04-30T08:44:52.812 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.812 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.812 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.812 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-30T08:44:52.812 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.812 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.812 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.812 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Prometheus container image", 2024-04-30T08:44:52.812 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.812 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.813 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.813 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.813 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_nvmeof": { 2024-04-30T08:44:52.813 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_nvmeof", 2024-04-30T08:44:52.813 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.813 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.813 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.813 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-30T08:44:52.813 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.813 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.813 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.813 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Nvme-of container image", 2024-04-30T08:44:52.813 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.813 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.814 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.814 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.814 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_prometheus": { 2024-04-30T08:44:52.814 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_prometheus", 2024-04-30T08:44:52.814 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.814 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.814 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.814 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-30T08:44:52.814 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.814 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.814 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.814 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Prometheus container image", 2024-04-30T08:44:52.814 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.815 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.815 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.815 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.815 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_promtail": { 2024-04-30T08:44:52.815 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_promtail", 2024-04-30T08:44:52.815 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.815 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.815 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.815 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-30T08:44:52.815 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.815 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.815 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.815 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Promtail container image", 2024-04-30T08:44:52.815 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.816 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.816 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.816 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.816 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_samba": { 2024-04-30T08:44:52.816 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_samba", 2024-04-30T08:44:52.816 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.816 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.816 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.816 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-30T08:44:52.816 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.816 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.816 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.816 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Samba/SMB container image", 2024-04-30T08:44:52.816 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.816 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.817 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.817 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.817 INFO:teuthology.orchestra.run.smithi032.stdout: "container_image_snmp_gateway": { 2024-04-30T08:44:52.817 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_image_snmp_gateway", 2024-04-30T08:44:52.817 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.817 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.817 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.817 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-30T08:44:52.817 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.817 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.817 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.817 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "SNMP Gateway container image", 2024-04-30T08:44:52.817 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.818 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.818 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.818 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.818 INFO:teuthology.orchestra.run.smithi032.stdout: "container_init": { 2024-04-30T08:44:52.818 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "container_init", 2024-04-30T08:44:52.818 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.818 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.818 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.818 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.818 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.818 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.818 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.818 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Run podman/docker with `--init`", 2024-04-30T08:44:52.819 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.819 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.819 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.819 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.819 INFO:teuthology.orchestra.run.smithi032.stdout: "daemon_cache_timeout": { 2024-04-30T08:44:52.819 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "daemon_cache_timeout", 2024-04-30T08:44:52.819 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.819 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.819 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.819 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2024-04-30T08:44:52.819 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.819 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.819 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.819 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-30T08:44:52.819 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.820 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.820 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.820 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.820 INFO:teuthology.orchestra.run.smithi032.stdout: "default_cephadm_command_timeout": { 2024-04-30T08:44:52.820 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "default_cephadm_command_timeout", 2024-04-30T08:44:52.820 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.820 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.820 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.820 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "900", 2024-04-30T08:44:52.820 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.820 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.820 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.820 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-30T08:44:52.820 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.821 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.821 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.821 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.821 INFO:teuthology.orchestra.run.smithi032.stdout: "default_registry": { 2024-04-30T08:44:52.821 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "default_registry", 2024-04-30T08:44:52.821 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.821 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.821 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.821 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "docker.io", 2024-04-30T08:44:52.821 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.821 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.821 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.821 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-30T08:44:52.821 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.822 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.822 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.822 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.822 INFO:teuthology.orchestra.run.smithi032.stdout: "device_cache_timeout": { 2024-04-30T08:44:52.822 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "device_cache_timeout", 2024-04-30T08:44:52.822 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.822 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.822 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.822 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "1800", 2024-04-30T08:44:52.822 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.822 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.822 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.822 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "seconds to cache device inventory", 2024-04-30T08:44:52.822 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.823 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.823 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.823 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.823 INFO:teuthology.orchestra.run.smithi032.stdout: "device_enhanced_scan": { 2024-04-30T08:44:52.823 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "device_enhanced_scan", 2024-04-30T08:44:52.823 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.823 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.823 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.823 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.823 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.823 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.823 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.823 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-30T08:44:52.823 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.824 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.824 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.824 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.824 INFO:teuthology.orchestra.run.smithi032.stdout: "facts_cache_timeout": { 2024-04-30T08:44:52.824 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "facts_cache_timeout", 2024-04-30T08:44:52.824 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.824 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.824 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.824 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2024-04-30T08:44:52.824 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.824 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.824 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.824 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "seconds to cache host facts data", 2024-04-30T08:44:52.824 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.825 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.825 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.825 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.825 INFO:teuthology.orchestra.run.smithi032.stdout: "grafana_dashboards_path": { 2024-04-30T08:44:52.825 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "grafana_dashboards_path", 2024-04-30T08:44:52.825 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.825 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.825 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.825 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-30T08:44:52.825 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.825 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.825 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.825 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-30T08:44:52.826 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.826 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.826 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.826 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.826 INFO:teuthology.orchestra.run.smithi032.stdout: "host_check_interval": { 2024-04-30T08:44:52.826 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "host_check_interval", 2024-04-30T08:44:52.826 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.826 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.826 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.826 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2024-04-30T08:44:52.826 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.826 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.826 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.826 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to perform a host check", 2024-04-30T08:44:52.827 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.827 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.827 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.827 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.827 INFO:teuthology.orchestra.run.smithi032.stdout: "hw_monitoring": { 2024-04-30T08:44:52.827 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "hw_monitoring", 2024-04-30T08:44:52.827 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.827 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.827 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.827 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.827 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.827 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.827 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.828 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-30T08:44:52.828 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.828 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.828 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.828 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.828 INFO:teuthology.orchestra.run.smithi032.stdout: "inventory_list_all": { 2024-04-30T08:44:52.828 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "inventory_list_all", 2024-04-30T08:44:52.828 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.828 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.828 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.828 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.828 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.828 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.828 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.829 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-30T08:44:52.829 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.829 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.829 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.829 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.829 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.829 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.829 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.829 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.829 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.829 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.829 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.829 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.829 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.830 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.830 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.830 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.830 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.830 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.830 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.830 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.830 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.830 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.830 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.830 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.830 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.830 INFO:teuthology.orchestra.run.smithi032.stdout: "log_refresh_metadata": { 2024-04-30T08:44:52.830 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_refresh_metadata", 2024-04-30T08:44:52.831 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.831 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.831 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.831 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.831 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.831 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.831 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.831 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-30T08:44:52.831 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.831 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.831 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.831 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.831 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.832 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.832 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.832 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.832 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.832 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.832 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.832 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.832 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.832 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-30T08:44:52.832 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.832 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.832 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.832 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.833 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.833 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.833 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.833 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.833 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.833 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.833 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.833 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.833 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.833 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.833 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.833 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.833 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.833 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.834 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.834 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.834 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.834 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.834 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.834 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.834 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.834 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.834 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.834 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.834 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.834 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.834 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.835 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.835 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.835 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.835 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.835 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.835 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.835 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.835 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.835 INFO:teuthology.orchestra.run.smithi032.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-30T08:44:52.835 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-30T08:44:52.835 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.835 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.835 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.835 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.836 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.836 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.836 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.836 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-30T08:44:52.836 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.836 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.836 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.836 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.836 INFO:teuthology.orchestra.run.smithi032.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-30T08:44:52.836 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-30T08:44:52.836 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.836 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.836 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.837 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "*", 2024-04-30T08:44:52.837 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.837 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.837 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.837 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-30T08:44:52.837 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.837 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.837 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.837 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.837 INFO:teuthology.orchestra.run.smithi032.stdout: "max_count_per_host": { 2024-04-30T08:44:52.837 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "max_count_per_host", 2024-04-30T08:44:52.837 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.837 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.837 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.838 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-30T08:44:52.838 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.838 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.838 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.838 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "max number of daemons per service per host", 2024-04-30T08:44:52.838 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.838 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.838 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.838 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.838 INFO:teuthology.orchestra.run.smithi032.stdout: "max_osd_draining_count": { 2024-04-30T08:44:52.838 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "max_osd_draining_count", 2024-04-30T08:44:52.838 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.838 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.839 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.839 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-30T08:44:52.839 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.839 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.839 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.839 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-30T08:44:52.839 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.839 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.839 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.839 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.839 INFO:teuthology.orchestra.run.smithi032.stdout: "migration_current": { 2024-04-30T08:44:52.839 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "migration_current", 2024-04-30T08:44:52.839 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.840 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.840 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.840 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.840 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.840 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.840 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.840 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "internal - do not modify", 2024-04-30T08:44:52.840 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.840 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.840 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.840 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.840 INFO:teuthology.orchestra.run.smithi032.stdout: "mode": { 2024-04-30T08:44:52.840 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mode", 2024-04-30T08:44:52.840 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.840 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.841 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.841 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "root", 2024-04-30T08:44:52.841 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.841 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.841 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.841 INFO:teuthology.orchestra.run.smithi032.stdout: "cephadm-package", 2024-04-30T08:44:52.841 INFO:teuthology.orchestra.run.smithi032.stdout: "root" 2024-04-30T08:44:52.841 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.841 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "mode for remote execution of cephadm", 2024-04-30T08:44:52.841 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.841 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.841 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.841 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.842 INFO:teuthology.orchestra.run.smithi032.stdout: "oob_default_addr": { 2024-04-30T08:44:52.842 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "oob_default_addr", 2024-04-30T08:44:52.842 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.842 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.842 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.842 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "169.254.1.1", 2024-04-30T08:44:52.842 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.842 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.842 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.842 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-30T08:44:52.842 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.842 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.842 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.843 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.843 INFO:teuthology.orchestra.run.smithi032.stdout: "prometheus_alerts_path": { 2024-04-30T08:44:52.843 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "prometheus_alerts_path", 2024-04-30T08:44:52.843 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.843 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.843 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.843 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-30T08:44:52.843 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.843 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.843 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.843 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-30T08:44:52.843 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.843 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.843 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.844 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.844 INFO:teuthology.orchestra.run.smithi032.stdout: "registry_insecure": { 2024-04-30T08:44:52.844 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "registry_insecure", 2024-04-30T08:44:52.844 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.844 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.844 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.844 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.844 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.844 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.844 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.844 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-30T08:44:52.844 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.844 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.844 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.845 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.845 INFO:teuthology.orchestra.run.smithi032.stdout: "registry_password": { 2024-04-30T08:44:52.845 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "registry_password", 2024-04-30T08:44:52.845 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.845 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.845 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.845 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.845 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.845 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.845 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.845 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-30T08:44:52.845 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.845 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.845 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.846 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.846 INFO:teuthology.orchestra.run.smithi032.stdout: "registry_url": { 2024-04-30T08:44:52.846 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "registry_url", 2024-04-30T08:44:52.846 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.846 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.846 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.846 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.846 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.846 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.846 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.846 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-30T08:44:52.846 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.846 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.847 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.847 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.847 INFO:teuthology.orchestra.run.smithi032.stdout: "registry_username": { 2024-04-30T08:44:52.847 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "registry_username", 2024-04-30T08:44:52.847 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.847 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.847 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.847 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.847 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.847 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.847 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.847 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-30T08:44:52.847 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.847 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.848 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.848 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.848 INFO:teuthology.orchestra.run.smithi032.stdout: "secure_monitoring_stack": { 2024-04-30T08:44:52.848 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "secure_monitoring_stack", 2024-04-30T08:44:52.848 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.848 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.848 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.848 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.848 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.848 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.848 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.848 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-30T08:44:52.848 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.849 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.849 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.849 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.849 INFO:teuthology.orchestra.run.smithi032.stdout: "service_discovery_port": { 2024-04-30T08:44:52.849 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "service_discovery_port", 2024-04-30T08:44:52.849 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.849 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.849 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.849 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8765", 2024-04-30T08:44:52.849 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.849 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.849 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.849 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "cephadm service discovery port", 2024-04-30T08:44:52.849 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.850 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.850 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.850 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.850 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.850 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.850 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.850 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.850 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.850 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.850 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.850 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.850 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.850 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.850 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.851 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.851 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.851 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.851 INFO:teuthology.orchestra.run.smithi032.stdout: "ssh_config_file": { 2024-04-30T08:44:52.851 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ssh_config_file", 2024-04-30T08:44:52.851 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.851 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.851 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.851 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.851 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.851 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.851 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.851 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-30T08:44:52.851 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.852 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.852 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.852 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.852 INFO:teuthology.orchestra.run.smithi032.stdout: "use_agent": { 2024-04-30T08:44:52.852 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "use_agent", 2024-04-30T08:44:52.852 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.852 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.852 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.852 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.852 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.852 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.852 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.852 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-30T08:44:52.853 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.853 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.853 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.853 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.853 INFO:teuthology.orchestra.run.smithi032.stdout: "use_repo_digest": { 2024-04-30T08:44:52.853 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "use_repo_digest", 2024-04-30T08:44:52.853 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.853 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.853 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.853 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.853 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.853 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.853 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.853 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-30T08:44:52.854 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.854 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.854 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.854 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.854 INFO:teuthology.orchestra.run.smithi032.stdout: "warn_on_failed_host_check": { 2024-04-30T08:44:52.854 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "warn_on_failed_host_check", 2024-04-30T08:44:52.854 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.854 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.854 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.854 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.854 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.854 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.854 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.855 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "raise a health warning if the host check fails", 2024-04-30T08:44:52.855 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.855 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.855 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.855 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.855 INFO:teuthology.orchestra.run.smithi032.stdout: "warn_on_stray_daemons": { 2024-04-30T08:44:52.855 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "warn_on_stray_daemons", 2024-04-30T08:44:52.855 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.855 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.855 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.855 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.855 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.855 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.855 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.855 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-30T08:44:52.856 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.856 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.856 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.856 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.856 INFO:teuthology.orchestra.run.smithi032.stdout: "warn_on_stray_hosts": { 2024-04-30T08:44:52.856 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "warn_on_stray_hosts", 2024-04-30T08:44:52.856 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.856 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.856 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.856 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.856 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.856 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.856 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.857 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-30T08:44:52.857 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.857 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.857 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.857 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.857 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.857 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.857 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.857 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crash", 2024-04-30T08:44:52.857 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.857 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.857 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.857 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.857 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.858 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.858 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.858 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.858 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.858 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.858 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.858 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.858 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.858 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.858 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.858 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.858 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.858 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.859 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.859 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.859 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.859 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.859 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.859 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.859 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.859 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.859 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.859 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.859 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.859 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.859 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.859 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.860 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.860 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.860 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.860 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.860 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.860 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.860 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.860 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.860 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.860 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.860 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.860 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.860 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.860 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.860 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.861 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.861 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.861 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.861 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.861 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.861 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.861 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.861 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.861 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.861 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.861 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.861 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.861 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.861 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.862 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.862 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.862 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.862 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.862 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.862 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.862 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.862 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.862 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.862 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.862 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.862 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.862 INFO:teuthology.orchestra.run.smithi032.stdout: "retain_interval": { 2024-04-30T08:44:52.862 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "retain_interval", 2024-04-30T08:44:52.863 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.863 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.863 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.863 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "31536000", 2024-04-30T08:44:52.863 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.863 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.863 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.863 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-30T08:44:52.863 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.863 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.863 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.863 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.863 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.863 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.864 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.864 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.864 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.864 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.864 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.864 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.864 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.864 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.864 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.864 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.864 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.865 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.865 INFO:teuthology.orchestra.run.smithi032.stdout: "warn_recent_interval": { 2024-04-30T08:44:52.865 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "warn_recent_interval", 2024-04-30T08:44:52.865 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.865 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.865 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.865 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "1209600", 2024-04-30T08:44:52.865 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.865 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.865 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.865 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-30T08:44:52.865 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.865 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.865 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.866 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.866 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.866 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.866 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.866 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "dashboard", 2024-04-30T08:44:52.866 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.866 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.866 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.866 INFO:teuthology.orchestra.run.smithi032.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-30T08:44:52.866 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-30T08:44:52.866 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.866 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.866 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.867 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-30T08:44:52.867 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.867 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.867 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.867 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.867 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.867 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.867 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.867 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.867 INFO:teuthology.orchestra.run.smithi032.stdout: "ALERTMANAGER_API_HOST": { 2024-04-30T08:44:52.867 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-30T08:44:52.867 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.867 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.867 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.868 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.868 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.868 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.868 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.868 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.868 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.868 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.868 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.868 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.868 INFO:teuthology.orchestra.run.smithi032.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-30T08:44:52.868 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-30T08:44:52.868 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.868 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.868 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.869 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.869 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.869 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.869 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.869 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.869 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.869 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.869 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.869 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.869 INFO:teuthology.orchestra.run.smithi032.stdout: "AUDIT_API_ENABLED": { 2024-04-30T08:44:52.869 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "AUDIT_API_ENABLED", 2024-04-30T08:44:52.869 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.869 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.869 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.870 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.870 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.870 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.870 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.870 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.870 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.870 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.870 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.870 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.870 INFO:teuthology.orchestra.run.smithi032.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-30T08:44:52.870 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-30T08:44:52.870 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.870 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.871 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.871 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.871 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.871 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.871 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.871 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.871 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.871 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.871 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.871 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.871 INFO:teuthology.orchestra.run.smithi032.stdout: "ENABLE_BROWSABLE_API": { 2024-04-30T08:44:52.871 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-30T08:44:52.871 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.871 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.872 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.872 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.872 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.872 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.872 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.872 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.872 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.872 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.872 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.872 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.872 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-30T08:44:52.872 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-30T08:44:52.872 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.872 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.872 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.873 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.873 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.873 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.873 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.873 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.873 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.873 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.873 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.873 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.873 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-30T08:44:52.873 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-30T08:44:52.873 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.873 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.873 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.874 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.874 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.874 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.874 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.874 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.874 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.874 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.874 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.874 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.874 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-30T08:44:52.874 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-30T08:44:52.874 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.874 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.875 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.875 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.875 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.875 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.875 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.875 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.875 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.875 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.875 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.875 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.875 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-30T08:44:52.875 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-30T08:44:52.875 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.875 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.876 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.876 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.876 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.876 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.876 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.876 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.876 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.876 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.876 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.876 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.876 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-30T08:44:52.876 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-30T08:44:52.876 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.876 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.876 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.877 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.877 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.877 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.877 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.877 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.877 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.877 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.877 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.877 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.877 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-30T08:44:52.877 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-30T08:44:52.877 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.877 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.877 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.878 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.878 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.878 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.878 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.878 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.878 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.878 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.878 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.878 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.878 INFO:teuthology.orchestra.run.smithi032.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-30T08:44:52.878 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-30T08:44:52.878 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.878 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.878 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.879 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.879 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.879 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.879 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.879 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.879 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.879 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.879 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.879 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.879 INFO:teuthology.orchestra.run.smithi032.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-30T08:44:52.879 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-30T08:44:52.879 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.879 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.879 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.880 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.880 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.880 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.880 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.880 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.880 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.880 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.880 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.880 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.880 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_API_PASSWORD": { 2024-04-30T08:44:52.880 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-30T08:44:52.880 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.880 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.880 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.881 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "admin", 2024-04-30T08:44:52.881 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.881 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.881 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.881 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.881 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.881 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.881 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.881 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.881 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-30T08:44:52.881 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-30T08:44:52.881 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.881 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.881 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.882 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.882 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.882 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.882 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.882 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.882 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.882 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.882 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.882 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.882 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_API_URL": { 2024-04-30T08:44:52.882 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_API_URL", 2024-04-30T08:44:52.882 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.882 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.883 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.883 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.883 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.883 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.883 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.883 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.883 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.883 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.883 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.883 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.883 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_API_USERNAME": { 2024-04-30T08:44:52.883 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-30T08:44:52.883 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.883 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.884 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.884 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "admin", 2024-04-30T08:44:52.884 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.884 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.884 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.884 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.884 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.884 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.884 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.884 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.884 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-30T08:44:52.884 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-30T08:44:52.884 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.884 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.885 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.885 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.885 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.885 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.885 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.885 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.885 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.885 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.885 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.885 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.885 INFO:teuthology.orchestra.run.smithi032.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-30T08:44:52.885 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-30T08:44:52.885 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.885 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.886 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.886 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.886 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.886 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.886 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.886 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.886 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.886 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.886 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.886 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.886 INFO:teuthology.orchestra.run.smithi032.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-30T08:44:52.886 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-30T08:44:52.886 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.887 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.887 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.887 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.887 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.887 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.887 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.887 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.887 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.887 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.887 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.887 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.887 INFO:teuthology.orchestra.run.smithi032.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-30T08:44:52.887 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-30T08:44:52.887 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.888 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.888 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.888 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.888 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.888 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.888 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.888 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.888 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.888 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.888 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.888 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.888 INFO:teuthology.orchestra.run.smithi032.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-30T08:44:52.888 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-30T08:44:52.888 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.888 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.888 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.889 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "[]", 2024-04-30T08:44:52.889 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.889 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.889 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.889 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.889 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.889 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.889 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.889 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.889 INFO:teuthology.orchestra.run.smithi032.stdout: "MULTICLUSTER_CONFIG": { 2024-04-30T08:44:52.889 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-30T08:44:52.889 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.889 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.889 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.890 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "{}", 2024-04-30T08:44:52.890 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.890 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.890 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.890 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.890 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.890 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.890 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.890 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.890 INFO:teuthology.orchestra.run.smithi032.stdout: "PROMETHEUS_API_HOST": { 2024-04-30T08:44:52.890 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-30T08:44:52.890 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.890 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.890 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.891 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.891 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.891 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.891 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.891 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.891 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.891 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.891 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.891 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.891 INFO:teuthology.orchestra.run.smithi032.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-30T08:44:52.891 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-30T08:44:52.891 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.891 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.891 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.892 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.892 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.892 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.892 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.892 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.892 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.892 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.892 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.892 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.892 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-30T08:44:52.892 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-30T08:44:52.892 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.892 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.892 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.892 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.893 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.893 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.893 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.893 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.893 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.893 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.893 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.893 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.893 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-30T08:44:52.893 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-30T08:44:52.893 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.893 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.893 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.893 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.893 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.894 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.894 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.894 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.894 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.894 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.894 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.894 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.894 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-30T08:44:52.894 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-30T08:44:52.894 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.894 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.894 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.894 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.894 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.895 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.895 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.895 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.895 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.895 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.895 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.895 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.895 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-30T08:44:52.895 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-30T08:44:52.895 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.895 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.895 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.895 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.895 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.895 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.896 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.896 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.896 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.896 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.896 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.896 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.896 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-30T08:44:52.896 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-30T08:44:52.896 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.896 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.896 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.896 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.896 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.896 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.897 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.897 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.897 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.897 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.897 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.897 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.897 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-30T08:44:52.897 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-30T08:44:52.897 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.897 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.897 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.897 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.897 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.897 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.898 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.898 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.898 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.898 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.898 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.898 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.898 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-30T08:44:52.898 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-30T08:44:52.898 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.898 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.898 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.898 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.898 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.899 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.899 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.899 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.899 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.899 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.899 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.899 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.899 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_ENABLED": { 2024-04-30T08:44:52.899 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-30T08:44:52.899 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.899 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.899 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.899 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.899 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.900 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.900 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.900 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.900 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.900 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.900 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.900 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.900 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-30T08:44:52.900 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-30T08:44:52.900 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.900 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.900 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.900 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-30T08:44:52.900 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.901 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.901 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.901 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.901 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.901 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.901 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.901 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.901 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-30T08:44:52.901 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-30T08:44:52.901 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.901 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.901 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.901 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-30T08:44:52.902 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.902 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.902 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.902 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.902 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.902 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.902 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.902 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.902 INFO:teuthology.orchestra.run.smithi032.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-30T08:44:52.902 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-30T08:44:52.902 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.902 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.902 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.902 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8", 2024-04-30T08:44:52.903 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.903 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.903 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.903 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.903 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.903 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.903 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.903 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.903 INFO:teuthology.orchestra.run.smithi032.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-30T08:44:52.903 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-30T08:44:52.903 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.903 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.903 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.903 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "45", 2024-04-30T08:44:52.903 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.904 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.904 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.904 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.904 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.904 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.904 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.904 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.904 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_ACCESS_KEY": { 2024-04-30T08:44:52.904 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-30T08:44:52.904 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.904 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.904 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.904 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.904 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.905 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.905 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.905 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.905 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.905 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.905 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.905 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.905 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-30T08:44:52.905 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-30T08:44:52.905 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.905 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.905 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.905 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "admin", 2024-04-30T08:44:52.905 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.906 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.906 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.906 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.906 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.906 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.906 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.906 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.906 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_SECRET_KEY": { 2024-04-30T08:44:52.906 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-30T08:44:52.906 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.906 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.906 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.906 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.906 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.907 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.907 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.907 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.907 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.907 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.907 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.907 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.907 INFO:teuthology.orchestra.run.smithi032.stdout: "RGW_API_SSL_VERIFY": { 2024-04-30T08:44:52.907 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-30T08:44:52.907 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.907 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.907 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.907 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.907 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.908 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.908 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.908 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.908 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.908 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.908 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.908 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.908 INFO:teuthology.orchestra.run.smithi032.stdout: "UNSAFE_TLS_v1_2": { 2024-04-30T08:44:52.908 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-30T08:44:52.908 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.908 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.908 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.908 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.908 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.908 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.909 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.909 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.909 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.909 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.909 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.909 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.909 INFO:teuthology.orchestra.run.smithi032.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-30T08:44:52.909 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-30T08:44:52.909 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.909 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.909 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.909 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.909 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.909 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.909 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.910 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.910 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.910 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.910 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.910 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.910 INFO:teuthology.orchestra.run.smithi032.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-30T08:44:52.910 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-30T08:44:52.910 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.910 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.910 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.910 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-30T08:44:52.910 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.910 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.910 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.911 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.911 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.911 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.911 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.911 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.911 INFO:teuthology.orchestra.run.smithi032.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-30T08:44:52.911 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-30T08:44:52.911 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.911 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.911 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.911 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5", 2024-04-30T08:44:52.911 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.911 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.911 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.912 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.912 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.912 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.912 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.809+0000 7fa438600640 1 -- 172.21.15.32:0/2063780753 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa4180afbb0 msgr2=0x7fa4180b20c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:44:52.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.809+0000 7fa438600640 1 --2- 172.21.15.32:0/2063780753 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa4180afbb0 0x7fa4180b20c0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fa43010c9e0 tx=0x7fa41c005f90 comp rx=0 tx=0).stop 2024-04-30T08:44:52.912 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.809+0000 7fa438600640 1 -- 172.21.15.32:0/2063780753 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa43010a040 msgr2=0x7fa43010d040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:44:52.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.809+0000 7fa438600640 1 --2- 172.21.15.32:0/2063780753 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa43010a040 0x7fa43010d040 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fa42400b450 tx=0x7fa42400b910 comp rx=0 tx=0).stop 2024-04-30T08:44:52.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.809+0000 7fa438600640 1 -- 172.21.15.32:0/2063780753 shutdown_connections 2024-04-30T08:44:52.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.809+0000 7fa438600640 1 --2- 172.21.15.32:0/2063780753 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fa4180afbb0 0x7fa4180b20c0 secure :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fa43010c9e0 tx=0x7fa41c005f90 comp rx=0 tx=0).stop 2024-04-30T08:44:52.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.809+0000 7fa438600640 1 --2- 172.21.15.32:0/2063780753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa430112140 0x7fa43010bc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:52.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.809+0000 7fa438600640 1 --2- 172.21.15.32:0/2063780753 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fa43010f7e0 0x7fa43010b6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:52.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.809+0000 7fa438600640 1 --2- 172.21.15.32:0/2063780753 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fa43010a040 0x7fa43010d040 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:44:52.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.809+0000 7fa438600640 1 -- 172.21.15.32:0/2063780753 >> 172.21.15.32:0/2063780753 conn(0x7fa430070560 msgr2=0x7fa430110720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:44:52.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.814+0000 7fa438600640 1 -- 172.21.15.32:0/2063780753 shutdown_connections 2024-04-30T08:44:52.913 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:44:51.814+0000 7fa438600640 1 -- 172.21.15.32:0/2063780753 wait complete. 2024-04-30T08:44:52.915 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:52 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2063780753' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-30T08:44:52.915 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:52 smithi071 ceph-mon[33385]: pgmap v716: 129 pgs: 129 active+clean; 8.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 694 op/s 2024-04-30T08:44:52.915 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:52 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2063780753' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-30T08:44:52.915 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:52 smithi189 ceph-mon[33038]: pgmap v716: 129 pgs: 129 active+clean; 8.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 694 op/s 2024-04-30T08:44:52.916 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.916 INFO:teuthology.orchestra.run.smithi032.stdout: "cross_origin_url": { 2024-04-30T08:44:52.916 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cross_origin_url", 2024-04-30T08:44:52.916 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.916 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.916 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.916 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.916 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.916 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.916 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.917 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.917 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.917 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.917 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.917 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.917 INFO:teuthology.orchestra.run.smithi032.stdout: "crt_file": { 2024-04-30T08:44:52.917 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "crt_file", 2024-04-30T08:44:52.917 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.917 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.917 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.917 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.917 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.917 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.918 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.918 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.918 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.918 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.918 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.918 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.918 INFO:teuthology.orchestra.run.smithi032.stdout: "debug": { 2024-04-30T08:44:52.918 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "debug", 2024-04-30T08:44:52.918 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.918 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.918 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.918 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.918 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.919 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.919 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.919 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Enable/disable debug options", 2024-04-30T08:44:52.919 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.919 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.919 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.919 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.919 INFO:teuthology.orchestra.run.smithi032.stdout: "jwt_token_ttl": { 2024-04-30T08:44:52.919 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "jwt_token_ttl", 2024-04-30T08:44:52.919 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.919 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.919 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.919 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "28800", 2024-04-30T08:44:52.919 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.920 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.920 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.920 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.920 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.920 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.920 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.920 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.920 INFO:teuthology.orchestra.run.smithi032.stdout: "key_file": { 2024-04-30T08:44:52.920 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "key_file", 2024-04-30T08:44:52.920 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.920 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.920 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.920 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.920 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.921 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.921 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.921 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.921 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.921 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.921 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.921 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.921 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.921 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.921 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.921 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.921 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.921 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.922 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.922 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.922 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.922 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.922 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.922 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.922 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.922 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.922 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.922 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.922 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.922 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.922 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.923 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.923 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.923 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.923 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.923 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.923 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.923 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.923 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.923 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.923 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.923 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.923 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.923 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.923 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.924 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.924 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.924 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.924 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.924 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.924 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.924 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.924 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.924 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.924 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.924 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.924 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.924 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.924 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.925 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.925 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.925 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.925 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.925 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.925 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.925 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.925 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.925 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.925 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.925 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.925 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.925 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.926 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.926 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.926 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.926 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.926 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.926 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.926 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.926 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.926 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.926 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.926 INFO:teuthology.orchestra.run.smithi032.stdout: "motd": { 2024-04-30T08:44:52.926 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "motd", 2024-04-30T08:44:52.926 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.926 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.927 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.927 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.927 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.927 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.927 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.927 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "The message of the day", 2024-04-30T08:44:52.927 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.927 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.927 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.927 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.927 INFO:teuthology.orchestra.run.smithi032.stdout: "redirect_resolve_ip_addr": { 2024-04-30T08:44:52.927 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "redirect_resolve_ip_addr", 2024-04-30T08:44:52.927 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.927 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.928 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.928 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.928 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.928 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.928 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.928 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.928 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.928 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.928 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.928 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.928 INFO:teuthology.orchestra.run.smithi032.stdout: "server_addr": { 2024-04-30T08:44:52.928 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_addr", 2024-04-30T08:44:52.928 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.928 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.929 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.929 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "::", 2024-04-30T08:44:52.929 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.929 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.929 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.929 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.929 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.929 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.929 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.929 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.929 INFO:teuthology.orchestra.run.smithi032.stdout: "server_port": { 2024-04-30T08:44:52.929 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_port", 2024-04-30T08:44:52.929 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.929 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.930 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.930 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8080", 2024-04-30T08:44:52.930 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.930 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.930 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.930 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.930 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.930 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.930 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.930 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.930 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.930 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.930 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.930 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.931 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.931 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.931 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.931 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.931 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.931 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.931 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.931 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.931 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.931 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.931 INFO:teuthology.orchestra.run.smithi032.stdout: "ssl": { 2024-04-30T08:44:52.931 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ssl", 2024-04-30T08:44:52.931 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.931 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.932 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.932 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.932 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.932 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.932 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.932 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.932 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.932 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.932 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.932 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.932 INFO:teuthology.orchestra.run.smithi032.stdout: "ssl_server_port": { 2024-04-30T08:44:52.932 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ssl_server_port", 2024-04-30T08:44:52.932 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.932 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.933 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.933 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8443", 2024-04-30T08:44:52.933 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.933 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.933 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.933 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.933 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.933 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.933 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.933 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.933 INFO:teuthology.orchestra.run.smithi032.stdout: "standby_behaviour": { 2024-04-30T08:44:52.933 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "standby_behaviour", 2024-04-30T08:44:52.933 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.934 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.934 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.934 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "redirect", 2024-04-30T08:44:52.934 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.934 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.934 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.934 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.934 INFO:teuthology.orchestra.run.smithi032.stdout: "redirect" 2024-04-30T08:44:52.934 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.934 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.934 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.934 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.934 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.935 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.935 INFO:teuthology.orchestra.run.smithi032.stdout: "standby_error_status_code": { 2024-04-30T08:44:52.935 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "standby_error_status_code", 2024-04-30T08:44:52.935 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.935 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.935 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.935 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "500", 2024-04-30T08:44:52.935 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "400", 2024-04-30T08:44:52.935 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "599", 2024-04-30T08:44:52.935 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.935 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.935 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.935 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.935 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.936 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.936 INFO:teuthology.orchestra.run.smithi032.stdout: "url_prefix": { 2024-04-30T08:44:52.936 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "url_prefix", 2024-04-30T08:44:52.936 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.936 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.936 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.936 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.936 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.936 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.936 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.936 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.936 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.936 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.936 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.937 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.937 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.937 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.937 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.937 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "devicehealth", 2024-04-30T08:44:52.937 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.937 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.937 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.937 INFO:teuthology.orchestra.run.smithi032.stdout: "enable_monitoring": { 2024-04-30T08:44:52.937 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "enable_monitoring", 2024-04-30T08:44:52.937 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.937 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.937 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.937 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.938 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.938 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.938 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.938 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "monitor device health metrics", 2024-04-30T08:44:52.938 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.938 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.938 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.938 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.938 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.938 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.938 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.938 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.938 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.939 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.939 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.939 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.939 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.939 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.939 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.939 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.939 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.939 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.939 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.939 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.939 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.939 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.939 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.940 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.940 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.940 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.940 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.940 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.940 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.940 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.940 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.940 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.940 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.940 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.940 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.940 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.940 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.941 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.941 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.941 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.941 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.941 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.941 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.941 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.941 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.941 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.941 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.941 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.941 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.941 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.942 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.942 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.942 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.942 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.942 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.942 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.942 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.942 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.942 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.942 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.942 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.942 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.942 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.943 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.943 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.943 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.943 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.943 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.943 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.943 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.943 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.943 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.943 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.943 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.943 INFO:teuthology.orchestra.run.smithi032.stdout: "mark_out_threshold": { 2024-04-30T08:44:52.943 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mark_out_threshold", 2024-04-30T08:44:52.943 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.944 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.944 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.944 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "2419200", 2024-04-30T08:44:52.944 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.944 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.944 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.944 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-30T08:44:52.944 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.944 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.944 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.944 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.944 INFO:teuthology.orchestra.run.smithi032.stdout: "pool_name": { 2024-04-30T08:44:52.944 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "pool_name", 2024-04-30T08:44:52.944 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.945 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.945 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.945 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "device_health_metrics", 2024-04-30T08:44:52.945 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.945 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.945 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.945 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-30T08:44:52.945 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.945 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.945 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.945 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.945 INFO:teuthology.orchestra.run.smithi032.stdout: "retention_period": { 2024-04-30T08:44:52.945 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "retention_period", 2024-04-30T08:44:52.946 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.946 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.946 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.946 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "15552000", 2024-04-30T08:44:52.946 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.946 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.946 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.946 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how long to retain device health metrics", 2024-04-30T08:44:52.946 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.946 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.946 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.946 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.946 INFO:teuthology.orchestra.run.smithi032.stdout: "scrape_frequency": { 2024-04-30T08:44:52.947 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "scrape_frequency", 2024-04-30T08:44:52.947 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.947 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.947 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.949 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "86400", 2024-04-30T08:44:52.949 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.949 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.949 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.949 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-30T08:44:52.949 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.949 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.949 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.950 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.950 INFO:teuthology.orchestra.run.smithi032.stdout: "self_heal": { 2024-04-30T08:44:52.950 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "self_heal", 2024-04-30T08:44:52.950 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.950 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.950 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.950 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:52.950 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.950 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.950 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.950 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-30T08:44:52.950 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.950 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.951 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.951 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.951 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2024-04-30T08:44:52.951 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2024-04-30T08:44:52.951 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.951 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.951 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.951 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2024-04-30T08:44:52.951 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.951 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.951 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.951 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how frequently to wake up and check device health", 2024-04-30T08:44:52.951 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.951 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.952 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.952 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.952 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.952 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.952 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.952 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.952 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.952 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.952 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.952 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.952 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.952 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.953 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.953 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.953 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.953 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.953 INFO:teuthology.orchestra.run.smithi032.stdout: "warn_threshold": { 2024-04-30T08:44:52.953 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "warn_threshold", 2024-04-30T08:44:52.953 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.953 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.953 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.953 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "7257600", 2024-04-30T08:44:52.953 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.953 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.953 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.953 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-30T08:44:52.954 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.954 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.954 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.954 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.954 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.954 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.954 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.954 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "diskprediction_local", 2024-04-30T08:44:52.954 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.954 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.954 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.954 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.955 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.955 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.955 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.955 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.955 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.955 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.955 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.955 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.955 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.955 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.955 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.955 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.955 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.955 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.956 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.956 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.956 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.956 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.956 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.956 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.956 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.956 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.956 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.956 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.956 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.956 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.956 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.956 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.956 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.957 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.957 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.957 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.957 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.957 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.957 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.957 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.957 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.957 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.957 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.957 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.957 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.957 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.958 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.958 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.958 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.958 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.958 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.958 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.958 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.958 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.958 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.958 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.958 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.958 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.958 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.959 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.959 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.959 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.959 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.959 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.959 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.959 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.959 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.959 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.959 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.959 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.959 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.959 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.959 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.960 INFO:teuthology.orchestra.run.smithi032.stdout: "predict_interval": { 2024-04-30T08:44:52.960 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "predict_interval", 2024-04-30T08:44:52.960 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.960 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.960 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.960 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "86400", 2024-04-30T08:44:52.960 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.960 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.960 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.960 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.960 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.960 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.960 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.960 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.961 INFO:teuthology.orchestra.run.smithi032.stdout: "predictor_model": { 2024-04-30T08:44:52.961 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "predictor_model", 2024-04-30T08:44:52.961 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.961 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.961 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.961 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "prophetstor", 2024-04-30T08:44:52.961 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.961 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.961 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.961 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.961 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.961 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.961 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.961 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.962 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2024-04-30T08:44:52.962 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2024-04-30T08:44:52.962 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.962 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.962 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.962 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "600", 2024-04-30T08:44:52.962 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.962 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.962 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.962 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.962 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.962 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.962 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.963 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.963 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.963 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.963 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.963 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.963 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.963 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.963 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.963 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.963 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.963 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.963 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.963 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.963 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.963 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.964 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.964 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.964 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.964 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "influx", 2024-04-30T08:44:52.964 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": false, 2024-04-30T08:44:52.964 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "influxdb python module not found", 2024-04-30T08:44:52.964 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.964 INFO:teuthology.orchestra.run.smithi032.stdout: "batch_size": { 2024-04-30T08:44:52.964 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "batch_size", 2024-04-30T08:44:52.964 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.964 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.964 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.964 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5000", 2024-04-30T08:44:52.964 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.965 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.965 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.965 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-30T08:44:52.965 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.965 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.965 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.965 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.965 INFO:teuthology.orchestra.run.smithi032.stdout: "database": { 2024-04-30T08:44:52.965 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "database", 2024-04-30T08:44:52.965 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.965 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.965 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.965 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "ceph", 2024-04-30T08:44:52.965 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.966 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.966 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.966 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-30T08:44:52.966 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.966 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.966 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.966 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.966 INFO:teuthology.orchestra.run.smithi032.stdout: "hostname": { 2024-04-30T08:44:52.966 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "hostname", 2024-04-30T08:44:52.966 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.966 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.966 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.966 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.967 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.967 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.967 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.967 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "InfluxDB server hostname", 2024-04-30T08:44:52.967 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.967 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.967 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.967 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.967 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2024-04-30T08:44:52.967 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2024-04-30T08:44:52.967 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:52.967 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.967 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.967 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "30", 2024-04-30T08:44:52.968 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "5", 2024-04-30T08:44:52.968 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.968 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.968 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-30T08:44:52.968 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.968 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.968 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.968 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.968 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.968 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.968 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.968 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.968 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.968 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.969 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.969 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.969 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.969 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.969 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.969 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.969 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.969 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.969 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.969 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.969 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.969 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.969 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.969 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.970 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.970 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.970 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.970 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.970 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.970 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.970 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.970 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.970 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.970 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.970 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.970 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.970 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.971 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.971 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.971 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.971 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.971 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.971 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.971 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.971 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.971 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.971 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.971 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.971 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.971 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.971 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.972 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.972 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.972 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.972 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.972 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.972 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.972 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.972 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.972 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.972 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.972 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.972 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.972 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.973 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.973 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.973 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.973 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.973 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.973 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.973 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.973 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.973 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.973 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.973 INFO:teuthology.orchestra.run.smithi032.stdout: "password": { 2024-04-30T08:44:52.973 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "password", 2024-04-30T08:44:52.973 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.974 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.974 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.974 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.974 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.974 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.974 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.974 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "password of InfluxDB server user", 2024-04-30T08:44:52.974 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.974 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.974 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.974 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.974 INFO:teuthology.orchestra.run.smithi032.stdout: "port": { 2024-04-30T08:44:52.974 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "port", 2024-04-30T08:44:52.975 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.975 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.975 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.975 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "8086", 2024-04-30T08:44:52.975 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.975 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.975 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.975 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "InfluxDB server port", 2024-04-30T08:44:52.975 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.975 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.975 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.975 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.975 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.975 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.976 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.976 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.976 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.976 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.976 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.976 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.976 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.976 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.976 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.976 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.976 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.976 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.976 INFO:teuthology.orchestra.run.smithi032.stdout: "ssl": { 2024-04-30T08:44:52.976 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ssl", 2024-04-30T08:44:52.977 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.977 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.977 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.977 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "false", 2024-04-30T08:44:52.977 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.977 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.977 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.977 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-30T08:44:52.977 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.977 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.977 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.977 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.977 INFO:teuthology.orchestra.run.smithi032.stdout: "threads": { 2024-04-30T08:44:52.977 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "threads", 2024-04-30T08:44:52.977 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.978 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.978 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.978 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5", 2024-04-30T08:44:52.978 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "1", 2024-04-30T08:44:52.978 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "32", 2024-04-30T08:44:52.978 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.978 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-30T08:44:52.978 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.978 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.978 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.978 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.978 INFO:teuthology.orchestra.run.smithi032.stdout: "username": { 2024-04-30T08:44:52.978 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "username", 2024-04-30T08:44:52.979 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.979 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.979 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.979 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.979 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.979 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.979 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.979 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "username of InfluxDB server user", 2024-04-30T08:44:52.979 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.979 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.979 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.979 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.979 INFO:teuthology.orchestra.run.smithi032.stdout: "verify_ssl": { 2024-04-30T08:44:52.979 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "verify_ssl", 2024-04-30T08:44:52.980 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.980 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.980 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.980 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "true", 2024-04-30T08:44:52.980 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.980 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.980 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.980 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-30T08:44:52.980 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.980 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.980 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.980 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.980 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.980 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.981 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.981 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "insights", 2024-04-30T08:44:52.981 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.981 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.981 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.981 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.981 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.981 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.981 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.981 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.981 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.981 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.981 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.981 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.982 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.982 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.982 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.982 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.982 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.982 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.982 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.982 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.982 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.982 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.982 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.982 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.982 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.983 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.983 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.983 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.983 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.983 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.983 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.983 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.983 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.983 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.983 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.983 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.983 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.983 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.983 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.983 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.984 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.984 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.984 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.984 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.984 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.984 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.984 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.984 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.984 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.984 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.984 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.984 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.984 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.985 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.985 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.985 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.985 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.985 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.985 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.985 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.985 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.985 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.985 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.985 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.985 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.985 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.985 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.986 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.986 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.986 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.986 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.986 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.986 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.986 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.986 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.986 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.986 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.986 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.986 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.986 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.987 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.987 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.987 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.987 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.987 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.987 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.987 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.987 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.987 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.987 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.987 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "iostat", 2024-04-30T08:44:52.987 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.987 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.987 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.988 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.988 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.988 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.988 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.988 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.988 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.988 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.988 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.988 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.988 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.988 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.988 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.988 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.988 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.988 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.989 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.989 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.989 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.989 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.989 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.989 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.989 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.989 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.989 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.989 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.989 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.989 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.989 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.990 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.990 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.990 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.990 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.990 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.990 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.990 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.990 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.990 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.990 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.990 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.990 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.990 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.990 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.991 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.991 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.991 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.991 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.991 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.991 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.991 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.991 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.991 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.991 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.991 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.991 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.991 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.992 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.992 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:52.992 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:52.992 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.992 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.992 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.992 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.992 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.992 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.992 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.992 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.992 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.992 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.992 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.993 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.993 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:52.993 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:52.993 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.993 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:52.993 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.993 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:52.993 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.993 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.993 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.993 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.993 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.993 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.993 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.994 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.994 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:52.994 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.994 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:52.994 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "k8sevents", 2024-04-30T08:44:52.994 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:52.994 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:52.994 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:52.994 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_event_retention_days": { 2024-04-30T08:44:52.994 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "ceph_event_retention_days", 2024-04-30T08:44:52.994 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.994 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.994 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.995 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "7", 2024-04-30T08:44:52.995 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.995 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.995 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.995 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-30T08:44:52.995 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.995 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.995 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.995 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.995 INFO:teuthology.orchestra.run.smithi032.stdout: "config_check_secs": { 2024-04-30T08:44:52.995 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "config_check_secs", 2024-04-30T08:44:52.995 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:52.995 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.995 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:52.996 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-30T08:44:52.996 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "10", 2024-04-30T08:44:52.996 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.996 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.996 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-30T08:44:52.996 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.996 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.996 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.996 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.996 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:52.996 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:52.996 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.996 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.996 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.997 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:52.997 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.997 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.997 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.997 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.997 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:52.997 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:52.997 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:52.997 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:52.997 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:52.997 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:52.997 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.997 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.997 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.998 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.998 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.998 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:52.998 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:52.998 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:52.998 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.998 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.998 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:52.998 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.998 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.998 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:52.998 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:52.998 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:52.998 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:52.999 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:52.999 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:52.999 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:52.999 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:52.999 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:52.999 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:52.999 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:52.999 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:52.999 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:52.999 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:52.999 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:52.999 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:52.999 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.000 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.000 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.000 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.000 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.000 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.000 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.000 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.000 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.000 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.000 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.000 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:53.000 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:53.000 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.000 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.001 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.001 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.001 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.001 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.001 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.001 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.001 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.001 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.001 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.001 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.001 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:53.001 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:53.001 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.001 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:53.002 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.002 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:53.002 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.002 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.002 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.002 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.002 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.002 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.002 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.002 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.002 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.002 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.002 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:53.002 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "localpool", 2024-04-30T08:44:53.003 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:53.003 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:53.003 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:53.003 INFO:teuthology.orchestra.run.smithi032.stdout: "failure_domain": { 2024-04-30T08:44:53.003 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "failure_domain", 2024-04-30T08:44:53.003 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.003 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.003 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.003 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "host", 2024-04-30T08:44:53.003 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.003 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.003 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.003 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "failure domain for any created local pool", 2024-04-30T08:44:53.004 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-30T08:44:53.004 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.004 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.004 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.004 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:53.004 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:53.004 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.004 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.004 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.004 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.004 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.004 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.004 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.004 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.004 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.005 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.005 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.005 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.005 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.005 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.005 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.005 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.005 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.005 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.005 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.005 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:53.005 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:53.005 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.006 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.006 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.006 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.006 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.006 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.006 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.006 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.006 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.006 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.006 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.006 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.006 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:53.006 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:53.007 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.007 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.007 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.007 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:53.007 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.007 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.007 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.007 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.007 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.007 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.007 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.007 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.007 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.007 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.008 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.008 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.008 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.008 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.008 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.008 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:53.008 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:53.008 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.008 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.008 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.008 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.008 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.008 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.008 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.008 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.008 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.009 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.009 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.009 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.009 INFO:teuthology.orchestra.run.smithi032.stdout: "min_size": { 2024-04-30T08:44:53.009 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "min_size", 2024-04-30T08:44:53.009 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.009 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.009 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.009 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.009 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.009 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.009 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.009 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "default min_size for any created local pool", 2024-04-30T08:44:53.010 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-30T08:44:53.010 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.010 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.010 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.010 INFO:teuthology.orchestra.run.smithi032.stdout: "num_rep": { 2024-04-30T08:44:53.010 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "num_rep", 2024-04-30T08:44:53.010 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.010 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.010 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.010 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "3", 2024-04-30T08:44:53.010 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.010 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.010 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.010 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "default replica count for any created local pool", 2024-04-30T08:44:53.011 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.011 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.011 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.011 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.011 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_num": { 2024-04-30T08:44:53.011 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "pg_num", 2024-04-30T08:44:53.011 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.011 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.011 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.011 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "128", 2024-04-30T08:44:53.011 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.011 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.011 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.012 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "default pg_num for any created local pool", 2024-04-30T08:44:53.012 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.012 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.012 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.012 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.012 INFO:teuthology.orchestra.run.smithi032.stdout: "prefix": { 2024-04-30T08:44:53.012 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "prefix", 2024-04-30T08:44:53.012 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.012 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.012 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.012 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.012 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.012 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.012 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.012 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "name prefix for any created local pool", 2024-04-30T08:44:53.013 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.013 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.013 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.013 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.013 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:53.013 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:53.013 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.013 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:53.013 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.013 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:53.013 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.013 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.013 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.013 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.014 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.014 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.014 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.014 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.014 INFO:teuthology.orchestra.run.smithi032.stdout: "subtree": { 2024-04-30T08:44:53.014 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "subtree", 2024-04-30T08:44:53.014 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.014 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.014 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.014 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "rack", 2024-04-30T08:44:53.014 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.014 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.014 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.015 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-30T08:44:53.015 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-30T08:44:53.015 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.015 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.015 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.015 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.015 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.015 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:53.015 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mds_autoscaler", 2024-04-30T08:44:53.015 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:53.015 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:53.015 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:53.016 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:53.016 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:53.016 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.016 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.016 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.016 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.016 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.016 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.016 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.016 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.016 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.016 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.016 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.016 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.016 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.017 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.017 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.017 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.017 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.017 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.017 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.017 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:53.017 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:53.017 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.017 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.017 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.017 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.017 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.017 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.018 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.018 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.018 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.018 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.018 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.018 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.018 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:53.018 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:53.018 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.018 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.018 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.018 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:53.018 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.019 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.019 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.019 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.019 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.019 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.019 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.019 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.019 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.019 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.019 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.019 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.019 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.019 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.019 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.020 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:53.020 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:53.020 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.020 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.020 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.020 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.020 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.020 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.020 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.020 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.020 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.020 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.020 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.021 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.021 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:53.021 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:53.021 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.021 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:53.021 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.021 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:53.021 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.021 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.021 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.021 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.021 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.021 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.021 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.021 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.022 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.022 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.022 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:53.022 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mirroring", 2024-04-30T08:44:53.022 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:53.022 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:53.022 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:53.022 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:53.022 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:53.022 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.022 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.022 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.022 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.023 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.023 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.023 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.023 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.023 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.023 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.023 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.023 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.023 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.023 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.023 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.023 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.023 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.023 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.024 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.024 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:53.024 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:53.024 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.024 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.024 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.024 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.024 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.024 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.024 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.024 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.024 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.024 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.024 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.025 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.025 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:53.025 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:53.025 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.025 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.025 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.025 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:53.025 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.025 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.025 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.025 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.025 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.025 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.025 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.025 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.026 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.026 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.026 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.026 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.026 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.026 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.026 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.026 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:53.026 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:53.026 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.026 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.026 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.026 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.027 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.027 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.027 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.027 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.027 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.027 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.027 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.027 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.027 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:53.027 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:53.027 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.027 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:53.027 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.027 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:53.028 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.028 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.028 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.028 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.028 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.028 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.028 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.028 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.028 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.028 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.028 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:53.028 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "nfs", 2024-04-30T08:44:53.028 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:53.028 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:53.028 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:53.029 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:53.029 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:53.029 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.029 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.029 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.029 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.029 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.029 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.029 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.029 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.029 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.029 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.029 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.029 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.030 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.030 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.030 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.030 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.030 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.030 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.030 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.030 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:53.030 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:53.030 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.030 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.030 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.030 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.030 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.030 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.031 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.031 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.031 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.031 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.031 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.031 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.031 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:53.031 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:53.031 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.031 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.031 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.031 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:53.031 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.032 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.032 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.032 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.032 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.032 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.032 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.032 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.032 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.032 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.032 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.032 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.032 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.032 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.032 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.032 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:53.033 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:53.033 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.033 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.033 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.033 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.033 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.033 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.033 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.033 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.033 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.033 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.033 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.033 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.033 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:53.034 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:53.034 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.034 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:53.034 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.034 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:53.034 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.034 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.034 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.034 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.034 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.034 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.034 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.034 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.034 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.035 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.035 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:53.035 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "orchestrator", 2024-04-30T08:44:53.035 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:53.035 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:53.035 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:53.035 INFO:teuthology.orchestra.run.smithi032.stdout: "fail_fs": { 2024-04-30T08:44:53.035 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "fail_fs", 2024-04-30T08:44:53.035 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.035 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.035 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.035 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.035 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.035 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.036 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.036 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-30T08:44:53.036 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.036 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.036 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.036 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.036 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:53.036 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:53.036 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.036 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.036 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.036 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.036 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.036 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.037 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.037 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.037 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.037 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.037 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.037 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.037 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.037 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.037 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.037 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.037 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.037 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.037 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.038 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:53.038 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:53.038 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.038 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.038 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.038 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.038 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.038 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.038 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.038 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.038 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.038 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.038 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.039 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.039 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:53.039 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:53.039 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.039 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.039 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.039 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:53.039 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.039 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.039 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.039 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.039 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.039 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.039 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.040 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.040 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.040 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.040 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.040 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.040 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.040 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.040 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.040 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:53.040 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:53.040 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.040 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.040 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.041 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.041 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.041 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.041 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.041 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.041 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.041 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.041 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.041 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.041 INFO:teuthology.orchestra.run.smithi032.stdout: "orchestrator": { 2024-04-30T08:44:53.041 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "orchestrator", 2024-04-30T08:44:53.041 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.041 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.041 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: "cephadm", 2024-04-30T08:44:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: "rook", 2024-04-30T08:44:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: "test_orchestrator" 2024-04-30T08:44:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Orchestrator backend", 2024-04-30T08:44:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.042 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.043 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:53.043 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:53.043 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.043 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:53.043 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.043 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:53.043 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.043 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.043 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.043 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.043 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.043 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.043 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.043 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.044 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.044 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.044 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:53.044 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "osd_perf_query", 2024-04-30T08:44:53.044 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:53.044 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:53.044 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:53.044 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:53.044 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:53.044 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.044 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.044 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.044 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.044 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.045 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.045 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.045 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.045 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.045 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.045 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.045 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.045 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.045 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.045 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.045 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.045 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.045 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.046 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.046 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:53.046 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:53.046 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.046 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.046 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.046 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.046 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.046 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.046 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.046 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.046 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.046 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.046 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.047 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.047 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:53.047 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:53.047 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.047 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.047 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.047 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:53.047 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.047 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.047 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.047 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.047 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.047 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.047 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.048 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.048 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.048 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.048 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.048 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.048 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.048 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.048 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.048 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:53.048 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:53.048 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.048 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.048 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.048 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.049 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.049 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.049 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.049 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.049 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.049 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.049 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.049 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.049 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:53.049 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:53.049 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.049 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:53.049 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.050 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:53.050 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.050 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.050 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.050 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.050 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.050 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.050 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.050 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.050 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.050 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.050 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:53.050 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "osd_support", 2024-04-30T08:44:53.050 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:53.051 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:53.051 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:53.051 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:53.051 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:53.051 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.051 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.051 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.051 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.051 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.051 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.051 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.051 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.051 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.051 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.052 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.052 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.052 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.052 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.052 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.052 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.052 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.052 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.052 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.052 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:53.052 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:53.052 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.052 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.052 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.053 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.053 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.053 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.053 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.053 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.053 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.053 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.053 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.053 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.053 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:53.053 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:53.053 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.053 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.053 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.054 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:53.054 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.054 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.054 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.054 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.054 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.054 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.054 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.054 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.054 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.054 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.054 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.054 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.055 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.055 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.055 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.055 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:53.055 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:53.055 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.055 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.055 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.055 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.055 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.055 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.055 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.055 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.056 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.056 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.056 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.056 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.056 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:53.056 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:53.056 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.056 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:53.056 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.056 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:53.056 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.056 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.056 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.056 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.057 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.057 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.057 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.057 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.057 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.057 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.057 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:53.057 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "pg_autoscaler", 2024-04-30T08:44:53.057 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:53.057 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:53.057 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:53.057 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:53.057 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:53.057 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.058 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.058 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.058 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.058 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.058 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.058 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.058 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.058 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.058 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.058 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.058 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.058 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.058 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.059 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.059 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.059 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.059 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.059 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.059 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:53.059 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:53.059 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.059 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.059 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.059 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.059 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.059 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.060 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.060 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.060 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.060 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.060 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.060 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.060 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:53.060 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:53.060 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.060 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.060 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.060 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:53.060 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.060 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.061 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.061 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.061 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.061 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.061 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.061 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.061 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.061 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.061 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.061 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.061 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.061 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.061 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.061 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:53.062 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:53.062 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.062 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.062 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.062 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.062 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.062 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.062 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.062 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.062 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.062 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.062 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.063 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.063 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2024-04-30T08:44:53.063 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2024-04-30T08:44:53.063 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:53.063 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.063 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.063 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2024-04-30T08:44:53.063 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.063 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.063 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.063 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.063 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.063 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.063 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.064 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.064 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:53.064 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:53.064 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.064 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:53.064 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.064 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:53.064 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.064 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.064 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.064 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.064 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.064 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.064 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.065 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.065 INFO:teuthology.orchestra.run.smithi032.stdout: "threshold": { 2024-04-30T08:44:53.065 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "threshold", 2024-04-30T08:44:53.065 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-30T08:44:53.065 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.065 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.065 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "3.0", 2024-04-30T08:44:53.065 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "1.0", 2024-04-30T08:44:53.065 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.065 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.065 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "scaling threshold", 2024-04-30T08:44:53.065 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-30T08:44:53.065 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.065 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.066 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.066 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.066 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.066 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:53.066 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "progress", 2024-04-30T08:44:53.066 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:53.066 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:53.066 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:53.066 INFO:teuthology.orchestra.run.smithi032.stdout: "allow_pg_recovery_event": { 2024-04-30T08:44:53.066 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "allow_pg_recovery_event", 2024-04-30T08:44:53.066 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.066 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.066 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.067 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.067 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.067 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.067 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.067 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-30T08:44:53.067 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.067 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.067 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.067 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.067 INFO:teuthology.orchestra.run.smithi032.stdout: "enabled": { 2024-04-30T08:44:53.067 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "enabled", 2024-04-30T08:44:53.067 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.067 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.067 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.068 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:53.068 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.068 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.068 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.068 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.068 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.068 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.068 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.068 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.068 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:53.068 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:53.068 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.068 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.068 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.069 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.069 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.069 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.069 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.069 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.069 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.069 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.069 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.069 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.069 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.069 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.069 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.069 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.069 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.070 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.070 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.070 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:53.070 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:53.070 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.070 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.070 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.070 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.070 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.070 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.070 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.070 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.070 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.070 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.071 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.071 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.071 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:53.071 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:53.071 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.071 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.071 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.071 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:53.071 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.071 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.071 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.071 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.071 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.071 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.072 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.072 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.072 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.072 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.072 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.072 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.072 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.072 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.072 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.072 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:53.072 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:53.072 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.072 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.073 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.073 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.073 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.073 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.073 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.073 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.073 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.073 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.073 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.073 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.073 INFO:teuthology.orchestra.run.smithi032.stdout: "max_completed_events": { 2024-04-30T08:44:53.073 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "max_completed_events", 2024-04-30T08:44:53.073 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.073 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.074 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.074 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "50", 2024-04-30T08:44:53.074 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.074 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.074 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.074 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "number of past completed events to remember", 2024-04-30T08:44:53.074 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.074 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.074 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.074 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.074 INFO:teuthology.orchestra.run.smithi032.stdout: "sleep_interval": { 2024-04-30T08:44:53.074 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sleep_interval", 2024-04-30T08:44:53.074 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:53.075 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.075 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.075 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "5", 2024-04-30T08:44:53.075 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.075 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.075 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.075 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "how long the module is going to sleep", 2024-04-30T08:44:53.075 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.075 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.075 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.075 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.075 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:53.075 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:53.075 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.076 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:53.076 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.076 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:53.076 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.076 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.076 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.076 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.076 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.076 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.076 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.076 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.076 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.076 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.076 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:53.077 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "prometheus", 2024-04-30T08:44:53.077 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:53.077 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:53.077 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:53.077 INFO:teuthology.orchestra.run.smithi032.stdout: "cache": { 2024-04-30T08:44:53.077 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "cache", 2024-04-30T08:44:53.077 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.077 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.077 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.077 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:53.077 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.077 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.077 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.078 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.078 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.078 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.078 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.078 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.078 INFO:teuthology.orchestra.run.smithi032.stdout: "exclude_perf_counters": { 2024-04-30T08:44:53.078 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "exclude_perf_counters", 2024-04-30T08:44:53.078 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.078 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.078 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.078 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:53.078 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.078 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.078 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.079 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-30T08:44:53.079 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-30T08:44:53.079 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.079 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.079 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.079 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:53.079 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:53.079 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.079 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.079 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.079 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.079 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.079 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.080 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.080 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.080 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.080 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.080 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.080 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.080 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.080 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.080 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.080 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.080 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.080 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.080 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.080 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:53.081 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:53.081 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.081 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.081 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.081 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.081 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.081 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.081 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.081 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.081 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.081 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.081 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.081 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.081 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:53.081 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:53.082 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.082 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.082 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.082 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:53.082 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.082 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.082 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.082 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.082 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.082 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.082 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.082 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.082 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.083 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.083 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.083 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.083 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.083 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.083 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.083 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:53.083 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:53.083 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.083 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.083 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.083 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.083 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.083 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.084 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.084 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.084 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.084 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.084 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.084 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.084 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_stats_pools": { 2024-04-30T08:44:53.084 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rbd_stats_pools", 2024-04-30T08:44:53.084 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.084 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.084 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.084 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.084 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.084 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.085 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.085 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.085 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.085 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.085 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.085 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.085 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-30T08:44:53.085 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-30T08:44:53.085 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.085 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.085 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.085 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "300", 2024-04-30T08:44:53.085 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.085 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.086 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.086 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.086 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.086 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.086 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.086 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.086 INFO:teuthology.orchestra.run.smithi032.stdout: "scrape_interval": { 2024-04-30T08:44:53.086 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "scrape_interval", 2024-04-30T08:44:53.086 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-30T08:44:53.086 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.086 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.086 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "15.0", 2024-04-30T08:44:53.086 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.087 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.087 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.087 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.087 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.087 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.087 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.087 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.087 INFO:teuthology.orchestra.run.smithi032.stdout: "server_addr": { 2024-04-30T08:44:53.087 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_addr", 2024-04-30T08:44:53.087 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.087 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.087 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.087 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "::", 2024-04-30T08:44:53.087 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.088 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.088 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.088 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-30T08:44:53.088 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.088 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.088 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.088 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.088 INFO:teuthology.orchestra.run.smithi032.stdout: "server_port": { 2024-04-30T08:44:53.088 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_port", 2024-04-30T08:44:53.088 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.088 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.088 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.088 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "9283", 2024-04-30T08:44:53.088 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.089 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.089 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.089 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-30T08:44:53.089 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.089 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.089 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.089 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.089 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:53.089 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:53.089 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.089 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:53.089 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.089 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:53.089 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.090 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.090 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.090 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.090 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.090 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.090 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.090 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.090 INFO:teuthology.orchestra.run.smithi032.stdout: "stale_cache_strategy": { 2024-04-30T08:44:53.090 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "stale_cache_strategy", 2024-04-30T08:44:53.090 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.090 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.090 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.090 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "log", 2024-04-30T08:44:53.091 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.091 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.091 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.091 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.091 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.091 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.091 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.091 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.091 INFO:teuthology.orchestra.run.smithi032.stdout: "standby_behaviour": { 2024-04-30T08:44:53.091 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "standby_behaviour", 2024-04-30T08:44:53.091 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.091 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.091 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.092 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "default", 2024-04-30T08:44:53.092 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.092 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.092 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.092 INFO:teuthology.orchestra.run.smithi032.stdout: "default", 2024-04-30T08:44:53.092 INFO:teuthology.orchestra.run.smithi032.stdout: "error" 2024-04-30T08:44:53.092 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.092 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.092 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.092 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.092 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.092 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.092 INFO:teuthology.orchestra.run.smithi032.stdout: "standby_error_status_code": { 2024-04-30T08:44:53.092 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "standby_error_status_code", 2024-04-30T08:44:53.093 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.093 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.093 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.093 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "500", 2024-04-30T08:44:53.093 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "400", 2024-04-30T08:44:53.093 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "599", 2024-04-30T08:44:53.093 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.093 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.093 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.093 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.093 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.093 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.093 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.093 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.094 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:53.094 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rbd_support", 2024-04-30T08:44:53.094 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:53.094 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:53.094 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:53.094 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:53.094 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:53.094 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.094 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.094 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.094 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.094 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.094 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.095 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.095 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.095 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.095 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.095 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.095 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.095 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.095 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.095 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.095 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.095 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.095 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.095 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.095 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:53.096 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:53.096 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.096 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.096 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.096 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.096 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.096 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.096 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.096 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.096 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.096 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.096 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.096 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.097 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:53.097 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:53.097 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.097 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.097 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.097 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:53.097 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.097 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.097 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.097 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.097 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.097 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.097 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.097 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.098 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.098 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.098 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.098 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.098 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.098 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.098 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.098 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:53.098 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:53.098 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.098 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.098 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.098 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.098 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.099 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.099 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.099 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.099 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.099 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.099 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.099 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.099 INFO:teuthology.orchestra.run.smithi032.stdout: "max_concurrent_snap_create": { 2024-04-30T08:44:53.099 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "max_concurrent_snap_create", 2024-04-30T08:44:53.099 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.099 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.099 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.099 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10", 2024-04-30T08:44:53.099 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.100 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.100 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.100 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.100 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.100 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.100 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.100 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.100 INFO:teuthology.orchestra.run.smithi032.stdout: "mirror_snapshot_schedule": { 2024-04-30T08:44:53.100 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "mirror_snapshot_schedule", 2024-04-30T08:44:53.100 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.100 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.100 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.100 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.100 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.101 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.101 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.101 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.101 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.101 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.101 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.101 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.101 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:53.101 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:53.101 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.101 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:53.101 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.101 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:53.101 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.102 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.102 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.102 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.102 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.102 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.102 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.102 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.102 INFO:teuthology.orchestra.run.smithi032.stdout: "trash_purge_schedule": { 2024-04-30T08:44:53.102 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "trash_purge_schedule", 2024-04-30T08:44:53.102 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.102 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.102 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.102 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.103 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.103 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.103 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.103 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.103 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.103 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.103 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.103 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.103 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.103 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.103 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:53.103 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "restful", 2024-04-30T08:44:53.103 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:53.103 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:53.104 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:53.104 INFO:teuthology.orchestra.run.smithi032.stdout: "enable_auth": { 2024-04-30T08:44:53.104 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "enable_auth", 2024-04-30T08:44:53.104 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.104 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.104 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.104 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:53.104 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.104 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.104 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.104 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.104 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.104 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.104 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.105 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.105 INFO:teuthology.orchestra.run.smithi032.stdout: "key_file": { 2024-04-30T08:44:53.105 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "key_file", 2024-04-30T08:44:53.105 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.105 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.105 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.105 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.105 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.105 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.105 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.105 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.105 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.105 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.105 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.106 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.106 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:53.106 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:53.106 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.106 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.106 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.106 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.106 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.106 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.106 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.106 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.106 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.106 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.107 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.107 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.107 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.107 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.107 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.107 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.107 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.107 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.107 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.107 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:53.107 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:53.107 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.107 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.107 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.108 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.108 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.108 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.108 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.108 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.108 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.108 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.108 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.108 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.108 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:53.108 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:53.108 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.108 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.108 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.109 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:53.109 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.109 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.109 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.109 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.109 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.109 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.109 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.109 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.109 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.109 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.109 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.109 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.109 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.110 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.110 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.110 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:53.110 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:53.110 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.110 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.110 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.110 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.110 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.110 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.110 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.110 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.110 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.110 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.111 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.111 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.111 INFO:teuthology.orchestra.run.smithi032.stdout: "server_addr": { 2024-04-30T08:44:53.111 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_addr", 2024-04-30T08:44:53.111 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.111 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.111 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.111 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.111 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.111 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.111 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.111 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.111 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.111 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.112 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.112 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.112 INFO:teuthology.orchestra.run.smithi032.stdout: "server_port": { 2024-04-30T08:44:53.112 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "server_port", 2024-04-30T08:44:53.112 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.112 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.112 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.112 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.112 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.112 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.112 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.112 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.112 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.113 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.113 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.113 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.113 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:53.113 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:53.113 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.113 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:53.113 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.113 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:53.113 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.113 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.113 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.113 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.113 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.114 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.114 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.114 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.114 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.114 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.114 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:53.114 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rgw", 2024-04-30T08:44:53.114 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:53.114 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:53.114 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:53.114 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:53.114 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:53.114 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.115 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.115 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.115 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.115 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.115 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.115 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.115 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.115 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.115 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.115 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.115 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.115 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.115 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.115 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.116 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.116 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.116 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.116 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.116 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:53.116 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:53.116 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.116 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.116 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.116 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.116 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.116 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.116 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.116 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.117 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.117 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.117 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.117 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.117 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:53.117 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:53.117 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.117 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.117 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.117 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:53.117 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.117 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.117 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.117 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.118 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.118 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.118 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.118 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.118 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.118 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.118 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.118 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.118 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.118 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.118 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.118 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:53.118 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:53.118 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.119 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.119 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.119 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.119 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.119 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.119 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.119 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.119 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.119 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.119 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.119 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.119 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:53.119 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:53.120 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.120 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:53.120 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.120 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:53.120 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.120 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.120 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.120 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.120 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.120 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.120 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.120 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.120 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.121 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.121 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:53.121 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rook", 2024-04-30T08:44:53.121 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:53.121 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:53.121 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:53.121 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:53.121 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:53.121 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.121 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.121 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.121 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.121 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.121 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.122 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.122 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.122 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.122 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.122 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.122 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.122 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.122 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.122 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.122 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.122 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.122 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.123 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.123 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:53.123 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:53.123 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.123 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.123 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.123 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.123 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.123 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.123 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.123 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.123 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.123 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.124 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.124 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.124 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:53.124 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:53.124 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.124 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.124 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.124 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:53.124 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.124 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.124 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.124 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.125 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.125 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.125 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.125 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.125 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.125 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.125 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.125 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.125 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.125 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.125 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.125 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:53.126 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:53.126 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.126 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.126 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.126 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.126 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.126 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.126 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.126 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.126 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.126 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.126 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.126 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.127 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:53.127 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:53.127 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.127 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:53.127 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.127 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:53.127 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.127 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.127 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.127 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.127 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.127 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.128 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.128 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.128 INFO:teuthology.orchestra.run.smithi032.stdout: "storage_class": { 2024-04-30T08:44:53.128 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "storage_class", 2024-04-30T08:44:53.128 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.128 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.128 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.128 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "local", 2024-04-30T08:44:53.128 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.128 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.128 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.128 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-30T08:44:53.128 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.129 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.129 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.129 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.129 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.129 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.129 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:53.129 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "selftest", 2024-04-30T08:44:53.129 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:53.129 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:53.129 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:53.129 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:53.129 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:53.129 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.130 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.130 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.130 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.130 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.130 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.130 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.130 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.130 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.130 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.130 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.130 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.130 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.131 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.131 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.131 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.131 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.131 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.131 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.131 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:53.131 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:53.131 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.131 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.131 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.131 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.131 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.132 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.132 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.132 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.132 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.132 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.132 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.132 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.132 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:53.132 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:53.132 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.132 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.132 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.133 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:53.133 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.133 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.133 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.133 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.133 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.133 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.134 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.134 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.134 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.134 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.134 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.134 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.134 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.134 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.134 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.134 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:53.135 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:53.135 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.135 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.135 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.135 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.135 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.135 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.135 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.135 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.135 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.135 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.135 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.135 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.136 INFO:teuthology.orchestra.run.smithi032.stdout: "roption1": { 2024-04-30T08:44:53.136 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "roption1", 2024-04-30T08:44:53.136 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.136 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.136 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.136 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.136 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.136 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.136 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.136 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.136 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.136 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.136 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.137 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.137 INFO:teuthology.orchestra.run.smithi032.stdout: "roption2": { 2024-04-30T08:44:53.137 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "roption2", 2024-04-30T08:44:53.137 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.137 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.137 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.137 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "xyz", 2024-04-30T08:44:53.137 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.137 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.137 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.137 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.137 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.137 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption1": { 2024-04-30T08:44:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption1", 2024-04-30T08:44:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.138 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.139 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.139 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.139 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.139 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption2": { 2024-04-30T08:44:53.139 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption2", 2024-04-30T08:44:53.139 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.139 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.139 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.139 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.139 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.139 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.139 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.139 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.139 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.140 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.140 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.140 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.140 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption3": { 2024-04-30T08:44:53.140 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption3", 2024-04-30T08:44:53.140 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "float", 2024-04-30T08:44:53.140 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.140 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.140 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.140 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.140 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.140 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.140 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.141 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.141 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.141 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.141 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.141 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption4": { 2024-04-30T08:44:53.141 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption4", 2024-04-30T08:44:53.141 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.141 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.141 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.141 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.141 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.141 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.142 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.142 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.142 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.142 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.142 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.142 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.142 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption5": { 2024-04-30T08:44:53.142 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption5", 2024-04-30T08:44:53.142 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.142 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.142 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.142 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.142 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.142 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.143 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.143 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.143 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.143 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.143 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.143 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.143 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption6": { 2024-04-30T08:44:53.143 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption6", 2024-04-30T08:44:53.143 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.143 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.143 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.143 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:53.143 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.144 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.144 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.144 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.144 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.144 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.144 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.144 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.144 INFO:teuthology.orchestra.run.smithi032.stdout: "rwoption7": { 2024-04-30T08:44:53.144 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rwoption7", 2024-04-30T08:44:53.144 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.144 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.144 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.144 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.145 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "1", 2024-04-30T08:44:53.145 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "42", 2024-04-30T08:44:53.145 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.145 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.145 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.145 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.145 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.145 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.145 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:53.145 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:53.145 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.145 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:53.145 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.146 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:53.146 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.146 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.146 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.146 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.146 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.146 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.146 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.146 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.146 INFO:teuthology.orchestra.run.smithi032.stdout: "testkey": { 2024-04-30T08:44:53.146 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "testkey", 2024-04-30T08:44:53.146 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.146 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.147 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.147 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.147 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.147 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.147 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.147 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.147 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.147 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.147 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.147 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.147 INFO:teuthology.orchestra.run.smithi032.stdout: "testlkey": { 2024-04-30T08:44:53.147 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "testlkey", 2024-04-30T08:44:53.147 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.148 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.148 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.148 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.148 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.148 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.148 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.148 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.148 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.148 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.148 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.148 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.148 INFO:teuthology.orchestra.run.smithi032.stdout: "testnewline": { 2024-04-30T08:44:53.148 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "testnewline", 2024-04-30T08:44:53.149 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.149 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.149 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.149 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.149 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.149 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.149 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.149 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.149 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.149 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.149 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.149 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.149 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.149 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.150 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:53.150 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "snap_schedule", 2024-04-30T08:44:53.150 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:53.150 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:53.150 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:53.150 INFO:teuthology.orchestra.run.smithi032.stdout: "allow_m_granularity": { 2024-04-30T08:44:53.150 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "allow_m_granularity", 2024-04-30T08:44:53.150 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.150 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.150 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.150 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.150 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.150 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.151 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.151 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "allow minute scheduled snapshots", 2024-04-30T08:44:53.151 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.151 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.151 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.151 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.151 INFO:teuthology.orchestra.run.smithi032.stdout: "dump_on_update": { 2024-04-30T08:44:53.151 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "dump_on_update", 2024-04-30T08:44:53.151 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.151 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.151 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.151 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.151 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.152 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.152 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.152 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "dump database to debug log on update", 2024-04-30T08:44:53.152 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.152 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.152 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.152 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.152 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:53.152 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:53.152 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.152 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.152 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.152 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.152 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.152 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.153 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.153 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.153 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.153 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.153 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.154 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.154 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.154 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.154 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.154 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.154 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.154 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.154 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.154 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:53.154 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:53.154 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.154 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.154 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.154 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.155 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.155 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.155 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.155 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.155 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.155 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.155 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.155 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.155 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:53.155 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:53.155 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.155 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.155 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.155 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:53.156 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.156 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.156 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.156 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.156 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.156 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.156 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.156 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.156 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.156 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.156 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.156 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.156 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.156 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.157 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.157 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:53.157 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:53.157 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.157 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.157 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.157 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.157 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.157 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.157 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.157 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.157 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.157 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.158 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.158 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.158 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:53.158 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:53.158 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.158 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:53.158 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.158 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:53.158 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.158 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.158 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.158 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.158 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.158 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.159 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.159 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.159 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.159 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.159 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:53.159 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "stats", 2024-04-30T08:44:53.159 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:53.159 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:53.159 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:53.159 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:53.159 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:53.159 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.159 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.159 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.160 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.160 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.160 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.160 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.160 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.160 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.160 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.160 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.160 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.160 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.160 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.160 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.160 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.160 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.161 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.161 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.161 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:53.161 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:53.161 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.161 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.161 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.161 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.161 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.161 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.161 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.161 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.161 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.161 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.162 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.162 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.162 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:53.162 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:53.162 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.162 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.162 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.162 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:53.162 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.162 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.162 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.162 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.163 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.163 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.163 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.163 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.163 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.163 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.163 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.163 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.163 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.163 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.163 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.163 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:53.163 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:53.163 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.164 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.164 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.164 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.164 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.164 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.164 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.164 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.164 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.164 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.164 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.164 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.164 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:53.164 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:53.165 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.165 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:53.165 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.165 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:53.165 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.165 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.165 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.165 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.165 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.165 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.165 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.165 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.165 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.165 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.166 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:53.166 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "status", 2024-04-30T08:44:53.166 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:53.166 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:53.166 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:53.166 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:53.166 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:53.166 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.166 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.166 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.166 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.166 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.166 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.167 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.167 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.167 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.167 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.167 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.167 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.167 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.167 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.167 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.167 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.167 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.167 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.167 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.167 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:53.168 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:53.168 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.168 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.168 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.168 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.168 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.168 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.168 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.168 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.168 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.168 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.168 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.168 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.168 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:53.169 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:53.169 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.169 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.169 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.169 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:53.169 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.169 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.169 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.169 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.169 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.169 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.169 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.169 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.170 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.170 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.170 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.170 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.170 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.170 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.170 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.170 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:53.170 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:53.170 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.170 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.170 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.170 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.170 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.171 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.171 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.171 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.171 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.171 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.171 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.171 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.171 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:53.171 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:53.171 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.171 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:53.171 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.171 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:53.171 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.172 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.172 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.172 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.172 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.172 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.172 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.172 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.172 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.172 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.172 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:53.172 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "telegraf", 2024-04-30T08:44:53.172 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:53.172 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:53.172 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:53.173 INFO:teuthology.orchestra.run.smithi032.stdout: "address": { 2024-04-30T08:44:53.173 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "address", 2024-04-30T08:44:53.173 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.173 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.173 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.173 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-30T08:44:53.173 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.173 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.173 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.173 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.173 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.173 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.174 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.174 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.174 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2024-04-30T08:44:53.174 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2024-04-30T08:44:53.174 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:53.174 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.174 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.174 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "15", 2024-04-30T08:44:53.174 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.174 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.174 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.174 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.174 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.174 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.175 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.175 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.175 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:53.175 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:53.175 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.175 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.175 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.175 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.175 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.175 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.175 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.175 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.175 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.175 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.176 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.176 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.176 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.176 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.176 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.176 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.176 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.176 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.176 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.176 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:53.176 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:53.176 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.176 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.176 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.177 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.177 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.177 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.177 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.177 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.177 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.177 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.177 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.177 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.177 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:53.177 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:53.177 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.177 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.177 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.178 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:53.178 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.178 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.178 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.178 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.178 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.178 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.178 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.178 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.178 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.178 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.178 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.178 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.178 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.179 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.179 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.179 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:53.179 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:53.179 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.179 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.179 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.179 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.179 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.179 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.179 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.179 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.179 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.180 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.180 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.180 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.180 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:53.180 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:53.180 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.180 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:53.180 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.180 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:53.180 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.180 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.180 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.180 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.181 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.181 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.181 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.181 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.181 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.181 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.181 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:53.181 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "telemetry", 2024-04-30T08:44:53.181 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:53.181 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:53.181 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:53.181 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_basic": { 2024-04-30T08:44:53.181 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_basic", 2024-04-30T08:44:53.181 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.182 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.182 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.182 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:53.182 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.182 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.182 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.182 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-30T08:44:53.182 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.182 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.182 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.182 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.182 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_crash": { 2024-04-30T08:44:53.182 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_crash", 2024-04-30T08:44:53.183 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.183 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.183 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.183 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:53.183 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.183 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.183 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.183 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-30T08:44:53.183 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.183 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.183 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.183 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.183 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_device": { 2024-04-30T08:44:53.183 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_device", 2024-04-30T08:44:53.184 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.184 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.184 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.184 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:53.184 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.184 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.184 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.184 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-30T08:44:53.184 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.184 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.184 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.184 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.184 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_ident": { 2024-04-30T08:44:53.184 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_ident", 2024-04-30T08:44:53.184 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.185 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.185 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.185 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.185 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.185 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.185 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.185 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-30T08:44:53.185 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.185 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.185 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.185 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.185 INFO:teuthology.orchestra.run.smithi032.stdout: "channel_perf": { 2024-04-30T08:44:53.185 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "channel_perf", 2024-04-30T08:44:53.186 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.186 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.186 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.186 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.186 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.186 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.186 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.186 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-30T08:44:53.186 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.186 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.186 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.186 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.186 INFO:teuthology.orchestra.run.smithi032.stdout: "contact": { 2024-04-30T08:44:53.187 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "contact", 2024-04-30T08:44:53.187 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.187 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.187 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.187 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.187 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.187 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.187 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.187 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.187 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.187 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.187 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.187 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.187 INFO:teuthology.orchestra.run.smithi032.stdout: "description": { 2024-04-30T08:44:53.188 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "description", 2024-04-30T08:44:53.188 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.188 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.188 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.188 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.188 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.188 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.188 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.188 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.188 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.188 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.188 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.188 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.188 INFO:teuthology.orchestra.run.smithi032.stdout: "device_url": { 2024-04-30T08:44:53.189 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "device_url", 2024-04-30T08:44:53.189 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.189 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.189 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.189 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-30T08:44:53.189 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.189 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.189 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.189 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.189 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.189 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.189 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.189 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.189 INFO:teuthology.orchestra.run.smithi032.stdout: "enabled": { 2024-04-30T08:44:53.190 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "enabled", 2024-04-30T08:44:53.190 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.190 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.190 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.190 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.190 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.190 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.190 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.190 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.190 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.190 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.190 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.190 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.191 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2024-04-30T08:44:53.191 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2024-04-30T08:44:53.191 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.191 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.191 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.191 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "24", 2024-04-30T08:44:53.191 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "8", 2024-04-30T08:44:53.191 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.191 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.191 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.191 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.191 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.191 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.191 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.192 INFO:teuthology.orchestra.run.smithi032.stdout: "last_opt_revision": { 2024-04-30T08:44:53.192 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "last_opt_revision", 2024-04-30T08:44:53.192 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.192 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.192 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.192 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "1", 2024-04-30T08:44:53.192 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.192 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.192 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.192 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.192 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.192 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.192 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.192 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.193 INFO:teuthology.orchestra.run.smithi032.stdout: "leaderboard": { 2024-04-30T08:44:53.193 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "leaderboard", 2024-04-30T08:44:53.193 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.193 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.193 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.193 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.193 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.193 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.193 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.193 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.193 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.193 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.193 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.193 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.193 INFO:teuthology.orchestra.run.smithi032.stdout: "leaderboard_description": { 2024-04-30T08:44:53.194 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "leaderboard_description", 2024-04-30T08:44:53.194 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.194 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.194 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.194 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.194 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.194 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.194 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.194 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.194 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.194 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.194 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.194 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.195 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:53.195 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:53.195 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.195 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.195 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.195 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.195 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.195 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.195 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.195 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.195 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.195 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.195 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.195 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.196 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.196 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.196 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.196 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.196 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.196 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.196 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.196 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:53.196 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:53.196 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.196 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.196 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.196 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.197 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.197 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.197 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.197 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.197 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.197 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.197 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.197 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.197 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:53.197 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:53.197 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.197 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.197 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.198 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:53.198 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.198 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.198 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.198 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.198 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.198 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.198 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.198 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.198 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.198 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.198 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.198 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.199 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.199 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.199 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.199 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:53.199 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:53.199 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.199 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.199 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.199 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.199 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.199 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.199 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.199 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.199 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.200 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.200 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.200 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.200 INFO:teuthology.orchestra.run.smithi032.stdout: "organization": { 2024-04-30T08:44:53.200 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "organization", 2024-04-30T08:44:53.200 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.200 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.200 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.200 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.200 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.200 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.200 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.200 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.200 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.201 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.201 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.201 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.201 INFO:teuthology.orchestra.run.smithi032.stdout: "proxy": { 2024-04-30T08:44:53.201 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "proxy", 2024-04-30T08:44:53.201 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.201 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.201 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.201 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.201 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.201 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.201 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.201 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.201 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.202 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.202 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.202 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.202 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:53.202 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:53.202 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.202 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:53.202 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.202 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:53.202 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.202 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.202 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.202 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.203 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.203 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.203 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.203 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.203 INFO:teuthology.orchestra.run.smithi032.stdout: "url": { 2024-04-30T08:44:53.203 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "url", 2024-04-30T08:44:53.203 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.203 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.203 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.203 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-30T08:44:53.203 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.203 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.203 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.203 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.204 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.204 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.204 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.204 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.204 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.204 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.204 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:53.204 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "test_orchestrator", 2024-04-30T08:44:53.204 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:53.204 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:53.204 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:53.204 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:53.204 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:53.204 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.205 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.205 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.205 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.205 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.205 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.205 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.205 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.205 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.205 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.205 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.205 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.205 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.205 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.205 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.206 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.206 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.206 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.206 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.206 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:53.206 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:53.206 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.206 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.206 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.206 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.206 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.206 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.206 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.206 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.207 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.207 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.207 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.207 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.207 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:53.207 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:53.207 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.207 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.207 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.207 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:53.207 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.207 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.207 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.208 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.208 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.208 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.208 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.208 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.208 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.208 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.208 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.208 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.208 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.208 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.208 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.208 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:53.208 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:53.209 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.209 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.209 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.209 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.209 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.209 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.209 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.209 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.209 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.209 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.209 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.209 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.209 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:53.209 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:53.210 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.210 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:53.210 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.210 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:53.210 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.210 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.210 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.210 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.210 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.210 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.210 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.210 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.210 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.211 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.211 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:53.211 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "volumes", 2024-04-30T08:44:53.211 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:53.211 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:53.211 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:53.211 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:53.211 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:53.211 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.211 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.211 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.211 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.211 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.211 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.212 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.212 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.212 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.212 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.212 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.212 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.212 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.212 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.212 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.212 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.212 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.212 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.212 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.212 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:53.213 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:53.213 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.213 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.213 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.213 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.213 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.213 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.213 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.213 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.213 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.213 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.213 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.213 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.213 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:53.214 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:53.214 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.214 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.214 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.214 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:53.214 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.214 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.214 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.214 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.214 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.214 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.214 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.214 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.215 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.215 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.215 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.215 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.215 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.215 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.215 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.215 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:53.215 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:53.215 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.215 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.215 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.215 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.215 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.216 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.216 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.216 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.216 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.216 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.216 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.216 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.216 INFO:teuthology.orchestra.run.smithi032.stdout: "max_concurrent_clones": { 2024-04-30T08:44:53.216 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "max_concurrent_clones", 2024-04-30T08:44:53.216 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.216 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.216 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.216 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "4", 2024-04-30T08:44:53.216 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.217 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.217 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.217 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-30T08:44:53.217 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.217 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.217 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.217 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.217 INFO:teuthology.orchestra.run.smithi032.stdout: "periodic_async_work": { 2024-04-30T08:44:53.217 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "periodic_async_work", 2024-04-30T08:44:53.217 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.217 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.217 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.217 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.218 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.218 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.218 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.218 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Periodically check for async work", 2024-04-30T08:44:53.218 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.218 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.218 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.218 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.218 INFO:teuthology.orchestra.run.smithi032.stdout: "snapshot_clone_delay": { 2024-04-30T08:44:53.218 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "snapshot_clone_delay", 2024-04-30T08:44:53.218 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.218 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.218 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.219 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:53.219 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.219 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.219 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.219 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-30T08:44:53.219 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.219 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.219 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.219 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.219 INFO:teuthology.orchestra.run.smithi032.stdout: "snapshot_clone_no_wait": { 2024-04-30T08:44:53.219 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "snapshot_clone_no_wait", 2024-04-30T08:44:53.219 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.219 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.219 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.220 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "True", 2024-04-30T08:44:53.220 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.220 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.220 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.220 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-30T08:44:53.220 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.220 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.220 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.220 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.220 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:53.220 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:53.220 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.220 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:53.220 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.221 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:53.221 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.221 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.221 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.221 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.221 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.221 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.221 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.221 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.221 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.221 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.221 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:53.221 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "zabbix", 2024-04-30T08:44:53.221 INFO:teuthology.orchestra.run.smithi032.stdout: "can_run": true, 2024-04-30T08:44:53.222 INFO:teuthology.orchestra.run.smithi032.stdout: "error_string": "", 2024-04-30T08:44:53.222 INFO:teuthology.orchestra.run.smithi032.stdout: "module_options": { 2024-04-30T08:44:53.222 INFO:teuthology.orchestra.run.smithi032.stdout: "discovery_interval": { 2024-04-30T08:44:53.222 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "discovery_interval", 2024-04-30T08:44:53.222 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "uint", 2024-04-30T08:44:53.222 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.222 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.222 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "100", 2024-04-30T08:44:53.222 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.222 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.222 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.222 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.222 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.222 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.223 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.223 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.223 INFO:teuthology.orchestra.run.smithi032.stdout: "identifier": { 2024-04-30T08:44:53.223 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "identifier", 2024-04-30T08:44:53.223 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.223 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.223 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.223 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.223 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.223 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.223 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.223 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.223 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.223 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.224 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.224 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.224 INFO:teuthology.orchestra.run.smithi032.stdout: "interval": { 2024-04-30T08:44:53.224 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "interval", 2024-04-30T08:44:53.224 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "secs", 2024-04-30T08:44:53.224 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.224 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.224 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "60", 2024-04-30T08:44:53.224 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.224 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.224 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.224 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.224 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.224 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.224 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.225 INFO:teuthology.orchestra.run.smithi032.stdout: "log_level": { 2024-04-30T08:44:53.225 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_level", 2024-04-30T08:44:53.225 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.225 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.225 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.225 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.225 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.225 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.225 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.225 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.225 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.225 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.225 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.225 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.226 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.226 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.226 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.226 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.226 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.226 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.226 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster": { 2024-04-30T08:44:53.226 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster", 2024-04-30T08:44:53.226 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.226 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.226 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.226 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.227 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.227 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.227 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.227 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.227 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.227 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.227 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.227 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.227 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_cluster_level": { 2024-04-30T08:44:53.227 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_cluster_level", 2024-04-30T08:44:53.227 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.227 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.227 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.227 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "info", 2024-04-30T08:44:53.227 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.228 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.228 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [ 2024-04-30T08:44:53.228 INFO:teuthology.orchestra.run.smithi032.stdout: "", 2024-04-30T08:44:53.228 INFO:teuthology.orchestra.run.smithi032.stdout: "critical", 2024-04-30T08:44:53.228 INFO:teuthology.orchestra.run.smithi032.stdout: "debug", 2024-04-30T08:44:53.228 INFO:teuthology.orchestra.run.smithi032.stdout: "error", 2024-04-30T08:44:53.228 INFO:teuthology.orchestra.run.smithi032.stdout: "info", 2024-04-30T08:44:53.228 INFO:teuthology.orchestra.run.smithi032.stdout: "warning" 2024-04-30T08:44:53.228 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.228 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.228 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.228 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.228 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.228 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.229 INFO:teuthology.orchestra.run.smithi032.stdout: "log_to_file": { 2024-04-30T08:44:53.229 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "log_to_file", 2024-04-30T08:44:53.229 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "bool", 2024-04-30T08:44:53.229 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.229 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.229 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "False", 2024-04-30T08:44:53.229 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.229 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.229 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.229 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.229 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.229 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.229 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.229 INFO:teuthology.orchestra.run.smithi032.stdout: "sqlite3_killpoint": { 2024-04-30T08:44:53.230 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "sqlite3_killpoint", 2024-04-30T08:44:53.230 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.230 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "dev", 2024-04-30T08:44:53.230 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 1, 2024-04-30T08:44:53.230 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "0", 2024-04-30T08:44:53.230 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.230 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.230 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.230 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.230 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.230 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.230 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.231 INFO:teuthology.orchestra.run.smithi032.stdout: "zabbix_host": { 2024-04-30T08:44:53.231 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "zabbix_host", 2024-04-30T08:44:53.231 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.231 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.231 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.231 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "", 2024-04-30T08:44:53.231 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.231 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.231 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.231 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.231 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.231 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.231 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.231 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.232 INFO:teuthology.orchestra.run.smithi032.stdout: "zabbix_port": { 2024-04-30T08:44:53.232 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "zabbix_port", 2024-04-30T08:44:53.232 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "int", 2024-04-30T08:44:53.232 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.232 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.232 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "10051", 2024-04-30T08:44:53.232 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.232 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.232 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.232 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.232 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.232 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.232 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.233 INFO:teuthology.orchestra.run.smithi032.stdout: "zabbix_sender": { 2024-04-30T08:44:53.233 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "zabbix_sender", 2024-04-30T08:44:53.233 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "str", 2024-04-30T08:44:53.233 INFO:teuthology.orchestra.run.smithi032.stdout: "level": "advanced", 2024-04-30T08:44:53.233 INFO:teuthology.orchestra.run.smithi032.stdout: "flags": 0, 2024-04-30T08:44:53.233 INFO:teuthology.orchestra.run.smithi032.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-30T08:44:53.233 INFO:teuthology.orchestra.run.smithi032.stdout: "min": "", 2024-04-30T08:44:53.233 INFO:teuthology.orchestra.run.smithi032.stdout: "max": "", 2024-04-30T08:44:53.233 INFO:teuthology.orchestra.run.smithi032.stdout: "enum_allowed": [], 2024-04-30T08:44:53.233 INFO:teuthology.orchestra.run.smithi032.stdout: "desc": "", 2024-04-30T08:44:53.233 INFO:teuthology.orchestra.run.smithi032.stdout: "long_desc": "", 2024-04-30T08:44:53.233 INFO:teuthology.orchestra.run.smithi032.stdout: "tags": [], 2024-04-30T08:44:53.233 INFO:teuthology.orchestra.run.smithi032.stdout: "see_also": [] 2024-04-30T08:44:53.233 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.234 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.234 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.234 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.234 INFO:teuthology.orchestra.run.smithi032.stdout: "services": { 2024-04-30T08:44:53.234 INFO:teuthology.orchestra.run.smithi032.stdout: "dashboard": "https://172.21.15.32:8443/" 2024-04-30T08:44:53.234 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.234 INFO:teuthology.orchestra.run.smithi032.stdout: "always_on_modules": { 2024-04-30T08:44:53.234 INFO:teuthology.orchestra.run.smithi032.stdout: "octopus": [ 2024-04-30T08:44:53.234 INFO:teuthology.orchestra.run.smithi032.stdout: "balancer", 2024-04-30T08:44:53.234 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-30T08:44:53.234 INFO:teuthology.orchestra.run.smithi032.stdout: "devicehealth", 2024-04-30T08:44:53.234 INFO:teuthology.orchestra.run.smithi032.stdout: "orchestrator", 2024-04-30T08:44:53.234 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscaler", 2024-04-30T08:44:53.234 INFO:teuthology.orchestra.run.smithi032.stdout: "progress", 2024-04-30T08:44:53.235 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_support", 2024-04-30T08:44:53.235 INFO:teuthology.orchestra.run.smithi032.stdout: "status", 2024-04-30T08:44:53.235 INFO:teuthology.orchestra.run.smithi032.stdout: "telemetry", 2024-04-30T08:44:53.235 INFO:teuthology.orchestra.run.smithi032.stdout: "volumes" 2024-04-30T08:44:53.235 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.235 INFO:teuthology.orchestra.run.smithi032.stdout: "pacific": [ 2024-04-30T08:44:53.235 INFO:teuthology.orchestra.run.smithi032.stdout: "balancer", 2024-04-30T08:44:53.235 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-30T08:44:53.235 INFO:teuthology.orchestra.run.smithi032.stdout: "devicehealth", 2024-04-30T08:44:53.235 INFO:teuthology.orchestra.run.smithi032.stdout: "orchestrator", 2024-04-30T08:44:53.235 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscaler", 2024-04-30T08:44:53.235 INFO:teuthology.orchestra.run.smithi032.stdout: "progress", 2024-04-30T08:44:53.235 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_support", 2024-04-30T08:44:53.235 INFO:teuthology.orchestra.run.smithi032.stdout: "status", 2024-04-30T08:44:53.236 INFO:teuthology.orchestra.run.smithi032.stdout: "telemetry", 2024-04-30T08:44:53.236 INFO:teuthology.orchestra.run.smithi032.stdout: "volumes" 2024-04-30T08:44:53.236 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.236 INFO:teuthology.orchestra.run.smithi032.stdout: "quincy": [ 2024-04-30T08:44:53.236 INFO:teuthology.orchestra.run.smithi032.stdout: "balancer", 2024-04-30T08:44:53.236 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-30T08:44:53.236 INFO:teuthology.orchestra.run.smithi032.stdout: "devicehealth", 2024-04-30T08:44:53.236 INFO:teuthology.orchestra.run.smithi032.stdout: "orchestrator", 2024-04-30T08:44:53.236 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscaler", 2024-04-30T08:44:53.236 INFO:teuthology.orchestra.run.smithi032.stdout: "progress", 2024-04-30T08:44:53.236 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_support", 2024-04-30T08:44:53.236 INFO:teuthology.orchestra.run.smithi032.stdout: "status", 2024-04-30T08:44:53.236 INFO:teuthology.orchestra.run.smithi032.stdout: "telemetry", 2024-04-30T08:44:53.236 INFO:teuthology.orchestra.run.smithi032.stdout: "volumes" 2024-04-30T08:44:53.237 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.237 INFO:teuthology.orchestra.run.smithi032.stdout: "reef": [ 2024-04-30T08:44:53.237 INFO:teuthology.orchestra.run.smithi032.stdout: "balancer", 2024-04-30T08:44:53.237 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-30T08:44:53.237 INFO:teuthology.orchestra.run.smithi032.stdout: "devicehealth", 2024-04-30T08:44:53.237 INFO:teuthology.orchestra.run.smithi032.stdout: "orchestrator", 2024-04-30T08:44:53.237 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscaler", 2024-04-30T08:44:53.237 INFO:teuthology.orchestra.run.smithi032.stdout: "progress", 2024-04-30T08:44:53.237 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_support", 2024-04-30T08:44:53.237 INFO:teuthology.orchestra.run.smithi032.stdout: "status", 2024-04-30T08:44:53.237 INFO:teuthology.orchestra.run.smithi032.stdout: "telemetry", 2024-04-30T08:44:53.237 INFO:teuthology.orchestra.run.smithi032.stdout: "volumes" 2024-04-30T08:44:53.237 INFO:teuthology.orchestra.run.smithi032.stdout: ], 2024-04-30T08:44:53.237 INFO:teuthology.orchestra.run.smithi032.stdout: "squid": [ 2024-04-30T08:44:53.237 INFO:teuthology.orchestra.run.smithi032.stdout: "balancer", 2024-04-30T08:44:53.238 INFO:teuthology.orchestra.run.smithi032.stdout: "crash", 2024-04-30T08:44:53.238 INFO:teuthology.orchestra.run.smithi032.stdout: "devicehealth", 2024-04-30T08:44:53.238 INFO:teuthology.orchestra.run.smithi032.stdout: "orchestrator", 2024-04-30T08:44:53.238 INFO:teuthology.orchestra.run.smithi032.stdout: "pg_autoscaler", 2024-04-30T08:44:53.238 INFO:teuthology.orchestra.run.smithi032.stdout: "progress", 2024-04-30T08:44:53.238 INFO:teuthology.orchestra.run.smithi032.stdout: "rbd_support", 2024-04-30T08:44:53.238 INFO:teuthology.orchestra.run.smithi032.stdout: "status", 2024-04-30T08:44:53.238 INFO:teuthology.orchestra.run.smithi032.stdout: "telemetry", 2024-04-30T08:44:53.238 INFO:teuthology.orchestra.run.smithi032.stdout: "volumes" 2024-04-30T08:44:53.238 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-30T08:44:53.238 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.238 INFO:teuthology.orchestra.run.smithi032.stdout: "last_failure_osd_epoch": 3, 2024-04-30T08:44:53.238 INFO:teuthology.orchestra.run.smithi032.stdout: "active_clients": [ 2024-04-30T08:44:53.238 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:53.239 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "devicehealth", 2024-04-30T08:44:53.239 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-30T08:44:53.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:53.239 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-30T08:44:53.239 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2024-04-30T08:44:53.239 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 1718012592 2024-04-30T08:44:53.239 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.239 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-30T08:44:53.239 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:53.239 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "libcephsqlite", 2024-04-30T08:44:53.239 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-30T08:44:53.239 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:53.239 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-30T08:44:53.239 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2024-04-30T08:44:53.240 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 2970162436 2024-04-30T08:44:53.240 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.240 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-30T08:44:53.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.240 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:53.240 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "rbd_support", 2024-04-30T08:44:53.240 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-30T08:44:53.240 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:53.240 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-30T08:44:53.240 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2024-04-30T08:44:53.240 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 3418667628 2024-04-30T08:44:53.240 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.240 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-30T08:44:53.240 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:44:53.240 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:53.240 INFO:teuthology.orchestra.run.smithi032.stdout: "name": "volumes", 2024-04-30T08:44:53.241 INFO:teuthology.orchestra.run.smithi032.stdout: "addrvec": [ 2024-04-30T08:44:53.241 INFO:teuthology.orchestra.run.smithi032.stdout: { 2024-04-30T08:44:53.241 INFO:teuthology.orchestra.run.smithi032.stdout: "type": "v2", 2024-04-30T08:44:53.241 INFO:teuthology.orchestra.run.smithi032.stdout: "addr": "172.21.15.32:0", 2024-04-30T08:44:53.241 INFO:teuthology.orchestra.run.smithi032.stdout: "nonce": 4288389191 2024-04-30T08:44:53.241 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.241 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-30T08:44:53.241 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:44:53.241 INFO:teuthology.orchestra.run.smithi032.stdout: ] 2024-04-30T08:44:53.241 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-30T08:44:53.283 DEBUG:tasks.check_counter:Getting stats from a 2024-04-30T08:44:53.284 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-30T08:44:53.284 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:52 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2063780753' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-30T08:44:53.284 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:52 smithi032 ceph-mon[28331]: pgmap v716: 129 pgs: 129 active+clean; 8.0 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 3.0 MiB/s wr, 694 op/s 2024-04-30T08:44:53.942 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:44:54.444 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:54 smithi032 ceph-mon[28331]: pgmap v717: 129 pgs: 129 active+clean; 6.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 192 KiB/s wr, 165 op/s 2024-04-30T08:44:54.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:54 smithi071 ceph-mon[33385]: pgmap v717: 129 pgs: 129 active+clean; 6.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 192 KiB/s wr, 165 op/s 2024-04-30T08:44:54.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:54 smithi189 ceph-mon[33038]: pgmap v717: 129 pgs: 129 active+clean; 6.9 GiB data, 18 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 192 KiB/s wr, 165 op/s 2024-04-30T08:44:56.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:56 smithi032 ceph-mon[28331]: pgmap v718: 129 pgs: 129 active+clean; 4.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 24 KiB/s wr, 195 op/s 2024-04-30T08:44:56.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:56 smithi071 ceph-mon[33385]: pgmap v718: 129 pgs: 129 active+clean; 4.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 24 KiB/s wr, 195 op/s 2024-04-30T08:44:56.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:56 smithi189 ceph-mon[33038]: pgmap v718: 129 pgs: 129 active+clean; 4.9 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 24 KiB/s wr, 195 op/s 2024-04-30T08:44:58.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:44:58 smithi032 ceph-mon[28331]: pgmap v719: 129 pgs: 129 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 16 KiB/s wr, 213 op/s 2024-04-30T08:44:58.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:44:58 smithi071 ceph-mon[33385]: pgmap v719: 129 pgs: 129 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 16 KiB/s wr, 213 op/s 2024-04-30T08:44:58.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:44:58 smithi189 ceph-mon[33038]: pgmap v719: 129 pgs: 129 active+clean; 3.0 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 16 KiB/s wr, 213 op/s 2024-04-30T08:45:00.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:00 smithi032 ceph-mon[28331]: pgmap v720: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 15 KiB/s wr, 217 op/s 2024-04-30T08:45:00.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:00 smithi071 ceph-mon[33385]: pgmap v720: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 15 KiB/s wr, 217 op/s 2024-04-30T08:45:00.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:00 smithi189 ceph-mon[33038]: pgmap v720: 129 pgs: 129 active+clean; 1.9 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 15 KiB/s wr, 217 op/s 2024-04-30T08:45:02.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:02 smithi032 ceph-mon[28331]: pgmap v721: 129 pgs: 129 active+clean; 371 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 3.4 KiB/s wr, 244 op/s 2024-04-30T08:45:02.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:02 smithi071 ceph-mon[33385]: pgmap v721: 129 pgs: 129 active+clean; 371 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 3.4 KiB/s wr, 244 op/s 2024-04-30T08:45:02.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:02 smithi189 ceph-mon[33038]: pgmap v721: 129 pgs: 129 active+clean; 371 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 3.4 KiB/s wr, 244 op/s 2024-04-30T08:45:04.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:04 smithi032 ceph-mon[28331]: pgmap v722: 129 pgs: 129 active+clean; 31 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.6 KiB/s wr, 218 op/s 2024-04-30T08:45:04.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:04 smithi071 ceph-mon[33385]: pgmap v722: 129 pgs: 129 active+clean; 31 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.6 KiB/s wr, 218 op/s 2024-04-30T08:45:04.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:04 smithi189 ceph-mon[33038]: pgmap v722: 129 pgs: 129 active+clean; 31 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 2.6 KiB/s wr, 218 op/s 2024-04-30T08:45:06.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:06 smithi032 ceph-mon[28331]: pgmap v723: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 1.3 KiB/s wr, 203 op/s 2024-04-30T08:45:06.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:06 smithi071 ceph-mon[33385]: pgmap v723: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 1.3 KiB/s wr, 203 op/s 2024-04-30T08:45:06.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:06 smithi189 ceph-mon[33038]: pgmap v723: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 KiB/s rd, 1.3 KiB/s wr, 203 op/s 2024-04-30T08:45:07.408 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-30T08:45:07.409 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-0": { 2024-04-30T08:45:07.409 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 0, 2024-04-30T08:45:07.409 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 2, 2024-04-30T08:45:07.409 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 0, 2024-04-30T08:45:07.409 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 0, 2024-04-30T08:45:07.409 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 4, 2024-04-30T08:45:07.409 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 0, 2024-04-30T08:45:07.409 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.006776159, 2024-04-30T08:45:07.409 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.000000000, 2024-04-30T08:45:07.409 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.001351101, 2024-04-30T08:45:07.409 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-30T08:45:07.409 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:07.410 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:07.410 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:07.410 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:07.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:07.410 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:07.410 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:07.410 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:07.410 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:07.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:07.410 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-30T08:45:07.410 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 0 2024-04-30T08:45:07.410 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:07.410 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-1": { 2024-04-30T08:45:07.411 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 1, 2024-04-30T08:45:07.411 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 164, 2024-04-30T08:45:07.411 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 160, 2024-04-30T08:45:07.411 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 107424, 2024-04-30T08:45:07.411 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 3, 2024-04-30T08:45:07.411 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 2, 2024-04-30T08:45:07.411 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.030403210, 2024-04-30T08:45:07.411 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.024541373, 2024-04-30T08:45:07.411 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.003271882, 2024-04-30T08:45:07.411 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-30T08:45:07.411 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:07.411 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 162, 2024-04-30T08:45:07.411 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.025030311, 2024-04-30T08:45:07.411 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000154508 2024-04-30T08:45:07.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:07.412 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:07.412 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:07.412 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:07.412 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:07.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:07.412 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-30T08:45:07.412 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 107424 2024-04-30T08:45:07.412 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:07.412 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-2": { 2024-04-30T08:45:07.412 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 225, 2024-04-30T08:45:07.412 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 218, 2024-04-30T08:45:07.412 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 805584, 2024-04-30T08:45:07.413 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 133472, 2024-04-30T08:45:07.413 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 2, 2024-04-30T08:45:07.413 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 1, 2024-04-30T08:45:07.413 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.093314837, 2024-04-30T08:45:07.413 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.038085698, 2024-04-30T08:45:07.413 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.054220426, 2024-04-30T08:45:07.413 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.012300377, 2024-04-30T08:45:07.413 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:07.413 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 218, 2024-04-30T08:45:07.413 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.053409299, 2024-04-30T08:45:07.413 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000244996 2024-04-30T08:45:07.413 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:07.413 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:07.414 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:07.414 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:07.414 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:07.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:07.414 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 805584, 2024-04-30T08:45:07.414 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 133472 2024-04-30T08:45:07.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:07.414 INFO:teuthology.orchestra.run.smithi032.stdout: "cct": { 2024-04-30T08:45:07.414 INFO:teuthology.orchestra.run.smithi032.stdout: "total_workers": 0, 2024-04-30T08:45:07.414 INFO:teuthology.orchestra.run.smithi032.stdout: "unhealthy_workers": 0 2024-04-30T08:45:07.414 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:07.414 INFO:teuthology.orchestra.run.smithi032.stdout: "mempool": { 2024-04-30T08:45:07.415 INFO:teuthology.orchestra.run.smithi032.stdout: "bloom_filter_bytes": 0, 2024-04-30T08:45:07.415 INFO:teuthology.orchestra.run.smithi032.stdout: "bloom_filter_items": 0, 2024-04-30T08:45:07.415 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_alloc_bytes": 0, 2024-04-30T08:45:07.415 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_alloc_items": 0, 2024-04-30T08:45:07.415 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_data_bytes": 0, 2024-04-30T08:45:07.415 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_data_items": 0, 2024-04-30T08:45:07.415 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-30T08:45:07.415 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_onode_items": 0, 2024-04-30T08:45:07.415 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-30T08:45:07.415 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_meta_items": 0, 2024-04-30T08:45:07.415 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_other_bytes": 0, 2024-04-30T08:45:07.415 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_other_items": 0, 2024-04-30T08:45:07.416 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-30T08:45:07.416 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_buffer_items": 0, 2024-04-30T08:45:07.416 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_extent_bytes": 0, 2024-04-30T08:45:07.416 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_extent_items": 0, 2024-04-30T08:45:07.416 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_blob_bytes": 0, 2024-04-30T08:45:07.416 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_blob_items": 0, 2024-04-30T08:45:07.416 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-30T08:45:07.416 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_shared_blob_items": 0, 2024-04-30T08:45:07.416 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-30T08:45:07.416 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_inline_bl_items": 0, 2024-04-30T08:45:07.416 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_bytes": 0, 2024-04-30T08:45:07.416 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_items": 0, 2024-04-30T08:45:07.416 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_txc_bytes": 0, 2024-04-30T08:45:07.417 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_txc_items": 0, 2024-04-30T08:45:07.417 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-30T08:45:07.417 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_deferred_items": 0, 2024-04-30T08:45:07.417 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_bytes": 0, 2024-04-30T08:45:07.417 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_items": 0, 2024-04-30T08:45:07.417 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_bytes": 0, 2024-04-30T08:45:07.417 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_items": 0, 2024-04-30T08:45:07.417 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_reader_bytes": 0, 2024-04-30T08:45:07.417 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_reader_items": 0, 2024-04-30T08:45:07.417 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_writer_bytes": 0, 2024-04-30T08:45:07.417 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_writer_items": 0, 2024-04-30T08:45:07.417 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_anon_bytes": 149998, 2024-04-30T08:45:07.417 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_anon_items": 30, 2024-04-30T08:45:07.418 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_meta_bytes": 0, 2024-04-30T08:45:07.418 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_meta_items": 0, 2024-04-30T08:45:07.418 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_bytes": 0, 2024-04-30T08:45:07.418 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_items": 0, 2024-04-30T08:45:07.418 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mapbl_bytes": 0, 2024-04-30T08:45:07.418 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mapbl_items": 0, 2024-04-30T08:45:07.418 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pglog_bytes": 0, 2024-04-30T08:45:07.418 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pglog_items": 0, 2024-04-30T08:45:07.418 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_bytes": 0, 2024-04-30T08:45:07.418 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_items": 0, 2024-04-30T08:45:07.418 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_mapping_bytes": 0, 2024-04-30T08:45:07.418 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_mapping_items": 0, 2024-04-30T08:45:07.418 INFO:teuthology.orchestra.run.smithi032.stdout: "pgmap_bytes": 0, 2024-04-30T08:45:07.419 INFO:teuthology.orchestra.run.smithi032.stdout: "pgmap_items": 0, 2024-04-30T08:45:07.419 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_co_bytes": 928, 2024-04-30T08:45:07.419 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_co_items": 2, 2024-04-30T08:45:07.419 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_1_bytes": 0, 2024-04-30T08:45:07.419 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_1_items": 0, 2024-04-30T08:45:07.419 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_2_bytes": 0, 2024-04-30T08:45:07.419 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_2_items": 0 2024-04-30T08:45:07.419 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:07.419 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-30T08:45:07.419 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-30T08:45:07.419 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 104857600, 2024-04-30T08:45:07.419 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-30T08:45:07.419 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 222, 2024-04-30T08:45:07.419 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 771384, 2024-04-30T08:45:07.420 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:07.420 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 222, 2024-04-30T08:45:07.420 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 0, 2024-04-30T08:45:07.420 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 0, 2024-04-30T08:45:07.420 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 222, 2024-04-30T08:45:07.420 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 771384, 2024-04-30T08:45:07.420 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-30T08:45:07.420 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:07.420 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:07.420 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:07.420 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:45:07.420 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:45:07.420 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-30T08:45:08.191 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:08 smithi032 ceph-mon[28331]: pgmap v724: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1023 B/s wr, 157 op/s 2024-04-30T08:45:08.359 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-04-30T08:45:08.360 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-04-30T08:45:08.360 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.a 2024-04-30T08:45:08.360 DEBUG:tasks.check_counter:Getting stats from d 2024-04-30T08:45:08.360 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-30T08:45:08.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:08 smithi071 ceph-mon[33385]: pgmap v724: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1023 B/s wr, 157 op/s 2024-04-30T08:45:08.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:08 smithi189 ceph-mon[33038]: pgmap v724: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1023 B/s wr, 157 op/s 2024-04-30T08:45:08.819 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:45:10.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:10 smithi032 ceph-mon[28331]: pgmap v725: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1023 B/s wr, 117 op/s 2024-04-30T08:45:10.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:10 smithi071 ceph-mon[33385]: pgmap v725: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1023 B/s wr, 117 op/s 2024-04-30T08:45:10.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:10 smithi189 ceph-mon[33038]: pgmap v725: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1023 B/s wr, 117 op/s 2024-04-30T08:45:12.336 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:12 smithi032 ceph-mon[28331]: pgmap v726: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 852 B/s wr, 91 op/s 2024-04-30T08:45:12.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:12 smithi071 ceph-mon[33385]: pgmap v726: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 852 B/s wr, 91 op/s 2024-04-30T08:45:12.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:12 smithi189 ceph-mon[33038]: pgmap v726: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 852 B/s wr, 91 op/s 2024-04-30T08:45:13.147 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-30T08:45:13.147 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-0": { 2024-04-30T08:45:13.147 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 346, 2024-04-30T08:45:13.147 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 234, 2024-04-30T08:45:13.147 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 1533586, 2024-04-30T08:45:13.147 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 139636, 2024-04-30T08:45:13.147 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 5, 2024-04-30T08:45:13.147 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 1, 2024-04-30T08:45:13.147 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.089782977, 2024-04-30T08:45:13.147 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.020333874, 2024-04-30T08:45:13.148 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.081470848, 2024-04-30T08:45:13.148 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.009220105, 2024-04-30T08:45:13.148 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:13.148 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 234, 2024-04-30T08:45:13.148 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.045630649, 2024-04-30T08:45:13.148 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000195002 2024-04-30T08:45:13.148 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.148 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:13.148 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.148 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.148 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.148 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.148 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 1532752, 2024-04-30T08:45:13.149 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 139312 2024-04-30T08:45:13.149 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.149 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-1": { 2024-04-30T08:45:13.149 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 629, 2024-04-30T08:45:13.149 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 798, 2024-04-30T08:45:13.149 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 154670, 2024-04-30T08:45:13.149 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 563101, 2024-04-30T08:45:13.149 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 5, 2024-04-30T08:45:13.149 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 4, 2024-04-30T08:45:13.149 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.170917010, 2024-04-30T08:45:13.149 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.080583373, 2024-04-30T08:45:13.149 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.028668714, 2024-04-30T08:45:13.150 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.046598954, 2024-04-30T08:45:13.150 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:13.150 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 796, 2024-04-30T08:45:13.150 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.110859885, 2024-04-30T08:45:13.150 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000139271 2024-04-30T08:45:13.150 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.150 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:13.150 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.150 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.150 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.150 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.150 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-30T08:45:13.151 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 363392 2024-04-30T08:45:13.151 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.151 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-2": { 2024-04-30T08:45:13.151 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 1443, 2024-04-30T08:45:13.151 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 1445, 2024-04-30T08:45:13.151 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 456422, 2024-04-30T08:45:13.151 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 458932, 2024-04-30T08:45:13.151 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 4, 2024-04-30T08:45:13.151 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 3, 2024-04-30T08:45:13.151 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.295103591, 2024-04-30T08:45:13.151 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.108983581, 2024-04-30T08:45:13.151 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.064014948, 2024-04-30T08:45:13.151 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.090515860, 2024-04-30T08:45:13.152 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:13.152 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 1443, 2024-04-30T08:45:13.152 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.226186601, 2024-04-30T08:45:13.152 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000156747 2024-04-30T08:45:13.152 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.152 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:13.152 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.152 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.152 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.152 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.152 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-30T08:45:13.152 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 0 2024-04-30T08:45:13.152 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.153 INFO:teuthology.orchestra.run.smithi032.stdout: "cct": { 2024-04-30T08:45:13.153 INFO:teuthology.orchestra.run.smithi032.stdout: "total_workers": 1, 2024-04-30T08:45:13.153 INFO:teuthology.orchestra.run.smithi032.stdout: "unhealthy_workers": 0 2024-04-30T08:45:13.153 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.153 INFO:teuthology.orchestra.run.smithi032.stdout: "finisher-MDSRank": { 2024-04-30T08:45:13.153 INFO:teuthology.orchestra.run.smithi032.stdout: "queue_len": 0, 2024-04-30T08:45:13.153 INFO:teuthology.orchestra.run.smithi032.stdout: "complete_latency": { 2024-04-30T08:45:13.153 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 3463, 2024-04-30T08:45:13.153 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.179633089, 2024-04-30T08:45:13.153 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000051872 2024-04-30T08:45:13.154 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:45:13.154 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.154 INFO:teuthology.orchestra.run.smithi032.stdout: "finisher-PurgeQueue": { 2024-04-30T08:45:13.154 INFO:teuthology.orchestra.run.smithi032.stdout: "queue_len": 0, 2024-04-30T08:45:13.154 INFO:teuthology.orchestra.run.smithi032.stdout: "complete_latency": { 2024-04-30T08:45:13.154 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.154 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.154 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.154 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:45:13.154 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.154 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-30T08:45:13.154 INFO:teuthology.orchestra.run.smithi032.stdout: "request": 0, 2024-04-30T08:45:13.154 INFO:teuthology.orchestra.run.smithi032.stdout: "reply": 0, 2024-04-30T08:45:13.155 INFO:teuthology.orchestra.run.smithi032.stdout: "reply_latency": { 2024-04-30T08:45:13.155 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.155 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.155 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.155 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.155 INFO:teuthology.orchestra.run.smithi032.stdout: "slow_reply": 0, 2024-04-30T08:45:13.155 INFO:teuthology.orchestra.run.smithi032.stdout: "forward": 0, 2024-04-30T08:45:13.155 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_fetch_complete": 0, 2024-04-30T08:45:13.155 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_fetch_keys": 0, 2024-04-30T08:45:13.155 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_commit": 0, 2024-04-30T08:45:13.155 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_split": 0, 2024-04-30T08:45:13.155 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_merge": 0, 2024-04-30T08:45:13.155 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes": 2, 2024-04-30T08:45:13.156 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_top": 0, 2024-04-30T08:45:13.156 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_bottom": 2, 2024-04-30T08:45:13.156 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_pin_tail": 0, 2024-04-30T08:45:13.156 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_pinned": 2, 2024-04-30T08:45:13.156 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_expired": 0, 2024-04-30T08:45:13.156 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_with_caps": 0, 2024-04-30T08:45:13.156 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": 0, 2024-04-30T08:45:13.156 INFO:teuthology.orchestra.run.smithi032.stdout: "subtrees": 1, 2024-04-30T08:45:13.156 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse": 0, 2024-04-30T08:45:13.156 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_hit": 0, 2024-04-30T08:45:13.156 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_forward": 0, 2024-04-30T08:45:13.156 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_discover": 0, 2024-04-30T08:45:13.156 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_dir_fetch": 0, 2024-04-30T08:45:13.156 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_remote_ino": 0, 2024-04-30T08:45:13.157 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_lock": 0, 2024-04-30T08:45:13.157 INFO:teuthology.orchestra.run.smithi032.stdout: "load_cent": 0, 2024-04-30T08:45:13.157 INFO:teuthology.orchestra.run.smithi032.stdout: "q": 0, 2024-04-30T08:45:13.157 INFO:teuthology.orchestra.run.smithi032.stdout: "exported": 0, 2024-04-30T08:45:13.157 INFO:teuthology.orchestra.run.smithi032.stdout: "exported_inodes": 0, 2024-04-30T08:45:13.157 INFO:teuthology.orchestra.run.smithi032.stdout: "imported": 0, 2024-04-30T08:45:13.157 INFO:teuthology.orchestra.run.smithi032.stdout: "imported_inodes": 0, 2024-04-30T08:45:13.157 INFO:teuthology.orchestra.run.smithi032.stdout: "openino_dir_fetch": 0, 2024-04-30T08:45:13.157 INFO:teuthology.orchestra.run.smithi032.stdout: "openino_backtrace_fetch": 0, 2024-04-30T08:45:13.157 INFO:teuthology.orchestra.run.smithi032.stdout: "openino_peer_discover": 0, 2024-04-30T08:45:13.157 INFO:teuthology.orchestra.run.smithi032.stdout: "root_rfiles": 0, 2024-04-30T08:45:13.157 INFO:teuthology.orchestra.run.smithi032.stdout: "root_rbytes": 0, 2024-04-30T08:45:13.157 INFO:teuthology.orchestra.run.smithi032.stdout: "root_rsnaps": 0, 2024-04-30T08:45:13.157 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_backtrace_fetch": 0, 2024-04-30T08:45:13.158 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_set_tag": 0, 2024-04-30T08:45:13.158 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_backtrace_repaired": 0, 2024-04-30T08:45:13.158 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_inotable_repaired": 0, 2024-04-30T08:45:13.158 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_dir_inodes": 0, 2024-04-30T08:45:13.158 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_dir_base_inodes": 0, 2024-04-30T08:45:13.158 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_dirfrag_rstats": 0, 2024-04-30T08:45:13.158 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_file_inodes": 0, 2024-04-30T08:45:13.158 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_inode_file_caps": 0, 2024-04-30T08:45:13.158 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_cap_op_revoke": 0, 2024-04-30T08:45:13.158 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_cap_op_grant": 0, 2024-04-30T08:45:13.158 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_cap_op_trunc": 0, 2024-04-30T08:45:13.158 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-30T08:45:13.159 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-30T08:45:13.159 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_client_caps": 0, 2024-04-30T08:45:13.159 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_client_caps_dirty": 0, 2024-04-30T08:45:13.159 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_client_cap_release": 0, 2024-04-30T08:45:13.159 INFO:teuthology.orchestra.run.smithi032.stdout: "process_request_cap_release": 0 2024-04-30T08:45:13.159 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.159 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_cache": { 2024-04-30T08:45:13.159 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_update": 0, 2024-04-30T08:45:13.159 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_update_receipt": 0, 2024-04-30T08:45:13.159 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_try_discover": 0, 2024-04-30T08:45:13.159 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_send_discover": 0, 2024-04-30T08:45:13.159 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_handle_discover": 0, 2024-04-30T08:45:13.159 INFO:teuthology.orchestra.run.smithi032.stdout: "num_strays": 0, 2024-04-30T08:45:13.159 INFO:teuthology.orchestra.run.smithi032.stdout: "num_strays_delayed": 0, 2024-04-30T08:45:13.160 INFO:teuthology.orchestra.run.smithi032.stdout: "num_strays_enqueuing": 0, 2024-04-30T08:45:13.160 INFO:teuthology.orchestra.run.smithi032.stdout: "strays_created": 0, 2024-04-30T08:45:13.160 INFO:teuthology.orchestra.run.smithi032.stdout: "strays_enqueued": 0, 2024-04-30T08:45:13.160 INFO:teuthology.orchestra.run.smithi032.stdout: "strays_reintegrated": 0, 2024-04-30T08:45:13.160 INFO:teuthology.orchestra.run.smithi032.stdout: "strays_migrated": 0, 2024-04-30T08:45:13.160 INFO:teuthology.orchestra.run.smithi032.stdout: "num_recovering_processing": 0, 2024-04-30T08:45:13.160 INFO:teuthology.orchestra.run.smithi032.stdout: "num_recovering_enqueued": 0, 2024-04-30T08:45:13.160 INFO:teuthology.orchestra.run.smithi032.stdout: "num_recovering_prioritized": 0, 2024-04-30T08:45:13.160 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_started": 0, 2024-04-30T08:45:13.160 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_completed": 0, 2024-04-30T08:45:13.160 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_quiesce_path": 0, 2024-04-30T08:45:13.160 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_quiesce_inode": 0, 2024-04-30T08:45:13.160 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_enqueue_scrub": 0, 2024-04-30T08:45:13.161 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_exportdir": 0, 2024-04-30T08:45:13.161 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_flush": 0, 2024-04-30T08:45:13.161 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_fragmentdir": 0, 2024-04-30T08:45:13.161 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_fragstats": 0, 2024-04-30T08:45:13.161 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_inodestats": 0 2024-04-30T08:45:13.161 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.161 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_log": { 2024-04-30T08:45:13.161 INFO:teuthology.orchestra.run.smithi032.stdout: "evlrg": 0, 2024-04-30T08:45:13.161 INFO:teuthology.orchestra.run.smithi032.stdout: "evadd": 0, 2024-04-30T08:45:13.161 INFO:teuthology.orchestra.run.smithi032.stdout: "evex": 0, 2024-04-30T08:45:13.161 INFO:teuthology.orchestra.run.smithi032.stdout: "evtrm": 0, 2024-04-30T08:45:13.161 INFO:teuthology.orchestra.run.smithi032.stdout: "ev": 19, 2024-04-30T08:45:13.161 INFO:teuthology.orchestra.run.smithi032.stdout: "evexg": 0, 2024-04-30T08:45:13.161 INFO:teuthology.orchestra.run.smithi032.stdout: "evexd": 0, 2024-04-30T08:45:13.162 INFO:teuthology.orchestra.run.smithi032.stdout: "segadd": 0, 2024-04-30T08:45:13.162 INFO:teuthology.orchestra.run.smithi032.stdout: "segex": 0, 2024-04-30T08:45:13.162 INFO:teuthology.orchestra.run.smithi032.stdout: "segtrm": 0, 2024-04-30T08:45:13.162 INFO:teuthology.orchestra.run.smithi032.stdout: "seg": 2, 2024-04-30T08:45:13.162 INFO:teuthology.orchestra.run.smithi032.stdout: "segmjr": 2, 2024-04-30T08:45:13.162 INFO:teuthology.orchestra.run.smithi032.stdout: "segexg": 0, 2024-04-30T08:45:13.162 INFO:teuthology.orchestra.run.smithi032.stdout: "segexd": 0, 2024-04-30T08:45:13.162 INFO:teuthology.orchestra.run.smithi032.stdout: "expos": 4194304, 2024-04-30T08:45:13.162 INFO:teuthology.orchestra.run.smithi032.stdout: "wrpos": 4219859, 2024-04-30T08:45:13.162 INFO:teuthology.orchestra.run.smithi032.stdout: "rdpos": 4217575, 2024-04-30T08:45:13.162 INFO:teuthology.orchestra.run.smithi032.stdout: "jlat": { 2024-04-30T08:45:13.162 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.162 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.163 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.163 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.163 INFO:teuthology.orchestra.run.smithi032.stdout: "replayed": 19 2024-04-30T08:45:13.163 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.163 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_mem": { 2024-04-30T08:45:13.163 INFO:teuthology.orchestra.run.smithi032.stdout: "ino": 4, 2024-04-30T08:45:13.163 INFO:teuthology.orchestra.run.smithi032.stdout: "ino+": 17, 2024-04-30T08:45:13.163 INFO:teuthology.orchestra.run.smithi032.stdout: "ino-": 13, 2024-04-30T08:45:13.163 INFO:teuthology.orchestra.run.smithi032.stdout: "dir": 2, 2024-04-30T08:45:13.163 INFO:teuthology.orchestra.run.smithi032.stdout: "dir+": 15, 2024-04-30T08:45:13.163 INFO:teuthology.orchestra.run.smithi032.stdout: "dir-": 13, 2024-04-30T08:45:13.163 INFO:teuthology.orchestra.run.smithi032.stdout: "dn": 2, 2024-04-30T08:45:13.163 INFO:teuthology.orchestra.run.smithi032.stdout: "dn+": 16, 2024-04-30T08:45:13.163 INFO:teuthology.orchestra.run.smithi032.stdout: "dn-": 14, 2024-04-30T08:45:13.164 INFO:teuthology.orchestra.run.smithi032.stdout: "cap": 0, 2024-04-30T08:45:13.164 INFO:teuthology.orchestra.run.smithi032.stdout: "cap+": 0, 2024-04-30T08:45:13.164 INFO:teuthology.orchestra.run.smithi032.stdout: "cap-": 0, 2024-04-30T08:45:13.164 INFO:teuthology.orchestra.run.smithi032.stdout: "rss": 118396, 2024-04-30T08:45:13.164 INFO:teuthology.orchestra.run.smithi032.stdout: "heap": 215440 2024-04-30T08:45:13.164 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.164 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_server": { 2024-04-30T08:45:13.164 INFO:teuthology.orchestra.run.smithi032.stdout: "dispatch_client_request": 0, 2024-04-30T08:45:13.164 INFO:teuthology.orchestra.run.smithi032.stdout: "dispatch_server_request": 0, 2024-04-30T08:45:13.164 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_client_request": 0, 2024-04-30T08:45:13.164 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_client_session": 0, 2024-04-30T08:45:13.164 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_peer_request": 0, 2024-04-30T08:45:13.164 INFO:teuthology.orchestra.run.smithi032.stdout: "req_create_latency": { 2024-04-30T08:45:13.165 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.165 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.165 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.165 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.165 INFO:teuthology.orchestra.run.smithi032.stdout: "req_getattr_latency": { 2024-04-30T08:45:13.165 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.165 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.165 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.165 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.165 INFO:teuthology.orchestra.run.smithi032.stdout: "req_getfilelock_latency": { 2024-04-30T08:45:13.165 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.165 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.165 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.166 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.166 INFO:teuthology.orchestra.run.smithi032.stdout: "req_link_latency": { 2024-04-30T08:45:13.166 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.166 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.166 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.166 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.166 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookup_latency": { 2024-04-30T08:45:13.166 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.166 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.166 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.166 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.166 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookuphash_latency": { 2024-04-30T08:45:13.166 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.167 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.167 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.167 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.167 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookupino_latency": { 2024-04-30T08:45:13.167 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.167 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.167 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.167 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.167 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookupname_latency": { 2024-04-30T08:45:13.167 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.167 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.167 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.167 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.168 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookupparent_latency": { 2024-04-30T08:45:13.168 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.168 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.168 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.168 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.168 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookupsnap_latency": { 2024-04-30T08:45:13.168 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.168 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.168 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.168 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.168 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lssnap_latency": { 2024-04-30T08:45:13.168 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.168 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.168 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.169 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.169 INFO:teuthology.orchestra.run.smithi032.stdout: "req_mkdir_latency": { 2024-04-30T08:45:13.169 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.169 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.169 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.169 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.170 INFO:teuthology.orchestra.run.smithi032.stdout: "req_mknod_latency": { 2024-04-30T08:45:13.170 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.170 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.170 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.170 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.170 INFO:teuthology.orchestra.run.smithi032.stdout: "req_mksnap_latency": { 2024-04-30T08:45:13.170 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.170 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.170 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.170 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.170 INFO:teuthology.orchestra.run.smithi032.stdout: "req_open_latency": { 2024-04-30T08:45:13.170 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.170 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.171 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.171 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.171 INFO:teuthology.orchestra.run.smithi032.stdout: "req_readdir_latency": { 2024-04-30T08:45:13.171 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.171 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.171 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.171 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.171 INFO:teuthology.orchestra.run.smithi032.stdout: "req_rename_latency": { 2024-04-30T08:45:13.171 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.171 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.171 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.171 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.171 INFO:teuthology.orchestra.run.smithi032.stdout: "req_renamesnap_latency": { 2024-04-30T08:45:13.172 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.172 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.172 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.172 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.172 INFO:teuthology.orchestra.run.smithi032.stdout: "req_snapdiff_latency": { 2024-04-30T08:45:13.172 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.172 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.172 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.172 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.172 INFO:teuthology.orchestra.run.smithi032.stdout: "req_rmdir_latency": { 2024-04-30T08:45:13.172 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.172 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.172 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.173 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.173 INFO:teuthology.orchestra.run.smithi032.stdout: "req_rmsnap_latency": { 2024-04-30T08:45:13.173 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.173 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.173 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.173 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.173 INFO:teuthology.orchestra.run.smithi032.stdout: "req_rmxattr_latency": { 2024-04-30T08:45:13.173 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.173 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.173 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.173 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.173 INFO:teuthology.orchestra.run.smithi032.stdout: "req_setattr_latency": { 2024-04-30T08:45:13.173 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.173 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.173 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.174 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.174 INFO:teuthology.orchestra.run.smithi032.stdout: "req_setdirlayout_latency": { 2024-04-30T08:45:13.174 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.174 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.174 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.174 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.174 INFO:teuthology.orchestra.run.smithi032.stdout: "req_setfilelock_latency": { 2024-04-30T08:45:13.174 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.174 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.174 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.174 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.174 INFO:teuthology.orchestra.run.smithi032.stdout: "req_setlayout_latency": { 2024-04-30T08:45:13.174 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.175 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.175 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.175 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.175 INFO:teuthology.orchestra.run.smithi032.stdout: "req_setxattr_latency": { 2024-04-30T08:45:13.175 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.175 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.175 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.175 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.175 INFO:teuthology.orchestra.run.smithi032.stdout: "req_symlink_latency": { 2024-04-30T08:45:13.175 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.175 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.175 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.175 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.176 INFO:teuthology.orchestra.run.smithi032.stdout: "req_unlink_latency": { 2024-04-30T08:45:13.176 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.176 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.176 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.176 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.176 INFO:teuthology.orchestra.run.smithi032.stdout: "cap_revoke_eviction": 0, 2024-04-30T08:45:13.176 INFO:teuthology.orchestra.run.smithi032.stdout: "cap_acquisition_throttle": 0, 2024-04-30T08:45:13.176 INFO:teuthology.orchestra.run.smithi032.stdout: "req_getvxattr_latency": { 2024-04-30T08:45:13.176 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.176 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.176 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.176 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:45:13.176 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.176 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_sessions": { 2024-04-30T08:45:13.177 INFO:teuthology.orchestra.run.smithi032.stdout: "session_count": 1, 2024-04-30T08:45:13.177 INFO:teuthology.orchestra.run.smithi032.stdout: "session_add": 3, 2024-04-30T08:45:13.177 INFO:teuthology.orchestra.run.smithi032.stdout: "session_remove": 2, 2024-04-30T08:45:13.177 INFO:teuthology.orchestra.run.smithi032.stdout: "sessions_open": 2, 2024-04-30T08:45:13.177 INFO:teuthology.orchestra.run.smithi032.stdout: "sessions_stale": 0, 2024-04-30T08:45:13.177 INFO:teuthology.orchestra.run.smithi032.stdout: "total_load": 0, 2024-04-30T08:45:13.177 INFO:teuthology.orchestra.run.smithi032.stdout: "average_load": 0, 2024-04-30T08:45:13.177 INFO:teuthology.orchestra.run.smithi032.stdout: "avg_session_uptime": 3367, 2024-04-30T08:45:13.177 INFO:teuthology.orchestra.run.smithi032.stdout: "mdthresh_evicted": 0 2024-04-30T08:45:13.177 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.177 INFO:teuthology.orchestra.run.smithi032.stdout: "mempool": { 2024-04-30T08:45:13.177 INFO:teuthology.orchestra.run.smithi032.stdout: "bloom_filter_bytes": 0, 2024-04-30T08:45:13.177 INFO:teuthology.orchestra.run.smithi032.stdout: "bloom_filter_items": 0, 2024-04-30T08:45:13.177 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_alloc_bytes": 0, 2024-04-30T08:45:13.178 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_alloc_items": 0, 2024-04-30T08:45:13.178 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_data_bytes": 0, 2024-04-30T08:45:13.178 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_data_items": 0, 2024-04-30T08:45:13.178 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-30T08:45:13.178 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_onode_items": 0, 2024-04-30T08:45:13.178 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-30T08:45:13.178 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_meta_items": 0, 2024-04-30T08:45:13.178 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_other_bytes": 0, 2024-04-30T08:45:13.178 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_other_items": 0, 2024-04-30T08:45:13.178 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-30T08:45:13.178 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_buffer_items": 0, 2024-04-30T08:45:13.178 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_extent_bytes": 0, 2024-04-30T08:45:13.178 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_extent_items": 0, 2024-04-30T08:45:13.179 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_blob_bytes": 0, 2024-04-30T08:45:13.179 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_blob_items": 0, 2024-04-30T08:45:13.179 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-30T08:45:13.179 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_shared_blob_items": 0, 2024-04-30T08:45:13.179 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-30T08:45:13.179 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_inline_bl_items": 0, 2024-04-30T08:45:13.179 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_bytes": 0, 2024-04-30T08:45:13.179 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_items": 0, 2024-04-30T08:45:13.179 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_txc_bytes": 0, 2024-04-30T08:45:13.179 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_txc_items": 0, 2024-04-30T08:45:13.179 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-30T08:45:13.179 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_deferred_items": 0, 2024-04-30T08:45:13.179 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_bytes": 0, 2024-04-30T08:45:13.179 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_items": 0, 2024-04-30T08:45:13.180 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_bytes": 0, 2024-04-30T08:45:13.180 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_items": 0, 2024-04-30T08:45:13.180 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_reader_bytes": 0, 2024-04-30T08:45:13.180 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_reader_items": 0, 2024-04-30T08:45:13.180 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_writer_bytes": 0, 2024-04-30T08:45:13.180 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_writer_items": 0, 2024-04-30T08:45:13.180 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_anon_bytes": 189383, 2024-04-30T08:45:13.180 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_anon_items": 50, 2024-04-30T08:45:13.180 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_meta_bytes": 0, 2024-04-30T08:45:13.180 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_meta_items": 0, 2024-04-30T08:45:13.180 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_bytes": 0, 2024-04-30T08:45:13.180 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_items": 0, 2024-04-30T08:45:13.180 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mapbl_bytes": 0, 2024-04-30T08:45:13.181 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mapbl_items": 0, 2024-04-30T08:45:13.181 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pglog_bytes": 0, 2024-04-30T08:45:13.181 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pglog_items": 0, 2024-04-30T08:45:13.181 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_bytes": 9088, 2024-04-30T08:45:13.181 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_items": 175, 2024-04-30T08:45:13.181 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_mapping_bytes": 0, 2024-04-30T08:45:13.181 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_mapping_items": 0, 2024-04-30T08:45:13.181 INFO:teuthology.orchestra.run.smithi032.stdout: "pgmap_bytes": 0, 2024-04-30T08:45:13.181 INFO:teuthology.orchestra.run.smithi032.stdout: "pgmap_items": 0, 2024-04-30T08:45:13.181 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_co_bytes": 13456, 2024-04-30T08:45:13.181 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_co_items": 36, 2024-04-30T08:45:13.181 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_1_bytes": 0, 2024-04-30T08:45:13.181 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_1_items": 0, 2024-04-30T08:45:13.181 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_2_bytes": 0, 2024-04-30T08:45:13.181 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_2_items": 0 2024-04-30T08:45:13.182 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.182 INFO:teuthology.orchestra.run.smithi032.stdout: "objecter": { 2024-04-30T08:45:13.182 INFO:teuthology.orchestra.run.smithi032.stdout: "op_active": 0, 2024-04-30T08:45:13.182 INFO:teuthology.orchestra.run.smithi032.stdout: "op_laggy": 0, 2024-04-30T08:45:13.182 INFO:teuthology.orchestra.run.smithi032.stdout: "op_send": 2072, 2024-04-30T08:45:13.182 INFO:teuthology.orchestra.run.smithi032.stdout: "op_send_bytes": 21, 2024-04-30T08:45:13.182 INFO:teuthology.orchestra.run.smithi032.stdout: "op_resend": 3, 2024-04-30T08:45:13.182 INFO:teuthology.orchestra.run.smithi032.stdout: "op_reply": 2072, 2024-04-30T08:45:13.182 INFO:teuthology.orchestra.run.smithi032.stdout: "op_latency": { 2024-04-30T08:45:13.182 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 2072, 2024-04-30T08:45:13.182 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 48.327048307, 2024-04-30T08:45:13.182 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.023323865 2024-04-30T08:45:13.182 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.183 INFO:teuthology.orchestra.run.smithi032.stdout: "op_inflight": 0, 2024-04-30T08:45:13.183 INFO:teuthology.orchestra.run.smithi032.stdout: "oplen_avg": { 2024-04-30T08:45:13.183 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 2072, 2024-04-30T08:45:13.183 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 2073 2024-04-30T08:45:13.183 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.183 INFO:teuthology.orchestra.run.smithi032.stdout: "op": 2072, 2024-04-30T08:45:13.183 INFO:teuthology.orchestra.run.smithi032.stdout: "op_r": 2072, 2024-04-30T08:45:13.183 INFO:teuthology.orchestra.run.smithi032.stdout: "op_w": 0, 2024-04-30T08:45:13.183 INFO:teuthology.orchestra.run.smithi032.stdout: "op_rmw": 0, 2024-04-30T08:45:13.183 INFO:teuthology.orchestra.run.smithi032.stdout: "op_pg": 0, 2024-04-30T08:45:13.183 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_stat": 1341, 2024-04-30T08:45:13.183 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_create": 0, 2024-04-30T08:45:13.183 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_read": 729, 2024-04-30T08:45:13.184 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_write": 0, 2024-04-30T08:45:13.184 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_writefull": 0, 2024-04-30T08:45:13.184 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_writesame": 0, 2024-04-30T08:45:13.184 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_append": 0, 2024-04-30T08:45:13.184 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_zero": 0, 2024-04-30T08:45:13.184 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_truncate": 0, 2024-04-30T08:45:13.184 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_delete": 0, 2024-04-30T08:45:13.184 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_mapext": 0, 2024-04-30T08:45:13.184 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_sparse_read": 0, 2024-04-30T08:45:13.184 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_clonerange": 0, 2024-04-30T08:45:13.184 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_getxattr": 1, 2024-04-30T08:45:13.184 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_setxattr": 0, 2024-04-30T08:45:13.184 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_cmpxattr": 0, 2024-04-30T08:45:13.184 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_rmxattr": 0, 2024-04-30T08:45:13.185 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_resetxattrs": 0, 2024-04-30T08:45:13.185 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_call": 0, 2024-04-30T08:45:13.185 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_watch": 0, 2024-04-30T08:45:13.185 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_notify": 0, 2024-04-30T08:45:13.185 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_src_cmpxattr": 0, 2024-04-30T08:45:13.185 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_pgls": 0, 2024-04-30T08:45:13.185 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_pgls_filter": 0, 2024-04-30T08:45:13.187 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_other": 0, 2024-04-30T08:45:13.187 INFO:teuthology.orchestra.run.smithi032.stdout: "linger_active": 0, 2024-04-30T08:45:13.187 INFO:teuthology.orchestra.run.smithi032.stdout: "linger_send": 0, 2024-04-30T08:45:13.187 INFO:teuthology.orchestra.run.smithi032.stdout: "linger_resend": 0, 2024-04-30T08:45:13.187 INFO:teuthology.orchestra.run.smithi032.stdout: "linger_ping": 0, 2024-04-30T08:45:13.187 INFO:teuthology.orchestra.run.smithi032.stdout: "poolop_active": 0, 2024-04-30T08:45:13.187 INFO:teuthology.orchestra.run.smithi032.stdout: "poolop_send": 0, 2024-04-30T08:45:13.187 INFO:teuthology.orchestra.run.smithi032.stdout: "poolop_resend": 0, 2024-04-30T08:45:13.187 INFO:teuthology.orchestra.run.smithi032.stdout: "poolstat_active": 0, 2024-04-30T08:45:13.187 INFO:teuthology.orchestra.run.smithi032.stdout: "poolstat_send": 0, 2024-04-30T08:45:13.187 INFO:teuthology.orchestra.run.smithi032.stdout: "poolstat_resend": 0, 2024-04-30T08:45:13.188 INFO:teuthology.orchestra.run.smithi032.stdout: "statfs_active": 0, 2024-04-30T08:45:13.188 INFO:teuthology.orchestra.run.smithi032.stdout: "statfs_send": 0, 2024-04-30T08:45:13.188 INFO:teuthology.orchestra.run.smithi032.stdout: "statfs_resend": 0, 2024-04-30T08:45:13.188 INFO:teuthology.orchestra.run.smithi032.stdout: "command_active": 0, 2024-04-30T08:45:13.188 INFO:teuthology.orchestra.run.smithi032.stdout: "command_send": 0, 2024-04-30T08:45:13.188 INFO:teuthology.orchestra.run.smithi032.stdout: "command_resend": 0, 2024-04-30T08:45:13.188 INFO:teuthology.orchestra.run.smithi032.stdout: "map_epoch": 85, 2024-04-30T08:45:13.188 INFO:teuthology.orchestra.run.smithi032.stdout: "map_full": 0, 2024-04-30T08:45:13.188 INFO:teuthology.orchestra.run.smithi032.stdout: "map_inc": 3, 2024-04-30T08:45:13.188 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_sessions": 5, 2024-04-30T08:45:13.188 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_session_open": 5, 2024-04-30T08:45:13.188 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_session_close": 0, 2024-04-30T08:45:13.188 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_laggy": 0, 2024-04-30T08:45:13.189 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_wr": 0, 2024-04-30T08:45:13.189 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_rd": 2, 2024-04-30T08:45:13.189 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_del": 0 2024-04-30T08:45:13.189 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.189 INFO:teuthology.orchestra.run.smithi032.stdout: "oft": { 2024-04-30T08:45:13.189 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_total_objs": 0, 2024-04-30T08:45:13.189 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_total_kv_pairs": 0, 2024-04-30T08:45:13.189 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_total_updates": 0, 2024-04-30T08:45:13.189 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_total_removes": 0 2024-04-30T08:45:13.189 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.189 INFO:teuthology.orchestra.run.smithi032.stdout: "osd-slow-ops": { 2024-04-30T08:45:13.189 INFO:teuthology.orchestra.run.smithi032.stdout: "slow_ops_count": 0 2024-04-30T08:45:13.189 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.189 INFO:teuthology.orchestra.run.smithi032.stdout: "purge_queue": { 2024-04-30T08:45:13.190 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executing_ops": 0, 2024-04-30T08:45:13.190 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executing_ops_high_water": 0, 2024-04-30T08:45:13.190 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executing": 0, 2024-04-30T08:45:13.190 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executing_high_water": 0, 2024-04-30T08:45:13.190 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executed_ops": 0, 2024-04-30T08:45:13.190 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executed": 0, 2024-04-30T08:45:13.190 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_item_in_journal": 0 2024-04-30T08:45:13.190 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.190 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-30T08:45:13.190 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-30T08:45:13.190 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 104857600, 2024-04-30T08:45:13.190 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-30T08:45:13.190 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 2414, 2024-04-30T08:45:13.191 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 1907974, 2024-04-30T08:45:13.191 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:13.191 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 2414, 2024-04-30T08:45:13.191 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 0, 2024-04-30T08:45:13.191 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 0, 2024-04-30T08:45:13.191 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 2414, 2024-04-30T08:45:13.191 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 1907974, 2024-04-30T08:45:13.191 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-30T08:45:13.193 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.193 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.193 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.193 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:45:13.193 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.194 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-objecter_bytes": { 2024-04-30T08:45:13.194 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-30T08:45:13.194 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 104857600, 2024-04-30T08:45:13.194 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-30T08:45:13.194 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 0, 2024-04-30T08:45:13.194 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 0, 2024-04-30T08:45:13.194 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:13.194 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 0, 2024-04-30T08:45:13.194 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 2072, 2024-04-30T08:45:13.194 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 25560, 2024-04-30T08:45:13.194 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 2072, 2024-04-30T08:45:13.194 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 25560, 2024-04-30T08:45:13.194 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-30T08:45:13.194 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.195 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.195 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.195 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:45:13.195 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.195 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-objecter_ops": { 2024-04-30T08:45:13.195 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-30T08:45:13.195 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 1024, 2024-04-30T08:45:13.195 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-30T08:45:13.195 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 0, 2024-04-30T08:45:13.195 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 0, 2024-04-30T08:45:13.195 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:13.195 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 0, 2024-04-30T08:45:13.195 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 2072, 2024-04-30T08:45:13.196 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 2072, 2024-04-30T08:45:13.196 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 2072, 2024-04-30T08:45:13.196 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 2072, 2024-04-30T08:45:13.196 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-30T08:45:13.196 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.196 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.196 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.196 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:45:13.196 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.196 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-write_buf_throttle": { 2024-04-30T08:45:13.196 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-30T08:45:13.196 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 3758096384, 2024-04-30T08:45:13.196 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-30T08:45:13.196 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 0, 2024-04-30T08:45:13.197 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 0, 2024-04-30T08:45:13.197 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:13.197 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 0, 2024-04-30T08:45:13.197 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 0, 2024-04-30T08:45:13.197 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 0, 2024-04-30T08:45:13.197 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 0, 2024-04-30T08:45:13.197 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 0, 2024-04-30T08:45:13.197 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-30T08:45:13.197 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.197 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.197 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.197 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:45:13.197 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:13.198 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-write_buf_throttle-0x55bfb0550410": { 2024-04-30T08:45:13.198 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-30T08:45:13.198 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 3758096384, 2024-04-30T08:45:13.198 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-30T08:45:13.198 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 0, 2024-04-30T08:45:13.198 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 0, 2024-04-30T08:45:13.198 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:13.198 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 0, 2024-04-30T08:45:13.198 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 0, 2024-04-30T08:45:13.198 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 0, 2024-04-30T08:45:13.198 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 0, 2024-04-30T08:45:13.198 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 0, 2024-04-30T08:45:13.198 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-30T08:45:13.199 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:13.199 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:13.199 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:13.199 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:45:13.199 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:45:13.199 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-30T08:45:14.162 INFO:tasks.check_counter:Daemon mds.d mds.exported=0 2024-04-30T08:45:14.162 INFO:tasks.check_counter:Daemon mds.d mds.imported=0 2024-04-30T08:45:14.162 INFO:tasks.check_counter:Daemon mds.d mds.dir_split=0 2024-04-30T08:45:14.162 DEBUG:tasks.check_counter:Getting stats from g 2024-04-30T08:45:14.163 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-30T08:45:14.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:14 smithi032 ceph-mon[28331]: pgmap v727: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.1 KiB/s wr, 39 op/s 2024-04-30T08:45:14.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:14 smithi071 ceph-mon[33385]: pgmap v727: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.1 KiB/s wr, 39 op/s 2024-04-30T08:45:14.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:14 smithi189 ceph-mon[33038]: pgmap v727: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.1 KiB/s wr, 39 op/s 2024-04-30T08:45:14.631 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:45:16.409 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:16 smithi032 ceph-mon[28331]: pgmap v728: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 511 B/s wr, 20 op/s 2024-04-30T08:45:16.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:16 smithi071 ceph-mon[33385]: pgmap v728: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 511 B/s wr, 20 op/s 2024-04-30T08:45:16.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:16 smithi189 ceph-mon[33038]: pgmap v728: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 511 B/s wr, 20 op/s 2024-04-30T08:45:17.185 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-30T08:45:17.185 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-0": { 2024-04-30T08:45:17.185 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 424, 2024-04-30T08:45:17.185 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 310, 2024-04-30T08:45:17.185 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 1569869, 2024-04-30T08:45:17.185 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 201613, 2024-04-30T08:45:17.185 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 13, 2024-04-30T08:45:17.186 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 4, 2024-04-30T08:45:17.186 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.151030078, 2024-04-30T08:45:17.186 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.043777205, 2024-04-30T08:45:17.186 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.101864006, 2024-04-30T08:45:17.186 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.010641336, 2024-04-30T08:45:17.186 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:17.186 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 305, 2024-04-30T08:45:17.186 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.061192512, 2024-04-30T08:45:17.186 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000200631 2024-04-30T08:45:17.186 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.186 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:17.187 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 37, 2024-04-30T08:45:17.187 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000027984, 2024-04-30T08:45:17.187 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000756 2024-04-30T08:45:17.187 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.187 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 1559056, 2024-04-30T08:45:17.187 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 143360 2024-04-30T08:45:17.187 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.187 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-1": { 2024-04-30T08:45:17.187 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 725, 2024-04-30T08:45:17.187 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 1087, 2024-04-30T08:45:17.187 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 229437, 2024-04-30T08:45:17.187 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 679688, 2024-04-30T08:45:17.187 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 7, 2024-04-30T08:45:17.188 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 4, 2024-04-30T08:45:17.188 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.181853242, 2024-04-30T08:45:17.188 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.088230679, 2024-04-30T08:45:17.188 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.088379438, 2024-04-30T08:45:17.188 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.006497562, 2024-04-30T08:45:17.188 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:17.188 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 1090, 2024-04-30T08:45:17.188 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.294673604, 2024-04-30T08:45:17.188 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000270342 2024-04-30T08:45:17.188 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.188 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:17.188 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 1507, 2024-04-30T08:45:17.188 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000821643, 2024-04-30T08:45:17.189 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000545 2024-04-30T08:45:17.189 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.189 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-04-30T08:45:17.189 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 366544 2024-04-30T08:45:17.189 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.189 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-2": { 2024-04-30T08:45:17.189 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 25, 2024-04-30T08:45:17.189 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 27, 2024-04-30T08:45:17.189 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 5562, 2024-04-30T08:45:17.189 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 131698, 2024-04-30T08:45:17.189 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 16, 2024-04-30T08:45:17.189 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 7, 2024-04-30T08:45:17.189 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.020464700, 2024-04-30T08:45:17.189 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.001849180, 2024-04-30T08:45:17.190 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.017881144, 2024-04-30T08:45:17.190 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.000603898, 2024-04-30T08:45:17.190 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:17.190 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 25, 2024-04-30T08:45:17.190 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.041205292, 2024-04-30T08:45:17.190 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.001648211 2024-04-30T08:45:17.190 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.190 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:17.190 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 10, 2024-04-30T08:45:17.190 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000002222, 2024-04-30T08:45:17.190 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000222 2024-04-30T08:45:17.190 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.190 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-30T08:45:17.190 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 0 2024-04-30T08:45:17.191 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.191 INFO:teuthology.orchestra.run.smithi032.stdout: "cct": { 2024-04-30T08:45:17.191 INFO:teuthology.orchestra.run.smithi032.stdout: "total_workers": 1, 2024-04-30T08:45:17.191 INFO:teuthology.orchestra.run.smithi032.stdout: "unhealthy_workers": 0 2024-04-30T08:45:17.191 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.191 INFO:teuthology.orchestra.run.smithi032.stdout: "finisher-MDSRank": { 2024-04-30T08:45:17.191 INFO:teuthology.orchestra.run.smithi032.stdout: "queue_len": 0, 2024-04-30T08:45:17.191 INFO:teuthology.orchestra.run.smithi032.stdout: "complete_latency": { 2024-04-30T08:45:17.191 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 42, 2024-04-30T08:45:17.191 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.004754171, 2024-04-30T08:45:17.191 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000113194 2024-04-30T08:45:17.191 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:45:17.192 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.192 INFO:teuthology.orchestra.run.smithi032.stdout: "finisher-PurgeQueue": { 2024-04-30T08:45:17.192 INFO:teuthology.orchestra.run.smithi032.stdout: "queue_len": 0, 2024-04-30T08:45:17.192 INFO:teuthology.orchestra.run.smithi032.stdout: "complete_latency": { 2024-04-30T08:45:17.192 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 2, 2024-04-30T08:45:17.192 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000007202, 2024-04-30T08:45:17.192 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000003601 2024-04-30T08:45:17.192 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:45:17.192 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.192 INFO:teuthology.orchestra.run.smithi032.stdout: "mds": { 2024-04-30T08:45:17.192 INFO:teuthology.orchestra.run.smithi032.stdout: "request": 16, 2024-04-30T08:45:17.192 INFO:teuthology.orchestra.run.smithi032.stdout: "reply": 1, 2024-04-30T08:45:17.192 INFO:teuthology.orchestra.run.smithi032.stdout: "reply_latency": { 2024-04-30T08:45:17.193 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 1, 2024-04-30T08:45:17.193 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000562563, 2024-04-30T08:45:17.193 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000562563 2024-04-30T08:45:17.193 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.193 INFO:teuthology.orchestra.run.smithi032.stdout: "slow_reply": 0, 2024-04-30T08:45:17.193 INFO:teuthology.orchestra.run.smithi032.stdout: "forward": 15, 2024-04-30T08:45:17.193 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_fetch_complete": 0, 2024-04-30T08:45:17.193 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_fetch_keys": 0, 2024-04-30T08:45:17.193 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_commit": 11, 2024-04-30T08:45:17.193 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_split": 0, 2024-04-30T08:45:17.193 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_merge": 0, 2024-04-30T08:45:17.193 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes": 15, 2024-04-30T08:45:17.193 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_top": 0, 2024-04-30T08:45:17.194 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_bottom": 15, 2024-04-30T08:45:17.194 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_pin_tail": 0, 2024-04-30T08:45:17.194 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_pinned": 13, 2024-04-30T08:45:17.194 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_expired": 6, 2024-04-30T08:45:17.194 INFO:teuthology.orchestra.run.smithi032.stdout: "inodes_with_caps": 0, 2024-04-30T08:45:17.194 INFO:teuthology.orchestra.run.smithi032.stdout: "caps": 0, 2024-04-30T08:45:17.194 INFO:teuthology.orchestra.run.smithi032.stdout: "subtrees": 3, 2024-04-30T08:45:17.194 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse": 19, 2024-04-30T08:45:17.194 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_hit": 3, 2024-04-30T08:45:17.194 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_forward": 13, 2024-04-30T08:45:17.194 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_discover": 2, 2024-04-30T08:45:17.194 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_dir_fetch": 0, 2024-04-30T08:45:17.194 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_remote_ino": 0, 2024-04-30T08:45:17.195 INFO:teuthology.orchestra.run.smithi032.stdout: "traverse_lock": 0, 2024-04-30T08:45:17.195 INFO:teuthology.orchestra.run.smithi032.stdout: "load_cent": 0, 2024-04-30T08:45:17.195 INFO:teuthology.orchestra.run.smithi032.stdout: "q": 0, 2024-04-30T08:45:17.195 INFO:teuthology.orchestra.run.smithi032.stdout: "exported": 0, 2024-04-30T08:45:17.195 INFO:teuthology.orchestra.run.smithi032.stdout: "exported_inodes": 0, 2024-04-30T08:45:17.195 INFO:teuthology.orchestra.run.smithi032.stdout: "imported": 0, 2024-04-30T08:45:17.195 INFO:teuthology.orchestra.run.smithi032.stdout: "imported_inodes": 0, 2024-04-30T08:45:17.195 INFO:teuthology.orchestra.run.smithi032.stdout: "openino_dir_fetch": 0, 2024-04-30T08:45:17.195 INFO:teuthology.orchestra.run.smithi032.stdout: "openino_backtrace_fetch": 0, 2024-04-30T08:45:17.195 INFO:teuthology.orchestra.run.smithi032.stdout: "openino_peer_discover": 1, 2024-04-30T08:45:17.195 INFO:teuthology.orchestra.run.smithi032.stdout: "root_rfiles": 325, 2024-04-30T08:45:17.196 INFO:teuthology.orchestra.run.smithi032.stdout: "root_rbytes": 13424230511, 2024-04-30T08:45:17.196 INFO:teuthology.orchestra.run.smithi032.stdout: "root_rsnaps": 0, 2024-04-30T08:45:17.196 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_backtrace_fetch": 0, 2024-04-30T08:45:17.196 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_set_tag": 0, 2024-04-30T08:45:17.196 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_backtrace_repaired": 0, 2024-04-30T08:45:17.196 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_inotable_repaired": 0, 2024-04-30T08:45:17.196 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_dir_inodes": 0, 2024-04-30T08:45:17.196 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_dir_base_inodes": 0, 2024-04-30T08:45:17.196 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_dirfrag_rstats": 0, 2024-04-30T08:45:17.196 INFO:teuthology.orchestra.run.smithi032.stdout: "scrub_file_inodes": 0, 2024-04-30T08:45:17.196 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_inode_file_caps": 0, 2024-04-30T08:45:17.196 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_cap_op_revoke": 0, 2024-04-30T08:45:17.196 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_cap_op_grant": 0, 2024-04-30T08:45:17.196 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_cap_op_trunc": 0, 2024-04-30T08:45:17.197 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-30T08:45:17.197 INFO:teuthology.orchestra.run.smithi032.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-30T08:45:17.197 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_client_caps": 0, 2024-04-30T08:45:17.197 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_client_caps_dirty": 0, 2024-04-30T08:45:17.197 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_client_cap_release": 0, 2024-04-30T08:45:17.197 INFO:teuthology.orchestra.run.smithi032.stdout: "process_request_cap_release": 1 2024-04-30T08:45:17.197 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.197 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_cache": { 2024-04-30T08:45:17.197 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_update": 0, 2024-04-30T08:45:17.197 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_update_receipt": 3, 2024-04-30T08:45:17.197 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_try_discover": 2, 2024-04-30T08:45:17.197 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_send_discover": 4, 2024-04-30T08:45:17.197 INFO:teuthology.orchestra.run.smithi032.stdout: "dir_handle_discover": 0, 2024-04-30T08:45:17.198 INFO:teuthology.orchestra.run.smithi032.stdout: "num_strays": 0, 2024-04-30T08:45:17.198 INFO:teuthology.orchestra.run.smithi032.stdout: "num_strays_delayed": 0, 2024-04-30T08:45:17.198 INFO:teuthology.orchestra.run.smithi032.stdout: "num_strays_enqueuing": 0, 2024-04-30T08:45:17.198 INFO:teuthology.orchestra.run.smithi032.stdout: "strays_created": 0, 2024-04-30T08:45:17.198 INFO:teuthology.orchestra.run.smithi032.stdout: "strays_enqueued": 0, 2024-04-30T08:45:17.198 INFO:teuthology.orchestra.run.smithi032.stdout: "strays_reintegrated": 0, 2024-04-30T08:45:17.198 INFO:teuthology.orchestra.run.smithi032.stdout: "strays_migrated": 0, 2024-04-30T08:45:17.198 INFO:teuthology.orchestra.run.smithi032.stdout: "num_recovering_processing": 0, 2024-04-30T08:45:17.198 INFO:teuthology.orchestra.run.smithi032.stdout: "num_recovering_enqueued": 0, 2024-04-30T08:45:17.198 INFO:teuthology.orchestra.run.smithi032.stdout: "num_recovering_prioritized": 0, 2024-04-30T08:45:17.198 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_started": 0, 2024-04-30T08:45:17.198 INFO:teuthology.orchestra.run.smithi032.stdout: "recovery_completed": 0, 2024-04-30T08:45:17.198 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_quiesce_path": 0, 2024-04-30T08:45:17.199 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_quiesce_inode": 0, 2024-04-30T08:45:17.199 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_enqueue_scrub": 0, 2024-04-30T08:45:17.199 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_exportdir": 0, 2024-04-30T08:45:17.199 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_flush": 0, 2024-04-30T08:45:17.199 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_fragmentdir": 0, 2024-04-30T08:45:17.199 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_fragstats": 0, 2024-04-30T08:45:17.199 INFO:teuthology.orchestra.run.smithi032.stdout: "ireq_inodestats": 0 2024-04-30T08:45:17.199 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.199 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_log": { 2024-04-30T08:45:17.199 INFO:teuthology.orchestra.run.smithi032.stdout: "evlrg": 0, 2024-04-30T08:45:17.199 INFO:teuthology.orchestra.run.smithi032.stdout: "evadd": 5, 2024-04-30T08:45:17.199 INFO:teuthology.orchestra.run.smithi032.stdout: "evex": 0, 2024-04-30T08:45:17.199 INFO:teuthology.orchestra.run.smithi032.stdout: "evtrm": 0, 2024-04-30T08:45:17.199 INFO:teuthology.orchestra.run.smithi032.stdout: "ev": 5, 2024-04-30T08:45:17.200 INFO:teuthology.orchestra.run.smithi032.stdout: "evexg": 0, 2024-04-30T08:45:17.200 INFO:teuthology.orchestra.run.smithi032.stdout: "evexd": 0, 2024-04-30T08:45:17.200 INFO:teuthology.orchestra.run.smithi032.stdout: "segadd": 2, 2024-04-30T08:45:17.200 INFO:teuthology.orchestra.run.smithi032.stdout: "segex": 0, 2024-04-30T08:45:17.200 INFO:teuthology.orchestra.run.smithi032.stdout: "segtrm": 0, 2024-04-30T08:45:17.200 INFO:teuthology.orchestra.run.smithi032.stdout: "seg": 2, 2024-04-30T08:45:17.200 INFO:teuthology.orchestra.run.smithi032.stdout: "segmjr": 2, 2024-04-30T08:45:17.200 INFO:teuthology.orchestra.run.smithi032.stdout: "segexg": 0, 2024-04-30T08:45:17.200 INFO:teuthology.orchestra.run.smithi032.stdout: "segexd": 0, 2024-04-30T08:45:17.200 INFO:teuthology.orchestra.run.smithi032.stdout: "expos": 4194304, 2024-04-30T08:45:17.200 INFO:teuthology.orchestra.run.smithi032.stdout: "wrpos": 4195734, 2024-04-30T08:45:17.200 INFO:teuthology.orchestra.run.smithi032.stdout: "rdpos": 0, 2024-04-30T08:45:17.200 INFO:teuthology.orchestra.run.smithi032.stdout: "jlat": { 2024-04-30T08:45:17.200 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 4, 2024-04-30T08:45:17.201 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.008992696, 2024-04-30T08:45:17.201 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.002248174 2024-04-30T08:45:17.201 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.201 INFO:teuthology.orchestra.run.smithi032.stdout: "replayed": 0 2024-04-30T08:45:17.201 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.201 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_mem": { 2024-04-30T08:45:17.201 INFO:teuthology.orchestra.run.smithi032.stdout: "ino": 19, 2024-04-30T08:45:17.201 INFO:teuthology.orchestra.run.smithi032.stdout: "ino+": 22, 2024-04-30T08:45:17.201 INFO:teuthology.orchestra.run.smithi032.stdout: "ino-": 3, 2024-04-30T08:45:17.201 INFO:teuthology.orchestra.run.smithi032.stdout: "dir": 18, 2024-04-30T08:45:17.201 INFO:teuthology.orchestra.run.smithi032.stdout: "dir+": 22, 2024-04-30T08:45:17.201 INFO:teuthology.orchestra.run.smithi032.stdout: "dir-": 4, 2024-04-30T08:45:17.201 INFO:teuthology.orchestra.run.smithi032.stdout: "dn": 15, 2024-04-30T08:45:17.202 INFO:teuthology.orchestra.run.smithi032.stdout: "dn+": 21, 2024-04-30T08:45:17.202 INFO:teuthology.orchestra.run.smithi032.stdout: "dn-": 6, 2024-04-30T08:45:17.202 INFO:teuthology.orchestra.run.smithi032.stdout: "cap": 0, 2024-04-30T08:45:17.202 INFO:teuthology.orchestra.run.smithi032.stdout: "cap+": 1, 2024-04-30T08:45:17.202 INFO:teuthology.orchestra.run.smithi032.stdout: "cap-": 1, 2024-04-30T08:45:17.202 INFO:teuthology.orchestra.run.smithi032.stdout: "rss": 120448, 2024-04-30T08:45:17.202 INFO:teuthology.orchestra.run.smithi032.stdout: "heap": 223632 2024-04-30T08:45:17.202 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.202 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_server": { 2024-04-30T08:45:17.202 INFO:teuthology.orchestra.run.smithi032.stdout: "dispatch_client_request": 17, 2024-04-30T08:45:17.202 INFO:teuthology.orchestra.run.smithi032.stdout: "dispatch_server_request": 0, 2024-04-30T08:45:17.202 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_client_request": 16, 2024-04-30T08:45:17.202 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_client_session": 12, 2024-04-30T08:45:17.203 INFO:teuthology.orchestra.run.smithi032.stdout: "handle_peer_request": 0, 2024-04-30T08:45:17.203 INFO:teuthology.orchestra.run.smithi032.stdout: "req_create_latency": { 2024-04-30T08:45:17.203 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:17.203 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:17.203 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:17.203 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.203 INFO:teuthology.orchestra.run.smithi032.stdout: "req_getattr_latency": { 2024-04-30T08:45:17.203 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 1, 2024-04-30T08:45:17.203 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000562563, 2024-04-30T08:45:17.203 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000562563 2024-04-30T08:45:17.203 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.203 INFO:teuthology.orchestra.run.smithi032.stdout: "req_getfilelock_latency": { 2024-04-30T08:45:17.203 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:17.203 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:17.204 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:17.204 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.204 INFO:teuthology.orchestra.run.smithi032.stdout: "req_link_latency": { 2024-04-30T08:45:17.204 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:17.204 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:17.204 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:17.204 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.204 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookup_latency": { 2024-04-30T08:45:17.204 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:17.204 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:17.204 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:17.204 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.204 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookuphash_latency": { 2024-04-30T08:45:17.204 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:17.205 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:17.205 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:17.205 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.205 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookupino_latency": { 2024-04-30T08:45:17.205 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:17.205 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:17.205 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:17.205 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.205 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookupname_latency": { 2024-04-30T08:45:17.205 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:17.205 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:17.205 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:17.205 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.205 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookupparent_latency": { 2024-04-30T08:45:17.206 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:17.206 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:17.206 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:17.206 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.206 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lookupsnap_latency": { 2024-04-30T08:45:17.206 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:17.206 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:17.206 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:17.206 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.206 INFO:teuthology.orchestra.run.smithi032.stdout: "req_lssnap_latency": { 2024-04-30T08:45:17.206 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:17.206 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:17.206 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:17.207 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.207 INFO:teuthology.orchestra.run.smithi032.stdout: "req_mkdir_latency": { 2024-04-30T08:45:17.207 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:17.207 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:17.207 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:17.207 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.208 INFO:teuthology.orchestra.run.smithi032.stdout: "req_mknod_latency": { 2024-04-30T08:45:17.208 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:17.208 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:17.208 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:17.208 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.208 INFO:teuthology.orchestra.run.smithi032.stdout: "req_mksnap_latency": { 2024-04-30T08:45:17.208 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:17.208 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:17.208 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:17.208 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.208 INFO:teuthology.orchestra.run.smithi032.stdout: "req_open_latency": { 2024-04-30T08:45:17.208 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:17.208 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:17.208 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:17.209 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.209 INFO:teuthology.orchestra.run.smithi032.stdout: "req_readdir_latency": { 2024-04-30T08:45:17.209 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:17.209 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:17.209 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:17.209 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.209 INFO:teuthology.orchestra.run.smithi032.stdout: "req_rename_latency": { 2024-04-30T08:45:17.209 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:17.209 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:17.209 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:17.209 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.209 INFO:teuthology.orchestra.run.smithi032.stdout: "req_renamesnap_latency": { 2024-04-30T08:45:17.209 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:17.210 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:17.210 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:17.210 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.210 INFO:teuthology.orchestra.run.smithi032.stdout: "req_snapdiff_latency": { 2024-04-30T08:45:17.210 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:17.210 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:17.210 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:17.210 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.210 INFO:teuthology.orchestra.run.smithi032.stdout: "req_rmdir_latency": { 2024-04-30T08:45:17.210 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:17.210 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:17.210 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:17.210 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.210 INFO:teuthology.orchestra.run.smithi032.stdout: "req_rmsnap_latency": { 2024-04-30T08:45:17.211 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:17.211 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:17.211 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:17.211 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.211 INFO:teuthology.orchestra.run.smithi032.stdout: "req_rmxattr_latency": { 2024-04-30T08:45:17.211 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:17.211 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:17.211 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:17.211 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.211 INFO:teuthology.orchestra.run.smithi032.stdout: "req_setattr_latency": { 2024-04-30T08:45:17.211 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:17.211 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:17.211 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:17.211 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.212 INFO:teuthology.orchestra.run.smithi032.stdout: "req_setdirlayout_latency": { 2024-04-30T08:45:17.212 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:17.212 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:17.212 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:17.212 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.212 INFO:teuthology.orchestra.run.smithi032.stdout: "req_setfilelock_latency": { 2024-04-30T08:45:17.212 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:17.212 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:17.212 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:17.212 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.212 INFO:teuthology.orchestra.run.smithi032.stdout: "req_setlayout_latency": { 2024-04-30T08:45:17.212 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:17.212 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:17.213 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:17.213 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.213 INFO:teuthology.orchestra.run.smithi032.stdout: "req_setxattr_latency": { 2024-04-30T08:45:17.213 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:17.213 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:17.213 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:17.213 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.213 INFO:teuthology.orchestra.run.smithi032.stdout: "req_symlink_latency": { 2024-04-30T08:45:17.213 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:17.213 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:17.213 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:17.213 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.213 INFO:teuthology.orchestra.run.smithi032.stdout: "req_unlink_latency": { 2024-04-30T08:45:17.213 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:17.214 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:17.214 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:17.214 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.214 INFO:teuthology.orchestra.run.smithi032.stdout: "cap_revoke_eviction": 0, 2024-04-30T08:45:17.214 INFO:teuthology.orchestra.run.smithi032.stdout: "cap_acquisition_throttle": 0, 2024-04-30T08:45:17.214 INFO:teuthology.orchestra.run.smithi032.stdout: "req_getvxattr_latency": { 2024-04-30T08:45:17.214 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:17.214 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:17.214 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:17.214 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:45:17.214 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.214 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_sessions": { 2024-04-30T08:45:17.214 INFO:teuthology.orchestra.run.smithi032.stdout: "session_count": 1, 2024-04-30T08:45:17.215 INFO:teuthology.orchestra.run.smithi032.stdout: "session_add": 2, 2024-04-30T08:45:17.215 INFO:teuthology.orchestra.run.smithi032.stdout: "session_remove": 1, 2024-04-30T08:45:17.215 INFO:teuthology.orchestra.run.smithi032.stdout: "sessions_open": 1, 2024-04-30T08:45:17.215 INFO:teuthology.orchestra.run.smithi032.stdout: "sessions_stale": 0, 2024-04-30T08:45:17.215 INFO:teuthology.orchestra.run.smithi032.stdout: "total_load": 1, 2024-04-30T08:45:17.215 INFO:teuthology.orchestra.run.smithi032.stdout: "average_load": 1, 2024-04-30T08:45:17.215 INFO:teuthology.orchestra.run.smithi032.stdout: "avg_session_uptime": 594, 2024-04-30T08:45:17.215 INFO:teuthology.orchestra.run.smithi032.stdout: "mdthresh_evicted": 0 2024-04-30T08:45:17.215 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.215 INFO:teuthology.orchestra.run.smithi032.stdout: "mempool": { 2024-04-30T08:45:17.215 INFO:teuthology.orchestra.run.smithi032.stdout: "bloom_filter_bytes": 0, 2024-04-30T08:45:17.215 INFO:teuthology.orchestra.run.smithi032.stdout: "bloom_filter_items": 0, 2024-04-30T08:45:17.215 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_alloc_bytes": 0, 2024-04-30T08:45:17.216 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_alloc_items": 0, 2024-04-30T08:45:17.216 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_data_bytes": 0, 2024-04-30T08:45:17.216 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_data_items": 0, 2024-04-30T08:45:17.216 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-30T08:45:17.216 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_onode_items": 0, 2024-04-30T08:45:17.216 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-30T08:45:17.216 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_meta_items": 0, 2024-04-30T08:45:17.216 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_other_bytes": 0, 2024-04-30T08:45:17.216 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_other_items": 0, 2024-04-30T08:45:17.216 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-30T08:45:17.216 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_buffer_items": 0, 2024-04-30T08:45:17.216 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_extent_bytes": 0, 2024-04-30T08:45:17.216 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_extent_items": 0, 2024-04-30T08:45:17.216 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_blob_bytes": 0, 2024-04-30T08:45:17.217 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_blob_items": 0, 2024-04-30T08:45:17.217 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-30T08:45:17.217 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_shared_blob_items": 0, 2024-04-30T08:45:17.217 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-30T08:45:17.217 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_inline_bl_items": 0, 2024-04-30T08:45:17.217 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_bytes": 0, 2024-04-30T08:45:17.217 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_items": 0, 2024-04-30T08:45:17.217 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_txc_bytes": 0, 2024-04-30T08:45:17.217 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_txc_items": 0, 2024-04-30T08:45:17.217 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-30T08:45:17.217 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_deferred_items": 0, 2024-04-30T08:45:17.217 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_bytes": 0, 2024-04-30T08:45:17.217 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_items": 0, 2024-04-30T08:45:17.217 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_bytes": 0, 2024-04-30T08:45:17.218 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_items": 0, 2024-04-30T08:45:17.218 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_reader_bytes": 0, 2024-04-30T08:45:17.218 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_reader_items": 0, 2024-04-30T08:45:17.218 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_writer_bytes": 0, 2024-04-30T08:45:17.218 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_writer_items": 0, 2024-04-30T08:45:17.218 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_anon_bytes": 241945, 2024-04-30T08:45:17.218 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_anon_items": 96, 2024-04-30T08:45:17.218 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_meta_bytes": 0, 2024-04-30T08:45:17.218 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_meta_items": 0, 2024-04-30T08:45:17.218 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_bytes": 0, 2024-04-30T08:45:17.218 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_items": 0, 2024-04-30T08:45:17.218 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mapbl_bytes": 0, 2024-04-30T08:45:17.219 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mapbl_items": 0, 2024-04-30T08:45:17.219 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pglog_bytes": 0, 2024-04-30T08:45:17.219 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pglog_items": 0, 2024-04-30T08:45:17.219 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_bytes": 9088, 2024-04-30T08:45:17.219 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_items": 175, 2024-04-30T08:45:17.219 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_mapping_bytes": 0, 2024-04-30T08:45:17.219 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_mapping_items": 0, 2024-04-30T08:45:17.219 INFO:teuthology.orchestra.run.smithi032.stdout: "pgmap_bytes": 0, 2024-04-30T08:45:17.219 INFO:teuthology.orchestra.run.smithi032.stdout: "pgmap_items": 0, 2024-04-30T08:45:17.219 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_co_bytes": 76733, 2024-04-30T08:45:17.219 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_co_items": 309, 2024-04-30T08:45:17.219 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_1_bytes": 0, 2024-04-30T08:45:17.219 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_1_items": 0, 2024-04-30T08:45:17.219 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_2_bytes": 0, 2024-04-30T08:45:17.220 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_2_items": 0 2024-04-30T08:45:17.220 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.220 INFO:teuthology.orchestra.run.smithi032.stdout: "objecter": { 2024-04-30T08:45:17.220 INFO:teuthology.orchestra.run.smithi032.stdout: "op_active": 0, 2024-04-30T08:45:17.220 INFO:teuthology.orchestra.run.smithi032.stdout: "op_laggy": 0, 2024-04-30T08:45:17.220 INFO:teuthology.orchestra.run.smithi032.stdout: "op_send": 40, 2024-04-30T08:45:17.220 INFO:teuthology.orchestra.run.smithi032.stdout: "op_send_bytes": 11732, 2024-04-30T08:45:17.220 INFO:teuthology.orchestra.run.smithi032.stdout: "op_resend": 33, 2024-04-30T08:45:17.220 INFO:teuthology.orchestra.run.smithi032.stdout: "op_reply": 40, 2024-04-30T08:45:17.220 INFO:teuthology.orchestra.run.smithi032.stdout: "op_latency": { 2024-04-30T08:45:17.220 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 40, 2024-04-30T08:45:17.220 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.320993617, 2024-04-30T08:45:17.220 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.008024840 2024-04-30T08:45:17.221 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.221 INFO:teuthology.orchestra.run.smithi032.stdout: "op_inflight": 0, 2024-04-30T08:45:17.221 INFO:teuthology.orchestra.run.smithi032.stdout: "oplen_avg": { 2024-04-30T08:45:17.221 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 40, 2024-04-30T08:45:17.221 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 62 2024-04-30T08:45:17.221 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.221 INFO:teuthology.orchestra.run.smithi032.stdout: "op": 40, 2024-04-30T08:45:17.221 INFO:teuthology.orchestra.run.smithi032.stdout: "op_r": 0, 2024-04-30T08:45:17.221 INFO:teuthology.orchestra.run.smithi032.stdout: "op_w": 40, 2024-04-30T08:45:17.221 INFO:teuthology.orchestra.run.smithi032.stdout: "op_rmw": 0, 2024-04-30T08:45:17.221 INFO:teuthology.orchestra.run.smithi032.stdout: "op_pg": 0, 2024-04-30T08:45:17.221 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_stat": 0, 2024-04-30T08:45:17.221 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_create": 10, 2024-04-30T08:45:17.221 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_read": 0, 2024-04-30T08:45:17.222 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_write": 4, 2024-04-30T08:45:17.222 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_writefull": 7, 2024-04-30T08:45:17.222 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_writesame": 0, 2024-04-30T08:45:17.222 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_append": 0, 2024-04-30T08:45:17.222 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_zero": 0, 2024-04-30T08:45:17.222 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_truncate": 0, 2024-04-30T08:45:17.222 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_delete": 6, 2024-04-30T08:45:17.222 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_mapext": 0, 2024-04-30T08:45:17.222 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_sparse_read": 0, 2024-04-30T08:45:17.222 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_clonerange": 0, 2024-04-30T08:45:17.222 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_getxattr": 0, 2024-04-30T08:45:17.222 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_setxattr": 20, 2024-04-30T08:45:17.222 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_cmpxattr": 0, 2024-04-30T08:45:17.223 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_rmxattr": 0, 2024-04-30T08:45:17.223 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_resetxattrs": 0, 2024-04-30T08:45:17.223 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_call": 0, 2024-04-30T08:45:17.223 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_watch": 0, 2024-04-30T08:45:17.223 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_notify": 0, 2024-04-30T08:45:17.223 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_src_cmpxattr": 0, 2024-04-30T08:45:17.223 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_pgls": 0, 2024-04-30T08:45:17.223 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_pgls_filter": 0, 2024-04-30T08:45:17.223 INFO:teuthology.orchestra.run.smithi032.stdout: "osdop_other": 0, 2024-04-30T08:45:17.223 INFO:teuthology.orchestra.run.smithi032.stdout: "linger_active": 0, 2024-04-30T08:45:17.223 INFO:teuthology.orchestra.run.smithi032.stdout: "linger_send": 0, 2024-04-30T08:45:17.223 INFO:teuthology.orchestra.run.smithi032.stdout: "linger_resend": 0, 2024-04-30T08:45:17.223 INFO:teuthology.orchestra.run.smithi032.stdout: "linger_ping": 0, 2024-04-30T08:45:17.223 INFO:teuthology.orchestra.run.smithi032.stdout: "poolop_active": 0, 2024-04-30T08:45:17.224 INFO:teuthology.orchestra.run.smithi032.stdout: "poolop_send": 0, 2024-04-30T08:45:17.224 INFO:teuthology.orchestra.run.smithi032.stdout: "poolop_resend": 0, 2024-04-30T08:45:17.224 INFO:teuthology.orchestra.run.smithi032.stdout: "poolstat_active": 0, 2024-04-30T08:45:17.224 INFO:teuthology.orchestra.run.smithi032.stdout: "poolstat_send": 0, 2024-04-30T08:45:17.224 INFO:teuthology.orchestra.run.smithi032.stdout: "poolstat_resend": 0, 2024-04-30T08:45:17.224 INFO:teuthology.orchestra.run.smithi032.stdout: "statfs_active": 0, 2024-04-30T08:45:17.224 INFO:teuthology.orchestra.run.smithi032.stdout: "statfs_send": 0, 2024-04-30T08:45:17.224 INFO:teuthology.orchestra.run.smithi032.stdout: "statfs_resend": 0, 2024-04-30T08:45:17.224 INFO:teuthology.orchestra.run.smithi032.stdout: "command_active": 0, 2024-04-30T08:45:17.224 INFO:teuthology.orchestra.run.smithi032.stdout: "command_send": 0, 2024-04-30T08:45:17.224 INFO:teuthology.orchestra.run.smithi032.stdout: "command_resend": 0, 2024-04-30T08:45:17.224 INFO:teuthology.orchestra.run.smithi032.stdout: "map_epoch": 85, 2024-04-30T08:45:17.224 INFO:teuthology.orchestra.run.smithi032.stdout: "map_full": 0, 2024-04-30T08:45:17.225 INFO:teuthology.orchestra.run.smithi032.stdout: "map_inc": 4, 2024-04-30T08:45:17.225 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_sessions": 9, 2024-04-30T08:45:17.225 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_session_open": 9, 2024-04-30T08:45:17.225 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_session_close": 0, 2024-04-30T08:45:17.225 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_laggy": 0, 2024-04-30T08:45:17.225 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_wr": 14, 2024-04-30T08:45:17.225 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_rd": 0, 2024-04-30T08:45:17.225 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_del": 1 2024-04-30T08:45:17.225 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.225 INFO:teuthology.orchestra.run.smithi032.stdout: "oft": { 2024-04-30T08:45:17.225 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_total_objs": 1, 2024-04-30T08:45:17.225 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_total_kv_pairs": 0, 2024-04-30T08:45:17.225 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_total_updates": 0, 2024-04-30T08:45:17.225 INFO:teuthology.orchestra.run.smithi032.stdout: "omap_total_removes": 0 2024-04-30T08:45:17.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.226 INFO:teuthology.orchestra.run.smithi032.stdout: "osd-slow-ops": { 2024-04-30T08:45:17.226 INFO:teuthology.orchestra.run.smithi032.stdout: "slow_ops_count": 0 2024-04-30T08:45:17.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.226 INFO:teuthology.orchestra.run.smithi032.stdout: "purge_queue": { 2024-04-30T08:45:17.226 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executing_ops": 0, 2024-04-30T08:45:17.226 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executing_ops_high_water": 0, 2024-04-30T08:45:17.226 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executing": 0, 2024-04-30T08:45:17.226 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executing_high_water": 0, 2024-04-30T08:45:17.226 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executed_ops": 0, 2024-04-30T08:45:17.226 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_executed": 0, 2024-04-30T08:45:17.226 INFO:teuthology.orchestra.run.smithi032.stdout: "pq_item_in_journal": 0 2024-04-30T08:45:17.226 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.227 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-30T08:45:17.227 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-30T08:45:17.227 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 104857600, 2024-04-30T08:45:17.227 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-30T08:45:17.227 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 1170, 2024-04-30T08:45:17.227 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 1678573, 2024-04-30T08:45:17.227 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:17.227 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 1170, 2024-04-30T08:45:17.227 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 0, 2024-04-30T08:45:17.227 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 0, 2024-04-30T08:45:17.227 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 1170, 2024-04-30T08:45:17.227 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 1678573, 2024-04-30T08:45:17.227 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-30T08:45:17.227 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:17.228 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:17.228 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:17.228 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:45:17.228 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.228 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-objecter_bytes": { 2024-04-30T08:45:17.228 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-30T08:45:17.228 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 104857600, 2024-04-30T08:45:17.228 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-30T08:45:17.228 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 0, 2024-04-30T08:45:17.228 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 0, 2024-04-30T08:45:17.228 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:17.228 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 0, 2024-04-30T08:45:17.228 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 40, 2024-04-30T08:45:17.229 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 11732, 2024-04-30T08:45:17.229 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 40, 2024-04-30T08:45:17.229 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 11732, 2024-04-30T08:45:17.229 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-30T08:45:17.229 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:17.229 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:17.229 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:17.229 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:45:17.229 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.229 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-objecter_ops": { 2024-04-30T08:45:17.229 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-30T08:45:17.229 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 1024, 2024-04-30T08:45:17.229 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-30T08:45:17.230 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 0, 2024-04-30T08:45:17.230 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 0, 2024-04-30T08:45:17.230 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:17.230 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 0, 2024-04-30T08:45:17.230 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 40, 2024-04-30T08:45:17.230 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 40, 2024-04-30T08:45:17.230 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 40, 2024-04-30T08:45:17.230 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 40, 2024-04-30T08:45:17.230 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-30T08:45:17.230 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:17.230 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:17.230 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:17.230 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:45:17.230 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.231 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-write_buf_throttle": { 2024-04-30T08:45:17.231 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-30T08:45:17.231 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 3758096384, 2024-04-30T08:45:17.231 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-30T08:45:17.231 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 0, 2024-04-30T08:45:17.231 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 0, 2024-04-30T08:45:17.231 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:17.231 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 0, 2024-04-30T08:45:17.231 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 0, 2024-04-30T08:45:17.231 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 0, 2024-04-30T08:45:17.231 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 0, 2024-04-30T08:45:17.231 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 0, 2024-04-30T08:45:17.231 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-30T08:45:17.232 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:17.232 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:17.232 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:17.232 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:45:17.232 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:17.232 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-write_buf_throttle-0x556f4cc52680": { 2024-04-30T08:45:17.232 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-30T08:45:17.232 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 3758096384, 2024-04-30T08:45:17.232 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-30T08:45:17.232 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 5, 2024-04-30T08:45:17.232 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 1430, 2024-04-30T08:45:17.232 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:17.232 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 5, 2024-04-30T08:45:17.232 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 0, 2024-04-30T08:45:17.233 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 0, 2024-04-30T08:45:17.233 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 4, 2024-04-30T08:45:17.233 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 1430, 2024-04-30T08:45:17.233 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-30T08:45:17.233 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:17.233 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:17.233 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:17.233 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:45:17.233 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:45:17.233 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-30T08:45:18.195 INFO:tasks.check_counter:Daemon mds.g mds.exported=0 2024-04-30T08:45:18.196 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-04-30T08:45:18.196 INFO:tasks.check_counter:Daemon mds.g mds.dir_split=0 2024-04-30T08:45:18.196 DEBUG:tasks.check_counter:Getting stats from j 2024-04-30T08:45:18.196 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-30T08:45:18.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:18 smithi032 ceph-mon[28331]: pgmap v729: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-30T08:45:18.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:18 smithi071 ceph-mon[33385]: pgmap v729: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-30T08:45:18.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:18 smithi189 ceph-mon[33038]: pgmap v729: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 426 B/s wr, 9 op/s 2024-04-30T08:45:18.676 INFO:teuthology.orchestra.run.smithi032.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.a/config 2024-04-30T08:45:20.434 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:20 smithi032 ceph-mon[28331]: pgmap v730: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-30T08:45:20.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:20 smithi071 ceph-mon[33385]: pgmap v730: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-30T08:45:20.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:20 smithi189 ceph-mon[33038]: pgmap v730: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-30T08:45:21.233 INFO:teuthology.orchestra.run.smithi032.stdout:{ 2024-04-30T08:45:21.233 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-0": { 2024-04-30T08:45:21.233 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 0, 2024-04-30T08:45:21.233 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 2, 2024-04-30T08:45:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 0, 2024-04-30T08:45:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 0, 2024-04-30T08:45:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 4, 2024-04-30T08:45:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 0, 2024-04-30T08:45:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.004155102, 2024-04-30T08:45:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.000000000, 2024-04-30T08:45:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.001259576, 2024-04-30T08:45:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-30T08:45:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:21.234 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-30T08:45:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 0 2024-04-30T08:45:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-1": { 2024-04-30T08:45:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 5, 2024-04-30T08:45:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 175, 2024-04-30T08:45:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 3152, 2024-04-30T08:45:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 109456, 2024-04-30T08:45:21.235 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 3, 2024-04-30T08:45:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 2, 2024-04-30T08:45:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.028876224, 2024-04-30T08:45:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.023973506, 2024-04-30T08:45:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.002192577, 2024-04-30T08:45:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-30T08:45:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 174, 2024-04-30T08:45:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.032174915, 2024-04-30T08:45:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000184913 2024-04-30T08:45:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:21.236 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-04-30T08:45:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 109456 2024-04-30T08:45:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "AsyncMessenger::Worker-2": { 2024-04-30T08:45:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_messages": 231, 2024-04-30T08:45:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages": 226, 2024-04-30T08:45:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_bytes": 805312, 2024-04-30T08:45:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_bytes": 139424, 2024-04-30T08:45:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_created_connections": 2, 2024-04-30T08:45:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_active_connections": 1, 2024-04-30T08:45:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_total_time": 0.090011719, 2024-04-30T08:45:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_send_time": 0.037155673, 2024-04-30T08:45:21.237 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_recv_time": 0.053208221, 2024-04-30T08:45:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_running_fast_dispatch_time": 0.009698796, 2024-04-30T08:45:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 225, 2024-04-30T08:45:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.028564002, 2024-04-30T08:45:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000126951 2024-04-30T08:45:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_recv_encrypted_bytes": 805312, 2024-04-30T08:45:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: "msgr_send_encrypted_bytes": 139424 2024-04-30T08:45:21.238 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "cct": { 2024-04-30T08:45:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "total_workers": 0, 2024-04-30T08:45:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "unhealthy_workers": 0 2024-04-30T08:45:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "mempool": { 2024-04-30T08:45:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "bloom_filter_bytes": 0, 2024-04-30T08:45:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "bloom_filter_items": 0, 2024-04-30T08:45:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_alloc_bytes": 0, 2024-04-30T08:45:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_alloc_items": 0, 2024-04-30T08:45:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_data_bytes": 0, 2024-04-30T08:45:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_data_items": 0, 2024-04-30T08:45:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-30T08:45:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_onode_items": 0, 2024-04-30T08:45:21.239 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-30T08:45:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_meta_items": 0, 2024-04-30T08:45:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_other_bytes": 0, 2024-04-30T08:45:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_other_items": 0, 2024-04-30T08:45:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-30T08:45:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_cache_buffer_items": 0, 2024-04-30T08:45:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_extent_bytes": 0, 2024-04-30T08:45:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_extent_items": 0, 2024-04-30T08:45:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_blob_bytes": 0, 2024-04-30T08:45:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_blob_items": 0, 2024-04-30T08:45:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-30T08:45:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_shared_blob_items": 0, 2024-04-30T08:45:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-30T08:45:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_inline_bl_items": 0, 2024-04-30T08:45:21.240 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_bytes": 0, 2024-04-30T08:45:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_fsck_items": 0, 2024-04-30T08:45:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_txc_bytes": 0, 2024-04-30T08:45:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_txc_items": 0, 2024-04-30T08:45:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-30T08:45:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_deferred_items": 0, 2024-04-30T08:45:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_bytes": 0, 2024-04-30T08:45:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "bluestore_writing_items": 0, 2024-04-30T08:45:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_bytes": 0, 2024-04-30T08:45:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_items": 0, 2024-04-30T08:45:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_reader_bytes": 0, 2024-04-30T08:45:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_reader_items": 0, 2024-04-30T08:45:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_writer_bytes": 0, 2024-04-30T08:45:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "bluefs_file_writer_items": 0, 2024-04-30T08:45:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_anon_bytes": 151029, 2024-04-30T08:45:21.241 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_anon_items": 31, 2024-04-30T08:45:21.242 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_meta_bytes": 0, 2024-04-30T08:45:21.242 INFO:teuthology.orchestra.run.smithi032.stdout: "buffer_meta_items": 0, 2024-04-30T08:45:21.242 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_bytes": 0, 2024-04-30T08:45:21.242 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_items": 0, 2024-04-30T08:45:21.242 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mapbl_bytes": 0, 2024-04-30T08:45:21.242 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_mapbl_items": 0, 2024-04-30T08:45:21.242 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pglog_bytes": 0, 2024-04-30T08:45:21.242 INFO:teuthology.orchestra.run.smithi032.stdout: "osd_pglog_items": 0, 2024-04-30T08:45:21.242 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_bytes": 0, 2024-04-30T08:45:21.242 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_items": 0, 2024-04-30T08:45:21.242 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_mapping_bytes": 0, 2024-04-30T08:45:21.242 INFO:teuthology.orchestra.run.smithi032.stdout: "osdmap_mapping_items": 0, 2024-04-30T08:45:21.242 INFO:teuthology.orchestra.run.smithi032.stdout: "pgmap_bytes": 0, 2024-04-30T08:45:21.243 INFO:teuthology.orchestra.run.smithi032.stdout: "pgmap_items": 0, 2024-04-30T08:45:21.243 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_co_bytes": 928, 2024-04-30T08:45:21.243 INFO:teuthology.orchestra.run.smithi032.stdout: "mds_co_items": 2, 2024-04-30T08:45:21.243 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_1_bytes": 0, 2024-04-30T08:45:21.243 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_1_items": 0, 2024-04-30T08:45:21.243 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_2_bytes": 0, 2024-04-30T08:45:21.243 INFO:teuthology.orchestra.run.smithi032.stdout: "unittest_2_items": 0 2024-04-30T08:45:21.243 INFO:teuthology.orchestra.run.smithi032.stdout: }, 2024-04-30T08:45:21.243 INFO:teuthology.orchestra.run.smithi032.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-30T08:45:21.243 INFO:teuthology.orchestra.run.smithi032.stdout: "val": 0, 2024-04-30T08:45:21.243 INFO:teuthology.orchestra.run.smithi032.stdout: "max": 104857600, 2024-04-30T08:45:21.243 INFO:teuthology.orchestra.run.smithi032.stdout: "get_started": 0, 2024-04-30T08:45:21.243 INFO:teuthology.orchestra.run.smithi032.stdout: "get": 232, 2024-04-30T08:45:21.243 INFO:teuthology.orchestra.run.smithi032.stdout: "get_sum": 772684, 2024-04-30T08:45:21.243 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:21.244 INFO:teuthology.orchestra.run.smithi032.stdout: "get_or_fail_success": 232, 2024-04-30T08:45:21.244 INFO:teuthology.orchestra.run.smithi032.stdout: "take": 0, 2024-04-30T08:45:21.244 INFO:teuthology.orchestra.run.smithi032.stdout: "take_sum": 0, 2024-04-30T08:45:21.244 INFO:teuthology.orchestra.run.smithi032.stdout: "put": 232, 2024-04-30T08:45:21.244 INFO:teuthology.orchestra.run.smithi032.stdout: "put_sum": 772684, 2024-04-30T08:45:21.244 INFO:teuthology.orchestra.run.smithi032.stdout: "wait": { 2024-04-30T08:45:21.244 INFO:teuthology.orchestra.run.smithi032.stdout: "avgcount": 0, 2024-04-30T08:45:21.244 INFO:teuthology.orchestra.run.smithi032.stdout: "sum": 0.000000000, 2024-04-30T08:45:21.244 INFO:teuthology.orchestra.run.smithi032.stdout: "avgtime": 0.000000000 2024-04-30T08:45:21.244 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:45:21.244 INFO:teuthology.orchestra.run.smithi032.stdout: } 2024-04-30T08:45:21.244 INFO:teuthology.orchestra.run.smithi032.stdout:} 2024-04-30T08:45:21.988 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-30T08:45:21.988 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-30T08:45:21.988 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-04-30T08:45:21.988 DEBUG:tasks.check_counter:Getting stats from b 2024-04-30T08:45:21.988 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-30T08:45:22.327 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.b/config 2024-04-30T08:45:22.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:22 smithi032 ceph-mon[28331]: pgmap v731: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-30T08:45:22.450 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:22 smithi071 ceph-mon[33385]: pgmap v731: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-30T08:45:22.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:22 smithi189 ceph-mon[33038]: pgmap v731: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 426 B/s wr, 10 op/s 2024-04-30T08:45:24.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:24 smithi032 ceph-mon[28331]: pgmap v732: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 682 B/s wr, 9 op/s 2024-04-30T08:45:24.455 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:24 smithi071 ceph-mon[33385]: pgmap v732: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 682 B/s wr, 9 op/s 2024-04-30T08:45:24.456 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-30T08:45:24.456 INFO:teuthology.orchestra.run.smithi071.stdout: "AsyncMessenger::Worker-0": { 2024-04-30T08:45:24.456 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_messages": 4999, 2024-04-30T08:45:24.456 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_messages": 4763, 2024-04-30T08:45:24.457 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_bytes": 2838051, 2024-04-30T08:45:24.457 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_bytes": 6371532, 2024-04-30T08:45:24.457 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_created_connections": 43, 2024-04-30T08:45:24.457 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_active_connections": 6, 2024-04-30T08:45:24.457 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_total_time": 0.408299565, 2024-04-30T08:45:24.457 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_send_time": 0.131925693, 2024-04-30T08:45:24.457 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_recv_time": 0.156630661, 2024-04-30T08:45:24.457 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_fast_dispatch_time": 0.110544353, 2024-04-30T08:45:24.457 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:24.457 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 4756, 2024-04-30T08:45:24.457 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 2.268289303, 2024-04-30T08:45:24.457 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000476932 2024-04-30T08:45:24.457 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.458 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:24.458 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 2027, 2024-04-30T08:45:24.458 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000440476, 2024-04-30T08:45:24.458 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000217 2024-04-30T08:45:24.458 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.458 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_encrypted_bytes": 1572192, 2024-04-30T08:45:24.458 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_encrypted_bytes": 154272 2024-04-30T08:45:24.458 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.458 INFO:teuthology.orchestra.run.smithi071.stdout: "AsyncMessenger::Worker-1": { 2024-04-30T08:45:24.458 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_messages": 4852, 2024-04-30T08:45:24.458 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_messages": 4946, 2024-04-30T08:45:24.459 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_bytes": 1534487, 2024-04-30T08:45:24.459 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_bytes": 13738945, 2024-04-30T08:45:24.459 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_created_connections": 9, 2024-04-30T08:45:24.459 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_active_connections": 5, 2024-04-30T08:45:24.459 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_total_time": 0.402415936, 2024-04-30T08:45:24.459 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_send_time": 0.143891463, 2024-04-30T08:45:24.459 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_recv_time": 0.133795363, 2024-04-30T08:45:24.459 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_fast_dispatch_time": 0.115189065, 2024-04-30T08:45:24.459 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:24.459 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 4950, 2024-04-30T08:45:24.459 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.768682579, 2024-04-30T08:45:24.459 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000155289 2024-04-30T08:45:24.459 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.460 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:24.460 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 3939, 2024-04-30T08:45:24.460 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000804858, 2024-04-30T08:45:24.460 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000204 2024-04-30T08:45:24.460 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.460 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-30T08:45:24.460 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_encrypted_bytes": 382016 2024-04-30T08:45:24.460 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.460 INFO:teuthology.orchestra.run.smithi071.stdout: "AsyncMessenger::Worker-2": { 2024-04-30T08:45:24.460 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_messages": 19475, 2024-04-30T08:45:24.460 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_messages": 22917, 2024-04-30T08:45:24.460 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_bytes": 6510673, 2024-04-30T08:45:24.460 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_bytes": 20817803, 2024-04-30T08:45:24.461 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_created_connections": 52, 2024-04-30T08:45:24.461 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_active_connections": 5, 2024-04-30T08:45:24.461 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_total_time": 1.202693043, 2024-04-30T08:45:24.461 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_send_time": 0.469812097, 2024-04-30T08:45:24.461 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_recv_time": 0.826225092, 2024-04-30T08:45:24.461 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_fast_dispatch_time": 0.160228411, 2024-04-30T08:45:24.461 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:24.461 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 22915, 2024-04-30T08:45:24.461 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 1.711691585, 2024-04-30T08:45:24.461 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000074697 2024-04-30T08:45:24.461 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.461 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:24.461 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 17034, 2024-04-30T08:45:24.462 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.004423361, 2024-04-30T08:45:24.462 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000259 2024-04-30T08:45:24.462 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.462 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_encrypted_bytes": 97472, 2024-04-30T08:45:24.462 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_encrypted_bytes": 31200 2024-04-30T08:45:24.462 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.462 INFO:teuthology.orchestra.run.smithi071.stdout: "cct": { 2024-04-30T08:45:24.462 INFO:teuthology.orchestra.run.smithi071.stdout: "total_workers": 1, 2024-04-30T08:45:24.462 INFO:teuthology.orchestra.run.smithi071.stdout: "unhealthy_workers": 0 2024-04-30T08:45:24.462 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.462 INFO:teuthology.orchestra.run.smithi071.stdout: "finisher-MDSRank": { 2024-04-30T08:45:24.462 INFO:teuthology.orchestra.run.smithi071.stdout: "queue_len": 0, 2024-04-30T08:45:24.462 INFO:teuthology.orchestra.run.smithi071.stdout: "complete_latency": { 2024-04-30T08:45:24.463 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 4055, 2024-04-30T08:45:24.463 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 5.161753923, 2024-04-30T08:45:24.463 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.001272935 2024-04-30T08:45:24.463 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:24.463 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.463 INFO:teuthology.orchestra.run.smithi071.stdout: "finisher-PurgeQueue": { 2024-04-30T08:45:24.463 INFO:teuthology.orchestra.run.smithi071.stdout: "queue_len": 0, 2024-04-30T08:45:24.463 INFO:teuthology.orchestra.run.smithi071.stdout: "complete_latency": { 2024-04-30T08:45:24.463 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 4807, 2024-04-30T08:45:24.463 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.241803613, 2024-04-30T08:45:24.463 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000050302 2024-04-30T08:45:24.463 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:24.463 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.463 INFO:teuthology.orchestra.run.smithi071.stdout: "mds": { 2024-04-30T08:45:24.464 INFO:teuthology.orchestra.run.smithi071.stdout: "request": 2368, 2024-04-30T08:45:24.464 INFO:teuthology.orchestra.run.smithi071.stdout: "reply": 2364, 2024-04-30T08:45:24.464 INFO:teuthology.orchestra.run.smithi071.stdout: "reply_latency": { 2024-04-30T08:45:24.464 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 2364, 2024-04-30T08:45:24.464 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 17.950921973, 2024-04-30T08:45:24.464 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.007593452 2024-04-30T08:45:24.464 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.464 INFO:teuthology.orchestra.run.smithi071.stdout: "slow_reply": 0, 2024-04-30T08:45:24.464 INFO:teuthology.orchestra.run.smithi071.stdout: "forward": 4, 2024-04-30T08:45:24.464 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_fetch_complete": 3, 2024-04-30T08:45:24.464 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_fetch_keys": 2, 2024-04-30T08:45:24.464 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_commit": 267, 2024-04-30T08:45:24.464 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_split": 1, 2024-04-30T08:45:24.464 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_merge": 0, 2024-04-30T08:45:24.464 INFO:teuthology.orchestra.run.smithi071.stdout: "inodes": 27, 2024-04-30T08:45:24.465 INFO:teuthology.orchestra.run.smithi071.stdout: "inodes_top": 6, 2024-04-30T08:45:24.465 INFO:teuthology.orchestra.run.smithi071.stdout: "inodes_bottom": 9, 2024-04-30T08:45:24.465 INFO:teuthology.orchestra.run.smithi071.stdout: "inodes_pin_tail": 12, 2024-04-30T08:45:24.465 INFO:teuthology.orchestra.run.smithi071.stdout: "inodes_pinned": 18, 2024-04-30T08:45:24.465 INFO:teuthology.orchestra.run.smithi071.stdout: "inodes_expired": 12, 2024-04-30T08:45:24.465 INFO:teuthology.orchestra.run.smithi071.stdout: "inodes_with_caps": 2, 2024-04-30T08:45:24.465 INFO:teuthology.orchestra.run.smithi071.stdout: "caps": 2, 2024-04-30T08:45:24.465 INFO:teuthology.orchestra.run.smithi071.stdout: "subtrees": 2, 2024-04-30T08:45:24.465 INFO:teuthology.orchestra.run.smithi071.stdout: "traverse": 2964, 2024-04-30T08:45:24.465 INFO:teuthology.orchestra.run.smithi071.stdout: "traverse_hit": 2808, 2024-04-30T08:45:24.465 INFO:teuthology.orchestra.run.smithi071.stdout: "traverse_forward": 0, 2024-04-30T08:45:24.465 INFO:teuthology.orchestra.run.smithi071.stdout: "traverse_discover": 0, 2024-04-30T08:45:24.465 INFO:teuthology.orchestra.run.smithi071.stdout: "traverse_dir_fetch": 0, 2024-04-30T08:45:24.466 INFO:teuthology.orchestra.run.smithi071.stdout: "traverse_remote_ino": 0, 2024-04-30T08:45:24.466 INFO:teuthology.orchestra.run.smithi071.stdout: "traverse_lock": 48, 2024-04-30T08:45:24.466 INFO:teuthology.orchestra.run.smithi071.stdout: "load_cent": 0, 2024-04-30T08:45:24.466 INFO:teuthology.orchestra.run.smithi071.stdout: "q": 0, 2024-04-30T08:45:24.466 INFO:teuthology.orchestra.run.smithi071.stdout: "exported": 4, 2024-04-30T08:45:24.466 INFO:teuthology.orchestra.run.smithi071.stdout: "exported_inodes": 8, 2024-04-30T08:45:24.466 INFO:teuthology.orchestra.run.smithi071.stdout: "imported": 4, 2024-04-30T08:45:24.466 INFO:teuthology.orchestra.run.smithi071.stdout: "imported_inodes": 0, 2024-04-30T08:45:24.466 INFO:teuthology.orchestra.run.smithi071.stdout: "openino_dir_fetch": 0, 2024-04-30T08:45:24.466 INFO:teuthology.orchestra.run.smithi071.stdout: "openino_backtrace_fetch": 651, 2024-04-30T08:45:24.466 INFO:teuthology.orchestra.run.smithi071.stdout: "openino_peer_discover": 0, 2024-04-30T08:45:24.466 INFO:teuthology.orchestra.run.smithi071.stdout: "root_rfiles": 4, 2024-04-30T08:45:24.466 INFO:teuthology.orchestra.run.smithi071.stdout: "root_rbytes": 234, 2024-04-30T08:45:24.467 INFO:teuthology.orchestra.run.smithi071.stdout: "root_rsnaps": 0, 2024-04-30T08:45:24.467 INFO:teuthology.orchestra.run.smithi071.stdout: "scrub_backtrace_fetch": 651, 2024-04-30T08:45:24.467 INFO:teuthology.orchestra.run.smithi071.stdout: "scrub_set_tag": 0, 2024-04-30T08:45:24.467 INFO:teuthology.orchestra.run.smithi071.stdout: "scrub_backtrace_repaired": 0, 2024-04-30T08:45:24.467 INFO:teuthology.orchestra.run.smithi071.stdout: "scrub_inotable_repaired": 0, 2024-04-30T08:45:24.467 INFO:teuthology.orchestra.run.smithi071.stdout: "scrub_dir_inodes": 202, 2024-04-30T08:45:24.467 INFO:teuthology.orchestra.run.smithi071.stdout: "scrub_dir_base_inodes": 5, 2024-04-30T08:45:24.467 INFO:teuthology.orchestra.run.smithi071.stdout: "scrub_dirfrag_rstats": 197, 2024-04-30T08:45:24.467 INFO:teuthology.orchestra.run.smithi071.stdout: "scrub_file_inodes": 449, 2024-04-30T08:45:24.467 INFO:teuthology.orchestra.run.smithi071.stdout: "handle_inode_file_caps": 9, 2024-04-30T08:45:24.467 INFO:teuthology.orchestra.run.smithi071.stdout: "ceph_cap_op_revoke": 353, 2024-04-30T08:45:24.467 INFO:teuthology.orchestra.run.smithi071.stdout: "ceph_cap_op_grant": 3786, 2024-04-30T08:45:24.467 INFO:teuthology.orchestra.run.smithi071.stdout: "ceph_cap_op_trunc": 6, 2024-04-30T08:45:24.468 INFO:teuthology.orchestra.run.smithi071.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-30T08:45:24.468 INFO:teuthology.orchestra.run.smithi071.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-30T08:45:24.468 INFO:teuthology.orchestra.run.smithi071.stdout: "handle_client_caps": 12075, 2024-04-30T08:45:24.468 INFO:teuthology.orchestra.run.smithi071.stdout: "handle_client_caps_dirty": 10493, 2024-04-30T08:45:24.468 INFO:teuthology.orchestra.run.smithi071.stdout: "handle_client_cap_release": 96, 2024-04-30T08:45:24.468 INFO:teuthology.orchestra.run.smithi071.stdout: "process_request_cap_release": 1335 2024-04-30T08:45:24.468 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.468 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_cache": { 2024-04-30T08:45:24.468 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_update": 1, 2024-04-30T08:45:24.468 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_update_receipt": 0, 2024-04-30T08:45:24.468 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_try_discover": 0, 2024-04-30T08:45:24.468 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_send_discover": 0, 2024-04-30T08:45:24.468 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_handle_discover": 12, 2024-04-30T08:45:24.468 INFO:teuthology.orchestra.run.smithi071.stdout: "num_strays": 2, 2024-04-30T08:45:24.469 INFO:teuthology.orchestra.run.smithi071.stdout: "num_strays_delayed": 0, 2024-04-30T08:45:24.469 INFO:teuthology.orchestra.run.smithi071.stdout: "num_strays_enqueuing": 0, 2024-04-30T08:45:24.469 INFO:teuthology.orchestra.run.smithi071.stdout: "strays_created": 930, 2024-04-30T08:45:24.469 INFO:teuthology.orchestra.run.smithi071.stdout: "strays_enqueued": 867, 2024-04-30T08:45:24.469 INFO:teuthology.orchestra.run.smithi071.stdout: "strays_reintegrated": 61, 2024-04-30T08:45:24.469 INFO:teuthology.orchestra.run.smithi071.stdout: "strays_migrated": 0, 2024-04-30T08:45:24.469 INFO:teuthology.orchestra.run.smithi071.stdout: "num_recovering_processing": 0, 2024-04-30T08:45:24.469 INFO:teuthology.orchestra.run.smithi071.stdout: "num_recovering_enqueued": 0, 2024-04-30T08:45:24.469 INFO:teuthology.orchestra.run.smithi071.stdout: "num_recovering_prioritized": 0, 2024-04-30T08:45:24.469 INFO:teuthology.orchestra.run.smithi071.stdout: "recovery_started": 0, 2024-04-30T08:45:24.469 INFO:teuthology.orchestra.run.smithi071.stdout: "recovery_completed": 0, 2024-04-30T08:45:24.469 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_quiesce_path": 0, 2024-04-30T08:45:24.469 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_quiesce_inode": 0, 2024-04-30T08:45:24.469 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_enqueue_scrub": 5, 2024-04-30T08:45:24.470 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_exportdir": 4, 2024-04-30T08:45:24.470 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_flush": 0, 2024-04-30T08:45:24.470 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_fragmentdir": 1, 2024-04-30T08:45:24.470 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_fragstats": 0, 2024-04-30T08:45:24.470 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_inodestats": 0 2024-04-30T08:45:24.470 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.470 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_log": { 2024-04-30T08:45:24.470 INFO:teuthology.orchestra.run.smithi071.stdout: "evlrg": 0, 2024-04-30T08:45:24.470 INFO:teuthology.orchestra.run.smithi071.stdout: "evadd": 14542, 2024-04-30T08:45:24.470 INFO:teuthology.orchestra.run.smithi071.stdout: "evex": 12634, 2024-04-30T08:45:24.470 INFO:teuthology.orchestra.run.smithi071.stdout: "evtrm": 5770, 2024-04-30T08:45:24.470 INFO:teuthology.orchestra.run.smithi071.stdout: "ev": 8772, 2024-04-30T08:45:24.470 INFO:teuthology.orchestra.run.smithi071.stdout: "evexg": 0, 2024-04-30T08:45:24.471 INFO:teuthology.orchestra.run.smithi071.stdout: "evexd": 8750, 2024-04-30T08:45:24.471 INFO:teuthology.orchestra.run.smithi071.stdout: "segadd": 28, 2024-04-30T08:45:24.471 INFO:teuthology.orchestra.run.smithi071.stdout: "segex": 26, 2024-04-30T08:45:24.471 INFO:teuthology.orchestra.run.smithi071.stdout: "segtrm": 19, 2024-04-30T08:45:24.471 INFO:teuthology.orchestra.run.smithi071.stdout: "seg": 9, 2024-04-30T08:45:24.471 INFO:teuthology.orchestra.run.smithi071.stdout: "segmjr": 1, 2024-04-30T08:45:24.471 INFO:teuthology.orchestra.run.smithi071.stdout: "segexg": 0, 2024-04-30T08:45:24.471 INFO:teuthology.orchestra.run.smithi071.stdout: "segexd": 10, 2024-04-30T08:45:24.471 INFO:teuthology.orchestra.run.smithi071.stdout: "expos": 17025879, 2024-04-30T08:45:24.471 INFO:teuthology.orchestra.run.smithi071.stdout: "wrpos": 35220218, 2024-04-30T08:45:24.471 INFO:teuthology.orchestra.run.smithi071.stdout: "rdpos": 0, 2024-04-30T08:45:24.471 INFO:teuthology.orchestra.run.smithi071.stdout: "jlat": { 2024-04-30T08:45:24.471 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 1720, 2024-04-30T08:45:24.471 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 3061.575337680, 2024-04-30T08:45:24.472 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 1.779985661 2024-04-30T08:45:24.472 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.472 INFO:teuthology.orchestra.run.smithi071.stdout: "replayed": 0 2024-04-30T08:45:24.472 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.472 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_mem": { 2024-04-30T08:45:24.472 INFO:teuthology.orchestra.run.smithi071.stdout: "ino": 25, 2024-04-30T08:45:24.472 INFO:teuthology.orchestra.run.smithi071.stdout: "ino+": 904, 2024-04-30T08:45:24.472 INFO:teuthology.orchestra.run.smithi071.stdout: "ino-": 879, 2024-04-30T08:45:24.472 INFO:teuthology.orchestra.run.smithi071.stdout: "dir": 20, 2024-04-30T08:45:24.472 INFO:teuthology.orchestra.run.smithi071.stdout: "dir+": 127, 2024-04-30T08:45:24.472 INFO:teuthology.orchestra.run.smithi071.stdout: "dir-": 107, 2024-04-30T08:45:24.472 INFO:teuthology.orchestra.run.smithi071.stdout: "dn": 27, 2024-04-30T08:45:24.472 INFO:teuthology.orchestra.run.smithi071.stdout: "dn+": 1597, 2024-04-30T08:45:24.472 INFO:teuthology.orchestra.run.smithi071.stdout: "dn-": 1570, 2024-04-30T08:45:24.473 INFO:teuthology.orchestra.run.smithi071.stdout: "cap": 2, 2024-04-30T08:45:24.473 INFO:teuthology.orchestra.run.smithi071.stdout: "cap+": 888, 2024-04-30T08:45:24.473 INFO:teuthology.orchestra.run.smithi071.stdout: "cap-": 886, 2024-04-30T08:45:24.473 INFO:teuthology.orchestra.run.smithi071.stdout: "rss": 146708, 2024-04-30T08:45:24.473 INFO:teuthology.orchestra.run.smithi071.stdout: "heap": 231824 2024-04-30T08:45:24.473 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.473 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_server": { 2024-04-30T08:45:24.473 INFO:teuthology.orchestra.run.smithi071.stdout: "dispatch_client_request": 3052, 2024-04-30T08:45:24.473 INFO:teuthology.orchestra.run.smithi071.stdout: "dispatch_server_request": 9, 2024-04-30T08:45:24.473 INFO:teuthology.orchestra.run.smithi071.stdout: "handle_client_request": 2368, 2024-04-30T08:45:24.473 INFO:teuthology.orchestra.run.smithi071.stdout: "handle_client_session": 27, 2024-04-30T08:45:24.473 INFO:teuthology.orchestra.run.smithi071.stdout: "handle_peer_request": 26, 2024-04-30T08:45:24.473 INFO:teuthology.orchestra.run.smithi071.stdout: "req_create_latency": { 2024-04-30T08:45:24.473 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 765, 2024-04-30T08:45:24.474 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.916777716, 2024-04-30T08:45:24.474 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.001198402 2024-04-30T08:45:24.474 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.474 INFO:teuthology.orchestra.run.smithi071.stdout: "req_getattr_latency": { 2024-04-30T08:45:24.474 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 92, 2024-04-30T08:45:24.474 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.021262648, 2024-04-30T08:45:24.474 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000231115 2024-04-30T08:45:24.474 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.474 INFO:teuthology.orchestra.run.smithi071.stdout: "req_getfilelock_latency": { 2024-04-30T08:45:24.474 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:24.474 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:24.474 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:24.474 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.475 INFO:teuthology.orchestra.run.smithi071.stdout: "req_link_latency": { 2024-04-30T08:45:24.475 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 61, 2024-04-30T08:45:24.475 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.053217856, 2024-04-30T08:45:24.475 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000872423 2024-04-30T08:45:24.475 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.475 INFO:teuthology.orchestra.run.smithi071.stdout: "req_lookup_latency": { 2024-04-30T08:45:24.475 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 76, 2024-04-30T08:45:24.475 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.032791457, 2024-04-30T08:45:24.475 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000431466 2024-04-30T08:45:24.475 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.475 INFO:teuthology.orchestra.run.smithi071.stdout: "req_lookuphash_latency": { 2024-04-30T08:45:24.475 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:24.475 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:24.475 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:24.476 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.476 INFO:teuthology.orchestra.run.smithi071.stdout: "req_lookupino_latency": { 2024-04-30T08:45:24.476 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 1, 2024-04-30T08:45:24.476 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000366100, 2024-04-30T08:45:24.476 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000366100 2024-04-30T08:45:24.476 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.476 INFO:teuthology.orchestra.run.smithi071.stdout: "req_lookupname_latency": { 2024-04-30T08:45:24.476 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:24.476 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:24.476 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:24.476 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.476 INFO:teuthology.orchestra.run.smithi071.stdout: "req_lookupparent_latency": { 2024-04-30T08:45:24.476 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:24.476 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:24.477 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:24.477 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.477 INFO:teuthology.orchestra.run.smithi071.stdout: "req_lookupsnap_latency": { 2024-04-30T08:45:24.477 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:24.477 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:24.477 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:24.477 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.477 INFO:teuthology.orchestra.run.smithi071.stdout: "req_lssnap_latency": { 2024-04-30T08:45:24.477 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:24.478 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:24.478 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:24.478 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.478 INFO:teuthology.orchestra.run.smithi071.stdout: "req_mkdir_latency": { 2024-04-30T08:45:24.478 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 105, 2024-04-30T08:45:24.478 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.119861310, 2024-04-30T08:45:24.478 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.001141536 2024-04-30T08:45:24.478 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.479 INFO:teuthology.orchestra.run.smithi071.stdout: "req_mknod_latency": { 2024-04-30T08:45:24.479 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:24.479 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:24.479 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:24.479 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.479 INFO:teuthology.orchestra.run.smithi071.stdout: "req_mksnap_latency": { 2024-04-30T08:45:24.479 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:24.479 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:24.479 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:24.479 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.479 INFO:teuthology.orchestra.run.smithi071.stdout: "req_open_latency": { 2024-04-30T08:45:24.479 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 49, 2024-04-30T08:45:24.479 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.013115626, 2024-04-30T08:45:24.480 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000267665 2024-04-30T08:45:24.480 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.480 INFO:teuthology.orchestra.run.smithi071.stdout: "req_readdir_latency": { 2024-04-30T08:45:24.480 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 171, 2024-04-30T08:45:24.480 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 1.001776753, 2024-04-30T08:45:24.480 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.005858343 2024-04-30T08:45:24.480 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.480 INFO:teuthology.orchestra.run.smithi071.stdout: "req_rename_latency": { 2024-04-30T08:45:24.480 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 113, 2024-04-30T08:45:24.480 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 12.658741807, 2024-04-30T08:45:24.480 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.112024263 2024-04-30T08:45:24.480 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.480 INFO:teuthology.orchestra.run.smithi071.stdout: "req_renamesnap_latency": { 2024-04-30T08:45:24.480 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:24.481 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:24.481 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:24.481 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.481 INFO:teuthology.orchestra.run.smithi071.stdout: "req_snapdiff_latency": { 2024-04-30T08:45:24.481 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:24.481 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:24.481 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:24.481 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.481 INFO:teuthology.orchestra.run.smithi071.stdout: "req_rmdir_latency": { 2024-04-30T08:45:24.481 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 99, 2024-04-30T08:45:24.481 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 1.893066495, 2024-04-30T08:45:24.481 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.019121883 2024-04-30T08:45:24.481 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.482 INFO:teuthology.orchestra.run.smithi071.stdout: "req_rmsnap_latency": { 2024-04-30T08:45:24.482 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:24.482 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:24.482 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:24.482 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.482 INFO:teuthology.orchestra.run.smithi071.stdout: "req_rmxattr_latency": { 2024-04-30T08:45:24.482 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:24.482 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:24.482 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:24.482 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.482 INFO:teuthology.orchestra.run.smithi071.stdout: "req_setattr_latency": { 2024-04-30T08:45:24.482 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 8, 2024-04-30T08:45:24.482 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.040454564, 2024-04-30T08:45:24.483 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.005056820 2024-04-30T08:45:24.483 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.483 INFO:teuthology.orchestra.run.smithi071.stdout: "req_setdirlayout_latency": { 2024-04-30T08:45:24.483 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:24.483 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:24.483 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:24.483 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.483 INFO:teuthology.orchestra.run.smithi071.stdout: "req_setfilelock_latency": { 2024-04-30T08:45:24.483 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 4, 2024-04-30T08:45:24.483 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.001644860, 2024-04-30T08:45:24.483 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000411215 2024-04-30T08:45:24.483 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.483 INFO:teuthology.orchestra.run.smithi071.stdout: "req_setlayout_latency": { 2024-04-30T08:45:24.483 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:24.484 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:24.484 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:24.484 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.484 INFO:teuthology.orchestra.run.smithi071.stdout: "req_setxattr_latency": { 2024-04-30T08:45:24.484 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 9, 2024-04-30T08:45:24.484 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.027995579, 2024-04-30T08:45:24.484 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.003110619 2024-04-30T08:45:24.484 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.484 INFO:teuthology.orchestra.run.smithi071.stdout: "req_symlink_latency": { 2024-04-30T08:45:24.484 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 13, 2024-04-30T08:45:24.484 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.009513908, 2024-04-30T08:45:24.484 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000731839 2024-04-30T08:45:24.485 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.485 INFO:teuthology.orchestra.run.smithi071.stdout: "req_unlink_latency": { 2024-04-30T08:45:24.485 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 798, 2024-04-30T08:45:24.485 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 1.160335294, 2024-04-30T08:45:24.485 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.001454054 2024-04-30T08:45:24.485 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.485 INFO:teuthology.orchestra.run.smithi071.stdout: "cap_revoke_eviction": 0, 2024-04-30T08:45:24.485 INFO:teuthology.orchestra.run.smithi071.stdout: "cap_acquisition_throttle": 0, 2024-04-30T08:45:24.485 INFO:teuthology.orchestra.run.smithi071.stdout: "req_getvxattr_latency": { 2024-04-30T08:45:24.485 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:24.485 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:24.485 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:24.485 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:24.486 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.486 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_sessions": { 2024-04-30T08:45:24.486 INFO:teuthology.orchestra.run.smithi071.stdout: "session_count": 1, 2024-04-30T08:45:24.486 INFO:teuthology.orchestra.run.smithi071.stdout: "session_add": 4, 2024-04-30T08:45:24.486 INFO:teuthology.orchestra.run.smithi071.stdout: "session_remove": 3, 2024-04-30T08:45:24.486 INFO:teuthology.orchestra.run.smithi071.stdout: "sessions_open": 1, 2024-04-30T08:45:24.486 INFO:teuthology.orchestra.run.smithi071.stdout: "sessions_stale": 0, 2024-04-30T08:45:24.486 INFO:teuthology.orchestra.run.smithi071.stdout: "total_load": 408, 2024-04-30T08:45:24.486 INFO:teuthology.orchestra.run.smithi071.stdout: "average_load": 408, 2024-04-30T08:45:24.486 INFO:teuthology.orchestra.run.smithi071.stdout: "avg_session_uptime": 615, 2024-04-30T08:45:24.486 INFO:teuthology.orchestra.run.smithi071.stdout: "mdthresh_evicted": 0 2024-04-30T08:45:24.486 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.486 INFO:teuthology.orchestra.run.smithi071.stdout: "mempool": { 2024-04-30T08:45:24.486 INFO:teuthology.orchestra.run.smithi071.stdout: "bloom_filter_bytes": 0, 2024-04-30T08:45:24.487 INFO:teuthology.orchestra.run.smithi071.stdout: "bloom_filter_items": 0, 2024-04-30T08:45:24.487 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_alloc_bytes": 0, 2024-04-30T08:45:24.487 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_alloc_items": 0, 2024-04-30T08:45:24.487 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_data_bytes": 0, 2024-04-30T08:45:24.487 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_data_items": 0, 2024-04-30T08:45:24.487 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-30T08:45:24.487 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_onode_items": 0, 2024-04-30T08:45:24.487 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-30T08:45:24.487 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_meta_items": 0, 2024-04-30T08:45:24.487 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_other_bytes": 0, 2024-04-30T08:45:24.487 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_other_items": 0, 2024-04-30T08:45:24.487 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-30T08:45:24.487 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_buffer_items": 0, 2024-04-30T08:45:24.488 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_extent_bytes": 0, 2024-04-30T08:45:24.488 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_extent_items": 0, 2024-04-30T08:45:24.488 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_blob_bytes": 0, 2024-04-30T08:45:24.488 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_blob_items": 0, 2024-04-30T08:45:24.488 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-30T08:45:24.488 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_shared_blob_items": 0, 2024-04-30T08:45:24.488 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-30T08:45:24.488 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_inline_bl_items": 0, 2024-04-30T08:45:24.488 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_fsck_bytes": 0, 2024-04-30T08:45:24.488 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_fsck_items": 0, 2024-04-30T08:45:24.488 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_txc_bytes": 0, 2024-04-30T08:45:24.488 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_txc_items": 0, 2024-04-30T08:45:24.488 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-30T08:45:24.488 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_writing_deferred_items": 0, 2024-04-30T08:45:24.489 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_writing_bytes": 0, 2024-04-30T08:45:24.489 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_writing_items": 0, 2024-04-30T08:45:24.489 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_bytes": 0, 2024-04-30T08:45:24.489 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_items": 0, 2024-04-30T08:45:24.489 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_file_reader_bytes": 0, 2024-04-30T08:45:24.489 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_file_reader_items": 0, 2024-04-30T08:45:24.489 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_file_writer_bytes": 0, 2024-04-30T08:45:24.489 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_file_writer_items": 0, 2024-04-30T08:45:24.489 INFO:teuthology.orchestra.run.smithi071.stdout: "buffer_anon_bytes": 332362, 2024-04-30T08:45:24.489 INFO:teuthology.orchestra.run.smithi071.stdout: "buffer_anon_items": 143, 2024-04-30T08:45:24.489 INFO:teuthology.orchestra.run.smithi071.stdout: "buffer_meta_bytes": 1144, 2024-04-30T08:45:24.489 INFO:teuthology.orchestra.run.smithi071.stdout: "buffer_meta_items": 13, 2024-04-30T08:45:24.489 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_bytes": 0, 2024-04-30T08:45:24.490 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_items": 0, 2024-04-30T08:45:24.490 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_mapbl_bytes": 0, 2024-04-30T08:45:24.490 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_mapbl_items": 0, 2024-04-30T08:45:24.490 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pglog_bytes": 0, 2024-04-30T08:45:24.490 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pglog_items": 0, 2024-04-30T08:45:24.490 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_bytes": 9088, 2024-04-30T08:45:24.490 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_items": 175, 2024-04-30T08:45:24.490 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_mapping_bytes": 0, 2024-04-30T08:45:24.490 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_mapping_items": 0, 2024-04-30T08:45:24.490 INFO:teuthology.orchestra.run.smithi071.stdout: "pgmap_bytes": 0, 2024-04-30T08:45:24.490 INFO:teuthology.orchestra.run.smithi071.stdout: "pgmap_items": 0, 2024-04-30T08:45:24.490 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_co_bytes": 109830, 2024-04-30T08:45:24.490 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_co_items": 966, 2024-04-30T08:45:24.491 INFO:teuthology.orchestra.run.smithi071.stdout: "unittest_1_bytes": 0, 2024-04-30T08:45:24.491 INFO:teuthology.orchestra.run.smithi071.stdout: "unittest_1_items": 0, 2024-04-30T08:45:24.491 INFO:teuthology.orchestra.run.smithi071.stdout: "unittest_2_bytes": 0, 2024-04-30T08:45:24.491 INFO:teuthology.orchestra.run.smithi071.stdout: "unittest_2_items": 0 2024-04-30T08:45:24.491 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.491 INFO:teuthology.orchestra.run.smithi071.stdout: "objecter": { 2024-04-30T08:45:24.491 INFO:teuthology.orchestra.run.smithi071.stdout: "op_active": 0, 2024-04-30T08:45:24.491 INFO:teuthology.orchestra.run.smithi071.stdout: "op_laggy": 0, 2024-04-30T08:45:24.491 INFO:teuthology.orchestra.run.smithi071.stdout: "op_send": 9454, 2024-04-30T08:45:24.491 INFO:teuthology.orchestra.run.smithi071.stdout: "op_send_bytes": 32476673, 2024-04-30T08:45:24.491 INFO:teuthology.orchestra.run.smithi071.stdout: "op_resend": 31, 2024-04-30T08:45:24.491 INFO:teuthology.orchestra.run.smithi071.stdout: "op_reply": 9454, 2024-04-30T08:45:24.491 INFO:teuthology.orchestra.run.smithi071.stdout: "op_latency": { 2024-04-30T08:45:24.491 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 9454, 2024-04-30T08:45:24.492 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 3351.774101847, 2024-04-30T08:45:24.492 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.354535022 2024-04-30T08:45:24.492 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.492 INFO:teuthology.orchestra.run.smithi071.stdout: "op_inflight": 0, 2024-04-30T08:45:24.492 INFO:teuthology.orchestra.run.smithi071.stdout: "oplen_avg": { 2024-04-30T08:45:24.492 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 9454, 2024-04-30T08:45:24.492 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 12024 2024-04-30T08:45:24.492 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.492 INFO:teuthology.orchestra.run.smithi071.stdout: "op": 9454, 2024-04-30T08:45:24.492 INFO:teuthology.orchestra.run.smithi071.stdout: "op_r": 695, 2024-04-30T08:45:24.492 INFO:teuthology.orchestra.run.smithi071.stdout: "op_w": 8759, 2024-04-30T08:45:24.492 INFO:teuthology.orchestra.run.smithi071.stdout: "op_rmw": 0, 2024-04-30T08:45:24.492 INFO:teuthology.orchestra.run.smithi071.stdout: "op_pg": 0, 2024-04-30T08:45:24.492 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_stat": 162, 2024-04-30T08:45:24.493 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_create": 1135, 2024-04-30T08:45:24.493 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_read": 34, 2024-04-30T08:45:24.493 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_write": 1749, 2024-04-30T08:45:24.493 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_writefull": 97, 2024-04-30T08:45:24.493 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_writesame": 0, 2024-04-30T08:45:24.493 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_append": 0, 2024-04-30T08:45:24.493 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_zero": 0, 2024-04-30T08:45:24.493 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_truncate": 1, 2024-04-30T08:45:24.493 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_delete": 5226, 2024-04-30T08:45:24.493 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_mapext": 0, 2024-04-30T08:45:24.493 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_sparse_read": 0, 2024-04-30T08:45:24.493 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_clonerange": 0, 2024-04-30T08:45:24.493 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_getxattr": 661, 2024-04-30T08:45:24.493 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_setxattr": 1758, 2024-04-30T08:45:24.494 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_cmpxattr": 0, 2024-04-30T08:45:24.494 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_rmxattr": 0, 2024-04-30T08:45:24.494 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_resetxattrs": 0, 2024-04-30T08:45:24.494 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_call": 0, 2024-04-30T08:45:24.494 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_watch": 0, 2024-04-30T08:45:24.494 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_notify": 0, 2024-04-30T08:45:24.494 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_src_cmpxattr": 0, 2024-04-30T08:45:24.494 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_pgls": 0, 2024-04-30T08:45:24.494 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_pgls_filter": 0, 2024-04-30T08:45:24.494 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_other": 6, 2024-04-30T08:45:24.494 INFO:teuthology.orchestra.run.smithi071.stdout: "linger_active": 0, 2024-04-30T08:45:24.494 INFO:teuthology.orchestra.run.smithi071.stdout: "linger_send": 0, 2024-04-30T08:45:24.494 INFO:teuthology.orchestra.run.smithi071.stdout: "linger_resend": 0, 2024-04-30T08:45:24.495 INFO:teuthology.orchestra.run.smithi071.stdout: "linger_ping": 0, 2024-04-30T08:45:24.495 INFO:teuthology.orchestra.run.smithi071.stdout: "poolop_active": 0, 2024-04-30T08:45:24.495 INFO:teuthology.orchestra.run.smithi071.stdout: "poolop_send": 0, 2024-04-30T08:45:24.495 INFO:teuthology.orchestra.run.smithi071.stdout: "poolop_resend": 0, 2024-04-30T08:45:24.495 INFO:teuthology.orchestra.run.smithi071.stdout: "poolstat_active": 0, 2024-04-30T08:45:24.495 INFO:teuthology.orchestra.run.smithi071.stdout: "poolstat_send": 0, 2024-04-30T08:45:24.495 INFO:teuthology.orchestra.run.smithi071.stdout: "poolstat_resend": 0, 2024-04-30T08:45:24.495 INFO:teuthology.orchestra.run.smithi071.stdout: "statfs_active": 0, 2024-04-30T08:45:24.495 INFO:teuthology.orchestra.run.smithi071.stdout: "statfs_send": 0, 2024-04-30T08:45:24.495 INFO:teuthology.orchestra.run.smithi071.stdout: "statfs_resend": 0, 2024-04-30T08:45:24.495 INFO:teuthology.orchestra.run.smithi071.stdout: "command_active": 0, 2024-04-30T08:45:24.495 INFO:teuthology.orchestra.run.smithi071.stdout: "command_send": 0, 2024-04-30T08:45:24.495 INFO:teuthology.orchestra.run.smithi071.stdout: "command_resend": 0, 2024-04-30T08:45:24.495 INFO:teuthology.orchestra.run.smithi071.stdout: "map_epoch": 85, 2024-04-30T08:45:24.496 INFO:teuthology.orchestra.run.smithi071.stdout: "map_full": 0, 2024-04-30T08:45:24.496 INFO:teuthology.orchestra.run.smithi071.stdout: "map_inc": 10, 2024-04-30T08:45:24.496 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_sessions": 12, 2024-04-30T08:45:24.496 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_session_open": 12, 2024-04-30T08:45:24.496 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_session_close": 0, 2024-04-30T08:45:24.496 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_laggy": 0, 2024-04-30T08:45:24.496 INFO:teuthology.orchestra.run.smithi071.stdout: "omap_wr": 941, 2024-04-30T08:45:24.496 INFO:teuthology.orchestra.run.smithi071.stdout: "omap_rd": 10, 2024-04-30T08:45:24.496 INFO:teuthology.orchestra.run.smithi071.stdout: "omap_del": 244 2024-04-30T08:45:24.496 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.496 INFO:teuthology.orchestra.run.smithi071.stdout: "oft": { 2024-04-30T08:45:24.496 INFO:teuthology.orchestra.run.smithi071.stdout: "omap_total_objs": 1, 2024-04-30T08:45:24.496 INFO:teuthology.orchestra.run.smithi071.stdout: "omap_total_kv_pairs": 0, 2024-04-30T08:45:24.496 INFO:teuthology.orchestra.run.smithi071.stdout: "omap_total_updates": 0, 2024-04-30T08:45:24.497 INFO:teuthology.orchestra.run.smithi071.stdout: "omap_total_removes": 0 2024-04-30T08:45:24.497 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.497 INFO:teuthology.orchestra.run.smithi071.stdout: "osd-slow-ops": { 2024-04-30T08:45:24.497 INFO:teuthology.orchestra.run.smithi071.stdout: "slow_ops_count": 0 2024-04-30T08:45:24.497 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.497 INFO:teuthology.orchestra.run.smithi071.stdout: "purge_queue": { 2024-04-30T08:45:24.497 INFO:teuthology.orchestra.run.smithi071.stdout: "pq_executing_ops": 0, 2024-04-30T08:45:24.497 INFO:teuthology.orchestra.run.smithi071.stdout: "pq_executing_ops_high_water": 27, 2024-04-30T08:45:24.497 INFO:teuthology.orchestra.run.smithi071.stdout: "pq_executing": 0, 2024-04-30T08:45:24.497 INFO:teuthology.orchestra.run.smithi071.stdout: "pq_executing_high_water": 9, 2024-04-30T08:45:24.497 INFO:teuthology.orchestra.run.smithi071.stdout: "pq_executed_ops": 4704, 2024-04-30T08:45:24.497 INFO:teuthology.orchestra.run.smithi071.stdout: "pq_executed": 867, 2024-04-30T08:45:24.497 INFO:teuthology.orchestra.run.smithi071.stdout: "pq_item_in_journal": 0 2024-04-30T08:45:24.497 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.498 INFO:teuthology.orchestra.run.smithi071.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-30T08:45:24.498 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 0, 2024-04-30T08:45:24.498 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 104857600, 2024-04-30T08:45:24.498 INFO:teuthology.orchestra.run.smithi071.stdout: "get_started": 0, 2024-04-30T08:45:24.498 INFO:teuthology.orchestra.run.smithi071.stdout: "get": 29322, 2024-04-30T08:45:24.498 INFO:teuthology.orchestra.run.smithi071.stdout: "get_sum": 8452835, 2024-04-30T08:45:24.498 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:24.498 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_success": 29322, 2024-04-30T08:45:24.498 INFO:teuthology.orchestra.run.smithi071.stdout: "take": 0, 2024-04-30T08:45:24.498 INFO:teuthology.orchestra.run.smithi071.stdout: "take_sum": 0, 2024-04-30T08:45:24.498 INFO:teuthology.orchestra.run.smithi071.stdout: "put": 29322, 2024-04-30T08:45:24.498 INFO:teuthology.orchestra.run.smithi071.stdout: "put_sum": 8452835, 2024-04-30T08:45:24.498 INFO:teuthology.orchestra.run.smithi071.stdout: "wait": { 2024-04-30T08:45:24.499 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:24.499 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:24.499 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:24.499 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:24.499 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.499 INFO:teuthology.orchestra.run.smithi071.stdout: "throttle-objecter_bytes": { 2024-04-30T08:45:24.499 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 0, 2024-04-30T08:45:24.499 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 104857600, 2024-04-30T08:45:24.499 INFO:teuthology.orchestra.run.smithi071.stdout: "get_started": 0, 2024-04-30T08:45:24.499 INFO:teuthology.orchestra.run.smithi071.stdout: "get": 0, 2024-04-30T08:45:24.499 INFO:teuthology.orchestra.run.smithi071.stdout: "get_sum": 0, 2024-04-30T08:45:24.499 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:24.499 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_success": 0, 2024-04-30T08:45:24.500 INFO:teuthology.orchestra.run.smithi071.stdout: "take": 9454, 2024-04-30T08:45:24.500 INFO:teuthology.orchestra.run.smithi071.stdout: "take_sum": 32570248, 2024-04-30T08:45:24.500 INFO:teuthology.orchestra.run.smithi071.stdout: "put": 9454, 2024-04-30T08:45:24.500 INFO:teuthology.orchestra.run.smithi071.stdout: "put_sum": 32570248, 2024-04-30T08:45:24.500 INFO:teuthology.orchestra.run.smithi071.stdout: "wait": { 2024-04-30T08:45:24.500 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:24.500 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:24.500 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:24.500 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:24.500 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.500 INFO:teuthology.orchestra.run.smithi071.stdout: "throttle-objecter_ops": { 2024-04-30T08:45:24.500 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 0, 2024-04-30T08:45:24.501 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 1024, 2024-04-30T08:45:24.501 INFO:teuthology.orchestra.run.smithi071.stdout: "get_started": 0, 2024-04-30T08:45:24.501 INFO:teuthology.orchestra.run.smithi071.stdout: "get": 0, 2024-04-30T08:45:24.501 INFO:teuthology.orchestra.run.smithi071.stdout: "get_sum": 0, 2024-04-30T08:45:24.501 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:24.501 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_success": 0, 2024-04-30T08:45:24.501 INFO:teuthology.orchestra.run.smithi071.stdout: "take": 9454, 2024-04-30T08:45:24.501 INFO:teuthology.orchestra.run.smithi071.stdout: "take_sum": 9454, 2024-04-30T08:45:24.501 INFO:teuthology.orchestra.run.smithi071.stdout: "put": 9454, 2024-04-30T08:45:24.501 INFO:teuthology.orchestra.run.smithi071.stdout: "put_sum": 9454, 2024-04-30T08:45:24.501 INFO:teuthology.orchestra.run.smithi071.stdout: "wait": { 2024-04-30T08:45:24.501 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:24.501 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:24.501 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:24.502 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:24.502 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.502 INFO:teuthology.orchestra.run.smithi071.stdout: "throttle-write_buf_throttle": { 2024-04-30T08:45:24.502 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 0, 2024-04-30T08:45:24.502 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 3758096384, 2024-04-30T08:45:24.502 INFO:teuthology.orchestra.run.smithi071.stdout: "get_started": 0, 2024-04-30T08:45:24.502 INFO:teuthology.orchestra.run.smithi071.stdout: "get": 867, 2024-04-30T08:45:24.502 INFO:teuthology.orchestra.run.smithi071.stdout: "get_sum": 93631, 2024-04-30T08:45:24.502 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:24.502 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_success": 867, 2024-04-30T08:45:24.502 INFO:teuthology.orchestra.run.smithi071.stdout: "take": 0, 2024-04-30T08:45:24.502 INFO:teuthology.orchestra.run.smithi071.stdout: "take_sum": 0, 2024-04-30T08:45:24.502 INFO:teuthology.orchestra.run.smithi071.stdout: "put": 29, 2024-04-30T08:45:24.503 INFO:teuthology.orchestra.run.smithi071.stdout: "put_sum": 93631, 2024-04-30T08:45:24.503 INFO:teuthology.orchestra.run.smithi071.stdout: "wait": { 2024-04-30T08:45:24.503 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:24.503 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:24.503 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:24.503 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:24.503 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:24.503 INFO:teuthology.orchestra.run.smithi071.stdout: "throttle-write_buf_throttle-0x55d1752ea4e0": { 2024-04-30T08:45:24.503 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 0, 2024-04-30T08:45:24.503 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 3758096384, 2024-04-30T08:45:24.503 INFO:teuthology.orchestra.run.smithi071.stdout: "get_started": 0, 2024-04-30T08:45:24.503 INFO:teuthology.orchestra.run.smithi071.stdout: "get": 14542, 2024-04-30T08:45:24.503 INFO:teuthology.orchestra.run.smithi071.stdout: "get_sum": 31025914, 2024-04-30T08:45:24.504 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:24.504 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_success": 14542, 2024-04-30T08:45:24.504 INFO:teuthology.orchestra.run.smithi071.stdout: "take": 0, 2024-04-30T08:45:24.504 INFO:teuthology.orchestra.run.smithi071.stdout: "take_sum": 0, 2024-04-30T08:45:24.504 INFO:teuthology.orchestra.run.smithi071.stdout: "put": 1720, 2024-04-30T08:45:24.504 INFO:teuthology.orchestra.run.smithi071.stdout: "put_sum": 31025914, 2024-04-30T08:45:24.504 INFO:teuthology.orchestra.run.smithi071.stdout: "wait": { 2024-04-30T08:45:24.504 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:24.504 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:24.504 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:24.504 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:24.504 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:24.504 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-30T08:45:24.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:24 smithi189 ceph-mon[33038]: pgmap v732: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 682 B/s wr, 9 op/s 2024-04-30T08:45:25.338 INFO:tasks.check_counter:Daemon mds.b mds.exported=4 2024-04-30T08:45:25.338 INFO:tasks.check_counter:Daemon mds.b mds.imported=4 2024-04-30T08:45:25.338 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=1 2024-04-30T08:45:25.338 DEBUG:tasks.check_counter:Getting stats from e 2024-04-30T08:45:25.338 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-30T08:45:25.586 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.b/config 2024-04-30T08:45:27.254 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:27 smithi071 ceph-mon[33385]: pgmap v733: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-30T08:45:27.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:27 smithi032 ceph-mon[28331]: pgmap v733: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-30T08:45:27.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:27 smithi189 ceph-mon[33038]: pgmap v733: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-30T08:45:27.564 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-30T08:45:27.565 INFO:teuthology.orchestra.run.smithi071.stdout: "AsyncMessenger::Worker-0": { 2024-04-30T08:45:27.565 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_messages": 734, 2024-04-30T08:45:27.565 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_messages": 735, 2024-04-30T08:45:27.565 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_bytes": 8609489, 2024-04-30T08:45:27.565 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_bytes": 233104, 2024-04-30T08:45:27.565 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_created_connections": 7, 2024-04-30T08:45:27.565 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_active_connections": 3, 2024-04-30T08:45:27.565 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_total_time": 0.075608676, 2024-04-30T08:45:27.565 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_send_time": 0.027879220, 2024-04-30T08:45:27.565 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_recv_time": 0.020406796, 2024-04-30T08:45:27.565 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_fast_dispatch_time": 0.020297394, 2024-04-30T08:45:27.565 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:27.566 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 733, 2024-04-30T08:45:27.566 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.035339045, 2024-04-30T08:45:27.566 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000048211 2024-04-30T08:45:27.566 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.566 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:27.566 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.566 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.566 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.566 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.566 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-30T08:45:27.566 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_encrypted_bytes": 0 2024-04-30T08:45:27.566 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.567 INFO:teuthology.orchestra.run.smithi071.stdout: "AsyncMessenger::Worker-1": { 2024-04-30T08:45:27.567 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_messages": 531, 2024-04-30T08:45:27.567 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_messages": 496, 2024-04-30T08:45:27.567 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_bytes": 15827286, 2024-04-30T08:45:27.567 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_bytes": 210829, 2024-04-30T08:45:27.567 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_created_connections": 4, 2024-04-30T08:45:27.567 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_active_connections": 3, 2024-04-30T08:45:27.567 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_total_time": 0.069806226, 2024-04-30T08:45:27.567 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_send_time": 0.017473675, 2024-04-30T08:45:27.567 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_recv_time": 0.047186278, 2024-04-30T08:45:27.567 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_fast_dispatch_time": 0.010549019, 2024-04-30T08:45:27.567 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:27.567 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 495, 2024-04-30T08:45:27.568 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.030869437, 2024-04-30T08:45:27.568 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000062362 2024-04-30T08:45:27.568 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.568 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:27.568 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.568 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.568 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.568 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.568 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_encrypted_bytes": 1530736, 2024-04-30T08:45:27.568 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_encrypted_bytes": 145840 2024-04-30T08:45:27.568 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.568 INFO:teuthology.orchestra.run.smithi071.stdout: "AsyncMessenger::Worker-2": { 2024-04-30T08:45:27.568 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_messages": 383, 2024-04-30T08:45:27.569 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_messages": 553, 2024-04-30T08:45:27.569 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_bytes": 8485613, 2024-04-30T08:45:27.569 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_bytes": 568015, 2024-04-30T08:45:27.569 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_created_connections": 6, 2024-04-30T08:45:27.569 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_active_connections": 4, 2024-04-30T08:45:27.569 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_total_time": 0.048154330, 2024-04-30T08:45:27.569 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_send_time": 0.015748209, 2024-04-30T08:45:27.569 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_recv_time": 0.016243974, 2024-04-30T08:45:27.569 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_fast_dispatch_time": 0.012152965, 2024-04-30T08:45:27.569 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:27.569 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 551, 2024-04-30T08:45:27.569 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.031332173, 2024-04-30T08:45:27.569 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000056864 2024-04-30T08:45:27.570 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.570 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:27.570 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.570 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.570 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.570 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.570 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-30T08:45:27.570 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_encrypted_bytes": 448128 2024-04-30T08:45:27.570 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.570 INFO:teuthology.orchestra.run.smithi071.stdout: "cct": { 2024-04-30T08:45:27.570 INFO:teuthology.orchestra.run.smithi071.stdout: "total_workers": 1, 2024-04-30T08:45:27.570 INFO:teuthology.orchestra.run.smithi071.stdout: "unhealthy_workers": 0 2024-04-30T08:45:27.570 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.571 INFO:teuthology.orchestra.run.smithi071.stdout: "finisher-MDSRank": { 2024-04-30T08:45:27.571 INFO:teuthology.orchestra.run.smithi071.stdout: "queue_len": 0, 2024-04-30T08:45:27.571 INFO:teuthology.orchestra.run.smithi071.stdout: "complete_latency": { 2024-04-30T08:45:27.571 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 2173, 2024-04-30T08:45:27.571 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.062788846, 2024-04-30T08:45:27.571 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000028895 2024-04-30T08:45:27.571 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:27.571 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.571 INFO:teuthology.orchestra.run.smithi071.stdout: "finisher-PurgeQueue": { 2024-04-30T08:45:27.571 INFO:teuthology.orchestra.run.smithi071.stdout: "queue_len": 0, 2024-04-30T08:45:27.571 INFO:teuthology.orchestra.run.smithi071.stdout: "complete_latency": { 2024-04-30T08:45:27.571 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.571 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.572 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.572 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:27.572 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.572 INFO:teuthology.orchestra.run.smithi071.stdout: "mds": { 2024-04-30T08:45:27.572 INFO:teuthology.orchestra.run.smithi071.stdout: "request": 0, 2024-04-30T08:45:27.572 INFO:teuthology.orchestra.run.smithi071.stdout: "reply": 0, 2024-04-30T08:45:27.572 INFO:teuthology.orchestra.run.smithi071.stdout: "reply_latency": { 2024-04-30T08:45:27.572 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.572 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.572 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.572 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.572 INFO:teuthology.orchestra.run.smithi071.stdout: "slow_reply": 0, 2024-04-30T08:45:27.572 INFO:teuthology.orchestra.run.smithi071.stdout: "forward": 0, 2024-04-30T08:45:27.573 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_fetch_complete": 0, 2024-04-30T08:45:27.573 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_fetch_keys": 0, 2024-04-30T08:45:27.573 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_commit": 0, 2024-04-30T08:45:27.573 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_split": 0, 2024-04-30T08:45:27.573 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_merge": 0, 2024-04-30T08:45:27.573 INFO:teuthology.orchestra.run.smithi071.stdout: "inodes": 1027, 2024-04-30T08:45:27.573 INFO:teuthology.orchestra.run.smithi071.stdout: "inodes_top": 380, 2024-04-30T08:45:27.573 INFO:teuthology.orchestra.run.smithi071.stdout: "inodes_bottom": 647, 2024-04-30T08:45:27.573 INFO:teuthology.orchestra.run.smithi071.stdout: "inodes_pin_tail": 0, 2024-04-30T08:45:27.573 INFO:teuthology.orchestra.run.smithi071.stdout: "inodes_pinned": 483, 2024-04-30T08:45:27.573 INFO:teuthology.orchestra.run.smithi071.stdout: "inodes_expired": 0, 2024-04-30T08:45:27.573 INFO:teuthology.orchestra.run.smithi071.stdout: "inodes_with_caps": 0, 2024-04-30T08:45:27.573 INFO:teuthology.orchestra.run.smithi071.stdout: "caps": 0, 2024-04-30T08:45:27.574 INFO:teuthology.orchestra.run.smithi071.stdout: "subtrees": 2, 2024-04-30T08:45:27.574 INFO:teuthology.orchestra.run.smithi071.stdout: "traverse": 0, 2024-04-30T08:45:27.574 INFO:teuthology.orchestra.run.smithi071.stdout: "traverse_hit": 0, 2024-04-30T08:45:27.574 INFO:teuthology.orchestra.run.smithi071.stdout: "traverse_forward": 0, 2024-04-30T08:45:27.574 INFO:teuthology.orchestra.run.smithi071.stdout: "traverse_discover": 0, 2024-04-30T08:45:27.574 INFO:teuthology.orchestra.run.smithi071.stdout: "traverse_dir_fetch": 0, 2024-04-30T08:45:27.574 INFO:teuthology.orchestra.run.smithi071.stdout: "traverse_remote_ino": 0, 2024-04-30T08:45:27.574 INFO:teuthology.orchestra.run.smithi071.stdout: "traverse_lock": 0, 2024-04-30T08:45:27.574 INFO:teuthology.orchestra.run.smithi071.stdout: "load_cent": 0, 2024-04-30T08:45:27.574 INFO:teuthology.orchestra.run.smithi071.stdout: "q": 0, 2024-04-30T08:45:27.574 INFO:teuthology.orchestra.run.smithi071.stdout: "exported": 0, 2024-04-30T08:45:27.574 INFO:teuthology.orchestra.run.smithi071.stdout: "exported_inodes": 0, 2024-04-30T08:45:27.574 INFO:teuthology.orchestra.run.smithi071.stdout: "imported": 0, 2024-04-30T08:45:27.575 INFO:teuthology.orchestra.run.smithi071.stdout: "imported_inodes": 0, 2024-04-30T08:45:27.575 INFO:teuthology.orchestra.run.smithi071.stdout: "openino_dir_fetch": 0, 2024-04-30T08:45:27.575 INFO:teuthology.orchestra.run.smithi071.stdout: "openino_backtrace_fetch": 0, 2024-04-30T08:45:27.575 INFO:teuthology.orchestra.run.smithi071.stdout: "openino_peer_discover": 0, 2024-04-30T08:45:27.575 INFO:teuthology.orchestra.run.smithi071.stdout: "root_rfiles": 4, 2024-04-30T08:45:27.575 INFO:teuthology.orchestra.run.smithi071.stdout: "root_rbytes": 234, 2024-04-30T08:45:27.575 INFO:teuthology.orchestra.run.smithi071.stdout: "root_rsnaps": 0, 2024-04-30T08:45:27.575 INFO:teuthology.orchestra.run.smithi071.stdout: "scrub_backtrace_fetch": 0, 2024-04-30T08:45:27.575 INFO:teuthology.orchestra.run.smithi071.stdout: "scrub_set_tag": 0, 2024-04-30T08:45:27.575 INFO:teuthology.orchestra.run.smithi071.stdout: "scrub_backtrace_repaired": 0, 2024-04-30T08:45:27.575 INFO:teuthology.orchestra.run.smithi071.stdout: "scrub_inotable_repaired": 0, 2024-04-30T08:45:27.575 INFO:teuthology.orchestra.run.smithi071.stdout: "scrub_dir_inodes": 0, 2024-04-30T08:45:27.575 INFO:teuthology.orchestra.run.smithi071.stdout: "scrub_dir_base_inodes": 0, 2024-04-30T08:45:27.576 INFO:teuthology.orchestra.run.smithi071.stdout: "scrub_dirfrag_rstats": 0, 2024-04-30T08:45:27.576 INFO:teuthology.orchestra.run.smithi071.stdout: "scrub_file_inodes": 0, 2024-04-30T08:45:27.576 INFO:teuthology.orchestra.run.smithi071.stdout: "handle_inode_file_caps": 0, 2024-04-30T08:45:27.576 INFO:teuthology.orchestra.run.smithi071.stdout: "ceph_cap_op_revoke": 0, 2024-04-30T08:45:27.576 INFO:teuthology.orchestra.run.smithi071.stdout: "ceph_cap_op_grant": 0, 2024-04-30T08:45:27.576 INFO:teuthology.orchestra.run.smithi071.stdout: "ceph_cap_op_trunc": 0, 2024-04-30T08:45:27.576 INFO:teuthology.orchestra.run.smithi071.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-30T08:45:27.576 INFO:teuthology.orchestra.run.smithi071.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-30T08:45:27.576 INFO:teuthology.orchestra.run.smithi071.stdout: "handle_client_caps": 0, 2024-04-30T08:45:27.576 INFO:teuthology.orchestra.run.smithi071.stdout: "handle_client_caps_dirty": 0, 2024-04-30T08:45:27.576 INFO:teuthology.orchestra.run.smithi071.stdout: "handle_client_cap_release": 0, 2024-04-30T08:45:27.576 INFO:teuthology.orchestra.run.smithi071.stdout: "process_request_cap_release": 0 2024-04-30T08:45:27.576 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.576 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_cache": { 2024-04-30T08:45:27.577 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_update": 0, 2024-04-30T08:45:27.577 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_update_receipt": 0, 2024-04-30T08:45:27.577 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_try_discover": 0, 2024-04-30T08:45:27.577 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_send_discover": 0, 2024-04-30T08:45:27.577 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_handle_discover": 0, 2024-04-30T08:45:27.577 INFO:teuthology.orchestra.run.smithi071.stdout: "num_strays": 0, 2024-04-30T08:45:27.577 INFO:teuthology.orchestra.run.smithi071.stdout: "num_strays_delayed": 0, 2024-04-30T08:45:27.577 INFO:teuthology.orchestra.run.smithi071.stdout: "num_strays_enqueuing": 0, 2024-04-30T08:45:27.577 INFO:teuthology.orchestra.run.smithi071.stdout: "strays_created": 0, 2024-04-30T08:45:27.577 INFO:teuthology.orchestra.run.smithi071.stdout: "strays_enqueued": 0, 2024-04-30T08:45:27.577 INFO:teuthology.orchestra.run.smithi071.stdout: "strays_reintegrated": 0, 2024-04-30T08:45:27.577 INFO:teuthology.orchestra.run.smithi071.stdout: "strays_migrated": 0, 2024-04-30T08:45:27.577 INFO:teuthology.orchestra.run.smithi071.stdout: "num_recovering_processing": 0, 2024-04-30T08:45:27.578 INFO:teuthology.orchestra.run.smithi071.stdout: "num_recovering_enqueued": 0, 2024-04-30T08:45:27.578 INFO:teuthology.orchestra.run.smithi071.stdout: "num_recovering_prioritized": 0, 2024-04-30T08:45:27.578 INFO:teuthology.orchestra.run.smithi071.stdout: "recovery_started": 0, 2024-04-30T08:45:27.578 INFO:teuthology.orchestra.run.smithi071.stdout: "recovery_completed": 0, 2024-04-30T08:45:27.578 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_quiesce_path": 0, 2024-04-30T08:45:27.578 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_quiesce_inode": 0, 2024-04-30T08:45:27.578 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_enqueue_scrub": 0, 2024-04-30T08:45:27.578 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_exportdir": 0, 2024-04-30T08:45:27.578 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_flush": 0, 2024-04-30T08:45:27.578 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_fragmentdir": 0, 2024-04-30T08:45:27.578 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_fragstats": 0, 2024-04-30T08:45:27.578 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_inodestats": 0 2024-04-30T08:45:27.578 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.579 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_log": { 2024-04-30T08:45:27.579 INFO:teuthology.orchestra.run.smithi071.stdout: "evlrg": 0, 2024-04-30T08:45:27.579 INFO:teuthology.orchestra.run.smithi071.stdout: "evadd": 0, 2024-04-30T08:45:27.579 INFO:teuthology.orchestra.run.smithi071.stdout: "evex": 0, 2024-04-30T08:45:27.579 INFO:teuthology.orchestra.run.smithi071.stdout: "evtrm": 0, 2024-04-30T08:45:27.579 INFO:teuthology.orchestra.run.smithi071.stdout: "ev": 14542, 2024-04-30T08:45:27.579 INFO:teuthology.orchestra.run.smithi071.stdout: "evexg": 0, 2024-04-30T08:45:27.579 INFO:teuthology.orchestra.run.smithi071.stdout: "evexd": 0, 2024-04-30T08:45:27.579 INFO:teuthology.orchestra.run.smithi071.stdout: "segadd": 0, 2024-04-30T08:45:27.579 INFO:teuthology.orchestra.run.smithi071.stdout: "segex": 0, 2024-04-30T08:45:27.579 INFO:teuthology.orchestra.run.smithi071.stdout: "segtrm": 0, 2024-04-30T08:45:27.579 INFO:teuthology.orchestra.run.smithi071.stdout: "seg": 9, 2024-04-30T08:45:27.579 INFO:teuthology.orchestra.run.smithi071.stdout: "segmjr": 15, 2024-04-30T08:45:27.580 INFO:teuthology.orchestra.run.smithi071.stdout: "segexg": 0, 2024-04-30T08:45:27.580 INFO:teuthology.orchestra.run.smithi071.stdout: "segexd": 0, 2024-04-30T08:45:27.580 INFO:teuthology.orchestra.run.smithi071.stdout: "expos": 17025879, 2024-04-30T08:45:27.580 INFO:teuthology.orchestra.run.smithi071.stdout: "wrpos": 35220218, 2024-04-30T08:45:27.580 INFO:teuthology.orchestra.run.smithi071.stdout: "rdpos": 35217721, 2024-04-30T08:45:27.580 INFO:teuthology.orchestra.run.smithi071.stdout: "jlat": { 2024-04-30T08:45:27.580 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.580 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.580 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.580 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.580 INFO:teuthology.orchestra.run.smithi071.stdout: "replayed": 14542 2024-04-30T08:45:27.580 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.580 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_mem": { 2024-04-30T08:45:27.580 INFO:teuthology.orchestra.run.smithi071.stdout: "ino": 21, 2024-04-30T08:45:27.581 INFO:teuthology.orchestra.run.smithi071.stdout: "ino+": 892, 2024-04-30T08:45:27.581 INFO:teuthology.orchestra.run.smithi071.stdout: "ino-": 871, 2024-04-30T08:45:27.581 INFO:teuthology.orchestra.run.smithi071.stdout: "dir": 16, 2024-04-30T08:45:27.581 INFO:teuthology.orchestra.run.smithi071.stdout: "dir+": 121, 2024-04-30T08:45:27.581 INFO:teuthology.orchestra.run.smithi071.stdout: "dir-": 105, 2024-04-30T08:45:27.581 INFO:teuthology.orchestra.run.smithi071.stdout: "dn": 1027, 2024-04-30T08:45:27.581 INFO:teuthology.orchestra.run.smithi071.stdout: "dn+": 1535, 2024-04-30T08:45:27.581 INFO:teuthology.orchestra.run.smithi071.stdout: "dn-": 508, 2024-04-30T08:45:27.581 INFO:teuthology.orchestra.run.smithi071.stdout: "cap": 0, 2024-04-30T08:45:27.581 INFO:teuthology.orchestra.run.smithi071.stdout: "cap+": 0, 2024-04-30T08:45:27.581 INFO:teuthology.orchestra.run.smithi071.stdout: "cap-": 0, 2024-04-30T08:45:27.581 INFO:teuthology.orchestra.run.smithi071.stdout: "rss": 124044, 2024-04-30T08:45:27.581 INFO:teuthology.orchestra.run.smithi071.stdout: "heap": 215440 2024-04-30T08:45:27.582 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.582 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_server": { 2024-04-30T08:45:27.582 INFO:teuthology.orchestra.run.smithi071.stdout: "dispatch_client_request": 0, 2024-04-30T08:45:27.582 INFO:teuthology.orchestra.run.smithi071.stdout: "dispatch_server_request": 0, 2024-04-30T08:45:27.582 INFO:teuthology.orchestra.run.smithi071.stdout: "handle_client_request": 0, 2024-04-30T08:45:27.582 INFO:teuthology.orchestra.run.smithi071.stdout: "handle_client_session": 0, 2024-04-30T08:45:27.582 INFO:teuthology.orchestra.run.smithi071.stdout: "handle_peer_request": 0, 2024-04-30T08:45:27.582 INFO:teuthology.orchestra.run.smithi071.stdout: "req_create_latency": { 2024-04-30T08:45:27.582 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.582 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.582 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.582 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.582 INFO:teuthology.orchestra.run.smithi071.stdout: "req_getattr_latency": { 2024-04-30T08:45:27.583 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.583 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.583 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.583 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.583 INFO:teuthology.orchestra.run.smithi071.stdout: "req_getfilelock_latency": { 2024-04-30T08:45:27.583 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.583 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.583 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.583 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.583 INFO:teuthology.orchestra.run.smithi071.stdout: "req_link_latency": { 2024-04-30T08:45:27.583 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.583 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.583 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.583 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.584 INFO:teuthology.orchestra.run.smithi071.stdout: "req_lookup_latency": { 2024-04-30T08:45:27.584 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.584 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.584 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.584 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.584 INFO:teuthology.orchestra.run.smithi071.stdout: "req_lookuphash_latency": { 2024-04-30T08:45:27.584 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.584 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.584 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.584 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.584 INFO:teuthology.orchestra.run.smithi071.stdout: "req_lookupino_latency": { 2024-04-30T08:45:27.584 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.584 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.585 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.585 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.585 INFO:teuthology.orchestra.run.smithi071.stdout: "req_lookupname_latency": { 2024-04-30T08:45:27.585 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.585 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.585 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.585 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.585 INFO:teuthology.orchestra.run.smithi071.stdout: "req_lookupparent_latency": { 2024-04-30T08:45:27.585 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.585 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.585 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.585 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.585 INFO:teuthology.orchestra.run.smithi071.stdout: "req_lookupsnap_latency": { 2024-04-30T08:45:27.585 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.586 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.586 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.586 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.586 INFO:teuthology.orchestra.run.smithi071.stdout: "req_lssnap_latency": { 2024-04-30T08:45:27.586 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.586 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.586 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.586 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.586 INFO:teuthology.orchestra.run.smithi071.stdout: "req_mkdir_latency": { 2024-04-30T08:45:27.587 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.587 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.587 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.587 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.587 INFO:teuthology.orchestra.run.smithi071.stdout: "req_mknod_latency": { 2024-04-30T08:45:27.587 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.587 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.587 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.588 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.588 INFO:teuthology.orchestra.run.smithi071.stdout: "req_mksnap_latency": { 2024-04-30T08:45:27.588 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.588 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.588 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.588 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.588 INFO:teuthology.orchestra.run.smithi071.stdout: "req_open_latency": { 2024-04-30T08:45:27.588 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.588 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.588 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.588 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.588 INFO:teuthology.orchestra.run.smithi071.stdout: "req_readdir_latency": { 2024-04-30T08:45:27.588 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.589 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.589 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.589 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.589 INFO:teuthology.orchestra.run.smithi071.stdout: "req_rename_latency": { 2024-04-30T08:45:27.589 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.589 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.589 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.589 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.589 INFO:teuthology.orchestra.run.smithi071.stdout: "req_renamesnap_latency": { 2024-04-30T08:45:27.589 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.589 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.589 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.589 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.589 INFO:teuthology.orchestra.run.smithi071.stdout: "req_snapdiff_latency": { 2024-04-30T08:45:27.590 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.590 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.590 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.590 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.590 INFO:teuthology.orchestra.run.smithi071.stdout: "req_rmdir_latency": { 2024-04-30T08:45:27.590 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.590 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.590 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.590 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.590 INFO:teuthology.orchestra.run.smithi071.stdout: "req_rmsnap_latency": { 2024-04-30T08:45:27.590 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.590 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.590 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.591 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.591 INFO:teuthology.orchestra.run.smithi071.stdout: "req_rmxattr_latency": { 2024-04-30T08:45:27.591 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.591 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.591 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.591 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.591 INFO:teuthology.orchestra.run.smithi071.stdout: "req_setattr_latency": { 2024-04-30T08:45:27.591 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.591 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.591 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.591 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.591 INFO:teuthology.orchestra.run.smithi071.stdout: "req_setdirlayout_latency": { 2024-04-30T08:45:27.591 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.592 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.592 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.592 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.592 INFO:teuthology.orchestra.run.smithi071.stdout: "req_setfilelock_latency": { 2024-04-30T08:45:27.592 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.592 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.592 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.592 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.592 INFO:teuthology.orchestra.run.smithi071.stdout: "req_setlayout_latency": { 2024-04-30T08:45:27.592 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.592 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.592 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.592 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.592 INFO:teuthology.orchestra.run.smithi071.stdout: "req_setxattr_latency": { 2024-04-30T08:45:27.593 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.593 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.593 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.593 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.593 INFO:teuthology.orchestra.run.smithi071.stdout: "req_symlink_latency": { 2024-04-30T08:45:27.593 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.593 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.593 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.593 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.593 INFO:teuthology.orchestra.run.smithi071.stdout: "req_unlink_latency": { 2024-04-30T08:45:27.593 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.593 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.594 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.594 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.594 INFO:teuthology.orchestra.run.smithi071.stdout: "cap_revoke_eviction": 0, 2024-04-30T08:45:27.594 INFO:teuthology.orchestra.run.smithi071.stdout: "cap_acquisition_throttle": 0, 2024-04-30T08:45:27.594 INFO:teuthology.orchestra.run.smithi071.stdout: "req_getvxattr_latency": { 2024-04-30T08:45:27.594 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.594 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.594 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.594 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:27.594 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.594 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_sessions": { 2024-04-30T08:45:27.594 INFO:teuthology.orchestra.run.smithi071.stdout: "session_count": 1, 2024-04-30T08:45:27.594 INFO:teuthology.orchestra.run.smithi071.stdout: "session_add": 4, 2024-04-30T08:45:27.595 INFO:teuthology.orchestra.run.smithi071.stdout: "session_remove": 3, 2024-04-30T08:45:27.595 INFO:teuthology.orchestra.run.smithi071.stdout: "sessions_open": 2, 2024-04-30T08:45:27.595 INFO:teuthology.orchestra.run.smithi071.stdout: "sessions_stale": 0, 2024-04-30T08:45:27.595 INFO:teuthology.orchestra.run.smithi071.stdout: "total_load": 0, 2024-04-30T08:45:27.595 INFO:teuthology.orchestra.run.smithi071.stdout: "average_load": 0, 2024-04-30T08:45:27.595 INFO:teuthology.orchestra.run.smithi071.stdout: "avg_session_uptime": 3872, 2024-04-30T08:45:27.595 INFO:teuthology.orchestra.run.smithi071.stdout: "mdthresh_evicted": 0 2024-04-30T08:45:27.595 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.595 INFO:teuthology.orchestra.run.smithi071.stdout: "mempool": { 2024-04-30T08:45:27.595 INFO:teuthology.orchestra.run.smithi071.stdout: "bloom_filter_bytes": 0, 2024-04-30T08:45:27.595 INFO:teuthology.orchestra.run.smithi071.stdout: "bloom_filter_items": 0, 2024-04-30T08:45:27.595 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_alloc_bytes": 0, 2024-04-30T08:45:27.595 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_alloc_items": 0, 2024-04-30T08:45:27.596 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_data_bytes": 0, 2024-04-30T08:45:27.596 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_data_items": 0, 2024-04-30T08:45:27.596 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-30T08:45:27.596 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_onode_items": 0, 2024-04-30T08:45:27.596 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-30T08:45:27.596 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_meta_items": 0, 2024-04-30T08:45:27.596 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_other_bytes": 0, 2024-04-30T08:45:27.596 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_other_items": 0, 2024-04-30T08:45:27.596 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-30T08:45:27.596 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_buffer_items": 0, 2024-04-30T08:45:27.596 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_extent_bytes": 0, 2024-04-30T08:45:27.596 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_extent_items": 0, 2024-04-30T08:45:27.596 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_blob_bytes": 0, 2024-04-30T08:45:27.596 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_blob_items": 0, 2024-04-30T08:45:27.597 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-30T08:45:27.597 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_shared_blob_items": 0, 2024-04-30T08:45:27.597 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-30T08:45:27.597 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_inline_bl_items": 0, 2024-04-30T08:45:27.597 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_fsck_bytes": 0, 2024-04-30T08:45:27.597 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_fsck_items": 0, 2024-04-30T08:45:27.597 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_txc_bytes": 0, 2024-04-30T08:45:27.597 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_txc_items": 0, 2024-04-30T08:45:27.597 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-30T08:45:27.597 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_writing_deferred_items": 0, 2024-04-30T08:45:27.597 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_writing_bytes": 0, 2024-04-30T08:45:27.597 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_writing_items": 0, 2024-04-30T08:45:27.597 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_bytes": 0, 2024-04-30T08:45:27.598 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_items": 0, 2024-04-30T08:45:27.598 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_file_reader_bytes": 0, 2024-04-30T08:45:27.598 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_file_reader_items": 0, 2024-04-30T08:45:27.598 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_file_writer_bytes": 0, 2024-04-30T08:45:27.598 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_file_writer_items": 0, 2024-04-30T08:45:27.598 INFO:teuthology.orchestra.run.smithi071.stdout: "buffer_anon_bytes": 205595, 2024-04-30T08:45:27.598 INFO:teuthology.orchestra.run.smithi071.stdout: "buffer_anon_items": 60, 2024-04-30T08:45:27.598 INFO:teuthology.orchestra.run.smithi071.stdout: "buffer_meta_bytes": 0, 2024-04-30T08:45:27.598 INFO:teuthology.orchestra.run.smithi071.stdout: "buffer_meta_items": 0, 2024-04-30T08:45:27.598 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_bytes": 0, 2024-04-30T08:45:27.598 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_items": 0, 2024-04-30T08:45:27.598 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_mapbl_bytes": 0, 2024-04-30T08:45:27.598 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_mapbl_items": 0, 2024-04-30T08:45:27.598 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pglog_bytes": 0, 2024-04-30T08:45:27.599 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pglog_items": 0, 2024-04-30T08:45:27.599 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_bytes": 9088, 2024-04-30T08:45:27.599 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_items": 175, 2024-04-30T08:45:27.599 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_mapping_bytes": 0, 2024-04-30T08:45:27.599 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_mapping_items": 0, 2024-04-30T08:45:27.599 INFO:teuthology.orchestra.run.smithi071.stdout: "pgmap_bytes": 0, 2024-04-30T08:45:27.599 INFO:teuthology.orchestra.run.smithi071.stdout: "pgmap_items": 0, 2024-04-30T08:45:27.599 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_co_bytes": 791151, 2024-04-30T08:45:27.599 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_co_items": 4596, 2024-04-30T08:45:27.599 INFO:teuthology.orchestra.run.smithi071.stdout: "unittest_1_bytes": 0, 2024-04-30T08:45:27.599 INFO:teuthology.orchestra.run.smithi071.stdout: "unittest_1_items": 0, 2024-04-30T08:45:27.599 INFO:teuthology.orchestra.run.smithi071.stdout: "unittest_2_bytes": 0, 2024-04-30T08:45:27.599 INFO:teuthology.orchestra.run.smithi071.stdout: "unittest_2_items": 0 2024-04-30T08:45:27.600 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.600 INFO:teuthology.orchestra.run.smithi071.stdout: "objecter": { 2024-04-30T08:45:27.600 INFO:teuthology.orchestra.run.smithi071.stdout: "op_active": 0, 2024-04-30T08:45:27.600 INFO:teuthology.orchestra.run.smithi071.stdout: "op_laggy": 0, 2024-04-30T08:45:27.600 INFO:teuthology.orchestra.run.smithi071.stdout: "op_send": 1296, 2024-04-30T08:45:27.600 INFO:teuthology.orchestra.run.smithi071.stdout: "op_send_bytes": 26, 2024-04-30T08:45:27.600 INFO:teuthology.orchestra.run.smithi071.stdout: "op_resend": 4, 2024-04-30T08:45:27.600 INFO:teuthology.orchestra.run.smithi071.stdout: "op_reply": 1296, 2024-04-30T08:45:27.600 INFO:teuthology.orchestra.run.smithi071.stdout: "op_latency": { 2024-04-30T08:45:27.600 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 1296, 2024-04-30T08:45:27.600 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 386.169205734, 2024-04-30T08:45:27.600 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.297970066 2024-04-30T08:45:27.600 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.600 INFO:teuthology.orchestra.run.smithi071.stdout: "op_inflight": 0, 2024-04-30T08:45:27.601 INFO:teuthology.orchestra.run.smithi071.stdout: "oplen_avg": { 2024-04-30T08:45:27.601 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 1296, 2024-04-30T08:45:27.601 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 1297 2024-04-30T08:45:27.601 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.601 INFO:teuthology.orchestra.run.smithi071.stdout: "op": 1296, 2024-04-30T08:45:27.601 INFO:teuthology.orchestra.run.smithi071.stdout: "op_r": 1296, 2024-04-30T08:45:27.601 INFO:teuthology.orchestra.run.smithi071.stdout: "op_w": 0, 2024-04-30T08:45:27.601 INFO:teuthology.orchestra.run.smithi071.stdout: "op_rmw": 0, 2024-04-30T08:45:27.601 INFO:teuthology.orchestra.run.smithi071.stdout: "op_pg": 0, 2024-04-30T08:45:27.601 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_stat": 783, 2024-04-30T08:45:27.601 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_create": 0, 2024-04-30T08:45:27.601 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_read": 510, 2024-04-30T08:45:27.601 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_write": 0, 2024-04-30T08:45:27.602 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_writefull": 0, 2024-04-30T08:45:27.602 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_writesame": 0, 2024-04-30T08:45:27.602 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_append": 0, 2024-04-30T08:45:27.602 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_zero": 0, 2024-04-30T08:45:27.602 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_truncate": 0, 2024-04-30T08:45:27.602 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_delete": 0, 2024-04-30T08:45:27.602 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_mapext": 0, 2024-04-30T08:45:27.602 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_sparse_read": 0, 2024-04-30T08:45:27.602 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_clonerange": 0, 2024-04-30T08:45:27.602 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_getxattr": 2, 2024-04-30T08:45:27.602 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_setxattr": 0, 2024-04-30T08:45:27.602 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_cmpxattr": 0, 2024-04-30T08:45:27.602 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_rmxattr": 0, 2024-04-30T08:45:27.602 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_resetxattrs": 0, 2024-04-30T08:45:27.603 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_call": 0, 2024-04-30T08:45:27.603 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_watch": 0, 2024-04-30T08:45:27.603 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_notify": 0, 2024-04-30T08:45:27.603 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_src_cmpxattr": 0, 2024-04-30T08:45:27.603 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_pgls": 0, 2024-04-30T08:45:27.603 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_pgls_filter": 0, 2024-04-30T08:45:27.603 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_other": 0, 2024-04-30T08:45:27.603 INFO:teuthology.orchestra.run.smithi071.stdout: "linger_active": 0, 2024-04-30T08:45:27.603 INFO:teuthology.orchestra.run.smithi071.stdout: "linger_send": 0, 2024-04-30T08:45:27.603 INFO:teuthology.orchestra.run.smithi071.stdout: "linger_resend": 0, 2024-04-30T08:45:27.603 INFO:teuthology.orchestra.run.smithi071.stdout: "linger_ping": 0, 2024-04-30T08:45:27.603 INFO:teuthology.orchestra.run.smithi071.stdout: "poolop_active": 0, 2024-04-30T08:45:27.603 INFO:teuthology.orchestra.run.smithi071.stdout: "poolop_send": 0, 2024-04-30T08:45:27.604 INFO:teuthology.orchestra.run.smithi071.stdout: "poolop_resend": 0, 2024-04-30T08:45:27.604 INFO:teuthology.orchestra.run.smithi071.stdout: "poolstat_active": 0, 2024-04-30T08:45:27.604 INFO:teuthology.orchestra.run.smithi071.stdout: "poolstat_send": 0, 2024-04-30T08:45:27.604 INFO:teuthology.orchestra.run.smithi071.stdout: "poolstat_resend": 0, 2024-04-30T08:45:27.604 INFO:teuthology.orchestra.run.smithi071.stdout: "statfs_active": 0, 2024-04-30T08:45:27.604 INFO:teuthology.orchestra.run.smithi071.stdout: "statfs_send": 0, 2024-04-30T08:45:27.604 INFO:teuthology.orchestra.run.smithi071.stdout: "statfs_resend": 0, 2024-04-30T08:45:27.604 INFO:teuthology.orchestra.run.smithi071.stdout: "command_active": 0, 2024-04-30T08:45:27.604 INFO:teuthology.orchestra.run.smithi071.stdout: "command_send": 0, 2024-04-30T08:45:27.604 INFO:teuthology.orchestra.run.smithi071.stdout: "command_resend": 0, 2024-04-30T08:45:27.604 INFO:teuthology.orchestra.run.smithi071.stdout: "map_epoch": 85, 2024-04-30T08:45:27.604 INFO:teuthology.orchestra.run.smithi071.stdout: "map_full": 0, 2024-04-30T08:45:27.604 INFO:teuthology.orchestra.run.smithi071.stdout: "map_inc": 3, 2024-04-30T08:45:27.604 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_sessions": 7, 2024-04-30T08:45:27.605 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_session_open": 7, 2024-04-30T08:45:27.605 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_session_close": 0, 2024-04-30T08:45:27.605 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_laggy": 0, 2024-04-30T08:45:27.605 INFO:teuthology.orchestra.run.smithi071.stdout: "omap_wr": 0, 2024-04-30T08:45:27.605 INFO:teuthology.orchestra.run.smithi071.stdout: "omap_rd": 2, 2024-04-30T08:45:27.605 INFO:teuthology.orchestra.run.smithi071.stdout: "omap_del": 0 2024-04-30T08:45:27.605 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.605 INFO:teuthology.orchestra.run.smithi071.stdout: "oft": { 2024-04-30T08:45:27.605 INFO:teuthology.orchestra.run.smithi071.stdout: "omap_total_objs": 0, 2024-04-30T08:45:27.605 INFO:teuthology.orchestra.run.smithi071.stdout: "omap_total_kv_pairs": 0, 2024-04-30T08:45:27.605 INFO:teuthology.orchestra.run.smithi071.stdout: "omap_total_updates": 0, 2024-04-30T08:45:27.605 INFO:teuthology.orchestra.run.smithi071.stdout: "omap_total_removes": 0 2024-04-30T08:45:27.605 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.605 INFO:teuthology.orchestra.run.smithi071.stdout: "osd-slow-ops": { 2024-04-30T08:45:27.606 INFO:teuthology.orchestra.run.smithi071.stdout: "slow_ops_count": 0 2024-04-30T08:45:27.606 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.606 INFO:teuthology.orchestra.run.smithi071.stdout: "purge_queue": { 2024-04-30T08:45:27.606 INFO:teuthology.orchestra.run.smithi071.stdout: "pq_executing_ops": 0, 2024-04-30T08:45:27.606 INFO:teuthology.orchestra.run.smithi071.stdout: "pq_executing_ops_high_water": 0, 2024-04-30T08:45:27.606 INFO:teuthology.orchestra.run.smithi071.stdout: "pq_executing": 0, 2024-04-30T08:45:27.606 INFO:teuthology.orchestra.run.smithi071.stdout: "pq_executing_high_water": 0, 2024-04-30T08:45:27.606 INFO:teuthology.orchestra.run.smithi071.stdout: "pq_executed_ops": 0, 2024-04-30T08:45:27.606 INFO:teuthology.orchestra.run.smithi071.stdout: "pq_executed": 0, 2024-04-30T08:45:27.606 INFO:teuthology.orchestra.run.smithi071.stdout: "pq_item_in_journal": 0 2024-04-30T08:45:27.606 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.606 INFO:teuthology.orchestra.run.smithi071.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-30T08:45:27.606 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 0, 2024-04-30T08:45:27.607 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 104857600, 2024-04-30T08:45:27.607 INFO:teuthology.orchestra.run.smithi071.stdout: "get_started": 0, 2024-04-30T08:45:27.607 INFO:teuthology.orchestra.run.smithi071.stdout: "get": 1644, 2024-04-30T08:45:27.607 INFO:teuthology.orchestra.run.smithi071.stdout: "get_sum": 32754739, 2024-04-30T08:45:27.607 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:27.607 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_success": 1644, 2024-04-30T08:45:27.607 INFO:teuthology.orchestra.run.smithi071.stdout: "take": 0, 2024-04-30T08:45:27.607 INFO:teuthology.orchestra.run.smithi071.stdout: "take_sum": 0, 2024-04-30T08:45:27.607 INFO:teuthology.orchestra.run.smithi071.stdout: "put": 1644, 2024-04-30T08:45:27.607 INFO:teuthology.orchestra.run.smithi071.stdout: "put_sum": 32754739, 2024-04-30T08:45:27.607 INFO:teuthology.orchestra.run.smithi071.stdout: "wait": { 2024-04-30T08:45:27.607 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.607 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.607 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.608 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:27.608 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.608 INFO:teuthology.orchestra.run.smithi071.stdout: "throttle-objecter_bytes": { 2024-04-30T08:45:27.608 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 0, 2024-04-30T08:45:27.608 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 104857600, 2024-04-30T08:45:27.608 INFO:teuthology.orchestra.run.smithi071.stdout: "get_started": 0, 2024-04-30T08:45:27.608 INFO:teuthology.orchestra.run.smithi071.stdout: "get": 0, 2024-04-30T08:45:27.608 INFO:teuthology.orchestra.run.smithi071.stdout: "get_sum": 0, 2024-04-30T08:45:27.608 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:27.608 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_success": 0, 2024-04-30T08:45:27.608 INFO:teuthology.orchestra.run.smithi071.stdout: "take": 1296, 2024-04-30T08:45:27.608 INFO:teuthology.orchestra.run.smithi071.stdout: "take_sum": 31025924, 2024-04-30T08:45:27.608 INFO:teuthology.orchestra.run.smithi071.stdout: "put": 1296, 2024-04-30T08:45:27.608 INFO:teuthology.orchestra.run.smithi071.stdout: "put_sum": 31025924, 2024-04-30T08:45:27.609 INFO:teuthology.orchestra.run.smithi071.stdout: "wait": { 2024-04-30T08:45:27.609 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.609 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.609 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.609 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:27.609 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.609 INFO:teuthology.orchestra.run.smithi071.stdout: "throttle-objecter_ops": { 2024-04-30T08:45:27.609 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 0, 2024-04-30T08:45:27.609 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 1024, 2024-04-30T08:45:27.609 INFO:teuthology.orchestra.run.smithi071.stdout: "get_started": 0, 2024-04-30T08:45:27.609 INFO:teuthology.orchestra.run.smithi071.stdout: "get": 0, 2024-04-30T08:45:27.609 INFO:teuthology.orchestra.run.smithi071.stdout: "get_sum": 0, 2024-04-30T08:45:27.609 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:27.610 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_success": 0, 2024-04-30T08:45:27.610 INFO:teuthology.orchestra.run.smithi071.stdout: "take": 1296, 2024-04-30T08:45:27.610 INFO:teuthology.orchestra.run.smithi071.stdout: "take_sum": 1296, 2024-04-30T08:45:27.610 INFO:teuthology.orchestra.run.smithi071.stdout: "put": 1296, 2024-04-30T08:45:27.610 INFO:teuthology.orchestra.run.smithi071.stdout: "put_sum": 1296, 2024-04-30T08:45:27.610 INFO:teuthology.orchestra.run.smithi071.stdout: "wait": { 2024-04-30T08:45:27.610 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.610 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.610 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.610 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:27.610 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.610 INFO:teuthology.orchestra.run.smithi071.stdout: "throttle-write_buf_throttle": { 2024-04-30T08:45:27.610 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 0, 2024-04-30T08:45:27.610 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 3758096384, 2024-04-30T08:45:27.611 INFO:teuthology.orchestra.run.smithi071.stdout: "get_started": 0, 2024-04-30T08:45:27.611 INFO:teuthology.orchestra.run.smithi071.stdout: "get": 0, 2024-04-30T08:45:27.611 INFO:teuthology.orchestra.run.smithi071.stdout: "get_sum": 0, 2024-04-30T08:45:27.611 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:27.611 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_success": 0, 2024-04-30T08:45:27.611 INFO:teuthology.orchestra.run.smithi071.stdout: "take": 0, 2024-04-30T08:45:27.611 INFO:teuthology.orchestra.run.smithi071.stdout: "take_sum": 0, 2024-04-30T08:45:27.611 INFO:teuthology.orchestra.run.smithi071.stdout: "put": 0, 2024-04-30T08:45:27.611 INFO:teuthology.orchestra.run.smithi071.stdout: "put_sum": 0, 2024-04-30T08:45:27.611 INFO:teuthology.orchestra.run.smithi071.stdout: "wait": { 2024-04-30T08:45:27.611 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.611 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.611 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.612 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:27.612 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:27.612 INFO:teuthology.orchestra.run.smithi071.stdout: "throttle-write_buf_throttle-0x557a951568f0": { 2024-04-30T08:45:27.612 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 0, 2024-04-30T08:45:27.612 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 3758096384, 2024-04-30T08:45:27.612 INFO:teuthology.orchestra.run.smithi071.stdout: "get_started": 0, 2024-04-30T08:45:27.612 INFO:teuthology.orchestra.run.smithi071.stdout: "get": 0, 2024-04-30T08:45:27.612 INFO:teuthology.orchestra.run.smithi071.stdout: "get_sum": 0, 2024-04-30T08:45:27.612 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:27.612 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_success": 0, 2024-04-30T08:45:27.612 INFO:teuthology.orchestra.run.smithi071.stdout: "take": 0, 2024-04-30T08:45:27.612 INFO:teuthology.orchestra.run.smithi071.stdout: "take_sum": 0, 2024-04-30T08:45:27.612 INFO:teuthology.orchestra.run.smithi071.stdout: "put": 0, 2024-04-30T08:45:27.612 INFO:teuthology.orchestra.run.smithi071.stdout: "put_sum": 0, 2024-04-30T08:45:27.613 INFO:teuthology.orchestra.run.smithi071.stdout: "wait": { 2024-04-30T08:45:27.613 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:27.613 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:27.613 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:27.613 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:27.613 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:27.613 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-30T08:45:28.364 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-04-30T08:45:28.365 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-04-30T08:45:28.365 INFO:tasks.check_counter:Daemon mds.e mds.dir_split=0 2024-04-30T08:45:28.365 DEBUG:tasks.check_counter:Getting stats from h 2024-04-30T08:45:28.365 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-30T08:45:28.604 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.b/config 2024-04-30T08:45:29.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:29 smithi032 ceph-mon[28331]: pgmap v734: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 9 op/s 2024-04-30T08:45:29.506 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:29 smithi071 ceph-mon[33385]: pgmap v734: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 9 op/s 2024-04-30T08:45:29.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:29 smithi189 ceph-mon[33038]: pgmap v734: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 9 op/s 2024-04-30T08:45:30.437 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-30T08:45:30.437 INFO:teuthology.orchestra.run.smithi071.stdout: "AsyncMessenger::Worker-0": { 2024-04-30T08:45:30.438 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_messages": 1384, 2024-04-30T08:45:30.438 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_messages": 1390, 2024-04-30T08:45:30.438 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_bytes": 1865646, 2024-04-30T08:45:30.438 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_bytes": 515445, 2024-04-30T08:45:30.438 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_created_connections": 7, 2024-04-30T08:45:30.438 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_active_connections": 2, 2024-04-30T08:45:30.438 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_total_time": 0.145387271, 2024-04-30T08:45:30.438 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_send_time": 0.055189940, 2024-04-30T08:45:30.438 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_recv_time": 0.090104698, 2024-04-30T08:45:30.438 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_fast_dispatch_time": 0.010637113, 2024-04-30T08:45:30.438 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:30.438 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 1393, 2024-04-30T08:45:30.439 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.078747918, 2024-04-30T08:45:30.439 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000056531 2024-04-30T08:45:30.439 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.439 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:30.439 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 1898, 2024-04-30T08:45:30.439 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000469603, 2024-04-30T08:45:30.439 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000247 2024-04-30T08:45:30.439 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.439 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_encrypted_bytes": 1550272, 2024-04-30T08:45:30.439 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_encrypted_bytes": 147600 2024-04-30T08:45:30.439 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.439 INFO:teuthology.orchestra.run.smithi071.stdout: "AsyncMessenger::Worker-1": { 2024-04-30T08:45:30.439 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_messages": 30, 2024-04-30T08:45:30.439 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_messages": 199, 2024-04-30T08:45:30.440 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_bytes": 10053, 2024-04-30T08:45:30.440 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_bytes": 461539, 2024-04-30T08:45:30.440 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_created_connections": 8, 2024-04-30T08:45:30.440 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_active_connections": 5, 2024-04-30T08:45:30.440 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_total_time": 0.012722819, 2024-04-30T08:45:30.440 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_send_time": 0.007375651, 2024-04-30T08:45:30.440 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_recv_time": 0.003077446, 2024-04-30T08:45:30.440 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_fast_dispatch_time": 0.000773408, 2024-04-30T08:45:30.440 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:30.440 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 198, 2024-04-30T08:45:30.440 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.026826911, 2024-04-30T08:45:30.440 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000135489 2024-04-30T08:45:30.440 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.441 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:30.441 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:30.441 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:30.441 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:30.441 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.441 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-30T08:45:30.441 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_encrypted_bytes": 449808 2024-04-30T08:45:30.441 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.441 INFO:teuthology.orchestra.run.smithi071.stdout: "AsyncMessenger::Worker-2": { 2024-04-30T08:45:30.441 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_messages": 69, 2024-04-30T08:45:30.441 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_messages": 80, 2024-04-30T08:45:30.441 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_bytes": 17489, 2024-04-30T08:45:30.441 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_bytes": 192824, 2024-04-30T08:45:30.441 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_created_connections": 22, 2024-04-30T08:45:30.442 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_active_connections": 8, 2024-04-30T08:45:30.442 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_total_time": 0.014038445, 2024-04-30T08:45:30.442 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_send_time": 0.002774898, 2024-04-30T08:45:30.442 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_recv_time": 0.010042871, 2024-04-30T08:45:30.442 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_fast_dispatch_time": 0.000733784, 2024-04-30T08:45:30.442 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:30.442 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 74, 2024-04-30T08:45:30.442 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.016790809, 2024-04-30T08:45:30.442 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000226902 2024-04-30T08:45:30.442 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.442 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:30.442 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 41, 2024-04-30T08:45:30.442 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000017729, 2024-04-30T08:45:30.443 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000432 2024-04-30T08:45:30.443 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.443 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-30T08:45:30.443 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_encrypted_bytes": 0 2024-04-30T08:45:30.443 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.443 INFO:teuthology.orchestra.run.smithi071.stdout: "cct": { 2024-04-30T08:45:30.443 INFO:teuthology.orchestra.run.smithi071.stdout: "total_workers": 1, 2024-04-30T08:45:30.443 INFO:teuthology.orchestra.run.smithi071.stdout: "unhealthy_workers": 0 2024-04-30T08:45:30.443 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.443 INFO:teuthology.orchestra.run.smithi071.stdout: "finisher-MDSRank": { 2024-04-30T08:45:30.443 INFO:teuthology.orchestra.run.smithi071.stdout: "queue_len": 0, 2024-04-30T08:45:30.443 INFO:teuthology.orchestra.run.smithi071.stdout: "complete_latency": { 2024-04-30T08:45:30.443 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 86, 2024-04-30T08:45:30.443 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.007259906, 2024-04-30T08:45:30.444 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000084417 2024-04-30T08:45:30.444 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:30.444 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.444 INFO:teuthology.orchestra.run.smithi071.stdout: "finisher-PurgeQueue": { 2024-04-30T08:45:30.444 INFO:teuthology.orchestra.run.smithi071.stdout: "queue_len": 0, 2024-04-30T08:45:30.444 INFO:teuthology.orchestra.run.smithi071.stdout: "complete_latency": { 2024-04-30T08:45:30.444 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 17, 2024-04-30T08:45:30.444 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000718686, 2024-04-30T08:45:30.444 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000042275 2024-04-30T08:45:30.444 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:30.444 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.444 INFO:teuthology.orchestra.run.smithi071.stdout: "mds": { 2024-04-30T08:45:30.444 INFO:teuthology.orchestra.run.smithi071.stdout: "request": 14, 2024-04-30T08:45:30.444 INFO:teuthology.orchestra.run.smithi071.stdout: "reply": 4, 2024-04-30T08:45:30.445 INFO:teuthology.orchestra.run.smithi071.stdout: "reply_latency": { 2024-04-30T08:45:30.445 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 4, 2024-04-30T08:45:30.445 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.007251840, 2024-04-30T08:45:30.445 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.001812960 2024-04-30T08:45:30.445 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.445 INFO:teuthology.orchestra.run.smithi071.stdout: "slow_reply": 0, 2024-04-30T08:45:30.445 INFO:teuthology.orchestra.run.smithi071.stdout: "forward": 10, 2024-04-30T08:45:30.445 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_fetch_complete": 0, 2024-04-30T08:45:30.445 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_fetch_keys": 0, 2024-04-30T08:45:30.445 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_commit": 11, 2024-04-30T08:45:30.445 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_split": 0, 2024-04-30T08:45:30.445 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_merge": 0, 2024-04-30T08:45:30.445 INFO:teuthology.orchestra.run.smithi071.stdout: "inodes": 15, 2024-04-30T08:45:30.446 INFO:teuthology.orchestra.run.smithi071.stdout: "inodes_top": 1, 2024-04-30T08:45:30.446 INFO:teuthology.orchestra.run.smithi071.stdout: "inodes_bottom": 14, 2024-04-30T08:45:30.446 INFO:teuthology.orchestra.run.smithi071.stdout: "inodes_pin_tail": 0, 2024-04-30T08:45:30.446 INFO:teuthology.orchestra.run.smithi071.stdout: "inodes_pinned": 13, 2024-04-30T08:45:30.446 INFO:teuthology.orchestra.run.smithi071.stdout: "inodes_expired": 14, 2024-04-30T08:45:30.446 INFO:teuthology.orchestra.run.smithi071.stdout: "inodes_with_caps": 0, 2024-04-30T08:45:30.446 INFO:teuthology.orchestra.run.smithi071.stdout: "caps": 0, 2024-04-30T08:45:30.446 INFO:teuthology.orchestra.run.smithi071.stdout: "subtrees": 3, 2024-04-30T08:45:30.446 INFO:teuthology.orchestra.run.smithi071.stdout: "traverse": 25, 2024-04-30T08:45:30.446 INFO:teuthology.orchestra.run.smithi071.stdout: "traverse_hit": 6, 2024-04-30T08:45:30.446 INFO:teuthology.orchestra.run.smithi071.stdout: "traverse_forward": 10, 2024-04-30T08:45:30.446 INFO:teuthology.orchestra.run.smithi071.stdout: "traverse_discover": 3, 2024-04-30T08:45:30.446 INFO:teuthology.orchestra.run.smithi071.stdout: "traverse_dir_fetch": 0, 2024-04-30T08:45:30.446 INFO:teuthology.orchestra.run.smithi071.stdout: "traverse_remote_ino": 0, 2024-04-30T08:45:30.447 INFO:teuthology.orchestra.run.smithi071.stdout: "traverse_lock": 0, 2024-04-30T08:45:30.447 INFO:teuthology.orchestra.run.smithi071.stdout: "load_cent": 0, 2024-04-30T08:45:30.447 INFO:teuthology.orchestra.run.smithi071.stdout: "q": 0, 2024-04-30T08:45:30.447 INFO:teuthology.orchestra.run.smithi071.stdout: "exported": 2, 2024-04-30T08:45:30.447 INFO:teuthology.orchestra.run.smithi071.stdout: "exported_inodes": 0, 2024-04-30T08:45:30.447 INFO:teuthology.orchestra.run.smithi071.stdout: "imported": 2, 2024-04-30T08:45:30.447 INFO:teuthology.orchestra.run.smithi071.stdout: "imported_inodes": 4, 2024-04-30T08:45:30.447 INFO:teuthology.orchestra.run.smithi071.stdout: "openino_dir_fetch": 0, 2024-04-30T08:45:30.447 INFO:teuthology.orchestra.run.smithi071.stdout: "openino_backtrace_fetch": 4, 2024-04-30T08:45:30.447 INFO:teuthology.orchestra.run.smithi071.stdout: "openino_peer_discover": 0, 2024-04-30T08:45:30.447 INFO:teuthology.orchestra.run.smithi071.stdout: "root_rfiles": 325, 2024-04-30T08:45:30.447 INFO:teuthology.orchestra.run.smithi071.stdout: "root_rbytes": 13424230511, 2024-04-30T08:45:30.447 INFO:teuthology.orchestra.run.smithi071.stdout: "root_rsnaps": 0, 2024-04-30T08:45:30.447 INFO:teuthology.orchestra.run.smithi071.stdout: "scrub_backtrace_fetch": 4, 2024-04-30T08:45:30.448 INFO:teuthology.orchestra.run.smithi071.stdout: "scrub_set_tag": 0, 2024-04-30T08:45:30.448 INFO:teuthology.orchestra.run.smithi071.stdout: "scrub_backtrace_repaired": 0, 2024-04-30T08:45:30.448 INFO:teuthology.orchestra.run.smithi071.stdout: "scrub_inotable_repaired": 0, 2024-04-30T08:45:30.448 INFO:teuthology.orchestra.run.smithi071.stdout: "scrub_dir_inodes": 0, 2024-04-30T08:45:30.448 INFO:teuthology.orchestra.run.smithi071.stdout: "scrub_dir_base_inodes": 0, 2024-04-30T08:45:30.448 INFO:teuthology.orchestra.run.smithi071.stdout: "scrub_dirfrag_rstats": 0, 2024-04-30T08:45:30.448 INFO:teuthology.orchestra.run.smithi071.stdout: "scrub_file_inodes": 4, 2024-04-30T08:45:30.448 INFO:teuthology.orchestra.run.smithi071.stdout: "handle_inode_file_caps": 0, 2024-04-30T08:45:30.448 INFO:teuthology.orchestra.run.smithi071.stdout: "ceph_cap_op_revoke": 4, 2024-04-30T08:45:30.448 INFO:teuthology.orchestra.run.smithi071.stdout: "ceph_cap_op_grant": 4, 2024-04-30T08:45:30.448 INFO:teuthology.orchestra.run.smithi071.stdout: "ceph_cap_op_trunc": 0, 2024-04-30T08:45:30.448 INFO:teuthology.orchestra.run.smithi071.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-30T08:45:30.448 INFO:teuthology.orchestra.run.smithi071.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-30T08:45:30.448 INFO:teuthology.orchestra.run.smithi071.stdout: "handle_client_caps": 6, 2024-04-30T08:45:30.448 INFO:teuthology.orchestra.run.smithi071.stdout: "handle_client_caps_dirty": 0, 2024-04-30T08:45:30.449 INFO:teuthology.orchestra.run.smithi071.stdout: "handle_client_cap_release": 2, 2024-04-30T08:45:30.449 INFO:teuthology.orchestra.run.smithi071.stdout: "process_request_cap_release": 4 2024-04-30T08:45:30.449 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.449 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_cache": { 2024-04-30T08:45:30.449 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_update": 0, 2024-04-30T08:45:30.449 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_update_receipt": 0, 2024-04-30T08:45:30.449 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_try_discover": 0, 2024-04-30T08:45:30.449 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_send_discover": 3, 2024-04-30T08:45:30.449 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_handle_discover": 0, 2024-04-30T08:45:30.449 INFO:teuthology.orchestra.run.smithi071.stdout: "num_strays": 0, 2024-04-30T08:45:30.449 INFO:teuthology.orchestra.run.smithi071.stdout: "num_strays_delayed": 0, 2024-04-30T08:45:30.449 INFO:teuthology.orchestra.run.smithi071.stdout: "num_strays_enqueuing": 0, 2024-04-30T08:45:30.449 INFO:teuthology.orchestra.run.smithi071.stdout: "strays_created": 2, 2024-04-30T08:45:30.449 INFO:teuthology.orchestra.run.smithi071.stdout: "strays_enqueued": 2, 2024-04-30T08:45:30.450 INFO:teuthology.orchestra.run.smithi071.stdout: "strays_reintegrated": 0, 2024-04-30T08:45:30.450 INFO:teuthology.orchestra.run.smithi071.stdout: "strays_migrated": 0, 2024-04-30T08:45:30.450 INFO:teuthology.orchestra.run.smithi071.stdout: "num_recovering_processing": 0, 2024-04-30T08:45:30.450 INFO:teuthology.orchestra.run.smithi071.stdout: "num_recovering_enqueued": 0, 2024-04-30T08:45:30.450 INFO:teuthology.orchestra.run.smithi071.stdout: "num_recovering_prioritized": 0, 2024-04-30T08:45:30.450 INFO:teuthology.orchestra.run.smithi071.stdout: "recovery_started": 0, 2024-04-30T08:45:30.450 INFO:teuthology.orchestra.run.smithi071.stdout: "recovery_completed": 0, 2024-04-30T08:45:30.450 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_quiesce_path": 0, 2024-04-30T08:45:30.450 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_quiesce_inode": 0, 2024-04-30T08:45:30.450 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_enqueue_scrub": 0, 2024-04-30T08:45:30.450 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_exportdir": 2, 2024-04-30T08:45:30.450 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_flush": 0, 2024-04-30T08:45:30.450 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_fragmentdir": 0, 2024-04-30T08:45:30.451 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_fragstats": 0, 2024-04-30T08:45:30.451 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_inodestats": 0 2024-04-30T08:45:30.451 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.451 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_log": { 2024-04-30T08:45:30.451 INFO:teuthology.orchestra.run.smithi071.stdout: "evlrg": 0, 2024-04-30T08:45:30.451 INFO:teuthology.orchestra.run.smithi071.stdout: "evadd": 25, 2024-04-30T08:45:30.451 INFO:teuthology.orchestra.run.smithi071.stdout: "evex": 0, 2024-04-30T08:45:30.451 INFO:teuthology.orchestra.run.smithi071.stdout: "evtrm": 0, 2024-04-30T08:45:30.451 INFO:teuthology.orchestra.run.smithi071.stdout: "ev": 25, 2024-04-30T08:45:30.451 INFO:teuthology.orchestra.run.smithi071.stdout: "evexg": 0, 2024-04-30T08:45:30.451 INFO:teuthology.orchestra.run.smithi071.stdout: "evexd": 0, 2024-04-30T08:45:30.451 INFO:teuthology.orchestra.run.smithi071.stdout: "segadd": 2, 2024-04-30T08:45:30.451 INFO:teuthology.orchestra.run.smithi071.stdout: "segex": 0, 2024-04-30T08:45:30.452 INFO:teuthology.orchestra.run.smithi071.stdout: "segtrm": 0, 2024-04-30T08:45:30.452 INFO:teuthology.orchestra.run.smithi071.stdout: "seg": 2, 2024-04-30T08:45:30.452 INFO:teuthology.orchestra.run.smithi071.stdout: "segmjr": 2, 2024-04-30T08:45:30.452 INFO:teuthology.orchestra.run.smithi071.stdout: "segexg": 0, 2024-04-30T08:45:30.452 INFO:teuthology.orchestra.run.smithi071.stdout: "segexd": 0, 2024-04-30T08:45:30.452 INFO:teuthology.orchestra.run.smithi071.stdout: "expos": 4194304, 2024-04-30T08:45:30.452 INFO:teuthology.orchestra.run.smithi071.stdout: "wrpos": 4240427, 2024-04-30T08:45:30.452 INFO:teuthology.orchestra.run.smithi071.stdout: "rdpos": 0, 2024-04-30T08:45:30.452 INFO:teuthology.orchestra.run.smithi071.stdout: "jlat": { 2024-04-30T08:45:30.452 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 16, 2024-04-30T08:45:30.452 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.074930831, 2024-04-30T08:45:30.452 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.004683176 2024-04-30T08:45:30.452 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.452 INFO:teuthology.orchestra.run.smithi071.stdout: "replayed": 0 2024-04-30T08:45:30.453 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.453 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_mem": { 2024-04-30T08:45:30.453 INFO:teuthology.orchestra.run.smithi071.stdout: "ino": 19, 2024-04-30T08:45:30.453 INFO:teuthology.orchestra.run.smithi071.stdout: "ino+": 28, 2024-04-30T08:45:30.453 INFO:teuthology.orchestra.run.smithi071.stdout: "ino-": 9, 2024-04-30T08:45:30.453 INFO:teuthology.orchestra.run.smithi071.stdout: "dir": 18, 2024-04-30T08:45:30.453 INFO:teuthology.orchestra.run.smithi071.stdout: "dir+": 26, 2024-04-30T08:45:30.453 INFO:teuthology.orchestra.run.smithi071.stdout: "dir-": 8, 2024-04-30T08:45:30.453 INFO:teuthology.orchestra.run.smithi071.stdout: "dn": 15, 2024-04-30T08:45:30.453 INFO:teuthology.orchestra.run.smithi071.stdout: "dn+": 35, 2024-04-30T08:45:30.453 INFO:teuthology.orchestra.run.smithi071.stdout: "dn-": 20, 2024-04-30T08:45:30.453 INFO:teuthology.orchestra.run.smithi071.stdout: "cap": 0, 2024-04-30T08:45:30.453 INFO:teuthology.orchestra.run.smithi071.stdout: "cap+": 4, 2024-04-30T08:45:30.454 INFO:teuthology.orchestra.run.smithi071.stdout: "cap-": 4, 2024-04-30T08:45:30.454 INFO:teuthology.orchestra.run.smithi071.stdout: "rss": 121580, 2024-04-30T08:45:30.454 INFO:teuthology.orchestra.run.smithi071.stdout: "heap": 223632 2024-04-30T08:45:30.454 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.454 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_server": { 2024-04-30T08:45:30.454 INFO:teuthology.orchestra.run.smithi071.stdout: "dispatch_client_request": 20, 2024-04-30T08:45:30.454 INFO:teuthology.orchestra.run.smithi071.stdout: "dispatch_server_request": 2, 2024-04-30T08:45:30.454 INFO:teuthology.orchestra.run.smithi071.stdout: "handle_client_request": 14, 2024-04-30T08:45:30.454 INFO:teuthology.orchestra.run.smithi071.stdout: "handle_client_session": 8, 2024-04-30T08:45:30.454 INFO:teuthology.orchestra.run.smithi071.stdout: "handle_peer_request": 8, 2024-04-30T08:45:30.454 INFO:teuthology.orchestra.run.smithi071.stdout: "req_create_latency": { 2024-04-30T08:45:30.454 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:30.454 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:30.455 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:30.455 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.455 INFO:teuthology.orchestra.run.smithi071.stdout: "req_getattr_latency": { 2024-04-30T08:45:30.455 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:30.455 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:30.455 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:30.455 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.455 INFO:teuthology.orchestra.run.smithi071.stdout: "req_getfilelock_latency": { 2024-04-30T08:45:30.455 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:30.455 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:30.455 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:30.455 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.455 INFO:teuthology.orchestra.run.smithi071.stdout: "req_link_latency": { 2024-04-30T08:45:30.456 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:30.456 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:30.456 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:30.456 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.456 INFO:teuthology.orchestra.run.smithi071.stdout: "req_lookup_latency": { 2024-04-30T08:45:30.456 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:30.456 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:30.456 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:30.456 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.456 INFO:teuthology.orchestra.run.smithi071.stdout: "req_lookuphash_latency": { 2024-04-30T08:45:30.456 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:30.456 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:30.456 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:30.456 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.456 INFO:teuthology.orchestra.run.smithi071.stdout: "req_lookupino_latency": { 2024-04-30T08:45:30.457 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:30.457 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:30.457 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:30.457 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.457 INFO:teuthology.orchestra.run.smithi071.stdout: "req_lookupname_latency": { 2024-04-30T08:45:30.457 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:30.457 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:30.457 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:30.457 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.457 INFO:teuthology.orchestra.run.smithi071.stdout: "req_lookupparent_latency": { 2024-04-30T08:45:30.457 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:30.457 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:30.457 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:30.457 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.457 INFO:teuthology.orchestra.run.smithi071.stdout: "req_lookupsnap_latency": { 2024-04-30T08:45:30.458 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:30.458 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:30.458 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:30.458 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.458 INFO:teuthology.orchestra.run.smithi071.stdout: "req_lssnap_latency": { 2024-04-30T08:45:30.458 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:30.458 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:30.458 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:30.458 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.458 INFO:teuthology.orchestra.run.smithi071.stdout: "req_mkdir_latency": { 2024-04-30T08:45:30.459 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:30.459 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:30.459 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:30.459 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.459 INFO:teuthology.orchestra.run.smithi071.stdout: "req_mknod_latency": { 2024-04-30T08:45:30.459 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:30.459 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:30.459 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:30.460 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.460 INFO:teuthology.orchestra.run.smithi071.stdout: "req_mksnap_latency": { 2024-04-30T08:45:30.460 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:30.460 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:30.460 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:30.460 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.460 INFO:teuthology.orchestra.run.smithi071.stdout: "req_open_latency": { 2024-04-30T08:45:30.460 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:30.460 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:30.460 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:30.460 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.460 INFO:teuthology.orchestra.run.smithi071.stdout: "req_readdir_latency": { 2024-04-30T08:45:30.460 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 2, 2024-04-30T08:45:30.460 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000534403, 2024-04-30T08:45:30.461 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000267201 2024-04-30T08:45:30.461 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.461 INFO:teuthology.orchestra.run.smithi071.stdout: "req_rename_latency": { 2024-04-30T08:45:30.461 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:30.461 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:30.461 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:30.461 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.461 INFO:teuthology.orchestra.run.smithi071.stdout: "req_renamesnap_latency": { 2024-04-30T08:45:30.461 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:30.461 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:30.461 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:30.461 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.461 INFO:teuthology.orchestra.run.smithi071.stdout: "req_snapdiff_latency": { 2024-04-30T08:45:30.461 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:30.461 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:30.462 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:30.462 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.462 INFO:teuthology.orchestra.run.smithi071.stdout: "req_rmdir_latency": { 2024-04-30T08:45:30.462 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:30.462 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:30.462 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:30.462 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.462 INFO:teuthology.orchestra.run.smithi071.stdout: "req_rmsnap_latency": { 2024-04-30T08:45:30.462 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:30.462 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:30.462 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:30.462 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.462 INFO:teuthology.orchestra.run.smithi071.stdout: "req_rmxattr_latency": { 2024-04-30T08:45:30.463 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:30.463 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:30.463 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:30.463 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.463 INFO:teuthology.orchestra.run.smithi071.stdout: "req_setattr_latency": { 2024-04-30T08:45:30.463 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:30.463 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:30.463 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:30.463 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.463 INFO:teuthology.orchestra.run.smithi071.stdout: "req_setdirlayout_latency": { 2024-04-30T08:45:30.463 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:30.463 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:30.463 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:30.463 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.463 INFO:teuthology.orchestra.run.smithi071.stdout: "req_setfilelock_latency": { 2024-04-30T08:45:30.464 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:30.464 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:30.464 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:30.464 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.464 INFO:teuthology.orchestra.run.smithi071.stdout: "req_setlayout_latency": { 2024-04-30T08:45:30.464 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:30.464 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:30.464 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:30.464 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.464 INFO:teuthology.orchestra.run.smithi071.stdout: "req_setxattr_latency": { 2024-04-30T08:45:30.464 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:30.464 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:30.464 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:30.465 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.465 INFO:teuthology.orchestra.run.smithi071.stdout: "req_symlink_latency": { 2024-04-30T08:45:30.465 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:30.465 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:30.465 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:30.465 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.465 INFO:teuthology.orchestra.run.smithi071.stdout: "req_unlink_latency": { 2024-04-30T08:45:30.465 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 2, 2024-04-30T08:45:30.465 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.006717437, 2024-04-30T08:45:30.465 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.003358718 2024-04-30T08:45:30.465 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.465 INFO:teuthology.orchestra.run.smithi071.stdout: "cap_revoke_eviction": 0, 2024-04-30T08:45:30.465 INFO:teuthology.orchestra.run.smithi071.stdout: "cap_acquisition_throttle": 0, 2024-04-30T08:45:30.466 INFO:teuthology.orchestra.run.smithi071.stdout: "req_getvxattr_latency": { 2024-04-30T08:45:30.466 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:30.466 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:30.466 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:30.466 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:30.466 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.466 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_sessions": { 2024-04-30T08:45:30.466 INFO:teuthology.orchestra.run.smithi071.stdout: "session_count": 1, 2024-04-30T08:45:30.466 INFO:teuthology.orchestra.run.smithi071.stdout: "session_add": 1, 2024-04-30T08:45:30.466 INFO:teuthology.orchestra.run.smithi071.stdout: "session_remove": 0, 2024-04-30T08:45:30.466 INFO:teuthology.orchestra.run.smithi071.stdout: "sessions_open": 1, 2024-04-30T08:45:30.466 INFO:teuthology.orchestra.run.smithi071.stdout: "sessions_stale": 0, 2024-04-30T08:45:30.466 INFO:teuthology.orchestra.run.smithi071.stdout: "total_load": 2, 2024-04-30T08:45:30.467 INFO:teuthology.orchestra.run.smithi071.stdout: "average_load": 2, 2024-04-30T08:45:30.467 INFO:teuthology.orchestra.run.smithi071.stdout: "avg_session_uptime": 2245, 2024-04-30T08:45:30.467 INFO:teuthology.orchestra.run.smithi071.stdout: "mdthresh_evicted": 0 2024-04-30T08:45:30.467 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.467 INFO:teuthology.orchestra.run.smithi071.stdout: "mempool": { 2024-04-30T08:45:30.467 INFO:teuthology.orchestra.run.smithi071.stdout: "bloom_filter_bytes": 0, 2024-04-30T08:45:30.467 INFO:teuthology.orchestra.run.smithi071.stdout: "bloom_filter_items": 0, 2024-04-30T08:45:30.467 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_alloc_bytes": 0, 2024-04-30T08:45:30.467 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_alloc_items": 0, 2024-04-30T08:45:30.467 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_data_bytes": 0, 2024-04-30T08:45:30.467 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_data_items": 0, 2024-04-30T08:45:30.467 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-30T08:45:30.467 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_onode_items": 0, 2024-04-30T08:45:30.467 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-30T08:45:30.468 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_meta_items": 0, 2024-04-30T08:45:30.468 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_other_bytes": 0, 2024-04-30T08:45:30.468 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_other_items": 0, 2024-04-30T08:45:30.468 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-30T08:45:30.468 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_buffer_items": 0, 2024-04-30T08:45:30.468 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_extent_bytes": 0, 2024-04-30T08:45:30.468 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_extent_items": 0, 2024-04-30T08:45:30.468 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_blob_bytes": 0, 2024-04-30T08:45:30.468 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_blob_items": 0, 2024-04-30T08:45:30.468 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-30T08:45:30.468 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_shared_blob_items": 0, 2024-04-30T08:45:30.468 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-30T08:45:30.468 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_inline_bl_items": 0, 2024-04-30T08:45:30.468 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_fsck_bytes": 0, 2024-04-30T08:45:30.469 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_fsck_items": 0, 2024-04-30T08:45:30.469 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_txc_bytes": 0, 2024-04-30T08:45:30.469 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_txc_items": 0, 2024-04-30T08:45:30.469 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-30T08:45:30.469 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_writing_deferred_items": 0, 2024-04-30T08:45:30.469 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_writing_bytes": 0, 2024-04-30T08:45:30.469 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_writing_items": 0, 2024-04-30T08:45:30.469 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_bytes": 0, 2024-04-30T08:45:30.469 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_items": 0, 2024-04-30T08:45:30.469 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_file_reader_bytes": 0, 2024-04-30T08:45:30.469 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_file_reader_items": 0, 2024-04-30T08:45:30.469 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_file_writer_bytes": 0, 2024-04-30T08:45:30.469 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_file_writer_items": 0, 2024-04-30T08:45:30.469 INFO:teuthology.orchestra.run.smithi071.stdout: "buffer_anon_bytes": 274335, 2024-04-30T08:45:30.470 INFO:teuthology.orchestra.run.smithi071.stdout: "buffer_anon_items": 102, 2024-04-30T08:45:30.470 INFO:teuthology.orchestra.run.smithi071.stdout: "buffer_meta_bytes": 0, 2024-04-30T08:45:30.470 INFO:teuthology.orchestra.run.smithi071.stdout: "buffer_meta_items": 0, 2024-04-30T08:45:30.470 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_bytes": 0, 2024-04-30T08:45:30.470 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_items": 0, 2024-04-30T08:45:30.470 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_mapbl_bytes": 0, 2024-04-30T08:45:30.470 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_mapbl_items": 0, 2024-04-30T08:45:30.470 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pglog_bytes": 0, 2024-04-30T08:45:30.470 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pglog_items": 0, 2024-04-30T08:45:30.470 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_bytes": 9088, 2024-04-30T08:45:30.470 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_items": 175, 2024-04-30T08:45:30.470 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_mapping_bytes": 0, 2024-04-30T08:45:30.470 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_mapping_items": 0, 2024-04-30T08:45:30.470 INFO:teuthology.orchestra.run.smithi071.stdout: "pgmap_bytes": 0, 2024-04-30T08:45:30.471 INFO:teuthology.orchestra.run.smithi071.stdout: "pgmap_items": 0, 2024-04-30T08:45:30.471 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_co_bytes": 76781, 2024-04-30T08:45:30.471 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_co_items": 315, 2024-04-30T08:45:30.471 INFO:teuthology.orchestra.run.smithi071.stdout: "unittest_1_bytes": 0, 2024-04-30T08:45:30.471 INFO:teuthology.orchestra.run.smithi071.stdout: "unittest_1_items": 0, 2024-04-30T08:45:30.471 INFO:teuthology.orchestra.run.smithi071.stdout: "unittest_2_bytes": 0, 2024-04-30T08:45:30.471 INFO:teuthology.orchestra.run.smithi071.stdout: "unittest_2_items": 0 2024-04-30T08:45:30.471 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.471 INFO:teuthology.orchestra.run.smithi071.stdout: "objecter": { 2024-04-30T08:45:30.471 INFO:teuthology.orchestra.run.smithi071.stdout: "op_active": 0, 2024-04-30T08:45:30.471 INFO:teuthology.orchestra.run.smithi071.stdout: "op_laggy": 0, 2024-04-30T08:45:30.471 INFO:teuthology.orchestra.run.smithi071.stdout: "op_send": 76, 2024-04-30T08:45:30.471 INFO:teuthology.orchestra.run.smithi071.stdout: "op_send_bytes": 58707, 2024-04-30T08:45:30.471 INFO:teuthology.orchestra.run.smithi071.stdout: "op_resend": 33, 2024-04-30T08:45:30.472 INFO:teuthology.orchestra.run.smithi071.stdout: "op_reply": 76, 2024-04-30T08:45:30.472 INFO:teuthology.orchestra.run.smithi071.stdout: "op_latency": { 2024-04-30T08:45:30.472 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 76, 2024-04-30T08:45:30.472 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.952983320, 2024-04-30T08:45:30.472 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.012539254 2024-04-30T08:45:30.472 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.472 INFO:teuthology.orchestra.run.smithi071.stdout: "op_inflight": 0, 2024-04-30T08:45:30.472 INFO:teuthology.orchestra.run.smithi071.stdout: "oplen_avg": { 2024-04-30T08:45:30.472 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 76, 2024-04-30T08:45:30.472 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 104 2024-04-30T08:45:30.472 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.472 INFO:teuthology.orchestra.run.smithi071.stdout: "op": 76, 2024-04-30T08:45:30.472 INFO:teuthology.orchestra.run.smithi071.stdout: "op_r": 5, 2024-04-30T08:45:30.472 INFO:teuthology.orchestra.run.smithi071.stdout: "op_w": 71, 2024-04-30T08:45:30.473 INFO:teuthology.orchestra.run.smithi071.stdout: "op_rmw": 0, 2024-04-30T08:45:30.473 INFO:teuthology.orchestra.run.smithi071.stdout: "op_pg": 0, 2024-04-30T08:45:30.473 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_stat": 0, 2024-04-30T08:45:30.473 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_create": 10, 2024-04-30T08:45:30.473 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_read": 1, 2024-04-30T08:45:30.473 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_write": 17, 2024-04-30T08:45:30.473 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_writefull": 10, 2024-04-30T08:45:30.473 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_writesame": 0, 2024-04-30T08:45:30.473 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_append": 0, 2024-04-30T08:45:30.473 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_zero": 0, 2024-04-30T08:45:30.473 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_truncate": 0, 2024-04-30T08:45:30.473 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_delete": 16, 2024-04-30T08:45:30.473 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_mapext": 0, 2024-04-30T08:45:30.473 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_sparse_read": 0, 2024-04-30T08:45:30.474 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_clonerange": 0, 2024-04-30T08:45:30.474 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_getxattr": 4, 2024-04-30T08:45:30.474 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_setxattr": 20, 2024-04-30T08:45:30.474 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_cmpxattr": 0, 2024-04-30T08:45:30.474 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_rmxattr": 0, 2024-04-30T08:45:30.474 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_resetxattrs": 0, 2024-04-30T08:45:30.474 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_call": 0, 2024-04-30T08:45:30.474 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_watch": 0, 2024-04-30T08:45:30.474 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_notify": 0, 2024-04-30T08:45:30.474 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_src_cmpxattr": 0, 2024-04-30T08:45:30.474 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_pgls": 0, 2024-04-30T08:45:30.474 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_pgls_filter": 0, 2024-04-30T08:45:30.474 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_other": 0, 2024-04-30T08:45:30.475 INFO:teuthology.orchestra.run.smithi071.stdout: "linger_active": 0, 2024-04-30T08:45:30.475 INFO:teuthology.orchestra.run.smithi071.stdout: "linger_send": 0, 2024-04-30T08:45:30.475 INFO:teuthology.orchestra.run.smithi071.stdout: "linger_resend": 0, 2024-04-30T08:45:30.475 INFO:teuthology.orchestra.run.smithi071.stdout: "linger_ping": 0, 2024-04-30T08:45:30.475 INFO:teuthology.orchestra.run.smithi071.stdout: "poolop_active": 0, 2024-04-30T08:45:30.475 INFO:teuthology.orchestra.run.smithi071.stdout: "poolop_send": 0, 2024-04-30T08:45:30.475 INFO:teuthology.orchestra.run.smithi071.stdout: "poolop_resend": 0, 2024-04-30T08:45:30.475 INFO:teuthology.orchestra.run.smithi071.stdout: "poolstat_active": 0, 2024-04-30T08:45:30.475 INFO:teuthology.orchestra.run.smithi071.stdout: "poolstat_send": 0, 2024-04-30T08:45:30.475 INFO:teuthology.orchestra.run.smithi071.stdout: "poolstat_resend": 0, 2024-04-30T08:45:30.475 INFO:teuthology.orchestra.run.smithi071.stdout: "statfs_active": 0, 2024-04-30T08:45:30.475 INFO:teuthology.orchestra.run.smithi071.stdout: "statfs_send": 0, 2024-04-30T08:45:30.475 INFO:teuthology.orchestra.run.smithi071.stdout: "statfs_resend": 0, 2024-04-30T08:45:30.476 INFO:teuthology.orchestra.run.smithi071.stdout: "command_active": 0, 2024-04-30T08:45:30.476 INFO:teuthology.orchestra.run.smithi071.stdout: "command_send": 0, 2024-04-30T08:45:30.476 INFO:teuthology.orchestra.run.smithi071.stdout: "command_resend": 0, 2024-04-30T08:45:30.476 INFO:teuthology.orchestra.run.smithi071.stdout: "map_epoch": 85, 2024-04-30T08:45:30.476 INFO:teuthology.orchestra.run.smithi071.stdout: "map_full": 0, 2024-04-30T08:45:30.476 INFO:teuthology.orchestra.run.smithi071.stdout: "map_inc": 4, 2024-04-30T08:45:30.476 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_sessions": 10, 2024-04-30T08:45:30.476 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_session_open": 10, 2024-04-30T08:45:30.476 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_session_close": 0, 2024-04-30T08:45:30.476 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_laggy": 0, 2024-04-30T08:45:30.476 INFO:teuthology.orchestra.run.smithi071.stdout: "omap_wr": 22, 2024-04-30T08:45:30.476 INFO:teuthology.orchestra.run.smithi071.stdout: "omap_rd": 0, 2024-04-30T08:45:30.476 INFO:teuthology.orchestra.run.smithi071.stdout: "omap_del": 4 2024-04-30T08:45:30.476 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.476 INFO:teuthology.orchestra.run.smithi071.stdout: "oft": { 2024-04-30T08:45:30.477 INFO:teuthology.orchestra.run.smithi071.stdout: "omap_total_objs": 1, 2024-04-30T08:45:30.477 INFO:teuthology.orchestra.run.smithi071.stdout: "omap_total_kv_pairs": 0, 2024-04-30T08:45:30.477 INFO:teuthology.orchestra.run.smithi071.stdout: "omap_total_updates": 0, 2024-04-30T08:45:30.477 INFO:teuthology.orchestra.run.smithi071.stdout: "omap_total_removes": 0 2024-04-30T08:45:30.477 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.477 INFO:teuthology.orchestra.run.smithi071.stdout: "osd-slow-ops": { 2024-04-30T08:45:30.477 INFO:teuthology.orchestra.run.smithi071.stdout: "slow_ops_count": 0 2024-04-30T08:45:30.477 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.477 INFO:teuthology.orchestra.run.smithi071.stdout: "purge_queue": { 2024-04-30T08:45:30.477 INFO:teuthology.orchestra.run.smithi071.stdout: "pq_executing_ops": 0, 2024-04-30T08:45:30.477 INFO:teuthology.orchestra.run.smithi071.stdout: "pq_executing_ops_high_water": 4, 2024-04-30T08:45:30.477 INFO:teuthology.orchestra.run.smithi071.stdout: "pq_executing": 0, 2024-04-30T08:45:30.477 INFO:teuthology.orchestra.run.smithi071.stdout: "pq_executing_high_water": 2, 2024-04-30T08:45:30.477 INFO:teuthology.orchestra.run.smithi071.stdout: "pq_executed_ops": 4, 2024-04-30T08:45:30.478 INFO:teuthology.orchestra.run.smithi071.stdout: "pq_executed": 2, 2024-04-30T08:45:30.478 INFO:teuthology.orchestra.run.smithi071.stdout: "pq_item_in_journal": 0 2024-04-30T08:45:30.478 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.478 INFO:teuthology.orchestra.run.smithi071.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-30T08:45:30.478 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 0, 2024-04-30T08:45:30.478 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 104857600, 2024-04-30T08:45:30.478 INFO:teuthology.orchestra.run.smithi071.stdout: "get_started": 0, 2024-04-30T08:45:30.478 INFO:teuthology.orchestra.run.smithi071.stdout: "get": 1479, 2024-04-30T08:45:30.478 INFO:teuthology.orchestra.run.smithi071.stdout: "get_sum": 1739187, 2024-04-30T08:45:30.478 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:30.478 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_success": 1479, 2024-04-30T08:45:30.478 INFO:teuthology.orchestra.run.smithi071.stdout: "take": 0, 2024-04-30T08:45:30.478 INFO:teuthology.orchestra.run.smithi071.stdout: "take_sum": 0, 2024-04-30T08:45:30.478 INFO:teuthology.orchestra.run.smithi071.stdout: "put": 1479, 2024-04-30T08:45:30.479 INFO:teuthology.orchestra.run.smithi071.stdout: "put_sum": 1739187, 2024-04-30T08:45:30.479 INFO:teuthology.orchestra.run.smithi071.stdout: "wait": { 2024-04-30T08:45:30.479 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:30.479 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:30.479 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:30.479 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:30.479 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.479 INFO:teuthology.orchestra.run.smithi071.stdout: "throttle-objecter_bytes": { 2024-04-30T08:45:30.479 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 0, 2024-04-30T08:45:30.479 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 104857600, 2024-04-30T08:45:30.479 INFO:teuthology.orchestra.run.smithi071.stdout: "get_started": 0, 2024-04-30T08:45:30.479 INFO:teuthology.orchestra.run.smithi071.stdout: "get": 0, 2024-04-30T08:45:30.479 INFO:teuthology.orchestra.run.smithi071.stdout: "get_sum": 0, 2024-04-30T08:45:30.479 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:30.479 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_success": 0, 2024-04-30T08:45:30.480 INFO:teuthology.orchestra.run.smithi071.stdout: "take": 76, 2024-04-30T08:45:30.480 INFO:teuthology.orchestra.run.smithi071.stdout: "take_sum": 58925, 2024-04-30T08:45:30.480 INFO:teuthology.orchestra.run.smithi071.stdout: "put": 76, 2024-04-30T08:45:30.480 INFO:teuthology.orchestra.run.smithi071.stdout: "put_sum": 58925, 2024-04-30T08:45:30.480 INFO:teuthology.orchestra.run.smithi071.stdout: "wait": { 2024-04-30T08:45:30.480 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:30.480 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:30.480 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:30.480 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:30.480 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.480 INFO:teuthology.orchestra.run.smithi071.stdout: "throttle-objecter_ops": { 2024-04-30T08:45:30.480 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 0, 2024-04-30T08:45:30.480 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 1024, 2024-04-30T08:45:30.480 INFO:teuthology.orchestra.run.smithi071.stdout: "get_started": 0, 2024-04-30T08:45:30.480 INFO:teuthology.orchestra.run.smithi071.stdout: "get": 0, 2024-04-30T08:45:30.481 INFO:teuthology.orchestra.run.smithi071.stdout: "get_sum": 0, 2024-04-30T08:45:30.481 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:30.481 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_success": 0, 2024-04-30T08:45:30.481 INFO:teuthology.orchestra.run.smithi071.stdout: "take": 76, 2024-04-30T08:45:30.481 INFO:teuthology.orchestra.run.smithi071.stdout: "take_sum": 76, 2024-04-30T08:45:30.481 INFO:teuthology.orchestra.run.smithi071.stdout: "put": 76, 2024-04-30T08:45:30.481 INFO:teuthology.orchestra.run.smithi071.stdout: "put_sum": 76, 2024-04-30T08:45:30.481 INFO:teuthology.orchestra.run.smithi071.stdout: "wait": { 2024-04-30T08:45:30.481 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:30.481 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:30.481 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:30.481 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:30.481 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.481 INFO:teuthology.orchestra.run.smithi071.stdout: "throttle-write_buf_throttle": { 2024-04-30T08:45:30.482 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 0, 2024-04-30T08:45:30.482 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 3758096384, 2024-04-30T08:45:30.482 INFO:teuthology.orchestra.run.smithi071.stdout: "get_started": 0, 2024-04-30T08:45:30.482 INFO:teuthology.orchestra.run.smithi071.stdout: "get": 2, 2024-04-30T08:45:30.482 INFO:teuthology.orchestra.run.smithi071.stdout: "get_sum": 218, 2024-04-30T08:45:30.482 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:30.482 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_success": 2, 2024-04-30T08:45:30.482 INFO:teuthology.orchestra.run.smithi071.stdout: "take": 0, 2024-04-30T08:45:30.482 INFO:teuthology.orchestra.run.smithi071.stdout: "take_sum": 0, 2024-04-30T08:45:30.482 INFO:teuthology.orchestra.run.smithi071.stdout: "put": 1, 2024-04-30T08:45:30.482 INFO:teuthology.orchestra.run.smithi071.stdout: "put_sum": 218, 2024-04-30T08:45:30.482 INFO:teuthology.orchestra.run.smithi071.stdout: "wait": { 2024-04-30T08:45:30.482 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:30.483 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:30.483 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:30.483 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:30.483 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:30.483 INFO:teuthology.orchestra.run.smithi071.stdout: "throttle-write_buf_throttle-0x5556f345a4e0": { 2024-04-30T08:45:30.483 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 0, 2024-04-30T08:45:30.483 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 3758096384, 2024-04-30T08:45:30.483 INFO:teuthology.orchestra.run.smithi071.stdout: "get_started": 0, 2024-04-30T08:45:30.483 INFO:teuthology.orchestra.run.smithi071.stdout: "get": 25, 2024-04-30T08:45:30.483 INFO:teuthology.orchestra.run.smithi071.stdout: "get_sum": 46123, 2024-04-30T08:45:30.483 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:30.483 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_success": 25, 2024-04-30T08:45:30.483 INFO:teuthology.orchestra.run.smithi071.stdout: "take": 0, 2024-04-30T08:45:30.483 INFO:teuthology.orchestra.run.smithi071.stdout: "take_sum": 0, 2024-04-30T08:45:30.484 INFO:teuthology.orchestra.run.smithi071.stdout: "put": 16, 2024-04-30T08:45:30.484 INFO:teuthology.orchestra.run.smithi071.stdout: "put_sum": 46123, 2024-04-30T08:45:30.484 INFO:teuthology.orchestra.run.smithi071.stdout: "wait": { 2024-04-30T08:45:30.484 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:30.484 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:30.484 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:30.484 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:30.484 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:30.484 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-30T08:45:31.278 INFO:tasks.check_counter:Daemon mds.h mds.exported=2 2024-04-30T08:45:31.279 INFO:tasks.check_counter:Daemon mds.h mds.imported=2 2024-04-30T08:45:31.279 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=0 2024-04-30T08:45:31.279 DEBUG:tasks.check_counter:Getting stats from k 2024-04-30T08:45:31.279 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-30T08:45:31.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:31 smithi071 ceph-mon[33385]: pgmap v735: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-30T08:45:31.521 INFO:teuthology.orchestra.run.smithi071.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.b/config 2024-04-30T08:45:31.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:31 smithi189 ceph-mon[33038]: pgmap v735: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-30T08:45:31.645 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:31 smithi032 ceph-mon[28331]: pgmap v735: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 341 B/s wr, 11 op/s 2024-04-30T08:45:33.250 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:33 smithi071 ceph-mon[33385]: pgmap v736: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-30T08:45:33.316 INFO:teuthology.orchestra.run.smithi071.stdout:{ 2024-04-30T08:45:33.316 INFO:teuthology.orchestra.run.smithi071.stdout: "AsyncMessenger::Worker-0": { 2024-04-30T08:45:33.317 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_messages": 753, 2024-04-30T08:45:33.317 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_messages": 754, 2024-04-30T08:45:33.317 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_bytes": 253043, 2024-04-30T08:45:33.317 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_bytes": 239137, 2024-04-30T08:45:33.317 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_created_connections": 6, 2024-04-30T08:45:33.317 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_active_connections": 2, 2024-04-30T08:45:33.317 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_total_time": 0.082287493, 2024-04-30T08:45:33.317 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_send_time": 0.030491791, 2024-04-30T08:45:33.317 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_recv_time": 0.019831144, 2024-04-30T08:45:33.317 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_fast_dispatch_time": 0.024330039, 2024-04-30T08:45:33.317 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:33.318 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 752, 2024-04-30T08:45:33.318 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.044896910, 2024-04-30T08:45:33.318 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000059703 2024-04-30T08:45:33.318 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.318 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:33.318 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.318 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.318 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.319 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.319 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-30T08:45:33.319 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_encrypted_bytes": 0 2024-04-30T08:45:33.319 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.319 INFO:teuthology.orchestra.run.smithi071.stdout: "AsyncMessenger::Worker-1": { 2024-04-30T08:45:33.319 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_messages": 1594, 2024-04-30T08:45:33.319 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_messages": 1484, 2024-04-30T08:45:33.319 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_bytes": 1896383, 2024-04-30T08:45:33.319 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_bytes": 539629, 2024-04-30T08:45:33.319 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_created_connections": 4, 2024-04-30T08:45:33.319 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_active_connections": 3, 2024-04-30T08:45:33.320 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_total_time": 0.137649490, 2024-04-30T08:45:33.320 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_send_time": 0.034530960, 2024-04-30T08:45:33.320 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_recv_time": 0.061586439, 2024-04-30T08:45:33.320 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_fast_dispatch_time": 0.042394275, 2024-04-30T08:45:33.320 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:33.320 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 1483, 2024-04-30T08:45:33.320 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.059633792, 2024-04-30T08:45:33.320 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000040211 2024-04-30T08:45:33.320 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.320 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:33.321 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.321 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.321 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.321 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.321 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_encrypted_bytes": 1531664, 2024-04-30T08:45:33.321 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_encrypted_bytes": 143344 2024-04-30T08:45:33.321 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.321 INFO:teuthology.orchestra.run.smithi071.stdout: "AsyncMessenger::Worker-2": { 2024-04-30T08:45:33.321 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_messages": 9, 2024-04-30T08:45:33.321 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_messages": 183, 2024-04-30T08:45:33.322 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_bytes": 4660, 2024-04-30T08:45:33.322 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_bytes": 453029, 2024-04-30T08:45:33.322 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_created_connections": 6, 2024-04-30T08:45:33.322 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_active_connections": 4, 2024-04-30T08:45:33.322 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_total_time": 0.011033495, 2024-04-30T08:45:33.322 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_send_time": 0.007653060, 2024-04-30T08:45:33.322 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_recv_time": 0.001775440, 2024-04-30T08:45:33.322 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_running_fast_dispatch_time": 0.000067202, 2024-04-30T08:45:33.322 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:33.323 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 181, 2024-04-30T08:45:33.323 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.017462523, 2024-04-30T08:45:33.323 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000096478 2024-04-30T08:45:33.323 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.323 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:33.323 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.323 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.323 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.323 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.323 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-04-30T08:45:33.323 INFO:teuthology.orchestra.run.smithi071.stdout: "msgr_send_encrypted_bytes": 452064 2024-04-30T08:45:33.323 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.323 INFO:teuthology.orchestra.run.smithi071.stdout: "cct": { 2024-04-30T08:45:33.324 INFO:teuthology.orchestra.run.smithi071.stdout: "total_workers": 1, 2024-04-30T08:45:33.324 INFO:teuthology.orchestra.run.smithi071.stdout: "unhealthy_workers": 0 2024-04-30T08:45:33.324 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.324 INFO:teuthology.orchestra.run.smithi071.stdout: "finisher-MDSRank": { 2024-04-30T08:45:33.324 INFO:teuthology.orchestra.run.smithi071.stdout: "queue_len": 0, 2024-04-30T08:45:33.324 INFO:teuthology.orchestra.run.smithi071.stdout: "complete_latency": { 2024-04-30T08:45:33.324 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 3475, 2024-04-30T08:45:33.324 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.093117435, 2024-04-30T08:45:33.324 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000026796 2024-04-30T08:45:33.324 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:33.324 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.324 INFO:teuthology.orchestra.run.smithi071.stdout: "finisher-PurgeQueue": { 2024-04-30T08:45:33.325 INFO:teuthology.orchestra.run.smithi071.stdout: "queue_len": 0, 2024-04-30T08:45:33.325 INFO:teuthology.orchestra.run.smithi071.stdout: "complete_latency": { 2024-04-30T08:45:33.325 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.325 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.325 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.325 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:33.325 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.325 INFO:teuthology.orchestra.run.smithi071.stdout: "mds": { 2024-04-30T08:45:33.325 INFO:teuthology.orchestra.run.smithi071.stdout: "request": 0, 2024-04-30T08:45:33.325 INFO:teuthology.orchestra.run.smithi071.stdout: "reply": 0, 2024-04-30T08:45:33.325 INFO:teuthology.orchestra.run.smithi071.stdout: "reply_latency": { 2024-04-30T08:45:33.326 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.326 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.326 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.326 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.326 INFO:teuthology.orchestra.run.smithi071.stdout: "slow_reply": 0, 2024-04-30T08:45:33.326 INFO:teuthology.orchestra.run.smithi071.stdout: "forward": 0, 2024-04-30T08:45:33.326 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_fetch_complete": 0, 2024-04-30T08:45:33.326 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_fetch_keys": 0, 2024-04-30T08:45:33.327 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_commit": 0, 2024-04-30T08:45:33.327 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_split": 0, 2024-04-30T08:45:33.327 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_merge": 0, 2024-04-30T08:45:33.327 INFO:teuthology.orchestra.run.smithi071.stdout: "inodes": 3, 2024-04-30T08:45:33.327 INFO:teuthology.orchestra.run.smithi071.stdout: "inodes_top": 0, 2024-04-30T08:45:33.327 INFO:teuthology.orchestra.run.smithi071.stdout: "inodes_bottom": 3, 2024-04-30T08:45:33.327 INFO:teuthology.orchestra.run.smithi071.stdout: "inodes_pin_tail": 0, 2024-04-30T08:45:33.327 INFO:teuthology.orchestra.run.smithi071.stdout: "inodes_pinned": 3, 2024-04-30T08:45:33.327 INFO:teuthology.orchestra.run.smithi071.stdout: "inodes_expired": 0, 2024-04-30T08:45:33.327 INFO:teuthology.orchestra.run.smithi071.stdout: "inodes_with_caps": 0, 2024-04-30T08:45:33.327 INFO:teuthology.orchestra.run.smithi071.stdout: "caps": 0, 2024-04-30T08:45:33.327 INFO:teuthology.orchestra.run.smithi071.stdout: "subtrees": 1, 2024-04-30T08:45:33.328 INFO:teuthology.orchestra.run.smithi071.stdout: "traverse": 0, 2024-04-30T08:45:33.328 INFO:teuthology.orchestra.run.smithi071.stdout: "traverse_hit": 0, 2024-04-30T08:45:33.328 INFO:teuthology.orchestra.run.smithi071.stdout: "traverse_forward": 0, 2024-04-30T08:45:33.328 INFO:teuthology.orchestra.run.smithi071.stdout: "traverse_discover": 0, 2024-04-30T08:45:33.328 INFO:teuthology.orchestra.run.smithi071.stdout: "traverse_dir_fetch": 0, 2024-04-30T08:45:33.328 INFO:teuthology.orchestra.run.smithi071.stdout: "traverse_remote_ino": 0, 2024-04-30T08:45:33.328 INFO:teuthology.orchestra.run.smithi071.stdout: "traverse_lock": 0, 2024-04-30T08:45:33.328 INFO:teuthology.orchestra.run.smithi071.stdout: "load_cent": 0, 2024-04-30T08:45:33.328 INFO:teuthology.orchestra.run.smithi071.stdout: "q": 0, 2024-04-30T08:45:33.328 INFO:teuthology.orchestra.run.smithi071.stdout: "exported": 0, 2024-04-30T08:45:33.328 INFO:teuthology.orchestra.run.smithi071.stdout: "exported_inodes": 0, 2024-04-30T08:45:33.328 INFO:teuthology.orchestra.run.smithi071.stdout: "imported": 0, 2024-04-30T08:45:33.328 INFO:teuthology.orchestra.run.smithi071.stdout: "imported_inodes": 0, 2024-04-30T08:45:33.329 INFO:teuthology.orchestra.run.smithi071.stdout: "openino_dir_fetch": 0, 2024-04-30T08:45:33.329 INFO:teuthology.orchestra.run.smithi071.stdout: "openino_backtrace_fetch": 0, 2024-04-30T08:45:33.329 INFO:teuthology.orchestra.run.smithi071.stdout: "openino_peer_discover": 0, 2024-04-30T08:45:33.329 INFO:teuthology.orchestra.run.smithi071.stdout: "root_rfiles": 0, 2024-04-30T08:45:33.329 INFO:teuthology.orchestra.run.smithi071.stdout: "root_rbytes": 0, 2024-04-30T08:45:33.329 INFO:teuthology.orchestra.run.smithi071.stdout: "root_rsnaps": 0, 2024-04-30T08:45:33.329 INFO:teuthology.orchestra.run.smithi071.stdout: "scrub_backtrace_fetch": 0, 2024-04-30T08:45:33.329 INFO:teuthology.orchestra.run.smithi071.stdout: "scrub_set_tag": 0, 2024-04-30T08:45:33.329 INFO:teuthology.orchestra.run.smithi071.stdout: "scrub_backtrace_repaired": 0, 2024-04-30T08:45:33.329 INFO:teuthology.orchestra.run.smithi071.stdout: "scrub_inotable_repaired": 0, 2024-04-30T08:45:33.329 INFO:teuthology.orchestra.run.smithi071.stdout: "scrub_dir_inodes": 0, 2024-04-30T08:45:33.329 INFO:teuthology.orchestra.run.smithi071.stdout: "scrub_dir_base_inodes": 0, 2024-04-30T08:45:33.329 INFO:teuthology.orchestra.run.smithi071.stdout: "scrub_dirfrag_rstats": 0, 2024-04-30T08:45:33.330 INFO:teuthology.orchestra.run.smithi071.stdout: "scrub_file_inodes": 0, 2024-04-30T08:45:33.330 INFO:teuthology.orchestra.run.smithi071.stdout: "handle_inode_file_caps": 0, 2024-04-30T08:45:33.330 INFO:teuthology.orchestra.run.smithi071.stdout: "ceph_cap_op_revoke": 0, 2024-04-30T08:45:33.330 INFO:teuthology.orchestra.run.smithi071.stdout: "ceph_cap_op_grant": 0, 2024-04-30T08:45:33.330 INFO:teuthology.orchestra.run.smithi071.stdout: "ceph_cap_op_trunc": 0, 2024-04-30T08:45:33.330 INFO:teuthology.orchestra.run.smithi071.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-30T08:45:33.330 INFO:teuthology.orchestra.run.smithi071.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-30T08:45:33.330 INFO:teuthology.orchestra.run.smithi071.stdout: "handle_client_caps": 0, 2024-04-30T08:45:33.330 INFO:teuthology.orchestra.run.smithi071.stdout: "handle_client_caps_dirty": 0, 2024-04-30T08:45:33.330 INFO:teuthology.orchestra.run.smithi071.stdout: "handle_client_cap_release": 0, 2024-04-30T08:45:33.330 INFO:teuthology.orchestra.run.smithi071.stdout: "process_request_cap_release": 0 2024-04-30T08:45:33.330 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.330 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_cache": { 2024-04-30T08:45:33.331 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_update": 0, 2024-04-30T08:45:33.331 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_update_receipt": 0, 2024-04-30T08:45:33.331 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_try_discover": 0, 2024-04-30T08:45:33.331 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_send_discover": 0, 2024-04-30T08:45:33.331 INFO:teuthology.orchestra.run.smithi071.stdout: "dir_handle_discover": 0, 2024-04-30T08:45:33.331 INFO:teuthology.orchestra.run.smithi071.stdout: "num_strays": 0, 2024-04-30T08:45:33.331 INFO:teuthology.orchestra.run.smithi071.stdout: "num_strays_delayed": 0, 2024-04-30T08:45:33.331 INFO:teuthology.orchestra.run.smithi071.stdout: "num_strays_enqueuing": 0, 2024-04-30T08:45:33.331 INFO:teuthology.orchestra.run.smithi071.stdout: "strays_created": 0, 2024-04-30T08:45:33.331 INFO:teuthology.orchestra.run.smithi071.stdout: "strays_enqueued": 0, 2024-04-30T08:45:33.331 INFO:teuthology.orchestra.run.smithi071.stdout: "strays_reintegrated": 0, 2024-04-30T08:45:33.331 INFO:teuthology.orchestra.run.smithi071.stdout: "strays_migrated": 0, 2024-04-30T08:45:33.331 INFO:teuthology.orchestra.run.smithi071.stdout: "num_recovering_processing": 0, 2024-04-30T08:45:33.331 INFO:teuthology.orchestra.run.smithi071.stdout: "num_recovering_enqueued": 0, 2024-04-30T08:45:33.331 INFO:teuthology.orchestra.run.smithi071.stdout: "num_recovering_prioritized": 0, 2024-04-30T08:45:33.332 INFO:teuthology.orchestra.run.smithi071.stdout: "recovery_started": 0, 2024-04-30T08:45:33.332 INFO:teuthology.orchestra.run.smithi071.stdout: "recovery_completed": 0, 2024-04-30T08:45:33.332 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_quiesce_path": 0, 2024-04-30T08:45:33.332 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_quiesce_inode": 0, 2024-04-30T08:45:33.332 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_enqueue_scrub": 0, 2024-04-30T08:45:33.332 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_exportdir": 0, 2024-04-30T08:45:33.332 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_flush": 0, 2024-04-30T08:45:33.332 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_fragmentdir": 0, 2024-04-30T08:45:33.332 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_fragstats": 0, 2024-04-30T08:45:33.332 INFO:teuthology.orchestra.run.smithi071.stdout: "ireq_inodestats": 0 2024-04-30T08:45:33.332 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.332 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_log": { 2024-04-30T08:45:33.332 INFO:teuthology.orchestra.run.smithi071.stdout: "evlrg": 0, 2024-04-30T08:45:33.332 INFO:teuthology.orchestra.run.smithi071.stdout: "evadd": 0, 2024-04-30T08:45:33.333 INFO:teuthology.orchestra.run.smithi071.stdout: "evex": 0, 2024-04-30T08:45:33.333 INFO:teuthology.orchestra.run.smithi071.stdout: "evtrm": 0, 2024-04-30T08:45:33.333 INFO:teuthology.orchestra.run.smithi071.stdout: "ev": 25, 2024-04-30T08:45:33.333 INFO:teuthology.orchestra.run.smithi071.stdout: "evexg": 0, 2024-04-30T08:45:33.333 INFO:teuthology.orchestra.run.smithi071.stdout: "evexd": 0, 2024-04-30T08:45:33.333 INFO:teuthology.orchestra.run.smithi071.stdout: "segadd": 0, 2024-04-30T08:45:33.333 INFO:teuthology.orchestra.run.smithi071.stdout: "segex": 0, 2024-04-30T08:45:33.333 INFO:teuthology.orchestra.run.smithi071.stdout: "segtrm": 0, 2024-04-30T08:45:33.333 INFO:teuthology.orchestra.run.smithi071.stdout: "seg": 2, 2024-04-30T08:45:33.333 INFO:teuthology.orchestra.run.smithi071.stdout: "segmjr": 2, 2024-04-30T08:45:33.333 INFO:teuthology.orchestra.run.smithi071.stdout: "segexg": 0, 2024-04-30T08:45:33.333 INFO:teuthology.orchestra.run.smithi071.stdout: "segexd": 0, 2024-04-30T08:45:33.333 INFO:teuthology.orchestra.run.smithi071.stdout: "expos": 4194304, 2024-04-30T08:45:33.333 INFO:teuthology.orchestra.run.smithi071.stdout: "wrpos": 4240427, 2024-04-30T08:45:33.334 INFO:teuthology.orchestra.run.smithi071.stdout: "rdpos": 4238776, 2024-04-30T08:45:33.334 INFO:teuthology.orchestra.run.smithi071.stdout: "jlat": { 2024-04-30T08:45:33.334 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.334 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.334 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.334 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.334 INFO:teuthology.orchestra.run.smithi071.stdout: "replayed": 25 2024-04-30T08:45:33.334 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.334 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_mem": { 2024-04-30T08:45:33.334 INFO:teuthology.orchestra.run.smithi071.stdout: "ino": 4, 2024-04-30T08:45:33.334 INFO:teuthology.orchestra.run.smithi071.stdout: "ino+": 19, 2024-04-30T08:45:33.334 INFO:teuthology.orchestra.run.smithi071.stdout: "ino-": 15, 2024-04-30T08:45:33.334 INFO:teuthology.orchestra.run.smithi071.stdout: "dir": 2, 2024-04-30T08:45:33.334 INFO:teuthology.orchestra.run.smithi071.stdout: "dir+": 16, 2024-04-30T08:45:33.335 INFO:teuthology.orchestra.run.smithi071.stdout: "dir-": 14, 2024-04-30T08:45:33.335 INFO:teuthology.orchestra.run.smithi071.stdout: "dn": 3, 2024-04-30T08:45:33.335 INFO:teuthology.orchestra.run.smithi071.stdout: "dn+": 21, 2024-04-30T08:45:33.335 INFO:teuthology.orchestra.run.smithi071.stdout: "dn-": 18, 2024-04-30T08:45:33.335 INFO:teuthology.orchestra.run.smithi071.stdout: "cap": 0, 2024-04-30T08:45:33.335 INFO:teuthology.orchestra.run.smithi071.stdout: "cap+": 0, 2024-04-30T08:45:33.335 INFO:teuthology.orchestra.run.smithi071.stdout: "cap-": 0, 2024-04-30T08:45:33.335 INFO:teuthology.orchestra.run.smithi071.stdout: "rss": 122204, 2024-04-30T08:45:33.335 INFO:teuthology.orchestra.run.smithi071.stdout: "heap": 215440 2024-04-30T08:45:33.335 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.335 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_server": { 2024-04-30T08:45:33.335 INFO:teuthology.orchestra.run.smithi071.stdout: "dispatch_client_request": 0, 2024-04-30T08:45:33.335 INFO:teuthology.orchestra.run.smithi071.stdout: "dispatch_server_request": 0, 2024-04-30T08:45:33.335 INFO:teuthology.orchestra.run.smithi071.stdout: "handle_client_request": 0, 2024-04-30T08:45:33.336 INFO:teuthology.orchestra.run.smithi071.stdout: "handle_client_session": 0, 2024-04-30T08:45:33.336 INFO:teuthology.orchestra.run.smithi071.stdout: "handle_peer_request": 0, 2024-04-30T08:45:33.336 INFO:teuthology.orchestra.run.smithi071.stdout: "req_create_latency": { 2024-04-30T08:45:33.336 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.336 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.336 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.336 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.336 INFO:teuthology.orchestra.run.smithi071.stdout: "req_getattr_latency": { 2024-04-30T08:45:33.336 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.336 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.336 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.336 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.336 INFO:teuthology.orchestra.run.smithi071.stdout: "req_getfilelock_latency": { 2024-04-30T08:45:33.336 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.337 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.337 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.337 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.337 INFO:teuthology.orchestra.run.smithi071.stdout: "req_link_latency": { 2024-04-30T08:45:33.337 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.337 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.337 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.337 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.337 INFO:teuthology.orchestra.run.smithi071.stdout: "req_lookup_latency": { 2024-04-30T08:45:33.337 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.337 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.337 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.337 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.337 INFO:teuthology.orchestra.run.smithi071.stdout: "req_lookuphash_latency": { 2024-04-30T08:45:33.338 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.338 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.338 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.338 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.338 INFO:teuthology.orchestra.run.smithi071.stdout: "req_lookupino_latency": { 2024-04-30T08:45:33.338 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.338 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.338 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.338 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.338 INFO:teuthology.orchestra.run.smithi071.stdout: "req_lookupname_latency": { 2024-04-30T08:45:33.338 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.338 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.339 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.339 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.339 INFO:teuthology.orchestra.run.smithi071.stdout: "req_lookupparent_latency": { 2024-04-30T08:45:33.339 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.339 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.339 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.339 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.339 INFO:teuthology.orchestra.run.smithi071.stdout: "req_lookupsnap_latency": { 2024-04-30T08:45:33.339 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.339 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.339 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.339 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.339 INFO:teuthology.orchestra.run.smithi071.stdout: "req_lssnap_latency": { 2024-04-30T08:45:33.339 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.340 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.340 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.340 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.340 INFO:teuthology.orchestra.run.smithi071.stdout: "req_mkdir_latency": { 2024-04-30T08:45:33.340 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.340 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.341 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.341 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.341 INFO:teuthology.orchestra.run.smithi071.stdout: "req_mknod_latency": { 2024-04-30T08:45:33.341 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.341 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.341 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.341 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.341 INFO:teuthology.orchestra.run.smithi071.stdout: "req_mksnap_latency": { 2024-04-30T08:45:33.341 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.341 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.341 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.341 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.342 INFO:teuthology.orchestra.run.smithi071.stdout: "req_open_latency": { 2024-04-30T08:45:33.342 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.342 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.342 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.342 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.342 INFO:teuthology.orchestra.run.smithi071.stdout: "req_readdir_latency": { 2024-04-30T08:45:33.342 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.342 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.342 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.342 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.342 INFO:teuthology.orchestra.run.smithi071.stdout: "req_rename_latency": { 2024-04-30T08:45:33.342 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.342 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.343 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.343 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.343 INFO:teuthology.orchestra.run.smithi071.stdout: "req_renamesnap_latency": { 2024-04-30T08:45:33.343 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.343 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.343 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.343 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.343 INFO:teuthology.orchestra.run.smithi071.stdout: "req_snapdiff_latency": { 2024-04-30T08:45:33.343 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.343 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.343 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.343 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.343 INFO:teuthology.orchestra.run.smithi071.stdout: "req_rmdir_latency": { 2024-04-30T08:45:33.344 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.344 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.344 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.344 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.344 INFO:teuthology.orchestra.run.smithi071.stdout: "req_rmsnap_latency": { 2024-04-30T08:45:33.344 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.344 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.344 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.344 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.344 INFO:teuthology.orchestra.run.smithi071.stdout: "req_rmxattr_latency": { 2024-04-30T08:45:33.344 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.344 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.344 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.345 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.345 INFO:teuthology.orchestra.run.smithi071.stdout: "req_setattr_latency": { 2024-04-30T08:45:33.345 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.345 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.345 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.345 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.345 INFO:teuthology.orchestra.run.smithi071.stdout: "req_setdirlayout_latency": { 2024-04-30T08:45:33.345 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.345 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.345 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.345 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.345 INFO:teuthology.orchestra.run.smithi071.stdout: "req_setfilelock_latency": { 2024-04-30T08:45:33.345 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.345 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.346 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.346 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.346 INFO:teuthology.orchestra.run.smithi071.stdout: "req_setlayout_latency": { 2024-04-30T08:45:33.346 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.346 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.346 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.346 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.346 INFO:teuthology.orchestra.run.smithi071.stdout: "req_setxattr_latency": { 2024-04-30T08:45:33.346 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.346 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.346 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.346 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.347 INFO:teuthology.orchestra.run.smithi071.stdout: "req_symlink_latency": { 2024-04-30T08:45:33.347 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.347 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.347 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.347 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.347 INFO:teuthology.orchestra.run.smithi071.stdout: "req_unlink_latency": { 2024-04-30T08:45:33.347 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.347 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.347 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.347 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.347 INFO:teuthology.orchestra.run.smithi071.stdout: "cap_revoke_eviction": 0, 2024-04-30T08:45:33.347 INFO:teuthology.orchestra.run.smithi071.stdout: "cap_acquisition_throttle": 0, 2024-04-30T08:45:33.347 INFO:teuthology.orchestra.run.smithi071.stdout: "req_getvxattr_latency": { 2024-04-30T08:45:33.347 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.348 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.348 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.348 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:33.348 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.348 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_sessions": { 2024-04-30T08:45:33.348 INFO:teuthology.orchestra.run.smithi071.stdout: "session_count": 1, 2024-04-30T08:45:33.348 INFO:teuthology.orchestra.run.smithi071.stdout: "session_add": 1, 2024-04-30T08:45:33.348 INFO:teuthology.orchestra.run.smithi071.stdout: "session_remove": 0, 2024-04-30T08:45:33.348 INFO:teuthology.orchestra.run.smithi071.stdout: "sessions_open": 1, 2024-04-30T08:45:33.348 INFO:teuthology.orchestra.run.smithi071.stdout: "sessions_stale": 0, 2024-04-30T08:45:33.348 INFO:teuthology.orchestra.run.smithi071.stdout: "total_load": 0, 2024-04-30T08:45:33.348 INFO:teuthology.orchestra.run.smithi071.stdout: "average_load": 0, 2024-04-30T08:45:33.348 INFO:teuthology.orchestra.run.smithi071.stdout: "avg_session_uptime": 2249, 2024-04-30T08:45:33.349 INFO:teuthology.orchestra.run.smithi071.stdout: "mdthresh_evicted": 0 2024-04-30T08:45:33.349 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.349 INFO:teuthology.orchestra.run.smithi071.stdout: "mempool": { 2024-04-30T08:45:33.349 INFO:teuthology.orchestra.run.smithi071.stdout: "bloom_filter_bytes": 0, 2024-04-30T08:45:33.349 INFO:teuthology.orchestra.run.smithi071.stdout: "bloom_filter_items": 0, 2024-04-30T08:45:33.349 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_alloc_bytes": 0, 2024-04-30T08:45:33.349 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_alloc_items": 0, 2024-04-30T08:45:33.349 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_data_bytes": 0, 2024-04-30T08:45:33.349 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_data_items": 0, 2024-04-30T08:45:33.350 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-30T08:45:33.350 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_onode_items": 0, 2024-04-30T08:45:33.350 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-30T08:45:33.350 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_meta_items": 0, 2024-04-30T08:45:33.350 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_other_bytes": 0, 2024-04-30T08:45:33.350 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_other_items": 0, 2024-04-30T08:45:33.350 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-30T08:45:33.350 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_cache_buffer_items": 0, 2024-04-30T08:45:33.350 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_extent_bytes": 0, 2024-04-30T08:45:33.350 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_extent_items": 0, 2024-04-30T08:45:33.350 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_blob_bytes": 0, 2024-04-30T08:45:33.350 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_blob_items": 0, 2024-04-30T08:45:33.350 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-30T08:45:33.350 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_shared_blob_items": 0, 2024-04-30T08:45:33.351 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-30T08:45:33.351 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_inline_bl_items": 0, 2024-04-30T08:45:33.351 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_fsck_bytes": 0, 2024-04-30T08:45:33.351 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_fsck_items": 0, 2024-04-30T08:45:33.351 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_txc_bytes": 0, 2024-04-30T08:45:33.351 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_txc_items": 0, 2024-04-30T08:45:33.351 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-30T08:45:33.351 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_writing_deferred_items": 0, 2024-04-30T08:45:33.351 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_writing_bytes": 0, 2024-04-30T08:45:33.351 INFO:teuthology.orchestra.run.smithi071.stdout: "bluestore_writing_items": 0, 2024-04-30T08:45:33.351 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_bytes": 0, 2024-04-30T08:45:33.352 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_items": 0, 2024-04-30T08:45:33.352 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_file_reader_bytes": 0, 2024-04-30T08:45:33.352 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_file_reader_items": 0, 2024-04-30T08:45:33.352 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_file_writer_bytes": 0, 2024-04-30T08:45:33.352 INFO:teuthology.orchestra.run.smithi071.stdout: "bluefs_file_writer_items": 0, 2024-04-30T08:45:33.352 INFO:teuthology.orchestra.run.smithi071.stdout: "buffer_anon_bytes": 197481, 2024-04-30T08:45:33.352 INFO:teuthology.orchestra.run.smithi071.stdout: "buffer_anon_items": 55, 2024-04-30T08:45:33.352 INFO:teuthology.orchestra.run.smithi071.stdout: "buffer_meta_bytes": 0, 2024-04-30T08:45:33.352 INFO:teuthology.orchestra.run.smithi071.stdout: "buffer_meta_items": 0, 2024-04-30T08:45:33.352 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_bytes": 0, 2024-04-30T08:45:33.352 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_items": 0, 2024-04-30T08:45:33.352 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_mapbl_bytes": 0, 2024-04-30T08:45:33.352 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_mapbl_items": 0, 2024-04-30T08:45:33.353 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pglog_bytes": 0, 2024-04-30T08:45:33.353 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_pglog_items": 0, 2024-04-30T08:45:33.353 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_bytes": 9088, 2024-04-30T08:45:33.353 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_items": 175, 2024-04-30T08:45:33.353 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_mapping_bytes": 0, 2024-04-30T08:45:33.353 INFO:teuthology.orchestra.run.smithi071.stdout: "osdmap_mapping_items": 0, 2024-04-30T08:45:33.353 INFO:teuthology.orchestra.run.smithi071.stdout: "pgmap_bytes": 0, 2024-04-30T08:45:33.353 INFO:teuthology.orchestra.run.smithi071.stdout: "pgmap_items": 0, 2024-04-30T08:45:33.353 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_co_bytes": 14160, 2024-04-30T08:45:33.353 INFO:teuthology.orchestra.run.smithi071.stdout: "mds_co_items": 40, 2024-04-30T08:45:33.353 INFO:teuthology.orchestra.run.smithi071.stdout: "unittest_1_bytes": 0, 2024-04-30T08:45:33.354 INFO:teuthology.orchestra.run.smithi071.stdout: "unittest_1_items": 0, 2024-04-30T08:45:33.354 INFO:teuthology.orchestra.run.smithi071.stdout: "unittest_2_bytes": 0, 2024-04-30T08:45:33.354 INFO:teuthology.orchestra.run.smithi071.stdout: "unittest_2_items": 0 2024-04-30T08:45:33.354 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.354 INFO:teuthology.orchestra.run.smithi071.stdout: "objecter": { 2024-04-30T08:45:33.354 INFO:teuthology.orchestra.run.smithi071.stdout: "op_active": 0, 2024-04-30T08:45:33.354 INFO:teuthology.orchestra.run.smithi071.stdout: "op_laggy": 0, 2024-04-30T08:45:33.354 INFO:teuthology.orchestra.run.smithi071.stdout: "op_send": 2001, 2024-04-30T08:45:33.354 INFO:teuthology.orchestra.run.smithi071.stdout: "op_send_bytes": 21, 2024-04-30T08:45:33.354 INFO:teuthology.orchestra.run.smithi071.stdout: "op_resend": 3, 2024-04-30T08:45:33.354 INFO:teuthology.orchestra.run.smithi071.stdout: "op_reply": 2001, 2024-04-30T08:45:33.355 INFO:teuthology.orchestra.run.smithi071.stdout: "op_latency": { 2024-04-30T08:45:33.355 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 2001, 2024-04-30T08:45:33.355 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 34.442395458, 2024-04-30T08:45:33.355 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.017212591 2024-04-30T08:45:33.355 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.355 INFO:teuthology.orchestra.run.smithi071.stdout: "op_inflight": 0, 2024-04-30T08:45:33.355 INFO:teuthology.orchestra.run.smithi071.stdout: "oplen_avg": { 2024-04-30T08:45:33.355 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 2001, 2024-04-30T08:45:33.355 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 2002 2024-04-30T08:45:33.355 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.355 INFO:teuthology.orchestra.run.smithi071.stdout: "op": 2001, 2024-04-30T08:45:33.355 INFO:teuthology.orchestra.run.smithi071.stdout: "op_r": 2001, 2024-04-30T08:45:33.356 INFO:teuthology.orchestra.run.smithi071.stdout: "op_w": 0, 2024-04-30T08:45:33.356 INFO:teuthology.orchestra.run.smithi071.stdout: "op_rmw": 0, 2024-04-30T08:45:33.356 INFO:teuthology.orchestra.run.smithi071.stdout: "op_pg": 0, 2024-04-30T08:45:33.356 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_stat": 1238, 2024-04-30T08:45:33.356 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_create": 0, 2024-04-30T08:45:33.356 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_read": 761, 2024-04-30T08:45:33.356 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_write": 0, 2024-04-30T08:45:33.356 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_writefull": 0, 2024-04-30T08:45:33.356 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_writesame": 0, 2024-04-30T08:45:33.356 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_append": 0, 2024-04-30T08:45:33.356 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_zero": 0, 2024-04-30T08:45:33.356 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_truncate": 0, 2024-04-30T08:45:33.357 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_delete": 0, 2024-04-30T08:45:33.357 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_mapext": 0, 2024-04-30T08:45:33.357 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_sparse_read": 0, 2024-04-30T08:45:33.357 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_clonerange": 0, 2024-04-30T08:45:33.357 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_getxattr": 1, 2024-04-30T08:45:33.357 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_setxattr": 0, 2024-04-30T08:45:33.357 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_cmpxattr": 0, 2024-04-30T08:45:33.357 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_rmxattr": 0, 2024-04-30T08:45:33.357 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_resetxattrs": 0, 2024-04-30T08:45:33.357 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_call": 0, 2024-04-30T08:45:33.358 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_watch": 0, 2024-04-30T08:45:33.358 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_notify": 0, 2024-04-30T08:45:33.358 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_src_cmpxattr": 0, 2024-04-30T08:45:33.358 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_pgls": 0, 2024-04-30T08:45:33.358 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_pgls_filter": 0, 2024-04-30T08:45:33.358 INFO:teuthology.orchestra.run.smithi071.stdout: "osdop_other": 0, 2024-04-30T08:45:33.358 INFO:teuthology.orchestra.run.smithi071.stdout: "linger_active": 0, 2024-04-30T08:45:33.358 INFO:teuthology.orchestra.run.smithi071.stdout: "linger_send": 0, 2024-04-30T08:45:33.358 INFO:teuthology.orchestra.run.smithi071.stdout: "linger_resend": 0, 2024-04-30T08:45:33.358 INFO:teuthology.orchestra.run.smithi071.stdout: "linger_ping": 0, 2024-04-30T08:45:33.359 INFO:teuthology.orchestra.run.smithi071.stdout: "poolop_active": 0, 2024-04-30T08:45:33.359 INFO:teuthology.orchestra.run.smithi071.stdout: "poolop_send": 0, 2024-04-30T08:45:33.359 INFO:teuthology.orchestra.run.smithi071.stdout: "poolop_resend": 0, 2024-04-30T08:45:33.359 INFO:teuthology.orchestra.run.smithi071.stdout: "poolstat_active": 0, 2024-04-30T08:45:33.359 INFO:teuthology.orchestra.run.smithi071.stdout: "poolstat_send": 0, 2024-04-30T08:45:33.359 INFO:teuthology.orchestra.run.smithi071.stdout: "poolstat_resend": 0, 2024-04-30T08:45:33.359 INFO:teuthology.orchestra.run.smithi071.stdout: "statfs_active": 0, 2024-04-30T08:45:33.359 INFO:teuthology.orchestra.run.smithi071.stdout: "statfs_send": 0, 2024-04-30T08:45:33.359 INFO:teuthology.orchestra.run.smithi071.stdout: "statfs_resend": 0, 2024-04-30T08:45:33.359 INFO:teuthology.orchestra.run.smithi071.stdout: "command_active": 0, 2024-04-30T08:45:33.360 INFO:teuthology.orchestra.run.smithi071.stdout: "command_send": 0, 2024-04-30T08:45:33.360 INFO:teuthology.orchestra.run.smithi071.stdout: "command_resend": 0, 2024-04-30T08:45:33.360 INFO:teuthology.orchestra.run.smithi071.stdout: "map_epoch": 85, 2024-04-30T08:45:33.360 INFO:teuthology.orchestra.run.smithi071.stdout: "map_full": 0, 2024-04-30T08:45:33.360 INFO:teuthology.orchestra.run.smithi071.stdout: "map_inc": 3, 2024-04-30T08:45:33.360 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_sessions": 6, 2024-04-30T08:45:33.360 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_session_open": 6, 2024-04-30T08:45:33.360 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_session_close": 0, 2024-04-30T08:45:33.360 INFO:teuthology.orchestra.run.smithi071.stdout: "osd_laggy": 0, 2024-04-30T08:45:33.360 INFO:teuthology.orchestra.run.smithi071.stdout: "omap_wr": 0, 2024-04-30T08:45:33.361 INFO:teuthology.orchestra.run.smithi071.stdout: "omap_rd": 2, 2024-04-30T08:45:33.361 INFO:teuthology.orchestra.run.smithi071.stdout: "omap_del": 0 2024-04-30T08:45:33.361 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.361 INFO:teuthology.orchestra.run.smithi071.stdout: "oft": { 2024-04-30T08:45:33.361 INFO:teuthology.orchestra.run.smithi071.stdout: "omap_total_objs": 0, 2024-04-30T08:45:33.361 INFO:teuthology.orchestra.run.smithi071.stdout: "omap_total_kv_pairs": 0, 2024-04-30T08:45:33.361 INFO:teuthology.orchestra.run.smithi071.stdout: "omap_total_updates": 0, 2024-04-30T08:45:33.361 INFO:teuthology.orchestra.run.smithi071.stdout: "omap_total_removes": 0 2024-04-30T08:45:33.361 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.362 INFO:teuthology.orchestra.run.smithi071.stdout: "osd-slow-ops": { 2024-04-30T08:45:33.362 INFO:teuthology.orchestra.run.smithi071.stdout: "slow_ops_count": 0 2024-04-30T08:45:33.362 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.362 INFO:teuthology.orchestra.run.smithi071.stdout: "purge_queue": { 2024-04-30T08:45:33.362 INFO:teuthology.orchestra.run.smithi071.stdout: "pq_executing_ops": 0, 2024-04-30T08:45:33.362 INFO:teuthology.orchestra.run.smithi071.stdout: "pq_executing_ops_high_water": 0, 2024-04-30T08:45:33.362 INFO:teuthology.orchestra.run.smithi071.stdout: "pq_executing": 0, 2024-04-30T08:45:33.362 INFO:teuthology.orchestra.run.smithi071.stdout: "pq_executing_high_water": 0, 2024-04-30T08:45:33.362 INFO:teuthology.orchestra.run.smithi071.stdout: "pq_executed_ops": 0, 2024-04-30T08:45:33.362 INFO:teuthology.orchestra.run.smithi071.stdout: "pq_executed": 0, 2024-04-30T08:45:33.363 INFO:teuthology.orchestra.run.smithi071.stdout: "pq_item_in_journal": 0 2024-04-30T08:45:33.363 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.363 INFO:teuthology.orchestra.run.smithi071.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-30T08:45:33.363 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 0, 2024-04-30T08:45:33.363 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 104857600, 2024-04-30T08:45:33.363 INFO:teuthology.orchestra.run.smithi071.stdout: "get_started": 0, 2024-04-30T08:45:33.363 INFO:teuthology.orchestra.run.smithi071.stdout: "get": 2352, 2024-04-30T08:45:33.363 INFO:teuthology.orchestra.run.smithi071.stdout: "get_sum": 1922555, 2024-04-30T08:45:33.363 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:33.363 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_success": 2352, 2024-04-30T08:45:33.363 INFO:teuthology.orchestra.run.smithi071.stdout: "take": 0, 2024-04-30T08:45:33.363 INFO:teuthology.orchestra.run.smithi071.stdout: "take_sum": 0, 2024-04-30T08:45:33.363 INFO:teuthology.orchestra.run.smithi071.stdout: "put": 2352, 2024-04-30T08:45:33.364 INFO:teuthology.orchestra.run.smithi071.stdout: "put_sum": 1922555, 2024-04-30T08:45:33.364 INFO:teuthology.orchestra.run.smithi071.stdout: "wait": { 2024-04-30T08:45:33.364 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.364 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.364 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.364 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:33.364 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.364 INFO:teuthology.orchestra.run.smithi071.stdout: "throttle-objecter_bytes": { 2024-04-30T08:45:33.364 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 0, 2024-04-30T08:45:33.364 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 104857600, 2024-04-30T08:45:33.364 INFO:teuthology.orchestra.run.smithi071.stdout: "get_started": 0, 2024-04-30T08:45:33.364 INFO:teuthology.orchestra.run.smithi071.stdout: "get": 0, 2024-04-30T08:45:33.364 INFO:teuthology.orchestra.run.smithi071.stdout: "get_sum": 0, 2024-04-30T08:45:33.364 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:33.364 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_success": 0, 2024-04-30T08:45:33.365 INFO:teuthology.orchestra.run.smithi071.stdout: "take": 2001, 2024-04-30T08:45:33.365 INFO:teuthology.orchestra.run.smithi071.stdout: "take_sum": 46128, 2024-04-30T08:45:33.365 INFO:teuthology.orchestra.run.smithi071.stdout: "put": 2001, 2024-04-30T08:45:33.365 INFO:teuthology.orchestra.run.smithi071.stdout: "put_sum": 46128, 2024-04-30T08:45:33.365 INFO:teuthology.orchestra.run.smithi071.stdout: "wait": { 2024-04-30T08:45:33.365 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.365 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.365 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.365 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:33.365 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.365 INFO:teuthology.orchestra.run.smithi071.stdout: "throttle-objecter_ops": { 2024-04-30T08:45:33.365 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 0, 2024-04-30T08:45:33.365 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 1024, 2024-04-30T08:45:33.366 INFO:teuthology.orchestra.run.smithi071.stdout: "get_started": 0, 2024-04-30T08:45:33.366 INFO:teuthology.orchestra.run.smithi071.stdout: "get": 0, 2024-04-30T08:45:33.366 INFO:teuthology.orchestra.run.smithi071.stdout: "get_sum": 0, 2024-04-30T08:45:33.366 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:33.366 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_success": 0, 2024-04-30T08:45:33.366 INFO:teuthology.orchestra.run.smithi071.stdout: "take": 2001, 2024-04-30T08:45:33.366 INFO:teuthology.orchestra.run.smithi071.stdout: "take_sum": 2001, 2024-04-30T08:45:33.366 INFO:teuthology.orchestra.run.smithi071.stdout: "put": 2001, 2024-04-30T08:45:33.366 INFO:teuthology.orchestra.run.smithi071.stdout: "put_sum": 2001, 2024-04-30T08:45:33.366 INFO:teuthology.orchestra.run.smithi071.stdout: "wait": { 2024-04-30T08:45:33.366 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.366 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.366 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.367 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:33.367 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.367 INFO:teuthology.orchestra.run.smithi071.stdout: "throttle-write_buf_throttle": { 2024-04-30T08:45:33.367 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 0, 2024-04-30T08:45:33.367 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 3758096384, 2024-04-30T08:45:33.367 INFO:teuthology.orchestra.run.smithi071.stdout: "get_started": 0, 2024-04-30T08:45:33.367 INFO:teuthology.orchestra.run.smithi071.stdout: "get": 0, 2024-04-30T08:45:33.367 INFO:teuthology.orchestra.run.smithi071.stdout: "get_sum": 0, 2024-04-30T08:45:33.367 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:33.367 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_success": 0, 2024-04-30T08:45:33.367 INFO:teuthology.orchestra.run.smithi071.stdout: "take": 0, 2024-04-30T08:45:33.367 INFO:teuthology.orchestra.run.smithi071.stdout: "take_sum": 0, 2024-04-30T08:45:33.367 INFO:teuthology.orchestra.run.smithi071.stdout: "put": 0, 2024-04-30T08:45:33.368 INFO:teuthology.orchestra.run.smithi071.stdout: "put_sum": 0, 2024-04-30T08:45:33.368 INFO:teuthology.orchestra.run.smithi071.stdout: "wait": { 2024-04-30T08:45:33.368 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.368 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.368 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.368 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:33.368 INFO:teuthology.orchestra.run.smithi071.stdout: }, 2024-04-30T08:45:33.368 INFO:teuthology.orchestra.run.smithi071.stdout: "throttle-write_buf_throttle-0x5575cd5b2410": { 2024-04-30T08:45:33.368 INFO:teuthology.orchestra.run.smithi071.stdout: "val": 0, 2024-04-30T08:45:33.368 INFO:teuthology.orchestra.run.smithi071.stdout: "max": 3758096384, 2024-04-30T08:45:33.368 INFO:teuthology.orchestra.run.smithi071.stdout: "get_started": 0, 2024-04-30T08:45:33.368 INFO:teuthology.orchestra.run.smithi071.stdout: "get": 0, 2024-04-30T08:45:33.369 INFO:teuthology.orchestra.run.smithi071.stdout: "get_sum": 0, 2024-04-30T08:45:33.369 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:33.369 INFO:teuthology.orchestra.run.smithi071.stdout: "get_or_fail_success": 0, 2024-04-30T08:45:33.369 INFO:teuthology.orchestra.run.smithi071.stdout: "take": 0, 2024-04-30T08:45:33.369 INFO:teuthology.orchestra.run.smithi071.stdout: "take_sum": 0, 2024-04-30T08:45:33.369 INFO:teuthology.orchestra.run.smithi071.stdout: "put": 0, 2024-04-30T08:45:33.369 INFO:teuthology.orchestra.run.smithi071.stdout: "put_sum": 0, 2024-04-30T08:45:33.369 INFO:teuthology.orchestra.run.smithi071.stdout: "wait": { 2024-04-30T08:45:33.369 INFO:teuthology.orchestra.run.smithi071.stdout: "avgcount": 0, 2024-04-30T08:45:33.369 INFO:teuthology.orchestra.run.smithi071.stdout: "sum": 0.000000000, 2024-04-30T08:45:33.369 INFO:teuthology.orchestra.run.smithi071.stdout: "avgtime": 0.000000000 2024-04-30T08:45:33.369 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:33.370 INFO:teuthology.orchestra.run.smithi071.stdout: } 2024-04-30T08:45:33.370 INFO:teuthology.orchestra.run.smithi071.stdout:} 2024-04-30T08:45:33.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:33 smithi189 ceph-mon[33038]: pgmap v736: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-30T08:45:33.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:33 smithi032 ceph-mon[28331]: pgmap v736: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 10 op/s 2024-04-30T08:45:34.025 INFO:tasks.check_counter:Daemon mds.k mds.exported=0 2024-04-30T08:45:34.025 INFO:tasks.check_counter:Daemon mds.k mds.imported=0 2024-04-30T08:45:34.025 INFO:tasks.check_counter:Daemon mds.k mds.dir_split=0 2024-04-30T08:45:34.026 DEBUG:tasks.check_counter:Getting stats from c 2024-04-30T08:45:34.026 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-30T08:45:34.315 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.c/config 2024-04-30T08:45:35.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:35 smithi071 ceph-mon[33385]: pgmap v737: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 8 op/s 2024-04-30T08:45:35.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:35 smithi189 ceph-mon[33038]: pgmap v737: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 8 op/s 2024-04-30T08:45:35.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:35 smithi032 ceph-mon[28331]: pgmap v737: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 8 op/s 2024-04-30T08:45:36.289 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-30T08:45:36.290 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-0": { 2024-04-30T08:45:36.290 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 60, 2024-04-30T08:45:36.290 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 130, 2024-04-30T08:45:36.290 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 15678, 2024-04-30T08:45:36.290 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 213318, 2024-04-30T08:45:36.290 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 24, 2024-04-30T08:45:36.290 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 5, 2024-04-30T08:45:36.290 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.016450138, 2024-04-30T08:45:36.290 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.003272093, 2024-04-30T08:45:36.290 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.008683802, 2024-04-30T08:45:36.291 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.001357322, 2024-04-30T08:45:36.291 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:36.291 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 129, 2024-04-30T08:45:36.291 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.019581804, 2024-04-30T08:45:36.291 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000151796 2024-04-30T08:45:36.291 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.292 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:36.292 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 21, 2024-04-30T08:45:36.292 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000003246, 2024-04-30T08:45:36.292 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000154 2024-04-30T08:45:36.292 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.292 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 3920, 2024-04-30T08:45:36.292 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 1184 2024-04-30T08:45:36.292 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.292 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-1": { 2024-04-30T08:45:36.292 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 1412, 2024-04-30T08:45:36.292 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 1325, 2024-04-30T08:45:36.292 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 1857410, 2024-04-30T08:45:36.292 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 487491, 2024-04-30T08:45:36.292 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 6, 2024-04-30T08:45:36.292 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 5, 2024-04-30T08:45:36.292 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.145015194, 2024-04-30T08:45:36.293 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.048864501, 2024-04-30T08:45:36.293 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.109375060, 2024-04-30T08:45:36.293 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.011613561, 2024-04-30T08:45:36.293 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:36.293 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 1324, 2024-04-30T08:45:36.293 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.073703012, 2024-04-30T08:45:36.293 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000055666 2024-04-30T08:45:36.293 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.293 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:36.293 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 1932, 2024-04-30T08:45:36.293 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000494437, 2024-04-30T08:45:36.293 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000255 2024-04-30T08:45:36.293 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.293 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 1544384, 2024-04-30T08:45:36.293 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 148240 2024-04-30T08:45:36.294 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.294 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-2": { 2024-04-30T08:45:36.294 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 41, 2024-04-30T08:45:36.294 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 215, 2024-04-30T08:45:36.294 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 11464, 2024-04-30T08:45:36.294 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 467712, 2024-04-30T08:45:36.294 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 8, 2024-04-30T08:45:36.294 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 5, 2024-04-30T08:45:36.294 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.018034289, 2024-04-30T08:45:36.294 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.009522399, 2024-04-30T08:45:36.294 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.006319142, 2024-04-30T08:45:36.294 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.000654795, 2024-04-30T08:45:36.295 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:36.295 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 213, 2024-04-30T08:45:36.295 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.026541708, 2024-04-30T08:45:36.295 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000124608 2024-04-30T08:45:36.295 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.295 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:36.295 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 15, 2024-04-30T08:45:36.295 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000012147, 2024-04-30T08:45:36.295 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000809 2024-04-30T08:45:36.295 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.295 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-30T08:45:36.295 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 451184 2024-04-30T08:45:36.295 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.295 INFO:teuthology.orchestra.run.smithi189.stdout: "cct": { 2024-04-30T08:45:36.296 INFO:teuthology.orchestra.run.smithi189.stdout: "total_workers": 1, 2024-04-30T08:45:36.296 INFO:teuthology.orchestra.run.smithi189.stdout: "unhealthy_workers": 0 2024-04-30T08:45:36.296 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.296 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-MDSRank": { 2024-04-30T08:45:36.296 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-04-30T08:45:36.296 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-04-30T08:45:36.296 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 85, 2024-04-30T08:45:36.296 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.005147742, 2024-04-30T08:45:36.296 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000060561 2024-04-30T08:45:36.296 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:36.296 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.296 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-PurgeQueue": { 2024-04-30T08:45:36.296 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-04-30T08:45:36.296 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-04-30T08:45:36.297 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 16, 2024-04-30T08:45:36.297 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000416703, 2024-04-30T08:45:36.297 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000026043 2024-04-30T08:45:36.297 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:36.297 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.297 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": { 2024-04-30T08:45:36.297 INFO:teuthology.orchestra.run.smithi189.stdout: "request": 3, 2024-04-30T08:45:36.297 INFO:teuthology.orchestra.run.smithi189.stdout: "reply": 3, 2024-04-30T08:45:36.297 INFO:teuthology.orchestra.run.smithi189.stdout: "reply_latency": { 2024-04-30T08:45:36.297 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 3, 2024-04-30T08:45:36.297 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.003640035, 2024-04-30T08:45:36.297 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.001213345 2024-04-30T08:45:36.297 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.297 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_reply": 0, 2024-04-30T08:45:36.297 INFO:teuthology.orchestra.run.smithi189.stdout: "forward": 0, 2024-04-30T08:45:36.298 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_complete": 0, 2024-04-30T08:45:36.298 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_keys": 0, 2024-04-30T08:45:36.298 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_commit": 11, 2024-04-30T08:45:36.298 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_split": 0, 2024-04-30T08:45:36.298 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_merge": 0, 2024-04-30T08:45:36.298 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes": 15, 2024-04-30T08:45:36.298 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_top": 0, 2024-04-30T08:45:36.298 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_bottom": 15, 2024-04-30T08:45:36.298 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pin_tail": 0, 2024-04-30T08:45:36.298 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pinned": 13, 2024-04-30T08:45:36.298 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_expired": 12, 2024-04-30T08:45:36.298 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_with_caps": 1, 2024-04-30T08:45:36.298 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": 1, 2024-04-30T08:45:36.298 INFO:teuthology.orchestra.run.smithi189.stdout: "subtrees": 3, 2024-04-30T08:45:36.299 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse": 7, 2024-04-30T08:45:36.299 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_hit": 3, 2024-04-30T08:45:36.299 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_forward": 0, 2024-04-30T08:45:36.299 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_discover": 1, 2024-04-30T08:45:36.299 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_dir_fetch": 0, 2024-04-30T08:45:36.299 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_remote_ino": 0, 2024-04-30T08:45:36.299 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_lock": 0, 2024-04-30T08:45:36.299 INFO:teuthology.orchestra.run.smithi189.stdout: "load_cent": 0, 2024-04-30T08:45:36.299 INFO:teuthology.orchestra.run.smithi189.stdout: "q": 0, 2024-04-30T08:45:36.299 INFO:teuthology.orchestra.run.smithi189.stdout: "exported": 1, 2024-04-30T08:45:36.299 INFO:teuthology.orchestra.run.smithi189.stdout: "exported_inodes": 0, 2024-04-30T08:45:36.299 INFO:teuthology.orchestra.run.smithi189.stdout: "imported": 1, 2024-04-30T08:45:36.299 INFO:teuthology.orchestra.run.smithi189.stdout: "imported_inodes": 2, 2024-04-30T08:45:36.299 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_dir_fetch": 0, 2024-04-30T08:45:36.299 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_backtrace_fetch": 2, 2024-04-30T08:45:36.300 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_peer_discover": 0, 2024-04-30T08:45:36.300 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rfiles": 325, 2024-04-30T08:45:36.300 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rbytes": 13424230511, 2024-04-30T08:45:36.300 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rsnaps": 0, 2024-04-30T08:45:36.300 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_fetch": 2, 2024-04-30T08:45:36.300 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_set_tag": 0, 2024-04-30T08:45:36.300 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_repaired": 0, 2024-04-30T08:45:36.300 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_inotable_repaired": 0, 2024-04-30T08:45:36.300 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_inodes": 0, 2024-04-30T08:45:36.300 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_base_inodes": 0, 2024-04-30T08:45:36.300 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dirfrag_rstats": 0, 2024-04-30T08:45:36.300 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_file_inodes": 2, 2024-04-30T08:45:36.300 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_inode_file_caps": 0, 2024-04-30T08:45:36.300 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_revoke": 1, 2024-04-30T08:45:36.301 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_grant": 2, 2024-04-30T08:45:36.301 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_trunc": 0, 2024-04-30T08:45:36.301 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-30T08:45:36.301 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-30T08:45:36.301 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps": 3, 2024-04-30T08:45:36.301 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps_dirty": 0, 2024-04-30T08:45:36.301 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_cap_release": 1, 2024-04-30T08:45:36.301 INFO:teuthology.orchestra.run.smithi189.stdout: "process_request_cap_release": 2 2024-04-30T08:45:36.301 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.301 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_cache": { 2024-04-30T08:45:36.301 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update": 0, 2024-04-30T08:45:36.301 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update_receipt": 0, 2024-04-30T08:45:36.301 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_try_discover": 0, 2024-04-30T08:45:36.301 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_send_discover": 2, 2024-04-30T08:45:36.301 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_handle_discover": 0, 2024-04-30T08:45:36.301 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays": 0, 2024-04-30T08:45:36.302 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_delayed": 0, 2024-04-30T08:45:36.302 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_enqueuing": 0, 2024-04-30T08:45:36.302 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_created": 1, 2024-04-30T08:45:36.302 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_enqueued": 1, 2024-04-30T08:45:36.302 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_reintegrated": 0, 2024-04-30T08:45:36.302 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_migrated": 0, 2024-04-30T08:45:36.302 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_processing": 0, 2024-04-30T08:45:36.302 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_enqueued": 0, 2024-04-30T08:45:36.302 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_prioritized": 0, 2024-04-30T08:45:36.302 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_started": 0, 2024-04-30T08:45:36.302 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_completed": 0, 2024-04-30T08:45:36.302 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_quiesce_path": 0, 2024-04-30T08:45:36.302 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_quiesce_inode": 0, 2024-04-30T08:45:36.302 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_enqueue_scrub": 0, 2024-04-30T08:45:36.302 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_exportdir": 1, 2024-04-30T08:45:36.303 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_flush": 0, 2024-04-30T08:45:36.303 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragmentdir": 0, 2024-04-30T08:45:36.303 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragstats": 0, 2024-04-30T08:45:36.303 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_inodestats": 0 2024-04-30T08:45:36.303 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.303 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_log": { 2024-04-30T08:45:36.303 INFO:teuthology.orchestra.run.smithi189.stdout: "evlrg": 0, 2024-04-30T08:45:36.303 INFO:teuthology.orchestra.run.smithi189.stdout: "evadd": 17, 2024-04-30T08:45:36.303 INFO:teuthology.orchestra.run.smithi189.stdout: "evex": 0, 2024-04-30T08:45:36.303 INFO:teuthology.orchestra.run.smithi189.stdout: "evtrm": 0, 2024-04-30T08:45:36.303 INFO:teuthology.orchestra.run.smithi189.stdout: "ev": 17, 2024-04-30T08:45:36.303 INFO:teuthology.orchestra.run.smithi189.stdout: "evexg": 0, 2024-04-30T08:45:36.303 INFO:teuthology.orchestra.run.smithi189.stdout: "evexd": 0, 2024-04-30T08:45:36.303 INFO:teuthology.orchestra.run.smithi189.stdout: "segadd": 2, 2024-04-30T08:45:36.303 INFO:teuthology.orchestra.run.smithi189.stdout: "segex": 0, 2024-04-30T08:45:36.304 INFO:teuthology.orchestra.run.smithi189.stdout: "segtrm": 0, 2024-04-30T08:45:36.304 INFO:teuthology.orchestra.run.smithi189.stdout: "seg": 2, 2024-04-30T08:45:36.304 INFO:teuthology.orchestra.run.smithi189.stdout: "segmjr": 2, 2024-04-30T08:45:36.304 INFO:teuthology.orchestra.run.smithi189.stdout: "segexg": 0, 2024-04-30T08:45:36.304 INFO:teuthology.orchestra.run.smithi189.stdout: "segexd": 0, 2024-04-30T08:45:36.304 INFO:teuthology.orchestra.run.smithi189.stdout: "expos": 4194304, 2024-04-30T08:45:36.304 INFO:teuthology.orchestra.run.smithi189.stdout: "wrpos": 4219327, 2024-04-30T08:45:36.304 INFO:teuthology.orchestra.run.smithi189.stdout: "rdpos": 0, 2024-04-30T08:45:36.304 INFO:teuthology.orchestra.run.smithi189.stdout: "jlat": { 2024-04-30T08:45:36.304 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 14, 2024-04-30T08:45:36.304 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.149118726, 2024-04-30T08:45:36.304 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.010651337 2024-04-30T08:45:36.304 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.304 INFO:teuthology.orchestra.run.smithi189.stdout: "replayed": 0 2024-04-30T08:45:36.304 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.304 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_mem": { 2024-04-30T08:45:36.305 INFO:teuthology.orchestra.run.smithi189.stdout: "ino": 19, 2024-04-30T08:45:36.305 INFO:teuthology.orchestra.run.smithi189.stdout: "ino+": 26, 2024-04-30T08:45:36.305 INFO:teuthology.orchestra.run.smithi189.stdout: "ino-": 7, 2024-04-30T08:45:36.305 INFO:teuthology.orchestra.run.smithi189.stdout: "dir": 18, 2024-04-30T08:45:36.305 INFO:teuthology.orchestra.run.smithi189.stdout: "dir+": 25, 2024-04-30T08:45:36.305 INFO:teuthology.orchestra.run.smithi189.stdout: "dir-": 7, 2024-04-30T08:45:36.305 INFO:teuthology.orchestra.run.smithi189.stdout: "dn": 15, 2024-04-30T08:45:36.305 INFO:teuthology.orchestra.run.smithi189.stdout: "dn+": 30, 2024-04-30T08:45:36.305 INFO:teuthology.orchestra.run.smithi189.stdout: "dn-": 15, 2024-04-30T08:45:36.305 INFO:teuthology.orchestra.run.smithi189.stdout: "cap": 1, 2024-04-30T08:45:36.305 INFO:teuthology.orchestra.run.smithi189.stdout: "cap+": 3, 2024-04-30T08:45:36.305 INFO:teuthology.orchestra.run.smithi189.stdout: "cap-": 2, 2024-04-30T08:45:36.305 INFO:teuthology.orchestra.run.smithi189.stdout: "rss": 124948, 2024-04-30T08:45:36.305 INFO:teuthology.orchestra.run.smithi189.stdout: "heap": 223632 2024-04-30T08:45:36.305 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.305 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_server": { 2024-04-30T08:45:36.306 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_client_request": 6, 2024-04-30T08:45:36.306 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_server_request": 1, 2024-04-30T08:45:36.306 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_request": 3, 2024-04-30T08:45:36.306 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_session": 13, 2024-04-30T08:45:36.306 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_peer_request": 4, 2024-04-30T08:45:36.306 INFO:teuthology.orchestra.run.smithi189.stdout: "req_create_latency": { 2024-04-30T08:45:36.306 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:36.306 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:36.306 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:36.306 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.306 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getattr_latency": { 2024-04-30T08:45:36.306 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:36.306 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:36.306 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:36.306 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.307 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getfilelock_latency": { 2024-04-30T08:45:36.307 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:36.307 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:36.307 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:36.307 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.307 INFO:teuthology.orchestra.run.smithi189.stdout: "req_link_latency": { 2024-04-30T08:45:36.307 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:36.307 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:36.307 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:36.307 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.307 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookup_latency": { 2024-04-30T08:45:36.307 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:36.307 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:36.307 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:36.307 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.307 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookuphash_latency": { 2024-04-30T08:45:36.308 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:36.308 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:36.308 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:36.308 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.308 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupino_latency": { 2024-04-30T08:45:36.308 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 1, 2024-04-30T08:45:36.308 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000158587, 2024-04-30T08:45:36.308 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000158587 2024-04-30T08:45:36.308 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.308 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupname_latency": { 2024-04-30T08:45:36.308 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:36.308 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:36.308 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:36.308 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.308 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupparent_latency": { 2024-04-30T08:45:36.309 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:36.309 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:36.309 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:36.309 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.309 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupsnap_latency": { 2024-04-30T08:45:36.309 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:36.309 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:36.309 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:36.309 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.309 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lssnap_latency": { 2024-04-30T08:45:36.309 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:36.309 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:36.309 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:36.309 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.309 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mkdir_latency": { 2024-04-30T08:45:36.310 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:36.310 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:36.310 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:36.310 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.310 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mknod_latency": { 2024-04-30T08:45:36.311 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:36.311 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:36.311 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:36.311 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.311 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mksnap_latency": { 2024-04-30T08:45:36.311 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:36.311 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:36.311 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:36.311 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.311 INFO:teuthology.orchestra.run.smithi189.stdout: "req_open_latency": { 2024-04-30T08:45:36.311 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:36.311 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:36.311 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:36.311 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.311 INFO:teuthology.orchestra.run.smithi189.stdout: "req_readdir_latency": { 2024-04-30T08:45:36.311 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 1, 2024-04-30T08:45:36.312 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000354000, 2024-04-30T08:45:36.312 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000354000 2024-04-30T08:45:36.312 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.312 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rename_latency": { 2024-04-30T08:45:36.312 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:36.312 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:36.312 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:36.312 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.312 INFO:teuthology.orchestra.run.smithi189.stdout: "req_renamesnap_latency": { 2024-04-30T08:45:36.312 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:36.312 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:36.312 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:36.312 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.312 INFO:teuthology.orchestra.run.smithi189.stdout: "req_snapdiff_latency": { 2024-04-30T08:45:36.313 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:36.313 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:36.313 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:36.313 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.313 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmdir_latency": { 2024-04-30T08:45:36.313 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:36.313 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:36.313 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:36.313 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.313 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmsnap_latency": { 2024-04-30T08:45:36.313 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:36.313 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:36.313 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:36.313 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.313 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmxattr_latency": { 2024-04-30T08:45:36.313 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:36.314 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:36.314 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:36.314 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.314 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setattr_latency": { 2024-04-30T08:45:36.314 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:36.314 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:36.314 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:36.314 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.314 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setdirlayout_latency": { 2024-04-30T08:45:36.314 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:36.314 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:36.314 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:36.314 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.314 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setfilelock_latency": { 2024-04-30T08:45:36.315 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:36.315 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:36.315 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:36.315 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.315 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setlayout_latency": { 2024-04-30T08:45:36.315 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:36.315 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:36.315 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:36.315 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.315 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setxattr_latency": { 2024-04-30T08:45:36.315 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:36.315 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:36.315 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:36.315 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.315 INFO:teuthology.orchestra.run.smithi189.stdout: "req_symlink_latency": { 2024-04-30T08:45:36.316 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:36.316 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:36.316 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:36.316 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.316 INFO:teuthology.orchestra.run.smithi189.stdout: "req_unlink_latency": { 2024-04-30T08:45:36.316 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 1, 2024-04-30T08:45:36.316 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.003127448, 2024-04-30T08:45:36.316 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.003127448 2024-04-30T08:45:36.316 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.316 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_revoke_eviction": 0, 2024-04-30T08:45:36.316 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_acquisition_throttle": 0, 2024-04-30T08:45:36.316 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getvxattr_latency": { 2024-04-30T08:45:36.316 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:36.316 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:36.316 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:36.317 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:36.317 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.317 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_sessions": { 2024-04-30T08:45:36.317 INFO:teuthology.orchestra.run.smithi189.stdout: "session_count": 1, 2024-04-30T08:45:36.317 INFO:teuthology.orchestra.run.smithi189.stdout: "session_add": 2, 2024-04-30T08:45:36.317 INFO:teuthology.orchestra.run.smithi189.stdout: "session_remove": 1, 2024-04-30T08:45:36.317 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_open": 1, 2024-04-30T08:45:36.317 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_stale": 0, 2024-04-30T08:45:36.317 INFO:teuthology.orchestra.run.smithi189.stdout: "total_load": 1, 2024-04-30T08:45:36.317 INFO:teuthology.orchestra.run.smithi189.stdout: "average_load": 1, 2024-04-30T08:45:36.317 INFO:teuthology.orchestra.run.smithi189.stdout: "avg_session_uptime": 613, 2024-04-30T08:45:36.317 INFO:teuthology.orchestra.run.smithi189.stdout: "mdthresh_evicted": 0 2024-04-30T08:45:36.317 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.317 INFO:teuthology.orchestra.run.smithi189.stdout: "mempool": { 2024-04-30T08:45:36.317 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_bytes": 0, 2024-04-30T08:45:36.318 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_items": 0, 2024-04-30T08:45:36.318 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_bytes": 0, 2024-04-30T08:45:36.318 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_items": 0, 2024-04-30T08:45:36.318 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_bytes": 0, 2024-04-30T08:45:36.318 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_items": 0, 2024-04-30T08:45:36.318 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-30T08:45:36.318 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_items": 0, 2024-04-30T08:45:36.318 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-30T08:45:36.318 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_items": 0, 2024-04-30T08:45:36.318 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_bytes": 0, 2024-04-30T08:45:36.318 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_items": 0, 2024-04-30T08:45:36.318 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-30T08:45:36.318 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_items": 0, 2024-04-30T08:45:36.318 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_bytes": 0, 2024-04-30T08:45:36.319 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_items": 0, 2024-04-30T08:45:36.319 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_bytes": 0, 2024-04-30T08:45:36.319 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_items": 0, 2024-04-30T08:45:36.319 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-30T08:45:36.319 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_items": 0, 2024-04-30T08:45:36.319 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-30T08:45:36.319 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_items": 0, 2024-04-30T08:45:36.319 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_bytes": 0, 2024-04-30T08:45:36.319 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_items": 0, 2024-04-30T08:45:36.319 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_bytes": 0, 2024-04-30T08:45:36.319 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_items": 0, 2024-04-30T08:45:36.319 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-30T08:45:36.319 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_items": 0, 2024-04-30T08:45:36.319 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_bytes": 0, 2024-04-30T08:45:36.319 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_items": 0, 2024-04-30T08:45:36.319 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_bytes": 0, 2024-04-30T08:45:36.320 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_items": 0, 2024-04-30T08:45:36.320 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_bytes": 0, 2024-04-30T08:45:36.320 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_items": 0, 2024-04-30T08:45:36.320 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_bytes": 0, 2024-04-30T08:45:36.320 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_items": 0, 2024-04-30T08:45:36.320 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_bytes": 275323, 2024-04-30T08:45:36.320 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_items": 94, 2024-04-30T08:45:36.320 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_bytes": 0, 2024-04-30T08:45:36.320 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_items": 0, 2024-04-30T08:45:36.320 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_bytes": 0, 2024-04-30T08:45:36.320 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_items": 0, 2024-04-30T08:45:36.320 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_bytes": 0, 2024-04-30T08:45:36.320 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_items": 0, 2024-04-30T08:45:36.320 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_bytes": 0, 2024-04-30T08:45:36.320 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_items": 0, 2024-04-30T08:45:36.320 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_bytes": 9088, 2024-04-30T08:45:36.321 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_items": 175, 2024-04-30T08:45:36.321 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_bytes": 0, 2024-04-30T08:45:36.321 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_items": 0, 2024-04-30T08:45:36.321 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_bytes": 0, 2024-04-30T08:45:36.321 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_items": 0, 2024-04-30T08:45:36.321 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_bytes": 77157, 2024-04-30T08:45:36.321 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_items": 318, 2024-04-30T08:45:36.321 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_bytes": 0, 2024-04-30T08:45:36.321 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_items": 0, 2024-04-30T08:45:36.321 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_bytes": 0, 2024-04-30T08:45:36.321 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_items": 0 2024-04-30T08:45:36.321 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.321 INFO:teuthology.orchestra.run.smithi189.stdout: "objecter": { 2024-04-30T08:45:36.321 INFO:teuthology.orchestra.run.smithi189.stdout: "op_active": 0, 2024-04-30T08:45:36.321 INFO:teuthology.orchestra.run.smithi189.stdout: "op_laggy": 0, 2024-04-30T08:45:36.322 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send": 71, 2024-04-30T08:45:36.322 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send_bytes": 37072, 2024-04-30T08:45:36.322 INFO:teuthology.orchestra.run.smithi189.stdout: "op_resend": 33, 2024-04-30T08:45:36.322 INFO:teuthology.orchestra.run.smithi189.stdout: "op_reply": 71, 2024-04-30T08:45:36.322 INFO:teuthology.orchestra.run.smithi189.stdout: "op_latency": { 2024-04-30T08:45:36.322 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 71, 2024-04-30T08:45:36.322 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 362.085574393, 2024-04-30T08:45:36.322 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 5.099796822 2024-04-30T08:45:36.322 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.322 INFO:teuthology.orchestra.run.smithi189.stdout: "op_inflight": 0, 2024-04-30T08:45:36.322 INFO:teuthology.orchestra.run.smithi189.stdout: "oplen_avg": { 2024-04-30T08:45:36.322 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 71, 2024-04-30T08:45:36.322 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 97 2024-04-30T08:45:36.322 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.322 INFO:teuthology.orchestra.run.smithi189.stdout: "op": 71, 2024-04-30T08:45:36.322 INFO:teuthology.orchestra.run.smithi189.stdout: "op_r": 3, 2024-04-30T08:45:36.323 INFO:teuthology.orchestra.run.smithi189.stdout: "op_w": 68, 2024-04-30T08:45:36.323 INFO:teuthology.orchestra.run.smithi189.stdout: "op_rmw": 0, 2024-04-30T08:45:36.323 INFO:teuthology.orchestra.run.smithi189.stdout: "op_pg": 0, 2024-04-30T08:45:36.323 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_stat": 0, 2024-04-30T08:45:36.323 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_create": 10, 2024-04-30T08:45:36.323 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_read": 1, 2024-04-30T08:45:36.323 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_write": 15, 2024-04-30T08:45:36.323 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writefull": 12, 2024-04-30T08:45:36.323 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writesame": 0, 2024-04-30T08:45:36.323 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_append": 0, 2024-04-30T08:45:36.323 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_zero": 0, 2024-04-30T08:45:36.323 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_truncate": 0, 2024-04-30T08:45:36.323 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_delete": 14, 2024-04-30T08:45:36.323 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_mapext": 0, 2024-04-30T08:45:36.323 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_sparse_read": 0, 2024-04-30T08:45:36.323 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_clonerange": 0, 2024-04-30T08:45:36.324 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_getxattr": 2, 2024-04-30T08:45:36.324 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_setxattr": 20, 2024-04-30T08:45:36.324 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_cmpxattr": 0, 2024-04-30T08:45:36.324 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_rmxattr": 0, 2024-04-30T08:45:36.324 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_resetxattrs": 0, 2024-04-30T08:45:36.324 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_call": 0, 2024-04-30T08:45:36.324 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_watch": 0, 2024-04-30T08:45:36.324 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_notify": 0, 2024-04-30T08:45:36.324 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_src_cmpxattr": 0, 2024-04-30T08:45:36.324 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls": 0, 2024-04-30T08:45:36.324 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls_filter": 0, 2024-04-30T08:45:36.324 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_other": 0, 2024-04-30T08:45:36.324 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_active": 0, 2024-04-30T08:45:36.324 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_send": 0, 2024-04-30T08:45:36.324 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_resend": 0, 2024-04-30T08:45:36.325 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_ping": 0, 2024-04-30T08:45:36.325 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_active": 0, 2024-04-30T08:45:36.325 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_send": 0, 2024-04-30T08:45:36.325 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_resend": 0, 2024-04-30T08:45:36.325 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_active": 0, 2024-04-30T08:45:36.325 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_send": 0, 2024-04-30T08:45:36.325 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_resend": 0, 2024-04-30T08:45:36.325 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_active": 0, 2024-04-30T08:45:36.325 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_send": 0, 2024-04-30T08:45:36.325 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_resend": 0, 2024-04-30T08:45:36.325 INFO:teuthology.orchestra.run.smithi189.stdout: "command_active": 0, 2024-04-30T08:45:36.325 INFO:teuthology.orchestra.run.smithi189.stdout: "command_send": 0, 2024-04-30T08:45:36.325 INFO:teuthology.orchestra.run.smithi189.stdout: "command_resend": 0, 2024-04-30T08:45:36.325 INFO:teuthology.orchestra.run.smithi189.stdout: "map_epoch": 85, 2024-04-30T08:45:36.325 INFO:teuthology.orchestra.run.smithi189.stdout: "map_full": 0, 2024-04-30T08:45:36.325 INFO:teuthology.orchestra.run.smithi189.stdout: "map_inc": 4, 2024-04-30T08:45:36.326 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_sessions": 10, 2024-04-30T08:45:36.326 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_open": 10, 2024-04-30T08:45:36.326 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_close": 0, 2024-04-30T08:45:36.326 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_laggy": 0, 2024-04-30T08:45:36.326 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_wr": 20, 2024-04-30T08:45:36.326 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_rd": 0, 2024-04-30T08:45:36.326 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_del": 3 2024-04-30T08:45:36.326 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.326 INFO:teuthology.orchestra.run.smithi189.stdout: "oft": { 2024-04-30T08:45:36.326 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_objs": 1, 2024-04-30T08:45:36.326 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_kv_pairs": 0, 2024-04-30T08:45:36.326 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_updates": 0, 2024-04-30T08:45:36.326 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_removes": 0 2024-04-30T08:45:36.326 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.326 INFO:teuthology.orchestra.run.smithi189.stdout: "osd-slow-ops": { 2024-04-30T08:45:36.327 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_ops_count": 0 2024-04-30T08:45:36.327 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.327 INFO:teuthology.orchestra.run.smithi189.stdout: "purge_queue": { 2024-04-30T08:45:36.327 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops": 0, 2024-04-30T08:45:36.327 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops_high_water": 2, 2024-04-30T08:45:36.327 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing": 0, 2024-04-30T08:45:36.327 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_high_water": 1, 2024-04-30T08:45:36.327 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed_ops": 2, 2024-04-30T08:45:36.327 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed": 1, 2024-04-30T08:45:36.327 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_item_in_journal": 0 2024-04-30T08:45:36.327 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.327 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-30T08:45:36.327 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-30T08:45:36.327 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-04-30T08:45:36.327 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-30T08:45:36.327 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 1509, 2024-04-30T08:45:36.328 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 1727480, 2024-04-30T08:45:36.328 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:36.328 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 1509, 2024-04-30T08:45:36.328 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-04-30T08:45:36.328 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-04-30T08:45:36.328 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 1509, 2024-04-30T08:45:36.328 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 1727480, 2024-04-30T08:45:36.328 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-30T08:45:36.328 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:36.328 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:36.328 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:36.328 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:36.328 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.328 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_bytes": { 2024-04-30T08:45:36.328 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-30T08:45:36.329 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-04-30T08:45:36.329 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-30T08:45:36.329 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-04-30T08:45:36.329 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-04-30T08:45:36.329 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:36.329 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-04-30T08:45:36.329 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 71, 2024-04-30T08:45:36.329 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 37181, 2024-04-30T08:45:36.329 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 71, 2024-04-30T08:45:36.329 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 37181, 2024-04-30T08:45:36.329 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-30T08:45:36.329 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:36.329 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:36.329 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:36.329 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:36.329 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.330 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_ops": { 2024-04-30T08:45:36.330 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-30T08:45:36.330 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 1024, 2024-04-30T08:45:36.330 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-30T08:45:36.330 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-04-30T08:45:36.330 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-04-30T08:45:36.330 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:36.330 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-04-30T08:45:36.330 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 71, 2024-04-30T08:45:36.330 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 71, 2024-04-30T08:45:36.330 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 71, 2024-04-30T08:45:36.330 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 71, 2024-04-30T08:45:36.330 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-30T08:45:36.330 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:36.330 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:36.331 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:36.331 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:36.331 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.331 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle": { 2024-04-30T08:45:36.331 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-30T08:45:36.331 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-04-30T08:45:36.331 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-30T08:45:36.331 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 1, 2024-04-30T08:45:36.331 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 109, 2024-04-30T08:45:36.331 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:36.331 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 1, 2024-04-30T08:45:36.331 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-04-30T08:45:36.331 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-04-30T08:45:36.331 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 1, 2024-04-30T08:45:36.331 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 109, 2024-04-30T08:45:36.332 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-30T08:45:36.332 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:36.332 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:36.332 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:36.332 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:36.332 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:36.332 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle-0x5634a616a4e0": { 2024-04-30T08:45:36.332 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-30T08:45:36.332 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-04-30T08:45:36.332 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-30T08:45:36.332 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 17, 2024-04-30T08:45:36.332 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 25023, 2024-04-30T08:45:36.332 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:36.332 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 17, 2024-04-30T08:45:36.332 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-04-30T08:45:36.332 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-04-30T08:45:36.333 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 14, 2024-04-30T08:45:36.333 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 25023, 2024-04-30T08:45:36.333 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-30T08:45:36.333 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:36.333 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:36.333 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:36.333 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:36.333 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:36.333 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-30T08:45:37.284 INFO:tasks.check_counter:Daemon mds.c mds.exported=1 2024-04-30T08:45:37.284 INFO:tasks.check_counter:Daemon mds.c mds.imported=1 2024-04-30T08:45:37.284 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=0 2024-04-30T08:45:37.284 DEBUG:tasks.check_counter:Getting stats from f 2024-04-30T08:45:37.284 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-30T08:45:37.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:37 smithi071 ceph-mon[33385]: pgmap v738: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 11 op/s 2024-04-30T08:45:37.534 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.c/config 2024-04-30T08:45:37.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:37 smithi189 ceph-mon[33038]: pgmap v738: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 11 op/s 2024-04-30T08:45:37.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:37 smithi032 ceph-mon[28331]: pgmap v738: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 85 B/s wr, 11 op/s 2024-04-30T08:45:39.473 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:39 smithi189 ceph-mon[33038]: pgmap v739: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:45:39.473 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-30T08:45:39.473 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-0": { 2024-04-30T08:45:39.473 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 2253, 2024-04-30T08:45:39.474 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 2255, 2024-04-30T08:45:39.474 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 638865, 2024-04-30T08:45:39.474 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 716461, 2024-04-30T08:45:39.474 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 6, 2024-04-30T08:45:39.474 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 2, 2024-04-30T08:45:39.474 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.203214379, 2024-04-30T08:45:39.474 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.059208670, 2024-04-30T08:45:39.474 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.053747169, 2024-04-30T08:45:39.474 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.076301104, 2024-04-30T08:45:39.474 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:39.474 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 2253, 2024-04-30T08:45:39.474 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.095436079, 2024-04-30T08:45:39.474 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000042359 2024-04-30T08:45:39.475 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.475 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:39.475 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.475 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.475 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.475 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.475 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-30T08:45:39.475 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 0 2024-04-30T08:45:39.475 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.475 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-1": { 2024-04-30T08:45:39.475 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 7, 2024-04-30T08:45:39.475 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 179, 2024-04-30T08:45:39.475 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 3708, 2024-04-30T08:45:39.475 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 454026, 2024-04-30T08:45:39.475 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 5, 2024-04-30T08:45:39.476 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 3, 2024-04-30T08:45:39.476 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.010749137, 2024-04-30T08:45:39.476 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.007198049, 2024-04-30T08:45:39.476 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.001976718, 2024-04-30T08:45:39.476 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.000076249, 2024-04-30T08:45:39.476 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:39.476 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 178, 2024-04-30T08:45:39.476 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.017040372, 2024-04-30T08:45:39.476 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000095732 2024-04-30T08:45:39.476 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.476 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:39.476 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.476 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.477 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.477 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.477 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 3152, 2024-04-30T08:45:39.477 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 453328 2024-04-30T08:45:39.477 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.477 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-2": { 2024-04-30T08:45:39.477 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 347, 2024-04-30T08:45:39.477 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 237, 2024-04-30T08:45:39.477 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 1531388, 2024-04-30T08:45:39.477 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 142382, 2024-04-30T08:45:39.477 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 3, 2024-04-30T08:45:39.477 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 2, 2024-04-30T08:45:39.477 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.044806250, 2024-04-30T08:45:39.477 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.013515342, 2024-04-30T08:45:39.478 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.031561918, 2024-04-30T08:45:39.478 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.006096656, 2024-04-30T08:45:39.478 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:39.478 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 236, 2024-04-30T08:45:39.478 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.016399947, 2024-04-30T08:45:39.478 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000069491 2024-04-30T08:45:39.478 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.478 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:39.478 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.478 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.478 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.478 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.478 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 1531120, 2024-04-30T08:45:39.479 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 142064 2024-04-30T08:45:39.479 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.479 INFO:teuthology.orchestra.run.smithi189.stdout: "cct": { 2024-04-30T08:45:39.479 INFO:teuthology.orchestra.run.smithi189.stdout: "total_workers": 1, 2024-04-30T08:45:39.479 INFO:teuthology.orchestra.run.smithi189.stdout: "unhealthy_workers": 0 2024-04-30T08:45:39.479 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.479 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-MDSRank": { 2024-04-30T08:45:39.479 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-04-30T08:45:39.479 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-04-30T08:45:39.479 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 3761, 2024-04-30T08:45:39.479 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.111777628, 2024-04-30T08:45:39.479 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000029720 2024-04-30T08:45:39.479 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:39.479 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.480 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-PurgeQueue": { 2024-04-30T08:45:39.480 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-04-30T08:45:39.480 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-04-30T08:45:39.480 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.480 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.480 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.480 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:39.480 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.480 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": { 2024-04-30T08:45:39.480 INFO:teuthology.orchestra.run.smithi189.stdout: "request": 0, 2024-04-30T08:45:39.480 INFO:teuthology.orchestra.run.smithi189.stdout: "reply": 0, 2024-04-30T08:45:39.480 INFO:teuthology.orchestra.run.smithi189.stdout: "reply_latency": { 2024-04-30T08:45:39.480 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.480 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.481 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.481 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.481 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_reply": 0, 2024-04-30T08:45:39.481 INFO:teuthology.orchestra.run.smithi189.stdout: "forward": 0, 2024-04-30T08:45:39.481 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_complete": 0, 2024-04-30T08:45:39.481 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_keys": 0, 2024-04-30T08:45:39.481 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_commit": 0, 2024-04-30T08:45:39.481 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_split": 0, 2024-04-30T08:45:39.481 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_merge": 0, 2024-04-30T08:45:39.481 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes": 0, 2024-04-30T08:45:39.481 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_top": 0, 2024-04-30T08:45:39.481 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_bottom": 0, 2024-04-30T08:45:39.481 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pin_tail": 0, 2024-04-30T08:45:39.481 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pinned": 0, 2024-04-30T08:45:39.482 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_expired": 0, 2024-04-30T08:45:39.482 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_with_caps": 0, 2024-04-30T08:45:39.482 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": 0, 2024-04-30T08:45:39.482 INFO:teuthology.orchestra.run.smithi189.stdout: "subtrees": 1, 2024-04-30T08:45:39.482 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse": 0, 2024-04-30T08:45:39.482 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_hit": 0, 2024-04-30T08:45:39.482 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_forward": 0, 2024-04-30T08:45:39.482 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_discover": 0, 2024-04-30T08:45:39.482 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_dir_fetch": 0, 2024-04-30T08:45:39.482 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_remote_ino": 0, 2024-04-30T08:45:39.482 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_lock": 0, 2024-04-30T08:45:39.482 INFO:teuthology.orchestra.run.smithi189.stdout: "load_cent": 0, 2024-04-30T08:45:39.482 INFO:teuthology.orchestra.run.smithi189.stdout: "q": 0, 2024-04-30T08:45:39.483 INFO:teuthology.orchestra.run.smithi189.stdout: "exported": 0, 2024-04-30T08:45:39.483 INFO:teuthology.orchestra.run.smithi189.stdout: "exported_inodes": 0, 2024-04-30T08:45:39.483 INFO:teuthology.orchestra.run.smithi189.stdout: "imported": 0, 2024-04-30T08:45:39.483 INFO:teuthology.orchestra.run.smithi189.stdout: "imported_inodes": 0, 2024-04-30T08:45:39.483 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_dir_fetch": 0, 2024-04-30T08:45:39.483 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_backtrace_fetch": 0, 2024-04-30T08:45:39.483 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_peer_discover": 0, 2024-04-30T08:45:39.483 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rfiles": 0, 2024-04-30T08:45:39.483 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rbytes": 0, 2024-04-30T08:45:39.483 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rsnaps": 0, 2024-04-30T08:45:39.483 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_fetch": 0, 2024-04-30T08:45:39.483 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_set_tag": 0, 2024-04-30T08:45:39.483 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_repaired": 0, 2024-04-30T08:45:39.483 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_inotable_repaired": 0, 2024-04-30T08:45:39.484 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_inodes": 0, 2024-04-30T08:45:39.484 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_base_inodes": 0, 2024-04-30T08:45:39.484 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dirfrag_rstats": 0, 2024-04-30T08:45:39.484 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_file_inodes": 0, 2024-04-30T08:45:39.484 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_inode_file_caps": 0, 2024-04-30T08:45:39.484 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_revoke": 0, 2024-04-30T08:45:39.484 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_grant": 0, 2024-04-30T08:45:39.484 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_trunc": 0, 2024-04-30T08:45:39.484 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-30T08:45:39.484 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-30T08:45:39.484 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps": 0, 2024-04-30T08:45:39.484 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps_dirty": 0, 2024-04-30T08:45:39.484 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_cap_release": 0, 2024-04-30T08:45:39.485 INFO:teuthology.orchestra.run.smithi189.stdout: "process_request_cap_release": 0 2024-04-30T08:45:39.485 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.485 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_cache": { 2024-04-30T08:45:39.485 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update": 0, 2024-04-30T08:45:39.485 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update_receipt": 0, 2024-04-30T08:45:39.485 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_try_discover": 0, 2024-04-30T08:45:39.485 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_send_discover": 0, 2024-04-30T08:45:39.485 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_handle_discover": 0, 2024-04-30T08:45:39.485 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays": 0, 2024-04-30T08:45:39.485 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_delayed": 0, 2024-04-30T08:45:39.485 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_enqueuing": 0, 2024-04-30T08:45:39.485 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_created": 0, 2024-04-30T08:45:39.485 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_enqueued": 0, 2024-04-30T08:45:39.485 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_reintegrated": 0, 2024-04-30T08:45:39.486 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_migrated": 0, 2024-04-30T08:45:39.486 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_processing": 0, 2024-04-30T08:45:39.486 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_enqueued": 0, 2024-04-30T08:45:39.486 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_prioritized": 0, 2024-04-30T08:45:39.486 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_started": 0, 2024-04-30T08:45:39.486 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_completed": 0, 2024-04-30T08:45:39.486 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_quiesce_path": 0, 2024-04-30T08:45:39.486 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_quiesce_inode": 0, 2024-04-30T08:45:39.486 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_enqueue_scrub": 0, 2024-04-30T08:45:39.486 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_exportdir": 0, 2024-04-30T08:45:39.486 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_flush": 0, 2024-04-30T08:45:39.486 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragmentdir": 0, 2024-04-30T08:45:39.486 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragstats": 0, 2024-04-30T08:45:39.487 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_inodestats": 0 2024-04-30T08:45:39.487 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.487 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_log": { 2024-04-30T08:45:39.487 INFO:teuthology.orchestra.run.smithi189.stdout: "evlrg": 0, 2024-04-30T08:45:39.487 INFO:teuthology.orchestra.run.smithi189.stdout: "evadd": 0, 2024-04-30T08:45:39.487 INFO:teuthology.orchestra.run.smithi189.stdout: "evex": 0, 2024-04-30T08:45:39.487 INFO:teuthology.orchestra.run.smithi189.stdout: "evtrm": 0, 2024-04-30T08:45:39.487 INFO:teuthology.orchestra.run.smithi189.stdout: "ev": 5, 2024-04-30T08:45:39.487 INFO:teuthology.orchestra.run.smithi189.stdout: "evexg": 0, 2024-04-30T08:45:39.487 INFO:teuthology.orchestra.run.smithi189.stdout: "evexd": 0, 2024-04-30T08:45:39.487 INFO:teuthology.orchestra.run.smithi189.stdout: "segadd": 0, 2024-04-30T08:45:39.487 INFO:teuthology.orchestra.run.smithi189.stdout: "segex": 0, 2024-04-30T08:45:39.487 INFO:teuthology.orchestra.run.smithi189.stdout: "segtrm": 0, 2024-04-30T08:45:39.487 INFO:teuthology.orchestra.run.smithi189.stdout: "seg": 2, 2024-04-30T08:45:39.487 INFO:teuthology.orchestra.run.smithi189.stdout: "segmjr": 2, 2024-04-30T08:45:39.488 INFO:teuthology.orchestra.run.smithi189.stdout: "segexg": 0, 2024-04-30T08:45:39.488 INFO:teuthology.orchestra.run.smithi189.stdout: "segexd": 0, 2024-04-30T08:45:39.488 INFO:teuthology.orchestra.run.smithi189.stdout: "expos": 4194304, 2024-04-30T08:45:39.488 INFO:teuthology.orchestra.run.smithi189.stdout: "wrpos": 4195734, 2024-04-30T08:45:39.488 INFO:teuthology.orchestra.run.smithi189.stdout: "rdpos": 4195424, 2024-04-30T08:45:39.488 INFO:teuthology.orchestra.run.smithi189.stdout: "jlat": { 2024-04-30T08:45:39.488 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.488 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.488 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.488 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.488 INFO:teuthology.orchestra.run.smithi189.stdout: "replayed": 5 2024-04-30T08:45:39.488 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.488 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_mem": { 2024-04-30T08:45:39.489 INFO:teuthology.orchestra.run.smithi189.stdout: "ino": 3, 2024-04-30T08:45:39.489 INFO:teuthology.orchestra.run.smithi189.stdout: "ino+": 3, 2024-04-30T08:45:39.489 INFO:teuthology.orchestra.run.smithi189.stdout: "ino-": 0, 2024-04-30T08:45:39.489 INFO:teuthology.orchestra.run.smithi189.stdout: "dir": 1, 2024-04-30T08:45:39.489 INFO:teuthology.orchestra.run.smithi189.stdout: "dir+": 1, 2024-04-30T08:45:39.489 INFO:teuthology.orchestra.run.smithi189.stdout: "dir-": 0, 2024-04-30T08:45:39.489 INFO:teuthology.orchestra.run.smithi189.stdout: "dn": 0, 2024-04-30T08:45:39.489 INFO:teuthology.orchestra.run.smithi189.stdout: "dn+": 0, 2024-04-30T08:45:39.489 INFO:teuthology.orchestra.run.smithi189.stdout: "dn-": 0, 2024-04-30T08:45:39.489 INFO:teuthology.orchestra.run.smithi189.stdout: "cap": 0, 2024-04-30T08:45:39.489 INFO:teuthology.orchestra.run.smithi189.stdout: "cap+": 0, 2024-04-30T08:45:39.489 INFO:teuthology.orchestra.run.smithi189.stdout: "cap-": 0, 2024-04-30T08:45:39.489 INFO:teuthology.orchestra.run.smithi189.stdout: "rss": 121472, 2024-04-30T08:45:39.489 INFO:teuthology.orchestra.run.smithi189.stdout: "heap": 215440 2024-04-30T08:45:39.489 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.490 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_server": { 2024-04-30T08:45:39.490 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_client_request": 0, 2024-04-30T08:45:39.490 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_server_request": 0, 2024-04-30T08:45:39.490 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_request": 0, 2024-04-30T08:45:39.490 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_session": 0, 2024-04-30T08:45:39.490 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_peer_request": 0, 2024-04-30T08:45:39.490 INFO:teuthology.orchestra.run.smithi189.stdout: "req_create_latency": { 2024-04-30T08:45:39.490 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.490 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.490 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.490 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.490 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getattr_latency": { 2024-04-30T08:45:39.490 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.490 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.491 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.491 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.491 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getfilelock_latency": { 2024-04-30T08:45:39.491 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.491 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.491 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.491 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.491 INFO:teuthology.orchestra.run.smithi189.stdout: "req_link_latency": { 2024-04-30T08:45:39.491 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.491 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.491 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.491 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.491 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookup_latency": { 2024-04-30T08:45:39.491 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.491 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.492 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.492 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.492 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookuphash_latency": { 2024-04-30T08:45:39.492 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.492 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.492 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.492 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.492 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupino_latency": { 2024-04-30T08:45:39.492 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.492 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.492 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.492 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.492 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupname_latency": { 2024-04-30T08:45:39.492 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.492 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.493 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.493 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.493 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupparent_latency": { 2024-04-30T08:45:39.493 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.493 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.493 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.493 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.493 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupsnap_latency": { 2024-04-30T08:45:39.493 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.493 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.493 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.493 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.493 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lssnap_latency": { 2024-04-30T08:45:39.493 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.493 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.493 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.494 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.494 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mkdir_latency": { 2024-04-30T08:45:39.494 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.494 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.494 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.494 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.494 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mknod_latency": { 2024-04-30T08:45:39.495 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.495 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.495 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.495 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.495 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mksnap_latency": { 2024-04-30T08:45:39.495 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.495 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.495 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.495 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.495 INFO:teuthology.orchestra.run.smithi189.stdout: "req_open_latency": { 2024-04-30T08:45:39.495 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.495 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.495 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.495 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.496 INFO:teuthology.orchestra.run.smithi189.stdout: "req_readdir_latency": { 2024-04-30T08:45:39.496 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.496 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.496 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.496 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.496 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rename_latency": { 2024-04-30T08:45:39.496 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.496 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.496 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.496 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.496 INFO:teuthology.orchestra.run.smithi189.stdout: "req_renamesnap_latency": { 2024-04-30T08:45:39.496 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.496 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.496 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.497 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.497 INFO:teuthology.orchestra.run.smithi189.stdout: "req_snapdiff_latency": { 2024-04-30T08:45:39.497 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.497 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.497 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.497 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.497 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmdir_latency": { 2024-04-30T08:45:39.497 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.497 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.497 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.497 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.497 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmsnap_latency": { 2024-04-30T08:45:39.497 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.497 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.497 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.498 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.498 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmxattr_latency": { 2024-04-30T08:45:39.498 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.498 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.498 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.498 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.498 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setattr_latency": { 2024-04-30T08:45:39.498 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.498 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.498 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.498 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.498 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setdirlayout_latency": { 2024-04-30T08:45:39.498 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.499 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.499 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.499 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.499 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setfilelock_latency": { 2024-04-30T08:45:39.499 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.499 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.499 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.499 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.499 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setlayout_latency": { 2024-04-30T08:45:39.499 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.499 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.499 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.499 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.500 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setxattr_latency": { 2024-04-30T08:45:39.500 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.500 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.500 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.500 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.500 INFO:teuthology.orchestra.run.smithi189.stdout: "req_symlink_latency": { 2024-04-30T08:45:39.500 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.500 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.500 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.500 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.500 INFO:teuthology.orchestra.run.smithi189.stdout: "req_unlink_latency": { 2024-04-30T08:45:39.500 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.500 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.500 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.501 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.501 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_revoke_eviction": 0, 2024-04-30T08:45:39.501 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_acquisition_throttle": 0, 2024-04-30T08:45:39.501 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getvxattr_latency": { 2024-04-30T08:45:39.501 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.501 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.501 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.501 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:39.501 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.501 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_sessions": { 2024-04-30T08:45:39.501 INFO:teuthology.orchestra.run.smithi189.stdout: "session_count": 1, 2024-04-30T08:45:39.501 INFO:teuthology.orchestra.run.smithi189.stdout: "session_add": 2, 2024-04-30T08:45:39.501 INFO:teuthology.orchestra.run.smithi189.stdout: "session_remove": 1, 2024-04-30T08:45:39.502 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_open": 1, 2024-04-30T08:45:39.502 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_stale": 0, 2024-04-30T08:45:39.502 INFO:teuthology.orchestra.run.smithi189.stdout: "total_load": 0, 2024-04-30T08:45:39.502 INFO:teuthology.orchestra.run.smithi189.stdout: "average_load": 0, 2024-04-30T08:45:39.502 INFO:teuthology.orchestra.run.smithi189.stdout: "avg_session_uptime": 2244, 2024-04-30T08:45:39.502 INFO:teuthology.orchestra.run.smithi189.stdout: "mdthresh_evicted": 0 2024-04-30T08:45:39.502 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.502 INFO:teuthology.orchestra.run.smithi189.stdout: "mempool": { 2024-04-30T08:45:39.502 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_bytes": 0, 2024-04-30T08:45:39.502 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_items": 0, 2024-04-30T08:45:39.502 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_bytes": 0, 2024-04-30T08:45:39.502 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_items": 0, 2024-04-30T08:45:39.502 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_bytes": 0, 2024-04-30T08:45:39.503 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_items": 0, 2024-04-30T08:45:39.503 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-30T08:45:39.503 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_items": 0, 2024-04-30T08:45:39.503 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-30T08:45:39.503 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_items": 0, 2024-04-30T08:45:39.503 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_bytes": 0, 2024-04-30T08:45:39.503 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_items": 0, 2024-04-30T08:45:39.503 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-30T08:45:39.503 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_items": 0, 2024-04-30T08:45:39.503 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_bytes": 0, 2024-04-30T08:45:39.503 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_items": 0, 2024-04-30T08:45:39.503 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_bytes": 0, 2024-04-30T08:45:39.503 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_items": 0, 2024-04-30T08:45:39.503 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-30T08:45:39.504 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_items": 0, 2024-04-30T08:45:39.504 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-30T08:45:39.504 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_items": 0, 2024-04-30T08:45:39.504 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_bytes": 0, 2024-04-30T08:45:39.504 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_items": 0, 2024-04-30T08:45:39.504 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_bytes": 0, 2024-04-30T08:45:39.504 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_items": 0, 2024-04-30T08:45:39.504 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-30T08:45:39.504 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_items": 0, 2024-04-30T08:45:39.504 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_bytes": 0, 2024-04-30T08:45:39.504 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_items": 0, 2024-04-30T08:45:39.504 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_bytes": 0, 2024-04-30T08:45:39.504 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_items": 0, 2024-04-30T08:45:39.504 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_bytes": 0, 2024-04-30T08:45:39.505 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_items": 0, 2024-04-30T08:45:39.505 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_bytes": 0, 2024-04-30T08:45:39.505 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_items": 0, 2024-04-30T08:45:39.505 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_bytes": 181321, 2024-04-30T08:45:39.505 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_items": 47, 2024-04-30T08:45:39.505 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_bytes": 0, 2024-04-30T08:45:39.505 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_items": 0, 2024-04-30T08:45:39.505 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_bytes": 0, 2024-04-30T08:45:39.505 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_items": 0, 2024-04-30T08:45:39.505 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_bytes": 0, 2024-04-30T08:45:39.505 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_items": 0, 2024-04-30T08:45:39.505 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_bytes": 0, 2024-04-30T08:45:39.505 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_items": 0, 2024-04-30T08:45:39.505 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_bytes": 9088, 2024-04-30T08:45:39.506 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_items": 175, 2024-04-30T08:45:39.506 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_bytes": 0, 2024-04-30T08:45:39.506 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_items": 0, 2024-04-30T08:45:39.506 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_bytes": 0, 2024-04-30T08:45:39.506 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_items": 0, 2024-04-30T08:45:39.506 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_bytes": 8512, 2024-04-30T08:45:39.506 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_items": 13, 2024-04-30T08:45:39.506 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_bytes": 0, 2024-04-30T08:45:39.506 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_items": 0, 2024-04-30T08:45:39.506 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_bytes": 0, 2024-04-30T08:45:39.506 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_items": 0 2024-04-30T08:45:39.506 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.506 INFO:teuthology.orchestra.run.smithi189.stdout: "objecter": { 2024-04-30T08:45:39.506 INFO:teuthology.orchestra.run.smithi189.stdout: "op_active": 0, 2024-04-30T08:45:39.507 INFO:teuthology.orchestra.run.smithi189.stdout: "op_laggy": 0, 2024-04-30T08:45:39.507 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send": 2256, 2024-04-30T08:45:39.507 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send_bytes": 21, 2024-04-30T08:45:39.507 INFO:teuthology.orchestra.run.smithi189.stdout: "op_resend": 3, 2024-04-30T08:45:39.507 INFO:teuthology.orchestra.run.smithi189.stdout: "op_reply": 2256, 2024-04-30T08:45:39.507 INFO:teuthology.orchestra.run.smithi189.stdout: "op_latency": { 2024-04-30T08:45:39.507 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 2256, 2024-04-30T08:45:39.507 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 15.963673658, 2024-04-30T08:45:39.507 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.007076096 2024-04-30T08:45:39.507 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.507 INFO:teuthology.orchestra.run.smithi189.stdout: "op_inflight": 0, 2024-04-30T08:45:39.507 INFO:teuthology.orchestra.run.smithi189.stdout: "oplen_avg": { 2024-04-30T08:45:39.507 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 2256, 2024-04-30T08:45:39.508 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 2257 2024-04-30T08:45:39.508 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.508 INFO:teuthology.orchestra.run.smithi189.stdout: "op": 2256, 2024-04-30T08:45:39.508 INFO:teuthology.orchestra.run.smithi189.stdout: "op_r": 2256, 2024-04-30T08:45:39.508 INFO:teuthology.orchestra.run.smithi189.stdout: "op_w": 0, 2024-04-30T08:45:39.508 INFO:teuthology.orchestra.run.smithi189.stdout: "op_rmw": 0, 2024-04-30T08:45:39.508 INFO:teuthology.orchestra.run.smithi189.stdout: "op_pg": 0, 2024-04-30T08:45:39.508 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_stat": 1469, 2024-04-30T08:45:39.508 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_create": 0, 2024-04-30T08:45:39.508 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_read": 785, 2024-04-30T08:45:39.508 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_write": 0, 2024-04-30T08:45:39.508 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writefull": 0, 2024-04-30T08:45:39.508 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writesame": 0, 2024-04-30T08:45:39.509 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_append": 0, 2024-04-30T08:45:39.509 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_zero": 0, 2024-04-30T08:45:39.509 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_truncate": 0, 2024-04-30T08:45:39.509 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_delete": 0, 2024-04-30T08:45:39.509 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_mapext": 0, 2024-04-30T08:45:39.509 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_sparse_read": 0, 2024-04-30T08:45:39.509 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_clonerange": 0, 2024-04-30T08:45:39.509 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_getxattr": 1, 2024-04-30T08:45:39.509 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_setxattr": 0, 2024-04-30T08:45:39.509 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_cmpxattr": 0, 2024-04-30T08:45:39.509 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_rmxattr": 0, 2024-04-30T08:45:39.509 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_resetxattrs": 0, 2024-04-30T08:45:39.509 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_call": 0, 2024-04-30T08:45:39.509 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_watch": 0, 2024-04-30T08:45:39.509 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_notify": 0, 2024-04-30T08:45:39.510 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_src_cmpxattr": 0, 2024-04-30T08:45:39.510 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls": 0, 2024-04-30T08:45:39.510 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls_filter": 0, 2024-04-30T08:45:39.510 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_other": 0, 2024-04-30T08:45:39.510 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_active": 0, 2024-04-30T08:45:39.510 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_send": 0, 2024-04-30T08:45:39.510 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_resend": 0, 2024-04-30T08:45:39.510 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_ping": 0, 2024-04-30T08:45:39.510 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_active": 0, 2024-04-30T08:45:39.510 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_send": 0, 2024-04-30T08:45:39.510 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_resend": 0, 2024-04-30T08:45:39.510 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_active": 0, 2024-04-30T08:45:39.510 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_send": 0, 2024-04-30T08:45:39.511 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_resend": 0, 2024-04-30T08:45:39.511 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_active": 0, 2024-04-30T08:45:39.511 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_send": 0, 2024-04-30T08:45:39.511 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_resend": 0, 2024-04-30T08:45:39.511 INFO:teuthology.orchestra.run.smithi189.stdout: "command_active": 0, 2024-04-30T08:45:39.511 INFO:teuthology.orchestra.run.smithi189.stdout: "command_send": 0, 2024-04-30T08:45:39.511 INFO:teuthology.orchestra.run.smithi189.stdout: "command_resend": 0, 2024-04-30T08:45:39.511 INFO:teuthology.orchestra.run.smithi189.stdout: "map_epoch": 85, 2024-04-30T08:45:39.511 INFO:teuthology.orchestra.run.smithi189.stdout: "map_full": 0, 2024-04-30T08:45:39.511 INFO:teuthology.orchestra.run.smithi189.stdout: "map_inc": 3, 2024-04-30T08:45:39.511 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_sessions": 4, 2024-04-30T08:45:39.511 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_open": 4, 2024-04-30T08:45:39.511 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_close": 0, 2024-04-30T08:45:39.511 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_laggy": 0, 2024-04-30T08:45:39.512 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_wr": 0, 2024-04-30T08:45:39.512 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_rd": 2, 2024-04-30T08:45:39.512 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_del": 0 2024-04-30T08:45:39.512 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.512 INFO:teuthology.orchestra.run.smithi189.stdout: "oft": { 2024-04-30T08:45:39.512 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_objs": 0, 2024-04-30T08:45:39.512 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_kv_pairs": 0, 2024-04-30T08:45:39.512 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_updates": 0, 2024-04-30T08:45:39.512 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_removes": 0 2024-04-30T08:45:39.512 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.512 INFO:teuthology.orchestra.run.smithi189.stdout: "osd-slow-ops": { 2024-04-30T08:45:39.512 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_ops_count": 0 2024-04-30T08:45:39.512 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.512 INFO:teuthology.orchestra.run.smithi189.stdout: "purge_queue": { 2024-04-30T08:45:39.513 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops": 0, 2024-04-30T08:45:39.513 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops_high_water": 0, 2024-04-30T08:45:39.513 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing": 0, 2024-04-30T08:45:39.513 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_high_water": 0, 2024-04-30T08:45:39.513 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed_ops": 0, 2024-04-30T08:45:39.513 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed": 0, 2024-04-30T08:45:39.513 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_item_in_journal": 0 2024-04-30T08:45:39.513 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.513 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-30T08:45:39.513 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-30T08:45:39.513 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-04-30T08:45:39.513 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-30T08:45:39.513 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 2603, 2024-04-30T08:45:39.513 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 1919987, 2024-04-30T08:45:39.514 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:39.514 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 2603, 2024-04-30T08:45:39.514 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-04-30T08:45:39.514 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-04-30T08:45:39.514 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 2603, 2024-04-30T08:45:39.514 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 1919987, 2024-04-30T08:45:39.514 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-30T08:45:39.514 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.514 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.514 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.514 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:39.514 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.514 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_bytes": { 2024-04-30T08:45:39.514 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-30T08:45:39.515 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-04-30T08:45:39.515 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-30T08:45:39.515 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-04-30T08:45:39.515 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-04-30T08:45:39.515 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:39.515 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-04-30T08:45:39.515 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 2256, 2024-04-30T08:45:39.515 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 1435, 2024-04-30T08:45:39.515 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 2256, 2024-04-30T08:45:39.515 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 1435, 2024-04-30T08:45:39.515 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-30T08:45:39.515 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.515 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.515 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.516 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:39.516 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.516 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_ops": { 2024-04-30T08:45:39.516 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-30T08:45:39.516 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 1024, 2024-04-30T08:45:39.516 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-30T08:45:39.516 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-04-30T08:45:39.516 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-04-30T08:45:39.516 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:39.516 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-04-30T08:45:39.516 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 2256, 2024-04-30T08:45:39.516 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 2256, 2024-04-30T08:45:39.516 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 2256, 2024-04-30T08:45:39.516 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 2256, 2024-04-30T08:45:39.517 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-30T08:45:39.517 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.517 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.517 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.517 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:39.517 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.517 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle": { 2024-04-30T08:45:39.517 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-30T08:45:39.517 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-04-30T08:45:39.517 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-30T08:45:39.517 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-04-30T08:45:39.517 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-04-30T08:45:39.517 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:39.517 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-04-30T08:45:39.518 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-04-30T08:45:39.518 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-04-30T08:45:39.518 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 0, 2024-04-30T08:45:39.518 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 0, 2024-04-30T08:45:39.518 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-30T08:45:39.518 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.518 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.518 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.518 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:39.518 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:39.518 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle-0x5561aa592410": { 2024-04-30T08:45:39.518 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-30T08:45:39.518 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-04-30T08:45:39.518 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-30T08:45:39.518 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-04-30T08:45:39.519 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-04-30T08:45:39.519 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:39.519 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-04-30T08:45:39.519 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-04-30T08:45:39.519 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-04-30T08:45:39.519 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 0, 2024-04-30T08:45:39.519 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 0, 2024-04-30T08:45:39.519 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-30T08:45:39.519 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:39.519 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:39.519 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:39.519 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:39.519 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:39.519 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-30T08:45:39.520 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:39 smithi071 ceph-mon[33385]: pgmap v739: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:45:39.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:39 smithi032 ceph-mon[28331]: pgmap v739: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:45:40.278 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-04-30T08:45:40.278 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-04-30T08:45:40.278 INFO:tasks.check_counter:Daemon mds.f mds.dir_split=0 2024-04-30T08:45:40.278 DEBUG:tasks.check_counter:Getting stats from i 2024-04-30T08:45:40.278 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-30T08:45:40.522 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.c/config 2024-04-30T08:45:41.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:41 smithi071 ceph-mon[33385]: pgmap v740: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-30T08:45:41.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:41 smithi189 ceph-mon[33038]: pgmap v740: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-30T08:45:41.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:41 smithi032 ceph-mon[28331]: pgmap v740: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 10 op/s 2024-04-30T08:45:42.274 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-30T08:45:42.274 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-0": { 2024-04-30T08:45:42.274 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 445, 2024-04-30T08:45:42.274 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 334, 2024-04-30T08:45:42.274 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 1571787, 2024-04-30T08:45:42.274 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 239409, 2024-04-30T08:45:42.275 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 13, 2024-04-30T08:45:42.275 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 2, 2024-04-30T08:45:42.275 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.059274346, 2024-04-30T08:45:42.275 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.017497475, 2024-04-30T08:45:42.275 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.038895533, 2024-04-30T08:45:42.275 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.006608276, 2024-04-30T08:45:42.275 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:42.275 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 333, 2024-04-30T08:45:42.275 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.028067010, 2024-04-30T08:45:42.275 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000084285 2024-04-30T08:45:42.275 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.275 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:42.275 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 39, 2024-04-30T08:45:42.276 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000022973, 2024-04-30T08:45:42.276 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000589 2024-04-30T08:45:42.276 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.276 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 1556960, 2024-04-30T08:45:42.276 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 148976 2024-04-30T08:45:42.276 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.276 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-1": { 2024-04-30T08:45:42.276 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 43, 2024-04-30T08:45:42.276 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 216, 2024-04-30T08:45:42.276 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 10807, 2024-04-30T08:45:42.276 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 548015, 2024-04-30T08:45:42.276 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 15, 2024-04-30T08:45:42.276 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 9, 2024-04-30T08:45:42.277 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.016980661, 2024-04-30T08:45:42.277 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.008538992, 2024-04-30T08:45:42.277 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.005918850, 2024-04-30T08:45:42.277 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.000757711, 2024-04-30T08:45:42.277 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:42.277 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 213, 2024-04-30T08:45:42.278 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.022129216, 2024-04-30T08:45:42.278 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000103893 2024-04-30T08:45:42.278 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.278 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:42.278 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 21, 2024-04-30T08:45:42.278 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000003603, 2024-04-30T08:45:42.278 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000171 2024-04-30T08:45:42.278 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.278 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 2496, 2024-04-30T08:45:42.278 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 457776 2024-04-30T08:45:42.278 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.278 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-2": { 2024-04-30T08:45:42.279 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 1036, 2024-04-30T08:45:42.279 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 1088, 2024-04-30T08:45:42.279 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 317625, 2024-04-30T08:45:42.279 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 410785, 2024-04-30T08:45:42.279 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 10, 2024-04-30T08:45:42.279 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 4, 2024-04-30T08:45:42.279 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.101103976, 2024-04-30T08:45:42.279 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.036385262, 2024-04-30T08:45:42.279 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.065980903, 2024-04-30T08:45:42.279 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.005831933, 2024-04-30T08:45:42.279 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:42.279 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 1087, 2024-04-30T08:45:42.279 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.062137756, 2024-04-30T08:45:42.279 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000057164 2024-04-30T08:45:42.280 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.280 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:42.280 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 1933, 2024-04-30T08:45:42.280 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000551873, 2024-04-30T08:45:42.280 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000285 2024-04-30T08:45:42.280 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.280 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-04-30T08:45:42.280 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 448 2024-04-30T08:45:42.280 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.280 INFO:teuthology.orchestra.run.smithi189.stdout: "cct": { 2024-04-30T08:45:42.280 INFO:teuthology.orchestra.run.smithi189.stdout: "total_workers": 1, 2024-04-30T08:45:42.280 INFO:teuthology.orchestra.run.smithi189.stdout: "unhealthy_workers": 0 2024-04-30T08:45:42.280 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.281 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-MDSRank": { 2024-04-30T08:45:42.281 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-04-30T08:45:42.281 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-04-30T08:45:42.281 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 92, 2024-04-30T08:45:42.281 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.005169699, 2024-04-30T08:45:42.281 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000056192 2024-04-30T08:45:42.281 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:42.281 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.281 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-PurgeQueue": { 2024-04-30T08:45:42.281 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-04-30T08:45:42.281 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-04-30T08:45:42.281 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 16, 2024-04-30T08:45:42.281 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000328120, 2024-04-30T08:45:42.281 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000020507 2024-04-30T08:45:42.282 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:42.282 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.282 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": { 2024-04-30T08:45:42.282 INFO:teuthology.orchestra.run.smithi189.stdout: "request": 15, 2024-04-30T08:45:42.282 INFO:teuthology.orchestra.run.smithi189.stdout: "reply": 6, 2024-04-30T08:45:42.282 INFO:teuthology.orchestra.run.smithi189.stdout: "reply_latency": { 2024-04-30T08:45:42.282 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 6, 2024-04-30T08:45:42.282 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.005678192, 2024-04-30T08:45:42.282 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000946365 2024-04-30T08:45:42.282 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.282 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_reply": 0, 2024-04-30T08:45:42.282 INFO:teuthology.orchestra.run.smithi189.stdout: "forward": 9, 2024-04-30T08:45:42.282 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_complete": 0, 2024-04-30T08:45:42.283 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_keys": 0, 2024-04-30T08:45:42.283 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_commit": 11, 2024-04-30T08:45:42.283 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_split": 0, 2024-04-30T08:45:42.283 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_merge": 0, 2024-04-30T08:45:42.283 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes": 15, 2024-04-30T08:45:42.283 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_top": 0, 2024-04-30T08:45:42.283 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_bottom": 15, 2024-04-30T08:45:42.283 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pin_tail": 0, 2024-04-30T08:45:42.283 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pinned": 14, 2024-04-30T08:45:42.283 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_expired": 12, 2024-04-30T08:45:42.283 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_with_caps": 1, 2024-04-30T08:45:42.283 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": 1, 2024-04-30T08:45:42.283 INFO:teuthology.orchestra.run.smithi189.stdout: "subtrees": 3, 2024-04-30T08:45:42.283 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse": 23, 2024-04-30T08:45:42.284 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_hit": 10, 2024-04-30T08:45:42.284 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_forward": 7, 2024-04-30T08:45:42.284 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_discover": 2, 2024-04-30T08:45:42.284 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_dir_fetch": 0, 2024-04-30T08:45:42.284 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_remote_ino": 0, 2024-04-30T08:45:42.284 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_lock": 0, 2024-04-30T08:45:42.284 INFO:teuthology.orchestra.run.smithi189.stdout: "load_cent": 0, 2024-04-30T08:45:42.284 INFO:teuthology.orchestra.run.smithi189.stdout: "q": 0, 2024-04-30T08:45:42.284 INFO:teuthology.orchestra.run.smithi189.stdout: "exported": 1, 2024-04-30T08:45:42.284 INFO:teuthology.orchestra.run.smithi189.stdout: "exported_inodes": 0, 2024-04-30T08:45:42.284 INFO:teuthology.orchestra.run.smithi189.stdout: "imported": 1, 2024-04-30T08:45:42.284 INFO:teuthology.orchestra.run.smithi189.stdout: "imported_inodes": 2, 2024-04-30T08:45:42.284 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_dir_fetch": 0, 2024-04-30T08:45:42.284 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_backtrace_fetch": 2, 2024-04-30T08:45:42.285 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_peer_discover": 0, 2024-04-30T08:45:42.285 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rfiles": 325, 2024-04-30T08:45:42.285 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rbytes": 13424230511, 2024-04-30T08:45:42.285 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rsnaps": 0, 2024-04-30T08:45:42.285 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_fetch": 2, 2024-04-30T08:45:42.285 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_set_tag": 0, 2024-04-30T08:45:42.285 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_repaired": 0, 2024-04-30T08:45:42.285 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_inotable_repaired": 0, 2024-04-30T08:45:42.285 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_inodes": 0, 2024-04-30T08:45:42.285 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_base_inodes": 0, 2024-04-30T08:45:42.285 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dirfrag_rstats": 0, 2024-04-30T08:45:42.285 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_file_inodes": 2, 2024-04-30T08:45:42.285 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_inode_file_caps": 0, 2024-04-30T08:45:42.285 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_revoke": 3, 2024-04-30T08:45:42.286 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_grant": 5, 2024-04-30T08:45:42.286 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_trunc": 0, 2024-04-30T08:45:42.286 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-30T08:45:42.286 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-30T08:45:42.286 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps": 5, 2024-04-30T08:45:42.286 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps_dirty": 0, 2024-04-30T08:45:42.286 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_cap_release": 1, 2024-04-30T08:45:42.286 INFO:teuthology.orchestra.run.smithi189.stdout: "process_request_cap_release": 3 2024-04-30T08:45:42.286 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.286 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_cache": { 2024-04-30T08:45:42.286 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update": 0, 2024-04-30T08:45:42.286 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update_receipt": 0, 2024-04-30T08:45:42.286 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_try_discover": 0, 2024-04-30T08:45:42.286 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_send_discover": 3, 2024-04-30T08:45:42.287 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_handle_discover": 0, 2024-04-30T08:45:42.287 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays": 0, 2024-04-30T08:45:42.287 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_delayed": 0, 2024-04-30T08:45:42.287 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_enqueuing": 0, 2024-04-30T08:45:42.287 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_created": 1, 2024-04-30T08:45:42.287 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_enqueued": 1, 2024-04-30T08:45:42.287 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_reintegrated": 0, 2024-04-30T08:45:42.287 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_migrated": 0, 2024-04-30T08:45:42.287 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_processing": 0, 2024-04-30T08:45:42.287 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_enqueued": 0, 2024-04-30T08:45:42.287 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_prioritized": 0, 2024-04-30T08:45:42.287 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_started": 0, 2024-04-30T08:45:42.287 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_completed": 0, 2024-04-30T08:45:42.287 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_quiesce_path": 0, 2024-04-30T08:45:42.288 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_quiesce_inode": 0, 2024-04-30T08:45:42.288 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_enqueue_scrub": 0, 2024-04-30T08:45:42.288 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_exportdir": 1, 2024-04-30T08:45:42.288 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_flush": 0, 2024-04-30T08:45:42.288 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragmentdir": 0, 2024-04-30T08:45:42.288 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragstats": 0, 2024-04-30T08:45:42.288 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_inodestats": 0 2024-04-30T08:45:42.288 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.288 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_log": { 2024-04-30T08:45:42.288 INFO:teuthology.orchestra.run.smithi189.stdout: "evlrg": 0, 2024-04-30T08:45:42.288 INFO:teuthology.orchestra.run.smithi189.stdout: "evadd": 19, 2024-04-30T08:45:42.288 INFO:teuthology.orchestra.run.smithi189.stdout: "evex": 0, 2024-04-30T08:45:42.288 INFO:teuthology.orchestra.run.smithi189.stdout: "evtrm": 0, 2024-04-30T08:45:42.288 INFO:teuthology.orchestra.run.smithi189.stdout: "ev": 19, 2024-04-30T08:45:42.288 INFO:teuthology.orchestra.run.smithi189.stdout: "evexg": 0, 2024-04-30T08:45:42.289 INFO:teuthology.orchestra.run.smithi189.stdout: "evexd": 0, 2024-04-30T08:45:42.289 INFO:teuthology.orchestra.run.smithi189.stdout: "segadd": 2, 2024-04-30T08:45:42.289 INFO:teuthology.orchestra.run.smithi189.stdout: "segex": 0, 2024-04-30T08:45:42.289 INFO:teuthology.orchestra.run.smithi189.stdout: "segtrm": 0, 2024-04-30T08:45:42.289 INFO:teuthology.orchestra.run.smithi189.stdout: "seg": 2, 2024-04-30T08:45:42.289 INFO:teuthology.orchestra.run.smithi189.stdout: "segmjr": 2, 2024-04-30T08:45:42.289 INFO:teuthology.orchestra.run.smithi189.stdout: "segexg": 0, 2024-04-30T08:45:42.289 INFO:teuthology.orchestra.run.smithi189.stdout: "segexd": 0, 2024-04-30T08:45:42.289 INFO:teuthology.orchestra.run.smithi189.stdout: "expos": 4194304, 2024-04-30T08:45:42.289 INFO:teuthology.orchestra.run.smithi189.stdout: "wrpos": 4219859, 2024-04-30T08:45:42.289 INFO:teuthology.orchestra.run.smithi189.stdout: "rdpos": 0, 2024-04-30T08:45:42.289 INFO:teuthology.orchestra.run.smithi189.stdout: "jlat": { 2024-04-30T08:45:42.289 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 15, 2024-04-30T08:45:42.289 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.034197871, 2024-04-30T08:45:42.290 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.002279858 2024-04-30T08:45:42.290 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.290 INFO:teuthology.orchestra.run.smithi189.stdout: "replayed": 0 2024-04-30T08:45:42.290 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.290 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_mem": { 2024-04-30T08:45:42.290 INFO:teuthology.orchestra.run.smithi189.stdout: "ino": 19, 2024-04-30T08:45:42.290 INFO:teuthology.orchestra.run.smithi189.stdout: "ino+": 26, 2024-04-30T08:45:42.290 INFO:teuthology.orchestra.run.smithi189.stdout: "ino-": 7, 2024-04-30T08:45:42.290 INFO:teuthology.orchestra.run.smithi189.stdout: "dir": 18, 2024-04-30T08:45:42.290 INFO:teuthology.orchestra.run.smithi189.stdout: "dir+": 25, 2024-04-30T08:45:42.290 INFO:teuthology.orchestra.run.smithi189.stdout: "dir-": 7, 2024-04-30T08:45:42.290 INFO:teuthology.orchestra.run.smithi189.stdout: "dn": 15, 2024-04-30T08:45:42.290 INFO:teuthology.orchestra.run.smithi189.stdout: "dn+": 30, 2024-04-30T08:45:42.291 INFO:teuthology.orchestra.run.smithi189.stdout: "dn-": 15, 2024-04-30T08:45:42.291 INFO:teuthology.orchestra.run.smithi189.stdout: "cap": 1, 2024-04-30T08:45:42.291 INFO:teuthology.orchestra.run.smithi189.stdout: "cap+": 6, 2024-04-30T08:45:42.291 INFO:teuthology.orchestra.run.smithi189.stdout: "cap-": 5, 2024-04-30T08:45:42.291 INFO:teuthology.orchestra.run.smithi189.stdout: "rss": 124956, 2024-04-30T08:45:42.291 INFO:teuthology.orchestra.run.smithi189.stdout: "heap": 223632 2024-04-30T08:45:42.291 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.291 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_server": { 2024-04-30T08:45:42.291 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_client_request": 22, 2024-04-30T08:45:42.291 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_server_request": 1, 2024-04-30T08:45:42.291 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_request": 15, 2024-04-30T08:45:42.291 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_session": 17, 2024-04-30T08:45:42.291 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_peer_request": 5, 2024-04-30T08:45:42.291 INFO:teuthology.orchestra.run.smithi189.stdout: "req_create_latency": { 2024-04-30T08:45:42.292 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:42.292 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:42.292 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:42.292 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.292 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getattr_latency": { 2024-04-30T08:45:42.292 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 2, 2024-04-30T08:45:42.292 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.002624070, 2024-04-30T08:45:42.292 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.001312035 2024-04-30T08:45:42.292 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.292 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getfilelock_latency": { 2024-04-30T08:45:42.292 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:42.293 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:42.293 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:42.293 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.293 INFO:teuthology.orchestra.run.smithi189.stdout: "req_link_latency": { 2024-04-30T08:45:42.293 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:42.293 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:42.293 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:42.293 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.293 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookup_latency": { 2024-04-30T08:45:42.293 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 2, 2024-04-30T08:45:42.293 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000322450, 2024-04-30T08:45:42.293 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000161225 2024-04-30T08:45:42.293 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.294 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookuphash_latency": { 2024-04-30T08:45:42.294 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:42.294 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:42.294 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:42.294 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.294 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupino_latency": { 2024-04-30T08:45:42.294 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:42.294 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:42.294 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:42.294 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.295 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupname_latency": { 2024-04-30T08:45:42.295 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:42.295 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:42.295 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:42.295 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.295 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupparent_latency": { 2024-04-30T08:45:42.295 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:42.295 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:42.295 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:42.295 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.295 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupsnap_latency": { 2024-04-30T08:45:42.295 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:42.295 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:42.295 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:42.296 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.296 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lssnap_latency": { 2024-04-30T08:45:42.296 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:42.296 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:42.296 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:42.296 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.296 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mkdir_latency": { 2024-04-30T08:45:42.297 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:42.297 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:42.297 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:42.297 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.297 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mknod_latency": { 2024-04-30T08:45:42.297 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:42.297 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:42.297 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:42.297 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.297 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mksnap_latency": { 2024-04-30T08:45:42.297 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:42.298 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:42.298 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:42.298 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.298 INFO:teuthology.orchestra.run.smithi189.stdout: "req_open_latency": { 2024-04-30T08:45:42.298 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:42.298 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:42.298 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:42.298 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.298 INFO:teuthology.orchestra.run.smithi189.stdout: "req_readdir_latency": { 2024-04-30T08:45:42.298 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 1, 2024-04-30T08:45:42.298 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000298856, 2024-04-30T08:45:42.298 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000298856 2024-04-30T08:45:42.298 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.299 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rename_latency": { 2024-04-30T08:45:42.299 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:42.299 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:42.299 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:42.299 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.299 INFO:teuthology.orchestra.run.smithi189.stdout: "req_renamesnap_latency": { 2024-04-30T08:45:42.299 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:42.299 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:42.299 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:42.299 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.299 INFO:teuthology.orchestra.run.smithi189.stdout: "req_snapdiff_latency": { 2024-04-30T08:45:42.299 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:42.300 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:42.300 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:42.300 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.300 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmdir_latency": { 2024-04-30T08:45:42.300 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:42.300 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:42.300 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:42.301 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.301 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmsnap_latency": { 2024-04-30T08:45:42.301 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:42.301 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:42.301 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:42.301 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.301 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmxattr_latency": { 2024-04-30T08:45:42.301 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:42.301 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:42.301 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:42.301 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.301 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setattr_latency": { 2024-04-30T08:45:42.302 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:42.302 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:42.302 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:42.302 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.302 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setdirlayout_latency": { 2024-04-30T08:45:42.302 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:42.302 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:42.302 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:42.302 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.302 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setfilelock_latency": { 2024-04-30T08:45:42.302 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:42.302 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:42.302 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:42.303 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.303 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setlayout_latency": { 2024-04-30T08:45:42.303 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:42.303 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:42.303 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:42.303 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.303 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setxattr_latency": { 2024-04-30T08:45:42.303 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:42.303 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:42.304 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:42.304 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.304 INFO:teuthology.orchestra.run.smithi189.stdout: "req_symlink_latency": { 2024-04-30T08:45:42.304 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:42.304 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:42.304 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:42.304 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.304 INFO:teuthology.orchestra.run.smithi189.stdout: "req_unlink_latency": { 2024-04-30T08:45:42.304 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 1, 2024-04-30T08:45:42.304 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.002432816, 2024-04-30T08:45:42.304 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.002432816 2024-04-30T08:45:42.304 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.304 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_revoke_eviction": 0, 2024-04-30T08:45:42.304 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_acquisition_throttle": 0, 2024-04-30T08:45:42.305 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getvxattr_latency": { 2024-04-30T08:45:42.305 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:42.305 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:42.305 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:42.305 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:42.305 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.305 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_sessions": { 2024-04-30T08:45:42.305 INFO:teuthology.orchestra.run.smithi189.stdout: "session_count": 1, 2024-04-30T08:45:42.305 INFO:teuthology.orchestra.run.smithi189.stdout: "session_add": 3, 2024-04-30T08:45:42.306 INFO:teuthology.orchestra.run.smithi189.stdout: "session_remove": 2, 2024-04-30T08:45:42.306 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_open": 1, 2024-04-30T08:45:42.306 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_stale": 0, 2024-04-30T08:45:42.306 INFO:teuthology.orchestra.run.smithi189.stdout: "total_load": 1, 2024-04-30T08:45:42.306 INFO:teuthology.orchestra.run.smithi189.stdout: "average_load": 1, 2024-04-30T08:45:42.306 INFO:teuthology.orchestra.run.smithi189.stdout: "avg_session_uptime": 634, 2024-04-30T08:45:42.306 INFO:teuthology.orchestra.run.smithi189.stdout: "mdthresh_evicted": 0 2024-04-30T08:45:42.306 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.306 INFO:teuthology.orchestra.run.smithi189.stdout: "mempool": { 2024-04-30T08:45:42.306 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_bytes": 0, 2024-04-30T08:45:42.306 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_items": 0, 2024-04-30T08:45:42.306 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_bytes": 0, 2024-04-30T08:45:42.307 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_items": 0, 2024-04-30T08:45:42.307 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_bytes": 0, 2024-04-30T08:45:42.307 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_items": 0, 2024-04-30T08:45:42.307 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-30T08:45:42.307 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_items": 0, 2024-04-30T08:45:42.307 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-30T08:45:42.307 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_items": 0, 2024-04-30T08:45:42.307 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_bytes": 0, 2024-04-30T08:45:42.307 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_items": 0, 2024-04-30T08:45:42.308 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-30T08:45:42.308 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_items": 0, 2024-04-30T08:45:42.308 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_bytes": 0, 2024-04-30T08:45:42.308 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_items": 0, 2024-04-30T08:45:42.308 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_bytes": 0, 2024-04-30T08:45:42.308 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_items": 0, 2024-04-30T08:45:42.308 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-30T08:45:42.308 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_items": 0, 2024-04-30T08:45:42.308 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-30T08:45:42.308 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_items": 0, 2024-04-30T08:45:42.308 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_bytes": 0, 2024-04-30T08:45:42.309 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_items": 0, 2024-04-30T08:45:42.309 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_bytes": 0, 2024-04-30T08:45:42.309 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_items": 0, 2024-04-30T08:45:42.309 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-30T08:45:42.309 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_items": 0, 2024-04-30T08:45:42.309 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_bytes": 0, 2024-04-30T08:45:42.309 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_items": 0, 2024-04-30T08:45:42.309 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_bytes": 0, 2024-04-30T08:45:42.309 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_items": 0, 2024-04-30T08:45:42.309 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_bytes": 0, 2024-04-30T08:45:42.310 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_items": 0, 2024-04-30T08:45:42.310 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_bytes": 0, 2024-04-30T08:45:42.310 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_items": 0, 2024-04-30T08:45:42.310 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_bytes": 270079, 2024-04-30T08:45:42.310 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_items": 107, 2024-04-30T08:45:42.310 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_bytes": 88, 2024-04-30T08:45:42.310 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_items": 1, 2024-04-30T08:45:42.310 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_bytes": 0, 2024-04-30T08:45:42.310 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_items": 0, 2024-04-30T08:45:42.310 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_bytes": 0, 2024-04-30T08:45:42.311 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_items": 0, 2024-04-30T08:45:42.311 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_bytes": 0, 2024-04-30T08:45:42.311 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_items": 0, 2024-04-30T08:45:42.311 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_bytes": 9088, 2024-04-30T08:45:42.311 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_items": 175, 2024-04-30T08:45:42.311 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_bytes": 0, 2024-04-30T08:45:42.311 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_items": 0, 2024-04-30T08:45:42.311 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_bytes": 0, 2024-04-30T08:45:42.311 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_items": 0, 2024-04-30T08:45:42.311 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_bytes": 77293, 2024-04-30T08:45:42.312 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_items": 335, 2024-04-30T08:45:42.312 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_bytes": 0, 2024-04-30T08:45:42.312 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_items": 0, 2024-04-30T08:45:42.312 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_bytes": 0, 2024-04-30T08:45:42.312 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_items": 0 2024-04-30T08:45:42.312 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.312 INFO:teuthology.orchestra.run.smithi189.stdout: "objecter": { 2024-04-30T08:45:42.312 INFO:teuthology.orchestra.run.smithi189.stdout: "op_active": 0, 2024-04-30T08:45:42.313 INFO:teuthology.orchestra.run.smithi189.stdout: "op_laggy": 0, 2024-04-30T08:45:42.313 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send": 78, 2024-04-30T08:45:42.313 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send_bytes": 38382, 2024-04-30T08:45:42.313 INFO:teuthology.orchestra.run.smithi189.stdout: "op_resend": 33, 2024-04-30T08:45:42.313 INFO:teuthology.orchestra.run.smithi189.stdout: "op_reply": 78, 2024-04-30T08:45:42.313 INFO:teuthology.orchestra.run.smithi189.stdout: "op_latency": { 2024-04-30T08:45:42.313 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 78, 2024-04-30T08:45:42.313 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.647986725, 2024-04-30T08:45:42.313 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.008307522 2024-04-30T08:45:42.314 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.314 INFO:teuthology.orchestra.run.smithi189.stdout: "op_inflight": 0, 2024-04-30T08:45:42.314 INFO:teuthology.orchestra.run.smithi189.stdout: "oplen_avg": { 2024-04-30T08:45:42.314 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 78, 2024-04-30T08:45:42.314 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 110 2024-04-30T08:45:42.314 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.314 INFO:teuthology.orchestra.run.smithi189.stdout: "op": 78, 2024-04-30T08:45:42.314 INFO:teuthology.orchestra.run.smithi189.stdout: "op_r": 3, 2024-04-30T08:45:42.314 INFO:teuthology.orchestra.run.smithi189.stdout: "op_w": 75, 2024-04-30T08:45:42.314 INFO:teuthology.orchestra.run.smithi189.stdout: "op_rmw": 0, 2024-04-30T08:45:42.314 INFO:teuthology.orchestra.run.smithi189.stdout: "op_pg": 0, 2024-04-30T08:45:42.314 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_stat": 0, 2024-04-30T08:45:42.314 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_create": 10, 2024-04-30T08:45:42.315 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_read": 1, 2024-04-30T08:45:42.315 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_write": 16, 2024-04-30T08:45:42.315 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writefull": 13, 2024-04-30T08:45:42.315 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writesame": 0, 2024-04-30T08:45:42.315 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_append": 0, 2024-04-30T08:45:42.315 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_zero": 0, 2024-04-30T08:45:42.315 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_truncate": 0, 2024-04-30T08:45:42.315 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_delete": 14, 2024-04-30T08:45:42.315 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_mapext": 0, 2024-04-30T08:45:42.315 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_sparse_read": 0, 2024-04-30T08:45:42.315 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_clonerange": 0, 2024-04-30T08:45:42.315 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_getxattr": 2, 2024-04-30T08:45:42.315 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_setxattr": 20, 2024-04-30T08:45:42.315 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_cmpxattr": 0, 2024-04-30T08:45:42.315 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_rmxattr": 0, 2024-04-30T08:45:42.315 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_resetxattrs": 0, 2024-04-30T08:45:42.316 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_call": 0, 2024-04-30T08:45:42.316 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_watch": 0, 2024-04-30T08:45:42.316 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_notify": 0, 2024-04-30T08:45:42.316 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_src_cmpxattr": 0, 2024-04-30T08:45:42.316 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls": 0, 2024-04-30T08:45:42.316 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls_filter": 0, 2024-04-30T08:45:42.316 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_other": 0, 2024-04-30T08:45:42.316 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_active": 0, 2024-04-30T08:45:42.316 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_send": 0, 2024-04-30T08:45:42.316 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_resend": 0, 2024-04-30T08:45:42.316 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_ping": 0, 2024-04-30T08:45:42.316 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_active": 0, 2024-04-30T08:45:42.316 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_send": 0, 2024-04-30T08:45:42.316 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_resend": 0, 2024-04-30T08:45:42.316 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_active": 0, 2024-04-30T08:45:42.317 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_send": 0, 2024-04-30T08:45:42.317 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_resend": 0, 2024-04-30T08:45:42.317 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_active": 0, 2024-04-30T08:45:42.317 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_send": 0, 2024-04-30T08:45:42.317 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_resend": 0, 2024-04-30T08:45:42.317 INFO:teuthology.orchestra.run.smithi189.stdout: "command_active": 0, 2024-04-30T08:45:42.317 INFO:teuthology.orchestra.run.smithi189.stdout: "command_send": 0, 2024-04-30T08:45:42.317 INFO:teuthology.orchestra.run.smithi189.stdout: "command_resend": 0, 2024-04-30T08:45:42.317 INFO:teuthology.orchestra.run.smithi189.stdout: "map_epoch": 85, 2024-04-30T08:45:42.317 INFO:teuthology.orchestra.run.smithi189.stdout: "map_full": 0, 2024-04-30T08:45:42.317 INFO:teuthology.orchestra.run.smithi189.stdout: "map_inc": 4, 2024-04-30T08:45:42.317 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_sessions": 9, 2024-04-30T08:45:42.317 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_open": 9, 2024-04-30T08:45:42.317 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_close": 0, 2024-04-30T08:45:42.318 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_laggy": 0, 2024-04-30T08:45:42.318 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_wr": 30, 2024-04-30T08:45:42.318 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_rd": 0, 2024-04-30T08:45:42.318 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_del": 4 2024-04-30T08:45:42.318 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.318 INFO:teuthology.orchestra.run.smithi189.stdout: "oft": { 2024-04-30T08:45:42.318 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_objs": 1, 2024-04-30T08:45:42.318 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_kv_pairs": 0, 2024-04-30T08:45:42.318 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_updates": 0, 2024-04-30T08:45:42.318 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_removes": 0 2024-04-30T08:45:42.318 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.318 INFO:teuthology.orchestra.run.smithi189.stdout: "osd-slow-ops": { 2024-04-30T08:45:42.318 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_ops_count": 0 2024-04-30T08:45:42.318 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.319 INFO:teuthology.orchestra.run.smithi189.stdout: "purge_queue": { 2024-04-30T08:45:42.319 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops": 0, 2024-04-30T08:45:42.319 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops_high_water": 2, 2024-04-30T08:45:42.319 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing": 0, 2024-04-30T08:45:42.319 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_high_water": 1, 2024-04-30T08:45:42.319 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed_ops": 2, 2024-04-30T08:45:42.319 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed": 1, 2024-04-30T08:45:42.319 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_item_in_journal": 0 2024-04-30T08:45:42.319 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.319 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-30T08:45:42.319 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-30T08:45:42.319 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-04-30T08:45:42.319 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-30T08:45:42.319 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 1520, 2024-04-30T08:45:42.319 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 1741735, 2024-04-30T08:45:42.320 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:42.320 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 1520, 2024-04-30T08:45:42.320 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-04-30T08:45:42.320 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-04-30T08:45:42.320 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 1520, 2024-04-30T08:45:42.320 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 1741735, 2024-04-30T08:45:42.320 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-30T08:45:42.320 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:42.320 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:42.320 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:42.320 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:42.320 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.320 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_bytes": { 2024-04-30T08:45:42.320 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-30T08:45:42.320 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-04-30T08:45:42.321 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-30T08:45:42.321 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-04-30T08:45:42.321 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-04-30T08:45:42.321 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:42.321 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-04-30T08:45:42.321 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 78, 2024-04-30T08:45:42.321 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 38491, 2024-04-30T08:45:42.321 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 78, 2024-04-30T08:45:42.321 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 38491, 2024-04-30T08:45:42.321 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-30T08:45:42.321 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:42.321 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:42.321 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:42.321 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:42.321 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.322 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_ops": { 2024-04-30T08:45:42.322 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-30T08:45:42.322 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 1024, 2024-04-30T08:45:42.322 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-30T08:45:42.322 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-04-30T08:45:42.322 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-04-30T08:45:42.322 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:42.322 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-04-30T08:45:42.322 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 78, 2024-04-30T08:45:42.322 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 78, 2024-04-30T08:45:42.322 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 78, 2024-04-30T08:45:42.322 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 78, 2024-04-30T08:45:42.322 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-30T08:45:42.322 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:42.322 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:42.323 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:42.323 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:42.323 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.323 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle": { 2024-04-30T08:45:42.323 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-30T08:45:42.323 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-04-30T08:45:42.323 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-30T08:45:42.323 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 1, 2024-04-30T08:45:42.323 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 109, 2024-04-30T08:45:42.323 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:42.323 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 1, 2024-04-30T08:45:42.323 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-04-30T08:45:42.323 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-04-30T08:45:42.323 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 1, 2024-04-30T08:45:42.323 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 109, 2024-04-30T08:45:42.324 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-30T08:45:42.324 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:42.324 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:42.324 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:42.324 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:42.324 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:42.324 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle-0x555deffa6680": { 2024-04-30T08:45:42.324 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-30T08:45:42.324 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-04-30T08:45:42.324 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-30T08:45:42.324 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 19, 2024-04-30T08:45:42.324 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 25555, 2024-04-30T08:45:42.324 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:42.324 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 19, 2024-04-30T08:45:42.324 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-04-30T08:45:42.324 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-04-30T08:45:42.325 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 15, 2024-04-30T08:45:42.325 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 25555, 2024-04-30T08:45:42.325 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-30T08:45:42.325 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:42.325 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:42.325 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:42.325 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:42.325 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:42.325 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-30T08:45:43.101 INFO:tasks.check_counter:Daemon mds.i mds.exported=1 2024-04-30T08:45:43.102 INFO:tasks.check_counter:Daemon mds.i mds.imported=1 2024-04-30T08:45:43.102 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=0 2024-04-30T08:45:43.102 DEBUG:tasks.check_counter:Getting stats from l 2024-04-30T08:45:43.102 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f shell --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-30T08:45:43.343 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/mon.c/config 2024-04-30T08:45:43.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:43 smithi032 ceph-mon[28331]: pgmap v741: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-30T08:45:43.463 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:43 smithi189 ceph-mon[33038]: pgmap v741: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-30T08:45:43.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:43 smithi071 ceph-mon[33385]: pgmap v741: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-04-30T08:45:45.281 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-04-30T08:45:45.281 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-0": { 2024-04-30T08:45:45.282 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 410, 2024-04-30T08:45:45.282 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 487, 2024-04-30T08:45:45.282 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 137705, 2024-04-30T08:45:45.282 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 136156, 2024-04-30T08:45:45.282 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 6, 2024-04-30T08:45:45.282 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 2, 2024-04-30T08:45:45.282 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.056145376, 2024-04-30T08:45:45.282 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.018039305, 2024-04-30T08:45:45.282 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.015254070, 2024-04-30T08:45:45.282 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.016532886, 2024-04-30T08:45:45.282 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:45.283 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 485, 2024-04-30T08:45:45.283 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.030831940, 2024-04-30T08:45:45.283 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000063571 2024-04-30T08:45:45.283 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.283 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:45.283 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.283 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.283 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.283 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.283 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-30T08:45:45.283 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 0 2024-04-30T08:45:45.283 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.283 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-1": { 2024-04-30T08:45:45.284 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 676, 2024-04-30T08:45:45.284 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 854, 2024-04-30T08:45:45.284 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 197841, 2024-04-30T08:45:45.284 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 590995, 2024-04-30T08:45:45.284 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 5, 2024-04-30T08:45:45.284 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 4, 2024-04-30T08:45:45.284 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.064926684, 2024-04-30T08:45:45.284 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.024206405, 2024-04-30T08:45:45.284 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.015412554, 2024-04-30T08:45:45.284 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.019665988, 2024-04-30T08:45:45.284 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:45.284 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 852, 2024-04-30T08:45:45.284 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.043627754, 2024-04-30T08:45:45.285 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000051206 2024-04-30T08:45:45.285 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.285 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:45.285 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.285 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.285 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.285 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.285 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-30T08:45:45.285 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 376288 2024-04-30T08:45:45.285 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.285 INFO:teuthology.orchestra.run.smithi189.stdout: "AsyncMessenger::Worker-2": { 2024-04-30T08:45:45.285 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_messages": 359, 2024-04-30T08:45:45.285 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages": 247, 2024-04-30T08:45:45.285 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_bytes": 1537092, 2024-04-30T08:45:45.285 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_bytes": 152213, 2024-04-30T08:45:45.286 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_created_connections": 4, 2024-04-30T08:45:45.286 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_active_connections": 3, 2024-04-30T08:45:45.286 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_total_time": 0.048946176, 2024-04-30T08:45:45.286 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_send_time": 0.015282368, 2024-04-30T08:45:45.286 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_recv_time": 0.034965458, 2024-04-30T08:45:45.286 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_running_fast_dispatch_time": 0.006051658, 2024-04-30T08:45:45.286 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_messages_queue_lat": { 2024-04-30T08:45:45.286 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 247, 2024-04-30T08:45:45.286 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.018214001, 2024-04-30T08:45:45.286 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000073740 2024-04-30T08:45:45.286 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.286 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_handle_ack_lat": { 2024-04-30T08:45:45.286 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.287 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.287 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.287 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.287 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_recv_encrypted_bytes": 1535744, 2024-04-30T08:45:45.287 INFO:teuthology.orchestra.run.smithi189.stdout: "msgr_send_encrypted_bytes": 151248 2024-04-30T08:45:45.287 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.287 INFO:teuthology.orchestra.run.smithi189.stdout: "cct": { 2024-04-30T08:45:45.287 INFO:teuthology.orchestra.run.smithi189.stdout: "total_workers": 1, 2024-04-30T08:45:45.287 INFO:teuthology.orchestra.run.smithi189.stdout: "unhealthy_workers": 0 2024-04-30T08:45:45.287 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.287 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-MDSRank": { 2024-04-30T08:45:45.287 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-04-30T08:45:45.287 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-04-30T08:45:45.288 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 1914, 2024-04-30T08:45:45.288 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.059782808, 2024-04-30T08:45:45.288 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000031234 2024-04-30T08:45:45.288 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:45.288 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.288 INFO:teuthology.orchestra.run.smithi189.stdout: "finisher-PurgeQueue": { 2024-04-30T08:45:45.288 INFO:teuthology.orchestra.run.smithi189.stdout: "queue_len": 0, 2024-04-30T08:45:45.288 INFO:teuthology.orchestra.run.smithi189.stdout: "complete_latency": { 2024-04-30T08:45:45.288 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.288 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.288 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.288 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:45.289 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.289 INFO:teuthology.orchestra.run.smithi189.stdout: "mds": { 2024-04-30T08:45:45.289 INFO:teuthology.orchestra.run.smithi189.stdout: "request": 0, 2024-04-30T08:45:45.289 INFO:teuthology.orchestra.run.smithi189.stdout: "reply": 0, 2024-04-30T08:45:45.289 INFO:teuthology.orchestra.run.smithi189.stdout: "reply_latency": { 2024-04-30T08:45:45.289 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.289 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.289 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.289 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.289 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_reply": 0, 2024-04-30T08:45:45.290 INFO:teuthology.orchestra.run.smithi189.stdout: "forward": 0, 2024-04-30T08:45:45.290 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_complete": 0, 2024-04-30T08:45:45.290 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_fetch_keys": 0, 2024-04-30T08:45:45.290 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_commit": 0, 2024-04-30T08:45:45.290 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_split": 0, 2024-04-30T08:45:45.290 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_merge": 0, 2024-04-30T08:45:45.290 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes": 2, 2024-04-30T08:45:45.290 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_top": 0, 2024-04-30T08:45:45.290 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_bottom": 2, 2024-04-30T08:45:45.290 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pin_tail": 0, 2024-04-30T08:45:45.290 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_pinned": 2, 2024-04-30T08:45:45.290 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_expired": 0, 2024-04-30T08:45:45.291 INFO:teuthology.orchestra.run.smithi189.stdout: "inodes_with_caps": 0, 2024-04-30T08:45:45.291 INFO:teuthology.orchestra.run.smithi189.stdout: "caps": 0, 2024-04-30T08:45:45.291 INFO:teuthology.orchestra.run.smithi189.stdout: "subtrees": 1, 2024-04-30T08:45:45.291 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse": 0, 2024-04-30T08:45:45.291 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_hit": 0, 2024-04-30T08:45:45.291 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_forward": 0, 2024-04-30T08:45:45.291 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_discover": 0, 2024-04-30T08:45:45.291 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_dir_fetch": 0, 2024-04-30T08:45:45.291 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_remote_ino": 0, 2024-04-30T08:45:45.291 INFO:teuthology.orchestra.run.smithi189.stdout: "traverse_lock": 0, 2024-04-30T08:45:45.291 INFO:teuthology.orchestra.run.smithi189.stdout: "load_cent": 0, 2024-04-30T08:45:45.291 INFO:teuthology.orchestra.run.smithi189.stdout: "q": 0, 2024-04-30T08:45:45.291 INFO:teuthology.orchestra.run.smithi189.stdout: "exported": 0, 2024-04-30T08:45:45.291 INFO:teuthology.orchestra.run.smithi189.stdout: "exported_inodes": 0, 2024-04-30T08:45:45.292 INFO:teuthology.orchestra.run.smithi189.stdout: "imported": 0, 2024-04-30T08:45:45.292 INFO:teuthology.orchestra.run.smithi189.stdout: "imported_inodes": 0, 2024-04-30T08:45:45.292 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_dir_fetch": 0, 2024-04-30T08:45:45.292 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_backtrace_fetch": 0, 2024-04-30T08:45:45.292 INFO:teuthology.orchestra.run.smithi189.stdout: "openino_peer_discover": 0, 2024-04-30T08:45:45.292 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rfiles": 0, 2024-04-30T08:45:45.292 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rbytes": 0, 2024-04-30T08:45:45.292 INFO:teuthology.orchestra.run.smithi189.stdout: "root_rsnaps": 0, 2024-04-30T08:45:45.292 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_fetch": 0, 2024-04-30T08:45:45.292 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_set_tag": 0, 2024-04-30T08:45:45.292 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_backtrace_repaired": 0, 2024-04-30T08:45:45.292 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_inotable_repaired": 0, 2024-04-30T08:45:45.292 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_inodes": 0, 2024-04-30T08:45:45.292 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dir_base_inodes": 0, 2024-04-30T08:45:45.293 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_dirfrag_rstats": 0, 2024-04-30T08:45:45.293 INFO:teuthology.orchestra.run.smithi189.stdout: "scrub_file_inodes": 0, 2024-04-30T08:45:45.293 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_inode_file_caps": 0, 2024-04-30T08:45:45.293 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_revoke": 0, 2024-04-30T08:45:45.293 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_grant": 0, 2024-04-30T08:45:45.293 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_trunc": 0, 2024-04-30T08:45:45.293 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-30T08:45:45.293 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-30T08:45:45.293 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps": 0, 2024-04-30T08:45:45.293 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_caps_dirty": 0, 2024-04-30T08:45:45.293 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_cap_release": 0, 2024-04-30T08:45:45.293 INFO:teuthology.orchestra.run.smithi189.stdout: "process_request_cap_release": 0 2024-04-30T08:45:45.293 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.293 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_cache": { 2024-04-30T08:45:45.294 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update": 0, 2024-04-30T08:45:45.294 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_update_receipt": 0, 2024-04-30T08:45:45.294 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_try_discover": 0, 2024-04-30T08:45:45.294 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_send_discover": 0, 2024-04-30T08:45:45.294 INFO:teuthology.orchestra.run.smithi189.stdout: "dir_handle_discover": 0, 2024-04-30T08:45:45.294 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays": 0, 2024-04-30T08:45:45.294 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_delayed": 0, 2024-04-30T08:45:45.294 INFO:teuthology.orchestra.run.smithi189.stdout: "num_strays_enqueuing": 0, 2024-04-30T08:45:45.294 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_created": 0, 2024-04-30T08:45:45.294 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_enqueued": 0, 2024-04-30T08:45:45.294 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_reintegrated": 0, 2024-04-30T08:45:45.294 INFO:teuthology.orchestra.run.smithi189.stdout: "strays_migrated": 0, 2024-04-30T08:45:45.295 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_processing": 0, 2024-04-30T08:45:45.295 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_enqueued": 0, 2024-04-30T08:45:45.295 INFO:teuthology.orchestra.run.smithi189.stdout: "num_recovering_prioritized": 0, 2024-04-30T08:45:45.295 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_started": 0, 2024-04-30T08:45:45.295 INFO:teuthology.orchestra.run.smithi189.stdout: "recovery_completed": 0, 2024-04-30T08:45:45.295 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_quiesce_path": 0, 2024-04-30T08:45:45.295 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_quiesce_inode": 0, 2024-04-30T08:45:45.295 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_enqueue_scrub": 0, 2024-04-30T08:45:45.295 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_exportdir": 0, 2024-04-30T08:45:45.295 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_flush": 0, 2024-04-30T08:45:45.295 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragmentdir": 0, 2024-04-30T08:45:45.295 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_fragstats": 0, 2024-04-30T08:45:45.295 INFO:teuthology.orchestra.run.smithi189.stdout: "ireq_inodestats": 0 2024-04-30T08:45:45.296 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.296 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_log": { 2024-04-30T08:45:45.296 INFO:teuthology.orchestra.run.smithi189.stdout: "evlrg": 0, 2024-04-30T08:45:45.296 INFO:teuthology.orchestra.run.smithi189.stdout: "evadd": 0, 2024-04-30T08:45:45.296 INFO:teuthology.orchestra.run.smithi189.stdout: "evex": 0, 2024-04-30T08:45:45.296 INFO:teuthology.orchestra.run.smithi189.stdout: "evtrm": 0, 2024-04-30T08:45:45.296 INFO:teuthology.orchestra.run.smithi189.stdout: "ev": 17, 2024-04-30T08:45:45.296 INFO:teuthology.orchestra.run.smithi189.stdout: "evexg": 0, 2024-04-30T08:45:45.296 INFO:teuthology.orchestra.run.smithi189.stdout: "evexd": 0, 2024-04-30T08:45:45.296 INFO:teuthology.orchestra.run.smithi189.stdout: "segadd": 0, 2024-04-30T08:45:45.296 INFO:teuthology.orchestra.run.smithi189.stdout: "segex": 0, 2024-04-30T08:45:45.297 INFO:teuthology.orchestra.run.smithi189.stdout: "segtrm": 0, 2024-04-30T08:45:45.297 INFO:teuthology.orchestra.run.smithi189.stdout: "seg": 2, 2024-04-30T08:45:45.297 INFO:teuthology.orchestra.run.smithi189.stdout: "segmjr": 2, 2024-04-30T08:45:45.297 INFO:teuthology.orchestra.run.smithi189.stdout: "segexg": 0, 2024-04-30T08:45:45.297 INFO:teuthology.orchestra.run.smithi189.stdout: "segexd": 0, 2024-04-30T08:45:45.297 INFO:teuthology.orchestra.run.smithi189.stdout: "expos": 4194304, 2024-04-30T08:45:45.297 INFO:teuthology.orchestra.run.smithi189.stdout: "wrpos": 4219327, 2024-04-30T08:45:45.297 INFO:teuthology.orchestra.run.smithi189.stdout: "rdpos": 4217676, 2024-04-30T08:45:45.297 INFO:teuthology.orchestra.run.smithi189.stdout: "jlat": { 2024-04-30T08:45:45.297 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.297 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.297 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.297 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.298 INFO:teuthology.orchestra.run.smithi189.stdout: "replayed": 17 2024-04-30T08:45:45.298 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.298 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_mem": { 2024-04-30T08:45:45.298 INFO:teuthology.orchestra.run.smithi189.stdout: "ino": 4, 2024-04-30T08:45:45.298 INFO:teuthology.orchestra.run.smithi189.stdout: "ino+": 17, 2024-04-30T08:45:45.298 INFO:teuthology.orchestra.run.smithi189.stdout: "ino-": 13, 2024-04-30T08:45:45.298 INFO:teuthology.orchestra.run.smithi189.stdout: "dir": 2, 2024-04-30T08:45:45.298 INFO:teuthology.orchestra.run.smithi189.stdout: "dir+": 15, 2024-04-30T08:45:45.298 INFO:teuthology.orchestra.run.smithi189.stdout: "dir-": 13, 2024-04-30T08:45:45.298 INFO:teuthology.orchestra.run.smithi189.stdout: "dn": 2, 2024-04-30T08:45:45.299 INFO:teuthology.orchestra.run.smithi189.stdout: "dn+": 16, 2024-04-30T08:45:45.299 INFO:teuthology.orchestra.run.smithi189.stdout: "dn-": 14, 2024-04-30T08:45:45.299 INFO:teuthology.orchestra.run.smithi189.stdout: "cap": 0, 2024-04-30T08:45:45.299 INFO:teuthology.orchestra.run.smithi189.stdout: "cap+": 0, 2024-04-30T08:45:45.299 INFO:teuthology.orchestra.run.smithi189.stdout: "cap-": 0, 2024-04-30T08:45:45.299 INFO:teuthology.orchestra.run.smithi189.stdout: "rss": 119764, 2024-04-30T08:45:45.299 INFO:teuthology.orchestra.run.smithi189.stdout: "heap": 215440 2024-04-30T08:45:45.299 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.299 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_server": { 2024-04-30T08:45:45.299 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_client_request": 0, 2024-04-30T08:45:45.300 INFO:teuthology.orchestra.run.smithi189.stdout: "dispatch_server_request": 0, 2024-04-30T08:45:45.300 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_request": 0, 2024-04-30T08:45:45.300 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_client_session": 0, 2024-04-30T08:45:45.300 INFO:teuthology.orchestra.run.smithi189.stdout: "handle_peer_request": 0, 2024-04-30T08:45:45.300 INFO:teuthology.orchestra.run.smithi189.stdout: "req_create_latency": { 2024-04-30T08:45:45.300 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.300 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.300 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.300 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.300 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getattr_latency": { 2024-04-30T08:45:45.301 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.301 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.301 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.301 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.301 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getfilelock_latency": { 2024-04-30T08:45:45.301 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.301 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.301 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.302 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.302 INFO:teuthology.orchestra.run.smithi189.stdout: "req_link_latency": { 2024-04-30T08:45:45.302 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.302 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.302 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.302 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.302 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookup_latency": { 2024-04-30T08:45:45.302 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.302 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.302 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.303 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.303 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookuphash_latency": { 2024-04-30T08:45:45.303 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.303 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.303 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.303 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.303 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupino_latency": { 2024-04-30T08:45:45.303 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.303 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.304 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.304 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.304 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupname_latency": { 2024-04-30T08:45:45.304 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.304 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.304 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.304 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.304 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupparent_latency": { 2024-04-30T08:45:45.304 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.304 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.304 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.305 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.305 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lookupsnap_latency": { 2024-04-30T08:45:45.305 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.305 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.305 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.305 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.305 INFO:teuthology.orchestra.run.smithi189.stdout: "req_lssnap_latency": { 2024-04-30T08:45:45.305 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.305 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.305 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.305 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.305 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mkdir_latency": { 2024-04-30T08:45:45.305 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.306 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.306 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.306 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.306 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mknod_latency": { 2024-04-30T08:45:45.307 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.307 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.307 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.307 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.307 INFO:teuthology.orchestra.run.smithi189.stdout: "req_mksnap_latency": { 2024-04-30T08:45:45.307 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.307 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.307 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.307 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.307 INFO:teuthology.orchestra.run.smithi189.stdout: "req_open_latency": { 2024-04-30T08:45:45.307 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.307 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.307 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.307 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.308 INFO:teuthology.orchestra.run.smithi189.stdout: "req_readdir_latency": { 2024-04-30T08:45:45.308 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.308 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.308 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.308 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.308 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rename_latency": { 2024-04-30T08:45:45.308 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.308 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.308 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.308 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.308 INFO:teuthology.orchestra.run.smithi189.stdout: "req_renamesnap_latency": { 2024-04-30T08:45:45.308 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.309 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.309 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.309 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.309 INFO:teuthology.orchestra.run.smithi189.stdout: "req_snapdiff_latency": { 2024-04-30T08:45:45.309 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.309 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.309 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.309 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.309 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmdir_latency": { 2024-04-30T08:45:45.309 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.309 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.309 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.309 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.310 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmsnap_latency": { 2024-04-30T08:45:45.310 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.310 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.310 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.310 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.310 INFO:teuthology.orchestra.run.smithi189.stdout: "req_rmxattr_latency": { 2024-04-30T08:45:45.310 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.310 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.310 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.310 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.310 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setattr_latency": { 2024-04-30T08:45:45.310 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.311 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.311 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.311 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.311 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setdirlayout_latency": { 2024-04-30T08:45:45.311 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.311 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.311 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.311 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.311 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setfilelock_latency": { 2024-04-30T08:45:45.311 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.311 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.311 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.311 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.312 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setlayout_latency": { 2024-04-30T08:45:45.312 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.312 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.312 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.312 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.312 INFO:teuthology.orchestra.run.smithi189.stdout: "req_setxattr_latency": { 2024-04-30T08:45:45.312 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.312 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.312 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.312 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.312 INFO:teuthology.orchestra.run.smithi189.stdout: "req_symlink_latency": { 2024-04-30T08:45:45.313 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.313 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.313 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.313 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.313 INFO:teuthology.orchestra.run.smithi189.stdout: "req_unlink_latency": { 2024-04-30T08:45:45.313 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.313 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.313 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.313 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.313 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_revoke_eviction": 0, 2024-04-30T08:45:45.313 INFO:teuthology.orchestra.run.smithi189.stdout: "cap_acquisition_throttle": 0, 2024-04-30T08:45:45.313 INFO:teuthology.orchestra.run.smithi189.stdout: "req_getvxattr_latency": { 2024-04-30T08:45:45.313 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.313 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.314 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.314 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:45.314 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.314 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_sessions": { 2024-04-30T08:45:45.314 INFO:teuthology.orchestra.run.smithi189.stdout: "session_count": 1, 2024-04-30T08:45:45.314 INFO:teuthology.orchestra.run.smithi189.stdout: "session_add": 2, 2024-04-30T08:45:45.314 INFO:teuthology.orchestra.run.smithi189.stdout: "session_remove": 1, 2024-04-30T08:45:45.314 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_open": 2, 2024-04-30T08:45:45.314 INFO:teuthology.orchestra.run.smithi189.stdout: "sessions_stale": 0, 2024-04-30T08:45:45.314 INFO:teuthology.orchestra.run.smithi189.stdout: "total_load": 0, 2024-04-30T08:45:45.314 INFO:teuthology.orchestra.run.smithi189.stdout: "average_load": 0, 2024-04-30T08:45:45.315 INFO:teuthology.orchestra.run.smithi189.stdout: "avg_session_uptime": 3867, 2024-04-30T08:45:45.315 INFO:teuthology.orchestra.run.smithi189.stdout: "mdthresh_evicted": 0 2024-04-30T08:45:45.315 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.315 INFO:teuthology.orchestra.run.smithi189.stdout: "mempool": { 2024-04-30T08:45:45.315 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_bytes": 0, 2024-04-30T08:45:45.315 INFO:teuthology.orchestra.run.smithi189.stdout: "bloom_filter_items": 0, 2024-04-30T08:45:45.315 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_bytes": 0, 2024-04-30T08:45:45.315 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_alloc_items": 0, 2024-04-30T08:45:45.316 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_bytes": 0, 2024-04-30T08:45:45.316 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_data_items": 0, 2024-04-30T08:45:45.316 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-30T08:45:45.316 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_onode_items": 0, 2024-04-30T08:45:45.316 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-30T08:45:45.316 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_meta_items": 0, 2024-04-30T08:45:45.316 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_bytes": 0, 2024-04-30T08:45:45.316 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_other_items": 0, 2024-04-30T08:45:45.316 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-30T08:45:45.316 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_cache_buffer_items": 0, 2024-04-30T08:45:45.316 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_bytes": 0, 2024-04-30T08:45:45.316 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_extent_items": 0, 2024-04-30T08:45:45.317 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_bytes": 0, 2024-04-30T08:45:45.317 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_blob_items": 0, 2024-04-30T08:45:45.317 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-30T08:45:45.317 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_shared_blob_items": 0, 2024-04-30T08:45:45.317 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-30T08:45:45.317 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_inline_bl_items": 0, 2024-04-30T08:45:45.317 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_bytes": 0, 2024-04-30T08:45:45.317 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_fsck_items": 0, 2024-04-30T08:45:45.317 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_bytes": 0, 2024-04-30T08:45:45.317 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_txc_items": 0, 2024-04-30T08:45:45.317 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-30T08:45:45.317 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_deferred_items": 0, 2024-04-30T08:45:45.317 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_bytes": 0, 2024-04-30T08:45:45.317 INFO:teuthology.orchestra.run.smithi189.stdout: "bluestore_writing_items": 0, 2024-04-30T08:45:45.318 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_bytes": 0, 2024-04-30T08:45:45.318 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_items": 0, 2024-04-30T08:45:45.318 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_bytes": 0, 2024-04-30T08:45:45.318 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_reader_items": 0, 2024-04-30T08:45:45.318 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_bytes": 0, 2024-04-30T08:45:45.318 INFO:teuthology.orchestra.run.smithi189.stdout: "bluefs_file_writer_items": 0, 2024-04-30T08:45:45.318 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_bytes": 197441, 2024-04-30T08:45:45.318 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_anon_items": 54, 2024-04-30T08:45:45.318 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_bytes": 0, 2024-04-30T08:45:45.319 INFO:teuthology.orchestra.run.smithi189.stdout: "buffer_meta_items": 0, 2024-04-30T08:45:45.319 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_bytes": 0, 2024-04-30T08:45:45.319 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_items": 0, 2024-04-30T08:45:45.319 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_bytes": 0, 2024-04-30T08:45:45.319 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_mapbl_items": 0, 2024-04-30T08:45:45.319 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_bytes": 0, 2024-04-30T08:45:45.319 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_pglog_items": 0, 2024-04-30T08:45:45.319 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_bytes": 9088, 2024-04-30T08:45:45.319 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_items": 175, 2024-04-30T08:45:45.319 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_bytes": 0, 2024-04-30T08:45:45.319 INFO:teuthology.orchestra.run.smithi189.stdout: "osdmap_mapping_items": 0, 2024-04-30T08:45:45.319 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_bytes": 0, 2024-04-30T08:45:45.319 INFO:teuthology.orchestra.run.smithi189.stdout: "pgmap_items": 0, 2024-04-30T08:45:45.319 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_bytes": 13456, 2024-04-30T08:45:45.319 INFO:teuthology.orchestra.run.smithi189.stdout: "mds_co_items": 36, 2024-04-30T08:45:45.320 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_bytes": 0, 2024-04-30T08:45:45.320 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_1_items": 0, 2024-04-30T08:45:45.320 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_bytes": 0, 2024-04-30T08:45:45.320 INFO:teuthology.orchestra.run.smithi189.stdout: "unittest_2_items": 0 2024-04-30T08:45:45.320 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.320 INFO:teuthology.orchestra.run.smithi189.stdout: "objecter": { 2024-04-30T08:45:45.320 INFO:teuthology.orchestra.run.smithi189.stdout: "op_active": 0, 2024-04-30T08:45:45.320 INFO:teuthology.orchestra.run.smithi189.stdout: "op_laggy": 0, 2024-04-30T08:45:45.320 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send": 1088, 2024-04-30T08:45:45.320 INFO:teuthology.orchestra.run.smithi189.stdout: "op_send_bytes": 21, 2024-04-30T08:45:45.320 INFO:teuthology.orchestra.run.smithi189.stdout: "op_resend": 3, 2024-04-30T08:45:45.320 INFO:teuthology.orchestra.run.smithi189.stdout: "op_reply": 1088, 2024-04-30T08:45:45.320 INFO:teuthology.orchestra.run.smithi189.stdout: "op_latency": { 2024-04-30T08:45:45.321 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 1088, 2024-04-30T08:45:45.321 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 385.643089569, 2024-04-30T08:45:45.321 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.354451369 2024-04-30T08:45:45.321 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.321 INFO:teuthology.orchestra.run.smithi189.stdout: "op_inflight": 0, 2024-04-30T08:45:45.321 INFO:teuthology.orchestra.run.smithi189.stdout: "oplen_avg": { 2024-04-30T08:45:45.321 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 1088, 2024-04-30T08:45:45.321 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 1089 2024-04-30T08:45:45.321 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.321 INFO:teuthology.orchestra.run.smithi189.stdout: "op": 1088, 2024-04-30T08:45:45.321 INFO:teuthology.orchestra.run.smithi189.stdout: "op_r": 1088, 2024-04-30T08:45:45.322 INFO:teuthology.orchestra.run.smithi189.stdout: "op_w": 0, 2024-04-30T08:45:45.322 INFO:teuthology.orchestra.run.smithi189.stdout: "op_rmw": 0, 2024-04-30T08:45:45.322 INFO:teuthology.orchestra.run.smithi189.stdout: "op_pg": 0, 2024-04-30T08:45:45.322 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_stat": 662, 2024-04-30T08:45:45.322 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_create": 0, 2024-04-30T08:45:45.322 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_read": 424, 2024-04-30T08:45:45.322 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_write": 0, 2024-04-30T08:45:45.322 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writefull": 0, 2024-04-30T08:45:45.322 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_writesame": 0, 2024-04-30T08:45:45.322 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_append": 0, 2024-04-30T08:45:45.322 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_zero": 0, 2024-04-30T08:45:45.322 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_truncate": 0, 2024-04-30T08:45:45.322 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_delete": 0, 2024-04-30T08:45:45.322 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_mapext": 0, 2024-04-30T08:45:45.323 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_sparse_read": 0, 2024-04-30T08:45:45.323 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_clonerange": 0, 2024-04-30T08:45:45.323 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_getxattr": 1, 2024-04-30T08:45:45.323 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_setxattr": 0, 2024-04-30T08:45:45.323 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_cmpxattr": 0, 2024-04-30T08:45:45.323 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_rmxattr": 0, 2024-04-30T08:45:45.323 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_resetxattrs": 0, 2024-04-30T08:45:45.323 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_call": 0, 2024-04-30T08:45:45.323 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_watch": 0, 2024-04-30T08:45:45.323 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_notify": 0, 2024-04-30T08:45:45.323 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_src_cmpxattr": 0, 2024-04-30T08:45:45.323 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls": 0, 2024-04-30T08:45:45.323 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_pgls_filter": 0, 2024-04-30T08:45:45.323 INFO:teuthology.orchestra.run.smithi189.stdout: "osdop_other": 0, 2024-04-30T08:45:45.323 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_active": 0, 2024-04-30T08:45:45.324 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_send": 0, 2024-04-30T08:45:45.324 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_resend": 0, 2024-04-30T08:45:45.324 INFO:teuthology.orchestra.run.smithi189.stdout: "linger_ping": 0, 2024-04-30T08:45:45.324 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_active": 0, 2024-04-30T08:45:45.324 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_send": 0, 2024-04-30T08:45:45.324 INFO:teuthology.orchestra.run.smithi189.stdout: "poolop_resend": 0, 2024-04-30T08:45:45.324 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_active": 0, 2024-04-30T08:45:45.324 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_send": 0, 2024-04-30T08:45:45.324 INFO:teuthology.orchestra.run.smithi189.stdout: "poolstat_resend": 0, 2024-04-30T08:45:45.324 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_active": 0, 2024-04-30T08:45:45.324 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_send": 0, 2024-04-30T08:45:45.324 INFO:teuthology.orchestra.run.smithi189.stdout: "statfs_resend": 0, 2024-04-30T08:45:45.325 INFO:teuthology.orchestra.run.smithi189.stdout: "command_active": 0, 2024-04-30T08:45:45.325 INFO:teuthology.orchestra.run.smithi189.stdout: "command_send": 0, 2024-04-30T08:45:45.325 INFO:teuthology.orchestra.run.smithi189.stdout: "command_resend": 0, 2024-04-30T08:45:45.325 INFO:teuthology.orchestra.run.smithi189.stdout: "map_epoch": 85, 2024-04-30T08:45:45.325 INFO:teuthology.orchestra.run.smithi189.stdout: "map_full": 0, 2024-04-30T08:45:45.325 INFO:teuthology.orchestra.run.smithi189.stdout: "map_inc": 3, 2024-04-30T08:45:45.325 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_sessions": 6, 2024-04-30T08:45:45.325 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_open": 6, 2024-04-30T08:45:45.326 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_session_close": 0, 2024-04-30T08:45:45.326 INFO:teuthology.orchestra.run.smithi189.stdout: "osd_laggy": 0, 2024-04-30T08:45:45.326 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_wr": 0, 2024-04-30T08:45:45.326 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_rd": 2, 2024-04-30T08:45:45.326 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_del": 0 2024-04-30T08:45:45.326 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.326 INFO:teuthology.orchestra.run.smithi189.stdout: "oft": { 2024-04-30T08:45:45.327 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_objs": 0, 2024-04-30T08:45:45.327 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_kv_pairs": 0, 2024-04-30T08:45:45.327 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_updates": 0, 2024-04-30T08:45:45.327 INFO:teuthology.orchestra.run.smithi189.stdout: "omap_total_removes": 0 2024-04-30T08:45:45.327 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.327 INFO:teuthology.orchestra.run.smithi189.stdout: "osd-slow-ops": { 2024-04-30T08:45:45.327 INFO:teuthology.orchestra.run.smithi189.stdout: "slow_ops_count": 0 2024-04-30T08:45:45.327 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.327 INFO:teuthology.orchestra.run.smithi189.stdout: "purge_queue": { 2024-04-30T08:45:45.327 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops": 0, 2024-04-30T08:45:45.327 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_ops_high_water": 0, 2024-04-30T08:45:45.327 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing": 0, 2024-04-30T08:45:45.328 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executing_high_water": 0, 2024-04-30T08:45:45.328 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed_ops": 0, 2024-04-30T08:45:45.328 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_executed": 0, 2024-04-30T08:45:45.328 INFO:teuthology.orchestra.run.smithi189.stdout: "pq_item_in_journal": 0 2024-04-30T08:45:45.328 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.329 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-30T08:45:45.329 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-30T08:45:45.329 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-04-30T08:45:45.329 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-30T08:45:45.329 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 1441, 2024-04-30T08:45:45.329 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 1722932, 2024-04-30T08:45:45.329 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:45.330 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 1441, 2024-04-30T08:45:45.330 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-04-30T08:45:45.330 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-04-30T08:45:45.330 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 1441, 2024-04-30T08:45:45.330 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 1722932, 2024-04-30T08:45:45.330 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-30T08:45:45.330 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.330 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.331 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.331 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:45.331 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.331 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_bytes": { 2024-04-30T08:45:45.331 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-30T08:45:45.331 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 104857600, 2024-04-30T08:45:45.331 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-30T08:45:45.331 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-04-30T08:45:45.331 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-04-30T08:45:45.331 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:45.332 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-04-30T08:45:45.332 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 1088, 2024-04-30T08:45:45.332 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 25028, 2024-04-30T08:45:45.332 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 1088, 2024-04-30T08:45:45.332 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 25028, 2024-04-30T08:45:45.332 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-30T08:45:45.332 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.332 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.332 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.332 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:45.332 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.332 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-objecter_ops": { 2024-04-30T08:45:45.332 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-30T08:45:45.332 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 1024, 2024-04-30T08:45:45.333 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-30T08:45:45.333 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-04-30T08:45:45.333 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-04-30T08:45:45.333 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:45.333 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-04-30T08:45:45.333 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 1088, 2024-04-30T08:45:45.333 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 1088, 2024-04-30T08:45:45.333 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 1088, 2024-04-30T08:45:45.333 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 1088, 2024-04-30T08:45:45.333 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-30T08:45:45.333 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.334 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.334 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.334 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:45.334 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.334 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle": { 2024-04-30T08:45:45.334 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-30T08:45:45.334 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-04-30T08:45:45.334 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-30T08:45:45.334 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-04-30T08:45:45.334 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-04-30T08:45:45.334 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:45.335 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-04-30T08:45:45.335 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-04-30T08:45:45.335 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-04-30T08:45:45.335 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 0, 2024-04-30T08:45:45.335 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 0, 2024-04-30T08:45:45.335 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-30T08:45:45.335 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.335 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.335 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.335 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:45.335 INFO:teuthology.orchestra.run.smithi189.stdout: }, 2024-04-30T08:45:45.335 INFO:teuthology.orchestra.run.smithi189.stdout: "throttle-write_buf_throttle-0x556f86546410": { 2024-04-30T08:45:45.335 INFO:teuthology.orchestra.run.smithi189.stdout: "val": 0, 2024-04-30T08:45:45.335 INFO:teuthology.orchestra.run.smithi189.stdout: "max": 3758096384, 2024-04-30T08:45:45.335 INFO:teuthology.orchestra.run.smithi189.stdout: "get_started": 0, 2024-04-30T08:45:45.336 INFO:teuthology.orchestra.run.smithi189.stdout: "get": 0, 2024-04-30T08:45:45.336 INFO:teuthology.orchestra.run.smithi189.stdout: "get_sum": 0, 2024-04-30T08:45:45.336 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_fail": 0, 2024-04-30T08:45:45.336 INFO:teuthology.orchestra.run.smithi189.stdout: "get_or_fail_success": 0, 2024-04-30T08:45:45.336 INFO:teuthology.orchestra.run.smithi189.stdout: "take": 0, 2024-04-30T08:45:45.336 INFO:teuthology.orchestra.run.smithi189.stdout: "take_sum": 0, 2024-04-30T08:45:45.336 INFO:teuthology.orchestra.run.smithi189.stdout: "put": 0, 2024-04-30T08:45:45.336 INFO:teuthology.orchestra.run.smithi189.stdout: "put_sum": 0, 2024-04-30T08:45:45.336 INFO:teuthology.orchestra.run.smithi189.stdout: "wait": { 2024-04-30T08:45:45.336 INFO:teuthology.orchestra.run.smithi189.stdout: "avgcount": 0, 2024-04-30T08:45:45.336 INFO:teuthology.orchestra.run.smithi189.stdout: "sum": 0.000000000, 2024-04-30T08:45:45.336 INFO:teuthology.orchestra.run.smithi189.stdout: "avgtime": 0.000000000 2024-04-30T08:45:45.336 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:45.337 INFO:teuthology.orchestra.run.smithi189.stdout: } 2024-04-30T08:45:45.337 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-04-30T08:45:45.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:45 smithi071 ceph-mon[33385]: pgmap v742: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-30T08:45:45.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:45 smithi189 ceph-mon[33038]: pgmap v742: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-30T08:45:45.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:45 smithi032 ceph-mon[28331]: pgmap v742: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-04-30T08:45:46.148 INFO:tasks.check_counter:Daemon mds.l mds.exported=0 2024-04-30T08:45:46.148 INFO:tasks.check_counter:Daemon mds.l mds.imported=0 2024-04-30T08:45:46.148 INFO:tasks.check_counter:Daemon mds.l mds.dir_split=0 2024-04-30T08:45:46.148 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-30T08:45:46.159 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-30T08:45:46.159 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-30T08:45:46.159 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-30T08:45:46.234 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-30T08:45:46.234 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-30T08:45:46.290 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-30T08:45:46.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.587+0000 7fc0e3e00640 1 -- 172.21.15.32:0/3891236874 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0e4109fc0 msgr2=0x7fc0e410acc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:46.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.587+0000 7fc0e3e00640 1 --2- 172.21.15.32:0/3891236874 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0e4109fc0 0x7fc0e410acc0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4009be0 tx=0x7fc0d402a6c0 comp rx=0 tx=0).stop 2024-04-30T08:45:46.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.587+0000 7fc0e3e00640 1 -- 172.21.15.32:0/3891236874 shutdown_connections 2024-04-30T08:45:46.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.587+0000 7fc0e3e00640 1 --2- 172.21.15.32:0/3891236874 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0e410ff00 0x7fc0e4112320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:46.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.587+0000 7fc0e3e00640 1 --2- 172.21.15.32:0/3891236874 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0e410b200 0x7fc0e410b610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:46.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.587+0000 7fc0e3e00640 1 --2- 172.21.15.32:0/3891236874 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0e4109fc0 0x7fc0e410acc0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:46.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.587+0000 7fc0e3e00640 1 -- 172.21.15.32:0/3891236874 >> 172.21.15.32:0/3891236874 conn(0x7fc0e4078bb0 msgr2=0x7fc0e4107a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:45:46.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.587+0000 7fc0e3e00640 1 -- 172.21.15.32:0/3891236874 shutdown_connections 2024-04-30T08:45:46.589 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.587+0000 7fc0e3e00640 1 -- 172.21.15.32:0/3891236874 wait complete. 2024-04-30T08:45:46.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.588+0000 7fc0e3e00640 1 Processor -- start 2024-04-30T08:45:46.590 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.588+0000 7fc0e3e00640 1 -- start start 2024-04-30T08:45:46.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.589+0000 7fc0e3e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0e4109fc0 0x7fc0e41a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:46.591 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.589+0000 7fc0e3e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0e410b200 0x7fc0e41a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:46.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.589+0000 7fc0e3e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0e410ff00 0x7fc0e41a7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:46.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.589+0000 7fc0e3e00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e4115ae0 con 0x7fc0e410ff00 2024-04-30T08:45:46.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.589+0000 7fc0e3e00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e4115960 con 0x7fc0e4109fc0 2024-04-30T08:45:46.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.589+0000 7fc0e3e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc0e4115c60 con 0x7fc0e410b200 2024-04-30T08:45:46.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.589+0000 7fc0e2a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0e4109fc0 0x7fc0e41a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:46.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.589+0000 7fc0e2a00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0e4109fc0 0x7fc0e41a06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:49414/0 (socket says 172.21.15.32:49414) 2024-04-30T08:45:46.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.589+0000 7fc0e3400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0e410ff00 0x7fc0e41a7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:46.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.589+0000 7fc0e2000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0e410b200 0x7fc0e41a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:46.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.589+0000 7fc0e2a00640 1 -- 172.21.15.32:0/1918868184 learned_addr learned my addr 172.21.15.32:0/1918868184 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:45:46.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.590+0000 7fc0e2a00640 1 -- 172.21.15.32:0/1918868184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0e410b200 msgr2=0x7fc0e41a0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:46.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.590+0000 7fc0e2a00640 1 --2- 172.21.15.32:0/1918868184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0e410b200 0x7fc0e41a0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:46.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.590+0000 7fc0e2a00640 1 -- 172.21.15.32:0/1918868184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0e410ff00 msgr2=0x7fc0e41a7b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:46.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.590+0000 7fc0e2a00640 1 --2- 172.21.15.32:0/1918868184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0e410ff00 0x7fc0e41a7b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:46.592 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.590+0000 7fc0e2a00640 1 -- 172.21.15.32:0/1918868184 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0e41a8200 con 0x7fc0e4109fc0 2024-04-30T08:45:46.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.590+0000 7fc0e2a00640 1 --2- 172.21.15.32:0/1918868184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0e4109fc0 0x7fc0e41a06c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4000f80 tx=0x7fc0d4030ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:45:46.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.590+0000 7fc0db400640 1 -- 172.21.15.32:0/1918868184 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d4041020 con 0x7fc0e4109fc0 2024-04-30T08:45:46.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.591+0000 7fc0db400640 1 -- 172.21.15.32:0/1918868184 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc0d4030e90 con 0x7fc0e4109fc0 2024-04-30T08:45:46.593 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.591+0000 7fc0e3e00640 1 -- 172.21.15.32:0/1918868184 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0e41a8490 con 0x7fc0e4109fc0 2024-04-30T08:45:46.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.591+0000 7fc0db400640 1 -- 172.21.15.32:0/1918868184 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0d4036040 con 0x7fc0e4109fc0 2024-04-30T08:45:46.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.591+0000 7fc0e3e00640 1 -- 172.21.15.32:0/1918868184 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0e41a8980 con 0x7fc0e4109fc0 2024-04-30T08:45:46.594 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.592+0000 7fc0e3e00640 1 -- 172.21.15.32:0/1918868184 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0e410ed10 con 0x7fc0e4109fc0 2024-04-30T08:45:46.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.594+0000 7fc0db400640 1 -- 172.21.15.32:0/1918868184 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7fc0d4044020 con 0x7fc0e4109fc0 2024-04-30T08:45:46.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.596+0000 7fc0db400640 1 --2- 172.21.15.32:0/1918868184 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc0c80afae0 0x7fc0c80b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:46.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.596+0000 7fc0db400640 1 -- 172.21.15.32:0/1918868184 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fc0d40fffe0 con 0x7fc0e4109fc0 2024-04-30T08:45:46.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.596+0000 7fc0e2000640 1 --2- 172.21.15.32:0/1918868184 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc0c80afae0 0x7fc0c80b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:46.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.597+0000 7fc0e2000640 1 --2- 172.21.15.32:0/1918868184 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc0c80afae0 0x7fc0c80b1ff0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc0d00096c0 tx=0x7fc0d0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:45:46.600 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.598+0000 7fc0db400640 1 -- 172.21.15.32:0/1918868184 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc0d404b020 con 0x7fc0e4109fc0 2024-04-30T08:45:46.805 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.803+0000 7fc0e3e00640 1 -- 172.21.15.32:0/1918868184 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc0e4063820 con 0x7fc0e4109fc0 2024-04-30T08:45:46.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.806+0000 7fc0db400640 1 -- 172.21.15.32:0/1918868184 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+24730 (secure 0 0 0) 0x7fc0d40c87b0 con 0x7fc0e4109fc0 2024-04-30T08:45:46.808 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:45:46.808 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":85,"fsid":"98c0a670-06ca-11ef-bc93-c7b262605968","created":"2024-04-30T08:22:11.122875+0000","modified":"2024-04-30T08:33:31.872704+0000","last_up_change":"2024-04-30T08:30:50.350439+0000","last_in_change":"2024-04-30T08:30:30.210620+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-30T08:26:19.728331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-30T08:31:49.800542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-30T08:31:51.211320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-30T08:31:55.716625+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a7e060c8-1cbb-418f-9a03-06693f6b3f52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6803","nonce":2624700402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6805","nonce":2624700402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6809","nonce":2624700402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6807","nonce":2624700402}]},"public_addr":"172.21.15.32:6803/2624700402","cluster_addr":"172.21.15.32:6805/2624700402","heartbeat_back_addr":"172.21.15.32:6809/2624700402","heartbeat_front_addr":"172.21.15.32:6807/2624700402","state":["exists","up"]},{"osd":1,"uuid":"72ef2fb1-e5ca-48ef-aa3a-4796a64b922f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6801","nonce":444649711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6803","nonce":444649711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6807","nonce":444649711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6805","nonce":444649711}]},"public_addr":"172.21.15.71:6801/444649711","cluster_addr":"172.21.15.71:6803/444649711","heartbeat_back_addr":"172.21.15.71:6807/444649711","heartbeat_front_addr":"172.21.15.71:6805/444649711","state":["exists","up"]},{"osd":2,"uuid":"704e5e8a-935e-47c7-ae28-27119876ac09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6801","nonce":3255528671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6803","nonce":3255528671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6807","nonce":3255528671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6805","nonce":3255528671}]},"public_addr":"172.21.15.189:6801/3255528671","cluster_addr":"172.21.15.189:6803/3255528671","heartbeat_back_addr":"172.21.15.189:6807/3255528671","heartbeat_front_addr":"172.21.15.189:6805/3255528671","state":["exists","up"]},{"osd":3,"uuid":"575004f7-6559-4a62-863e-e33078b0eebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6811","nonce":1384702112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6813","nonce":1384702112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6817","nonce":1384702112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6815","nonce":1384702112}]},"public_addr":"172.21.15.32:6811/1384702112","cluster_addr":"172.21.15.32:6813/1384702112","heartbeat_back_addr":"172.21.15.32:6817/1384702112","heartbeat_front_addr":"172.21.15.32:6815/1384702112","state":["exists","up"]},{"osd":4,"uuid":"e1c1e894-074d-4c78-89ef-9e5a114268bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6809","nonce":4250534574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6811","nonce":4250534574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6815","nonce":4250534574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6813","nonce":4250534574}]},"public_addr":"172.21.15.71:6809/4250534574","cluster_addr":"172.21.15.71:6811/4250534574","heartbeat_back_addr":"172.21.15.71:6815/4250534574","heartbeat_front_addr":"172.21.15.71:6813/4250534574","state":["exists","up"]},{"osd":5,"uuid":"12c022d4-fac6-40c4-ae7a-691571bee5d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6809","nonce":4245346198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6811","nonce":4245346198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6815","nonce":4245346198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6813","nonce":4245346198}]},"public_addr":"172.21.15.189:6809/4245346198","cluster_addr":"172.21.15.189:6811/4245346198","heartbeat_back_addr":"172.21.15.189:6815/4245346198","heartbeat_front_addr":"172.21.15.189:6813/4245346198","state":["exists","up"]},{"osd":6,"uuid":"b413f8ee-3d4a-415a-a2ba-d68437b7330d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6819","nonce":1910389190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6821","nonce":1910389190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6825","nonce":1910389190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6823","nonce":1910389190}]},"public_addr":"172.21.15.32:6819/1910389190","cluster_addr":"172.21.15.32:6821/1910389190","heartbeat_back_addr":"172.21.15.32:6825/1910389190","heartbeat_front_addr":"172.21.15.32:6823/1910389190","state":["exists","up"]},{"osd":7,"uuid":"aac64fb3-fc13-4126-b842-7b06ad5ce238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6817","nonce":692607380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6819","nonce":692607380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6823","nonce":692607380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6821","nonce":692607380}]},"public_addr":"172.21.15.71:6817/692607380","cluster_addr":"172.21.15.71:6819/692607380","heartbeat_back_addr":"172.21.15.71:6823/692607380","heartbeat_front_addr":"172.21.15.71:6821/692607380","state":["exists","up"]},{"osd":8,"uuid":"cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6817","nonce":3973697069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6819","nonce":3973697069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6823","nonce":3973697069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6821","nonce":3973697069}]},"public_addr":"172.21.15.189:6817/3973697069","cluster_addr":"172.21.15.189:6819/3973697069","heartbeat_back_addr":"172.21.15.189:6823/3973697069","heartbeat_front_addr":"172.21.15.189:6821/3973697069","state":["exists","up"]},{"osd":9,"uuid":"ef4a1645-ae6e-45c3-82ad-161510e62aa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6827","nonce":46066685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6829","nonce":46066685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6833","nonce":46066685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6831","nonce":46066685}]},"public_addr":"172.21.15.32:6827/46066685","cluster_addr":"172.21.15.32:6829/46066685","heartbeat_back_addr":"172.21.15.32:6833/46066685","heartbeat_front_addr":"172.21.15.32:6831/46066685","state":["exists","up"]},{"osd":10,"uuid":"25b26a38-a644-409f-b941-59efdf04a3ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6825","nonce":1886290293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6827","nonce":1886290293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6831","nonce":1886290293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6829","nonce":1886290293}]},"public_addr":"172.21.15.71:6825/1886290293","cluster_addr":"172.21.15.71:6827/1886290293","heartbeat_back_addr":"172.21.15.71:6831/1886290293","heartbeat_front_addr":"172.21.15.71:6829/1886290293","state":["exists","up"]},{"osd":11,"uuid":"ab96b3ff-3688-455b-af64-60226c5bfbff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6825","nonce":850185001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6827","nonce":850185001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6831","nonce":850185001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6829","nonce":850185001}]},"public_addr":"172.21.15.189:6825/850185001","cluster_addr":"172.21.15.189:6827/850185001","heartbeat_back_addr":"172.21.15.189:6831/850185001","heartbeat_front_addr":"172.21.15.189:6829/850185001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:27.220308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:53.230783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:17.190642+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:52.603799+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:20.568671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:46.158363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:22.706933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:50.665173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:16.602410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:53.814698+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:22.435189+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:48.651949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.e","mappings":[{"from":4,"to":7}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.28","mappings":[{"from":3,"to":6},{"from":5,"to":8}]},{"pgid":"3.30","mappings":[{"from":1,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1266897222":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/2915689533":"2024-05-01T08:23:29.526664+0000","172.21.15.32:6801/4019351306":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/425836193":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/1134075722":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6801/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/21059253":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/3245550894":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/4019351306":"2024-05-01T08:23:29.526664+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-30T08:45:46.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.808+0000 7fc0e3e00640 1 -- 172.21.15.32:0/1918868184 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc0c80afae0 msgr2=0x7fc0c80b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:46.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.808+0000 7fc0e3e00640 1 --2- 172.21.15.32:0/1918868184 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc0c80afae0 0x7fc0c80b1ff0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc0d00096c0 tx=0x7fc0d0009260 comp rx=0 tx=0).stop 2024-04-30T08:45:46.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.808+0000 7fc0e3e00640 1 -- 172.21.15.32:0/1918868184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0e4109fc0 msgr2=0x7fc0e41a06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:46.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.808+0000 7fc0e3e00640 1 --2- 172.21.15.32:0/1918868184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0e4109fc0 0x7fc0e41a06c0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4000f80 tx=0x7fc0d4030ca0 comp rx=0 tx=0).stop 2024-04-30T08:45:46.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.809+0000 7fc0e3e00640 1 -- 172.21.15.32:0/1918868184 shutdown_connections 2024-04-30T08:45:46.810 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.809+0000 7fc0e3e00640 1 --2- 172.21.15.32:0/1918868184 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc0c80afae0 0x7fc0c80b1ff0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:46.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.809+0000 7fc0e3e00640 1 --2- 172.21.15.32:0/1918868184 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc0e410ff00 0x7fc0e41a7b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:46.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.809+0000 7fc0e3e00640 1 --2- 172.21.15.32:0/1918868184 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc0e410b200 0x7fc0e41a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:46.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.809+0000 7fc0e3e00640 1 --2- 172.21.15.32:0/1918868184 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc0e4109fc0 0x7fc0e41a06c0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:46.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.809+0000 7fc0e3e00640 1 -- 172.21.15.32:0/1918868184 >> 172.21.15.32:0/1918868184 conn(0x7fc0e4078bb0 msgr2=0x7fc0e410bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:45:46.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.809+0000 7fc0e3e00640 1 -- 172.21.15.32:0/1918868184 shutdown_connections 2024-04-30T08:45:46.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:46.809+0000 7fc0e3e00640 1 -- 172.21.15.32:0/1918868184 wait complete. 2024-04-30T08:45:46.837 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-30T08:26:19.728331+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-30T08:31:49.800542+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '75', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-30T08:31:51.211320+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '76', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-04-30T08:31:55.716625+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-04-30T08:45:46.838 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-30T08:45:47.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.120+0000 7f1eaec00640 1 -- 172.21.15.32:0/2663147273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ea810b900 msgr2=0x7f1ea810dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:47.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.120+0000 7f1eaec00640 1 --2- 172.21.15.32:0/2663147273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ea810b900 0x7f1ea810dd40 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0009ca0 tx=0x7f1ea002a8c0 comp rx=0 tx=0).stop 2024-04-30T08:45:47.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.120+0000 7f1eaec00640 1 -- 172.21.15.32:0/2663147273 shutdown_connections 2024-04-30T08:45:47.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.120+0000 7f1eaec00640 1 --2- 172.21.15.32:0/2663147273 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ea810b900 0x7f1ea810dd40 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:47.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.120+0000 7f1eaec00640 1 --2- 172.21.15.32:0/2663147273 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ea8075960 0x7f1ea810b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:47.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.120+0000 7f1eaec00640 1 --2- 172.21.15.32:0/2663147273 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea8075010 0x7f1ea8075420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:47.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.120+0000 7f1eaec00640 1 -- 172.21.15.32:0/2663147273 >> 172.21.15.32:0/2663147273 conn(0x7f1ea80feba0 msgr2=0x7f1ea8101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:45:47.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.120+0000 7f1eaec00640 1 -- 172.21.15.32:0/2663147273 shutdown_connections 2024-04-30T08:45:47.122 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.120+0000 7f1eaec00640 1 -- 172.21.15.32:0/2663147273 wait complete. 2024-04-30T08:45:47.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.121+0000 7f1eaec00640 1 Processor -- start 2024-04-30T08:45:47.124 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.121+0000 7f1eaec00640 1 -- start start 2024-04-30T08:45:47.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.121+0000 7f1eaec00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ea8075010 0x7f1ea819c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:47.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.121+0000 7f1eaec00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea8075960 0x7f1ea819c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:47.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.121+0000 7f1eaec00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ea810b900 0x7f1ea81a37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:47.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.121+0000 7f1eaec00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea81116a0 con 0x7f1ea8075010 2024-04-30T08:45:47.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.121+0000 7f1eaec00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea8111520 con 0x7f1ea8075960 2024-04-30T08:45:47.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.121+0000 7f1eaec00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f1ea8111820 con 0x7f1ea810b900 2024-04-30T08:45:47.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.122+0000 7f1ead800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ea8075010 0x7f1ea819c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:47.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.122+0000 7f1ead800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ea8075010 0x7f1ea819c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58310/0 (socket says 172.21.15.32:58310) 2024-04-30T08:45:47.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.122+0000 7f1ead800640 1 -- 172.21.15.32:0/4249337263 learned_addr learned my addr 172.21.15.32:0/4249337263 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:45:47.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.122+0000 7f1eae200640 1 --2- 172.21.15.32:0/4249337263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ea810b900 0x7f1ea81a37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:47.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.122+0000 7f1eace00640 1 --2- 172.21.15.32:0/4249337263 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea8075960 0x7f1ea819c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:47.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.122+0000 7f1ead800640 1 -- 172.21.15.32:0/4249337263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ea810b900 msgr2=0x7f1ea81a37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:47.125 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.122+0000 7f1ead800640 1 --2- 172.21.15.32:0/4249337263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ea810b900 0x7f1ea81a37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:47.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.122+0000 7f1ead800640 1 -- 172.21.15.32:0/4249337263 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea8075960 msgr2=0x7f1ea819c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:47.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.122+0000 7f1ead800640 1 --2- 172.21.15.32:0/4249337263 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea8075960 0x7f1ea819c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:47.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.122+0000 7f1ead800640 1 -- 172.21.15.32:0/4249337263 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ea81a3e10 con 0x7f1ea8075010 2024-04-30T08:45:47.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.122+0000 7f1ead800640 1 --2- 172.21.15.32:0/4249337263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ea8075010 0x7f1ea819c2d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c00e9c0 tx=0x7f1e9c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:45:47.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.123+0000 7f1ea6000640 1 -- 172.21.15.32:0/4249337263 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c00cd80 con 0x7f1ea8075010 2024-04-30T08:45:47.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.123+0000 7f1ea6000640 1 -- 172.21.15.32:0/4249337263 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f1e9c0054d0 con 0x7f1ea8075010 2024-04-30T08:45:47.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.123+0000 7f1ea6000640 1 -- 172.21.15.32:0/4249337263 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e9c010430 con 0x7f1ea8075010 2024-04-30T08:45:47.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.123+0000 7f1eaec00640 1 -- 172.21.15.32:0/4249337263 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ea81a4100 con 0x7f1ea8075010 2024-04-30T08:45:47.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.123+0000 7f1eaec00640 1 -- 172.21.15.32:0/4249337263 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ea81a4650 con 0x7f1ea8075010 2024-04-30T08:45:47.126 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.124+0000 7f1eaec00640 1 -- 172.21.15.32:0/4249337263 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ea810aac0 con 0x7f1ea8075010 2024-04-30T08:45:47.127 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.126+0000 7f1ea6000640 1 -- 172.21.15.32:0/4249337263 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f1e9c004fd0 con 0x7f1ea8075010 2024-04-30T08:45:47.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.128+0000 7f1ea6000640 1 --2- 172.21.15.32:0/4249337263 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f1e900afae0 0x7f1e900b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:47.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.128+0000 7f1ea6000640 1 -- 172.21.15.32:0/4249337263 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f1e9c0df400 con 0x7f1ea8075010 2024-04-30T08:45:47.129 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.128+0000 7f1eace00640 1 --2- 172.21.15.32:0/4249337263 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f1e900afae0 0x7f1e900b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:47.130 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.129+0000 7f1eace00640 1 --2- 172.21.15.32:0/4249337263 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f1e900afae0 0x7f1e900b1ff0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1ea819d370 tx=0x7f1e9400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:45:47.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.133+0000 7f1ea6000640 1 -- 172.21.15.32:0/4249337263 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1e9c0a7bd0 con 0x7f1ea8075010 2024-04-30T08:45:47.333 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.331+0000 7f1eaec00640 1 -- 172.21.15.32:0/4249337263 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1ea8063820 con 0x7f1ea8075010 2024-04-30T08:45:47.334 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.332+0000 7f1ea6000640 1 -- 172.21.15.32:0/4249337263 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v85) ==== 93+0+10 (secure 0 0 0) 0x7f1e9c0a74d0 con 0x7f1ea8075010 2024-04-30T08:45:47.334 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 1 2024-04-30T08:45:47.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.334+0000 7f1eaec00640 1 -- 172.21.15.32:0/4249337263 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f1e900afae0 msgr2=0x7f1e900b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:47.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.334+0000 7f1eaec00640 1 --2- 172.21.15.32:0/4249337263 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f1e900afae0 0x7f1e900b1ff0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f1ea819d370 tx=0x7f1e9400b390 comp rx=0 tx=0).stop 2024-04-30T08:45:47.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.334+0000 7f1eaec00640 1 -- 172.21.15.32:0/4249337263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ea8075010 msgr2=0x7f1ea819c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:47.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.334+0000 7f1eaec00640 1 --2- 172.21.15.32:0/4249337263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ea8075010 0x7f1ea819c2d0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f1e9c00e9c0 tx=0x7f1e9c00ee80 comp rx=0 tx=0).stop 2024-04-30T08:45:47.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.335+0000 7f1eaec00640 1 -- 172.21.15.32:0/4249337263 shutdown_connections 2024-04-30T08:45:47.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.335+0000 7f1eaec00640 1 --2- 172.21.15.32:0/4249337263 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f1e900afae0 0x7f1e900b1ff0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:47.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.335+0000 7f1eaec00640 1 --2- 172.21.15.32:0/4249337263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ea810b900 0x7f1ea81a37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:47.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.335+0000 7f1eaec00640 1 --2- 172.21.15.32:0/4249337263 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f1ea8075960 0x7f1ea819c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:47.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.335+0000 7f1eaec00640 1 --2- 172.21.15.32:0/4249337263 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f1ea8075010 0x7f1ea819c2d0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:47.337 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.335+0000 7f1eaec00640 1 -- 172.21.15.32:0/4249337263 >> 172.21.15.32:0/4249337263 conn(0x7f1ea80feba0 msgr2=0x7f1ea80ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:45:47.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.335+0000 7f1eaec00640 1 -- 172.21.15.32:0/4249337263 shutdown_connections 2024-04-30T08:45:47.338 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.335+0000 7f1eaec00640 1 -- 172.21.15.32:0/4249337263 wait complete. 2024-04-30T08:45:47.363 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-30T08:45:47.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:47 smithi071 ceph-mon[33385]: pgmap v743: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-30T08:45:47.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:47 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1918868184' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:45:47.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:47 smithi189 ceph-mon[33038]: pgmap v743: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-30T08:45:47.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:47 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1918868184' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:45:47.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.646+0000 7f4c95200640 1 -- 172.21.15.32:0/4270934299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c90075010 msgr2=0x7f4c90075420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:47.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.646+0000 7f4c95200640 1 --2- 172.21.15.32:0/4270934299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c90075010 0x7f4c90075420 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f4c80009900 tx=0x7f4c8002a410 comp rx=0 tx=0).stop 2024-04-30T08:45:47.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.646+0000 7f4c95200640 1 -- 172.21.15.32:0/4270934299 shutdown_connections 2024-04-30T08:45:47.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.646+0000 7f4c95200640 1 --2- 172.21.15.32:0/4270934299 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c9010b900 0x7f4c9010dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:47.648 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.646+0000 7f4c95200640 1 --2- 172.21.15.32:0/4270934299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c90075960 0x7f4c9010b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:47.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.646+0000 7f4c95200640 1 --2- 172.21.15.32:0/4270934299 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c90075010 0x7f4c90075420 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:47.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.646+0000 7f4c95200640 1 -- 172.21.15.32:0/4270934299 >> 172.21.15.32:0/4270934299 conn(0x7f4c900feba0 msgr2=0x7f4c90101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:45:47.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.646+0000 7f4c95200640 1 -- 172.21.15.32:0/4270934299 shutdown_connections 2024-04-30T08:45:47.649 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.646+0000 7f4c95200640 1 -- 172.21.15.32:0/4270934299 wait complete. 2024-04-30T08:45:47.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.647+0000 7f4c95200640 1 Processor -- start 2024-04-30T08:45:47.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.647+0000 7f4c95200640 1 -- start start 2024-04-30T08:45:47.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.648+0000 7f4c95200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c90075010 0x7f4c9019c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:47.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.648+0000 7f4c95200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c90075960 0x7f4c9019c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:47.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.648+0000 7f4c95200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c9010b900 0x7f4c901a3810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:47.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.648+0000 7f4c95200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f4c90111710 con 0x7f4c9010b900 2024-04-30T08:45:47.651 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.648+0000 7f4c95200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f4c90111590 con 0x7f4c90075010 2024-04-30T08:45:47.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.648+0000 7f4c95200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f4c90111890 con 0x7f4c90075960 2024-04-30T08:45:47.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.648+0000 7f4c8fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c9010b900 0x7f4c901a3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:47.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.648+0000 7f4c8fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c9010b900 0x7f4c901a3810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58332/0 (socket says 172.21.15.32:58332) 2024-04-30T08:45:47.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.648+0000 7f4c8ea00640 1 --2- 172.21.15.32:0/1686981274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c90075960 0x7f4c9019c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:47.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.648+0000 7f4c8f400640 1 --2- 172.21.15.32:0/1686981274 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c90075010 0x7f4c9019c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:47.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.648+0000 7f4c8fe00640 1 -- 172.21.15.32:0/1686981274 learned_addr learned my addr 172.21.15.32:0/1686981274 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:45:47.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.649+0000 7f4c8fe00640 1 -- 172.21.15.32:0/1686981274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c90075960 msgr2=0x7f4c9019c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:47.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.649+0000 7f4c8fe00640 1 --2- 172.21.15.32:0/1686981274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c90075960 0x7f4c9019c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:47.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.649+0000 7f4c8fe00640 1 -- 172.21.15.32:0/1686981274 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c90075010 msgr2=0x7f4c9019c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:47.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.649+0000 7f4c8fe00640 1 --2- 172.21.15.32:0/1686981274 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c90075010 0x7f4c9019c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:47.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.649+0000 7f4c8fe00640 1 -- 172.21.15.32:0/1686981274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c901a3e80 con 0x7f4c9010b900 2024-04-30T08:45:47.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.649+0000 7f4c8fe00640 1 --2- 172.21.15.32:0/1686981274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c9010b900 0x7f4c901a3810 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4c8800dac0 tx=0x7f4c8800df80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:45:47.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.649+0000 7f4c87e00640 1 -- 172.21.15.32:0/1686981274 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c88017070 con 0x7f4c9010b900 2024-04-30T08:45:47.652 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.649+0000 7f4c87e00640 1 -- 172.21.15.32:0/1686981274 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f4c8800be30 con 0x7f4c9010b900 2024-04-30T08:45:47.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.649+0000 7f4c95200640 1 -- 172.21.15.32:0/1686981274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c901a4170 con 0x7f4c9010b900 2024-04-30T08:45:47.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.649+0000 7f4c87e00640 1 -- 172.21.15.32:0/1686981274 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c88010460 con 0x7f4c9010b900 2024-04-30T08:45:47.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.650+0000 7f4c95200640 1 -- 172.21.15.32:0/1686981274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c901a4670 con 0x7f4c9010b900 2024-04-30T08:45:47.653 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.651+0000 7f4c95200640 1 -- 172.21.15.32:0/1686981274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4c9010aac0 con 0x7f4c9010b900 2024-04-30T08:45:47.655 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.653+0000 7f4c87e00640 1 -- 172.21.15.32:0/1686981274 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f4c880040b0 con 0x7f4c9010b900 2024-04-30T08:45:47.656 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.655+0000 7f4c87e00640 1 --2- 172.21.15.32:0/1686981274 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4c740afae0 0x7f4c740b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:47.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.655+0000 7f4c87e00640 1 -- 172.21.15.32:0/1686981274 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f4c880deca0 con 0x7f4c9010b900 2024-04-30T08:45:47.657 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.656+0000 7f4c8f400640 1 --2- 172.21.15.32:0/1686981274 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4c740afae0 0x7f4c740b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:47.658 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.657+0000 7f4c8f400640 1 --2- 172.21.15.32:0/1686981274 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4c740afae0 0x7f4c740b1ff0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f4c80031440 tx=0x7f4c800314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:45:47.659 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.658+0000 7f4c87e00640 1 -- 172.21.15.32:0/1686981274 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4c880a6e20 con 0x7f4c9010b900 2024-04-30T08:45:47.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:47 smithi032 ceph-mon[28331]: pgmap v743: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 11 op/s 2024-04-30T08:45:47.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:47 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1918868184' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:45:47.857 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.855+0000 7f4c95200640 1 -- 172.21.15.32:0/1686981274 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4c9002dd60 con 0x7f4c9010b900 2024-04-30T08:45:47.858 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.856+0000 7f4c87e00640 1 -- 172.21.15.32:0/1686981274 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v85) ==== 104+0+11 (secure 0 0 0) 0x7f4c880ab120 con 0x7f4c9010b900 2024-04-30T08:45:47.858 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-30T08:45:47.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.858+0000 7f4c95200640 1 -- 172.21.15.32:0/1686981274 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4c740afae0 msgr2=0x7f4c740b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:47.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.858+0000 7f4c95200640 1 --2- 172.21.15.32:0/1686981274 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4c740afae0 0x7f4c740b1ff0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f4c80031440 tx=0x7f4c800314a0 comp rx=0 tx=0).stop 2024-04-30T08:45:47.860 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.858+0000 7f4c95200640 1 -- 172.21.15.32:0/1686981274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c9010b900 msgr2=0x7f4c901a3810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:47.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.858+0000 7f4c95200640 1 --2- 172.21.15.32:0/1686981274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c9010b900 0x7f4c901a3810 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f4c8800dac0 tx=0x7f4c8800df80 comp rx=0 tx=0).stop 2024-04-30T08:45:47.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.859+0000 7f4c95200640 1 -- 172.21.15.32:0/1686981274 shutdown_connections 2024-04-30T08:45:47.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.859+0000 7f4c95200640 1 --2- 172.21.15.32:0/1686981274 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f4c740afae0 0x7f4c740b1ff0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:47.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.859+0000 7f4c95200640 1 --2- 172.21.15.32:0/1686981274 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f4c9010b900 0x7f4c901a3810 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:47.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.859+0000 7f4c95200640 1 --2- 172.21.15.32:0/1686981274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4c90075960 0x7f4c9019c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:47.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.859+0000 7f4c95200640 1 --2- 172.21.15.32:0/1686981274 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f4c90075010 0x7f4c9019c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:47.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.859+0000 7f4c95200640 1 -- 172.21.15.32:0/1686981274 >> 172.21.15.32:0/1686981274 conn(0x7f4c900feba0 msgr2=0x7f4c900ff9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:45:47.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.859+0000 7f4c95200640 1 -- 172.21.15.32:0/1686981274 shutdown_connections 2024-04-30T08:45:47.861 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:47.859+0000 7f4c95200640 1 -- 172.21.15.32:0/1686981274 wait complete. 2024-04-30T08:45:47.887 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-30T08:45:48.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.166+0000 7f0b42400640 1 -- 172.21.15.32:0/2189161010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b3c109aa0 msgr2=0x7f0b3c10bec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:48.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.166+0000 7f0b42400640 1 --2- 172.21.15.32:0/2189161010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b3c109aa0 0x7f0b3c10bec0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f0b34009900 tx=0x7f0b3402a410 comp rx=0 tx=0).stop 2024-04-30T08:45:48.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.166+0000 7f0b42400640 1 -- 172.21.15.32:0/2189161010 shutdown_connections 2024-04-30T08:45:48.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.166+0000 7f0b42400640 1 --2- 172.21.15.32:0/2189161010 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b3c109aa0 0x7f0b3c10bec0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:48.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.166+0000 7f0b42400640 1 --2- 172.21.15.32:0/2189161010 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b3c104da0 0x7f0b3c1051b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:48.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.166+0000 7f0b42400640 1 --2- 172.21.15.32:0/2189161010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b3c103b60 0x7f0b3c104860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:48.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.166+0000 7f0b42400640 1 -- 172.21.15.32:0/2189161010 >> 172.21.15.32:0/2189161010 conn(0x7f0b3c0fed50 msgr2=0x7f0b3c101200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:45:48.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.167+0000 7f0b42400640 1 -- 172.21.15.32:0/2189161010 shutdown_connections 2024-04-30T08:45:48.169 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.167+0000 7f0b42400640 1 -- 172.21.15.32:0/2189161010 wait complete. 2024-04-30T08:45:48.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.168+0000 7f0b42400640 1 Processor -- start 2024-04-30T08:45:48.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.168+0000 7f0b42400640 1 -- start start 2024-04-30T08:45:48.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.168+0000 7f0b42400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b3c103b60 0x7f0b3c19a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:48.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.168+0000 7f0b42400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b3c104da0 0x7f0b3c19a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:48.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.168+0000 7f0b42400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b3c109aa0 0x7f0b3c1a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:48.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.168+0000 7f0b42400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3c10f6e0 con 0x7f0b3c103b60 2024-04-30T08:45:48.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.168+0000 7f0b42400640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3c10f560 con 0x7f0b3c109aa0 2024-04-30T08:45:48.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.168+0000 7f0b42400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3c10f860 con 0x7f0b3c104da0 2024-04-30T08:45:48.171 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.169+0000 7f0b3aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b3c104da0 0x7f0b3c19a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:48.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.169+0000 7f0b3b400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b3c103b60 0x7f0b3c19a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:48.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.169+0000 7f0b3aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b3c104da0 0x7f0b3c19a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:56702/0 (socket says 172.21.15.32:56702) 2024-04-30T08:45:48.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.169+0000 7f0b3aa00640 1 -- 172.21.15.32:0/1544856469 learned_addr learned my addr 172.21.15.32:0/1544856469 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:45:48.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.169+0000 7f0b3be00640 1 --2- 172.21.15.32:0/1544856469 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b3c109aa0 0x7f0b3c1a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:48.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.169+0000 7f0b3aa00640 1 -- 172.21.15.32:0/1544856469 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b3c109aa0 msgr2=0x7f0b3c1a1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:48.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.169+0000 7f0b3aa00640 1 --2- 172.21.15.32:0/1544856469 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b3c109aa0 0x7f0b3c1a1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:48.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.169+0000 7f0b3aa00640 1 -- 172.21.15.32:0/1544856469 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b3c103b60 msgr2=0x7f0b3c19a250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:48.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.169+0000 7f0b3aa00640 1 --2- 172.21.15.32:0/1544856469 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b3c103b60 0x7f0b3c19a250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:48.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.169+0000 7f0b3aa00640 1 -- 172.21.15.32:0/1544856469 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b3c1a1d90 con 0x7f0b3c104da0 2024-04-30T08:45:48.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.169+0000 7f0b3b400640 1 --2- 172.21.15.32:0/1544856469 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b3c103b60 0x7f0b3c19a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:45:48.172 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.169+0000 7f0b3aa00640 1 --2- 172.21.15.32:0/1544856469 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b3c104da0 0x7f0b3c19a790 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0b2400ebc0 tx=0x7f0b2400c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:45:48.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.170+0000 7f0b33e00640 1 -- 172.21.15.32:0/1544856469 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b24010630 con 0x7f0b3c104da0 2024-04-30T08:45:48.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.170+0000 7f0b42400640 1 -- 172.21.15.32:0/1544856469 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b3c1a2080 con 0x7f0b3c104da0 2024-04-30T08:45:48.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.170+0000 7f0b42400640 1 -- 172.21.15.32:0/1544856469 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b3c1a2620 con 0x7f0b3c104da0 2024-04-30T08:45:48.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.170+0000 7f0b33e00640 1 -- 172.21.15.32:0/1544856469 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f0b24010c80 con 0x7f0b3c104da0 2024-04-30T08:45:48.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.170+0000 7f0b33e00640 1 -- 172.21.15.32:0/1544856469 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b24018aa0 con 0x7f0b3c104da0 2024-04-30T08:45:48.173 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.171+0000 7f0b42400640 1 -- 172.21.15.32:0/1544856469 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b18005290 con 0x7f0b3c104da0 2024-04-30T08:45:48.176 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.174+0000 7f0b33e00640 1 -- 172.21.15.32:0/1544856469 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f0b240107d0 con 0x7f0b3c104da0 2024-04-30T08:45:48.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.176+0000 7f0b33e00640 1 --2- 172.21.15.32:0/1544856469 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f0b1c0afaf0 0x7f0b1c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:48.177 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.176+0000 7f0b33e00640 1 -- 172.21.15.32:0/1544856469 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f0b240dfbb0 con 0x7f0b3c104da0 2024-04-30T08:45:48.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.176+0000 7f0b3b400640 1 --2- 172.21.15.32:0/1544856469 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f0b1c0afaf0 0x7f0b1c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:48.178 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.177+0000 7f0b3b400640 1 --2- 172.21.15.32:0/1544856469 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f0b1c0afaf0 0x7f0b1c0b2000 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c104760 tx=0x7f0b2c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:45:48.179 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.178+0000 7f0b33e00640 1 -- 172.21.15.32:0/1544856469 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0b240a7d30 con 0x7f0b3c104da0 2024-04-30T08:45:48.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.394+0000 7f0b42400640 1 -- 172.21.15.32:0/1544856469 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f0b18005850 con 0x7f0b3c104da0 2024-04-30T08:45:48.397 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.395+0000 7f0b33e00640 1 -- 172.21.15.32:0/1544856469 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v85) ==== 100+0+11 (secure 0 0 0) 0x7f0b240ac030 con 0x7f0b3c104da0 2024-04-30T08:45:48.397 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 64 2024-04-30T08:45:48.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.399+0000 7f0b42400640 1 -- 172.21.15.32:0/1544856469 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f0b1c0afaf0 msgr2=0x7f0b1c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:48.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.399+0000 7f0b42400640 1 --2- 172.21.15.32:0/1544856469 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f0b1c0afaf0 0x7f0b1c0b2000 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f0b3c104760 tx=0x7f0b2c005f90 comp rx=0 tx=0).stop 2024-04-30T08:45:48.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.399+0000 7f0b42400640 1 -- 172.21.15.32:0/1544856469 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b3c104da0 msgr2=0x7f0b3c19a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:48.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.399+0000 7f0b42400640 1 --2- 172.21.15.32:0/1544856469 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b3c104da0 0x7f0b3c19a790 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0b2400ebc0 tx=0x7f0b2400c6a0 comp rx=0 tx=0).stop 2024-04-30T08:45:48.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.399+0000 7f0b42400640 1 -- 172.21.15.32:0/1544856469 shutdown_connections 2024-04-30T08:45:48.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.399+0000 7f0b42400640 1 --2- 172.21.15.32:0/1544856469 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f0b1c0afaf0 0x7f0b1c0b2000 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:48.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.399+0000 7f0b42400640 1 --2- 172.21.15.32:0/1544856469 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f0b3c109aa0 0x7f0b3c1a1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:48.401 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.399+0000 7f0b42400640 1 --2- 172.21.15.32:0/1544856469 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b3c104da0 0x7f0b3c19a790 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:48.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.399+0000 7f0b42400640 1 --2- 172.21.15.32:0/1544856469 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f0b3c103b60 0x7f0b3c19a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:48.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.399+0000 7f0b42400640 1 -- 172.21.15.32:0/1544856469 >> 172.21.15.32:0/1544856469 conn(0x7f0b3c0fed50 msgr2=0x7f0b3c105a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:45:48.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.399+0000 7f0b42400640 1 -- 172.21.15.32:0/1544856469 shutdown_connections 2024-04-30T08:45:48.402 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.399+0000 7f0b42400640 1 -- 172.21.15.32:0/1544856469 wait complete. 2024-04-30T08:45:48.427 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-04-30T08:45:48.494 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:48 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/4249337263' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:45:48.494 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:48 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1686981274' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-30T08:45:48.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:48 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/4249337263' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:45:48.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:48 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1686981274' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-30T08:45:48.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:48 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/4249337263' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-30T08:45:48.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:48 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1686981274' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-30T08:45:48.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.703+0000 7f854f200640 1 -- 172.21.15.32:0/510117439 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8550103990 msgr2=0x7f8550104690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:48.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.703+0000 7f854f200640 1 --2- 172.21.15.32:0/510117439 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8550103990 0x7f8550104690 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f8540009900 tx=0x7f854002a3f0 comp rx=0 tx=0).stop 2024-04-30T08:45:48.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.703+0000 7f854f200640 1 -- 172.21.15.32:0/510117439 shutdown_connections 2024-04-30T08:45:48.705 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.703+0000 7f854f200640 1 --2- 172.21.15.32:0/510117439 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85501098d0 0x7f855010bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:48.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.703+0000 7f854f200640 1 --2- 172.21.15.32:0/510117439 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8550104bd0 0x7f8550104fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:48.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.703+0000 7f854f200640 1 --2- 172.21.15.32:0/510117439 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8550103990 0x7f8550104690 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:48.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.703+0000 7f854f200640 1 -- 172.21.15.32:0/510117439 >> 172.21.15.32:0/510117439 conn(0x7f85500feba0 msgr2=0x7f8550101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:45:48.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.703+0000 7f854f200640 1 -- 172.21.15.32:0/510117439 shutdown_connections 2024-04-30T08:45:48.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.703+0000 7f854f200640 1 -- 172.21.15.32:0/510117439 wait complete. 2024-04-30T08:45:48.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.704+0000 7f854f200640 1 Processor -- start 2024-04-30T08:45:48.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.704+0000 7f854f200640 1 -- start start 2024-04-30T08:45:48.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.705+0000 7f854f200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8550103990 0x7f85500768d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:48.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.705+0000 7f854f200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8550104bd0 0x7f8550074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:48.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.705+0000 7f854f200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85501098d0 0x7f85500754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:48.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.705+0000 7f854f200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f855010f4c0 con 0x7f85501098d0 2024-04-30T08:45:48.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.705+0000 7f854f200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f855010f340 con 0x7f8550103990 2024-04-30T08:45:48.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.705+0000 7f854f200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f855010f640 con 0x7f8550104bd0 2024-04-30T08:45:48.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.705+0000 7f854d400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8550104bd0 0x7f8550074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:48.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.705+0000 7f854de00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8550103990 0x7f85500768d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:48.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.705+0000 7f854e800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85501098d0 0x7f85500754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:48.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.705+0000 7f854d400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8550104bd0 0x7f8550074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:56724/0 (socket says 172.21.15.32:56724) 2024-04-30T08:45:48.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.705+0000 7f854de00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8550103990 0x7f85500768d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:34606/0 (socket says 172.21.15.32:34606) 2024-04-30T08:45:48.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.705+0000 7f854d400640 1 -- 172.21.15.32:0/2784957699 learned_addr learned my addr 172.21.15.32:0/2784957699 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:45:48.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.705+0000 7f854e800640 1 -- 172.21.15.32:0/2784957699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8550104bd0 msgr2=0x7f8550074f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:48.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.705+0000 7f854e800640 1 --2- 172.21.15.32:0/2784957699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8550104bd0 0x7f8550074f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:48.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.705+0000 7f854e800640 1 -- 172.21.15.32:0/2784957699 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8550103990 msgr2=0x7f85500768d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:48.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.705+0000 7f854e800640 1 --2- 172.21.15.32:0/2784957699 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8550103990 0x7f85500768d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:48.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.706+0000 7f854e800640 1 -- 172.21.15.32:0/2784957699 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8550075be0 con 0x7f85501098d0 2024-04-30T08:45:48.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.706+0000 7f854de00640 1 --2- 172.21.15.32:0/2784957699 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8550103990 0x7f85500768d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:45:48.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.706+0000 7f854e800640 1 --2- 172.21.15.32:0/2784957699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85501098d0 0x7f85500754b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f854800ec50 tx=0x7f854800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:45:48.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.706+0000 7f8546a00640 1 -- 172.21.15.32:0/2784957699 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8548003b70 con 0x7f85501098d0 2024-04-30T08:45:48.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.706+0000 7f854f200640 1 -- 172.21.15.32:0/2784957699 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85501a41c0 con 0x7f85501098d0 2024-04-30T08:45:48.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.706+0000 7f8546a00640 1 -- 172.21.15.32:0/2784957699 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f8548003d10 con 0x7f85501098d0 2024-04-30T08:45:48.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.706+0000 7f8546a00640 1 -- 172.21.15.32:0/2784957699 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8548010830 con 0x7f85501098d0 2024-04-30T08:45:48.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.706+0000 7f854f200640 1 -- 172.21.15.32:0/2784957699 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85501a4630 con 0x7f85501098d0 2024-04-30T08:45:48.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.707+0000 7f854f200640 1 -- 172.21.15.32:0/2784957699 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85501086e0 con 0x7f85501098d0 2024-04-30T08:45:48.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.710+0000 7f8546a00640 1 -- 172.21.15.32:0/2784957699 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f85480109d0 con 0x7f85501098d0 2024-04-30T08:45:48.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.711+0000 7f8546a00640 1 --2- 172.21.15.32:0/2784957699 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f85340afae0 0x7f85340b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:48.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.712+0000 7f8546a00640 1 -- 172.21.15.32:0/2784957699 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f85480df5f0 con 0x7f85501098d0 2024-04-30T08:45:48.713 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.712+0000 7f854de00640 1 --2- 172.21.15.32:0/2784957699 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f85340afae0 0x7f85340b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:48.714 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.713+0000 7f854de00640 1 --2- 172.21.15.32:0/2784957699 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f85340afae0 0x7f85340b1ff0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f854002a9e0 tx=0x7f85400314a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:45:48.716 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.714+0000 7f8546a00640 1 -- 172.21.15.32:0/2784957699 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f85480a7e50 con 0x7f85501098d0 2024-04-30T08:45:48.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.933+0000 7f854f200640 1 -- 172.21.15.32:0/2784957699 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f8550063820 con 0x7f85501098d0 2024-04-30T08:45:48.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.934+0000 7f8546a00640 1 -- 172.21.15.32:0/2784957699 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v85) ==== 103+0+11 (secure 0 0 0) 0x7f85480a7750 con 0x7f85501098d0 2024-04-30T08:45:48.936 INFO:teuthology.orchestra.run.smithi032.stdout:pg_num: 32 2024-04-30T08:45:48.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.937+0000 7f854f200640 1 -- 172.21.15.32:0/2784957699 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f85340afae0 msgr2=0x7f85340b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:48.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.937+0000 7f854f200640 1 --2- 172.21.15.32:0/2784957699 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f85340afae0 0x7f85340b1ff0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f854002a9e0 tx=0x7f85400314a0 comp rx=0 tx=0).stop 2024-04-30T08:45:48.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.937+0000 7f854f200640 1 -- 172.21.15.32:0/2784957699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85501098d0 msgr2=0x7f85500754b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:48.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.937+0000 7f854f200640 1 --2- 172.21.15.32:0/2784957699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85501098d0 0x7f85500754b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f854800ec50 tx=0x7f854800c6a0 comp rx=0 tx=0).stop 2024-04-30T08:45:48.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.937+0000 7f854f200640 1 -- 172.21.15.32:0/2784957699 shutdown_connections 2024-04-30T08:45:48.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.937+0000 7f854f200640 1 --2- 172.21.15.32:0/2784957699 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f85340afae0 0x7f85340b1ff0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:48.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.937+0000 7f854f200640 1 --2- 172.21.15.32:0/2784957699 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f85501098d0 0x7f85500754b0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:48.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.937+0000 7f854f200640 1 --2- 172.21.15.32:0/2784957699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8550104bd0 0x7f8550074f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:48.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.937+0000 7f854f200640 1 --2- 172.21.15.32:0/2784957699 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f8550103990 0x7f85500768d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:48.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.937+0000 7f854f200640 1 -- 172.21.15.32:0/2784957699 >> 172.21.15.32:0/2784957699 conn(0x7f85500feba0 msgr2=0x7f85501058b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:45:48.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.937+0000 7f854f200640 1 -- 172.21.15.32:0/2784957699 shutdown_connections 2024-04-30T08:45:48.939 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:48.937+0000 7f854f200640 1 -- 172.21.15.32:0/2784957699 wait complete. 2024-04-30T08:45:48.966 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-30T08:45:49.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.226+0000 7fb90fe00640 1 -- 172.21.15.32:0/3663780599 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb910103990 msgr2=0x7fb910104690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:49.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.226+0000 7fb90fe00640 1 --2- 172.21.15.32:0/3663780599 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb910103990 0x7fb910104690 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb900009900 tx=0x7fb90002a3f0 comp rx=0 tx=0).stop 2024-04-30T08:45:49.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.226+0000 7fb90fe00640 1 -- 172.21.15.32:0/3663780599 shutdown_connections 2024-04-30T08:45:49.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.226+0000 7fb90fe00640 1 --2- 172.21.15.32:0/3663780599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9101098d0 0x7fb91010bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:49.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.226+0000 7fb90fe00640 1 --2- 172.21.15.32:0/3663780599 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb910104bd0 0x7fb910104fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:49.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.226+0000 7fb90fe00640 1 --2- 172.21.15.32:0/3663780599 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb910103990 0x7fb910104690 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:49.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.226+0000 7fb90fe00640 1 -- 172.21.15.32:0/3663780599 >> 172.21.15.32:0/3663780599 conn(0x7fb9100feba0 msgr2=0x7fb910101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:45:49.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.226+0000 7fb90fe00640 1 -- 172.21.15.32:0/3663780599 shutdown_connections 2024-04-30T08:45:49.228 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.226+0000 7fb90fe00640 1 -- 172.21.15.32:0/3663780599 wait complete. 2024-04-30T08:45:49.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.227+0000 7fb90fe00640 1 Processor -- start 2024-04-30T08:45:49.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.227+0000 7fb90fe00640 1 -- start start 2024-04-30T08:45:49.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.228+0000 7fb90fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb910103990 0x7fb91019c350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:49.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.228+0000 7fb90fe00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb910104bd0 0x7fb91019c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:49.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.228+0000 7fb90fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9101098d0 0x7fb9101a37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:49.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.228+0000 7fb90fe00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb91010f4c0 con 0x7fb910103990 2024-04-30T08:45:49.230 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.228+0000 7fb90fe00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb91010f340 con 0x7fb910104bd0 2024-04-30T08:45:49.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.228+0000 7fb90fe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb91010f640 con 0x7fb9101098d0 2024-04-30T08:45:49.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.228+0000 7fb90ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb910103990 0x7fb91019c350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:49.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.228+0000 7fb90ea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb910103990 0x7fb91019c350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58382/0 (socket says 172.21.15.32:58382) 2024-04-30T08:45:49.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.228+0000 7fb90ea00640 1 -- 172.21.15.32:0/3909119705 learned_addr learned my addr 172.21.15.32:0/3909119705 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:45:49.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.228+0000 7fb90e000640 1 --2- 172.21.15.32:0/3909119705 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb910104bd0 0x7fb91019c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:49.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.228+0000 7fb90f400640 1 --2- 172.21.15.32:0/3909119705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9101098d0 0x7fb9101a37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:49.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.228+0000 7fb90ea00640 1 -- 172.21.15.32:0/3909119705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9101098d0 msgr2=0x7fb9101a37b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:49.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.228+0000 7fb90ea00640 1 --2- 172.21.15.32:0/3909119705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9101098d0 0x7fb9101a37b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:49.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.228+0000 7fb90ea00640 1 -- 172.21.15.32:0/3909119705 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb910104bd0 msgr2=0x7fb91019c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:49.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.228+0000 7fb90ea00640 1 --2- 172.21.15.32:0/3909119705 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb910104bd0 0x7fb91019c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:49.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.228+0000 7fb90ea00640 1 -- 172.21.15.32:0/3909119705 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9101a3e20 con 0x7fb910103990 2024-04-30T08:45:49.231 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.228+0000 7fb90e000640 1 --2- 172.21.15.32:0/3909119705 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb910104bd0 0x7fb91019c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:45:49.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.229+0000 7fb90ea00640 1 --2- 172.21.15.32:0/3909119705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb910103990 0x7fb91019c350 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb900031440 tx=0x7fb900003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:45:49.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.229+0000 7fb907400640 1 -- 172.21.15.32:0/3909119705 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb90003f070 con 0x7fb910103990 2024-04-30T08:45:49.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.229+0000 7fb907400640 1 -- 172.21.15.32:0/3909119705 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb900031c50 con 0x7fb910103990 2024-04-30T08:45:49.232 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.229+0000 7fb90fe00640 1 -- 172.21.15.32:0/3909119705 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9101a40b0 con 0x7fb910103990 2024-04-30T08:45:49.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.229+0000 7fb907400640 1 -- 172.21.15.32:0/3909119705 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb900035030 con 0x7fb910103990 2024-04-30T08:45:49.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.229+0000 7fb90fe00640 1 -- 172.21.15.32:0/3909119705 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9101a45f0 con 0x7fb910103990 2024-04-30T08:45:49.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.231+0000 7fb90fe00640 1 -- 172.21.15.32:0/3909119705 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9101a4910 con 0x7fb910103990 2024-04-30T08:45:49.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.235+0000 7fb907400640 1 -- 172.21.15.32:0/3909119705 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7fb900042070 con 0x7fb910103990 2024-04-30T08:45:49.239 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.238+0000 7fb907400640 1 --2- 172.21.15.32:0/3909119705 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb8f40afb60 0x7fb8f40b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:49.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.239+0000 7fb907400640 1 -- 172.21.15.32:0/3909119705 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fb9000ffe60 con 0x7fb910103990 2024-04-30T08:45:49.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.239+0000 7fb90e000640 1 --2- 172.21.15.32:0/3909119705 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb8f40afb60 0x7fb8f40b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:49.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.239+0000 7fb907400640 1 -- 172.21.15.32:0/3909119705 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb9000c8020 con 0x7fb910103990 2024-04-30T08:45:49.241 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.240+0000 7fb90e000640 1 --2- 172.21.15.32:0/3909119705 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb8f40afb60 0x7fb8f40b2070 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fb91019d3f0 tx=0x7fb8fc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:45:49.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.477+0000 7fb90fe00640 1 -- 172.21.15.32:0/3909119705 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fb9101a4ae0 con 0x7fb910103990 2024-04-30T08:45:49.479 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.478+0000 7fb907400640 1 -- 172.21.15.32:0/3909119705 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v85) ==== 81+0+587 (secure 0 0 0) 0x7fb9000cc320 con 0x7fb910103990 2024-04-30T08:45:49.480 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/1266897222 2024-05-01T08:23:29.526664+0000 2024-04-30T08:45:49.480 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/2915689533 2024-05-01T08:23:29.526664+0000 2024-04-30T08:45:49.480 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6801/4019351306 2024-05-01T08:23:29.526664+0000 2024-04-30T08:45:49.480 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/425836193 2024-05-01T08:23:29.526664+0000 2024-04-30T08:45:49.480 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/1134075722 2024-05-01T08:22:53.639893+0000 2024-04-30T08:45:49.480 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6800/464872591 2024-05-01T08:22:53.639893+0000 2024-04-30T08:45:49.480 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6801/464872591 2024-05-01T08:22:53.639893+0000 2024-04-30T08:45:49.480 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/21059253 2024-05-01T08:22:53.639893+0000 2024-04-30T08:45:49.480 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:0/3245550894 2024-05-01T08:22:53.639893+0000 2024-04-30T08:45:49.480 INFO:teuthology.orchestra.run.smithi032.stdout:172.21.15.32:6800/4019351306 2024-05-01T08:23:29.526664+0000 2024-04-30T08:45:49.481 INFO:teuthology.orchestra.run.smithi032.stderr:listed 10 entries 2024-04-30T08:45:49.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.481+0000 7fb90fe00640 1 -- 172.21.15.32:0/3909119705 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb8f40afb60 msgr2=0x7fb8f40b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:49.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.481+0000 7fb90fe00640 1 --2- 172.21.15.32:0/3909119705 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb8f40afb60 0x7fb8f40b2070 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fb91019d3f0 tx=0x7fb8fc005f90 comp rx=0 tx=0).stop 2024-04-30T08:45:49.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.481+0000 7fb90fe00640 1 -- 172.21.15.32:0/3909119705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb910103990 msgr2=0x7fb91019c350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:49.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.481+0000 7fb90fe00640 1 --2- 172.21.15.32:0/3909119705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb910103990 0x7fb91019c350 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb900031440 tx=0x7fb900003c10 comp rx=0 tx=0).stop 2024-04-30T08:45:49.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.481+0000 7fb90fe00640 1 -- 172.21.15.32:0/3909119705 shutdown_connections 2024-04-30T08:45:49.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.481+0000 7fb90fe00640 1 --2- 172.21.15.32:0/3909119705 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb8f40afb60 0x7fb8f40b2070 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:49.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.481+0000 7fb90fe00640 1 --2- 172.21.15.32:0/3909119705 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb9101098d0 0x7fb9101a37b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:49.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.481+0000 7fb90fe00640 1 --2- 172.21.15.32:0/3909119705 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb910104bd0 0x7fb91019c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:49.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.481+0000 7fb90fe00640 1 --2- 172.21.15.32:0/3909119705 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb910103990 0x7fb91019c350 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:49.483 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.481+0000 7fb90fe00640 1 -- 172.21.15.32:0/3909119705 >> 172.21.15.32:0/3909119705 conn(0x7fb9100feba0 msgr2=0x7fb9100fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:45:49.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.481+0000 7fb90fe00640 1 -- 172.21.15.32:0/3909119705 shutdown_connections 2024-04-30T08:45:49.484 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:49.481+0000 7fb90fe00640 1 -- 172.21.15.32:0/3909119705 wait complete. 2024-04-30T08:45:49.503 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:49 smithi071 ceph-mon[33385]: pgmap v744: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:45:49.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:49 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1544856469' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-30T08:45:49.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:49 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:45:49.504 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:49 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2784957699' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-30T08:45:49.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:49 smithi032 ceph-mon[28331]: pgmap v744: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:45:49.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:49 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1544856469' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-30T08:45:49.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:49 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:45:49.508 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:49 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2784957699' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-30T08:45:49.511 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-30T08:45:49.511 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-30T08:45:49.511 DEBUG:teuthology.orchestra.run.smithi032:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-30T08:45:49.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:49 smithi189 ceph-mon[33038]: pgmap v744: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 9 op/s 2024-04-30T08:45:49.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:49 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1544856469' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-30T08:45:49.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:49 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-30T08:45:49.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:49 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2784957699' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-04-30T08:45:49.646 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi032.front.sepia.ceph.com 2024-04-30T08:45:49.646 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-30T08:45:49.646 DEBUG:teuthology.orchestra.run.smithi032:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-30T08:45:49.695 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-30T08:45:49.695 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-04-30T08:45:49.770 INFO:teuthology.orchestra.run.smithi032.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-30T08:45:49.771 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-30T08:45:49.771 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-30T08:45:49.843 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-30T08:45:49.843 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link delete ceph-brx 2024-04-30T08:45:49.940 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-30T08:45:49.941 DEBUG:teuthology.orchestra.run.smithi032:> set -ex 2024-04-30T08:45:49.941 DEBUG:teuthology.orchestra.run.smithi032:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-30T08:45:49.980 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053506,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700,uid=1000,gid=1316 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/655dcfd70762c28e0fd55f22e9899475aa4c44a6e171d6b217b6e4c45cf4ed53/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/4ZQNEVY3JVRZLGHGU22GRSP6ES:/var/lib/containers/storage/overlay/l/MRUJMHAPM6RJKYDFRAHTS6AF4O,upperdir=/var/lib/containers/storage/overlay/655dcfd70762c28e0fd55f22e9899475aa4c44a6e171d6b217b6e4c45cf4ed53/diff,workdir=/var/lib/containers/storage/overlay/655dcfd70762c28e0fd55f22e9899475aa4c44a6e171d6b217b6e4c45cf4ed53/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/305ef99cd576959b8c7a33096bd1a308140c966128fcbcf15325a10b2abecca6/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/4ZQNEVY3JVRZLGHGU22GRSP6ES:/var/lib/containers/storage/overlay/l/MRUJMHAPM6RJKYDFRAHTS6AF4O,upperdir=/var/lib/containers/storage/overlay/305ef99cd576959b8c7a33096bd1a308140c966128fcbcf15325a10b2abecca6/diff,workdir=/var/lib/containers/storage/overlay/305ef99cd576959b8c7a33096bd1a308140c966128fcbcf15325a10b2abecca6/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249616k,nr_inodes=812404,mode=700 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 overlay /var/lib/containers/storage/overlay/f7cbdd9817d386562affbbdb7b6d2f49f3e3cfa6ebbbe83e6d82362e9040335b/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/4ZQNEVY3JVRZLGHGU22GRSP6ES:/var/lib/containers/storage/overlay/l/MRUJMHAPM6RJKYDFRAHTS6AF4O,upperdir=/var/lib/containers/storage/overlay/f7cbdd9817d386562affbbdb7b6d2f49f3e3cfa6ebbbe83e6d82362e9040335b/diff,workdir=/var/lib/containers/storage/overlay/f7cbdd9817d386562affbbdb7b6d2f49f3e3cfa6ebbbe83e6d82362e9040335b/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/186456821c6aaa2f0e77ce5d1c9818097ce5136349e14b20d9cac22a1daab496/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/4ZQNEVY3JVRZLGHGU22GRSP6ES:/var/lib/containers/storage/overlay/l/MRUJMHAPM6RJKYDFRAHTS6AF4O,upperdir=/var/lib/containers/storage/overlay/186456821c6aaa2f0e77ce5d1c9818097ce5136349e14b20d9cac22a1daab496/diff,workdir=/var/lib/containers/storage/overlay/186456821c6aaa2f0e77ce5d1c9818097ce5136349e14b20d9cac22a1daab496/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/313a2d52f90bf15f3ed13d29a73951681f19e53a1422fe4fec9cf39fa7969932/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/4ZQNEVY3JVRZLGHGU22GRSP6ES:/var/lib/containers/storage/overlay/l/MRUJMHAPM6RJKYDFRAHTS6AF4O,upperdir=/var/lib/containers/storage/overlay/313a2d52f90bf15f3ed13d29a73951681f19e53a1422fe4fec9cf39fa7969932/diff,workdir=/var/lib/containers/storage/overlay/313a2d52f90bf15f3ed13d29a73951681f19e53a1422fe4fec9cf39fa7969932/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/33f1f2d434abd75169f58761b6cf744f11a86f83280e9e14e73e1bb1bbe8d9dc/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/4ZQNEVY3JVRZLGHGU22GRSP6ES:/var/lib/containers/storage/overlay/l/MRUJMHAPM6RJKYDFRAHTS6AF4O,upperdir=/var/lib/containers/storage/overlay/33f1f2d434abd75169f58761b6cf744f11a86f83280e9e14e73e1bb1bbe8d9dc/diff,workdir=/var/lib/containers/storage/overlay/33f1f2d434abd75169f58761b6cf744f11a86f83280e9e14e73e1bb1bbe8d9dc/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/b5c232a13935efa166a37f0c9facf199401b43a530f22deb4afcc90b4a0eb92a/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/4ZQNEVY3JVRZLGHGU22GRSP6ES:/var/lib/containers/storage/overlay/l/MRUJMHAPM6RJKYDFRAHTS6AF4O,upperdir=/var/lib/containers/storage/overlay/b5c232a13935efa166a37f0c9facf199401b43a530f22deb4afcc90b4a0eb92a/diff,workdir=/var/lib/containers/storage/overlay/b5c232a13935efa166a37f0c9facf199401b43a530f22deb4afcc90b4a0eb92a/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/6e1c00c34cdc09db8646deace3032d23b037f1f07560cd583d5ab75f540a6de0/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/4ZQNEVY3JVRZLGHGU22GRSP6ES:/var/lib/containers/storage/overlay/l/MRUJMHAPM6RJKYDFRAHTS6AF4O,upperdir=/var/lib/containers/storage/overlay/6e1c00c34cdc09db8646deace3032d23b037f1f07560cd583d5ab75f540a6de0/diff,workdir=/var/lib/containers/storage/overlay/6e1c00c34cdc09db8646deace3032d23b037f1f07560cd583d5ab75f540a6de0/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/a87b50f7886eee3f513277639787ac14c794596e6271165b8037b96873500b3a/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/4ZQNEVY3JVRZLGHGU22GRSP6ES:/var/lib/containers/storage/overlay/l/MRUJMHAPM6RJKYDFRAHTS6AF4O,upperdir=/var/lib/containers/storage/overlay/a87b50f7886eee3f513277639787ac14c794596e6271165b8037b96873500b3a/diff,workdir=/var/lib/containers/storage/overlay/a87b50f7886eee3f513277639787ac14c794596e6271165b8037b96873500b3a/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/6e69d365f1e7755d98afe0dc54d8db09fcefb0cdc0909acc7c74b9c5ca79b1a3/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/4ZQNEVY3JVRZLGHGU22GRSP6ES:/var/lib/containers/storage/overlay/l/MRUJMHAPM6RJKYDFRAHTS6AF4O,upperdir=/var/lib/containers/storage/overlay/6e69d365f1e7755d98afe0dc54d8db09fcefb0cdc0909acc7c74b9c5ca79b1a3/diff,workdir=/var/lib/containers/storage/overlay/6e69d365f1e7755d98afe0dc54d8db09fcefb0cdc0909acc7c74b9c5ca79b1a3/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6499240k,nr_inodes=819200,mode=755 0 0 2024-04-30T08:45:49.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-30T08:45:49.981 DEBUG:teuthology.orchestra.run.smithi032:> ip netns list 2024-04-30T08:45:50.063 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-30T08:45:50.063 DEBUG:teuthology.orchestra.run.smithi032:> sudo ip link delete ceph-brx 2024-04-30T08:45:50.257 INFO:teuthology.orchestra.run.smithi032.stderr:Cannot find device "ceph-brx" 2024-04-30T08:45:50.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-30T08:45:50.262 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-30T08:45:50.273 INFO:tasks.cephadm:Teardown begin 2024-04-30T08:45:50.273 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-30T08:45:50.273 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-30T08:45:50.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:50 smithi032 ceph-mon[28331]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:45:50.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:50 smithi032 ceph-mon[28331]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:45:50.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:50 smithi032 ceph-mon[28331]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:45:50.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:50 smithi032 ceph-mon[28331]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:45:50.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:50 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/3909119705' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-30T08:45:50.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:50 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:45:50.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:50 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:45:50.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:50 smithi032 ceph-mon[28331]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:45:50.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.543+0000 7f6105800640 1 -- 172.21.15.32:0/1995314487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6100104bd0 msgr2=0x7f6100104fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:50.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.543+0000 7f6105800640 1 --2- 172.21.15.32:0/1995314487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6100104bd0 0x7f6100104fe0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f60e8009860 tx=0x7f60e802a340 comp rx=0 tx=0).stop 2024-04-30T08:45:50.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.544+0000 7f6105800640 1 -- 172.21.15.32:0/1995314487 shutdown_connections 2024-04-30T08:45:50.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.544+0000 7f6105800640 1 --2- 172.21.15.32:0/1995314487 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f61001098d0 0x7f610010bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:50.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.544+0000 7f6105800640 1 --2- 172.21.15.32:0/1995314487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6100104bd0 0x7f6100104fe0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:50.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.544+0000 7f6105800640 1 --2- 172.21.15.32:0/1995314487 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f6100103990 0x7f6100104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:50.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.544+0000 7f6105800640 1 -- 172.21.15.32:0/1995314487 >> 172.21.15.32:0/1995314487 conn(0x7f61000feba0 msgr2=0x7f6100101030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:45:50.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.544+0000 7f6105800640 1 -- 172.21.15.32:0/1995314487 shutdown_connections 2024-04-30T08:45:50.546 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.544+0000 7f6105800640 1 -- 172.21.15.32:0/1995314487 wait complete. 2024-04-30T08:45:50.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.545+0000 7f6105800640 1 Processor -- start 2024-04-30T08:45:50.547 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.545+0000 7f6105800640 1 -- start start 2024-04-30T08:45:50.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.546+0000 7f6105800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6100103990 0x7f61000768d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:50.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.546+0000 7f6105800640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6100104bd0 0x7f6100074f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:50.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.546+0000 7f6105800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61001098d0 0x7f61000754b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:50.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.546+0000 7f6105800640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f610010f490 con 0x7f61001098d0 2024-04-30T08:45:50.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.546+0000 7f6105800640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f610010f310 con 0x7f6100104bd0 2024-04-30T08:45:50.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.546+0000 7f6105800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f610010f610 con 0x7f6100103990 2024-04-30T08:45:50.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.546+0000 7f6104e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61001098d0 0x7f61000754b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:50.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.546+0000 7f60f7400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6100104bd0 0x7f6100074f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:50.549 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.546+0000 7f60f7400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6100104bd0 0x7f6100074f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:34652/0 (socket says 172.21.15.32:34652) 2024-04-30T08:45:50.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.546+0000 7f60f7400640 1 -- 172.21.15.32:0/684707548 learned_addr learned my addr 172.21.15.32:0/684707548 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:45:50.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.546+0000 7f60ffe00640 1 --2- 172.21.15.32:0/684707548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6100103990 0x7f61000768d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:50.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.547+0000 7f60f7400640 1 -- 172.21.15.32:0/684707548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6100103990 msgr2=0x7f61000768d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:50.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.547+0000 7f60f7400640 1 --2- 172.21.15.32:0/684707548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6100103990 0x7f61000768d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:50.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.547+0000 7f60f7400640 1 -- 172.21.15.32:0/684707548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61001098d0 msgr2=0x7f61000754b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:50.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.547+0000 7f60f7400640 1 --2- 172.21.15.32:0/684707548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61001098d0 0x7f61000754b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:50.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.547+0000 7f60f7400640 1 -- 172.21.15.32:0/684707548 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6100075be0 con 0x7f6100104bd0 2024-04-30T08:45:50.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.547+0000 7f60ffe00640 1 --2- 172.21.15.32:0/684707548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6100103990 0x7f61000768d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-30T08:45:50.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.547+0000 7f60f7400640 1 --2- 172.21.15.32:0/684707548 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6100104bd0 0x7f6100074f70 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f60e8032d50 tx=0x7f60e8032d80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:45:50.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.547+0000 7f60fd600640 1 -- 172.21.15.32:0/684707548 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e8037040 con 0x7f6100104bd0 2024-04-30T08:45:50.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.547+0000 7f60fd600640 1 -- 172.21.15.32:0/684707548 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f60e8030550 con 0x7f6100104bd0 2024-04-30T08:45:50.550 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.547+0000 7f6105800640 1 -- 172.21.15.32:0/684707548 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61001a4180 con 0x7f6100104bd0 2024-04-30T08:45:50.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.547+0000 7f60fd600640 1 -- 172.21.15.32:0/684707548 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60e8003850 con 0x7f6100104bd0 2024-04-30T08:45:50.551 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.547+0000 7f6105800640 1 -- 172.21.15.32:0/684707548 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61001a4530 con 0x7f6100104bd0 2024-04-30T08:45:50.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:50 smithi189 ceph-mon[33038]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:45:50.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:50 smithi189 ceph-mon[33038]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:45:50.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:50 smithi189 ceph-mon[33038]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:45:50.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:50 smithi189 ceph-mon[33038]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:45:50.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:50 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/3909119705' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-30T08:45:50.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:50 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:45:50.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:50 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:45:50.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:50 smithi189 ceph-mon[33038]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:45:50.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.548+0000 7f6105800640 1 -- 172.21.15.32:0/684707548 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f61001086e0 con 0x7f6100104bd0 2024-04-30T08:45:50.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.551+0000 7f60fd600640 1 -- 172.21.15.32:0/684707548 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f60e8043020 con 0x7f6100104bd0 2024-04-30T08:45:50.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.552+0000 7f60fd600640 1 --2- 172.21.15.32:0/684707548 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f60e40afae0 0x7f60e40b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:50.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.553+0000 7f60fd600640 1 -- 172.21.15.32:0/684707548 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f60e80fe940 con 0x7f6100104bd0 2024-04-30T08:45:50.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.553+0000 7f60ffe00640 1 --2- 172.21.15.32:0/684707548 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f60e40afae0 0x7f60e40b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:50.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.554+0000 7f60ffe00640 1 --2- 172.21.15.32:0/684707548 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f60e40afae0 0x7f60e40b1ff0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f60f0009a60 tx=0x7f60f00095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:45:50.556 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.555+0000 7f60fd600640 1 -- 172.21.15.32:0/684707548 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f60e804a060 con 0x7f6100104bd0 2024-04-30T08:45:50.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:50 smithi071 ceph-mon[33385]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:45:50.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:50 smithi071 ceph-mon[33385]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:45:50.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:50 smithi071 ceph-mon[33385]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:45:50.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:50 smithi071 ceph-mon[33385]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:45:50.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:50 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/3909119705' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-30T08:45:50.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:50 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-30T08:45:50.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:50 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-30T08:45:50.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:50 smithi071 ceph-mon[33385]: from='mgr.14152 ' entity='mgr.x' 2024-04-30T08:45:50.806 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.804+0000 7f6105800640 1 -- 172.21.15.32:0/684707548 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f610002dbd0 con 0x7f6100104bd0 2024-04-30T08:45:50.807 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.806+0000 7f60fd600640 1 -- 172.21.15.32:0/684707548 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) ==== 95+0+11938 (secure 0 0 0) 0x7f60e80c7110 con 0x7f6100104bd0 2024-04-30T08:45:50.808 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:45:50.808 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":132,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2235842676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2235842676},{"type":"v1","addr":"172.21.15.32:6835","nonce":2235842676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/3577009355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":3577009355},{"type":"v1","addr":"172.21.15.32:6841","nonce":3577009355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":132,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-30T08:31:52.236668+0000","modified":"2024-04-30T08:44:48.772575+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24541,"mds_3":24533,"mds_4":24517},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":205,"addr":"172.21.15.189:6833/463203038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":463203038},{"type":"v1","addr":"172.21.15.189:6833","nonce":463203038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6833/3635143591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3635143591},{"type":"v1","addr":"172.21.15.71:6833","nonce":3635143591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6837/2569449994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2569449994},{"type":"v1","addr":"172.21.15.32:6837","nonce":2569449994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/278686565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":278686565},{"type":"v1","addr":"172.21.15.189:6835","nonce":278686565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":201,"addr":"172.21.15.71:6835/1558303056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1558303056},{"type":"v1","addr":"172.21.15.71:6835","nonce":1558303056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.189:6837/686520873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":686520873},{"type":"v1","addr":"172.21.15.189:6837","nonce":686520873}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.71:6837/2324039932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2324039932},{"type":"v1","addr":"172.21.15.71:6837","nonce":2324039932}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6839/352970107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":352970107},{"type":"v1","addr":"172.21.15.189:6839","nonce":352970107}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.32:6839/4160386524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4160386524},{"type":"v1","addr":"172.21.15.32:6839","nonce":4160386524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":197,"addr":"172.21.15.71:6839/3517409993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3517409993},{"type":"v1","addr":"172.21.15.71:6839","nonce":3517409993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24517,24533,24560,24590,24541]},"id":1}]} 2024-04-30T08:45:50.808 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 132 2024-04-30T08:45:50.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.808+0000 7f6105800640 1 -- 172.21.15.32:0/684707548 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f60e40afae0 msgr2=0x7f60e40b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:50.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.808+0000 7f6105800640 1 --2- 172.21.15.32:0/684707548 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f60e40afae0 0x7f60e40b1ff0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f60f0009a60 tx=0x7f60f00095c0 comp rx=0 tx=0).stop 2024-04-30T08:45:50.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.808+0000 7f6105800640 1 -- 172.21.15.32:0/684707548 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6100104bd0 msgr2=0x7f6100074f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:50.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.808+0000 7f6105800640 1 --2- 172.21.15.32:0/684707548 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6100104bd0 0x7f6100074f70 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f60e8032d50 tx=0x7f60e8032d80 comp rx=0 tx=0).stop 2024-04-30T08:45:50.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.809+0000 7f6105800640 1 -- 172.21.15.32:0/684707548 shutdown_connections 2024-04-30T08:45:50.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.809+0000 7f6105800640 1 --2- 172.21.15.32:0/684707548 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f60e40afae0 0x7f60e40b1ff0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:50.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.809+0000 7f6105800640 1 --2- 172.21.15.32:0/684707548 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f61001098d0 0x7f61000754b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:50.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.809+0000 7f6105800640 1 --2- 172.21.15.32:0/684707548 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f6100104bd0 0x7f6100074f70 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:50.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.809+0000 7f6105800640 1 --2- 172.21.15.32:0/684707548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6100103990 0x7f61000768d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:50.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.809+0000 7f6105800640 1 -- 172.21.15.32:0/684707548 >> 172.21.15.32:0/684707548 conn(0x7f61000feba0 msgr2=0x7f61001058b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:45:50.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.809+0000 7f6105800640 1 -- 172.21.15.32:0/684707548 shutdown_connections 2024-04-30T08:45:50.811 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:50.809+0000 7f6105800640 1 -- 172.21.15.32:0/684707548 wait complete. 2024-04-30T08:45:50.840 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-30T08:45:51.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.128+0000 7faa73200640 1 -- 172.21.15.32:0/1766603022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa7410baf0 msgr2=0x7faa7410df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:51.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.128+0000 7faa73200640 1 --2- 172.21.15.32:0/1766603022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa7410baf0 0x7faa7410df10 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7faa6c009be0 tx=0x7faa6c02a6c0 comp rx=0 tx=0).stop 2024-04-30T08:45:51.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.129+0000 7faa73200640 1 -- 172.21.15.32:0/1766603022 shutdown_connections 2024-04-30T08:45:51.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.129+0000 7faa73200640 1 --2- 172.21.15.32:0/1766603022 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa7410baf0 0x7faa7410df10 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:51.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.129+0000 7faa73200640 1 --2- 172.21.15.32:0/1766603022 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa74106df0 0x7faa74107200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:51.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.129+0000 7faa73200640 1 --2- 172.21.15.32:0/1766603022 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa74105bb0 0x7faa741068b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:51.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.129+0000 7faa73200640 1 -- 172.21.15.32:0/1766603022 >> 172.21.15.32:0/1766603022 conn(0x7faa74100da0 msgr2=0x7faa74103250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:45:51.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.129+0000 7faa73200640 1 -- 172.21.15.32:0/1766603022 shutdown_connections 2024-04-30T08:45:51.131 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.129+0000 7faa73200640 1 -- 172.21.15.32:0/1766603022 wait complete. 2024-04-30T08:45:51.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.130+0000 7faa73200640 1 Processor -- start 2024-04-30T08:45:51.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.130+0000 7faa73200640 1 -- start start 2024-04-30T08:45:51.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.130+0000 7faa73200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa74105bb0 0x7faa7419c1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:51.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.130+0000 7faa73200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa74106df0 0x7faa7419c730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:51.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.130+0000 7faa73200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa7410baf0 0x7faa741a36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:51.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.130+0000 7faa73200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7faa741116e0 con 0x7faa74105bb0 2024-04-30T08:45:51.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.130+0000 7faa73200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7faa74111560 con 0x7faa7410baf0 2024-04-30T08:45:51.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.130+0000 7faa73200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7faa74111860 con 0x7faa74106df0 2024-04-30T08:45:51.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.131+0000 7faa71400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa74106df0 0x7faa7419c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:51.133 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.131+0000 7faa71400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa74106df0 0x7faa7419c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.32:56792/0 (socket says 172.21.15.32:56792) 2024-04-30T08:45:51.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.131+0000 7faa71400640 1 -- 172.21.15.32:0/253013798 learned_addr learned my addr 172.21.15.32:0/253013798 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:45:51.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.131+0000 7faa72800640 1 --2- 172.21.15.32:0/253013798 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa7410baf0 0x7faa741a36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:51.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.131+0000 7faa71e00640 1 --2- 172.21.15.32:0/253013798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa74105bb0 0x7faa7419c1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:51.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.131+0000 7faa71400640 1 -- 172.21.15.32:0/253013798 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa7410baf0 msgr2=0x7faa741a36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:51.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.131+0000 7faa71400640 1 --2- 172.21.15.32:0/253013798 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa7410baf0 0x7faa741a36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:51.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.131+0000 7faa71400640 1 -- 172.21.15.32:0/253013798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa74105bb0 msgr2=0x7faa7419c1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:51.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.131+0000 7faa71400640 1 --2- 172.21.15.32:0/253013798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa74105bb0 0x7faa7419c1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:51.134 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.131+0000 7faa71400640 1 -- 172.21.15.32:0/253013798 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa741a3d30 con 0x7faa74106df0 2024-04-30T08:45:51.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.132+0000 7faa71400640 1 --2- 172.21.15.32:0/253013798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa74106df0 0x7faa7419c730 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7faa5c00e9c0 tx=0x7faa5c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:45:51.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.132+0000 7faa6aa00640 1 -- 172.21.15.32:0/253013798 <== mon.1 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa5c00cde0 con 0x7faa74106df0 2024-04-30T08:45:51.135 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.132+0000 7faa6aa00640 1 -- 172.21.15.32:0/253013798 <== mon.1 v2:172.21.15.189:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7faa5c0035f0 con 0x7faa74106df0 2024-04-30T08:45:51.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.132+0000 7faa73200640 1 -- 172.21.15.32:0/253013798 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa741a3fc0 con 0x7faa74106df0 2024-04-30T08:45:51.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.132+0000 7faa6aa00640 1 -- 172.21.15.32:0/253013798 <== mon.1 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa5c010690 con 0x7faa74106df0 2024-04-30T08:45:51.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.132+0000 7faa73200640 1 -- 172.21.15.32:0/253013798 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa741a4500 con 0x7faa74106df0 2024-04-30T08:45:51.136 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.133+0000 7faa73200640 1 -- 172.21.15.32:0/253013798 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa7410a900 con 0x7faa74106df0 2024-04-30T08:45:51.137 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.136+0000 7faa6aa00640 1 -- 172.21.15.32:0/253013798 <== mon.1 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7faa5c004000 con 0x7faa74106df0 2024-04-30T08:45:51.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.137+0000 7faa6aa00640 1 --2- 172.21.15.32:0/253013798 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7faa580afb60 0x7faa580b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:51.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.137+0000 7faa6aa00640 1 -- 172.21.15.32:0/253013798 <== mon.1 v2:172.21.15.189:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7faa5c0df4d0 con 0x7faa74106df0 2024-04-30T08:45:51.139 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.138+0000 7faa71e00640 1 --2- 172.21.15.32:0/253013798 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7faa580afb60 0x7faa580b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:51.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.138+0000 7faa71e00640 1 --2- 172.21.15.32:0/253013798 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7faa580afb60 0x7faa580b2070 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7faa741067b0 tx=0x7faa64005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:45:51.140 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.139+0000 7faa6aa00640 1 -- 172.21.15.32:0/253013798 <== mon.1 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faa5c076ff0 con 0x7faa74106df0 2024-04-30T08:45:51.391 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.390+0000 7faa73200640 1 -- 172.21.15.32:0/253013798 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa7402dbd0 con 0x7faa74106df0 2024-04-30T08:45:51.392 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.391+0000 7faa6aa00640 1 -- 172.21.15.32:0/253013798 <== mon.1 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 132 v132) ==== 95+0+11938 (secure 0 0 0) 0x7faa5c0a7ca0 con 0x7faa74106df0 2024-04-30T08:45:51.393 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:45:51.393 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":132,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14379,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6835/2235842676","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6834","nonce":2235842676},{"type":"v1","addr":"172.21.15.32:6835","nonce":2235842676}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14457,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.32:6841/3577009355","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6840","nonce":3577009355},{"type":"v1","addr":"172.21.15.32:6841","nonce":3577009355}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12}],"filesystems":[{"mdsmap":{"epoch":132,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-30T08:31:52.236668+0000","modified":"2024-04-30T08:44:48.772575+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24590,"mds_1":24560,"mds_2":24541,"mds_3":24533,"mds_4":24517},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24442":{"gid":24442,"name":"l","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":205,"addr":"172.21.15.189:6833/463203038","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":463203038},{"type":"v1","addr":"172.21.15.189:6833","nonce":463203038}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24458":{"gid":24458,"name":"k","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.71:6833/3635143591","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6832","nonce":3635143591},{"type":"v1","addr":"172.21.15.71:6833","nonce":3635143591}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24484":{"gid":24484,"name":"d","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.32:6837/2569449994","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6836","nonce":2569449994},{"type":"v1","addr":"172.21.15.32:6837","nonce":2569449994}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24505":{"gid":24505,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6835/278686565","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":278686565},{"type":"v1","addr":"172.21.15.189:6835","nonce":278686565}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24506":{"gid":24506,"name":"e","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":201,"addr":"172.21.15.71:6835/1558303056","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6834","nonce":1558303056},{"type":"v1","addr":"172.21.15.71:6835","nonce":1558303056}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24517":{"gid":24517,"name":"c","rank":4,"incarnation":25,"state":"up:active","state_seq":17,"addr":"172.21.15.189:6837/686520873","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":686520873},{"type":"v1","addr":"172.21.15.189:6837","nonce":686520873}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24533":{"gid":24533,"name":"h","rank":3,"incarnation":23,"state":"up:active","state_seq":16,"addr":"172.21.15.71:6837/2324039932","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6836","nonce":2324039932},{"type":"v1","addr":"172.21.15.71:6837","nonce":2324039932}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24541":{"gid":24541,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":15,"addr":"172.21.15.189:6839/352970107","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":352970107},{"type":"v1","addr":"172.21.15.189:6839","nonce":352970107}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24560":{"gid":24560,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":14,"addr":"172.21.15.32:6839/4160386524","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6838","nonce":4160386524},{"type":"v1","addr":"172.21.15.32:6839","nonce":4160386524}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24590":{"gid":24590,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":197,"addr":"172.21.15.71:6839/3517409993","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6838","nonce":3517409993},{"type":"v1","addr":"172.21.15.71:6839","nonce":3517409993}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24590,"qdb_cluster":[24517,24533,24560,24590,24541]},"id":1}]} 2024-04-30T08:45:51.393 INFO:teuthology.orchestra.run.smithi032.stderr:dumped fsmap epoch 132 2024-04-30T08:45:51.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.394+0000 7faa73200640 1 -- 172.21.15.32:0/253013798 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7faa580afb60 msgr2=0x7faa580b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:51.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.394+0000 7faa73200640 1 --2- 172.21.15.32:0/253013798 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7faa580afb60 0x7faa580b2070 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7faa741067b0 tx=0x7faa64005f90 comp rx=0 tx=0).stop 2024-04-30T08:45:51.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.394+0000 7faa73200640 1 -- 172.21.15.32:0/253013798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa74106df0 msgr2=0x7faa7419c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:51.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.394+0000 7faa73200640 1 --2- 172.21.15.32:0/253013798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa74106df0 0x7faa7419c730 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7faa5c00e9c0 tx=0x7faa5c00ee80 comp rx=0 tx=0).stop 2024-04-30T08:45:51.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.394+0000 7faa73200640 1 -- 172.21.15.32:0/253013798 shutdown_connections 2024-04-30T08:45:51.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.394+0000 7faa73200640 1 --2- 172.21.15.32:0/253013798 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7faa580afb60 0x7faa580b2070 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:51.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.394+0000 7faa73200640 1 --2- 172.21.15.32:0/253013798 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7faa7410baf0 0x7faa741a36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:51.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.394+0000 7faa73200640 1 --2- 172.21.15.32:0/253013798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faa74106df0 0x7faa7419c730 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:51.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.394+0000 7faa73200640 1 --2- 172.21.15.32:0/253013798 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7faa74105bb0 0x7faa7419c1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:51.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.394+0000 7faa73200640 1 -- 172.21.15.32:0/253013798 >> 172.21.15.32:0/253013798 conn(0x7faa74100da0 msgr2=0x7faa74107ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:45:51.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.394+0000 7faa73200640 1 -- 172.21.15.32:0/253013798 shutdown_connections 2024-04-30T08:45:51.396 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.394+0000 7faa73200640 1 -- 172.21.15.32:0/253013798 wait complete. 2024-04-30T08:45:51.428 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-30T08:45:51.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:51 smithi189 ceph-mon[33038]: pgmap v745: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-30T08:45:51.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:51 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/684707548' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:45:51.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:51 smithi032 ceph-mon[28331]: pgmap v745: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-30T08:45:51.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:51 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/684707548' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:45:51.706 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.704+0000 7f5d2ca00640 1 -- 172.21.15.32:0/1534256183 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d28102260 msgr2=0x7f5d28102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:51.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.704+0000 7f5d2ca00640 1 --2- 172.21.15.32:0/1534256183 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d28102260 0x7f5d28102670 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f5d18009ca0 tx=0x7f5d1802a8b0 comp rx=0 tx=0).stop 2024-04-30T08:45:51.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.705+0000 7f5d2ca00640 1 -- 172.21.15.32:0/1534256183 shutdown_connections 2024-04-30T08:45:51.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.705+0000 7f5d2ca00640 1 --2- 172.21.15.32:0/1534256183 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d28103500 0x7f5d28103910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:51.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.705+0000 7f5d2ca00640 1 --2- 172.21.15.32:0/1534256183 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d28102bb0 0x7f5d28102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:51.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.705+0000 7f5d2ca00640 1 --2- 172.21.15.32:0/1534256183 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d28102260 0x7f5d28102670 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:51.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.705+0000 7f5d2ca00640 1 -- 172.21.15.32:0/1534256183 >> 172.21.15.32:0/1534256183 conn(0x7f5d280fda70 msgr2=0x7f5d280fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:45:51.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.705+0000 7f5d2ca00640 1 -- 172.21.15.32:0/1534256183 shutdown_connections 2024-04-30T08:45:51.707 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.705+0000 7f5d2ca00640 1 -- 172.21.15.32:0/1534256183 wait complete. 2024-04-30T08:45:51.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.706+0000 7f5d2ca00640 1 Processor -- start 2024-04-30T08:45:51.708 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.706+0000 7f5d2ca00640 1 -- start start 2024-04-30T08:45:51.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.707+0000 7f5d2ca00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d28102260 0x7f5d2819c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:51.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.707+0000 7f5d2ca00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d28102bb0 0x7f5d2819c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:51.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.707+0000 7f5d2ca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d28103500 0x7f5d281a3790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:51.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.707+0000 7f5d2ca00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f5d28111690 con 0x7f5d28102260 2024-04-30T08:45:51.709 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.707+0000 7f5d2ca00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f5d28111510 con 0x7f5d28102bb0 2024-04-30T08:45:51.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.707+0000 7f5d2ca00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f5d28111810 con 0x7f5d28103500 2024-04-30T08:45:51.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.707+0000 7f5d1ea00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d28102bb0 0x7f5d2819c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:51.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.707+0000 7f5d27400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d28102260 0x7f5d2819c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:51.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.707+0000 7f5d1ea00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d28102bb0 0x7f5d2819c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:34670/0 (socket says 172.21.15.32:34670) 2024-04-30T08:45:51.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.707+0000 7f5d1ea00640 1 -- 172.21.15.32:0/4191469167 learned_addr learned my addr 172.21.15.32:0/4191469167 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:45:51.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.707+0000 7f5d27e00640 1 --2- 172.21.15.32:0/4191469167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d28103500 0x7f5d281a3790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:51.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.707+0000 7f5d1ea00640 1 -- 172.21.15.32:0/4191469167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d28103500 msgr2=0x7f5d281a3790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:51.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.707+0000 7f5d1ea00640 1 --2- 172.21.15.32:0/4191469167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d28103500 0x7f5d281a3790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:51.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.707+0000 7f5d1ea00640 1 -- 172.21.15.32:0/4191469167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d28102260 msgr2=0x7f5d2819c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:51.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.707+0000 7f5d1ea00640 1 --2- 172.21.15.32:0/4191469167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d28102260 0x7f5d2819c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:51.710 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.707+0000 7f5d1ea00640 1 -- 172.21.15.32:0/4191469167 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d281a3e00 con 0x7f5d28102bb0 2024-04-30T08:45:51.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.708+0000 7f5d1ea00640 1 --2- 172.21.15.32:0/4191469167 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d28102bb0 0x7f5d2819c800 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f5d1000e9f0 tx=0x7f5d1000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:45:51.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.708+0000 7f5d24c00640 1 -- 172.21.15.32:0/4191469167 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d1000cd80 con 0x7f5d28102bb0 2024-04-30T08:45:51.711 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.708+0000 7f5d2ca00640 1 -- 172.21.15.32:0/4191469167 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d281a40f0 con 0x7f5d28102bb0 2024-04-30T08:45:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.708+0000 7f5d24c00640 1 -- 172.21.15.32:0/4191469167 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f5d100108e0 con 0x7f5d28102bb0 2024-04-30T08:45:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.708+0000 7f5d24c00640 1 -- 172.21.15.32:0/4191469167 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d10018980 con 0x7f5d28102bb0 2024-04-30T08:45:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.708+0000 7f5d2ca00640 1 -- 172.21.15.32:0/4191469167 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d281a4640 con 0x7f5d28102bb0 2024-04-30T08:45:51.712 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.709+0000 7f5d2ca00640 1 -- 172.21.15.32:0/4191469167 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d2810d2a0 con 0x7f5d28102bb0 2024-04-30T08:45:51.715 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.714+0000 7f5d24c00640 1 -- 172.21.15.32:0/4191469167 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f5d10010430 con 0x7f5d28102bb0 2024-04-30T08:45:51.717 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.716+0000 7f5d24c00640 1 --2- 172.21.15.32:0/4191469167 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f5d0c0afae0 0x7f5d0c0b1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:51.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.716+0000 7f5d24c00640 1 -- 172.21.15.32:0/4191469167 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7f5d100df7e0 con 0x7f5d28102bb0 2024-04-30T08:45:51.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.716+0000 7f5d27400640 1 --2- 172.21.15.32:0/4191469167 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f5d0c0afae0 0x7f5d0c0b1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:51.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.716+0000 7f5d24c00640 1 -- 172.21.15.32:0/4191469167 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5d100a7960 con 0x7f5d28102bb0 2024-04-30T08:45:51.718 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.717+0000 7f5d27400640 1 --2- 172.21.15.32:0/4191469167 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f5d0c0afae0 0x7f5d0c0b1ff0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f5d180094e0 tx=0x7f5d18005d80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:45:51.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:51 smithi071 ceph-mon[33385]: pgmap v745: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 426 B/s wr, 11 op/s 2024-04-30T08:45:51.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:51 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/684707548' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:45:51.928 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.927+0000 7f5d2ca00640 1 -- 172.21.15.32:0/4191469167 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5d2802dda0 con 0x7f5d28102bb0 2024-04-30T08:45:51.930 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.929+0000 7f5d24c00640 1 -- 172.21.15.32:0/4191469167 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+24730 (secure 0 0 0) 0x7f5d100abc60 con 0x7f5d28102bb0 2024-04-30T08:45:51.931 INFO:teuthology.orchestra.run.smithi032.stdout: 2024-04-30T08:45:51.931 INFO:teuthology.orchestra.run.smithi032.stdout:{"epoch":85,"fsid":"98c0a670-06ca-11ef-bc93-c7b262605968","created":"2024-04-30T08:22:11.122875+0000","modified":"2024-04-30T08:33:31.872704+0000","last_up_change":"2024-04-30T08:30:50.350439+0000","last_in_change":"2024-04-30T08:30:30.210620+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-30T08:26:19.728331+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-30T08:31:49.800542+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"75","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-30T08:31:51.211320+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"76","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-04-30T08:31:55.716625+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"a7e060c8-1cbb-418f-9a03-06693f6b3f52","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6802","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6803","nonce":2624700402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6804","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6805","nonce":2624700402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6808","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6809","nonce":2624700402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6806","nonce":2624700402},{"type":"v1","addr":"172.21.15.32:6807","nonce":2624700402}]},"public_addr":"172.21.15.32:6803/2624700402","cluster_addr":"172.21.15.32:6805/2624700402","heartbeat_back_addr":"172.21.15.32:6809/2624700402","heartbeat_front_addr":"172.21.15.32:6807/2624700402","state":["exists","up"]},{"osd":1,"uuid":"72ef2fb1-e5ca-48ef-aa3a-4796a64b922f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6800","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6801","nonce":444649711}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6802","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6803","nonce":444649711}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6806","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6807","nonce":444649711}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6804","nonce":444649711},{"type":"v1","addr":"172.21.15.71:6805","nonce":444649711}]},"public_addr":"172.21.15.71:6801/444649711","cluster_addr":"172.21.15.71:6803/444649711","heartbeat_back_addr":"172.21.15.71:6807/444649711","heartbeat_front_addr":"172.21.15.71:6805/444649711","state":["exists","up"]},{"osd":2,"uuid":"704e5e8a-935e-47c7-ae28-27119876ac09","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6801","nonce":3255528671}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6803","nonce":3255528671}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6807","nonce":3255528671}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":3255528671},{"type":"v1","addr":"172.21.15.189:6805","nonce":3255528671}]},"public_addr":"172.21.15.189:6801/3255528671","cluster_addr":"172.21.15.189:6803/3255528671","heartbeat_back_addr":"172.21.15.189:6807/3255528671","heartbeat_front_addr":"172.21.15.189:6805/3255528671","state":["exists","up"]},{"osd":3,"uuid":"575004f7-6559-4a62-863e-e33078b0eebf","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6810","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6811","nonce":1384702112}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6812","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6813","nonce":1384702112}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6816","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6817","nonce":1384702112}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6814","nonce":1384702112},{"type":"v1","addr":"172.21.15.32:6815","nonce":1384702112}]},"public_addr":"172.21.15.32:6811/1384702112","cluster_addr":"172.21.15.32:6813/1384702112","heartbeat_back_addr":"172.21.15.32:6817/1384702112","heartbeat_front_addr":"172.21.15.32:6815/1384702112","state":["exists","up"]},{"osd":4,"uuid":"e1c1e894-074d-4c78-89ef-9e5a114268bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6808","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6809","nonce":4250534574}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6810","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6811","nonce":4250534574}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6814","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6815","nonce":4250534574}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6812","nonce":4250534574},{"type":"v1","addr":"172.21.15.71:6813","nonce":4250534574}]},"public_addr":"172.21.15.71:6809/4250534574","cluster_addr":"172.21.15.71:6811/4250534574","heartbeat_back_addr":"172.21.15.71:6815/4250534574","heartbeat_front_addr":"172.21.15.71:6813/4250534574","state":["exists","up"]},{"osd":5,"uuid":"12c022d4-fac6-40c4-ae7a-691571bee5d0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":36,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6809","nonce":4245346198}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6811","nonce":4245346198}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6815","nonce":4245346198}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4245346198},{"type":"v1","addr":"172.21.15.189:6813","nonce":4245346198}]},"public_addr":"172.21.15.189:6809/4245346198","cluster_addr":"172.21.15.189:6811/4245346198","heartbeat_back_addr":"172.21.15.189:6815/4245346198","heartbeat_front_addr":"172.21.15.189:6813/4245346198","state":["exists","up"]},{"osd":6,"uuid":"b413f8ee-3d4a-415a-a2ba-d68437b7330d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":42,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6818","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6819","nonce":1910389190}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6820","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6821","nonce":1910389190}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6824","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6825","nonce":1910389190}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6822","nonce":1910389190},{"type":"v1","addr":"172.21.15.32:6823","nonce":1910389190}]},"public_addr":"172.21.15.32:6819/1910389190","cluster_addr":"172.21.15.32:6821/1910389190","heartbeat_back_addr":"172.21.15.32:6825/1910389190","heartbeat_front_addr":"172.21.15.32:6823/1910389190","state":["exists","up"]},{"osd":7,"uuid":"aac64fb3-fc13-4126-b842-7b06ad5ce238","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":48,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6816","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6817","nonce":692607380}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6818","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6819","nonce":692607380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6822","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6823","nonce":692607380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6820","nonce":692607380},{"type":"v1","addr":"172.21.15.71:6821","nonce":692607380}]},"public_addr":"172.21.15.71:6817/692607380","cluster_addr":"172.21.15.71:6819/692607380","heartbeat_back_addr":"172.21.15.71:6823/692607380","heartbeat_front_addr":"172.21.15.71:6821/692607380","state":["exists","up"]},{"osd":8,"uuid":"cace7fec-ce53-4cdf-a6a4-2c7d8326e0e5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":54,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6817","nonce":3973697069}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6819","nonce":3973697069}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6823","nonce":3973697069}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3973697069},{"type":"v1","addr":"172.21.15.189:6821","nonce":3973697069}]},"public_addr":"172.21.15.189:6817/3973697069","cluster_addr":"172.21.15.189:6819/3973697069","heartbeat_back_addr":"172.21.15.189:6823/3973697069","heartbeat_front_addr":"172.21.15.189:6821/3973697069","state":["exists","up"]},{"osd":9,"uuid":"ef4a1645-ae6e-45c3-82ad-161510e62aa9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6826","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6827","nonce":46066685}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6828","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6829","nonce":46066685}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6832","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6833","nonce":46066685}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.32:6830","nonce":46066685},{"type":"v1","addr":"172.21.15.32:6831","nonce":46066685}]},"public_addr":"172.21.15.32:6827/46066685","cluster_addr":"172.21.15.32:6829/46066685","heartbeat_back_addr":"172.21.15.32:6833/46066685","heartbeat_front_addr":"172.21.15.32:6831/46066685","state":["exists","up"]},{"osd":10,"uuid":"25b26a38-a644-409f-b941-59efdf04a3ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":65,"up_thru":82,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6824","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6825","nonce":1886290293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6826","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6827","nonce":1886290293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6830","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6831","nonce":1886290293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.71:6828","nonce":1886290293},{"type":"v1","addr":"172.21.15.71:6829","nonce":1886290293}]},"public_addr":"172.21.15.71:6825/1886290293","cluster_addr":"172.21.15.71:6827/1886290293","heartbeat_back_addr":"172.21.15.71:6831/1886290293","heartbeat_front_addr":"172.21.15.71:6829/1886290293","state":["exists","up"]},{"osd":11,"uuid":"ab96b3ff-3688-455b-af64-60226c5bfbff","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":71,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6825","nonce":850185001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6827","nonce":850185001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6831","nonce":850185001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":850185001},{"type":"v1","addr":"172.21.15.189:6829","nonce":850185001}]},"public_addr":"172.21.15.189:6825/850185001","cluster_addr":"172.21.15.189:6827/850185001","heartbeat_back_addr":"172.21.15.189:6831/850185001","heartbeat_front_addr":"172.21.15.189:6829/850185001","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:27.220308+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:25:53.230783+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:17.190642+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:26:52.603799+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:20.568671+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:27:46.158363+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:22.706933+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:28:50.665173+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:16.602410+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:29:53.814698+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:22.435189+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-30T08:30:48.651949+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":4,"to":8}]},{"pgid":"3.e","mappings":[{"from":4,"to":7}]},{"pgid":"3.17","mappings":[{"from":4,"to":6}]},{"pgid":"3.18","mappings":[{"from":3,"to":9}]},{"pgid":"3.1a","mappings":[{"from":5,"to":7}]},{"pgid":"3.28","mappings":[{"from":3,"to":6},{"from":5,"to":8}]},{"pgid":"3.30","mappings":[{"from":1,"to":9}]},{"pgid":"3.36","mappings":[{"from":4,"to":9}]},{"pgid":"4.8","mappings":[{"from":5,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.13","mappings":[{"from":9,"to":8}]},{"pgid":"4.14","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.32:0/1266897222":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/2915689533":"2024-05-01T08:23:29.526664+0000","172.21.15.32:6801/4019351306":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/425836193":"2024-05-01T08:23:29.526664+0000","172.21.15.32:0/1134075722":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6801/464872591":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/21059253":"2024-05-01T08:22:53.639893+0000","172.21.15.32:0/3245550894":"2024-05-01T08:22:53.639893+0000","172.21.15.32:6800/4019351306":"2024-05-01T08:23:29.526664+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-30T08:45:51.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.932+0000 7f5d2ca00640 1 -- 172.21.15.32:0/4191469167 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f5d0c0afae0 msgr2=0x7f5d0c0b1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:51.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.932+0000 7f5d2ca00640 1 --2- 172.21.15.32:0/4191469167 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f5d0c0afae0 0x7f5d0c0b1ff0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f5d180094e0 tx=0x7f5d18005d80 comp rx=0 tx=0).stop 2024-04-30T08:45:51.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.932+0000 7f5d2ca00640 1 -- 172.21.15.32:0/4191469167 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d28102bb0 msgr2=0x7f5d2819c800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:51.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.932+0000 7f5d2ca00640 1 --2- 172.21.15.32:0/4191469167 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d28102bb0 0x7f5d2819c800 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f5d1000e9f0 tx=0x7f5d1000eeb0 comp rx=0 tx=0).stop 2024-04-30T08:45:51.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.932+0000 7f5d2ca00640 1 -- 172.21.15.32:0/4191469167 shutdown_connections 2024-04-30T08:45:51.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.932+0000 7f5d2ca00640 1 --2- 172.21.15.32:0/4191469167 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f5d0c0afae0 0x7f5d0c0b1ff0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:51.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.932+0000 7f5d2ca00640 1 --2- 172.21.15.32:0/4191469167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5d28103500 0x7f5d281a3790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:51.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.932+0000 7f5d2ca00640 1 --2- 172.21.15.32:0/4191469167 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f5d28102bb0 0x7f5d2819c800 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:51.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.932+0000 7f5d2ca00640 1 --2- 172.21.15.32:0/4191469167 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f5d28102260 0x7f5d2819c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:51.934 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.932+0000 7f5d2ca00640 1 -- 172.21.15.32:0/4191469167 >> 172.21.15.32:0/4191469167 conn(0x7f5d280fda70 msgr2=0x7f5d2810a470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:45:51.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.932+0000 7f5d2ca00640 1 -- 172.21.15.32:0/4191469167 shutdown_connections 2024-04-30T08:45:51.935 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:51.932+0000 7f5d2ca00640 1 -- 172.21.15.32:0/4191469167 wait complete. 2024-04-30T08:45:51.963 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-30T08:45:52.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.230+0000 7fb8aea00640 1 -- 172.21.15.32:0/1718641090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8a810bb60 msgr2=0x7fb8a810df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:52.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.230+0000 7fb8aea00640 1 --2- 172.21.15.32:0/1718641090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8a810bb60 0x7fb8a810df80 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0009900 tx=0x7fb8a002a3f0 comp rx=0 tx=0).stop 2024-04-30T08:45:52.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.231+0000 7fb8aea00640 1 -- 172.21.15.32:0/1718641090 shutdown_connections 2024-04-30T08:45:52.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.231+0000 7fb8aea00640 1 --2- 172.21.15.32:0/1718641090 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8a810bb60 0x7fb8a810df80 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:52.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.231+0000 7fb8aea00640 1 --2- 172.21.15.32:0/1718641090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8a80769f0 0x7fb8a810b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:52.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.231+0000 7fb8aea00640 1 --2- 172.21.15.32:0/1718641090 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb8a80760a0 0x7fb8a80764b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:52.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.231+0000 7fb8aea00640 1 -- 172.21.15.32:0/1718641090 >> 172.21.15.32:0/1718641090 conn(0x7fb8a80fed60 msgr2=0x7fb8a8101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:45:52.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.231+0000 7fb8aea00640 1 -- 172.21.15.32:0/1718641090 shutdown_connections 2024-04-30T08:45:52.233 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.231+0000 7fb8aea00640 1 -- 172.21.15.32:0/1718641090 wait complete. 2024-04-30T08:45:52.234 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.232+0000 7fb8aea00640 1 Processor -- start 2024-04-30T08:45:52.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.232+0000 7fb8aea00640 1 -- start start 2024-04-30T08:45:52.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.232+0000 7fb8aea00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8a80760a0 0x7fb8a819c410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:52.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.232+0000 7fb8aea00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb8a80769f0 0x7fb8a819c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:52.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.233+0000 7fb8aea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8a810bb60 0x7fb8a81a38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:52.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.233+0000 7fb8aea00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a8111870 con 0x7fb8a80760a0 2024-04-30T08:45:52.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.233+0000 7fb8aea00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a81116f0 con 0x7fb8a80769f0 2024-04-30T08:45:52.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.233+0000 7fb8aea00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb8a81119f0 con 0x7fb8a810bb60 2024-04-30T08:45:52.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.233+0000 7fb8a7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8a80760a0 0x7fb8a819c410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:52.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.233+0000 7fb89f400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb8a80769f0 0x7fb8a819c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:52.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.233+0000 7fb8a7e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8a80760a0 0x7fb8a819c410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58446/0 (socket says 172.21.15.32:58446) 2024-04-30T08:45:52.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.233+0000 7fb89f400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb8a80769f0 0x7fb8a819c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:34684/0 (socket says 172.21.15.32:34684) 2024-04-30T08:45:52.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.233+0000 7fb8a7e00640 1 -- 172.21.15.32:0/457399556 learned_addr learned my addr 172.21.15.32:0/457399556 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:45:52.235 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.233+0000 7fb8aca00640 1 --2- 172.21.15.32:0/457399556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8a810bb60 0x7fb8a81a38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:52.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.233+0000 7fb89f400640 1 -- 172.21.15.32:0/457399556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8a810bb60 msgr2=0x7fb8a81a38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:52.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.233+0000 7fb89f400640 1 --2- 172.21.15.32:0/457399556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8a810bb60 0x7fb8a81a38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:52.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.233+0000 7fb89f400640 1 -- 172.21.15.32:0/457399556 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8a80760a0 msgr2=0x7fb8a819c410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:52.236 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.233+0000 7fb89f400640 1 --2- 172.21.15.32:0/457399556 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8a80760a0 0x7fb8a819c410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:52.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.233+0000 7fb89f400640 1 -- 172.21.15.32:0/457399556 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8a81a3f50 con 0x7fb8a80769f0 2024-04-30T08:45:52.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.234+0000 7fb89f400640 1 --2- 172.21.15.32:0/457399556 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb8a80769f0 0x7fb8a819c950 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fb89000e9f0 tx=0x7fb89000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:45:52.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.234+0000 7fb8a5600640 1 -- 172.21.15.32:0/457399556 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb89000cd80 con 0x7fb8a80769f0 2024-04-30T08:45:52.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.234+0000 7fb8aea00640 1 -- 172.21.15.32:0/457399556 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb8a81a4240 con 0x7fb8a80769f0 2024-04-30T08:45:52.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.234+0000 7fb8aea00640 1 -- 172.21.15.32:0/457399556 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb8a81a47e0 con 0x7fb8a80769f0 2024-04-30T08:45:52.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.234+0000 7fb8a5600640 1 -- 172.21.15.32:0/457399556 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb8900035f0 con 0x7fb8a80769f0 2024-04-30T08:45:52.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.234+0000 7fb8a5600640 1 -- 172.21.15.32:0/457399556 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb890018690 con 0x7fb8a80769f0 2024-04-30T08:45:52.237 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.235+0000 7fb8aea00640 1 -- 172.21.15.32:0/457399556 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb8a810ad20 con 0x7fb8a80769f0 2024-04-30T08:45:52.240 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.238+0000 7fb8a5600640 1 -- 172.21.15.32:0/457399556 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7fb890004020 con 0x7fb8a80769f0 2024-04-30T08:45:52.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.240+0000 7fb8a5600640 1 --2- 172.21.15.32:0/457399556 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb88c0afaf0 0x7fb88c0b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:52.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.240+0000 7fb8a5600640 1 -- 172.21.15.32:0/457399556 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 9216+0+0 (secure 0 0 0) 0x7fb8900df2f0 con 0x7fb8a80769f0 2024-04-30T08:45:52.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.240+0000 7fb8a7e00640 1 --2- 172.21.15.32:0/457399556 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb88c0afaf0 0x7fb88c0b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:52.242 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.241+0000 7fb8a7e00640 1 --2- 172.21.15.32:0/457399556 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb88c0afaf0 0x7fb88c0b2000 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fb898005f40 tx=0x7fb898009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:45:52.243 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.241+0000 7fb8a5600640 1 -- 172.21.15.32:0/457399556 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb8900a7b50 con 0x7fb8a80769f0 2024-04-30T08:45:52.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:52 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/253013798' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:45:52.441 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:52 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/4191469167' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:45:52.492 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.490+0000 7fb8aea00640 1 -- 172.21.15.32:0/457399556 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fb8a8063820 con 0x7fb8a80769f0 2024-04-30T08:45:52.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.522+0000 7fb8a5600640 1 -- 172.21.15.32:0/457399556 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v133) ==== 161+0+0 (secure 0 0 0) 0x7fb8900a7450 con 0x7fb8a80769f0 2024-04-30T08:45:52.523 INFO:teuthology.orchestra.run.smithi032.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-30T08:45:52.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.525+0000 7fb8aea00640 1 -- 172.21.15.32:0/457399556 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb88c0afaf0 msgr2=0x7fb88c0b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:52.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.525+0000 7fb8aea00640 1 --2- 172.21.15.32:0/457399556 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb88c0afaf0 0x7fb88c0b2000 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fb898005f40 tx=0x7fb898009500 comp rx=0 tx=0).stop 2024-04-30T08:45:52.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.525+0000 7fb8aea00640 1 -- 172.21.15.32:0/457399556 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb8a80769f0 msgr2=0x7fb8a819c950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:52.527 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.525+0000 7fb8aea00640 1 --2- 172.21.15.32:0/457399556 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb8a80769f0 0x7fb8a819c950 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fb89000e9f0 tx=0x7fb89000eeb0 comp rx=0 tx=0).stop 2024-04-30T08:45:52.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.525+0000 7fb8aea00640 1 -- 172.21.15.32:0/457399556 shutdown_connections 2024-04-30T08:45:52.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.525+0000 7fb8aea00640 1 --2- 172.21.15.32:0/457399556 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb88c0afaf0 0x7fb88c0b2000 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:52.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.525+0000 7fb8aea00640 1 --2- 172.21.15.32:0/457399556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb8a810bb60 0x7fb8a81a38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:52.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.525+0000 7fb8aea00640 1 --2- 172.21.15.32:0/457399556 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb8a80769f0 0x7fb8a819c950 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:52.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.525+0000 7fb8aea00640 1 --2- 172.21.15.32:0/457399556 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb8a80760a0 0x7fb8a819c410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:52.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.525+0000 7fb8aea00640 1 -- 172.21.15.32:0/457399556 >> 172.21.15.32:0/457399556 conn(0x7fb8a80fed60 msgr2=0x7fb8a80ffba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:45:52.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.526+0000 7fb8aea00640 1 -- 172.21.15.32:0/457399556 shutdown_connections 2024-04-30T08:45:52.528 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.526+0000 7fb8aea00640 1 -- 172.21.15.32:0/457399556 wait complete. 2024-04-30T08:45:52.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:52 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/253013798' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:45:52.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:52 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/4191469167' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:45:52.558 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-30T08:45:52.638 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:52 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/253013798' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-30T08:45:52.638 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:52 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/4191469167' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-30T08:45:52.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.811+0000 7fcca7400640 1 -- 172.21.15.32:0/2457893217 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcca0109ab0 msgr2=0x7fcca010bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:52.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.811+0000 7fcca7400640 1 --2- 172.21.15.32:0/2457893217 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcca0109ab0 0x7fcca010bed0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7fcc98009be0 tx=0x7fcc9802a6c0 comp rx=0 tx=0).stop 2024-04-30T08:45:52.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.811+0000 7fcca7400640 1 -- 172.21.15.32:0/2457893217 shutdown_connections 2024-04-30T08:45:52.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.811+0000 7fcca7400640 1 --2- 172.21.15.32:0/2457893217 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcca0109ab0 0x7fcca010bed0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:52.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.811+0000 7fcca7400640 1 --2- 172.21.15.32:0/2457893217 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcca0104db0 0x7fcca01051c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:52.813 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.811+0000 7fcca7400640 1 --2- 172.21.15.32:0/2457893217 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcca0103b70 0x7fcca0104870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:52.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.811+0000 7fcca7400640 1 -- 172.21.15.32:0/2457893217 >> 172.21.15.32:0/2457893217 conn(0x7fcca00fed60 msgr2=0x7fcca0101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:45:52.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.811+0000 7fcca7400640 1 -- 172.21.15.32:0/2457893217 shutdown_connections 2024-04-30T08:45:52.814 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.811+0000 7fcca7400640 1 -- 172.21.15.32:0/2457893217 wait complete. 2024-04-30T08:45:52.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.812+0000 7fcca7400640 1 Processor -- start 2024-04-30T08:45:52.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.812+0000 7fcca7400640 1 -- start start 2024-04-30T08:45:52.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.813+0000 7fcca7400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcca0103b70 0x7fcca019a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:52.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.813+0000 7fcca7400640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcca0104db0 0x7fcca019a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:52.816 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.813+0000 7fcca7400640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcca0109ab0 0x7fcca01a1710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:52.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.813+0000 7fcca7400640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fcca010f680 con 0x7fcca0104db0 2024-04-30T08:45:52.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.813+0000 7fcca7400640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fcca010f500 con 0x7fcca0109ab0 2024-04-30T08:45:52.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.813+0000 7fcca7400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fcca010f800 con 0x7fcca0103b70 2024-04-30T08:45:52.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.813+0000 7fcc9fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcca0104db0 0x7fcca019a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:52.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.813+0000 7fcc9fe00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcca0104db0 0x7fcca019a7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58512/0 (socket says 172.21.15.32:58512) 2024-04-30T08:45:52.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.813+0000 7fcc9fe00640 1 -- 172.21.15.32:0/1330382249 learned_addr learned my addr 172.21.15.32:0/1330382249 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:45:52.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.813+0000 7fcca4a00640 1 --2- 172.21.15.32:0/1330382249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcca0103b70 0x7fcca019a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:52.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.813+0000 7fcca5400640 1 --2- 172.21.15.32:0/1330382249 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcca0109ab0 0x7fcca01a1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:52.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.813+0000 7fcc9fe00640 1 -- 172.21.15.32:0/1330382249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcca0103b70 msgr2=0x7fcca019a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:52.817 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.813+0000 7fcc9fe00640 1 --2- 172.21.15.32:0/1330382249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcca0103b70 0x7fcca019a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:52.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.813+0000 7fcc9fe00640 1 -- 172.21.15.32:0/1330382249 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcca0109ab0 msgr2=0x7fcca01a1710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:52.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.813+0000 7fcc9fe00640 1 --2- 172.21.15.32:0/1330382249 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcca0109ab0 0x7fcca01a1710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:52.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.813+0000 7fcc9fe00640 1 -- 172.21.15.32:0/1330382249 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcca01a1d80 con 0x7fcca0104db0 2024-04-30T08:45:52.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.814+0000 7fcc9fe00640 1 --2- 172.21.15.32:0/1330382249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcca0104db0 0x7fcca019a7f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c00e9c0 tx=0x7fcc8c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:45:52.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.814+0000 7fcc9d600640 1 -- 172.21.15.32:0/1330382249 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc8c00cde0 con 0x7fcca0104db0 2024-04-30T08:45:52.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.814+0000 7fcc9d600640 1 -- 172.21.15.32:0/1330382249 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fcc8c0035f0 con 0x7fcca0104db0 2024-04-30T08:45:52.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.814+0000 7fcca7400640 1 -- 172.21.15.32:0/1330382249 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcca01a2070 con 0x7fcca0104db0 2024-04-30T08:45:52.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.814+0000 7fcc9d600640 1 -- 172.21.15.32:0/1330382249 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc8c010690 con 0x7fcca0104db0 2024-04-30T08:45:52.818 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.814+0000 7fcca7400640 1 -- 172.21.15.32:0/1330382249 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcca01a25c0 con 0x7fcca0104db0 2024-04-30T08:45:52.819 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.816+0000 7fcca7400640 1 -- 172.21.15.32:0/1330382249 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcca01088c0 con 0x7fcca0104db0 2024-04-30T08:45:52.821 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.820+0000 7fcc9d600640 1 -- 172.21.15.32:0/1330382249 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7fcc8c004000 con 0x7fcca0104db0 2024-04-30T08:45:52.822 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.821+0000 7fcc9d600640 1 --2- 172.21.15.32:0/1330382249 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fcc880afaf0 0x7fcc880b2000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:52.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.821+0000 7fcc9d600640 1 -- 172.21.15.32:0/1330382249 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9646+0+0 (secure 0 0 0) 0x7fcc8c0df5f0 con 0x7fcca0104db0 2024-04-30T08:45:52.823 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.821+0000 7fcca4a00640 1 --2- 172.21.15.32:0/1330382249 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fcc880afaf0 0x7fcc880b2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:52.824 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.822+0000 7fcca4a00640 1 --2- 172.21.15.32:0/1330382249 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fcc880afaf0 0x7fcc880b2000 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fcca0104770 tx=0x7fcc94005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:45:52.825 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:52.823+0000 7fcc9d600640 1 -- 172.21.15.32:0/1330382249 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcc8c0a7c80 con 0x7fcca0104db0 2024-04-30T08:45:52.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:52 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mon-a[28327]: 2024-04-30T08:45:52.511+0000 7f343f200700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-30T08:45:52.942 INFO:journalctl@ceph.mds.g.smithi032.stdout:Apr 30 08:45:52 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-g[70109]: ignoring --setuser ceph since I am not root 2024-04-30T08:45:52.942 INFO:journalctl@ceph.mds.g.smithi032.stdout:Apr 30 08:45:52 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-g[70109]: ignoring --setgroup ceph since I am not root 2024-04-30T08:45:52.942 INFO:journalctl@ceph.mds.g.smithi032.stdout:Apr 30 08:45:52 smithi032 ceph-mds[70113]: -- 172.21.15.32:0/2434243182 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ebf8475a40 con 0x55ebf84bda80 2024-04-30T08:45:52.942 INFO:journalctl@ceph.mds.g.smithi032.stdout:Apr 30 08:45:52 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-g[70109]: starting mds.g at 2024-04-30T08:45:52.942 INFO:journalctl@ceph.mds.d.smithi032.stdout:Apr 30 08:45:52 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-d[69485]: ignoring --setuser ceph since I am not root 2024-04-30T08:45:52.942 INFO:journalctl@ceph.mds.d.smithi032.stdout:Apr 30 08:45:52 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-d[69485]: ignoring --setgroup ceph since I am not root 2024-04-30T08:45:52.942 INFO:journalctl@ceph.mds.d.smithi032.stdout:Apr 30 08:45:52 smithi032 ceph-mds[69489]: -- 172.21.15.32:0/2083949983 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556afd28ba40 con 0x556afd2d3180 2024-04-30T08:45:52.943 INFO:journalctl@ceph.mds.d.smithi032.stdout:Apr 30 08:45:52 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-d[69485]: starting mds.d at 2024-04-30T08:45:53.004 INFO:journalctl@ceph.mds.h.smithi071.stdout:Apr 30 08:45:52 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-h[63716]: ignoring --setuser ceph since I am not root 2024-04-30T08:45:53.004 INFO:journalctl@ceph.mds.h.smithi071.stdout:Apr 30 08:45:52 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-h[63716]: ignoring --setgroup ceph since I am not root 2024-04-30T08:45:53.004 INFO:journalctl@ceph.mds.h.smithi071.stdout:Apr 30 08:45:52 smithi071 ceph-mds[63720]: -- 172.21.15.71:0/2779302883 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561cf1169a40 con 0x561cf11b1180 2024-04-30T08:45:53.004 INFO:journalctl@ceph.mds.h.smithi071.stdout:Apr 30 08:45:52 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-h[63716]: starting mds.h at 2024-04-30T08:45:53.004 INFO:journalctl@ceph.mds.k.smithi071.stdout:Apr 30 08:45:52 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-k[63071]: ignoring --setuser ceph since I am not root 2024-04-30T08:45:53.004 INFO:journalctl@ceph.mds.k.smithi071.stdout:Apr 30 08:45:52 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-k[63071]: ignoring --setgroup ceph since I am not root 2024-04-30T08:45:53.004 INFO:journalctl@ceph.mds.k.smithi071.stdout:Apr 30 08:45:52 smithi071 ceph-mds[63075]: -- 172.21.15.71:0/3112515851 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cd1a20ba40 con 0x55cd1a253180 2024-04-30T08:45:53.004 INFO:journalctl@ceph.mds.k.smithi071.stdout:Apr 30 08:45:52 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-k[63071]: starting mds.k at 2024-04-30T08:45:53.005 INFO:journalctl@ceph.mds.b.smithi071.stdout:Apr 30 08:45:52 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-b[64026]: ignoring --setuser ceph since I am not root 2024-04-30T08:45:53.005 INFO:journalctl@ceph.mds.b.smithi071.stdout:Apr 30 08:45:52 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-b[64026]: ignoring --setgroup ceph since I am not root 2024-04-30T08:45:53.005 INFO:journalctl@ceph.mds.b.smithi071.stdout:Apr 30 08:45:52 smithi071 ceph-mds[64030]: -- 172.21.15.71:0/35780204 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563851285a40 con 0x5638512cd600 2024-04-30T08:45:53.005 INFO:journalctl@ceph.mds.b.smithi071.stdout:Apr 30 08:45:52 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-b[64026]: starting mds.b at 2024-04-30T08:45:53.005 INFO:journalctl@ceph.mds.e.smithi071.stdout:Apr 30 08:45:52 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-e[63391]: ignoring --setuser ceph since I am not root 2024-04-30T08:45:53.005 INFO:journalctl@ceph.mds.e.smithi071.stdout:Apr 30 08:45:52 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-e[63391]: ignoring --setgroup ceph since I am not root 2024-04-30T08:45:53.005 INFO:journalctl@ceph.mds.e.smithi071.stdout:Apr 30 08:45:52 smithi071 ceph-mds[63395]: -- 172.21.15.71:0/146359414 <== mon.2 v2:172.21.15.71:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563c70e25a40 con 0x563c70e6d600 2024-04-30T08:45:53.006 INFO:journalctl@ceph.mds.e.smithi071.stdout:Apr 30 08:45:52 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-e[63391]: starting mds.e at 2024-04-30T08:45:53.051 INFO:journalctl@ceph.mds.f.smithi189.stdout:Apr 30 08:45:52 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-f[63232]: ignoring --setuser ceph since I am not root 2024-04-30T08:45:53.051 INFO:journalctl@ceph.mds.f.smithi189.stdout:Apr 30 08:45:52 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-f[63232]: ignoring --setgroup ceph since I am not root 2024-04-30T08:45:53.051 INFO:journalctl@ceph.mds.f.smithi189.stdout:Apr 30 08:45:52 smithi189 ceph-mds[63236]: -- 172.21.15.189:0/3287432544 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55f0e82efa40 con 0x55f0e8337180 2024-04-30T08:45:53.052 INFO:journalctl@ceph.mds.f.smithi189.stdout:Apr 30 08:45:52 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-f[63232]: starting mds.f at 2024-04-30T08:45:53.052 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 30 08:45:52 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-i[63845]: ignoring --setuser ceph since I am not root 2024-04-30T08:45:53.052 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 30 08:45:52 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-i[63845]: ignoring --setgroup ceph since I am not root 2024-04-30T08:45:53.052 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 30 08:45:52 smithi189 ceph-mds[63849]: -- 172.21.15.189:0/4170399716 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e0af3c9a40 con 0x55e0af411a80 2024-04-30T08:45:53.052 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 30 08:45:52 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-i[63845]: starting mds.i at 2024-04-30T08:45:53.053 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 30 08:45:52 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-c[63536]: ignoring --setuser ceph since I am not root 2024-04-30T08:45:53.053 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 30 08:45:52 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-c[63536]: ignoring --setgroup ceph since I am not root 2024-04-30T08:45:53.053 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 30 08:45:52 smithi189 ceph-mds[63540]: -- 172.21.15.189:0/2490227610 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e0befe3a40 con 0x55e0bf02b180 2024-04-30T08:45:53.053 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 30 08:45:52 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-c[63536]: starting mds.c at 2024-04-30T08:45:53.053 INFO:journalctl@ceph.mds.l.smithi189.stdout:Apr 30 08:45:52 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-l[62922]: ignoring --setuser ceph since I am not root 2024-04-30T08:45:53.053 INFO:journalctl@ceph.mds.l.smithi189.stdout:Apr 30 08:45:52 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-l[62922]: ignoring --setgroup ceph since I am not root 2024-04-30T08:45:53.053 INFO:journalctl@ceph.mds.l.smithi189.stdout:Apr 30 08:45:52 smithi189 ceph-mds[62926]: -- 172.21.15.189:0/2107415 <== mon.1 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c63532da40 con 0x55c635375180 2024-04-30T08:45:53.053 INFO:journalctl@ceph.mds.l.smithi189.stdout:Apr 30 08:45:52 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-l[62922]: starting mds.l at 2024-04-30T08:45:53.065 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.064+0000 7fcca7400640 1 -- 172.21.15.32:0/1330382249 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7fcca002dd60 con 0x7fcca0104db0 2024-04-30T08:45:53.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:53 smithi032 ceph-mon[28331]: pgmap v746: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 767 B/s wr, 11 op/s 2024-04-30T08:45:53.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:53 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/457399556' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-30T08:45:53.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:53 smithi032 ceph-mon[28331]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-30T08:45:53.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:53 smithi032 ceph-mon[28331]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-30T08:45:53.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:53 smithi032 ceph-mon[28331]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-30T08:45:53.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:53 smithi032 ceph-mon[28331]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-30T08:45:53.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:53 smithi032 ceph-mon[28331]: osdmap e86: 12 total, 12 up, 12 in 2024-04-30T08:45:53.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:53 smithi032 ceph-mon[28331]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-30T08:45:53.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:53 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1330382249' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-30T08:45:53.520 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.518+0000 7fcc9d600640 1 -- 172.21.15.32:0/1330382249 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v134) ==== 104+0+0 (secure 0 0 0) 0x7fcc8c0a7580 con 0x7fcca0104db0 2024-04-30T08:45:53.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.522+0000 7fcca7400640 1 -- 172.21.15.32:0/1330382249 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fcc880afaf0 msgr2=0x7fcc880b2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:53.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.522+0000 7fcca7400640 1 --2- 172.21.15.32:0/1330382249 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fcc880afaf0 0x7fcc880b2000 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fcca0104770 tx=0x7fcc94005f90 comp rx=0 tx=0).stop 2024-04-30T08:45:53.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.522+0000 7fcca7400640 1 -- 172.21.15.32:0/1330382249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcca0104db0 msgr2=0x7fcca019a7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:53.523 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.522+0000 7fcca7400640 1 --2- 172.21.15.32:0/1330382249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcca0104db0 0x7fcca019a7f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fcc8c00e9c0 tx=0x7fcc8c00ee80 comp rx=0 tx=0).stop 2024-04-30T08:45:53.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.522+0000 7fcca7400640 1 -- 172.21.15.32:0/1330382249 shutdown_connections 2024-04-30T08:45:53.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.522+0000 7fcca7400640 1 --2- 172.21.15.32:0/1330382249 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fcc880afaf0 0x7fcc880b2000 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:53.524 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.522+0000 7fcca7400640 1 --2- 172.21.15.32:0/1330382249 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fcca0109ab0 0x7fcca01a1710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:53.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.522+0000 7fcca7400640 1 --2- 172.21.15.32:0/1330382249 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fcca0104db0 0x7fcca019a7f0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:53.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.522+0000 7fcca7400640 1 --2- 172.21.15.32:0/1330382249 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcca0103b70 0x7fcca019a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:53.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.522+0000 7fcca7400640 1 -- 172.21.15.32:0/1330382249 >> 172.21.15.32:0/1330382249 conn(0x7fcca00fed60 msgr2=0x7fcca0105a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:45:53.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.522+0000 7fcca7400640 1 -- 172.21.15.32:0/1330382249 shutdown_connections 2024-04-30T08:45:53.525 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.522+0000 7fcca7400640 1 -- 172.21.15.32:0/1330382249 wait complete. 2024-04-30T08:45:53.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:53 smithi189 ceph-mon[33038]: pgmap v746: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 767 B/s wr, 11 op/s 2024-04-30T08:45:53.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:53 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/457399556' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-30T08:45:53.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:53 smithi189 ceph-mon[33038]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-30T08:45:53.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:53 smithi189 ceph-mon[33038]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-30T08:45:53.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:53 smithi189 ceph-mon[33038]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-30T08:45:53.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:53 smithi189 ceph-mon[33038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-30T08:45:53.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:53 smithi189 ceph-mon[33038]: osdmap e86: 12 total, 12 up, 12 in 2024-04-30T08:45:53.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:53 smithi189 ceph-mon[33038]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-30T08:45:53.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:53 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1330382249' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-30T08:45:53.557 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-30T08:45:53.753 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:53 smithi071 ceph-mon[33385]: pgmap v746: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 767 B/s wr, 11 op/s 2024-04-30T08:45:53.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:53 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/457399556' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-30T08:45:53.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:53 smithi071 ceph-mon[33385]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-30T08:45:53.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:53 smithi071 ceph-mon[33385]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-30T08:45:53.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:53 smithi071 ceph-mon[33385]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-30T08:45:53.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:53 smithi071 ceph-mon[33385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-30T08:45:53.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:53 smithi071 ceph-mon[33385]: osdmap e86: 12 total, 12 up, 12 in 2024-04-30T08:45:53.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:53 smithi071 ceph-mon[33385]: fsmap cephfs:0/5 2 up:standby, 5 failed 2024-04-30T08:45:53.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:53 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1330382249' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-30T08:45:53.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.828+0000 7f185b200640 1 -- 172.21.15.32:0/4063707781 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f185c075010 msgr2=0x7f185c075420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:53.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.828+0000 7f185b200640 1 --2- 172.21.15.32:0/4063707781 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f185c075010 0x7f185c075420 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f184c009ca0 tx=0x7f184c02a8c0 comp rx=0 tx=0).stop 2024-04-30T08:45:53.830 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.828+0000 7f185b200640 1 -- 172.21.15.32:0/4063707781 shutdown_connections 2024-04-30T08:45:53.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.828+0000 7f185b200640 1 --2- 172.21.15.32:0/4063707781 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f185c110110 0x7f185c112530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:53.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.828+0000 7f185b200640 1 --2- 172.21.15.32:0/4063707781 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f185c075960 0x7f185c10fbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:53.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.828+0000 7f185b200640 1 --2- 172.21.15.32:0/4063707781 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f185c075010 0x7f185c075420 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:53.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.828+0000 7f185b200640 1 -- 172.21.15.32:0/4063707781 >> 172.21.15.32:0/4063707781 conn(0x7f185c0feba0 msgr2=0x7f185c101050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:45:53.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.828+0000 7f185b200640 1 -- 172.21.15.32:0/4063707781 shutdown_connections 2024-04-30T08:45:53.831 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.828+0000 7f185b200640 1 -- 172.21.15.32:0/4063707781 wait complete. 2024-04-30T08:45:53.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.829+0000 7f185b200640 1 Processor -- start 2024-04-30T08:45:53.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.830+0000 7f185b200640 1 -- start start 2024-04-30T08:45:53.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.830+0000 7f185b200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f185c075010 0x7f185c1a0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:53.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.830+0000 7f185b200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f185c075960 0x7f185c1a0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:53.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.830+0000 7f185b200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f185c110110 0x7f185c1a7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:53.833 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.830+0000 7f185b200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7f185c115af0 con 0x7f185c110110 2024-04-30T08:45:53.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.830+0000 7f185b200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7f185c115970 con 0x7f185c075010 2024-04-30T08:45:53.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.830+0000 7f185b200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7f185c115c70 con 0x7f185c075960 2024-04-30T08:45:53.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.830+0000 7f185a800640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f185c110110 0x7f185c1a7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:53.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.830+0000 7f1859e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f185c075010 0x7f185c1a0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:53.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.830+0000 7f1859e00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f185c075010 0x7f185c1a0700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:34772/0 (socket says 172.21.15.32:34772) 2024-04-30T08:45:53.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.830+0000 7f1859e00640 1 -- 172.21.15.32:0/2462191442 learned_addr learned my addr 172.21.15.32:0/2462191442 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:45:53.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.830+0000 7f1851400640 1 --2- 172.21.15.32:0/2462191442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f185c075960 0x7f185c1a0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:53.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.831+0000 7f1859e00640 1 -- 172.21.15.32:0/2462191442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f185c075960 msgr2=0x7f185c1a0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:53.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.831+0000 7f1859e00640 1 --2- 172.21.15.32:0/2462191442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f185c075960 0x7f185c1a0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:53.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.831+0000 7f1859e00640 1 -- 172.21.15.32:0/2462191442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f185c110110 msgr2=0x7f185c1a7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:53.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.831+0000 7f1859e00640 1 --2- 172.21.15.32:0/2462191442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f185c110110 0x7f185c1a7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:53.834 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.831+0000 7f1859e00640 1 -- 172.21.15.32:0/2462191442 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f185c1a8240 con 0x7f185c075010 2024-04-30T08:45:53.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.831+0000 7f1859e00640 1 --2- 172.21.15.32:0/2462191442 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f185c075010 0x7f185c1a0700 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f184c02ade0 tx=0x7f184c003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:45:53.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.832+0000 7f1853400640 1 -- 172.21.15.32:0/2462191442 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f184c02f380 con 0x7f185c075010 2024-04-30T08:45:53.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.832+0000 7f1853400640 1 -- 172.21.15.32:0/2462191442 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7f184c0317c0 con 0x7f185c075010 2024-04-30T08:45:53.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.832+0000 7f185b200640 1 -- 172.21.15.32:0/2462191442 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f185c1a84d0 con 0x7f185c075010 2024-04-30T08:45:53.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.832+0000 7f1853400640 1 -- 172.21.15.32:0/2462191442 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f184c02fc60 con 0x7f185c075010 2024-04-30T08:45:53.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.832+0000 7f185b200640 1 -- 172.21.15.32:0/2462191442 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f185c1a8a20 con 0x7f185c075010 2024-04-30T08:45:53.835 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.833+0000 7f185b200640 1 -- 172.21.15.32:0/2462191442 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f185c105da0 con 0x7f185c075010 2024-04-30T08:45:53.837 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.835+0000 7f1853400640 1 -- 172.21.15.32:0/2462191442 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7f184c004b40 con 0x7f185c075010 2024-04-30T08:45:53.838 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.837+0000 7f1853400640 1 --2- 172.21.15.32:0/2462191442 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f18400afc80 0x7f18400b2190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:53.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.837+0000 7f1853400640 1 -- 172.21.15.32:0/2462191442 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 9646+0+0 (secure 0 0 0) 0x7f184c100850 con 0x7f185c075010 2024-04-30T08:45:53.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.837+0000 7f1851400640 1 --2- 172.21.15.32:0/2462191442 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f18400afc80 0x7f18400b2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:53.839 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.838+0000 7f1851400640 1 --2- 172.21.15.32:0/2462191442 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f18400afc80 0x7f18400b2190 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f185c1a17a0 tx=0x7f1844005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:45:53.841 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:53.839+0000 7f1853400640 1 -- 172.21.15.32:0/2462191442 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f184c0c8e50 con 0x7f185c075010 2024-04-30T08:45:54.039 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.037+0000 7f185b200640 1 -- 172.21.15.32:0/2462191442 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f185c02dbd0 con 0x7f185c075010 2024-04-30T08:45:54.538 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.536+0000 7f1853400640 1 -- 172.21.15.32:0/2462191442 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v87) ==== 181+0+0 (secure 0 0 0) 0x7f184c0c8750 con 0x7f185c075010 2024-04-30T08:45:54.538 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_metadata' removed 2024-04-30T08:45:54.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.541+0000 7f185b200640 1 -- 172.21.15.32:0/2462191442 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f18400afc80 msgr2=0x7f18400b2190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:54.543 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.541+0000 7f185b200640 1 --2- 172.21.15.32:0/2462191442 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f18400afc80 0x7f18400b2190 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f185c1a17a0 tx=0x7f1844005f90 comp rx=0 tx=0).stop 2024-04-30T08:45:54.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.541+0000 7f185b200640 1 -- 172.21.15.32:0/2462191442 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f185c075010 msgr2=0x7f185c1a0700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:54.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.541+0000 7f185b200640 1 --2- 172.21.15.32:0/2462191442 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f185c075010 0x7f185c1a0700 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f184c02ade0 tx=0x7f184c003ba0 comp rx=0 tx=0).stop 2024-04-30T08:45:54.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.542+0000 7f185b200640 1 -- 172.21.15.32:0/2462191442 shutdown_connections 2024-04-30T08:45:54.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.542+0000 7f185b200640 1 --2- 172.21.15.32:0/2462191442 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7f18400afc80 0x7f18400b2190 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:54.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.542+0000 7f185b200640 1 --2- 172.21.15.32:0/2462191442 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7f185c110110 0x7f185c1a7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:54.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.542+0000 7f185b200640 1 --2- 172.21.15.32:0/2462191442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f185c075960 0x7f185c1a0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:54.544 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.542+0000 7f185b200640 1 --2- 172.21.15.32:0/2462191442 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7f185c075010 0x7f185c1a0700 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:54.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.542+0000 7f185b200640 1 -- 172.21.15.32:0/2462191442 >> 172.21.15.32:0/2462191442 conn(0x7f185c0feba0 msgr2=0x7f185c102f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:45:54.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.543+0000 7f185b200640 1 -- 172.21.15.32:0/2462191442 shutdown_connections 2024-04-30T08:45:54.545 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.543+0000 7f185b200640 1 -- 172.21.15.32:0/2462191442 wait complete. 2024-04-30T08:45:54.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:54 smithi189 ceph-mon[33038]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-30T08:45:54.550 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:54 smithi189 ceph-mon[33038]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-30T08:45:54.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:54 smithi189 ceph-mon[33038]: Cluster is now healthy 2024-04-30T08:45:54.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:54 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/1330382249' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-30T08:45:54.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-30T08:45:54.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:54 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6832/1028350478,v1:172.21.15.189:6834/1028350478] up:boot 2024-04-30T08:45:54.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:54 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6833/3867656310,v1:172.21.15.189:6835/3867656310] up:boot 2024-04-30T08:45:54.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:54 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6836/2703548161,v1:172.21.15.189:6838/2703548161] up:boot 2024-04-30T08:45:54.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:54 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.189:6837/1151682346,v1:172.21.15.189:6839/1151682346] up:boot 2024-04-30T08:45:54.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:54 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6833/2386823876,v1:172.21.15.71:6834/2386823876] up:boot 2024-04-30T08:45:54.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:54 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6832/1847024308,v1:172.21.15.71:6835/1847024308] up:boot 2024-04-30T08:45:54.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:54 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6836/782496760,v1:172.21.15.71:6838/782496760] up:boot 2024-04-30T08:45:54.551 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:54 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.71:6837/2357772129,v1:172.21.15.71:6839/2357772129] up:boot 2024-04-30T08:45:54.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:54 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.32:6836/2214894175,v1:172.21.15.32:6837/2214894175] up:boot 2024-04-30T08:45:54.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:54 smithi189 ceph-mon[33038]: mds.? [v2:172.21.15.32:6838/1222684735,v1:172.21.15.32:6839/1222684735] up:boot 2024-04-30T08:45:54.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-30T08:45:54.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:54 smithi189 ceph-mon[33038]: fsmap 12 up:standby 2024-04-30T08:45:54.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-30T08:45:54.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-30T08:45:54.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-30T08:45:54.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-30T08:45:54.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-30T08:45:54.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-30T08:45:54.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-30T08:45:54.552 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:54 smithi189 ceph-mon[33038]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-30T08:45:54.553 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:54 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/2462191442' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-30T08:45:54.553 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:54 smithi189 ceph-mon[33038]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-30T08:45:54.574 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:54 smithi032 ceph-mon[28331]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-30T08:45:54.574 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:54 smithi032 ceph-mon[28331]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-30T08:45:54.574 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:54 smithi032 ceph-mon[28331]: Cluster is now healthy 2024-04-30T08:45:54.574 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:54 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/1330382249' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-30T08:45:54.574 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-30T08:45:54.574 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:54 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6832/1028350478,v1:172.21.15.189:6834/1028350478] up:boot 2024-04-30T08:45:54.574 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:54 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6833/3867656310,v1:172.21.15.189:6835/3867656310] up:boot 2024-04-30T08:45:54.575 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:54 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6836/2703548161,v1:172.21.15.189:6838/2703548161] up:boot 2024-04-30T08:45:54.575 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:54 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.189:6837/1151682346,v1:172.21.15.189:6839/1151682346] up:boot 2024-04-30T08:45:54.575 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:54 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6833/2386823876,v1:172.21.15.71:6834/2386823876] up:boot 2024-04-30T08:45:54.575 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:54 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6832/1847024308,v1:172.21.15.71:6835/1847024308] up:boot 2024-04-30T08:45:54.575 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:54 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6836/782496760,v1:172.21.15.71:6838/782496760] up:boot 2024-04-30T08:45:54.575 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:54 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.71:6837/2357772129,v1:172.21.15.71:6839/2357772129] up:boot 2024-04-30T08:45:54.575 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:54 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.32:6836/2214894175,v1:172.21.15.32:6837/2214894175] up:boot 2024-04-30T08:45:54.575 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:54 smithi032 ceph-mon[28331]: mds.? [v2:172.21.15.32:6838/1222684735,v1:172.21.15.32:6839/1222684735] up:boot 2024-04-30T08:45:54.575 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-30T08:45:54.575 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:54 smithi032 ceph-mon[28331]: fsmap 12 up:standby 2024-04-30T08:45:54.575 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-30T08:45:54.576 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-30T08:45:54.576 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-30T08:45:54.576 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-30T08:45:54.576 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-30T08:45:54.576 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-30T08:45:54.576 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-30T08:45:54.576 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:54 smithi032 ceph-mon[28331]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-30T08:45:54.576 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:54 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/2462191442' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-30T08:45:54.576 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:54 smithi032 ceph-mon[28331]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-30T08:45:54.577 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-30T08:45:54.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:54 smithi071 ceph-mon[33385]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-30T08:45:54.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:54 smithi071 ceph-mon[33385]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-30T08:45:54.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:54 smithi071 ceph-mon[33385]: Cluster is now healthy 2024-04-30T08:45:54.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:54 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/1330382249' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-30T08:45:54.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-30T08:45:54.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:54 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6832/1028350478,v1:172.21.15.189:6834/1028350478] up:boot 2024-04-30T08:45:54.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:54 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6833/3867656310,v1:172.21.15.189:6835/3867656310] up:boot 2024-04-30T08:45:54.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:54 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6836/2703548161,v1:172.21.15.189:6838/2703548161] up:boot 2024-04-30T08:45:54.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:54 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.189:6837/1151682346,v1:172.21.15.189:6839/1151682346] up:boot 2024-04-30T08:45:54.754 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:54 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6833/2386823876,v1:172.21.15.71:6834/2386823876] up:boot 2024-04-30T08:45:54.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:54 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6832/1847024308,v1:172.21.15.71:6835/1847024308] up:boot 2024-04-30T08:45:54.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:54 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6836/782496760,v1:172.21.15.71:6838/782496760] up:boot 2024-04-30T08:45:54.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:54 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.71:6837/2357772129,v1:172.21.15.71:6839/2357772129] up:boot 2024-04-30T08:45:54.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:54 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.32:6836/2214894175,v1:172.21.15.32:6837/2214894175] up:boot 2024-04-30T08:45:54.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:54 smithi071 ceph-mon[33385]: mds.? [v2:172.21.15.32:6838/1222684735,v1:172.21.15.32:6839/1222684735] up:boot 2024-04-30T08:45:54.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-30T08:45:54.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:54 smithi071 ceph-mon[33385]: fsmap 12 up:standby 2024-04-30T08:45:54.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-30T08:45:54.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-30T08:45:54.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-30T08:45:54.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-30T08:45:54.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-30T08:45:54.755 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-30T08:45:54.756 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-30T08:45:54.756 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:54 smithi071 ceph-mon[33385]: from='mgr.14152 172.21.15.32:0/1898469735' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-30T08:45:54.756 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:54 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/2462191442' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-30T08:45:54.756 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:54 smithi071 ceph-mon[33385]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-30T08:45:54.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.861+0000 7fc2b8200640 1 -- 172.21.15.32:0/1678229626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2b00769f0 msgr2=0x7fc2b010b620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:54.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.861+0000 7fc2b8200640 1 --2- 172.21.15.32:0/1678229626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2b00769f0 0x7fc2b010b620 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc29c009900 tx=0x7fc29c02a3f0 comp rx=0 tx=0).stop 2024-04-30T08:45:54.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.861+0000 7fc2b8200640 1 -- 172.21.15.32:0/1678229626 shutdown_connections 2024-04-30T08:45:54.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.861+0000 7fc2b8200640 1 --2- 172.21.15.32:0/1678229626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2b010bb60 0x7fc2b010df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:54.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.861+0000 7fc2b8200640 1 --2- 172.21.15.32:0/1678229626 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2b00769f0 0x7fc2b010b620 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:54.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.861+0000 7fc2b8200640 1 --2- 172.21.15.32:0/1678229626 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2b00760a0 0x7fc2b00764b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:54.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.861+0000 7fc2b8200640 1 -- 172.21.15.32:0/1678229626 >> 172.21.15.32:0/1678229626 conn(0x7fc2b00fed60 msgr2=0x7fc2b0101210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:45:54.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.861+0000 7fc2b8200640 1 -- 172.21.15.32:0/1678229626 shutdown_connections 2024-04-30T08:45:54.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.861+0000 7fc2b8200640 1 -- 172.21.15.32:0/1678229626 wait complete. 2024-04-30T08:45:54.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.862+0000 7fc2b8200640 1 Processor -- start 2024-04-30T08:45:54.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.862+0000 7fc2b8200640 1 -- start start 2024-04-30T08:45:54.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.863+0000 7fc2b8200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2b00760a0 0x7fc2b019c4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:54.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.863+0000 7fc2b8200640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2b00769f0 0x7fc2b019ca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:54.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.863+0000 7fc2b8200640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2b010bb60 0x7fc2b01a39a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:54.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.863+0000 7fc2b8200640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b0111870 con 0x7fc2b00769f0 2024-04-30T08:45:54.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.863+0000 7fc2b8200640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b01116f0 con 0x7fc2b010bb60 2024-04-30T08:45:54.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.863+0000 7fc2b8200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fc2b01119f0 con 0x7fc2b00760a0 2024-04-30T08:45:54.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.863+0000 7fc2b4e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2b00769f0 0x7fc2b019ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:54.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.863+0000 7fc2b4e00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2b00769f0 0x7fc2b019ca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.32:3300/0 says I am v2:172.21.15.32:58546/0 (socket says 172.21.15.32:58546) 2024-04-30T08:45:54.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.863+0000 7fc2b4e00640 1 -- 172.21.15.32:0/4207611030 learned_addr learned my addr 172.21.15.32:0/4207611030 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:45:54.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.863+0000 7fc2b5800640 1 --2- 172.21.15.32:0/4207611030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2b00760a0 0x7fc2b019c4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:54.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.863+0000 7fc2b6200640 1 --2- 172.21.15.32:0/4207611030 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2b010bb60 0x7fc2b01a39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:54.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.864+0000 7fc2b4e00640 1 -- 172.21.15.32:0/4207611030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2b00760a0 msgr2=0x7fc2b019c4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:54.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.864+0000 7fc2b4e00640 1 --2- 172.21.15.32:0/4207611030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2b00760a0 0x7fc2b019c4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:54.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.864+0000 7fc2b4e00640 1 -- 172.21.15.32:0/4207611030 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2b010bb60 msgr2=0x7fc2b01a39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:54.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.864+0000 7fc2b4e00640 1 --2- 172.21.15.32:0/4207611030 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2b010bb60 0x7fc2b01a39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:54.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.864+0000 7fc2b4e00640 1 -- 172.21.15.32:0/4207611030 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc2b01a4010 con 0x7fc2b00769f0 2024-04-30T08:45:54.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.864+0000 7fc2b6200640 1 --2- 172.21.15.32:0/4207611030 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2b010bb60 0x7fc2b01a39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:45:54.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.864+0000 7fc2b5800640 1 --2- 172.21.15.32:0/4207611030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2b00760a0 0x7fc2b019c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-30T08:45:54.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.864+0000 7fc2b4e00640 1 --2- 172.21.15.32:0/4207611030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2b00769f0 0x7fc2b019ca10 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc29c006010 tx=0x7fc29c031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:45:54.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.864+0000 7fc2ae000640 1 -- 172.21.15.32:0/4207611030 <== mon.0 v2:172.21.15.32:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc29c03f030 con 0x7fc2b00769f0 2024-04-30T08:45:54.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.865+0000 7fc2ae000640 1 -- 172.21.15.32:0/4207611030 <== mon.0 v2:172.21.15.32:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fc29c003aa0 con 0x7fc2b00769f0 2024-04-30T08:45:54.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.865+0000 7fc2b8200640 1 -- 172.21.15.32:0/4207611030 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc2b01a42a0 con 0x7fc2b00769f0 2024-04-30T08:45:54.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.865+0000 7fc2ae000640 1 -- 172.21.15.32:0/4207611030 <== mon.0 v2:172.21.15.32:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc29c035070 con 0x7fc2b00769f0 2024-04-30T08:45:54.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.865+0000 7fc2b8200640 1 -- 172.21.15.32:0/4207611030 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc2b01a47e0 con 0x7fc2b00769f0 2024-04-30T08:45:54.869 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.866+0000 7fc2b8200640 1 -- 172.21.15.32:0/4207611030 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc2b010ad20 con 0x7fc2b00769f0 2024-04-30T08:45:54.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.868+0000 7fc2ae000640 1 -- 172.21.15.32:0/4207611030 <== mon.0 v2:172.21.15.32:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7fc29c042030 con 0x7fc2b00769f0 2024-04-30T08:45:54.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.870+0000 7fc2ae000640 1 --2- 172.21.15.32:0/4207611030 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc2980afb70 0x7fc2980b2080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:54.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.870+0000 7fc2ae000640 1 -- 172.21.15.32:0/4207611030 <== mon.0 v2:172.21.15.32:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 9150+0+0 (secure 0 0 0) 0x7fc29c0ffb70 con 0x7fc2b00769f0 2024-04-30T08:45:54.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.870+0000 7fc2b5800640 1 --2- 172.21.15.32:0/4207611030 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc2980afb70 0x7fc2980b2080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:54.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.871+0000 7fc2b5800640 1 --2- 172.21.15.32:0/4207611030 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc2980afb70 0x7fc2980b2080 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fc2a40095f0 tx=0x7fc2a4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:45:54.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:54.873+0000 7fc2ae000640 1 -- 172.21.15.32:0/4207611030 <== mon.0 v2:172.21.15.32:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc29c0c7d10 con 0x7fc2b00769f0 2024-04-30T08:45:55.081 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.080+0000 7fc2b8200640 1 -- 172.21.15.32:0/4207611030 --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fc2b002dbd0 con 0x7fc2b00769f0 2024-04-30T08:45:55.548 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.546+0000 7fc2ae000640 1 -- 172.21.15.32:0/4207611030 <== mon.0 v2:172.21.15.32:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v88) ==== 169+0+0 (secure 0 0 0) 0x7fc29c0d03e0 con 0x7fc2b00769f0 2024-04-30T08:45:55.548 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data' removed 2024-04-30T08:45:55.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.551+0000 7fc2b8200640 1 -- 172.21.15.32:0/4207611030 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc2980afb70 msgr2=0x7fc2980b2080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:55.552 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.551+0000 7fc2b8200640 1 --2- 172.21.15.32:0/4207611030 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc2980afb70 0x7fc2980b2080 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fc2a40095f0 tx=0x7fc2a4005f90 comp rx=0 tx=0).stop 2024-04-30T08:45:55.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.551+0000 7fc2b8200640 1 -- 172.21.15.32:0/4207611030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2b00769f0 msgr2=0x7fc2b019ca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:55.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.551+0000 7fc2b8200640 1 --2- 172.21.15.32:0/4207611030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2b00769f0 0x7fc2b019ca10 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc29c006010 tx=0x7fc29c031a50 comp rx=0 tx=0).stop 2024-04-30T08:45:55.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.551+0000 7fc2b8200640 1 -- 172.21.15.32:0/4207611030 shutdown_connections 2024-04-30T08:45:55.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.551+0000 7fc2b8200640 1 --2- 172.21.15.32:0/4207611030 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fc2980afb70 0x7fc2980b2080 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:55.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.551+0000 7fc2b8200640 1 --2- 172.21.15.32:0/4207611030 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fc2b010bb60 0x7fc2b01a39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:55.553 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.551+0000 7fc2b8200640 1 --2- 172.21.15.32:0/4207611030 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fc2b00769f0 0x7fc2b019ca10 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:55.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.551+0000 7fc2b8200640 1 --2- 172.21.15.32:0/4207611030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2b00760a0 0x7fc2b019c4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:55.554 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.551+0000 7fc2b8200640 1 -- 172.21.15.32:0/4207611030 >> 172.21.15.32:0/4207611030 conn(0x7fc2b00fed60 msgr2=0x7fc2b00ffba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:45:55.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.554+0000 7fc2b8200640 1 -- 172.21.15.32:0/4207611030 shutdown_connections 2024-04-30T08:45:55.555 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.554+0000 7fc2b8200640 1 -- 172.21.15.32:0/4207611030 wait complete. 2024-04-30T08:45:55.590 DEBUG:teuthology.orchestra.run.smithi032:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-04-30T08:45:55.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:55 smithi189 ceph-mon[33038]: pgmap v748: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-04-30T08:45:55.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:55 smithi189 ceph-mon[33038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-30T08:45:55.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:55 smithi189 ceph-mon[33038]: osdmap e87: 12 total, 12 up, 12 in 2024-04-30T08:45:55.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:55 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/4207611030' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-30T08:45:55.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.861+0000 7fb19be00640 1 -- 172.21.15.32:0/3315796959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb19c075010 msgr2=0x7fb19c075420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:55.862 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.861+0000 7fb19be00640 1 --2- 172.21.15.32:0/3315796959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb19c075010 0x7fb19c075420 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb18c009900 tx=0x7fb18c02a3f0 comp rx=0 tx=0).stop 2024-04-30T08:45:55.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.861+0000 7fb19be00640 1 -- 172.21.15.32:0/3315796959 shutdown_connections 2024-04-30T08:45:55.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.861+0000 7fb19be00640 1 --2- 172.21.15.32:0/3315796959 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb19c10b920 0x7fb19c10dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:55.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.861+0000 7fb19be00640 1 --2- 172.21.15.32:0/3315796959 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb19c075960 0x7fb19c10b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:55.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.861+0000 7fb19be00640 1 --2- 172.21.15.32:0/3315796959 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb19c075010 0x7fb19c075420 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:55.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.861+0000 7fb19be00640 1 -- 172.21.15.32:0/3315796959 >> 172.21.15.32:0/3315796959 conn(0x7fb19c0feba0 msgr2=0x7fb19c101050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:45:55.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.861+0000 7fb19be00640 1 -- 172.21.15.32:0/3315796959 shutdown_connections 2024-04-30T08:45:55.863 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.861+0000 7fb19be00640 1 -- 172.21.15.32:0/3315796959 wait complete. 2024-04-30T08:45:55.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.862+0000 7fb19be00640 1 Processor -- start 2024-04-30T08:45:55.864 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.863+0000 7fb19be00640 1 -- start start 2024-04-30T08:45:55.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.863+0000 7fb19be00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb19c075010 0x7fb19c19c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:55.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.863+0000 7fb19be00640 1 --2- >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb19c075960 0x7fb19c19c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:55.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.863+0000 7fb19be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb19c10b920 0x7fb19c1a37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:55.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.863+0000 7fb19be00640 1 -- --> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] -- mon_getmap magic: 0 -- 0x7fb19c111730 con 0x7fb19c075960 2024-04-30T08:45:55.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.863+0000 7fb19be00640 1 -- --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_getmap magic: 0 -- 0x7fb19c1115b0 con 0x7fb19c075010 2024-04-30T08:45:55.865 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.863+0000 7fb19be00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 -- 0x7fb19c1118b0 con 0x7fb19c10b920 2024-04-30T08:45:55.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.863+0000 7fb19aa00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb19c075010 0x7fb19c19c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:55.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.863+0000 7fb19aa00640 1 --2- >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb19c075010 0x7fb19c19c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.71:3300/0 says I am v2:172.21.15.32:34806/0 (socket says 172.21.15.32:34806) 2024-04-30T08:45:55.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.863+0000 7fb19aa00640 1 -- 172.21.15.32:0/535301379 learned_addr learned my addr 172.21.15.32:0/535301379 (peer_addr_for_me v2:172.21.15.32:0/0) 2024-04-30T08:45:55.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.864+0000 7fb192000640 1 --2- 172.21.15.32:0/535301379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb19c075960 0x7fb19c19c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:55.866 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.864+0000 7fb19b400640 1 --2- 172.21.15.32:0/535301379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb19c10b920 0x7fb19c1a37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:55.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.864+0000 7fb19aa00640 1 -- 172.21.15.32:0/535301379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb19c10b920 msgr2=0x7fb19c1a37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:55.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.864+0000 7fb19aa00640 1 --2- 172.21.15.32:0/535301379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb19c10b920 0x7fb19c1a37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:55.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.864+0000 7fb19aa00640 1 -- 172.21.15.32:0/535301379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb19c075960 msgr2=0x7fb19c19c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:55.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.864+0000 7fb19aa00640 1 --2- 172.21.15.32:0/535301379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb19c075960 0x7fb19c19c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:55.867 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.864+0000 7fb19aa00640 1 -- 172.21.15.32:0/535301379 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb19c1a3e10 con 0x7fb19c075010 2024-04-30T08:45:55.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.864+0000 7fb19aa00640 1 --2- 172.21.15.32:0/535301379 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb19c075010 0x7fb19c19c2d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fb18c031440 tx=0x7fb18c003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:45:55.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.865+0000 7fb193e00640 1 -- 172.21.15.32:0/535301379 <== mon.2 v2:172.21.15.71:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb18c03f070 con 0x7fb19c075010 2024-04-30T08:45:55.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.865+0000 7fb193e00640 1 -- 172.21.15.32:0/535301379 <== mon.2 v2:172.21.15.71:3300/0 2 ==== config(26 keys) ==== 1101+0+0 (secure 0 0 0) 0x7fb18c031c50 con 0x7fb19c075010 2024-04-30T08:45:55.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.865+0000 7fb193e00640 1 -- 172.21.15.32:0/535301379 <== mon.2 v2:172.21.15.71:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb18c035030 con 0x7fb19c075010 2024-04-30T08:45:55.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.865+0000 7fb19be00640 1 -- 172.21.15.32:0/535301379 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb19c1a40a0 con 0x7fb19c075010 2024-04-30T08:45:55.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.865+0000 7fb19be00640 1 -- 172.21.15.32:0/535301379 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb19c1a4540 con 0x7fb19c075010 2024-04-30T08:45:55.868 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.866+0000 7fb19be00640 1 -- 172.21.15.32:0/535301379 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb19c10a1f0 con 0x7fb19c075010 2024-04-30T08:45:55.870 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.869+0000 7fb193e00640 1 -- 172.21.15.32:0/535301379 <== mon.2 v2:172.21.15.71:3300/0 4 ==== mgrmap(e 20) ==== 147520+0+0 (secure 0 0 0) 0x7fb18c042070 con 0x7fb19c075010 2024-04-30T08:45:55.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.870+0000 7fb193e00640 1 --2- 172.21.15.32:0/535301379 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb1800afb60 0x7fb1800b2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-30T08:45:55.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.870+0000 7fb193e00640 1 -- 172.21.15.32:0/535301379 <== mon.2 v2:172.21.15.71:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8496+0+0 (secure 0 0 0) 0x7fb18c0ffbb0 con 0x7fb19c075010 2024-04-30T08:45:55.872 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.870+0000 7fb192000640 1 --2- 172.21.15.32:0/535301379 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb1800afb60 0x7fb1800b2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-30T08:45:55.873 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.871+0000 7fb192000640 1 --2- 172.21.15.32:0/535301379 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb1800afb60 0x7fb1800b2070 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fb19c19d370 tx=0x7fb184009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-30T08:45:55.874 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:55.872+0000 7fb193e00640 1 -- 172.21.15.32:0/535301379 <== mon.2 v2:172.21.15.71:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb18c0c8000 con 0x7fb19c075010 2024-04-30T08:45:55.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:55 smithi032 ceph-mon[28331]: pgmap v748: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-04-30T08:45:55.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:55 smithi032 ceph-mon[28331]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-30T08:45:55.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:55 smithi032 ceph-mon[28331]: osdmap e87: 12 total, 12 up, 12 in 2024-04-30T08:45:55.941 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:55 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/4207611030' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-30T08:45:56.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:55 smithi071 ceph-mon[33385]: pgmap v748: 129 pgs: 129 active+clean; 18 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.1 KiB/s wr, 11 op/s 2024-04-30T08:45:56.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:55 smithi071 ceph-mon[33385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-30T08:45:56.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:55 smithi071 ceph-mon[33385]: osdmap e87: 12 total, 12 up, 12 in 2024-04-30T08:45:56.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:55 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/4207611030' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-30T08:45:56.083 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:56.081+0000 7fb19be00640 1 -- 172.21.15.32:0/535301379 --> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7fb19c02dbd0 con 0x7fb19c075010 2024-04-30T08:45:56.580 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:56.578+0000 7fb193e00640 1 -- 172.21.15.32:0/535301379 <== mon.2 v2:172.21.15.71:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v89) ==== 178+0+0 (secure 0 0 0) 0x7fb18c0cc300 con 0x7fb19c075010 2024-04-30T08:45:56.582 INFO:teuthology.orchestra.run.smithi032.stderr:pool 'cephfs_data_ec' removed 2024-04-30T08:45:56.596 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:56.595+0000 7fb19be00640 1 -- 172.21.15.32:0/535301379 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb1800afb60 msgr2=0x7fb1800b2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:56.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:56.595+0000 7fb19be00640 1 --2- 172.21.15.32:0/535301379 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb1800afb60 0x7fb1800b2070 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fb19c19d370 tx=0x7fb184009500 comp rx=0 tx=0).stop 2024-04-30T08:45:56.597 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:56.595+0000 7fb19be00640 1 -- 172.21.15.32:0/535301379 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb19c075010 msgr2=0x7fb19c19c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-30T08:45:56.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:56.595+0000 7fb19be00640 1 --2- 172.21.15.32:0/535301379 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb19c075010 0x7fb19c19c2d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fb18c031440 tx=0x7fb18c003c10 comp rx=0 tx=0).stop 2024-04-30T08:45:56.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:56.595+0000 7fb19be00640 1 -- 172.21.15.32:0/535301379 shutdown_connections 2024-04-30T08:45:56.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:56.595+0000 7fb19be00640 1 --2- 172.21.15.32:0/535301379 >> [v2:172.21.15.32:6800/2930853529,v1:172.21.15.32:6801/2930853529] conn(0x7fb1800afb60 0x7fb1800b2070 secure :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fb19c19d370 tx=0x7fb184009500 comp rx=0 tx=0).stop 2024-04-30T08:45:56.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:56.595+0000 7fb19be00640 1 --2- 172.21.15.32:0/535301379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb19c10b920 0x7fb19c1a37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:56.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:56.595+0000 7fb19be00640 1 --2- 172.21.15.32:0/535301379 >> [v2:172.21.15.32:3300/0,v1:172.21.15.32:6789/0] conn(0x7fb19c075960 0x7fb19c19c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-30T08:45:56.598 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:56.595+0000 7fb19be00640 1 --2- 172.21.15.32:0/535301379 >> [v2:172.21.15.71:3300/0,v1:172.21.15.71:6789/0] conn(0x7fb19c075010 0x7fb19c19c2d0 secure :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fb18c031440 tx=0x7fb18c003c10 comp rx=0 tx=0).stop 2024-04-30T08:45:56.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:56.595+0000 7fb19be00640 1 -- 172.21.15.32:0/535301379 >> 172.21.15.32:0/535301379 conn(0x7fb19c0feba0 msgr2=0x7fb19c1073c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-30T08:45:56.599 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:56.596+0000 7fb19be00640 1 -- 172.21.15.32:0/535301379 shutdown_connections 2024-04-30T08:45:56.601 INFO:teuthology.orchestra.run.smithi032.stderr:2024-04-30T08:45:56.600+0000 7fb19be00640 1 -- 172.21.15.32:0/535301379 wait complete. 2024-04-30T08:45:56.653 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-30T08:45:56.786 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-30T08:45:56.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:56 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/4207611030' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-30T08:45:56.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:56 smithi071 ceph-mon[33385]: osdmap e88: 12 total, 12 up, 12 in 2024-04-30T08:45:56.810 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:56 smithi071 ceph-mon[33385]: from='client.? 172.21.15.32:0/535301379' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-30T08:45:56.811 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:56 smithi071 ceph-mon[33385]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-30T08:45:56.812 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-30T08:45:56.834 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:56 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/4207611030' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-30T08:45:56.834 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:56 smithi189 ceph-mon[33038]: osdmap e88: 12 total, 12 up, 12 in 2024-04-30T08:45:56.834 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:56 smithi189 ceph-mon[33038]: from='client.? 172.21.15.32:0/535301379' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-30T08:45:56.834 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:56 smithi189 ceph-mon[33038]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-30T08:45:56.839 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-30T08:45:56.839 DEBUG:teuthology.orchestra.run.smithi032:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-30T08:45:56.915 DEBUG:teuthology.orchestra.run.smithi071:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-30T08:45:56.929 DEBUG:teuthology.orchestra.run.smithi189:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-30T08:45:56.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:56 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/4207611030' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-30T08:45:56.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:56 smithi032 ceph-mon[28331]: osdmap e88: 12 total, 12 up, 12 in 2024-04-30T08:45:56.942 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:56 smithi032 ceph-mon[28331]: from='client.? 172.21.15.32:0/535301379' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-30T08:45:56.943 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:56 smithi032 ceph-mon[28331]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-30T08:45:56.950 INFO:tasks.cephadm:Stopping all daemons... 2024-04-30T08:45:56.950 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-30T08:45:56.950 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mon.a 2024-04-30T08:45:57.442 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:57 smithi032 systemd[1]: Stopping Ceph mon.a for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:45:57.863 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:57 smithi032 ceph-mon[28331]: pgmap v751: 33 pgs: 33 active+clean; 577 KiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:45:57.863 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:57 smithi032 ceph-mon[28331]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-04-30T08:45:57.863 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:57 smithi032 ceph-mon[28331]: osdmap e89: 12 total, 12 up, 12 in 2024-04-30T08:45:58.003 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:57 smithi071 ceph-mon[33385]: pgmap v751: 33 pgs: 33 active+clean; 577 KiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:45:58.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:57 smithi071 ceph-mon[33385]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-04-30T08:45:58.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:57 smithi071 ceph-mon[33385]: osdmap e89: 12 total, 12 up, 12 in 2024-04-30T08:45:58.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:57 smithi189 ceph-mon[33038]: pgmap v751: 33 pgs: 33 active+clean; 577 KiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-30T08:45:58.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:57 smithi189 ceph-mon[33038]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]': finished 2024-04-30T08:45:58.050 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:45:57 smithi189 ceph-mon[33038]: osdmap e89: 12 total, 12 up, 12 in 2024-04-30T08:45:58.143 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:57 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mon-a[28327]: 2024-04-30T08:45:57.861+0000 7f3446000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-30T08:45:58.144 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:57 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mon-a[28327]: 2024-04-30T08:45:57.861+0000 7f3446000700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-30T08:45:58.429 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:58 smithi032 podman[94553]: 2024-04-30 08:45:58.142964251 +0000 UTC m=+0.514138952 container died 9a3de84cb1e909976d461d7e88cd0ad533211e8248979910e513c4fd2f0e933f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True) 2024-04-30T08:45:58.691 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:58 smithi032 podman[94553]: 2024-04-30 08:45:58.575506147 +0000 UTC m=+0.946680832 container cleanup 9a3de84cb1e909976d461d7e88cd0ad533211e8248979910e513c4fd2f0e933f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-30T08:45:58.692 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:58 smithi032 bash[94553]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mon-a 2024-04-30T08:45:59.171 INFO:journalctl@ceph.mon.a.smithi032.stdout:Apr 30 08:45:58 smithi032 podman[94564]: 2024-04-30 08:45:58.827144675 +0000 UTC m=+0.666940730 container remove 9a3de84cb1e909976d461d7e88cd0ad533211e8248979910e513c4fd2f0e933f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True) 2024-04-30T08:45:59.202 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mon.a.service' 2024-04-30T08:45:59.327 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-30T08:45:59.327 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-30T08:45:59.327 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-30T08:45:59.327 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mon.b 2024-04-30T08:45:59.708 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:59 smithi071 systemd[1]: Stopping Ceph mon.b for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:46:00.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:59 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mon-b[33381]: 2024-04-30T08:45:59.705+0000 7fac38e26700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-30T08:46:00.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:59 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mon-b[33381]: 2024-04-30T08:45:59.705+0000 7fac38e26700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-30T08:46:00.004 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:45:59 smithi071 podman[68670]: 2024-04-30 08:45:59.819681103 +0000 UTC m=+0.233219736 container died 7615c446928d9168f2c655b7af67defffaa49ddbc2afe0a94052bb819dde3244 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mon-b, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-30T08:46:00.485 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:46:00 smithi071 podman[68670]: 2024-04-30 08:46:00.224504876 +0000 UTC m=+0.638043515 container cleanup 7615c446928d9168f2c655b7af67defffaa49ddbc2afe0a94052bb819dde3244 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-04-30T08:46:00.485 INFO:journalctl@ceph.mon.b.smithi071.stdout:Apr 30 08:46:00 smithi071 bash[68670]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mon-b 2024-04-30T08:46:00.600 DEBUG:teuthology.orchestra.run.smithi071:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mon.b.service' 2024-04-30T08:46:00.658 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-30T08:46:00.658 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-30T08:46:00.658 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-30T08:46:00.658 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mon.c 2024-04-30T08:46:01.006 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:46:00 smithi189 systemd[1]: Stopping Ceph mon.c for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:46:01.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:46:01 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mon-c[33034]: 2024-04-30T08:46:01.004+0000 7fa352cd6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-30T08:46:01.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:46:01 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mon-c[33034]: 2024-04-30T08:46:01.004+0000 7fa352cd6700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-04-30T08:46:01.301 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:46:01 smithi189 podman[68401]: 2024-04-30 08:46:01.178379062 +0000 UTC m=+0.291759153 container died bf88bed5fe55d41eb96b923f63c7f14003672d82eafa509460d7a8f06986f14d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mon-c, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-30T08:46:01.800 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:46:01 smithi189 podman[68401]: 2024-04-30 08:46:01.597253834 +0000 UTC m=+0.710633922 container cleanup bf88bed5fe55d41eb96b923f63c7f14003672d82eafa509460d7a8f06986f14d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mon-c, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d) 2024-04-30T08:46:01.801 INFO:journalctl@ceph.mon.c.smithi189.stdout:Apr 30 08:46:01 smithi189 bash[68401]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mon-c 2024-04-30T08:46:02.048 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mon.c.service' 2024-04-30T08:46:02.093 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-30T08:46:02.094 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-30T08:46:02.094 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-30T08:46:02.094 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mgr.x 2024-04-30T08:46:02.442 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:46:02 smithi032 systemd[1]: Stopping Ceph mgr.x for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:46:03.430 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:46:03 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x[28571]: 2024-04-30T08:46:03.012+0000 7fbbb8a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-30T08:46:03.430 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:46:03 smithi032 podman[94663]: 2024-04-30 08:46:03.116280229 +0000 UTC m=+0.561949185 container died 55eff9d8db7ceca692cabe245b81c28a16acf7f28fc49dde0d53b36da014ee1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-30T08:46:04.192 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:46:03 smithi032 podman[94663]: 2024-04-30 08:46:03.765450513 +0000 UTC m=+1.211119493 container cleanup 55eff9d8db7ceca692cabe245b81c28a16acf7f28fc49dde0d53b36da014ee1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430) 2024-04-30T08:46:04.192 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:46:03 smithi032 bash[94663]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x 2024-04-30T08:46:04.192 INFO:journalctl@ceph.mgr.x.smithi032.stdout:Apr 30 08:46:03 smithi032 podman[94680]: 2024-04-30 08:46:03.983376007 +0000 UTC m=+0.853146587 container remove 55eff9d8db7ceca692cabe245b81c28a16acf7f28fc49dde0d53b36da014ee1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:8feaf7d9864e980041b042fd5d964ac18adfd20f, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-x, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True) 2024-04-30T08:46:04.525 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-30T08:46:04.641 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-30T08:46:04.641 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-30T08:46:04.642 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-30T08:46:04.642 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mgr.y 2024-04-30T08:46:04.747 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:46:04 smithi071 systemd[1]: Stopping Ceph mgr.y for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:46:05.253 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:46:05 smithi071 podman[68797]: 2024-04-30 08:46:05.000452554 +0000 UTC m=+0.167956768 container died b69f6e6706d923cc833b8d2f70598cbae2555ca76f34772da9070ec2a372e30b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-y, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-30T08:46:05.674 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:46:05 smithi071 podman[68797]: 2024-04-30 08:46:05.405494017 +0000 UTC m=+0.572998215 container cleanup b69f6e6706d923cc833b8d2f70598cbae2555ca76f34772da9070ec2a372e30b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-30T08:46:05.674 INFO:journalctl@ceph.mgr.y.smithi071.stdout:Apr 30 08:46:05 smithi071 bash[68797]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-y 2024-04-30T08:46:05.816 DEBUG:teuthology.orchestra.run.smithi071:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-30T08:46:05.852 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-30T08:46:05.852 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-30T08:46:05.853 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-30T08:46:05.853 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mgr.z 2024-04-30T08:46:06.207 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:46:05 smithi189 systemd[1]: Stopping Ceph mgr.z for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:46:06.495 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:46:06 smithi189 podman[68527]: 2024-04-30 08:46:06.206646727 +0000 UTC m=+0.154550641 container died b90455e4a74856c308ad9119d2b65297cff0a79e8a07fe42a68add492ee481c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.build-date=20240430, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-30T08:46:06.800 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:46:06 smithi189 podman[68527]: 2024-04-30 08:46:06.594382497 +0000 UTC m=+0.542286406 container cleanup b90455e4a74856c308ad9119d2b65297cff0a79e8a07fe42a68add492ee481c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-z, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-30T08:46:06.800 INFO:journalctl@ceph.mgr.z.smithi189.stdout:Apr 30 08:46:06 smithi189 bash[68527]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mgr-z 2024-04-30T08:46:07.016 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-30T08:46:07.048 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-30T08:46:07.049 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-30T08:46:07.049 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-30T08:46:07.049 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.0 2024-04-30T08:46:07.441 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 30 08:46:07 smithi032 systemd[1]: Stopping Ceph osd.0 for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:46:07.942 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 30 08:46:07 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-0[43279]: 2024-04-30T08:46:07.599+0000 7f7f34600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-30T08:46:07.942 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 30 08:46:07 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-0[43279]: 2024-04-30T08:46:07.600+0000 7f7f34600700 -1 osd.0 89 *** Got signal Terminated *** 2024-04-30T08:46:07.942 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 30 08:46:07 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-0[43279]: 2024-04-30T08:46:07.600+0000 7f7f34600700 -1 osd.0 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-30T08:46:13.786 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 30 08:46:13 smithi032 podman[94789]: 2024-04-30 08:46:13.484062584 +0000 UTC m=+5.994762312 container died db5407ef6167e5e03abef844c6fd6e858ebd27317c2b5a5896ad1260197bc5e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d) 2024-04-30T08:46:14.165 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 30 08:46:13 smithi032 podman[94789]: 2024-04-30 08:46:13.912369426 +0000 UTC m=+6.423069148 container cleanup db5407ef6167e5e03abef844c6fd6e858ebd27317c2b5a5896ad1260197bc5e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d) 2024-04-30T08:46:14.165 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 30 08:46:13 smithi032 bash[94789]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-0 2024-04-30T08:46:14.441 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 30 08:46:14 smithi032 podman[94810]: 2024-04-30 08:46:14.163876854 +0000 UTC m=+0.666967154 container remove db5407ef6167e5e03abef844c6fd6e858ebd27317c2b5a5896ad1260197bc5e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, maintainer=Guillaume Abrioux ) 2024-04-30T08:46:15.431 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 30 08:46:15 smithi032 podman[94898]: 2024-04-30 08:46:14.978997684 +0000 UTC m=+0.036758224 image pull 16056ace0edc7f73516cf28297f1db04dda872801fb5b17b9157b0d1ccf38f9b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de 2024-04-30T08:46:15.432 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 30 08:46:15 smithi032 podman[94898]: 2024-04-30 08:46:15.098427515 +0000 UTC m=+0.156188129 container create 85e5e77a42225743b1cd15d5559cbb513f9f1b4e2344229c9fe71d22811b1f10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240430) 2024-04-30T08:46:15.432 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 30 08:46:15 smithi032 podman[94898]: 2024-04-30 08:46:15.429872737 +0000 UTC m=+0.487633320 container init 85e5e77a42225743b1cd15d5559cbb513f9f1b4e2344229c9fe71d22811b1f10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, ceph=True, org.label-schema.schema-version=1.0) 2024-04-30T08:46:15.691 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 30 08:46:15 smithi032 podman[94898]: 2024-04-30 08:46:15.441834308 +0000 UTC m=+0.499594856 container start 85e5e77a42225743b1cd15d5559cbb513f9f1b4e2344229c9fe71d22811b1f10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-30T08:46:15.692 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 30 08:46:15 smithi032 podman[94898]: 2024-04-30 08:46:15.542873295 +0000 UTC m=+0.600633841 container attach 85e5e77a42225743b1cd15d5559cbb513f9f1b4e2344229c9fe71d22811b1f10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, maintainer=Guillaume Abrioux ) 2024-04-30T08:46:16.692 INFO:journalctl@ceph.osd.0.smithi032.stdout:Apr 30 08:46:16 smithi032 podman[94898]: 2024-04-30 08:46:16.26855499 +0000 UTC m=+1.326315598 container died 85e5e77a42225743b1cd15d5559cbb513f9f1b4e2344229c9fe71d22811b1f10 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-30T08:46:17.145 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.0.service' 2024-04-30T08:46:17.263 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-30T08:46:17.263 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-30T08:46:17.263 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-30T08:46:17.263 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.1 2024-04-30T08:46:17.586 INFO:journalctl@ceph.osd.1.smithi071.stdout:Apr 30 08:46:17 smithi071 systemd[1]: Stopping Ceph osd.1 for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:46:18.003 INFO:journalctl@ceph.osd.1.smithi071.stdout:Apr 30 08:46:17 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-1[37970]: 2024-04-30T08:46:17.584+0000 7f8f00eac700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-30T08:46:18.003 INFO:journalctl@ceph.osd.1.smithi071.stdout:Apr 30 08:46:17 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-1[37970]: 2024-04-30T08:46:17.584+0000 7f8f00eac700 -1 osd.1 89 *** Got signal Terminated *** 2024-04-30T08:46:18.004 INFO:journalctl@ceph.osd.1.smithi071.stdout:Apr 30 08:46:17 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-1[37970]: 2024-04-30T08:46:17.584+0000 7f8f00eac700 -1 osd.1 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-30T08:46:23.470 INFO:journalctl@ceph.osd.1.smithi071.stdout:Apr 30 08:46:23 smithi071 podman[68931]: 2024-04-30 08:46:23.135637686 +0000 UTC m=+5.670910610 container died ddbbffda8919d432805fe7d578818f020caafc5c9f9418b5382f14f61aab91e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-1, org.label-schema.build-date=20240430, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True) 2024-04-30T08:46:23.754 INFO:journalctl@ceph.osd.1.smithi071.stdout:Apr 30 08:46:23 smithi071 podman[68931]: 2024-04-30 08:46:23.568333429 +0000 UTC m=+6.103606357 container cleanup ddbbffda8919d432805fe7d578818f020caafc5c9f9418b5382f14f61aab91e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-1, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-30T08:46:23.754 INFO:journalctl@ceph.osd.1.smithi071.stdout:Apr 30 08:46:23 smithi071 bash[68931]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-1 2024-04-30T08:46:24.228 INFO:journalctl@ceph.osd.1.smithi071.stdout:Apr 30 08:46:23 smithi071 podman[68951]: 2024-04-30 08:46:23.811822061 +0000 UTC m=+0.671613153 container remove ddbbffda8919d432805fe7d578818f020caafc5c9f9418b5382f14f61aab91e8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-1, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d) 2024-04-30T08:46:24.504 INFO:journalctl@ceph.osd.1.smithi071.stdout:Apr 30 08:46:24 smithi071 podman[69042]: 2024-04-30 08:46:24.13035175 +0000 UTC m=+0.017864948 image pull 16056ace0edc7f73516cf28297f1db04dda872801fb5b17b9157b0d1ccf38f9b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de 2024-04-30T08:46:24.504 INFO:journalctl@ceph.osd.1.smithi071.stdout:Apr 30 08:46:24 smithi071 podman[69042]: 2024-04-30 08:46:24.262779036 +0000 UTC m=+0.150292225 container create a99b9ad54e254da8cc803a850bc902d645c3c17ae4ed53944a7f93e419526ea0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-30T08:46:25.004 INFO:journalctl@ceph.osd.1.smithi071.stdout:Apr 30 08:46:24 smithi071 podman[69042]: 2024-04-30 08:46:24.541071802 +0000 UTC m=+0.428585000 container init a99b9ad54e254da8cc803a850bc902d645c3c17ae4ed53944a7f93e419526ea0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-30T08:46:25.004 INFO:journalctl@ceph.osd.1.smithi071.stdout:Apr 30 08:46:24 smithi071 podman[69042]: 2024-04-30 08:46:24.544537578 +0000 UTC m=+0.432050773 container start a99b9ad54e254da8cc803a850bc902d645c3c17ae4ed53944a7f93e419526ea0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-30T08:46:25.004 INFO:journalctl@ceph.osd.1.smithi071.stdout:Apr 30 08:46:24 smithi071 podman[69042]: 2024-04-30 08:46:24.683377987 +0000 UTC m=+0.570891190 container attach a99b9ad54e254da8cc803a850bc902d645c3c17ae4ed53944a7f93e419526ea0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-1-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-30T08:46:25.349 INFO:journalctl@ceph.osd.1.smithi071.stdout:Apr 30 08:46:25 smithi071 podman[69042]: 2024-04-30 08:46:25.079228541 +0000 UTC m=+0.966741732 container died a99b9ad54e254da8cc803a850bc902d645c3c17ae4ed53944a7f93e419526ea0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-30T08:46:25.755 DEBUG:teuthology.orchestra.run.smithi071:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.1.service' 2024-04-30T08:46:25.791 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-30T08:46:25.791 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-30T08:46:25.792 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-30T08:46:25.792 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.2 2024-04-30T08:46:26.107 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 30 08:46:25 smithi189 systemd[1]: Stopping Ceph osd.2 for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:46:26.382 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 30 08:46:26 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-2[38240]: 2024-04-30T08:46:26.105+0000 7f5b33e07700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-30T08:46:26.382 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 30 08:46:26 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-2[38240]: 2024-04-30T08:46:26.105+0000 7f5b33e07700 -1 osd.2 89 *** Got signal Terminated *** 2024-04-30T08:46:26.382 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 30 08:46:26 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-2[38240]: 2024-04-30T08:46:26.105+0000 7f5b33e07700 -1 osd.2 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-30T08:46:32.025 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 30 08:46:31 smithi189 podman[68652]: 2024-04-30 08:46:31.733463499 +0000 UTC m=+5.750476169 container died ba827a09709769c9aace10b9875946a452edb44079b5045ca0e0fbcddc131de6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-30T08:46:32.300 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 30 08:46:32 smithi189 podman[68652]: 2024-04-30 08:46:32.114861692 +0000 UTC m=+6.131874373 container cleanup ba827a09709769c9aace10b9875946a452edb44079b5045ca0e0fbcddc131de6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True) 2024-04-30T08:46:32.300 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 30 08:46:32 smithi189 bash[68652]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-2 2024-04-30T08:46:32.694 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 30 08:46:32 smithi189 podman[68681]: 2024-04-30 08:46:32.383288942 +0000 UTC m=+0.652033468 container remove ba827a09709769c9aace10b9875946a452edb44079b5045ca0e0fbcddc131de6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True) 2024-04-30T08:46:32.962 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 30 08:46:32 smithi189 podman[68772]: 2024-04-30 08:46:32.597639055 +0000 UTC m=+0.017911077 image pull 16056ace0edc7f73516cf28297f1db04dda872801fb5b17b9157b0d1ccf38f9b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de 2024-04-30T08:46:32.962 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 30 08:46:32 smithi189 podman[68772]: 2024-04-30 08:46:32.709510041 +0000 UTC m=+0.129782128 container create 41db1e25f0c26f4eaed6065d69fd6174ee11d8ae44d55f9749c9a1ac5fda4915 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-2-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-30T08:46:32.962 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 30 08:46:32 smithi189 podman[68772]: 2024-04-30 08:46:32.961951396 +0000 UTC m=+0.382223418 container init 41db1e25f0c26f4eaed6065d69fd6174ee11d8ae44d55f9749c9a1ac5fda4915 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.schema-version=1.0) 2024-04-30T08:46:33.300 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 30 08:46:32 smithi189 podman[68772]: 2024-04-30 08:46:32.965306745 +0000 UTC m=+0.385578774 container start 41db1e25f0c26f4eaed6065d69fd6174ee11d8ae44d55f9749c9a1ac5fda4915 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-30T08:46:33.301 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 30 08:46:33 smithi189 podman[68772]: 2024-04-30 08:46:33.078869599 +0000 UTC m=+0.499141635 container attach 41db1e25f0c26f4eaed6065d69fd6174ee11d8ae44d55f9749c9a1ac5fda4915 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, GIT_CLEAN=True) 2024-04-30T08:46:33.800 INFO:journalctl@ceph.osd.2.smithi189.stdout:Apr 30 08:46:33 smithi189 podman[68772]: 2024-04-30 08:46:33.49848829 +0000 UTC m=+0.918760315 container died 41db1e25f0c26f4eaed6065d69fd6174ee11d8ae44d55f9749c9a1ac5fda4915 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=) 2024-04-30T08:46:34.133 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.2.service' 2024-04-30T08:46:34.172 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-30T08:46:34.172 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-30T08:46:34.172 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-30T08:46:34.172 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.3 2024-04-30T08:46:34.562 INFO:journalctl@ceph.osd.3.smithi032.stdout:Apr 30 08:46:34 smithi032 systemd[1]: Stopping Ceph osd.3 for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:46:34.942 INFO:journalctl@ceph.osd.3.smithi032.stdout:Apr 30 08:46:34 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-3[50179]: 2024-04-30T08:46:34.807+0000 7fd995400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-30T08:46:34.942 INFO:journalctl@ceph.osd.3.smithi032.stdout:Apr 30 08:46:34 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-3[50179]: 2024-04-30T08:46:34.807+0000 7fd995400700 -1 osd.3 89 *** Got signal Terminated *** 2024-04-30T08:46:34.942 INFO:journalctl@ceph.osd.3.smithi032.stdout:Apr 30 08:46:34 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-3[50179]: 2024-04-30T08:46:34.807+0000 7fd995400700 -1 osd.3 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-30T08:46:41.192 INFO:journalctl@ceph.osd.3.smithi032.stdout:Apr 30 08:46:40 smithi032 podman[95003]: 2024-04-30 08:46:40.868442535 +0000 UTC m=+6.223411125 container stop dd27e0c3eec116814809e20ed8aa4bec55d2e5a291a2218a10412eeff5eab3de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-3, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-30T08:46:41.192 INFO:journalctl@ceph.osd.3.smithi032.stdout:Apr 30 08:46:40 smithi032 podman[95003]: 2024-04-30 08:46:40.870273002 +0000 UTC m=+6.225241606 container died dd27e0c3eec116814809e20ed8aa4bec55d2e5a291a2218a10412eeff5eab3de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, CEPH_POINT_RELEASE=) 2024-04-30T08:46:41.691 INFO:journalctl@ceph.osd.3.smithi032.stdout:Apr 30 08:46:41 smithi032 podman[95003]: 2024-04-30 08:46:41.350487774 +0000 UTC m=+6.705456364 container cleanup dd27e0c3eec116814809e20ed8aa4bec55d2e5a291a2218a10412eeff5eab3de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-3, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-30T08:46:41.692 INFO:journalctl@ceph.osd.3.smithi032.stdout:Apr 30 08:46:41 smithi032 bash[95003]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-3 2024-04-30T08:46:42.191 INFO:journalctl@ceph.osd.3.smithi032.stdout:Apr 30 08:46:41 smithi032 podman[95034]: 2024-04-30 08:46:41.700706198 +0000 UTC m=+0.810637356 container remove dd27e0c3eec116814809e20ed8aa4bec55d2e5a291a2218a10412eeff5eab3de (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-3, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240430, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-30T08:46:42.685 INFO:journalctl@ceph.osd.3.smithi032.stdout:Apr 30 08:46:42 smithi032 podman[95120]: 2024-04-30 08:46:42.238849594 +0000 UTC m=+0.036424748 image pull 16056ace0edc7f73516cf28297f1db04dda872801fb5b17b9157b0d1ccf38f9b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de 2024-04-30T08:46:42.685 INFO:journalctl@ceph.osd.3.smithi032.stdout:Apr 30 08:46:42 smithi032 podman[95120]: 2024-04-30 08:46:42.378690272 +0000 UTC m=+0.176265484 container create 6ac4d04c3113d40768245d1bdcc67fb72b271866e6f1784a6d896d3ee7f4fc8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d) 2024-04-30T08:46:42.942 INFO:journalctl@ceph.osd.3.smithi032.stdout:Apr 30 08:46:42 smithi032 podman[95120]: 2024-04-30 08:46:42.6834429 +0000 UTC m=+0.481018125 container init 6ac4d04c3113d40768245d1bdcc67fb72b271866e6f1784a6d896d3ee7f4fc8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-3-deactivate, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-30T08:46:42.942 INFO:journalctl@ceph.osd.3.smithi032.stdout:Apr 30 08:46:42 smithi032 podman[95120]: 2024-04-30 08:46:42.696286235 +0000 UTC m=+0.493861423 container start 6ac4d04c3113d40768245d1bdcc67fb72b271866e6f1784a6d896d3ee7f4fc8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240430, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-30T08:46:42.942 INFO:journalctl@ceph.osd.3.smithi032.stdout:Apr 30 08:46:42 smithi032 podman[95120]: 2024-04-30 08:46:42.772903597 +0000 UTC m=+0.570478825 container attach 6ac4d04c3113d40768245d1bdcc67fb72b271866e6f1784a6d896d3ee7f4fc8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-30T08:46:43.441 INFO:journalctl@ceph.osd.3.smithi032.stdout:Apr 30 08:46:43 smithi032 podman[95120]: 2024-04-30 08:46:43.16606856 +0000 UTC m=+0.963643710 container died 6ac4d04c3113d40768245d1bdcc67fb72b271866e6f1784a6d896d3ee7f4fc8f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-30T08:46:43.939 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.3.service' 2024-04-30T08:46:44.061 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-30T08:46:44.062 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-30T08:46:44.062 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-30T08:46:44.062 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.4 2024-04-30T08:46:44.386 INFO:journalctl@ceph.osd.4.smithi071.stdout:Apr 30 08:46:44 smithi071 systemd[1]: Stopping Ceph osd.4 for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:46:44.753 INFO:journalctl@ceph.osd.4.smithi071.stdout:Apr 30 08:46:44 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-4[44913]: 2024-04-30T08:46:44.384+0000 7f07165d4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-30T08:46:44.754 INFO:journalctl@ceph.osd.4.smithi071.stdout:Apr 30 08:46:44 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-4[44913]: 2024-04-30T08:46:44.384+0000 7f07165d4700 -1 osd.4 89 *** Got signal Terminated *** 2024-04-30T08:46:44.754 INFO:journalctl@ceph.osd.4.smithi071.stdout:Apr 30 08:46:44 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-4[44913]: 2024-04-30T08:46:44.384+0000 7f07165d4700 -1 osd.4 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-30T08:46:50.254 INFO:journalctl@ceph.osd.4.smithi071.stdout:Apr 30 08:46:49 smithi071 podman[69163]: 2024-04-30 08:46:49.971255256 +0000 UTC m=+5.707831924 container died 5624c66ec459bc67bb105986aca7b79ea3dd5a203f240fe18bc8757e400e92e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, ceph=True) 2024-04-30T08:46:50.629 INFO:journalctl@ceph.osd.4.smithi071.stdout:Apr 30 08:46:50 smithi071 podman[69163]: 2024-04-30 08:46:50.36035243 +0000 UTC m=+6.096929080 container cleanup 5624c66ec459bc67bb105986aca7b79ea3dd5a203f240fe18bc8757e400e92e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-4, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, io.buildah.version=1.33.5) 2024-04-30T08:46:50.630 INFO:journalctl@ceph.osd.4.smithi071.stdout:Apr 30 08:46:50 smithi071 bash[69163]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-4 2024-04-30T08:46:50.975 INFO:journalctl@ceph.osd.4.smithi071.stdout:Apr 30 08:46:50 smithi071 podman[69180]: 2024-04-30 08:46:50.629225543 +0000 UTC m=+0.657592415 container remove 5624c66ec459bc67bb105986aca7b79ea3dd5a203f240fe18bc8757e400e92e1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-4, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-30T08:46:51.233 INFO:journalctl@ceph.osd.4.smithi071.stdout:Apr 30 08:46:50 smithi071 podman[69273]: 2024-04-30 08:46:50.877808317 +0000 UTC m=+0.018318758 image pull 16056ace0edc7f73516cf28297f1db04dda872801fb5b17b9157b0d1ccf38f9b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de 2024-04-30T08:46:51.233 INFO:journalctl@ceph.osd.4.smithi071.stdout:Apr 30 08:46:50 smithi071 podman[69273]: 2024-04-30 08:46:50.996846734 +0000 UTC m=+0.137357167 container create 4a525b66e29c2b9b8fa52fd711775a57153b03b277747f77628d39cb4ecd47db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240430, ceph=True) 2024-04-30T08:46:51.233 INFO:journalctl@ceph.osd.4.smithi071.stdout:Apr 30 08:46:51 smithi071 podman[69273]: 2024-04-30 08:46:51.232728885 +0000 UTC m=+0.373239318 container init 4a525b66e29c2b9b8fa52fd711775a57153b03b277747f77628d39cb4ecd47db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-04-30T08:46:51.506 INFO:journalctl@ceph.osd.4.smithi071.stdout:Apr 30 08:46:51 smithi071 podman[69273]: 2024-04-30 08:46:51.235900303 +0000 UTC m=+0.376410739 container start 4a525b66e29c2b9b8fa52fd711775a57153b03b277747f77628d39cb4ecd47db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2) 2024-04-30T08:46:51.506 INFO:journalctl@ceph.osd.4.smithi071.stdout:Apr 30 08:46:51 smithi071 podman[69273]: 2024-04-30 08:46:51.315997238 +0000 UTC m=+0.456507681 container attach 4a525b66e29c2b9b8fa52fd711775a57153b03b277747f77628d39cb4ecd47db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-4-deactivate, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux ) 2024-04-30T08:46:51.861 INFO:journalctl@ceph.osd.4.smithi071.stdout:Apr 30 08:46:51 smithi071 podman[69299]: 2024-04-30 08:46:51.552645796 +0000 UTC m=+0.018673548 container died 4a525b66e29c2b9b8fa52fd711775a57153b03b277747f77628d39cb4ecd47db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.license=GPLv2) 2024-04-30T08:46:52.198 DEBUG:teuthology.orchestra.run.smithi071:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.4.service' 2024-04-30T08:46:52.239 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-30T08:46:52.239 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-30T08:46:52.239 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-30T08:46:52.239 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.5 2024-04-30T08:46:52.550 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 30 08:46:52 smithi189 systemd[1]: Stopping Ceph osd.5 for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:46:53.050 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 30 08:46:52 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-5[45058]: 2024-04-30T08:46:52.559+0000 7f45f862e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-30T08:46:53.050 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 30 08:46:52 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-5[45058]: 2024-04-30T08:46:52.559+0000 7f45f862e700 -1 osd.5 89 *** Got signal Terminated *** 2024-04-30T08:46:53.050 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 30 08:46:52 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-5[45058]: 2024-04-30T08:46:52.559+0000 7f45f862e700 -1 osd.5 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-30T08:46:58.744 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 30 08:46:58 smithi189 podman[68876]: 2024-04-30 08:46:58.456568126 +0000 UTC m=+6.024161341 container died c4e2487a1068789ea7f84ee4f4fba7bb6b8a91e6ac36fea23a93a4f4fd6e858e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d) 2024-04-30T08:46:59.028 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 30 08:46:58 smithi189 podman[68876]: 2024-04-30 08:46:58.834545879 +0000 UTC m=+6.402139094 container cleanup c4e2487a1068789ea7f84ee4f4fba7bb6b8a91e6ac36fea23a93a4f4fd6e858e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-5, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.vendor=CentOS) 2024-04-30T08:46:59.028 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 30 08:46:58 smithi189 bash[68876]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-5 2024-04-30T08:46:59.300 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 30 08:46:59 smithi189 podman[68893]: 2024-04-30 08:46:59.027590931 +0000 UTC m=+0.568681999 container remove c4e2487a1068789ea7f84ee4f4fba7bb6b8a91e6ac36fea23a93a4f4fd6e858e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-5, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-30T08:46:59.715 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 30 08:46:59 smithi189 podman[68982]: 2024-04-30 08:46:59.340054954 +0000 UTC m=+0.017326535 image pull 16056ace0edc7f73516cf28297f1db04dda872801fb5b17b9157b0d1ccf38f9b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de 2024-04-30T08:46:59.715 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 30 08:46:59 smithi189 podman[68982]: 2024-04-30 08:46:59.453413747 +0000 UTC m=+0.130685313 container create 7b631e6ea781cf18710aadab354b6c684fc9ae42147367439b13953b4125ce6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux ) 2024-04-30T08:46:59.715 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 30 08:46:59 smithi189 podman[68982]: 2024-04-30 08:46:59.71459347 +0000 UTC m=+0.391865036 container init 7b631e6ea781cf18710aadab354b6c684fc9ae42147367439b13953b4125ce6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240430, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-30T08:47:00.008 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 30 08:46:59 smithi189 podman[68982]: 2024-04-30 08:46:59.717899763 +0000 UTC m=+0.395171338 container start 7b631e6ea781cf18710aadab354b6c684fc9ae42147367439b13953b4125ce6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d) 2024-04-30T08:47:00.008 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 30 08:46:59 smithi189 podman[68982]: 2024-04-30 08:46:59.797773338 +0000 UTC m=+0.475044905 container attach 7b631e6ea781cf18710aadab354b6c684fc9ae42147367439b13953b4125ce6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_CLEAN=True, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-30T08:47:00.291 INFO:journalctl@ceph.osd.5.smithi189.stdout:Apr 30 08:47:00 smithi189 podman[68982]: 2024-04-30 08:47:00.00781148 +0000 UTC m=+0.685083054 container died 7b631e6ea781cf18710aadab354b6c684fc9ae42147367439b13953b4125ce6d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-5-deactivate, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-30T08:47:00.699 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.5.service' 2024-04-30T08:47:00.736 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-30T08:47:00.736 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-30T08:47:00.736 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-30T08:47:00.736 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.6 2024-04-30T08:47:01.191 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 30 08:47:00 smithi032 systemd[1]: Stopping Ceph osd.6 for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:47:01.942 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 30 08:47:01 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-6[57124]: 2024-04-30T08:47:01.528+0000 7f3a47a00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-30T08:47:01.942 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 30 08:47:01 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-6[57124]: 2024-04-30T08:47:01.528+0000 7f3a47a00700 -1 osd.6 89 *** Got signal Terminated *** 2024-04-30T08:47:01.942 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 30 08:47:01 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-6[57124]: 2024-04-30T08:47:01.528+0000 7f3a47a00700 -1 osd.6 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-30T08:47:07.670 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 30 08:47:07 smithi032 podman[95225]: 2024-04-30 08:47:07.358912917 +0000 UTC m=+6.107987084 container died 9dfc7b047d1066d8066f9278845e411fb30daf27dfc313366a7b3eac50cd5892 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-6, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux ) 2024-04-30T08:47:07.941 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 30 08:47:07 smithi032 podman[95225]: 2024-04-30 08:47:07.787082119 +0000 UTC m=+6.536156306 container cleanup 9dfc7b047d1066d8066f9278845e411fb30daf27dfc313366a7b3eac50cd5892 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-6, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-30T08:47:07.942 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 30 08:47:07 smithi032 bash[95225]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-6 2024-04-30T08:47:08.441 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 30 08:47:08 smithi032 podman[95243]: 2024-04-30 08:47:08.013375549 +0000 UTC m=+0.641728820 container remove 9dfc7b047d1066d8066f9278845e411fb30daf27dfc313366a7b3eac50cd5892 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-6, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-30T08:47:09.192 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 30 08:47:08 smithi032 podman[95328]: 2024-04-30 08:47:08.816653362 +0000 UTC m=+0.035108557 image pull 16056ace0edc7f73516cf28297f1db04dda872801fb5b17b9157b0d1ccf38f9b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de 2024-04-30T08:47:09.192 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 30 08:47:08 smithi032 podman[95328]: 2024-04-30 08:47:08.930804859 +0000 UTC m=+0.149260067 container create c9c3bcaa690e72c293e980a610aeeaccfadfc13e438f1e5e849cc11abc31d3f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d) 2024-04-30T08:47:09.486 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 30 08:47:09 smithi032 podman[95328]: 2024-04-30 08:47:09.220977942 +0000 UTC m=+0.439433165 container init c9c3bcaa690e72c293e980a610aeeaccfadfc13e438f1e5e849cc11abc31d3f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-30T08:47:09.486 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 30 08:47:09 smithi032 podman[95328]: 2024-04-30 08:47:09.232419327 +0000 UTC m=+0.450874550 container start c9c3bcaa690e72c293e980a610aeeaccfadfc13e438f1e5e849cc11abc31d3f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-6-deactivate, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux ) 2024-04-30T08:47:09.486 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 30 08:47:09 smithi032 podman[95328]: 2024-04-30 08:47:09.338570513 +0000 UTC m=+0.557025776 container attach c9c3bcaa690e72c293e980a610aeeaccfadfc13e438f1e5e849cc11abc31d3f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-30T08:47:09.812 INFO:journalctl@ceph.osd.6.smithi032.stdout:Apr 30 08:47:09 smithi032 podman[95328]: 2024-04-30 08:47:09.714782147 +0000 UTC m=+0.933237394 container died c9c3bcaa690e72c293e980a610aeeaccfadfc13e438f1e5e849cc11abc31d3f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-30T08:47:10.511 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.6.service' 2024-04-30T08:47:10.604 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-30T08:47:10.604 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-30T08:47:10.605 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-30T08:47:10.605 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.7 2024-04-30T08:47:10.904 INFO:journalctl@ceph.osd.7.smithi071.stdout:Apr 30 08:47:10 smithi071 systemd[1]: Stopping Ceph osd.7 for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:47:11.253 INFO:journalctl@ceph.osd.7.smithi071.stdout:Apr 30 08:47:10 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-7[51771]: 2024-04-30T08:47:10.903+0000 7f7773cb2700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-30T08:47:11.253 INFO:journalctl@ceph.osd.7.smithi071.stdout:Apr 30 08:47:10 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-7[51771]: 2024-04-30T08:47:10.903+0000 7f7773cb2700 -1 osd.7 89 *** Got signal Terminated *** 2024-04-30T08:47:11.254 INFO:journalctl@ceph.osd.7.smithi071.stdout:Apr 30 08:47:10 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-7[51771]: 2024-04-30T08:47:10.903+0000 7f7773cb2700 -1 osd.7 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-30T08:47:16.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:15 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:15.776+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:16.921 INFO:journalctl@ceph.osd.7.smithi071.stdout:Apr 30 08:47:16 smithi071 podman[69381]: 2024-04-30 08:47:16.607069017 +0000 UTC m=+5.814355111 container died 330b77d31c1e6ff00bac1fc176c81980eab6eb3581dd6672d0b40bd1ad79057c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-7, org.label-schema.build-date=20240430, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-30T08:47:17.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:16 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:16.775+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:17.253 INFO:journalctl@ceph.osd.7.smithi071.stdout:Apr 30 08:47:17 smithi071 podman[69381]: 2024-04-30 08:47:17.020040444 +0000 UTC m=+6.227326539 container cleanup 330b77d31c1e6ff00bac1fc176c81980eab6eb3581dd6672d0b40bd1ad79057c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-7, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True) 2024-04-30T08:47:17.253 INFO:journalctl@ceph.osd.7.smithi071.stdout:Apr 30 08:47:17 smithi071 bash[69381]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-7 2024-04-30T08:47:17.685 INFO:journalctl@ceph.osd.7.smithi071.stdout:Apr 30 08:47:17 smithi071 podman[69408]: 2024-04-30 08:47:17.279413799 +0000 UTC m=+0.669290321 container remove 330b77d31c1e6ff00bac1fc176c81980eab6eb3581dd6672d0b40bd1ad79057c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-7, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , ceph=True) 2024-04-30T08:47:17.958 INFO:journalctl@ceph.osd.7.smithi071.stdout:Apr 30 08:47:17 smithi071 podman[69500]: 2024-04-30 08:47:17.588428579 +0000 UTC m=+0.018080483 image pull 16056ace0edc7f73516cf28297f1db04dda872801fb5b17b9157b0d1ccf38f9b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de 2024-04-30T08:47:17.959 INFO:journalctl@ceph.osd.7.smithi071.stdout:Apr 30 08:47:17 smithi071 podman[69500]: 2024-04-30 08:47:17.705709168 +0000 UTC m=+0.135361118 container create 437fa958c25f0c4435ed7f2e54f874ab26feafee29ff3f068cf04bad9fac23e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, ceph=True) 2024-04-30T08:47:17.959 INFO:journalctl@ceph.osd.7.smithi071.stdout:Apr 30 08:47:17 smithi071 podman[69500]: 2024-04-30 08:47:17.958129338 +0000 UTC m=+0.387781243 container init 437fa958c25f0c4435ed7f2e54f874ab26feafee29ff3f068cf04bad9fac23e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-30T08:47:18.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:17 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:17.742+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:18.254 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:18 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:18.157+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.248135+0000 front 2024-04-30T08:46:12.247837+0000 (oldest deadline 2024-04-30T08:47:18.147418+0000) 2024-04-30T08:47:18.254 INFO:journalctl@ceph.osd.7.smithi071.stdout:Apr 30 08:47:17 smithi071 podman[69500]: 2024-04-30 08:47:17.961521897 +0000 UTC m=+0.391173802 container start 437fa958c25f0c4435ed7f2e54f874ab26feafee29ff3f068cf04bad9fac23e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-7-deactivate, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-30T08:47:18.254 INFO:journalctl@ceph.osd.7.smithi071.stdout:Apr 30 08:47:18 smithi071 podman[69500]: 2024-04-30 08:47:18.083214266 +0000 UTC m=+0.512866177 container attach 437fa958c25f0c4435ed7f2e54f874ab26feafee29ff3f068cf04bad9fac23e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-30T08:47:18.620 INFO:journalctl@ceph.osd.7.smithi071.stdout:Apr 30 08:47:18 smithi071 podman[69500]: 2024-04-30 08:47:18.253622597 +0000 UTC m=+0.683274500 container died 437fa958c25f0c4435ed7f2e54f874ab26feafee29ff3f068cf04bad9fac23e3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-30T08:47:18.942 DEBUG:teuthology.orchestra.run.smithi071:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.7.service' 2024-04-30T08:47:18.979 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-30T08:47:18.980 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-30T08:47:18.980 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-30T08:47:18.980 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.8 2024-04-30T08:47:19.006 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:18 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:18.714+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:19.253 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:19 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:19.120+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.248135+0000 front 2024-04-30T08:46:12.247837+0000 (oldest deadline 2024-04-30T08:47:18.147418+0000) 2024-04-30T08:47:19.295 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 30 08:47:19 smithi189 systemd[1]: Stopping Ceph osd.8 for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:47:19.550 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 30 08:47:19 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-8[51785]: 2024-04-30T08:47:19.294+0000 7fd624b74700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-30T08:47:19.550 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 30 08:47:19 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-8[51785]: 2024-04-30T08:47:19.294+0000 7fd624b74700 -1 osd.8 89 *** Got signal Terminated *** 2024-04-30T08:47:19.550 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 30 08:47:19 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-8[51785]: 2024-04-30T08:47:19.294+0000 7fd624b74700 -1 osd.8 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-30T08:47:20.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:19 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:19.700+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:20.504 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:20 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:20.158+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.248135+0000 front 2024-04-30T08:46:12.247837+0000 (oldest deadline 2024-04-30T08:47:18.147418+0000) 2024-04-30T08:47:21.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:20 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:20.679+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:21.503 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:21 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:21.116+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.248135+0000 front 2024-04-30T08:46:12.247837+0000 (oldest deadline 2024-04-30T08:47:18.147418+0000) 2024-04-30T08:47:22.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:21 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:21.656+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:22.503 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:22 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:22.126+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.248135+0000 front 2024-04-30T08:46:12.247837+0000 (oldest deadline 2024-04-30T08:47:18.147418+0000) 2024-04-30T08:47:23.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:22 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:22.645+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:23.503 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:23 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:23.112+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.248135+0000 front 2024-04-30T08:46:12.247837+0000 (oldest deadline 2024-04-30T08:47:18.147418+0000) 2024-04-30T08:47:24.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:23 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:23.649+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:24.191 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:47:23 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:47:23.851+0000 7ff432400700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.71:6804 osd.1 since back 2024-04-30T08:46:22.186976+0000 front 2024-04-30T08:46:22.186942+0000 (oldest deadline 2024-04-30T08:47:23.287096+0000) 2024-04-30T08:47:24.503 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:24 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:24.075+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.248135+0000 front 2024-04-30T08:46:12.247837+0000 (oldest deadline 2024-04-30T08:47:18.147418+0000) 2024-04-30T08:47:24.912 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:24 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:24.634+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:25.191 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:47:24 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:47:24.891+0000 7ff432400700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.71:6804 osd.1 since back 2024-04-30T08:46:22.186976+0000 front 2024-04-30T08:46:22.186942+0000 (oldest deadline 2024-04-30T08:47:23.287096+0000) 2024-04-30T08:47:25.261 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 30 08:47:24 smithi189 podman[69100]: 2024-04-30 08:47:24.911743645 +0000 UTC m=+5.741996210 container died 034b9a88b5935b49a7dcee7faf2eda25e93ce901ff205fe6c5a83c72ed91ae99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-8, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-30T08:47:25.503 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:25 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:25.072+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.248135+0000 front 2024-04-30T08:46:12.247837+0000 (oldest deadline 2024-04-30T08:47:18.147418+0000) 2024-04-30T08:47:25.503 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:25 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:25.072+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.71:6804 osd.1 since back 2024-04-30T08:46:22.248549+0000 front 2024-04-30T08:46:22.248440+0000 (oldest deadline 2024-04-30T08:47:24.548499+0000) 2024-04-30T08:47:25.550 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 30 08:47:25 smithi189 podman[69100]: 2024-04-30 08:47:25.260966334 +0000 UTC m=+6.091218902 container cleanup 034b9a88b5935b49a7dcee7faf2eda25e93ce901ff205fe6c5a83c72ed91ae99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-8, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-30T08:47:25.550 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 30 08:47:25 smithi189 bash[69100]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-8 2024-04-30T08:47:25.913 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 30 08:47:25 smithi189 podman[69117]: 2024-04-30 08:47:25.578851149 +0000 UTC m=+0.667636678 container remove 034b9a88b5935b49a7dcee7faf2eda25e93ce901ff205fe6c5a83c72ed91ae99 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-8, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2) 2024-04-30T08:47:25.913 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:25 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:25.660+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:26.183 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 30 08:47:25 smithi189 podman[69211]: 2024-04-30 08:47:25.912936362 +0000 UTC m=+0.107591879 container create eb7db1fad275b09d52bc929d428c74b1f9794f47a74191cc6ca2f59753e6ab03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True) 2024-04-30T08:47:26.183 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 30 08:47:25 smithi189 podman[69211]: 2024-04-30 08:47:25.823436744 +0000 UTC m=+0.018092276 image pull 16056ace0edc7f73516cf28297f1db04dda872801fb5b17b9157b0d1ccf38f9b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de 2024-04-30T08:47:26.191 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:47:25 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:47:25.862+0000 7ff432400700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.71:6804 osd.1 since back 2024-04-30T08:46:22.186976+0000 front 2024-04-30T08:46:22.186942+0000 (oldest deadline 2024-04-30T08:47:23.287096+0000) 2024-04-30T08:47:26.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 30 08:47:26 smithi189 podman[69211]: 2024-04-30 08:47:26.182796277 +0000 UTC m=+0.377451824 container init eb7db1fad275b09d52bc929d428c74b1f9794f47a74191cc6ca2f59753e6ab03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-30T08:47:26.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 30 08:47:26 smithi189 podman[69211]: 2024-04-30 08:47:26.186151501 +0000 UTC m=+0.380807023 container start eb7db1fad275b09d52bc929d428c74b1f9794f47a74191cc6ca2f59753e6ab03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-30T08:47:26.487 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 30 08:47:26 smithi189 podman[69211]: 2024-04-30 08:47:26.265257435 +0000 UTC m=+0.459912966 container attach eb7db1fad275b09d52bc929d428c74b1f9794f47a74191cc6ca2f59753e6ab03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS) 2024-04-30T08:47:26.503 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:26 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:26.031+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.248135+0000 front 2024-04-30T08:46:12.247837+0000 (oldest deadline 2024-04-30T08:47:18.147418+0000) 2024-04-30T08:47:26.503 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:26 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:26.031+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.71:6804 osd.1 since back 2024-04-30T08:46:22.248549+0000 front 2024-04-30T08:46:22.248440+0000 (oldest deadline 2024-04-30T08:47:24.548499+0000) 2024-04-30T08:47:26.793 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:26 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:26.700+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:26.794 INFO:journalctl@ceph.osd.8.smithi189.stdout:Apr 30 08:47:26 smithi189 podman[69211]: 2024-04-30 08:47:26.486866306 +0000 UTC m=+0.681521820 container died eb7db1fad275b09d52bc929d428c74b1f9794f47a74191cc6ca2f59753e6ab03 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_CLEAN=True) 2024-04-30T08:47:27.115 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.8.service' 2024-04-30T08:47:27.191 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:47:26 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:47:26.888+0000 7ff432400700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.71:6804 osd.1 since back 2024-04-30T08:46:22.186976+0000 front 2024-04-30T08:46:22.186942+0000 (oldest deadline 2024-04-30T08:47:23.287096+0000) 2024-04-30T08:47:27.193 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-30T08:47:27.193 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-30T08:47:27.193 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-30T08:47:27.193 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.9 2024-04-30T08:47:27.503 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:27 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:27.007+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.248135+0000 front 2024-04-30T08:46:12.247837+0000 (oldest deadline 2024-04-30T08:47:18.147418+0000) 2024-04-30T08:47:27.503 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:27 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:27.007+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.71:6804 osd.1 since back 2024-04-30T08:46:22.248549+0000 front 2024-04-30T08:46:22.248440+0000 (oldest deadline 2024-04-30T08:47:24.548499+0000) 2024-04-30T08:47:27.691 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:47:27 smithi032 systemd[1]: Stopping Ceph osd.9 for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:47:28.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:27 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:27.691+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:28.191 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:47:27 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:47:27.796+0000 7ff436c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-30T08:47:28.191 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:47:27 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:47:27.796+0000 7ff436c00700 -1 osd.9 89 *** Got signal Terminated *** 2024-04-30T08:47:28.192 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:47:27 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:47:27.796+0000 7ff436c00700 -1 osd.9 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-30T08:47:28.192 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:47:27 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:47:27.856+0000 7ff432400700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.71:6804 osd.1 since back 2024-04-30T08:46:22.186976+0000 front 2024-04-30T08:46:22.186942+0000 (oldest deadline 2024-04-30T08:47:23.287096+0000) 2024-04-30T08:47:28.321 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:28 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:28.014+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.248135+0000 front 2024-04-30T08:46:12.247837+0000 (oldest deadline 2024-04-30T08:47:18.147418+0000) 2024-04-30T08:47:28.321 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:28 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:28.014+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.71:6804 osd.1 since back 2024-04-30T08:46:22.248549+0000 front 2024-04-30T08:46:22.248440+0000 (oldest deadline 2024-04-30T08:47:24.548499+0000) 2024-04-30T08:47:29.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:28 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:28.721+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:29.191 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:47:28 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:47:28.878+0000 7ff432400700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.71:6804 osd.1 since back 2024-04-30T08:46:22.186976+0000 front 2024-04-30T08:46:22.186942+0000 (oldest deadline 2024-04-30T08:47:23.287096+0000) 2024-04-30T08:47:29.253 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:28 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:28.991+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.248135+0000 front 2024-04-30T08:46:12.247837+0000 (oldest deadline 2024-04-30T08:47:18.147418+0000) 2024-04-30T08:47:29.253 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:28 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:28.991+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.71:6804 osd.1 since back 2024-04-30T08:46:22.248549+0000 front 2024-04-30T08:46:22.248440+0000 (oldest deadline 2024-04-30T08:47:24.548499+0000) 2024-04-30T08:47:30.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:29 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:29.695+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:30.191 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:47:29 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:47:29.910+0000 7ff432400700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.71:6804 osd.1 since back 2024-04-30T08:46:22.186976+0000 front 2024-04-30T08:46:22.186942+0000 (oldest deadline 2024-04-30T08:47:23.287096+0000) 2024-04-30T08:47:30.503 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:30 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:30.025+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.248135+0000 front 2024-04-30T08:46:12.247837+0000 (oldest deadline 2024-04-30T08:47:18.147418+0000) 2024-04-30T08:47:30.503 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:30 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:30.025+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.71:6804 osd.1 since back 2024-04-30T08:46:22.248549+0000 front 2024-04-30T08:46:22.248440+0000 (oldest deadline 2024-04-30T08:47:24.548499+0000) 2024-04-30T08:47:31.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:30 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:30.737+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:31.253 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:30 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:30.976+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.248135+0000 front 2024-04-30T08:46:12.247837+0000 (oldest deadline 2024-04-30T08:47:18.147418+0000) 2024-04-30T08:47:31.254 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:30 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:30.976+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.71:6804 osd.1 since back 2024-04-30T08:46:22.248549+0000 front 2024-04-30T08:46:22.248440+0000 (oldest deadline 2024-04-30T08:47:24.548499+0000) 2024-04-30T08:47:31.441 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:47:30 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:47:30.953+0000 7ff432400700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.71:6804 osd.1 since back 2024-04-30T08:46:22.186976+0000 front 2024-04-30T08:46:22.186942+0000 (oldest deadline 2024-04-30T08:47:23.287096+0000) 2024-04-30T08:47:32.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:31 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:31.755+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:32.254 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:31 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:31.945+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.248135+0000 front 2024-04-30T08:46:12.247837+0000 (oldest deadline 2024-04-30T08:47:18.147418+0000) 2024-04-30T08:47:32.254 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:31 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:31.945+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.71:6804 osd.1 since back 2024-04-30T08:46:22.248549+0000 front 2024-04-30T08:46:22.248440+0000 (oldest deadline 2024-04-30T08:47:24.548499+0000) 2024-04-30T08:47:32.441 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:47:32 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9[63681]: 2024-04-30T08:47:31.999+0000 7ff432400700 -1 osd.9 89 heartbeat_check: no reply from 172.21.15.71:6804 osd.1 since back 2024-04-30T08:46:22.186976+0000 front 2024-04-30T08:46:22.186942+0000 (oldest deadline 2024-04-30T08:47:23.287096+0000) 2024-04-30T08:47:33.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:32 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:32.751+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:33.253 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:32 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:32.948+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.248135+0000 front 2024-04-30T08:46:12.247837+0000 (oldest deadline 2024-04-30T08:47:18.147418+0000) 2024-04-30T08:47:33.254 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:32 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:32.948+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.71:6804 osd.1 since back 2024-04-30T08:46:22.248549+0000 front 2024-04-30T08:46:22.248440+0000 (oldest deadline 2024-04-30T08:47:24.548499+0000) 2024-04-30T08:47:33.254 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:32 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:32.948+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-30T08:46:29.050968+0000 front 2024-04-30T08:46:29.050902+0000 (oldest deadline 2024-04-30T08:47:32.550749+0000) 2024-04-30T08:47:34.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:33 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:33.721+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:34.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:33 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:33.721+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-30T08:46:28.696366+0000 front 2024-04-30T08:46:28.696395+0000 (oldest deadline 2024-04-30T08:47:33.396486+0000) 2024-04-30T08:47:34.192 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:47:33 smithi032 podman[95446]: 2024-04-30 08:47:33.829195681 +0000 UTC m=+6.164188025 container died 7a2dccf97f0e1e2580fdf5941f6a1e68a2a0c96c8b1dc095ad239a16fa1a7dd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-30T08:47:34.253 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:33 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:33.921+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.248135+0000 front 2024-04-30T08:46:12.247837+0000 (oldest deadline 2024-04-30T08:47:18.147418+0000) 2024-04-30T08:47:34.253 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:33 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:33.921+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.71:6804 osd.1 since back 2024-04-30T08:46:22.248549+0000 front 2024-04-30T08:46:22.248440+0000 (oldest deadline 2024-04-30T08:47:24.548499+0000) 2024-04-30T08:47:34.254 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:33 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:33.921+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-30T08:46:29.050968+0000 front 2024-04-30T08:46:29.050902+0000 (oldest deadline 2024-04-30T08:47:32.550749+0000) 2024-04-30T08:47:34.649 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:47:34 smithi032 podman[95446]: 2024-04-30 08:47:34.389097895 +0000 UTC m=+6.724090250 container cleanup 7a2dccf97f0e1e2580fdf5941f6a1e68a2a0c96c8b1dc095ad239a16fa1a7dd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-30T08:47:34.650 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:47:34 smithi032 bash[95446]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9 2024-04-30T08:47:34.941 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:47:34 smithi032 podman[95464]: 2024-04-30 08:47:34.648295022 +0000 UTC m=+0.802974755 container remove 7a2dccf97f0e1e2580fdf5941f6a1e68a2a0c96c8b1dc095ad239a16fa1a7dd0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, ceph=True) 2024-04-30T08:47:35.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:34 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:34.748+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:35.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:34 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:34.748+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-30T08:46:28.696366+0000 front 2024-04-30T08:46:28.696395+0000 (oldest deadline 2024-04-30T08:47:33.396486+0000) 2024-04-30T08:47:35.253 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:34 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:34.902+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.248135+0000 front 2024-04-30T08:46:12.247837+0000 (oldest deadline 2024-04-30T08:47:18.147418+0000) 2024-04-30T08:47:35.254 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:34 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:34.902+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.71:6804 osd.1 since back 2024-04-30T08:46:22.248549+0000 front 2024-04-30T08:46:22.248440+0000 (oldest deadline 2024-04-30T08:47:24.548499+0000) 2024-04-30T08:47:35.254 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:34 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:34.902+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-30T08:46:29.050968+0000 front 2024-04-30T08:46:29.050902+0000 (oldest deadline 2024-04-30T08:47:32.550749+0000) 2024-04-30T08:47:35.823 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:47:35 smithi032 podman[95552]: 2024-04-30 08:47:35.418189387 +0000 UTC m=+0.033512764 image pull 16056ace0edc7f73516cf28297f1db04dda872801fb5b17b9157b0d1ccf38f9b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de 2024-04-30T08:47:35.823 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:47:35 smithi032 podman[95552]: 2024-04-30 08:47:35.540821691 +0000 UTC m=+0.156145105 container create eb1e440f15a1511530338e8f750cac7098860f524ae452e0e1b877cfda985750 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9-deactivate, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-30T08:47:35.823 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:47:35 smithi032 podman[95552]: 2024-04-30 08:47:35.820861656 +0000 UTC m=+0.436185059 container init eb1e440f15a1511530338e8f750cac7098860f524ae452e0e1b877cfda985750 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9-deactivate, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-30T08:47:36.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:35 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:35.797+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:36.071 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:35 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:35.797+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-30T08:46:28.696366+0000 front 2024-04-30T08:46:28.696395+0000 (oldest deadline 2024-04-30T08:47:33.396486+0000) 2024-04-30T08:47:36.191 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:47:35 smithi032 podman[95552]: 2024-04-30 08:47:35.833652663 +0000 UTC m=+0.448976049 container start eb1e440f15a1511530338e8f750cac7098860f524ae452e0e1b877cfda985750 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-30T08:47:36.191 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:47:35 smithi032 podman[95552]: 2024-04-30 08:47:35.917938723 +0000 UTC m=+0.533262134 container attach eb1e440f15a1511530338e8f750cac7098860f524ae452e0e1b877cfda985750 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-30T08:47:36.253 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:35 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:35.936+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.248135+0000 front 2024-04-30T08:46:12.247837+0000 (oldest deadline 2024-04-30T08:47:18.147418+0000) 2024-04-30T08:47:36.254 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:35 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:35.936+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.71:6804 osd.1 since back 2024-04-30T08:46:22.248549+0000 front 2024-04-30T08:46:22.248440+0000 (oldest deadline 2024-04-30T08:47:24.548499+0000) 2024-04-30T08:47:36.254 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:35 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:35.936+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-30T08:46:29.050968+0000 front 2024-04-30T08:46:29.050902+0000 (oldest deadline 2024-04-30T08:47:32.550749+0000) 2024-04-30T08:47:36.653 INFO:journalctl@ceph.osd.9.smithi032.stdout:Apr 30 08:47:36 smithi032 podman[95552]: 2024-04-30 08:47:36.300702754 +0000 UTC m=+0.916026214 container died eb1e440f15a1511530338e8f750cac7098860f524ae452e0e1b877cfda985750 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-9-deactivate, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-04-30T08:47:37.171 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.9.service' 2024-04-30T08:47:37.253 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:36 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:36.914+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.248135+0000 front 2024-04-30T08:46:12.247837+0000 (oldest deadline 2024-04-30T08:47:18.147418+0000) 2024-04-30T08:47:37.253 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:36 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:36.914+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.71:6804 osd.1 since back 2024-04-30T08:46:22.248549+0000 front 2024-04-30T08:46:22.248440+0000 (oldest deadline 2024-04-30T08:47:24.548499+0000) 2024-04-30T08:47:37.253 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:36 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:36.914+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-30T08:46:29.050968+0000 front 2024-04-30T08:46:29.050902+0000 (oldest deadline 2024-04-30T08:47:32.550749+0000) 2024-04-30T08:47:37.278 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-30T08:47:37.278 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-30T08:47:37.278 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-30T08:47:37.278 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.10 2024-04-30T08:47:37.300 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:36 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:36.807+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:37.300 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:36 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:36.807+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-30T08:46:28.696366+0000 front 2024-04-30T08:46:28.696395+0000 (oldest deadline 2024-04-30T08:47:33.396486+0000) 2024-04-30T08:47:37.572 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:37 smithi071 systemd[1]: Stopping Ceph osd.10 for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:47:37.957 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:37 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:37.571+0000 7f89f3da9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-30T08:47:37.957 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:37 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:37.571+0000 7f89f3da9700 -1 osd.10 89 *** Got signal Terminated *** 2024-04-30T08:47:37.957 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:37 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:37.571+0000 7f89f3da9700 -1 osd.10 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-30T08:47:37.957 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:37 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:37.956+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.248135+0000 front 2024-04-30T08:46:12.247837+0000 (oldest deadline 2024-04-30T08:47:18.147418+0000) 2024-04-30T08:47:37.957 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:37 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:37.956+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.71:6804 osd.1 since back 2024-04-30T08:46:22.248549+0000 front 2024-04-30T08:46:22.248440+0000 (oldest deadline 2024-04-30T08:47:24.548499+0000) 2024-04-30T08:47:37.957 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:37 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:37.956+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-30T08:46:29.050968+0000 front 2024-04-30T08:46:29.050902+0000 (oldest deadline 2024-04-30T08:47:32.550749+0000) 2024-04-30T08:47:38.300 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:37 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:37.842+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:38.300 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:37 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:37.842+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-30T08:46:28.696366+0000 front 2024-04-30T08:46:28.696395+0000 (oldest deadline 2024-04-30T08:47:33.396486+0000) 2024-04-30T08:47:39.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:38 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:38.819+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:39.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:38 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:38.819+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-30T08:46:28.696366+0000 front 2024-04-30T08:46:28.696395+0000 (oldest deadline 2024-04-30T08:47:33.396486+0000) 2024-04-30T08:47:39.253 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:38 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:38.987+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.248135+0000 front 2024-04-30T08:46:12.247837+0000 (oldest deadline 2024-04-30T08:47:18.147418+0000) 2024-04-30T08:47:39.253 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:38 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:38.987+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.71:6804 osd.1 since back 2024-04-30T08:46:22.248549+0000 front 2024-04-30T08:46:22.248440+0000 (oldest deadline 2024-04-30T08:47:24.548499+0000) 2024-04-30T08:47:39.254 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:38 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:38.987+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-30T08:46:29.050968+0000 front 2024-04-30T08:46:29.050902+0000 (oldest deadline 2024-04-30T08:47:32.550749+0000) 2024-04-30T08:47:40.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:39 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:39.775+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:40.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:39 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:39.775+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-30T08:46:28.696366+0000 front 2024-04-30T08:46:28.696395+0000 (oldest deadline 2024-04-30T08:47:33.396486+0000) 2024-04-30T08:47:40.253 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:39 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:39.955+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.248135+0000 front 2024-04-30T08:46:12.247837+0000 (oldest deadline 2024-04-30T08:47:18.147418+0000) 2024-04-30T08:47:40.254 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:39 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:39.955+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.71:6804 osd.1 since back 2024-04-30T08:46:22.248549+0000 front 2024-04-30T08:46:22.248440+0000 (oldest deadline 2024-04-30T08:47:24.548499+0000) 2024-04-30T08:47:40.254 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:39 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:39.955+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-30T08:46:29.050968+0000 front 2024-04-30T08:46:29.050902+0000 (oldest deadline 2024-04-30T08:47:32.550749+0000) 2024-04-30T08:47:41.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:40 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:40.745+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:41.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:40 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:40.745+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-30T08:46:28.696366+0000 front 2024-04-30T08:46:28.696395+0000 (oldest deadline 2024-04-30T08:47:33.396486+0000) 2024-04-30T08:47:41.253 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:41 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:41.002+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.248135+0000 front 2024-04-30T08:46:12.247837+0000 (oldest deadline 2024-04-30T08:47:18.147418+0000) 2024-04-30T08:47:41.254 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:41 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:41.002+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.71:6804 osd.1 since back 2024-04-30T08:46:22.248549+0000 front 2024-04-30T08:46:22.248440+0000 (oldest deadline 2024-04-30T08:47:24.548499+0000) 2024-04-30T08:47:41.254 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:41 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:41.002+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-30T08:46:29.050968+0000 front 2024-04-30T08:46:29.050902+0000 (oldest deadline 2024-04-30T08:47:32.550749+0000) 2024-04-30T08:47:42.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:41 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:41.725+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:42.051 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:41 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:41.725+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-30T08:46:28.696366+0000 front 2024-04-30T08:46:28.696395+0000 (oldest deadline 2024-04-30T08:47:33.396486+0000) 2024-04-30T08:47:42.051 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:41 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:41.725+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-30T08:46:35.697638+0000 front 2024-04-30T08:46:35.697762+0000 (oldest deadline 2024-04-30T08:47:40.997646+0000) 2024-04-30T08:47:42.504 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:42 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:42.049+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.248135+0000 front 2024-04-30T08:46:12.247837+0000 (oldest deadline 2024-04-30T08:47:18.147418+0000) 2024-04-30T08:47:42.504 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:42 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:42.049+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.71:6804 osd.1 since back 2024-04-30T08:46:22.248549+0000 front 2024-04-30T08:46:22.248440+0000 (oldest deadline 2024-04-30T08:47:24.548499+0000) 2024-04-30T08:47:42.504 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:42 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10[58786]: 2024-04-30T08:47:42.049+0000 7f89f035f700 -1 osd.10 89 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-30T08:46:29.050968+0000 front 2024-04-30T08:46:29.050902+0000 (oldest deadline 2024-04-30T08:47:32.550749+0000) 2024-04-30T08:47:43.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:42 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:42.760+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:43.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:42 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:42.760+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-30T08:46:28.696366+0000 front 2024-04-30T08:46:28.696395+0000 (oldest deadline 2024-04-30T08:47:33.396486+0000) 2024-04-30T08:47:43.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:42 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:42.760+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-30T08:46:35.697638+0000 front 2024-04-30T08:46:35.697762+0000 (oldest deadline 2024-04-30T08:47:40.997646+0000) 2024-04-30T08:47:43.589 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:43 smithi071 podman[69606]: 2024-04-30 08:47:43.269529843 +0000 UTC m=+5.801361614 container died 9779d2ada4913ef643a6650197d00f1a25615df156bc357f3d4dfb78e42a5f7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-04-30T08:47:43.914 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:43 smithi071 podman[69606]: 2024-04-30 08:47:43.695756921 +0000 UTC m=+6.227588679 container cleanup 9779d2ada4913ef643a6650197d00f1a25615df156bc357f3d4dfb78e42a5f7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d) 2024-04-30T08:47:43.914 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:43 smithi071 bash[69606]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10 2024-04-30T08:47:44.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:43 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:43.783+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:44.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:43 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:43.783+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-30T08:46:28.696366+0000 front 2024-04-30T08:46:28.696395+0000 (oldest deadline 2024-04-30T08:47:33.396486+0000) 2024-04-30T08:47:44.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:43 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:43.783+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-30T08:46:35.697638+0000 front 2024-04-30T08:46:35.697762+0000 (oldest deadline 2024-04-30T08:47:40.997646+0000) 2024-04-30T08:47:44.253 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:43 smithi071 podman[69635]: 2024-04-30 08:47:43.913501323 +0000 UTC m=+0.639744355 container remove 9779d2ada4913ef643a6650197d00f1a25615df156bc357f3d4dfb78e42a5f7c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-30T08:47:44.562 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:44 smithi071 podman[69722]: 2024-04-30 08:47:44.212797613 +0000 UTC m=+0.017682272 image pull 16056ace0edc7f73516cf28297f1db04dda872801fb5b17b9157b0d1ccf38f9b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de 2024-04-30T08:47:44.562 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:44 smithi071 podman[69722]: 2024-04-30 08:47:44.314267291 +0000 UTC m=+0.119151940 container create 5746515d902a79da8c8af70b43d3109744548c7dc730658c279d44b0fd52e2d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10-deactivate, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-30T08:47:44.562 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:44 smithi071 podman[69722]: 2024-04-30 08:47:44.5581394 +0000 UTC m=+0.363024053 container init 5746515d902a79da8c8af70b43d3109744548c7dc730658c279d44b0fd52e2d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-30T08:47:44.562 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:44 smithi071 podman[69722]: 2024-04-30 08:47:44.561228982 +0000 UTC m=+0.366113635 container start 5746515d902a79da8c8af70b43d3109744548c7dc730658c279d44b0fd52e2d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10-deactivate, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True) 2024-04-30T08:47:44.853 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:44 smithi071 podman[69722]: 2024-04-30 08:47:44.675674078 +0000 UTC m=+0.480558732 container attach 5746515d902a79da8c8af70b43d3109744548c7dc730658c279d44b0fd52e2d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_BRANCH=HEAD) 2024-04-30T08:47:45.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:44 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:44.744+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:45.051 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:44 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:44.744+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-30T08:46:28.696366+0000 front 2024-04-30T08:46:28.696395+0000 (oldest deadline 2024-04-30T08:47:33.396486+0000) 2024-04-30T08:47:45.051 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:44 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:44.744+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-30T08:46:35.697638+0000 front 2024-04-30T08:46:35.697762+0000 (oldest deadline 2024-04-30T08:47:40.997646+0000) 2024-04-30T08:47:45.136 INFO:journalctl@ceph.osd.10.smithi071.stdout:Apr 30 08:47:44 smithi071 podman[69722]: 2024-04-30 08:47:44.853249097 +0000 UTC m=+0.658133755 container died 5746515d902a79da8c8af70b43d3109744548c7dc730658c279d44b0fd52e2d7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-30T08:47:45.419 DEBUG:teuthology.orchestra.run.smithi071:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.10.service' 2024-04-30T08:47:45.457 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-30T08:47:45.457 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-30T08:47:45.457 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-30T08:47:45.458 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.11 2024-04-30T08:47:45.772 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:45 smithi189 systemd[1]: Stopping Ceph osd.11 for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:47:46.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:45 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:45.770+0000 7fe3a83f0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-30T08:47:46.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:45 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:45.770+0000 7fe3a83f0700 -1 osd.11 89 *** Got signal Terminated *** 2024-04-30T08:47:46.051 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:45 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:45.770+0000 7fe3a83f0700 -1 osd.11 89 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-30T08:47:46.051 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:45 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:45.776+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:46.051 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:45 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:45.776+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-30T08:46:28.696366+0000 front 2024-04-30T08:46:28.696395+0000 (oldest deadline 2024-04-30T08:47:33.396486+0000) 2024-04-30T08:47:46.051 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:45 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:45.776+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-30T08:46:35.697638+0000 front 2024-04-30T08:46:35.697762+0000 (oldest deadline 2024-04-30T08:47:40.997646+0000) 2024-04-30T08:47:47.300 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:46 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:46.810+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:47.301 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:46 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:46.810+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-30T08:46:28.696366+0000 front 2024-04-30T08:46:28.696395+0000 (oldest deadline 2024-04-30T08:47:33.396486+0000) 2024-04-30T08:47:47.301 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:46 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:46.810+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-30T08:46:35.697638+0000 front 2024-04-30T08:46:35.697762+0000 (oldest deadline 2024-04-30T08:47:40.997646+0000) 2024-04-30T08:47:48.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:47 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:47.772+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:48.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:47 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:47.772+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-30T08:46:28.696366+0000 front 2024-04-30T08:46:28.696395+0000 (oldest deadline 2024-04-30T08:47:33.396486+0000) 2024-04-30T08:47:48.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:47 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:47.772+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-30T08:46:35.697638+0000 front 2024-04-30T08:46:35.697762+0000 (oldest deadline 2024-04-30T08:47:40.997646+0000) 2024-04-30T08:47:49.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:48 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:48.758+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:49.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:48 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:48.758+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-30T08:46:28.696366+0000 front 2024-04-30T08:46:28.696395+0000 (oldest deadline 2024-04-30T08:47:33.396486+0000) 2024-04-30T08:47:49.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:48 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:48.758+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-30T08:46:35.697638+0000 front 2024-04-30T08:46:35.697762+0000 (oldest deadline 2024-04-30T08:47:40.997646+0000) 2024-04-30T08:47:50.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:49 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:49.718+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:50.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:49 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:49.718+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-30T08:46:28.696366+0000 front 2024-04-30T08:46:28.696395+0000 (oldest deadline 2024-04-30T08:47:33.396486+0000) 2024-04-30T08:47:50.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:49 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:49.718+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-30T08:46:35.697638+0000 front 2024-04-30T08:46:35.697762+0000 (oldest deadline 2024-04-30T08:47:40.997646+0000) 2024-04-30T08:47:51.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:50 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:50.679+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6806 osd.0 since back 2024-04-30T08:46:12.091386+0000 front 2024-04-30T08:46:12.091195+0000 (oldest deadline 2024-04-30T08:47:14.991147+0000) 2024-04-30T08:47:51.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:50 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:50.679+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.189:6804 osd.2 since back 2024-04-30T08:46:28.696366+0000 front 2024-04-30T08:46:28.696395+0000 (oldest deadline 2024-04-30T08:47:33.396486+0000) 2024-04-30T08:47:51.050 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:50 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11[58554]: 2024-04-30T08:47:50.679+0000 7fe3a099e700 -1 osd.11 89 heartbeat_check: no reply from 172.21.15.32:6814 osd.3 since back 2024-04-30T08:46:35.697638+0000 front 2024-04-30T08:46:35.697762+0000 (oldest deadline 2024-04-30T08:47:40.997646+0000) 2024-04-30T08:47:51.671 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:51 smithi189 podman[69328]: 2024-04-30 08:47:51.409113429 +0000 UTC m=+5.762292707 container died b149d40573b8dd1bcb5ae0d2e41d55134353dd9d12d2780758314a06ca8e5ad7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430) 2024-04-30T08:47:52.013 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:51 smithi189 podman[69328]: 2024-04-30 08:47:51.761850405 +0000 UTC m=+6.115029682 container cleanup b149d40573b8dd1bcb5ae0d2e41d55134353dd9d12d2780758314a06ca8e5ad7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True) 2024-04-30T08:47:52.013 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:51 smithi189 bash[69328]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11 2024-04-30T08:47:52.300 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:52 smithi189 podman[69345]: 2024-04-30 08:47:52.012847957 +0000 UTC m=+0.602750569 container remove b149d40573b8dd1bcb5ae0d2e41d55134353dd9d12d2780758314a06ca8e5ad7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True) 2024-04-30T08:47:52.633 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:52 smithi189 podman[69436]: 2024-04-30 08:47:52.275396642 +0000 UTC m=+0.018111947 image pull 16056ace0edc7f73516cf28297f1db04dda872801fb5b17b9157b0d1ccf38f9b quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de 2024-04-30T08:47:52.634 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:52 smithi189 podman[69436]: 2024-04-30 08:47:52.414060833 +0000 UTC m=+0.156776125 container create dc48174aa38fb8e4238271b74fc7ba1cc373d9e587abc244f9240e60f495b243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11-deactivate, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-04-30T08:47:52.634 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:52 smithi189 podman[69436]: 2024-04-30 08:47:52.632918767 +0000 UTC m=+0.375634060 container init dc48174aa38fb8e4238271b74fc7ba1cc373d9e587abc244f9240e60f495b243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11-deactivate, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.build-date=20240430, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-04-30T08:47:52.927 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:52 smithi189 podman[69436]: 2024-04-30 08:47:52.636133311 +0000 UTC m=+0.378848611 container start dc48174aa38fb8e4238271b74fc7ba1cc373d9e587abc244f9240e60f495b243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS) 2024-04-30T08:47:52.927 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:52 smithi189 podman[69436]: 2024-04-30 08:47:52.732995972 +0000 UTC m=+0.475711277 container attach dc48174aa38fb8e4238271b74fc7ba1cc373d9e587abc244f9240e60f495b243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-30T08:47:53.211 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:52 smithi189 podman[69436]: 2024-04-30 08:47:52.927186605 +0000 UTC m=+0.669901901 container died dc48174aa38fb8e4238271b74fc7ba1cc373d9e587abc244f9240e60f495b243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240430) 2024-04-30T08:47:53.462 INFO:journalctl@ceph.osd.11.smithi189.stdout:Apr 30 08:47:53 smithi189 podman[69462]: 2024-04-30 08:47:53.444344869 +0000 UTC m=+0.508329270 container remove dc48174aa38fb8e4238271b74fc7ba1cc373d9e587abc244f9240e60f495b243 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-30T08:47:53.467 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@osd.11.service' 2024-04-30T08:47:53.504 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-30T08:47:53.504 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-30T08:47:53.504 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-30T08:47:53.504 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.a 2024-04-30T08:47:53.941 INFO:journalctl@ceph.mds.a.smithi032.stdout:Apr 30 08:47:53 smithi032 systemd[1]: Stopping Ceph mds.a for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:47:54.441 INFO:journalctl@ceph.mds.a.smithi032.stdout:Apr 30 08:47:54 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-a[70456]: 2024-04-30T08:47:54.113+0000 7fa27d600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-30T08:47:54.442 INFO:journalctl@ceph.mds.a.smithi032.stdout:Apr 30 08:47:54 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-a[70456]: 2024-04-30T08:47:54.113+0000 7fa27d600700 -1 mds.a *** got signal Terminated *** 2024-04-30T08:47:58.442 INFO:journalctl@ceph.mds.a.smithi032.stdout:Apr 30 08:47:58 smithi032 podman[95670]: 2024-04-30 08:47:58.151724892 +0000 UTC m=+4.157185792 container stop 5234a090b9bb0665b92884a9b6a46f6b6c14473dabc3087360376d433fc12e69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-a, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-30T08:47:58.442 INFO:journalctl@ceph.mds.a.smithi032.stdout:Apr 30 08:47:58 smithi032 podman[95670]: 2024-04-30 08:47:58.17809025 +0000 UTC m=+4.183551127 container died 5234a090b9bb0665b92884a9b6a46f6b6c14473dabc3087360376d433fc12e69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-a, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-30T08:47:58.825 INFO:journalctl@ceph.mds.a.smithi032.stdout:Apr 30 08:47:58 smithi032 podman[95670]: 2024-04-30 08:47:58.622782997 +0000 UTC m=+4.628243927 container cleanup 5234a090b9bb0665b92884a9b6a46f6b6c14473dabc3087360376d433fc12e69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-a, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-30T08:47:58.825 INFO:journalctl@ceph.mds.a.smithi032.stdout:Apr 30 08:47:58 smithi032 bash[95670]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-a 2024-04-30T08:47:59.191 INFO:journalctl@ceph.mds.a.smithi032.stdout:Apr 30 08:47:58 smithi032 podman[95685]: 2024-04-30 08:47:58.823666361 +0000 UTC m=+0.646467966 container remove 5234a090b9bb0665b92884a9b6a46f6b6c14473dabc3087360376d433fc12e69 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-a, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux ) 2024-04-30T08:47:59.341 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.a.service' 2024-04-30T08:47:59.462 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-30T08:47:59.463 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-30T08:47:59.463 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-30T08:47:59.463 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.d 2024-04-30T08:47:59.941 INFO:journalctl@ceph.mds.d.smithi032.stdout:Apr 30 08:47:59 smithi032 systemd[1]: Stopping Ceph mds.d for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:48:00.441 INFO:journalctl@ceph.mds.d.smithi032.stdout:Apr 30 08:48:00 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-d[69485]: 2024-04-30T08:48:00.159+0000 7f3778000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-30T08:48:00.442 INFO:journalctl@ceph.mds.d.smithi032.stdout:Apr 30 08:48:00 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-d[69485]: 2024-04-30T08:48:00.159+0000 7f3778000700 -1 mds.d *** got signal Terminated *** 2024-04-30T08:48:04.566 INFO:journalctl@ceph.mds.d.smithi032.stdout:Apr 30 08:48:04 smithi032 podman[95791]: 2024-04-30 08:48:04.190470455 +0000 UTC m=+4.201703626 container died c92cfdfac4b557a78dc31f4ad7d4dc967a4260d11c34d178ea2b33f367a61fdd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-d, ceph=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-30T08:48:04.928 INFO:journalctl@ceph.mds.d.smithi032.stdout:Apr 30 08:48:04 smithi032 podman[95791]: 2024-04-30 08:48:04.692603863 +0000 UTC m=+4.703837032 container cleanup c92cfdfac4b557a78dc31f4ad7d4dc967a4260d11c34d178ea2b33f367a61fdd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-d, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-30T08:48:04.928 INFO:journalctl@ceph.mds.d.smithi032.stdout:Apr 30 08:48:04 smithi032 bash[95791]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-d 2024-04-30T08:48:05.191 INFO:journalctl@ceph.mds.d.smithi032.stdout:Apr 30 08:48:04 smithi032 podman[95805]: 2024-04-30 08:48:04.926908256 +0000 UTC m=+0.724159660 container remove c92cfdfac4b557a78dc31f4ad7d4dc967a4260d11c34d178ea2b33f367a61fdd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-30T08:48:05.429 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.d.service' 2024-04-30T08:48:05.532 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-30T08:48:05.532 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-30T08:48:05.532 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-30T08:48:05.533 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.g 2024-04-30T08:48:05.691 INFO:journalctl@ceph.mds.g.smithi032.stdout:Apr 30 08:48:05 smithi032 systemd[1]: Stopping Ceph mds.g for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:48:06.441 INFO:journalctl@ceph.mds.g.smithi032.stdout:Apr 30 08:48:06 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-g[70109]: 2024-04-30T08:48:06.095+0000 7fa609800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-30T08:48:06.442 INFO:journalctl@ceph.mds.g.smithi032.stdout:Apr 30 08:48:06 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-g[70109]: 2024-04-30T08:48:06.095+0000 7fa609800700 -1 mds.g *** got signal Terminated *** 2024-04-30T08:48:09.076 INFO:journalctl@ceph.mds.g.smithi032.stdout:Apr 30 08:48:08 smithi032 podman[95913]: 2024-04-30 08:48:08.786220285 +0000 UTC m=+2.829193321 container died 42dc38ed484fb3750e5dddc1f73258edbee51de89147ecec3dced15d223e9684 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-g, CEPH_POINT_RELEASE=, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True) 2024-04-30T08:48:09.441 INFO:journalctl@ceph.mds.g.smithi032.stdout:Apr 30 08:48:09 smithi032 podman[95913]: 2024-04-30 08:48:09.204066645 +0000 UTC m=+3.247039694 container cleanup 42dc38ed484fb3750e5dddc1f73258edbee51de89147ecec3dced15d223e9684 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-g, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240430, ceph=True, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-30T08:48:09.441 INFO:journalctl@ceph.mds.g.smithi032.stdout:Apr 30 08:48:09 smithi032 bash[95913]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-g 2024-04-30T08:48:09.941 INFO:journalctl@ceph.mds.g.smithi032.stdout:Apr 30 08:48:09 smithi032 podman[95927]: 2024-04-30 08:48:09.580017023 +0000 UTC m=+0.776764635 container remove 42dc38ed484fb3750e5dddc1f73258edbee51de89147ecec3dced15d223e9684 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-g, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240430) 2024-04-30T08:48:10.092 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.g.service' 2024-04-30T08:48:10.205 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-30T08:48:10.205 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-30T08:48:10.206 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-30T08:48:10.206 DEBUG:teuthology.orchestra.run.smithi032:> sudo systemctl stop ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.j 2024-04-30T08:48:10.691 INFO:journalctl@ceph.mds.j.smithi032.stdout:Apr 30 08:48:10 smithi032 systemd[1]: Stopping Ceph mds.j for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:48:11.191 INFO:journalctl@ceph.mds.j.smithi032.stdout:Apr 30 08:48:10 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-j[69120]: 2024-04-30T08:48:10.763+0000 7f3a0b800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-30T08:48:11.191 INFO:journalctl@ceph.mds.j.smithi032.stdout:Apr 30 08:48:10 smithi032 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-j[69120]: 2024-04-30T08:48:10.763+0000 7f3a0b800700 -1 mds.j *** got signal Terminated *** 2024-04-30T08:48:14.779 INFO:journalctl@ceph.mds.j.smithi032.stdout:Apr 30 08:48:14 smithi032 podman[96033]: 2024-04-30 08:48:14.46288315 +0000 UTC m=+3.826291496 container died e7e3c5f5b7232d911f86850fce0d024f4761ecadefa013cf26d5c1df4f7f1ffd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-j, ceph=True, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-30T08:48:15.159 INFO:journalctl@ceph.mds.j.smithi032.stdout:Apr 30 08:48:14 smithi032 podman[96033]: 2024-04-30 08:48:14.8991003 +0000 UTC m=+4.262508643 container cleanup e7e3c5f5b7232d911f86850fce0d024f4761ecadefa013cf26d5c1df4f7f1ffd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-j, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b) 2024-04-30T08:48:15.159 INFO:journalctl@ceph.mds.j.smithi032.stdout:Apr 30 08:48:14 smithi032 bash[96033]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-j 2024-04-30T08:48:15.441 INFO:journalctl@ceph.mds.j.smithi032.stdout:Apr 30 08:48:15 smithi032 podman[96047]: 2024-04-30 08:48:15.158284315 +0000 UTC m=+0.670863338 container remove e7e3c5f5b7232d911f86850fce0d024f4761ecadefa013cf26d5c1df4f7f1ffd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-j, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-30T08:48:15.701 DEBUG:teuthology.orchestra.run.smithi032:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.j.service' 2024-04-30T08:48:15.811 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-30T08:48:15.811 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-30T08:48:15.811 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-30T08:48:15.812 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.b 2024-04-30T08:48:16.128 INFO:journalctl@ceph.mds.b.smithi071.stdout:Apr 30 08:48:15 smithi071 systemd[1]: Stopping Ceph mds.b for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:48:16.503 INFO:journalctl@ceph.mds.b.smithi071.stdout:Apr 30 08:48:16 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-b[64026]: 2024-04-30T08:48:16.126+0000 7efc39be6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-30T08:48:16.504 INFO:journalctl@ceph.mds.b.smithi071.stdout:Apr 30 08:48:16 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-b[64026]: 2024-04-30T08:48:16.126+0000 7efc39be6700 -1 mds.b *** got signal Terminated *** 2024-04-30T08:48:20.427 INFO:journalctl@ceph.mds.b.smithi071.stdout:Apr 30 08:48:20 smithi071 podman[69827]: 2024-04-30 08:48:20.141551901 +0000 UTC m=+4.132865427 container died 562a91b76601bfa2f3e437103889bab3af607fe26861f7b526e61f2dd21912f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-b, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, ceph=True, org.label-schema.build-date=20240430, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-30T08:48:20.753 INFO:journalctl@ceph.mds.b.smithi071.stdout:Apr 30 08:48:20 smithi071 podman[69827]: 2024-04-30 08:48:20.534589549 +0000 UTC m=+4.525903071 container cleanup 562a91b76601bfa2f3e437103889bab3af607fe26861f7b526e61f2dd21912f9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.license=GPLv2, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-30T08:48:20.754 INFO:journalctl@ceph.mds.b.smithi071.stdout:Apr 30 08:48:20 smithi071 bash[69827]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-b 2024-04-30T08:48:20.915 DEBUG:teuthology.orchestra.run.smithi071:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.b.service' 2024-04-30T08:48:20.951 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-30T08:48:20.951 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-30T08:48:20.951 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-30T08:48:20.951 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.e 2024-04-30T08:48:21.253 INFO:journalctl@ceph.mds.e.smithi071.stdout:Apr 30 08:48:21 smithi071 systemd[1]: Stopping Ceph mds.e for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:48:21.753 INFO:journalctl@ceph.mds.e.smithi071.stdout:Apr 30 08:48:21 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-e[63391]: 2024-04-30T08:48:21.254+0000 7fa5feb51700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-30T08:48:21.753 INFO:journalctl@ceph.mds.e.smithi071.stdout:Apr 30 08:48:21 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-e[63391]: 2024-04-30T08:48:21.254+0000 7fa5feb51700 -1 mds.e *** got signal Terminated *** 2024-04-30T08:48:25.004 INFO:journalctl@ceph.mds.e.smithi071.stdout:Apr 30 08:48:24 smithi071 podman[69959]: 2024-04-30 08:48:24.661626133 +0000 UTC m=+3.500101115 container died fe42b55cc699347dd25c04c28bca38b032bc11133afc52fd740a5df408232123 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240430, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-30T08:48:25.497 INFO:journalctl@ceph.mds.e.smithi071.stdout:Apr 30 08:48:25 smithi071 podman[69959]: 2024-04-30 08:48:25.304391854 +0000 UTC m=+4.142866843 container cleanup fe42b55cc699347dd25c04c28bca38b032bc11133afc52fd740a5df408232123 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-e, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-04-30T08:48:25.498 INFO:journalctl@ceph.mds.e.smithi071.stdout:Apr 30 08:48:25 smithi071 bash[69959]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-e 2024-04-30T08:48:25.725 DEBUG:teuthology.orchestra.run.smithi071:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.e.service' 2024-04-30T08:48:25.753 INFO:journalctl@ceph.mds.e.smithi071.stdout:Apr 30 08:48:25 smithi071 podman[69973]: 2024-04-30 08:48:25.497275904 +0000 UTC m=+0.827198774 container remove fe42b55cc699347dd25c04c28bca38b032bc11133afc52fd740a5df408232123 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-e, org.label-schema.build-date=20240430, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-04-30T08:48:25.753 INFO:journalctl@ceph.mds.e.smithi071.stdout:Apr 30 08:48:25 smithi071 systemd[1]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.e.service: Deactivated successfully. 2024-04-30T08:48:25.753 INFO:journalctl@ceph.mds.e.smithi071.stdout:Apr 30 08:48:25 smithi071 systemd[1]: Stopped Ceph mds.e for 98c0a670-06ca-11ef-bc93-c7b262605968. 2024-04-30T08:48:25.754 INFO:journalctl@ceph.mds.e.smithi071.stdout:Apr 30 08:48:25 smithi071 systemd[1]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.e.service: Consumed 4.192s CPU time. 2024-04-30T08:48:25.757 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-30T08:48:25.757 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-30T08:48:25.757 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-30T08:48:25.757 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.h 2024-04-30T08:48:26.075 INFO:journalctl@ceph.mds.h.smithi071.stdout:Apr 30 08:48:25 smithi071 systemd[1]: Stopping Ceph mds.h for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:48:26.503 INFO:journalctl@ceph.mds.h.smithi071.stdout:Apr 30 08:48:26 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-h[63716]: 2024-04-30T08:48:26.074+0000 7f6db4cf5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-30T08:48:26.504 INFO:journalctl@ceph.mds.h.smithi071.stdout:Apr 30 08:48:26 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-h[63716]: 2024-04-30T08:48:26.074+0000 7f6db4cf5700 -1 mds.h *** got signal Terminated *** 2024-04-30T08:48:28.942 INFO:journalctl@ceph.mds.h.smithi071.stdout:Apr 30 08:48:28 smithi071 podman[70079]: 2024-04-30 08:48:28.680904837 +0000 UTC m=+2.708463821 container died 478762dd0b2a628b4afdca4f410480b202edbbdb814a91530c2e5b349ffcc9e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-h, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.vendor=CentOS) 2024-04-30T08:48:29.254 INFO:journalctl@ceph.mds.h.smithi071.stdout:Apr 30 08:48:29 smithi071 podman[70079]: 2024-04-30 08:48:29.04122578 +0000 UTC m=+3.068784763 container cleanup 478762dd0b2a628b4afdca4f410480b202edbbdb814a91530c2e5b349ffcc9e2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-h, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240430, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, ceph=True, org.label-schema.schema-version=1.0) 2024-04-30T08:48:29.254 INFO:journalctl@ceph.mds.h.smithi071.stdout:Apr 30 08:48:29 smithi071 bash[70079]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-h 2024-04-30T08:48:29.472 DEBUG:teuthology.orchestra.run.smithi071:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.h.service' 2024-04-30T08:48:29.502 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-30T08:48:29.502 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-30T08:48:29.503 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-30T08:48:29.503 DEBUG:teuthology.orchestra.run.smithi071:> sudo systemctl stop ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.k 2024-04-30T08:48:29.753 INFO:journalctl@ceph.mds.k.smithi071.stdout:Apr 30 08:48:29 smithi071 systemd[1]: Stopping Ceph mds.k for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:48:30.254 INFO:journalctl@ceph.mds.k.smithi071.stdout:Apr 30 08:48:29 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-k[63071]: 2024-04-30T08:48:29.827+0000 7f6264a3f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-30T08:48:30.254 INFO:journalctl@ceph.mds.k.smithi071.stdout:Apr 30 08:48:29 smithi071 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-k[63071]: 2024-04-30T08:48:29.827+0000 7f6264a3f700 -1 mds.k *** got signal Terminated *** 2024-04-30T08:48:32.995 INFO:journalctl@ceph.mds.k.smithi071.stdout:Apr 30 08:48:32 smithi071 podman[70199]: 2024-04-30 08:48:32.675728242 +0000 UTC m=+2.957325362 container died 188e029109ea2f6d81f5b7d365df309d4206f8a762dcc38b7ef529d0aa0869ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-k, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430) 2024-04-30T08:48:33.253 INFO:journalctl@ceph.mds.k.smithi071.stdout:Apr 30 08:48:33 smithi071 podman[70199]: 2024-04-30 08:48:33.069500028 +0000 UTC m=+3.351097154 container cleanup 188e029109ea2f6d81f5b7d365df309d4206f8a762dcc38b7ef529d0aa0869ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-30T08:48:33.254 INFO:journalctl@ceph.mds.k.smithi071.stdout:Apr 30 08:48:33 smithi071 bash[70199]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-k 2024-04-30T08:48:33.450 DEBUG:teuthology.orchestra.run.smithi071:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.k.service' 2024-04-30T08:48:33.480 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-30T08:48:33.480 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-30T08:48:33.480 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-30T08:48:33.480 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.c 2024-04-30T08:48:33.784 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 30 08:48:33 smithi189 systemd[1]: Stopping Ceph mds.c for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:48:34.049 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 30 08:48:33 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-c[63536]: 2024-04-30T08:48:33.783+0000 7fab78266700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-30T08:48:34.049 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 30 08:48:33 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-c[63536]: 2024-04-30T08:48:33.783+0000 7fab78266700 -1 mds.c *** got signal Terminated *** 2024-04-30T08:48:37.050 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 30 08:48:36 smithi189 podman[69540]: 2024-04-30 08:48:36.66401214 +0000 UTC m=+2.984042458 container died 2d6e1472fdad879e1c1cde2e4ce8871cb533c0b2987b0d65b0a716345ded38f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-c, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-30T08:48:37.376 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 30 08:48:37 smithi189 podman[69540]: 2024-04-30 08:48:37.192035656 +0000 UTC m=+3.512066018 container cleanup 2d6e1472fdad879e1c1cde2e4ce8871cb533c0b2987b0d65b0a716345ded38f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-c, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.build-date=20240430, io.buildah.version=1.33.5) 2024-04-30T08:48:37.376 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 30 08:48:37 smithi189 bash[69540]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-c 2024-04-30T08:48:37.610 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.c.service' 2024-04-30T08:48:37.631 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 30 08:48:37 smithi189 podman[69566]: 2024-04-30 08:48:37.376331336 +0000 UTC m=+0.712277853 container remove 2d6e1472fdad879e1c1cde2e4ce8871cb533c0b2987b0d65b0a716345ded38f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-c, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5) 2024-04-30T08:48:37.631 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 30 08:48:37 smithi189 systemd[1]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.c.service: Deactivated successfully. 2024-04-30T08:48:37.631 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 30 08:48:37 smithi189 systemd[1]: Stopped Ceph mds.c for 98c0a670-06ca-11ef-bc93-c7b262605968. 2024-04-30T08:48:37.631 INFO:journalctl@ceph.mds.c.smithi189.stdout:Apr 30 08:48:37 smithi189 systemd[1]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.c.service: Consumed 2.151s CPU time. 2024-04-30T08:48:37.645 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-30T08:48:37.645 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-30T08:48:37.645 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-30T08:48:37.645 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.f 2024-04-30T08:48:37.955 INFO:journalctl@ceph.mds.f.smithi189.stdout:Apr 30 08:48:37 smithi189 systemd[1]: Stopping Ceph mds.f for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:48:38.300 INFO:journalctl@ceph.mds.f.smithi189.stdout:Apr 30 08:48:37 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-f[63232]: 2024-04-30T08:48:37.953+0000 7fc998353700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-30T08:48:38.300 INFO:journalctl@ceph.mds.f.smithi189.stdout:Apr 30 08:48:37 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-f[63232]: 2024-04-30T08:48:37.953+0000 7fc998353700 -1 mds.f *** got signal Terminated *** 2024-04-30T08:48:40.922 INFO:journalctl@ceph.mds.f.smithi189.stdout:Apr 30 08:48:40 smithi189 podman[69672]: 2024-04-30 08:48:40.667615153 +0000 UTC m=+2.807678140 container died c9190cc367e4b7a614cbd637841661c0dac2e8e3e0a677d1b1608ea031099663 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240430, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d) 2024-04-30T08:48:41.280 INFO:journalctl@ceph.mds.f.smithi189.stdout:Apr 30 08:48:41 smithi189 podman[69672]: 2024-04-30 08:48:41.012029996 +0000 UTC m=+3.152092991 container cleanup c9190cc367e4b7a614cbd637841661c0dac2e8e3e0a677d1b1608ea031099663 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-f, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240430, org.label-schema.schema-version=1.0, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-04-30T08:48:41.280 INFO:journalctl@ceph.mds.f.smithi189.stdout:Apr 30 08:48:41 smithi189 bash[69672]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-f 2024-04-30T08:48:41.428 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.f.service' 2024-04-30T08:48:41.460 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-30T08:48:41.461 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-30T08:48:41.461 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-30T08:48:41.461 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.i 2024-04-30T08:48:41.800 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 30 08:48:41 smithi189 systemd[1]: Stopping Ceph mds.i for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:48:41.801 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 30 08:48:41 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-i[63845]: 2024-04-30T08:48:41.772+0000 7f9a1c8f7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-30T08:48:41.801 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 30 08:48:41 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-i[63845]: 2024-04-30T08:48:41.772+0000 7f9a1c8f7700 -1 mds.i *** got signal Terminated *** 2024-04-30T08:48:44.800 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 30 08:48:44 smithi189 podman[69791]: 2024-04-30 08:48:44.646129318 +0000 UTC m=+2.972917303 container died 7800c09b89351ddddbc70e5fe35e0f433ca1aa3030af4abe43d07492b371ecf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-i, org.label-schema.build-date=20240430, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=) 2024-04-30T08:48:45.300 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 30 08:48:44 smithi189 podman[69791]: 2024-04-30 08:48:44.990265729 +0000 UTC m=+3.317053706 container cleanup 7800c09b89351ddddbc70e5fe35e0f433ca1aa3030af4abe43d07492b371ecf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-i, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240430, GIT_BRANCH=HEAD, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-30T08:48:45.300 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 30 08:48:44 smithi189 bash[69791]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-i 2024-04-30T08:48:45.300 INFO:journalctl@ceph.mds.i.smithi189.stdout:Apr 30 08:48:45 smithi189 podman[69805]: 2024-04-30 08:48:45.166321047 +0000 UTC m=+0.510536592 container remove 7800c09b89351ddddbc70e5fe35e0f433ca1aa3030af4abe43d07492b371ecf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-i, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.build-date=20240430, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-30T08:48:45.410 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.i.service' 2024-04-30T08:48:45.440 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-30T08:48:45.440 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-30T08:48:45.441 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-30T08:48:45.441 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.l 2024-04-30T08:48:45.753 INFO:journalctl@ceph.mds.l.smithi189.stdout:Apr 30 08:48:45 smithi189 systemd[1]: Stopping Ceph mds.l for 98c0a670-06ca-11ef-bc93-c7b262605968... 2024-04-30T08:48:46.050 INFO:journalctl@ceph.mds.l.smithi189.stdout:Apr 30 08:48:45 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-l[62922]: 2024-04-30T08:48:45.751+0000 7f80d84fd700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-30T08:48:46.050 INFO:journalctl@ceph.mds.l.smithi189.stdout:Apr 30 08:48:45 smithi189 ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-l[62922]: 2024-04-30T08:48:45.751+0000 7f80d84fd700 -1 mds.l *** got signal Terminated *** 2024-04-30T08:48:48.936 INFO:journalctl@ceph.mds.l.smithi189.stdout:Apr 30 08:48:48 smithi189 podman[69913]: 2024-04-30 08:48:48.65287911 +0000 UTC m=+3.000267602 container died 40bea0c1edd36b00c9309a5545600bb0f8cb3db9dbd178b1a479f45274fe2d22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-l, io.buildah.version=1.33.5, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240430, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-04-30T08:48:49.294 INFO:journalctl@ceph.mds.l.smithi189.stdout:Apr 30 08:48:49 smithi189 podman[69913]: 2024-04-30 08:48:49.018505423 +0000 UTC m=+3.365893907 container cleanup 40bea0c1edd36b00c9309a5545600bb0f8cb3db9dbd178b1a479f45274fe2d22 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:ce4c3a10333d5c81164e1786fc61a2c110567e8ef5432867b15425bd8b6d69de, name=ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240430, io.buildah.version=1.33.5, GIT_COMMIT=318fe5928e744a8c3fe408a2f9ccaca3c31b552b, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-pdonnell-testing-20240429.210911-debug-8feaf7d, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-30T08:48:49.295 INFO:journalctl@ceph.mds.l.smithi189.stdout:Apr 30 08:48:49 smithi189 bash[69913]: ceph-98c0a670-06ca-11ef-bc93-c7b262605968-mds-l 2024-04-30T08:48:49.454 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-98c0a670-06ca-11ef-bc93-c7b262605968@mds.l.service' 2024-04-30T08:48:49.484 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-30T08:48:49.485 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-30T08:48:49.485 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-30T08:48:49.813 INFO:teuthology.orchestra.run.smithi032.stdout:Deleting cluster with fsid: 98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:48:55.246 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-30T08:48:55.438 INFO:teuthology.orchestra.run.smithi071.stdout:Deleting cluster with fsid: 98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:48:57.805 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-30T08:48:57.997 INFO:teuthology.orchestra.run.smithi189.stdout:Deleting cluster with fsid: 98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T08:49:00.372 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-30T08:49:00.468 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-30T08:49:00.499 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-30T08:49:00.524 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-30T08:49:00.527 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/pdonnell-2024-04-30_05:04:19-fs-wip-pdonnell-testing-20240429.210911-debug-distro-default-smithi/7680601/remote/smithi032/crash 2024-04-30T08:49:00.528 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/crash -- . 2024-04-30T08:49:00.630 INFO:teuthology.orchestra.run.smithi032.stderr:tar: /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-30T08:49:00.631 INFO:teuthology.orchestra.run.smithi032.stderr:tar: Error is not recoverable: exiting now 2024-04-30T08:49:00.635 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/pdonnell-2024-04-30_05:04:19-fs-wip-pdonnell-testing-20240429.210911-debug-distro-default-smithi/7680601/remote/smithi071/crash 2024-04-30T08:49:00.636 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/crash -- . 2024-04-30T08:49:00.676 INFO:teuthology.orchestra.run.smithi071.stderr:tar: /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-30T08:49:00.676 INFO:teuthology.orchestra.run.smithi071.stderr:tar: Error is not recoverable: exiting now 2024-04-30T08:49:00.678 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/pdonnell-2024-04-30_05:04:19-fs-wip-pdonnell-testing-20240429.210911-debug-distro-default-smithi/7680601/remote/smithi189/crash 2024-04-30T08:49:00.679 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/crash -- . 2024-04-30T08:49:00.714 INFO:teuthology.orchestra.run.smithi189.stderr:tar: /var/lib/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-30T08:49:00.714 INFO:teuthology.orchestra.run.smithi189.stderr:tar: Error is not recoverable: exiting now 2024-04-30T08:49:00.715 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-30T08:49:00.716 DEBUG:teuthology.orchestra.run.smithi032:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-30T08:49:00.886 INFO:tasks.cephadm:Compressing logs... 2024-04-30T08:49:00.886 DEBUG:teuthology.orchestra.run.smithi032:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-30T08:49:00.892 DEBUG:teuthology.orchestra.run.smithi071:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-30T08:49:00.894 DEBUG:teuthology.orchestra.run.smithi189:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-30T08:49:00.921 INFO:teuthology.orchestra.run.smithi189.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-30T08:49:00.922 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-30T08:49:00.922 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph.log 2024-04-30T08:49:00.924 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-30T08:49:00.924 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-30T08:49:00.924 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-30T08:49:00.924 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-30T08:49:00.924 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-30T08:49:00.924 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-30T08:49:00.924 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-30T08:49:00.925 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-30T08:49:00.925 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mon.c.log: /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-30T08:49:00.926 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-30T08:49:00.926 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-30T08:49:00.926 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-30T08:49:00.926 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-30T08:49:00.931 INFO:teuthology.orchestra.run.smithi071.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-30T08:49:00.932 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-30T08:49:00.932 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph.log 2024-04-30T08:49:00.933 INFO:teuthology.orchestra.run.smithi071.stderr:gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-30T08:49:00.934 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-30T08:49:00.934 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-30T08:49:00.934 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-30T08:49:00.934 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-30T08:49:00.937 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-30T08:49:00.937 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-30T08:49:00.937 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-30T08:49:00.937 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-30T08:49:00.937 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-30T08:49:00.937 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-30T08:49:00.937 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-30T08:49:00.937 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-30T08:49:00.969 INFO:teuthology.orchestra.run.smithi032.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-30T08:49:00.969 INFO:teuthology.orchestra.run.smithi032.stderr:: No such file or directory 2024-04-30T08:49:00.978 INFO:teuthology.orchestra.run.smithi032.stderr:gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph.log 2024-04-30T08:49:00.979 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-30T08:49:00.980 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-30T08:49:00.981 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-30T08:49:00.982 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-30T08:49:00.983 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-30T08:49:00.984 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-30T08:49:00.985 INFO:teuthology.orchestra.run.smithi032.stderr: 88.2% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-30T08:49:00.985 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-30T08:49:00.986 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-30T08:49:00.987 INFO:teuthology.orchestra.run.smithi032.stderr: 94.6% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-30T08:49:00.987 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-30T08:49:00.988 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-30T08:49:00.989 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-30T08:49:00.990 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-30T08:49:00.991 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-30T08:49:00.992 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-04-30T08:49:00.994 INFO:teuthology.orchestra.run.smithi032.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.9.log: /var/log/ceph/ceph-client.admin.log: 94.5% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-30T08:49:01.061 INFO:teuthology.orchestra.run.smithi032.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-04-30T08:49:01.066 INFO:teuthology.orchestra.run.smithi071.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mon.b.log: /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.k.log: /var/log/ceph/cephadm.log: 92.9% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-30T08:49:01.070 INFO:teuthology.orchestra.run.smithi032.stderr: 90.1% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-30T08:49:01.105 INFO:teuthology.orchestra.run.smithi071.stderr: 82.0% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-30T08:49:01.115 INFO:teuthology.orchestra.run.smithi071.stderr: 88.2% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-30T08:49:01.119 INFO:teuthology.orchestra.run.smithi032.stderr: 82.4% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-30T08:49:01.134 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.8.log: /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-volume.log: /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.l.log: 82.0% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-30T08:49:01.134 INFO:teuthology.orchestra.run.smithi189.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-30T08:49:01.147 INFO:teuthology.orchestra.run.smithi189.stderr: 93.0% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-30T08:49:01.169 INFO:teuthology.orchestra.run.smithi071.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-30T08:49:01.178 INFO:teuthology.orchestra.run.smithi071.stderr: 90.3% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-30T08:49:01.198 INFO:teuthology.orchestra.run.smithi189.stderr: 90.3% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-30T08:49:01.222 INFO:teuthology.orchestra.run.smithi032.stderr: 93.0% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-30T08:49:01.268 INFO:teuthology.orchestra.run.smithi032.stderr: 93.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-30T08:49:01.286 INFO:teuthology.orchestra.run.smithi189.stderr: 91.0% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-30T08:49:01.303 INFO:teuthology.orchestra.run.smithi071.stderr: 91.0% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-30T08:49:01.303 INFO:teuthology.orchestra.run.smithi032.stderr: 94.0% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-30T08:49:01.306 INFO:teuthology.orchestra.run.smithi032.stderr: 90.9% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-30T08:49:01.492 INFO:teuthology.orchestra.run.smithi189.stderr: 92.9% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-30T08:49:01.535 INFO:teuthology.orchestra.run.smithi189.stderr: 88.2% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-30T08:49:01.614 INFO:teuthology.orchestra.run.smithi071.stderr: 96.0% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-30T08:49:01.713 INFO:teuthology.orchestra.run.smithi189.stderr: 94.5% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-30T08:49:01.823 INFO:teuthology.orchestra.run.smithi071.stderr: 94.0% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-30T08:49:01.827 INFO:teuthology.orchestra.run.smithi189.stderr: 93.1% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-30T08:49:02.084 INFO:teuthology.orchestra.run.smithi032.stderr: 90.0% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-30T08:49:02.393 INFO:teuthology.orchestra.run.smithi071.stderr: 93.2% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-30T08:49:02.996 INFO:teuthology.orchestra.run.smithi189.stderr: 93.8% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-30T08:49:05.291 INFO:teuthology.orchestra.run.smithi189.stderr: 91.5% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-30T08:49:05.839 INFO:teuthology.orchestra.run.smithi071.stderr: 91.6% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-30T08:49:08.116 INFO:teuthology.orchestra.run.smithi032.stderr: 90.3% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-30T08:49:32.654 INFO:teuthology.orchestra.run.smithi071.stderr: 94.4% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-30T08:51:35.358 INFO:teuthology.orchestra.run.smithi071.stderr: 91.7% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-30T08:57:13.066 INFO:teuthology.orchestra.run.smithi071.stderr: 91.1% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-30T08:57:40.408 INFO:teuthology.orchestra.run.smithi071.stderr: 91.7% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-30T08:58:10.759 INFO:teuthology.orchestra.run.smithi071.stderr: 91.6% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-30T08:58:10.761 INFO:teuthology.orchestra.run.smithi071.stderr: 2024-04-30T08:58:10.761 INFO:teuthology.orchestra.run.smithi071.stderr:real 9m9.852s 2024-04-30T08:58:10.761 INFO:teuthology.orchestra.run.smithi071.stderr:user 6m42.945s 2024-04-30T08:58:10.761 INFO:teuthology.orchestra.run.smithi071.stderr:sys 0m15.589s 2024-04-30T09:00:12.733 INFO:teuthology.orchestra.run.smithi189.stderr: 91.5% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-30T09:03:26.023 INFO:teuthology.orchestra.run.smithi189.stderr: 91.5% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-30T09:03:55.058 INFO:teuthology.orchestra.run.smithi189.stderr: 91.7% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-30T09:05:07.274 INFO:teuthology.orchestra.run.smithi189.stderr: 91.3% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-30T09:05:07.276 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-04-30T09:05:07.276 INFO:teuthology.orchestra.run.smithi189.stderr:real 16m6.367s 2024-04-30T09:05:07.276 INFO:teuthology.orchestra.run.smithi189.stderr:user 7m12.922s 2024-04-30T09:05:07.276 INFO:teuthology.orchestra.run.smithi189.stderr:sys 0m18.266s 2024-04-30T09:05:29.820 INFO:teuthology.orchestra.run.smithi032.stderr: 91.6% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-30T09:06:51.500 INFO:teuthology.orchestra.run.smithi032.stderr: 91.4% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-30T09:07:01.457 INFO:teuthology.orchestra.run.smithi032.stderr: 91.6% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-30T09:08:03.095 INFO:teuthology.orchestra.run.smithi032.stderr: 90.6% -- replaced with /var/log/ceph/98c0a670-06ca-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-30T09:08:03.098 INFO:teuthology.orchestra.run.smithi032.stderr: 2024-04-30T09:08:03.098 INFO:teuthology.orchestra.run.smithi032.stderr:real 19m2.169s 2024-04-30T09:08:03.098 INFO:teuthology.orchestra.run.smithi032.stderr:user 22m50.058s 2024-04-30T09:08:03.099 INFO:teuthology.orchestra.run.smithi032.stderr:sys 2m45.018s 2024-04-30T09:08:03.100 INFO:tasks.cephadm:Archiving logs... 2024-04-30T09:08:03.101 DEBUG:teuthology.misc:Transferring archived files from smithi032:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-30_05:04:19-fs-wip-pdonnell-testing-20240429.210911-debug-distro-default-smithi/7680601/remote/smithi032/log 2024-04-30T09:08:03.102 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-30T09:09:33.835 DEBUG:teuthology.misc:Transferring archived files from smithi071:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-30_05:04:19-fs-wip-pdonnell-testing-20240429.210911-debug-distro-default-smithi/7680601/remote/smithi071/log 2024-04-30T09:09:33.836 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-30T09:10:44.999 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-04-30_05:04:19-fs-wip-pdonnell-testing-20240429.210911-debug-distro-default-smithi/7680601/remote/smithi189/log 2024-04-30T09:10:45.000 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-30T09:12:00.950 INFO:tasks.cephadm:Removing cluster... 2024-04-30T09:12:00.950 DEBUG:teuthology.orchestra.run.smithi032:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 --force 2024-04-30T09:12:01.842 INFO:teuthology.orchestra.run.smithi032.stdout:Deleting cluster with fsid: 98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T09:12:04.050 DEBUG:teuthology.orchestra.run.smithi071:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 --force 2024-04-30T09:12:04.535 INFO:teuthology.orchestra.run.smithi071.stdout:Deleting cluster with fsid: 98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T09:12:05.255 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 98c0a670-06ca-11ef-bc93-c7b262605968 --force 2024-04-30T09:12:05.518 INFO:teuthology.orchestra.run.smithi189.stdout:Deleting cluster with fsid: 98c0a670-06ca-11ef-bc93-c7b262605968 2024-04-30T09:12:06.269 INFO:tasks.cephadm:Removing cephadm ... 2024-04-30T09:12:06.270 DEBUG:teuthology.orchestra.run.smithi032:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-30T09:12:06.338 DEBUG:teuthology.orchestra.run.smithi071:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-30T09:12:06.357 DEBUG:teuthology.orchestra.run.smithi189:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-30T09:12:06.373 INFO:tasks.cephadm:Teardown complete 2024-04-30T09:12:06.374 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-30T09:12:06.387 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-30T09:12:06.387 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-30T09:12:06.395 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-30T09:12:06.400 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-30T09:12:06.670 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-30T09:12:06.682 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-30T09:12:06.682 DEBUG:teuthology.orchestra.run.smithi032:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-30T09:12:06.691 DEBUG:teuthology.orchestra.run.smithi071:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-30T09:12:06.695 DEBUG:teuthology.orchestra.run.smithi189:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-30T09:12:06.708 INFO:teuthology.orchestra.run.smithi189.stderr:bash: line 1: ntpq: command not found 2024-04-30T09:12:06.709 INFO:teuthology.orchestra.run.smithi071.stderr:bash: line 1: ntpq: command not found 2024-04-30T09:12:06.727 INFO:teuthology.orchestra.run.smithi189.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-30T09:12:06.727 INFO:teuthology.orchestra.run.smithi189.stdout:=============================================================================== 2024-04-30T09:12:06.727 INFO:teuthology.orchestra.run.smithi189.stdout:^* hv01.front.sepia.ceph.com 3 6 377 44 +671us[ +635us] +/- 59ms 2024-04-30T09:12:06.727 INFO:teuthology.orchestra.run.smithi189.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 47 -408us[ -444us] +/- 71ms 2024-04-30T09:12:06.728 INFO:teuthology.orchestra.run.smithi189.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 49 -296us[ -332us] +/- 71ms 2024-04-30T09:12:06.728 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-30T09:12:06.739 INFO:teuthology.orchestra.run.smithi032.stderr:bash: line 1: ntpq: command not found 2024-04-30T09:12:06.741 INFO:teuthology.orchestra.run.smithi071.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-30T09:12:06.742 INFO:teuthology.orchestra.run.smithi071.stdout:=============================================================================== 2024-04-30T09:12:06.742 INFO:teuthology.orchestra.run.smithi071.stdout:^* hv01.front.sepia.ceph.com 3 6 377 48 +650us[ +659us] +/- 59ms 2024-04-30T09:12:06.742 INFO:teuthology.orchestra.run.smithi071.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 46 -436us[ -436us] +/- 71ms 2024-04-30T09:12:06.742 INFO:teuthology.orchestra.run.smithi071.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 47 -365us[ -365us] +/- 71ms 2024-04-30T09:12:06.742 INFO:teuthology.orchestra.run.smithi071.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-30T09:12:06.881 INFO:teuthology.orchestra.run.smithi032.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-30T09:12:06.881 INFO:teuthology.orchestra.run.smithi032.stdout:=============================================================================== 2024-04-30T09:12:06.881 INFO:teuthology.orchestra.run.smithi032.stdout:^* hv01.front.sepia.ceph.com 3 6 377 43 +667us[ +645us] +/- 59ms 2024-04-30T09:12:06.881 INFO:teuthology.orchestra.run.smithi032.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 112 -381us[ -368us] +/- 70ms 2024-04-30T09:12:06.881 INFO:teuthology.orchestra.run.smithi032.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 49 -359us[ -381us] +/- 71ms 2024-04-30T09:12:06.881 INFO:teuthology.orchestra.run.smithi032.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-30T09:12:06.883 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-30T09:12:06.894 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-30T09:12:06.895 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-30T09:12:06.905 DEBUG:teuthology.orchestra.run.smithi032:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-30T09:12:07.255 DEBUG:teuthology.orchestra.run.smithi071:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-30T09:12:07.352 DEBUG:teuthology.orchestra.run.smithi189:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-30T09:12:07.459 DEBUG:teuthology.orchestra.run.smithi032:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-30T09:12:07.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-30T09:12:07.559 DEBUG:teuthology.orchestra.run.smithi071:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-30T09:12:07.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-30T09:12:07.590 DEBUG:teuthology.orchestra.run.smithi189:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-30T09:12:07.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-30T09:12:07.619 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-30T09:12:07.631 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-30T09:12:07.646 INFO:teuthology.task.internal:Duration was 3563.335881 seconds 2024-04-30T09:12:07.646 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-30T09:12:07.657 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-30T09:12:07.657 DEBUG:teuthology.orchestra.run.smithi032:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-30T09:12:07.666 DEBUG:teuthology.orchestra.run.smithi071:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-30T09:12:07.669 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-30T09:12:07.744 INFO:teuthology.orchestra.run.smithi189.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-30T09:12:07.748 INFO:teuthology.orchestra.run.smithi071.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-30T09:12:07.816 INFO:teuthology.orchestra.run.smithi032.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-30T09:12:08.103 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-30T09:12:08.103 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi032.front.sepia.ceph.com 2024-04-30T09:12:08.103 DEBUG:teuthology.orchestra.run.smithi032:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-30T09:12:08.185 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi071.front.sepia.ceph.com 2024-04-30T09:12:08.185 DEBUG:teuthology.orchestra.run.smithi071:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-30T09:12:08.233 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi189.front.sepia.ceph.com 2024-04-30T09:12:08.234 DEBUG:teuthology.orchestra.run.smithi189:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-30T09:12:08.273 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-30T09:12:08.274 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-30T09:12:08.281 DEBUG:teuthology.orchestra.run.smithi071:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-30T09:12:08.284 DEBUG:teuthology.orchestra.run.smithi189:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-30T09:12:08.371 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-30T09:12:08.371 DEBUG:teuthology.orchestra.run.smithi032:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-30T09:12:08.377 DEBUG:teuthology.orchestra.run.smithi071:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-30T09:12:08.380 DEBUG:teuthology.orchestra.run.smithi189:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-30T09:12:08.924 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-30T09:12:08.936 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-30T09:12:08.936 DEBUG:teuthology.orchestra.run.smithi032:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-30T09:12:09.029 DEBUG:teuthology.orchestra.run.smithi071:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-30T09:12:09.070 DEBUG:teuthology.orchestra.run.smithi189:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-30T09:12:09.099 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-30T09:12:09.111 DEBUG:teuthology.orchestra.run.smithi032:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-30T09:12:09.119 DEBUG:teuthology.orchestra.run.smithi071:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-30T09:12:09.122 DEBUG:teuthology.orchestra.run.smithi189:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-30T09:12:09.152 INFO:teuthology.orchestra.run.smithi071.stdout:kernel.core_pattern = core 2024-04-30T09:12:09.183 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = core 2024-04-30T09:12:09.262 INFO:teuthology.orchestra.run.smithi032.stdout:kernel.core_pattern = core 2024-04-30T09:12:09.313 DEBUG:teuthology.orchestra.run.smithi032:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-30T09:12:09.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-30T09:12:09.365 DEBUG:teuthology.orchestra.run.smithi071:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-30T09:12:09.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-30T09:12:09.381 DEBUG:teuthology.orchestra.run.smithi189:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-30T09:12:09.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-30T09:12:09.397 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-30T09:12:09.409 INFO:teuthology.task.internal:Transferring archived files... 2024-04-30T09:12:09.411 DEBUG:teuthology.misc:Transferring archived files from smithi032:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-30_05:04:19-fs-wip-pdonnell-testing-20240429.210911-debug-distro-default-smithi/7680601/remote/smithi032 2024-04-30T09:12:09.411 DEBUG:teuthology.orchestra.run.smithi032:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-30T09:12:09.518 DEBUG:teuthology.misc:Transferring archived files from smithi071:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-30_05:04:19-fs-wip-pdonnell-testing-20240429.210911-debug-distro-default-smithi/7680601/remote/smithi071 2024-04-30T09:12:09.518 DEBUG:teuthology.orchestra.run.smithi071:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-30T09:12:09.557 DEBUG:teuthology.misc:Transferring archived files from smithi189:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-04-30_05:04:19-fs-wip-pdonnell-testing-20240429.210911-debug-distro-default-smithi/7680601/remote/smithi189 2024-04-30T09:12:09.558 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-30T09:12:09.597 INFO:teuthology.task.internal:Removing archive directory... 2024-04-30T09:12:09.597 DEBUG:teuthology.orchestra.run.smithi032:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-30T09:12:09.605 DEBUG:teuthology.orchestra.run.smithi071:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-30T09:12:09.608 DEBUG:teuthology.orchestra.run.smithi189:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-30T09:12:09.660 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-30T09:12:09.674 INFO:teuthology.task.internal:Not uploading archives. 2024-04-30T09:12:09.675 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-30T09:12:09.686 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-30T09:12:09.686 DEBUG:teuthology.orchestra.run.smithi032:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-30T09:12:09.692 DEBUG:teuthology.orchestra.run.smithi071:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-30T09:12:09.694 DEBUG:teuthology.orchestra.run.smithi189:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-30T09:12:09.709 INFO:teuthology.orchestra.run.smithi071.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 30 09:12 /home/ubuntu/cephtest 2024-04-30T09:12:09.711 INFO:teuthology.orchestra.run.smithi189.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 30 09:12 /home/ubuntu/cephtest 2024-04-30T09:12:09.742 INFO:teuthology.orchestra.run.smithi032.stdout: 393940 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 30 09:12 /home/ubuntu/cephtest 2024-04-30T09:12:09.746 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-30T09:12:09.765 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-30T09:12:09.790 INFO:teuthology.nuke:Checking targets against current locks 2024-04-30T09:12:09.816 DEBUG:teuthology.nuke:shortname: smithi032 2024-04-30T09:12:09.817 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-30T09:12:09.841 DEBUG:teuthology.nuke:shortname: smithi071 2024-04-30T09:12:09.841 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-30T09:12:09.846 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi032.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-30_05:04:19-fs-wip-pdonnell-testing-20240429.210911-debug-distro-default-smithi/7680601', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-30 08:03:51.713627', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEU3Y/7URMK0Jo+Za/Cngq+21csc7ZJUzb/JDPnzUl01zYOQiFmUXAflXU88IPq7tql29/6zYUAEKA4YzeYGahY='} 2024-04-30T09:12:09.864 DEBUG:teuthology.nuke:shortname: smithi189 2024-04-30T09:12:09.864 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-30T09:12:09.872 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi071.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-30_05:04:19-fs-wip-pdonnell-testing-20240429.210911-debug-distro-default-smithi/7680601', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-30 08:03:51.712458', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCwOv6xI7VHUgInYJlYquG4or26SYiH/PW8rDSDC7Cm9qaMTgr+P8l1M3oepRFSVIBhWr07qy/paGUqt1CWv3UQ='} 2024-04-30T09:12:09.904 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi189.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-04-30_05:04:19-fs-wip-pdonnell-testing-20240429.210911-debug-distro-default-smithi/7680601', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-30 08:03:51.710685', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCnlkcjxTxuYmuUCdpJxn4UaVQC4NPv7vfYg3VsGfGtOr72bwi2gmaUSpofdjWuo0ynIGlYWKVhoNs+6brFD0Os='} 2024-04-30T09:12:09.914 INFO:teuthology.orchestra.console.smithi032:Power off 2024-04-30T09:12:09.914 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-30T09:12:09.944 DEBUG:teuthology.orchestra.console.smithi032:power off output: Chassis Power Control: Down/Off 2024-04-30T09:12:09.945 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-30T09:12:09.963 INFO:teuthology.orchestra.console.smithi071:Power off 2024-04-30T09:12:09.963 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-30T09:12:09.977 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is on 2024-04-30T09:12:09.988 DEBUG:teuthology.orchestra.console.smithi071:power off output: Chassis Power Control: Down/Off 2024-04-30T09:12:09.988 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-30T09:12:10.005 INFO:teuthology.orchestra.console.smithi189:Power off 2024-04-30T09:12:10.005 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-30T09:12:10.016 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is on 2024-04-30T09:12:10.030 DEBUG:teuthology.orchestra.console.smithi189:power off output: Chassis Power Control: Down/Off 2024-04-30T09:12:10.030 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-30T09:12:10.057 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is on 2024-04-30T09:12:13.977 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-30T09:12:14.017 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-30T09:12:14.058 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-30T09:12:14.089 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is on 2024-04-30T09:12:14.129 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is on 2024-04-30T09:12:14.168 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is on 2024-04-30T09:12:18.094 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-30T09:12:18.131 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-30T09:12:18.169 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-30T09:12:18.205 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is on 2024-04-30T09:12:18.241 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is on 2024-04-30T09:12:18.280 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is on 2024-04-30T09:12:22.210 DEBUG:teuthology.orchestra.console.smithi032:pexpect command: ipmitool -H smithi032.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-30T09:12:22.242 DEBUG:teuthology.orchestra.console.smithi071:pexpect command: ipmitool -H smithi071.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-30T09:12:22.281 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-30T09:12:22.321 DEBUG:teuthology.orchestra.console.smithi032:check power output: Chassis Power is off 2024-04-30T09:12:22.351 DEBUG:teuthology.orchestra.console.smithi071:check power output: Chassis Power is off 2024-04-30T09:12:22.392 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is off 2024-04-30T09:12:22.422 INFO:teuthology.orchestra.console.smithi032:Power off completed 2024-04-30T09:12:22.452 INFO:teuthology.orchestra.console.smithi071:Power off completed 2024-04-30T09:12:22.493 INFO:teuthology.orchestra.console.smithi189:Power off completed 2024-04-30T09:12:22.599 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 806ec36b21129f113ce356c007dd59694d340b37 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-comp-ec-root omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/5 replication/default} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/ffsb}} duration: 3563.335881471634 failure_reason: 'error during scrub thrashing: Command failed on smithi032 with status 124: ''sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls''' flavor: default owner: scheduled_pdonnell@teuthology status: fail success: false 2024-04-30T09:12:22.600 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-30T09:12:22.700 INFO:teuthology.run:FAIL